Rules/limitations for deriving serializability

I have the following template & types:

data Obligation f a = ... -- a recursive ADT. Has some Bools and Decimals and manual Eq/Show instances.

-- Identity monad
newtype Id a = Id a deriving (Eq, Show)

-- Specialized effect-less obligation for DvP
type Obligations = Obligation Id (Either Ccy Security)

-- Asset types
data Ccy = GBP | USD deriving  (Eq, Show)

data Security = Security with
    isin: Text
    claims: Obligations
  deriving (Eq, Show)

-- The financial contracts
template DeliveryVersusPayment 
  with
    bearer: Party
    counterparty: Party
    obligations: Obligations
  where
    signatory bearer, counterparty

And the compiler is complaining that the template is not serializable:

error type checking template DvPExamples.DeliveryVersusPayment :
expected serializable type:

• reason: template argument
• found: DvPExamples:DeliveryVersusPayment
• problem:
  unserializable data type DvPExamples:DeliveryVersusPayment

Which leads me to the question, how is serializability derived, and why does the compiler fail here? Could it be the recursion in the ADT?

Is it something I can prove to the compiler explicitly through a typeclass?

Is there some trick that can help me determine which type is not serializable here?

The issue is not recursion, it’s the fact that Obligation takes a higher-kinded type parameter. For a type to be serializable, all type parameters must be serializable even phantom type parameters. Id Int is serializable but Id which is the type parameter is not. The rules for serializability are documented in the LF specification.

I’m attempting to understand the hieroglyphics on the linked page , to find a workaround for this.

But from what you’re saying, it sounds like the only way to make this work is to remove f and avoid using higher-kinds. Is there any instance of f that is serializable? Such as the built-in ones like Update for instance?

I also tried using a type synonym originally Id a = a but your link notes that these aren’t serializable (and it didn’t even work - I got the idea from scala, where that definition is used).

Types with higher-kinded type parameters are never serializable. I’m not aware of any workaround for this, Update which you mention isn’t serializable either.

It would help if you showed your Obligation type. The workaround here will be to modify it such that you don’t need to pass in Id with a free type parameter.

Thanks Bernhard, that’s what I’m going to end up doing. The problem is that I still haven’t thought through what the f effect does precisely and I just wanted to abstract it away so I would have an initial implementation without it.

The exact rules are as @cocreature said. The “trick” is to ask yourself “is this defunctionalized, and is the one-to-one translation of this type to Java obvious and unambiguous?” Thus ruling out all HKTs.

So I’ve continued thinking about this … I’m curious whether this is a limitation of the compiler / by design, or whether it’s in fact something that’s impossible to implement.

As you said in your example, if I have a higher-kinded type parameter (e.g. Id : * -> *), and know what the concrete types in this case are (e.g. Id Int) at the point of instantiation, then is there something that prevents me from inferring that the template is serializable?

Apologies if this is something obvious !

It is a deliberate restriction, put in place so that ledger interactions from languages other than DAML can be sensibly typed.

The fact that you have a value for the type variable at all isn’t considered a relevant factor; the problem for serializability is that you have a type parameter of kind * -> * in the first place. In other words, the problem is not the value you are passing for f, it is f: * -> *'s existence itself.

We also forbid type parameters of kind Nat for the same reason; the occurrence within the Numeric constructor itself is treated as a special exception, precisely because having that exception does not cause the problems with ledger interactions that general Nat parameters would.