Daml 3 Contract Keys and lookupNByKey completeness?

What are some scenarios in which we know that lookupAllByKey will return all contracts for a given key?

For example, is this such a scenario:

• A single synchronizer (i.e., the contracts never cross “domains”) 1
• A single-hosted maintainer (i.e., the same participant node is used for all submissions and validations by the maintainer) 2

If so, can we push the single-synchronizer scenario further?

• What if the maintainer is multi-hosted, but always submits through the same participant node? 3
• What if the maintainer is multi-hosted, but might submit to any of its hosting nodes? 4

Related to: Canton 3 Contract Keys

Even a single synchronizer and a single-hosted maintainer submitting through its one participant node does not suffice here. That’s because lookupAllByKey will not return any contracts created by commands that are in flight concurrently to the submission. Say if the maintainer submits a command to create a contract c with a key k and while this command is making its way through the Canton protocol, the maintainer submits another command with lookupAllByKey. This second command will not find the contract c even though by the time the second command appears as an accepted transaction on the ledger API, c is active. You can in theory prevent such concurrency problems in this scenario by using the same command ID for deduplication on all commands that create/lookup a particular key, but that’s not particularly scalable.

Moreover, the maintainer should not grant any other party (hosted on another node) the right to create such contracts via a Daml delegation contracts because this would again open up such concurrency problems again.

Andreas_Lochbihler:

That’s because lookupAllByKey will not return any contracts created by commands that are in flight concurrently to the submission.

@Andreas_Lochbihler Could this be solved by using a consuming choice on a single factory contract that performs the lookup? This would prevent concurrent submissions.

I realize this still has the same scalability concerns but they’re not relevant for our use case.

Could this be solved by using a consuming choice on a single factory contract that performs the lookup? This would prevent concurrent submissions.

This very much depends on the concrete implementation of the approach and the particular deployment of Canton. Ledger API commands are converted into transactions not against a snapshot of the ledger state. For example, you could have the case where you first perform a lookupByKey or fetchByKey operation that gives you a list of resolved contracts and then later perform the lookupAllByKey through the factory contract, whose creating transaction just has been ingested into the DB. Then the interpreter may refer to the cached resolution without rechecking that another contract has only been created.

Essentially, such an approach would rely on implementation details in the indexer that are not officially specified and therefore could change in the future as part of how things are implemented there.

Interesting. To use a concrete example to ensure I understand:

template Item
  with
    a : Party
    b : Party
    id : Text
  where
    signatory a, b
    key (a, id) : (Party, Text)
    maintainer key._1

template ItemFactory
  with
    a : Party
    b : Party
  where
    signatory a, b

    choice ItemFactory_Create : (ContractId Item, ContractId ItemFactory)
      with id : Text
      controller a, b
      do
        existing <- lookupByKey @Item (a, id)
        assertMsg ("Item " <> id <> " already exists") (isNone existing)
        (,) <$> create Item with a, b, id
            <*> create this

template Service
  with
    a : Party
    b : Party
  where
    signatory a, b

    nonconsuming choice Service_CreateItem : (ContractId Item, ContractId ItemFactory)
      with
        factoryCid : ContractId ItemFactory
        id : Text
      controller b
      do exercise factoryCid ItemFactory_Create with id

b submits Service_CreateItem as the only command; nothing else in the transaction touches the key, and all Item creates go through ItemFactory_Create. Since the factory is fetched before lookupByKey runs, does the interpreter guarantee that the lookup sees any Item created by the transaction that produced this factory version? Or could the key lookup still be served from a stale cache or index, so two Items with the same key could end up being created?

@Andreas_Lochbihler
Given a single synchronizer and maintainer, what additional conditions are required for a lookupAllByKey to be exhaustive with regard to previously committed transactions (leaving the problem of in-flight submissions aside)?