Lookupbykey vs fetchbykey - why can I fetch a key but not do a lookup

I am following Aithorization pattern as described and in place of passing the contractid as per the example, I want to use contractKeys.

template AccreditedTenantToken
  with
    agency: Party
    tenant: Party
    landlord: Party
  where
    signatory agency
    observer landlord
    key (agency, landlord, tenant) : (Party, Party, Party)
    maintainer key._1
    controller agency can
      WithdrawAccreditedTenantToken : ()
        do
          return ()

where lanlord relies on agency to issue these tokens.
in another place landlord wants to check if these tokens exist before finalizing a contract…I am using a foldlA

          foldlA (\a b -> do
                    lookupByKey @AccreditedTenantToken (agency, finalContract.landlord, b )
                    return (a && True)
            )True finalContract.tenants

in here if I use lookupbyKey I get Authorization error which is inline with documentation but if I use fetchBykey I either can fetch the contracts if they do exist or fail which is fine and inline with documentation.

ignore the return part there…
My question is if I fetch a contract why I cannot do a lookup?

          foldlA (\a b -> do
                    fetchByKey @AccreditedTenantToken (agency, finalContract.landlord, b )
                    return (a && True)
            )True finalContract.tenants

          -- gothrough the tenant list
          -- fetch tokens for each 
          -- if the token exist 
          res<-foldlA (\ a b -> do 
                      cid<- lookupByKey @AccreditedTenantToken (agency, finalContract.landlord, b )
                      case cid of
                        None -> return (a && False)
                        _-> return (a && True)) True finalContract.tenants
          assert (res == True )

image2474×2104 471 KB

lookupByKey and fetchByKey are authorized differently as described in the docs.

• fetchByKey is authorized like fetch so it needs to be authorized by at least one stakeholder.
• lookupByKey needs to be authorized by all maintainers of the contract you are trying to lookup.

The reason is that lookupByKey allows negative lookups, and validating any kind of lookup involves work. Imagine we didn’t have that restriction and both you and I were on a distributed ledger with a template

template T
  with
    sig : Party
  where
    signatory sig
    key = sig : Party
    maintainer = key

You could then hammer me with transactions containing lookupByKey @T bernhard without me ever having agreed to validate a key lookup for you.

Thanks that explains it well, may be we can add this explanation in the documentation as well.

bernhard:

allows negative lookups, and validating any kind of lookup involves work.

one more clarification… does this additional cost on lookup not exist on fetch or given its based on access check its much cheaper?

Yes, but fetch only allows for positive lookups so unless you manipulate your participant node to be dishonest (which I can detect), I only need to do that work if I signed a contract that says I’m OK with that.

@bernhard @Vivek_Srivastava - having lookupByKey behave differently to fetchByKey make it super confusing. I don’t think the benefit of avoiding spamming is worth the confusion that this causes. I’ve distilled this to a trivial example where I even make alice a signatory of the contract she is trying to lookup and fails to do so.

template Foo
  with
    admin : Party
    user : Party
    taker : Party
  where
    signatory admin, user
    key (admin, user) : (Party, Party)
    maintainer key._1

    controller taker can
      Foo_Take : ContractId Foo
        do create this with user = taker


lookupVsFetch = scenario do
  alice <- getParty "Alice"
  admin <- getParty "Admin"
  let foo = Foo with user = alice, taker = alice, ..

  -- create the foo
  fooCid <- admin `submit` create foo with user = admin

  -- alice becomes a signatory
  alice `submit` exercise fooCid Foo_Take

  -- alice can fetch it
  (fooCid, fooCdata) <- alice `submit` fetchByKey @Foo (admin, alice)
  assert $ fooCdata == foo

  -- she can even fetch if she has the contract id
  fooCdata2 <- alice `submit` fetch fooCid
  assert $ fooCdata2 == foo

  -- alice is a signatory of foo
  assert $ alice `elem` signatory foo

  -- but she can't look it up!
  -- does this make any sense?
  optFooCid <- alice `submitMustFail` lookupByKey @Foo (admin, alice)
  assert $ optFooCid == ()

Alice is indeed a signatory, but she is not the maintainer so as per @bernhard 's comment, the lookupByKey @Foo (admin, alice) will fail unless you switch her with maintainer key._2 - but I can see how this might cause confusion.

dliakakos:

having lookupByKey behave differently to fetchByKey make it super confusing . I don’t think the benefit of avoiding spamming is worth the confusion that this causes.

As much as I share your sentiment on this, I don’t know how to align the two, even ignoring the DoS angle.

One line of argument would be to say that positive and negative lookupByKey instances should have different authorization rules so that positive lookupByKey behaves like fetchByKey. That seems even worse and entirely pointless to me. lookupByKey then just falls back to fetchByKey in contexts where negative lookups are not properly authorized. The behaviour now becomes implicit from the context rather than being explicit through fetchByKey and lookupByKey.

Authorizing lookupByKey like fetchByKey is also not an option. fetchByKey needs the authority of at least one stakeholder (which then becomes the actor on the fetch node). In a negative lookupByKey, the stakeholders are unknown so this check can’t be performed.

One could then consider removing all authorization from negative lookupByKey. However, that would effectively make existence of keys public information. This is problematic as the existence of keys can convey valuable information (like two parties being in a relationship).

The only reasonable weakening I can imagine is to say we only require the authority of a single maintainer, not of all maintainers. I’m not sure that solves anything, though.

@bernhard (Or someone more knowledgeable than me ) can you please precisely define what you mean by “positive” and “negative” here? I’ve seen it in several parts of the documentation but nothing definitive.

My understanding by context is that a positive lookup is one for a contract that does exist and a negative is for one that does not. I’m asking because based on my understanding there would be some ex-post interpretation of the result that I find confusing. Ie.

bernhard:

In a negative lookupByKey , the stakeholders are unknown so this check can’t be performed

Contract doesn’t exist, lookupByKey should be None, so why would be care about checking against the stakeholders?

Positive means lookupByKey returns Some cid during interpretation. Negative means lookupByKey returns None.

At the transaction level, the None becomes a NoSuchKey node in the transaction tree. A positive one becomes a simple Fetch with the maintainers as actors.

As a simple example say we implement a lock on a choice as a separate contract. Note that the below fails due to the authorization checks under discussion here.

template Lock
  with
    locker : Party
    obs : [Party]
  where
    signatory locker
    key locker : Party
    maintainer key

template Lockable
  with
    sig : Party
    con : Party
    locker : Party
  where
    signatory sig

    controller con can
      Guarded : ()
        do
          oLock <- lookupByKey @Lock locker
          case oLock of
            Some lock -> abort "Choice locked"
            None -> return ()

s : Scenario () = do
  [locker, sig, con] <- mapA getParty ["locker", "sig", "con"]
  
  lock <- submit locker do
    create Lock with
      locker
      obs = []
    
  locked <- submit sig do
    create Lockable with ..
  
  submitMustFail con do
    exercise locked Guarded

  submit locker do
    archive lock

  submit con do
    exercise locked Guarded

Neither sig nor con are stakeholders on the Lock, nor is locker a stakeholder on Lockable. When con submits the exercise locked Guarded, they fill in the NoSuchKey or Fetch nodes. Assuming they are malicious, they can submit the transaction with either. So unless sig can validate that con submits the right thing, the lock is worthless.

But asking locker is not an option. locker has given neither con, nor obs the authority to know about the existence of Lock.

So what if we give them that authority by adding them to the obs field (as in obs = [sig, con]? Now sig could validate the first submission attempt as they know Lock exists. But how about the second one? It may be that locker has created a new Lock with obs = [con], and con is maliciously submitting NoSuchKey. Again, the only way to be sure for sig is to ask locker, but locker hasn’t agreed to share any information about the existence of Lock.

So in short: to validate Guarded in the case where the lookup results in a NoSuchKey, locker needs to validate this. But doing so reveals information about the existence of Lock so authorization is needed. In other words: locker needs to authorize Guarded eg by delegating lookups.

template Lock
  with
    locker : Party
    obs : [Party]
  where
    signatory locker
    key locker : Party
    maintainer key

template LockDelegation
  with
    locker : Party
    con : Party
  where
    signatory locker
    key locker : Party
    maintainer key

    controller con can
      nonconsuming CheckLock : Optional (ContractId Lock)
        do
          lookupByKey @Lock locker

template Lockable
  with
    sig : Party
    con : Party
    locker : Party
  where
    signatory sig

    controller con can
      Guarded : ()
        do
          oLock <- exerciseByKey @LockDelegation locker CheckLock
          case oLock of
            Some lock -> abort "Choice locked"
            None -> return ()

s : Scenario () = do
  [locker, sig, con] <- mapA getParty ["locker", "sig", "con"]
  
  lock <- submit locker do
    create Lock with
      locker
      obs = [con]

  submit locker do
    create LockDelegation with ..
    
  locked <- submit sig do
    create Lockable with ..
  
  submitMustFail con do
    exercise locked Guarded

  submit locker do
    archive lock

  submit con do
    exercise locked Guarded

In the above, locker has agreed to validate the node resulting from lookupByKey so sig will have a guarantee that con can’t submit an incorrect result without being caught during validation.

I agree with @Leonid_Rozenberg that the difference between fetchByKey and lookupByKey is subtle and a source of confusion when learning DAML. Also I agree with @bernhard that the semantics and authority requirement of both updates are correct. But I do think a major source of confusion is that these functions are named and look almost the same, even their functionality is very similar - up to authorization. So here is a suggestion we might consider: We entangle the existence check from actually fetching the contract by

• leaving fetchByKey as is
• dropping lookupByKey and introduce doesContractExist: k -> Update Bool (or a better name)

This would make the distinction in functionality very clear and also why it needs different authority rules. We can reintroduce lookupByKey as a stdlib function:

lookupByKey k = do
  contractExists <- doesContractExist k
  if contractExists
    then 
      Some <$> fetchByKey k
    else 
      return None

Again, this makes it very clear why lookupByKey needs more authorization than fetchByKey. Also, this seems closer to what is actually written in the transaction.

If you think it helps, we could also implement

doesContractExist = fmap isSome lookupByKey

and favour that in the docs over lookupByKey. I don’t think there’s an important difference in performance, nor does it matter which function is native vs derived.

Yes, that’d be the other option. Would be interesting to know if this is more clear and intuitive to (new) users.

@bernhard Thank you for the detailed answer.

In the first scenario that you describe the server responds back with

Scenario execution failed on commit at Lock:52:3:
  1: lookup by key of Lock:Lock at DA.Internal.Prelude:381:26
     failed due to a missing authorization from 'locker'

which is odd since that contract should have been archived in the previous submit by locker. But I think this issue is beside your point? I will ignore it.

I am confused by

bernhard:

When con submits the exercise locked Guarded , they fill in the NoSuchKey or Fetch nodes. Assuming they are malicious, they can submit the transaction with either. So unless sig can validate that con submits the right thing, the lock is worthless.

In the first clause, con is submitting an exercise and then you refer to they. Who is "they" ?

What I think that you are referring to is that when we perform an Update we need to translate that into a sequence of transactions that will be stored in the ledger (in order to maintain a consistent ledger). Each action against a key needs to be replaced with a NoSuchKey or Fetch node (or KeyLookup per this issue). Consequently, in order for both con and sig (the "they") to arrive at the same transaction the lookupByKey must resolve to the same node for both? (I think that I understood this part after I figured out the rest, but I want to clear it up for anyone else trying to follow along).

I think that you forgot to add observer = obs for Lock. Otherwise when I do set obs = [sig, con] the first submit by con fails with

Scenario execution failed on commit at Lock:46:3:
  Attempt to fetch, lookup or exercise a key associated with a contract not visible to the committer.
  Contract:  #0:0 (Lock:Lock)
  Key:  'locker'
  Committer: 'con'
  Stakeholders: 'locker'

Weirdly, after I do set the observer field on Lock and modify the abort to a benign debug I get the previous error (failed due to a missing authorization from 'locker') but this time on con’s first submit,

bernhard:

Again, the only way to be sure for sig is to ask locker , but locker hasn’t agreed to share any information about the existence of Lock .

Is this in the scenario where con and sig are observers on Lock ? If both of those parties are observers I fail to see why we wouldn’t want the lock to function? Or more precisely, why we wouldn’t want the semantics of DAML to allow the lock to work.

Again, thank you for helping to elucidate this material.

Leonid_Rozenberg:

which is odd since that contract should have been archived in the previous submit by locker . But I think this issue is beside your point? I will ignore it.

Authorization checks happen before contracts are checked for being active to prevent privacy leaks. The whole point is that con is not entitled to learn whether the Lock is active or not.

Leonid_Rozenberg:

In the first clause, con is submitting an exercise and then you refer to they . Who is "they" ?

I’m using they as the gender neutral pronoun. It refers to con.

Leonid_Rozenberg:

I think that you forgot to add observer = obs for Lock . Otherwise when I do set obs = [sig, con] the first submit by con fails with

I did and I didn’t. If you add them as observers, you could reasonably argue that sig could validate the positive lookup which is indeed true. If you switched to fetchByKey and added the observers, the first Guarded would work. But the negative lookup is still problematic. I went back and fourth over exactly how to structure the “hypothetical” first version. There’s really no point looking at the scenario error messages for it as the current implementation doesn’t allow it for a whole range of reasons.

Leonid_Rozenberg:

Weirdly, after I do set the observer field on Lock and modify the abort to a benign debug I get the previous error ( failed due to a missing authorization from 'locker' ) but this time on con 's first submit,

The first version is a hypothetical one. It’s explicitly prevented by the current authorization rules, which is what that error is telling you.

Leonid_Rozenberg:

Is this in the scenario where con and sig are observers on Lock ? If both of those parties are observers I fail to see why we wouldn’t want the lock to function? Or more precisely, why we wouldn’t want the semantics of DAML to allow the lock to work.

As I said above, if both are observers, you could argue that the positive lookup should work. But in the negative case there are no observers so you can’t make it work. Having rules that allow for positive, but not negative lookups is pointless. You may as well use fetchByKey in that case. This is what I was saying further up in the thread:

bernhard:

One line of argument would be to say that positive and negative lookupByKey instances should have different authorization rules so that positive lookupByKey behaves like fetchByKey . That seems even worse and entirely pointless to me. lookupByKey then just falls back to fetchByKey in contexts where negative lookups are not properly authorized. The behaviour now becomes implicit from the context rather than being explicit through fetchByKey and lookupByKey .

I find this sentence and the one that follows (quoted later) confusing,

bernhard:

But in the negative case there are no observers so you can’t make it work.

This might be the crux of why I don’t understand your explanation. If

bernhard:

Negative means lookupByKey returns None .

and we’re in the negative case then what do the observers have to do with the return of Guarded? From the docs of lookupByKey I can reason that the Lock contract does not exist. If you meant to write that we’re in the negative case because there are no observers (stakeholders on the Lock to authorize), then why can’t we interpret that as “working”? Maybe our interpretation of “work” is different.

bernhard:

Having rules that allow for positive, but not negative lookups is pointless.

Don’t we want the authorization rules to determine whether lookupByKey is positive or negative, but in many of your examples, you assume a resulting state and talk about different authorization rules for those cases.

bernhard:

One line of argument would be to say that positive and negative lookupByKey instances should have different authorization rules…

Isn’t that backwards? Or what do the authorization rules do after we’ve executed a lookupByKey and have a result?

Let me try a different line of inquiry. When lookupByKey was added to DAML, why did it need to have different authorization from fetchByKey ?

Leonid_Rozenberg:

Let me try a different line of inquiry. When lookupByKey was added to DAML, why did it need to have different authorization from fetchByKey ?

I’m not sure I understand all (or any) of this, but let me try. This is all speculation, but it sort of makes sense to me (now).

For both, you have three cases to consider:

1. There is an active contract and you are allowed to see it.
2. There is no active contract.
3. There is an active contract and you are not allowed to see it.

At first glance, if you only think at the DAML level, there is no reason for them to be different. In the happy case (1) they return essentially the same thing; in the unhappy cases (2, 3), fetchByKey fails the transaction, whereas lookupByKey returns None. Both functions give you no way to know why things failed, which is great.

But DAML is not just a language, it’s also an architecture for distributed applications. From that perspective, every DAML choice is evaluated first in a “local” context (local to the submitting party) and then sent for confirmation to a “more global” context (validation by all the stakeholders; note that this is not (necessarily) a single global context).

Say Alice wants to exercise a choice, and Bob is a stakeholder on the contract. Bob will need to validate the transaction that Alice submits. The transaction says, simplifying a lot: “I have executed this choice on this contract, and the result is the following list (tree) of fetches, archives and creates”. Given the choice name and the contract, Bob has the exact code Alice should have run.

Let’s assume Alice is dishonest (because Bob has to).

How does authorization work for fetchByKey? Locally, Alice needs to create a transaction tree that contains the corresponding contract ID (in a Fetch node). fetchByKey explicitly only works if Alice is a stakeholder on the contract, so conceptually she (locally) scans all the contracts she knows about to find one with the correct type and key, and which is currently active.

If we’re in case 1, where the contract exists and Alice can see it, Alice knows about it because she is a stakeholder: if the contract had been archived, she would have been notified. There is of course a chance that contention would result in the transaction being rejected, but there is no opportunity for Alice to cheat here: either she can produce a transaction with the correct Fetch node, or she can’t. As a validator, you can ask Alice to prove that the given contract ID exists.

For cases 2 and 3, Alice does not have a contract ID, so she cannot produce a valid transaction to send. Further, fetchByKey is defined as only succeeding if the submitting party is a stakeholder on the corresponding contract, meaning that it would be trivial for any validating party to reject a fraudulent transaction Alice would try to create, either using obsolete IDs or invented ones.

Now, let’s think through how we should authorize lookupByKey. At first glance, the happy case stays the same: we validate it just like fetchByKey. If Alice is a stakeholder on the matching contract, she knows about it and can include the corresponding Fetch node.

For cases 2 and 3, however, things are less clear. Because lookupByKey does not fail the transaction when the contract cannot be found, we need to record, as part of the transaction, that the contract could not be found, so that other parties trying to validate the transaction can verify they get the same result.

So you’re Bob, and Alice sends you a transaction containing a NoSuchKey block. How do you validate that? As per the rules of fetchByKey, the only constraint here is that Alice be a stakeholder on the contract. There’s nothing about Bob, so it’s entirely possible that the contract exists and you don’t know about it (and Alice is lying to you). Now, based on the key (which you have), you know who the maintainers are; let’s say it’s Carol. You could ask her. But she may have nothing to do with the transaction you’re looking at. Not only do you not have authorization to pester Carol (possible DoS attack), you also should really not be telling Carol about this transaction you’re validating (possible information leak: Carol learning of the transaction), and she may think you really have no business knowing about the existence of that contract anyway (other possible information leak: what if the contract is between Carol and Alice, and Bob has nothing to do with it?).

So essentially, if we tried to keep the same permissions for lookupByKey as for fetchByKey, there would be no way to validate transactions in the negative case.

This is why fetchByKey and lookupByKey need different permissions for the negative case. From this point, we have a few options:

• Make all keys public at all time. This is tantamount to maintaining a global lookup map for the entire ledger, so it’s not great from a performance perspective in a distributed system (or making everyone queryable at all times, which is also not great from a DoS perspective). It’s also not a great option for privacy, as keys are likely to contain privileged information along the lines of who is dealing with whom.
• Have different permission requirements for positive lookupByKey (same as fetchByKey) and negative lookupByKey (as we evidently need more to validate those). This seems like a very messy model, as you need to know what the result of executing an action is before you know what permissions you need to run it.
• Change the permissions for the positive lookupByKey to match what we need for the negative lookupByKey. This means lookupByKey is more restrictive than fetchByKey even though they superficially look like they do the same thing (especially in the positive case), but that seems better than the other two options.

So then, assuming you agree with our choice of the third option, what should the permissions for lookupByKey be? In other words, what does the validator need to be able to validate a negative lookup? Well, they need to be able to check if the contract exists, and for that they have to be able to talk about that key with at least one of the maintainers. The only way to resolve both the possible information leaks and the unwanted DoS attack is for the maintainer to be part of the transaction.

I’m not sure why “all the maintainers” rather than just any one, but I suppose if we assume Alice is dishonest we may not want to let her choose which maintainer is going to validate the transaction, especially if that could be herself.

Even ignoring validation and dishonest participants, a model where lookupByKey has the same authorization rules fetchByKey currently has doesn’t make sense conceptually:

• fetchByKey needs to be authorized by a stakeholder.
• To calculate stakeholders, you need the contract. There is no way to determine this from the key alone.
• If there is no contract with a given key, there is no contract. Therefore you cannot calculate the stakeholders.
• If you don’t know what the stakeholders are, you cannot possibly check if you have authorization from them.

Since the maintainers have to be inferable from the key, the authorization rules we currently have don’t have this issue: We can calculate maintainers just from the key alone without needing the contract.

This doesn’t answer why we have the exact authorization rules we have and others in this thread have done a much better job at answering this than I can. But it does provide a very simple answer as to why giving lookupByKey the same authorization rules as fetchByKey is not possible and the problem is more complex than that.

You make it sound like the issue really is with the permissions of fetchByKey.

I suppose now there is “backwards compatibility”, but ignoring that one, is there a fundamental reason why fetchByKey does not have the same requirements as lookupBuyKey? It seems like “all the maintainers” as lexical scope on the call to lookupByKey makes more sense than the dynamic scope of “the current submitter” used in fetchByKey.

Also am I reading it right that if I am not a stakeholder on the contract I’m trying to get, but I do have the required permissions in the current scope, fetchByKey would fail whereas lookupByKey would actually return me a contract ID? I.e. lookupByKey is not “more” restrictive, it’s just different restrictive?

I don’t think the authorization of fetchByKey is wrong. It is exactly the same as fetch which I find very sensible. It is really lookupByKey that needs more restrictions since you have to differentiate authorization failures from negative key lookups. Note that the somewhat dynamic nature (not sure scope is the right term here) of authorization checks is unavoidable in DAML. You always need to first fetch the contract whenever you need signatories, observers or actors of a choice.

We did experiment with different authorization rules for fetchByKey where the submitter needed to be a maintainer. However, that broke a lot of sensible models so eventually we dropped that restriction (it never made it past DAML-LF 1.dev). I’m not too familiar with all the details but Loosen restriction that the submitter is in the maintainers for all contract keys operations · Issue #2311 · digital-asset/daml · GitHub and linked issues are probably a good starting point.

As for your question about lookupByKey, it really is more restrictive than fetchByKey:

• Maintainers need to be signatories so if you have authorization from all maintainers, you definitely have authorization from a stakeholder.
• That leaves you with the restriction that the submitter must be a stakeholder. This applies to both lookupByKey and fetchByKey (but admittedly the docs for lookupByKey could point this out more clearly).

What I meant was that I was expecting this would work:

template Secret
  with
    owner: Party
    name: Text
    secret: Text
  where
    signatory owner
    key (owner, name): (Party, Text)
    maintainer key._1

template AllowFetch
  with
    assetOwner: Party
    obs: Party
  where
    signatory assetOwner
    observer obs
    controller obs can
      nonconsuming FetchAsset: Secret
        with
          id: ContractId Secret
        do
          fetch @Secret id

testSecret = scenario do
  alice <- getParty "Alice"
  bob <- getParty "Bob"
  secret <- submit alice do
    create Secret with owner = alice, name = "label", secret = "my secret"
  allow <- submit alice do
    create AllowFetch with assetOwner = alice, obs = bob
  submit alice do
    fetch secret
  submitMustFail bob do -- this is the one I expected to succeed
    exercise allow FetchAsset with id = secret
  return ()

because at the point of fetching secret, I do have the explicit authorization of alice. I suppose I need to adjust my mental model, but I will note that looking for information on fetch through the search function on the docs site is surprisingly unhelpful.

@Gary_Verhaegen Thank you for your reply. I follow your argument up to the paragraph starting with

Gary_Verhaegen:

So you’re Bob, and Alice sends you a transaction containing a NoSuchKey block. How do you validate that?

if you’ll bear with me, hopefully you can explain where I’m confused.

Gary_Verhaegen:

As per the rules of fetchByKey , the only constraint here is that Alice be a stakeholder on the contract. There’s nothing about Bob, so it’s entirely possible that the contract exists and you don’t know about it (and Alice is lying to you).

One way to clear up this constraint, is for a NoSuchKey block to require that Alice delegate to Bob the ability to validate it. I’m purposefully using “delegate” to confer the same rights as what we would mean in a delegation contract. This doesn’t necessarily have to be explicit, but implicit in the semantics of DAML. If Alice is malicious this would mean that Bob would learn that. And if she is honest, she wants her transaction validated, so whatever reason led her to using lookupByKey as opposed to fetchByKey should merit that right.

Tangentially, I think that this implicit stakeholders right-of-validation is a desired feature of DAML. I assumed that it was everywhere in the language, so confusion around this is what has been driving my questions.

Gary_Verhaegen:

Now, based on the key (which you have), you know who the maintainers are; let’s say it’s Carol. You could ask her. But she may have nothing to do with the transaction you’re looking at. Not only do you not have authorization to pester Carol (possible DoS attack), you also should really not be telling Carol about this transaction you’re validating (possible information leak: Carol learning of the transaction), and she may think you really have no business knowing about the existence of that contract anyway (other possible information leak: what if the contract is between Carol and Alice, and Bob has nothing to do with it?).

Here you raise many fine points. I think some are overstated (DoS) while others don’t really make sense from the perspective of Alice wanting her transaction validated, Bob clearly has something to do with the transaction if he’s validating it, so Carol should acknowledge that as a consequence of granting Alice authorization on the keyed contract in the first place. But at root, what I find confusing, is this piece of documentation that you recently committed (the paragraph just above here):

For the negative case, however, the transaction submitted for execution cannot say which contract it has not found (as, by definition, it has not found it, and it may not even exist). Still, validators have to be able to reproduce the result of not finding the contract, and therefore they need to be able to look for it, which means having the authorization to ask the maintainers about it.

This seems to contradict your points with respect to Carol. I will accept the more recent documentation as authoritative.

Leonid_Rozenberg:

One way to clear up this constraint, is for a NoSuchKey block to require that Alice delegate to Bob the ability to validate it. I’m purposefully using “delegate” to confer the same rights as what we would mean in a delegation contract. This doesn’t necessarily have to be explicit, but implicit in the semantics of DAML.

You’re right that it doesn’t need to be explicit in principle, but it looks like whoever designed lookupByKey did make the choice of making this explicit. This is exactly what is currently happening: in order for the transaction to be valid, the lookupByKey call must have the authority to query the maintainers, i.e. the submitter is delegating to the validators the right to query the maintainers.

You may think implicit would have been better. I would disagree: Alice being malicious is not the only problematic case. Let’s say we want to make the choice that lookupByKey is special, and implicitly confers validators the right to check for key existence with the same rights as the submitter. What if the contract does exist but it is not visible to the submitter? We still want to refuse the transaction, but if the transaction is between Bob and Alice and the maintainer is Carol, nobody can tell either Alice or Bob that the contract does, in fact, exist. So we take the conservative approach of saying that, if the lookupByKey call does not explicitly have all the rights it would need to make sure that the contract does, in fact, not exist, then the transaction is invalid.

Leonid_Rozenberg:

This seems to contradict your points with respect to Carol. I will accept the more recent documentation as authoritative.

Would you mind clarifying where the contradiction lies? I’m not aware of any, but my understanding of this issue did evolve quite a bit over the past week or so so it’s definitely possible I don’t fully agree with my past self anymore.

@Gary_Verhaegen This is really great, I’m starting to understand much more technically about the argument and that our different interpretations stem from a difference of what I think lookupByKey should mean.

Gary_Verhaegen:

This is exactly what is currently happening: in order for the transaction to be valid, the lookupByKey call must have the authority to query the maintainers, i.e. the submitter is delegating to the validators the right to query the maintainers.

Now I understand Bernhard’s point about the LockDelegation. This makes explicit the differences between what we want; I’m arguing for an implicit delegation whereas y’all are for an explicit one.

Gary_Verhaegen:

We still want to refuse the transaction, but if the transaction is between Bob and Alice and the maintainer is Carol, nobody can tell either Alice or Bob that the contract does, in fact, exist. So we take the conservative approach of saying that, if the lookupByKey call does not explicitly have all the rights it would need to make sure that the contract does, in fact, not exist, then the transaction is invalid.

My response in this case is why would we want to refuse it? If it is not visible to both Alice and Bob, how does its existence change the business logic that they would want to write in a contract between them? They’re told that “these are not the drones you seek” and continue doing what they’re doing.

W.r.t the contradiction, I think that you are using

Gary_Verhaegen:

Not only do you not have authorization to pester Carol …

as an argument against for why validating a non existence lookup is difficult, but then you document that you do have to pester Carol. I think that I am being dense here as you may be using different contexts.

After reading through some of the back issues that Moritz linked to I realized that lookupByKey was a stand alone idea, and not a wrapper for a potentially failing fetchByKey. So I should stop shoehorning it into that frame. I find the DoS arguments confusing, because couldn’t one fetchByKey out-of-band to perform this attack too, or to check for the existence of keys?

I still find the current abstraction awkward. I think that having implicit delegation for validation so that the authorization for the two functions is the same is a worthwhile trade-off, but I think that I’m in the minority on this opinion so I’ll stop.

Leonid_Rozenberg:

our different interpretations stem from a difference of what I think lookupByKey should mean.

To clarify: I am not trying to argue for anything here, and I am not presenting what I would want, I’m merely trying to explain how I think things currently work and what a reasoning for them working the way they do could be. In the following (and most of the above), when I say “we”, I mean the company / daml language design team; I was not involved in any of this until last week.

Leonid_Rozenberg:

My response in this case is why would we want to refuse it? If it is not visible to both Alice and Bob, how does its existence change the business logic that they would want to write in a contract between them?

For better or worse, we have defined contract keys as globally unique over the entire ledger. Ideally, even when that ledger is a world-spanning, distributed, aggregated network of DAML ledgers. You may disagree with that, but I’m afraid that ship has sailed.

So that’s constraint 1. Constraint 2 is that we also do not want keys to be discoverable without the proper authorization, because the mere existence of a key could leak information about the corresponding contract. So we have made the decision that the existence of a key is a private thing that requires authorization to discover.

Alice may know that the contract does indeed exist; similarly, Alice may not know, but perhaps Bob does and Bob is the dishonest one. Either of them could learn of that contract afterwards. The ledger as a whole is not consistent with constraint 1 if we do not block that transaction, and being able to block that transaction without violating constraint 2 gets us back to needing the authorization of the maintainers.

Note that, as those things are currently defined, the maintainers of the contract could be observers, meaning you could have a transaction between Alice and Bob that checks for the existence of a contract (which they can’t see), and then, if it does not exist, attempts to create it (with Carol as observer/maintainer). What then?

I’m not claiming the current design is great UX, but every time I try to think of a different one, I hit a wall like that.

Leonid_Rozenberg:

W.r.t the contradiction, I think that you are using

Gary_Verhaegen:

Not only do you not have authorization to pester Carol …

as an argument against for why validating a non existence lookup is difficult, but then you document that you do have to pester Carol.

If Carol is the maintainer, asking her is the only way to know. So, yes, we do have to ask her, and therefore we need her authority before we can attempt the lookupByKey call.

If we were to allow lookupByKey without Carol’s authorization, we’d be stuck, because we would have to ask her and we couldn’t. Call this “Constraint 3”: we want DAML to be able to run truly distributed ledgers where different nodes are managed by different organizations, and you cannot make requests against a node without that node having given you consent first, through the mechanism of DAML signatories. Having her authority does not mean we don’t have to ask her; we still have to, but at least now we can (under constraint 3).

Leonid_Rozenberg:

I find the DoS arguments confusing, because couldn’t one fetchByKey out-of-band to perform this attack too, or to check for the existence of keys?

The way it is currently defined, fetchByKey actually resolves the key to a contract ID locally (this is possible because the submitting party must be a stakeholder), so it does not query anyone and cannot be used for DoS or discovery.

Leonid_Rozenberg:

I still find the current abstraction awkward.

I agree with that sentiment, though I do hope that the switch of emphasis to visibleByKey helps a little bit with that.

Leonid_Rozenberg:

I think that having implicit delegation for validation so that the authorization for the two functions is the same is a worthwhile trade-off

I hope I’ve been able to at least convince you that this would not be compatible with three constraints we have set for ourselves. Whether those constraints are worth their cost is a separate discussion, I suppose, but not one I am well equipped to have.

Gary_Verhaegen:

the maintainers of the contract could be observers

which contract are you referring to here? If you mean the keyed one are you sure? The docs say that maintainers must be signatories. Or are you referring to maintainers being observers of the contract between Alice and Bob? In which case they’re not under obligation to help Alice and Bob.

Gary_Verhaegen:

…if it does not exist, attempts to create it (with Carol as observer/maintainer). What then?

“it” is the keyed contract? But then Alice and Bob couldn’t create it without Carol’s authorization. If they had Carol’s authorization for the query, they would be able to see the original contract.

Gary_Verhaegen:

I hope I’ve been able to at least convince you that this would not be compatible with three constraints we have set for ourselves.

I’m sorry to disappoint you but you have haven’t. I support your effort but I’d like to see a more full fledged example of the negative consequences of what I’m suggesting.

On a previous project I know that I ran into the fetchByKey vs lookupByKey distinction, but because of time pressure, I worked around it. I will try to resurrect that work into a meaningful example.

I have to say that I’m completely lost as to where the confusion is at this point. Let me try a completely fresh explanation.

There are two places where fetches and lookups can succeed or fail:

During interpretation, the Participant node of the requester (aka submitting party) looks inside their index database to see whether the requester knows of a contract with that key or not. Since we have to assume the Participant is malicious, any fetch or lookup can return just about anything in theory.
In practice, honest Participants currently succeed on fetch if the requestor is a witness on a contract, and succeeds on fetchByKey and lookupByKey if the requestor is a stakeholder in a contract with the given key.

Why the distinction, you ask? Because by witnessing creates, but not archives, you could be a witness on multiple contracts with the same key, and the events creating those keys can’t necessarily be ordered so there is no good way to decide what a key lookup should do if multiple key instances are witnessed.

During validation the result of that key or contractId resolution at interpretation time is validated by the signatories or maintainers. To see why authorization matters, let’s say there’s a template

template WeMeetTonightAt7UnderTheLimeTreeOnCentralPlaza
  with
    sigs : [Party]
    obs : [Party]
  where
    signatory sigs
    observer obs
    key sigs : [Party]
    maintainer key

The mere existence of a contract instance confers information, so I, bernhard, am interested to know whether you, leonid and gary have signed such a contract instance. I have a severe case of fomo.

Now let’s say I send out a transaction that is a single NoSuchKey node for the key [leonid, gary] on that template. If the two of you respond with “CONTENTION: Actually, there is such a key”, or “Yep, fine, go ahead and commit”, I got what I was looking for. I know whether to crash your party or not.

So you need an authorization rule that allows you to say “Na’uh! You are not invited, we are not telling you”.

The only information avaialble on the NoSuchKey node are the key and template. The only party-related information you can pull out of those is the set of maintainers. So the authorization rule cannot be based on stakeholders. You don’t know who the stakeholders of a non-existing contract are.

Ie The authorization rule cannot be the same as for fetch or fetchByKey.

But there’s still some wiggle room: Do you require authorization from any maintainer, or from all maintainers? The rationale for this is more subtle and based around DoS considerations. I’d stop reading here if all you care about is the difference between fetchByKey and lookupByKey.

Suppose we just need one maintainer. A lookupByKey with key [bernhard, leonid, gary] would now be well-authorized, as I submitted it. I could now submit an endless stream of negative lookups of that key. All of them are correct, all of them are well authorized, and each time the two of you have to go to your index database and check even though you have never agreed to anything that may possibly, maybe have led to an agreement to meet me.

bernhard:

template WeMeetTonightAt7UnderTheLimeTreeOnCentralPlaza

Excellent spycraft

Although I still have small questions about specific examples that are unanswered, I am not confused about why things are. But I am confused about why things cannot be different. For example,

bernhard:

Now let’s say I send out a transaction that is a single NoSuchKey node for the key [leonid, gary] on that template. If the two of you respond with “CONTENTION: Actually, there is such a key”, or “Yep, fine, go ahead and commit”, I got what I was looking for. I know whether to crash your party or not.

What if [leonid, gary] respond with “No” (None) in the cases where bernhard is not authorized and when no such contract exists? What will bernhard do then? bernhard can’t force [leonid, gary] to be maintainers on this contract. Maybe bernhard could use that information to ask to meetup with [martin, moritz] instead?

bernhard:

The only information avaialble on the NoSuchKey node are the key and template.

Is that the only way that things can be? Could the NoSuchKey also contain the party who makes this claim (bernhard), so that to validate this transaction we have to take into account who is making this claim. To leonid this transaction makes sense (I don’t want you to know about our get together) and it also makes sense to martin who has the same information as bernhard.

bernhard:

Suppose we just need one maintainer. A lookupByKey with key [bernhard, leonid, gary] would now be well-authorized,

Would this situation also be fixed if we stipulate any maintainer as long as it isn’t the one in the new NoSuchKey that I propose ?

Leonid_Rozenberg:

What if [leonid, gary] respond with “No” ( None ) in the cases where bernhard is not authorized and when no such contract exists?

If you response depends on the existence of the contract, I have learnt the information I was looking for. The response leonid and gary give must not reveal whether the key exists or not.

That should also answer your second question. The premise of my example is that bernhard must not be able to learn whether leonid and gary are meeting up or not. Don’t get hung up on the whole meeting thing, it’s just an example. The important thing is that bernhard is not entitled to find out whether gary and leonid have an active contract with a given key.

Leonid_Rozenberg:

Could the NoSuchKey also contain the party who makes this claim ( bernhard ),

That would let the transaction submitter leak into the Ledger Model beyond top-level authorization, which harms composability. We try to guarantee that if bernhard has the right to perform an action a, bernhard can delegate that action to another party. If you let the submitter/requester leak into the auth model, that’s no longer the case.

But I also don’t fully understand what you have in mind. If you are saying the transaction NoSuchKey should just be accepted because bernhard doesn’t have the right to do the lookup, I’d infer that lookupByKey falls back to const None in any ill-authorized context, which is probably not what you had in mind.

Leonid_Rozenberg:

Would this situation also be fixed if we stipulate any maintainer as long as it isn’t the one in the new NoSuchKey that I propose ?

I don’t understand this question. The key always contains all maintainers so the NoSuchKey node contains all maintainers.

bernhard:

If you response depends on the existence of the contract, I have learnt the information I was looking for. The response leonid and gary give must not reveal whether the key exists or not.

If the response is None in two cases how do you learn which one it was?

Leonid_Rozenberg:

If the response is None in two cases how do you learn which one it was?

In that case I don’t, but if you mean to say that you’d like lookupByKey to return None in all cases where it’s ill-authorized, I don’t think that’s a good idea. It’s better for things to fail than to behave in unexpected ways without warning.

How is that unexpected? The specification of DAML gets to define what the caller of lookupByKey should expect. I think being able to fetch something that I can’t lookup is unexpected.

Could you make precise what your proposal is? I may be misunderstanding. Under which circumstances should lookupByKey return what value during interpretation (evaluation of the command on the submitting node), under which circumstances should the resulting transaction be accepted, and what should the error message be if it doesn’t get accepted?

Current Behaviour:

Interpretation

• Fails with authorization error if not authorized by all maintainers
• Returns None if there is no matching contract the submitting party is a stakeholder on
• Returns a Contract ID if there is a matching contract the submitting party is a stakeholder on

Validation

• If returned None during Interpretateion
  • Fails with authorization error if not authorized by all maintainers
  • Fails with contention if the key does actually exist
  • Succeeds if there is (still) no contract with a matching key
• If returned a ContractId during interpretation
  • Fails with authorization error if not authorized by all maintainers
  • Fails with contention if the ContractId doesn’t exist (anymore), or the referenced contract doesn’t match
  • Succeeds if the ContractId is still active and the key matches

Thank you for considering, and prompting me to write down this proposal. It isn’t necessarily something that I had clearly in my mind when I started asking questions; I was (still am) curious about the choices and trade-offs that have been made to arrive at the current abstraction. What do you think of:

Interpretation

Does contract of the requested type and given key exist ?

• No -> None
• Yes -> Is the submitter authorized by at least one stakeholder ?
  • No -> None
  • Yes -> Some (ContractId _)

Validation

If None

• Succeeds

If Some (ContractId _)

• Fails with authorization error if not authorized by at least one stakeholder
• Fails with contention if the ContractId doesn’t exist (anymore), or the referenced contract doesn’t match
• Succeeds if the ContractId is still active and the key matches

As I promised Gary, here is an example of where I want the power of lookupByKey to match fetchByKey:

template OpeningBid 
  with
    bidder : Party
    seller : Party
    bid : Decimal
    kId : Int
    b_workers : [Party]
  where
    signatory bidder 
    observer seller, b_workers

    key (bidder, kId) : (Party, Int)
    maintainer key._1

    controller seller can
      Accept : ContractId Negotiation
        with
          offer : Decimal
          s_workers : [Party]
        do create Negotiation with ..

template Negotiation
  with
    bidder : Party
    seller : Party
    bid : Decimal
    offer : Decimal
    kId : Int
    b_workers : [Party]
    s_workers : [Party]
  where
    signatory bidder, seller
    observer b_workers, s_workers

    key (bidder, seller, kId) : (Party, Party, Int)
    maintainer key._1

    choice Agree : ()
        with
          who : Party
      controller who
        do
          assert $ bid > offer
          return ()

    controller bidder can
      MakeBid : ContractId Negotiation
        with
          newBid : Decimal
        do
          assert $ newBid > bid
          create this with bid = newBid

    controller seller can
      MakeOffer : ContractId Negotiation
        with
          newOffer : Decimal
        do
          assert $ newOffer < offer
          create this with offer = newOffer

template BidDelegate
  with
    bidder : Party
    worker : Party
  where
    signatory bidder

    controller worker can
      nonconsuming WorkBidL : ContractId Negotiation
        with
          kId : Int
          seller : Party
          newBid : Decimal
        do
          nOpt <- lookupByKey @Negotiation (bidder, seller, kId) 
          case nOpt of
            None -> abort "missing"
            Some fId -> exercise fId MakeBid with ..

template OfferDelegate
  with
    seller : Party
    worker : Party
  where
    signatory seller

    controller worker can
      nonconsuming WorkOfferL : ContractId Negotiation
        with
          kId : Int
          bidder : Party
          newOffer : Decimal
        do
          nOpt <- lookupByKey @Negotiation (bidder, seller, kId) 
          case nOpt of
            None -> abort "missing"
            Some fId -> exercise fId MakeOffer with ..

t1 = scenario do

  [b, s, b1, s1] <- mapA getParty ["b", "s", "b1", "s1"]
  ob <- b `submit` do
          create OpeningBid with
            bidder = b
            seller = s
            bid = 10.0
            kId = 1
            b_workers = [b1]

  n <- s `submit` do
          exercise ob Accept with
            offer = 20.0
            s_workers = [s1]

  bd <- b `submit` do
          create BidDelegate with
            bidder = b
            worker = b1

  n <- b1 `submit` do
        exercise bd WorkBidL with
          kId = 1
          seller = s
          newBid = 15.0

  sd <- s `submit` do
          create OfferDelegate with
            seller = s
            worker = s1

 -- I want this last submit to work.
  n <- s1 `submit` do
        exercise sd WorkOfferL with
          kId = 1
          bidder = b
          newOffer = 14.0
 
  pure ()

In this example, we describe a simple 2 sided market/negotiation, but with delegation. A couple of things combine to create the failure:

• Delegation. This is fundamental to this example, but not necessarily to the original difference in authorizations.
• Using lookupByKey as opposed to fetchByKey/ exerciseByKey. One can write the WorkOffer methods using the other Update's, and the scenario will succeed. But what I really want to write is this method:

      nonconsuming WorkBid2L : Either Text (ContractId Negotiation)
        with
          kid_fst_choice : Int
          kid_snd_choice : Int 
          seller : Party
          by : Decimal
        do
          nOpt1 <- lookupByKey @Negotiation (bidder, seller, kid_fst_choice) 
          case nOpt1 of
            Some fId1 -> do 
              f1 <- fetch fId1
              Right <$> exercise fId1 MakeBid with newBid = f1.bid * by
            None -> do
              nOpt2 <- lookupByKey @Negotiation (bidder, seller, kid_snd_choice) 
              case nOpt2 of
                Some fId2 -> do
                  f2 <- fetch fId2 
                  Right <$> exercise fId2 MakeBid with newBid = f2.bid * by
                None -> do return $ Left "Nothing"

I want to be able to write choices that have non-aborting functionality. This is pretty fundamental in finance where a lot of your exposure is hedged (though this is not that kind of example ) and making more than one transaction (trade) in a transaction (atomic DB event) is key (pun intended).

For fun consider changing the key maintainers on Negotiation to maintainer [key._1, key._2].

Leonid_Rozenberg:

Does contract of the requested type and given key exist ?

You don’t have this knowledge available. Data in DAML Ledgers is distributed. You don’t know whether a key exists, you only know whether you have local knowledge of it. The only people that are guarantee to have knowledge of key existence are its maintainers.

But let’s assume you keep this the same as current fetchByKey:

bernhard:

• Returns None if there is no matching contract the submitting party is a stakeholder on

Leonid_Rozenberg:

Validation

If None

• Succeeds

In this case, transaction submitters have a choice. Wherever there is a lookupByKey, I could choose to insert a None. This changes the nature of the function fundamentally. Have another look at my Lock Example above. It would no longer work.

Un- or partially checked queries like you propose are something that’s been discussed before, but there is currently no such thing in DAML, and that’s not what lookupByKey is. We currently have the following principle:

• Given a an action (ie create, exercise, etc), and a ledger state, there is at most one valid transaction resulting from that action.

This would no longer hold under your proposal because no matter the state of a key, you can always submit None. Ie the None case is unchecked.

Leonid_Rozenberg:

But what I really want to write is this method

Could you explain what the idea here is?

worker here is an observer (ie stakeholder) of the negotiations as part of b_workers or s_workers. Ie they know which negotiations exist. Why would they now pass in a kid_fst_choice that doesn’t exist? Ie in what case would you expect this to fall through to kid_snd_choice?

If you are trying to circumvent contention, that won’t work. Ie if you are saying “take the first choice if it’s still available, and otherwise go for my second choice”.

Keys are resolved on the submitters participant node during interpretation. Contention happens during validation. For example:

• s1 looks what negotiations are active and sees two: kid_fst_choice and kid_snd_choice. They call WorkOfferL with those.
• s1’s Participant interprets the command. Since kid_fst_choice was just queried moments before, it’s probably still there. The first lookupByKey resolves to a contract Id.
• Another worker elsewhere does the same simultaneously. Their offer/bid,accept gets there first.
• s1’s transaction will now fail due to contention.

The above is independent of authorization rules or whether we always accept None. Contract keys do not remove contention.

If you want to make this bidding model work, you need to remove contention from the Negotiation contract by collecting bids and offers in side contracts. The below version is contention free on a single negotiation. The only contention is between making bids/offers and a negotiation being agreed.

module Market where

import DA.Optional
import DA.Foldable

data NegotiationId = NegotiationId with
  bidder : Party
  seller : Party
  n : Int
    deriving (Eq, Show)


template OpeningBid 
  with
    nId : NegotiationId
    bid : Decimal
    b_workers : [Party]
  where
    signatory nId.bidder
    observer nId.seller, b_workers

    key nId : NegotiationId
    maintainer key.bidder

    controller nId.seller can
      Accept : ContractId Negotiation
        with
          offer : Decimal
          s_workers : [Party]
        do
          create Bid with worker = nId.bidder; bid = Some bid; ..
          forA b_workers (\w -> create Bid with worker = w; bid = None; ..)
          create Offer with worker = nId.seller; offer = Some offer; ..
          forA s_workers (\w -> create Offer with worker = w; offer = None; ..)
          create Negotiation with ..

template Bid
  with
    bid : Optional Decimal
    nId : NegotiationId
    b_workers : [Party]
    s_workers : [Party]
    worker : Party
  where
    signatory nId.bidder
    observer nId.seller, b_workers, s_workers
    key (nId, worker) : (NegotiationId, Party)
    maintainer key._1.bidder

    controller worker can
      ChangeBid : ContractId Bid
        with newBid : Decimal
        do create this with bid = Some newBid

template Offer
  with
    offer : Optional Decimal
    nId : NegotiationId
    b_workers : [Party]
    s_workers : [Party]
    worker : Party
  where
    signatory nId.seller
    observer nId.bidder, b_workers, s_workers
    key (nId, worker) : (NegotiationId, Party)
    maintainer key._1.seller

    controller worker can
      ChangeOffer : ContractId Offer
        with newOffer : Decimal
        do create this with offer = Some newOffer

template Negotiation
  with
    nId : NegotiationId
    b_workers : [Party]
    s_workers : [Party]
  where
    signatory nId.bidder, nId.seller
    observer b_workers, s_workers

    key nId : NegotiationId
    maintainer [key.bidder, key.seller]

    let
      getBid = do
        let keys = map (\w -> (nId, w)) (nId.bidder::b_workers)
        bids <- mapA (fetchByKey @Bid) keys
        return (maximum (mapOptional (\(_, b) -> b.bid) bids))
      getOffer = do
        let keys = map (\w -> (nId, w)) (nId.seller::s_workers)
        offers <- mapA (fetchByKey @Offer) keys
        return (minimum (mapOptional (\(_, o) -> o.offer) offers))
       
    choice Agree : ()
        with
          who : Party
      controller who
        do
          bid <- getBid
          offer <- getOffer
          assert $ bid >= offer
          forA (nId.bidder::b_workers) (\w -> exerciseByKey @Bid (nId, w) Archive)
          forA (nId.seller::s_workers) (\w -> exerciseByKey @Offer (nId, w) Archive)
          return ()

t1 = scenario do

  [b, s, b1, s1, b2] <- mapA getParty ["b", "s", "b1", "s1", "b2"]
  let nId = NegotiationId with
        bidder = b
        seller = s
        n = 1

  b `submit` do
    create OpeningBid with
      bid = 10.0
      b_workers = [b1, b2]
      nId

  s `submit` do
    exerciseByKey @OpeningBid nId Accept with
      offer = 20.0
      s_workers = [s1]

  b1 `submit` do
    exerciseByKey @Bid (nId, b1) ChangeBid with newBid = 13.0

  s1 `submit` do
    exerciseByKey @Offer (nId, s1) ChangeOffer with newOffer = 14.0

  submitMustFail b do
    exerciseByKey @Negotiation nId Agree with who = b

  b2 `submit` do
    exerciseByKey @Bid (nId, b2) ChangeBid with newBid = 14.0

  submit b do
    exerciseByKey @Negotiation nId Agree with who = b 
  
  pure ()

@Leonid_Rozenberg I’m not sure I fully understand what you’re getting at, so let me try to rephrase it.

I believe the disagreement/misunderstanding here is around the following two ideas: the meaning of NoSuchKey and how to interact with shared state. There are roughly two ways of (safely) interacting with shared state: optimistic updates (with a risk of rejection if state has changed in-between read and (attempted) write) and critical sections (with no risk of rejection but some need for queueing). Similarly, there are two possible meanings for NoSuchKey: no contract with that key exists globally, or no contract with that key was visible to the submitter at submission time.

It seems to me you are asking why we chose optimistic updates with globally-meaningful NoSuchKey nodes rather than critical sections with observer-dependent NoSuchKey nodes. Am I getting this right?