Reassignment Risk with Positive Key Lookups

We think we’ve run into a multi-synchronizer issue that affects workflows relying on lookupByKey.

A contract can be reassigned independently of the Daml workflow operating on it. Once that happens, a workflow expecting to resolve that contract by key on its current synchronizer can break.

We wrote up our understanding below. Curious whether we’re misunderstanding anything, whether there is an existing pattern or mechanism we should be using, or whether it would make sense for reassignment policy to be expressible at the template level. For example, allowing a contract to opt out of reassignment or restrict the synchronizers to which it may be reassigned.

On Reassignment Risk

Reassigning a contract between synchronizers is different from an ordinary Daml state change. A participant hosting a stakeholder can initiate reassignment, provided it is a reassigning participant for that stakeholder, and observers are stakeholders.

The reassignment protocol includes confirmation by participants hosting the contract’s signatories, but no Daml choice executes and the template itself has no opportunity to approve or reject the move. As a result, the application cannot express an invariant such as “this contract must remain on this synchronizer” or “reassignment requires authorization from this party.”

There are limits on where a contract can be reassigned. Every stakeholder must be appropriately hosted across the source and target synchronizers, the required signatory confirmation topology must exist, and the contract package must be vetted on the target.

We rely on those constraints today by keeping our protocol parties and protocol packages on the private synchronizer only. That prevents our market contracts from being reassigned to the Global Synchronizer.

The concern is that this protection is an operational topology invariant rather than an application invariant. We have to maintain the correct hosting and package-vetting configuration across every participant indefinitely. A topology or vetting change can turn a reassignment that was previously impossible into one that the protocol permits, without any change to the Daml contract itself.

Doing some testing on testnet. Our participant is connected to both the Global Synchronizer and a private synchronizer, with multi-synchronizer reassignment enabled (alpha). One DAR holds Prober, ProbeResult, and KeyedThing templates. The DAR is vetted on both synchronizers. All parties used are also hosted on both synchronizers.

1. Deployed Prober and KeyedThing to PrivateSync
2. Prober creates ProbeResult{found: true}
3. Manual reassign KeyedThing to GlobalSync and confirm with ACS
4. Prober transaction reverts with AUTOMATIC_REASSIGNMENT_FOR_TRANSACTION_FAILED and no ProbeResult is created
5. End result is Prober and KeyedThing are both on the PrivateSync

So I tried again…

1. Prober creates ProbeResult{found: true}
2. Manual reassign KeyedThing to GlobalSync and confirm with ACS
3. Prober creates ProbeResult{found: true}
4. End result is Prober and KeyedThing are both on the PrivateSync

This suggests that lookupByKey is not scoped to a contract’s assigned synchronizer, and the synchronizer reassigns that contract so the transaction can execute.

Separately, the rejected transaction in round 1 step 4 appears to have reassigned the contract even thought the transaction reverted. We observed the end state but did not isolate the mechanism.

I ran this scenario 20 times and the looked-up contract was reassigned to the submitting transaction’s synchronizer every time, within about 5.3 seconds of submission.

The transaction itself committed in 15/20 cases. The rest were rejected with AUTOMATIC_REASSIGNMENT_FOR_TRANSACTION_FAILED. The reassignment occurred regardless, including in every rejected case.

This is good news that the reassignment risk (of looking up keys for contracts assigned to another synchronizer) we originally were worried about is not a problem. It is odd how transactions doing this fail ~25% of the time

Dear @bmino ,

Thanks for your message and the feedback. Because multi-synchronizer is still early stages, feedback is particularly appreciated.

I need a bit of time to finalize my answer but it should come soon.

Best,

Rafael

Now with the proper answer

Authorization and rules

We decided to allow any stakeholder (including observers) to submit reassignments because it simplifies workflows and thinking about edge cases of reassignments: if the reassignment is not what was required/desired, then any stakeholder can submit the reassignment opposite “direction”.

We considered allowing to describe rules for reassignments (e.g., for package pkg, then reassignment should be possible only from synchronizer S1 to synchronizer S2 and not the opposite) but this is not currently on the roadmap. Note that such rules would most likely be in the topology state as well.

About your testing

I’d like to understand a bit more what you are trying to do and the various failures. Would you agree to share:

• The relevant templates
• Logs that surround one of the unsuccessful run (DEBUG level please)

You can do that by writing to da-support@digitalasset.com (discretionary support) and your ticket will find its way to me. Please add as much details about your workflow as possible.

Thanks.

Rafael