Lesson 2: Distributed Systems and Concurrency Concerns

Captions as YouTube publishes them, usually machine-generated, so names such as Daml and Canton can be misheard.

1. 0:01

   a daml application is inherently a distributed concurrent system and therefore shares the same concerns as any other distributed system in exchange for the advantages distribution provides the fact that there are multiple nodes inherently introduces concurrency networks and therefore unavoidable propagation delays as a result every distributed technology platform needs to provide the means to address the resulting concerns propagation delays require the platform

2. 0:33

   provide facilities to synchronize shared State and some degree of consensus on updates to that state concurrency even without distribution introduces the risk of data races propagation delays make them more likely and therefore facilities to avoid or protect against them essential with networks involved both Network partitions and node failures part IAL and transient will happen this necessitates dedicated facilities in the

3. 1:05

   platform to ensure at most once execution and uniqueness finally when building on a distributed technology platform your components can be oriented around materialized shared state or streams of events it is important to understand when and why one of these will be preferred over the other over the course of this lesson we will investigate how daml addresses each of these concerns transactions are created by

4. 1:37

   interpreting commands submitted to The Ledger API in this example this starts with a backend process querying The Ledger state in pqs it uses the results of this query to compose a DL command it then submits the command to The Ledger API on its participant node where a daml engine evaluates the command into a transaction using node's local private contract store the participant node then sends the transaction to the participant nodes

5. 2:09

   of the transaction stakeholders using the synchronizer multiple transactions are processed concurrently stakeholder participant nodes use the caned commit protocol to reach consensus on each transactions commit order using the synchronizer to share data and coordinate the process it is this transaction commit order that determines when the transaction is deemed to have updated the shared State during the commit protocol each participant node validates the

6. 2:40

   transactions based on the consensus transaction order confirming they maintain The Ledger invariance valid transactions are added to the private contract store and at this point become visible to Future command evaluations shortly after that the transactions are indexed by P eqs and become visible to queries from the backend process for the first time the Canton transaction commit protocol ensures that updates are synchronized the invariance in the DL

7. 3:13

   Ledger model provided a deterministic definition of what is a valid transaction together they ensure that only valid transactions get committed and The Ledger evolves valid transaction by valid transaction which leads us directly to the next set of concerns a daml application is an eventually consistent system while the system is in use no individual component can guarantee their view of the current state has not been changed elsewhere in

8. 3:48

   the system the Canton synchronization protocol does ensure that if the entire system has been quiet for a minute or two every node will agree on the current state of that portion of The Ledger they are authorized to observe in other words The daml Ledger is eventually consistent otherwise the propagation delay inherent in this Loop can mean thousands of inflight transactions appended to The Ledger between us querying pqs and the eventual verification of our

9. 4:20

   transaction if one of those inflight transactions ends up conflicting with ours then our transaction will be rejected by the transaction verifier this is a concurrent system and contention will happen a worse outcome is where there is no contention between the transactions but the combined result of both commands violates functional business requirements this is what a data Ras defect looks like in a Danel application as an example consider a

10. 4:52

    buggy loan application the goal is to issue loans via overdraft facilities provided the total amount of issued loans is below a per facility limit borrow Choice delegates the limit check to the backend process that executes it creating the loan with the required Bank client and account fields to illustrate why this is a problem we have a system with a single overdraft facility already created in the PCS and indexed in pqs two loone

11. 5:24

    front and uis both send borrow requests against the facility at similar time for each request the loan application backend queries the overdraft and existing loans and if the sum of the existing loans leave sufficient credit converts the request into an exercise command to The Ledger API in this case with no existing loans both requests satisfy the check and so we have two commands issued the daml engine converts each command into a

12. 5:57

    transaction in doing so it checks the overdraft facility is active for now and evaluates the choice which allows it to determine that each transaction results in creating a loan contract during the Canton synchronization protocol the transaction verifier will run the same checks as the D engine however at this point due to the protocol the verifier knows the transactions proposed place in The Ledger the verifier is therefore able to determine with absolute

13. 6:27

    certainty if every trans action dependency remains valid in this case because the only check required by The Ledger invariance is the check on the facility both transactions will verify successfully and thus both loan contracts will be created exceeding the limit the propagation delay between the backend process and the verifier meant this bugy version of borrow Choice permits at a data race that violates the business rules of the

14. 6:58

    application the Sol solution is to use the daml ledger invariance to convert each silent data Ras into transaction contention that will be detected by the verifier start by ensuring your business constraint is represented on The Ledger in this case we add a loan balance contract make sure these constraint contracts are passed as contract IDs to choices required to satisfy those constraints then ensure your choice performs a fetch or exercise excise for

15. 7:30

    every contract whose State you depend on for the choice to be valid finally check your validation conditions using assertions in the choices body before performing your business Logic the transaction verifier will then ensure your transaction will only be committed provided the fetched contracts are still active if they are no longer active then that's a sign a data Ras occurred and her command submission should be retried using the most recent state of The

16. 8:01

    Ledger this fetch validate and act pattern is very common within daml choices this is at the heart of our recommendation to structure Ledger interactions around the larger pattern query Ledger state from pqs include resulting contract IDs in the generated command in the damel choice fetch the contract IDs and validate their Associated contracts only then update The Ledger allowing DL transaction verification to prevent violation of

17. 8:34

    business constraints consistent use of this pattern allows daml and Canton to protect us against data races within a daml application query generate fetch validate update then was the updated state is visible in pqs rinse and repeat and by restarting from the initial pqs query whenever validation fails this core Loop drives resilient performant and reliable daml applications recall pattern achieves reliability and resilience by converting business constraint violations into ler

18. 9:15

    invariant violations violations of business constraints are detected as contention between D transactions recognize that contention is not a bug in a distributed system some contention is inevitable if the system is to remain consistent in a damel application contention will be detected at one of the two places daml is evaluated either during command interpretation in the daml engine preparing the transaction or

19. 9:47

    in the transaction verifier during the transaction commit protocol either way the result is The Ledger API reporting a command failure to the backend process this failure indicates the PCS has changed invalidating the state the backend relied on that state change is or will shortly be reflected in the pqs so the back end should respond by retrying the entire Loop starting with rerunning the pqs queries against the latest Ledger State other common reasons

20. 10:22

    for retries include command timeouts flow control back off and backend service restarts regardless of the reason it is likely that The Ledger has changed in the interim it is rarely worth trying to guess if the original command is still valid it is almost always better to regenerate the command with a new query retries introduce the possibility of duplicate command submission normally duplication is going

21. 10:53

    to cause a business constraint violation we can already see this in our overdraft example with duplicate loans being a problem to avoid this ensure your commands are itm potent at the level of the business operation as we saw with the overdraft facility the easiest way to achieve this is ensuring that a contract associated with the business operation will be consumed by the command in our overdraft example we exercise a consuming choice on the loan

22. 11:25

    balance archiving The Loan balance ensures any subsequent use of the overdraft must take this use into account when you can't represent the business operation as a contract each participant node provides its own command ID D duplication facility this is common when submitting create commands as obviously these cannot consume anything to utilize this generate a deterministic command ID based on the

23. 11:55

    business operation the participant node will reject any command that duplicates a previously successful or pending command ID from the same submitting participant user in some cases you can add consumption to an otherwise non-consuming transaction this can be a useful approach for situations such as when ingesting messages from a CFA partition bundling an additional transaction containing a consuming exercise into an aggregate command can

24. 12:26

    allow non-consuming transactions to benefit from the item potency of the bundled command you can store the last ingested offset in a contract that is consumed and updated when ingesting new messages all three of these approaches can be used to ensure uniqueness the choice between them will generally be made to satisfy other scalability and atomicity requirements when trying to ensure uniqueness it is also worth considering

25. 12:57

    if your business function can Toler eventual uniqueness models that allow for duplicates to be merged or cleaned up in the background are often simpler and allow for higher throughput item potency also matters when your backend process is sending data from The Ledger to an external process you will need to choose an appropriate identifier and track your progress in most cases the obvious choice is the participant node

26. 13:27

    offset this identifier is scoped to each participant node offset increases monotonically with transactions being added to the PCS and hence the order they are seen by the back end which is why it is so useful for tracking the last seen Ledger state it is not comparable between participant nodes so you will need to factor that into scaling and failover planning also available but less common option is synchronizer sequencing time

27. 13:59

    while ordered its relationship to observed order from the back end is Less Direct however it does have the advantage of being consistent across all participant nodes sharing a synchronizer finally you can use the fact that Demo's contract IDs transaction IDs and event IDs are globally unique and therefore globally comparable we recommend you base your application interaction loop on querying the current Ledger state provided by

28. 14:34

    pqs given that The Ledger is canonically defined as a stream of events it is worth considering why we make this recommendation the experience of many years building advising and supporting DL applications has shown us that a state-based approach has many advantages in design generating commands from queries lends itself to a common approach to dealing with contention that doesn't require knowledge of the internals of the workflow also designing in terms of

29. 15:06

    transitions from observed state to desired State tends to result in designs that are easier to understand and reason about in development the global view available to a state-based query is inherently more tolerant of minor variations in observed transaction and event order similarly when an implementation is based on rest storing desired Global invariance it becomes easier to verify that the total system complies with business requirements when testing the

30. 15:38

    same case-based reasoning that assisted with design now supports case-based testing this makes it much easier to keep tests deterministic an important requirement to support Dev velocity also the traditional query response structure fits well with existing test infrastructure and simple stubs we find these advantages along with the deterministic nature of daml itself substantially improves the team's confidence in their test coverage

31. 16:09

    finally in production we see significant benefits as an increased focus on the ACs Over The Ledger stream increases tolerance of pruning we have found the same Focus also leads to faster and more stable recovery after incidents and the highly deterministic nature of of historical state to historical command generation makes diagnosis of unexpected Behavior much easier we have seen how the synchronization and consensus guarantees

32. 16:41

    of damel and the Canton protocol can be used to solve the various concerns inherent in distributed and concurrent systems in particular we have explored how these combine to form the recommended interaction Loop how it works and the advantages generally associated with it