Lesson 5: Daml Workflow Scalability

Captions as YouTube publishes them, usually machine-generated, so names such as Daml and Canton can be misheard.

1. 0:04

   we have decomposed the problem of analyzing workflows into understanding the events people decisions and consequences involved and how they are observed and authorized and we've looked at how this creates a core Loop for people to make business decisions now we'll turn our attention towards the need for testing how to think about scaling and managing the situations where contention between work work flows can occur so far we've used an example application a digital Trading

2. 0:35

   Card Market to explore how we might create a workflow appropriate for a distributed Ledger application as we've analyzed this simple workflow we've come up with a simple action-oriented design after some adaptation to accommodate the pragmatic concerns associated with production this sort of simple design will work for the vast majority of workflows in a daml application the places where this approach won't be sufficient are the workflows that drive the scaling and performance ceiling in

3. 1:07

   your application requirements using either a data oriented design or this sort of simple action oriented design for these requirements defining workflows will at best result in a need to over-provision hardware and at worst can result in levels of contention that can make meeting the requirements impossible first some general advice nothing beats testing so start testing for scale early and often unlike software optimization this sort of information schema

4. 1:41

   optimization is something that needs to be done early in the development process the Dale models developed at the beginning of the project become the foundations that structure a lot of Downstream code and application development as in most things changing foundations can be difficult and invasive so leaving scaling and performance considerations to the end of a project can result in significant rework so is important to include

5. 2:11

   testing of key performance and scalability benchmarks as early as is feasible and to keep testing throughout the development process you can get a lot of dam application skillability from vertical and horizontal scaling of the participant and domain nodes with a suitable Dam model you can truly provide incredible performance and scalability by a combination of vertical and horizontal scaling of participant and domain nodes critical to efficient

6. 2:42

   scaling is ensuring your model supports full utilization of the resources deployed to support it the primary cause of underutilization is workflow contention in the worst case excessive contention between workflows can prevent any ability to scale the application contention occurs when multiple workflows attempt to exercise consuming choices on the same contract this issue is almost inevitable if you have used data Centric modeling

7. 3:14

   techniques especially the relational database schema design techniques we are all used to a common example of this is when a model is denormalized resulting in a centralized holding contract every transfer in and every transfer out of the holding now has to queue to adjust the balance on this single contract as if we had a supermarket with only a single cash register that can only process one item at a time in a supermarket the solution is simple allow

8. 3:47

   multiple items to be sold in a single transaction and allow multiple cash registers to operate simultaneously and with most Dale models for example the problematic holding cont contract the solution is analogous allow multiple transfers to be applied to the holding in a single transaction and allow the holding to be split over multiple contracts that can be updated simultaneously now we will look at these in more detail but first to reiterate if

9. 4:18

   you take nothing else away from this lesson Let It Be these three things test for scale early and often allow vertical and horizontal scaling by managing intention with consistent attention to these three things the vast majority of performance and scalability issues will be avoided to try to gain an intuition for D optimization let's consider again the point of sale example we are setting up a store that will sell things and settle transactions for money we have

10. 4:52

    workflows that involve staff providing advisory services that ultimately result in a sale that will use a cash Reg register The Advisory Services may be very complicated and take several minutes to complete but ultimately every advisory workflow will need to use a cash register to process the transaction when we design for a workflow it is common to simplify things in this example we'll have one customer one advisor one sale and our design will

11. 5:23

    end up with one cach register we may well Implement a small proof of concept along these lines and prove that one cash register is sufficient for a small Store with only one staff member serving One customer at a time this is where a failure to test the Real Performance and scalability requirements can bite as long as we only need to support one staff member and two or three concurrent customers this model will prove incredibly simple and

12. 5:53

    flexible in fact at that scale it is an ideal solution to the problem at as we increase the number of customers though we will find we have to increase the number of Staff if we don't then the high latency of the long running advisory services will dominate and our throughput will be limited if the advisory Services take five minutes then a single staff member sets a hard upper

13. 6:19

    limit of 12 transactions per hour and it doesn't take a lot of customers to see latencies climb to half an hour or more now we have to revisit our design because we have multiple staff we are going to need to consider how our workflow allocates a customer to a staff member and ensure the staff processes have the resources they need to operate independently still even with three to

14. 6:46

    four staff members we can probably get by with only one cash register if we allow a short but manageable queue of course we may need to revisit our simple design again and include this Q or else a disorderly rush for the register may cause problems eventually as we increase the number of customers and staff to match the que stops being a transient occasional thing and becomes a permanent part of the workflow that every transaction has to navigate and still it

15. 7:19

    might be manageable for a while if our latency requirements are flexible enough if our scaling requirements demand we service customers Beyond this point then adding more staff will not help at some point the contention for the cash register comes to dominate the entire workflow experience if it takes 30 seconds to process a transaction through the register that sets a hard limit of 120 transactions per hour but the system will be unworkable long before we reach

16. 7:52

    that limit instead of per hour let's assume that we want to handle 120 trans transactions per minute how might we achieve this this is where the two approaches we mentioned at the outset come in allow multiple items to be sold in a single transaction and allow multiple cash registers to operate simultaneously the first approach Finds Its use in almost every performance remediation as it generally has the smallest impact on the existing system

17. 8:25

    this might seem counterintuitive because adding more cash rate registers probably doesn't affect the sales workflow much but recall what we said earlier daml models form the foundations of a daml application and foundational changes late in a project can be expensive the Assumption of a single register quickly becomes embedded in other workflows like endof day closeout procedures or realtime balance reporting once this assumption is there changing it may

18. 8:56

    require substantial changes to these other workflows in a physical store we might dedicate a staff member to collecting a batch of sales and processing them in a single transaction at the cash register if we want to handle 120 transactions per minute we will need to be able to process at least 60 transactions per batch while this approach avoids changing the other dependent workflows it does affect the sales workflow we need to revisit our simple

19. 9:29

    design as staff no longer access the register directly instead they add their request to the batch and wait for it to be processed alternatively we can use the second approach installing multiple independent cash registers most of the time this will produce a more flexible and scalable design a daml application is inherently a distributed application and reducing the number of centralized points of contention is necessary NE AR to leverage DL to its full potential the

20. 10:02

    impact of this decentralization on workflows that haven't been written yet is often minor which is why this is something that becomes both more difficult and more expensive the longer you leave it it is important to be considering your true scalability requirements as early as possible then start testing to find this point of contention now rather than in 6 or 12 months after a dozen other workflows have been implemented assuming access to this single centralized

21. 10:33

    contract so if we can run 60 cash registers in parallel we can meet the 120 transactions per minute throughput of course we can do both batching and distributed concurrency and our throughput limit would now be 120 transactions per second a number in excess of requirements for our physical store but a very reasonable expectation for a daml workflow that has been designed to scale remember the vast majority of workflows in your

22. 11:05

    application will not need this treatment therefore we don't want to add this complexity to every workflow how do we identify where we need to apply these principles start by estimating the business transaction performance requirements how many holding transfers how many inventory acknowledgements and how many sale completions for each business transaction identify which contracts SL templates can be consumed by the relevant step in the relevant

23. 11:37

    workflow then consider possible contention sources what happens when we run multiple instances of a workflow concurrently do they end up needing to update or consume the same shared contract what happens when we run the transaction in parallel with other workflows again do they end up with multi multiple updates to the same contract that would therefore need to be cued or serialized and are there any consequences for one step that become inputs that another workflow will need

24. 12:10

    to wait for note that all of these involve updating a contact which in Dam always involves contention on an archive operation so that makes a good guide identify the workflows and workflow steps that want to Archive the same contract then compare your number against a critical rule of thumb your design should not require better than 1 second roundtrip latency to The Ledger a roundtrip has to prepare then submit a transaction to the Canton

25. 12:42

    Ledger then validate and confirm the transaction was correctly committed to a ledger this will normally take less than 1 second however to depend on being able to sustain better than that when the system is under stress will add risk and instability to your application consequently any business transaction you estimate to have a contended archive performance requirement greater than or even close to one per second should be prioritized for early performance in scalability

26. 13:14

    testing keep in mind Peak load can be 10 to 100 times higher than the average load over a full day or full week when doing this analysis there is no substitute for understanding your business process using these principles on the targeted workflows that you identify as key to your system's performance and scalability requirements it is possible to run thousands of business transactions per second but only if you decentralize the shared contracts in critical workflows so they admit

27. 13:47

    concurrent transactions and then maximize the throw put per Dale transaction using batching to take full advantage of that concurrency then test this availability of your designs early and often