Lesson 3: Daml Application Backend

Captions as YouTube publishes them, usually machine-generated, so names such as Daml and Canton can be misheard.

1. 0:02

   a d application backend serves three primary purposes provisioning of higher level apis for communication with The Ledger automation of on Ledger workflows integration with off Ledger systems in this lesson you will learn about the key design considerations with regard to these purposes you will also learn how to select the appropriate text stack for implementing the D app back end when an app needs to communicate with The Ledger there's a number of common considerations that you may want

2. 0:37

   to abstract to higher level apis which can then be used by other logical components of the back end by the app front end or by external apps or systems these considerations are different for the read path that is when you want to read data from The Ledger and for the right path that is when you need to write to The Ledger on the read path the most important consideration is to read Ledger data using a component of a participant node called participant query store or

3. 1:10

   pqs pqs extends and complements capabilities available through Ledger API Ledger API is optimized for efficiently exposing the current state of The Ledger and the changes to it without filtering however deml apps typically also have applications specific query needs which often include retrieval of individual contracts pqs is designed to serve as an operational data store it provides flexible and efficient query capabilities for Ledger data on the

4. 1:44

   participant node with pqs you retrieve Ledger data using SQL over a jdbc connection for example this means implementing the read path is as simple and standard as implementing any other web service backed by an SQL database pqs has access to all data it is configured and allowed to ingest from the PCS but doesn't Implement any extra access control for pqs clients however when presenting the

5. 2:16

   retrieved data to Upstream components you need to ensure appropriate enduser authentication and access controls are enforced by the API service the reason for delegating the implementation of end user access controls to the developers of the D app backend and not including it in pqs is that this allows implementing access controls at any level of granularity crucially it allows access controls to be based on

6. 2:46

   intraorganizational business requirements it allows different users within the same organization to have access to different data sets as opposed to constraining access control to the schemas encoded in the D model which are focused on interorganizational access control authentication of requests for Ledger data is discussed in detail in the next course on this certification path pqs supports accessing both the active contract set or ACS and The

7. 3:19

   Ledger history within the pring window of pqs we recommend choosing that pring window based on your business needs and the expected data volume if required pqs can be used as a flexible source of filtered events for analytical processing or to populate a data warehouse pqs stores both the contracts and the exercise events that justify the contract's creation and archival as a daml model design consideration we recommend against

8. 3:51

   storing information about completed workflow steps in the form of active contracts as this leads to unbounded growth of the ACs instead you can use the historic events in the pqs as the golden source to serve for example an application specific transaction log or to deliver notifications about completed workflow steps to end users sometimes it is useful to include additional non-consuming exercise events to represent particular workflow events

9. 4:23

   or notifications that should be communicated to stakeholders in the workflow pqs provides a general purpose queriable view of Ledger history and state suitable for a wide range of use cases it is the recommended path for Ledger data reads from Automation and integration components however for some specialized high-scale read use cases a custom ODS solution may need to be designed that stores and indexes data in a format

10. 4:53

    tailored to the specific read Requirements on the right path the most important consideration when implementing a higher level API is reliability which breaks down to two related items retry behavior on command failure and itap potency of command submission since retry on failed command submissions is required by every component of your application that needs to send commands to The Ledger it usually makes sense to package up this

11. 5:24

    capability in a reusable fashion since commands to The Ledger may be computed and resubmitted due to retries and or crashes it is important to ensure that rights to The Ledger are it impotant a very simple way of achieving it is to make the command sent to The Ledger consume some of its input for example exercise a consuming choice on the contract that led to sending the command another technique is to use command D

12. 5:55

    duplication participant nodes provide a mechanism in The Ledger API to ensure that they execute a command at most once this mechanism is known as command D duplication the mechanism works by the participant node storing the command ID and D duplicating later submissions with the same command ID for details on the mechanics of command D duplication see The Ledger API documentation daml code has no independent thread of execution contracts on a ledger are

13. 6:29

    passive records of synchronized data and wres specifying who can advance the shared workflows these workflows remain static until Advanced by external components any action on The Ledger must be initiated by external components when implementing Automation in a daml app backend we recommend modularizing it into retriable tasks well-defined independent units of work that the automation is guaranteed to be a to complete tasks can then be processed

14. 7:03

    with bounded parallelism based on this code automation tasks can be triggered by external events common examples of external events are messages received from Off lger Systems and time events which allow automation tasks to run at a given time or on a given schedule tasks often represent the need of a backend to advance an on lger workflow for example to hand an app user onboarding request by checking the provided information against an off lger

15. 7:34

    know your customer system and auto accepting the request when that check succeeds given that workflows represent their state using DL contracts these tasks are triggered by the creation of DL contracts or by the backend discovering their existence after it has started we call these kinds of tasks State triggered automation it is important to to note that daml contracts that trigger automation should be consumed in the triggered command processing to avoid

16. 8:06

    the automation looping technically State triggered automation can often be considered time triggered as well as it is typically implemented by periodically querying the pqs for new tasks in a polling fashion when retrying such tasks it is important to rerun the whole query against pqs to ensure that your new attempt is based on the most recent data as The Ledger state may be changing due to concurrent actions an automation task should always utilize the most recent Ledger State

17. 8:39

    available on pqs when multiple queries need to be executed as part of the same automation task run make sure that consistent Ledger offset is utilized in all queries each task should be automatically retried on retryable errors up to a limit the whole code code block that processes the task should be retried not just the Ledger command submission within the block this ensures that the most recent Ledger state is reflected in

18. 9:10

    the command being submitted or even discovering that due to concurrent actions on The Ledger the task has already been completed or is otherwise no longer valid in which case the retrying loop should exit we've looked at two functions that a DL app backend performs provisioning of higher level apis and automation of on Ledger tasks the third function is integration with off lger systems for this function there are no special

19. 9:42

    considerations you can use your existing integration Technologies as appropriate following the it landscape where the daml app backend is expected to be deployed there are various integration options a common case is for the backend to query Off lger Systems as part of automating on lger workflows for example to query and know your customer database or to run a complex computation using the system that already implements this

20. 10:13

    computation an example of such computation is a margin calculation on a financial position or a netting computation for optimization of transaction settlement in financial markets data can be pushed from Off lger Systems to The Ledger using API calls or by having the backend consume messages from a message queue this can be used for example to ingest pricing data from a message queue and create reference data contracts providing this pricing data for on

21. 10:44

    Ledger workflows data can be pushed from The Ledger to Off lger Systems by the backend using web hooks or by writing to message cues a pull-based consumption is also possible leveraging the offset based ACC access to events provided by both pqs and The Ledger API an example of this kind of data flow would be replicating the securi registration data from an on lger registry to the order book matching Services other examples are feeding

22. 11:15

    accounting systems or populating reporting databases the options we mentioned are non-exhaustive there are other ways of exchanging data between The Ledger and Off lger Systems use whatever technology makes development and deployment of the app easier for the app provider and the app users whatever technology you use we recommend keeping your read and write paths separate and that integration between off Ledger systems and The Ledger is implemented via a backend

23. 11:48

    service rather than allowing off Ledger systems to use Ledger API directly the last topic we're going to discuss in this lesson is how to choose the text stack for the implementation of a DL app backend as you seen there's nothing out of the ordinary about the functions that a DL app backend serves and the tasks that need to be implemented for these functions hence there's nothing special about the tech stack required to implement the DL app backend we

24. 12:20

    recommend that you use your standard stack for building an Enterprise application to interact with a ledger API use whatever Library you consider standard for interacting with grpc Services similarly use your Tech stack standard for I am integration one thing you should expect is that you'll need some background processing to react to on Ledger State changes and to ingest data from external sources Into The Ledger you should also invest in making all interactions with The Ledger crash

25. 12:53

    fault tolerant by always using retries with it imp poent commands any programming language or any combination of languages can be used to implement daml app backend Services however for some languages there are readily available higher level programming tools such as the daml code gen the daml code gen is very helpful to interact with the payload of DL contracts as the Cod gen utility generates the mapping between the types implemented in daml models and language

26. 13:25

    types for example the Coden utility can generate Java classes corresponding to DL contract templates in your DL models these classes incorporate all boilerplate code for encoding and decoding Ledger API representation of daml contract arguments and creating commands for exercising the contract's choices please check daml documentation for the latest list of languages where higher level programming tools are readily available if you use other languages

27. 13:58

    course responding infrastructure code may not yet have been built and you may need to use grpc directly in this case you may want to consider creating The Ledger bindings including a coach and utility or other metaprogramming feature for the language of your choice to summarize in this lesson we have learned that a DL application backend serves three primary purposes provisioning of higher level apis for communication with The Ledger automation of on lger

28. 14:32

    workflows integration with Off lger Systems when implementing a higher level API use pqs to read from The Ledger on the right path to the Ledger the most important consideration is reliability which breaks down to two related items retry behavior on command failure and it imp potency of command submission there's nothing unusual about the function that a DL app backend serves hence there's nothing special

29. 15:04

    about the tech stack required to implement the daml app backend we recommend that you use your standard stack for building an Enterprise application