{"id":"NOJkCFNyY4s","title":"Lesson 6: Data in Canton Network","description":"For a better learning experience, sign up and enroll in this free course at: https://daml.talentlms.com/plus/catalog/courses/153","channel_name":"Digital Asset Training","published_at":"2025-02-20T00:00:00.000Z","thumbnail_url":"https://i.ytimg.com/vi/NOJkCFNyY4s/hqdefault.jpg","transcript_text":"managing and querying the data created by daml application requires understanding the concept of active contract set data pruning and using an operational data store in this lesson we will learn about these Concepts which are critical for designing a high-performing daml application there are two important Concepts to understand about the data created by Dam applications first what is stored and second where it is stored as the contracts in daml are immutable they are not deleted or updated if there is any change in a contract the old contract is archived and a new one is created therefore a ledger at any point in time has an active contract set or ACS which is a set of all active contracts at that time The Ledger also stores the history of transactions that led to this ACS this history is represented in the form of transaction trees a transaction tree is a hierarchical structure in which a highlevel transaction for example making a change to a contract is carried out in a series of subtransaction such as archiving old contract and creating a new contract for example let us say we want to create an account and deposit $20 in it so transaction zero creates an account contract which becomes an active contract then another transaction is carried out to deposit some money in it this has subtransaction exercising the deposit choice on the account contract which further has a subtransaction of archiving The Old account contract and creating a new account contract with $20 in it at the end of these transactions we have one active contract for the bank account with $20 in it and a history of the transactions carried out before it was created as you can see this can lead to the ledger size reaching orders of magnitude higher than similar loads in traditional database systems which can negatively impact the daml application performance to overcome this challenge we recommend some best practices and techniques first when designing workflows pay attention to the lifetime of contracts if there are any supporting or auxiliary contracts make sure they get archived when their purpose is done to further control the lifetime of workflows make them complete within a limited time frame and add a choice to Archive their contracts once the time frame has expired this way workflows that get stuck can always be cleaned up after a certain amount of time and second set up a data pruning schedule pruning refers to the selective removal of of archived and unneeded data from participant sequencer and mediator nodes automatic pruning schedules can be created as cron jobs at the frequencies desired for each node pruning deletes the data but it does not perform any database maintenance such as table resizing so for that you will need to use the database systems features for further optimization and vacuuming of storage space pruning is an IO heavy operation so it should be scheduled during maintenance Windows you will find more details about how to schedule these pruning activities in daml documentation at docs. dam.com now the next question to think about is how can we access the data stored in the Canton Network and for that let us look at where it is stored when a command is submitted it is submitted to a sync domain used by the participant node hosting the submitting party the command FL close through the sequencers and mediators to reach participants hosting the affected parties the contracts resulting from this command are stored in the private contract stores of all the participant nodes where the involved parties are hosted they are also stored in an encrypted form on the sequencer backends for delivery to involved participant nodes it is important to note that the applications use the data in participant nodes and not on the sequencer backend as a sequencer backend is a relatively temporary storage to access this data stored in the private contract store web UI use The Ledger API including Json API for getting the current state of The Ledger in bulk streaming events and state changes and point-wise queries like fetching contracts by key or ID they can query over the ACs for example to get all active contracts where field X is set to value why we can also query the historical data which can both include actually showing the history directly for example the transaction history in a wallet as well as Aggregates for example how much did I spend in the last month however submitting complex queries directly to the Ledger API can adversely affect the daml application performance and the reasons are first the Json API used by daml applications to access The Ledger is not designed for high performance complex queries and second it cannot access historical data and full transaction trees the recommended approach is to use an operational data store or ODS having an independent ODS that mirrors The Ledger will allow efficient querying without negatively impacting the daml application's performance one option for implementing an ODS is to use the participant query store or pqs which is part of the daml SDK pqs ingests the contracts and transactions into a postgis database it thus allows the backend to use jgbc and the rich querying capabilities of SQL query execution can then be accelerated using standard tools like creating indices you can find more detailed guidance on how to set up an ODS in Docs dl.com let us now summarize as The Ledger data has contracts that are immutable they are never deleted or overwritten but are archived so the data storage at the nodes has an active contract set along with historical data of archived contracts which continues to grow at a fast pace to handle this data growth effectively you should pay attention to lifetime of contracts when designing them and also use the data pruning feature through which you can selectively remove archived or unneeded contracts from the storage daml applications create data in the form of contracts that are stored in the private contract stores of participant nodes where the parties involved in a transaction or hosted this data is also stored in the sequencer backend in the sync domain finally to query the data stored in The Ledger it is recommended that you set up an operational data store and use that as the source for performing complex queries instead of querying The Ledger directly as it may impact the daml applications performance daml offers participant query store or pqs as an ODS that can be set up in a participant node","transcript_synced_at":"2026-10-26T04:01:26.263Z","segments":[{"start_seconds":5,"text":"managing and querying the data created by daml application requires understanding the concept of active","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=5s"},{"start_seconds":12,"text":"contract set data pruning and using an operational data store in this lesson we will learn about these Concepts which","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=12s"},{"start_seconds":20,"text":"are critical for designing a high-performing daml application there are two important Concepts to understand about the data","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=20s"},{"start_seconds":29,"text":"created by Dam applications first what is stored and second where it is stored as the","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=29s"},{"start_seconds":36,"text":"contracts in daml are immutable they are not deleted or updated if there is any change in a contract the old contract is","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=36s"},{"start_seconds":45,"text":"archived and a new one is created therefore a ledger at any point in time has an active contract set or ACS which","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=45s"},{"start_seconds":54,"text":"is a set of all active contracts at that time The Ledger also stores the history of transactions that led to this","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=54s"},{"start_seconds":62,"text":"ACS this history is represented in the form of transaction trees a transaction tree is a hierarchical structure in","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=62s"},{"start_seconds":71,"text":"which a highlevel transaction for example making a change to a contract is carried out in a series of","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=71s"},{"start_seconds":77,"text":"subtransaction such as archiving old contract and creating a new contract for example let us say we want to create an","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=77s"},{"start_seconds":86,"text":"account and deposit $20 in it so transaction zero creates an account contract which becomes an active","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=86s"},{"start_seconds":95,"text":"contract then another transaction is carried out to deposit some money in it this has subtransaction","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=95s"},{"start_seconds":102,"text":"exercising the deposit choice on the account contract which further has a subtransaction of archiving The Old","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=102s"},{"start_seconds":109,"text":"account contract and creating a new account contract with $20 in it at the end of these transactions we have one","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=109s"},{"start_seconds":117,"text":"active contract for the bank account with $20 in it and a history of the transactions carried out before it was","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=117s"},{"start_seconds":125,"text":"created as you can see this can lead to the ledger size reaching orders of magnitude higher than similar loads in","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=125s"},{"start_seconds":133,"text":"traditional database systems which can negatively impact the daml application performance to overcome this challenge","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=133s"},{"start_seconds":141,"text":"we recommend some best practices and techniques first when designing workflows pay attention to the lifetime","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=141s"},{"start_seconds":148,"text":"of contracts if there are any supporting or auxiliary contracts make sure they get archived when their purpose is done to further","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=148s"},{"start_seconds":157,"text":"control the lifetime of workflows make them complete within a limited time frame and add a choice to Archive their","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=157s"},{"start_seconds":164,"text":"contracts once the time frame has expired this way workflows that get stuck can always be cleaned up after a","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=164s"},{"start_seconds":171,"text":"certain amount of time and second set up a data pruning schedule pruning refers to the selective removal of of archived","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=171s"},{"start_seconds":180,"text":"and unneeded data from participant sequencer and mediator nodes automatic pruning schedules can be created as cron","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=180s"},{"start_seconds":189,"text":"jobs at the frequencies desired for each node pruning deletes the data but it does not perform any database","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=189s"},{"start_seconds":197,"text":"maintenance such as table resizing so for that you will need to use the database systems features for further","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=197s"},{"start_seconds":203,"text":"optimization and vacuuming of storage space pruning is an IO heavy operation so it should be scheduled during","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=203s"},{"start_seconds":211,"text":"maintenance Windows you will find more details about how to schedule these pruning activities in daml documentation","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=211s"},{"start_seconds":218,"text":"at docs. dam.com now the next question to think about is how can we access the data stored in the Canton Network and","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=218s"},{"start_seconds":227,"text":"for that let us look at where it is stored when a command is submitted it is submitted to a sync domain used by the","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=227s"},{"start_seconds":235,"text":"participant node hosting the submitting party the command FL close through the sequencers and mediators to reach","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=235s"},{"start_seconds":242,"text":"participants hosting the affected parties the contracts resulting from this command are stored in the private contract stores of all the participant","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=242s"},{"start_seconds":251,"text":"nodes where the involved parties are hosted they are also stored in an encrypted form on the sequencer backends","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=251s"},{"start_seconds":259,"text":"for delivery to involved participant nodes it is important to note that the applications use the data in participant","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=259s"},{"start_seconds":266,"text":"nodes and not on the sequencer backend as a sequencer backend is a relatively temporary storage to access this data stored in","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=266s"},{"start_seconds":276,"text":"the private contract store web UI use The Ledger API including Json API for getting the current state of The Ledger","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=276s"},{"start_seconds":284,"text":"in bulk streaming events and state changes and point-wise queries like fetching contracts by key or ID they can","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=284s"},{"start_seconds":292,"text":"query over the ACs for example to get all active contracts where field X is set to value why we can also query the","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=292s"},{"start_seconds":302,"text":"historical data which can both include actually showing the history directly for example the transaction history in a","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=302s"},{"start_seconds":309,"text":"wallet as well as Aggregates for example how much did I spend in the last month however submitting complex queries","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=309s"},{"start_seconds":318,"text":"directly to the Ledger API can adversely affect the daml application performance and the reasons are first the Json API","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=318s"},{"start_seconds":327,"text":"used by daml applications to access The Ledger is not designed for high performance complex queries and second","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=327s"},{"start_seconds":335,"text":"it cannot access historical data and full transaction trees the recommended approach is to use an operational data","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=335s"},{"start_seconds":343,"text":"store or ODS having an independent ODS that mirrors The Ledger will allow efficient querying without negatively","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=343s"},{"start_seconds":352,"text":"impacting the daml application's performance one option for implementing an ODS is to use the participant query","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=352s"},{"start_seconds":360,"text":"store or pqs which is part of the daml SDK pqs ingests the contracts and","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=360s"},{"start_seconds":367,"text":"transactions into a postgis database it thus allows the backend to use jgbc and","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=367s"},{"start_seconds":374,"text":"the rich querying capabilities of SQL query execution can then be accelerated using standard tools like creating","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=374s"},{"start_seconds":383,"text":"indices you can find more detailed guidance on how to set up an ODS in Docs","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=383s"},{"start_seconds":389,"text":"dl.com let us now summarize as The Ledger data has contracts that are immutable they are never deleted or","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=389s"},{"start_seconds":398,"text":"overwritten but are archived so the data storage at the nodes has an active contract set along with historical data","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=398s"},{"start_seconds":406,"text":"of archived contracts which continues to grow at a fast pace to handle this data growth effectively you should pay","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=406s"},{"start_seconds":414,"text":"attention to lifetime of contracts when designing them and also use the data pruning feature through which you can","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=414s"},{"start_seconds":421,"text":"selectively remove archived or unneeded contracts from the storage daml applications create data in the form of","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=421s"},{"start_seconds":429,"text":"contracts that are stored in the private contract stores of participant nodes where the parties involved in a transaction or hosted this data is also","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=429s"},{"start_seconds":439,"text":"stored in the sequencer backend in the sync domain finally to query the data stored in The Ledger it is recommended","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=439s"},{"start_seconds":447,"text":"that you set up an operational data store and use that as the source for performing complex queries instead of","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=447s"},{"start_seconds":453,"text":"querying The Ledger directly as it may impact the daml applications performance daml offers participant","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=453s"},{"start_seconds":461,"text":"query store or pqs as an ODS that can be set up in a participant","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=461s"},{"start_seconds":478,"text":"node","url":"https://www.youtube.com/watch?v=NOJkCFNyY4s&t=478s"}],"youtube_url":"https://youtube.com/watch?v=NOJkCFNyY4s","url":"https://ccpedia.xyz/videos/NOJkCFNyY4s"}