How to manage large ACS?

Hi guys, we got a new problem here. The Java application we are building needs to read a large ACS. Is there anyway to process the ACS information and throw it away before the end of the ACS?

The GetActiveContractsResponse() in com.daml.ledger.javaapi.data appears to read the whole ACS and return it. Any one know if there is a different way to handle it? e.g. paging or streaming?

The ACS gRPC stream is split up based on the workflow ID. What happens if there is one workflow ID with a disproportionate number of contract? Is there any way we can split the stream up further? It can be very memory intensive if one workflow ID has a huge number of contracts. To clarify, our strategy is to stream the ACS, for each contract from the stream we call a function on it, producing an integer result which we store in memory (an integer being much smaller than the contract payloads). By consuming the stream message by message, we don’t need to read the entire ACS in memory (assuming the ACS is split up over many messages). We are using the rxjava bindings.

I recommend to take a look at Gerolf’s answer in Active Contract Set Service returns a Stream Arrays/List: how big can the arrays/List be per event in the stream? - #6 by gerolf. The Java APIs give you a stream of GetActiveContractsResponse. Usually those are singleton lists. I think under high load it can include more elments up to a constant batch size. I believe that batch size is 1000 from a quick look at the code but @gerolf can probably confirm that.

Ignore my comment on batching, I misread the code. it should always be a singleton list in the current implementation. This is not guaranteed though but I think you can reasonably expect that each stream element can fit in memory so it at most goes up to some reasonable batch size.

Let us try it on a later version. We were using 1.18 to test it. It probably behave differently now. @gerolf , would you know in which version the batch size was added?

Citing @gerolf from the thread @cocreature linked:

Active Contract Set Service returns a Stream Arrays/List: how big can the arrays/List be per event in the stream?

In practice across recent releases of the Daml SDK, you will actually always get a single contract in each GetActiveContractsResponse.

So despite the type indicating it may return multiple elements, it is by no means returning “the entire ACS”.

Does that answer your question?

Thanks for the clarification. That answers the question. It would great to have a little more documentation around this.