Why is the offset sent last in the GRPC ACS rpc?

I’m using the ledger API directly to fetch some contracts, and I observe that the GRPC messages returned consists of

1. One or more responses with empty offset and non-empty activeContracts
2. A final response with a non-empty offset and empty activeContracts.

Here’s a discussion describing this:

Offset is null while getting Active contracts Questions

Hi, we are using the LedgerClient(gRPC) to retrieve current active contracts. As per documentation the response object should contain an offset value(unless the ledger is not empty). In our case this value is coming as null always eventhough we have some data in the ledger. This behavior is not reproducible for Sandbox but it is for Daml on Postgres. What are the cases when the offset may come as null? We are using daml version 1.11.1

I think there is a design flaw here which can cause resource issues on the client side.

The problem is that you have to buffer the entire acs, potentially 1000’s of contracts, because you won’t know the offset until the end. This is most obvious if for example, you request the starting point as LEDGEREND. Then you can’t use this to e.g. exercise a choice and provide an offset for deduplication purposes (The LEDGEREND may have changed by the time you send out your transaction).

More generally, if you want to synchronize two concurrent processes that are working in a different context i.e. ledger offset, you need to know the offset of both processes. There are ways to mitigate this - for example: request the ACS for the second process with the known offset from the first - but then you are making assumptions about the internals of the ledger API, and you need to make assertions after the fact to ensure the offset you received matches what you requested (I don’t believe there is any guarantee of this, from reading the docs - happy to be proved wrong though). (This still doesn’t solve the LEDGEREND problem, when starting without any ledger context - you’d need to request a ‘Dummy’ templateId to discover the ledger boundary first).

What do you think?

Is this a valid argument for requesting to change the semantics of this rpc in the future?

You raise some valid points. One potential fix may be [DPP-1022] ACS active_at_offset by pbatko-da · Pull Request #15764 · digital-asset/daml · GitHub which will let you query the ACS at a given offset. Then you can query ledger end and then query ACS at that offset and you clearly know the offset upfront.

Just including the offset in the first message would be a breaking change so we would need to encode this is a backwards compatible way (a different offset field for example) which would work but arguably is more clunky than request offset first + query acs at that offset.