Canton 2.8.3

Canton 2.8.3 has been released on February 21, 2024. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.

Summary

This is a maintenance release, fixing critical issue which can occur if overly large transactions are submitted to the participant. The bootstrap domain command has been slightly improved, but it may now throw an error if you try to bootstrap an already initialized domain while the domain manager node is still starting up.

Console changes

Console commands that allow to download an ACS snapshot now take a new mandatory argument to indicate whether the snapshot will be used in the context of a party offboarding (party replication or not). This allows Canton to performance additional checks and makes party offboarding safer.

Affected console commands:

• participant.repair.export_acs
• participant.repair_download_acs (deprecated method)

New argument: partiesOffboarding: Boolean.

Bugfixes

(24-003, Moderate): Cannot exercise keyed contract after party replication

Issue Description

When replicating a party who is a maintainer on a contract key, the contract cannot be exercised anymore

Affected Deployments

Participant

Affected Versions

• 2.7
• 2.8.0-2.8.1

Impact

Contract become unusable

Symptom

The following error is emitted when trying to exercise a choice on the contract:

java.lang.IllegalStateException:
Unknown keys are to be reassigned. Either the persisted ledger state corrupted
or this is a malformed transaction. Unknown keys:

Workaround

None.

Likeliness

Deterministic.

Recommendation

Do not use the party migration macros with contract keys in version 2.8.1. Upgrade to 2.8.2 if you want to use them.

(24-002, Critical): Looping DB errors for transactions with many (32k) key updates

Issue Description

If a transaction with a very large number of key updates is submitted to the participant, the SQL query to update the contract keys table will fail, leading to a database retry loop, stopping transaction processing. The issue is caused by a limit of 32k rows in a prepared statement in the JDBC Postgres driver.

Affected Deployments

Participant on Postgres

Affected Versions

2.3-2.6, 2.7.0-2.7.6 2.8.0-2.8.1

Impact

Transactions can be submitted to the participant, but the transaction stream stops emitting transactions and commands don't appear to succeed.

Symptom

A stuck participant, continuously logging about "Now retrying operation 'com.digitalasset.canton.participant.store.db.DbContractKeyJournal.addKeyStateUpdates'" and "Tried to send an out-of-range integer as a 2-byte value: 40225"

Workaround

Upgrade to a version containing the fix. Alternatively, such a transaction may be ignored, but this must be done with care to avoid a ledger fork in case of multiple involved participants. Contact support.

Likeliness

Deterministic with very large (32k key updates) transactions.

Recommendation

Upgrade at your convenience or if experiencing the error.

(24-004, Major): DB write operations fail due to read-only connections as part of a DB failover

Issue Description

Given a Postgres HA setup with a write and read replica using AWS RDS. During DB failover, Canton connects and remains connected to the former write replica, which results in the DB connections being read-only. This fails all write DB operations on which we retry indefinitely.

Affected Deployments

All nodes with Postgres HA. Only observed with AWS RDS so far, not on Azure.

Affected Versions

All 2.3-2.6 2.7.0-2.7.6 2.8.0-2.8.1

Impact

A node becomes unusable due to all DB write operations failing and being retried indefinitely.

Symptom

The following kind of exceptions are logged: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction

Workaround

Restart the node to reconnect to the current write replica. Set DNS TTL in JVM networkaddress.cache.ttl=0 reduces the likelihood of this problem, but disables DNS caching entirely

Likeliness

Probable in case of DB failover, only observed in AWS RDS so far.

Recommendation

Upgrade to 2.8.2 if you are using AWS RDS

(24-005, Major): Race condition during domain manager startup may prevent startup after upgrade

Issue Description

A race condition in the domain bootstrap macro and domain manager node initialisation may result in a domain manager overriding the previously stored domain parameters. This can cause the domain manager to be unable to reconnect to the domain after upgrading to version 2.7 or beyond without a hard domain migration to pv=5, as with 2.7, the default protocol version changed to pv=5.

Affected Deployments

Domain Manager Nodes

Impact

The domain manager node is unable to connect to the sequencer. Topology transactions such as party additions have no effect on the domain, as they are not verified and forwarded by the domain manager.

Symptom

Parties added to the participant do not appear on the domain. Transactions referring to such parties are rejected with UNKNOWN_INFORMEE. The domain manager logs complain about "The unversioned subscribe endpoints must be used with protocol version 4"

Workaround

If you don't run bootstrap_domain after the domain was already initialised, the issue won't happen. If a node is affected, you can recover the node explicitly by seting the protocol version to v=4 in the configuration and use the reset-stored-static-config parameter to correct the wrongly stored parameters.

Likeliness

Race condition which can happen if you run the bootstrap_domain command repeatedly on an already bootstrapped domain during startup of the domain manager node. As there is no need to run the bootstrap_domain command repeatedly, this issue can easily be avoided.

Recommendation

Do not run the bootstrap_domain command after the domain was initialised. A fix has been added to prevent future mistakes.

(24-006, Minor): Large error returned from ACS query cannot be stuffed into HTTP2 headers

Issue Description

A bad ledger api client request results in an error that contains a large metadata resource list and fails the serialization in the netty layer. As a consequence what is sent to the client is an INTERNAL error that doesn't correspond to the actual problem.

Affected Deployments

Participant Nodes

Impact

Incorrect error returned to the ledger client which may complicate trouble-shooting

Symptom

Ledger client observes following error:

INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR

At the same time the participant node reports following error:

io.grpc.netty.NettyServerHandler - Stream Error
io.netty.handler.codec.http2.Http2Exception$HeaderListSizeException: Header size exceeded max allowed size (8192)

Workaround

Rely on participant logs when investigating ledger client issues with logs containing the message

INTERNAL: RST_STREAM closed stream. HTTP/2 error code: PROTOCOL_ERROR

Likeliness

It is rather difficult for ledger api clients to create bad requests that will end up in errors with long list of resources. One example is when client requests a long list of non existing templates in a filter of a ACS ledger api request. There was an unrelated bug in the trigger service that could produce long list of templates when AllInDar construction was used in the trigger installation invocation.

Recommendation

Upgrade at your convenience.

Compatibility

The following Canton protocol versions are supported:

Dependency	Version
Canton protocol versions	3, 4, 5

Canton has been tested against the following versions of its dependencies:

Dependency	Version
Java Runtime	OpenJDK 64-Bit Server VM Zulu11.66+15-CA (build 11.0.20+8-LTS, mixed mode)
Postgres	Recommended: PostgreSQL 12.18 (Debian 12.18-1.pgdg120+2) – Also tested: PostgreSQL 11.16 (Debian 11.16-1.pgdg90+1), PostgreSQL 13.14 (Debian 13.14-1.pgdg120+2), PostgreSQL 14.11 (Debian 14.11-1.pgdg120+2), PostgreSQL 15.6 (Debian 15.6-1.pgdg120+2)
Oracle	19.20.0