Canton 2.8.4

Canton 2.8.4 has been released on April 16, 2024. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.

Summary

This is a maintenance release containing small improvements and bug fixes.

What’s New

Node's Exit on Fatal Failures

When a node encounters a fatal failure that Canton unexpectedly cannot handle gracefully, the new default behavior is that the node will exit/stop the process and relies on an external process or service monitor to restart the node's process. Without this exit, the node would remain in a half-broken state, requiring a manual restart.

Failures that are currently logged as an error but not automatically recovered from are:

1. Unhandled exceptions when processing events from a domain, which currently leads to a disconnect from that domain.
2. Failed transition from an active replica to a passive replica, which may result in an invalid state of the node.

The new behavior can be reverted by setting: canton.parameters.exit-on-fatal-failures = false in the configuration, but we encourage users to adopt the new behaviour.

Minor Improvements

Error code changes

• When an access token expires and ledger api stream is terminated an ABORTED(ACCESS_TOKEN_EXPIRED) error is returned.

Fixed error cause truncation

Before, error causes of the documented errors were truncated after 512 characters. This behaviour is necessary when transporting errors through GRPC, as GRPC imposes size limits, but the limit was also applied to errors in logs, which caused relevant information to be truncated. Now, the 512 bytes limit is not applied to the errors written to the logs anymore.

Configuration Changes

Token Expiry Grace Period for Streams

When a token used in the ledger api request to open a stream expires, the stream is terminated. This normally happens several minutes or hours after the stream initiation. Users can now configure a grace period that will protect the stream termination beyond the token expiry:

   canton.participants.participant1.parameters.ledger-api-server-parameters.token-expiry-grace-period-for-streams=600.seconds

Grace period can be any non-negative duration where both the value and the units must be defined e.g. "600.seconds" or "10.minutes". When parameter is omitted, grace period defaults to zero. When the configured value is Inf the stream is never terminated.

(24-007, Major): Domain reconnect does not complete timely during participant failover

Issue Description

During participant failover the newly active participant does not complete reconnecting to the domain and fail to declare itself as active in a timely manner. This can happen when the passive replica, which has become active, has been idle for a longer time while there has been many transactions processed by the former active replica.

Affected Deployments

Participant

Affected Versions

2.3-2.6, 2.7.0-2.7.7 2.8.0-2.8.3

Impact

During participant failover the newly active participant does not become active and does not serve commands and transactions in a timely manner.

Symptom

"One of the last log entries before the participant only outputs storage health checks anymore is like:

INFO c.d.c.p.s.d.DbMultiDomainEventLog:participant=participant1 Fetch unpublished from domain::122345... up to Some(1234)

The participant reports itself as not active and not connected to any domains in its health status."

Workaround

Restart the participant node.

Likeliness

It depends on how long the passive replica has been idle, the number and size of transactions that the active replica has processed in the meantime. These factors influence how timely the failover and in particular reconnect to the domain completes.

Recommendation

Upgrade to 2.8.4

Compatibility

The following Canton protocol versions are supported:

Dependency	Version
Canton protocol versions	3, 4, 5

Canton has been tested against the following versions of its dependencies:

Dependency	Version
Java Runtime	OpenJDK 64-Bit Server VM Zulu11.70+15-CA (build 11.0.22+7-LTS, mixed mode)
Postgres	Recommended: PostgreSQL 12.18 (Debian 12.18-1.pgdg120+2) – Also tested: PostgreSQL 11.16 (Debian 11.16-1.pgdg90+1), PostgreSQL 13.14 (Debian 13.14-1.pgdg120+2), PostgreSQL 14.11 (Debian 14.11-1.pgdg120+2), PostgreSQL 15.6 (Debian 15.6-1.pgdg120+2)
Oracle	19.20.0