Canton 2.9.1

Canton 2.9.1 has been released on July 15, 2024. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.

Summary

We are excited to announce Canton 2.9.1, which offers additional features and improvements:

• KMS drivers (Beta)
• support for smart contract upgrades (Beta)
• operational improvements around monitoring, liveness, and logging

See below for details.

What’s New

Breaking: Protocol version should be set explicitly

Until now, the configuration of a domain was picking the latest protocol version by default. Since the protocol version is an important parameter of the domain, having this value set behind the scenes caused unwanted behavior.

You now must specify the protocol version for your domain:

myDomain {
  init.domain-parameters.protocol-version = 5
}

For a domain manager:

domainManager {
  init.domain-parameters.protocol-version = 5
}

You can read more about protocol version in the docs. If you are unsure which protocol version to pick:

• Use the last one supported by your binary (see docs).
• Ensure all your environments use the same protocol version: you should not use one protocol version in your test environment and another one in production.

Breaking: Protocol version 3 and 4 discontinuation

This Canton version requires protocol version at least 5.

If your domain is running protocol version 5, you can replace the binaries and apply the database migrations.

If you have a domain running protocol version 3 or 4, you first need to bootstrap a new domain running protocol version at least 5 and then perform a hard domain migration.

Upgrading instructions can be found in the documentation: upgrading manual

KMS Drivers

The Canton protocol relies on a number of cryptographic operations such as asymmetric encryption and digital signatures. To maximize the operational security of a Canton node the corresponding private keys should not be stored or processed in cleartext. A Key Management System (KMS) or Hardware Security Module (HSM) allows us to perform such cryptographic operations where the private key resides securely inside the KMS/HSM. All nodes in Canton can make use of a KMS.

AWS KMS and Google Cloud KMS are supported as of Canton v2.7. To broaden the support of other KMSs and HSMs, Canton v2.9 introduces a plugin approach, called KMS Drivers, which allows the implementation of custom integrations. You can find more information on how to develop a KMS driver in the KMS Driver Guide.

Smart Contract Updates

The feature allows Daml models (packages in DAR files) to be updated on Canton transparently, provided some guidelines in making the changes are followed. For example, you can fix an application bug by uploading the DAR of the fixed package. This is a Beta feature that requires LF 1.16 & Canton Protocol version 6. Please refer to the Daml enterprise release notes for more information on this feature.

Mediator liveness health service and built-in watchdog

Previously a mediator node that had irrecoverably lost its connection to a Canton domain would not exit and would continue to report SERVING on liveness health endpoint. This lead to mediator nodes not being able to automatically recover from unexpected failures. Now sequencer connection status of a mediator node is connected to the liveness health endpoint, allowing for external monitoring and automated intervention (i.e. by setting up k8s liveness probes). Additionally, for systems not using k8s, it is possible to enable a built-in node watchdog that will monitor liveness health endpoint and will forcefully make the node exit if it's no longer alive. By default, the watchdog is disabled and can be enabled by setting the following configuration:

canton.mediators.<mediator_node>.parameters.watchdog = {
    enabled = true
    checkInterval = 15s // default value
    killDelay = 30s // default value
}

Configuration parameters are:

• checkInterval - interval at which the watchdog will check the liveness health endpoint
• killDelay - delay after the watchdog has detected that the node is no longer alive before it forcefully exits the node

Paging in Party Management

Background

Being able to retrieve all parties known to a participant in a paging fashion has been a frequently requested feature. When the number of parties on a participant exceeds tens of thousands, trying to deliver them all in a single message can present many challenges: corresponding db operation can take a long time, internal memory buffers within the participant or the client application can be exhausted, finally, the maximum size of the gRPC message can be exceeded. In extreme cases this could lead to an OOM crash.

Specific Changes

The ListKnownParties method on the PartyManagementService now takes two additional parameters. The new page_size field determines the maximum number of results to be returned by the server. The new page_token field on the other hand is a continuation token that signals to the server to fetch the next page containing the results. Each ListKnownPartiesResponse response contains a page of parties and a next_page_token field that can be used to populate the page_token field for a subsequent request. When the last page is reached, the next_page_token is empty. The parties on each page are sorted in ascending order according to their ids. The pages themselves are sorted as well.

The GetLedgerApiVersion method of the VersionService contains new features.party_management field within the returned GetLedgerApiVersionResponse message. It describes the capabilities of the party management through a sub-message called PartyManagementFeature. At the moment it contains just one field the max_parties_page_size which specifies the maximum number of parties that will be sent per page by default.

Configuration

The default maximum size of the page returned by the participant in response to the ListKnownParties call has been set to 10'000. It can be modified through the max-parties-page-size entry

canton.participants.participant.ledger-api.party-management-service.max-parties-page-size=777

Impact and Migration

The change may have an impact on your workflow if your participant contains more than 10'000 parties and you rely on the results of ListKnownParties containing all parties known to the participant. You will need to do one of two things:

• Change your workflow to utilize a series of ListKnownParties calls chained by page tokens instead of one, This is the recommended approach.
• Change your configuration to increase the maximum page returned by the participant.

Node's Exit on Fatal Failures

When a node encounters a fatal failure that Canton cannot handle gracefully yet, the new default behavior is that the node will exit/stop the process and relies on an external process or service monitor to restart the node's process.

The following failures are considered fatal and now leads to an exit of the process:

1. Unhandled exceptions when processing events from a domain, which previously lead to a disconnect from that domain.
2. Failed transition from an active replica to a passive replica, which may result in an invalid state of the node.
3. Failed transition from a passive replica to an active replica, which may result in an invalid state of the node.

The new behavior can be reverted by setting: canton.parameters.exit-on-fatal-failures = false in the configuration.

Minor Improvements

Logging of Conflict Reason

When a command is rejected due to conflict (e.g. usage of an inactive contract), every participant detecting the conflict will now log the resource causing the conflict at INFO level. This change affects the following error codes: LOCAL_VERDICT_LOCKED_CONTRACTS, LOCAL_VERDICT_LOCKED_KEYS, LOCAL_VERDICT_INACTIVE_CONTRACTS, LOCAL_VERDICT_DUPLICATE_KEY, LOCAL_VERDICT_INCONSISTENT_KEY, LOCAL_VERDICT_CREATES_EXISTING_CONTRACTS

Repair service improvements

• Prevent concurrent execution of a domain (re-)connection while repair operations are in flight.
• Commands ignore_events and unignore_events are now also available on remote nodes.

Error code changes

• PACKAGE_NAMES_NOT_FOUND is introduced for reporting package-name identifiers that could not be found.
• When an access token expires and ledger api stream is terminated an ABORTED(ACCESS_TOKEN_EXPIRED) error is returned.
• DAR_NOT_VALID_UPGRADE is introduced for reporting that the uploaded DAR is not upgrade-compatible with other existing DARs on the participant.
• KNOWN_DAR_VERSION is introduced for reporting that the uploaded DAR name and version is already known to the participant.
• NO_INTERNAL_PARTICIPANT_DATA_BEFORE is introduced and returned when participant.pruning.find_safe_offset is invoked with a timestamp before the earliest known internal participant data.

Remote Log Level Changes

The log levels and last errors can now be accessed remotely from the logging command group on remote consoles.

New Block Sequencer Metrics

As an early access feature, the block sequencer now exposes various labeled metrics that can be used to monitor which node is sending how many submissions to the sequencer. The new metrics have the prefix canton.sequencer.block.

Alpha: Failed Command Inspection

In order to improve debugging of failed commands, the participant now stores the last few commands (successes, failures and pending) in memory for debug inspection. The data is accessible through the command inspection service on the ledger api.

Enterprise HA sequencers expose admin status

HA sequencers only allow one of the sequencers to make administrative changes such as configuring the pruning schedule. Now the sequencer node admin status identifies the admin sequencer via the health status as described in the pruning docs.

Upload DAR checks

DAR uploads via com.digitalasset.canton.participant.admin.v0.PackageService.UploadDar and com.daml.ledger.api.v1.admin.PackageManagementService.UploadDar now check that the main package uploaded is upgrade-compatible with the rest of the packages currently stored on the participant.

Configuration Changes

Changed Create Key Retries configuration

We changed the retry policy for checking the creation of KMS crypto keys to use exponential backoff, so the configuration for the retry-config.create-key-check is now done similarly as the retry-config.failures

canton.participants.participant1.crypto.kms.retries.create-key-check {
      initial-delay = "0.1s",
      max-delay = "10 seconds",
      max-retries = 20,
}

Token Expiry Grace Period for Streams

When a token used in the ledger api request to open a stream expires, the stream is terminated. This normally happens several minutes or hours after the stream initiation. Users can now configure a grace period that will protract the stream termination beyond the token expiry:

   canton.participants.participant1.parameters.ledger-api-server-parameters.token-expiry-grace-period-for-streams=600.seconds

Grace period can be any non-negative duration where both the value and the units must be defined e.g. "600.seconds" or "10.minutes". When parameter is omitted, grace period defaults to zero. When the configured value is Inf the stream is never terminated.

Updates to database sequencer writer and reader configuration defaults

Sequencers with writer.type = high-throughput now have default settings optimized to lower the latency.

Previous configuration is equivalent to specifying:

writer {
    payload-write-batch-max-duration = 50ms
    event-write-batch-max-duration = 50ms
}

New defaults are:

writer {
    payload-write-batch-max-duration = 5ms
    event-write-batch-max-duration = 5ms
}

Multi-instance sequencers with high-availability.enabled = true now have default settings optimized to lower the latency.

Previous defaults are:

high-availability {
    keep-alive-interval = 100ms
    polling-interval = 100ms

New defaults are:

high-availability {
    keep-alive-interval = 50ms
    polling-interval = 50ms
}

Canton Console Changes

• The party migration macros have been improved to be more intuitive and safer to use.
• The new repair method participant.repair.purge_deactivated_domain allows removing data from the deactivated domain after a hard domain migration.
• Repair method participant.repair.add to add contracts is also available on remote nodes.
• Repair method participant.repair.migrate_domain features a force flag. When set true it forces a domain migration ignoring in-flight transactions. See the upgrading manual for details.
• A new connect method connect_single to connect a participant to a domain using a single sequencer connection.

ACS snapshot commands

Console commands that allow to download an ACS snapshot now take a new mandatory argument to indicate whether the snapshot will be used in the context of a party offboarding (party replication or not). This allows Canton to performance additional checks and makes party offboarding safer.

Affected console commands:

• participant.repair.export_acs
• participant.repair_download_acs (deprecated method)

New argument: partiesOffboarding: Boolean.

Compatibility

The following Canton protocol and Ethereum sequencer contract versions are supported:

Dependency	Version
Canton protocol versions	5, 6*

Canton has been tested against the following versions of its dependencies:

Dependency	Version
Java Runtime	OpenJDK 64-Bit Server VM Zulu11.72+19-CA (build 11.0.23+9-LTS, mixed mode)
Postgres	Recommended: PostgreSQL 12.19 (Debian 12.19-1.pgdg120+1) – Also tested: PostgreSQL 11.16 (Debian 11.16-1.pgdg90+1), PostgreSQL 13.15 (Debian 13.15-1.pgdg120+1), PostgreSQL 14.12 (Debian 14.12-1.pgdg120+1), PostgreSQL 15.7 (Debian 15.7-1.pgdg120+1)
Oracle	19.20.0