Canton 2.8.0

Canton 2.8.0 has been released on December 15, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.

Summary

We are excited to announce Canton 2.8.0, which offers some great additional features:

• distributed tracing enhancements
• event query service
• many operational, quality, security and performance improvements.

See below for details.

Please upgrade to this version as soon as possible. Version 2.7 will not be maintained after March 2024.

What’s New

Explicit contract disclosure is now available in Beta

Background

Explicit contract disclosure allows you to delegate contract read rights to a non-stakeholder using off-ledger data distribution.

The feature is now available in Beta and enabled by default. or more information, see the documentation.

Specific Changes

• The participant.ledger-api.explicit-disclosure-unsafe feature flag is replaced by participant.ledger-api.enable-explicit-disclosure which is by default set to true. To disable the feature, set the new flag to false.

Distributed Tracing Enhancements

Background

Distributed tracing is a technique for troubleshooting performance issues in a microservices environment like Daml Enterprise. Canton supports distributed tracing, and the support on the Ledger API is improving. In this release, the client applications gain the ability to extract trace and span IDs from past transactions and completions, so that distributed traces can continue in follow-up commands.

To learn how to extend your application to support distributed tracing, see the documentation.

Specific Changes

Trace contexts are now included in the gRPC messages return in Ledger API streams and point-wise queries. This change affects the following transaction and command completion service calls:

• TransactionService.GetTransactions
• TransactionService.GetTransactionTrees
• TransactionService.GetTransactionByEventId
• TransactionService.GetTransactionById
• TransactionService.GetFlatTransactionByEventId
• TransactionService.GetFlatTransactionById
• CompletionService.CompletionStream

Trace context enables client applications that were not the submitters of the original request to pick up the initial spans and continue them in follow-up requests. This allows multi-step workflows to be adorned with contiguous chains of related spans.

Impact and Migration

This is a purely additive change.

Event Query Service is now available as Beta

Background

Use the event query service to obtain a party-specific view of contract events.

The gRPC API provides ledger streams to off-ledger components that maintain a queryable state. This service allows you to make simple event queries without off-ledger components like the JSON API.

Using the event query service, you can create, retrieve, and archive events associated with a contract ID or contract key. The API returns only those events where at least one of the requesting parties is a stakeholder of the contract. If the contract is still active, the archive_event is unset.

Contract keys can be used by multiple contracts over time. The latest contract events are returned first. To access earlier contract key events, use the continuation_token returned in the GetEventsByContractKeyResponse in a subsequent GetEventsByContractKeyRequest.

If no events match the request criteria or the requested events are not visible to the requesting parties, an empty structure is returned. Events associated with consumed contracts are returned until they are pruned.

Specific Changes

• Retrieve contract events by contract ID via the GetEventsByContractId request.
• Retrieve contract events by contract key via the GetEventsByContractKey request.

Impact and Migration

This is a purely additive change.

Automatic participant pruning support for pruning internal-only state

Background

Pruning presents a trade-off between limiting ledger storage space and being able to query ledger history far into the past. Only pruning participant-internal state strikes a balance by deleting exclusively internal state. As a result, applications can continue to query historic portions of the ledger, but internal-only pruning frees up less storage space than regular pruning. Previously, internal pruning was available only via the "manual" participant.pruning.prune_internally command. With this release, pruning participant internal-only state also becomes available through automatic pruning as well.

Specific Changes

• Configure automatic, internal-only pruning using the new participant.pruning.set_participant_schedule command's prune_internally_only parameter.
• Retrieve the currently active participant schedule including the prune_internally_only setting via the newly introduced participant.pruning.get_participant_schedule command.

Impact and Migration

This is a purely additive change.

Minor Changes

Logging Cleanup

We've cleaned up our transaction processing logging to make it easier to understand what is happening. Now, significant events on all nodes are logged at INFO levels and including relevant information.

Ledger API Command Submission Changes

CommandService gRPC deadline logic

Commands submitted to the Ledger API now respects the grpc deadlines: If a request reaches the command processing layer with an already-expired gRPC deadline, the command will not be sent for submission. Instead, the request is rejected with a new self-service error code REQUEST_DEADLINE_EXCEEDED, which informs the client that the command is guaranteed not to have been sent for execution to the ledger.

Command Interpretation Timeouts

If you submit a command that runs for a very long time, the Ledger API will now reject the command with the new self-service error code INTERPRETATION_TIME_EXCEEDED when the transaction would reach the ledger time tolerance limit based on the submission time.

Protocol versions 3 and 4 are deprecated

Protocol versions 3 and 4 are now marked as deprecated and will be removed in the next minor release of Canton. Protocol version 5 should be preferred for any new deployment.

Configuration Changes

Breaking: KMS wrapper-key configuration value now accepts a simple string

The expected KMS wrapper-key configuration value has changed from:

    crypto.private-key-store.encryption.wrapper-key-id = { str = "..."}

to a simple string:

    crypto.private-key-store.encryption.wrapper-key-id = "..."

Parallel Node Startup

Nodes will now start up in parallel. The startup parallelism can be configured by setting:

    canton.parameters.startup-parallelism = 1 // defaults to number of threads.

Breaking: Schema migration attempts configuration for the indexer

The configuration fields schema-migration-attempt-backoff and schema-migration-attempts for the indexer were removed. The following config lines will have to be removed, if they exist:

participants.participant.parameters.ledger-api-server-parameters.indexer.schema-migration-attempt-backoff
participants.participant.parameters.ledger-api-server-parameters.indexer.schema-migration-attempts

Breaking: Cache weight configuration for the Ledger API server

The configuration fields max-event-cache-weight and max-contract-cache-weight for the ledger api server were removed. The following config lines will have to be removed, if they exist:

participants.participant.ledger-api.max-event-cache-weight
participants.participant.ledger-api.max-contract-cache-weight

Config Logging On Startup

By default, Canton will log the config values on startup, as this has turned out to be useful for troubleshooting, in order to understand what configuration is related to the given log-file.

This feature can be turned off by setting

    canton.monitoring.logging.log-config-on-startup = false

Logging the configuration including all default values can be turned on using

    canton.monitoring.logging.log-config-with-defaults = true

Note that this will log all available settings, which includes parameters which we do not recommend changing. Confidential data will not be logged but replaced by xxxx.

Mediator Finalized Response Cache

The mediator finalized response cache can now be configured using:

    canton.mediators.mediator.caching.finalized-mediator-requests = 1000 // default

which allows you to limit memory consumption in the presence of a transaction with a very large number of views.

Default Size of Ledger API Caches

The default sizes of the contract state and contract key state caches has been decreased by one order of magnitude from 100'000 to 10'000.

   canton.participants.participant.ledger-api.index-service.max-contract-state-cache-size
   canton.participants.participant.ledger-api.index-service.max-contract-key-state-cache-size

The size of these caches determines the likelihood that a transaction using a contract/contract-key that was recently created or read will still find it in memory rather than need to query it from the database. Larger caches might be of interest in use cases where there is a big pool of ambient contracts that are consistently being fetched or used for non consuming exercises. It may also benefit those use cases where a big pool of contracts is being rotated through a create -> archive -> create-successor cycle. Consider adjusting these parameters explicitly if the performance of your specific workflow depends on large caches, and you were relying on the defaults thus far. Beware that increasing these caches will increase the memory consumption of the Ledger API, which in turn might lead to garbage collection pauses if you run the node with insufficient heap memory reserves.

Default Number of Max Transactions in the In-Memory Fan-Out

The default maximum number of transactions stored in the in-memory fan-out has been decreased by one order of magnitude from 10'000 to 1'000.

   canton.participants.participant.ledger-api.index-service.max-transactions-in-memory-fan-out-buffer-size

The in-memory fan-out allows serving the transaction streams from memory as they are finalized, rather than using the database. You should choose this buffer to be large enough such that the likeliness of applications having to stream transactions from the database is low. Generally, having a 10s buffer is sensible. Therefore, if you expect e.g. a throughput of 20 tx/s, then setting this number to 200 is sensible. The new default setting of 1000 assumes 100 tx/s. Consider adjusting these parameters explicitly if the performance of your specific workflow foresees transaction rates larger than 100 tx/s.

Target Scope for JWT Authorization

The default scope (scope field in the scope based token) for authenticating on the Ledger API using JWT is daml_ledger_api. Other scopes can be configured explicitly using the custom target scope configuration option:

   canton.participants.participant.ledger-api.auth-services.0.target-scope="custom/Scope-5:with_special_characters"

Target scope can be any case-sensitive string containing alphanumeric characters, hyphens, slashes, colons and underscores. Either the target-scope or target-audience parameter can be configured, but not both.

SQL Batching Parameter has been moved

The expert mode sql batching parameter

  canton.participants.participant.parameters.stores.max-items-in-sql-clause

has been moved to:

  canton.participants.participant.parameters.batching.max-items-in-sql-clause

Generally, we recommend to not change this parameter unless advised by support.

Explicit Settings for Database Connection Pool Sizes

The sizes of the connection pools used for interactions with database storage inside Canton nodes are determined using a dedicated formula described in the documentation article on max connection settings:

The values obtained from that formula can now be overridden using explicit configuration settings for the read, write and ledger-api connection pool sizes:

canton.participants.participant.storage.parameters.connection-allocation.num-reads
canton.participants.participant.storage.parameters.connection-allocation.num-writes
canton.participants.participant.storage.parameters.connection-allocation.num-ledger-api

Similar parameters are available for other Canton node types:

canton.sequencers.sequencer.storage.parameters.connection-allocation...
canton.mediators.mediator.storage.parameters.connection-allocation...
canton.domain-managers.domain_manager.storage.parameters.connection-allocation...

The effective connection pool sizes are reported by the Canton nodes at start-up

INFO  c.d.c.r.DbStorageMulti$:participant=participant_b - Creating storage, num-reads: 5, num-writes: 4

Console Changes

Ledger API commands

Usage of the applicationId in command submissions and completion subscriptions

Previously, the Canton console used a hard-coded "CantonConsole" as an applicationId in the command submissions and the completion subscriptions performed against the Ledger API. Now, if an access token is provided to the console, it will extract the userId from that token and use it instead. A local console will use the adminToken provided in canton.participants.<participant>.ledger-api.admin-token, whereas a remote console will use the token from canton.remote-participants.<remoteParticipant>.token

This affects the following console commands:

• ledger_api.commands.submit
• ledger_api.commands.submit_flat
• ledger_api.commands.submit_async
• ledger_api.completions.list
• ledger_api.completions.list_with_checkpoint
• ledger_api.completions.subscribe

You can also override the applicationId by supplying it explicitly to these commands.

Introduction of Java Bindings Compatible Console Commands

The following console commands were added to support actions with java codegen compatible data:

• participant.ledger_api.javaapi.commands.submit
• participant.ledger_api.javaapi.commands.submit_flat
• participant.ledger_api.javaapi.commands.submit_async
• participant.ledger_api.javaapi.transactions.trees
• participant.ledger_api.javaapi.transactions.flat
• participant.ledger_api.javaapi.acs.await
• participant.ledger_api.javaapi.acs.filter
• participant.ledger_api.javaapi.event_query.by_contract_id
• participant.ledger_api.javaapi.event_query.by_contract_key

The following commands were replaced by their java bindings compatible equivalent (in parentheses):

• participant.ledger_api.acs.await (participant.ledger_api.javaapi.acs.await)
• participant.ledger_api.acs.filter (participant.ledger_api.javaapi.acs.filter)

Please note that the Scala codegen and bindings are not an officially supported feature and will be removed in a future release. These commands here are provided for your convenience but are subject to change.

New Functions to Specify a Full-blown Transaction Filter for Flat Transactions

ledger_api.transactions.flat_with_tx_filter and ledger_api.javaapi.transactions.flat_with_tx_filter are more sophisticated alternatives to ledger_api.transactions.flat and ledger_api.javaapi.transactions.flat respectively that allow to specify a full transaction filter instead of a set of parties. Consider using this if you need to specify more fine-grained filters that include template IDs, interface IDs, and/or whether you want to retrieve create event blobs for explicit disclosure.

Repair Commands

Commands around ACS migration

Console commands for ACS migration can now be used with remote nodes. This change applies to the commands in the repair namespace.

New ACS export / import repair commands

The new ACS export / import commands, repair.export_acs and repair.import_acs provide similar functionality as the existing repair.download and repair.upload commands. However, its implementation allows to evolve it better over time.

Consequently, the existing download / upload functionality has been deprecated for 2.8.0 and is going to be removed with a next release.

Transactions generated by importing an ACS have a configurable workflow ID to track ongoing imports

Contract added via the repair.party_migration.step2_import_acs and repair.import_acs commands now include a workflow ID. The ID is in the form prefix-${n}-${m}, where m is the number of transactions generated as part of the import process and n is a sequential number from 1 to m inclusive. Each transaction contains 1 or more contracts that share the ledger time of their creation. The two numbers allow you to track whether an import is being processed. You can specify a prefix with the workflow_id_prefix string parameter defined on both commands. If not specified, the prefix defaults to import-${randomly-generated-unique-identifier}.

Key Management Commands

keys.secret.rotate_node_key() console command

The console command keys.secret.rotate_node_key can now accept a name for the newly generated key.

Breaking: owner_to_key_mappings.rotate_key command expects a node reference

The previous owner_to_key_mappings.rotate_key is deprecated and now expects a node reference (InstanceReferenceCommon) to avoid any dangerous and/or unwanted key rotations.

Miscellaneous Commands

Domain filtering in testing commands

To improve consistency and code safety, some testing console commands now expect an optional domain alias (rather than a plain domain alias). For example, the following call

participant.testing.event_search("da")

needs to be rewritten to

participant.testing.event_search(Some("da"))

The impacted console commands are:

• participant.testing.event_search
• participant.testing.transaction_search

DAR vetting and unvetting commands

DAR vetting and unvetting convenience commands have been added to:

• Canton admin API as PackageService.VetDar and PackageService.UnvetDar
• Canton console as participant.dars.vetting.enable and participant.dars.vetting.disable

Additionally, two error codes have been introduced to allow better error reporting to the client when working with DAR vetting/unvetting: DAR_NOT_FOUND and PACKAGE_MISSING_DEPENDENCIES. Please note that these commands are alpha only and subject to change.

Deprecations

• SequencerConnection.addConnection is deprecated. Use SequencerConnection.addEndpoints instead.
• repair.download and repair.upload Console Commands are deprecated. Use repair.export_acs and repair.import_acs commands instead.
• Removed obsolete dar sharing service.

Metrics Changes

• The DB metric lookup_active_contracts is removed in favor of lookup_created_contracts and lookup_archived_contracts. This reflects the change of active contract lookup from DB: switching for a single batched active DB query to two parallel executed batch queries targeting archived and created events.
• The sequencer client metric load is removed without replacement, as it was not measuring anything meaningful.

Error Code Changes

Breaking: Submission service error code change

The error code SEQUENCER_DELIVER_ERROR that could be received when submitting a transaction has been superseded by two separate new error codes: SEQUENCER_SUBMISSION_REQUEST_MALFORMED and SEQUENCER_SUBMISSION_REQUEST_REFUSED. Please migrate client applications code if you rely on the older error code.

Packaging

We have reverted the change from 2.7.0 that introduced the distribution of a separate bcprov-jdk15on-1.70.jar along the canton jar in the lib folder. This revert was also enacted in the version 2.7.1. If you use the Oracle JRE, beware that you will now need to explicitly add the bouncycastle library to the classpath when running canton.

Faster emission of command rejections

Some commands that the participant rejected locally, e.g., command deduplication errors, produced a rejection completion only after the participant observed some other traffic from the domain. Such commands now produce the rejection completion immediately.

Bugfixes

(23-023, Critical): Crash recovery issue in command deduplication store

Issue Description

On restart of a sync domain, the participant will replay pending transactions, updating the stores in case some writes were not persisted. Within the command deduplication store, existing records are compared with to be written records for internal consistency checking purposes. This comparison includes the trace context which differs on a restart and hence can cause the check to fail, aborting the startup with an IllegalArgumentException.

Affected Deployments

Participant

Impact

An affected participant can not reconnect to the given domain, which means that transaction processing is blocked.

Symptom

Upon restart, the participant refuses to reconnect to the domain, writing the following log message: ERROR c.d.c.p.s.d.DbCommandDeduplicationStore ... - An internal error has occurred. java.lang.IllegalArgumentException: Cannot update command deduplication data for ChangeIdHash

Workaround

No workaround exists. You need to upgrade to a version not affected by this issue.

Likelihood

The error happens if a participant crashes with a particular state not yet written to the database. The bug has been present since end of Nov 21 and has never been observed before, not even during testing.

Recommendation

Upgrade during your next maintenance window to a patch version not affected by this issue.

(23-022, Major): - rotate_keys command is dangerous and does not work when there are multiple sequencers

Issue Description

We allow the replacement of key A with key B, but we cannot guarantee that the node using key A will actually have access to key B.

Furthermore, when attempting to rotate the keys of a sequencer using the rotate_node_keys(domainManagerRef) method, it will fail if we have more than one sequencer in our environment. This occurs because they share a unique identifier (UID), and as a result, this console command not only rotates the keys of the sequencer it is called on but also affects the keys of the other sequencers.

Modified the process of finding keys for rotation in the rotate_node_keys(domainManagerRef) function to prevent conflicts among multiple sequencers that share the same UID. Additionally, we have updated the console command owner_to_key_mappings.rotate_key to expect a node reference (InstanceReferenceCommon), thereby ensuring that both the current and new keys are associated with the correct node.

Affected Deployments

All nodes (but mostly sequencer)

Impact

A node can mistakenly rotate keys that do not pertain to it. Using rotate_node_keys(domainManagerRef) to rotate a sequencer's keys when other sequencers are present will also fail and break Canton.

Symptom

When you try to rotate a sequencer's key, it catastrophically fails with a java.lang.RuntimeException: KeyNotAvailable.

Workaround

For rotate_node_keys(domainManagerRef), we have ensured that we filter the correct keys for rotation by checking both the authorized store and the local private key store.

Additionally, we have deprecated the existing owner_to_key_mappings.rotate_key and introduced a new method that requires the user to provide the node instance for which they intend to apply the key rotation. We have also implemented a validation check within this function to ensure that the current and new keys are associated with this node.

Likelihood

Every use of rotate_node_keys to rotate the keys of sequencer(s) in a multiple sequencer environment.

Recommendation

Upgrade the Canton console that you use to administrate the domain, in particular the sequencer and mediator, to a Canton version with the bug fix.

(23-019, Minor): Fixed rotate_node_keys command when it is used to rotate keys of sequencer(s) and mediator(s)

Issue Description

Canton has a series of console commands to rotate keys, in particular, rotate_node_keys that is used to rotate the keys of a node.

When we use the console macro rotate_node_keys to rotate the keys of a sequencer or mediator the keys are actually not rotated because we are: (1) looking at the wrong topology management store and (2) not using the associated domainManager to actually rotate the keys.

Affected deployments

Mediator and Sequencer nodes.

Impact

The rotation of the keys for a sequencer or mediator will not succeed.

Workaround

Select the correct topology management Active store (the one from the domainManager) to look for keys to rotate and call the rotate_key console command with the domainManager reference.

Likelihood

Everytime we use rotate_node_keys to rotate the keys of sequencer(s) or mediator(s).

Recommendation

Upgrade the Canton console that you use to administrate the domain, in particular the sequencer and mediator, to a Canton version with the bug fix.

(23-020, Minor): Core contract input stakeholders bug

Background

The Canton protocol includes in ViewParticipantData contracts that required to re-interpret command evaluation. These inputs are known as coreInputs. Included as part of this input contract data is contract metadata the includes details of signatories and stakeholders.

Issue Description

Where the evaluation of a command results in a contract key being resolved to a contract identifier but that identifier is not in turn resolved to a contract instance that distributed metadata associated with contract will incorrectly have the key maintainers as both the signatory and the stakeholders. A way to do this would be to execute a choice on a contract other than the keyed contract that only issues a lookupByKey on the keyed contract.

Affected protocol versions

This problem is observed in canton protocol version 4 and fixed in version 5.

Impact

The impact of this bug is invalid contract metadata is distributed with transaction views.

Workaround

A workaround would be to ensure that whenever a contract key is resolved to a contract identifier that identifier is always resolved to a contract (even if not needed). For example following the lookupByKey if the case where a contract identifier is returned the issue a fetch command on this identifier, discarding the result.

Likelihood

Unlikely. Most of the time a contract key is resolved to a contract so that some action can be performed on that contract. In this situation the metadata would be correct. The only situation this has occurred observed is in test scenarios.

Recommendation

The recommendation is that no action is required. The problem will be naturally fixed in protocol version 5 and the workaround above can be used in the unlikely case it is observed.

(23-021, Minor): Transaction view decomposition bug

Background

Transaction view decomposition is the process of taking a transaction and generating a view hierarchy whereby each view has a common set of informees. Each view additionally has a rollback scope. A child view having a rollback scope that is different from that of the parent indicates that any changes to the liveness of contracts that occurred within the child view should be disregarded upon exiting the child view. For example it would be valid for a contract to be consumed in a rolled back child view and then consumed again in a subsequent child view.

Issue Description

As the activeness of contracts is preserved across views with the same rollback scope every source transaction rollback node should be allocated a unique scope. In certain circumstances this is not happening resulting in contract activeness inconsistency.

Effected protocol versions

This problem is observed in canton protocol version 4 and fixed in version 5.

Impact

The impact of this bug is that a inconsistent transaction view hierarchy can be generated from a consistent transaction. This in turn can result in a valid transaction being rejected. The symptoms of this would be the mediator rejecting a valid transaction request on the basis of inconsistency.

Workaround

This bug only effects transactions that contain rollbacks, if use of rollbacks can be avoided this bug will not occur.

Likelihood

To encounter this bug requires a transaction that has multiple rolled back nodes in which overlapping contracts and/or keys are used. For this reason the likelihood of encountering the bug is low.

Recommendation

If this bug was to effect a client development then the best course of action would be to finalize and release canton protocol 5.

(23-024, Moderate) Participant state topology transaction may be silently ignored during cascading update

Issue Description

In some cases, participant state and mediator domain state topology transactions were silently ignored when they were sent as part of a cascading topology update (which means they were sent together with a namespace certificate). As a result, the nodes had a different view on the topology state and not all daml transactions could be run.

Affected Deployments

All nodes

Impact

A participant node might consider another participant node as inactive and therefore refuse to send transactions or invalidate transactions.

Symptom

A Daml transaction might be rejected with UNKNOWN_INFORMEES.

Workaround

Flush the topology state by running domain.participants.set_state(pid, Submission, Vip (and back to Ordinary)). This will run the update through the "incremental" update code path that is behaving correctly, which fixes the topology state of the broken node.

Likelihood

The bug is deterministic and can occur when using permissioned domains, when the participant state is received together with the namespace delegation of the domain but without the namespace delegation of the participant.

Recommendation

Upgrade to this version if you intend to use permissioned domains. If you need to fix a broken system, then upgrade to a version fixing the issue and apply the work-around to ""flush"" the topology state.

(23-025, Minor) PingService stops working after a LedgerAPI crash

After an Indexer restart in the Ledger API or any error causing the client transaction streams to fail, the PingService stops working.

Affected Deployments

The participant node

Impact

When the Ledger API encounters an error that leads to cancelling the client connections while the participant node does not become passive, the PingService cannot continue processing commands.

Symptom

Ping commands issued in the PingService are timing out. Additionally, the participant might appear unhealthy if configured to report health by using the PingService (i.e. configured with monitoring.health.check.type = ping).

Workaround

Restart the participant node.

Likelihood

This bug occurs consistently when there is an error in the Ledger API, such as a DB overloaded issue that causes the Ledger API Indexer to restart. For this bug to occur, the participant node must not transition to passive state. If it transitions to passive and then back to active, the bug should not reproduce.

Recommendation

If the system is subject to frequent transient errors in the Ledger API (e.g. flaky Index database) or consistently high load, update to this version in order to avoid reproducibility.

(23-026, Minor) Non-graceful shutdown of the participant or the Ledger API

Participant may shut down ungracefully if there are still completing CommandService submissions or, in extreme cases, the Ledger API can restart during normal operations.

Affected Deployments

The participant node

Impact

No significant operational impact.

Symptoms

• On shutdown, an exception including IllegalStateException("Promise already completed.") is logged.
• Pending CommandService submissions are not completed gracefully with SERVER_IS_SHUTTING_DOWN.
• In extreme cases, the issue can trigger a Ledger API restart during normal operation.

Workaround

Not applicable as the effect is mostly aesthetic.

Likelihood

This ungraceful shutdown is only likely under heavy usage of CommandService at the same time with the participant shutdown. The likelihood of this bug triggering a Ledger API restart is very small as multiple conditions need to be met:

• Submissions with the same (submissionId, commandId, applicationId, submitters) change key are sent concurrently to both the CommandSubmissionService and the CommandService.
• The chosen deduplication duration for the submissions is small enough to allow them to succeed within a small timeframe (rather concurrently).

Recommendation

Upgrade when convenient.

(23-027, Minor) Expired gRPC request deadlines crash requests in CommandService

The CommandService errors out when confronted with an expired gRPC request deadline.

Affected Deployments

The participant node

Impact

If encountered repeatedly (up to the maximum-in-flight configuration limit for the CommandService), the CommandService can appear saturated and reject new commands.

Symptoms

When a command request is submitted via CommandService.submitAndWait and its variants with providing a gRPC request deadline, the request can fail with an INTERNAL error reported to the client and a log message with ERROR level is logged on the participant.

Workaround

Restart the participant and use a higher gRPC request deadline.

Likelihood

This bug is likely to happen if the gRPC request deadline is small enough for it to expire upon arriving at the participant's Ledger API.

Recommendation

Upgrade when convenient.

(23-028, Minor) PingService completions re-subscription loop

A Ledger API restart following a pruning event can lead to a non-functioning PingService

Affected Deployments

The participant node

Impact

The PingService is not functional after this error is encountered.

Symptoms

• Ping commands issued in the PingService are timing out.
• The participant node is continuously logging re-subscription errors with PARTICIPANT_PRUNED_DATA_ACCESSED every one second.
• Additionally, the participant might appear unhealthy if configured to report health by using the PingService.

Workaround

Restart the participant node.

Likelihood

This bug is likely when in the same participant active session, a pruning request is followed by Ledger API restart (caused by transient errors such as a DB connectivity). For this bug to occur, the participant node must not transition to passive state. If it transitions to passive and then back to active, the bug should not reproduce.

Recommendation

Upgrade when convenient.

Minor UX fixes

• Added a log warning on scheduled background pruning problems, so that persistent pruning issues are more easily detected by Canton node operators.
• Invalid or duplicate node names in configuration files will now be reported on startup before exiting.

Compatibility

The following Canton protocol versions are supported:

Dependency	Version
Canton protocol versions	3, 4, 5

Canton has been tested against the following versions of its dependencies:

Dependency	Version
Java Runtime	OpenJDK 64-Bit Server VM Zulu11.66+15-CA (build 11.0.20+8-LTS, mixed mode)
Postgres	Recommended: PostgreSQL 12.15 (Debian 12.15-1.pgdg120+1) – Also tested: PostgreSQL 11.16 (Debian 11.16-1.pgdg90+1), PostgreSQL 13.12 (Debian 13.12-1.pgdg120+1), PostgreSQL 14.8 (Debian 14.8-1.pgdg120+1), PostgreSQL 15.3 (Debian 15.3-1.pgdg120+1)
Oracle	19.20.0

Note: We no longer test with Postgres 10 as its support has been discontinued.

What's Coming

We are currently working on

• zero downtime distributed smart contract upgrading (for 2.9)
• multi-domain general availability (per 3.0)
• performance improvements (for 3.0)