DamlLedgerClient grpc issues

Hi,

We have begun getting the below error messages when making use of the ledger client. It has manifested itself in a way in that making a query on any ledgerclient we have, results in the error appearing in all ledger clients connected to the same domain. We believe this is a consequence of another issue within canton as certain commands submissions via the client fail silently when we start seeing these.

Versions
canton: 2.3.2
java bindings: 2.3.2

2022-09-07 15:01:19.098 ERROR 12 --- [     parallel-1] i.g.i.ManagedChannelOrphanWrapper        : *~*~*~ Channel ManagedChannelImpl{logId=649, target=HOST:PORT} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93) ~[grpc-core-1.44.0.jar!/:1.44.0]
	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53) ~[grpc-core-1.44.0.jar!/:1.44.0]
	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44) ~[grpc-core-1.44.0.jar!/:1.44.0]
	at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630) ~[grpc-core-1.44.0.jar!/:1.44.0]
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297) ~[grpc-core-1.44.0.jar!/:1.44.0]
	at com.daml.ledger.rxjava.DamlLedgerClient.<init>(DamlLedgerClient.java:134) ~[bindings-rxjava-2.3.2.jar!/:na]
	at com.daml.ledger.rxjava.DamlLedgerClient$Builder.build(DamlLedgerClient.java:83) ~[bindings-rxjava-2.3.2.jar!/:na]

Hi Turan, do you have the logs of the Participant node that ledger client was connected to to hand? It looks to me like the Participant shut down the connection, which suggests some sort of catastrophic failure. Such catastrophic failures can occur for example when running in containers with very little allocated memory and the JVM throws OOMs. But I’m only speculating here. Participant (and ideally also domain) node logs would help diagnose.

We are indeed running in containers. Will look to extract the logs.

Do you have a recommended core/thread/memory count for canton (or JVM args to override)?

By the way this only began manifesting itself after we went from daml 2.0.0 to 2.3.2

If you are running with n CPU cores, try setting 2+n GB memory.

Is there a way to limit a canton node from being taken down by a large number of connections?

Is this what you are after? Static Configuration — Daml SDK 2.3.4 documentation