Canton-on-postgres - Failed to initialize sequencer node

Hi team,

I’m having an issue of connecting my sequencer which uses postgresql.
When I onboard my domain manager, sequencer and mediator using:

domain.setup.bootstrap_domain(sequencers.all, mediators.all)

I’m facing a “Failed to initialize sequencer node” error.

Please find below the sequencer configuration file:

canton {
  sequencers {
    sequencer {
      public-api.address = 0.0.0.0
      public-api.port = 3010
      admin-api.address = 0.0.0.0
      admin-api.port = 4001
      storage = ${_shared.storage}
      storage.config.properties.databaseName = cantondb
    }
  }
}

Please find the postgres configuration file:

_shared {
  storage {
    type = postgres
    config {
      dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
      queueSize = 50000
      properties = {
        serverName = "localhost"
        portNumber = "5432"
        databaseName = "cantondb" # default value, can be overwritten after importing
        # ApplicationName = "canton" # default value, can be overwritten after importing
        user = "canton"
        password = "supersafe"
      }
    }
    # The following is an educated guess of a sane default for the number of DB connections.
    # https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
    max-connections = 110
  }
}

Please find the error below:

ERROR c.d.c.c.EnterpriseConsoleEnvironment - Request failed for sequencer.
  GrpcServerError: INTERNAL/Failed to initialize sequencer node: ConnectionError(
  ConnectionFailure(
    Failed to create connection: java.sql.SQLTransientConnectionException:  - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at slick.jdbc.hikaricp.HikariCPJdbcDataSource.createConnection(HikariCPJdbcDataSource.scala:14)
	at com.digitalasset.canton.resource.SingleConnectionDatabase$.$anonfun$createConnection$3(SingleConnectionDatabase.scala:346)
	at cats.syntax.EitherSyntax$CatchOnlyPartiallyApplied$.apply$extension(either.scala:31)
	at com.digitalasset.canton.resource.SingleConnectionDatabase$.$anonfun$createConnection$2(SingleConnectionDatabase.scala:346)
	at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:59)
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118)
	at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:70)
	at scala.concurrent.package$.blocking(package.scala:124)
	at com.digitalasset.canton.resource.SingleConnectionDatabase$.$anonfun$createConnection$1(SingleConnectionDatabase.scala:348)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
	at org.postgresql.Driver$ConnectThread.getResult(Driver.java:416)
	at org.postgresql.Driver.connect(Driver.java:271)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:103)
	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:87)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
	at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:706)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:692)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

  )
)
  Request: Initialize(domain::12206ea33fa0...,Seq(),StaticDomainParameters(1m,1000000,10485760,false,TreeSet(ECDSA-P256, ECDSA-P384, Ed25519),TreeSet(ECIES-P256_HMAC256_AES128-GCM),TreeSet(AES128-GCM),TreeSet(Sha256),TreeSet(Tink),1.0.0-rc6),None)
  Trailers: Metadata(content-type=application/grpc)
  Command SetupAdministration$Setup.$anonfun$bootstrap_domain$3$adapted invoked from LinearSeq.scala:169
ERROR c.d.c.ConsoleScriptRunner - Script execution failed: com.digitalasset.canton.console.CommandFailure: Command execution failed.

Could you please advise ?

Thanks and Regards,
Jean-Paul

You have sql connection issues:

jaypeeda:

Failed to create connection: java.sql.SQLTransientConnectionException:  - Connection is not available, request timed out after 30000ms.
...
Caused by: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections

In particular, you need to configure Postgres with more db connections, or limit the number of connections the process is using: Static Configuration — Daml SDK 2.7.6 documentation