DAML on SQL: Connection not available on startup

We came across the following issue today, started working fine after restarting the container.

ERROR: get_event_sequential_id_range: Got an exception while executing a SQL query. Rolled back the transaction. (context: {participantId=daml-on-sql})

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:128)

at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)

at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)a

at scala.util.Success.$anonfun$map$1(Try.scala:255)

at scala.util.Success.map(Try.scala:213)

at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

ERROR: Unhandled internal error (context: {participantId=daml-on-sql, startExclusive=00000000000002ff, endInclusive=, parties=[test2]})

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:128)

at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)

at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)

at scala.util.Success.$anonfun$map$1(Try.scala:255)

at scala.util.Success.map(Try.scala:213)

at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

ERROR: get_event_sequential_id_range: Got an exception while executing a SQL query. Rolled back the transaction. (context: {participantId=daml-on-sql})

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:128)

at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)

at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)

at scala.util.Success.$anonfun$map$1(Try.scala:255)

at scala.util.Success.map(Try.scala:213)

at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

mersalin:

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

This error means that a database connection couldn’t be acquired from the connection pool within 250ms for a flat transactions or a transaction tree request. Do you have many transaction streams running at the same time? It’s difficult to say without knowing more details about how your applications interact with the ledger api.

If I may revive this topic, I’m experiencing something similar on my end using Daml-on-SQL 1.18.1:

INDEX_DB_SQL_TRANSIENT_ERROR(1,0): Processing the request failed due to a transient database error: daml.index.db.connection.sandbox - Connection is not available, request timed out after 2397ms.
java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 2397ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at com.daml.platform.store.appendonlydao.DataSourceConnectionProvider$$anon$2.runSQL(HikariJdbcConnectionProvider.scala:73)
	at com.daml.platform.store.appendonlydao.DbDispatcher.$anonfun$executeSql$2(DbDispatcher.scala:57)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:672)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:431)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.

We are running some load tests fetching thousands of contracts at once from different parties’ perspective.

@Matheus
Please note that old threads on this forum are not monitored. If you need to ask a new question or raise a new issue, always start a new thread. If you still need help with this issue, please make a new post on a new thread. If you need to reference an old thread, include the link to it in your post.