Request timeout encountered for request [POST v1/exercise]

HI Daml team

We are encountering a strange issue in our enviroment where we received a timeout on our POST v1/exercise
while we are not receiveing any timeouts on POST v1/query

ANy insights on what could be the root cause ?

I would take a look at the JSON API and probably more important the participant logs to see which parts takes long enough to produce a timeout. /v1/exercise actually commits a transaction while /v1/query doesn’t so it’s not surprising that they behave differently.

Can you remand me on how we can access the participants logs ? I like to investigate this further. Our timeout is set to 30 seconds, so a simple “Archive” Exercise should not take the full 30 seconds

We fixed the issue by re-starting our EKG pod for daml-json-api

The strange thing is that this issue occurred in 2 different environments, but with fairly large (same size) databases, but we had not seen any performance degradation recently in either one recently.

How did you configure your participant? By default it writes it to a log file. In a cluster deployment you usually configure it to write to stdout and rely on your cluster infrastructure to capture logs from all containers.

We are still on 1.13 Daml SDK, so we have a farly simple EKG architecture.
I just did a

kubectl logs

on the pod to get the log information

Hopefully soon we get to migrate our production servers to Canton