Health monitoring of the JSON API Server

I understand the JSON API Health Check endpoints (HTTP JSON API Service — Daml SDK 1.18.1 documentation) call the Ledger API Health Check and if the latter fails, the former indicates the unhealthy status as well.

But what happens if the underlying Ledger API is healthy but the JSON API server is not, specifically:

1. What are the kinds of issues to monitor for on the JSON API?
2. How to detect cases where the JSON API server is unhealthy but the underlying Ledger API still reports healthy, outside of the JSON API process crashing - so for example, if it stays alive but stops responding?

meet:

What are the kinds of issues to monitor for on the JSON API?

There’s nothing specific to it the you ought to monitor for it that sets it apart from a normal JVM process, resource usage being the most important one. If you’re interested in metrics exposed by the HTTP JSON API, the documentation on the production setup lists them all.

meet:

How to detect cases where the JSON API server is unhealthy but the underlying Ledger API still reports healthy, outside of the JSON API process crashing - so for example, if it stays alive but stops responding?

Both the Ledger API server and the HTTP JSON API have a liveness endpoint that should be able to tell you whether each service is currently available. If it’s reported as available but it’s not, you could be tempted to infer from missing responses that the liveness check is somehow lying, but distributed systems are hard and many other things could be happening. Checking the logs and metrics is probably your best shot.

Thanks @stefanobaghino-da
What are the kinds of health checks done by the JSON API health endpoint over and above calling the gPRC API health endpoint?

It also ensures that the query store (if set up) is also alive and well.

Ok, so to confirm, the JSON API health endpoint reports healthy if both of the following are met (and no other conditions):

• The underlying gRPC health endpoint reports healthy (liveness check + readiness check)
• The persistent query store is live and connected

Correct?

Yes. I further add this sentence because posts must be at least five characters long.