HA setup with DAML on relational database

Hi I realize I can setup the sandbox to connect to a Database for persistence. I have a way to solve high avaliability for the database, but what about the sandbox, are there any options avaliable to me?

Hi Nate, welcome to the DAML discussion forums.

Sandbox currently doesn’t support running in a hot/hot kind of scenario. So currently the only real option is to monitor the process and restart it if something goes down.

So the question is, how can one detect an issue with the Sandbox?

For that reason Sandbox (and all DAML ledgers using the Participant Server stack in the daml repository) implement the gRPC health checking protocol. This allows you to query the health check endpoint and react accordingly (e.g. by restarting the sandbox process). This should also be compatible with Kubernetes.

Here’s how you can query the health check endpoint with grpcurl:

$ grpcurl -plaintext localhost:6865 grpc.health.v1.Health/Check
{
  "status": "SERVING"
}

What you can do in addition is to cater for the situation where your entire host / machine goes down. In this case you can keep other VMs / machines ready for the sandbox process to be started there instead. Of course, you’ll need some kind of gateway that will route incoming traffic to the new node upon failure, but your standard proxies (Envoy, NGINX, etc.) can do that.

Thanks for the responses. Very helpful. In the scenario where I start another sandbox up in the event that one fails, can I confirm:

1. No in-flight transactions will be lost
2. No required state is written locally by the sandbox, it’s all written to db.

Regards

Nathan

It is paramount that only one sandbox process writes to the database at a time. If you make sure that the first process is definitely not running anymore, you can safely start another process.

Nate:

1. No in-flight transactions will be lost

With the first process being terminated, this means inflight transactions were cancelled and the clients should have had their gRPC connections terminated as well.

Nate:

1. No required state is written locally by the sandbox, it’s all written to db.

That’s correct.

Hi Nate. A bit off-topic, but just wanted to remind you that if you’re considering this for production/commercial use, you could reach out to our ledger partners and inquire whether the underlying ledger supports HA, and consider that as an alternative to sandbox.

Hi Luciano,

If things go to plan, happy to look at various ledger options. Current place of work is heavily invested in relation databases as persistence store. Does anyone offer a commercial support offering around that?

Hi Nathan - short answer is yes, there’s a couple commercial options just being spun up now. Hit me up directly. I think you know how to find me

Tim