Documentation/DamlView on Daml ↗
Daml

Monitoring

​
Monitoring

The Covalidation Service contains metrics/traces out of the box to support monitoring. This behavior is enabled by adding the following block to your configuration: 
  monitoring:
    otel:
      endpointURL: <OTEL Monitoring URL>
The endpointURL field is an OpenTelemetry (OTEL) URL and port for sending metrics and traces. Different solutions are available to configure an OTEL endpoint, i.e. Datadog-agents. Logs in the Covalidation Service can be made more verbose by adding the following to your configuration: 
  log:
    level: debug
It is good practice to monitor for things like:
  • Kubernetes liveness/readiness probes for the covalidation-service are passing
  • No ERROR log entries observed
  • Log indicating last checkpoint processed is progressing

​
Metrics

The covalidation service offers metrics out of the box for both successful and failing operations. Metrics embedded in the code include:
  • covalidation.error: Metric counting all the errors encountered during the various syncs that occur during the covalidation process
  • covalidation.github.dars_uploaded: Metric measuring the total sum of dars uploaded to participant
  • sync: Metric measuring the total number of sync attempts by the covalidation service
  • offset: Metric capturing the current highest offset of the ledger API
Additionally, a sync trace is also included to capture a sync and raise any errors that may occur during the three covalidation processes:
  • sync: Overarching trace that includes 3 subtraces for the 3 processes of covalidation
  • github: Trace for the Github dar upload service
  • vetter: Trace for the Vetter package vetting service
  • partysyncer: Trace for the Partysyncer transaction sync service
Any errors that occur during one of the services are captured as a span event and appear under the respective trace to help with debugging service issues.

​
Alerts

We recommend creating the following alerts for overall infrastructure and application health:
  • Memory usage over time: Set alert on memory usage of the node the service is hosted on
  • Kubernetes Deployment Replica Pods: Ensure pods are READY and healthy
  • Covalidation Error Rate Over Time: Change in error rate in the previous X amount of minutes compared to the last Y minutes