Setting up alerts

A client wants to setup alerts based on log messages.

• Do we have a canonical or suggested list for the various components?
• Should we treat everything at a WARN or ERROR level as alert worthy? Anything at a lower level that can be used for diagnostics?
• The documentation answers the how but not the what.

My recommendation:

• Alert on every ERROR (though be sure to not pick up errors wrapped in warns)
• Alert on continuous rates of WARNs. Eg if your system runs at 100 TPS, warn if there are more than 5 warns/s average for 30+ seconds or similar.

But the second one is really something you need to make a judgement call on your tolerance for false positives/negatives. If you alert on every warn, better be sure there are no intermittent network failures, DB disconnects, contention, etc. If you don’t alert on any warns, you may be failing all your transactions because your clock skews are off and you wouldn’t notice.