... and use a smarter Prometheus query to combine the two.

This should implement my actual intentions: Alert when
backups run for longer than 3h, and the the repo check
for more than 6h.

Fixes privatestorageops#287

Forgot to add a second alert when I added the workaround to
not count the ZFS ARC into used memory :/

... which is both governed by our retention policy.

This is a bit buggy still in our version of Grafana, but already
nice to look at / maybe useful.

Refs privatestorageops#429

This adds alerting to the backup job duration graph:  Grafana alerting
works with systemd unit metrics, i.e. a backup job unit being "active"
for too long.  Use that fact for alerting on long-running backup jobs.

... instead of connected lines default, also with working label for host

Refs privatestorageops#429

Refs privatestorageops#429

Failed backups now have a filled red area instead of a thin
yellow line.

Refs privatestorageops#429.

, a dashboard that "displays a lot of data about one single host".

This is

One query for hosts with ZFS and one for those without.

Since ZoL frees ARC under memory pressure, let's not
count it as "used" but instead as "free" memory.

node_memory_MemAvailable_bytes is a better estimator than the sum I used before
says some Prometheus documentation.  It is also almost the same, but reads nicer.

This is only semi correct. We keep logs up to what's specified in the privacy
policy.  Currently, that is implemented as keeping them for up to 30 days on
the individual nodes as well as on the central Loki server.

This should fix the current alerts for our RAID arrays.
It's only "should" because I can't test it proper without said
RAID arrays in the dev or staging machines.

Tom Prince authored 3 years ago and Florian Sesser committed 3 years ago

This includes the host-based metrics collector, and the VPN client setup (including key deployment).

The newer "Time Series" panel does not support two axes.

... and fix that option the other commit introduced