Recently this alarm fired a couple of times when the backup self-
check ran.

Set it to alert only when load is over 1 for longer than 2 hours.

Refs #129

This comes from nixpgks commit 81291cc793cf88bd6eff3fd8512e5eb9d037066c
and will be included with nixos 22.11.

... and use a smarter Prometheus query to combine the two.

This should implement my actual intentions: Alert when
backups run for longer than 3h, and the the repo check
for more than 6h.

Fixes privatestorageops#287

Forgot to add a second alert when I added the workaround to
not count the ZFS ARC into used memory :/

... which is both governed by our retention policy.

This is a bit buggy still in our version of Grafana, but already
nice to look at / maybe useful.

Refs privatestorageops#429

This adds alerting to the backup job duration graph:  Grafana alerting
works with systemd unit metrics, i.e. a backup job unit being "active"
for too long.  Use that fact for alerting on long-running backup jobs.

... instead of connected lines default, also with working label for host

Refs privatestorageops#429

Refs privatestorageops#429

Failed backups now have a filled red area instead of a thin
yellow line.

Refs privatestorageops#429.

, a dashboard that "displays a lot of data about one single host".

This is

One query for hosts with ZFS and one for those without.

Since ZoL frees ARC under memory pressure, let's not
count it as "used" but instead as "free" memory.

node_memory_MemAvailable_bytes is a better estimator than the sum I used before
says some Prometheus documentation.  It is also almost the same, but reads nicer.

This is only semi correct. We keep logs up to what's specified in the privacy
policy.  Currently, that is implemented as keeping them for up to 30 days on
the individual nodes as well as on the central Loki server.