This comes from nixpgks commit 81291cc793cf88bd6eff3fd8512e5eb9d037066c
and will be included with nixos 22.11.

... and use a smarter Prometheus query to combine the two.

This should implement my actual intentions: Alert when
backups run for longer than 3h, and the the repo check
for more than 6h.

Forgot to add a second alert when I added the workaround to
not count the ZFS ARC into used memory :/

... which is both governed by our retention policy.

This is a bit buggy still in our version of Grafana, but already
nice to look at / maybe useful.

Refs privatestorageops#429

This adds alerting to the backup job duration graph:  Grafana alerting
works with systemd unit metrics, i.e. a backup job unit being "active"
for too long.  Use that fact for alerting on long-running backup jobs.

... instead of connected lines default, also with working label for host

Refs privatestorageops#429

Refs privatestorageops#429

Failed backups now have a filled red area instead of a thin
yellow line.

Refs privatestorageops#429.

, a dashboard that "displays a lot of data about one single host".

This is

One query for hosts with ZFS and one for those without.

Since ZoL frees ARC under memory pressure, let's not
count it as "used" but instead as "free" memory.

node_memory_MemAvailable_bytes is a better estimator than the sum I used before
says some Prometheus documentation.  It is also almost the same, but reads nicer.

This should fix the current alerts for our RAID arrays.
It's only "should" because I can't test it proper without said
RAID arrays in the dev or staging machines.

The newer "Time Series" panel does not support two axes.

- Also alert on "negative" (== receiving) errors
- Use 'rate' so we can get out of the reporting if situation normalizes

Grafana 8 recommends these for added performance and capabilities.

- (tryfix) Switch to Loki datasource;
- Filter requests to metrics instead of all GET;
- Show the last week by default
- Show log times

Interesting and not quickly fixable as it looks like: the
GET ... line comes *after*  the result.  We might be logging
this wrong to begin with, but probably this is some next
gen AI line ordering intelligence

Or rather, let it listen only on localhost.
I thought Grafana needed it, or Promtail needed it, but I don't remember
clearly, the web doesn't say clearly, and on my local dev stack
Promtail/Loki seems to still work just fine without GRPC on the network.

Besides migrating all charts to the faster and more capable Grafana 8
TimeSeries charting tool this mainly introduces two panels to view
the logs of PaymentServer.  It has some naive and minimal filtering
in place to only show lines that are not caused by the metrics gathering
itself or that I just deemed to be OK (like "GET" requests) so we
hopefully catch those dreaded "Unexpected Exception" lines.

Fixes privatestorageops#207.