Monitoring: Fix backup duration alert

See merge request !349

This should implement my actual intentions: Alert when
backups run for longer than 3h, and the the repo check
for more than 6h.

Add dashboards for Tahoe Incident Report count and rate

Closes privatestorageops#288

See merge request !347

Publish number of Tahoe Incident Reports

Closes privatestorageops#287

See merge request !346

Fixes privatestorageops#287

Also alert when hosts that run ZFS run out of RAM

See merge request !345

Prometheus should keep metrics as long as Loki keeps logs

See merge request !344

Forgot to add a second alert when I added the workaround to
not count the ZFS ARC into used memory :/

... which is both governed by our retention policy.

Monitoring: Count Tahoe's corruption advisories

Closes privatestorageops#287

See merge request !341

Monitoring: Add Tahoe-LAFS corruption advisory count + rate + alert on rate > 0

Closes privatestorageops#288

See merge request !342

Monitoring: Fix resources dashboard

See merge request !343

Monitoring: Backup duration and backup set size

Closes privatestorageops#429

See merge request !339

This is a bit buggy still in our version of Grafana, but already
nice to look at / maybe useful.

Refs privatestorageops#429

This adds alerting to the backup job duration graph:  Grafana alerting
works with systemd unit metrics, i.e. a backup job unit being "active"
for too long.  Use that fact for alerting on long-running backup jobs.

... instead of connected lines default, also with working label for host

Refs privatestorageops#429

Refs privatestorageops#429

Backup: Borg: All logs as JSON please

See merge request !338

That makes working with its output in Grafana easier.
Good documentation can be found at
https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#json-output

Backup: Add --stats to borgbackup create options

See merge request !337

We want to have this in our own monitoring

Monitoring: Backup: Clearer coloring

See merge request !336

Failed backups now have a filled red area instead of a thin
yellow line.

Refs privatestorageops#429.

Disregard ZFS ARC cache when monitoring free RAM

Closes #119

See merge request !331

Monitoring: Add a "Node Exporter Full" dashboard

See merge request !332

, a dashboard that "displays a lot of data about one single host".

This is

One query for hosts with ZFS and one for those without.

Since ZoL frees ARC under memory pressure, let's not
count it as "used" but instead as "free" memory.

Set minimal nixos profile for our grid machines

See merge request !330