The default log level ("WARN") means borgbackup does not output
anything (and exits with a "0" return code) on success.

We'd like to have some more insight into long-running monthly
backup repo checks and turn up the log level to "INFO" with
this change.

See https://borgbackup.readthedocs.io/en/stable/usage/general.html#common-options .

Fixes #136.

That makes working with its output in Grafana easier.
Good documentation can be found at
https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#json-output

We want to have this in our own monitoring

Fixes #118

This reverts commit c8a13555.

Sadly, Borg says this is not a good idea: https://borgbackup.readthedocs.io/en/stable/faq.html#can-i-backup-from-multiple-servers-into-a-single-repository

Spoiler:  This won't work, Borg does not guarantee consistency with
parallel writers to the same repo as I now found out.

It's not really a key, and not really public, but not secret either.
This way, the tests can run.

Since that describes it better, especially now that I have
moved the one line that wasn't about the VPN out of it.

Tom Prince authored 3 years ago and Florian Sesser committed 3 years ago

This includes the host-based metrics collector, and the VPN client setup (including key deployment).

... and fix that option the other commit introduced

Do some name resolution for those poor computers

Improve factoring by removing those `/etc/hosts` entries

And avoid the infinite loop @tomprince was encountering in !258

morph offers an auto-passed `nodes` parameter from which we can read all of
the nodes in the morph "network".  We can dig around in this to find the
monitoring node and then read its configured IP address.

It may be worth noting that this address appears in at least two places in the
configuration.  It appears in our "input" configuration which grid.nix
defines.  This change reads it from there.  We also have a NixOS module
`nixos/modules/monitoring/vpn/server.nix` which reads this "input" and uses it
to configure `networking.wireguard.interfaces.monitoringvpn`.  This is fed
onwards to <nixos> to generate actual system configuration.

It seems better to use the former than the latter because we have more direct
control over it and if we consider the whole configuration system a function
then it is more like an argument we are supplying rather than an obscure
implementation detail.

This reverts commit 3113a3e6.

Thinking a second time:  While the staging grid is deployed to
by an automaton anyway and it taking a minute longer won't bother
us, 20 % space savings is probably not worth slowing down
development interactivity.

Should be pure refactoring

This is my latest version of this, updated to work with the
packages in NixOS 21.05.

Note that changing the origin repository in the on-node deployment checkout
is also required.

See https://whetstone.private.storage/privatestorage/privatestorageops/-/merge_requests/197#note_19071

The current code evaluates our custom packages once for each node, which adds
signifcant amount of time to evaluate a grid. We can reduce this, by adding the
custom package set as an attribute to the nixpkgs set we pass to morph.

This doesn't change how we refer to those packages, as we continue to expose the
custom package set as a module attribute.

These are the times to evaluate all three grids (on a partially loaded system),
when there was nothing new to build:

```
before:
real	2m27.837s
user	3m35.528s
sys	0m3.722s

after:
real	1m12.748s
user	1m34.047s
sys	0m3.346s
```

Grafana 8 on NixOS 21.11 fails harder than 7.x on 21.05 when its
`grafana-admin.password` file is missing.  Provide it regardless
whether Google auth is configured or not.