- Also alert on "negative" (== receiving) errors
- Use 'rate' so we can get out of the reporting if situation normalizes

Don't display the port number in Grafana dash tooltips

See merge request !267

If I understand the behavior current Prometheus and Grafana
correctly, this should remove the port number from the
paymentserver labels in Prometheus TDSB.

nixpkgs update 2022-02-21

See merge request !266

Grafana 8: Migrate Dashboards from Graph to TimeSeries panels

See merge request !263

Grafana 8 recommends these for added performance and capabilities.

Specify ssh-keys as list of strings, rather than having a single string.

See merge request !254

Grafana: Payments dashboard log fix & fine tune

See merge request !260

- (tryfix) Switch to Loki datasource;
- Filter requests to metrics instead of all GET;
- Show the last week by default
- Show log times

Interesting and not quickly fixable as it looks like: the
GET ... line comes *after*  the result.  We might be logging
this wrong to begin with, but probably this is some next
gen AI line ordering intelligence

Update nixpkgs.

See merge request !259

Add a derivation that builds all three grids.

See merge request !253

Loki/Promtail: Towards centralized logging

Closes privatestorageops#207 and privatestorageops#389

See merge request !251

Or rather, let it listen only on localhost.
I thought Grafana needed it, or Promtail needed it, but I don't remember
clearly, the web doesn't say clearly, and on my local dev stack
Promtail/Loki seems to still work just fine without GRPC on the network.

OMG look at teh genius of it

Do some name resolution for those poor computers

Improve factoring by removing those `/etc/hosts` entries

And avoid the infinite loop @tomprince was encountering in !258

Merge remote-tracking branch 'whetstone/207.payment-server-exception-reporting-update' into 207.payment-server-exception-reporting-update

Merge branch '207.reuse-vpn-server-ip-configuration' into '207.payment-server-exception-reporting-update'

Read the VPN server IP address from the monitoring node config

See merge request !257

morph offers an auto-passed `nodes` parameter from which we can read all of
the nodes in the morph "network".  We can dig around in this to find the
monitoring node and then read its configured IP address.

It may be worth noting that this address appears in at least two places in the
configuration.  It appears in our "input" configuration which grid.nix
defines.  This change reads it from there.  We also have a NixOS module
`nixos/modules/monitoring/vpn/server.nix` which reads this "input" and uses it
to configure `networking.wireguard.interfaces.monitoringvpn`.  This is fed
onwards to <nixos> to generate actual system configuration.

It seems better to use the former than the latter because we have more direct
control over it and if we consider the whole configuration system a function
then it is more like an argument we are supplying rather than an obscure
implementation detail.

Auto-optimize nix stores on dev and staging VMs

See merge request !255

This reverts commit 3113a3e6.

Thinking a second time:  While the staging grid is deployed to
by an automaton anyway and it taking a minute longer won't bother
us, 20 % space savings is probably not worth slowing down
development interactivity.

rename sshPrivateKeyFile also in other files

Use the qualified `.monitoringvpn` domains for collecting metrics.

See merge request !236

Update nixpkgs.

See merge request !252

This is the derivation I use when running `nix store diff-closures` for the
weekly nixpkgs update.

The derivation also includes some attributes that are useful for exploring the
various grid configurations in the nix repl.

Besides migrating all charts to the faster and more capable Grafana 8
TimeSeries charting tool this mainly introduces two panels to view
the logs of PaymentServer.  It has some naive and minimal filtering
in place to only show lines that are not caused by the metrics gathering
itself or that I just deemed to be OK (like "GET" requests) so we
hopefully catch those dreaded "Unexpected Exception" lines.

Fixes privatestorageops#207.

Should be pure refactoring