This is a step towards adding centralized logging to our distributed system.
The components have been chosen for their integration with our Grafana/Prometheus system: It consists of Loki log aggregator on the monitoring host and Promtail agents on all nodes. The existing Grafana web app is used to comb through the log data.
For some reason the network architecture is opposite of that of Prometheus, where a central TDSB server (data sink) acts as a client and polls a lot of data sources (each node in the system).
I can't say for sure yet how much storage this will cost on the central system. For privacy reasons, logs are kept for only two weeks. This also should keep storage requirements modest and performance reasonable.
Fixes privatestorageops#207. In turn, fixes privatestorageops#389.