update production

Changes from develop since 2022-05-23T16:19:20+00:00

Deployment Notes

Diff Stat

 docs/ops/backup-recovery.rst                       |  62 ++++
 morph/grid/local/private-keys/README.rst           |   3 +-
 .../grid/local/private-keys/borgbackup.passphrase  |   1 +
 morph/grid/local/private-keys/borgbackup.ssh-key   |   7 +
 .../local/public-keys/borgbackup/storage1.repopath |   1 +
 .../local/public-keys/borgbackup/storage2.repopath |   1 +
 .../public-keys/borgbackup/storage001.repopath     |   1 +
 .../public-keys/borgbackup/storage002.repopath     |   1 +
 .../public-keys/borgbackup/storage003.repopath     |   1 +
 .../public-keys/borgbackup/storage004.repopath     |   1 +
 .../public-keys/borgbackup/storage005.repopath     |   1 +
 .../public-keys/borgbackup/storage001.repopath     |   1 +
 morph/lib/borgbackup.nix                           |  81 +++++
 morph/lib/default.nix                              |   1 +
 morph/lib/storage.nix                              |   3 +
 nixos/lib/ip-util.nix                              | 219 +++++++++++++
 nixos/modules/issuer.nix                           |  13 +-
 .../server/grafana-dashboards/meta-monitoring.json |   2 +-
 .../grafana-dashboards/resources-overview.json     | 348 +++++++++++++++------
 .../grafana-dashboards/services-overview.json      |   2 +-
 nixos/pkgs/zkapissuer/repo.json                    |   4 +-
 nixos/tests/private-storage.nix                    |   5 +-
 nixos/tests/run-client.py                          |   7 +-
 nixos/tests/test_privatestorage.py                 |  13 +-
 nixpkgs.json                                       |   4 +-
 tools/get-vpn-config.nix                           |  10 +-
 26 files changed, 665 insertions(+), 128 deletions(-)

docs/ops/backup-recovery.rst

@@ -113,3 +113,65 @@ Recovery
 #. Clean up the remote copies of the backup file ::
 
      [REMOTE]$ rm -iv recovery.tar.bz2
+
+Storage Directories
+~~~~~~~~~~~~~~~~~~~
+
+The user ciphertext is backed up using `Borg backup <https://borgbackup.readthedocs.io/>`_ to a separate location - currently a SaaS backup storage service (`borgbase.com <https://borgbase.com>`_).
+
+Borg backup uses a *RepoKey* secured by a *passphrase* to encrypt the backup data and an *SSH key* to authenticate against the backup storage service.
+Each Borg backup job requires one *backup repository*.
+
+The backups are automatically checked periodically.
+
+SSH keys
+````````
+
+Borgbase `recommends creating ed25519 ssh keys with one hundred KDF rounds <https://www.borgbase.com/ssh>`_.
+We create one key pair per grid (not per host)::
+
+    $ ssh-keygen -f borgbackup-appendonly-staging -t ed25519 -a 100
+    $ ssh-keygen -f borgbackup-appendonly-production -t ed25519 -a 100
+
+Save the key without a passphrase and upload the public part to `Borgbase SSH keys <https://www.borgbase.com/ssh>`_.
+
+Passphrase
+``````````
+
+Make up a passphrase to encrypt our repository key with. Use computer help if you like::
+
+    nix-shell --packages pwgen --command 'pwgen --secure 83 1'  # 83 is the year I was born. Very random.
+
+Create & initialize the backup repository
+`````````````````````````````````````````
+
+Borgbase.com offers a `borgbase.com GraphQL API <https://docs.borgbase.com/api/>`_.
+Since our current number of repositories is small we save time by creating the repositories by clicking a few buttons in the `borgbase.com Web Interface <https://www.borgbase.com/repositories>`_:
+
+* Set up one repository per backup job.
+* Set the *Repository Name* to the FQDN of the host to be backed up.
+* Add the SSH key created earlier as *Append-Only Access* key.
+* Leave the other settings at their defaults.
+
+Then initialize those repositories with our chosen parameters::
+
+    export BORG_PASSCOMMAND="cat borgbackup-passphrase-staging"
+    export BORG_RSH="ssh -i borgbackup-appendonly-staging"
+    borg init -e repokey-blake2 xyxyx123@xyxyx123.repo.borgbase.com:repo
+
+Reliability checks
+``````````````````
+
+Borg handles large amounts of data.
+Given enough bits rare, spurious bit flips become a problem.
+That is why regular runs of ``borg check`` are recommended
+(see the `borgbase FAQ <https://docs.borgbase.com/faq/#how-often-should-i-run-borg-check>`_).
+
+
+
+Recovery
+````````
+
+Borg offers various methods to restore backups.
+A very convenient method is to mount a backup set using FUSE.
+Please consult the restore documentation at `Borgbase <https://docs.borgbase.com/restore/>`_ and `Borg <https://borgbackup.readthedocs.io/en/stable/usage/mount.html>`_.