diff --git a/docs/ops/backup-recovery.rst b/docs/ops/backup-recovery.rst
index a39c96dfa859203d6b54c1812e70414715b920e9..2201fcb7ab7463a73cd30730b01c045b5c85d72e 100644
--- a/docs/ops/backup-recovery.rst
+++ b/docs/ops/backup-recovery.rst
@@ -113,3 +113,65 @@ Recovery
#. Clean up the remote copies of the backup file ::
[REMOTE]$ rm -iv recovery.tar.bz2
+
+Storage Directories
+~~~~~~~~~~~~~~~~~~~
+
+The user ciphertext is backed up using `Borg backup <https://borgbackup.readthedocs.io/>`_ to a separate location - currently a SaaS backup storage service (`borgbase.com <https://borgbase.com>`_).
+
+Borg backup uses a *RepoKey* secured by a *passphrase* to encrypt the backup data and an *SSH key* to authenticate against the backup storage service.
+Each Borg backup job requires one *backup repository*.
+
+The backups are automatically checked periodically.
+
+SSH keys
+````````
+
+Borgbase `recommends creating ed25519 ssh keys with one hundred KDF rounds <https://www.borgbase.com/ssh>`_.
+We create one key pair per grid (not per host)::
+
+ $ ssh-keygen -f borgbackup-appendonly-staging -t ed25519 -a 100
+ $ ssh-keygen -f borgbackup-appendonly-production -t ed25519 -a 100
+
+Save the key without a passphrase and upload the public part to `Borgbase SSH keys <https://www.borgbase.com/ssh>`_.
+
+Passphrase
+``````````
+
+Make up a passphrase to encrypt our repository key with. Use computer help if you like::
+
+ nix-shell --packages pwgen --command 'pwgen --secure 83 1' # 83 is the year I was born. Very random.
+
+Create & initialize the backup repository
+`````````````````````````````````````````
+
+Borgbase.com offers a `borgbase.com GraphQL API <https://docs.borgbase.com/api/>`_.
+Since our current number of repositories is small we save time by creating the repositories by clicking a few buttons in the `borgbase.com Web Interface <https://www.borgbase.com/repositories>`_:
+
+* Set up one repository per backup job.
+* Set the *Repository Name* to the FQDN of the host to be backed up.
+* Add the SSH key created earlier as *Append-Only Access* key.
+* Leave the other settings at their defaults.
+
+Then initialize those repositories with our chosen parameters::
+
+ export BORG_PASSCOMMAND="cat borgbackup-passphrase-staging"
+ export BORG_RSH="ssh -i borgbackup-appendonly-staging"
+ borg init -e repokey-blake2 xyxyx123@xyxyx123.repo.borgbase.com:repo
+
+Reliability checks
+``````````````````
+
+Borg handles large amounts of data.
+Given enough bits rare, spurious bit flips become a problem.
+That is why regular runs of ``borg check`` are recommended
+(see the `borgbase FAQ <https://docs.borgbase.com/faq/#how-often-should-i-run-borg-check>`_).
+
+
+
+Recovery
+````````
+
+Borg offers various methods to restore backups.
+A very convenient method is to mount a backup set using FUSE.
+Please consult the restore documentation at `Borgbase <https://docs.borgbase.com/restore/>`_ and `Borg <https://borgbackup.readthedocs.io/en/stable/usage/mount.html>`_.
diff --git a/morph/grid/local/private-keys/borgbackup.passphrase b/morph/grid/local/private-keys/borgbackup.passphrase
new file mode 100644
index 0000000000000000000000000000000000000000..c66b798a01fd511ae71532452a6933a2e5c709d1
--- /dev/null
+++ b/morph/grid/local/private-keys/borgbackup.passphrase
@@ -0,0 +1 @@
+The most interesting passphrase in the world.
diff --git a/morph/grid/local/private-keys/borgbackup.ssh-key b/morph/grid/local/private-keys/borgbackup.ssh-key
new file mode 100644
index 0000000000000000000000000000000000000000..d63f7803d35ef226f5c20a2a7944051627b56d86
--- /dev/null
+++ b/morph/grid/local/private-keys/borgbackup.ssh-key
@@ -0,0 +1,7 @@
+-----BEGIN OPENSSH PRIVATE KEY-----
+ratatatratatatratatatratatatratatatratatatratatatratatatratatatratatat
+ratatatratatatratatatratatatratatatratatatratatatratatatratatatratatat
+ratatatratatatratatatratatatratatatratatatratatatratatatratatatratatat
+ratatatratatatratatatratatatratatatratatatratatatratatatratatatratatat
+ratatatratatatratatatratatatratatatratatatc=
+-----END OPENSSH PRIVATE KEY-----
diff --git a/morph/grid/local/public-keys/borgbackup/storage1.repopath b/morph/grid/local/public-keys/borgbackup/storage1.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..5f5df8d1954eeefee7e733783d4d9f92571e199d
--- /dev/null
+++ b/morph/grid/local/public-keys/borgbackup/storage1.repopath
@@ -0,0 +1 @@
+abc123de@abc123de.repo.borgbase.com:repo
diff --git a/morph/grid/local/public-keys/borgbackup/storage2.repopath b/morph/grid/local/public-keys/borgbackup/storage2.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..e73b10d01f5502f943a0d9bce220cbdf758d113f
--- /dev/null
+++ b/morph/grid/local/public-keys/borgbackup/storage2.repopath
@@ -0,0 +1 @@
+vwx789yz@vwx789yz.repo.borgbase.com:repo
diff --git a/morph/grid/production/public-keys/borgbackup/storage001.repopath b/morph/grid/production/public-keys/borgbackup/storage001.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..e3342643a66dfaaedee7071c511e6d5ff4fc97f2
--- /dev/null
+++ b/morph/grid/production/public-keys/borgbackup/storage001.repopath
@@ -0,0 +1 @@
+gye1flhy@gye1flhy.repo.borgbase.com:repo
diff --git a/morph/grid/production/public-keys/borgbackup/storage002.repopath b/morph/grid/production/public-keys/borgbackup/storage002.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..e2a6ec180927dc73eb5c6c6e790f45ccededac67
--- /dev/null
+++ b/morph/grid/production/public-keys/borgbackup/storage002.repopath
@@ -0,0 +1 @@
+l4642x1g@l4642x1g.repo.borgbase.com:repo
diff --git a/morph/grid/production/public-keys/borgbackup/storage003.repopath b/morph/grid/production/public-keys/borgbackup/storage003.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..26bd44969458dbc933ec0617f0236df176c3834b
--- /dev/null
+++ b/morph/grid/production/public-keys/borgbackup/storage003.repopath
@@ -0,0 +1 @@
+c7400xl6@c7400xl6.repo.borgbase.com:repo
diff --git a/morph/grid/production/public-keys/borgbackup/storage004.repopath b/morph/grid/production/public-keys/borgbackup/storage004.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..cc1e8017b31ceaf80a7d6938e11c4d8c12db5592
--- /dev/null
+++ b/morph/grid/production/public-keys/borgbackup/storage004.repopath
@@ -0,0 +1 @@
+sbn13vf8@sbn13vf8.repo.borgbase.com:repo
diff --git a/morph/grid/production/public-keys/borgbackup/storage005.repopath b/morph/grid/production/public-keys/borgbackup/storage005.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..74b928d15fd781fd2e7ed61e0d0e02a8cee7697a
--- /dev/null
+++ b/morph/grid/production/public-keys/borgbackup/storage005.repopath
@@ -0,0 +1 @@
+wg8x4po7@wg8x4po7.repo.borgbase.com:repo
diff --git a/morph/grid/testing/public-keys/borgbackup/storage001.repopath b/morph/grid/testing/public-keys/borgbackup/storage001.repopath
new file mode 100644
index 0000000000000000000000000000000000000000..27c74225521a74fa6a8e60e19f44d0968f7c816b
--- /dev/null
+++ b/morph/grid/testing/public-keys/borgbackup/storage001.repopath
@@ -0,0 +1 @@
+p2kt6691@p2kt6691.repo.borgbase.com:repo
diff --git a/morph/lib/borgbackup.nix b/morph/lib/borgbackup.nix
new file mode 100644
index 0000000000000000000000000000000000000000..e71ca55ef90bb3aa50efb3d48b4cd6268a18daf7
--- /dev/null
+++ b/morph/lib/borgbackup.nix
@@ -0,0 +1,81 @@
+# Importing this adds a daily borg backup job to a node.
+# It has all the common config and keys, and can be configured
+# to back up more (or entirely different) folders.
+
+
+{ lib, config, pkgs, ...}:
+let
+ cfg = config.services.private-storage.borgbackup;
+ inherit (config.grid) publicKeyPath privateKeyPath;
+
+ # Get a per-host number so backup jobs don't all run at the
+ # same time.
+ ip-util = import ../../nixos/lib/ip-util.nix;
+ backupDelay = with builtins; bitAnd (ip-util.fromHexString
+ (hashString "md5" config.networking.hostName)) 15;
+
+in {
+ options.services.private-storage.borgbackup = {
+ enable = lib.mkEnableOption "Borgbackup daily backup job";
+ paths = lib.mkOption {
+ type = lib.types.listOf lib.types.str;
+ description = ''
+ A list of directories to back up using Borg.
+ '';
+ default = [ "/storage" ];
+ };
+ };
+
+ config = lib.mkIf cfg.enable {
+ deployment = {
+ secrets = {
+ "borgbackup-passphrase" = {
+ # The passphrase is used to encrypt the repo key
+ # https://borgbackup.readthedocs.io/en/stable/usage/init.html
+ destination = "/run/keys/borgbackup/passphrase";
+ source = "${privateKeyPath}/borgbackup.passphrase";
+ };
+ "borgbackup-appendonly-ssh-key" = {
+ # The ssh key is used to authenticate to the remote repo server
+ destination = "/run/keys/borgbackup/ssh-key";
+ source = "${privateKeyPath}/borgbackup.ssh-key";
+ };
+ };
+ };
+
+ services.borgbackup.jobs = {
+ daily = {
+ paths = cfg.paths;
+ repo = lib.fileContents "${publicKeyPath}/borgbackup/${config.networking.hostName}.repopath";
+ doInit = false;
+ encryption = {
+ mode = "repokey-blake2";
+ passCommand = "cat /run/keys/borgbackup/passphrase";
+ };
+ environment = {
+ BORG_RSH = "ssh -i /run/keys/borgbackup/ssh-key";
+ };
+
+ # Ciphertext doesn't compress well
+ compression = "none";
+
+ # Start the backup at a different time per machine,
+ # and not at the full hour, but somewhat later
+ startAt = "*-*-* " + toString backupDelay + ":22:11 UTC";
+ };
+ };
+
+ # Check repo once a month
+ systemd.services.borgbackup-check-repo = {
+ # Once a month, 3h after last backup started
+ startAt = "*-*-" + toString backupDelay + " 18:33:22 UTC";
+ path = [ pkgs.borgbackup ];
+ environment = {
+ BORG_PASSCOMMAND = "cat /run/keys/borgbackup/passphrase";
+ BORG_RSH = "ssh -i /run/keys/borgbackup/ssh-key";
+ BORG_REPO = lib.fileContents "${publicKeyPath}/borgbackup/${config.networking.hostName}.repopath";
+ };
+ script = ''${pkgs.borgbackup}/bin/borg check'';
+ };
+ };
+}
diff --git a/morph/lib/default.nix b/morph/lib/default.nix
index f236b8cada99b71cd1c5ab851f3c081421c4b717..c99c19a57e45a27e585830a8dfff95fa3d9d2efb 100644
--- a/morph/lib/default.nix
+++ b/morph/lib/default.nix
@@ -10,6 +10,7 @@
issuer = import ./issuer.nix;
storage = import ./storage.nix;
monitoring = import ./monitoring.nix;
+ borgbackup = import ./borgbackup.nix;
modules = builtins.toString ../../nixos/modules;
diff --git a/morph/lib/storage.nix b/morph/lib/storage.nix
index 83c12f55cc077abb683482b8435dbcbd5025be10..1fd8c26ce3db81ae405025d97abe6068ce2a6903 100644
--- a/morph/lib/storage.nix
+++ b/morph/lib/storage.nix
@@ -7,6 +7,7 @@ in {
# Any extra NixOS modules to load on this server.
imports = [
./monitoringvpn-client.nix
+ ./borgbackup.nix
];
options.grid.storage = {
@@ -46,6 +47,8 @@ in {
services.private-storage.monitoring.exporters.node.enable = true;
services.private-storage.monitoring.exporters.tahoe.enable = true;
+ services.private-storage.borgbackup.enable = true;
+
# Turn on the Private Storage (Tahoe-LAFS) service.
services.private-storage = {
# Yep. Turn it on.
diff --git a/nixos/lib/ip-util.nix b/nixos/lib/ip-util.nix
new file mode 100644
index 0000000000000000000000000000000000000000..9d9c47890ae9a0f7b49a946422248a243ce8692e
--- /dev/null
+++ b/nixos/lib/ip-util.nix
@@ -0,0 +1,219 @@
+# Thank you: https://gist.github.com/petabyteboy/558ffddb9aeb24e1eab2d5d6d021b5d7
+
+with import <nixpkgs/lib>;
+
+rec {
+ # FIXME: add case for negative numbers
+ pow = base: exponent: if exponent == 0 then 1 else fold (
+ x: y: y * base
+ ) base (
+ range 2 exponent
+ );
+
+ fromHexString = hex: foldl (
+ x: y: 16 * x + (
+ (
+ listToAttrs (
+ map (
+ x: nameValuePair (
+ toString x
+ ) x
+ ) (
+ range 0 9
+ )
+ ) // {
+ "a" = 10;
+ "b" = 11;
+ "c" = 12;
+ "d" = 13;
+ "e" = 14;
+ "f" = 15;
+ }
+ ).${y}
+ )
+ ) 0 (
+ stringToCharacters (
+ removePrefix "0x" (
+ hex
+ )
+ )
+ );
+
+ ipv4 = rec {
+
+ decode = address: foldl (
+ x: y: 256 * x + y
+ ) 0 (
+ map toInt (
+ splitString "." address
+ )
+ );
+
+ encode = num: concatStringsSep "." (
+ map (
+ x: toString (mod (num / x) 256)
+ ) (
+ reverseList (
+ genList (
+ x: pow 2 (x * 8)
+ ) 4
+ )
+ )
+ );
+
+ netmask = prefixLength: (
+ foldl (
+ x: y: 2 * x + 1
+ ) 0 (
+ range 1 prefixLength
+ )
+ ) * (
+ pow 2 (
+ 32 - prefixLength
+ )
+ );
+
+ reverseZone = net: (
+ concatStringsSep "." (
+ reverseList (
+ splitString "." net
+ )
+ )
+ ) + ".in-addr.arpa";
+
+ eachAddress = net: prefixLength: genList (
+ x: decode (
+ x + (
+ decode net
+ )
+ )
+ ) (
+ pow 2 (
+ 32 - prefixLength
+ )
+ );
+
+ networkOf = address: prefixLength: encode (
+ bitAnd (
+ decode address
+ ) (
+ netmask prefixLength
+ )
+ );
+
+ isInNetwork = net: address: networkOf address == net;
+
+ /* nixos-specific stuff */
+
+ findOwnAddress = config: net: head (
+ filter (
+ isInNetwork net
+ ) (
+ configuredAddresses config
+ )
+ );
+
+ configuredAddresses = config: concatLists (
+ mapAttrsToList (
+ name: iface: iface.ipv4.addresses
+ ) config.networking.interfaces
+ );
+
+ };
+
+ ipv6 = rec {
+
+ expand = address: (
+ replaceStrings ["::"] [(
+ concatStringsSep "0" (
+ genList (x: ":") (
+ 9 - (count (x: x == ":") (stringToCharacters address))
+ )
+ )
+ )] address
+ ) + (
+ if hasSuffix "::" address then
+ "0"
+ else
+ ""
+ );
+
+ decode = address: map fromHexString (
+ splitString ":" (
+ expand address
+ )
+ );
+
+ encode = address: toLower (
+ concatStringsSep ":" (
+ map toHexString address
+ )
+ );
+
+ netmask = prefixLength: map (
+ x: if prefixLength > x + 16 then
+ (pow 2 16) - 1
+ else if prefixLength < x then
+ 0
+ else
+ (
+ foldl (
+ x: y: 2 * x + 1
+ ) 0 (
+ range 1 (prefixLength - x)
+ )
+ ) * (
+ pow 2 (
+ 16 - (prefixLength - x)
+ )
+ )
+ ) (
+ genList (
+ x: x * 16
+ ) 8
+ );
+
+ reverseZone = net: (
+ concatStringsSep "." (
+ concatLists (
+ reverseList (
+ map (
+ x: stringToCharacters (fixedWidthString 4 "0" x)
+ ) (
+ splitString ":" (
+ expand net
+ )
+ )
+ )
+ )
+ )
+ ) + ".ip6.arpa";
+
+ networkOf = address: prefixLength: encode (
+ zipListsWith bitAnd (
+ decode address
+ ) (
+ netmask prefixLength
+ )
+ );
+
+ isInNetwork = net: address: networkOf address == (expand net);
+
+ /* nixos-specific stuff */
+
+ findOwnAddress = config: net: head (
+ filter (
+ isInNetwork net
+ ) (
+ configuredAddresses config
+ )
+ );
+
+ configuredAddresses = config: concatLists (
+ mapAttrsToList (
+ name: iface: iface.ipv6.addresses
+ ) config.networking.interfaces
+ );
+
+ };
+}