diff --git a/docs/source/ops/generating-keys.rst b/docs/source/ops/generating-keys.rst
index afe2ece4009f761aea56acd24fcbf627b985cadb..47a1f4e91a876ac1919252c099654886f0bd128a 100644
--- a/docs/source/ops/generating-keys.rst
+++ b/docs/source/ops/generating-keys.rst
@@ -1,7 +1,28 @@
Generating keys
===============
-``config.json`` has the paths for the Ristretto and the Stripe secret key files.
+There's an example ``secrets`` repo in ``morph/grid/local/secrets``.
+``<grid>/config.json`` has the paths for the key files for the respective grid.
+Create a symlink named ``secrets`` to your secret key repository for the deployment you are working on.
+
+
+Stripe
+``````
+
+For the Stripe key any random bytes with a little light formatting "work" - at least to make our software happy - but if you want to be able to interact with Stripe and have payments (even pretend payments) move all the way through the system you should get a Stripe account and generate a key w/ them.
+Lauri can get you added to our "dev" Stripe account, too, though I forget how important that is for ad hoc dev/testing.
+
+I think this will work for generating random Stripe secret keys (that our software will load, I think, but Stripe will reject)::
+
+ >>> import base64, os
+ >>> print((b"sk_test_" + base64.b64encode(os.urandom(25)).strip(b"=")).decode("ascii"))
+ sk_test_Dr+XLVjkC0oO3Zw8Ws0yWtDLqR1sM+/fmw
+
+Public keys are the same but "pk_test" instead of "sk_test" ("test" is for "test mode" key that can only process pretend txns; for real txns there are keys with "live" embedded).
+
+
+ZKAP-Issuer Ristretto
+`````````````````````
Here is a Ristretto key you can use, randomly generated just now::
@@ -19,16 +40,9 @@ For example::
echo -n "SILOWzbnkBjxC1hGde9d5Q3Ir/4yLosCLEnEQGAxEQE=" > ristretto.signing-key
-For the Stripe key any random bytes with a little light formatting "work" - at least to make our software happy - but if you want to be able to interact with Stripe and have payments (even pretend payments) move all the way through the system you should get a Stripe account and generate a key w/ them.
-Lauri can get you added to our "dev" Stripe account, too, though I forget how important that is for ad hoc dev/testing.
-
-I think this will work for generating random Stripe secret keys (that our software will load, I think, but Stripe will reject)::
-
- >>> import base64, os
- >>> print((b"sk_test_" + base64.b64encode(os.urandom(25)).strip(b"=")).decode("ascii"))
- sk_test_Dr+XLVjkC0oO3Zw8Ws0yWtDLqR1sM+/fmw
-Public keys are the same but "pk_test" instead of "sk_test" ("test" is for "test mode" key that can only process pretend txns; for real txns there are keys with "live" embedded).
+ZKAP-Issuer TLS
+```````````````
The ZKAPIssuer.service needs a working TLS certificate and expects it in the certbot directory for the domain you configured, in my case::
@@ -37,14 +51,26 @@ The ZKAPIssuer.service needs a working TLS certificate and expects it in the cer
Move the three .pem files into the payment's server ``/var/lib/letsencrypt/live/payments.localdev/`` directory and issue a ``sudo systemctl restart zkapissuer.service``.
-Create Wireguard VPN key pairs in ``PrivateStorageSecrets/monitoringvpn/`` or where you have them::
- for i in "172.23.23.11" "172.23.23.12" "172.23.23.13" "server"; do
- wg genkey | tee ${i}.key | wg pubkey > ${i}.pub
+Monitoring VPN
+``````````````
+
+Create Wireguard VPN key pairs in ``secrets/monitoringvpn/`` or where you have them.
+
+``tools/create-vpn-keys.sh`` holds a script to rotate all VPN keys at once::
+
+ ./tools/create-vpn-keys.sh morph/grid/testing/grid.nix
+
+Or do it manually::
+
+ cd secrets/monitoringvpn
+ for i in 1 11 12 13 ; do
+ wg genkey | tee 172.23.23.${i}.key | wg pubkey > 172.23.23.${i}.pub
done
+ ln -s 172.23.23.1.key server.key
+ ln -s 172.23.23.1.pub server.pub
+
And a shared VPN key for "post-quantum resistance"::
wg genpsk > preshared.key
-
-
diff --git a/morph/grid/local/README.rst b/morph/grid/local/README.rst
index 59f032a0fad5bf2fd328955db7649b3cd8b288e4..345547244635734278aa76cb5cd59946f2afd37f 100644
--- a/morph/grid/local/README.rst
+++ b/morph/grid/local/README.rst
@@ -14,7 +14,7 @@ Use the local development environment
0. Add VirtualBox to your NixOs system configuration at ``/etc/nixos/configuration.nix``::
virtualisation.virtualbox.host.enable = true;
- # Use VirtualBox installation without GUI and Qt dependency:
+ # Save bytes and build time, optional but recommended:
virtualisation.virtualbox.host.headless = true;
1. Enter the morph local grid directory::
@@ -35,7 +35,7 @@ Use the local development environment
5. Edit the generated configuration: Add the ``publicIP`` addresses from ``grid.nix`` to ssh config **Host** match blocks (**not** HostName) so the ``Host`` lines all read like::
- Host payments1 192.168.67.21
+ Host payments 192.168.67.21
HostName 127.0.0.1
User vagrant
[...]
diff --git a/morph/grid/local/Vagrantfile b/morph/grid/local/Vagrantfile
index 82bbef1063b108829261670fdceb2e27af8d6764..7ad95ca872a72e5da6c11b3269e2a824cf8a55f9 100644
--- a/morph/grid/local/Vagrantfile
+++ b/morph/grid/local/Vagrantfile
@@ -8,8 +8,8 @@ Vagrant.configure("2") do |config|
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
- config.vm.define "payments1" do |config|
- config.vm.hostname = "payments1"
+ config.vm.define "payments" do |config|
+ config.vm.hostname = "payments"
config.vm.box = "esselius/nixos"
config.vm.box_version = "20.09"
config.vm.box_check_update = false
@@ -36,8 +36,8 @@ Vagrant.configure("2") do |config|
config.vm.network "private_network", ip: "192.168.67.23"
end
- config.vm.define "monitoring1" do |config|
- config.vm.hostname = "monitoring1"
+ config.vm.define "monitoring" do |config|
+ config.vm.hostname = "monitoring"
config.vm.box = "esselius/nixos"
config.vm.box_version = "20.09"
config.vm.box_check_update = false
diff --git a/morph/grid/local/grid.nix b/morph/grid/local/grid.nix
index 0c114ccf11b9ddb963d5e2e6297b6a7c83792aba..fdc0cde55be4f1b644c212ce20f6c3e44af8e3df 100644
--- a/morph/grid/local/grid.nix
+++ b/morph/grid/local/grid.nix
@@ -13,16 +13,16 @@ import ../../lib/make-grid.nix {
# TBD: derive these automatically:
hostsMap = {
- "172.23.23.1" = [ "monitoring1" "monitoring1.monitoringvpn" ];
- "172.23.23.11" = [ "payments1" "payments1.monitoringvpn" ];
+ "172.23.23.1" = [ "monitoring" "monitoring.monitoringvpn" ];
+ "172.23.23.11" = [ "payments" "payments.monitoringvpn" ];
"172.23.23.12" = [ "storage1" "storage1.monitoringvpn" ];
"172.23.23.13" = [ "storage2" "storage2.monitoringvpn" ];
};
vpnClientIPs = [ "172.23.23.11" "172.23.23.12" "172.23.23.13" ];
- nodeExporterTargets = [ "monitoring1" "payments1" "storage1" "storage2" ];
+ nodeExporterTargets = [ "monitoring" "payments" "storage1" "storage2" ];
in {
- "payments1" = import ../../lib/make-issuer.nix (cfg // rec {
+ "payments" = import ../../lib/make-issuer.nix (cfg // rec {
publicIPv4 = "192.168.67.21";
monitoringvpnIPv4 = "172.23.23.11";
hardware = import ./virtual-hardware.nix ({ inherit publicIPv4; });
@@ -49,7 +49,7 @@ import ../../lib/make-grid.nix {
inherit sshUsers;
});
- "monitoring1" = import ../../lib/make-monitoring.nix (cfg // rec {
+ "monitoring" = import ../../lib/make-monitoring.nix (cfg // rec {
publicIPv4 = "192.168.67.24";
monitoringvpnIPv4 = "172.23.23.1";
inherit vpnClientIPs;
diff --git a/morph/grid/production/config.json b/morph/grid/production/config.json
index e71cb8b4b5f999e3059f0669c2bc3f92f29242a6..ef7dc53649febcd7beb7901bb3608204df197059 100644
--- a/morph/grid/production/config.json
+++ b/morph/grid/production/config.json
@@ -1,6 +1,8 @@
{ "publicStoragePort": 8898
, "ristrettoSigningKeyPath": "./secrets/ristretto.signing-key"
, "stripeSecretKeyPath": "./secrets/stripe.secret"
+, "monitoringvpnKeyDir": "./secrets/monitoringvpn"
+, "monitoringvpnEndpoint": "monitoring.private.storage:51820"
, "passValue": 1000000
, "issuerDomains": [
"payments.privatestorage.io"
diff --git a/morph/grid/production/grid.nix b/morph/grid/production/grid.nix
index f5735d259dbff27f1d9cabbbca512af81d4550bb..fee0c9be6faed47d4a702b5b53c2419cbb677ba6 100644
--- a/morph/grid/production/grid.nix
+++ b/morph/grid/production/grid.nix
@@ -7,6 +7,38 @@ import ../../lib/make-grid.nix {
nodes = cfg:
let
sshUsers = import ./secrets/users.nix;
+
+ # Get absolute vpn key directory path, as a string:
+ monitoringvpnKeyDir = toString ./. + "/${cfg.monitoringvpnKeyDir}";
+
+ # TBD: derive these automatically:
+ hostsMap = {
+ "172.23.23.1" = [ "monitoring" "monitoring.monitoringvpn" ];
+ "172.23.23.11" = [ "payments" "payments.monitoringvpn" ];
+ "172.23.23.21" = [ "storage001" "storage001.monitoringvpn" ];
+ "172.23.23.22" = [ "storage002" "storage002.monitoringvpn" ];
+ "172.23.23.23" = [ "storage003" "storage003.monitoringvpn" ];
+ "172.23.23.24" = [ "storage004" "storage004.monitoringvpn" ];
+ "172.23.23.25" = [ "storage005" "storage005.monitoringvpn" ];
+ };
+ vpnClientIPs = [
+ "172.23.23.11"
+ "172.23.23.21"
+ "172.23.23.22"
+ "172.23.23.23"
+ "172.23.23.24"
+ "172.23.23.25"
+ ];
+ nodeExporterTargets = [
+ "monitoring"
+ "payments"
+ "storage001"
+ "storage002"
+ "storage003"
+ "storage004"
+ "storage005"
+ ];
+
in {
# Here are the hosts that are in this morph network. This is sort of like
# a server manifest. We try to keep as many of the specific details as
@@ -20,42 +52,66 @@ import ../../lib/make-grid.nix {
# doesn't specify one.
#
# The names must be unique!
- "payments.privatestorage.io" = import ../../lib/make-issuer.nix ({
+ "payments.privatestorage.io" = import ../../lib/make-issuer.nix (cfg // {
publicIPv4 = "18.184.142.208";
+ monitoringvpnIPv4 = "172.23.23.11";
+ inherit monitoringvpnKeyDir;
inherit sshUsers;
hardware = ../../lib/issuer-aws.nix;
stateVersion = "19.03";
- } // cfg);
+ });
- "storage001" = import ../../lib/make-storage.nix ({
+ "storage001" = import ../../lib/make-storage.nix (cfg // {
cfg = import ./storage001-config.nix;
inherit sshUsers;
hardware = ./storage001-hardware.nix;
stateVersion = "19.09";
- } // cfg);
- "storage002" = import ../../lib/make-storage.nix ({
+ monitoringvpnIPv4 = "172.23.23.21";
+ inherit monitoringvpnKeyDir;
+ });
+ "storage002" = import ../../lib/make-storage.nix (cfg // {
cfg = import ./storage002-config.nix;
inherit sshUsers;
hardware = ./storage002-hardware.nix;
stateVersion = "19.09";
- } // cfg);
- "storage003" = import ../../lib/make-storage.nix ({
+ monitoringvpnIPv4 = "172.23.23.22";
+ inherit monitoringvpnKeyDir;
+ });
+ "storage003" = import ../../lib/make-storage.nix (cfg // {
cfg = import ./storage003-config.nix;
inherit sshUsers;
hardware = ./storage003-hardware.nix;
stateVersion = "19.09";
- } // cfg);
- "storage004" = import ../../lib/make-storage.nix ({
+ monitoringvpnIPv4 = "172.23.23.23";
+ inherit monitoringvpnKeyDir;
+ });
+ "storage004" = import ../../lib/make-storage.nix (cfg // {
cfg = import ./storage004-config.nix;
inherit sshUsers;
hardware = ./storage004-hardware.nix;
stateVersion = "19.09";
- } // cfg);
- "storage005" = import ../../lib/make-storage.nix ({
+ monitoringvpnIPv4 = "172.23.23.24";
+ inherit monitoringvpnKeyDir;
+ });
+ "storage005" = import ../../lib/make-storage.nix (cfg // {
cfg = import ./storage005-config.nix;
inherit sshUsers;
hardware = ./storage005-hardware.nix;
stateVersion = "19.03";
- } // cfg);
+ monitoringvpnIPv4 = "172.23.23.25";
+ inherit monitoringvpnKeyDir;
+ });
+
+ "monitoring" = import ../../lib/make-monitoring.nix (cfg // {
+ publicIPv4 = "monitoring.private.storage";
+ monitoringvpnIPv4 = "172.23.23.1";
+ inherit monitoringvpnKeyDir;
+ inherit vpnClientIPs;
+ inherit hostsMap;
+ inherit nodeExporterTargets;
+ hardware = ../../lib/issuer-aws.nix;
+ stateVersion = "19.09";
+ inherit sshUsers;
+ });
};
}
diff --git a/morph/lib/make-monitoring.nix b/morph/lib/make-monitoring.nix
index 8eb53c6db9552e65a84e5d3e5564449db437e902..592a859657e624e8fdf5632f8144c5acc6919e8c 100644
--- a/morph/lib/make-monitoring.nix
+++ b/morph/lib/make-monitoring.nix
@@ -61,7 +61,7 @@ in rec {
};
services.private-storage.monitoring.grafana = {
- domain = "grafana.grid.private.storage";
+ domain = "monitoring.private.storage";
prometheusUrl = "http://localhost:9090/";
lokiUrl = "http://localhost:3100/";
};
diff --git a/morph/lib/make-storage.nix b/morph/lib/make-storage.nix
index af0867c8b8342e31393f19a76a7cbfc4c95f86c9..6619336d758f69a677e9178592357480aed3f0c8 100644
--- a/morph/lib/make-storage.nix
+++ b/morph/lib/make-storage.nix
@@ -11,8 +11,36 @@
# to avoid breaking some software such as
# database servers. You should change this only
# after NixOS release notes say you should.
+, monitoringvpnKeyDir ? null # The directory that holds the VPN keys.
+, monitoringvpnIPv4 ? null # This node's IP in the monitoring VPN.
+, monitoringvpnEndpoint ? null # The VPN server and port.
, ...
-}: rec {
+}: let
+
+ enableVpn = monitoringvpnKeyDir != null &&
+ monitoringvpnIPv4 != null &&
+ monitoringvpnEndpoint != null;
+
+ vpnSecrets = if !enableVpn then {} else {
+ "monitoringvpn-secret-key" = {
+ source = monitoringvpnKeyDir + "/${monitoringvpnIPv4}.key";
+ destination = "/run/keys/monitoringvpn/client.key";
+ owner.user = "root";
+ owner.group = "root";
+ permissions = "0400";
+ action = ["sudo" "systemctl" "restart" "wireguard-monitoringvpn.service"];
+ };
+ "monitoringvpn-preshared-key" = {
+ source = monitoringvpnKeyDir + "/preshared.key";
+ destination = "/run/keys/monitoringvpn/preshared.key";
+ owner.user = "root";
+ owner.group = "root";
+ permissions = "0400";
+ action = ["sudo" "systemctl" "restart" "wireguard-monitoringvpn.service"];
+ };
+ };
+
+in rec {
deployment = {
targetHost = cfg.publicIPv4;
@@ -28,7 +56,7 @@
# extract it from the tahoe-lafs nixos module somehow?
action = ["sudo" "systemctl" "restart" "tahoe.storage.service"];
};
- };
+ } // vpnSecrets;
};
# Any extra NixOS modules to load on this server.
@@ -40,6 +68,10 @@
# Bring in our module for configuring the Tahoe-LAFS service and other
# Private Storage-specific things.
../../nixos/modules/private-storage.nix
+ # Connect to the monitoringvpn.
+ ../../nixos/modules/monitoring/vpn/client.nix
+ # Expose base system metrics over the monitoringvpn.
+ ../../nixos/modules/monitoring/exporters/node.nix
];
# Pass the configuration specific to this host to the 100TB module to be
@@ -67,4 +99,11 @@
};
system.stateVersion = stateVersion;
+
+ services.private-storage.monitoring.vpn.client = if !enableVpn then {} else {
+ enable = true;
+ ip = monitoringvpnIPv4;
+ endpoint = monitoringvpnEndpoint;
+ endpointPublicKeyFile = monitoringvpnKeyDir + "/server.pub";
+ };
}
diff --git a/shell.nix b/shell.nix
index 6e46c9ca0feaa3ab6fbd22c1228ec786a49e79b6..2c1c5123da656d34fafe0883b50ef49c578c6c8b 100644
--- a/shell.nix
+++ b/shell.nix
@@ -8,5 +8,6 @@ pkgs.mkShell {
buildInputs = [
pkgs.morph
stable2105.vagrant
+ pkgs.jp
];
}
diff --git a/tools/create-vpn-keys.sh b/tools/create-vpn-keys.sh
new file mode 100755
index 0000000000000000000000000000000000000000..e092a8ced698bd3a3bb2d4acc3ca07a3a8e6032d
--- /dev/null
+++ b/tools/create-vpn-keys.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# Scope: Create wireguard keys for all monitoringVPN hosts
+# Parameters:
+# file: path to grid.nix of morph deployment
+#
+# Output: Key files for all monitoring VPN hosts in secrets/monitoringvpn
+# relative to the grid.nix
+#
+# The server key will also be symlinked to server.{key,pub}.
+
+set -euxo pipefail
+
+umask 077
+
+if [[ $# -ne 1 ]]; then
+ echo "Illegal number of parameters. Expected: file (path of grid.nix)"
+ exit 2
+fi
+
+SRC=$(dirname $0)
+VPN_SECRETS=$(dirname $1)/secrets/monitoringvpn
+
+CONFIG=$(nix-instantiate --strict --json --eval "${SRC}"/get-vpn-config.nix --arg pathToGrid "${1}")
+
+MONITORING_IPS=$(echo $CONFIG | jp --unquoted "join(' ', clientIPs)")
+VPNSERVER_IP=$(echo $CONFIG | jp --unquoted "serverIP")
+
+mkdir -p "${VPN_SECRETS}"
+
+for i in $MONITORING_IPS $VPNSERVER_IP; do
+ wg genkey | tee "${VPN_SECRETS}"/${i}.key | wg pubkey > "${VPN_SECRETS}"/${i}.pub
+done
+
+wg genpsk > "${VPN_SECRETS}"/preshared.key
+
+ln -fs $VPNSERVER_IP.key "${VPN_SECRETS}"/server.key
+ln -fs $VPNSERVER_IP.pub "${VPN_SECRETS}"/server.pub
+
+# EOF
diff --git a/tools/get-vpn-config.nix b/tools/get-vpn-config.nix
new file mode 100644
index 0000000000000000000000000000000000000000..7753292aa83c4b63be7457228de0cd84e6eeefa2
--- /dev/null
+++ b/tools/get-vpn-config.nix
@@ -0,0 +1,19 @@
+# A function that accepts a path to a grid.nix-style file and returns a set
+# with two attributes:
+#
+# * serverIP - a string giving the VPN IP address of the grid's VPN server.
+#
+# * clientIPs - a list of strings giving the VPN IP addresses of all of the
+# grid's VPN clients.
+#
+{ pathToGrid }:
+let
+ grid = import pathToGrid;
+ vpnConfig = node: node.services.private-storage.monitoring.vpn or null;
+ vpnClientIP = node: (vpnConfig node).client.ip or null;
+ vpnServerIP = node: (vpnConfig node).server.ip or null;
+in
+{
+ "serverIP" = vpnServerIP grid.monitoring;
+ "clientIPs" = builtins.filter (x: x != null) (map vpnClientIP (builtins.attrValues grid));
+}