.gitlab-ci.yml

@@ -23,6 +23,7 @@ unit-tests:
 vulnerability-scan:
   stage: "test"
   script:
+    - "sed -i 's/undefined/\"unundefined\"/' morph/grid/local/secrets/users.nix"
     - "ci-tools/vulnerability-scan security-report.json"
     - "ci-tools/count-vulnerabilities <security-report.json"
   artifacts:

ci-tools/vulnerability-scan

@@ -21,7 +21,7 @@ OUTPUT=$1
 [ -e scan-target ] && rm -v scan-target
 nix-shell --run '
 set -x
-if morph_result=$(morph build morph/grid/testing/grid.nix 2>&1); then
+if morph_result=$(morph build morph/grid/local/grid.nix 2>&1); then
   object=$(echo "$morph_result" | tail -n 1)
   ln -s "$object" scan-target
 else

morph/README.rst

@@ -42,6 +42,23 @@ grid
 
 Specific grid definitions live in subdirectories beneath this directory.
 
+secrets
+~~~~~~~
+
+This must be created and populated before the grid can be built or deployed.
+
+This directory contains all of the secrets necessary to deploy the grid.
+Secrets beneath this directory are referenced by ``config.json`` and ``grid.nix``
+(and possibly elsewhere).
+Some of the paths are configurable and some are just convention.
+This path is **ignored** by git.
+The intended workflow is that the secrets will be maintained on secure storage and a symlink to the correct location created here.
+This keeps the secrets themselves out of the git working tree as an extra protection against unintentionally committing them.
+
+An exception is the ``secrets`` directory in the ``local`` morph grid:
+That directory is fully populated, provided as an example, and mostly: not very secret.
+Do not deploy these keys to machines reachable via the internet.
+
 config.json
 ~~~~~~~~~~~
 

morph/grid/local/.gitignore

 .vagrant
-

morph/grid/local/README.rst

 Set up and use a network of local development VMs
 -------------------------------------------------
 
-... using `Vagrant <https://www.vagrantup.com/>`_ to manage VirtualBox VMs [#]_.
-To get started, first install Vagrant and make sure it works.
-One possible way to do it in NixOS:
+... using `Vagrant <https://www.vagrantup.com/>`_ to manage VirtualBox VMs.
+(The author of this documentation wasted a lot of time trying to get Vagrant to work with KVM/libvirt.
+Issues with networking that looked like guest misconfigurations vanished after changing to the better-tested combination of Vagrant and VirtualBox.)
 
-1. Install Vagrant, by adding the packages:
-
-  - ``vagrant`` (orchestrating virtual machines on the command line)
-     - Only use when version >= 2.2.16 has become available.  Else see below.
-  - Optional: ``packer`` (for creating your own VM images)
-
-2. Add configuration to install and enable VirtualBox:
-
-  - ``virtualisation.virtualbox.host.enable = true;``
-
-3. Add your user to the ``vboxusers`` group, for example:
-
-  - ``users.extraGroups.vboxusers.members = [ "flo" "jp" ];``
-
-
-.. [#] The author of this documentation wasted a lot of time trying to get Vagrant to work with KVM/libvirt.  Issues with networking that looked like guest misconfigurations vanished after changing to the better-tested combination of Vagrant and VirtualBox.
 
+Use the local development environment
+`````````````````````````````````````
 
-Pre-Vagrant 2.2.16: Get Vagrant with the required fixes for NixOS guests
-````````````````````````````````````````````````````````````````````````
+1. Enter the morph local grid directory::
 
-The Vagrant nixos-guest template `received a critical update on 2021-03-08 <https://github.com/hashicorp/vagrant/commit/990d94ed9d0b3092e855bc1bb9deeeb7aa7792cf>`_ which came out with Vagrant version 2.2.16.
+    cd morph/grid/local
 
-If you run an older Nixpkgs, retrieve and use the latest Vagrant development version like so::
+2. Enter the project's nix-shell::
 
-  NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/heads/master.tar.gz nix-shell -p vagrant
+    nix-shell ../../../shell.nix
 
-
-Use the local development environment
-`````````````````````````````````````
-
-1. Build and start the VMs::
+3. Build and start the VMs::
 
     VAGRANT_DEFAULT_PROVIDER=virtualbox vagrant up
 
-2. Then, once::
+4. Then, add the Vagrant SSH configuration to your user's ``~/.ssh/config`` file::
 
-    vagrant ssh-config > ./vagrant-ssh-config
+    install -d ~/.ssh ; vagrant ssh-config >> ~/.ssh/config
 
-3. Edit the output: Add the IPs from ``grid.nix`` to the ``vagrant-ssh-config`` **Host match blocks** so the config reads like::
+5. Edit the generated configuration: Add the ``publicIP`` addresses from ``grid.nix`` to ssh config **Host** match blocks (**not** HostName) so the ``Host`` lines all read like::
 
     Host payments1 192.168.67.21
-      HostName 192.168.67.21
+      HostName 127.0.0.1
       User vagrant
-      Port 22
       [...]
 
-4.  Then, make morph use this ssh config either - with newer morph [#]_ - by pointing it to it::
-
-     export SSH_CONFIG_FILE=./vagrant-ssh-config
+  Latest Morph honors the ``SSH_CONFIG_FILE`` environment variable (`since 3f90aa88 (March 2020, v 1.5.0) <https://github.com/DBCDK/morph/commit/3f90aa885fac1c29fce9242452fa7c0c505744ef#diff-d155ad793bd62e6ea4c44ba985049ecb13a4f4f32f799791b2bce695a16c0101>`_), so in the future this should get a bit more convenient.
 
-  Or, with older morph, adding the config to your user's ``~/.ssh/config`` file.
+6. Add your SSH key to ``users.nix`` so you'll be able to log in after deploying the new configuration::
 
-  .. [#]  Morph honors the ``SSH_CONFIG_FILE`` environment variable `since 3f90aa88 (March 2020, v 1.5.0) <https://github.com/DBCDK/morph/commit/3f90aa885fac1c29fce9242452fa7c0c505744ef#diff-d155ad793bd62e6ea4c44ba985049ecb13a4f4f32f799791b2bce695a16c0101>`_.
+    $EDITOR secrets/users.nix
 
-5. Then, build and deploy our software to the Vagrant VMs::
+7. Then, build and deploy our software to the Vagrant VMs::
 
     morph build grid.nix
     morph push grid.nix
+    morph deploy grid.nix boot
+    vagrant halt
+    vagrant up
     morph upload-secrets grid.nix
-    morph deploy grid.nix switch
 
-  You will now be able to log in with the users and keys you set in your ``localdev-users.nix`` file.
+  You should now be able to log in with the users and keys you set in your ``users.nix`` file.
 

morph/grid/local/Vagrantfile

@@ -14,6 +14,10 @@ Vagrant.configure("2") do |config|
     config.vm.box_version = "20.09"
     config.vm.box_check_update = false
     config.vm.network "private_network", ip: "192.168.67.21"
+    # Add self signed SSL key for zkap-issuer:
+    config.vm.provision "file", source: "secrets/payments-localdev-ssl", destination: "/tmp/payments-localdev-ssl"
+    config.vm.provision "shell", inline: "sudo mkdir -p /var/lib/letsencrypt/live/payments.localdev/"
+    config.vm.provision "shell", inline: "sudo mv /tmp/payments-localdev-ssl/* /var/lib/letsencrypt/live/payments.localdev/"
   end
 
   config.vm.define "storage1" do |config|

morph/grid/local/config.json

 { "publicStoragePort": 8898
-, "ristrettoSigningKeyPath": "../../PrivateStorageSecrets/ristretto.signing-key"
-, "stripeSecretKeyPath": "../../PrivateStorageSecrets/privatestorageio-testing-stripe.secret"
-, "monitoringvpnKeyDir": "../../PrivateStorageSecrets/monitoringvpn"
+, "ristrettoSigningKeyPath": "./secrets/ristretto.signing-key"
+, "stripeSecretKeyPath": "./secrets/stripe.secret"
+, "monitoringvpnKeyDir": "./secrets/monitoringvpn"
 , "monitoringvpnEndpoint": "192.168.67.24:51820"
 , "passValue": 1000000
-, "issuerDomain": "payments.localdev"
+, "issuerDomains": ["payments.localdev"]
 , "letsEncryptAdminEmail": "florian@privatestorage.io"
 , "allowedChargeOrigins": [
     "http://localhost:5000"

morph/grid/local/grid.nix

@@ -6,40 +6,46 @@ import ../../lib/make-grid.nix {
   config = ./config.json;
   nodes = cfg:
   let
-    sshUsers = import ../../../../PrivateStorageSecrets/localdev-users.nix;
+    sshUsers = import ./secrets/users.nix;
     vpnClientIPs = [ "172.23.23.11" "172.23.23.12" "172.23.23.13" ]; # TBD: derive automatically
+    # Get absolute vpn key directory path, as a string:
+    monitoringvpnKeyDir = toString ./. + "/${cfg.monitoringvpnKeyDir}";
   in {
-    "payments1" = import ../../lib/make-issuer.nix (rec {
+    "payments1" = import ../../lib/make-issuer.nix (cfg // rec {
       publicIPv4 = "192.168.67.21";
       monitoringvpnIPv4 = "172.23.23.11";
+      inherit monitoringvpnKeyDir;
       inherit sshUsers;
       hardware = import ./virtual-hardware.nix ({ inherit publicIPv4; });
       stateVersion = "19.03";
-    } // cfg);
+    });
 
-    "storage1" = import ../../lib/make-testing.nix (rec {
+    "storage1" = import ../../lib/make-testing.nix (cfg // rec {
       publicIPv4 = "192.168.67.22";
       monitoringvpnIPv4 = "172.23.23.12";
+      inherit monitoringvpnKeyDir;
       inherit sshUsers;
       hardware = import ./virtual-hardware.nix ({ inherit publicIPv4; });
       stateVersion = "19.09";
-    } // cfg);
+    });
 
-    "storage2" = import ../../lib/make-testing.nix (rec {
+    "storage2" = import ../../lib/make-testing.nix (cfg // rec {
       publicIPv4 = "192.168.67.23";
       monitoringvpnIPv4 = "172.23.23.13";
+      inherit monitoringvpnKeyDir;
       inherit sshUsers;
       hardware = import ./virtual-hardware.nix ({ inherit publicIPv4; });
       stateVersion = "19.09";
-    } // cfg);
+    });
 
-    "monitoring1" = import ../../lib/make-monitoring.nix (rec {
+    "monitoring1" = import ../../lib/make-monitoring.nix (cfg // rec {
       publicIPv4 = "192.168.67.24";
       monitoringvpnIPv4 = "172.23.23.1";
       inherit vpnClientIPs;
       inherit sshUsers;
+      inherit monitoringvpnKeyDir;
       hardware = import ./virtual-hardware.nix ({ inherit publicIPv4; });
       stateVersion = "19.09";
-    } // cfg);
+    });
   };
 }

morph/grid/local/secrets/monitoringvpn/172.23.23.11.key

+cLP62YAYoA7FY+OhSLR64DIHekOjGGQlfJAWp5cYP00=

morph/grid/local/secrets/monitoringvpn/172.23.23.11.pub

+GYNjLkoyQ1d3OMymYbgq40WAHIUzrSEGBWXvxqceF00=

morph/grid/local/secrets/monitoringvpn/172.23.23.12.key

+qFjBtvJKBchzl2HwFvEDoe3zFzyc10osiRlP8HOk2n0=

morph/grid/local/secrets/monitoringvpn/172.23.23.12.pub

+veio/0E0sJYOjwp3E8EccCyME1pqjkZr4R6whFMdrhs=

morph/grid/local/secrets/monitoringvpn/172.23.23.13.key

+8HlKTvxZBAZeww6JaNk9kBPjSfT0pVMbDJbzV67yUGE=

morph/grid/local/secrets/monitoringvpn/172.23.23.13.pub

+4VlUMl9FubrLWaN0pRvfdNjjRBQzfCVLMA2lU7OwPzA=

morph/grid/local/secrets/monitoringvpn/preshared.key

+E7KTLVnWMmP/mIEkU8WX2DBZJaeMS2+sYArRZuGT1o4=

morph/grid/local/secrets/monitoringvpn/server.key

+iOp2pk2HWyNgRnke7nJgFwodkTWMyHRIKwe8pk+bN3M=

morph/grid/local/secrets/monitoringvpn/server.pub

+ojo+p9ZE03GN66ewoIlrHmyV7ICt+2LV32Prs66JsA4=

morph/grid/local/secrets/payments-localdev-ssl/cert.pem

+-----BEGIN CERTIFICATE-----
+MIIDGTCCAgGgAwIBAgIUM9YnOMe2yYQuCLpYiI2TpZg21AMwDQYJKoZIhvcNAQEL
+BQAwHDEaMBgGA1UEAwwRcGF5bWVudHMubG9jYWxkZXYwHhcNMjEwNjA5MTQzNzU4
+WhcNMzEwNjA3MTQzNzU4WjAcMRowGAYDVQQDDBFwYXltZW50cy5sb2NhbGRldjCC
+ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBCvkMprdRjD8ECeweMVTFT
+pILu5VA8HC8XK15Y4iE8G48cSYE3m/e6w5g4JvfXgED9eY+1/ZazQXVI6sojnfHL
+Vj9XqsUww3IVtq2Zn+0B8YTrbNxcDH77mUx0mUziAm1bglmoNbd4+q0Fe8FhK/EN
+Jj7UppRjD2ziV9Mw7UT1JjMbfSo7/7ZOV2cWk+hezOywsE+3BM6mju1aHbbYryAd
+EzRwfQKcI/9PC84Bck9e+tEiWQImBf8DguQ+ChuSOmGtP1rI+hq5HfCimH0NCCNS
+iUfNVcubz920FRBeAx9oM0G/u4feeS8T5a1PThZrdhjIycEmU5wsM7F34RzkuEkC
+AwEAAaNTMFEwHQYDVR0OBBYEFAbBfAFOlx4c53CNgsRlobFwUIBbMB8GA1UdIwQY
+MBaAFAbBfAFOlx4c53CNgsRlobFwUIBbMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
+hvcNAQELBQADggEBACD+47l30gGaZBPvuCMc/CENLmuqZqj7WGGmWcmUAzsdEgHt
+Q1zhj4un4u4qQ78jeT+cjqNK9qartQIjxzYTpn6lKDBpM9sS2G7RdLxOWWRVAXQj
+Cv6l0tEQArIqqMnzmECe0VDCc97kTg6tnMLFeyQL29XSSNIpS0DiQ+NC69fyrF2E
+vt2bAi+QTlN1U1ZGDJqWlwqkpI3xTRDJnmRPuVNrt07gkMWeSJHEI98Qcve7Ujf3
+hFcAnbshJ8iXxbNjTFIQcCJzHq1UF+eZo8BF1ixYmrk/jmaLNBOSy93tOk1XeJfS
+XtV6X9jmAE20V0XV8uCli6cDxQQYthll/1q5JYQ=
+-----END CERTIFICATE-----

morph/grid/local/secrets/payments-localdev-ssl/privkey.pem

+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDgQr5DKa3UYw/B
+AnsHjFUxU6SC7uVQPBwvFyteWOIhPBuPHEmBN5v3usOYOCb314BA/XmPtf2Ws0F1
+SOrKI53xy1Y/V6rFMMNyFbatmZ/tAfGE62zcXAx++5lMdJlM4gJtW4JZqDW3ePqt
+BXvBYSvxDSY+1KaUYw9s4lfTMO1E9SYzG30qO/+2TldnFpPoXszssLBPtwTOpo7t
+Wh222K8gHRM0cH0CnCP/TwvOAXJPXvrRIlkCJgX/A4LkPgobkjphrT9ayPoauR3w
+oph9DQgjUolHzVXLm8/dtBUQXgMfaDNBv7uH3nkvE+WtT04Wa3YYyMnBJlOcLDOx
+d+Ec5LhJAgMBAAECggEARI3in6FkFCLcNAJQHbSWbmfFSIlC7E4Tx4lrpoHBTquT
+OSJKjgez0/zxwdyYfPcRq8xQls/pX2IYxoOt0nEk3T9tdBuWhoUrmfptR5BIxSjs
+7dcSBiLVZxP+ftK98jS8zTVGGaZEFXwUFUQx2qGbzypX4Kkc6wuFMaHXeyXfwk4j
+v3lOTpT7kX/3iVXzQU9Y5m45W1bbt3lSV4yCndOt/4Xg5d4ue6UcUsyg/r8m5wsU
+nCc14L0HsY/VCCRISmDwa4UoqHGfQv6+XXxLT8gJM7H8PkZK8TRV/ZG2Yug63wDS
+xkhMZBodGoYuQpB19Y+NxjwG8Se7RxcCAyS7zcYcCQKBgQD/VWesc2/TplikifQf
+LJl+utiQ9G/o3wAquAKInxQOcUxOm1fsvjgsLfrjpAVj0D3qD0Sf3lSp7z3tpUMf
+MSi9MgNunSU7pKST7jex6BJNU0TTRcFnoqcIlUYiDoLLbkkVlf8Nome88gkwJRKZ
+RjI+6ES9hGaM85QEgjdtRNGXIwKBgQDg2JPfVXYyrOIITus/DHHv8em1RuGCIe6X
+T3M0+GgJ27hOEeneAyzk4aWvVzE3/3lxx55ePGRAz43jPrUxn0kfy/Jx4UMKsEW+
+XzR7CfxTxamUx/mq2bbqk+bbnXZZQedAip4eQ23GbWdrE4/v83JXzWj355yUQCQk
+uuIcda7fowKBgEY20CmmHOxQ5DNrFEy2UQd+jitebJ/XIw6cR2YWiMdn9JnxMf6S
+WJQdmM6cvjayfzQsOqzT0OhiN99wAMNFG3TbmgIDCMgcAH4Flh9AODg3W8fVeNfs
+7I35rq2S2/jhPQvIkbjIHkrhLBGnQDQSD6Mo8C5FiIXePaf3vxI3SIONAoGAZdOv
+pEUf8nM5KmoTP8pzDyePn/kpx7V2SDBDDIozE8PeA/043MKzYjSOxInIUIPyjATL
+RAI1pORabb/Ib2CjzTKf6dMKeZy6+SxEqDQtggLSef7WovlWTYYN1wfIwUOHZ0Nf
+uHTxEhwZ6fRCC3lFH153W04ZK0qhE8FPBXSGbeECgYBcY0QmOd3ao3v0xscKmPlM
+E9MICgwgEJ6pAsaOI30OHqUAtlqrlw5ph3skxlLUpHWW6OUczUUV6nwguk4piuee
+1J9uuA02FlIvvUy5Bsrgqlu0Lj5vJ8Im2Go9dJ8k/m9J5BUgDdPuNwnmjiyNQurd
+Fl50FWx/3WeJB2qK981vmw==
+-----END PRIVATE KEY-----