Newer
Older
This directory contains Nix-based configuration for the grid.
This takes the form of Nix expressions in ``.nix`` files
and some JSON-based configuration in ``.json`` files.
This configuration is fed to `morph`_ to make changes to the deployment.
The deployment consists of the public software packages and the private secrets.
You can deploy these together::
morph deploy --upload-secrets morph/grid/<testing|production|...>/grid.nix test
Or separately::
morph deploy morph/grid/<testing|production|...>/grid.nix test
morph upload-secrets morph/grid/<testing|production|...>/grid.nix
Separate deployment is useful when the software deploy is done from system which may not be sufficiently secure to host the secrets
(such as a cloud build machine).
Secrets should only be hosted on an extremely secure system
(XXX write the document for what this means).
Note secrets only need to be uploaded after a host in the grid has been rebooted or when the secrets have changed.
See the ``morph`` and ``nixos-rebuild`` documentation for more details about these commands.
Filesystem Layout
`````````````````
lib
---
This contains Nix library code for defining the grids.
It has all the details of how each type of node in our grid is configured, and does know about morph (so defines deployment.secrets, and has the logic for collecting data defined by other nodes); and defines options (i.e. grid.*) for things specific to how we configure grids (i.e. grid.publicKeyPath), metadata about nodes that we use on other nodes (i.e. grid.monitoringvpnIPv4 which is used to define various things on the monitoring node).
Each top-level module here defines one type of node, with all (or at least most) of the configuration necessary for that node.
grid
----
Specific grid definitions live in subdirectories beneath this directory.
They consist almost exclusively setting options defined in morph/lib (and few options defined elsewhere), and then delegating to the morph/lib modules.
private-keys
~~~~~~~~~~~~
Jean-Paul Calderone
committed
This must be created and populated before the grid can be built or deployed.
This directory contains all of the secrets necessary to deploy the grid.
Secrets beneath this directory are referenced by ``config.json`` and ``grid.nix``
(and possibly elsewhere).
Some of the paths are configurable and some are just convention.
This path is **ignored** by git.
The intended workflow is that the secrets will be maintained on secure storage and a symlink to the correct location created here.
This keeps the secrets themselves out of the git working tree as an extra protection against unintentionally committing them.
An exception is the ``private-keys`` directory in the ``local`` morph grid:
That directory is fully populated, provided as an example, and mostly: not very secret.
Do not deploy these keys to machines reachable via the internet.
Strictly speaking,
this path is configurable in the grid's ``config.json`` but all three grids currently use this name.
public-keys
~~~~~~~~~~~
This must be created and populated before the grid can be built or deployed.
This directory contains any public key material necessary for operation of the grid.
This includes the public keys corresponding to any private keys held in ``private-keys``.
As for ``private-keys``,
this path can be configured in the grid's ``config.json``.
Star-crossed Keys
^^^^^^^^^^^^^^^^^
Where the system uses keypairs,
the public and private parts of those keypairs are stored in different locations
(``public-keys`` and ``private-keys`` mentioned above).
This somewhat complicates key management because any key rotation involves changing key material in two location instead of just one.
This complication is balanced against a specific operational goal:
that our build systems operate without copies of our private keys.
Our system configurations do currently have build-time dependencies on public keys.
Splitting public keys and private keys across two different storage locations provides a simple mechanism for providing build systems with the public keys but withholding the private keys.
In the future we may:
* be sufficiently confident in the security of our build systems to let them have our private keys; or
* remove the dependency upon public keys from the build process.
Either of these directions would let us re-unify public/private-key storage and remove this complication.
config.json
~~~~~~~~~~~
As much as possible of the static configuration for the PrivateStorage.io application is provided in this file.
It is read by **grid.nix**.
~~~~~~~~
This is the `morph`_ entrypoint for the grid.
This defines all of the servers that are part of the grid.
The actual configuration is split into separate files that are imported from this one.
You can do things like build the network::
morph build grid.nix
~~~~~~~~~~~~~~~~~~~~~~~
These are the generated hardware-related configuration files for servers in the grid.
These files are referenced from the corresponding ``<hostname>.nix`` files.
<hostname>-config.nix
~~~~~~~~~~~~~~~~~~~~~
Each such file contains a minimal Nix expression supplying critical system configuration details.
"Critical" roughly corresponds to anything which must be specified to have a bootable system.
These files are referenced by the corresponding ``<hostname>.nix`` files.
Configuring New Storage Nodes
Storage nodes are brought into the grid in a multi-step process.
Here are the steps to configure a new node,
starting from a minimal NixOS 19.03 or 19.09 installation.
#. Copy the remote file ``/etc/nixos/hardware-configuration.nix`` to the local file ``storageNNN-hardware.nix``.
In the case of an EC2 instance, copy the remote file ``/etc/nixos/configuration.nix`` instead.
#. Add ``"zfs"`` to ``boot.supportedFilesystems`` in ``storageNNN-hardware.nix``.
#. Add a unique value for ``networking.hostId`` in ``storageNNN-hardware.nix``.
#. Copy ``storageNNN-hardware.nix`` back to ``/etc/nixos/hardware-configuration.nix``.
#. Run ``nixos-rebuild test``.
#. Manually create a storage zpool::
zpool create -m legacy -o ashift=12 root raidz /dev/disk/by-id/{...}
#. Mount the new ZFS filesystem to verify it is working::
mkdir /storage
mount -t zfs root /storage
#. Add a new filesystem entry to ``storageNNN-hardware.nix``::
# Manually created using:
# zpool create -f -m legacy -o ashift=12 root raidz ...
fileSystems."/storage" = {
device = "root";
fsType = "zfs";
};
#. Create a ``storageNNN-config.nix`` containing further configuration for the new host.
#. Add an entry for the new host to ``grid.nix`` referencing the new files.
#. Deploy to the new host with ``morph deploy morph/.../grid.nix --on <identifier> boot --upload-secrets --reboot``.
.. _`morph`: https://github.com/DBCDK/morph