Find a way to minimize the closure size of our deployments.
We should minimize the amount of software deployed on our servers, in particular, including GUI libraries.
From @jcalderone on https://whetstone.privatestorage.io/privatestorage/PrivateStorageio/-/merge_requests/182#note_16061
I see there is a sibling config option,
x11, which defaults to true. Maybe we can cut down the closure size by setting it to false. This could also be fine as a follow-up. I don't think this changeset adds any x11 dependencies (at least not according todiff-closures).
and https://whetstone.privatestorage.io/privatestorage/PrivateStorageio/-/merge_requests/182#note_16134
I'm not sure if there's a way to disable all GUI dependencies at once but a related idea might be having a safety net - we could probably have some part of the system fail loudly if it discovers some basic GUI stuff has made its way into our closure (eg libX11 or something similar). Then at least someone would have to stop and fix the issue before proceeding...
A different (maybe complementary) approach could be to have the closure size of various expressions tracked by a monitoring system? Then we can see how we're doing over time, and maybe understand contributing factors other than GUI stuff.
Activity
added Tech-Debt label
One thing I noticed while writing up #93, is that we pull in w3m (which has a vulnerabilitiy), as a dependency of
nixos-help(since it is the default browser used by that script).
We really install much too much software.
Example: Samba
zfs-user(land) depends on Samba (which has multiple CVEs)
$ nix why-depends /nix/store/4cia66md9qha0inqnc68bg9094425n41-nixos-system-storage004-21.11.33 7971.a1fe662eb26.drv /nix/store/b2c2yk0rzr5z1s381npc60kjq8ynjhn3-samba-4.15.5.drv/nix/store/4cia66md9qha0inqnc68bg9094425n41-nixos-system-storage004-21.11.337971.a1fe662eb26.drv ╚═══/: …1-p8.drv",["out"]),("/nix/store/2gy7m3c5vybfmrk7cd1ricnzxxs7bxjs-system-path.drv",["out"]),("/ni… => /nix/store/2gy7m3c5vybfmrk7cd1ricnzxxs7bxjs-system-path.drv ╚═══/: …v",["info","out"]),("/nix/store/69ifhwh7jvylvf62wdcpcjgbsqgj8mj9-zfs-user-2.1.4.drv",["out"]),("… => /nix/store/69ifhwh7jvylvf62wdcpcjgbsqgj8mj9-zfs-user-2.1.4.drv ╚═══/: …9.13.drv",["out"]),("/nix/store/b2c2yk0rzr5z1s381npc60kjq8ynjhn3-samba-4.15.5.drv",["out"]),("/n… => /nix/store/b2c2yk0rzr5z1s381npc60kjq8ynjhn3-samba-4.15.5.drvExample: QEMU
cloud-utils depends on QEMU-utils depends on QEMU depends on Mesa depends on Wayland depends on Graphviz depends on OpenEXR (or so)... and GTK... and Cups,the standards-based, open source printing system.
$ nix why-depends /nix/store/l421fbjvnx30swybfnxwimpwrs9p5mmr-nixos-system-payments-21.11.3379 71.a1fe662eb26.drv /nix/store/vbs4zhgi8v924n2d7079vfbsyba7x1hj-openexr-2.5.7.drv/nix/store/l421fbjvnx30swybfnxwimpwrs9p5mmr-nixos-system-payments-21.11.337971.a1fe662eb26.drv ╚═══/: …2.10.drv",["out"]),("/nix/store/d5z01w6x6m4smqack9xs7v05zv1l9ddm-initrd-linux-5.10.126.drv",["ou… => /nix/store/d5z01w6x6m4smqack9xs7v05zv1l9ddm-initrd-linux-5.10.126.drv ╚═══/: …2.13.drv",["out"]),("/nix/store/9z6819v5gv7zxcdmdlda1z9j05i7581p-stage-1-init.sh.drv",["out"]),(… => /nix/store/9z6819v5gv7zxcdmdlda1z9j05i7581p-stage-1-init.sh.drv ╚═══/: …nits.drv",["out"]),("/nix/store/qxlai0n38ybg7v8awh75m63x47bd1cgn-extra-utils.drv",["out"]),("/ni… => /nix/store/qxlai0n38ybg7v8awh75m63x47bd1cgn-extra-utils.drv ╚═══/: …37.4.drv",["bin"]),("/nix/store/1frijx0cafc1zp3497vn52mwplf5frx3-cloud-utils-0.32.drv",["guest"]… => /nix/store/1frijx0cafc1zp3497vn52mwplf5frx3-cloud-utils-0.32.drv ╚═══/: …1.11.drv",["out"]),("/nix/store/q0vdindk04qcf87ln737bc4jv5ibgbvz-qemu-utils-6.1.1.drv",["out"]),… => /nix/store/q0vdindk04qcf87ln737bc4jv5ibgbvz-qemu-utils-6.1.1.drv ╚═══/: …inux.drv",["out"]),("/nix/store/zgvf44mgwb3y9i91bp5mbsab3mz8lq9s-qemu-6.1.1.drv",["out"])],["/ni… => /nix/store/zgvf44mgwb3y9i91bp5mbsab3mz8lq9s-qemu-6.1.1.drv ╚═══/: …19.0.drv",["out"]),("/nix/store/g72qpl1bp78x1n5qhbg6zkp4f1y6hxy4-mesa-21.2.6.drv",["dev"]),("/ni… => /nix/store/g72qpl1bp78x1n5qhbg6zkp4f1y6hxy4-mesa-21.2.6.drv ╚═══/: …5.41.drv",["out"]),("/nix/store/6kl4ia9nxnn206jb6qmfqk89r3ccbdc2-wayland-1.19.0.drv",["bin","dev… => /nix/store/6kl4ia9nxnn206jb6qmfqk89r3ccbdc2-wayland-1.19.0.drv ╚═══/: …1.34.drv",["dev"]),("/nix/store/9w53rqac91nypirn4ig9aaxlypcyf7xq-graphviz-2.49.3.drv",["out"]),(… => /nix/store/9w53rqac91nypirn4ig9aaxlypcyf7xq-graphviz-2.49.3.drv ╚═══/: …hook.drv",["out"]),("/nix/store/kcfsc5xhz202m41d0jxhfdxa4yj13div-libdevil-1.7.8.drv",["dev"]),("… => /nix/store/kcfsc5xhz202m41d0jxhfdxa4yj13div-libdevil-1.7.8.drv ╚═══/: …inux.drv",["out"]),("/nix/store/vbs4zhgi8v924n2d7079vfbsyba7x1hj-openexr-2.5.7.drv",["dev"]),("/… => /nix/store/vbs4zhgi8v924n2d7079vfbsyba7x1hj-openexr-2.5.7.drvExample: libde265
LeaseReport depends on libde265 (via imagemagick, sphinx and ghc)
$ nix why-depends /nix/store/4cia66md9qha0inqnc68bg9094425n41-nixos-system-storage004-21.11.337 971.a1fe662eb26.drv /nix/store/7a6ry7k946r0g9z5ijc2zan6zcpsiqnr-libde265-1.0.8.drv/nix/store/4cia66md9qha0inqnc68bg9094425n41-nixos-system-storage004-21.11.337971.a1fe662eb26.drv ╚═══/: …1-p8.drv",["out"]),("/nix/store/2gy7m3c5vybfmrk7cd1ricnzxxs7bxjs-system-path.drv",["out"]),("/ni… => /nix/store/2gy7m3c5vybfmrk7cd1ricnzxxs7bxjs-system-path.drv ╚═══/: …nfig.drv",["out"]),("/nix/store/fj3l19h9mqwljn69y397zcgsaxh4jv24-LeaseReport-exe-LeaseReport-0.1… => /nix/store/fj3l19h9mqwljn69y397zcgsaxh4jv24-LeaseReport-exe-LeaseReport-0.1.0.0.drv ╚═══/: …-p23.drv",["out"]),("/nix/store/dspkn307mzgdpp99n3v40jn65lggjrgm-default-Setup-setup.drv",["out"… => /nix/store/dspkn307mzgdpp99n3v40jn65lggjrgm-default-Setup-setup.drv ╚═══/: …inux.drv",["out"]),("/nix/store/klr1ixwy239yn8n6x8i4clpyywdg1qvk-ghc-8.10.6.drv",["out"])],["/ni… => /nix/store/klr1ixwy239yn8n6x8i4clpyywdg1qvk-ghc-8.10.6.drv ╚═══/: …4.18.drv",["out"]),("/nix/store/l6wcsx49xp5wf50yfralyq2fdcknwr59-python3.8-sphinx-3.5.4.drv",["o… => /nix/store/l6wcsx49xp5wf50yfralyq2fdcknwr59-python3.8-sphinx-3.5.4.drv ╚═══/: …7.12.drv",["out"]),("/nix/store/g8c7bj9pjyvj2z8dhh8qnc4iz33pplg1-imagemagick-7.0.11-13.drv",["de… => /nix/store/g8c7bj9pjyvj2z8dhh8qnc4iz33pplg1-imagemagick-7.0.11-13.drv ╚═══/: ….4.6.drv",["out"]),("/nix/store/xmdir350f3k1pb3mbzh5ssmj75z0gw61-libheif-1.12.0.drv",["dev"])],[… => /nix/store/xmdir350f3k1pb3mbzh5ssmj75z0gw61-libheif-1.12.0.drv ╚═══/: …hook.drv",["out"]),("/nix/store/7a6ry7k946r0g9z5ijc2zan6zcpsiqnr-libde265-1.0.8.drv",["out"]),("… => /nix/store/7a6ry7k946r0g9z5ijc2zan6zcpsiqnr-libde265-1.0.8.drvEdited by Florian SesserRegarding the QEMU chain, we could fairly easily break it at Mesa by using a QEMU with some custom options.
Looking at
nixpkgs/pkgs/applications/virtualization/qemu/default.nix, I see a lot of options which we could turn off:{ lib, stdenv, fetchurl, fetchpatch, python3, python3Packages, zlib, pkg-config, glib, buildPackages , perl, pixman, vde2, alsa-lib, texinfo, flex , bison, lzo, snappy, libaio, libtasn1, gnutls, nettle, curl, ninja, meson, sigtool , makeWrapper, runtimeShell, removeReferencesTo , attr, libcap, libcap_ng, socat , CoreServices, Cocoa, Hypervisor, rez, setfile , guestAgentSupport ? with stdenv.hostPlatform; isLinux || isSunOS || isWindows , numaSupport ? stdenv.isLinux && !stdenv.isAarch32, numactl , seccompSupport ? stdenv.isLinux, libseccomp , alsaSupport ? lib.hasSuffix "linux" stdenv.hostPlatform.system && !nixosTestRunner , pulseSupport ? !stdenv.isDarwin && !nixosTestRunner, libpulseaudio , sdlSupport ? !stdenv.isDarwin && !nixosTestRunner, SDL2, SDL2_image , jackSupport ? !stdenv.isDarwin && !nixosTestRunner, libjack2 , gtkSupport ? !stdenv.isDarwin && !xenSupport && !nixosTestRunner, gtk3, gettext, vte, wrapGAppsHook , vncSupport ? !nixosTestRunner, libjpeg, libpng , smartcardSupport ? !nixosTestRunner, libcacard , spiceSupport ? !stdenv.isDarwin && !nixosTestRunner, spice, spice-protocol , ncursesSupport ? !nixosTestRunner, ncurses , usbredirSupport ? spiceSupport, usbredir , xenSupport ? false, xen , cephSupport ? false, ceph , glusterfsSupport ? false, glusterfs, libuuid , openGLSupport ? sdlSupport, mesa, libepoxy, libdrm , virglSupport ? openGLSupport, virglrenderer , libiscsiSupport ? true, libiscsi , smbdSupport ? false, samba , tpmSupport ? true , uringSupport ? stdenv.isLinux, liburing , hostCpuOnly ? false , hostCpuTargets ? (if hostCpuOnly then (lib.optional stdenv.isx86_64 "i386-softmmu" ++ ["${stdenv.hostPlatform.qemuArch}-softmmu"]) else null) , nixosTestRunner ? false , doCheck ? false , qemu # for passthru.tests }:We could do without many of those
*Supportoptions. The expression is such that if we turn off a flag, the related dependencies don't appear in the derivation.The mechanism to toggling these flags is probably to override the top-level package with a new version. Right now we get our nixpkgs by importing the top-level
nixpkgs.nixand calling it with an empty set. I think we can call it with some overlays instead. For example, shell.nix could start with:let pinned-pkgs = import ./nixpkgs.nix { overlays = [ (import ./overlays.nix) ]; }And
overlays.nixcould be something like:self: super: { qemu = super.qemu.override { sdlSupport = false; # also turns off OpenGL and "virgl", whatever that is }; }Or, at least, something like this
When I did mess with overlays, I was mostly using them to upgrade Python libraries and this usually spiraled into unmanageable levels of complexity. However, overriding QEMU to turn off some features is pretty much what the overlay system is for so maybe it is simple enough to be feasible.
Another thing to look out for is build-time dependencies vs runtime dependencies. I think QEMU's Mesa dependency is runtime so it translates into software we actually deploy. I think LeaseReport's dependency on imagemagick is build-time so while it does add cost and complexity to our build system I don't think we will actually deploy imagemagick to our servers as a result (but I'm not sure!).
There is probably a way to ask
nixto differentiate between these two things.mentioned in merge request !325 (closed)
mentioned in commit c76bac86
mentioned in merge request !326 (closed)
-
Tools
Some commands I like to use to find out the size of things. There must be much better tools out there, but this is what I used so far.
du
[root@storage001:~]# du -ms /nix/store/* | sort -rn | head -23Executed on a grid machine, this returns the size per nix-store object in megabytes, sorts descending, and returns the first 23 (arbitrarily).
Quite interesting here I find that the biggest six objects, together 3,6 GB (over 40% of the size of
/nix/storeon staging storage001), look like source packages, and as such might be build dependencies that we deploy by accident?[root@storage001:~]# du -ms /nix/store/* | sort -rn | head -6 992 /nix/store/p735m5inf57wlqxa6zjc0kd8l3147qcm-pypi-deps-db-src 773 /nix/store/f8v50iw3r5nc4sdnl7lav24dwa0w7jzw-nix-pypi-fetcher 772 /nix/store/bxxh7964pvipmpi5dhld1wc6sxyj4y1l-source 623 /nix/store/xpgy7kf9prc5hchbirjyc1iam9piclzv-source 261 /nix/store/2g1ppmcribsyshvbxa0l6a81pjbdjzfp-source 216 /nix/store/j8yxa77jqxiy4m7aqfx5zfr4a14jcycv-nix-pypi-fetchernix why-depends
Should we find a big nix store object that does not look relevant and we want to find out why it's installed at all,
nix why-dependscan help:nix why-depends /nix/store/l421fbjvnx30swybfnxwimpwrs9p5mmr-nixos-system-payments-21.11.337971.a1fe662eb26.drv /nix/store/zgvf44mgwb3y9i91bp5mbsab3mz8lq9s-qemu-6.1.1.drvnix show-derivation
When I find an interesting store object with
why-dependsI sometimes look into the derivation to get a (bit) better idea of what's going on:nix show-derivation -r /nix/store/zgvf44mgwb3y9i91bp5mbsab3mz8lq9s-qemu-6.1.1.drvThe vulnerability digest
I started to look into this not because of disk space used, but because we install too much software that has security vulnerabilities. The
vulnerability-scanCI job conveniently provides a useful artefact that lists vulnerable software, much of which we shouldn't have installed in the first place... Find the latest such job and artefact here.Wanted
- A tool (or process) to discern between run-time and build-time dependencies
- Knowhow how to turn miscategorized run-time into build-time dependencies
- Find out why those source packages above land on our grid machines -
why-dependscomes up empty. (Probably I am using it wrong?)
Edited by Florian Sesser -
This might be interesting too: https://nixos.org/manual/nixos/stable/#ch-profiles
In some cases, it may be desirable to take advantage of commonly-used, predefined configurations provided by nixpkgs, but different from those that come as default. [...]
Even if some of these profiles seem only useful in the context of install media, many are actually intended to be used in real installs.There's a headless profile intended for virtual machines:
Common configuration for headless machines (e.g., Amazon EC2 instances).
Disables sound, vesa, serial consoles, emergency mode, grub splash images and configures the kernel to reboot automatically on panic.And a minimal profile that keeps the serial console and might thus be better suited for our bare metal servers:
This profile defines a small NixOS configuration. It does not contain any graphical stuff. It’s a very short file that enables noXlibs, sets i18n.supportedLocales to only support the user-selected locale, disables packages’ documentation, and disables sound.
(sound seems to be disabled by default now already, so the impact of these settings probably isn't huge)
Edited by Florian Sesser mentioned in merge request !330 (merged)
marked this issue as related to LeaseReport#1 (closed)
-
Numtide's SrvOS nix configs have some maybe useful configurations for servers: https://github.com/nix-community/srvos/blob/main/nixos/server/default.nix
mentioned in merge request !522 (merged)
-
https://discourse.nixos.org/t/disable-sage-doc-build/32730/2
Try to find out how your system is depending on sagedoc
nix path-info -r /run/current-system | grep sage
if it does not appear there then
nix path-info -r /run/current-system --derivation | grep sage
then once you have the pathdo
nix why-depends /run/current-system "storepathhere" --all --precise
if the store path ends in .drv then add --derivation to the endEdited by Florian Sesser -
https://github.com/astro/microvm.nix/blob/caac7808d1e31f8a0fa408338cd3736947cb226d/nixos-modules/microvm/optimization.nix#L20-L28 has some nix options to reduce the closure size of micro VMs:
Enables some optimizations by default to closure size and startup time: - defaults documentation to off - defaults to using systemd in initrd - use systemd-networkd - disables systemd-network-wait-online - disables NixOS system switching if the host store is not mounted This takes a few hundred MB off the closure size, including qemu, allowing for putting MicroVMs inside Docker containers.
