Skip to content
Snippets Groups Projects

Update haskell.nix version used in CI and deployment.

Merged Administrator requested to merge github/fork/tp-la/update-haskell.nix into main

Created by: tomprince

  • Update haskell.nix to a recent version (including hackage.nix and stackage.nix pins therein).
    • When setting up a dev environment for [PrivateStorageio](https://whetstone.privatestorage.io/privatestorage/PrivateStorageio), downloading and extracting hackage.nix took 1+h, and wasn't cached[1]. The newer version of haskell.nix pins the versions in a way that is cached.
  • Add binary caches from privatestorage and haskell.nix.
    • This should make build time in the CI of the project shorter.
  • Use a more recent nixpkgs (in particular, I picked the same version as is currently pinned in PrivateStorageio
    • This should not make a difference in practice, as the build uses a pinned version of nixpkgs from haskell.nix.
  • Update the stackage version and haskell.nix nixpkgs version to a haskell.nix supported[2] (but not tested) configuration.
    • This was the minimal update to get to a supported configuration, it would probably make sense to update even further, but that required code changes.
  • Reduce the instance size used in CI, since the build is now <15m, when rebuilding all dependencies.

[1] Specifically, it used the equivalent of fetchTarball which forces a redownload/build after tarball-ttl or if ~/.cache is reset.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Jean-Paul Calderone
  • 90 91
    91 92 - run:
    92 93 name: "Building with Nix"
    93 94 command: |
    94 95 nix-build \
    95 --option extra-substituters https://hydra.iohk.io/ \
    96 -j 4 \
    96 97 ./nix/ \
    97 98 -A PaymentServer.components.exes."PaymentServer-exe"
    98 99
    99 100 - run:
    100 101 name: "Building Tests"
    101 102 command: |
    102 103 nix-build \
    103 --option extra-substituters https://hydra.iohk.io/ \
    104 -j 4 \
    • xlarge has 8 vCPUs. If the cache hits then I doubt -j4 vs -j8 is going to matter much. If the cache misses I wonder if even xlarge has enough RAM (16GB) to stand -j8. I could wonder the same about -j4 I guess... Hm. So much fun trying to juggle resource constraints in multiple dimensions with a single parameter and with "your build was killed" as the only failure mode.

  • 111 112
    112 113 - save_cache:
    113 114 name: "Cache Nix Store Paths"
    114 key: paymentserver-nix-store-v3-{{ checksum "nixpkgs.rev" }}-{{ checksum "nix/challenge-bypass-ristretto-repo.nix" }}
    115 key: paymentserver-nix-store-v5-{{ checksum "nixpkgs.rev" }}-{{ checksum "nix/challenge-bypass-ristretto-repo.nix" }}
    • Having saxtons configured as a binary substituter and using the same revision of nixpkgs, I wonder if this cache adds anything at all now - saxtons should give us the challenge-bypass-ristretto binaries, I think?

    • And getting the packages from a nix binary substituter should be a lot faster than using a CircleCI cache of all of /nix/store.

    • Created by: tomprince

      I'd be inclined to replace it with cachix (at least assuming we don't soon move this to whetsone). We do still need to build all of the haskell packages, when we update something like the stackage version, but not GHC.

    • Created by: tomprince

      (But I thought this was a large enough change on its own, so didn't do anything but bump the cache revision here)

  • Review: Commented

    I tried building this locally for the first time in a while and I still had an uncomfortably large amount of stuff to build (maybe 300 Haskell libs, if my counting methodology is valid - probably approximately all of the Haskell lib deps).

    This was with NIX_PATH set by PrivateStorageio's shell.nix and saxtons and iohk both configured as binary substituters.

  • Created by: tomprince

    I tried building this locally for the first time in a while and I still had an uncomfortably large amount of stuff to build (maybe 300 Haskell libs, if my counting methodology is valid - probably approximately all of the Haskell lib deps).

    This was with NIX_PATH set by PrivateStorageio's shell.nix and saxtons and iohk both configured as binary substituters.

    I'm not sure if saxtons has built this version. With just iohk's substituter, I don't think building all the haskell packages is surprising. I think iohk builds GHC (and it looks like the package bundled with it), but not anything beyond that.

  • Created by: tomprince

    This was with NIX_PATH set by PrivateStorageio's shell.nix and saxtons and iohk both configured as binary substituters.

    Having just pushed to circleci again, this build shows it pulling a bunch of stuff from saxtons, so I'm not sure why your local build wouldn't do the same.

  • Review: Approved

    Thanks!

  • Jean-Paul Calderone approved this merge request

    approved this merge request

  • Please register or sign in to reply
    Loading