libnix cargo-libnix release

motivation

unlike crane / naersk / crate2nix / cargo2nix this cargo (libnix) uses nix internally, making wrappers obsolete.

this is a status update covering the difference to the last blog post and the code can be found here:

https://github.com/nixcloud/cargo/blob/master-libnix/README.md

improvements

release

this release is not upstream in cargo, for now it is only a fork intended for a PR.

it is now easy to get cargo (libnix) mixed into your fenix setup, see the README.md for instructions.

see https://github.com/nixcloud/cargo/blob/master-libnix/README.md#using-cargo-libnix.

nix buildsystem export

the buildsystem can be exported now:

cargo build --release write-nix-buildsystem \
  --out-dir ./libnix-1.87.0+v1 \
  --url https://github.com/nixcloud/cargo/archive/refs/tags/libnix-1.87.0+v1.tar.gz \
  --hash 1blw87myixqdkxz9wv5dv8r0g9xxcdprwzrr1l0r7sj29shrzc8n

when starting this export it will rewrite the src attribute to download it from the given url+hash instead. this way no IFD is needed and the build system can be placed near the actual source code. the release for cargo (libnix) has been done this way, see https://github.com/nixcloud/cargo/blob/c0a84a1a1978c1c65e0e4449b41c179319a99c76/flake.nix#L21 and https://github.com/nixcloud/cargo/blob/master-libnix/README.md#release-workflow.

   ┌─────────────────┐
   │  flake.nix      │
   └────────┬────────┘
            │
            ▼
┌────────────────────────────┐
│  Nix toolchain             │
│  • cargo_build_caller.nix  │
│  • derivations/default.nix │
│  • derivations/ ...        │
└───────────┬────────────────┘
            │
            ▼
┌───────────────────────┐
│  Project source code  │
│  • src/               │
│  • Cargo.toml / ...   │
└───────────────────────┘

 { pkgs }:
    with pkgs;
    {
        deps = {
            "markup5ever_rcdom" = {
                "0.3.0" =
                    [ pkg-config openssl ];
            };
            "unicode-ident" = [ pkg-config curl ];
            "xml5ever" = {
                "0.20.0" = [];
            };
        };
        envs = {
            "cargo" = {
                "0.88.0" = {
                    "RUST_BACKTRACE" = "1";
                };
            };
            "cargo-platform" = {
                "RUST_BACKTRACE" = "full";
            };
        };
    }

logone improvements

logone shows the ‘build type’ now, see:

Compiling proc-macro2 (build.rs run)
Compiling quote (build.rs run)
Compiling proc-macro2
Compiling quote
Compiling libc (build.rs run)
Compiling tz-rs
Compiling const_format_proc_macros
Compiling itoa
Compiling libc
Compiling tzdb_data
Compiling yaml-rust2
Compiling starship
Compiling starship (bin)

incremental builds

still a hot topic of research but with some promising preliminary results in the branch master-libnix_3_incremental_compile.

to build, use one of the two commands:

time nix build target \
  --file nix/cargo_build_caller.nix \
  --out-link target/debug/nix/gc/result \
  --option extra-sandbox-paths '/incremental-target=/cargo-incremental-target' \
  -L'

time nix build target \
  --file nix/cargo_build_caller.nix \
  --out-link target/debug/nix/gc/result \
  --option extra-sandbox-paths '/incremental-target=/cargo-incremental-target' \
  --json --log-format internal-json 2>&1 | ~/logone/target/debug/logone \
  --json --level errors

notable changes:

• the filepaths in nix/cargo-0.88.0-b9aa49f38b781d3e.nix and for the other root crates have been hardcoded with pkgs.lib.fileset.toSource (this speed things up a lot)

• incremental builds require this host change

  mkdir -p /cargo-incremental-target
  chown root:nixbld /cargo-incremental-target

• the --out-dir writes to /tmp/out (instead of $out) so incremental compiles work

  # excerpt of nix/cargo-0.88.0-b9aa49f38b781d3e.nix
  \
  mkdir -p /tmp/out
  set -x +e
  
  ${RUSTC} \
          --crate-name cargo \
          $(if [ -d /incremental-target ]; then echo "-C incremental=/incremental-target"; fi) \
          --edition=2021 src/cargo/lib.rs \
          --error-format=json \
          --json=diagnostic-rendered-ansi,artifacts,future-incompat \
          --crate-type lib \
          ...
          --out-dir /tmp/out \
          ...
  
  mkdir -p $out/
  cp -r /tmp/out/* $out

issue with specification is https://github.com/nixcloud/cargo/issues/3

outlook

next steps could involve:

• fix the ‘parallel’ execution bug https://github.com/nixcloud/cargo/issues/11
• fully support incremental builds as described in https://github.com/nixcloud/cargo/issues/3
• IFD support (so one calls cargo build from nix, skipping the write-nix-buildsystem call)
• logone improvements
  • fix issues in nix upstream
  • better error messages
• GC improvements
• full x86_64-unknown-linux-musl support
• replace bash with nushell
• cargo test & cargo install support
• rework -C metadata=8abf83ef020a3059 / -C extra-filename=-27e7993d9cf32df7 as we might not need it in nix
• support more projects

there is a full list with ideas here:

https://github.com/nixcloud/cargo/blob/master-libnix/README-DEV.md

thanks

i would like to express my heartfelt thanks to:

• nlnet for sponsoring this effort

summary

it is easy to use cargo (libnix) now! this is great, the next thing should be to make it fast.

honestly, for small projects this works already great but for compiling “cargo” it is way too slow on iterative builds.