diff --git a/sources/tech/20230303.2 ⭐️⭐️⭐️ How do Nix builds work.md b/sources/tech/20230303.2 ⭐️⭐️⭐️ How do Nix builds work.md index c5923e2034..ab8551a30f 100644 --- a/sources/tech/20230303.2 ⭐️⭐️⭐️ How do Nix builds work.md +++ b/sources/tech/20230303.2 ⭐️⭐️⭐️ How do Nix builds work.md @@ -10,8 +10,7 @@ How do Nix builds work? ====== -Hello! For some reason after the last [nix post][1] I got nerdsniped by trying to understand how Nix builds -work under the hood, so here’s a quick exploration I did today. There are probably some mistakes in here. +Hello! For some reason after the last [nix post][1] I got nerdsniped by trying to understand how Nix builds work under the hood, so here’s a quick exploration I did today. There are probably some mistakes in here. I started by [complaining on Mastodon][2]: @@ -31,24 +30,18 @@ complicated C program. #### the goal: compile a C program, without using Nix’s standard machinery -Our goal is to compile a C program called `paperjam`. This is a real C program -that wasn’t in the Nix repository already. I already figured out how to -compile it in [this post][1] by copying and pasting a bunch of stuff I didn’t understand, but this time I wanted to do -it in a more principled way where I actually understand more of the steps. +Our goal is to compile a C program called `paperjam`. This is a real C program that wasn’t in the Nix repository already. I already figured out how to +compile it in [this post][1] by copying and pasting a bunch of stuff I didn’t understand, but this time I wanted to do it in a more principled way where I actually understand more of the steps. We’re going to avoid using most of Nix’s helpers for compiling C programs. -The plan is to start with an almost empty build script, and then resolve errors -until we have a working build. +The plan is to start with an almost empty build script, and then resolve errors until we have a working build. #### first: what’s a derivation? I said that we weren’t going to talk about too many Nix abstractions (and we won’t!), but understanding what a derivation is really helped me. -Everything I read about Nix talks about derivations all the time, but I was -really struggling to figure out what a derivation _is_. It turns out that `derivation` -is a function in the Nix language. But not just any function! The whole point of the Nix language seems to be to -to call this function. The [official documentation for the `derivation` function][5] is actually extremely clear. Here’s what I took away: +Everything I read about Nix talks about derivations all the time, but I was really struggling to figure out what a derivation _is_. It turns out that `derivation` is a function in the Nix language. But not just any function! The whole point of the Nix language seems to be to to call this function. The [official documentation for the `derivation` function][5] is actually extremely clear. Here’s what I took away: `derivation` takes a bunch of keys and values as input. There are 3 required keys: @@ -56,8 +49,7 @@ to call this function. The [official documentation for the `derivation` function - `name`: the name of the package you’re building - `builder`: a program (usually a bash script) that runs the build -Every other key is an arbitrary string that gets passed as an environment -variable to the `builder` shell script. +Every other key is an arbitrary string that gets passed as an environment variable to the `builder` shell script. #### derivations automatically build all their inputs @@ -69,15 +61,12 @@ Nix will: - put the resulting output directory somewhere like `/nix/store/4garxzr1rpdfahf374i9p9fbxnx56519-qpdf-11.1.0` - expand `pkgs.qpdf` into that output directory (as a string), so that I can reference it in my build script -The derivation function does some other things (described in the -[documentation][5]), but “it builds all of its inputs” is all we really need to know +The derivation function does some other things (described in the [documentation][5]), but “it builds all of its inputs” is all we really need to know for now. #### step 1: write a derivation file -Let’s write a very simple build script and call the `derivation` function. These don’t work yet, -but I found it pretty fun to go through all the errors, fix them one at a time, -and learn a little more about how Nix works by fixing them. +Let’s write a very simple build script and call the `derivation` function. These don’t work yet, but I found it pretty fun to go through all the errors, fix them one at a time, and learn a little more about how Nix works by fixing them. Here’s the build script (`build_paperjam.sh`). This just unpacks the tarball and runs `make install`. @@ -115,9 +104,7 @@ The main things here are: #### problem 1: tar: command not found -Nix needs you to declare all the dependencies for your builds. It forces this -by removing your `PATH` environment variable so that you have no binaries in -your PATH at all. +Nix needs you to declare all the dependencies for your builds. It forces this by removing your `PATH` environment variable so that you have no binaries in your PATH at all. This is pretty easy to fix: we just need to edit our `PATH`. @@ -150,11 +137,9 @@ The next error was: > #include ``` -Makes sense: everything is isolated, so it can’t access my system header files. -Figuring out how to handle this was a little more confusing though. +Makes sense: everything is isolated, so it can’t access my system header files. Figuring out how to handle this was a little more confusing though. -It turns out that the way Nix handles header files is that it has a shell -script wrapper around `clang`. So when you run `clang++`, you’re actually +It turns out that the way Nix handles header files is that it has a shell script wrapper around `clang`. So when you run `clang++`, you’re actually running a shell script. On my system, the `clang++` wrapper script was at `/nix/store/d929v59l9a3iakvjccqpfqckqa0vflyc-clang-wrapper-11.1.0/bin/clang++`. I searched that file for `LDFLAGS` and found that it uses 2 environment variables: @@ -194,22 +179,15 @@ Here’s the next error: I started by adding `-L ${pkgs.libiconv}/lib` to my `NIX_LDFLAGS` environment variable, but that didn’t fix it. Then I spent a while going around in circles and being confused. -I eventually figured out how to fix this by taking a working version of the `paperjam` build that I’d made before -and editing my `clang++` wrapper file to print out all of its environment -variables. The `LDFLAGS` environment variable in the working version was different from mine: it had `-liconv` in it. +I eventually figured out how to fix this by taking a working version of the `paperjam` build that I’d made before and editing my `clang++` wrapper file to print out all of its environment variables. The `LDFLAGS` environment variable in the working version was different from mine: it had `-liconv` in it. So I added `-liconv` to `NIX_LDFLAGS` as well and that fixed it. #### why doesn’t the original Makefile have -liconv? -I was a bit puzzled by this `-liconv` thing though: the original Makefile links -in `libqpdf` and `libpaper` by passing `-lqpdf -lpaper`. So why doesn’t it link in iconv, if it requires the -iconv library? +I was a bit puzzled by this `-liconv` thing though: the original Makefile links in `libqpdf` and `libpaper` by passing `-lqpdf -lpaper`. So why doesn’t it link in iconv, if it requires the iconv library? -I think the reason for this is that the original Makefile assumed that you were -running on Linux and using glibc, and glibc includes these iconv functions by -default. But I guess Mac OS libc doesn’t include iconv, so we need to -explicitly set the linker flag `-liconv` to add the iconv library. +I think the reason for this is that the original Makefile assumed that you were running on Linux and using glibc, and glibc includes these iconv functions by default. But I guess Mac OS libc doesn’t include iconv, so we need to explicitly set the linker flag `-liconv` to add the iconv library. #### problem 6: missing codesign_allocate @@ -219,8 +197,7 @@ Time for the next error: libc++abi: terminating with uncaught exception of type std::runtime_error: Failed to spawn codesign_allocate: No such file or directory ``` -I guess this is some kind of Mac code signing thing. I used `find /nix/store -name codesign_allocate` to find `codesign_allocate` on my system. It’s at -`/nix/store/a17dwfwqj5ry734zfv3k1f5n37s4wxns-cctools-binutils-darwin-973.0.1/bin/codesign_allocate`. +I guess this is some kind of Mac code signing thing. I used `find /nix/store -name codesign_allocate` to find `codesign_allocate` on my system. It’s at `/nix/store/a17dwfwqj5ry734zfv3k1f5n37s4wxns-cctools-binutils-darwin-973.0.1/bin/codesign_allocate`. But this doesn’t tell us what the package is called – we need to be able to refer to it as `${pkgs.XXXXXXX}` and `${pkgs.cctools-binutils-darwin}` doesn’t work. @@ -289,8 +266,7 @@ make install PREFIX="$out" #### let’s look at our compiled derivation! -Now that we understand this configuration a little better, let’s talk about -what `nix-build` is doing a little more. +Now that we understand this configuration a little better, let’s talk about what `nix-build` is doing a little more. Behind the scenes, `nix-build paperjam.nix` actually runs `nix-instantiate` and `nix-store --realize`: @@ -300,11 +276,7 @@ $ nix-instantiate paperjam.nix $ nix-store --realize /nix/store/xp8kibpll55s0bm40wlpip51y7wnpfs0-paperjam-fake.drv ``` -I think what this means is that `paperjam.nix` get compiled to some -intermediate representation (also called a derivation?), and then the Nix -runtime takes over and is in charge of actually running the build scripts. - -We can look at this `.drv` intermediate representation with `nix show-derivation` +I think what this means is that `paperjam.nix` get compiled to some intermediate representation (also called a derivation?), and then the Nix runtime takes over and is in charge of actually running the build scripts. We can look at this `.drv` intermediate representation with `nix show-derivation` ``` { @@ -345,13 +317,11 @@ We can look at this `.drv` intermediate representation with `nix show-derivation } ``` -This feels surprisingly easy to understand – you can see that there are a -bunch of environment variables, our bash script, and the paths to our inputs. +This feels surprisingly easy to understand – you can see that there are a bunch of environment variables, our bash script, and the paths to our inputs. #### the compilation helpers we’re not using: stdenv -Normally when you build a package with Nix, you don’t do all of this stuff -yourself. Instead, you use a helper called `stdenv`, which seems to have two parts: +Normally when you build a package with Nix, you don’t do all of this stuff yourself. Instead, you use a helper called `stdenv`, which seems to have two parts: - a function called `stdenv.mkDerivation` which takes some arguments and generates a bunch of environment variables (it seems to be [documented here][6]) - a 1600-line bash build script ([setup.sh][7]) that consumes those environment variables. This is like our `build-paperjam.sh`, but much more generalized. @@ -370,8 +340,7 @@ and probably lots more useful things I don’t know about yet #### let’s look at the derivation for jq -Let’s look at one more compiled derivation, for `jq`. This is quite long but there -are some interesting things in here. I wanted to look at this because I wanted to see what a more typical derivation generated by `stdenv.mkDerivation` looked like. +Let’s look at one more compiled derivation, for `jq`. This is quite long but there are some interesting things in here. I wanted to look at this because I wanted to see what a more typical derivation generated by `stdenv.mkDerivation` looked like. ``` $ nix show-derivation /nix/store/q9cw5rp0ibpl6h4i2qaq0vdjn4pyms3p-jq-1.6.drv @@ -451,8 +420,7 @@ $ nix show-derivation /nix/store/q9cw5rp0ibpl6h4i2qaq0vdjn4pyms3p-jq-1.6.drv } ``` -I thought it was interesting that some of the environment variables in here are actually bash scripts themselves – for example the `postInstallCheck` environment variable is a bash script. -Those bash script environment variables are `eval`ed in the main bash script (you can [see that happening in setup.sh here][8]) +I thought it was interesting that some of the environment variables in here are actually bash scripts themselves – for example the `postInstallCheck` environment variable is a bash script. Those bash script environment variables are `eval`ed in the main bash script (you can [see that happening in setup.sh here][8]) The `postInstallCheck` environment variable in this particular derivation starts like this: @@ -469,11 +437,7 @@ All of my compiler experiments used about 3GB of disk space, but `nix-collect-ga #### let’s recap the process! -I feel like I understand Nix a bit better after going through this. I still -don’t feel very motivated to learn the Nix language, but now I have some -idea of what Nix programs are actually doing under the hood! - -My understanding is: +I feel like I understand Nix a bit better after going through this. I still don’t feel very motivated to learn the Nix language, but now I have some idea of what Nix programs are actually doing under the hood! My understanding is: - First, `.nix` files get compiled into a `.drv` file, which is mostly a bunch of inputs and outputs and environment variables. This is where the Nix language stops being relevant. - Then all the environment variables get passed to a build script, which is in charge of doing the actual build