fix formats for 20230303.2

sources/tech/20230303.2 ⭐️⭐️⭐️ How do Nix builds work.md

@@ -10,8 +10,7 @@
 How do Nix builds work?
 ======
 
-Hello! For some reason after the last [nix post][1] I got nerdsniped by trying to understand how Nix builds
+Hello! For some reason after the last [nix post][1] I got nerdsniped by trying to understand how Nix builds work under the hood, so here’s a quick exploration I did today. There are probably some mistakes in here.
-work under the hood, so here’s a quick exploration I did today. There are probably some mistakes in here.
 
 I started by [complaining on Mastodon][2]:
 
@@ -31,24 +30,18 @@ complicated C program.
 
 #### the goal: compile a C program, without using Nix’s standard machinery
 
-Our goal is to compile a C program called `paperjam`. This is a real C program
+Our goal is to compile a C program called `paperjam`. This is a real C program that wasn’t in the Nix repository already. I already figured out how to
-that wasn’t in the Nix repository already. I already figured out how to
+compile it in [this post][1] by copying and pasting a bunch of stuff I didn’t understand, but this time I wanted to do it in a more principled way where I actually understand more of the steps.
-compile it in [this post][1] by copying and pasting a bunch of stuff I didn’t understand, but this time I wanted to do
-it in a more principled way where I actually understand more of the steps.
 
 We’re going to avoid using most of Nix’s helpers for compiling C programs.
 
-The plan is to start with an almost empty build script, and then resolve errors
+The plan is to start with an almost empty build script, and then resolve errors until we have a working build.
-until we have a working build.
 
 #### first: what’s a derivation?
 
 I said that we weren’t going to talk about too many Nix abstractions (and we won’t!), but understanding what a derivation is really helped me.
 
-Everything I read about Nix talks about derivations all the time, but I was
+Everything I read about Nix talks about derivations all the time, but I was really struggling to figure out what a derivation _is_. It turns out that `derivation` is a function in the Nix language. But not just any function! The whole point of the Nix language seems to be to to call this function. The [official documentation for the `derivation` function][5] is actually extremely clear. Here’s what I took away:
-really struggling to figure out what a derivation _is_. It turns out that `derivation`
-is a function in the Nix language. But not just any function! The whole point of the Nix language seems to be to
-to call this function. The [official documentation for the `derivation` function][5] is actually extremely clear. Here’s what I took away:
 
 `derivation` takes a bunch of keys and values as input. There are 3 required keys:
 
@@ -56,8 +49,7 @@ to call this function. The [official documentation for the `derivation` function
 - `name`: the name of the package you’re building
 - `builder`: a program (usually a bash script) that runs the build
 
-Every other key is an arbitrary string that gets passed as an environment
+Every other key is an arbitrary string that gets passed as an environment variable to the `builder` shell script.
-variable to the `builder` shell script.
 
 #### derivations automatically build all their inputs
 
@@ -69,15 +61,12 @@ Nix will:
 - put the resulting output directory somewhere like `/nix/store/4garxzr1rpdfahf374i9p9fbxnx56519-qpdf-11.1.0`
 - expand `pkgs.qpdf` into that output directory (as a string), so that I can reference it in my build script
 
-The derivation function does some other things (described in the
+The derivation function does some other things (described in the [documentation][5]), but “it builds all of its inputs” is all we really need to know
-[documentation][5]), but “it builds all of its inputs” is all we really need to know
 for now.
 
 #### step 1: write a derivation file
 
-Let’s write a very simple build script and call the `derivation` function. These don’t work yet,
+Let’s write a very simple build script and call the `derivation` function. These don’t work yet, but I found it pretty fun to go through all the errors, fix them one at a time, and learn a little more about how Nix works by fixing them.
-but I found it pretty fun to go through all the errors, fix them one at a time,
-and learn a little more about how Nix works by fixing them.
 
 Here’s the build script (`build_paperjam.sh`). This just unpacks the tarball and runs `make install`.
 
@@ -115,9 +104,7 @@ The main things here are:
 
 #### problem 1: tar: command not found
 
-Nix needs you to declare all the dependencies for your builds. It forces this
+Nix needs you to declare all the dependencies for your builds. It forces this by removing your `PATH` environment variable so that you have no binaries in your PATH at all.
-by removing your `PATH` environment variable so that you have no binaries in
-your PATH at all.
 
 This is pretty easy to fix: we just need to edit our `PATH`.
 
@@ -150,11 +137,9 @@ The next error was:
  > #include <qpdf/QPDF.hh>
 ```
 
-Makes sense: everything is isolated, so it can’t access my system header files.
+Makes sense: everything is isolated, so it can’t access my system header files. Figuring out how to handle this was a little more confusing though.
-Figuring out how to handle this was a little more confusing though.
 
-It turns out that the way Nix handles header files is that it has a shell
+It turns out that the way Nix handles header files is that it has a shell script wrapper around `clang`. So when you run `clang++`, you’re actually
-script wrapper around `clang`. So when you run `clang++`, you’re actually
 running a shell script.
 
 On my system, the `clang++` wrapper script was at `/nix/store/d929v59l9a3iakvjccqpfqckqa0vflyc-clang-wrapper-11.1.0/bin/clang++`. I searched that file for `LDFLAGS` and found that it uses 2 environment variables:
@@ -194,22 +179,15 @@ Here’s the next error:
 
 I started by adding `-L ${pkgs.libiconv}/lib` to my `NIX_LDFLAGS` environment variable, but that didn’t fix it. Then I spent a while going around in circles and being confused.
 
-I eventually figured out how to fix this by taking a working version of the `paperjam` build that I’d made before
+I eventually figured out how to fix this by taking a working version of the `paperjam` build that I’d made before and editing my `clang++` wrapper file to print out all of its environment variables. The `LDFLAGS` environment variable in the working version was different from mine: it had `-liconv` in it.
-and editing my `clang++` wrapper file to print out all of its environment
-variables. The `LDFLAGS` environment variable in the working version was different from mine: it had `-liconv` in it.
 
 So I added `-liconv` to `NIX_LDFLAGS` as well and that fixed it.
 
 #### why doesn’t the original Makefile have -liconv?
 
-I was a bit puzzled by this `-liconv` thing though: the original Makefile links
+I was a bit puzzled by this `-liconv` thing though: the original Makefile links in `libqpdf` and `libpaper` by passing `-lqpdf -lpaper`. So why doesn’t it link in iconv, if it requires the iconv library?
-in `libqpdf` and `libpaper` by passing `-lqpdf -lpaper`. So why doesn’t it link in iconv, if it requires the
-iconv library?
 
-I think the reason for this is that the original Makefile assumed that you were
+I think the reason for this is that the original Makefile assumed that you were running on Linux and using glibc, and glibc includes these iconv functions by default. But I guess Mac OS libc doesn’t include iconv, so we need to explicitly set the linker flag `-liconv` to add the iconv library.
-running on Linux and using glibc, and glibc includes these iconv functions by
-default. But I guess Mac OS libc doesn’t include iconv, so we need to
-explicitly set the linker flag `-liconv` to add the iconv library.
 
 #### problem 6: missing codesign_allocate
 
@@ -219,8 +197,7 @@ Time for the next error:
 libc++abi: terminating with uncaught exception of type std::runtime_error: Failed to spawn codesign_allocate: No such file or directory
 ```
 
-I guess this is some kind of Mac code signing thing. I used `find /nix/store -name codesign_allocate` to find `codesign_allocate` on my system. It’s at
+I guess this is some kind of Mac code signing thing. I used `find /nix/store -name codesign_allocate` to find `codesign_allocate` on my system. It’s at `/nix/store/a17dwfwqj5ry734zfv3k1f5n37s4wxns-cctools-binutils-darwin-973.0.1/bin/codesign_allocate`.
-`/nix/store/a17dwfwqj5ry734zfv3k1f5n37s4wxns-cctools-binutils-darwin-973.0.1/bin/codesign_allocate`.
 
 But this doesn’t tell us what the package is called – we need to be able to refer to it as `${pkgs.XXXXXXX}` and `${pkgs.cctools-binutils-darwin}` doesn’t work.
 
@@ -289,8 +266,7 @@ make install PREFIX="$out"
 
 #### let’s look at our compiled derivation!
 
-Now that we understand this configuration a little better, let’s talk about
+Now that we understand this configuration a little better, let’s talk about what `nix-build` is doing a little more.
-what `nix-build` is doing a little more.
 
 Behind the scenes, `nix-build paperjam.nix` actually runs `nix-instantiate` and `nix-store --realize`:
 
@@ -300,11 +276,7 @@ $ nix-instantiate paperjam.nix
 $ nix-store --realize /nix/store/xp8kibpll55s0bm40wlpip51y7wnpfs0-paperjam-fake.drv
 ```
 
-I think what this means is that `paperjam.nix` get compiled to some
+I think what this means is that `paperjam.nix` get compiled to some intermediate representation (also called a derivation?), and then the Nix runtime takes over and is in charge of actually running the build scripts. We can look at this `.drv` intermediate representation with `nix show-derivation`
-intermediate representation (also called a derivation?), and then the Nix
-runtime takes over and is in charge of actually running the build scripts.
-
-We can look at this `.drv` intermediate representation with `nix show-derivation`
 
 ```
 {
@@ -345,13 +317,11 @@ We can look at this `.drv` intermediate representation with `nix show-derivation
 }
 ```
 
-This feels surprisingly easy to understand – you can see that there are a
+This feels surprisingly easy to understand – you can see that there are a bunch of environment variables, our bash script, and the paths to our inputs.
-bunch of environment variables, our bash script, and the paths to our inputs.
 
 #### the compilation helpers we’re not using: stdenv
 
-Normally when you build a package with Nix, you don’t do all of this stuff
+Normally when you build a package with Nix, you don’t do all of this stuff yourself. Instead, you use a helper called `stdenv`, which seems to have two parts:
-yourself. Instead, you use a helper called `stdenv`, which seems to have two parts:
 
 - a function called `stdenv.mkDerivation` which takes some arguments and generates a bunch of environment variables (it seems to be [documented here][6])
 - a 1600-line bash build script ([setup.sh][7]) that consumes those environment variables. This is like our `build-paperjam.sh`, but much more generalized.
@@ -370,8 +340,7 @@ and probably lots more useful things I don’t know about yet
 
 #### let’s look at the derivation for jq
 
-Let’s look at one more compiled derivation, for `jq`. This is quite long but there
+Let’s look at one more compiled derivation, for `jq`. This is quite long but there are some interesting things in here. I wanted to look at this because I wanted to see what a more typical derivation generated by `stdenv.mkDerivation` looked like.
-are some interesting things in here. I wanted to look at this because I wanted to see what a more typical derivation generated by `stdenv.mkDerivation` looked like.
 
 ```
 $ nix show-derivation /nix/store/q9cw5rp0ibpl6h4i2qaq0vdjn4pyms3p-jq-1.6.drv
@@ -451,8 +420,7 @@ $ nix show-derivation /nix/store/q9cw5rp0ibpl6h4i2qaq0vdjn4pyms3p-jq-1.6.drv
 }
 ```
 
-I thought it was interesting that some of the environment variables in here are actually bash scripts themselves – for example the `postInstallCheck` environment variable is a bash script.
+I thought it was interesting that some of the environment variables in here are actually bash scripts themselves – for example the `postInstallCheck` environment variable is a bash script. Those bash script environment variables are `eval`ed in the main bash script (you can [see that happening in setup.sh here][8])
-Those bash script environment variables are `eval`ed in the main bash script (you can [see that happening in setup.sh here][8])
 
 The `postInstallCheck` environment variable in this particular derivation starts like this:
 
@@ -469,11 +437,7 @@ All of my compiler experiments used about 3GB of disk space, but `nix-collect-ga
 
 #### let’s recap the process!
 
-I feel like I understand Nix a bit better after going through this. I still
+I feel like I understand Nix a bit better after going through this. I still don’t feel very motivated to learn the Nix language, but now I have some idea of what Nix programs are actually doing under the hood! My understanding is:
-don’t feel very motivated to learn the Nix language, but now I have some
-idea of what Nix programs are actually doing under the hood!
-
-My understanding is:
 
 - First, `.nix` files get compiled into a `.drv` file, which is mostly a bunch of inputs and outputs and environment variables. This is where the Nix language stops being relevant.
 - Then all the environment variables get passed to a build script, which is in charge of doing the actual build