Unicorn Edition (CC, by ladyfem)
A previous post showed how to use guix shell to create development environments. I said they'd be reproducible, frictionless and fun - but ... I lied! Not about the fun part, obviously. But, about the reproducible element - because we're missing one step for guaranteed reproducibility! We'll put that right in this post.
We'll also show how to use it practically in our example development project, "Unicorn Edition" Tmux! Maybe changing the version string isn't as impressive as it sounds - but who doesn't love unicorns!
If you missed the previous couple of posts on guix shell and development environments, check them out:
Guix is a rolling release distribution meaning that new versions of packages are added continuously. As users this brings the latest features from across the free software ecosystem to our finger tips! However, as package versions change over time, the make-up of new environments we create will also change: for example defining an environment with 'Vim' in it will pull in a different version of Vim if the environment is created on January 1st, than if it's created six months later. This is a problem for reproducibility, as we need certainty that all inputs will remain the same.
One aspect is that to be fully reproducible we must specify the precise package version of each dependency in our manifests and package definitions:
package@<version> vim@9.0.2001
This has the benefit of being explicit, and is the best way to define package inputs.
Another aspect is that Guix is always rolling forward and generally there is only one version of a package in the distribution at any point in time. For example, we only see one version of Vim if we do a guix search vim: when a new version is released Guix is updated to this new version and the old version is no longer visible. This is a problem because for complete reproducibility we need to identify the versions of dependencies in our package and we must use a version of the Guix repository that contains those specific versions.
We can do this by identifying a specific commit to the Guix repository, perhaps one from when we defined the build environment for our project. Then every time we create the environment the packages used will be the exact same versions that were available at that point in time.
Guix has a command to set the version of the Guix repository that's used called Guix time-machine, which sounds as cool as it is! We use it to run a Guix command at a specified commit of the Guix repository.
Internally Guix knows what version of the repository it's using and various details about it, we can see this with guix describe:
$ guix describe
Generation 39 Oct 17 2023 11:28:22 (current)
nonguix 14656d6
repository URL: https://gitlab.com/nonguix/nonguix
branch: master
commit: 14656d642dc113c73f9b144ccba366376a274a2b
guix a0d2ecd
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: a0d2ecde943bd1854ddbb7c0cad35a1fc64dc5abGuix describe
The time-travelling command alters the commit that Guix knows about, Guix uses Git to checkout that revision of the repository, making all the packages and services from that commit available. We can then interact with those packages (e.g. install them). In some cases there will still be binary substitutions available, if not Guix will build the package locally. The further back we time-travel the more likely it is that we'll have to build locally.
To use guix time-machine we provide two parts:
The format is:
guix time-machine [options] -- [guix command [args]]
🏆 TIP: for the guix command part we don't specify guix as that's implicit - so we would do shell not guix shell
The simplest test is to go back in time a few commits from the tip of the Guix repository - essentially, moving the state of the Guix repository back a few days. To do this browse the Guix git repository, look at the master branch and pick a commit from a few days earlier.
We don't want to set our entire Guix (default profile) to an earlier time, instead we can explore the command safely within a Guix shell container. You'll have noticed earlier that when I ran guix describe it showed I'm using the nonguix channel. It's important that we only have the Guix channel available when using guix time-machine. If there are multiple channels then it will try and return each channel to the specific git commit, which won't work. To avoid this make sure to use guix shell --container and not a plain guix shell:
# create a temporary project directory somewhere - I do this in ~/tmp/ $ mkdir <some location>/guix-timeplay $ cd <some location>/guix-timeplay $ guix shell --container --nesting --network openssl nss-certs coreutils
⚠️ WARNING: Always use a container so that it only has the guix channel. Using time-machine with multiple channels doesn't work from the command line as the same commit won't be in multiple repositories. To use multiple channels/repositories use a channels file.
We've created a container shell which has access to the network, with the Guix command nested inside it. Guix uses git and SSL to download the repository, which is why we're installing openssl and nss-certs. Now when we do guix describe we only have the guix channel:
[env]$ guix describe
guix a0d2ecd
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: a0d2ecde943bd1854ddbb7c0cad35a1fc64dc5ab
For our simple test we call guix time-machine at the commit from a few days ago, and tell it to build the hello package using whatever version of the source was available in the Guix repository at that commit. The guix time-machine command isn't very fast as it will often have to build the whole of Guix - on my laptop it takes about 12 minutes - so be prepared:
[env]$ guix time-machine --commit=2b5c6e1a41e4ddcf4cfa53a319ed784a856eac5d -- build hello --no-substitutes Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'... Authenticating channel 'guix', commits 9edb3f6 to 2b5c6e1 (779 new commits)... [ ... lots of output ...] The following derivations will be built: /gnu/store/iamy6p0qybm1z3fvwiw39dj74wyh1ncc-profile.drv /gnu/store/nsbl82945psdydq8gb6zdl1k2wmrwzz0-guix-2b5c6e1a4.drv /gnu/store/26px4ryhjx17aq7777adbr2ywhrwl0q4-guix-2b5c6e1a4-modules.drv [ ... lots of output as it builds guix ...] building /gnu/store/rslhq5mbxxbwa4zpf8rmyay418nny9k2u-guix-command.drv... building /gnu/store/m1fkys0fy2akkjlp6a67f8xsk3yl4l2r-guix-daemon.drv... building /gnu/store/nsbl82945psdydq8gb6zdl1k2wmrwzz0-guix-2b5c6e1a4.drv... [ ... output as it build the specific hello version ...] The following graft will be made: /gnu/store/1qxiibc83lxxmxd40ixpl652fn09yg40-hello-2.12.1.drv applying 2 grafts for hello-2.12.1 ... grafting '/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1' -> '/gnu/store/8bzzc70vgzdvj6qdzhdpd709m4y2kw7z-hello-2.12.1'... successfully built /gnu/store/1qxiibc83lxxmxd40ixpl652fn09yg40-hello-2.12.1.drv /gnu/store/8bzzc70vgzdvj6qdzhdpd709m4y2kw7z-hello-2.12.1
We've built version 2.12.1 of the hello package, we can install it with:
[env] guix package --install /gnu/store/8bzzc70vgzdvj6qdzhdpd709m4y2kw7z-hello-2.12.1 [env] /home/<user>/.guix-profile/bin/hello Hello, world!
While conceptually time travel is quite complicated - the command itself is straightforward to use!
This demonstrates we can go back in time to earlier versions of the Guix repository, allowing us to build and run packages from that point in time. Of course, since we're just going back a couple of days, the version of the hello package is probably exactly the same as the one in normal Guix - but we're just proving a point.
If we want to use the same version of Guix with other developers we don't want to have to remember the commit we're all using, instead we can specify it in a file.
We normally tell Guix about additional channels using ~/.config/guix/channels.scm, we can use this same concept with guix time-machine to specify exactly which channels and commits we want time-machine to use. First, create a channels file with the current channels and commits in it:
guix describe --format=channels > guix-channel-pinned.scm
Edit the file to remove any channels that are not guix (for example the nonguix channel). For the guix channel we replace the 'commit' with the commit that we used earlier. The file will be similar to this:
(list (channel
(name 'guix)
(url "https://git.savannah.gnu.org/git/guix.git")
(branch "master")
(commit
"2b5c6e1a41e4ddcf4cfa53a319ed784a856eac5d")
(introduction
(make-channel-introduction
"9edb3f66fd807b096b48283debdcddccfea34bad"
(openpgp-fingerprint
"BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA")))))
Each channel section defines the details for that particular channel, with a name, url, branch and commit. The introduction section specifies an OpenPGP fingerprint and a commit which Guix uses to determine that the channels commits are legitimate. We use the channel file with guix time-machine like so:
# make sure it's accessible from inside the guix shell container $ mv guix-channel-pinned.scm tmp/guix-timeplay $ guix shell --container --nesting --network openssl nss-certs coreutils [env]$ guix time-machine --channels=guix-channel-pinned.scm -- build hello --no-substitutes [env] guix package --install /gnu/store/8bzzc70vgzdvj6qdzhdpd709m4y2kw7z-hello-2.12.1
We move the guix-channel-pinned.scm file into our project directory so that we can see it when we start the container. Now we can run guix time-machine command providing the --channels option so that we don't have to cut-n-paste the commit we want around.
🏆 TIP: if we've previously built this version of the hello package then it will be in the Guix Store and Guix will show us this location, rather than rebuilding the package. This is because the version in the Store will be bit-for-bit the same. Yay, for functional build systems and reproducibility! When we build with the same inputs we get exactly the same package (output). We can delete the one we built earlier with guix gc --delete /gnu/store/8bzzc<etc> and then run the time machine command to build it again.
So far we've been using the guix time-machine command to run a single command at a set point in the Guix repositories history.
To run a series of commands we'll create a Guix shell container that uses the commit of the Guix repository we've defined in the channels file:
$ guix time-machine --channels=guix-channel-pinned.scm -- \
shell --container --nesting --network nss-certs openssl coreutils
[env]$ guix describe
guix 2b5c6e1
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 2b5c6e1a41e4ddcf4cfa53a319ed784a856eac5d
[env]$ guix build hello
/gnu/store/8bzzc70vgzdvj6qdzhdpd709m4y2kw7z-hello-2.12.1
The time-machine command is looking a bit more complicated now, but that's just because we're doing it in a single step. The first part calls guix time-machine with the channels file which pins the Guix repository to the requested commit; and the guix command is shell with its options. In the shell that's created we can use the nested guix, as we can see from the guix describe it's using the pinned version of the repository.
When we tell Guix to build the hello package it correctly identifies that we built it earlier, we can simply do guix install hello to get the correct version.
Now we have the capability to run a series of commands at a set point in the Guix repositories history.
🏆 TIP: We can find out what's installed in the environment by providing the current profile which is specified in $GUIX_ENVIRONMENT. To see what was installed as part of creating the environment we'd do guix package --list-installed --profile=$GUIX_ENVIRONMENT.
For our final step lets make our development projects completely reproducible, by having everyone use the same pinned revision of the Guix repository. Essentially, we create a channels file which defines a commit to use, check that into our project and then everyone uses that file in the guix shell commands.
In the development environments post we did some development on Tmux, creating some local changes (Unicorn Edition!) and building our altered package for testing locally. Lets add our pinned channel file to the project:
$ cd tmux-experiment $ mv <some location>/guix-channel-pinned.scm> ./ $ git add guix-channel-pinned.scm
Now any developer can easily create their build and test environments with:
$ guix time-machine --channels=guix-channel-pinned.scm -- \
shell --container --nesting --network --preserve=^TERM$ \
--development --file=guix.scm openssl nss-certs coreutils bash git
Note that we're adding coreutils, bash and git because these are needed to run the guix.scm, we could also put these into a manifest file. We're preserving the $TERM setting from our environment as Tmux needs this. To build Tmux we do:
[env]$ make clean [env]$ ./configure [env]$ make [env]$ ./tmux -V tmux unicorn-edition-3.4
Unicorn Edition!
We used guix time-machine to switch us back to an earlier revision of Guix, then we started a guix shell container which was provided with some prerequisite packages to install, finally it evaluated the package in the guix.scm file, installing any build dependencies.
Rather, than having to remember the command here's a short guix-build-env.sh which can also be added to the project:
#!/usr/bin/env bash set -ex guix time-machine --channels=guix-channel-pinned.scm -- \ shell --container --nesting --network --preserve=^TERM$ --development \ --file=guix.scm openssl nss-certs coreutils bash git
This time rather than building the raw source lets use the package definition:
$ cd <some project> $ ./guix-build-env.sh [env]$ guix build --file=guix.scm --no-substitutes --no-grafts [env]$ guix package --install-from-file=guix.scm [env]$ /home/steve/.guix-profile/bin/tmux -V tmux unicorn-edition-3.4
Similar to working in languages like Rust and Python we can update the guix-channel-pinned.scm to later revisions when we're ready to move the dependencies forward.
That's it, we have time-travel at our finger-tips - who says technology is boring these days!
I found Guix's time-travel capability quite complicated to explain in this post, in practise it's simple to use. It's a unique capability, imagine trying to handle this in a traditional Linux distribution! And, the pay-off is avoiding the many problems caused by version skew between different environments - whether that's one person working on their own, or a big team collaborating together.
We've focused on using time travel for reproducible development environments, time-machine has a few more interesting tricks - we'll explore those next time.