We can use guix shell to make development reproducible, frictionless and fun. Frictionless, because it's one command (or less) to start the right environment - speeding the path from a thought to being in our editor ready to go. Reproducible, because we'll use exactly the same versions of libraries and tools every time. As the environments are declarative and reproducible we can share them with other developers and testers. And, if you don't think that's "fun", then you're reading the wrong blog!
We'll be creating separate build, test and development environments: this approach keeps things simple and ensures we can set exactly what is needed for each phase of development.
⚠️ Conceptually some parts of this post depend on knowing the format of Guix packages - unfortunately, we haven't covered packaging yet! Hopefully, the examples are simple enough.
There are two scenarios when building software using Guix:
In either case, the assumption is that we've cloned our source code locally and are trying to build it. The Guix commands know how to build software from a defined version of the source which is set in the package definition. A prerequisite is that we've cloned our project and are making changes locally that we want to build and test - the cloning part and pulling down the source code are outside the scope of Guix's tooling.
Our examples, assume that we're doing a git clone into a local development directory or similar.
If the software we're developing has a package in Guix, then that package can be used to define a build environment. A Guix package specifies the build time dependencies and any build time tools needed to create the package. We can use this capability to create an environment and build our source code.
Normally when we call guix shell it creates an environment and installs the specified package. For example to install Vitetris we would do:
$ guix shell vitetris [env]$ vitetris
However, we can also use guix shell to install all the development libraries (inputs) that are needed to build the software (output). For example, if we wanted to work on Vitetris we would do this:
$ git clone https://github.com/vicgeralds/vitetris.git $ guix shell --container --development vitetris [env]$ cd vitetris [env]$ CC=gcc make
This will create the build environment that is needed to build vitetris itself. The CC=gcc before make is a quirk that some software looks for cc and Guix makes gcc available [1].
The --development flag works for any package that is in the Guix archive.
What about customising the build dependencies for a package? It's often the case that as development moves forward dependencies change, or perhaps we want to build the software using different libraries.
The best way to do this is to provide a file named guix.scm within the project directory (or one of the ancestors). If guix shell finds this file then it will automatically evaluate it, using the output to create the environment. This is similar to the manifest.scm capability we looked at in the previous post, but a guix.scm must be a package definition.
A package definition in a guix.scm defines all the inputs needed to build the output package. It's written in Guile Scheme. It's reasonably easy to understand, think of it as a DSL with funny brackets! There are fields for inputs and native inputs which are libraries and tools that need to be installed at build time. The result is that we can use guix shell to create a build environment to build a git checkout of our software.
As an example, lets develop a new feature for Tmux. We start by pulling down the source code:
$ mkdir tmux-experiment $ cd tmux-experiment $ git clone https://github.com/tmux/tmux.git
Then we create a guix.scm in our source checkout directory (tmux-experiment/tmux), as follows:
1 (use-modules (guix) 2 (guix packages) 3 (guix git) 4 (guix git-download) 5 (gnu packages tmux) 6 (gnu packages autotools) 7 (gnu packages texinfo) 8 (gnu packages pkg-config) 9 (gnu packages bison) 10 (gnu packages version-control) 11 (gnu packages bash) 12 (ice-9 popen) 13 (ice-9 rdelim)) 14 15 (define %git-commit (read-string (open-pipe "git show HEAD | head -1 | cut -d ' ' -f 2" OPEN_READ))) 16 17 (package 18 (inherit tmux) 19 (version (git-version "3.4-git" "HEAD" %git-commit)) 20 (source (git-checkout (url (dirname (current-filename))))) 21 (synopsis "Terminal multiplexer - built from source") 22 (native-inputs (modify-inputs (package-native-inputs tmux) 23 (prepend autoconf automake pkg-config bison texinfo bash))) 24 (inputs (modify-inputs (package-inputs tmux) 25 (append git))))
It's easiest to understand if we start with the package definition at the bottom: that's the section from line 17 to 24, which starts with (package ... ) and ending with last bracket on line 25. As we can see there's a set of fields to define elements like the synopsis (what it is), source (where to get it) and native-inputs (what the build dependencies are).
We can use inheritance in Guix packages, which is why this package definition is so short. Line 18 (inherit tmux) lets us take advantage of the existing package in Guix: for example there's a Description field in Guix's Tmux package that we're reusing - so we don't have to define it here.
The version line (line 19) uses the git SHA which it pulls in by using the variable in line 15. It's needed so that Guix knows it's building or downloading a unique version of the software that it doesn't already have in the Store.
The source line (line 20) tells the package where the source code is, in this case we're just telling it to find the source in the same directory as this file. Note that means this file has to be moved into the sources git checkout directory. Normally, we would place guix.scm in the source directory and check it into the project. That way every developer can use the same tools to create their environment.
⚠️ The guix.scm MUST be in the top of the Tmux source tree for this example to work. This is due to the version using shell commands, and source using the current directory that the guix.scm is in to find the source code.
Lines 22-23 and 24-25 are the two important statements where we change the inputs of the package that is in Guix. The native-inputs section are build time packages that are needed to build the package. We're prepending the existing list that Guix has with our own requirements using (modify inputs (package native inputs tmux) (prepend <list>)). The inputs line (line 24-25) is the same idea, it has Git in it so we can use it to compute the version.
Finally, lines 1-13 are the various Guile modules that Guix uses to build the package. We won't cover those in this post!
Next, we authorise that Guix can automatically load the guix.scm file:
echo /home/steve/tmux-experiment/tmux >> /home/steve/.config/guix/shell-authorized-directories/tmux
Then we can start the shell with:
$ guix shell --container --preserve='^TERM$' guix shell: loading environment from '/home/steve/workspace/guix-games/tmux-experiment/tmux/guix.scm'... The following derivation will be built: /gnu/store/arc2asi29xh3lamk5ws0cw5lmnkrl534-profile.drv building CA certificate bundle... listing Emacs sub-directories... building fonts directory... building directory of Info manuals... building profile with 23 packages...
Now that we've authorised the directory Guix automatically uses the guix.scm file when it creates the environment. We're using --preserve=^TERM$ so that we can run the compiled tmux in our build environment as a quick test, and Tmux needs the TERM environment.
We have an environment with the build dependencies we specified and customised in the package. To build our Tmux source we do:
[env]$ ./autogen.sh [env]$ ./configure [env]$ make [env]$ ./tmux
That's it - a completely clean environment for building our project - and it's reproducible because everything is defined declaratively!
At this point we've created a customised guix package in guix.scm and we can build the source. The implication is that we can hack away at the source, check in any changes and then do a local build. To go a step further by building a package and installing that package in a separated environment for testing we need nested containers.
A nested container means that Guix shell starts a container which has has the guix cli within it (and is allowed to use the Guix build daemon). This is a key capability for building and testing, as we can create a clean environment and then build or test out software.
Here's an example:
$ guix shell --container --nesting [env]$ guix install which less coreutils [env]$ GUIX_PROFILE="/home/steve/.guix-profile" [env]$ source"$GUIX_PROFILE/etc/profile" [env]$ guix package --list-generations Generation 1 May 18 2023 14:24:51 (current) which 2.21 out /gnu/store/6vxk0i5j9w8mik4l6gx3cbw33f9x4l24-which-2.21 less 608 out /gnu/store/2zzjawni90xwb0p6pwa8cpywacb3fplk-less-608 coreutils 9.1 out /gnu/store/yr39rh6wihd1wv6gzf7w4w687dwzf3vb-coreutils-9.1
Alright, lets dive into changing Tmux's source code and building a package.
In another terminal make a change to the Tmux source code, the simplest one is to alter the configure.ac:
# change the next-X.X to something else, for example 'unicorn-edition-3.4 AC_INIT([tmux], unicorn-edition-3.4)
Commit this change so we have an altered version from the upstream:
$ git commit -a -m "Created unicorn edition"
Lets build our altered source code into a guix package.
Create a new Guix shell, this time using the --nesting option: this will enable us to use the guix command within the environment.
$ guix shell --container --nesting --development --file=guix.scm coreutils
One thing to notice is that we specified some packages to install into the environment (e.g. coreutils): due to this the guix shell command won't also process the guix.scm file automatically, which is why we specify it with the --file option. Guix will only processes the guix.scm file if you don't specify other packages on the command line.
📝NOTE: there's a behaviour difference between guix shell automatically finding the guix.scm in the working directory, and when we provide the --file option. When guix shell finds a guix.scm automatically it's as if we've asked for the development dependencies. To achieve the equivalent on the command line we do guix shell --development --file=guix.scm. We must provide the --development option when using it this way, otherwise it will install the package into the environment.
We're now ready to build our version of Tmux, in our build environment we do:
[env] $ guix build --file=guix.scm --no-substitutes --dry-run
updating checkout of '/home/steve/workspace/guix-games/tmux-experiment/tmux'...
retrieved commit a40660dfc1494ec8022814829529ee6c5e59ae87
The following derivation would be built:
/gnu/store/30g9jrpad4nj7czm1kqlhdiqr3w6spa9-tmux-3.4-unicorn-edition-HEAD.a40660d.drv
This command tells Guix to build the package, as we have the --dry-run option on it will just tell us what it will do. We've provided the package definition that we created in guix.scm using the --file option - technically this could be any filename we want. The no-substitutes prevents Guix from finding a build we've already done and just giving that to us - while that's normally efficient in this case we want it to build the software.
Repeat the command without the --dry-run to build the package which will put it into the /gnu/guix store. At the end of compilation we see:
successfully built /gnu/store/30g9jrpad4nj7czm1kqlhdiqr3w6spa9-tmux-3.4-unicorn-edition-HEAD.a40660d.drv /gnu/store/m61ia3l6f5p62nh91qc0m70rscx7f2py-tmux-3.4-unicorn-edition-HEAD.a40660d
We've built a customised version of Tmux in a Guix package. The next step is to create a test environment and then use that to install the package and perform whatever testing we want.
First, we create a separate test environment using Guix shell:
# start a nested guix shell container, # we preserve everything because Tmux needs locale $ guix shell --container --nesting --preserve=$ coreutils
Again, we're using the --nesting option so that we have access to the guix command itself. There are two ways to install the custom built package:
Have Guix process the package from the file again but without --no-substitutes so that it finds the already built binary in /gnu/store and installs that. If we do this we have to add git to the env
Having noted down the full /gnu/store path earlier - directly install the package using the path.
# option 1: process the file again, and let it use a substitute
# it needs git to see the version, so we have to install git
[env]$ guix package -i git
[env]$ guix package --install-from-file=guix.scm
# option 2: note down the path of the build and install it
[env]$ guix package -i /gnu/store/zynrim4c1jm83xpbk4ls6cqbc8x8scxz-tmux-3.4-unicorn-edition-HEAD.47b0519
The following package will be installed:
tmux-3.4-unicorn-edition HEAD.47b0519
The following derivation will be built:
/gnu/store/7pw3r5ilc9s2ar3h1825s3nchp1hb5dc-profile.drv
It's installed into ~/.guix-profile so we can source this or just run it directly:
[env]$ $HOME/.guix-profile/bin/tmux -V tmux unicorn-edition-3.4
All our examples use the inputs of packages that are already in Guix, with some alterations.
If we're working on a project that is not packaged in Guix, then it's the same process but we have to create a package from scratch.
I'll cover packaging in later posts, but if you'd like to get started now read the Guix manuals section on packaging, and the Guix Cookbook has a packaging tutorial.
We have a build enviroment, and a test environment, but what about a development environment? There's three options:
I'm currently using either simple or hybrid manifest files as I find it easier than maintaining a separate package: the main situation where a developer tooling package would be useful is if a tool isn't part of Guix so keeping a local package would be useful.
One quirk to be aware of is that a manifest.scm takes precedence over a guix.scm in the same directory. In fact, if Guix processes a manifest.scm it won't process the guix.scm. This means you have to use the --file=./guix.scm switch, but this will assume you want to install the defined package rather than install the build dependencies - so you have to pair it with --development --file=./guix.scm.
The hybrid manifest is what I'm calling a file which has both a shell command and a Guix manifest in it. Originally suggested on the Guix devel mailing list by Kaelyn. I'm finding this a super clever and useful hack!
In the Tmux source directory I create a guix-dev-env.sh script with the following:
#!/usr/bin/env bash
set -ex
exec guix shell --container --network --preserve='^DISPLAY$' \
--preserve='^XAUTHORITY$' --expose=$XAUTHORITY \
--preserve='XDG_RUNTIME_DIR' --expose=$XDG_RUNTIME_DIR \
--share=$HOME/.vim \
--development --file=./guix.scm \
--manifest="$0"
!#
(use-modules (gnu))
(manifest
(map (lambda (spec)
(apply package->manifest-entry
(cond
((pair? spec) (let ((pkg (car spec))
(output (cadr spec)))
(list
(if (string? pkg)
(specification->package pkg)
pkg)
output)))
((string? spec) (list (specification->package spec)))
(else (list spec)))))
`("bash" ; always required
"coreutils" ; always required
"util-linux"
"findutils"
"nss-certs"
"git"
"vim"
"which"
"lesspipe"
"exa"
"patman")))
When we run this script it starts a guix shell with all the development utilities we've specified (e.g. Vim), it also installs the development dependencies in the guix.scm. We discussed the functions of this script in the Guix shell post.
Our goal at the start of this post was to get to one command (or less) to start the right environment. Well we're there. Now when we want to work on a project we can create our development environment with just one command:
$ cd <project> $ ./guix-dev-env.sh [env]$ vim <whatever we want>
When we've made our changes we create our build environment with one command:
$ guix shell --container
And for a clean package building and test environment we do:
$ guix shell --container --nesting --preserve=$ coreutils
To fully automate creating the development environment something like Direnv might be useful, and to simplify the commands further they could be integrated into a Makefile.
Despite covering development environments extensively in this post, there's plenty we haven't had room to go over! Nonetheless, over the last couple of posts we've shown how to create build, test and development environments in a reproducible way. Using the techniques in this post we can reduce the friction of working on software - hmm must do that and stop writing this post!
Next time we'll cover how pin Guix to a particular version - this is useful for sharing development environments and keeping multiple Guix environments in sync.
| [1] | https://issues.guix.gnu.org/28629 |