Each package in Guix is defined using a package definition. When we run a build the recipe is processed by the build system and it outputs the package. We can think of the code that creates the packaging machinery as a couple of layers. At the lowest layer Guix is written in Guile Scheme, so we can use Guile's functions and capabilities in our packages. Above this Guix provides a layer of functions that are focused on the specifics of packaging. While we have the power of a full programming language if we need to use it - we often don't. In fact, it's not necessary to know lots of Scheme to package, instead we can focus on learning the packaging format and functions that Guix provides.
Guix package recipes have a structure:
In this post we're going to cover the basic information that we need in a package, and how we get the source for it - the source/origin inputs.
This post is Part 6 of the Guix Packaging Blog Series - click through for the rest!
Lets start by looking at the overall structure of a package. We'll use GNU Go as our example. To view the package recipe for yourself do guix edit gnugo.
The gnugo package is defined in the file games.scm along with multiple other packages. Here's the package definition I have:
1 (define-public gnugo 2 (package 3 (name "gnugo") 4 (version "3.8") 5 (source (origin 6 (method url-fetch) 7 (uri (string-append "mirror://gnu/gnugo/gnugo-" version 8 ".tar.gz")) 9 (sha256 10 (base32 11 "0wkahvqpzq6lzl5r49a4sd4p52frdmphnqsfdv7gdp24bykdfs6s")))) 12 (build-system gnu-build-system) 13 (inputs 14 (list readline)) 15 (arguments 16 `(#:configure-flags '("CFLAGS=-fcommon"))) 17 (synopsis "Play the game of Go") 18 (description 19 "GNU Go is a program that plays the game of Go, in which players 20 place stones on a grid to form territory or capture other stones. While 21 it can be played directly from the terminal, rendered in ASCII characters, 22 it is also possible to play GNU Go with 3rd party graphical interfaces or 23 even in Emacs. It supports the standard game storage format (SGF, Smart 24 Game Format) and inter-process communication format (GMP, Go Modem 25 Protocol).") 26 (home-page "https://www.gnu.org/software/gnugo/") 27 (license license:gpl3+)))
We can already see that there's a simple structure at play. There's some basic information that is needed by the user (like the name and version). There's the source section, where we can see a URL for downloading the source from. We've also got some inputs, which are libraries needed to compile the package. There's also a build-system and some arguments that go to it, to build the package.
This DSL is provided by Guix and the modules that are imported.
As we know Guix uses Guile modules to separate and organise code. At the top of games.scm the file defines itself as a module (define-module (gnu packages games) ...). The Guix repository keeps the packages in gnu/packages/ as they are all part of the GNU system. There's nothing special about this directory structure, if you have local packages you can keep them wherever you want.
Guix's modules provide various functions to define a package, how to download source code and many other capabilities. Consequently, the rest of the import section pulls in various modules to use, these are the lines - #use-module (guix packages).
The package definition for gnugo starts with:
Line 1: (define-public gnugo ... ) - each package starts with a define-public which tells Guile that we're creating a public variable that will be exported from the module. In this case it exports the gnugo variable. The whole section from the starting bracket all the way to the last bracket (after the license at the end of line 27) is the definition.
Line 2: (package ... ) - we're defining a variable of type package. It's a package Guile Record and there are a set of fields that have to be defined. Records are a standard Guile type, and Guix uses them extensively. Fundamentally, they give us a set of fields with functions to query them. Ultimately our package definition becomes an instance of the package record type.
Basic information about a package is in the string fields name, version, synopsis, description and license.
Line 3: (name "gnugo") - the name field is the package name that users interact with when using the command line tools e.g. guix install gnugo. Commonly it's the same name as the variable we define e.g. define-public gnugo.
Line 4: (version "3.8") - the version is a string that Guix will use to determine upgrades. Often, this is the version from the projects release page, or if there isn't a release sometimes a shortened Git hash will be used.
Each package variable name is unique, we can't define-public gnugo twice: this makes sense since we're exporting them within Guix so if the same variable name was used there'd be a clash. Commonly, Guix only packages one version of any project, for example there's just one version of Vim in the archive at any time. However, in some instances we need more than one version - for example when I search for rust-rustc I currently find three versions of this package with 0.4.0, 0.3.3 and 0.2.3 available. In those cases, the variable name will be set as the name and a short version (rust-rustc-version-0.4) to differentiate them. For our own local packages we also need to define different variable names from the ones in Guix's archive (e.g. gnugo-local).
Line 17: (synopsis "Play the game of Go") - the synopsis string is a short summary description of the package.
Line 18: (description "Gnu Go is a ...") - the description string is a longer description of the package. Both of these are written in Texinfo so we can use various forms of mark-up (e.g. bullets, bold, etc).
Line 26: (homepage "https://www.gnu.org/software/gnugo/") a URL for the package.
Line 27: (license license:gpl3+) - the license of the package. These are kept in licenses.scm and consist of the licenses recognised by the FSF. This module is always imported with the prefix license - I actually have no idea why.
A package needs source code and an input, which is generally downloaded.
Line 5-11: (source ... ) - this section of the package definition specifies where we get the source code of the package from, along with anything that needs to done to that source to prepare it for building (e.g. applying patches). We commonly want to download the source, so it's specified through an <origin> Record. If we were getting the source from a local file we would just specify it here and not use an origin Record.
Line 5-11: (origin (method ...) (uri ...) (sha256 ...) - the <origin> record consists of three parts, the method that we're doing the download with, the URI for the download and a hash of the download. For gnugo the method for the download is (method url-fetch).
Line 7: we specify a URI (uri (string-append "mirror://gnu/gnugo/gnugo-" version ".tar.gz")) to download the source from, as the second part of the origin record. The URI statement could just be a long string. Here it's using a string-append to join the URL with the version number - this is a pretty common technique as it makes it easy to update a package by simply changing the version number. For GNU Go we're downloading the source from a mirror which is why the URL is "mirror://...".
Line 9-11: to check that the source hasn't been MITM altered there's a hash field (sha256 (base32 "0wk...")). We specify the hash as a string and use base32 to convert it. When packaging the fastest way to find the hash is to put in a hash from another package and then run guix build - it will tell you the hash is wrong and give you the correct one that you can then put into the package. Presumably you're confident you're not being MITM attacked - if you want to be make sure grab the source and use the guix hash command.
Line 6: the (method url-fetch) defines the specific procedure used to retrieve the source. There are alternatives to url-fetch, the most common is git-fetch. For example, the game Wesnoth in games.scm uses git-fetch like this:
(define-public wesnoth (package (name "wesnoth") (version "1.16.11") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/wesnoth/wesnoth") (commit version))) (file-name (git-file-name name version)) (sha256 (base32 "0z0y2il4xq8fdj20fwfggpf6286hb099jh1kdywap9rlrybq142d")))) (build-system cmake-build-system) (arguments (list #:tests? #f)) ;no test target (inputs (list boost dbus fribidi libvorbis openssl pango (sdl-union (list sdl2 sdl2-image sdl2-mixer sdl2-ttf)))) (native-inputs (list gettext-minimal pkg-config)) (home-page "https://www.wesnoth.org/") (synopsis "Turn-based strategy game") (description "The Battle for Wesnoth is a fantasy, turn based tactical strategy game, with several single player campaigns, and multiplayer games (both networked and local).
The <origin> record receives the same three parts, a method, uri and sha256. The (method git-fetch) uses a git repository and a commit through the (git-reference (url "https://...") (commit version)) record. Within that we're defining the URL and the commit to grab - it's pretty common to see the 'version' be the git commit that we want. The commit can be any git reference such as a tag (in Wesnoth it's using the tag) or a git commit hash as string.
The source code is downloaded the put into the Store, it's optional to define a (file-name ..) for that source code to be stored under. The Wesnoth package does, but Gnugo doesn't.
There are two other fields we sometimes see in <origin> definitions which are patches and snippet - both are ways to alter the source before we start the build. Here's an example of patches being applied from the lightdm package in display-managers.scm:
(source (origin (method git-fetch) (uri (git-reference (url "https://github.com/canonical/lightdm") (commit version))) (file-name (git-file-name name version)) (sha256 (base32 "1wr60c946p8jz9kb8zi4cd8d4mkcy7infbvlfzwajiglc22nblxn")) (patches (search-patches "lightdm-arguments-ordering.patch" "lightdm-vncserver-check.patch" "lightdm-vnc-color-depth.patch" "lightdm-vnc-ipv6.patch"))))
As we can see the origin record specifies four patches, and these are found using the (search-patches ...) function which looks in the patches directory.
The other option is to use a snippet. A snippet allows us to use Guile Scheme code on the source. For example, to remove a directory, or adjust paths to libraries. Just like patches being applied this happens during the build when the source is being processed (as a fixed-output derivation). To make sure the right modules are available we tell the build process to load certain Guix modules and then run the code within the (snippet ...).
Here's a nice example of a snippet from the package gemmi (in chemistry.scm):
(modules '((guix build utils))) (snippet '(begin (delete-file-recursively "include/gemmi/third_party") (delete-file-recursively "third_party")))
First it uses (modules ...) to load the ((guix build utils)) module. We want that line to be run as code during the build, so it's quoted '() to make sure that it's not evaluated when the variable is evaluated.
The (snippet ...) contains the specific Guile Scheme code to run - again it's passed in as lines of code to run using a quoted list '(). The (begin ..) part is to handle having two separate execution statements. Those two statements use a Guix function delete-file-recursively from the utils module to delete some directories.
There's also an alternative way to specify the code to run. Rather than using a quoted list we can use a G-expression. We won't dig into the full details today, but here's how python-pynbody (in astronomy.scm) defines a snippet using a G-expression:
(modules '((guix build utils))) (snippet ;; Symlink goes to not existing directory. #~(for-each delete-file '("docs/testdata" "docs/tutorials/example_code/testdata")))))
What we see here is that rather than using a quoted list '(), it's using a G-expression #~ which says the following section to the matching bracket are build instructions. The code starts with a for-each that runs on each file in the list '("docs/testdata" ...), and it executes the delete-file function on each item in that list. We still have to quote the list of files so they are not evaluated.
That's everything on source and origin that we need to know - lets play with some practical examples!
If you'd like a practical example of using url-fetch have a look at the last post Modifying Guix packages using inheritance where we used it to update the calcurse package to a later version.
To demonstrate git-fetch we're going to update Vifm to a specific commit. Create a file called local-vifm.scm with the following contents:
1 (define-module (local-vifm) 2 #:use-module (guix download) 3 #:use-module (guix git-download) 4 #:use-module (guix packages) 5 #:use-module (gnu packages vim)) 6 7 (define-public vifm-local 8 (package 9 (inherit vifm) 10 (version "733b9c9") 11 (source 12 (origin 13 (method git-fetch) 14 (uri (git-reference 15 (url "https://github.com/vifm/vifm") 16 (commit "733b9c97ea6ee3c37d15b0c9abf8f811d09f0640"))) 17 (sha256 18 (base32 19 "1a47q9aslgpbrs86s5j61cp07scp97985fm7kcblh4r9hfh0h0cv"))))))
As we've discussed previously the file name and the module name have to be the same: so the file is called local-vifm.scm and the module local-vifm. For the package itself we're using the variable name vifm-local so that it's different from standard vifm. We're keeping the name vifm, as this is one of the fields that we're inheriting - any field we don't alter is inherited. But, we're using a different version - as I wanted this to be reasonably short I've explicitly set it and also used the long commit hash in the git-reference.
To build and test the local version we do:
$ guix shell --container --nesting --network --preserve=^TERM$ coreutils [env]$ guix build --source --no-substitutes --load-path=./ vifm@733b9c9 The following derivation will be built: /gnu/store/66wdxrxn5mbysff9f5fmm0sv35cli5xr-git-checkout.drv building /gnu/store/66wdxrxn5mbysff9f5fmm0sv35cli5xr-git-checkout.drv... Initialized empty Git repository in /gnu/store/ddyyyamp3bqsy3v9ghb82km8r22sih94-git-checkout/.git/ From https://github.com/vifm/vifm * branch 733b9c97ea6ee3c37d15b0c9abf8f811d09f0640 -> FETCH_HEAD [... few lines of git stdout about being on a detached head ...] HEAD is now at 733b9c9 Merge branch 'packer-plugin' successfully built /gnu/store/66wdxrxn5mbysff9f5fmm0sv35cli5xr-git-checkout.drv /gnu/store/ddyyyamp3bqsy3v9ghb82km8r22sih94-git-checkout
This demonstrates that we are getting the right source, we can even inspect that checkout in our Store. Then we can build it with:
[env]$ guix build --no-substitutes --load-path=./ vifm@733b9c9 [... all our normal build output ...] successfully built /gnu/store/ls7lrgf7i6hh0x5ij3jjfb00m7mqci3n-vifm-733b9c9.drv /gnu/store/znv7brf957hd221x0iapsks7ghnwphpy-vifm-733b9c9
To install it we do:
[env]$ guix package --load-path=./ --install vifm@733b9c9 [env]$ GUIX_PROFILE="/home/steve/.guix-profile" [env]$ . "$GUIX_PROFILE/etc/profile" [env]$ vifm
Hopefully, this is feeling pretty straightforward and it's a good demonstration of inheriting and using a different git reference.
For a snippet example we're going to make an alteration to Vifm's source. When we check the version we get:
[env]$ vifm --version Version: 0.13
As you know I love a unicorn edition, so lets make a change to Vifm to reflect our love of unicorns! The gnu-build-system runs configure and then make - so we can alter the version by changing the configure file with a snippet:
(define-module (local-vifm) #:use-module (guix download) #:use-module (guix gexp) #:use-module (guix git-download) #:use-module (guix packages) #:use-module (gnu packages vim)) (define-public vifm-local-unicorn (package (inherit vifm) (version "unicorn-733b9c9") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/vifm/vifm") (commit "733b9c97ea6ee3c37d15b0c9abf8f811d09f0640"))) (sha256 (base32 "1a47q9aslgpbrs86s5j61cp07scp97985fm7kcblh4r9hfh0h0cv")) (modules '((guix build utils))) (snippet #~(substitute* "configure" (("0.13") "Unicorn Edition@733b9c9")))))))
As we're using a Gexp in our snippet we have to add #:use-module (guix gexp) so we can use the capabilities. We name the variable something else (vifm-local-unicorn) so it's different from our other example, and we change the version field.
The snippet itself is just one expression (called a form in Lisp). We start with a Gexp #~ which tells Guix the rest of the form is a build instruction (from the bracket next to it, to the matching bracket). It uses the substitute* function, the parameters are the file that we want to search in (configure in this case), a regular expression to match ("0.13"), and a replacement.
Build the source derivation with:
$ guix shell --container --nesting --network --preserve=^TERM$ coreutils [env]$ guix build --source --no-substitutes --load-path=./ vifm@unicorn-733b9c9
After building the source Guix tells us where the checkout is, we can inspect the checkout to see if the snippet worked correctly. For me:
$ less /gnu/store/i21acskcnc2qk01x6icjw58y4jrd4jx4-git-checkout/configure
I have a line in the configure script of `PACKAGE_VERSION='Unicorn Edition@733b9c9`. Consequently, I know it will build with the right version string. To build the rest:
[env]$ guix build --no-substitutes --load-path=./ vifm@unicorn-733b9c9 [env]$ guix package --load-path=./ --install vifm@unicorn-733b9c9 [env]$ GUIX_PROFILE="/home/steve/.guix-profile" [env]$ . "$GUIX_PROFILE/etc/profile" [env]$ vifm --version Version: Unicorn Edition@733b9c9
There we go - a Unicorn Edition of vifm!
To show an example of using patches we'll return to patching Mutt which we previously did with a transformation. Create a file called mutt-local.scm, with the following:
1 (define-module (mutt-local) 2 #:use-module (guix gexp) 3 #:use-module (guix git-download) 4 #:use-module (guix packages) 5 #:use-module (gnu packages autotools) 6 #:use-module (gnu packages mail)) 7 8 (define-public mutt-local 9 (package 10 (inherit mutt) 11 (version "local-00d5628") 12 (source 13 (origin 14 (method git-fetch) 15 (uri (git-reference 16 (url "https://gitlab.com/muttmua/mutt.git") 17 (commit "00d56288d33005b7412c5fd8b36ccc1d27d12c2f"))) 18 (sha256 19 (base32 20 "1a0kdnk5kpwv6h95sby4c3qqmx0xx3fsiy8ki3x7spq8izkgqb74")) 21 (patches 22 (append 23 (origin-patches (package-source mutt)) 24 (list (local-file "882690-use_fqdn_from_etc_mailname.patch")))))) 25 (native-inputs 26 (modify-inputs (package-native-inputs mutt) 27 (append autoconf automake)))))
The patches section (line 21-24) will take some explaining - or at least I needed some help, thanks to Civodul for the assistance. As usual it may be easier to read it from "inside to outside".
Line 24: we have a new patch we want to add, as it's a string we use Guix's local-file function to change it into a file-like-object. Then we put it into a list using the list function. The result is a list with our patch in it.
We want to add the patch (in it's list) to the ones the inherited Mutt package has.
Line 23: gets the package-source for Mutt (this is part of the <origin> record). Then it extracts the patches using the origin-patches procedure - this returns a list of patches. The result is a list of patches from the Mutt package.
Line 22: the final step is to use append to create a single list, consisting of all the patches from the inherited Mutt package and our additional patch.
Line 21: the consolidated list is used in the patches field.
The inputs to the Mutt package have also been altered in lines 25-27. The Mutt package in Guix uses url-fetch, but we are getting the latest source from Git. The source archive that's downloaded using url-fetch was processed by the upstream developer, as part of their release process, so that it can be built by running configure then make. However, the source from Git hasn't been processed so it doesn't have the necessary build artefacts. It needs to be processed as part of our build - this is the job of autoconf and automake. On lines 25-27 we add autoconf and automake to the inherited list of inputs (native-inputs as these are build tools), so that the build can run correctly. We'll come back to modifying inputs in a future post.
To build the source we do:
$ guix shell --container --nesting --network --preserve=^TERM$ coreutils curl nss-certs [env]$ curl --remote-name \ https://sources.debian.org/data/main/m/mutt/2.2.12-0.1~deb12u1/debian/patches/debian-specific/882690-use_fqdn_from_etc_mailname.patch [env]$ guix build --source --no-substitutes --load-path=./ mutt@local-00d5628 [... lots of output and then ....] patching file init.c source is at 'git-checkout' applying '/gnu/store/vh1jb9z26vg14q5qym1izjpbwnjbpf6b-mutt-store-references.patch'... applying '/gnu/store/92w6h5vqzz8cvi2x2d2xpl1y35dbh4c9-882690-use_fqdn_from_etc_mailname.patch'... [... more output ...] successfully built /gnu/store/jgw0j23rnjnmcmxkdsbsaybalspqs0kz-git-checkout.drv /gnu/store/933gds4avgjdzzipal4pkl11p4644f3g-git-checkout
To check it's correct have a look at /gnu/store/933gds4avgjdzzipal4pkl11p4644f3g-git-checkout/init.c and look for the getmailname comment from the patch. With the patch applied correctly, to build the package do:
[env]$ guix build --no-substitutes --load-path=./ mutt@local-00d5628 [... all our normal build output ...] successfully built /gnu/store/xphyz7ad2h9zrwkvxrc3xfdsc6xbbgs2-mutt-local-00d5628.drv /gnu/store/l8r4ln241c1d4i4aq27v20n5i2ci17qf-mutt-local-00d5628
To install it we do:
[env]$ guix package --load-path=./ --install mutt@local-00d5628 [env]$ GUIX_PROFILE="/home/steve/.guix-profile" [env]$ . "$GUIX_PROFILE/etc/profile" [env]$ mutt
Hopefully, that's clear enough - we can add patches to inherited packages, and we've changed a package from using one type of method for grabbing the source to another.
The most useful resource to read is the manual which has lots of content about the package format: the whole of the Programming Interface section is worth reading. It's also useful to start reading the functions themselves as these have document strings, and look at packages within Guix (under gnu/packages) to find examples.
We've made good progress on understanding the structure of package definitions! We know the high level layout of package definitions and we've really dug into how we can define the source of a package. We've covered a little more of how Guix functions provide a DSL-like experience when defining package recipes. Did I miss anything out that should be covered, was everything clear or is some part confusing? Feel free to email or leave comments futurile@mastodon.social
Next time we'll look at package inputs, the various libraries that we need to specify to build a package.