Manifests are code

A manifest contains a collection of packages. We already know that lots of the Guix commands accept a manifest, such as guix shell and guix package. From the commands perspective, it receives a collection of packages and then takes the action using the collection as input. Consequently, we use manifests to install a collection of packages into a profile or shell environment.

Lets create the simplest manifest:

$ guix shell calcurse --export-manifest

This runs the guix shell command specifying calcurse as the package to install, but with the --export-manifest option provided it outputs a manifest of the packages that would be installed into that environment. The manifest itself is:

(specifications->manifest (list "calcurse"))

A manifest like this is installed into a guix shell like so:

$ guix shell --container --nesting --manifest=basic-manifest.scm

[env]$ guix package --list-installed --profile=$GUIX_ENVIRONMENT
guix            29c94dd522833b2603a651c14a5b06120bcf1829        out     /gnu/store/5di95cr5qlf3gk0imgzwwb8qrsrlmgi0-profile
calcurse        4.5.1                   out     /gnu/store/sf2dk1y33gls2s00217qxh0kds20k26n-calcurse-4.5.1

The calcurse package has been installed into the environment. Since --nesting was specified a copy of the guix command line is also installed into the environment.

Note that the manifest specifies a list with (list "calcurse"). A list is a collection which is generally used for having more than one thing in it: technically we have a list consisting of one item (calcurse). Lets create a manifest with a package list of two items, and look at the manifest:

guix shell calcurse cbonsai --export-manifest

The manifest looks like this:

(specifications->manifest (list "calcurse" "cbonsai"))

The list is a data type from GNU Guile. You might have guessed that the specifications->manifest is a function from Gnu Guile. In fact, a Guix manifest file is a Guile Scheme file that evaluates to a collection of packages - that's why it has the .scm file extension.

As we're going to use Guile as we create manifests and packages - it's time explore the language a little!

A bit of Scheme

Guix is written in GNU Guile, an implementation of Scheme. Scheme itself is a dialect of Lisp: sharing the simple, parentheses-based syntax and functional programming approach. And, a lot - I mean a lot - of brackets!

Scheme's syntax consists of statements in between brackets (parentheses). Each pair of brackets is a statement, so for the famous hello world we'd have:

(display "Hello World!")

Each statement uses prefix notation [1] where the function precedes what we're operating on. Notice how the function (display) is called before the text ("Hello World!"). Where prefix notation has more impact is when we start doing maths:

(+ 4 4) ; evaluates to 8

In this case we have the addition function and then the two numbers we want to act on. The nice thing is that we could have as many numbers as we wanted (e.g. + 4 4 4 7 8) - it doesn't matter.

Guile Scheme in the Lisp family of languages. Lisp is a contraction of "list processing" which comes from the fact that each statement - some items between brackets - is a list. A list is the unit of computation. The whole structure is lists [2].

A lot of the time, we want more than one statement. To create multiple statements we nest lists:

;; how we run a function with two parameters
(function a-parameter b-parameter)
(+ 4 4)

;; if we want to calculate the value of one of the parameters we nest another list
(one-function (two-function a-parameter b-parameter) one-function-b-parameter)
(+ (- 5 2) 4)

As we can see we can split a parameter with another statement, by nesting another statement inside it. In this example, we start with a simple addition statement. The second line nests another calculation inside: (- 5 2).

Nesting can be difficult to read if you're used to languages like Java or Python that have a lot of structure in the text and different keys words. Luckily, Lisp code doesn't care how it's formatted - so early on I suggest you format it how you want so you can see the nesting. I often vertically align brackets so I can see what's happening and use comments to remind myself:

;; sometimes putting in some line breaks makes the structure easier to see
(+
   (- 5 2)  ; a nested statement - the result becomes the outer function's first parameter
   4        ; the second parameter
)

One tip is that it's generally easiest to read a section of code from right to left, and often from inside to outside. As it's a tree structure, following each branch back along to the root. For example, look at the manifest again:

(specifications->manifest
        (list "calcurse")
)

Examining it from inside to outside we can see there are two statements:

1. Inner statement: create a list with calcurse in it: (list "calcurse")
2. Outer statement: take that list and create a manifest: (specification->manifest <inner statements list>)

Lets consider the inner statement - the one that's creating a list. In Guile (and all Schemes) lists are both used to create the structure of the program, and are an important collection data type for storing a group of items. If you're interested have a look at the Guile Scheme Lists manual page. There are two ways to create a list:

(list "calcurse")
'("calcurse")

These two are the same, they create a list with a single string inside the collection.

The second version is interesting. Technically, a list is simply anything inside parentheses. However, lists are used as both a data store and as a statement. Consequently, when using a list as a data store it's prefaced with a quote so that the interpreter doesn't try to run the statement. Here's an example of the two ways:

("calcurse")  ;; this line has no quote: the interpreter will try and use the string as it's a statement
'("calcurse") ;; this is a list literal storing a string

Generally, in my examples I'll try to use the list function to create a list where possible. But, you'll see quoting used extensively in Guile code that is trying to say that the list is a data store.

List are for storing a collection of items - generally that's more than one item - here's how to create a list with two items in it:

(list "calcurse" "cbonsai")     ; creates a list with two items in it
'("calcurse" "cbonsai")         ; the same using the list literal and a single quote

Hopefully, it's clear how to use lists to create a collection of items.

The second statement (form) in the manifest is calling the specifications->packages function:

(specifications->manifest <inner statements list>)

Having evaluated the inner statement to become a list, the outer statement evaluates. Here we're looking at a function called specifications->manifest which is part of Guix's API. Guix itself provides a set of modules that have functions and data types that we can use in our manifests or package recipes.

If you have a local copy of the Guix source code, this function is defined in gnu/packages.scm. The document string says that it accepts a List as a parameter, and outputs a manifest data type. Each list item is a spec which is some text that's resolved to a package. It can be in the form:

"name"               ;; "git"
"name@version"       ;; "git@2.41.0
"name:out"           ;; "git:gui"
"name@version:out    ;; "git@2.41.0:send-email"

A few different ways we could call this function are:

;; these two are equivalent - just different ways of defining a list
(specifications->manifest (list "calcurse"))
(specifications->manifest `("calcurse"))

;; multiple items in the list
(specifications->manifest (list "calcurse" "cbonsai@1.3.1" "git:send-email"))

The final output is a manifest, which the Guix command can accept as an input. Technically a manifest is a special data type (called a record) that Guix has defined in guix/profiles.scm.

Of course, there's much more to know about GNU Guile. But, for now we have enough to continue - we have enough basic understanding to create manifests. Lets move onto to creating package transformations.

[1]	Prefix notation

[2]	A statement within parentheses is called an S-expression. See the Wikipedia - S-expressions article. The main benefit is that S-expressions form a tree and easily manipulated by other programs.

Modules in manifests

As we're using Guix's API it can be useful to have a copy of the source code to see the functions documentation and source. To grab the source code do:

git clone https://git.savannah.gnu.org/git/guix.git

The functions for package transformations are all in the guix/transformations.scm file. We can use these functions to create all the same transformations that are available through the command line. We'll start by creating a --with-source transformation that updates the Calcurse package to a specific version.

Create a file called calcurse-basic.scm and add the following to it:

1  (use-modules (guix profiles)
2               (guix transformations)
3               (gnu packages calcurse)
4 
5  (packages->manifest (list calcurse))

The first thing that's different from our previous manifest is that we're explicitly telling Guix to use some modules. Guile Scheme has a module system which Guix uses extensively. A module is a single file, and we directly refer to the path of the module when we ask the interpreter to load it. So (guix profiles) is the path in the source <top-level-dir>/guix/profiles.scm.

The (guix profiles) module contains functions for dealing with profiles, but also how to deal with manifests. All of the different transformations are in (guix transformations) see the guix/transformations.scm file. We're using (gnu packages calcurse) which is <top-level-dir>/gnu/packages/calcurse.scm as this contains the package definition for Calcurse: this allow us to use the variable calcurse directly rather than having to search by the name - it just simplifies our manifests code.

Line 5 calls the function packages->manifest which returns a list of manifest entries (just the same as specifications->manifest) but rather than a text name (spec) it has to receive a package variable.

Run this manifest:

$ guix shell --container --nesting --network --manifest=calcurse-basic.scm

When the shell environment is created we can test Calcurse was installed with:

[env]$ calcurse --version
calcurse 4.5.1 -- text-based organizer

[env]$ guix package --list-installed --environment=$GUIX_ENVIRONMENT
guix        29c94dd     out     /gnu/store/5di95cr5qlf3gk0imgzwwb8qrsrlmgi0-profile
calcurse    4.5.1       out     /gnu/store/sf2dk1y33gls2s00217qxh0kds20k26n-calcurse-4.5.1

This demonstrates that the manifest can install the package for us.

Package source transform

Now we add our first source transformation to calcurse-basic.scm so it looks like this:

 1 (use-modules (guix profiles)
 2              (guix transformations)
 3              (gnu packages calcurse)
 4 
 5 (define calcurse-src '(with-source . "https://calcurse.org/files/calcurse-4.8.1.tar.gz"))
 6 
 7 (define calcurse-transform
 8     (options->transformation (list calcurse-src)))
 9 
10 (packages->manifest (list (calcurse-transform calcurse)))

Line 5 is new, it creates a variable called calcurse-src. It's a list with a single Pair in it. A Pair is a Guile Scheme data type which consists of a Symbol and some value. All we really need to know is we have a Symbol of with-source and we have the source location as a string. A pair is created inside a list, so we have to use the quoted list '().

The next line down (line 6) defines a variable called calcurse-transform. It uses the calcurse-src pair and puts it into a list. The function options->transformation is called with the calcurse-src pair as an argument. The options->transformation function is defined in guix/transformations.scm. It accepts as a parameter a list of pairs, where each pair is a symbolic name and a string. When the function is called, it returns a package record with the transformation applied: I'm slightly simplifying here, because it returns a function and we don't currently need to know the ins-and-outs of that to use it.

Finally, in line 10 - reading from the inside - we initially call calcurse-transform on the standard calcurse variable, this applies the transformation. We then wrap this into a list and call packages->manifest to create the manifest.

Technically, we don't need two defined variables like this, but it does make it a bit easier to explain. When we run it we get:

$ guix shell --container --nesting --network --manifest=calcurse-basic2.scm

Then testing in the shell environment:

[env]$ guix package --list-installed --profile=$GUIX_ENVIRONMENT
guix        29c94dd  out   /gnu/store/5di95cr5qlf3gk0imgzwwb8qrsrlmgi0-profile
calcurse    4.8.1    out   /gnu/store/clvzsax51ycy4awya6nhf7sn35f932j9-calcurse-4.8.1

[env]$ calcurse --version
calcurse 4.8.1 - text-based organizer

We've successfully created a package variant that does the same as the --with-source package transformation option on the command line. We can also provide our own version number, by changing the text spec part of the Pair, for example:

(define calcurse-src '(with-source . "calcurse@20231207=https://calcurse.org/files/calcurse-4.8.1.tar.gz")).

Transforming inputs

We can also use Guix's API to apply transformations to package inputs. In the previous post on using the command line tools for package transformations we showed this example:

guix build jnettop --with-source=libpcap@1.10.2=https://www.tcpdump.org/release/libpcap-1.10.2.tar.gz

To create the equivalent using a manifest create a file called jnettop-local.scm with the following contents:

 1  use-modules (guix profiles)
 2              (guix transformations)
 3              (gnu packages admin))
 4 
 5  (define libpcap-src '(with-source . "libpcap@1.10.2=https://www.tcpdump.org/release/libpcap-1.10.2.tar.gz"))
 6 
 7  (define jnettop-transform (options->transformation (list libpcap-src)))
 8 
 9  (packages->manifest (list (jnettop-transform jnettop)))

It's very similar to the previous transformation. We define the transformation we want in the Pair as with-source, and provide a spec. The Spec is exactly the same as for the command line transformation, we're giving the details of one of the build inputs.

Line 7 and Line 8 define the transformation we're doing is to the jnettop package. The jnettop variable is from the (gnu packages admin) module. The module that contains a package can be found by doing guix edit jnettop and looking at the file it opens, we translate gnu/packages/admin.scm to the module line (gnu packages admin).

To see the build details do:

$ guix shell --container --nesting --network --manifest=jnettop-local.scm --no-substitutes --no-grafts

[env]$ guix package --list-installed --profile=$GUIX_ENVIRONMENT
guix    29c94dd522833b2603a651c14a5b06120bcf1829        out     /gnu/store/5di95cr5qlf3gk0imgzwwb8qrsrlmgi0-profile
jnettop 0.13.0                  out     /gnu/store/3b5x5x8w0j0id20n1pbzamyn31s1j1b4-jnettop-0.13.0

In a normal shell environment inspect the build by looking at the build log associated with the derivation we see above:

$ guix build --log-file /gnu/store/3b5x5x8w0j0id20n1pbzamyn31s1j1b4-jnettop-0.13.0
/var/log/guix/drvs/vi/2yiirwxj5irx7wgrwxpkxvs58gn4am-jnettop-0.13.0.drv.gz
$ less /var/log/guix/drvs/vi/2yiirwxj5irx7wgrwxpkxvs58gn4am-jnettop-0.13.0.drv.gz

Here we can see references to the libpcap derivation:

/gnu/store/jgc6jpp0gc4nqn5y6qlhq3fwziamchj1-libpcap-1.10.2/bin

We can inspect this build log with:

$ guix package --log-file  /gnu/store/jgc6jpp0gc4nqn5y6qlhq3fwziamchj1-libpcap-1.10.2
/var/log/guix/drvs/hx/nz8k6hp8ccd041vks2hx0hhxqal3wk-libpcap-1.10.2.drv.gz

$ less /var/log/guix/drvs/hx/nz8k6hp8ccd041vks2hx0hhxqal3wk-libpcap-1.10.2.drv.gz

The testing procedure is the same as in the previous blog - as libpcap needs admin access to the network interfaces we use a normal Guix shell:

$ guix shell coreutils
# install the package into a temporary profile so that the main profile is not polluted
[guix-dev]$ guix package --profile=./tmp-profile --install /gnu/store/3b5x5x8w0j0id20n1pbzamyn31s1j1b4-jnettop-0.13.0

# run the program as the root user so that it can access the interfaces
[guix-dev]$ sudo /home/steve/tmp-profile/bin/jnettop

# to clean up exit the shell and remove the profile files
$ rm tmp-profile

Half time summary

We've covered a lot in this post and made progress recreating the basic package transformations we used on the command line using the API and manifests. We've also continued becoming familiar with Guile. Don't worry about learning Guile - particularly if you don't have a functional programming background - we only need to know enough Guile to be dangerous. Guix provides a lot of functions and capabilities we can use without having to write Guile Scheme.

In the next post we'll continue to explore using the package transformations through the API, and we'll investigate how Guix creates packages from the information we give it - something called Derivations.