Use-cases

There are two options for upgrading between one version of a distribution's release and another: fresh install and upgrade. If you do a fresh install then the installer will write over the system partition and install completely new versions of the packages. With an upgrade the installer leaves the partitions alone, installs new system packages and leaves the configuration files in place.

Trusty Tahr to Xenial Xerus

These instructions work for Ubuntu, and should work for Debian. I'm moving from one LTS release to another. With a little alteration the metholodology will work on other systems e.g. CentOS. In this case we're upgrade from Trusty Tahr (Ubuntu 14.04 LTS) to Xenial Xerus (16.04 LTS), but the instructions are applicable to any upgrade.

Strengths

• Keep system configuration

If you've changed your system configuration then an upgrade will for the most part keep those configuration settings for you. Where I alter server daemons like SSH or hardware like power settings this is very useful.

• Maintains installed applications

Upgrades keep all the applications that you've installed for you. If you do a plain install then you'll have to remember what you've installed. That impacts me a lot as I use a lot of non-standard applications.

• Deal with major system changes

Between major releases there are often substantial system changes. The upgrade methodology keeps you aware of those alterations.

Weaknesses

• Misses new defaults

Sometimes maintainers change to new defaults, particularly where there's been a major new version. Often we're over-riding those changes during a change so you miss them.

• Complicated to undertake

Upgrades are more complicated to do than simply installing the system over the top.

• Can fail if interrupted

Upgrades can fail badly if the Internet connection is interrupted, or the system gets stuck for some other reason. This is a general weakness with package management systems and filesystems where they can't revert the changes.

Keep Calm, image by Miss Nixie, CC BY-NC-ND 2.0

Major system affecting changes

• Systemd (Vivid)

  Replaces Upstart with Systemd. This is a major change for how we manage system daemons (and a host of connected elements) on Linux systems. I did a bit of reading systemd on Ubuntu . As this so significant I also looked at the major systemd-boot bugs on Launchpad to see if there was anything that might impact me.

• Python 3 default (Xenial)

  Xenial installs Python 3 and uses it as the default. This will impact my development environment.

• Python 3 for vim (Xenial)

  Vim has been built with Python 3. Plugins that depend on Python 2 will struggle. There's a suggested workaround. I've taken the pain and gone to everything on Python 3.

• Major SSH change (Xenial)

  The new version should be fine for my usage which is very basic.

Significant bugs

• USB Creator

  There's been a major change of behaviour meaning that USB sticks created on 14.04 LTS won't work on 16.04 LTS. This is very painful for me as my upgrade process dictates I need to have a backup to boot the system. I did lots of research done to find out what was going on and what the recommended solution was: pretty inconclusive, confusing and annoying. Start with installing from USD, also the usb-creator bugs list.

• Keyboard selection may be incorrect

  Possibility of a bad selection, where the workaround is to go to the keyboard selection control panel and change it back. This can impact me as I mess around with my Kinesis Advantage keyboard configuration.

New features

• Snap packages (Xenial)

  A new packaging format for applications that can live in parallel to the traditional ones. Snap packages have been coming for a while. I'll probably ignore this for now.

• Software Center replacement (Xenial)

  Sad to see this go as I worked on it. It's replaced by GNOME Software labelled Ubuntu Software. Doesn't impact me directly as I mostly use Synaptic.

• GNOME Calendar (Xenial)

  Installed by default. Probably doesn't impact me at the moment.

• Docker 1.10 (Xenial)

  This is the system installed version rather than using one provided directly by Docker.

• Blueman 2.0 (Wily)

  Noted this down but didn't think much of it. Had some Bluetooth problems later where it was important to know more about the changes.

• Ubuntu Make

  Upgraded Ubuntu Make which is interesting and noted so that I could take a look at it.

Be happy if you don't know why this is funny! And thankful for Keith Packard's contributions, image by Jacobo Tarrío, CC BY-SA 2.0

Obsolete Packages

I use Synaptic as it's easy to find the Obsolete packages:

In the Sections area find:
Status
    'Installed (auto removable)'
    'Installed (local or obsolete)'
    'Not installed (residual config)'
Custom Filters
    Broken  # Check for broken applications
    Origin  # Check through for odd things

In each Status area and with Broken or an odd Origin Make for complete removal

• Be careful with 'Local or obsolete'

  These may be applications that you use but have put in local installs. Note that Local installs are ones where Dpkg doesn't know the repository where this package came from.

• Removing 'Not installed'

  To speed things up use <Shift>+<Arrow keys> to highlight a group and then right click to mark them for complete removal.

• Remove old Kernel versions

  The /boot area can fill-up very easily. It's worth searching for linux-headers, linux-headers-generic and linux-image to find old version. Keep a couple around.

• Remove applications that you don't use

  If we're not actively using an application it's worth removing them. The simpler the system the fewer libraries and the less complicated the upgrade calculation will be. I didn't do this as I was short of time.

Clean the system

There are a variety of tools that can examine the system directly, or through Apt's database and try to simplify or clean the set-up. These tools can be dangerous, so be careful.

• Clean with Bleachbit

  Bleachbit tries to find files that are no longer required. I've used it in the past but it's pretty aggressive so skipped it this time. Basic instructions are:

  $ sudo apt-get install bleachbit
  $ sudo bleachbit
  

• Use GTKorphan

  Gktorphan examines the Apt database and tries to find files that aren't required. It's worth running. I don't trust it enough to allow it to remove anything, I do that directly through Synaptic.

  $ sudo gtkorphan
  

It gives output a bit like this:

* compizconfig-backend-gconf
* compiz-plugins-main-default
* gwibber-service
* gwiber-service-facebook
* gwibber-service-identica
* gwibber-service-twiter
* libavahi-gobject0
* libavahi-ui-gtk3-o
[very long list redacted]

• Clean out the local caches

  The default is for Apt to maintain caches, if we want to free up space we can clean this out with.

  $ sudo apt-get clean
  

• Use Cruft

  There are recommendations to use Cruft. I personally don't as it can be dangerous.

• Find altered packages

  This is from Debian Cleanup Tip #4: find broken packages and reinstall them. I only found a couple of packages that didn't have md5sums.

  $ debsums --changed
  

• Use adequate

  I hadn't heard of this before. Adequate checks packages for policy and bugs issues. Felt similar to Gtkorphan.

adequate --tag library-not-found --all

The output was this:

spotify-client: library-not-found /usr/bin/spotify => libcef.so
uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libreglo.so
uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libunoidllo.so
uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libxmlreaderlo.so
libsvn1:amd64: library-not-found /usr/lib/x86_64-linux-gnu/libsvn_auth_kwallet-1.so.1.0.0 => libkdecore.so.5
libsvn1:amd64: library-not-found /usr/lib/x86_64-linux-gnu/libsvn_auth_kwallet-1.so.1.0.0 => libkdeui.so.5

You can then figure out what to do about each library problem. I couldn't figure out spotify, for libsvn I just removed the library and subversion as I'm not using them.

Not sure how useful it is for me as it took a long time and I didn't really do anything when it told me this. Could be because I'd already run GTKOrphan.

Fix packages

It's worth making sure that the package database is correct.

In Synaptic go to Edit and Fix Broken Packages, then click Mark all Upgrades button, finally hit the Apply button.

Apply updates

Make sure the whole system is up to date by updating all sources and doing any updates that are required.

Reboot the system

Probably best to reboot the system at this point to check that all software is fine: generally you'll have removed some kernels along the way!

Ubuntu installation media, Ubuntu 5.04 CD by Elizabeth K. Joseph, CC BY-NC-ND 2.0

Use LVM to snapshot

I'm using LVM which means I get snapshots "for free". I snapshot everything except for any swap LV's.

• Make snapshots of everything

  Make snapshots of all the partitions.:

  $ sudo lvcreate -L10G -s -n rootsnapshot /dev/vgUbuntu/root-ubuntu-lv
  $ sudo lvcreate -L10G -s -n homesnapshot /dev/vgUbuntu/home-ubuntu-lv
  

• Create a backup image of each partition

  We're using cat to create the images so they are full disk images, this means they take up a lot of space but can simply be copied back over should we have to recover. The whole thing takes a very long time, with the main /home backup taking over two hours to do!

  $ sudo sh -c "cat /dev/vgUbuntu/rootsnapshot | pv -s 15G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`rootsnapshot.dd.gz"
  $ sudo sh -c "cat /dev/vgUbuntu/homesnapshot | pv -s 123G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`homesnapshot.dd.gz"
  $ sudo sh -c "cat /dev/vgUbuntu/powersgsnapshot | pv -s 35G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`powersgsnapshot.dd.gz"
  

• Remove the temporary snapshots

  Before we copy the drive information to our backup we remove them so there's not confusing information in the backup.

  $ sudo lvremove /dev/vgUbuntu/rootsnapshot
  $ sudo lvremove /dev/vgUbuntu/homesnapshot
  $ sudo lvremove /dev/vgUbuntu/powersgsnapshot
  $ sudo lvs
  

• Create a backup of your /boot partition

  We can't snapshot /boot because it's not within LVM. To guarantee the integrity of the backup the partition can't be written to during the backup. Depending on how conservative we are, either drop to runlevel 1 or boot the system from a USB drive.

  $ sudo sh -c "dd if=/dev/sda1 | gzip > /mnt/backups/lenovosg/snapshots/\`date +%Y%m%d\`bootsnapshot.dd.gz"
  

• Record drive information

  Record various bits of drive set-up information which are useful when restoring.

  $ sudo sh -c "pvscan --uuid > /mnt/backups/snapshots/`date +%Y%m%d`-snapshotinfo.txt"
  $ sudo sh -c "sudo lvdisplay --maps >> /mnt/backups/snapshots/`date +%Y%m%d`-snapshotinfo.txt"
  $ sudo sh -c "sfdisk --dump /dev/sda > /mnt/backups/snapshots/`date +%Y%m%d`-partitions-backup-sda.sfdisk"
  

If you do the install and it all messes up then you can restore from snapshot.

No longer needed

In many cases it will propose to remove applications that have been installed from repositories it doesn't know about, or if there are fundamental changes in the system. In this case I use LaTeX a little bit and the distribution version had changed. It notified me that it wanted to remove xscreensaver for some reason, I noted this down as I thought I'd need it later.

• luatex
• xchat-indicator
• xscreenserver

No longer supported by Canonical

Changes in whether Canonical supports a component point to alterations in fundamental components. The interesting one was Consolekit as this was previously important.

• consolekit?
• lynx?
• sshfs
• ssh-askpass-gnome
• texlive
• texlive-latex-extra

Remove

The Removes are almost all programmes where Dpkg doesn't know the repository. The important one for me is Freemind, but changes in the JDK were also worth knowing about.

• airvpn
• bzr-fastimport
• openjdk-6-jre; openjdk-7-jre
• xchat
• freemind

Changes

System programs have configuration files that are in the package. Dpkg will notify you if your version is different to the distributions packaged version. The trade-off is that my version has my configuration, but sometimes new versions will have important configuration changes.

For each change I inspect it and decide if I want the distribution version or my own. In this example it wanted to change /etc/init/lightdm.conf as my version had been configured along the way.

start on ((filesystem
-           and runlevel [!036]
+           and runlevel [!06]
            and started dbus
            and plymouth-ready)
           or runlevel PREVLEVEL=S)

 stop on runlevel [016]

Strange apt commands, sometimes called the Esoteric rites of penguin worship, by LONnptech.org

Bluetooth Sound

A good example of the sorts of issues that happen from time to time was that bluetooth sound appeared not to be working correctly. In the end it turned out to be a change in the way that Bluetooth drivers work and my own hardware configuration.

See: X220 Community page and Bluetooth on Ubuntu 16.04

Application software

The main application software that I test is:

Development / Utility

The main utilities and development software that I check are:

Extensive use of PPA's

Too many PPA's confusing the upgrade process. The problem with PPA's are that they are outside the Ubuntu official repositories so the upgrade paths and conflicts aren't tested. Either don't use them, or strip them out ahead of an upgrade (downgrading to the Ubuntu repository version) and then reintroduce them after the upgrade.

Lack of space

Not enough space to download and install. This happens to me quite a bit. When I ran the update-manager on my first upgrade I got this message:

The upgrade has aborted. The upgrade needs a total of 4,407 M free space on disk '/'. Please free at least an additional 747 M of disk space on '/'. Empty your trash and remove temporary packages of former installations using 'sudo apt-get clean'.

The problem was that my / directory didn't have enough space to download all the packages to for the upgrade process. I needed some temporary space for it to download the packages into.

• Create a temporary TMP

  Create a new temporary location, in my case I'm using LVM so add a new filesystem and mount it, but I've also previously used a USB drive.

  sudo lvcreate -L10G -n aptstore vgUbuntu
  sudo mkfs.ext3 /dev/vgUbuntu/aptstore
  sudo mkdir /mnt/aptstore
  #sudo chmod 777 /mnt/aptstore
  
  sudo mount /dev/vgUbuntu/aptstore /mnt/aptstore
  sudo mkdir /mnt/aptstore/archives
  

• Backup the APT archive locally:

  Copy the apt archive to the new location so that we don't have to redownload anything later.

  $ sudo cp -pR /var/cache/apt/archives/* /mnt/aptstore/archives/
  

• Rebind

  We rebind the /var/cache/apt/archives location to our new filesystem.

  sudo mount --bind /mnt/aptstore/archives /var/cache/apt/archives
  

• Run apt autoclean to make sure everything is fine

  The check command updates the package cache and checks for broken depenencies, it should catch any problems with the set-up. To be extra cautious run autoclean which will delete anything from the cache packages that can no longer be downloaded (e.g. because the apt configuration has changed). Finally run clean if you want to redownload everything.:

  $ sudo apt-get check
  #$ sudo apt-get clean
  $ sudo apt-get autoclean
  

• Run the upgrade

  We can re-run the upgrade at this point and at but don't reboot at the end as we have to refix the apt cache location.

  $ update-manager -d
  # don't let it reboot when it comes to an end
  

• Unbind the /var/cache/apt mount

  Restore /var/cache/apt so we're back to the smaller size.

  $ sudo umount /var/cache/apt/archives
  

• Make sure apt is clean

  We don't want any old files so run apt-clean to remove any old downloaded packages from before the upgrade.

  $ sudo apt-get clean
  

• Sync apt status back

  Move across any packages that are still in the cache. If there's not enough space this isn't really required.

  sudo rsync -av /mnt/aptstore /var/cache/apt
  

• Run apt to check everything is fine

  Update and do any final updates before you reboot the system.

  $ sudo apt-get update
  $ sudo apt-get upgrade
  

• Reboot

  Now you can reboot and check everything is OK.

The system is blown up

If the system has completely blown up and it's impossible to figure out what's happening then it's time to go back and do a clean install. This will wipe out everything except for home.

Technically, we have the backup which we can use to restore back to our previous state, I've never had to use this, but the commands are (roughly):

• Boot the machine using a USB stick

  Boot the system using the Ubuntu USB stick that you created earlier.

• Go to the rescue environment

  Ubuntu's installer has a rescue option.

• Mount the partitions and chroot

  Get all the partitions mounted under the right environment and chroot into them:

  $ sudo mount -t ext3 /dev/mapper/vgUbuntu-root--ubuntu--lv /target
  $ sudo mount -t ext3 /dev/sda1 /target/boot
  $ sudo mount -t ext3 /dev/mapper/vgUbuntu-home--Ubuntu--lv /target/home
  $ sudo mount -t proc proc /target/proc
  $ sudo mount -t sysfs sysfs /target/sys
  $ mount -o bind /dev /newroot/dev
  $ sudo chroot /target
  

• Restore the boot environment

  Grab the boot image from where you backed it up to and write it over the current boot.

  $ rm -Rf /boot/*
  $ gunzip /mnt/backups/lenovosg/snapshots/`date +%Y%M%d`rootsnapshot.dd.gz
  $ sudo dd if=/target/rootsnapshot.dd of=/dev/sda1
  $ sudo grub-install /dev/sda
  $ exit
  

• Restore to the snapshots

  Restore the rest of the system. Umount all the LV's that are mounted so they can merge.

  $ sudo umount /target/*
  $ sudo lvconvert -merge homesnapshot -i -1
  $ sudo lvconvert -merge rootsnapshot -i -1