If you're running Ubuntu (Linux) as a desktop environment the general advice on the Internet to to reinstall every time there's a new release. It's certainly simpler, but it misses the opportunity to keep system configuration and to enjoy a piece of magic - the entire operating system and all applications upgrading from one version to another!
When I started using Linux, being able to upgrade was one of the main differences over Microsoft Windows. Perhaps due to the server heritage upgrades were the most popular methodology - plus getting hold of floppy disks and then CD's for a full fresh install wasn't always easy!
For many years I had very slow connection to the Net - it could take 24 hours to download the 7 floppy disks, nursing a 27.7k modem (relive the sound). Consequently, I've sat through many sweaty-palmed weekends hoping that the connection stayed up while the system tried to download a new glibc. It's fair to say then that my approach is battle tested!
It's not complex, but it is long-winded and conservative. Probably, due to that history!
There are two options for upgrading between one version of a distribution's release and another: fresh install and upgrade. If you do a fresh install then the installer will write over the system partition and install completely new versions of the packages. With an upgrade the installer leaves the partitions alone, installs new system packages and leaves the configuration files in place.
These instructions work for Ubuntu, and should work for Debian. I'm moving from one LTS release to another. With a little alteration the metholodology will work on other systems e.g. CentOS. In this case we're upgrade from Trusty Tahr (Ubuntu 14.04 LTS) to Xenial Xerus (16.04 LTS), but the instructions are applicable to any upgrade.
If you've changed your system configuration then an upgrade will for the most part keep those configuration settings for you. Where I alter server daemons like SSH or hardware like power settings this is very useful.
Upgrades keep all the applications that you've installed for you. If you do a plain install then you'll have to remember what you've installed. That impacts me a lot as I use a lot of non-standard applications.
Between major releases there are often substantial system changes. The upgrade methodology keeps you aware of those alterations.
Sometimes maintainers change to new defaults, particularly where there's been a major new version. Often we're over-riding those changes during a change so you miss them.
Upgrades are more complicated to do than simply installing the system over the top.
Upgrades can fail badly if the Internet connection is interrupted, or the system gets stuck for some other reason. This is a general weakness with package management systems and filesystems where they can't revert the changes.
This document is broken down into broad areas. The first is the upgrade process itself, this is numbered from a. through to p. specifying each step that I take with some reasoning for it. The other section is some common upgrade problems that I've run into over the year - these aren't all, or perhaps even the most common ones, but they are the ones I've run into!
The main structure of these notes is:
The first decision is when to upgrade. It's worth researching and understanding Ubuntu's upgrade paths. I use Ubuntu for work, so stay on the Long Term Support (LTS) releases. I wait for the .1 release which happens 3 months after the initial launch. Some bugs don't show up in the testing that the distributions software teams undertake, you only see them when mass usage takes place and users expand all the different types of use. This is particularly true for bugs that depend on interactions between different pieces of hardware. For the LTS releases I worked on we kept a lot of engineers working on it, resolving bugs for the first 3 months, so the .1 release stabilised the release a lot.
It's worth reading through the release notes which are part of the release announcement. It lists any major changes that the release team thinks users should know about, along with any common problems (and the workaround).
Bear in mind that LTS announcements only cover changes from the previous short release, not for the whole LTS to LTS cycle. To know everything that has changed we have to read the release notes for each interim release. For this upgrade that meant reading:
For Xenial's release:
Note down any major system changes, significant bugs or new features. These are the ones that I noted down:
The benefit of waiting a couple of months before going to the new version is that the distribution developers and users should have solved many of the common problems. To make sure that we're not going to find anything terrible search the Web for the common tools and hardware that you use.
Commonly, I search using DuckDuckGo for the release names and version number a key area and terms like "problem" or "upgrade". For example, I'd search on "16.04 Ubuntu x200 Lenovo problem" to see if anything came up. To decide what key areas you should search for I'd choose any hardware, any commonly problematic area and anything unusual that you use.
My searches were:
In Debian we use apt to upgrade the system. This calculates the upgrades between Debian packages and then performs them in the most suitable way. In Ubuntu we use upgrade-manager which layers on an additional level of intelligence: the distribution engineers often add workarounds for known issues. We can help make the upgrade as simple as possible by removing any old or obsolete packages - it's a good opportunity to do some housekeeping.
I use Synaptic as it's easy to find the Obsolete packages:
In the Sections area find: Status 'Installed (auto removable)' 'Installed (local or obsolete)' 'Not installed (residual config)' Custom Filters Broken # Check for broken applications Origin # Check through for odd things
In each Status area and with Broken or an odd Origin Make for complete removal
There are a variety of tools that can examine the system directly, or through Apt's database and try to simplify or clean the set-up. These tools can be dangerous, so be careful.
Bleachbit tries to find files that are no longer required. I've used it in the past but it's pretty aggressive so skipped it this time. Basic instructions are:
$ sudo apt-get install bleachbit $ sudo bleachbit
Gktorphan examines the Apt database and tries to find files that aren't required. It's worth running. I don't trust it enough to allow it to remove anything, I do that directly through Synaptic.
$ sudo gtkorphan
It gives output a bit like this:
* compizconfig-backend-gconf * compiz-plugins-main-default * gwibber-service * gwiber-service-facebook * gwibber-service-identica * gwibber-service-twiter * libavahi-gobject0 * libavahi-ui-gtk3-o [very long list redacted]
It might be fine to do this directly from through gtkorphan but I'm just being cautious
The default is for Apt to maintain caches, if we want to free up space we can clean this out with.
$ sudo apt-get clean
This is too complicated and may be dangerous.
This is from Debian Cleanup Tip #4: find broken packages and reinstall them. I only found a couple of packages that didn't have md5sums.
$ debsums --changed
I hadn't heard of this before. Adequate checks packages for policy and bugs issues. Felt similar to Gtkorphan.
adequate --tag library-not-found --all
The output was this:
spotify-client: library-not-found /usr/bin/spotify => libcef.so uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libreglo.so uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libunoidllo.so uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libxmlreaderlo.so libsvn1:amd64: library-not-found /usr/lib/x86_64-linux-gnu/libsvn_auth_kwallet-1.so.1.0.0 => libkdecore.so.5 libsvn1:amd64: library-not-found /usr/lib/x86_64-linux-gnu/libsvn_auth_kwallet-1.so.1.0.0 => libkdeui.so.5
You can then figure out what to do about each library problem. I couldn't figure out spotify, for libsvn I just removed the library and subversion as I'm not using them.
Not sure how useful it is for me as it took a long time and I didn't really do anything when it told me this. Could be because I'd already run GTKOrphan.
It's worth making sure that the package database is correct.
In Synaptic go to Edit and Fix Broken Packages, then click Mark all Upgrades button, finally hit the Apply button.
Make sure the whole system is up to date by updating all sources and doing any updates that are required.
Probably best to reboot the system at this point to check that all software is fine: generally you'll have removed some kernels along the way!
We need backups of the whole system. It's worth doing this as a normal backup and an image. Disk images are much easier to recover with. We're also going to backup the whole system, including /boot, so that we can recover to this point.
If I'm short of time the alternative is to make sure I have the previous installer on a USB drive and to backup my home directory: since I previously installed this version it should be safe to depend on it.
I'm using LVM which means I get snapshots "for free". I snapshot everything except for any swap LV's.
Make snapshots of all the partitions.:
$ sudo lvcreate -L10G -s -n rootsnapshot /dev/vgUbuntu/root-ubuntu-lv $ sudo lvcreate -L10G -s -n homesnapshot /dev/vgUbuntu/home-ubuntu-lv
We're using cat to create the images so they are full disk images, this means they take up a lot of space but can simply be copied back over should we have to recover. The whole thing takes a very long time, with the main /home backup taking over two hours to do!
$ sudo sh -c "cat /dev/vgUbuntu/rootsnapshot | pv -s 15G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`rootsnapshot.dd.gz" $ sudo sh -c "cat /dev/vgUbuntu/homesnapshot | pv -s 123G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`homesnapshot.dd.gz" $ sudo sh -c "cat /dev/vgUbuntu/powersgsnapshot | pv -s 35G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`powersgsnapshot.dd.gz"
See this discussion on using cat instead of dd
Before we copy the drive information to our backup we remove them so there's not confusing information in the backup.
$ sudo lvremove /dev/vgUbuntu/rootsnapshot $ sudo lvremove /dev/vgUbuntu/homesnapshot $ sudo lvremove /dev/vgUbuntu/powersgsnapshot $ sudo lvs
We can't snapshot /boot because it's not within LVM. To guarantee the integrity of the backup the partition can't be written to during the backup. Depending on how conservative we are, either drop to runlevel 1 or boot the system from a USB drive.
$ sudo sh -c "dd if=/dev/sda1 | gzip > /mnt/backups/lenovosg/snapshots/\`date +%Y%m%d\`bootsnapshot.dd.gz"
Record various bits of drive set-up information which are useful when restoring.
$ sudo sh -c "pvscan --uuid > /mnt/backups/snapshots/`date +%Y%m%d`-snapshotinfo.txt" $ sudo sh -c "sudo lvdisplay --maps >> /mnt/backups/snapshots/`date +%Y%m%d`-snapshotinfo.txt" $ sudo sh -c "sfdisk --dump /dev/sda > /mnt/backups/snapshots/`date +%Y%m%d`-partitions-backup-sda.sfdisk"
If you do the install and it all messes up then you can restore from snapshot.
Download the new version and verify the image.
Download the new version using the Web or a Torrent.
$ mkdir ~/tmp/download $ transmission-gtk http://releases.ubuntu.com/16.04/ubuntu-16.04.2-desktop-amd64.iso.torrent?_ga=2.200209574.949229349.1493635885-657005342.1481661640
Both the SHA256SUMS and SHA256SUMS.gpg.
$ cd ~/tmp/download $ wget http://releases.ubuntu.com/16.04/SHA256SUMS $ wget http://releases.ubuntu.com/16.04/SHA256SUMS.gpg
If we don't have it already we have to add the official Ubuntu GPG key. Verify that the key has the correct fingerprint.
$ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys "8439 38DF 228D 22F7 B374 2BC0 D94A A3F0 EFE2 1092" "C598 6B4F 1257 FFA8 6632 CBA7 4618 1433 FBB7 5451" $ gpg --list-keys --with-fingerprint 0xFBB75451 0xEFE21092
Verify that the SHA sums file is correct by checking the GPG signature on them.
$ gpg --verify SHA256SUMS.gpg SHA256SUMS
Check that the SHAs match with the file.:
$ sha256sum -c SHA256SUMS 2>&1 | grep OK
We create a bootable USB of the current system so that we can restore back to the current state. The easiest way is to use the disk creator in the menu.
$ lsblk $ sudo ddrescue -D --force --verbose <ubuntu-15.04-image> /dev/sdb
To generate a list of devices that can be used to boot press F12 at start-up and then select USB. To access the BIOS do F1
Turn off screen lock and blank screen as these can mess up the upgrade process. In XFCE this means:
It's useful to run SSHD if you have access to another machine. If there's a problem we can login remotely and save the system properly. I haven't had to do this in a long, long time but it pays to be cautious.
Basic steps are:
$ sudo apt install openssh-client $ sudo apt-get install openssh-server
Configure the server /etc/sshd_config:
Port 2222 #PasswordAuthentication yes AllowUsers bob sarah PermitRootLogin no
Restart the server:
$ sudo restart ssh $ sudo systemctl restart ssh
Test that it's working:
$ ssh localhost
Note if you have strict host checking running ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 127.0.0.1.
With everything up to date, simplified and backed up it's time to run the upgrade process. In Ubuntu update-manager is a front-end to the upgrade process, it uses dpkg but also a set of heuristics to improve the upgrade process. In other words use it!
the Ubuntu development team work on update-manager, making sure that it's aware of specific upgrade problems and enabling it to resolve problems. You should use it and not just plain apt upgrades
Open up a terminal and do:
$ update-manager -d
A lot of upgrades happen without a problem but it's not unusual to get some problems with the process. The upgrader is trying to find a conservative path through all the applications you've installed, updated and altered - sometimes it needs some help.
It's best to note down any changes you see in packages where you think you'd like to reinstall them later. I use the -d switch to figure out any changes I want to be aware of first. Here are some of the changes I noted down:
In many cases it will propose to remove applications that have been installed from repositories it doesn't know about, or if there are fundamental changes in the system. In this case I use LaTeX a little bit and the distribution version had changed. It notified me that it wanted to remove xscreensaver for some reason, I noted this down as I thought I'd need it later.
Changes in whether Canonical supports a component point to alterations in fundamental components. The interesting one was Consolekit as this was previously important.
The Removes are almost all programmes where Dpkg doesn't know the repository. The important one for me is Freemind, but changes in the JDK were also worth knowing about.
System programs have configuration files that are in the package. Dpkg will notify you if your version is different to the distributions packaged version. The trade-off is that my version has my configuration, but sometimes new versions will have important configuration changes.
For each change I inspect it and decide if I want the distribution version or my own. In this example it wanted to change /etc/init/lightdm.conf as my version had been configured along the way.
start on ((filesystem - and runlevel [!036] + and runlevel [!06] and started dbus and plymouth-ready) or runlevel PREVLEVEL=S) stop on runlevel 
If you used the -d switch earlier then all the changes it proposes should be known.
After the upgrade is complete it asks to reboot. Before I do this I always update and apply any upgrades:
$ sudo apt-get update $ sudo apt-get upgrade
I always do this before I reboot just in case there was some terrible kernel issue and an emergency update had to be sent out.
Next reboot and check that everything is fine.
Earlier we noted down applications that were being removed during upgrade. We now go back and re-install them. This will involve searching the Web and finding the PPA or repository that they came from.
In some cases we land up with applications where the dpkg database doesn't know the repository that the applications came from: common if you removed the repository from the deb configuration or if you installed it manually (e.g. dpkg install from the command line). To see these go into Synaptic in 'Installed (local or obsolete)' and then see if these are available from other sources as apt sources so that they're upgraded to the right versions.
The best way to do this is from within Synaptic, in the Settings->Repositories->Other Software section.
We've changed the system from the top to the bottom so it's worth testing both the hardware and software to check that everything works. It's much better to catch a problem now rather than to be confused months later about some changed behaviour.
In reality, the upgrade process has already take a while so there may not be time (or energy) to test everything at once. I keep a file with everything in and then test it all over a couple of days. I've noted some of the things I test, but not all as this tutorial is already long enough!
For hardware tests, first install hardinfo, and then run it, you can also generate a report for later review. Mostly you just check through the main Summary information:
|Environment||Tested under XFCE as it has some additional hardware information which is easy to access.|
|Processor||Double check that both CPU's are operating. 2x Intel Core 2 Duo CPU P8600 @ 2.40 Ghz||Pass|
|Memory||Check that it can address all 8GB. I always double check in top.||Pass|
|Hard drive||ATA Samsung SSD 840. You'll know about it if this isn't working.||Pass|
Mesa DRI Intel GM45 Express Chipset.
Open the 'Displays' applet in the Systems Settings and check. Resolution: 1920x1200 (16:10). The resolutions or dual-screen get messed up in upgrades some times.
HD Pro Webcam C920 (/dev/video0)
On laptop (not on dock). Check list with lsusb
USB Kinesis Advantage Pro
Logitech wireless receivers: software is Solaar. Small USB plugin and the Anywhere MX mouse
Works, selecting text is a bit broken but this is an old issue which is likely the hardware.
|Printers||Check printer shows: system-config-printer - HP LaserJet P2015 (NPI8261AA; 192.168.1.253); via HPLIP||Pass|
A good example of the sorts of issues that happen from time to time was that bluetooth sound appeared not to be working correctly. In the end it turned out to be a change in the way that Bluetooth drivers work and my own hardware configuration.
Check that all the applications have installed and are working.
|Display manager||Test that the display manager (LightDM) shows and logs in the desktop environments I use (Xfce, i3, Unity).||Pass|
Check that I can alter the X configuration; output to my main screen; turn-off the laptop screen.
xrandr --query xrandr --output VGA1 --mode 1920x1200 xrandr --output LVDS1 --off
|Desktop Env XFCE||Check that XFCE desktop environment works properly Function - Change screen: GUI in accessories - Switch windows: <Super key>1-0||Pass|
Check that i3 configuration works properly.
|Multiple logins||Check that I can login on multiple VT's so I can run more one session at a time.|
|Encrypted files||Check that encryption is working correctly.||Pass|
|Synaptic||Check that the system software is all fine.||Pass|
|SSH||Check we can SSH to standard external sites.||Pass|
|Password agent||Check that password agent is working.||Pass|
|Sudo||Check that gksudo is working correctly.||Pass|
|Power management||Check power management is up and running. sudo tlp-stat -s||Pass|
|Application containers||Check that application containers are all fine.||Pass|
Check through the most important applications and check that they work correctly. I haven't included the test scenarios for all the applications that I use as it would make this page too long (and they're pretty rough). Here's an example set of test cases for Mutt:
- Version 1.5 in archive; changed to PPA for vim 1.7
- Receive email
- Check for email regularly (mbsync daemonize)
- Delete email (mbsync sync back)
- Read email
- Write email
- Send email
- Send email (check .msmtp/log/msmtp.log and msmtp.queue.log)
- Runs queue regularly (check .msmtp/log/msmtp-queu-daemonize-output.log)
- Keychain for ssh etc
The main application software that I test is:
|Browser - Firefox||Browser checks.||Pass|
|Email - Thunderbird||Check that email is fine.||Pass|
|Browser - Chromium||Check that alternative browser is running.||Pass|
|Email - Mutt||Check that alternative email is running.||Pass|
|Filemanager - Thunar||Check that the file manager works.||Pass|
|IRC - weechat||Check that the IRC client is all fine.||Pass|
|Music - Spotify||Check we have music.||Pass|
|Personal Cloud - Google Cloud||Check we can store files online.||Pass|
|Personal Cloud - Dropbox||Alternative Cloud storage.||Pass|
|Personal Cloud - SpiderOak||Alternative Cloud storage.||Pass|
|Text editor - Vim||Primary text and programming editor.||Pass|
|Note Taking - Vim Wiki||Primary Note taking system.||Pass|
|Office Productivity - LibreOffice||LibreOffice office productivity||Pass|
|Version control - Git||Version control for files and programs||Pass|
|Todo management - task||Todo management for the command line||Pass|
|Mindmaps - Freemind||Mindmapping for organisation||Pass|
The main utilities and development software that I check are:
|Python||Python development environment||Pass|
|Clojure||Clojure development environment||Pass|
|Diff||Diff for comparison of code||Pass|
|Bash||Command line environment||Pass|
|Ranger||Command line file manager||Pass|
|nmcli||Network manager cli client||Pass|
There's generally some reconfiguration that has to be done at this stage. For example, resetting up ssh-agent and fixing hibernation. I keep a log of all changes I make, so it's easy to go back through this and double check that nothing has been missed.
I learnt a long time ago that if it isn't written down then it didn't happen. So I keep a log of all changes that are made to the system and specifically for each upgrade I keep a file. It doesn't have to be snazzy but some notes about what you changed can really save you later when something goes wrong and you need to know the last few stages.
You're changing the entire operating system and applications so it's possible you might have some problems. But since this is open source there's always a way to solve them!
Too many PPA's confusing the upgrade process. The problem with PPA's are that they are outside the Ubuntu official repositories so the upgrade paths and conflicts aren't tested. Either don't use them, or strip them out ahead of an upgrade (downgrading to the Ubuntu repository version) and then reintroduce them after the upgrade.
Not enough space to download and install. This happens to me quite a bit. When I ran the update-manager on my first upgrade I got this message:
The upgrade has aborted. The upgrade needs a total of 4,407 M free space on disk '/'. Please free at least an additional 747 M of disk space on '/'. Empty your trash and remove temporary packages of former installations using 'sudo apt-get clean'.
The problem was that my / directory didn't have enough space to download all the packages to for the upgrade process. I needed some temporary space for it to download the packages into.
Create a new temporary location, in my case I'm using LVM so add a new filesystem and mount it, but I've also previously used a USB drive.
sudo lvcreate -L10G -n aptstore vgUbuntu sudo mkfs.ext3 /dev/vgUbuntu/aptstore sudo mkdir /mnt/aptstore #sudo chmod 777 /mnt/aptstore sudo mount /dev/vgUbuntu/aptstore /mnt/aptstore sudo mkdir /mnt/aptstore/archives
Copy the apt archive to the new location so that we don't have to redownload anything later.
$ sudo cp -pR /var/cache/apt/archives/* /mnt/aptstore/archives/
We rebind the /var/cache/apt/archives location to our new filesystem.
sudo mount --bind /mnt/aptstore/archives /var/cache/apt/archives
The check command updates the package cache and checks for broken depenencies, it should catch any problems with the set-up. To be extra cautious run autoclean which will delete anything from the cache packages that can no longer be downloaded (e.g. because the apt configuration has changed). Finally run clean if you want to redownload everything.:
$ sudo apt-get check #$ sudo apt-get clean $ sudo apt-get autoclean
We can re-run the upgrade at this point and at but don't reboot at the end as we have to refix the apt cache location.
$ update-manager -d # don't let it reboot when it comes to an end
Restore /var/cache/apt so we're back to the smaller size.
$ sudo umount /var/cache/apt/archives
We don't want any old files so run apt-clean to remove any old downloaded packages from before the upgrade.
$ sudo apt-get clean
Move across any packages that are still in the cache. If there's not enough space this isn't really required.
sudo rsync -av /mnt/aptstore /var/cache/apt
Update and do any final updates before you reboot the system.
$ sudo apt-get update $ sudo apt-get upgrade
Now you can reboot and check everything is OK.
If the system has completely blown up and it's impossible to figure out what's happening then it's time to go back and do a clean install. This will wipe out everything except for home.
Technically, we have the backup which we can use to restore back to our previous state, I've never had to use this, but the commands are (roughly):
Boot the system using the Ubuntu USB stick that you created earlier.
Ubuntu's installer has a rescue option.
Get all the partitions mounted under the right environment and chroot into them:
$ sudo mount -t ext3 /dev/mapper/vgUbuntu-root--ubuntu--lv /target $ sudo mount -t ext3 /dev/sda1 /target/boot $ sudo mount -t ext3 /dev/mapper/vgUbuntu-home--Ubuntu--lv /target/home $ sudo mount -t proc proc /target/proc $ sudo mount -t sysfs sysfs /target/sys $ mount -o bind /dev /newroot/dev $ sudo chroot /target
Grab the boot image from where you backed it up to and write it over the current boot.
$ rm -Rf /boot/* $ gunzip /mnt/backups/lenovosg/snapshots/`date +%Y%M%d`rootsnapshot.dd.gz $ sudo dd if=/target/rootsnapshot.dd of=/dev/sda1 $ sudo grub-install /dev/sda $ exit
Restore the rest of the system. Umount all the LV's that are mounted so they can merge.
$ sudo umount /target/* $ sudo lvconvert -merge homesnapshot -i -1 $ sudo lvconvert -merge rootsnapshot -i -1
Alright! We're all done with a new upgraded system - ready to enjoy the latest goodies! As you've seen this approach is cautious, trying to ensure success, but the cost is time and effort. Using this process makes it simple to move from each successive release of Ubuntu or from LTS-to-LTS with a working system and all your configurations intact. If you have tips or ideas on how to do upgrades, or you used this process, leave a comment below!blog comments powered by Disqus