Introduction

Msmtp is a small mail delivery agent [1] that command line email applications use to send email via the users smarthost. Email clients (MUA's) such as Mutt, Alpine or Sup send email through it as if it were a standard Sendmail interface. It uses far fewer resources than running a full MTA, with the caveat that it can only hand-off email to an internet provider for delivery.

Email by Jurgen Appelo, CC BY 2.0

I'm using Mutt which has basic support for sending email but it's recommended configuration is to use a separate agent, particularly if mail is being sent out through multiple accounts. Some people will use the full capability of an MTA such as Postfix, but I wanted something less resource intensive.

There are a variety of light-weight mail transfer agents. Msmtp is very popular with Mutt users and there didn't seem to be an obviously better option. Having used it for a few days I'm pretty satisfied.

Weaknesses

• Complicated to set-up

It's a command line application dealing with a fairly complex set of protocols, so the opposite side of configurable is that it's complex. Be prepared to set aside a couple of hours for a set-up that sings!

• Can fail if there's a lock file

The downside of not being part of the email client (MUA) is there's a point when email is handed off from the client to Msmtp. The one bug I see reported in multiple places is that users are told to set Mutt to return immediately without checking the status of the send from Msmtp - most of the time this will be fine, but if Msmtp has a problem with start-up then it may not fail gracefully (most commonly this is due to it having problems with it's lock files). This means email would go missing!

• Queueing extension and automated sending are shell scripts

Msmtp is designed to be just enough to act as a sendmail interface, the question is how much is 'just enough'. Queueing isn't part of the core and the ability to automate sending when the system goes online is another set of shell scripts. This could be brittle and I just don't like shell scripts much!

You got mail, image by Bruno Girin, CC BY-SA 2.0

Table of contents

Email security

It was quite confusing trying to figure out why tutorials had such different settings for securely sending SMTP over SSL/TLS. The reason is that encryption is always complex and the way it interacts with email has been through a few iterations. This results in variations between email providers set-ups.

It's worth noting that in 2016 email transmission remains insecure! Although, it's now standard to secure the communication from the client to their smarthost, the transmission between host providers is commonly not over a TLS tunnel. There is more to do!

The terms SSL and TLS are often used interchangeably, TLS is the successor to SSL. Transport Layer Security (TLS) provides two ways for using it to secure a protocol:

• The protocol session is initiated in the clear then the STARTLS command begins encryption
• The protocol session immediately initialises TLS and only then does the normal protocol begin

To add transport security to the existing email protocols new ports were assigned:

• IMAP SSL/TLS on 993
• POP SSL/TLS on 995
• SMTP SSL/TLS on 465

IMAPS and POPS are pretty standard, everyone uses the correct ports and the only variability is the strength of the encryption. The situation with sending email is more complicated.

The Internet standards bodies have favoured using the standard port and using STARTTLS to upgrade the conversation. Consequently, port 465 is not an official port (it was reassigned to a different protocol)! The problem is that some client software wouldn't upgrade the connection properly and that some attackers would purposefully 'downgrade' the communication (responding that they didn't support STARTTLS) to do man-in-the-middle attacks. As the whole conversation isn't wrapped there's at least some leaking of meta data such as what the hostname is and how often it send email.

A slight kink is that port 587 was defined for authenticated submission of mail for onward transmission (being a smarthost), rather than transfer of a fully complete message (with all headers and FQDN). So port 587 is supposed to be used by an email client (an MUA) to talk to its smarthost for sending email. It was done around the time that using SSL/TLS was becoming popular so it became a defacto standard that communication on port 587 requires SSL/TLS with STARTTLS upgrading and (generally) authentication. This leaves us with:

• Encrypted SSL/TLS wrapped session on port 465
• Clear session initiation with required STARTLS upgrade on port 587

Where it's a completely wrapped session it's often called SMTPS. Due to the risks of STARTLS, it's best to use a completely encrypted session (SMTPS). For most providers this will mean using port 465 which is unfortunate as it's not an official port.

Installing MSMTP

Msmtp is part of a lot of distributions, on Ubuntu install it with:

$ sudo apt install msmtp

Create some core files, ~/.msmtprc is for configuration and has to have 0600 permissions. The log file location is optional, I'm creating an ~/.msmtp/ directory and then storing everything under this:

$ touch ~/.msmtprc
$ mkdir ~/.msmtp/log/msmtp.log
$ chmod 0600 $HOME/.msmtprc

A short example configuration for ~/.msmtprc:

# From this line to the first 'account X' line are settings for all accounts
defaults
tls on

# This is where the TLS CERTs reside
tls_trust_file /etc/ssl/certs/ca-certificates.crt

# With tls_starttls on it will be a clear session and then upgrade to TLS
# using the STARTTLS command. If set to 'off' then the whole session is
# encrypted inside a TLS session (SMTPS)
tls_starttls on

# This is the log file - comment it out to stop it collecting
logfile ~/.msmtp/log/msmtp.log

# Authentication must be used for sending
# Each account will use a different method
auth on

## Account ##
account account-name
host mail.runbox.com
port 465

user username@domain
password somepassword

The Account section provides configuration for each specific email provider. To check the configuration is working do:

$ msmtp -Sd

This displays information on the configuration and on certificates that the remote host is using. If you have more than one account add --account=<accountname>. To send a test email, create a file with the recipient email address:

To: username@mydomain.com
From: username@mydomain.com
Subject: A test to see if MSMTP is delivering

Hello there.

Then from the commandline do the following:

cat email.txt | msmtp --account=default <recipient@somedomain.com>

Note that the <recipient@somedomain.com> is where the email will be sent using SMTP - logically the To in the email and the one for Msmtp to use on the SMTP conversation will be the same. For testing it's useful to send to your email address at your email provider as this checks that the smarthost is receiving and that it's processing it correctly within your providers email system. A second check is to send email to an external address (e.g. a free account like Gmail or Yahoo) as this checks that everything is fine for relaying outside your providers email system.

Less stressed, but less employed! Image by Ron Madder, CC BY-SA 2.0

As discussed above I want to use a completely wrapped TLS session which means using port 465 in the account section of the file. I tell Msmtp to wrap the session with this setting:

tls_starttls off

Smarthost fingerprinting

As part of TLS process we want Msmtp to check the validity of the TLS certificate. The general default is to use a system CA certificates file, on Ubuntu this is part of the ca-certificates package which places them at /etc/ssl/certs/ca-certificates.crt, so often in the defaults section we have:

tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt

The problem is that the system provided CA certificates could be out of date or revoked, there's no way to check that they're still valid. If we know the details for the specific smarthost that we're using then we can tell Msmtp the fingerprint it should use. The first step is to get the smarthosts certificate signature:

# Either run this to show all the details of the default account or
# use --account=XX to choose a specific one
$ msmtp -dS

# Alternatively
$ msmtp --serverinfo --tls --tls-certcheck=off --host=mail.someserver.com --port=587

In my case I'm using a completely wrapped TLS session, so the first command option works, but the second one didn't. Anyway, this returns an MD5 and SHA1 fingerprint, either works in the configuration file:

tls on
tls_fingerprint 4A:2B:DD:BE:B5:98:6B:C1:37:51:89:B0:83:27:FC:40:FF:B8:F3:46

Password storage

In the example above we stored the accounts password in the configuration file. A more secure configuration is to either store the password in the system keyring (gnome-keyring on Linux or Mac OSX's keychain), or to store it with a system such as GnuPG.

To store the configuration in gnome-keyring (Linux) or Mac OSX keychain, we need the following command:

$ secret-tool store --label=msmtp host smtp.your.domain service smtp user yourusername

The 'host' argument should match the host in the account section and 'user' is the username that it will use. As long as there's no password set in the configuration file it will automatically try and access the users keyring.

It's slightly easier for me to use GnuPG as it means I don't have to run the GNOME libraries. To store the password in gpg agent we first have to create a file for the secret password to be stored in, then gpg encrypt it:

$ touch ~/.msmtp/msmtp-myaccount-pass.gpg
$ gpg --encrypt --output ~/.msmtp/msmtp-myaccount-pass.gpg --recipient my@gpg-email -

In the Msmtp configuration file we tell it to use the output of the passwordeval command:

passwordeval "gpg --quiet --for-your-eyes-only --no-tty --decrypt ~/.msmtp/msmtp-myaccount-pass.gpg"

If you don't have gpg-agent running as part of your session you'll have to unlock it:

eval $(gpg-agent --daemon)
gpg-agent

To make sure it's available for future sessions, I added this to my .xinitrc:

# Start gpg-agent for X session
eval $(gpg-agent --daemon)

At sign by Howard Lake, CC BY-SA 2.0

Configure Mutt to use Msmtp

Having tested the configuration we can now configure Mutt to send through Msmtp. In our .muttrc we add the following:

set sendmail="/path/to/msmtp"
set use_from=yes
set realname="Your Name"
set from=you@example.com
set envelope_from=yes

The setting envelope_from=yes lets Msmtp check the envelope from address that Mutt supplies and matches this against the from setting in Msmtp (it's equivalent to using Msmtp's -f option). If you have multiple accounts this will match each accounts email address.

At this point we can send email when we're on-line, but I also want to write and queue email when my laptops not on the network.

Queuing outgoing email

Msmtp ships with two sets of user contributed scripts which will queue outgoing email. I'm using the msmtpq ones created by Chris Gianniotis which are easy to get running. There are other options such as msmtp-offline but msmtpq is popular so I've stuck with it. The scripts are shipped with Ubuntu under /usr/share/doc/msmtp. To install them copy them to a location in your path:

$ cp /usr/share/doc/msmtp/examples/msmtpq/msmtpq.gz ~/bin/
$ cp /usr/share/doc/msmtp/examples/msmtpq/msmtp-queue ~/bin/
$ gunzip ~/bin/msmtpq.gz
$ chmod u+x ~/bin/msmtpq
$ chmod u+x ~/bin/msmtp-queue

To configure the msmtpq script we have to configure two variables. The first one is the directory that mail will will be queued in before it's sent out. As Msmtp is running as a user application this should be somewhere under the users home directory. The default is ~/.msmtp.queue, I'm using .msmtp/ and created a mailqueue/ directory underneath. Edit the msmtpq file:

Q=~/.msmtp/mailqueue
LOG=~/.msmtp/log/msmtp.queue.log

Create the directory:

$ mkdir ~/.msmtp/mailqueue
$ chmod 0700 ~/.msmtp/mailqueue
$ mkdir ~/.msmtp/log/

Note that the mailqueue directorys permissions have to be 0700. If it's a system install of Msmtp the MSMTP variable shouldn't need to be edited to directly find the binary as it's in the system path. I had to change it later to point to the binary as I had some problems with where I defined my paths.

Mutt should call the msmtpq script so alter muttrc:

set sendmail = /path/to/msmtpq'
set sendmail_wait = -1

To test it's working we switch off the Internet and write an email with Mutt - Mutt should say the email has been sent, it's just being queued.

To see the queue in action we can do:

# Display the current queue
$ msmtp-queue

# Run the queue
$ msmtp-queue -r

# Send selected mails in the queue
$  msmtp-queue -R

# Delete selected emails from the queue
$ msmtp-queue -p

The script supports doing connection tests and some other functionality but I didn't need any of this - it all worked first time!

Running email when the network is active

Postal mail stamps by Hersson Piratoba, CC BY-NC-ND 2.0

Most posts I read about Msmtp queuing suggested using Cron to send email every five minutes - that's fine but my laptop isn't online all the time so it's a bit wasteful. Then I ran into a post by Reza Jelveh whose created automation to check email when his system is online and when it's offline email is queued as normal. It's a fantastic solution and worth the effort!

Reza's solution uses Network Managers built in capability to run a script when the network status changes. When the network status changes to online Network Manager runs a dispatch script which kicks of an Msmtp script, this flushes the Msmtp queue and re-flushes it every five minutes. Later, when the network status changes to offline Network Manager re-runs the dispatch script and this time it stops the Msmtp queue flushing script so that email queues.

There are two components (scripts), the first is the Network Manager dispatch script, and the second is a script to make msmtpq into a daemon that flushes the msmtp queue every five minutes. There are lots of ways to write daemons, but if you want to run a normal program as a daemon you need something that will wrap it and provide various bits of support (e.g. handle PID's). I did a lot of research on how people create daemons:

• There are general recommendations to use the shell scripting support libraries associated with init.d (on Debian it's start-stop-daemon) to create daemons. This didn't feel right to me as the context for this script is being run as part of Network Manager under my userid rather than as part of init.
• Libslacks Daemon was an obvious options as it's in Ubuntu already, has been around a long time and appeared to be very stable. I spent a few hours trying to get it working but it wouldn't manage the process properly.
• Reza used Daemonize which is in OSX but isn't in my version of Ubuntu. It turned out it was easy to install and now I wish I hadn't wasted so many hours on Daemon!

The first thing is to create a script to run the mailqueue as a daemon, mine is called msmtp-queue-daemon.sh:

#!/bin/sh
# For use with the daemon command to run msmtp-queue every 5 minutes

# Check msmtp-queue is fine
if [ -x /home/yourusername/bin/msmtp-queue ]; then
    while true; do
        /home/yourusername/bin/msmtp-queue -r &
        date -n `date +%c`
        echo ": Running msmtp-queue"
        sleep 300
    done
fi

# Not totally sure on the syntax here - signal handling
trap "echo -n exiting... ;kill $!;exit" INT TERM

Its function is to run the standard msmtp-queue script flushing the msmtp queue, it then sleeps for 5 minutes and does it again. Our next step is to convert this script into a Daemon so that it can run continuously.

As Daemonize isn't in 14.04 LTS the best option was to install one from a later version of Ubuntu:

• Find Daemonize on Launchpad
• Download the right package
• Install it with dpkg --install daemonize_1.7.7-1_amd64.deb

At the most basic level to test out daemonize we run it from the commandline with something such as this:

daemonize -v -a -p /home/yourusername/.msmtp/msmtp-queue-daemonize.pid -l /home/yourusername/.msmtp/msmtp-queue-daemonize.pid \
-e /home/steve/.msmtp/log/msmtp-queue-daemonize-stderr.log -o /home/yourusername/.msmtp/log/msmtp-queue-daemonize-output.log \
/home/yourusername/bin/msmtp-queue-daemon.sh

Where -v puts it into verbose mode so that it sends some output to stdout; -a appends log files rather than over-writing them; -p this is the file location to store the process PID; -l is the lock file, if this file exists daemonize will not start a second service, note that I'm using the same file as the PID file; -e this is the stderr log, I didn't get any output from this so removed it later; -o this is the output of the command; and the final parameter is the msmtp queue daemon script that we created earlier. A successful run will create the PID file that you can use to check the process, and in the output file it shows msmtp-queue running every five minutes.

If that all works, then the next step is to test the command as root (sudo bash):

daemonize -v -a -u <yourusername> -p /home/yourusername/.msmtp/msmtp-queue-daemonize.pid -l /home/yourusername/.msmtp/msmtp-queue-daemonize.pid \
-e /home/yourusername/.msmtp/log/msmtp-queue-daemonize.log -o /home/yourusername/.msmtp/log/msmtp-queue-daemonize-output.log \
/home/yourusername/bin/msmtp-queue-daemon.sh

The only difference is the use of -u <yourusername> which tells daemonize to switch to the specified username to run the rest of the command. We're now ready to integrate the daemonized msmtp-queue-daemon.sh into Network Manager.

$ sudo touch /etc/NetworkManager/dispatcher.d/99smtp-queue-run.sh
$ sudo chown root:root /etc/NetworkManager/dispatcher.d/99smtp-queue-run.sh
$ sudo chmod u+x /etc/NetworkManager/dispatcher.d/99smtp-queue-run.sh

#!/bin/sh

INTERFACE=$1    # The interface that's being brought up or down
STATUS=$2       # Interface status - up or down

USER=myuser
HOME=/home/$USER
PIDFILE=$HOME/.msmtp/msmtp-queue-daemonize.pid
LOCKFILE=$HOME/.msmtp/msmtp-queue-daemonize.pid
STDOUTFILE=$HOME/.msmtp/log/msmtp-queue-daemonize-output.log
DAEMONIZE=/usr/sbin/daemonize
NOTIFY=/usr/bin/notify-send

# If the display variable is unset, find the correct display number
# and set the variable so we can send messages
#if [ -z "$DISPLAY" ]; then
#    console=`fgconsole`
#    dispnum=`ps t tty$console | sed -n -re 's,.*/X(org)? .*:([0-9]+).*,\2,p'`
#    export DISPLAY=":$dispnum"
#fi
export DISPLAY=":0"

logger -p daemon.err "MSMTP queue: dispatcher running"

case "$STATUS" in
    'up')
        # Network interfaces are active, so run daemonize
        daemonize -a -u $USER -p $PIDFILE -l $LOCKFILE -o $STDOUTFILE /home/steve/bin/msmtp-queue-daemon.sh
        if [ $? -eq 0 ];then
            su - $USER -c "$NOTIFY -i info --expire-time 180 'Network up' 'Running mail'"
        else
            # Could be that the PID file is acting as a lock file, this doesn't show on the stderr log
            su - $USER -c "$NOTIFY -i error 'MSMTP queue: Error daemonising email - intervention required'"
            logger -s -p daemon.err "MSMTP queue: Unable to daemonize"
        fi
        ;;
    'down')
        # Network interfaces are down
        NETWORKSTATUS=`nm-tool|grep State|cut -f2 -d' '`
        # Note that you have to have quote marks around the variable
        if [ "$NETWORKSTATUS" != "connected" ];then
            # Remove the PIDFILE or we can't run again
            kill -TERM `cat $PIDFILE`
            if [ $? -eq 0 ];then
                su - $USER -c "$NOTIFY -i info --expire-time 180 'Network down' 'Stopping mail'"
                rm $PIDFILE || su - $USER -c "$NOTIFY -i error 'MSMTP queue: Error removing the lock - intervention required'"
                # No ideal why this doesn't work
                #if [ $? -eq 0 ];then
                #    #$NOTIFY -i info 'MSMTP queue: Error removing the lock file - intervention required'
                #    logger -s -p daemon.err "MSMTP queue: Unable to remove lock file - may not be able to run again"
                #fi
            else
                su - $USER -c "$NOTIFY -i info 'MSMTP queue: Unable to kill the PID - intervention required'"
                logger -s -p daemon.err "MSMTP queue: Unable to kill the PID - may not be able to run again"
            fi
        fi
            ;;
esac

Multiple accounts

A lone mail box, by Finntasia

I have more than one email account, which means msmtp needs to select the correct SMTP email server depending on the email address. The msmtp queue script we set-up earlier supports multiple email accounts.

First, tell Mutt to provide an SMTP from address:

set envelope_from=yes

Then in the .msmtprc set-up the accounts. Make sure that the from address for each account in Mutt is the same as the one in each Account is Msmtp. For example, lets say the from address we use in Mutt for this account is exampleuser@gmail.com, we would have settings such as:

## GMail Account ##
account gmail
host smtp.googlemail.com
port 465
from exampleuser@gmail.com
user exampleuser@gmail.com
password somelongpassword

To temporarily set a from line in Mutt, start it and do:

:set from=exampleuser@gmail.com

Then write an email as that user, it's best to turn off the Internet connection for testing first, then run the queue:

$ msmtp-queue -r

This should show that it's trying to deliver to the smarthost depending on the SMTP envelope that is set in Mutt. When you turn on the Internet the log file should show something like this:

Aug 11 23:50:40 host=smtp.googlemail.com tls=on auth=on user=exampleuser@gmail.com from=exampleuser@gmail.com recipients=exampleuser@funnydomain.net mailsize=391 smtpstatus=250 smtpmsg='250 2.0.0 OK 1470952239 t188sm1633646wme.19 - gsmtp' exitcode=EX_OK
Aug 11 23:50:42 host=mail.runbox.com tls=on auth=on user=example%runbox.net from=exampleuser@runbox.net recipients=example@runbox.net mailsize=383 smtpstatus=250 smtpmsg='250 OK id=1bXxsA-0000o6-8N' exitcode=EX_OK

To regularly use this in Mutt we can set-up a macro:

macro generic "<esc>1" ":set from=you@example.com"
macro generic "<esc>2" ":set from=you@your-employer.example"
macro generic "<esc>3" ":set from=you@some-other-provider.example"

For those using Emacs there's a write-up on the Gnus MSMTP page.

Msmtp log rotation

We're logging email delivery and how our daemon is running so it's useful to rotate the logs. Create a file in /etc/logrotate.d name it msmtp to set-up log rotation:

/home/myuser/.msmtp/log/msmtp.log /home/myuser/.msmtp/log/msmtp.queue.log  \
/home/myuser/.msmtp/log/msmtp-queue-daemonize-output.log {
rotate 12
weekly
compress
notifempty
missingok
}

For readability the files are spread over two lines in the listing above, but in reality should be on one line.

System aliases

If you're running Msmtp at a system level then the configuration with the benefit of use system wide aliases. Msmtp supports system wide aliases (/etc/aliases) if you run it in a system wide configuration. The core issue is that you have to put SMTP authentication somewhere unless there's a local Smarthost on the network. There's a good write-up of this in Configuring msmtp on Ubuntu. As the process is running as root it can also log to the /var/log/ directory.

Msmtp example configuration

The final example Msmtp configuration file:

# Set default values for all following accounts.
## MSMTP Configuration ##
# From this line to the first 'account X' line are settings for all accounts
defaults
# Always use TLS
tls on
# This is where the TLS CERTs reside
# Either this or tls_fingerprint can be used
#tls_trust_file /etc/ssl/certs/ca-certificates.crt

# With tls_starttls on it will be a clear session and then upgrade to TLS
# using the STARTTLS command. If set to 'off' then the whole session is
# encrypted inside a TLS session (SMTPS)
# For Runbox you can't have this and be in port 587
tls_starttls off
# This is the log file - comment it out to stop it collecting
logfile ~/.msmtp/log/msmtp.log

# Authentication must be used for sending
# Each account will use a different method
auth on

## Runbox Account ##
account runbox
#tls_starttls off
tls_fingerprint 4A:2B:DD:BE:B5:98:6B:C1:37:51:89:B0:83:27:FC:40:FF:B8:F3:46
host mail.runbox.com
port 465
domain myserver.mydomain.com

# Specifies the 'FROM' in the SMTP envelope
# Either this is set, or use auto_from
from myemail@mydomain.com

# Authentication for this account
user myemail%mydomain.com
passwordeval "gpg --quiet --for-your-eyes-only --no-tty --decrypt ~/.msmtp-runbox.gpg"

## GMail Account ##
account gmail
tls_fingerprint 19:41:05:2D:AB:F1:73:7D:5F:18:7D:6E:F4:20:20:41:FF:FF:01:D3
host smtp.gmail.com
port 465
domain myserverhostname.mydomain.com
from exampleuser@gmail.com
user exampleuser@gmail.com
password somefunnylongpassword

# A second mail address at Runbox
# The colon tells it to inherit the settings from the previous account
account runbox2: runbox
from different-email@mydomain.com

# Set a default account
account default: runbox

MSMTP resources and footnotes

Email in lights by Chris Goldberg, CC BY-NC 2.0

• ArchWiki - Msmtp

  Very good page on the ArchWiki covering all aspects of configuring Msmtp.

• Installing and configuring mSMTP

  Well written and easy to follow short guide.

• Offlineimap and Msmtp

  Using Msmtp on a remote server with some clever tricks like caching the passwords in environmental variables.

Comments

If you have any comments or experiences with Msmtp please leave your thoughts below.