Hack 59. Convert Your Mailbox

Don't let your old email become stranded in a format that your new mail program can't read.

Switching email clients, MTAs, or mail servers can be a traumatic process. And getting used to the different interface of an email client and its way of working can be awkward. Aside from the cosmetic changes, you might have a far more serious problem if the new client stores its mail differently.

This hack takes you through the steps of converting between the two most popular formats on Linux: mbox and maildir. This can often be a daunting and confusing process, but with this hack by your side, you will be converting your mail in no time.

The most common reasons to switch mailbox formats are as follows:

Switching email clients

Not all email clients support both mailbox formats. If you really want to use a client that doesn't support your current format, you need to covert your old mail to a format that is supported.

Switching your local delivery agent

Many Linux users run an MTA on their desktops to handle mail delivery to local users. Like email clients, not all email servers support each mailbox format. When switching servers, it might be necessary to convert your mail from one format to another.

Switching mail servers

This is like switching your local MTA, only on a larger scale. If you administer users on a mail server, it might be necessary to convert their mailboxes when you switch MTAs.

Generally, the mailbox format is dictated by the MTA, and you choose your client accordingly. Of course, if you want a particular client for its unique feature, you might need to switch (or reconfigure) your MTA. You might also decide to switch between the two based purely on technical merits, so here is a brief rundown of the differences.

8.6.1. mbox Versus maildir

mbox is the older of the two mail storage formats. It is actually the generic name for a family of related formats. Although the formats are slightly different, they both store multiple messages within a single file. These multiple formats came about because of different Mail User Agents (MUAs) implementing their own variations of the original format. Although they are very similar, they are generally not compatible between MUAs. Because of the single file format, there can be problems with file-locking and storage on networked filesystems such as NFS, which might cause the mail file to be corrupted.

maildir, on the other hand, stores one message per file. This removes the locking problems of mbox, which means it should be your first choice is your mail is stored over NFS. Because of its relative youth compared to some of the MUAs that support it, there are no variants of the format, so your mail should be portable between clients.

Generally, maildir is considered to be superior to mbox.

8.6.2. Converting mbox to maildir

You can convert between the two formats with the mb2md.pl Perl script, which you can find at http://batleth.sapienti-sat.org/projects/mb2md. To use this tool, you need to have both Perl and its TimeDate module installed, but packages of the script are available for various distributions that should take care of these dependencies for you.

First, you need to know where your mail is coming from. Your mbox mail is normally stored in two locations: /var/spool/mail/$user for new mail and an mbox file in your home folder for read mail, although the paths and filenames can be distribution-dependent. If you sort your mail into folders, each folder is represented by a single file within your mail directory.

When run, mb2md.pl discovers your new and read mail locations automatically, but you need to point it at any other folders containing mbox files. If necessary, you can also point it at specific files. The following examples assume you've downloaded the script from its site.

To convert your new and read mail into a newmaildir folder:

foo@bar:~$ perl mb2md.pl -m

To send the output to a different folder:

foo@bar:~$ perl mb2md.pl -m -d  somefolder

To tell it where to convert the files in a specific folder:

foo@bar:~$ perl mb2md.pl -s  sourcefolder

To convert the files in a folder recursively (for nested folders):

foo@bar:~$ perl mb2md.pl -s  sourcefolder  -R

8.6.3. Converting maildir to mbox

The main conversion from maildir to mbox is achieved in a much more hack-ish way, by writing a small shell script that uses the formail tool to filter input into mbox format. Create a file called md2mb.sh in you home folder, making sure you replace maildir with your own mailbox. Insert the following text into the file:

#!/bin/bash
for file in `find maildir/.$1/cur/ -type f`
do
    cat $file | formail -a Date: >> mbox
done

Make the script executable with:

foo@bar:~$ chmod +x md2mb.sh

Then run it with:

foo@bar:~$ ./md2mb.sh

To process a particular folder, run it with:

foo@bar:~$ ./md2mb.sh 
foldername

This appends any email in the maildir folder to an mbox file that can be read with mail or your favorite MUA.

David Murphy