Backing up e-mail

E-mail, for many of us, is very important and accumulates forever, making it a large mess when it comes to backing it up.

The importance of my e-mail snuck up on me. Once upon a time, I opened my old reliable e-mail program and was confronted with an error message. The net effect of the problem was that the last four days of incoming mail had disappeared from my in-box. This was, for me, a very big deal. In large part, my in-box is my "to do" list. As a consultant, my incoming e-mail is too important to ever allow a repeat of this problem.

Suffice it say, this made me think about backing up my e-mail perhaps more than most people.

The need for reliable and redundant e-mail backups dictates the use of a client side e-mail program such as Outlook Express, Thunderbird or Eudora. Web based e-mail systems such as Gmail, Yahoo mail and Hotmail, have their advantages but backup is not one of them.

To begin with, I have an external hard disk attached to my computer and every morning I copy all of my e-mail from the internal hard disk to the external one. This is a destructive backup. That is, every morning the backup is totally re-created on the external hard disk. The advantage of this is that I never have to worry about running out of space on the external hard disk. The disadvantage is that I can't use it to recover e-mail from three days ago. Everything is a trade-off when it comes to backups.

Also, this backup doesn't manipulate the original files in any way; they aren't combined, compressed or re-formatted. Thus, I can easily copy e-mail from the external hard disk back to my computer and use it immediately. And simple means there is less that can go wrong. The downside is that the backup is the same size as the original, but external hard disks have a huge capacity and transferring files over a USB2 connection is more than fast enough for this purpose.

One of my prime rules for backups is to never to copy a file while it's in use. That is, I never copy e-mail when my e-mail program is running and never copy Word documents when Word is running. The morning backup of my e-mail is scheduled by the Windows scheduler and since it runs first thing after Windows starts up my e-mail program is not running.

This however, is just a starting point as it still allows for the loss of an entire day's worth of e-mail. To cut my potential loss in half, I also backup my e-mail midday. This backup is also scheduled using the Windows scheduler, but it's very different from the morning backup. Rather than backing up all my e-mail, here I only copy the most important folders (the in-box and a few others). Also, the backup is sent via FTP to an online file storage company.

This limits my worst case scenario to the loss of a half day's worth of e-mail. It also means that no matter what happens to my computer and the external hard disk, I always have the most important e-mail stored a thousand miles away. And since my e-mail is sensitive, online storage space is limited and uploads are slow, I compress, encrypt and password protect the e-mail before it leaves my computer and travels over the Internet to the file storage company.

The midday backup is different in other ways too. For one, all the e-mail is combined into a single file. In addition, I keep multiple copies of the midday backup. The backup program tags the daily file with the current day of the week. Thus every backup made on a Monday will result in the same file name. When the backup is sent offsite, the backup program is instructed to delete older versions of files with the same names. I end up with seven off-site copies of my most important folders and, again, don't have to worry about running out of space.

Finally, once a month I compress and encrypt all my e-mail and send it off-site to another file storage company.

No one approach is right for everyone. For example, I have chosen to limit my worst-case loss to a half day of e-mail, which may not work for you. And my approach requires constantly filing e-mail in folders, something not everyone wants to do.

After living with the above scheme for a while, I modified it a bit to prevent the most important folders from growing in size forever.

I manually archive the in-box, sent folder and a few other important folders by moving old messages to new folders tagged with the year. For example, all the messages in my in-box from 2005 are stored in a folder called inbox2005. Likewise there are folders called inbox2004, inbox2006 and inbox2007. A couple months ago I moved messages in my in-box from January through March of this year into the inbox2007 folder. Later this year, I'll again move old messages from this year into it.

With this approach, I can eventually delete the inbox2004 and inbox 2005 folders from my computer. They remain on the external hard disk and are also stored off-site if need be. Without some type of archiving scheme, e-mail will grow forever. I find that manipulating a few folders this way a couple times a year is well worth the effort.

Of course, you can't use this approach, or anything remotely similar, unless your e-mail program stores each folder as a separate file (or two). But who would use an e-mail program that stored all your mail in a single file? :-)

[grakn]

One option to consider...

Instead of the mid-day backup, you could set your POP or Exchange (or whatever you use) hold on to each email for 1 day before purging. If your email goes down, you can then retrieve those emails in between backups.

[ssegal123]

My Email Solution

Mike, just like you I rely on email for my living... not just my life. No email, no money.

I am not about to start using Gmail for my day to day email duty. I need multiple addresses from multiple domains.

Here is how I manage my email.

My domains are hosted with a local ISP. Every email that comes in is automatically forwarded to my blackberry, and to a backup Gmail account. I have over 20k messages in my Gmail account, and am at about 75% capacity. I then check my mail on my notebook and desktop with Thunderbird.

This works great for me because, if I am away from my computers, I can always find the email by just logging in to Gmail and searching. Same goes for the Blackberry.

I run a 0-Message Inbox policy on my Blackberry, and delete things as they come in. If I need a phone number or address from an email I use the Gmail for Blackberry client and just search for it.

On the desktop side I have Thunderbird set to download my messages. I then backup my settings and messages to a 500GB local USB disk and also to an online backup account. I am using BackupRight, $9.99 I backup like 40GB, its insurance, but cheap and they won't deny your claims! ;)

[acahyadi]

sorry ... wrong place ... my bad

im reading few articles in same time ... sorry wrong place

[dbjohnson2]

Gmail and Thumbdrives

Michael,

Thank you for sharing your email backup strategy with us.

I have a comment and a couple of questions.

Provided one uses Gmail for their SMTP server and one fetches all email to the Gmail site and uses POP3 to download this email to the client on his PC, then a copy of all incoming and outgoing email (even that composed and sent from the client) resides on the Gmail server. So the backup pretty much occurs automatically. Plus you automatically get the benefit of the anti-SPAM filters employed by Gmail. What are the drawbacks to this approach?

In a separate conversation Michael suggested I use a thumbdrive to eliminate email synchronization issues between PCs I encounter when traveling between residences. Could someone elaborate on exactly how to do this. Would the email client (Thunderbird) and the email directory be set up on the thumbdrive. Could the hard disk on the PC then be used as the backup?

Doug

ps - could anyone suggest a good step-by-step tutorial on setting up a thumbdrive to run applications. Is a launcher required? Why?

[mhinnewyork]

Portable Thunderbird post coming soon

To dbjohnson2: I'll soon make a posting about the portable version of Thunderbird.

[MichieS]

One question....

Michael,

Thanks for your series of articles.

I, too, love TBird and use it. BTW, I just backup emails by copying the whole \Profile folder to CD (every so often, but I have no need for daily backups, like you do).

BUT your comment:
>>But who would use an e-mail program that stored all your mail in a single file? :-) <<

But Thunderbird *does* store all emails in a single file!; i.e., all emails in the Inbox are linked together in one (huge) file. Or, I maybe don't understand something.

OH, you can read one message's text body by searching and viewing with a text editor (I use Metapad). But you need Thunderbird if you want to restore the whole Inbox (somehow a backup copy to let TB get at it).

[dotcomguy00]

Better solution: IMAP and a simple procmail recipe

Backup and archival of e-mail is always of critical importance. I never rely on a local computer, storage, or most importantly a proprietary e-mail file storage format to maintain my e-mail archive.

Instead, I use IMAP and keep an easily accessible and easily backed up archive server-side. With IMAP I can access my current e-mail as well as any archived mail from any compatible e-mail client anywhere. If my flash drive goes bust, I haven't lost anything.

Rather than manually moving and archiving e-mails, I use a simple procmail or maildrop recipe which copies any incoming message to a folder such as "Archive/2007/09". I can delete the e-mails from my inbox after acting on them knowing that each and every one is archived.

And backing up is as simple as making sure the Maildir is backed up. Since the e-mails are in standard plain text, they can be compressed very easily by any back up software (BackupPC in my case) Another great advantage is that since the e-mails are in Maildir format, they are portable and not locked into a mail client's file format.

[airspeed13]

Great post dot com guy. Since I am not as savy as you, could you please explain what a procmail or maildrop is that copies all you incoming messages to a archive folder??
I assume Maildir, is a mail directory. I use outlook express and my email directory doesnot provide email in plain text. Your method is exactly what I would like to do (except for the imap), is there anyway anyone could explain this to me.

thanks so much airspeed

thanks so much