Simple off-site backup for photography

Hard drives fail. It’s not a matter of IF, but a matter of WHEN. Learn how to develop a bulletproof backup plan and be prepared for this moment.
Published by Fábio Pili on September 15th, 2010. Last updated on February 14th, 2013.

I’ve recently had a 3-month-old 1tb hard drive fail on me. Fortunately it was a backup drive and I lost no important files, but all my Time Machine backups for the last 10 months were gone. This prompted me to rethink the backup strategies for my work.

My previous backup plan consisted of:

A) External hard drive constantly in sync with Time Machine.

B) DVD copies of raw images, which I was slow to update due to the hassle of copying my ever-growing files to DVDs, a small media by current standards. For example, I can fit only about 330 14bit Nikon D300 NEF files on a single disc. Those DVDs were stored in the same room as my main computer and external backup drive.

Obviously, there are lots of failure points in that strategy:

  1. Time required to burn lots of RAW files to DVDs. Backups should be effortless, or else you’re not going to update them regularly.
  2. The only files with double backups were those on DVD. All my processed images, catalog files, e-mails, design work, and other files relied only on the external drive.
  3. The fact that both backups were in the same place.

The first thing to avoid in a backup strategy is having a single point of failure. Storing all my copies in the same room was a big shortcoming. An electrical problem could have busted my internal HD and external backup. A fire in the office or natural catastrophe could have destroyed everything. Or a common theft could have left me with no data.

Clearly I needed a better backup scheme—something easier to update and with off-site backups.

The first idea was to back up online. There are plenty of services—like Mozy, Amazon S3, etc.—but they’re less suited for big files because most internet connections have limited upload speeds. My ADSL connection, for example, has a 10mb download speed and just a 1mb upload. At this pace, I could upload only about 324mb per hour, and a 500gb backup would take months to complete. An alternative idea was to place a server in a relative’s house and back up online there, but it would have the same upload time problem and I’d have to rely on their less than optimal internet connection and mess with their router configuration for the incoming connection.

I ended up opting for a simple off-site backup plan by cloning the main hard drive to another one and depositing that copy at a friend’s house. This copy is refreshed every two weeks or after important jobs. Smaller files—like catalogs—are copied daily to a backup server by SFTP. There’s a program called ExpanDrive that mounts remote SFTP servers as transparent local drives, making the access very easy.

Current backup scheme

  • External hard drive constantly synced by Time Machine.
  • Online copies of important smaller files that change frequently.
  • Off-site external drive that has its data refreshed every two weeks or after important jobs.

Backup options

External hard drives

It’s safer to have a backup copy on an external enclosure than inside your computer case. Those enclosures have separate power supplies and can be turned off when not in use, prolonging the drive mechanism life and protecting it from electrical problems. They can also be stored away from the computer when not in use, protecting them from physical damage.

RAID drives

Some RAID modes, like RAID-1, offer constantly mirrored data between two drives. If one of them fails, it can be replaced and the array is automatically rebuilt without data loss. The problem here is that if the RAID controller suffers a failure you might end up with a corrupted and unrecoverable array. Write speeds on RAID-1 arrays are also slower than single drives and it doesn’t protect you from a software problem, accidental deletion, or virus attack. RAID is just an additional layer of security and cannot be considered an alternative to backing up data.

NAS—Network Attached Storage

These boxes work like external enclosures, but are more robust, use built-in operating systems, and have management options. NAS boxes can be accessed from any computer on the same network or even remotely, depending on the software and configuration, so they are handy for multiple users’ backup. The main drawback is that they use specialized operating systems and, in the case of a controller failure, that data might not be directly readable on a standard computer. Also, most of the less expensive NAS boxes are three to four times slower than a directly connected drive.

Amazon S3

Amazon offers an online storage service. All files are encrypted and replicated among a cloud of servers, making it a very secure and reliable storage medium. Storing 100gb costs US$15 per month and you can use their online calculator to plan your storage spending in advance. Online backups are viable only if you don’t work on big files or have a fast internet upload speed. Long term cost is higher than maintaining an off-site hard drive or drive array.

DIY SFTP server

Don’t fall into the temptation of using an all-you-can-eat cheap hosting plan as your backup space. Sysadmins at the data center could access your precious unencrypted files, and most of those services forbid their use as a backup only server. Sign up for a simple VPS server with lots of storage space or use a dedicated storage service like Amazon S3. I’ve opted for the SFTP option because I already had the server available.

Useful software

Time Machine

Built in on the Mac OS Leopard. Maintains an incremental backup and also protects you from accidental file damage by keeping all file versions on the backup. Innovative interface and effortless usage.

SuperDuper or Carbon Copy Cloner—Mac

Simple hard drive cloning. Supports incremental backups. I own a copy of SuperDuper and use it frequently, but Carbon Copy Cloner is free and that’s a hard price to beat.

XXCopy—Windows

Windows command line copy utility. Can be used on a batch process to sync an internal hard drive to an external copy.

Microsoft SyncToy—Windows

Free download from Microsoft. Easy-to-use graphical interface. Synchronizes multiple folders or whole hard drives.

ExpanDrive—Mac and Windows

Mounts remote SFTP servers as local drives.

Transmit—Mac

One of the best file transfer clients for the Mac. Supports SFTP, Amazon S3, and most common file transfer protocols.

Update

Since writing this article I’ve become frustrated with the hassle of picking up my off-site hard drive at my friend’s house, updating, and returning it. It works, but takes too much time, although having this excuse for visiting my friend and chatting a little bit is nice.

In the search for an automated and brainless solution I’ve decided to sign up for Backblaze backup. It’s been three months already and it has worked very well so far. Even with my 1mb upload speed, I managed to complete my initial 500gb backup in about 60 days. I’ve excluded from this backup any unessential files to keep the size manageable. The only drawback is that my electricity bill has doubled during the initial backup while I’ve kept my MacPro on constantly. I’ll write more about it in a separate article.

Update 2: CrashPlan looks great

If you're a looking for a great online backup alternative, make sure you check out CrashPlan. They offer unlimited online backup for a very competitive price - on par with Backblaze, Mozy and Carbonite - with some unique features, like the ability to backup your files on other computers running CrashPlan for free. This is a much easier route than deploying your own server or messing with SFTP and rsync.

Thank you, DaveB (@davidbbitton), for tip.

Comments

DaveB
Sep 19th, 2010 - 18h56

You totally missed CrashPlan. It rocks. For your initial backup, they'll send you a drive that you fill and they'll seed their servers with your data. Beats waiting days to move many gigs of data. Also, you can backup to a friend's computer anywhere on the Internet as long as they are running CrashPlan too, or simply another machine in house/office. To just backup locally or with a friend, it's free. To use CrashPlan's server, it's dirt cheap and they don't ding you on space (read: unlimited). There is a $59 version of the app that adds features Check them out at www.crashplan.com. Also, they are awesome at answering questions via Twitter at @crashplan.

Reply
DaveB
Sep 19th, 2010 - 18h58

BTW you can find me on Twitter at @davidbbitton

Reply
Fernando
Sep 21st, 2010 - 13h03

How about www.dropbox.com?

I've been using their free version and am amazed at how easy it is to work with it (specially with the iPhone app). I'm seriously thinking of paying to get the full back up features and regularly dump my photo library there.

Reply
Fábio
Sep 21st, 2010 - 14h23

Fernando, Dropbox is more of a file sharing solution than a real backup tool, even if they also promote it this way. The main limitation is that their biggest plan is only 100gb and costs US$19.99 / month. Other problem is that you'd have to manually copy your files to the Dropbox folder. My experience is that backups should be automatic, or else we might end up forgetting to update them.

I'd recommend you to try Backblaze or CrashPlan, both around US$ 5 per month for unlimited automatic backups.

Reply
hi
Mar 7th, 2011 - 21h43

Hi, your points are valid about dropbox, but you can also create a symbolic link to your "home" folder and just place that in the dropbox and it will automatically back everything up and it stores revisions of all files.

Reply
Amos
Nov 30th, 2010 - 14h45

I use Arq for Mac Backups to Amazon S3 Servers and now a Synology NAS Box. You should try Arq, its only 25$ and you will get full upload speed instead of only some KB with Crashplan.

Reply
Fábio
Dec 5th, 2010 - 09h36

Thank you for your comment, Amos. Arq for Mac seems to be a great piece of software. My main concern on this approach is the fixed cost of Amazon S3 storage. It's arguably the most reliable online storage available, but the monthly storage costs make it much more expensive than providers that use fixed prices for unlimited storage, like Backblaze.

Reply
Raphael M
Jul 7th, 2011 - 14h57

Since I read this article a few months ago, I realized that I really didn't have a good back up plan other than an external hard drive sitting behind my computer.

I've been using CrashPlan (http://www.crashplan.com/) ever since and I'm pretty happy so far (well, my computer hasn't crashed yet... knock on wood). If you decide to go for Crash Plan and you are in the U.S. for sure take on their offer to send you an external hard drive and courier it back to them, the upload via internet took good 2 months for over 1 TB (@ 2.5 Mbps upload connection). After that, the backups have been quite seamless and you can setup an email report/ warning to make sure all your computers are being backed up regularly. The support answers are quick (and polite) as well.

Reply
Karo
Nov 3rd, 2012 - 09h36

Thanks for all the infos!

I'm wondering which CrashPlan you guys use - the crashplan+ which is officially only for personal datas or do you use a business-account - crashplan Pro?

Reply
Fabio
Nov 3rd, 2012 - 12h13

Karo, Crashplan Pro is geared towards business that need user management and the possibility to backup several computers under the same account. Other than that, it has the same underlying backup system as Crashplan+. You can find a comparison between the two versions here.

Reply

Leave a Comment

Name

E-mail

Comment

Submit comment