Simple off-site backup for photography
I’ve recently had a 3-month-old 1tb hard drive fail on me. Fortunately it was a backup drive and I lost no important files, but all my Time Machine backups for the last 10 months were gone. This prompted me to rethink the backup strategies for my work.
My previous backup plan consisted of:
A) External hard drive constantly in sync with Time Machine.
B) DVD copies of raw images, which I was slow to update due to the hassle of copying my ever-growing files to DVDs, a small media by current standards. For example, I can fit only about 330 14bit Nikon D300 NEF files on a single disc. Those DVDs were stored in the same room as my main computer and external backup drive.
Obviously, there are lots of failure points in that strategy:
- Time required to burn lots of RAW files to DVDs. Backups should be effortless, or else you’re not going to update them regularly.
- The only files with double backups were those on DVD. All my processed images, catalog files, e-mails, design work, and other files relied only on the external drive.
- The fact that both backups were in the same place.
The first thing to avoid in a backup strategy is having a single point of failure. Storing all my copies in the same room was a big shortcoming. An electrical problem could have busted my internal HD and external backup. A fire in the office or natural catastrophe could have destroyed everything. Or a common theft could have left me with no data.
Clearly I needed a better backup scheme—something easier to update and with off-site backups.
The first idea was to back up online. There are plenty of services—like Mozy, Amazon S3, etc.—but they’re less suited for big files because most internet connections have limited upload speeds. My ADSL connection, for example, has a 10mb download speed and just a 1mb upload. At this pace, I could upload only about 324mb per hour, and a 500gb backup would take months to complete. An alternative idea was to place a server in a relative’s house and back up online there, but it would have the same upload time problem and I’d have to rely on their less than optimal internet connection and mess with their router configuration for the incoming connection.
I ended up opting for a simple off-site backup plan by cloning the main hard drive to another one and depositing that copy at a friend’s house. This copy is refreshed every two weeks or after important jobs. Smaller files—like catalogs—are copied daily to a backup server by SFTP. There’s a program called ExpanDrive that mounts remote SFTP servers as transparent local drives, making the access very easy.
Current backup scheme
- External hard drive constantly synced by Time Machine.
- Online copies of important smaller files that change frequently.
- Off-site external drive that has its data refreshed every two weeks or after important jobs.
External hard drives
It’s safer to have a backup copy on an external enclosure than inside your computer case. Those enclosures have separate power supplies and can be turned off when not in use, prolonging the drive mechanism life and protecting it from electrical problems. They can also be stored away from the computer when not in use, protecting them from physical damage.
Some RAID modes, like RAID-1, offer constantly mirrored data between two drives. If one of them fails, it can be replaced and the array is automatically rebuilt without data loss. The problem here is that if the RAID controller suffers a failure you might end up with a corrupted and unrecoverable array. Write speeds on RAID-1 arrays are also slower than single drives and it doesn’t protect you from a software problem, accidental deletion, or virus attack. RAID is just an additional layer of security and cannot be considered an alternative to backing up data.
NAS—Network Attached Storage
These boxes work like external enclosures, but are more robust, use built-in operating systems, and have management options. NAS boxes can be accessed from any computer on the same network or even remotely, depending on the software and configuration, so they are handy for multiple users’ backup. The main drawback is that they use specialized operating systems and, in the case of a controller failure, that data might not be directly readable on a standard computer. Also, most of the less expensive NAS boxes are three to four times slower than a directly connected drive.
Amazon offers an online storage service. All files are encrypted and replicated among a cloud of servers, making it a very secure and reliable storage medium. Storing 100gb costs US$15 per month and you can use their online calculator to plan your storage spending in advance. Online backups are viable only if you don’t work on big files or have a fast internet upload speed. Long term cost is higher than maintaining an off-site hard drive or drive array.
DIY SFTP server
Don’t fall into the temptation of using an all-you-can-eat cheap hosting plan as your backup space. Sysadmins at the data center could access your precious unencrypted files, and most of those services forbid their use as a backup only server. Sign up for a simple VPS server with lots of storage space or use a dedicated storage service like Amazon S3. I’ve opted for the SFTP option because I already had the server available.
Built in on the Mac OS Leopard. Maintains an incremental backup and also protects you from accidental file damage by keeping all file versions on the backup. Innovative interface and effortless usage.
SuperDuper or Carbon Copy Cloner—Mac
Simple hard drive cloning. Supports incremental backups. I own a copy of SuperDuper and use it frequently, but Carbon Copy Cloner is free and that’s a hard price to beat.
Windows command line copy utility. Can be used on a batch process to sync an internal hard drive to an external copy.
Free download from Microsoft. Easy-to-use graphical interface. Synchronizes multiple folders or whole hard drives.
ExpanDrive—Mac and Windows
Mounts remote SFTP servers as local drives.
One of the best file transfer clients for the Mac. Supports SFTP, Amazon S3, and most common file transfer protocols.
Since writing this article I’ve become frustrated with the hassle of picking up my off-site hard drive at my friend’s house, updating, and returning it. It works, but takes too much time, although having this excuse for visiting my friend and chatting a little bit is nice.
In the search for an automated and brainless solution I’ve decided to sign up for Backblaze backup. It’s been three months already and it has worked very well so far. Even with my 1mb upload speed, I managed to complete my initial 500gb backup in about 60 days. I’ve excluded from this backup any unessential files to keep the size manageable. The only drawback is that my electricity bill has doubled during the initial backup while I’ve kept my MacPro on constantly. I’ll write more about it in a separate article.
Update 2: CrashPlan looks great
If you're a looking for a great online backup alternative, make sure you check out CrashPlan. They offer unlimited online backup for a very competitive price - on par with Backblaze, Mozy and Carbonite - with some unique features, like the ability to backup your files on other computers running CrashPlan for free. This is a much easier route than deploying your own server or messing with SFTP and rsync.
Thank you, DaveB (@davidbbitton), for tip.