Backing up your digital world
April 9th, 2009. Filed under: Technical posts.
I’m sure I’ve blogged about this somewhere else, but it’s probably time to revisit it.
I was catching up with a friend earlier this week, who does most of his photography off a laptop, and the conversation turned to the vexing question of backing up the Lightroom catalogue. He asked me how I did it, and when he realised my approach was somewhat more complex than his (it seems simple to me!), it struck me I should talk about it. So here is how I do it.
Firstly I don’t back up to DVD. I’ve given up on that, and here’s why. I run around 40 GB of memory cards, and I’ve known days where I have used the lot. So call me trigger-happy! I’ll admit it; some days I see things that excite me so much I want to shoot them, and often there are so many potential possibilities I shoot them all. Often I will shoot round the subject, varying my composition subtly. I can always throw it away later. So backing up to DVD, each with a capacity of around 3 1/2 GB is just not an option, unless I win the lottery, when I can afford to buy a warehouse to store them! Filing them for later retrieval is going to be a nightmare, especially given the read/write speeds of most DVD players. Blu-Ray is an option, but the price of the drives and disks makes it frankly uneconomical. Bang for your buck, hard drives deliver faster access, speed and more capacity. At least that’s the way I see it.
I use Adobe Lightroom as my primary cataloguing software. It has got so damn good (for the money it’s amazing value) that I pretty much don’t bother with anything hours. Don’t get me wrong; I’ve tried the opposition, but frankly at this point they don’t measure up. And I include Adobe Bridge. Essentially it’s a file browser with a hotline to PhotoShop. Lightroom on the other hand is a database with processing stuff built on top. At the front-end, the downloading end, it makes downloading so easy that I’m happy to stay with it. What’s more it keeps getting better. So here is my process.
When I download, I use the import dialogue box to rename, convert to DNG, keyword and into my IPTC data. At the same time I set it to duplicate files to an external hard drive. Since I do it on my desktop, the backup drive uses an ESATA connection rather than USB 2.0. ESATA moves the data at around 7 to 8 times the speed of USB 2 (note that USB 3 is not far away). When the backup drive gets full, I label it carefully and put it away in a safe. I put a new one on the line and take up with the old one left off.
When each of the hard drives and my desktop gets full, I carefully copy the data onto a new larger hard drive, then label and file the old one in my safe. As a matter of interest, the safe I have is a class four gun safe, suitable from the New Zealand police point of view, I’m told, for storing my .44 Magnum, RPG and AK-47, if I had one! A steel filing cabinet would be good, too. From time to time I take the hard drives and file them in a fireproof safe I swapped for a darkroom years ago, which lives off site. Recreating my computer in the event of a major meltdown is going to be a major odyssey, but at least it’s achievable.
My Lightroom catalogue is on a second ESATA external drive, which also stores my master files, the ones I will use for a future exhibition. You might ask why not leave it on the scratch drive which holds the rest of my applications. With the number of files I have (> than 120,000), I have created a number of separate catalogues, to hasten start-up and probably to enable me to pigeonhole what I do. Even though the engineers at Adobe maintain that Lightroom catalogues can reach up to 100,000 files, my experience suggests that they get clunky over 30,000. As a side note it’s worth mentioning that from time to time, optimising the catalogue (go to catalogue settings) will help speed things up. Whichever way you look at it however, Lightroom catalogues can get very large. You will begin to notice this when Lightroom takes ages to load up and/or has wee hissy fits when you move around within it. That may be the time to start splitting out your catalogue into two or more. Apologies for the long side note!
You might ask why I still my backups on a separate external hard drive. Two reasons really. Firstly, every time Lightroom creates a backup of your catalogue(s), it creates a full backup. If your catalogue is sitting on, say, 10 GB, this means your new backup is 10 GB as well. Run three or four large catalogues, backup, say, weekly basis (I do), and you can imagine that it won’t be long before your main drive is full. Secondly, if your main drive goes down, and you lose all your data, there is little point in having both your main catalogue and its backup on the same drive. Keeping it on an external drive means that you keep plenty of space free on your main drive. As an aside, (here I go again) it’s worth noting that you should have between 30 and 50% of your main hard drive space free to get optimum performance from your machine.
Matt Kloskowski over at Photoshop® Lightroom® KILLER TIPS has his own take on backing up your catalogue. He reckons you shouldn’t do it at all. Well having read your article Matt, I might agree with much of it, but I don’t agree with that. One of the people who commented on his post had this to say:
Here is a little story that happened to me a couple of month ago. While I was doing tasks in Lightroom, my Mac suddenly crashed (no idea why) (I can think of some reasons-ed). Anyway, after I restarted the computed I started Lightroom again and Lightroom was saying: “Unable to open a corrupt catalog.” Fortunately, I had just done a backup of the catalog file. So it may be useful even if it is on the same hard drive.
I’m with you, philou, whoever you are.
Whatever method you use, what is important is that you do consider the whole question of backing up your digital world … and do something about it.
April 9th, 2009 at 6:47 pm
Sez Tony:
“Lightroom on the other hand is a database with processing stuff built on top.”
When the database runs on a server with multi-user access, that may come true. Until then, I find it hard to see Lightroom as much more than Camera Raw with a cut-down Bridge bolted on. I try out new releases when they appear, and wonder why I bothered. Bridge, of course, is useless for the same reasons, plus it seems to need more resources to run than Photoshop itself.
We use ThumbsPlus as a front end to Photoshop, with MySQL on the fileserver, and 178,000 records in the database is not a big load for it. Also, the database and the image files are completely separate, with all metadata in the files as IPTC, or ACR settings in DNGs, so a total database loss would be only a matter of rescanning the disks. The only risk is losing data from offline CDs and DVDs, and the database is backed up regularly – it’s only 1.74GB. Backing up raw and finished images is a different matter, and I agree with your methods there.
Colin
April 9th, 2009 at 7:48 pm
Hi Colin:
Many thanks for your input. I guess I must have a blind spot in this area. I had forgotten all these options ( iMatch, Portfolio, etc), but to be honest, LR works for me ( imperfect as it is) because of its integration with the Adobe workflow. The others hang on as best they can.
Let’s be honest. CSx is still the 800lb gorilla of editing apps. Nothing else comes close. So while I may bitch about paying the (to my mind) excessive PhotoShop tax, it is still the best game in town. So I have to play along with Adobe. And Whatever I may think of LR’s cataloguing, it still enables me to move fast from download to CS4 and back.
Remember too, that very few of my readers work in a server environment (although I am about to move to one, so your comments are appreciated). A single-machine user may not find your structure that useful.
If you feel otherwise, let me know your thoughts, and I will add them to the original article, so they are more visible.
better still, write an article, and I will post it. I love guest bloggers!!!!
April 9th, 2009 at 10:56 pm
Well, I’m with Tony. Of course we can all see the value and simplicity of a program like ThumbsPlus, but anything not ported to the Mac misses a fair percentage of worlds creative output and we don’t even realise it’s capabilities.
The Adobe suite is both a blessing in it’s integration and rapid development (albeit with a few hiccups along the way) and the achilles heel of our workflow with it’s massive resource demands. I’m not interested in replacing hardware every 2 years just to keep up with the demands of poorly designed, energy hungry software.
I’ve trailed all the cataloguing systems for my platform and it seems Lightroom is currently the most painless system that integrates file handling and RAW developing into a workable system.
I use a similar file handling system to Tony apparently with multiple SATA hard drives and offsite backups, I use a rack system for hard drive swapping and storage is very inexpensive, currently at around 25 – 50c per GB. There really are no excuses these days for not having a decent backup regime. You can have an off site emergency system restore disk and should be able to be back in action within hours.
I don’t think Lightroom is useful for file ingesting, I use Photo Mechanic for ingestion and metadata and import into Lightroom after first edit.
I consider files out of Lightroom processing too ‘Rough’ and will almost always put files through a Photoshop batch edit to finish.
That said, Lightroom has massively reduced my time in image handling with it’s intuitive file handling, pre sets and workflow integration. Now Adobe need to reduce computer resource demands to that of Photo Mechanic and life would be sweet in the workshop.
Have a great Easter guys.
Andrew
April 9th, 2009 at 11:10 pm
ASP:
couldn’t agree more. Like you, I find files ex LR rough, especially the sharpening,so I outsource to CSx for finishing and final sharpening ( I use zero LR sharpening ( too crude for me).
Interested in QM. Maybe the next time I a passing, you can take me through your workflow…
have great Easter ourself in beautiful Kaikoura.
April 10th, 2009 at 1:26 am
This last bit ‘Whatever method you use, what is important is that you do consider the whole question of backing up your digital world … and do something about it.’ is the key thing that most people aren’t doing.
The rest is gravy after that.
Certainly having off-site backups is key. The main problem with using hard drives is their likelihood of failure increases dramatically the longer they are stored without spinning up. Ideally you need to power them up at least once every 6 months to stop them sticking and failing.
It is a somewhat depressingly complex situation, with no actual, good, painless and affordable solutions out there. Tapes get chewed. CDs/DVDs have terrible shelflife, even when stored carefully. Hard drives will always fail – a matter of when, not if. RAID systems suffer lightning strike or water damage. Off site backups get bit rot.
The main solution is multiple copies, of everything.
Gordon McGregors last blog post..infinite monkeys
April 11th, 2009 at 7:34 pm
I love love love Lightroom.
Admittedly, I’m a P-Shop dolt but, P-shop or not, there is no program I am aware of that allows me to put one image in multiple places wearing multiple disguises without increasing the load on my storage capacity. One of the most frustrating things I encountered pre-digital was trying to place one chrome in two or three stacks at the same time. Lightroom says “sure, why not?” – a virtual genie in a bottle.
Regarding backup of LR catalogs – don’t even think of not doing it!!! I’ve had the “corrupt catalogue” message more than once and the BU was the only thing that saved my day (actually week) to rebuild. Why would you not bother? It’s effortless and a small drop in the pond (ocean?) of memory on your system. If you’re really worried about storage – dump the old backups.
Back to P-shop: I do actually venture there on occasion, and I find that after diddling around in megabyte-land I can open the file in LR and make tiny, simple, lovely, easy adjustments to finish the work for me. It’s a weird concept to you CS boys, but it works to shuttle back and forth between programs to utilize the best of each, and it’s so easy even I can do it.
April 17th, 2009 at 8:59 am
Tony, I’m kind of doing the same in terms f lightroom as the database, it really is fantastic, a word of caution to people using the metadata there though, stick to the real stuff, i.e. the IPTC some of the early stuff isn’t really meta & doesnt trnsfer to other programs.
In terms of backups, another option exists if your happy to use other backup software. you can use any std back up software to back up all the files lightroom does but do incremental backups, i.e. only the new files & those that have changed. This can be scheduled as well so you don’t even have to remember!
April 19th, 2009 at 7:42 pm
As John says, there is a bewildering array of third-party backup software out there at equally bewildering prices that you can use.
I do not make my living from photography, it is merely a hobby, but I have had success with a neat little program called “NTI Shadow”. This is very reasonably priced & can be set up to either copy your files as you create or re-save them immediately (hence the name shadow) or, as I use it, as a conventional back-up software.
It does have a couple of nifty features that not all backup software has, you can choose to keep multiple versions of files as you change them, and after the initial save it only copies new or altered files creating in effect incremental backups. However, it automatically adds them to the original file(s) so you end up with a “Mirror” or “Shadow” copy of your selected files/folders, not an increasing collection of original & incremental backups, this means you only have to look in one place to retrieve any lost files & not multiple backups.
I have mine set up to copy all my documents & images (& a few other directories) to a second hard drive once every day, then copy from that to a USB drive once a week. This suits me fine, but may not satisfy heavy duty commercial users. ( NFI )
Just my $0.02 worth
April 25th, 2009 at 7:32 am
There are actually a couple of free(a Scotman’s Fav word!) pluing’s for lightroom that will do some simple eporting / archiving for you as well, they aren’t fast but for the casual user they do a good job. NIT Shadow looks pretty impressive as well.