A guide to keeping your digital photos


If you use digital cameras and haven't yet managed to lose some photographs, don't wait. It happens to everyone eventually. But with a bit of foresight and planning, digital storage should be both more convenient than, and at least as reliable as, conventional storage of film material. However there's quite a lot to think about and get right.

One big thing to consider is that the standard backup routines that people use for written data are not really so suitable for photographers. These routines are for backing up large numbers of small files, that might be altered every day, to hard drives that remain attached. Photographers have much larger sizes of files that usually need to be archived in a pristine state. You should of course still use a back up routine such as Retrospect, SuperDuper or Silverkeeper for your data files.


FLASH CARD USE

Do I need better flash cards? This is a relatively minor consideration - it's rare (apart from Microdrives) that a card will fail completely; somewhat more common that you will lose one or two pictures for no obvious reason; much more likely that you will accidentally forget which one has your pictures on; or take it out of the camera or card reader too quickly and lose pictures that way. So there's some advantage in using a faster card if you're absent-minded or in a hurry. Rob Galbraith's website has an excellent guide to Compact Flash and Secure Digital cards (the CF Database).

A few big cards or lots of small ones? Some photographers use a lot of small cards so that if one goes, something can be salvaged from the rest. If you're well ordered and efficient, having lots of smaller cards gives a theoretical advantage. Others, perhaps those working in difficult or fast-moving environments where changing a card is not straightforward, prefer to work with a few big ones.

Anything else about cards? Make sure they all look different! It's all too easy to put a full one back in a camera and format it before you realise. Have a system so you can tell that a card is full just by looking at it - for instance put the full ones in a card wallet back to front; the empty ones are the right way round. Formatting in the camera is by far the most reliable way of clearing a card. Removing them prematurely from the camera or the card reader almost always causes some corruption. And if you put them through a washing machine (many have, accidentally left in the wrong pocket), you probably won't get your images off.

What if I accidentally delete images from a card? Don't take any more photos using that card. Normal "deletion" (and quick formatting) won't actually delete the pictures; they are simply marked for over-writing. Any pictures that haven't been over-written can nearly always be retrieved with special software - but this can take a while to work, so if you're in a rush, stop using the card and come back to it when you've got time. Photorescue is one of the best recovery applications.


TRANSFERRING IMAGES TO COMPUTER

What's the safest way to transfer files to a computer? Most losses are caused by operator error rather than equipment failure. The most important thing is to be consistent. Develop a system so that you always do the same thing, in the same order, with new pictures. Regarding equipment, it's usually faster and sometimes safer to use a card reader than to connect the camera direct; and many people prefer to transfer the files manually rather than let some software do it automatically. Very occasionally the card driver can malfunction and copy a folder of images with some missing. Check that the folder on your desktop is the same size as the one on the flash card. If you're using software to transfer images, test it thoroughly to find out how it works and whether it's reliable on your system. The more advanced software such as Capture One is customisable and gives you a session report so you know what's happened.

What's a good system to develop? Many photographers first transfer their pictures to a volume or drive that's separate from the system disc. Have a foolproof way of naming or numbering the folders you put them in. It's then good practice to have some way of checking the pictures to see that they're all there. Check them in cataloguing software (see below). Look at a few of them full size to check for any problems. Some applications, for instance iView MediaPro, allow you to rebuild large thumbnails direct from the high-res image, and this is a very good way to tell that all the images are there.

It's useful to colour-code the job files: no colour - not archived; green - fully archived, and so on. That way it's possible to see what files are at risk in case of hard drive failure.

Once you have them on your disk, you might choose to renumber them (Try not to use names or numbers that are too long; for maximum compatibility use a standard filename system such as the "8.3" convention). Choose a system in which every photo you every take has a unique number; this avoids a situation later when someone orders a copy of IMG_0001, and you have dozens on your system called that. You can do this in cataloguing software (see below); some allow you to renumber the original files on your hard disc in various different ways. Some photographers prefer not to alter the original files; provided you trust your folder naming system, that's fine.

Then, straightaway, before you re-use the card, back up the files to another hard drive. Hard drives do fail, and it costs very little to have an extra copy. Do all your Photoshop (or other software) work on the second set, keeping the first set untouched. Be extra careful of the "Save" command in Photoshop - it overwrites all your changes and the original file is lost. Use "Save As..." instead. For more information the Prodig mailing list contains many well-informed discussions on workflow generally.


CATALOGUING

While it's beyond the scope of this article to go into this subject in depth, it's worth noting that if you can't find a photograph on your system, then it might as well not be there. Good cataloguing is a vital stage of preservation. This particularly applies if you have tens or hundreds of thousands of photos to store. Cataloguing (also known as Digital Asset Management) applications vary in speed and features; read reviews to find out which one is for you. Some examples are Photoshop's file browser or Bridge, Capture One, iView MediaPro, PhotoMechanic, Fotostation, Aperture, iPhoto, Cumulus or Portfolio.

But it's not usually sufficient to use one of them as your only method of cataloguing. This is because once a catalogue grows to more than a few hundred photos it's very difficult to scroll through to find the one you want. It is possible in certain cataloguing software to arrange your work in "sets" so you can view selected portions of a catalogue. But this depends on the software remaining current and useable. Better also to have some sort of text database, probably like the one you already use for cataloguing your films, to home in on a particular catalogue that might contain the photos from just one job.

A side issue, but remember to caption images you send out. This is done in the File Info area of a photograph; also known as the metadata, XMP or IPTC information. Again, some cataloguing software can do this with varying degrees of ease and automation, as can Photoshop. If you don't caption, it's a lot easier for people to steal your images.


ARCHIVING

So far it's mostly common sense. Now it gets more tricky. Stories abound of failures of every sort - unreadable files, inaccessible CDs, failed hard drives. It's an area where we're still finding things out; also a really good archiving system takes some time and effort. But there are some simple rules:
-- Use common file formats and technologies.
-- Archive in two or three different ways.
-- Pay attention to handling and storage.

What are the common file formats? JPEG for compressed files and TIFF for uncompressed. It's nearly certain that these formats will continue to be supported, and thus readable, for many years to come. It's more risky archiving camera raw formats - this is a secretive area, as camera manufacturers like to guard their own formulae, so third-party software that reads every kind of camera raw file is not that common. There are already over 100 different raw formats. Adobe is trying to push an "open" raw format, DNG, but at the time of writing this has not been widely adopted. If you want to know more or do something about this "raw problem" visit the Openraw website.

What are the common archiving technologies? Hard drive, CD and DVD. There are a couple of others, MO and tape, which are in more limited use. But it's much better to stick to the most common ones, as it's going to be a lot easier to find a way of reading them on your computer in decades time.

Why use two or three different methods? It's not unknown for hard drives to fail, and batches of CD-Rs to become unreadable, through a number of causes - poor manufacture, bad burning, poor storage. So if you're making two sets of optical backups, it's good practice to use different burners, CD or DVD makes and storage for each method.


HARD DRIVES

What are the issues with using hard drives? They're big, getting cheaper and the most important thing is that you can keep the files on your system, so you can check their integrity at any time. Internal hard drives from reliable manufacturers such as Seagate and Western Digital have long guarantees and high MTBF (mean time between failure) figures. However it's worth remembering that shock, dropping or bashing a hard drive, especially while it's in use, is the most common cause of drive failure. A technology on recent drives, SMART, means you can get advance notice of failure - it works in about 60 percent of failures. The downside to relying on internal hard drives is they are more likely to get stolen along with your computer; and of course most photographers will outgrow their capacity. See the Storage Review website for more info.

Individual external hard drives have more problems; they lead a rougher life, they tend to get hotter, and some well-known manufacturers put cheap drives into their fancy cases, or worse, put two hard drives in a Raid 0 configuration (so it appears as one large faster drive on your computer, but with a higher chance of total failure). On the other hand you can take them with you, and store or hide them. If you store them unused for a long time, check them every so often. If you're moving them between areas of very different temperature, allow them to acclimatise for a day before switching them on. Treat them as the delicate pieces of machinery that they are.

If you're faced with apparent failure of an external hard drive, don't panic - at least until you’ve tried a few things. If the drive appears to be running but you can't get to your files, you might just have "logical damage" or data corruption. There's lots of software to sort this out; first try the drive tools and utility software in your system; then you might consider trying some of the independent data rescue and drive recovery utilities. If the drive cuts out suddenly, it might be overheating; once you've let it cool down you can try running it with a fan pointing at it - it sounds daft, and it's not ideal long-term but it's helped the author get data off more than one dodgy drive. The port might have failed, so if you have a choice of connections (Firewire instead of USB for instance), try another one, or plug it into a friend's computer. The power supply on the drive could have failed - again this is a simple matter of trying a different one.

If none of these work, and the drive is out of warranty, you might consider opening the case, taking out the hard drive, and installing it in different enclosure. Although only recommended if you know what you are doing, this very often does the trick. External enclosures are widely available and cheap, and it's probably sensible to have one in your emergency toolkit. Make sure the one you choose supports your drive type (either IDE or SATA). If this also fails, and particularly if the drive is making scraping or other mechanical noises (other than the whirring and clicking that is normal), then your final option is professional data recovery - taking the drive to a specialist who has difficult and time-consuming ways of recovering your data. The cost of this is high, at least as much as the high-quality RAID enclosure you should've bought in the first place...

More photographers are starting to use RAID - a redundant array of independent discs often in purpose built cooled enclosures. Although they're not as easy to transport as external hard drives, the right configuration will give you a great boost in reliability. RAID comes in different levels; for increased reliability a basic RAID 1 setup simply mirrors two hard drives, so that if one fails, you still keep your data. You can control this with system software and any two attached drives. RAID 5 is a popular and more sophisticated solution as it allows one drive in an array to fail completely with no loss of data. With RAID 5 you effectively only give up the storage space of one drive, no matter how many you have in an array (up to a maximum or 24 drives, though 4 or 5 is a typical number). So four 250GB drives configured as RAID 5 will give you 750GB storage space (in practice a little less), and still allow one drive to fail.

There are two ways of attaching RAID, depending on whether you want speed or flexibility. The array can be wired directly to one computer using Firewire, USB, or external SATA. This is known as DAS - directly attached storage - and maximises the speed of operation, but of course you need the computer switched on to use the storage. Even cleverer is to put the RAID on your network (NAS - network attached storage), then you can access the storage from any computer on the network or even via the internet, and it can be kept somewhere safe away from your computer. The slight disadvantage of NAS is that the read and write speed is limited to the speed of your ethernet. The cleverest ready built RAID 5 NAS boxes such as ReadyNAS from Infrant also have other tricks, for instance emailing you if there's a problem. There's lots about RAID on the web; Google "Raid for photographers"; look through web forums. Other RAID 5 NAS boxes are available from Buffalo.

Why do I need another type of archiving? It's possible for several hard drives to fail or be corrupted at once, through lightning strike, software failure, human error or malice. CD or DVD back up provides a theoretically unalterable set of your images. They can also be stored off site.


OPTICAL MEDIA (CDs or DVDs)

CDs or DVDs? As photographers shoot more and more files the bother of CD archiving becomes apparent. A single shoot of several gigabytes has to be divided among several different CD-Rs. But in some ways CD-R provides a more reliable solution than DVD-R; with a larger sized laser track, there is more tolerance for the equipment reading or writing discs. Some consider the design of DVD-Rs more durable with the dye layer in the middle of the disc rather than near the top as with CDs. But the consensus of opinion at the moment is that CDs will last longer than DVDs.

Are some makes of CD or DVD better than others? Absolutely. But a totally reliable test has not been devised. A number of websites do their own tests. Brand name discs tend to be better than unbranded; some cheap discs have an imperfect dye layer; some CDs have a top layer (the label side) that can be scratched off easily (try it on unwanted discs with a pen or paperclip; you might be surprised). The type of dye makes a slight difference but different dyes are better for longevity and compatibility.

Any tips for burning? In theory the slower you burn the better the result. If you've got time, burn below the maximum speed of your writer. If you start to get regular rejects of different brands of media it might be time to replace your burner. They don't seem to last very long - a couple of years typically. A burner on the edge of its tolerance can produce CDs or DVDs that don't work in other readers.

What about storage and handling? This is vital. It's likely to be a far bigger factor in longevity than manufacture. Don't use labels; write on a disc with caution. Ideally store the discs vertically in their jewel cases, away from heat and light. Be careful of paper or plastic sleeves that make contact with the disc. Try not to touch the disc surface. Apparently discs are more susceptible to stress straight out of the burner, so handle carefully if they come out warm. Much more detail on this subject can be found in the report "Care and Handling of CDs and DVDs" by Fred Byers.

As mentioned before, try and keep a set off site. If you've got a friend nearby arrange to swap backup drives or discs. Check them every so often - take one out of storage and see if it's still readable. If you are considering a firesafe then make sure it is up to "data" standard. Firesafes were originally designed around the flashpoint of paper but CD, DVD and HD will melt at lower temperatures so require better safes.



CONCLUSION

Help! I wanna go back to film. Are you sure? There's only ever one original; if a client loses it, or you have a burglary or fire, it's gone for ever. Negs and trannies are delicate (see what happens if you put them in a damp cellar!). Have you forgotten how long good neg scanning takes?

I might as well delete the images I don't need... Please don't. You're bound to make mistakes, or need a photo you've deleted. If you're really short of space, certainly keep multiple copies of your favourite pictures, and at least one copy of everything.

Isn't this all overkill? So much time and effort! Well, the assumption is you're reading this because you're serious about storage. Transfer, cataloguing and archiving all take time. If it's commissioned work, it's something you can charge onto the client. They will be absolutely delighted you can lay your hands on an image five years later in just a few seconds. They'll certainly have lost it.

Are there any other methods of storage? You can print out your best pictures and store them archivally. Wilhelm Imaging Research, who investigate digital print lifespan, recommends this. With the best printers, materials and storage, lifespan extends into hundreds or thousands of years. Or you can put them onto the web. Enough people will copy them and circulate them that you can be pretty sure someone somewhere will have a copy... if you can find it. A less extreme version of this is some sort of internet-based back up or online hard drive storage; but your space and bandwidth is limited; at the kind of volumes photographers need, it's very expensive. Still, it might be worth sending your favourite images to a Gmail account.


Can you sum up good practice for preserving digital images?

-- Develop a foolproof system for copying your pictures off the camera, cataloguing and archiving.
-- Use common file formats and technologies.
-- Keep at least one copy on hard drive and at least one copy on optical media (CD-R or DVD-R). Then have a third copy on another hard drive or different optical medium. Keep one copy off-site.
-- Handle and store your discs and hard drives with care.


Further reading:

General advice on digital imaging:
http://www.robgalbraith.com
http://www.prodig.org/

Optical (CD/DVD):
http://www.cdrfaq.org/
http://www.clir.org/pubs/reports/pub121/contents.html
http://www.mscience.com/

Hard Drives/RAID:
http://www.storagereview.com/
http://en.wikipedia.org/wiki/Redundant_array_of_independent_disks
http://www.naturescapes.net/032004/jp0304.htm

Rescue/Recovery:
Photorescue: http://www.datarescue.com/photorescue/

Asset Management:
The DAM Book is a comprehensive and brand new look at this whole subject: http://www.oreilly.com/catalog/dambk/


[This article published 4.12.05]
[Updated info on RAID added 27.6.06]
[Revised with additions 9.12.07]


Article written by Paul Herrmann
© Paul Herrmann/Redeye.

With thanks to Jonathan Keenan and David Eaton for additional contributions.

If you have any suggestions for amending or adding to this article, please use the form below.

If you would like to suggest a resource please click here or choose another category