Menu

DNG vs RAW vs Jpeg-the mission continues…

DNG vs RAW vs Jpeg-the mission continues…frustration-798907.jpg

Kia ora:

in a post not long ago, I pointed out that sooner or later, I would respond to Mary Jo’s request for some sort of statement on the question of the DNG format and why you might want to use it instead of Raw or JPEG format. Well, I thought the answer would be relatively straightforward.

It isn’t.

I guess the best way to think of it is to imagine going out. late one night for a swim in a clear blue mountain stream sparkling in the moonlight, but finding instead that you have jumped into a muddy, swollen torrent. The more I have attempted to swim, the more I’ve found that I am trying to cross the Mississippi!

Nonetheless, I have kept on going, realising that the answer (if any) is as important for me as it probably is for you.

With that in mind, here is a summary of what I’ve learned to date. It is not an easy read, far from it, and at the end of this you may well be as confused as you were at the beginning, but I have done my best to try and unpack the whole vexed process.

I would strongly suggest that you do not try reading this after a full meal and a few wines. This is the sort of read you should only attempt after a light salad and half a dozen Red Bulls.

Firstly, we need to break the different file formats into two categories:

1. lossy formats,

and

2. lossless formats

In the beginning there was the JPEG, an attempt by an organisation known as the Joint Photographic Experts Group, which developed a standard for the file format (no, you do not want to know its correct ISO-name!). The reason why this particular file format came into being was to enable people to send a picture files across the Internet, at a time when the de-facto speed was 14.4 K. To do this, they had to find a way to shrink the file size down, and so the JPEG came in to being. Depending on the chosen compression ratio, it was and it is possible to compress the file by up to 90%. The JPEG does this by writing a series of shorthand algorithms for the data in a particular file. Thus a block of pixels 10 x 10, with a common RGB value of 120, 120, 120 is written in a short form. The degree of compression you use in a particular file tells your software how far it may vary beyond those values. A low degree of compression may only allow values between 116 and 124; a high degree of compression may allow values between 100 and 140. You can see that the greater the amount of compression, the more a room there is for variation in the interpretation of the file by a piece of software rendering it in another computer at a later time. Because the software you’re using has to read and rewrite the algorithms, there is obviously room for error, and that is what happens. Put bluntly, the more you open, edit and save a JPEG file, the more the original information is degraded. But don’t take my word for it, try it for yourselves.

At the same time as the JPEG, the de-facto lossless standard was the Tiff (Tagged Image File Format). This particular file format records the RGB data for each pixel in the image, along with its location. As a consequence of this, you can open and close it as many times as you want without degrading the information present in the file. The downside of the tiff, however, is that it is large, and as a consequence, takes up a lot of storage space. As an example, the JPEG (full-size) from a Canon 1 D mark III is about 3.5 MB in size, while a full-blown TIFF from the same camera in 16BIT is around 60 to 70 megabytes in size. Jpegs, because they take up less space, mean that you can get more images on a card and fill up less of your hard drive(s).

Because the JPEG, (which is processed in-camera) is rendered instantly and is significantly smaller in size, it is popular with a certain section of the photographic community, namely news photographers and journalists. It does however have another downside, in that it can only be output in 8-bit, which means that the image quality will never be quite as good as that obtained from either a native Raw or TIFF file. But it does have its uses.

In the early days, when 256 MB cards cost in excess of $1000, it was common for some cameras to be able to write directly to Tiff. Very few if any do that these days, for obvious reasons.

For a period of time, the Good Ship Digital Photography sailed serenely along, with one camp happily espousing the virtues of JPEGs, while the other realised the value of TIFF and worshipped at its shrine. It was a sea calm and untroubled.

Then the waters began to muddy. A number of manufacturers, among them Fuji, began releasing their cameras with proprietary raw formats. Fuji used a .RAF file extension, while Canon released of their own formats, .CRW (Canon raw) and Nikon the .NEF. The others soon followed suit. The advantage of the raw file format is that processing can be done at a later stage in the computer, and that all one records is pure data. This means that capture errors (for example, setting the correct white balance) can be corrected at a later stage in a variety of different software (provided, of course, that the software recognizes this particular file format!) In the early days, this was not always the case. Furthermore, the data can be processed in a 16-bit form, which gives a better tonality’s and subtleties to the finished image. It allows the use of colour spaces with a wider gamut, meaning the ability to render subtleties and shadows and highlights is improved.

The software manufacturers, including Adobe, the 800-lb gorilla of the image editing world, were obviously less and less impressed, and here is why. Because a raw file format is machine-specific rather than manufacturer-specific, each camera has its own raw format. Thus, the .CR2 for a Canon 5D is different to that for a 40D and for a 350D. Add in cameras from Olympus, Panasonic, Sony, Minolta and Leica, and you can see that having to rewrite a raw-file definition for each model on the market must be a never-ending nightmare for the manufacturers.

Enter the Digital Negative Format. DNG for short, it was first released in 2004, along worth a converter to save files in the new format. It was an attempt by Adobe to create an open standard for raw file formats that could be adopted by all manufacturers, a good idea in principle that would have found great favour with the camera-buying public. Alas, it was not to be. A number of manufacturers, including Canon and Nikon, refused to adopt it, claiming that it would give away too many of their commercial secrets! As a consequence, DNG limps along, trying to gain favour and universal adoption. While a couple of manufacturers have begun to openly support it as a format (eg. Leica), most are sticking to the old way.

Dammit.

For the foreseeable future it looks as if manufacturers will continue to develop proprietary raw formats for every new camera. As they are released, manufacturers like Adobe will continue to released updated converters to allow these cameras to be used. What this means, of course, is limited retrofitting. Adobe does not issue updated converters for its older programs. If you use CS2, for example, and you want to use raw files, and you’ve just bought yourself with the new and Nikon D300, then, to put it bluntly, you are screwed. The versions of Adobe Camera Raw compatible with CS2 do not support any cameras released after the introduction of CS3. You either upgrade or trade down or use one of the many excellent alternative products, such as Bibble, ACDSee, or Lightroom.

To sum up then, the files available to us can be divided into two distinct camps; on the one hand we have lossy formats such as the JPEG, on the other we have lossless formats such as TIFF, DNG and the huge plethora of different raw file formats. No wonder film still has a certain following!

But wait, there’s more.

The question I set out to answer was whether I, who shoot almost exclusively in Raw, should consider saving all my work in DNG, which on the surface promises to be a file format with some longevity. Adobe have, after all, been in the business for some 17 or 18 years now, so presumably they will be there for a little but longer. Will I be able to access my Canon raw files in 20 years, or for that matter the files of whatever camera I am using at the time? Will any of the software I am using at the time recognize DNG files? Frankly I don’t know the answer to that, and the reading I have done to date hasn’t really encouraged me all that much. Is the DNG a bomb about to go off and explode through the digital world, conquering by storm, or rather a sputtering fuse? Frankly I don’t know.

And there is another issue (well several).

When you save to DNG, you have the option of writing to a pure DNG file or embedding your original raw file inside it. On the face of it, the latter would appear to be the safer of the two options, a sort of have-your-cake-and-eat-it scenario. Of course, the question remains: Will there be software in 20 years time, capable of extracting the original raw file, and does the converter at the moment copy the data exactly? Fishing around on the Internet, or should that be swimming in shark infested waters, I found some suggestion, particularly with some brands, that the original raw file in the DNG may not necessarily always be exactly the same. From what I can tell some of the difficulties for Adobe come from the reluctance of certain camera manufacturers to release information about the way in which their cameras work.

Foveon appear to be a case in point.

“There is no way to support Foveon X3 data in DNG without publishing intimate details of the off-sensor data, which is Foveon’s intellectual property”.Zalman Stern:

Writing an embedded-raw DNG file leads to a whopping increase in file size. A Canon 5D raw file for example, increases from 13 MB to 28 MB! If you shoot the way that I do (probably you don’t), then the implications for file storage requirements are immense. With something in the order of 21/2 TB of data already in my machine, the implications for backup and archiving are significant if I switch to this format.

But wait, there is even more. The day of the JPEG, as we know it may be coming to an end. While it will continue to be supported, the heir to the throne is already waiting in the wings. You may have wondered what JPEG 2000 was. It was a format developed and approved by the JPEG organisation (which still exists!), which allowed 12 and 16 bit images, and was essentially lossless while it has stayed absent from still photography, digital cinema has embraced it, I believe.. Imagine that -a lossless JPEG! Believe me, they have not been sleeping out there, and you might be surprised to know that there are at least four different versions of the JPEG already. A.png format already exists, which is a much higher quality version of the JPEG, allowing 16 bit storage. But the JPEG 2000 is not the successor I had in mind, not the Crown Prince.

It is the new HD photo format, a brainchild of Microsoft and an attempt to produce a new standard, which allows the JPEG to be lifted to a new level. It allows for 32-bit processing, and is already supported by Microsoft’s Vista. I know what you’re thinking; it’s another powerplay by Microsoft (and have we not had enough of those!), and an attempt to take over the digital world. Goodness knows they haven’t had much success: they bought iVIew Media Pro, wrote more bugs into it then doubled the price, but this may actually be something useful. It has certainly gained the JPEG organisation’s approval, and we should see it within another year or so, as the JPEG XR format.

….One important aspect regarding the standardization of HD Photo is Microsoft’s commitment to make its patents that are required to implement the specification available without charge. Microsoft’s royalty free commitment will help the JPEG committee foster widespread adoption of the specification and help ensure that it can be implemented by the widest possible audience. The JPEG committee hopes and encourages all participants in its meetings to consider this royalty free approach when offering patented technology as a candidate for standardization….

Jpeg Committee, July 6, 2007

And before you run away, to those of you still awake, if it should come off, and it shows a lot of promise. Firstly, it is a lossless compression format, secondly it allows for 32-bit processing, and an in-camera, processed image that comes very close to a full tiff. On the face of it, it seems to be something worth being able to access. Whether it gets off the ground is another matter.

So there it is at the moment.

To sum up:

  • JPEG is an in-camera format, rather like slide film. You make all the choices at the moment of exposure and live with the consequences.
  • JPEG is an excellent choice for people who need a compressed format, for limited reproduction, and for transmission.
  • JPEG is an excellent choice for Web images in for images to be e-mailed.
  • JPEG has been a de-facto standard for some 18 years now, and as such is probably a very safe choice from an archival point of view.
  • JP is can only be output as 8-bit, and as such are not able to deliver the same level of quality as a raw or DNG file.
  • Raw files are processed in-computer within the of a wide range of different converters, each passing their own character to the final image. Think of it as a black and white film, which can be processed in a variety of different developers, each yielding a different result.
  • Raw files require software that can read them. Retrofitting can often be a major issue, and tends to require an ongoing upgrade process in terms of software.
  • Raw files generally offer a wider range of colour gamut options and tend to be the choice of people concerned with extracting the very best quality from the finished image.
  • Raw files are camera-specific rather than manufacturer-specific. Each new camera requires specific support from the software manufacturers. There is no universal standards here.
  • DNG has only been in existence for four years. A lack of support from camera manufacturers (with one or two exceptions), has meant that its writers are constantly having to release new versions as new cameras are released. In this regard it does not appear to me to be that different from the current status quo with raw files.
  • DNG files can either be written natively, or with the original raw file embedded. Choosing the latter option file size is dramatically increased. There is some talk of releasing a compressed DNG file, but that may be some time away.
  • DNG files offer enormous potential, but only if camera manufacturers come on board. Pigs might fly.

I guess the question you’re asking is, would you go to DNG, Tony?

Right now? I don’t think so. As much as I want to, I remain to be convinced.

Perhaps, when I win Lotto, and can afford a Big Blue to do the processing for me.

10 Responses

  1. garry says:

    i still like film!

  2. Peregrina says:

    Garry, so do I.

  3. Tony Bridge says:

    Good to know that the Age of The Dinosaurs has not yet passed (snigger)

  4. Tony Bridge says:

    Actually I like it too, especially in large 8″ x 10″ sheets…

  5. garry says:

    are you suggesting that i am a dinosaur? when we are out testing my deardorrf i can assure you there will only be two dinsaurs prssent – including the camera.

  6. Bikerchris says:

    Perhaps I should not have sold my F100………….

  7. mj says:

    Many thanks Tony. Got it!

  8. Ray says:

    I still love my silvestri.

  9. Tom McGhee says:

    hey Tony

    you brought up some very good points. as a stock shooter and relatively
    new digital shooter 3 yr on & off mostly due to requests from impatient clients. I really do worry about the future of RAW files and having access to all the information from these in the future with the development of new programs coming out every 2 mins so the corps can keep bleeding us dry…I will continue to shoot my fine art work with Neg me thinks!!! just as long as the greenies don’t put out a complete ban on the chemisty I will be safe :))) thanks for the info .

  10. Phil says:

    Hate to throw more mud into the mix, but another crucial point for me is the metadata. Currently, if I do some quick edits in Bridge, such as a crop, WB correction, or exposure correction, this information is saved in a sidecar file. I also believe that the keywords, headlines, and other metadata are stored in this file as well. DNG can save all this information in itself which is extremely convenient. When I make edits to the actual image though, I cannot save them into the file, which I suppose is wishful thinking on my part. But I can hope that if adobe controls that standard that it might be able to store the lossless edits into the file sometime in the future 😛

Leave a reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

All original content on these pages is fingerprinted and certified by Digiprove