DxO sheds light on camera sensor performance

A correction was made to this story. See below for details.

DxO Labs, a French company with deep experience measuring cameras' technical performance, has launched a Web site called DxOMark.com that features detailed information on the performance of the image sensor at the heart of many higher-end digital cameras.

Many Web sites and magazines measure camera image quality with varying degrees of rigor, typically examining either the JPEG that the camera produces or a processed version of the camera's raw. But with its DxOMark Sensor measurement, DxO takes a new approach by judging the sensor performance based on the unprocessed "raw" image file from higher-end cameras such as SLRs.

That's significant, because raw images typically must go through a conversion process called demosaicing before they're useful for viewing. Most digital cameras capture only a single color--red, green, or blue--for each sensor pixel. Demosaicing fills in the gaps in this colored checkerboard pattern so each pixel gets all three color components, but this processing stage can disguise sensor performance.

The detail-obsessed camera crowd has begun eagerly chomping on the new data. On Sunday, there were 220 mentions of DxOmark on the Digital Photography Review forums, a popular location for impassioned technical discussions.

Nikon's D90 sensor beats out the one in Canon's 50D, judged on the basis of the raw files it produces.

(Credit: DxO Labs)

New tests coming
More measurements are coming, added Nicolas Touchard, vice president of marketing for DxO Labs' image quality evaluation business. First, in two or three weeks, will come measurements for medium-format digital camera sensors from companies including Hasselblad, Mamiya, Phase One, and Leaf. Then will come more high-end compact "bridge" cameras.

DxOMark Image Processing for the camera's computer, whose job it is to perform tasks such as converting raw images to JPEG, and DxOMark Optics for lenses.

The latter measurement will go beyond most lens tests by showing how well each lens works on each camera rather than one or two reference models. DxO takes that approach because lenses behave differently because different cameras have different attributes such as the geometry of the microlenses that help each sensor pixel gather more light, Touchard said.

DxO makes a business out of detailed measurements of camera performance, selling the data to camera and chip companies and incorporating it into its own DxO Optics Pro raw-processing software for photographers. So why give some of the data away for free on a Web site? Publicity.

"It's a marketing program to communicate to the public as well to the industry our expertise in measuring cameras," Touchard said.

The DxOMark Sensor benchmark measures a sensor's color depth (useful for studio photographers), high-ISO performance (useful for photojournalists), and dynamic range (useful for landscape photographers)--the span between dark and bright. DxO rolls the three parameters up into a composite DxOMark Sensor score, but it provides other detailed measurements for each camera as well. (DxO also offers detailed descriptions of the three DxOMark sensor components.)

The Bayer pattern, invented decades ago by Kodak, defines which of three colors each pixel on a camera sensor can detect. The camera or software such as Photoshop must be used to 'demosaic' the resulting 'raw' image into one that has red, green, and blue values for each pixel.

(Credit: DxO Labs)

DxOMark Sensor doesn't measure many camera attributes--durability, autofocus, video and live-view modes, battery life, LCD quality, weatherproofing, user interface--and DxO doesn't pretend it does. Indeed, that's part of the reason for the later benchmarks for image processing and optics that the company envisions.

Intrigue
But even in its limited form, the tests are interesting. For example, the full-frame 12-megpixel sensors in Nikon's top-end D3 and D700 models currently rule the roost, beating out the full frame sensors in Canon's top-end 21-megpixel EOS 1Ds Mark III and Sony's top-end 24-megpixel Alpha A900. The D3 has a 80.6, the D700 80.5, the 1Ds Mark III 80.3, and the A900 78.9. Those differences are relatively narrow.

More interesting, perhaps, is the difference in performance lower down the line. With a score of 72.6, the sensor in Nikon's new 12-megapixel D90 is head and shoulders over the Canon 50D's 15-megapixel sensor, which scored 62.9, even though the D90 is a bit cheaper. Indeed, the D90's sensor scored sixth out of 50 cameras measured.

Sensor results can be compared two different ways: "screen," where performance is compared pixel-by-pixel, and "print," which compares the sensor's full-size images. In the print mode, to compensate for different sensor sizes and resolutions, the DxOMark sensor test normalizes results mathematically to the equivalent of an 8MP image.

Smaller pixels suffer from more noise, which shows up as speckles in the image. But having more pixels also improves the amount of information in an image, so high-resolution sensors get an edge in the "print" measurements. Thus, for example, the Canon 50D makes up some of the ground on the Nikon D90 when viewed in "print" mode to consider the whole image.

In camera sensors' signal-to-noise ratio (measured in decibels), Nikon sensors on average has now erased Canon's years-long lead.

(Credit: DxO Labs)

Comparing the Canon 40D to the newer 50D is also instructive. Though the 40D gets an overall score that's higher, presumably because of stronger weighting of the color depth, the 50D fares a smidgen better with dynamic range and significantly better with low-light performance.

Also worth a look
Also interesting at the site are a couple explanations, one about "the paradoxical evolution of sensor signal-to-noise ratio over time" and another that argues that "contrary to conventional wisdom, higher resolution actually compensates for noise." (Dust off your logarithms and statistical analysis for the latter.)

The diversity of camera sensor designs shines through in DxO's measurements. The SuperCCD sensors used in Fujifilm's SLRs, for example, use two sensors for each photo's pixel to increase the dynamic range of the images. That design shines through when it comes to dynamic range: the Fujifilm FinePix S5 Pro spans a range of 13.5 stops compared with 11.5 for the Nikon D200 and 12 for the Nikon D300. However, the Fujifilm camera's sensor pixels are larger, reducing resolution.

And there's still no result for the Sony Alpha A700, which performs some image processing on the sensor before the raw image is created. "It's not really raw. I'd say it's medium raw," Touchard said.

A final note: DxO provides some fodder for the enthusiasts who prefer to shoot raw images instead of JPEGs. That's because demosaicing algorithms get better over time, so more information can be recovered with newer software.

"For example, over the past two years, DxO Optics Pro raw conversion (with its denoising algorithm) showed a gain of 4 decibels signal-to-noise ratio, a much larger improvement than that achieved for sensor technology," DxO said (For reference, 4 decibels is about a 1-1/3 stop in exposure.) "These advances in raw conversion algorithms will continue, underscoring the advantages of shooting in raw, as raw images can always be reprocessed with every new release of a raw converter."

DxO's data is fun to pore through and should will be helpful for those considering upgrades or new purchases. Just don't let the technical analysis get in the way of actually taking pictures, too.

Correction on November 24: This story initially mentioned a Nikon camera with a 12-stop dynamic range. That camera is the D300.

[ibeetle]

Nice artical. Thanks for the information. And great website.

I did get the impression from your article that this site is for prosumer and professional photographers. There are testing and benchmark results for many entry level DSLR's (such as the Nikon D40) and even some high end point and shot cameras.

If these testing results are accurate, and if DxO runs with it DxOMark could become the first place everybody should go to when comparing cameras.

This site should DxOMark be bookmarked in your browser.

[Lukas_C]

The hard-fact advantage of this approach is very interesting indeed, but it's usefulness to an average enthusiast or amateur is limited, even though the simple 1-number-to-rule-them-all would suggest otherwise.
Even though the site states that it only measures the raw output, the simplicity of one number will cater especially to the amateurs and enthusiasts who do not often (if at all) shoot raw.
The mathematical approach to noise is all good and dandy, but in photography there is noise and then there is noise - meaning the subjective look of various types of noise, one more pleasing than the other. Objective mathematical tests would be hard pressed to reflect this subjective quality of a picture, which nevertheless is also part of photography. Then there is also the issue of Sony a900 suberb low-ligh noise rating which raised some eyebrows, but that may just be our canon fanboyism ;)

It is a great tool, but I fear that the consumers will treat it as a buying guide.

[Shankland]

One number to rule them all is indeed a problem, as we've seen with the not-terribly-helpful megapixel race over the years (or CPU clock speed, or TV diagonal), but I suspect that the DxOMark site will appeal chiefly to the kinds of people who already are poring through 30-page reviews at DPreview--the kind of people who probably are aware that there's more to a camera than a few benchmarks.

[canberra_photographer]

A JPEG is still a product of the camera's sensor, so one can still learn from this which camera is capturing the most and therefore, has the most to work with.

[Hep Cat]

Yay. More minutiae for the numbers-obsessed.

For some reason, it seems like the photographers who spend more time, you know, actually going out and making pictures do the best. The gear queers who obsess over this stuff tend to make subpar photographs when they go shooting at all.

[Shankland]

I agree that some people can go off the deep end about camera specs--thus my closing line of "Just don't let the technical analysis get in the way of actually taking pictures, too." But for technically savvy people torn between two camera models, this information actually could be helpful. When DxO adds some lens measurement tests, I for one will pay attention before plonking down $1400 for some telephoto.

[peterblaise]

.

Earlier: "... Most digital cameras capture only a single color--red, green, or blue--for each sensor pixel ..."

Peter Blaise responds:

No. Except for Foveon, all digital cameras capture a simple brightness value range (tone) that happens to be filtered through only a single color per pixel, via red, green, green, and blue lenses (chrome) that separate solo color values for each capture pixel.

Demosiacing extrapolates the missing color values for each pixel, so the captured tone (non color) through a red filter (chrome) gets a blue and a green value added from nearby blue- and green- filtered pixels, and so on for each pixel.

A single color is NOT captured per pixel; a single color VALUE is captured. Demosaicing extrapolated the missing color values to reconstruct a total red, green, blue set of chrome values for each captured tone pixel.

Slight difference, but critical once we try to understand the inner difference between tone and chrome in our imaging.

==

The reason to ***** Raw image files themselves without demosaicing is to assess the imaging sensor potential, rather then assessing the demosaicing algorithm.

See sites such as Dalibor Jelinek's conversion for Minolta Raw - http://www.dalibor.cz/minolta/plugin_user_guide.htm - one of the few Raw converters that permit us to try different algorithms. How complex does it get? There are 17 algorithms available:
- none (chose chrome or tone)
- average (half size)
- nearest neighbor, chose north-west, north-east, south-west, or south-east
- bilinear
- smooth hue transition, logarithmic or not
- edge sensing
- color correction
- pixel grouping
- variable number of gradients ... and so on.
And that's just antique algorithms; there are new software approaches to Raw conversion every day. There's also, of course, custom algorithms for each camera, including such arcana as "Custom white balance ? defined by coefficients" versus "Custom white balance ? defined by temperature". There's NO WAY to compare non-Raw image files without being totally overwhelmed by having to also compare the incredible variety and choices in Raw conversion software.

So, it's good that there are "total camera" reviews out there, and it's good that there are "Raw conversion software" reviews out there. DxOMark.com is the first "raw" Raw review prior to conversion I have seen. Why is this important? Because 10, 20, 30 years from now, we may be using completely different Raw conversion software, so todays reviews of today's Raw conversion software will hardly matter then. But, we'll still be going back into our image archives and opening our original Raw capture files, just as we now are revisiting and scanning our image archives of old negatives and slides for reuse.

Using DxOMark.com, we may get reassurance that the Raw capture itself is as pristine as possible in today's marketplace, even if the maker's Raw conversion software is not as sophisticated as the competition, or if current commercial Raw conversion software is lacking in some way. For me, I want the highest qualities inside the Raw image file I capture and archive today, and then I know that whatever and whenever I convert to an editable file (16-bit lossless TIF, PSD), and export to a distribution file (8-bit lossy JPG), I will have the minimum compromise possible.