How we test digital cameras

Ever wondered what goes on in the lab?
 
Learn more
 
 
 
 
 

01 .Testing digital cameras

test_camera_lead
Camera testing is a complicated and expensive process. To help with the high cost we share the results between a number of consumer organisations around the world. All the tests and procedures described in this article are carried out, or overseen by professional testers in consumer laboratories in Europe.

To try to keep up with the rapidly evolving digital camera market, we report on batches of cameras every couple of months. Our lab is constantly testing cameras, but each batch can take some months to complete. We only test products we can purchase from retail outlets, which means we're often a bit slower than magazines that take samples from manufacturers, but at least we know that what we're testing is representative of what you're likely to see on the shelves.

The descriptions that follow are of the Basic and High-end tests that make up the scores you'll find in our reviews of basic compact digital cameras and High-end cameras which includes SLRs. 

All cameras are subject to the 'Basic' tests that follow. Cameras that fit with our criteria for 'High-end' cameras are also put through the High-end tests.

The criteria for a High-end camera are as follows:
Manual override for focus and exposure controls
RAW format recording (to allow you more control over the image once it’s taken)
Manual white balance control
Hot shoe for an external flash

Although the two tests share a number of elements, the weighting for each test (Basic and High-end) is different and it isn't possible to compare scores from one test to the other directly.  However, a score of 70% or more is considered a good score and anything less than 40% is a poor score.

In each test description there's a run down of all the basic functions or features we appraise and how we decide what it will score in each category. 

Before any actual testing begins, we do an exhaustive list of each cameras features and specifications. For the most part we don't rely on manufacturers specifications. For example, we take measurements that include all protruding parts when the camera is off and weigh them with their memory, batteries and neck straps etc. that come as standard. Manufactures specifications often leave these sorts of things out, but our intention is to try to give you information that relates to actual use.

All computer related tests are performed on an up to date PC with Microsoft WIN XP Professional SP3 operating system. The five monitors used are profiled and calibrated EIZO ColorEdge CG21 or EIZO S2433W and both monitors and video cards are adjusted to sRGB colour space.

In order to compare the test results to those of the previous tests using this method, in all relevant tests, especially those where a subjective score is given (for example the outdoor shots), samples from the last test are re-tested in certain aspects and used as anchor for scoring.

Unless otherwise noted, subjective tests are evaluated by five people, among them at least three interested users of cameras who are trained in assessing images.
Each panel member has their own monitor to provide best possible viewing conditions.

We hope this article helps to explain why our results sometimes differ from others who rely on an individual opinion to appraise these complex products. See also our digital camera buying guide. Check our article on compact digital camera reviews for more details.

 
 

Sign up to our free
e-Newsletter

Receive FREE email updates of our latest tests, consumer news and CHOICE marketing promotions.

 

All evaluations of picture quality are performed after the photos are transferred to the PC. For all Basic tests the cameras are set to full automatic function, including exposure, white balance and focus function with maximum resolution and image quality setting in JPEG, except where otherwise stated. Test charts are adjusted to fill the full monitor display of the camera.
If not stated otherwise a tripod is used.

Resolution

resolution chartResolution is determined by the evaluation of a black and white test pattern. This is a visual appraisal of the pattern in the middle and at the four corners of the test chart. Additionally, there is a score for aberration and colour problems at the edges of the image, with comments where appropriate.
The pictures are taken in maximum and minimum optical zoom setting at ISO200 (High-end cameras take pictures at ISO1600 and ISO3200 and in both JPEG and RAW formats where possible). The test chart is evenly illuminated by 2 light sources with D65 light characteristic (daylight), illumination level is at 2000 cd/m2.

 

Colour reproduction

Colour reproduction is tested using an IT8 colour test chart. The lighting is similar to Resolution (above) and the camera is set to ISO200 and Auto or its equivIT8 chartalent. For High-end cameras the test is repeated at ISO1600.
There's a subjective score of the colour balance for the IT8 test chart for luminance, contrast, colour saturation and definition.
The pure white field of the test chart is evaluated for colour balance of the primary colours. This is done using a software calculation (histogram) of the test chart differences in brightness for the three primary colours in the white field (ideally they should be equal).

Low light performance

res chart on turntablePicture brightness in very low light conditions is measured by the electronic evaluation of pictures taken of a plain screen with a dimmable white light source. The pictures are taken at a very low luminance setting of 1 cd/m². The sensitivity of the camera is evaluated by measuring the average brightness and the standard deviation of the evaluated pixels in the central part of the digital image.
Resolution in low light conditions is tested at the same light level (1 cd/m²) using the resolution test chart, shown above. This test chart is photographed at the maximum wide position of the cameras lens and scores are given for sharpness and image noise.
Low light performance is evaluated with a visual inspection of an image of a test chart taken in a low light environment. The test chart is placed on a turntable which rotates at 45 seconds per rotation. Photos are taken at an illumination level of 11 lux and evaluated for loss of sharpness and image noise.
Image Noise (High-end)
Is a measurement of the luminance and colour noise (red, green, blue) in the central grey field of the test chart at ISO 1600 and ISO3200 (JPEG and raw format). The camera is set to its minimum focal length and white balance is done manually.

Distortion

41Measures vertical and horizontal grid lines on a target at 1/3 and at 2/3 image height and width in both JPEG and RAW format at ISO 200. The lens is set to minimum and maximum focal length.

 

 



 

Vignetting

vignettingIs a measurement according to DIN 58188 (1991-11) at maximum aperture (smallest F-number) and at an aperture of 5.6 (in both JPEG and raw format) at ISO 200. The lens is set to minimum and maximum focal length. The light level is compared at the optical axis and the average at four appropriate corners at 2/3 image height and width.

 

 

 

Saturated Colour Reproduction (High-end cameras only)

SpectrumWe generate a continuous, optical spectrum from a white light source on a screen and add a bright invisible infrared light source. We take pictures at the highest JPEG quality and in sRGB colour space. These pictures are displayed on a profiled monitor and evaluated for their respective differences to the original spectrum.
This is done at ISO 200 and ISO 1600 and always with manual white balance (measured at the projection screen without dispersive element and infrared source) for all High-end cameras. Scores are given for the colour characteristics at low and high frequencies (infrared to ultraviolet) and the colour continuity (steps or holes in the spectrum).

 

Veiling Glare (High-end cameras only)

Is measured according to ISO 9358:1994 (see also DIN 58186 (1982-10)) at maximum and minimum aperture and at normal focal length with ISO 200 setting and using manual white balance.
This measures the amount of stray light in a lens caused by reflections of light between the surface of the lens element and the inside of the barrel of the lens. A lens that performs poorly will often show up as producing images with a washed-out look over most of the photo. This is often not as noticeable as other types of glare which may leave a distinctive light spot on an image. Veiling glare can result in a reduction in the contrast of the image as well as the colour saturation.

Reflections (High-end cameras only)

An image is taken at maximum aperture (minimum F-number) and minimum and maximum focal length of a dark screen with a bright light source at 10° right or left of the maximum horizontal image angle. We do not use a lens hood, even if one is delivered.

Practical pictures / viewing tests

08Practical pictures are performed using five images:
• Images one and two are portraits under medium light conditions (D65 daylight, about 140 lux) in a room with a dark background, both with and without the use of the integrated flash. The tests are performed with the camera set to Auto or its equivalent. The subject is 2 meters away and the focal length is adjusted to fill the screen with the subject. Scores are given for skin tone quality, colour quality in general, exposure level, sharpness and image noise.
• The third test picture is an outdoor landscape picture with sky, a building, a natural background of grass lawn and trees, taken in bright sunshine. The camera is focused at infinity. Scores are given for detail in highlights and shadows, colour quality, exposure level, sharpness and image noise.
• The fourth test picture is an indoor image of different items such as skin, clothes, toys with artificial colours, high contrasts and a heat source at 450°C temperature (with almost invisible infra red emission). Scores are given for detail in highlights and shadows, colour quality, exposure level, sharpness and image noise. The subject is 2 meters away and the focal le09ngth is adjusted to fill the screen with the subject.
• The fifth test picture is for the evaluation of the face detection function. An image is taken of two people left and right of the image centre, in front of a bright white screen with some high contrast objects. The image is taken as close as possible to normal viewing angle (47°). The two people are at a distance of 1 and 3 meters and the rear screen is at a distance of 5 meters. Scores are given for sharpness and the exposure of the two faces, with preference of the person closest to the camera.

Monitor and viewfinder

The quality of the monitor and viewfinder are evaluated by a panel for readability under different light conditions (sunshine, low light indoor). Adjustments for users wearing glasses (diopter correction) is evaluated as well. In both cases we look for picture quality in terms of colour performance, brightness, contrast, resolution and the speed of the refresh rate (important for precise pictures when zooming or taking photos in panning situations)..

Viewfinder

The evaluation of the viewfinder is performed by scoring the usability in bright sunlight conditions outside, the usability in poor light conditions inside, the quality of resolution and refresh rate at panning scenes and the size of the image in the viewfinder. Basic cameras are not penalized for not having a viewfinder, but High-end cameras are.
For High-end cameras:
We measure the difference of the size of the image in the viewfinder related to the width stored in the image. The camera is positioned as close as possible to a rectangular test chart. The test chart is adjusted to full viewfinder size by adjusting the distance to the test chart at the maximum wide setting and maximum telescopic setting of the zoom lens.

Monitor

The monitor's loss of readability (contrast, brightness, colour) when viewed from an angle other than 90° from above (evaluation range is +/-30° to left and right and +/-20° to above and below related to view from right angle). This test is performed indoors in good light with no light from behind the monitor and in poor light conditions. This is repeated outside in bright conditions.
The quality of the monitor in Playback mode is evaluated using a test image in JPEG format with fine details and high contrast. Each camera tested has the test file displayed in full screen mode and without additional information (if possible), which is photographed with a high resolution digital camera. All cameras’ monitors are shot using the same focal length of the recording camera, so that different sizes of the file image show the different size of the display. These shots of the tested cameras monitors are evaluated on a high quality PC screen for resolution, sharpness, contrast and brightness of the display as well as some other errors such as a visible cell structure of the LCD panel.
In this test only the quality of the monitor itself is evaluated. In the tests previously described the cameras are in record mode, which means that the effects of the image sensor, the lens and the exposure electronics also have an influence on the result.
The following pictures show some examples of different monitor displays.

Poor monitor Middling monitor Good monitor

Image stabilisation

Image stabilisation rigThe test signal is recorded from acceleration sensors applied the arms of two men holding a camera in their hands at a normal shooting position. This signal is averaged for the two persons and measured over a lengthy period. We test by mounting the camera on a shaker and applying the recorded movement in different directions and at different strengths while taking images. Half the images are taken at normal viewing angle (47°) of an 80cm target at around one meter distance. This is done with a light level of 11 Lux and the cameras flash is turned off. Where possible the test is repeated with the lens set to a viewing angle of 12°, which puts the target at around four meters from the camera and the light level is adjusted to 1000 Lux.

Focusing

A score is given for the autofocus behavior in a room with an illumination level of 11 lux, with flash suppressed for; speed of focus, ringing, a stable final setting and sharpness of the images.

Autofocus Sensitivity (High-end cameras only)

We take an image in automatic mode and at normal focal length (image angle ≈ 47°) of a test chart at a distance of one metre illuminated by photo lamps. There's a pattern in the centre of the screen to allow the camera to focus automatically. The power of the photo lamps is reduced which in turn reduces the contrast of the pattern on the test chart. We measure the luminance of the test chart in cd/m² when the auto focus doesn't reliably find the focus point any more.

Macro image

The macro function of the camera is performed by measuring the smallest sharp area which the camera can take in macro mode. We also measure the distance between test chart and lens front. The test is performed using automatic focus.

Flash

We check to see if the flash is "intelligent" (the power of the flash is adapted to the distance from the subject automatically).
Flash quality Flash targets
A picture of the test chart is taken in a dark room. The camera is adjusted to fit to the full test chart in the viewfinder/monitor of the camera by using the zoom lens.
Pictures are taken at one, three and five metres.
The resulting images are appraised for the amount of light put out by the flash, evenness of illumination and ability of the flash to cope with varying distances. If necessary we also Flash evennessnote reflections/shadows from the camera's body.
Flash quality long distance
A complete dark room with white walls has distance marks (small grey cards) at different positions within the room. Their brightness is evaluated for the flatness of the distribution of the flash light across the scene.

Video viewing test

Monitors, projectors and video cards are adjusted to sRGB colour space. All measurements and recordings are made using automatic exposure and automatic white balance settings. Two one minute video recordings are made of a scene featuring people with colourful clothes, other objects with fine details and realistic artificial plants at different distances, both at rest and in motion. The automatic focus system and white balance is switched on – if possible. The camera is mounted on a tripod, about a third of the take is recorded while panning and another third while zooming. Lighting for the tests is under the following lighting conditions at the targets:
- Daylight conditions, 3500 Lux (D65)
- Low light, 12 Lux. A scene from video testing
These recordings are replayed on profiled and calibrated, high-quality liquid crystal monitors or projectors with high resolution (FullHD with 1920 x 1080 pixels and at least 25 (24) frames per second). The test panelists evaluate colour fidelity, the test persons are allowed to observe the actual objects (colourful objects and the like) used in making the recordings with appropriate illumination. The recorded images are rated according to the following criteria:
1. Sharpness (how faithfully structures are reproduced)
2. Colour fidelity (colour neutrality and colour noise)
3. Exposure and contrast gradation (also in highlights and in dark areas)
4. Overall image quality
4. Image and focusing irregularities.

Resolution

We take video recordings at maximum and minimum focal length of the same test chart used for still images and rate them for the resolution of the images at the centre and corners.

Macro Recording

We take video recordings at maximum magnification and rate them for the resolution and the overall quality of the images. Paying particular attention to any issues with focusing, exposure or contrast etc.

Audio Quality

All measurements and recordings are made using the cameras automatic gain control settings. An audio stereo sample from CD (classical music with high dynamic range), with a duration of about 30 seconds is played over two high-quality loudspeakers in a room. First, a reference recording from these samples is produced on a digital recorder with a high-quality stereo microphone at a distance of two metres. The sound pressure level at the location of the microphone is at an average of 65 dB(A). Two audio recordings are made with each camera being tested, under the same conditions. Recording is done using the built-in microphone in automatic mode and we create possible noise by using the zoom button or zoom drive.
We then listen to the recordings using high-quality headphones and rate the “overall audio quality” of the sample recordings, at the original volume. Problems with motor-, zoom- and button-noises are commented on separately.

Device ready for operation

The device is switched off for at least a minute then switched on and we measure the time it takes before the first video frame is taken.

Ease of use is evaluated by five people, at least three of whom have some formal photographic training. Scores range through a five point scale from bad to very good. This section of the test is broken into the following:

Manuals

The printed documentation and the manuals on CD-ROM supplied are evaluated for the following:

  • Complete documentation with detailed explanations and examples for the use of the camera?
  • Appearance, paper quality, readability of text and sketches etc..
  • Does the manual correspond to the delivered camera, its structure, sufficient and easily understood content, are technical expressions explained for 'non experts'?
  • Are there safety hints for electrical danger or damages by static discharge?
  • Are there helpful hints in case of malfunction?
  • Quality of explanations for the use of digital cameras in general and the use of the camera itself.
  • Quality of the description for connecting to a PC and file transfer.
  • Is there a table of contents or equivalent available?
  • Are there hints for the care of rechargeable batteries?

Shutter delay

All the cameShutter delay triggerras functions are set to automatic and the lens focused at a target at 10 metres. Then, with the camera positioned one metre from a stop watch, the shutter release button is pressed in one motion and the time it takes for the shutter to open is measured. A device is used to start the stopwatch at the instant the shutter is pressed and the resulting image shows how far the watch had progressed before the image was captured. We then repeat this process with the target set to one metre to help assess the cameras ability to cope with different circumstances. This whole process is repeated 10 times per camera and the results are averaged.

High end cameras such as SLRs may have a choice of phase detection or contrast detection autofocus depending on their mode (usually LiveView for the latter). Where this option is available the test is carried out in both modes.

Camera ready for operation

The camera is switched off for at least one minute. Then we measure the time it takes before the first picture can be taken. The monitor is on and has to preview the live scene before the release button is operated.

Picture taking speed (single shooting mode)

We measure the time taken from pressing the shutter release to take a picture until the camera indicates it's ready to take next picture. It's set to its highest quality setting with the flash off.

Picture sequence speed (continuous shooting mode)

Using a consistent normal illumination, with the camera set at medium focal length, a subject at three metres and the flash off we measure:
a) the number of pictures / sec or the shortest possible time interval between 2 pictures.
b) number of pictures till the buffer is full.

Inserting and removing memory card

The insertion and the removal of the memory cards is evaluated for: Battery_SDcard
• How easy is it to grip when inserted?
• Is the eject button accessible and easy to use?
• Is there a clear indication of which way/side the memory card should be inserted?
• Does the card compartment have a cover and is it easy to open?

Release button

Is the release button easy to handle? Does it need much force to depress (if it's too stiff there's a danger of moving the camera while releasing)? Is there a clear indication when the shutter releases?

Autofocus

We give a rating for the quality of the feedback of the toggle point when using the focus lock by half pressing the shutter release.

Manual focusing

How easy is it to use the focus adjustment in manual mode? Are there easily readable scales or other information aids for determining the object distance?

Zooming

We give a rating for the adjustment of the optical zoom control, speed, responsiveness, number of steps and precision.

Battery change and control of the remaining power

We check how easy it is to change the battery and what indications there are when the battery is getting low.

Lens Change

We check for precision, mechanical play and friction when changing larger and smaller lenses.

Video Recording

We rate the following:
• How the device fits into the hand and its comfort for taking video
• Delay of video recording starting after pressing the record button
• Feedback during video take
• Control of the zoom
• Control of the record button
• Control of other functions such as focusing

Video playback

We rate the following:
• Access of playback, pause and stop function
• Possibility of fast forward and rewind
• Video clip selection (including sorting, searching and filtering)

Your say - Choice voice

Make a Comment

Members – Sign in on the top right to contribute to comments