All evaluations of picture quality are performed after the photos are transferred to the PC. For all Basic tests the cameras are set to full automatic function, including exposure, white balance and focus function with maximum resolution and image quality setting in JPEG, except where otherwise stated. Test charts are adjusted to fill the full monitor display of the camera.
If not stated otherwise a tripod is used.
Resolution is determined by the evaluation of a black and white test pattern. This is a visual appraisal of the pattern in the middle and at the four corners of the test chart. Additionally, there is a score for aberration and colour problems at the edges of the image, with comments where appropriate.
The pictures are taken in maximum and minimum optical zoom setting at ISO200 (High-end cameras take pictures at ISO1600 and ISO3200 and in both JPEG and RAW formats where possible). The test chart is evenly illuminated by 2 light sources with D65 light characteristic (daylight), illumination level is at 2000 cd/m2.
Colour reproduction is tested using an IT8 colour test chart. The lighting is similar to Resolution (above) and the camera is set to ISO200 and Auto or its equivalent. For High-end cameras the test is repeated at ISO1600.
There's a subjective score of the colour balance for the IT8 test chart for luminance, contrast, colour saturation and definition.
The pure white field of the test chart is evaluated for colour balance of the primary colours. This is done using a software calculation (histogram) of the test chart differences in brightness for the three primary colours in the white field (ideally they should be equal).
Low light performance
Picture brightness in very low light conditions is measured by the electronic evaluation of pictures taken of a plain screen with a dimmable white light source. The pictures are taken at a very low luminance setting of 1 cd/m². The sensitivity of the camera is evaluated by measuring the average brightness and the standard deviation of the evaluated pixels in the central part of the digital image.
Resolution in low light conditions is tested at the same light level (1 cd/m²) using the resolution test chart, shown above. This test chart is photographed at the maximum wide position of the cameras lens and scores are given for sharpness and image noise.
Low light performance is evaluated with a visual inspection of an image of a test chart taken in a low light environment. The test chart is placed on a turntable which rotates at 45 seconds per rotation. Photos are taken at an illumination level of 11 lux and evaluated for loss of sharpness and image noise.
Image Noise (High-end)
Is a measurement of the luminance and colour noise (red, green, blue) in the central grey field of the test chart at ISO 1600 and ISO3200 (JPEG and raw format). The camera is set to its minimum focal length and white balance is done manually.
Measures vertical and horizontal grid lines on a target at 1/3 and at 2/3 image height and width in both JPEG and RAW format at ISO 200. The lens is set to minimum and maximum focal length.
Is a measurement according to DIN 58188 (1991-11) at maximum aperture (smallest F-number) and at an aperture of 5.6 (in both JPEG and raw format) at ISO 200. The lens is set to minimum and maximum focal length. The light level is compared at the optical axis and the average at four appropriate corners at 2/3 image height and width.
Saturated Colour Reproduction (High-end cameras only)
We generate a continuous, optical spectrum from a white light source on a screen and add a bright invisible infrared light source. We take pictures at the highest JPEG quality and in sRGB colour space. These pictures are displayed on a profiled monitor and evaluated for their respective differences to the original spectrum.
This is done at ISO 200 and ISO 1600 and always with manual white balance (measured at the projection screen without dispersive element and infrared source) for all High-end cameras. Scores are given for the colour characteristics at low and high frequencies (infrared to ultraviolet) and the colour continuity (steps or holes in the spectrum).
Veiling Glare (High-end cameras only)
Is measured according to ISO 9358:1994 (see also DIN 58186 (1982-10)) at maximum and minimum aperture and at normal focal length with ISO 200 setting and using manual white balance.
This measures the amount of stray light in a lens caused by reflections of light between the surface of the lens element and the inside of the barrel of the lens. A lens that performs poorly will often show up as producing images with a washed-out look over most of the photo. This is often not as noticeable as other types of glare which may leave a distinctive light spot on an image. Veiling glare can result in a reduction in the contrast of the image as well as the colour saturation.
Reflections (High-end cameras only)
An image is taken at maximum aperture (minimum F-number) and minimum and maximum focal length of a dark screen with a bright light source at 10° right or left of the maximum horizontal image angle. We do not use a lens hood, even if one is delivered.
Practical pictures / viewing tests
Practical pictures are performed using five images:
• Images one and two are portraits under medium light conditions (D65 daylight, about 140 lux) in a room with a dark background, both with and without the use of the integrated flash. The tests are performed with the camera set to Auto or its equivalent. The subject is 2 meters away and the focal length is adjusted to fill the screen with the subject. Scores are given for skin tone quality, colour quality in general, exposure level, sharpness and image noise.
• The third test picture is an outdoor landscape picture with sky, a building, a natural background of grass lawn and trees, taken in bright sunshine. The camera is focused at infinity. Scores are given for detail in highlights and shadows, colour quality, exposure level, sharpness and image noise.
• The fourth test picture is an indoor image of different items such as skin, clothes, toys with artificial colours, high contrasts and a heat source at 450°C temperature (with almost invisible infra red emission). Scores are given for detail in highlights and shadows, colour quality, exposure level, sharpness and image noise. The subject is 2 meters away and the focal length is adjusted to fill the screen with the subject.
• The fifth test picture is for the evaluation of the face detection function. An image is taken of two people left and right of the image centre, in front of a bright white screen with some high contrast objects. The image is taken as close as possible to normal viewing angle (47°). The two people are at a distance of 1 and 3 meters and the rear screen is at a distance of 5 meters. Scores are given for sharpness and the exposure of the two faces, with preference of the person closest to the camera.