When looking at high end industrial cameras, there are obvious attributes that distinguish the cameras from one another; resolution in mega pixels, speed or number of frames per second, type of sensor CCD versus CMOS, dynamic range and sensitivity, data interface type, mechanical form factor and lens mount, image processing capabilities if any, and the list goes on.

CMOS, dynamic range and sensitivity, data interface type, mechanical form factor and lens mount, image processing capabilities if any, and the list goes on.

The above attributes define the general performance characteristics of a camera. However, non-idealealities of the image sensor and camera are not obvious from list of camera specifications. These non-ideal performance characteristics and artefacts can sometimes become major system implementation hurdles. In this article we discuss some of these artefacts as they pertain to CMOS cameras and methods to correct for them. These factors include fixed pattern noise and photo response non-uniformity as related to exposure time, static and dynamic bad pixel identification and replacement, analog and digital gain adjustment and sensor timing flexibility.

In-Camera Image Correction

Having image correction feature in a camera has many advantages over correction by the frame grabber or the host software. There are some disadvantages as well, but one can always disable camera corrections (if allowed by the camera features) and use correction on the raw output image of the camera. As such, the advantages of camera image correction overshadow disadvantages of it in many cases. For example, in camera corrections could be more precise due to additional sensor data bit depth in the camera than the bit depth of the camera output image. Another advantage is that the camera manufacturers have a better understanding of the sensor characteristics and optimize their calibration algorithms to specific sensor performance. However there are some disadvantages associated with in camera image corrections; for example the real time processing power available within the camera may be limited or the computational precision could be not as good as off camera processing due to limited processor bit depth availability. Other disadvantages may include increased power dissipation or camera cost.

One of the image processing features that can effectively be implemented within a camera is to correct for pixel with non-ideal response behaviour, or bad pixels such as hot pixels or dead pixels. These types of non ideal pixels are present in every CMOS image sensor and there is no “perfect” CMOS image sensor. Typical algorithms can simply replace the pixel value by its neighbouring pixel, or a weighted average of neighbouring pixels or the median value of neighbouring pixels.

Fixed Pattern Noise, Photo Response Non-Uniformity, Flat Field Correction

In the absence of incident light image sensor pixel outputs exhibit a fixed pattern noise. The presence of dark current in CMOS sensors and variation between pixels is the main reason for fixed pattern noise (FPN).  FPN is corrected for by measuring the pixel values at dark and deducting the recorded value as an offset from the raw image after exposed image readout. Cameras that incorporate flat field correction ability (FFC) perform FPN correction in the camera as part of FFC correction. The second element in FFC correction is photo response non-uniformity correction or PRNU, where pixel specific gains are applied to each pixel to compensate for differences in photo responsivity of each pixel. FFC offset and gain coefficients are determined during FFC calibration, where several images at dark and several uniform bright images are taken at certain exposure time.

FFC correction must be done at the actual exposure time that the camera uses. Pixel offsets determined during FPN calibration increase as the exposure time is increased. As such if we use a lower exposure time during actual imaging, FPN correction applied to dark pixels in the image, will create negative values. Since negative values are not meaningful these pixels are clipped to zero value, which leads to loss of image data. One solution to this problem is to store several FFC coefficients for different exposure times or dynamically adjust the FFC coefficients for different exposure times. For applications where a range of exposure times are used, it is preferred to perform FFC calibration at lower end of the exposure time bracket.

FFC correction can only correct for pixels with linear response behaviour. There are two major non-linear responses that image sensor pixels exhibit. The first type of pixels is known as hot pixels. During FPN calibration some of the pixels are saturated or show excessive output values. These pixels may exhibit normal behaviour at low exposure times, however quickly become saturated as the exposure time during FPN calibration at dark is increased. A camera with pixel replacement capability can identify hot pixels and replace them with a weighted average of adjacent pixels. However some pixels may not show up as hot pixels at low exposure times, and therefore not corrected for when imaging at long exposure times.

One advanced calibration feature is to identify those pixels by performing a second FPN calibration at higher end of the exposure time range. Subsequently, the camera uses the FPN data from low exposure time calibration for actual calibration, however replaces the hot pixel introduced at high exposure times as the camera exposure time is increased. The advantage of this method is that the valid pixel data from the linear portion of its response curve is used and it is only replaced when the pixel data is not valid.

There are several methods that can be used for replacing pixels marked as invalid, including median filtering and weighted averaging of adjacent pixels.

The value of a pixel during FPN calibration, or calibration offset, can be used to determine whether a pixel is usable or not. A pixel calibration offset threshold determines the maximum expected offset increase at long exposure times, before it is marked for pixel replacement. If a pixel offset at long exposure times is increased beyond the threshold, it is marked for replacement at long exposure times.

The exhibit below shows how the additional FPN correction at long exposure times and the threshold value helps in detecting misbehaving pixels at longer exposure times:

Figure 1a shows example of four pixels with typical offset values at low exposure times. Figure 1b shows the FPN values of the same pixels at longer exposure times. As we can see the offset value of the pixel on the lower right corner has increased substantially compared to its adjacent pixels. By comparing the offset increase for each pixel with a user defined FPN calibration offset threshold, the camera can identify and mark the pixel for replacement at high exposure times.

The image in figure 2a is an example of FPN at short exposure time. Figure 2b shows the FPN image taken at a longer exposure time. As we can see there are pixels with excessive values visible in the image. By enabling the advanced FPN calibration at longer exposure time, those hot pixels are corrected for at longer exposure time as it is shown in figure 2c.

There is another type of nonlinear pixels, which are called dead pixels. Dead pixels are pixels that exhibit very low responsivity to light. As part of PRNU calibration pixels responsivity to light is measured. Since the sensitivity to incident light is not the same for all pixels, a correction gain must be applied to each pixel to correct for photo response non-uniformity (PRNU) of the image sensor. Pixels that require excessive PRNU correction are considered bad pixels and are marked for pixel replacement.

FFC Statistics

One useful feature for the cameras that do have in camera flat field correction feature is to provide FFC statistics. FFC statistics helps the user better understand the impact of camera FFC correction and adjust the parameters accordingly, or decide whether camera FFC must be enabled or an off camera FFC would be better for specific application.

Dynamic Pixel Replacement

Another type of pixel replacement can be done dynamically where the value of a pixel is compared with its adjacent pixels and if the pixel value difference is greater than a preset threshold, the pixel can be replaced by weighted average of its neighbouring pixels. The dynamic pixel replacement is useful in situations where there are blinking pixels; pixels that behave normally except for rare occasions when they show abnormally high output.

Gain and Offset Resolution

The precision of camera image processing depends on the number of bits assigned to each parameter during image manipulation calculations. The more bit depth resolution assigned to each calculation the more internal bandwidth and memory is required within the camera. On the other hand the error in image processing increases as the number of bits used in the processing is reduced. Therefore there is a balance between parameters bit depth and camera internal required bandwidth and memory. As the number of bits assigned to each parameter is increased a point of diminishing return is approached where the additional bits would not provide material benefit in image processing precision. For example, if the typical range for FPN calibration offset is 64 DN (digital number) in a 10 bit camera, at least 6 bits are needed to represent values between 0 and 64 DN. If we use less than 6 bits we cannot represent pixels with offset values above 64 DN and those pixels cannot be properly corrected.

A similar situation exists with PRNU gain calibration calculation. For example if the camera applies a calibration gain between 1X and 2X, the precision of the gain factor, or minimum gain increment step is defined by the number of bits assigned to the gain factor. In applications where the camera PRNU gain range is limited, the gain value range can be reduced, for example from 1X - 2X to 1X – 1.5X to increase the resolution of gain factor for more precise processing. Some cameras provide a high gain and a high resolution mode for PRNU gain factor that helps with better FFC calibration.

Programmable Gain

Real time control of the camera gain can be a useful feature in many applications. Camera general purpose inputs and outputs can be used to control camera gain in real time. For example if the user needs to change the gin of the camera between subsequent frames, they can signal the camera the new gain before the next frame starts. In some applications the object image at two different exposures can provide better data for subsequent contrast enhancement that can be effective to detect image features with both high and low intensities.

First Frame Offset

The first frame from camera may suffer from accumulated dark current from period of inactivity, resulting in first frame bring different from the subsequent frames.

This excess dark current can be removed by various techniques. One common technique is to dump the first frame and start image output from the second frame onwards. However this technique may not be acceptable in many applications. For example it may be vital for the system to start imaging right after the trigger signal is received and the first frame may contain crucial data. The second reason could be that that excessive delay between the times of imaging trigger to the start of image sequence caused by first frame dump may not be acceptable. Another reason for the first frame being different is that typically after prolonged reset periods, the camera response to light is not the same.

Various mechanisms in the camera can be incorporated to overcome the first frame issue. Employing specific sensor timing can reduce the effect of first frame offset while maintaining the first frame readout delay to a minimum. This is an example of image processing that cannot be implemented outside the camera. Teledyne DALSA’s Falcon2 camera incorporates first frame clear mode feature that utilizes such sensor timing.

In conclusion, when choosing a CMOS camera, the features need to be considered, but also the artefacts and non-ideal performance characteristics need to be considered before they become major system implementation hurdles.