Machine Vision Meets Color
Color machine vision can be used successfully for generalized object recognition.
Color machine vision has two principal uses. The first is to verify that the correct color product is being produced; the second is to use color properties for object recognition. In the latter case, it is not the colors themselves that are of interest, but identification of the object that generates them.
Quite a few recent articles and webinars have covered the basics of color machine vision. These usually discuss methods well suited for color measurements to verify that the correct color product is being produced. These methods also can be used for identification of single colored objects; however, they are usually impractical for multicolored objects and those where the region being inspected may include complex boundaries. Common methods of color-based recognition (CBR), including ones that make it easy to recognize multicolored objects or those with complex boundaries, are the focus here.
For simplicity this will focus on classification, however, the methods involved generally cover both types of recognition.
To cope with the broadening, it is traditional to transform color coordinates from RGB, native to cameras and image monitors, into HSI (hue, saturation, intensity), LAB color space (color-opponent space with dimension L for lightness and A and B for the color-opponent dimensions) or other coordinates that conform more closely to human’s perceptions of color. The transformation usually makes it easier to set color boundary thresholds because hue tends to remain relatively constant as lighting details vary. Systems using this approach are often demonstrated sorting such items as brightly colored M&M candies or other high-contrast single colored items.
Extensive discussions of the coordinate transformation and threshold approach appear in textbooks and system suppliers’ literature. It is both intuitively appealing and appropriate for differentiating simple single-colored items. However, system engineering and maintenance costs can quickly become overwhelming when one tries to use it for recognizing multicolored objects or those with complex shapes or unpredictable orientation.
Another approach well suited to high speed sorting of items with limited acceptable color range is to assign each possible color to a specific class of interest. Although this approach can be time consuming to train and is best implemented in specialized hardware, it can be extremely fast and efficient for detection of imperfections in high throughput sorting.
This article will focus on methods specifically designed for generalized color-based recognition. By abandoning the assumption of single colored items and the need to report actual color values, these systems gain the ability to provide robust and cost-effective recognition. While not widely known, the methods involved have a long record of accomplishment in industrial applications where they have demonstrated recognition accuracy rivaling that of humans but much faster and more reliable over extended periods.
Other factors, such as effects of an imager’s Bayer pattern, effects of pixels falling on item boundaries or uncertainties about the boundaries themselves can increase the likelihood of overlap among the color distributions of the different classes, whether single colored or multicolored.
The bottom line is that for all but a very few color-based recognition problems involving simple boundaries, contrasting colors and carefully designed lighting, the assumptions on which the traditional CBR methods are based rarely hold.
These systems use powerful concepts from information theory, involving hundreds of parameters, to characterize the color distributions of each reference class. The concepts include well-defined methods for classification and anomaly detection. By comparing the color distribution statistics of an unknown object with the statistics for each reference class, the systems can find the reference class most likely to have been the source of the color pattern captured by the camera.
In the case of anomaly detection, the theory provides a comprehensive measure of the anomalousness related to the degree to which a test distribution falls inside or outside the reference distribution.
Implementation in a commercial software package requires substantial expertise and investment by the original developers. However, in well-designed packages all the underlying complexity is hidden. Training by example, much as one would train a human inspector, means little special user knowledge is required. It becomes no more difficult to train on, and recognize, items with the most complex color distributions than to do the same for the simplest single-colored item.
Engineering cost reduction comes primarily from system simplification. The general nature of the method used virtually eliminates the need for color space conversions, special algorithm development, color picking and threshold setting. Requirements of lighting uniformity are often significantly reduced, and glints and shadows may be an aid instead of a hindrance in some applications.
Modern information theory based CBR is equally at home sorting based on large color contrasts or distinguishing subtle color shades near the limits of human perception. The powerful “looks most like” matching can greatly reduce or eliminate the need for equipment to accurately position items to be identified.
Reduced setup and system training time can significantly reduce operator costs. Even manual, train-by-show training is fast, and automated training or retraining for a single class of object may take only a small fraction of a second; inspection is even faster. Even for such complex inspections as fabric sample cards containing 30 or more different swatches, one-time setup may require less than an hour of off-line assistant time, pre-inspection on-line training less than three minutes of operator time. Actual inspection rates can be as high as several cards per second, limited usually by the time necessary to move the sample into position.
Any identification application in which differences in color or form of items create statistically significant color differences in their images.
Any routine sorting, packing or inspection task where valid decisions can be made using color information alone by a minimally skilled worker.
Whenever simple class information at one or more locations in an image is all that is required, particularly when the item may appear in more than one orientation.
Experienced integrators of color vision systems are familiar with the need for:
Stable, broad spectrum illumination and dealing with the effects of degradation of phosphors and other lighting components.
Proper shielding from the effects of ambient lighting variations.
Strict environmental controls to minimize temperature dependent drift of lighting, camera and other electronics.
Newcomers to machine vision would do well to consult with such an integrator, at least for advice on lighting and imaging. By heeding that advice and devoting the necessary resources to creating and maintaining the appropriate inspection environment, one can expect a high level of success.
Ask each potential supplier to demonstrate how their approach will meet requirements using a representative set of samples or sample images that have been chosen.
If possible, pick these samples so they can serve as the basis of system acceptance tests. For simple applications there may be little or no charge for demonstrations, particularly if they can be done at the supplier’s site. For applications that are more complex or if the demonstration must be performed at the customer’s facilities, a modest charge may be involved.
Color-based recognition makes it easy to recognize multicolored objects and those with complex boundaries. For those who have not worked with it before, it is an option worth looking into. V&S
Stable, broad spectrum illumination and dealing with the effects of degradation of phosphors and other lighting components.
Proper shielding from the effects of ambient lighting variations.
Strict environmental controls to minimize temperature dependent drift of lighting, camera and other electronics.
Color machine vision has two principal uses. The first is to verify that the correct color product is being produced; the second is to use color properties for object recognition. In the latter case, it is not the colors themselves that are of interest, but identification of the object that generates them.
Quite a few recent articles and webinars have covered the basics of color machine vision. These usually discuss methods well suited for color measurements to verify that the correct color product is being produced. These methods also can be used for identification of single colored objects; however, they are usually impractical for multicolored objects and those where the region being inspected may include complex boundaries. Common methods of color-based recognition (CBR), including ones that make it easy to recognize multicolored objects or those with complex boundaries, are the focus here.
This figure shows a simple example of modern color-based recognition (CBR) applied to solder joint inspection. Figures A and B show two classes of joints, unsoldered and soldered. The presence of the solder significantly alters the colors and their proportions in a small region around the crimp connector opening. By simply showing the CBR system examples of each class of joints in that region it learns to distinguish these classes based on the proportions of each color present. Figures C and D show the region where both training and inspection take place and where the results are displayed; red indicates regions interpreted by the system as unsoldered, green indicates soldered. In addition to the on-screen display the system provides digital output for process control. Source: WAY-2C Color Machine Vision
Types of Recognition Applications
There are two basic types of recognition applications: classification and anomaly detection. For classification, the system must decide between a known set of possible reference classes, for example a red apple, a green apple, a rotten apple or no apple at all. Once trained on each class, the system must reliably assign test items it has not seen before to the appropriate reference class. For anomaly detection the system is trained on examples of only one reference class; it must decide whether or not each test item belongs to that class.For simplicity this will focus on classification, however, the methods involved generally cover both types of recognition.
Traditional Single-Color Approaches to Recognition
Even when items of interest are single colored, the same is seldom true of their images. Texture, topography, positioning and lighting gradients all contribute to broadening of the range of colors that fall on the image sensor.To cope with the broadening, it is traditional to transform color coordinates from RGB, native to cameras and image monitors, into HSI (hue, saturation, intensity), LAB color space (color-opponent space with dimension L for lightness and A and B for the color-opponent dimensions) or other coordinates that conform more closely to human’s perceptions of color. The transformation usually makes it easier to set color boundary thresholds because hue tends to remain relatively constant as lighting details vary. Systems using this approach are often demonstrated sorting such items as brightly colored M&M candies or other high-contrast single colored items.
Extensive discussions of the coordinate transformation and threshold approach appear in textbooks and system suppliers’ literature. It is both intuitively appealing and appropriate for differentiating simple single-colored items. However, system engineering and maintenance costs can quickly become overwhelming when one tries to use it for recognizing multicolored objects or those with complex shapes or unpredictable orientation.
Another approach well suited to high speed sorting of items with limited acceptable color range is to assign each possible color to a specific class of interest. Although this approach can be time consuming to train and is best implemented in specialized hardware, it can be extremely fast and efficient for detection of imperfections in high throughput sorting.
This article will focus on methods specifically designed for generalized color-based recognition. By abandoning the assumption of single colored items and the need to report actual color values, these systems gain the ability to provide robust and cost-effective recognition. While not widely known, the methods involved have a long record of accomplishment in industrial applications where they have demonstrated recognition accuracy rivaling that of humans but much faster and more reliable over extended periods.
The Multicolor World
Traditional recognition methods are based on the assumption that items of interest are distinguishable in their images by a unique set of pixel colors, usually colors that are closely clustered in hue space. If the colors are not unique, so that the color distributions of the classes to be distinguished overlap or are hard to separate with well-defined thresholds, the traditional approaches can experience significant problems.Other factors, such as effects of an imager’s Bayer pattern, effects of pixels falling on item boundaries or uncertainties about the boundaries themselves can increase the likelihood of overlap among the color distributions of the different classes, whether single colored or multicolored.
The bottom line is that for all but a very few color-based recognition problems involving simple boundaries, contrasting colors and carefully designed lighting, the assumptions on which the traditional CBR methods are based rarely hold.
Abandoning the Single-Color Approximation
Rather than relying on the simple, single-color model, modern CBR systems assume from the start that items of interest may be characterized by a virtually unlimited combination of colors. They also assume that some, if not all, colors may appear in more than one class; only the proportions of the colors are diagnostic.These systems use powerful concepts from information theory, involving hundreds of parameters, to characterize the color distributions of each reference class. The concepts include well-defined methods for classification and anomaly detection. By comparing the color distribution statistics of an unknown object with the statistics for each reference class, the systems can find the reference class most likely to have been the source of the color pattern captured by the camera.
In the case of anomaly detection, the theory provides a comprehensive measure of the anomalousness related to the degree to which a test distribution falls inside or outside the reference distribution.
Implementation in a commercial software package requires substantial expertise and investment by the original developers. However, in well-designed packages all the underlying complexity is hidden. Training by example, much as one would train a human inspector, means little special user knowledge is required. It becomes no more difficult to train on, and recognize, items with the most complex color distributions than to do the same for the simplest single-colored item.
Though these images might not appear relevant to quality inspection, they do illustrate the use of modern color-based recognition (CBR) to locate complex colored items against a complex colored background, a concept applicable to many inspection and vision-guided robotics tasks. Here the user has specified there are only two classes of interest, apples and background; both are multicolored. Figure B shows a completely processed version of the results of Figure A using 40 by 40 pixel regions. First each region was classified; regions most resembling the apple reference were marked in white, those most resembling background were marked in black. Then all white blobs in the image were located, sorted by size and the blob with the largest area marked in red. Source: WAY-2C Color Machine Vision
Modern CBR Characteristics
Modern CBR systems are available both for PCs and smart cameras. Although they may offer the ability to return mean numerical color coordinates for a pixel or region, this ability is rarely useful. Instead they are distinguished by intuitive training by example and identifications that tend to closely match those of human inspectors. The combination can result in advantages in reduced engineering, equipment and operating costs together with improved performance.Engineering cost reduction comes primarily from system simplification. The general nature of the method used virtually eliminates the need for color space conversions, special algorithm development, color picking and threshold setting. Requirements of lighting uniformity are often significantly reduced, and glints and shadows may be an aid instead of a hindrance in some applications.
Modern information theory based CBR is equally at home sorting based on large color contrasts or distinguishing subtle color shades near the limits of human perception. The powerful “looks most like” matching can greatly reduce or eliminate the need for equipment to accurately position items to be identified.
Reduced setup and system training time can significantly reduce operator costs. Even manual, train-by-show training is fast, and automated training or retraining for a single class of object may take only a small fraction of a second; inspection is even faster. Even for such complex inspections as fabric sample cards containing 30 or more different swatches, one-time setup may require less than an hour of off-line assistant time, pre-inspection on-line training less than three minutes of operator time. Actual inspection rates can be as high as several cards per second, limited usually by the time necessary to move the sample into position.
When to Consider CBR
CBR should be considered for:Color Image Requirements
Up to this point the discussion has focused on methods available to identify objects from their color images. However, whether one is concerned with color verification or color-based recognition, it is important to note that both require consistent stable imaging systems. The more stable the image, the less often retraining is required. The subtler the differences between objects to be distinguished, the more stringent the need for image acquisition system stability.Experienced integrators of color vision systems are familiar with the need for:
Newcomers to machine vision would do well to consult with such an integrator, at least for advice on lighting and imaging. By heeding that advice and devoting the necessary resources to creating and maintaining the appropriate inspection environment, one can expect a high level of success.
Some Words of Caution
The history of color machine vision is filled with unkept promises, and one should take marketing claims with a grain of salt. That being said, do not dismiss any vision system supplier’s claims out of hand. Most have some unique technologies to offer; some may be better suited to the application than others.Ask each potential supplier to demonstrate how their approach will meet requirements using a representative set of samples or sample images that have been chosen.
If possible, pick these samples so they can serve as the basis of system acceptance tests. For simple applications there may be little or no charge for demonstrations, particularly if they can be done at the supplier’s site. For applications that are more complex or if the demonstration must be performed at the customer’s facilities, a modest charge may be involved.
Color-based recognition makes it easy to recognize multicolored objects and those with complex boundaries. For those who have not worked with it before, it is an option worth looking into. V&S
Tech Tips
Experienced integrators of color vision systems are familiar with the need for:Looking for a reprint of this article?
From high-res PDFs to custom plaques, order your copy today!
