Quality Magazine

Cameras Get Smart

May 1, 2007
Some smart cameras such as the Sony XCI can work with third-party software such as the ActivVisionTools from MVTec Software. Source: Sony


The term smart camera has been somewhat of a misnomer. It is true that it is a camera, but it has not necessarily been smart. But that is changing as the cameras of the past, which could only work at less than optimal feed speeds and do less than complete image analysis, have evolved into a powerful quality assurance tool.

Today’s smart cameras can perform imaging tasks from the most mundane and repetitive to the most critical and complex. Most experts feel that they are capable of doing 70 to 90% of the machine vision tasks that a PC-based vision system can do.

The Iris P-Series is a line of smart cameras that incorporate a powerful embedded Intel architecture processor and Windows CE .NET real-time operating system. Source: Matrox Imaging

These systems’ capabilities range from a simple presence and absence function-yes, it is there; no, it is not-to gaging functions, to sophisticated analysis using powerful algorithms and vision application routines such as blog analysis, pattern recognition, optical character recognition, ID reading, surface inspection and geometric pattern matching.

In addition to the individual quality assurance and metrology functions, most of the smart cameras can simultaneously do multiple measurements.

Smart cameras are being run on a variety of operating systems including Linux and Windows, as well as DSPs, MS-DOS and proprietary systems. Some companies’ cameras can be run on multiple systems such as Sony that can run on both Linux and Windows XPe.

Input/output functions also have improved with these smart cameras. They can be tied into PLCs for closed-looped feedback functions, or networked with multiple smart cameras. Data sets can be transferred over serial links or TCP/IP-enabled Ethernet interfaces.

John Salls, president of Vision ICS (Woodbury, MN), a machine vision integration and consulting company, began working with DVT smart cameras more than a decade ago and still works extensively with DVT products, says that the capabilities of smart cameras had been limited by a lack of processing power. In the past, comparing PC-based vision systems to smart camera systems, the smart camera could only do about 5 to 10% of the jobs, but he now puts that figure at greater than 70%. “What has really changed,” he says, “are the processors that are available.”

He says that there are only a few types of tasks that a smart camera still cannot do, that still need the processing power of the PC. For instance, he says, there is no counterpart for inspecting “mouse bites”-flaws in metal wires on integrated circuit chips. “Smart cameras are not good at that, but the capabilities are starting to be there,” Salls says. “That kind of technology, object location, 50th of a pixel accuracy, that is coming.”

The top right hook does not fall within accepted tolerances when compared to the template (left hook). The bottom right hook passes the inspection. Source: Matrox Imaging

Building Block

The history of smart cameras-those devices with built-in sensors, image acquisition cards, onboard memory, operating systems and processors-are tied to the growth of, or more accurately, the miniaturization of, the semiconductor component. These blazing fast chips such as FPGA, DSPs, and chips from Intel and Texas Instruments and others have boosted the processing power of a smart camera and in turn boosted the capabilities of the camera, says Joseph Christenson, president of PPT Vision Inc. (Eden Prairie, MN).

The top right hook does not fall within accepted tolerances when compared to the template (left hook). The bottom right hook passes the inspection. Source: Matrox Imaging

PPT Vision’s new A-10 smart camera features an FPGA (Field-Programmable Gate Array), a type of logic chip that can be programmed, and can process up to 700 MIPS (million instructions per second) and has onboard memory of 256 MB RAM; 256 MB Flash. It has a 1/3-inch CMOS sensor and works at up to 60 frames per second (fps).

Matrox Imaging’s (Dorval, Canada) Iris P-series line of smart cameras features an embedded Intel Celeron processor and operates in a Windows CE.Net format. It has effective resolution of 640 by 480 up to 1280 by 1024, and frame rates of between 7.5 and 100 fps.

The Sony smart camera XCI series, which is a hardware platform only, features a 400 Mhz Geode GX533 processor with an x86-compatible architecture that allows it to be run with multiple third-party software utilities.

The smart camera is used in a pharmaceutical inspection application. Source: Matrox Imaging

Basler AG’s (Ahrensburg, Germany) eXcite intelligent camera features a 64-bit MIPS processor that runs at 1 gigahertz (GHz). It is based on a CMOS sensor with VGA resolution with frame rates that vary in range from 60 to 176 fps. Its resolution ranges from 656 by 490 to 1623 by 1236.

The DVT 535C from Cognex is a color vision sensor with a resolution of 640 by 480 and 23 fps with a 1/3-inch CCD sensor. It has a set of color tools for sorting, color matching and defect detection.

Brake pads are conveyed under the 3-D smart camera. Source: Sick Inc.

Some smart cameras even offer power that may go beyond the needs of most machine-vision applications. The Lynx IPX-11M5-L smart camera from ImperX (Boca Raton, FL) has a resolution of 10.7 megapixels (4,000 by 2,672 pixels) and can zoom from low-resolution at video frame rates (200 lines at 30 fps) to ultra-high resolution (2,672 lines at 5 fps).

Feed speeds also are increasing because of the camera’s ability to do partial scanning and vertical and horizontal binning. Partial scans allow operators to select a region of interest to reduce the data size and as a consequence, increase frame rates. Binning functions, such as those found on Sony’s cameras, allows operators to combine image data. Vertical binning combines image data for every two lines vertically to increase frame rates. Horizontal binning combines image data for every two pixels horizontally, increasing sensitivity and shortening capture speed.

In this application, the software checks the presence and absence of pins and their location. Source: National Instruments

Ease of Use

Software is key to the improved capabilities of smart cameras. Some cameras are configurable, with point-and-clink functions, some are programmable in various programming languages and some do both. What is common among most smart cameras is its ease of use and ability to tap into powerful software utilities.

Software libraries such as the Matrox Imaging Library (MIL), National Instrument’s Vision Builder AI, and MVTec’s Halcon, can manipulate and analyze images throughout the process.

This image manipulation, or image processing, begins at the image acquisition stage and, if required, image processing is performed before running analysis routines. That is because the results of image acquisition can vary widely; Often images are taken in areas that have poor illumination, such as an assembly line. A manufacturing environment is not only noisy and dirty, it offers challenges like aligning the target object, centering the object for best imaging, and lighting it for best results. Because assembly line imaging is literally a moving target, imaging systems must shift images left, right, up, or down. And some parts produced in a factory can be problematic because of variation in gray values. The variation in gray values occurs because of light reflecting off the objects.

Multiple types of gears are inspected simultaneously. Source: National Instruments

According to a Matrox Imaging whitepaper, today’s software corrects images with functions for image filtering and morphology as well as sophisticated techniques such as convolution that smoothes or sharpens an image by replacing pixels by weighting the sum of itself and nearby pixels.

A smart camera’s capabilities are enhanced even more by using a vision library that contains sophisticated algorithms and vision inspection tools. These tools add enormous power to a smart camera such as the preprocessing functions, and real-time image analysis. The Matrox Imaging smart camera, for instance, uses the company’s MIL. This program, with more than 10 years of software developments and application algorithms built-in, allows operators with little experience in traditional programming environments to do vision inspection without having to write lines of code. MIL is a flowchart-based system that allows for programs to be developed using a variety of modules and tools such as the OCR module or the geometric module.

Using 3-D profiling offers the capability to see differences in height. In this example, a group of parts are imaged with a 2-D camera (left) and with a 3-D camera (right). The 3-D camera makes it easy to determine which part is on top. Source: Sick Inc.

Matrox Imaging Product Manager Fabio Perelli acknowledges that while processing power has given the smart camera market a boost, “the configurable software [point and click] approach has its limits because the manufacturer decides which functionality the user needs.” MIL, therefore, is a key component of Matrox Imaging’s smart-camera philosophy. The C-based library is built on more than 10 years of algorithm research and software development, and each module of the Matrox library contains scores of functions and parameters; for Matrox’s smart camera users, this means they have the flexibility to fine tune the software to their applications’s exact needs. A customer can use the Geometric Model Finder (feature-based pattern recognition) to locate the position of a gear and “then call the metrology functions to measure and ensure the teeth are 36 degrees apart,” says Perelli. A machine vision user programs the application on a desktop, compiles it, and has the Matrox Iris smart camera run the executable.

Other cameras have point-and-click functionality. PPT Vision’s Impact Inspection Builder software suite, available on its A-10 smart camera, allows for vision applications to be developed through a series of clickable icons such as blob analysis, contrast sensing, data matrix reading and character inspection, gaging and edge detection.

Using 3-D profiling offers the capability to see differences in height. In this example, a group of parts are imaged with a 2-D camera (left) and with a 3-D camera (right). The 3-D camera makes it easy to determine which part is on top. Source: Sick Inc.

Christenson of PPT Vision says that the Impact Software suite features three modules: the Impact KickStart, which has preconfigured setup panels that allow inspection applications to be developed in minutes; the Vision Program Manager (VPM), which has a selection of vision tools of comprehensive inspection requirements; and the Control Panel Manager (CPM) that allows for quick setup, storage and viewing of captured images through the entire application, even as inspection requirements change.

Cameras can run a variety of third-party software packages, and each adds capability to a camera. Basler’s eXcite cameras can use many different packages. So too can Sony’s new smart cameras. The XCI series runs a variety of software packages including National Instrument’s Vision Builder, MVTec’s ActivVisionTools and Euresys’ e-vision.

This model was developed using the 3-D smart camera. Source: Sick Inc.

MVTec’s ActivVisionTools is typical of these types of software. It has a number of modules such as quality assurance, surface inspection, blob analysis and metrology. An application example might be verifying a screw joint to determine if the end of the screw is at the same location within a mounting hole. Using a dark-field illumination, the upper part of the screw joint and the screw itself are highlighted to determine that the screw joint meets specifications.

Kyle Voosen, National Instrument’s (Austin, TX) vision product manager, says that there is no programming involved with Vision Builder. The software is entirely menu-driven to build a complete inspection application from acquiring to process, all the way to communication. Image processing includes 50 machine-vision functions such as pattern matching, OCR, gaging and reading bar codes. In addition, if the application requires something more than the standard offering, a LabView program can be written and incorporated into the package without having to buy the whole package. LabView is NI’s higher level programming environment.

The DVT 535C vision sensor from Cognex is an entry-level color vision sensor that uses Intellect software. Source: Cognex

Vision Builder is based on branching and looping functions. Branching is similar to if/then statements. In a particular application, the system will read a bar code or the data matrix, and that will determine what part is in front of the camera. The information it gleans from this will determine the appropriate inspection program. Looping helps to determine how often an inspection application runs. If the camera is presented multiple gears, it will first determine the number of gears and then run the program on each gear.

Operating in a dynamic environment is a pivotal aspect of using a smart camera with high-level program capabilities. Salls of Vision ICS believes that this is only going to continue. In fact, he says that in the next five to 10 years, smart cameras will have progressed to the point where “my services are no longer needed.”

sidebar: tech tips

  • The increase in computer power and software has given smart cameras a variety of new capabilities.
  • Smart cameras now can do quality control functions such as gaging, pattern recognition, optical character readings and blob analysis.
  • Many smart cameras can now use the power of third-party software platforms.