An automated IC chip marking inspection system for surface mounted devices on taping machines

*Author for correspondence

Tel: +886-6-59779566#5830; Fax: +886-6-5977510 E-mail: ttliao@cc.feu.edu.tw

An automated IC chip marking inspection system for surface mounted devices on taping machines

Shih-Hung Chen¹ and Te-Tan Liao²*

1Department of Automation and Control Engineering, ²Department of Mechanical Engineering, Far East University, No. 49, Chung Hua Road, Hsin-Shin, Tainan County 744, Taiwan

Received 20 August 2008; revised 17 February 2009; accepted 25 February 2009

This study presents a new automated system for inspecting markings on surface mounted devices (SMDs) prior to packaging.

In proposed design, marking region is identified using a normalized cross-correlation template-matching scheme. A multi- resolution pyramid image processing approach is used to enhance speed of search process. Target image is filtered using a hybrid digital logic filter (DLC) / mean and standard deviation gray scale (MSDGS) algorithm for noise filtering. Individual characters in marking are segmented and fed to a neural network for automatic recognition. DLC / MSDGS filtering scheme is found more straightforward and far more robust toward a noise filtering than conventional image processing schemes. System achieves a recognition rate of 99.14% with identifying each IC chip marking within 0.05 sec. System provides an ideal solution for real-time inspection of IC markings in high-throughput SMD packaging applications.

Keywords: Learning vector quantization, On-line inspection, Surface mounted device

Introduction

In manufacturing industries, conventional human inspection causes errors in long term and monotonous processing, besides inducing fatigue stress to human inspectors^1-6. In manual inspecting markings on IC chips, an incorrect decision on marking may result in inappropriate placement of chip on printed circuit board during assembly process. Thus automatic inspection of IC markings attracted considerable interest^6-10. In character recognition task, segmentation process is carried using one out of four different techniques (projection-based¹¹, pitch-based^12-14, recognition-based¹⁵ and region-based¹⁶). After character segmentation, character recognition is performed by feature-based projection method¹⁷, contour chain code method using neural network (NN)^18,19, distance function and template matching method²⁰, relaxation matching method²¹, and vector quantization and compression method²².

Present study describes a NN and automated vision- based system for inspecting IC markings on surface mounted devices (SMD) on a taping machine prior to packaging. Design objectives were: a) high rate of

recognition; b) high speed of recognition; and c) real- time inspection capability.

Proposed Automated IC Marking Inspection System (AICMIS)

Main components of AICMIS include a loading/

unloading mechanism, an image acquisition system, and a marking inspection system (Fig. 1). In inspection process, unpacked SMD assemblies are loaded in a single- line fashion on conveyor belt and are driven sequentially for passing beneath the image acquisition device. Once beneath this device, a frame grabber is triggered to acquire frames from CCD camera. These frames are then interfaced to a PC, which processes corresponding images and determines markings on IC chips. After correct marking, packaging can take place. However, if inspection system fails to determine marking or detects an incorrect (unanticipated) marking, inspection / packaging process is halted so that appropriate remedial action can be taken.

Marking Inspection Procedure

Inspection procedure (Fig. 2) comprises image acquisition, image filtering / character segregation, and automatic character recognition. In proposed approach, positional coordinates of target image are determined using a template matching process conducted using

normalized cross-correction (CC) algorithm^23,24. Speed of search procedure is enhanced by using of a multi- resolution pyramid image processing technique²⁵. In template matching process, correlation coefficient between source image and template image is formulated as

( ) ( ) ( )

( ) ( )

∑∑ ∑∑

∑∑

−

=

−

=

−

=

−

=

−

=

−

=

+ +

+ +

= h 1

0 y'

1 w

0 x'

1 h

0 y'

1 w

0 x'

2 2

1 h

0 y'

1 w

0 x'

y' y , x' x I y'

, x' T

y' y , x' x I y' , x' T y

x, R

~

~

~

~

~

.

…(1) where, T~(x',y') is a template image comprising w×h pixels, and is given as

(

x',y'

)

T

(

x',y'

)

T

T~ = −

…(2) where T

(

x',y'

)

is gray scale of pixel located at coordinates (x,y) and _T is mean gray scale of pixels within template image. Difference between gray scale and mean gray scale is given as

(

x x',y y'

) (

I x x',y y'

)

I

~I + + = + + − …(3)

In object recognition, using normalized CC algorithm, an instance of a small reference template is located in

source image by sliding template window over source image on a pixel-by-pixel basis, and computing correlation coefficient between template window and source image at each position. Having scanned entire source image, window position of maximum correlation coefficient is judged to represent position of searched object within source image. A pyramid image processing technique is used to enhance accuracy and efficiency of CC search process performed in AICMIS.

Template matching process is performed initially at a coarse level of resolution in order to locate approximate

Load/Unload mechanism

Stepping Motor Torque Motor

Tray(1) Tray(2)

Conveyor Computer

Marking inspection system

Light Source Monitor

CCD Camera (JAI CV-M10BX) Image Acquisition device

Frame grabber Card (Matrox Meteor II)

MCU Board RS232

Photo sensor Hardware:

1.Pentium III 1GHz /256MB RAM SoftWare:

1.Windows XP 2.Open CV 3.MIL-7.1 LITE 4.Borland C++ Builder 6.0

CCTV Lens

Image Filter Thresholding Contour Search

Image Segmentation for

Character Image Zoom

Character Recognition

Output Find the target

image Grab for source

image

Fig. 1—Surface mounted device (SMD) IC chip marking inspection system

Fig. 2—Block diagram showing major data processing steps in marking inspection process

image (Fig. 1) of SMD assembly is acquired by CCD camera, transformed into a corresponding digital signal, sent to image card, and then input to inspection system software installed on PC. However, image captured by CCD device is subject to various sources of noise, like focusing errors, variations in light source intensity, stray illuminance from interfering light sources, strong reflections from cover tape on carrier tape, and so forth.

Effects of these noise sources on target image are analogous to effects of electrical and magnetic fields on signals input to DLC. Thus, present target image filtering process is based on noise filtering approach implemented in conventional DLCs. From an inspection of Fig. 3, following generic image filtering equation can be inferred:

V

OH

p if min value

p

= < …(5)

where p is gray scale of any pixel in image and

V

OH ^is

a minimum threshold voltage value. This basic filtering technique provides a means of pre-classifying background regions of target image so as to improve performance of subsequent segmentation process.

Furthermore, filtering scheme improves robustness of segmentation process. In order to pre-classify background regions of target image, present study utilizes digital logic filtering scheme (Fig. 4) as

p 0.48 p

if 0

p

= < …(6)

where

p

is mean gray scale value and 0.48 is a constant whose value is calculated in accordance with Transistor- Transistor Logic (TTL) design standard over a 0~5 V coordinates of target image and search procedure is

repeated at original image resolution (1.2 times higher than that used in coarse search) in order to establish precise location of target image.

Target image is filtered prior to its segmentation in order to improve quality of segmentation process, thereby yielding an effective improvement in detection performance of inspection system. Target images acquired in AICMIS presents a number of quite specific challenges. For example, reflection of image acquisition light source from cover tape on SMD assemblies causes IC image to be significantly brighter than background.

In addition, subtle differences may exist in color of background and text on various ICs attached to same roll of carrier tape. Finally, even within same IC chip, same character may have slightly different characteristics and / or defects. In AICMIS, these problems are resolved using filtering scheme (Fig. 2). Target image is supplied to a digital logic filter (DLF) designed to enhance contrast between background pixels in target image and marking pixels. Target image is further processed using a mean and standard deviation gray scale (MSDGS) algorithm to generate a binary version of target image.

Filtering method is not only simpler and more efficient than traditional image filtering schemes, but is also considerably more robust in noise filtering.

Input signals of digital logic circuits used throughout the electronics field are interfered by surrounding electrical or magnetic fields causing noise. If noise is too large, input to digital logic circuit (DLC) exceeds allowable voltage level (Fig. 3) and causes DLC to malfunction or yield an inappropriate response. Optical

Voltage Logical 1

Undefined Range VOH

VOL

Logical 0

p

0.48 xp

0 255

Gray scale

The Size of The Images-1 The Point of The Images

0

Set 0

Fig. 3—Allowable voltage range for digital logic

circuit input signal Fig. 4—Noise filtering theorem implemented using digital logic circuit analogy

range. In present digital signal processing scheme, gray scale image data are derived directly from voltage signal of CCD. However, CCD image inevitably contains a certain amount of noise, which introduces a margin of error into transformed gray scale image data.

Logical output of TTL is therefore set equal to one whenever TTL voltage exceeds 2.4 V as a consequence of noise. Linear relationship (Fig. 5) between voltage signal of TTL and gray scale image data can then be used to calculate threshold value of 0.48 (

0V 5V

2.4V

− ^).

In practice, filter described in Eq. (6) implies that any pixel within target image whose gray scale is less than 0.48 is automatically assigned a gray scale value of 0 (black). In this way, background regions of target image are readily distinguished from non-background regions.

DLF is applied to classify target image pixels as either background pixels or non-background pixels. An algorithm based on MSDGS values of target image pixels is applied to further enhance contrast between IC surface background and IC marking characters.

Mean gray scale,

p

, and standard deviation, s, can be expressed as

∑

=

= ⁿ

1 j

p

j

n

p 1

_…(7)

∑

=

− −

= ⁿ

1 j

2 j

2

(p p )

1 n

s 1

_…(8)

where n is total number of pixels in target image and p_j is gray scale of j-th pixel in target image. In accordance with Eqs (7) and (8), image produced by DLF is further filtered as







<

− +

≥

−

≤

=

Ns p p if P

Ns p p if max value

Ns p p if value min

p

…(9)

where N is multiplier of standard deviation. Having executed Eq. (9) for every pixel within target image, the image is corrected into binary accordance with



 >

=

0 otherwise threshold p

if max value

p

_…(10)

Character segmentation process commences by grabbing the region of target image containing IC marking in accordance with a user-defined set of coordinate data.

Each character within IC marking is then segmented using contour search method^26,27. In order to facilitate an on- line IC marking inspection capability, segmented characters within IC marking are identified using a NN.

However, before NN can be applied to recognize individual characters within IC marking, it is first trained.

Training process is performed using linear vector quantization (LVQ) scheme²¹.

Experiments and Results

Inspection system required an inspection time of < 1 sec per IC marking, and an ability to inspect 3 IC chips concurrently. Performance of inspection system was evaluated using IC chips marked with identifier, BTU00052S. Thus, marking comprises 7 different characters (B, T, U, S, 0, 5 and 2). During system set-up phase, NN was trained using 13 sample images for each of 7 different characters. System achieves a recognition rate of 99.14% (corresponding to just 6 misjudged characters) when implemented using DLF and 88%

(corresponding to 84 misjudged characters) without DLF.

DLF in image processing procedure yields a significant improvement in resolution of individual characters, and therefore, prompts a corresponding improvement in detection performance of NN. Source of detection errors in on-line inspection process can be eliminated using proposed technique as follows:

Character Defects

Individual characters within IC marking are invariably affected to a greater or less extent by a range of defects, including irregular line widths, varying lightness,

5V

2.4V

0V

p

0.48 x p

0 Volt Gray scale

Fig. 5—Correlation between voltage and gray scale value

incomplete or broken outlines, distortion, and so forth.

These defects not only vary from one IC marking to another, but also amongst identical characters within same marking. As a result, it is virtually impossible to develop a comprehensive set of rules with which to compensate for these errors during image recognition process. Selected images confirm effectiveness of DFL / MSDGS scheme in detecting complete contour of individual characters, thereby improving reliability of subsequent character recognition process.

Light Source Angle

Variations in angle of illuminating light source cause apparent background color of IC component to change.

Therefore, traditional image processing algorithms may fail to properly distinguish IC marking characters from background. However, current results suggest that DFL / MSDGS scheme improves resolution of individual characters and therefore renders detection performance more robust to illumination noise.

Reflection of Light Source From Cover Tape

A transparent tape covers carrier tape used in the packaging of surface mounted components. Any non- uniformity in reflective properties of this tape (scratches, creases, oil traces, etc.) will have a direct effect on characteristics of source image acquired by CCD camera.

In practice, cover tape has both advantages and disadvantages in terms of its impact on detection performance of inspection system. Localized reflection effects caused by defects in carrier tape introduce noise into grabbed image, whereas carrier tape also enhances color consistency of individual characters and therefore facilitates their separation from background during image processing procedure. Detection performance falls when cover tape is removed. Overall recognition rate still retains a relatively high value of 98.28%. With cover tape, overall recognition rate of 99.14% was achieved.

In on-line tests, recognition speed of inspection system, defined as time between moment at which source image was grabbed by CCD and moment at which an inspection result was obtained, was determined to be 0.15 sec (each image contain 3 separate IC components;

recognition time of each component, 0.05 sec). Since each IC marking comprised 9 characters, average character recognition time was found to be 0.006 sec.

These results compare extremely favorably with recognition speed of commercial detection systems, which generally require order of 1 sec to inspect markings on 3 ICs.

Conclusions

This study presented a new NN based automated inspection system for detecting markings of IC chips during carrier tape packaging of SMDs. In proposed approach, target images are processed using a combined DLF / MSDGS scheme prior to their segmentation and individual characters are then detected using a LVQ- trained NN. DLF / MSDGS filtering scheme is far simpler than conventional image processing algorithms, and yields a significant improvement in both detection time and detection performance of inspection system.

Experimental results have shown that inspection system is capable of detecting an IC marking with 9 characters in just 0.05 sec and achieves a maximum throughput of 10800 ICs / h in a practical test run. System attains a maximum recognition rate of 99.14%. AICMIS has a faster inspection speed and a comparable recognition rate to that of existing techniques and other commercial detection systems. As a result, it provides an ideal solution for on-line inspection of IC markings in commercial SMD taping systems.

Acknowledgment

Financial support provided by National Science Council of Taiwan under grant NSC96-2221-E-269-003 is gratefully acknowledged.

References

1 Wetherill G B, Sampling Inspection and Quality Control, 2^nd edn (Chapman and Hall, London) 1977, 106-107.

2 Freeman H, Machine Vision: Algorithms, Architectures, and Systems (Academic Press, Boston, USA) 1988, 109-148.

3 Batchelor B G, Hill D A & Hodgson D C, Automated Visual Inspection (IFS Publications Ltd, Kempston, Bedford, UK) 1985, 27-35.

4 Vernon D, Machine Vision: Automated Visual Inspection and Robot Vision (Prentice-Hall, New York) 1991, 118-123.

5 Nayar S K, Weiss L E, Simon D A & Sanderson A C, Structural highlight inspection of specular surfaces using extended Gaussian images, Proc of SPIE, Opt, Illumin, Image Sensing Mach Vis III, 1005 (1988) 235.

6 Newman T S & Jain A K, A survey of automated visual inspection, Comput Vis Image Underst, 61 (1995) 231-262.

7 Sablatnig R & Hansen C, Machine vision applications in industrial inspection III, Proc Int Soc Opt Eng, 2423 (1995) 356-366.

8 Cho C-S, Chung B-M & Park M-J, Development of real-time vision-based fabric inspection system, IEEE Trans Ind Electron, 52 (2005) 1073-1079.

9 Jiang B C, Wang Y M & Wang C C, Bootstrap sampling techniques applied to the PCB golden fingers defects

classification study, Int J Prod Res, 39 (2001) 2215-2230.

10 Tsai S L, Grey system theory applied to IC code recognition, MS dissertation (in Chinese), Yuan-Ze University, Taiwan, 1999.

11 Pervez A & Suen C Y, Computer recognition of totally unconstrained handwritten zip codes, Int J Pattern Recogn Artif Intell, 1 (1987) 1-15.

12 Ascher R N, Koppelman G M, Miller M J, Nagy G & Shelton G L, An interactive system for reading unformatted printer text, IEEE Trans Comput, 20 (1971) 1527-1543.

13 Hoffman R L & McCullough J W, Segmentation methods for recognition of machine-printed characters, IBM J Res Develop, 15 (1971) 153-165.

14 Miura T, Umeda T, Egami K, Nishijima Y, Tsuji Y, Tsuda K &

Tsukakoshi N, Development of OCR address reading/coding system, NEC R & D, (1985) 62-71.

15 Casey R G & Nagy G, Recursive segmentation and classification of composite character patterns, in Proc Sixth Int Conf Pattern Recogn (Munich, Germany) 1982, 1023-1026.

16 Hosking K H, A contour method for the recognition of hand printed characters, in Machine Perception of Patterns and Pictures (The Institute of Physics, London, England) 1972, 19- 27.

17 Chu J-U, Moon I, Lee Y-J, Kim S-K & Mun M-S, A supervised feature-projection-based real-time EMG pattern recognition for multifunction myoelectric hand control, IEEE/ASME Trans on Mechatronics, 12 (2007) 282-290.

18 Chiu S D, A study of an automatic IC inspection and recognition system, MS dissertation (in Chinese), Cheng-Kung University, Tainan, Taiwan, 1992.

19 Edinbarough I, Balderas R & Bose S, A vision and robot based on-line inspection monitoring system for electronic manufacturing, Comput Ind, 56 (2005) 986-996.

20 Jeng B S & Lin L K, Optical Chinese character recognition system based on a stability learning and a two stage classification, Tech Rep Telecommun Labs, 20 (1990) 85-93.

21 Lee H-J & Chen B, Recognition of handwritten Chinese character via short line segments, Pattern Recogn, 25 (1992) 543-552.

22 Gersho A & Gray R M, Vector Quantization and Signal Compression (Kluwer, Boston, MA) 1991.

23 Lewis J P, Fast template matching, Vision Interface, (1995) 120- 123.

24 Gonzalez R C & Woods R E, Digital Image Processing, 2nd edn (Prentice-Hall, Inc., New York) 2002.

25 Adelson et al, Pyramid methods in image processing, RCA Engr, 29 (6) (1984) 33-41.

26 Suzuki S & Abe K, Topological structural analysis of digitized binary image by border following, Computr Graphics Image Process, 30 (1985) 32-46.

27 Wu S-T & Mercedes R G M, A non-self-intersection Douglas- Peucker algorithm, in 16th Brazilian Symp on Computer Graphics and Image Process, Sao Carlos, Brazil, 12-15 October 2003, 60-66.