Profilometry for the measurement of three-dimensional object shape using radial basis function, and multi-layer
perceptron neural networks
Dinesh Ganotra, Joby Joseph, Kehar Singh*
Photonics group, Physics Department, Indian Institute of Technology - Delhi, New Delhi 110 016, India Received 24 January 2002; received in revised form 10 June 2002; accepted 18 June 2002
Abstract
Neural networks have been used to carryout calibration process in fringe projection profilometry for the mea- surement of three-dimensional object shape. The calibration procedure uses several calibration planes whose positions in space are known. Radial basis function based networks and multi-layer perceptron networks are investigated for the phase recovery. Preliminary studies are also presented for the direct reconstruction of the object without the use of the intermediate step of phase plane calculations. Experimental results are presented for diffuse objects.
Keywords: Neural networks; Profilometry; 3-D shape measurement; Phase measurement
1. Introduction
In manufacturing applications, non-contact, automated surface measurement is a desirable al- ternative to micrometers and calipers. Fringe pro- jection techniques may be used to recover depth range data, other methods being Moire topography [1], laser triangulation measurement [2], stereo dis- parity [3], texture gradient [4], occlusion [5], etc. No one method seems clearly superior to the rest. Jarvis [6] and Marshall and Gilby [7] have discussed the performance of non-contact 3D data capture sys-
tems. All methods appear to have drawbacks which fall in one or more of the following categories:
missing part problem, computational complexity, time-consuming in improvement of signal/noise ratio, limited indoor application, limited to highly textured or line structured scenes, limited surface orientation, and limited spatial resolution. The use of projection gratings for the measurement is an easy and efficient way to characterize three-dimen- sional information because the grating is phases- modulated according to the topography of the ob- ject. The method has advantage of being contact free. It has applications in machine vision for au- tomated manufacturing, component quality con- trol, biomedicine, robotics, solid modeling, etc.
In fringe projection profilometry a grating (Ronchi or sinusoidal) is projected onto the object
under study. It is observed from a direction dif- ferent from that of the projector, and a phase- modulated fringe grating is recorded. There are several techniques to demodulate the phase from the images of the projected grating. It can be es- timated using conventional demodulation tech- niques such as phase-shifting [8], phase-locked loop (PLL) [9], spatial synchronous detection [10], and Fourier transform profilometry [11]. The phase-to-depth conversion requires the physical parameters and optical aberrations of the system to be known.
Calibration process in optical system often poses problems. Sometimes in dynamical robot vision systems, parameters such as focal distance of the video camera being used, reference plane position, grating frequency, distance and angle between the camera and the projector, are un- known, making the calibration task difficult.
Some experimental factors arise when the object under analysis is located near the projector- camera system. These factors include diverging fringe projection camera perspective distortion and the use of crossed-optical axes geometry.
Problems like camera calibrations, perspective distortions and false matches, delineations, and geometrical consequences of substantially com- pressed images in photogrammetry are discussed in [12]. Cuevas et al. [13] have presented a method to calculate the object depth distribution from the demodulated phase by the use of radial basis functions (RBFs) based neural networks.
Input to RBFs based neural network is the phase distribution from a set of planes inside the region where the depth data are being measured. The network operates as an interpo- lation function of phase values to the object height in places other than the calibrating planes. Cuevas et al. [14] in another application used multi-layer neural network for such cali- brations, by sampling the fringe pattern with a 5 x 5 pixel window. Set of such patterns and height directional gradients (calculated from pixels within that window) were used as training set to the multi-layer perceptron neural network.
They determined the depth of the object using path independent integration of the phase re- covered using neural networks.
In the present work we have investigated RBF [15], and multi-layer perceptron neural network models using Fourier transform and PLL phase recovery methods for object depth recovery. We have also shown that it is possible to recover the depth information by training the neural network with one spatial period of the projected grating over the object and a reference plane as the ele- ments of the input vectors.
2. The fringe profilometry
In fringe profilometry a grating pattern is pro- jected on a 3-D surface. The deformed grating image intensity distribution modulated by height distribution is captured with a CCD camera. Let f be the spatial frequency of the projected Ronchi grating and /0 and / respectively the phases of the reference plane and object. The images of the grating for the reference plane and object can be written [11,16] respectively as
go(x,y) = ro(x,y) J^ An exp{i«[2n/x + 4>0(x,y)]},
n=—oo
(1)
n=oo
g{x,y) = r(x,y) J^ Anexp {in[2nfx + (f>(x,y)]}.
n=—oo
(2) Here rðx;yÞ, and r0ðx;yÞ are non-uniform distribu- tions of reflectivity on the diffuse object and on the reference plane respectively and An are the Fourier coefficients for the projected Ronchi grating.
2.1. Phase detection using Fourier transform profil- ometry (FTP)
FTP can work for both Ronchi grating, and sinusoidal grating projection. The deformed fringe pattern is Fourier transformed and processed in its spatial frequency domain to demodulate the object shape from the fundamental frequency component in the Fourier spectra. With suitable filtering, the spectra are filtered to let only the fundamental component (n = 1) pass through. The reference and object planes after the filtering and inverse Fourier transform are written respectively as
go(x,y) =Air(x,y)exp{i[2nfx + 4>o(x,y)}}, (3) g(x,y) =A1r(x,y)exp{i[2nfx + <f>(x,y)}}. (4) The phase change due to the presence of the object with respect to the reference plane is thus
y) = 4>(x,y)-U^y)- (5)
grating
This can be recovered using the above two filtered functions as
y) = Im(log(g(x,y)-g*0(x,y))y (6)
The depth change can be calculated from the phase change using the transformation
2pfd1 (7)
where d0 and d1 are the distances as shown in Fig.
1. The phase on point 'O' is same as on point 'A'.
Point B and A are imaged by same sensor on CCD. Sensor plane and reference planes are con- jugate to each other.
If instead of a Ronchi grating a sinusoidal grating is projected on the object, then the mod- ulated image of the grating as seen by the imaging system can be expressed as
g(x,y) = a(x,y)+b(x,y) cos (<j)(x,y)), (9) where aðx;yÞ and bðx;yÞ are respectively the background intensity and fringe contrast obtained by the system. A quasi-sine grating can be gener- ated from a Ronchi grating by projecting a defo- cused Ronchi grating on the object surface [17].
Two-dimensional Fourier transform and 2-D Hanning filtering can be applied to provide a better separation of the height information from noise when speckle-like structure and discontinu- ities exist in the fringe pattern. FTP cannot work well for measurements of steep object slopes, and step discontinuities. Also it needs high-resolution imaging systems.
2.2. Phase detection using phase-locked loop This technique is based on first order digital PLL. In phase-locked detection, the signal is pas- sed through a low pass filter. The phase difference is assumed small enough to consider the linear approximation valid. Cuevas et al. [13] have used a
reference plane
object
(a)
grating
reference plane
A..---"
\ object
das
(b)
Fig. 1. (a) Experimental setup. (b) Schematic diagram showing points A, B, O and distances d0 and d1.
PLL based phase recovery method using the equations below.
A forward calculation along the x-axis from left to right is accomplished by the following updating rule:
\,y) = <j>{x,y)-x[g{x+\,y)
ð10Þ A second iteration is required in a backward di- rection through the same line to recover incorrect phase values due to setting time in the dynamic system
4>(x,y) = 4>(x+l,y)-z[g(x+ \,y)
^(x+l,y)}. (11) In this technique an appropriate value of the gain loop parameter s is required, which occa-
sionally is difficult to determine with wide spec- trum and noisy image of the grating.
2.3. Other techniques for phase detection
Some of the other techniques for phase detec- tion from the images of the projected grating are spatial synchronous detection, and discrete phase- shifting technique. In our simulations and experi- ments, we have restricted ourselves to the above two techniques, i.e. Fourier transform profilome- try and phase detection using PPL.
2.3.1. Spatial synchronous detection
In spatial synchronous detection [10] the phase can be retrieved assuming that the height information varies slowly as compared with the phase carrier. The phase is calculated by first multiplying the fringe pattern by the sine and then cosine of the carrier signal. Then a low pass filter is applied to both signals to remove the high frequency terms. The ratio of these two signals is then the tangent of the modulated phase. The detected phase given by this ratio is 2p wrapped.
! [ h{x,y) * [g{x,y) sin(27i/x)] 1 4>(x,y) = tan
h(x,y)* \g(x,y) cosð2pfxÞ;
ð12Þ where * denotes the convolution and hðx;yÞ transfer function for a low pass filter.
2.3.2. Discrete phase-shifting technique
Upon translation of the original grating by a fraction 1=N of its period p, the phase of the pat- tern represented by Eq. (9) is shifted [6] by 2p=N.
A new intensity function In ðx; yÞ is observed. Using N intensity functions (N = 3; 5;...), the phase / can be retrieved using
<j){x,y) = tan- i ð13Þ
The selection of an odd number of phase is advantageous and should not be compromised under any circumstances. Here also the phase is wrapped in its principal value — p to p. The slope of height variation of the measured object
must be limited to prevent spectral components from overlapping when filtering in frequency domain.
The phase-shifting phase-measuring profilome- try gets its accuracy from the high spatial sampling rate of the evaluated profile. This sampling rate is equal to N times the number of pixels per frame of the high-resolution image of the projected and captured grating, where N is the number of phase steps used in an N-step phase-shifting algorithm.
The N-step phase-shifting algorithm is inherently free of errors only if combined with sinusoidal patterns.
2.4. Phase unwrapping
Most of the phase detecting formulae use arctan functions to calculate the phase (Eqs. (12) and (13)). The arctan function has a range between — p and p. Even FTP gives the phase wrapped in this range. Thus the phases calculated are wrapped in this 2p range. The traditional approach to phase unwrapping is by removal of discontinuities. To reconstitute the continuous phase, the discontinu- ities are searched sequentially. Discontinuous fringe patterns are not handled even when phase is recovered using PLL properly, because the PLL may go out of the lock creating unwanted tran- sients in the recovered phase [9]. Noise generated phase inconsistencies may be removed by iterative PLL [18]. Here the unknown phase, which modu- lates the fringe pattern is modified in order to de- modulate the fringe pattern iteratively. For the successive iteration, the estimated phase found in the previous iteration is used as a new reference.
After each iteration the constant s may be reduced to improve the signal-to-noise ratio in the esti- mated phase. For unwrapping extremely noisy phase maps, the regularized phase unwrapping technique presented by Servin et al. [19] can be used. Here two phase-shifted fringe patterns are obtained from the unwrapped phase. Phase and frequencies are sequentially adapted to optimise a cost function. It becomes almost impossible to distinguish between discontinuities, which are genuine surface discontinuities and the disconti- nuities due to the limited range of the arctan function. Other problems that lead to failure in
phase unwrapping are discontinuous phase jumps, high-frequency high amplitude noise of region under sampling of the signal, shadow, speckle- noise, and height discontinuity.
3. p-phase shifting for reflectivity independent measurements
In most of the profilometry experiments, the object is assumed to be a perfect diffuser and the changes in the reflectivity 'aðx;yÞ' are neglected.
When the measured object is more complicated, with sharper rises and falls, the zero component resulting from aðx;yÞ cannot be neglected (see Eq. (14)). To eliminate the influence of aðx;yÞ, a grating p phase shift technique [20] is em- ployed. In this technique the grating is moved half of the grating period along the x-axis while the other conditions remain unchanged. Thus the optical field is sampled twice. With the use of a spatial light modulator (SLM), by using a con- trast reversed image of the grating, mechanical shifting to introduce p phase shifting can be avoided.
Let the two images be
gi(x,y) = a(x,y) + b(x,y) cos {Infox + <f>(x,y)), ð14Þ gi{x,y) = a(x,y) + b(x,y) cos {2nf0x + 4>(x,y) + n).
ð15Þ Subtracting Eq. (15) from Eq. (14), we get g(x,y) =2b(x,y)cos(2nf0x+4>(x,y)). (16) Thus by using this p phase-shifting technique, gðx;yÞ becomes independent of aðx;yÞ and also the search for the first order spectrum in the Fourier transformation of the above becomes easier.
4. Experiment
In our experiments we have used objects of size less than 6 mm in height and a laser has been used as the light source. An electrically addressed trans- mission type SLM (Jenoptik model SLM-M) has been used to project the grating. Though laser light source introduces speckle noise, it was satisfactory for comparison of one process/network from an-
Neural
Network Low Passfilter
Fig. 2. Modified multi-layer perceptron network for direct object reconstruction, using fringes.
other. Pulnix TMC-76 camera was used to capture the images.
A square wave Ronchi grating can be conve- niently programmed and displayed in transmission type SLM. The grating periods as well as grating shifts are similarly accomplished by appropriate computer control. The CCD captures four pairs of images. The two images within a pair are p spatial phase shifted with respect to each other. Three pairs are of three reference planes and one pair for the object. The image within the pairs are subtracted from each other using Eqs. (14) and (15) to get re- flectivity independent images (Eq. (16)). The phase difference of the reference planes and the object with respect to one of the above three planes are calcu- lated using FTP and also using PLL method. The lowest plane was chosen as the reference plane and the rest of the planes' heights were calculated with respect to it. The differences between the heights of the planes were kept typically 1-2 mm.
5. Neural network models used for phase to depth conversion
A neural network is a multivariate nonlinear mapping. Once the network has been trained, then it can interpolate the intermediate vectors and even extrapolate to some extent. We have used RBFs based networks and multi-layer perceptron network to calculate the height when the phase information is given.
5. 1. Radial basis functions based network
RBFs based neural network calculate the depth of the object by adjusting weights of a linear com- bination of Gaussian functions using the experi- mental reference planes fixing the centers of the RBFs over samples of detected phases from cali- bration planes. RBFs based network is compara- tively less time consuming, as the learning process does not require any iteration. Two matrices are calculated and the inverse of one is multiplied with another. The details of the calculations are de- scribed by Cuevas et al. [13]. Though the network operates very fast (as it does not require any itera- tion) but numbers computed by such networks be-
come very large in magnitude. This leads to a floating-point overflow and often to nearly singular matrices. Their inverse is needed for calculations.
For training the RBFs based neural network, we have used the same equations as used by Cuevas et al. [13]. The difference being that their network uses PLL technique to calculate the phase change but we have included results using the FTP tech- nique to calculate the phase change.
5.2. Multi-layer perceptron network
Cuevas et al. [13] preferred a RBFs based neural network with Gaussian processor over a multi-
400
y-pixel x-pixel
£00
y-pix
(b)
Fig. 3. 3-D reconstructed object using (a) RBF based neural networks and (b) multi-layer perceptron neural network. For both reconstruction same set of reference planes were used.
Spatial period of projected fringes 13 pixels.
layer perceptron network with sigmoid processors and backpropagation training with gradient de- scent since the learning process requires very little time comparatively. We have included multi-layer perceptron network for comparison of one process with the other. Though these are just visual com- parisons of the reconstructed 3-D objects, but they still carry useful information.
Seventy-five phase values are sampled randomly from each phase plane as input to the network. Each sample is a three element input vector. Two elements are the indices of the matrix and the third the cor- responding phase. The desired output for each sample is the mechanically set height/depth of the
corresponding plane. Note that both inputs and outputs are re-scaled to the range (—1, þ1Þ. A three layer perceptron neural network with three input neurons, three hidden neurons and one output neuron is trained with the above set of input-output pairs. The output of the network for an arbitrarily shaped object is discussed in Section 6.
5.3. Multi-layer perceptron network for direct mapping from fringe patterns
In this method for direct mapping of depth from fringe patterns using multi-layer perceptron net- work, the phases of the object or of the calibration
VXt
v-pixel 0 Ci
(a)
2 IT
y-\:i<&\
(b)
Fig. 4. (a)-(c) Approximate profiles of the previously measured real objects used for 3-D reconstruction.
planes are not required to be calculated. Instead, the images were sampled horizontally for the number of pixels same as the period of the image of the grating. The computation of the phases from the image of the grating and then the height from the calculated phase is assumed to be a complicated nonlinear mapping, where neural networks seem to be a suitable choice. A neural network with number of input neurons equal to double the number of pixels in one period of the images of the grating is constructed. We have chosen one period of the projected grating as the inputs to the network. Two such periods from different planes carry sufficient information for the phase change. The neural net- works do this mapping on their own. The first half
elements of the input vector are the samples of one period length of the reference plane and the second half are the elements with the same indices but from a randomly chosen plane (see Fig. 2). The desired output to the neural network was chosen as the difference between the heights of the two planes.
6. Results and discussion
Different diffuse objects like fold of paper, chalks and ground glasses were imaged. They were shaped to have slopes with different gradients within 3-6 mm height. Figs. 3(a) and (b) show 3-D reconstructed object using RBFs based network
400
y-pixel y-pixel x-pixel
400
y-pixel <-pixel
Fig. 5. 3-D reconstructed object using (a) RBF based neural networks, (b) multi-layer perceptron neural network and (c) reference planes used for the above reconstruction. Spatial period of the projected fringes 10 pixels.
and multi-layer perceptron network respectively.
A profile of the previously measured real object used is shown in Fig. 4(a). Both figures are for the same object and have been reconstructed using same set of phase planes. It can be observed that RBFs based network gives discontinuities at the corners. These discontinuities were less severe in the multi-layer perceptron training based network.
The mean square error between Figs. 3(a) and 4(a) is 0.39 whereas between Figs. 4(a) and 3(b) it is 0.06. Fig. 5(a) is the reconstructed object using RBFs based network and Fig. 5(b) using multi- layer perceptron network, for the profile of the previously measured real object shown in Fig.
4(b). The corresponding phase planes are shown in Fig. 5(c). There are three phase planes of the three reference planes shown in this figure along with the phases of the object. These figures are different from Figs. 3(a) and (b) in spatial frequency of the
projected grating. The mean square error (mse) between Figs. 4(b) and 5(a) is 0.54 whereas be- tween Figs. 4(b) and 5(b) it is 0.27. A plot of standard deviation of the differences in depth values previously measured object and the mea- sured values for different heights for the two net- works is shown in Fig. 8. The lower curve corresponds to heights measured using multi-layer perceptron based networks and the upper curve to heights measured using RBF based network. We have tried with the spatial periods from 8 pixels to 16 pixels. Figs. 3(a) and (b) had the spatial period of the projected grating as 13 pixels and Figs. 5(a) and (b) of 10 pixels.
A side view of the object Fig. 4(c), recon- structed using multi-layer perceptron algorithm, without filtering is shown in Fig. 6. It can be seen that though the complete profile of the object is not reconstructed, but variation in height can be
1 0 r
50 100 150 200
x-pixel
250 300 350
Fig. 6. Side view of reconstructed object (Fig. 4(c)) by direct reconstruction with fringe images and modified multi-layer perceptron algorithm (without filtering).
seen clearly from the base plane corrupted by the high frequency noise. There are a large number of factors whose effects can be studied to optimize the result, such that coming to definite conclusions in neural networks is very difficult. We attempted to filter these frequencies using Fourier transforma- tion but it worked only partially. The absolute
height calculated using the neural network was thus lost during filtering. Since the object retains the shape, we can choose some plane/height/shape within the object and re-scale accordingly. Our future studies will concentrate on improving this reconstruction. Fig. 7(a) shows the same object reconstructed using RBFs based network and the
400 300
y-pixel x-Dixel
8-, 6-, 4 - , 2 -.
•2^
-4,, - 6 ,
200 300
400
y-pixel x-pixel
Fig. 7. 3-D reconstructed object using (a) RBF based neural network and (b) direct mapping of depth from fringe patterns and multi- layer perceptron network.
Fig. 8. Plot of standard deviation between the depth values of the object and the measured values for various heights. The lower curve corresponds to multi-layer perceptron network calculated values and the upper curve for RBF based network.
same phase planes. Fig. 7(b) is the high frequency filtered image of the reconstructed object.
7. Conclusion
A multi-layer perceptron neural network seems to be promising tool in fringe projection profilom- etry. The calibration procedure using several planes whose positions in space are known can be used to train the neural network. RBFs based networks and multi-layer perceptron networks are studied for the phase recovery. Figs. 3(a) and (b) are reconstructed from the same set of fringe images. They belong to the same object whose sketch is shown in Fig. 4(a).
The axes and viewpoints of all the three pseudo 3-D plots are kept same for comparison. It can be easily seen that Fig. 3(b), which is reconstructed using multi-layer perceptron neural network resembles more closely Fig. 4(a) as compared to Fig. 3(a).
Similarly, Fig. 5(b), which is reconstructed using multi-layer perceptron neural network resembles more closely Fig. 4(b) as compared to Fig. 5(a). A comparison of the mse for these figures also con- firms this. Thus we find that multi-layer perceptron neural networks perform better. Preliminary studies presented for the direct reconstruction of the object
without the use of the intermediate step of phase plane calculations also show that it is possible to get depth/height information directly. The results are found to be encouraging and require further inves- tigations.
Acknowledgements
One of the authors, D. Ganotra, acknowledges financial support from the Council of Scientific and Industrial Research (CSIR) Government of India.
References
[1] D.M. Meadows, W.O. Johnson, J.B. Allen, Appl. Opt. 9 (1970) 942.
[2] X.-X. Cheng, X.-Y. Su, L.-R. Guo, Appl. Opt. 30 (1991) 1274.
[3] M.D. Levine, D.A. O'handley, G.M. Yagi, Comput.
Graph. Image Process. 2 (1973) 134.
[4] R. Bajcsy, L. Lieberman, Comput. Graph. Image Process.
5 (1976) 52.
[5] A. Rosenfeld, R.A. Hummel, S.W. Zucker, IEEE Trans.
Syst. Man Cybern. SMC-6 (1976) 420.
[6] R.A. Jarvis, IEEE Trans. PAMI-5 (1983) 122.
[7] S.J. Marshall, J.H. Gilby, New opportunities in non- contact 3D measurement, in: Proceedings of the National Measurement Conference 2001, Harrogate, UK.
[8] V. Srinivasan, H.C. Liu, M. Halioua, Appl. Opt. 23 (1984) 3105.
[9] M. Servin, R. Rodriguez-Vera, J. Mod. Opt. 40 (1993) 2087.
[10] S. Tang, Y.Y. Hung, Appl. Opt. 29 (1990) 3012.
[11] M. Takeda, K. Mutoh, Appl. Opt. 22 (1983) 3977.
[12] D.M. McKeown, I.J. Dowman, Proc. SPIE 2486 (1995).
[13] F.J. Cuevas, M. Servin, R. Rodriguez-Vera, Opt. Com- mun. 163 (1999) 270.
[14] F.J. Cuevas, M. Servin, O.N. Stavroudis, R. Rodriguez- Vera, Opt. Commun. 181 (2000) 239.
[15] S. Haykin, Neural Networks; A Comprehensive Founda- tion, MacMillan, NJ, 1994.
[16] X. Su, W. Chen, Opt. Lasers Eng. 35 (2001) 263.
[17] X.-Y. Su, W.-S. Zhou, G. von Bally, D. Vukicevic, Opt.
Commun. 94 (1992) 561.
[18] M. Servin, R. Rodriguez-Vera, D. Malacara, Opt. Lasers Eng. 23 (1995) 355.
[19] M. Servin, F.J. Cuevas, D. Malacara, J.L. Marroquin, R.
Rodriguez-Vera, Appl. Opt. 38 (1999) 1934.
[20] J. Li, X.-Y. Su, L.-R. Guo, Opt. Eng. 29 (1990) 1439.
