Research of Camera Calibration Based on DSP

To take advantage of the high-efficiency and stability of DSP in the data processing and the functions of OpenCV library, this study brought forward a scheme that camera calibration in DSP embedded system calibration. An arithmetic of camera calibration based on OpenCV is designed by analyzing the camera model and lens distortion. The transplantation of EMCV to DSP is completed and the arithmetic of camera calibration is migrated and optimized based on the CCS development environment and the DSP/BIOS system. On the premise of realizing calibration function, this arithmetic improves the efficiency of program execution and the precision of calibration and lays the foundation for further research of the visual location based on DSP embedded system.


INTRODUCTION
Camera calibration is one of the most important work of the stereo vision research.A complete stereo vision system usually can be divided into six parts, such as image acquisition, camera calibration, feature extraction, stereo matching, depth determined and interpolation (Forsyth and Ponce, 2003;Yin et al., 2007;Li and Ma, 2011).Camera calibration is the basis and prerequisite of stereoscopic vision research that its accuracy determines the accuracy of the stereo vision system directly.The mapping relationship of corresponding point between the image coordinate system and three-dimensional coordinate system is described by the imaging geometry model of camera, which determined by camera intrinsic parameters and extrinsic parameters and the process of determining these parameters is called camera calibration (Steger and Ulrich, 2008;Bradski and Kaehler, 2008;Zhang, 2000;Yu et al., 2009;Zhang et al., 2000).
With the development of integrated circuits and embedded technology, DSP promote the development of the field of image processing with its peculiar hardware multiplier, powerful data processing capabilities, high computing speed and computing precision.
OpenCV is an open source computer vision library which can realize many common algorithms for image processing and computer vision and have dedicated camera calibration module to provide users with a good interface.Meanwhile, EMCV ports for the embedded system improves constantly, which makes more and more function of OpenCV can be used on the DSP experiment platform.In this study, the migration of EMCV library to DSP is completed and the camera calibration based on the functions of OpenCV is realized, for the purpose of making use of the functions of OpenCV library and DSP processing technology.

CALIBRATION PRINCIPLE
Camera model: According to the relationship of the feature points between the image pixel coordinate and the world coordinate, the process of calculating camera intrinsic and extrinsic parameters in camera model is called camera calibration, therefore, it is necessary to choose the right camera model.A simple but useful model for camera calibration is the pinhole camera model.In this model, with the radial distortion and tangential distortion considered, the camera geometry model and lens distortion model are given, which define the intrinsic parameters of the camera.The coordinate of the point P is converted from (x w , y w , z w ) in the world coordinate system to (u, v) in the pixel coordinate system, the transformation of coordinate among four coordinate system is shown in Fig. 1.
There are four coordinates in this model.The world coordinate system (O w -X w Y w Z w ) is defined as a reference coordinate system defined in space.The origin of the camera coordinate system is in the optical center of camera and Z c -axis is the optical axis of the camera.The imaging plane is perpendicular to the optical axis of the retinal coordinate system (O o -XY) and the origin of the retinal coordinate system is on the optical axis, the distance between the origin of coordinate and the projective centre of camera is focal length f, the physical coordinate system measured by millimeter and the image pixel coordinate system (O p -UV) measured by pixel are in the imaging plane.The origin of the physical coordinate system is principle point, the origin of the image pixel coordinate system is the center of the pixel, which is in the first row and first column of the image and the coordinate of the principal point in the pixel coordinates system is (c x , c y ).Point P w in the world coordinate system is projected to point P u in the imaging plane through the projective center O c .
In the ideal case, point P, point P u and point O c are three collinear, however, the point P u is skewed to the point P d due to the lens distortion, which caused by the lens manufacturing and assembly accuracy of camera.

Coordinate transformation and the calculation of intrinsic and extrinsic parameters:
 According to the single camera model shown in Fig. 1, the process of coordinate transformation of any point P in the world coordinate system is as follows: o The rigid transformation between the point P w (x w , y w , z w ) in the world coordinate system and the point P c (x c , y c , z c ) in the camera coordinate system is carried out and it's expression in the form of homogeneous coordinate is as follows:   where, R = The 3-by-3 rotation matrix T = The translation vector o The projection transformation between the point P c in the camera coordinate system and the point P u (x u , y u ) in the physical coordinate system is carried out and it's expression in the form of homogeneous coordinate is as follows: Considering the radial distortion and tangential distortion of camera lens, the point P u is skewed to the point P d (x d , y d ), as follows: In the formula: , k 2 and k 3 are the radial distortion coefficients, p 1 and p 2 are the tangential distortion coefficients.o The coordinate transformation of the point P d between physics coordinate system and pixel coordinate system is presented and it's expression in terms of homogeneous coordinate is as follows: In the formula above: d x and d y are the physical size of each pixel in the X-axis and Y-axis, of which the unit is mm/pixel.In summary, the mapping of points on different coordinate systems is derived, of which the expression in the form of matrix multiplication is as follows: Therefore, there are four intrinsic parameters (f u , f v , c x , c v ) and five distortion coefficients (k 1 , k 2 , p 1 , p 2 , k 3 ) should be solved in single camera calibration and three rotation parameters (r 1 , r 2 , r 3 ) and three translation parameters (t x , t y , t z ) should also be solved for each view. In OpenCV, the homograph matrix H relates the positions of the points on the image plane to the points on the image plane (usually the imager plane) and computing multiple homograph matrixes from multiple views is the method used to solve for the camera extrinsic and intrinsic parameters.According to the formula (5), then, the process of coordinate transformation is expressed as follows: The homography matrix H is written out as column vectors, H = [h 1 h 2 h 3 ], where each h is a 3-by-1 vector.Without loss of generality, defining the object plane so that Z w = 0.According to formulas (1) and ( 5), so there is: Reading off these equations, the relations obtained are as follows: Because the rotation matrix is orthogonal to each other by construction, r 1 and r 2 are orthonormal.There are two constraints, as follows: Substituting for r 1 and r 2 both constraints can be expressed as follows: Supposing B = M -T M -1 and writing this out, as follows: Thus, the formula (10) can be expressed in the general form h i T Bh j .Because matrix B is symmetric, it can be written as one six-dimensional vector dot product.Arranging the necessary elements of B into the vector b, so there are:

h h B h h h h B h h B h h h h B h h h h
Using this definition of v ij T , the two constrains can be written as: If collecting K images of chessboards together, these equations could be stacked together as Vb = 0, where V is a 2K-by-6 matrix.If K≥2, then this equation can be solved for b = [B 11 , B 12 , B 22 , B 13 , B 23 , B 33 ] T .In fact, matrix B has a general closed-form solution, it is as follows: Therefore, the camera intrinsic parameters are pulled directly out of the closed-form solution for the matrix B and the camera extrinsic parameters are computed from the constructions of the homography.According to the formula (3), a large list of these equations are collected and solved to find the distortion parameters, after which the intrinsic parameters and extrinsic parameters are re-estimated.

THE TRANSPLANTATION OF THE OPENCV AND THE IMPLEMENTATION OF ALGORITHM
The EMCV, which is called the Embedded Computer Vision Library, is a library of computer vision running on TI DM64x series DSP (Hu, 2011;Xia et al., 2012).The EMCV provides exactly the same function interfaces as OpenCV and contains most of the basic algorithm for digital image processing.
Very often, the procedure of porting EMCV to DM642 requires the following steps: 1.A available demo project in DM642 works is found, in which all *.C file are modified to *.cpp file and then added to the CCS simulator for compiling.
Table 1: The result of calibration experiment 2. According to the error prompted from CCS compiler, the programs are modified and debugged repeatedly until error disappears.Thus, the work of changing the demo project into C++ project is completed.3. The cpp files are added to the project modified in the step (2) to be debugged, then the errors should be dealt with until compiled, which are in the cv directory and cxcore directory of EMCV.
As mentioned above, the EMCV are successfully transplanted to DSP.However, owing to the limited hardware sources and the data processing characteristic of DM642, the improvements that needed for transplantation of the algorithm based on OpenCV are shown below:  The redefinition of data structures and functions: To ensure the implementation of the camera calibration algorithm, the data structures and functions do not conform to the DSP application are tailored by using the layer-by-layer analysis method; Meanwhile, the new data structures and functions need to be redefined in practical application. The development of driver program: Using the DSP/BIOS system provided by the CCS development environment, the driver program is designed to drive the video equipment.

 The modification and optimization of code:
According to the characteristic of DM642, the operation speed of system is improved by converting the floating point number into the fixed one, the memory address of the data and program are reasonably allocated by programming the cmd file, the real-time interaction of data is realized by using EDMA and the pipeline feature of DSP is in full use by using loop unrolling.

EXPERIMENTAL RESULTS
In this study, the hardware used in the experiment includes the personal computer and the DSP platform, the design and verification of the algorithm are completed on the PC and the transplantation of the algorithm is achieved on DSP platform.According to the calibration principle and the step of transplantation illustrated above, the transplantation of EMCV to DSP is completed and the algorithm of camera calibration based on OpenCV is achieved on DSP platform.Through the test of the algorithm, the memory leaks do not appear and all the corners are found successfully.The result of calibration experiment with the chessboards used is as follows (Table 1).

CONCLUSION
According to the principle of camera calibration and transplantation illustrated above, the arithmetic of camera calibration is designed and migrated, which makes use of the characteristics of DSP and OpenCV.Compared with the traditional ways of camera calibration, the method based on the combination of DSP and OpenCV promotes the portability of vision system, improves executing efficiency, shortens period of development and accelerates the development of machine vision in DSP embedded systems.

Fig. 1 :
Fig. 1: The camera model considers lens radial distortion and tangential distortion P w = The expression of the points P u and P w under homogenous coordinate M = The parameter matrix W = [R T] is the extrinsic parameters matrix H = The homography matrix and s is the scaling factor