Enhanced Machine Vision System for Ripe Fruit Detection Based on Robotic Harvesting

: The proposed study intends to provide an efficient algorithm for the instruction of an automatic robot arm to choose the ripe fruits on the tree. Steps involved in this study are recognizing and locating the ripe fruits from the leaf and branch portions by using an efficient machine vision algorithm. Initially, discrete wavelet transform is used for better preserving of edges and fine details in the given input image. Then RGB, HSV, L*a*b* and YIQ color spaces were studied to segment the ripe fruits from the surrounding objects. Finally, the results showed that ‘I’ component of the YIQ color space has the best criterion for recognizing the fruit from the foliage. The fruit segmentation based on machine vision has an occlusion problem. In this proposed method these problems are also examined.


INTRODUCTION
India is an expansive agrarian nation.Agribusiness has given careful consideration since old.Our precursor has created a lot of better approaches to make the marvelous rural human progress.With the advancement of horticulture engineering, farming modernization raises new prerequisites for agribusiness improvement.Then again, previous customary recognition strategies have not fulfilled the prerequisite of up-to-date horticulture, which promotes modern detection technology applied.Around them machine vision technology can give proficient and dependable items recognition approach.Fruits picking by humans is a time-consuming, tiresome and expensive process.Because of this, the automation of fruit harvesting has accomplished great popularity in the last decade.Therefore, image processing and use of automation techniques in agriculture have become a major issue in recent years.
System designs based on automatic image analysis technology that are already getting used in other areas of agriculture (Cubero et al., 2011;Lorente et al., 2012), including applications in the field.For instance, Mizushima and Lu (2010) designed system for the presorting of apples respectively in the field.Further examples of the usage of machine vision technology operation in the field are those applied to automate the harvesting task.Basic research on robotic harvesting initiated with orchard fruits (Schertz and Brown, 1968;Parrish and Goksel, 1977); after that, this kind of studies have been ongoing in several countries (Sarig, 1993).This technology has then been useful for vegetable fruits.Tillett (1993) reviewed several prototype robots and clarified the importance of the manipulator design and its application to practical use.Several researchers have applied robotic technology to fields in greenhouses; For example, Ling et al. (2004) for tomatoes, Muscato et al. (2005) for oranges, Edan et al. (2000) for melons and Van Henten et al. (2002) for cucumbers.A thorough review with regard to fruit recognition systems can be found in Jimenez et al. (2000).However, the performance and cost have not satisfied commercial requirements.Hanan et al. (2009) designed a vision system to pick orange using a harvesting robot.They used the R/ (R+G+B) feature for recognition of orange fruits on the tree.Wang et al. (2008) applied RGB model for recognition of cotton.They applied the R-B feature for this purpose.Bulanon et al. (2002) designed an algorithm for the automatic recognition of Fuji apples on the tree for a robotic harvesting system.The color of Fuji apple was red; the difference between luminance and red color (R-Y) was only used.Guo et al. (2008) used Ohta et al. (1980) color spaces based image segmentation algorithm for robotic strawberry harvesting system.Moradi et al. (2011) and Pawar and Deshpande (2012) developed systems for the skin defect detection of apples by L*a*b* color space features and pomegranates by HSI color space features, respectively.
Localization of fruit on trees is an important and still challenging issue in agriculture, which has potential applications ranging from fruit load estimation to yield forecasting and robotic harvesting.Automated vision-based localization of fruit has been studied intensively (Jimenez et al., 2000).In this present study, we focused on the development of an algorithm in order to recognize and localize the ripe fruits that have a smooth surface, such as apples, oranges and pomegranates using only color analysis.The objectives of the proposed work are as follows: • To compare the twelve different color components for harvesting based background removal with histograms of the color channels (R, G, B, H, S, V, L*, a*, b*, Y, I, Q) of the input image.The color component with the best peak histogram is selected in order to recognize the ripe fruits on the tree • To develop an algorithm for fruit harvesting robot arm to locate the fruit on the tree

MATERIALS AND METHODS
This section shows the development of an algorithm based on machine vision that can be used for the guidance of a harvesting robot arm to pick the ripe fruits on the tree.

Acquisition of the image:
To validate the proposed fruit detection algorithm, 40 digitalized images of different on tree fruits (orange, apple and pomegranate) were randomly selected from the internet; each image has more fruits.Here, the natural environments of the greenhouse were preferred for getting the images without any extra lighting technique was used.These images were then transferred to the computer and all proposed algorithms were developed in the MATLAB environment using wavelet and image processing toolbox version 7.0.
Preprocessing operation: An image is considered as a collection of information and the occurrence of noises in the image causes degradation in the quality of the images.So the information related to an image tends to loss or damage.It must be important to restore the image from noises for acquiring maximum information from images.Filtering is a technique for enhancing the image.In this study, six different image filtering methods are compared using the Root Mean Square Error (RMSE) and Peak Signal to Noise Ratio (PSNR).If the value of RMSE is low and the value of PSNR is higher than the de-noising method it is better: where, im (x, y) : The original image f (x, y) : The reconstructed image n×n : The picture size: Here, MAX is the peak value of the pixels within an image.In the 8-bit pixel format image, MAX value is represented by 255.
The statistical measurement of input and denoised input image of oranges and apples are reported in Table 1 and 2. From the statistical measurement, DWT has the high PSNR value and low RMSE value.The experimental evaluation of our proposed 2D-DWT decomposition and reconstruction shows that it removes noise and preserve fine details of an image more effectively than the other filters.This filtering output serves as input to succeeding processes.
Segmentation: Image segmentation acts as the key of image analysis and pattern recognition.It is a process of dividing an image into different regions such that every single region is uniform, but the union of any two regions is not (Cheng et al., 2001;Pal and Pal, 1993).A proper definition of image segmentation is as follows: If P ( ) is a homogeneity predicate described on groups of connected pixels, then segmentation is a separator of the set F into connected subsets or regions (S 1 , S 2 ,…, S n ) such that: The uniformity predicate (  ) = true for all regions,   and �  ∪   �= false, when ( ≠ ) and   and   are neighbours.
Color of an image can have much more information than the gray level.In most pattern recognition and computer vision applications, the additional information provided by color is able to help the image analysis method yield better results than approaches using only gray scale information (Gauch and Hsia, 1992).Usually color spaces have three components or channels for representing all possible color and intensity information.Selecting the best color space still is one of the difficulties in color image segmentation for each application.In this study, we compare the twelve different color spaces and choose the suitable color space for separate the ripe fruits from leaves and tree branches.The fruit regions in most images were under the shadow of the leaves and branches.
Primary space: RGB is a very commonly used threedimensional color space with color components or channels red, green and blue.Red, green and blue elements can be described by the illumination values of the location obtained through three separate filters (red, green and blue filters) depending on the following equations: where, S R , S G , S B are the color filters on the incoming light or radiance E (λ) and λ is the wavelength.The R, G, B color components of an acquired image is device dependent (Trussell et al., 2005).Lighting condition of greenhouse was not equal in the time of image acquisition.The RGB color model could not be lonely used to recognize mature fruits because of the high correlation among the R, G and B components (Pietikainen, 1996;Littmann and Ritter, 1997).Figure 1 shows the monochrome images of red, green and blue color components and its corresponding onedimensional histograms.
Perceptual space: Human describes colors by hue, saturation and brightness.Hue (H) and saturation (S) define chrominance, while intensity or Value (V) specifies luminance.The HSV color space is defined as follows (Smith, 1978): HSV is computed from a nonlinear transformation of RGB color space and normalized to a range of 0 to 255.This is consistent with the histogram representation in intensity with the values from 0 to 255. Figure 2 shows the monochrome images of hue, saturation and value color components and its corresponding one-dimensional histograms.(Ohta et al., 1980).It is obtained through a nonlinear transformation on XYZ: where, ( 0 ,  0 ,  0 ) are the XYZ values for the standard white (Cheng et al., 2001;Gauch and Hsia, 1992).The 'L' component in the L*a*b color space corresponds to lightness ranging from 0 (black) to 100 (white), the 'a' component corresponds to the measurement of redness (positive values) or greenness (negative values) and the 'b' component corresponds to the measurement of yellowness (positive values) or blueness (negative values).
CIE spaces have metric color difference sensitivity to a good approximation and are very convenient to measure the small color difference, while the RGB space does not (Robinson, 1977).Figure 3 shows the monochrome images of L*, a* and b* color components and its corresponding one-dimensional histograms.

YIQ space:
The linear transformation of the RGB to YIQ conversion is defined by the following matrix transformation:  , 2004).The YIQ color model can partly neutralize the interrelation of the red, green and blue components in an image.Figure 4 shows the monochrome images of Y, I and Q color components and its corresponding one-dimensional histograms.
Among all the planes, except in the instance of 'I' component, the fruits and other objects were highly presented in the images.So it was just about difficult to find an appropriate fruit area from the other color components.So from the YIQ color space 'I' color plane (Fig. 4c) was utilized for the contour detection of fruit regions because this plane was considered as pixels of fruit regions and small amount of canopies.So, this step was used for the contour detection of probable fruit regions within a given input image.

RESULTS AND DISCUSSION
Recognition of ripe fruits: This Preprocessed output RGB image (Fig. 5a) transformed into the YIQ color model and separate the 'I' plane (Fig. 5b) to recognize the ripe fruits.The recognized pixels were represented by the value of '1' while the rest of the pixels were represented by the value of '0'.This resulted in the binary representation of the 'I' plane (Fig. 5c) in which the fruit areas are represented as white and the background was represented by black color.Due to the variation in illumination among the image in the normal-view category and the presence of some dead leaves, certain pixels were falsely classified as fruits.After the binary conversion, the ripe fruits are in the image, but still some objects are available these are not fruits.It was essential to filter out or else reduce these parts.These unwanted parts are eliminated by applying the dilation and erosion methods on binary image.Resultant image has only the fruit regions (Fig. 5d) and the background was fully removed by morphological operations.
Localization of ripe fruits: Localization of ripe fruit is another key task in robotic applications.In some cases, the ripe fruits are clustered.This leads to multiple ripe fruits be detected as a one big fruit.To overcome this problem, the watershed algorithm was applied (Gonzalez and Woods, 2008).The goal of the watershed transform is to identify regions of highintensity gradients (watersheds) that divide neighbored local minima (basins).This algorithm can separate the joined objects into individual ones.But, the watershed algorithm splits an individual ripe fruit into several slices.To solve this problem, the binary image was first eroded using a morphological operation.Then, watershed segmentation was applied to the binary image.This method could successfully separate clustered fruits into individual ones (Fig. 5e).Based on these results, label the connected components in a binary image.A labeling algorithm was used on the segmented image to separate out regions of pixels in the binary image which may correspond to physical fruits.
This binary mask image changed over to the same type of the input image.To remove the background, the binary mask image was multiplied in R, G and B channels separately.The color image was reconstructed by composition of R, G and B channels got from the past step.The resultant image (Fig. 5f) shows the recognized fruit regions only.Finally, measure the properties of each connected component region in the binary image for locating center of the fruit (Fig. 5g). Figure 5 photographically demonstrates the proposed algorithm for ripe fruit detection using an orange sample and the output of apple sample by using this same algorithm.

Performance of the proposed algorithm:
The development of this algorithm with it is able to detect fruits in varying lighting condition and occlusion would increase the overall performance of robotic fruit harvesting.The performance of this algorithm was validated with 40 samples of different fruit images.Overall, localization of fruits was carried out with acceptable accuracy and the algorithm was truly able to   3).The accuracy of the algorithm was 93%.
The execution times for each step of this proposed algorithm were measured to examine the performance of this system.The measured execution time for each step is shown in Table 4.The average execution times for the recognition and localization of a single fruit was 2.9 sec.

CONCLUSION
In this study, a vision algorithm was designed to recognize the ripe fruits from the other objects of image as well as to determine their location.Recognition algorithm developed in this study used color difference ('I' channel) as criteria for discriminating the ripe fruits from the leaves and boughs.This algorithm could identify ripe fruits by high accuracy in different lighting conditions of a greenhouse.Almost all the fruits are spherical, so the centroid of the fruit was accurate to be considered as the target point for a picking arm.Moreover, the algorithm showed reliable for robotic harvesting operations.About 93% area of a ripe fruits was extracted by the proposed algorithm.This shows the suitability of the algorithm to use in machine vision guidance based harvesting robots.The required time for processing of an image was 2.9 sec.This low processing time makes the algorithm to be suitable for real time applications.

λ
Fig. 1: (a) and (b) red color component and its corresponding histogram plot analysis, (c) and (d) blue color component and its corresponding histogram plot analysis, (e) and (f) green color component and its corresponding histogram plot analysis

Fig. 2 :
Fig. 2: (a) and (b) hue color component and its corresponding histogram plot analysis, (c) and (d) saturation color component and its corresponding histogram plot analysis, (e) and (f) value color component and its corresponding histogram plot analysis

Fig. 3 :
Fig. 3: (a) and (b) L* color component and its corresponding histogram plot analysis, (c) and (d) a* color component and its corresponding histogram plot analysis, (e) and (f) b* color component and its corresponding histogram plot analysis

Fig. 4 :
Fig. 4: (a) and (b) Y color component and its corresponding histogram plot analysis, (c) and (d) I color component and its corresponding histogram plot analysis, (e) and (f) Q color component and its corresponding histogram plot analysis Luminance-chrominance spaces: CIE L*a*b space: The Commission International de l'Eclairage (CIE) color system defines three primary colors, denoted as X, Y and Z. XYZ coordinates originate from a linear transformation of RGB space, as indicated by Tenenbaum et al. (1974): , 0≤G≤1, 0≤B≤1.'Y' component corresponds to luminance (lightness) and 'I' component corresponds to the orange-cyan axis and 'Q' component corresponds to the magenta-green axis (Che-Yen and Chun-Ming

Fig. 5 :
Fig. 5: Flow chart of the key steps involved in ripe fruit detection algorithm S 2 , …, S n = Partitioning of the set F into a set of connected regions S i , S j S i = Adjacent to S j pixel P () = Uniformity predicate defined on group of connected pixels E (λ)= Incoming light or radiance λ =

Table 1 :
RMSE, PSNR value between given input and denoised input

Table 3 :
Performance of proposed fruit detection algorithm

Table 4 :
Execution time of each image processing steps