Decorative Line and Edge Extraction in Cartoon Images

Decorative Lines and edges in a cartoon image are the most important elements to convey its semantic information. In this study, we propose a method to extract both decorative lines and edges of the cartoon image at the same time. Firstly, we use the Hessian matrix to compute the direction perpendicular to the decorative lines in the image. Then, non-maximum suppression is applied to the image’s second derivative of Gaussian to locate the possible decorative line’s center. Secondly, we use directional zero-crossing (the zero-crossing of the first Gaussian derivative on the precomputed direction) to verify the existence of the decorative lines. Finally, decorative lines and the edges detected by non-maximum suppression on the first Gaussian derivative are fused together. The experimental results show that our method is more effective than other existed methods in literature for complicated cartoon images.


INTRODUCTION
Cartoon, as a very popular visual art, is well-liked by both children and adults for its comic characters and the funny drawing style.In history, lots of classic cartoon characters had been produced by artists and shown to readers, which form a big industry.Nowadays, these classic arts draw more and more attention of researchers for variant reasons (Ping and Wang, 2011).For the picture studios, they are eager to excavate the potential commercial value of the classic cartoon characters by re-editing them to produce more artistic study at a much higher speed.Additionally, they also want their copyrights to be protected, especially when the study are transformed on the internet.For the computer scientists, they want the cartoon easy to be compressed, retrieved and used on various devices (Liu et al., 2007;Elder and Goldberg, 2001).To achieve the aims mentioned above, accurate cartoon image decomposition gives a shortcut.
Compared to the natural images, 2D cartoon has simpler, more artificial contents.In general, there are mainly two kinds of curve in the cartoon image.One is decorative lines, which is usually thick lines used to highlight certain structure.Another is the edge describing the sharp discontinuity between regions (Zhang et al., 2009) as shown in Fig. 1a.Generally speaking, the decorative lines, together with the edges, convey most of the semantic information in a cartoon image.Artists can present the facial features, motion and the action, only using the decorative lines and edges.Some study has already focused on, or related to the extraction of these semantic elements.There have been a series of outstanding study on the edge detection (Marr and Hildreth, 1980;Canny, 1986;Martin et al., 2004;Pablo et al., 2011), which did not consider the decorative line extraction problem.Steger (1998) presents a method to extract curvilinear structure, such as road, vessel in satellite and medical image.Sykora et al. (2003Sykora et al. ( , 2005) ) makes significant advances in vectorizing cartoon images, but their approach relies on a particular drawing style in which meaningful regions are enclosed by dark thick bounding lines.However, modern cartoon regions are usually delimited by edges in the image rather than explicit decorative lines.They consider only one curve type but not both, which is not appropriate to many modern cartoon styles.Zhang et al. (2009) promote another meaningful method to locate both decorative lines and edge.However, their method depends on the assumption that the local maxima of the second Gaussian derivative of an image are the center of a decorative line, which is not exact because the local maxima may also present near a significant edge, as shown in Fig. 1b.Ping and Wang (2011) proposes a method to extract the two types of curve by construct Delaunay triangulation on the edge points detected by edge detection method, such as Canny detector.This method uses the property of Delaunay triangulation subtly, but while the edge points of decorative lines are not detected, the method may fall into local disorder.Huang et al. (2010) also proposes an isotropic nonlinear filter based edge-enhance method to extract the curves in cartoon images, which only restrict to the curves that are similar to decorative lines.We note that specific challenges exist while performing 2D cartoon semantic elements extraction.There are not only decorative lines but also boundaries which consists of edge points in modern cartoon images, so that local maxima of second Gaussian derivative may mistake edge's neighbor as decorative lines, as show in Fig. 1c, on the eyebrow.
In this study, we promote a new method to tackle the problem mentioned above.The new method extracts decorative line and edge one by one.Firstly, we compute the eigenvector of the Hessian matrix to give the direction of each pixel in the image.Then we use the directional zero-crossing of the first Gaussian derivative on the direction to verify if a local maxima of second Gaussian derivative is on the decorative lines.Finally, we combine the edge and the decorative lines together to generate the final results.
Problem in decorative lines detection: The red circles in Fig. 2a illustrate the decorative line profile perpendicular to excircle of the eye of Fig. 1a.We resize the x coordinate to make the bar shape of the profile into width 1.It is clear that the edge of the bar shape is much sharper than that in the natural image, which is also a common style for most of the cartoon image.Figure 2c and 2e shows the results convolved with the first and second derivative of Gaussian kernel respectively.An edge profile perpendicular to the inner circle of the eye of Fig. 1a, is shown in Fig. 2b and so is the results convolved with the first and second derivative of Gaussian kernel respectively in (d) and (f).It is clear that there is also a local maximum in Fig. 2f.
The bar shape will arise together with a zero value in the first directional derivative and a local maxima in the second directional derivative, which is also the main principle for the curvature detection in literatures (Zhang et al., 2009;Steger, 1998).However, in the cartoon image, the edge is so sharp that an edge arise within two or three pixels, as shown in Fig. 2b.The local maxima of edge's second directional and the zeros value are so close in x coordinate, so there will probably be a false detection probably in this case.
In order to avoid this problem, we use a directional zero-crossing, instead of zero value, to verify the existence of a bar shape.Compared to the bar shape, there will be no zero-crossing in the first directional derivative for edge in the profile direction as shown in Fig. 2d.

PROPOSED METHODOLOGY
Our method consists of the following steps.Firstly, the direction of the second derivative of Gaussian is gained by computing the Hessian matrix's eigenvector corresponding to the largest eigenvalue.Secondly, the local maxima of the second derivative of Gaussian of the image are extracted.Then the zero-crossing pixels along the direction are gained to verify the decorative line.Finally Canny edge detector is used to get the edge.Together with the decorative lines, we get the both the two types of curve.

Direction calculation based on Hessian matrix:
When the image I (x, y) is convolved with a second derivative of Gaussian, we denote the result as G 2 (x, y), where x∈1, 2…m, y∈1, 2…n.Here, we employ the Hessian matrix H (x, y) to compute the direction: where, r xx , r xy and r yy denote the second order partial derivatives.The direction of G 2 (x, y) taking on the maximum absolute value can be determined by the eigenvectors (t x , t y ) of the Hessian matrix H (x, y), corresponding to the maximum eigenvalue.We normalize the direction as (n x , n y ) by Eq. ( 2 (2) Directional zero-crossing detection: We use G 1 (x, y) = (d x , d y ) to denote the result that cartoon image is convolved with the first directional derivative of Gaussian at position (x, y).Along the direction (n x , n y ), directional component is described Eq. ( 3): ( , ) Then we detect the directional zero-crossing Z (x, y) on the m×n matrix L using a zero-crossing detector.The zero-crossing detector looks for places on L where the elements of the L passes through zero, i.e., points where the function value changes sign at coordinate (0, 0) in Fig. 2c.
The zero-crossing detector often presents bias to certain edges.In practice, we locate the pixel (x, y) with smaller absolute value of L (x, y) as the result.For good measure, we also check the absolute value of the difference between the neighborhoods.A bigger absolute value is preferred.In summarize, the zerocrossing detector can be described as Eq. ( 4): Non-maximum suppressing on the second derivative of Gaussian: Given the direction of second derivative of Gaussian, we use a non-maximum suppression to locate the local maxima on the G 2 (x, y).While the nonmaximum suppressing, we do not quantize the direction into 4 or 8 averaged divided circle bins, but use the actual direction (n x , n y ): where, (x j , y j ) is the 8-connected neighbors of (x i , y i ) with the close angle bias less than π/6 and i, j∈ [1, 2…m×n], i ≠ j.The results are shown in Fig. 3b.Decorative line center detection: In Fig. 3, it is obviously that non-maximum suppression of G 2 (x, y) not only mark the center of the decorative lines but also so edges with certain pixels bias, which is similar to situation in Fig. 2f.Therefore, some examinations on M (x, y) are needed.
Given the result of non-maximum suppression M (x, y) and the directional zero-crossing Z (x, y), it is easy to use Z (x, y) to verify the existence of a decorative line in M (x, y): While detecting, M (x, y) and Z (x, y) may bias with in 1 pixel, it should be tolerable.Figure 3d shows decorative lines detected from Fig. 1a, in which isolated points and edge fragment are removed.

Edge and decorative line fusion:
There are two types of curve existing in cartoon image.One is the common edge; another is the edge of decorative lines.Decorative lines have continuous edges on both sides of themselves.Based on this principle, we fill the gap between the edge and the center of the decorative lines by flood filling.What is more, some broken edges are linked according to the average distance to the decorative lines.
We use the Canny detector to detect both the two type edges based on G 1 (x, y), as shown in Fig. 4.Then, Fig. 4: Shows the result of our algorithm, running on Fig. 1a The thick lines are the decorative lines with their actual thickness and the thin lines present the edge if there are edge fragments near the center of the decorative lines, we link them.Finally we use flood filling with decorative line center as seed to fulfill the decorative lines.The result can be shown in Fig. 4.

EXPERIMENTAL RESULTS
We test our method on various famous cartoon images.For the limitation of length, only three, the Simpsons, Saint Seiya and the Xi Yang, are given as examples (Fig. 5).We compare our method with two impressive methods proposed by Sykora et al. (2003) while extracting the decorative lines and the edges; we assume that the decorative lines are no more than 12 pixels thick.The decorative lines and boundaries, together with regions coherent to the decorative lines with the same color will be marked as black.Empirically, we use σ = line width/1.5 for the decorative line extraction which is relatively bigger than that used in Steger (1998) to deal with the sharp bar shape profile of decorative lines in cartoon.Sykora et al. (2005) method usually misses the edge with less contrast and mistakes all the edge as decorative lines and locates them with some bias.The method proposed by Zhang et al. (2009) produces better results, but it usually mistakes some of the edges as decorative lines when the decorative lines are very thin.Our method produces the best of the three methods, with clear distinction between the edge and the decorative lines.

CONCLUSION
In this study, we proposed a method for decorative line and edge extraction.We use the directional zerocrossing on the first Gaussian derivative of cartoon image to verify the existence of a cartoon decorative line.The direction is obtained by computing the eigenvector the Hessian matrix.The results show that our method is effective on various famous cartoon characters, from simple to complicated ones.Our future study will focus on high level semantic elements extraction, such as closed region and other structures.

Fig. 1 :
Fig. 1: (a) is the original image, (b) the pixel in green is the local maxima of second Gaussian derivative of the original image, (c) reveals that the local maxima exists not only on the decorative lines, such as the excircle of the eyes, but also near the edge, the inner circle of the eye and the eyebrow, with several pixels bias from the actual edge

Fig. 2 :
Fig. 2: (a) and (b) show typical decorative line profile and edge profile in cartoon image, (c) and (d) show the first derivative of Guassian of (a) and (b) respectively, (e) and (f) show their second derivative of Gaussian

Fig. 3 :
Fig. 3: (a) shows results of the zerocrosing detector on Fig. 1a, (b) shows the result of non-maximum suppression on the second derivative of Guassian, (c) shows the two together, (d) give the final results for decortive line detection

Fig. 5 :
Fig. 5: Comparison of the detection results about decorative lines and edges for three cartoon images between our algorithm and other method The thick line is the decorative line with its actual thickness and the thin lines presents the edge; The first row shows the original image, the second and third row show results of Steger (1998) and Zhang et al. (2009) method respectively; The last row shows the results of our method