A Night Time Application for a Real-time Vehicle Detection Algorithm Based on Computer Vision

Vehicle detection technology is the key technology of intelligent transportation systems, attracting the attention of many researchers. Although much literature has been published concerning daytime vehicle detection, little has been published concerning nighttime vehicle detection. In this study, a nighttime vehicle detection algorithm, consisting of headlight segmentation, headlight pairing and headlight tracking, is proposed. First, the pixels of the headlights are segmented in nighttime traffic images, through the use of the thresholding method. Then the pixels of the headlights are grouped and labeled, to analyze the characteristics of related components, such as area, location and size. Headlights are paired based on their location and size and then tracked via a tracking procedure designed to detect vehicles. Vehicles with only one headlight or those with three or four headlights are also detected. Experimental results show that the proposed algorithm is robust and effective in detecting vehicles in nighttime traffic.


INTRODUCTION
Vehicle detection is an important problem in many related applications, such as self-guided vehicles, driver assistance systems, intelligent parking systems, or in the measurement of traffic parameters, such as vehicle count, speed and flow.Due to the decreasing costs and increasing power of computers, computer vision technology plays an increasingly important role in traffic monitoring and intelligent transportation systems.However, developing a robust and effective system of vision-based, vehicle detection is very challenging, due to the shadows of the vehicles, the variable illumination conditions and the variable weather conditions.In sunlight, for instance, the shadows that accompany the moving vehicles may easily be regarded as part of the vehicles, resulting in incorrect segmentation.At night, vehicle headlights and bad illumination may both cause many difficulties for accurate vehicle detection.
There is much published literature concerning vehicle detection.Tsai et al. (2007) proposed an approach for detecting vehicles using still images, based on color and edge features.This approach can detect vehicles without motion information, allowing static or slowly moving vehicles to be efficiently detected from image sequences.Zhou et al. (2007), Toral et al. (2009) and Vargas et al. (2010) utilized background subtraction to extract motion information from and detect moving vehicles in, video sequences.Zhu et al. (2000) applied an edge detector to separate shadows from vehicles, with the knowledge that a shadow is edgeless and chose a threshold to eliminate the shadows.To adapt to different characteristics of vehicle appearances under various lighting conditions, four cues, including underside shadows, vertical edges, symmetry and rear lights, were combined for effective daylight vehicle detection (Chan et al., 2012).
However, most of the features employed for vehicle detection, such as color, shadows, edges and motion information, are difficult or impossible to extract in dark or nighttime situations.Hence, the aforementioned methods are inadequate in dark or nighttime traffic conditions.In contrast to daytime traffic environments, headlights and rear lights become the salient features of moving vehicles in nighttime traffic conditions.However, nighttime traffic conditions are complicated and chaotic, with many potential light sources that are not vehicle headlights, such as traffic lights, street lights and reflections from vehicle headlights.
To detect vehicles in nighttime traffic conditions, Zhang et al. (2012) applied a reflection intensity map and a suppressed reflection map, based on the analysis of the light attenuation model, in order to extract the headlights.Headlights were tracked and paired utilizing a simple yet effective bidirectional reasoning algorithm.Although the accuracy rate of headlight detection was 95.2%, the vehicle tracking rate was only 88.2%.Wan et al. (2011) presented a new algorithm to extract, pair and track headlights.Two thresholds were applied to extract headlights.The Kanade-Lucas-Tomasi (KLT) tracker was utilized for measuring traffic flow and speed.However, the authors did not consider vehicles without lights, with only light, or with all lights shielded by others.Moreover, they did not fully consider the reflections of the headlights.A method (Lin et al., 2011) that identified and classified vehicles in nighttime traffic conditions was implemented on a TI DM642 DSP-based, embedded platform.Although the vehicle detection rate was high, this method relies on the performance of a lane detection algorithm (Wu et al., 2009).Therefore, the method may be ineffective when lane markings become indistinct due to bad illumination, or when there are no lane markings, on rural roads.
In this study, we concentrated on detecting vehicles with one headlight, or those with two, three, or four headlights in grayscale images.The proposed algorithm, including headlight segmentation, headlight pairing and headlight tracking, does not rely on the performance of a lane detection algorithm.First, pixels of headlights are extracted from the captured image sequences by utilizing the thresholding method.Second, the pixels of the headlights are grouped and labeled to obtain characteristics of the related components.The locations and sizes of the related components are employed for headlight pairings.A related component of the headlight is indicated by enclosure within a bounding box.Finally, the bounding boxes are tracked by a tracking procedure to detect vehicles.Experimental results show that the proposed algorithm can robustly and effectively detect vehicles in complicated nighttime traffic conditions.

HEADLIGHT SEGMENTATION
The headlight is a strong and consistent feature in revealing the presence of a vehicle at night.Hence, the primary task is to segment the pixels of headlights in traffic image sequences.In nighttime traffic, headlights appear as the brightest regions, whether on highways or on urban roads.Regardless of the type of street lighting or the weather conditions, the vehicle headlight feature remains relatively stable.In this study, the pixels of the headlight images are segmented in grayscale images by the thresholding method.The thresholding method is described as follows: where, f (x, y) denotes the intensity of the pixel (x, y) in the grayscale image and B (x, y) indicates the corresponding segmentation result.Threshold T is set at The headlights were successfully segmented, whereas other common light sources, such as rear lights, street lamps and some reflections of headlights, were excluded.However, a few reflections were also segmented, especially the strong reflections found in rainy, nighttime traffic conditions.Reflections are the main source of interference with headlight detection.The difference between headlights and reflections is that the shapes of headlights are stable, while the shapes of reflections are unstable.For instance, at the upper part of Fig. 1b, the shapes of the reflections are irregular, while in the lower part of Fig. 1b, reflections have a nearly circular shape.To initially filter out reflections, the detection region is adopted, as shown in Fig. 2. The detection region is applied in each image sequence.The detection region is set at the lower part of the image.The detection region should cover the lanes that are being monitored.Subsequent processes will be performed in the detection region.

HEADLIGHT PAIRING
Utilizing the thresholding method yields bright pixels.The bright pixels are then grouped and labeled Fig. 3: The parameters of a related component (Chang et al., 2004) to analyze the characteristics of related components, such as the area, location and size.Because headlights should have a predictable area, the area of related components is employed to further filter out the non-vehicle related components.A related component is filtered out if it does not meet the following condition: where CA denotes the area of the related component.Thresholds TA L and TA H are set to 50 and 150, respectively.A related component that meets condition (2) is represented by a bounding box.
Utilizing the location and size of each related component to pair off headlights, the relevant parameters of related components are defined as fellows: • C i denotes the i th related component at the detection region • The location of the bounding box surrounding C i is given by the x and y coordinates, denoted as C i (x) and C i (y), respectively (Fig. 3) • C i (W) and C i (H) denote the width and height of C i , respectively (Fig. 3) • CS denotes the set of related components at the detection region, CS = {C i , i = 0, 1, 2, …} In this study, a vehicle is defined as a pair of headlights at an appropriate distance from each other.Two headlights belonging to the same vehicle have very similar size.The two related components C i and C j are paired as belonging to the same vehicle based on their size and distance from each other, if the following pairing conditions are satisfied simultaneously.
• The two headlights of the same vehicle are aligned horizontally on the image plane: where, threshold TJ is selected as 5.
• The two headlights of the same vehicle are an appropriate distance away from each other: where, thresholds TX L and TX H are set to 30 and 80, respectively.
• The two headlights of the same vehicle have very similar size: where, thresholds TW and TH are set to 5 and 5, respectively.If two related components C i and C j meet the aforementioned pairing conditions simultaneously, they are represented by bounding box CP k t .The bounding box CP k t also represents the potential vehicle which C i and C j belong to.The parameters of the bounding box CP k t are defined as follows: • The locations of CP k t at current frame t are x and y coordinates, denoted as CP k t (x) and CP k t (y), respectively: • Width and height of CP k t are denoted as CP k t (W) and CP k t (H), respectively. ) However, a vehicle with only one headlight also needs to be detected.A related component C l that does not pair with any other related components can be represented by bounding box CDP g t .CDP t denotes the set of bounding box CDP g t , CDP t = {CDP g t , g = 0, 1, 2, …}.
Figure 4 shows the results of headlight detection and headlight pairing.Figure 4a shows that the related components that meet condition (2) are surrounded by blue bounding boxes.As shown in Fig. 4b, the paired headlights are surrounded by a white bounding box and the unpaired headlight is surrounded by a yellow bounding box.

HEADLIGHT TRACKING
The aforementioned processes, including headlight segmentation and headlight pairing, do not alone provide enough information for determining the presence of vehicles.Hence, a tracking procedure is applied to analyze the motion of the potential vehicle, based on successive image frames.The bounding boxes, CP k t and CDP g t , represent the potential vehicles being tracked across the detection region.When a bounding box, CP k t or CDP g t , is initially identified in the detection region, a tracker will be created to associate this bounding box with those in subsequent frames.The relevant parameters of the tracking procedure are defined as follows: • Tracker TP u t denotes the trajectory of CP u t , which has been tracked m (1≤m≤t) frames from t-m+1 to t and is defined as follows: • Tracker TDP v t denotes the trajectory of CDP v t , which has been tracked n (1≤n≤t) frames from t-n+1 to t and is defined as follows: The potential vehicle bounding boxes CP t = {CP k t , k = 0, 1, 2, …} and CDP t = {CDP g t , g = 0, 1, 2, …} appearing in the incoming frame t, will be analyzed and associated with the tracker sets TP t-1 = {TP u t-1 | u = 0, 1, 2, …} and TDP t-1 = {TDP v t-1 | v = 0, 1, 2, …} in the previous frame t-1, respectively.Tracker sets will be updated in subsequent processes.In the tracking process of the potential vehicle bounding boxes, trackers may be in one of the following four states: Update, Appear, Disappear, or Exit.The tracking states and relevant tracking operations are described below.o The moving distance of a potential vehicle bounding box in a horizontal direction between two successive frames is limited.
o The moving distance of a potential vehicle bounding box in a vertical direction between two successive frames depends on the vehicle speed.If the vehicle movement is fast, the moving distance may be larger than the size of the bounding box.If the vehicle movement is slow, there will be overlaps between the bounding boxes of the current and previous frames.To address the aforementioned situations, the matching condition for movement in a vertical direction is: o A potential vehicle bounding box has a similar size in two successive frames: In formulations (13-16), thresholds TCJ, TCV, TCW and TCH are set to 5, 10, 5 and 5, respectively.
• Appear: If a new bounding box does not match any tracker, a new tracker is created for this bounding box, as well as updating the tracker sets • Disappear: If an existing tracker can not be matched to any new potential vehicle bounding box, for more than ten successive frames, it is deleted from the tracker sets • Exit: When a tracker moves to the boundary of the detection region, it should stop tracking and be deleted from the tracker sets.The conditions that determine tracker TP u t ∈TP t or TDP v t ∈TDP t movement to the boundary of the detection region are described as follows: where, ROI (y) denotes the boundary of the detection region.Threshold TB is set to 15.However, as shown in Fig. 5, a vehicle with three or four headlights is tracked by two trackers, which results in false detection.It is necessary to delete the redundant tracker in a timely manner.The white lines in Fig. 5 indicate the trajectories of potential vehicle bounding boxes.However, tracker TDP v t ∈TDP t is sensitive to the reflections of headlights.To strengthen the robustness of the proposed algorithm, the tracker TDP v t ∈TDP t , which comprises a vehicle, satisfies the additional condition: where, ROI (H) denotes the height of the detection region.

EXPERIMENTAL RESULTS
The performance of the proposed algorithm was evaluated through implementation in a Microsoft Visual Studio 2008 environment, on a computer with 2.0-GHz CPU and 2-GB RAM.Nighttime traffic videos were captured by Charge Coupled Device (CCD) cameras in Tianjin City, for different traffic conditions (crowded and sparse), different weather conditions and under different lighting conditions.The FPS (frames per second) of the CCD camera was 25 frames per second and the resolution of each video was 720×576 pixels.The proposed algorithm can process 64 FPS and effectively satisfy the demands of real-time processing.
We compared the real number of vehicles against the number of vehicles detected.To objectively evaluate the performance of the proposed algorithm, the time span of each video in the experiment was more than 10 min.Figure 7 shows typical samples of nighttime traffic scenes from the performance evaluation.Table 1 shows the experimental results corresponding to the typical samples.Figure 7a shows a sample of an urban road in dim lighting conditions.Since motorcycles are not permitted on urban roads in Tianjin City, some vehicles with single headlights are classified as cars.Although illumination was dim, headlights were segmented effectively by the thresholding method.There are two trackers tracking a vehicle with three or four headlights, but the redundant tracker was deleted in time, by the redundant tracker detection procedure.
Figure 7b shows a nighttime traffic scene from a highway at rush hour under illuminated conditions.As shown in Fig. 7b, the detection region covers four lanes with heavy traffic flow.Although vehicles often change lanes while passing through the detection region, the proposed algorithm correctly detects and tracks almost all the moving vehicles.The few missed detections occurred when the light intensity of a vehicle was weak.
An urban crossroad traffic video was employed for a comparative performance evaluation of vehicle detection.The following part compares the proposed method to the method of Zhang et al. (2012) and the method of Wan et al. (2011).Figure 8 shows the comparative results of nighttime vehicle detection produced by the proposed approach, the method of Zhang et al. (2012) and the method of Wan et al. (2011).As shown in Fig. 8, the traffic conditions are complicated and chaotic.There are many non-vehicular illuminating objects, such as reflections on the road and street lamps, which interfere with vehicle detection.Figure 8 shows that the method of Zhang et al. (2012) and the method of Wan et al. (2011) do not perform well if there are reflections of headlights.
Table 2 shows the quantitative evaluation results for the proposed approach, the method of Zhang et al.
(2012) and the method of Wan et al. (2011).According to Table 2, the proposed method can provide better vehicle detection performance for nighttime traffic surveillance than other exiting methods.The experimental and comparative results can demonstrate that the proposed algorithm can quickly, effectively and robustly detect vehicles in different nighttime traffic environments.

CONCLUSION
In this study, the proposed vehicle detection algorithm consists of headlight segmentation, headlight pairing and headlight tracking.Multiple vehicles are tracked simultaneously in an efficient manner.Vehicles with only one headlight, or those with two, three or four headlights, are correctly detected.The experimental results demonstrate the effectiveness and robustness of the proposed algorithm for headlight segmentation, pairing and tracking.In comparison with state-of-theart methods, the proposed method can detect the vehicle robustly in complicated traffic scene.

Fig. 5 :Fig. 7
Fig. 5: Examples of redundant trackers (a) two trackers tracking a vehicle (b) two trackers tracking a truck

Fig. 8 :
Fig. 8: Comparative results of vehicle detection for an urban crossroad traffic scene (a) proposed method (b) method of Zhang et al. (2012) (c) method of Wan et al. (2011)

Table 1 :
Experimental data of the proposed algorithm for four samples from Fig.7