Estimating the Scene-wise Reliability of LiDAR Pedestrian Detectors

Haruya Kyutoku1, Yasutomo Kawanishi1, Daisuke Deguchi2, Ichiro Ide1, Kazuki Kato3 and Hiroshi Murase1

Abstract— Nowadays, development of driving support sys-tems and autonomous driving systems have become active.Pedestrian detection from in-vehicle sensors is one of the mostimportant technologies for these systems. However, outputsof pedestrian detectors can not be fully trusted in real en-vironments. Therefore, we propose an estimation system ofpedestrian detector’s reliabilities for a given scene. This paperproposes a scene-wise reliability calculation method for LiDAR-based detectors, and a construction method for their estimators.Here, the problem is how we can define the reliability. Theproposed method defines the reliability considering oversightsas the strictest threshold without oversights. Meanwhile, itdefines the reliability considering false detections as the loosestthreshold without false detections. Experimental results showedthat the proposed method could properly represent the relia-bility of a given scene, and estimate their reliability.

I. INTRODUCTION

In this paper, we address the problem of estimating howmuch the output of a pedestrian detector can be trusted fora given scene.

Nowadays, development of driving support systems andautonomous driving systems have become an active researchtopic. The main focus of these systems is to reduce trafficaccidents. Especially, overlooking pedestrians directly linksto deadly accidents, so pedestrian detection from in-vehiclesensors is one of the most important technologies for thesesystems.

Accordingly, various attempts to detect pedestrians fromin-vehicle cameras have been made [1], [2]. In addition,pedestrian detection using LiDAR has been actively re-searched with the growing interest in automatic drivingsystems [3]–[8].

However, outputs of pedestrian detectors can not be fullytrusted in real environments. For example, in the case ofusing a camera, the detector could not detect pedestriansin difficult scenes as shown in Fig. 1; Figure 1(a) is anexample of a scene with blown out highlights due to lensflare. There are two pedestrians at the left front, but thepedestrian detector could not detect them. Figure 1(b) is anexample of clipped shadows. There are two pedestrians atthe right front, but the pedestrian detector will not be ableto correctly detect them. As described above, the output ofa detector can not be completely trusted. Thus, in order tomake proper use of the pedestrian detection technology, we

1Haruya Kyutoku, Yasutomo Kawanishi, Ichiro Ide, and Hiroshi Muraseare with Graduate School of Informatics, Nagoya University, Nagoya, Aichi,Japan kyutoku@murase.is.i.nagoya-u.ac.jp

2Daisuke Deguchi is with Information Strategy Office, Information &Communications, Nagoya University, Nagoya, Aichi, Japan

3Kazuki Kato is with DENSO CORPORATION, Kariya, Aichi, Japan

(a) Blown out highlights. (b) Clipped shadows.

Fig. 1. Examples of difficult scenes for detection with a camera.

need to consider the reliability of a detector for a given scene,in addition to its output.

Likewise, there are difficult situations in the case ofpedestrian detection using a LiDAR. Here, there are twokinds of difficult situations for a detector; oversights and falsedetections as shown in Fig. 2. Figure 2(a) is the full view ofa scene, and Fig. 2(b) and Fig. 2(c) are examples showingeach of the difficult situations. Details of each situation aredescribed below.

• Difficult situations due to oversightsFigure 2(b) is a magnified view of the right front partof Fig. 2(a). It is an example where a pedestrian existsclose to a wall. The arrow indicates the pedestrian. Sucha pedestrian will most likely be integrated with thewall in the detection process, and will be overlooked.However, such situations exist often in practice, so ifa vehicle travels trusting the detector’s output, it couldlead to a serious accident. On the other hand, it maybe possible to take actions such as running carefully ifthe information that oversight is likely to be occurringis provided to the driver.

• Difficult situations due to false detectionsFigure 2(c) is a magnified view of the left front partof Fig. 2(a). It is an example where there are manystructures similar to pedestrians. The arrows indicatesuch structures. In this case, there are many rod-shaped structures. A detector will most likely detectsuch pedestrian-like structures as pedestrians. Such falsedetections will obstruct smooth traveling.

As discussed above, information on the reliability of adetector is very important. If we can estimate the reliabilityof a detector directly from the input independent of itsoutput, safer autonomous driving and driver assistance can berealized. For example, the system can slow down when thereliability considering oversights is low, because there maybe oversights. Moreover, the system can lower the urgencyof the detection result when the reliability considering falsedetections is low, because unnecessary stops will increasethe risk of accidents. As a matter of fact, a number of

2018 21st International Conference on Intelligent Transportation Systems (ITSC)Maui, Hawaii, USA, November 4-7, 2018

978-1-7281-0322-8/18/$31.00 ©2018 IEEE 3511

(a) Full view. (b) Close to large structures such as a wall. (c) Existing pedestrian-like structures such as poles.

Fig. 2. Examples of difficult scenes for detection with a LiDAR. (Bottom right images are corresponding camera images for reference.)

Points DetectorDetection

Result

Annotation

Reliability

Reliability Estimator

EstimatorConstructor

Calculation of Reliability

Construction of Reliability Estimator

DetectionAccuracy

(a) Learning step

Points

Detector

DetectionResult

Reliability Estimator

Reliability

Detection Result with Reliability

(b) Estimation step

Fig. 3. Proposed framework.

techniques to estimate the reliability have been developed forsensor fusion [9]–[11]. However, they are techniques for thereliability of the sensor itself or inputs. Therefore, we pro-pose a framework for estimating the reliability of a LiDAR-based pedestrian detector for a given scene. In the followingsections, we present the definitions of scene-wise reliabilities(probability of oversight and misdetection) of a detector. Theproposed framework is shown in Fig. 3. It is composed oftwo steps; a learning step (Fig. 3(a)) and an estimation step(Fig. 3(b)). The learning step calculates the reliabilities andconstructs reliability estimators. In the estimation step, theconstructed estimator estimates the reliability of the detectordirectly from a given scene, and the system outputs it withthe detection results.

This paper is organized as follows: Section II introducesthe definition of reliabilities in detail. Section III explains thedetails of constructing the reliability estimator. Experimentalresults and discussions are presented in Section IV. Finally,Section V summarizes the paper.

II. DEFINITION OF SCENE-WISE RELIABILITIESOF A DETECTOR

In this section, we introduce the definition of scene-wisereliabilities of a detector.

As errors in pedestrian detection, there are oversightsand misdetections of non-pedestrian objects in the outputof a pedestrian detector. Thus, when the number of theseerrors are small, we can say that the reliability of thedetector for a given scene is high. The metric of reliabilityconsidering oversights indicates the degree of reliability incorrectly detecting pedestrians, namely, the extent to whichthe oversights do not exist. On the other hand, the metric of

reliability considering false detections indicates the degreeof reliability considering misdetections, namely, the extent towhich false detections do not exist. Each kind of reliabilitywill be described in detail below.

A. Reliability Considering Oversights

Firstly, the detection rate (recall) can be considered as themetric representing the degree of reliability considering over-sights. A scene with lower recall will have more undetectedpedestrians, so the scene will have more oversights. Thus, ascene with high recall can be considered as highly reliable.So recall can be used as the metric for the reliability consid-ering oversights. However, since recall is difficult to calculatefrom a single scene, we propose a metric approximating itas follows.

A detector’s threshold where the number of oversights isminimized is expressed as follows.

To = max{arg mint

NFN(t)} (1)

Here, t is a detector’s threshold normalized in the range of[0.00, 1.00], and NFN(t) is the number of false negatives(oversights). An example of the change in the number of truepositives with changing the threshold t is shown in Fig. 4(a).As shown here, in general, the number of positive detectionsincrease with lower (looser) threshold values, and decreasewith higher (tighter) threshold values. The output from alower threshold will include more false detections, so wecan say that To (the maximum threshold value with minimaloversights) is better if its value is higher. Therefore, To isused as the reliability considering oversights as follows.

Ro = To (2)

3512

�������� ��� ��������

��� ��� ����������

�����������

������� ����� ����

(a) Oversights.

������������

�������������� ���

�������� ����� ���� ��

������� �� ����������������

(b) False detections.

Fig. 4. Reliability metrics.

B. Reliability Considering False Detections

Meanwhile, precision can be considered as the metricrepresenting the degree of the reliability considering falsedetections. A scene with lower precision will have morefalse detections, so the scene will have more fake detections.Thus, a scene with high precision can be considered ashighly reliabile. So precision can be used as the metric forthe reliability considering false detections. However, sinceprecision is difficult to calculate from a single scene, wepropose a metric approximating it as follows.

Next, a detector’s threshold where a false detection occursis expressed as follows.

Tf = min{arg mint

NFP(t)} (3)

Here, t is a detector’s threshold normalized in the rangeof [0.00, 1.00], and NFP(t) is the number of false posi-tives (false detections). An example of the change in thenumber of false positives with changing the threshold t isshown in Fig. 4(b). As explained in Section II-A, in general,results with higher (tighter) threshold values will includemore oversights. Conversely, oversights will be avoided withlower (looser) threshold values, so we can say that Tf (theminimum threshold value with minimal false detections) isbetter if its value is lower. Therefore, inverted Tf is used asthe reliability considering false detections as follows.

Rf = 1− Tf (4)

III. RELIABILITY ESTIMATION

In this section, we describe the proposed scene-wisereliability estimation method.

The proposed scene-wise reliability estimator extracts sev-eral features from the point cloud of a given scene measuredby a LiDAR and outputs the estimated reliability. Here,following the methology of general detection tasks, we firstextract sets of points by clustering as pre-processing. Wedefined two types of features representing a scene, namelyglobal features and pedestrian-like features, both describedbelow. Both features are extracted from each cluster andin the end concatenated into a 271 dimensions vector. Thereliability regressors of the estimators for (2) and (4) areconstructed using ε-SVR (Epsilon Support Vector Regres-sion) with RBF (Radial Basis Function) kernel. They aretrained with these features as input and reliabilities as output.

Note that the coordinate axes are the X-axis in the horizontaldirection, the Y-axis in the depth direction, and the Z-axisin the height direction.

A. Global Features

The following four features are extracted as global featuresfrom each cluster, and in the end concatenated into a 118dimensions vector.

• 3D covariance matrixCalculated from the three-dimensional covariance ma-trix of each cluster’s centroid. Since this is a 3 × 3diagonal matrix, six elements eliminating duplicates areused as features.

• 3D momentCalculated from the three-dimensional moment of eachcluster’s centroid. Six elements eliminating duplicatesare used as features.

• Distance to candidateCalculated from the set of distances to all of theclusters’ centroid. To represent the statistics of this set,the median, mean, and standard deviation are used. Inaddition, a histogram of distances to all points projectedonto the X-Y plane is used as feature. The histogram iscomposed of 100 bins with 1 m interval.

• Number of pointsCalculated from the number of points in each cluster.The mean, median, and standard deviation of thesevalues are used as feature.

B. Pedestrian-like Features

The following three features are extracted as pedestrian-like features from each cluster, and in the end concatenatedinto a 153 dimensions vector.

• Statistics for tall objectsGenerally, pedestrians have a tall shape. Therefore, theratio of clusters whose height is larger than the widthand the depth when the cluster is represented by arectangular solid is used as the feature. In addition, theareas of each cluster projected onto the X-Y plane areused. The mean, median, and standard deviation of thesemetrics are used as features.

• Statistics for heightsCalculated from the maximum, minimum, centroid, andheight (maximum − minimum) of the Z coordinatesof points in each cluster. The mean, median, standarddeviation, and histogram of these values are used asfeatures. The histograms of the position (maximum,minimum, centroid) are composed of nine bins with0.5 m interval. Only the histogram of the height iscomposed of ten bins with 0.5 m interval.

• Statistics for reflection intensityCalculated from the maximum, minimum, mean, andmedian of the reflection intensity of points in eachcluster. Their histograms are composed of 25 bins andused as features.

3513

LiDAR

Fig. 5. Experimental vehicle

IV. EXPERIMENTS

To confirm the effectiveness of the proposed method,experiments on scene-wise reliability estimation were con-ducted by using actual point clouds measured using a LiDARas input.

A. Experimental Conditions

To acquire point clouds, we prepared a vehicle thatmounted Velodyne LiDAR HDL-64e on the roof as shownin Fig. 5. The vehicle traveled through urban areas duringthe day time, and obtained 17 sequences composed of 3,871frames. Each experiment was carried out with a leave-one-out cross-validation scheme where one sequence was usedfor evaluation and the others for learning.

In this experiment, pedestrians with no occlusion withinapproximately 40 m from the sensor were targets for thedetection, which were annotated manually.

B. Pedestrian Detector

As pre-processing, road plane removal and clusteringusing Euclidean distance were performed for candidatesextraction. Next, these candidates were classified whetherthey were pedestrian or not by a classifier. The pedestrianclassifier used in this experiment was constructed with RealAdaBoost [12]. The following 225 dimensions feature wasused for this.

• Features proposed by Kidono et al. [3]213-dimensional features.

• Features proposed by Liu et al. [13]Seven-dimensional features based on eigenvalues.

• Additional featuresFive-dimensional features. (The height of the cluster, themaximum and minimum of the Z coordinate of eachpoint, the area in the X-Y plane, the median of theintensity of each point.)

C. Validity of the Proposed Reliabilities

Here we present an experiment to confirm the validity ofthe proposed reliabilities described in Section II.

The proposed method defined “threshold where over-sights or false detections are minimized” as reliabilities. Forevaluation of these criteria, the thresholds in each framewere calculated with the pedestrian detector constructed in

Section IV-B. Note that the pedestrian detector applied toeach sequence was trained with other sequences. In addition,clusters extracted from the pre-processing were used forevaluating oversights and false detections.

As an example, the result from sequence #14 is shown inFig. 6. The threshold of the detector is in the range of [0.00,1.00]. Figure 6(a) shows the maximum thresholds withoutoversights defined in (1) for each frame. As mentioned inSection II-A, we can say that outputs from scenes withhigher threshold values can be trusted. On the other hand,Figure 6(b) shows the minimum thresholds without falsedetections defined in (3) for each frame. As mentioned inSection II-B, we can say that outputs from scenes with lowerthreshold values can be trusted. Note that blanks in Fig. 6(a)are frames with no pedestrian to be detected.

From Fig. 6, we can see that the threshold changes acrossframes in accordance with the change of scene followingthe travelling of the vehicle. A frame indicated by a solidcircle in Fig. 6(a) is shown in Fig. 7(a) as an example ofa low reliability scene considering oversights. A pedestrianexisting near a wall, so it was difficult to detect. Similarly,a frame indicated by a dotted circle in Fig. 6(a) is shown inFig. 7(b) as an example of a high reliability scene consider-ing oversights. A pedestrian existing in an open intersection,so it was easy to detect.

Meanwhile, a frame indicated by a solid circle in Fig. 6(b)is shown in Fig. 7(c) as an example of a low reliabilityscene considering false detections. Since there were manystructures similar to pedestrians such as poles, trees, and soon, false detections were likely to occur. Similarly, a frameindicated by a dotted circle in Fig. 6(b) is shown in Fig. 7(d)as an example of a high reliability scene considering falsedetections. Although there are trees along the street, theywere easy to be separated from pedestrians because they werefew and thin.

As described above, we confirmed the validity of the pro-posed reliabilities. In the remaining part of the experiments,the reliabilities calculated here are used as ground truths.

D. Effectiveness of the Proposed Reliability Estimation

Here we present an experiment to confirm the effectivenessof the proposed reliability estimator described in Section III.The estimators were constructed with the reliabilities calcu-lated in Section IV-C as ground truths. Note that the estima-tors for each sequence were learned with the reliabilities ofother sequences.

As a result, the proposed estimator could estimate thereliability considering oversights with 0.055 mean absoluteerror, and that considering false detections with 0.033 meanabsolute error. As an example, the estimation result forsequence #14 is shown in Fig. 8. From Fig. 8(b), we cansee that the reliability estimator considering false detectionswas able to estimate quite well. Meanwhile, from Fig. 8(a),we can see that the reliability estimator considering over-sights failed to estimate well when the reliability decreaseddrastically. The following two reasons could be the cause forthis.

3514

����

����

����

����

� �� ��� ��� ��� ��� �� �� ��� ���

�� � �����

���������������� ��

��� � �������

����

(a) Considering oversights.

����

����

����

����

� �� ��� ��� ��� ��� �� �� ��� ���

�� � �����

����������������� �

�������� ����������

����

(b) Considering false detections.

Fig. 6. Thresholds where oversights or false detections are minimized.

• Lack of featuresFeatures leading to oversights such as integration ofclusters were lacking.

• Insufficient amount of training dataThe reliability considering oversights is calculated fromonly frames including pedestrians, so the size of thetraining data was insufficient.

V. SUMMARY

In this paper, we focused on the problem that outputsfrom pedestrian detectors can not be fully trusted in realenvironments. Considering this, we defined reliabilities ofdetectors for a given scene, and proposed a method toconstruct a scene-wise reliability estimator. The proposedmethod defined the “threshold where oversights or falsedetections are minimized” as scene-wise reliabilities.

To demonstrate the effectiveness of the proposed method,experiments were conducted by applying the proposedmethod to actual point clouds measured from a LiDAR.The experimental results showed that the definitions ofreliabilities were reasonable, and that the proposed estimatorcan estimate the reliabilities properly.

For future work, we need to extend features of the relia-bility estimator for more accurate estimation. For example,rather than outputting a global reliability for a scene, wemay calculate estimation features and the reliability of localregions. Furthermore, we should also design a method thatconsiders the final detection results considering the reliabil-ity.

ACKNOWLEDGMENT

Parts of this research were supported by the Center ofInnovation Program from Japan Science and TechnologyAgency, JST.

REFERENCES

[1] H. Yoshida, D. Suzuo, D. Deguchi, I. Ide, H. Murase, T. Machida,and Y. Kojima, “Pedestrian detection based on deep convolutionalneural network with ensemble inference network,” in Proc. 2013 IEEEIntelligent Vehicles Symposium, Jun. 2013, pp. 654–659.

[2] H. Fukui, T. Yamashita, Y. Yamauchi, H. Fujiyoshi, and H. Murase,“Pedestrian detection based on deep convolutional neural network withensemble inference network,” in Proc. 2015 IEEE Intelligent VehiclesSymposium, Jul. 2015, pp. 223–228.

[3] K. Kidono, T. Miyasaka, A. Watanabe, T. Naito, and J. Miura,“Pedestrian recognition using high-definition LIDAR,” in Proc. 2011IEEE Intelligent Vehicles Symposium, Jun. 2011, pp. 405–410.

[4] T. Ogawa, H. Sakai, Y. Suzuki, K. Takagi, and K. Morikawa, “Pedes-trian detection and tracking using in-vehicle LIDAR for automotiveapplication,” in Proc. 2011 IEEE Intelligent Vehicles Symposium, Jun.2011, pp. 734–739.

[5] M. Engelcke, D. Rao, D. Z. Wang, C. H. Tong, and I. Posner,“Vote3Deep: Fast object detection in 3D point clouds using efficientconvolutional neural networks,” in Proc. 2017 Int. Conf. on Roboticsand Automation, May 2017, pp. 1355–1361.

[6] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deephierarchical feature learning on point sets in a metric space,” Advancesin Neural Information Processing Systems 30, pp. 5099–5108, 2017.

[7] T. Yamamoto, F. Shinmura, D. Deguchi, Y. Kawanishi, I. Ide, andH. Murase, “Efficient pedestrian scanning by active scan LIDAR,” inProc. 2018 Int. Workshop on Advanced Image Technology, no. C4-2,Jan. 2018, pp. 1–4.

[8] Y. Tatebe, D. Deguchi, Y. Kawanishi, I. Ide, H. Murase, and U. Sakai,“Pedestrian detection from sparse point-cloud using 3DCNN,” in Proc.2018 Int. Workshop on Advanced Image Technology, no. C4-4, Jan.2018, pp. 1–4.

[9] A.-L. Jousselme, A.-C. Boury-Brisset, B. Debaque, and D. Prevost,“Characterization of hard and soft sources of information: A practicalillustration,” in Proc. 2014 Int. Conf. on Information Fusion, Oct.2014, pp. 1–8.

[10] N.-E. E. Faouzi and L. A. Klein, “Data fusion for ITS: Techniquesand research needs,” Transportation Research Procedia, vol. 15, pp.495–512, Jun. 2016.

[11] A. R. Hilal, “Context-aware source reliability estimation for multi-sensor management,” in Proc. 2017 Annual Int. Systems Conf., Apr.2017, pp. 1–4.

[12] R. E. Schapire and Y. Singer, “Improved boosting algorithms usingconfidence-rated predictions,” Machine Learning, vol. 37, no. 3, pp.297–336, Dec. 1999.

[13] X. Liu, G. Zhao, J. Yao, and C. Qi, “Background subtraction based onlow-rank and structured sparse decomposition,” IEEE Trans. on ImageProcessing, vol. 24, no. 8, pp. 2502–2514, Apr. 2015.

3515

(a) Low reliability scene considering oversights due to large structures. (b) High reliability scene considering oversights due to open space.

(c) Low reliability scene considering misdetectionsdue to existence of similar objects.

(d) High reliability scene considering misdetectionsdue to inexistence of similar objects.

Fig. 7. Examples of scenes with low and high reliabilities.

����

����

����

����

����

����

���

� �� �� �� �� �� ��� ��� ��� ���

� �� ������

������ �����

���������

������������� �������

���� ���� ��

��

(a) Considering oversights.

����

����

����

����

����

����

� �� ��� ��� ��� ��� ��� ��� ��� ���

���� ���

����������

����� ���

������������� �������

��� ��������� ����

(b) Considering false detections.

Fig. 8. Estimated reliabilities.

3516