learn from iot: pedestrian detection and intention ...gsolmaz/publications/mobicom19... ·...
Post on 25-Aug-2020
4 Views
Preview:
TRANSCRIPT
Learn from IoT: Pedestrian Detection and Intention Prediction for Autonomous Driving
Gürkan Solmaz, Everton Luis Berz, Marzieh Farahani Dolatabadi*, Samet Aytac, Jonathan Fürst, Bin Cheng, Jos den Ouden*
IoT Research Group
NEC Laboratories Europe
* Eindhoven University of Technology (TU/e)
1st ACM Workshop on Emerging Smart Technologies and Infrastructures for Smart Mobility and Sustainability (SMAS), ACM MobiCom 2019
Los Cabos, Mexico, October 21, 2019
This work has received funding from the European Union’s Horizon 2020 research and innovation programme within the project AUTOPILOT (Automated Driving Progressed by Internet Of Things) under the grant agreement No 731993. Responsibility for the information and views set out in this document lies entirely with the authors.
2 © NEC Laboratories Europe GmbH 2019
Motivation
▌Why “learn from IoT”?Uber accident (2018): Both car sensors
and driver can easily fail
Mobile sensing, vast amount of sensors, RSUs, V2V, V2X communication, 5G, edge computing
IoT can support traffic safety
▌Major advancements in ML due to increased availability of data, computation, and software TensorFlow, PyTorch
▌IoT expansion to many domainsSmart cities and smart mobility
IoT platforms: Azure, AWS IoT
Large-scale tesbeds
▌Aim: Combine IoT and ML for enhanced AD safety
3 © NEC Laboratories Europe GmbH 2019
Proposed approach
▌ System learning from data collected from
1) in-vehicle sensors (e.g., cameras)
2) external IoT sources such as mobile devices of VRUs
▌ Components included in the system setup:
1) autonomous car
2) mobile devices with the mobility app
3) in-vehicle IoT platform
4) VRU detection and intention prediction components
5) cloud IoT platform
▌ ML leverages data sources for training
Data collected by the vehicle at the TU/e campus
Smartphone apps of TU/e and NEC for pedestrian movement tracking
Mobile ITS-G5 location devices
▌ We propose methods for accurate VRU detection and pedestrian intention prediction
1) creating a world model (WM) by combining the vehicle and mobile device data and performing VRU detection and localization
2) using the latest trajectory data (in-vehicle or mobile device) to predict the pedestrians’ intended movement steps based on historical measurements
▌ Petri Net models to combine VRU detection and intention prediction
Decide under which conditions which inputs can be combined for the autonomous decision-making
4 © NEC Laboratories Europe GmbH 2019
VRU detection through WM
▌WM contains the vehicle itself and objects around
▌The formalism adopted is WIRE where the
WM aims to track semantic objects such as VRUs http://wiki.ros.org/wire
(World Information for Robot Environments framework by TU/e)
▌Multiple Hypothesis Tracker (MHT) algorithm [2] combines evidences to a common world representation dynamically
▌Objects’ attributes, classification, and prior knowledge are associated in the hypotheses treeEvery hypothesis contains a list of anchors and has a correctness probability
Each anchor contains an individual symbol, a set of measurements and a probabilistic signature that consists out of a mixture of probability density functions generated by a set of behavior models
The predicate attribute space represents predicate grounding relations that link attribute values and predicate symbols [2]
5 © NEC Laboratories Europe GmbH 2019
Pedestrian intention prediction method
▌This method predicts the next location of pedestrians based on historical data and the current position
▌ML model uses the mobile device data
Speed of pedestrian
GPS trajectory values
▌The model predicts nf future locations
▌The representation of pedestrian trajectory is inspired by the model in [8]
the path modeling with an
example pedestrian
trajectory during the TU/e
experiments
6 © NEC Laboratories Europe GmbH 2019
Intention prediction ML model
▌ Three input features, three concatenation layers, and nf output features
▌ Adam algorithm [5] for the optimization process with ReLU activation function
▌ Embed encoder: Map the inputs into vectors and then forward to the (intermediary) concatenation layers
▌ Intermediary layers: Concatenate all outputs of the feature encoders and pass the concatenation through fully connected layers
▌ The data is randomly partitioned into training (70%), validation (20%), and testing (10%) subsets.
We defined np = 10 and nf = 5
Approximately equivalent to 11m
and 5.5m in a straight-walking
distance
7 © NEC Laboratories Europe GmbH 2019
Combining VRU detection and intention prediction
▌ Combining the inputs using stochastic priority Petri Nets from the two previous parts to support the autonomous decision-making
States/places: big circles, transitions: rectangles
Probabilistic var. for transitions: λ
Priorities: Curly-braced numbers
# tokens: Numbers w/o curly braces
▌ The model has two types of VRUs
1. users of mobile devices and our app (w/ IoT data)
2. people w/o the app
▌ Three possible cases exist during the objects come to vicinity of each other
1) both the mobile device and vehicle sensor data,
2) only mobile device data
3) only vehicle sensor (camera) data available in the in-vehicle platform
8 © NEC Laboratories Europe GmbH 2019
TU/e experiments
▌ We conduct the pilot tests mainly at the TU/e campus with 2km road network, speed limit 15km/h
▌ Three pilot tests: Each ~1-2 weeks long 21 controlled experiments (total of 70 runs)
Predefined pedestrian movements
Autonomous driving behaviors are mostly uncontrolled
▌ Custom-built autonomous car prototype (Toyota Prius)
▌ The car has an ITS-G5 device connected to the in-vehicle IoT platform Two pedestrians also carry mobile ITS-G5 devices
▌ IoT gateway connects the in-vehicle platform with two cloud IoT plaftormsvia MQTT and HTTP using cellular 4G
▌ Robot operating system (ROS) operates on the in-vehicle IoT platform
9 © NEC Laboratories Europe GmbH 2019
Message delay and WM
▌Smartphone to vehicle delay ~0.6secFrom smartphone to cloud and lastly in-vehicle IoT platform (ROS timestamps)
▌In the WM, vehicle and mobile devices together has more consistency compared to only vehicleVehicle receives the global location of a pedestrian a few seconds before the
camera detection
10 © NEC Laboratories Europe GmbH 2019
Intention detection results
▌ Ludwig and TensorFlow to train and validate the model
▌ Each step is a consecutive data point in the range of [0.5,1] sec Most predictions for 1st step is <1m
▌ Controlled scenarios: Walking-straight has the best accuracy Max error of 1.9m in the furthest prediction step
▌ 18% and 48% decrease in the error compared to the NARX and Dead Reckoning approaches 0.2m better on average than [8]
11 © NEC Laboratories Europe GmbH 2019
Conclusion & future work
▌We propose learning from IoT data to improve the safety of autonomous driving
Key problems: pedestrian detection and pedestrian intention prediction
▌The proposed system can complement existing safety solutions
▌Future work:
extracting features from other data sources such as OpenStreetMap and traffic operation center data (e.g., traffic lights)
Old problems also still exist
12 © NEC Laboratories Europe GmbH 2019
This work has received funding from the European Union’s Horizon 2020 research and innovation programme within the project AUTOPILOT (Automated Driving Progressed by Internet Of Things) under the grant agreement No 731993. Responsibility for the information and views set out in this document lies entirely with the authors.
Questions?
gurkan.solmaz@neclab.eu
top related