IVSIAC 產學小聯盟成果發表會

Advanced ADAS Technology

Prof. Jiun-In GuoDept. of Electronics Engineering, National Chiao Tung University

Hsinchu, Taiwan

June 26th, 2015

1

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

2

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

3

Design Trend for Car Safety: From Passive to Active

4

Euro NCAP Regulation

5

6

Technology for Self Driving

Car Tech News in CES 2015

• Self-driving cars, Apple CarPlay and Android Auto take over CES 2015

• Delphi's computer chauffeur drives me around Las Vegas at CES 2015

• CES 2015: Cruise control gains self-steering with Valeo'sCruise4U

• Volkswagen's Trained Parking is a robotic valet for your home

• CNET's Connected Car panel explores the future of transportation at CES 2015

• Hands off with the Volkswagen Golf R Touch at CES 2015

• Cars of the future won't need drivers: CES 2015

7

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

8

Mobileye Technology

Features:1. Headway Monitoring and Warning2. Intelligent High-Beam Control

3. Daylight Pedestrian Collision Warning,

including Bicycle Detection

Vision SoC for Safety Driving

4. Forward Collision Warning,

both in Highway and Urban areas,

including Motorcycle Detection

5. Lane Departure Warning

6. Traffic light recognition

7. Speed limit indication

9

TI TDA2x SoC for Advanced Driver Assistance Systems (ADAS)

1010

TI’s Automotive Technologies

Advanced ADAS based on Machine Learning

• Nvidia Drive PX

11

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

12

Research Team

13

Dr. Jiun-In Guo

Dr. Ching-Lung Su

Dr. Ching-Wei Yeh

Dr. Kuan-Hung Chen

NCTUProfessor and Director of Institute of Electronics Department of Electronics Engineering

CCUProfessor

Department of Electronics Engineering

FCUProfessor

Department of Electronics Engineering

YuntechProfessor Department & Graduate school of Electronics Engineering

ADAS Technology We Developed

14

Blind Spot Detection

Auto headlight control

Auto High Beam Control

Auto Windshield Wiper Control

Animated Car-Backing Track

Hands-free Trunk Lift gate

Inclement Weather Processing

Wide-view Video Stitching

HDR Night Vision(Single-CAM, Dual-CAM, Micro Camera Array)

Traffic Light Detection

Speed Limit Detection(Taiwan)

USA Driver Dangerous Behavior Detection

Car License Plate Detection

Pedestrian and Scooter Detection

(On IMX6)

Lane Departure Warning (Day, Night, Rain)

Forward Collision Warning(Day, Night, Rain)

LDWS+FCWS(Day, Night, Rain)

Stop & Go

2D/3D Hand Tracking (Single-CAM and Dual-CAM)

Hand Writing

ADAS Technology Developed in NCTU

15

Performance Summary

16

Function Feature

Performance

CPU Resolution FPS

Lane Departure Warning System (LDWS)

• Support variety of road lanes(Solid, dashed, cat-eyes)

• Support variety of weather conditions(Sunny, rainy, foggy days, nights, etc.)

ARM Cortex-A9 1GHz D1 30

Forward Collision Warning System (FCWS)

• Detection distance at day time: 0~50m• Detection distance at night time: 0~40m

ARM Cortex-A9 1GHz D1 30

Stop & Go System (working with FCWS)

• Working distance at day: 0~10m ARM Cortex-A9 1GHz D1 30

Pedestrian and Motorcycle Detection

System

• Obstacle type: pedestrian, motorcycle, bicycle

• Multiple ROI detection• Moving Object Detection IP

ARM Cortex-A9 1GHz D1 30

AMD A10-7850K GPU

D1 30

Accelerator (200MHz in TSMC 90nm)

D1 30

Blind Spot Warning System (BSWS)

• Detection region: outward 0~3m, backward 0~20m

• Object type: vehicle, motorcycleARM Cortex A9 1GHz VGA 30

Inclement Weather Processing Technology

• Dynamic local contrast enhancement • Automatic harsh environment judgment• Support a variety of inclement weather

conditions (nights, rainy, cloudy, foggy days, sandstorm)

ARM Cortex A9 1GHz D1 30

Performance Summary

17

Function FeaturePerformance

CPU Resolution FPS

Traffic Light Detection System

• Vehicle speed: 0~100 km/h ARM Cortex A9 1GHz D1 30

Speed Limit Detection System

• Vehicle speed: 0~150 km/h• Support speed limit sign of North America,

Taiwan, Europe, and China

Intel Atom 1.6GHz (USA)

D1 50

ARM Cortex A9 1GHz (Taiwan, EU)

D1 15

Wide-view Video Stitching System

• Support variety of weather conditions (Sunny, rainy, cloudy, nights, etc.)

• Extending Panoramic to 360 degree surround view

ASIC4ch HD720@

178Hz 30

ASIC4ch HD1080@

202MHz30

Single-camera HDR Night Vision System

• Good for scenes with high variation in lighting (Tunnel, night view with strong light, etc. )

ARM Cortex-A9 1 GHz VGA 13

Dual-Camera HDR Night Vision System

• Good for scenes with high variation in lighting (Tunnel, night view with strong light, etc. )

ASIC HD1080 30

Driver Dangerous Behavior Detection

• To detect eating, phone calling and dozingTREK-668 (Intel Atom

1.6GHz)360x240 15

2D/3D Hand Tracking for Gesture Control

• Single-camera(2D)/Dual-camera(3D) vision based system

Intel Core i7-2600 3.40GHz (3D)

VGA 24

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

18

Highlight on Advanced ADAS Technology

19

Speed Limit Detection(Taiwan)

USA

Pedestrian and Scooter Detection

Lane Departure Warning

Forward Collision Warning

Cross Traffic Detection

Vision-based system

Support resolution: Up to 720x480

Support variety of weather conditions(Sunny, rainy, foggy days, nights, etc.)

Support variety of road lanes(Solid, dashed, cat-eyes)

Lane Departure Warning System (LDWS)

20

Demo Video (Day, Night, Rain)

100 fps D1@ Intel Atom 1.6GHz (using one core)30 fps D1@ Cortex-A9 (Freescale iMX6 1GHz using one core)

Detected rate: 97.43%False alarm rate: 7.89%

LDWS Flow chart

21

Image Capture

Adaptivethreshold

Integral Image

Line-thinning

Houghtransform

Departure

Right laneFSM

Left laneFSM

Line Collection

Lane detectprocess

line detectprocess

Patent Comparison: LDWSMobileye

WO0180068A1、US6704621、US8254635、US2013141580A1、US2012069185A1

NCTU iVS LAB(Patent pending)

Process

Steps Methods Steps Methods

XImage pre-processing

ROI setting

Detect lane boundaries

(3-D)

1. Road skeleton estimation system

(1) Roadway is modeled as a circular/parabolic arc parallel to XZ plane in 3-D space

(2) Two-phase model(3) Value cost function

determine the line (Warped image)

Lane detection(2-D)

Line detection1. Pixel-based Adaptive Threshold

(PAT): dynamically distribute image into lane and non-lane part.

2. Sobel and Randomized Hough transform (RHT)

Lane definition1. Slope2. Vanishing point distance3. Previous frame lane

information4. Lane information finite state

machineFind road curvature XVehicle relationship

to laneX

Time to lane crossing (TLC)

Wheel-to-lane distanceDeparture warning

Image center to Lanes center distance

22

Vision-based system

Detection distance at day time: 0~50m

Detection distance at night time: 0~40m

Support resolution: Up to 720x480

Forward Collision Warning System (FCWS)

23

Demo Video (Day, Night, Rain)

30 fps D1@ Intel Atom 1.6GHz (using one core) with LDWS30 fps D1 @ Cortex-A9 (Freescale iMX6 1GHz) with LDWS

Detected rate: 90.24%False alarm rate: 5.40%

FCWS Flow chart

ROI setting

Image capture

Day or Night

Edge feature

Shadow Feature

Combination

Tracking

Distance estimation

Light feature

Edge feature

FCWSDay

FCWSNight

Patent Comparison: FCWS

Mobileye (Note) NCTU-IVS

Detect Vehicle

Generate candidate regions of interest:1. Rectangular shaped region for all

position and all size2. Filter out candidates lack of texture

and in-compliance with perspective constrains on range and size.

Single frame classification:1. 2-stage classification algorithm.2. SVM + Adaboost.Multi-frame Approval Process:1. Additional information collected

over a number of frames are used in system.

Generate candidate regions of interest:1. Noon or Night judgement.2. Noon : Shadow feature extraction.3. Night : Light-pair feature extraction.Single frame classification:1. Vertical feature verification for noon

and night.Candidate Vehicle buffer:1. Accumulate feature frequency

according to spatial information.2. Select maximum temporal frequency

candidate as vehicle.Tracking state:1. Finite state machine and kalman filter.

Range estimation

Pinhole model and position-based geometry.

Horizontal line + position-based geometry.

Time to collision

Estimate TTC by Scale change and frame rate.

X

25

Note: Itay Gat, Meny Benady, and Amnon Shashua, “A Monocular Vision Advance Warning System for the Automotive Aftermarket,” SAE Technical Paper 2005-01-1470, 2005, doi:10.4271/2005-01-1470.

Vision-based system using machine learning

Support resolution: Up to 720x480

30 fps D1 @ Intel Atom 1.6GHz (using single core)30 fps D1 @ Cortex-A9 (Freescale iMX6 1GHz)30 fps D1 @ AMD A10-7850K GPU Realization30 fps D1 @ Accelerator (200MHz in TSMC 90nm)

Obstacle type: pedestrian, motorcycle, bicycle

Multiple ROI detection

Pedestrian and Motorcycle Detection System

26

Demo Video (PC)

Demo Video (IMX6)

Detected rate: 85 %False alarm rate: 3.6 %

Detection distance: 0~40m (D1)

Proposed Adaboost-based Detection Algorithm

27

Y

Y

Feature value

Calculation

Stage Sum

> thresholdReject

Accumulate

Weighting Value

for Stage Sum

Final Stage?

Choosing

“Left”or“Right”Weighting Value

Adjusted Threshold =

Variance * Feature Threshold

Start a New

Stage

Accept

Feature

Coordinate

Decoding

Integral & Square

Integral Image

Calculation

N

N

3m~5m (360x65)

1m~3m(180x55)

0.5m~1m (180x80)

ROI

Inner

Down-sample

Image

Original

Image

Choose Source

Image

No More

Feature

Y

N

Stage Number

from Classifier

Node number

from Classifier

1. Outside of 3m

2. Within 3m

(full-length)

3. Within 3m

(half-length)

Classifiers

Stage Threshold

from Classifier

Next stage

ROI

Dynamic setup

1. Classifier type

2. Classifier stage

Search size : 14x28~40x80

Next node

Off-line

14x28

40x80

Classifier Stages

Feature Node

12 types

Moving Object Detection IP

Pedestrian and Motorcycle Detection in ASIC

28

Specification

Technology

Gate Count

On-Chip Memory

Capability D1@30fps

TSMC 90nm CMOS

173K Gates

35.7K Bytes

Memory Interface AXI 32bit Bus

Max Frequency 200MHz

Video Format YUV 4:2:0

Detection Rate 85% (based on classifiers)

SRAM (17.4KB)

Search

Window

Buffer

(14.5KB)

Mesh &

Integral pixel

Calculator

(48x88x21bit)

(48x88x8bit)

Feature

Coordinate

Calculator

Classifier 2

Classifier 1

Coordinate

Buffer

Object

Recognition

Engine

AXI Master AXI Slave

Register

Files

(feature number,

feature type)

Parameter Bus

Candidate

Window

DRAM

RISCSource Images

Unique Features

· Supporting pedestrian/motorcyclist detection

· Supporting close-range detection

· Dynamic ROI region

· Fully HW computation

Verification

· RTL simulation pass

· Altera Stratix4 FPGA testing pass

· ESL Aldec FPGA emulation pass

Vision-based system

Support resolution: HD720/D1/VGA

Vehicle speed: 0~150 km/h

D1 23fps@Intel Atom 1.6GHz (Taiwan, EU)D1 50fps@Intel Atom 1.6GHz (USA)

Support speed limit sign of different countries

Speed Limit Detection System

29

Demo Video (Taiwan)

Demo Video (USA)

Detected rate: 94.34 % (USA), 94.52% (Taiwan, EU)False alarm rate: 5.66 % (USA), 5.48% (Taiwan, EU)

Proposed Algorithm for USA Speed Limit Detection

1. Rectangle Detector

2. Achromatic Decomposition

3. Adaptive Threshold

4. Digit Segmentation

5. Digit Normalization

6. Digit Recognition

30

• Behavior analysis• Eating detection

• Detect the distance between hands and mouth

• Phone calling detection

• Detect if hand tracking blob locate beside the face

• Dozing off detection

• The system can not detect eyes for more than 15 frames in the face region

• Dangerous driving warning• Eating detection

• Phone calling detection

• Dozing off detection

31

Driver Behavior Analysis System (DBAS)

Alley View for Cross Traffic Detection

• Goal: • To show the left/right sides of view through view

transformation on a fish-eye camera

• Could be combined with object detection

32

Input: Fish-eye view Output: Alley view

Multiple People Detection, Tracking, and Behavior Analysis

33

• Major Functions• People Detection

• People Tracking

• Behavior Analysis

• Challenges• Lightness variation

• Strong lightness variation by turn on/off the light indoors

• People keeping static• People will be updated to

background if they keep static

• People crossing• The labels often exchange

when people switch

• Moving objects• When objects move(like chair),

it will be detected

Functions

Behavior Analysis

Video Sequence

PeopleTracking

Falling Down

Detection

People detection

People Labeling

Posture Analysis

Applications

Virtual Fence

Trajectory Recording

Aware！

Falling down detection

Virtual fence

Demo Video for Multiple People Detection, Tracking, and Behavior Analysis

34

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

35

ADAS for Fleet Management

• Concept

36

ADAS Algorithms We Adopted

Lane Departure Warning System (LDWS)

Forward Collision Warning System (FCWS)

Driver Behavior Analysis System (DBAS)

Speed Limit Detection (SLD)

37

ADAS for Fleet Management

• Dangerous driving behaviors

38

車道偏移警示

前車防撞警示

速限標示偵測

危險駕駛行為警示

左車道偏移

右車道偏移

前方車距

道路速限

手部偵測

臉部偵測

五官偵測

蛇行單位時間內，訊號交互

系統 功能 時間資訊 車速資訊 危險行為

單位時間內，維持短距 高速 逼車

<當前車速 超速

使用手機

進食

打瞌睡

左車道偏移警示

前車過近警示

方向燈訊息

右車道偏移警示

無左方向燈

無右方向燈

ADAS for Fleet Management

•危險行為回傳資料

39

危險行為

回傳資料

車輛ID 時間 地點(GPS) 當前車速 規定速限 偏移資訊 距離資訊 駕駛行為

蛇行 ● ● ● ●

車道偏移 ● ● ● ●

逼車 ● ● ● ● ●

前車過近 ● ● ● ● ●

超速 ● ● ● ● ●

使用手機 ● ● ● ●

進食 ● ● ● ●

打瞌睡 ● ● ● ●

Multiple ADAS Functions in Freescale i.MX6 with Multithreading

• Approaches• Multiple threads

are created• Captured thread

• Worker threads• LDWS, FCWS, SLD,

DBAS

• Display thread

• Thread pipelining• Synchronized by

FIFO in data transfer

40

pth_DLCE

pth_LDWS

pth_PDS

Write data packet to display

thread

Read data packet from the last

worker thread

Draw the boundaries of driving

lane and the marking windows of

detected pedestrians

Capture thread

Display thread

pth_ADAS

working loop

initialization

Create worker threads and FIFOs

working pipeline

pth_FCWS

Pth_LDWS

Pth_FCWS

Pth_SLD

Pth_DBAS

Performance Summary

41

Freescale iMX6 Quad Core Cortex A9 (BD-SL-i.MX6)

(Automotive grade)

系統 平台 CPU 解析度 效能

A. 車道偏移警示

Freescale i.MX6 Cortex A9 1 GHz D1(720x480)

30fps

B. 前車防撞警示 30fps

C. 速限標誌偵測 24fps

D. 駕駛行為分析 15fps

E. A+B 30fps

F. A+B+C 24fps

G. A+B+C+D 15fps

System Integration

42

• ADAS box (IMX6) + IDU-300 (3G) + Cloud Server

Client Side

43

車隊管理 I.MX6 demo (camera on road)功能: FCWS + LDWS + SLD

系統: 40km/h + (-1~+1)km/h

資訊: 統計五分鐘內偏移次數、未保持安全車距、超速等行為

Server Side

• GCP(receive message from IDU-300)

• GCP_AP(decode G2A data)

44

GCP

GCP_AP

Our waring message

Vision Radar for ADAS

• LDWS+FCWS in traffic map

45

System Initial

Parameter Setting

Line model Vehicle model

Image capture

Not initial Initial already

Lane departure warning system(Multiple lanes detection)

Forward collision warning system(Multiple vehicles detection)

Lane width estimation

Locate vehicle x position

Build traffic map(vehicle position& departure

judgement)

Outline

• Design trends

• State-of-the-art technology

• Overview on NCTU iVS intelligent vision for automobiles

• Advanced ADAS technology

• Some application scenarios

• Conclusion

46

Conclusion - On-going MOST Project (ViDAR) (2014/8~2017/7)

47

IVVS: Intelligent Vehicle ViDAR System

Multiple Object Identification Multiple Object Recognition Multiple Object Ranging Multiple Object Tracking Multiple Object Behavior Analysis HDR Night Vision 360/540 Degree Surround View

Multiple Object Identification

Multiple Object RecognitionMultiple Object Ranging

Multiple Object Tracking

Multiple Object Behavior Analysis

HDR Night Vision

360/540 Degree Surround View

Conclusion - Roadmap

• Intelligent Vision Technology for Advanced ADAS Applications

48

ADAS (2013-2015) Advanced ADAS (ViDAR) with deep learning(2015-2017)

Advanced ADAS (ViDAR) with deep learning and 360 degree

surround view by SPADscanning

(2017-2019)

Conclusion

• Intelligent Vision is an emerging topic in both academia and industry.• ADAS has become an attractive feature in the vehicles

developed by the major automobile companies in the world.

• Machine learning is also a hot topic in image/vision content analysis.

• Advanced ADAS is the trend for automotive applications.

• Sensor fusion is required (Vision + mmwave + Laser (SPAD))

• More and more intelligence is required and will be invented in the near future.• Technology always begins from Human.

49

50

Thank you very much for your attention !

http://ivs.ee.nctu.edu.tw/iac/

Our Vision, Your Intelligence !

Q&A