車載向け高速カメラモニタリング システ …...japan display inc. 2018/03/02...

Japan Display Inc.

2018/03/02

車載向け高速カメラモニタリング システム（CMS）の開発

Takayuki Nakanishi,

Kazunori Yamaguchi,

Toshiharu Matsushima,

Fumitaka Goto,

Daichi Suzuki, and Satoshi Matsushima

Development of

Fast Response,

Low Latency

Real-time Camera and Display System

for Automotive Applications

2 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Outline

1. Introduction

• Background

• Primary experiment

2. Subject-Evaluation Results of the Latency

• Latency time test

3. Technologies of Our Prototype

4. Summary

3 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

1. Introduction

• Background

• Primary experiment

2. Subject-Evaluation Results of the Latency

• Latency time test

3. Technologies of Our Prototype

4. Summary

4 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Background

Opportunity:

• Improving fuel economy

• Reducing blind area

• Image recognition

• Designing flexibility

100 ms Delay

10 ms Delay

5.6 m

A few tens of centimeters

Moving Distance at 200 km/hr

200 ms Delay 11 m

5 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Primary Experiment - Setup

DSLR

60 fps (estimation)

latency: 280~320 ms (removed)

CMS120

120 fps

latency: <40 ms CMS240

240 fps

latency: <10 ms

Camcorder

60 fps (estimation)

latency: 40~80 ms

Tripod External display of DSLR

AMOLED, by Wi-Fi connection

CMS240

TN-LCD, HDMI

External display

for camcorder

IPS-LCD, HDMI Side

mirror

CMS120

in-house

IPS LCD,

stripe

RGB

Difficult to compare!

6 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Observation – In the cockpit

Optical mirror

CMS120

CMS240

Camcorder

• Ambient light at 100k Lux

• Comparing to side mirror, display brightness is too low

and barely see nothing

• Strong surface reflection occurs on KAMUY’s display

Problems

Noticeable delay using camcorder : 40-80ms delay

Noticeable motion blur : 240Hz case looks good

Strong surface reflection : need anti-reflection

Insufficient panel brightness : more luminance!

7 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Observation – Night view

Optical mirror

CMS120

CMS240

Problems

Poor contrast using digital system : AE control is important

Dark & poor image using optical mirror

Benefit

Good visibility at camcoder : Image processing effect

Camcorder

8 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

1. Introduction

• Background

• Primary experiment

2. Subject-Evaluation Results of the Latency

• Latency time test

3. Technologies of Our Prototype

4. Summary

9 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Latency Time Test

Participants

• 20 persons

Condition

• Delay from 0 to 117 ms

(0 to 7 frames)

• 6 type of Images, Random

10 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 33 50 67 83 100 117P

erc

en

tage

Latency Time (ms)

Accuracy rate of latency test : Video 1

Wrong

No Diff

Correct

Video 1: Waving Hands

• Accuracy rate significantly increases after latency time of 50 ms

• Fast and periodic moving object is easy to judge latency differences

Speed: 13.18 deg/sec

Periodic: Yes

Reference point: Yes

Can detect time difference correctly

Can’t detect time difference correctly

Say “same” means wrong answer

11 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Video 2: Person Passing By

• The accuracy rate is unstable, while “No diff” is around 30% until 117 ms

• Judgment for slowly moving object seems not easy

Speed: 7.14 deg/sec

Periodic: No

Reference point: Yes 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 33 50 67 83 100 117P

erc

en

tage

Latency Time (ms)

Accuracy rate of latency test : Video 2

Wrong

No Diff

Correct

12 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 33 50 67 83 100 117P

erc

en

tage

Latency Time (ms)

Accuracy rate of latency test : Video 3

Wrong

No Diff

Correct

Video 3: Train Moving Quickly

• The fastest moving object and no reference point in this movie

70 km/hr speed and train becomes blurred and looks all the same

• Obviously it’s a difficult scenario.

The percentage of “No Diff” is over 40% for all delay

Speed: 23.33 deg/sec

Periodic: No

Reference point: Insignificant

13 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 33 50 67 83 100 117P

erc

en

tage

Latency Time (ms)

Accuracy rate of latency test : Video 4

Wrong

No Diff

Correct

Video 4: Train Moving Slowly

• Two trains run in opposite direction at a speed around 30 km/hr

• Lots of reference point exists in this video,

e.g., the relative movement of two trains, or fixed electric pole

• The tester can judge the latency easily over 50 ms

Speed: 1.74 deg/sec

Periodic: Yes (relative movement of two trains)

Reference point: Yes

14 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Experimental Result – Latency time test -

Experiment Operation Type ND of Latency

Waving Hands Indirect Periodic

And w/ reference point

50 ms

Train Moving Slowly

Indirect Periodic

and w/ reference point

50 ms

Train Moving Quickly

Indirect Non periodic

And w/o reference point

> 100 ms

People Passing By

Indirect Non periodic

And w/o reference point

> 117 ms or unstable

• People can detect time difference from around 50 ms

• Without reference point, noticeable difference increase Over 100 ms

System Latency Requirement: < 50 ms

15 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Demands for CMS - from primary and latency time test -

Low Motion Blur

Low Latency

Maximum allowance: 50 ms

High Luminance and High Contrast Ratio

Need more brightness in daylight

Need more contrast at night driving

16 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

1. Introduction

• Background

• Primary experiment

2. Subject-Evaluation Results of the Latency

• Latency time test

3. Technologies of Our Prototype

4. Summary

17 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

High Frame Rate Driving

60 Hz

240 Hz

frame1

Image sensor

exposure

Panel driving

frame1

0

768 Line

Exposure time

frame2 frame3

frame2 frame3 time frame4 frame5

frame4 frame5

frame6

frame6

frame7

frame7

frame8

frame8

frame1

Image sensor

exposure

Panel driving

0

Exposure time

frame1

time frame2

768 Line

768 Line

768 Line

Low Motion Blur

Clear Image is Captured by Shorter Scanning Time

18 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Fast Response Time LC

BET(blur edge time) vs. frame rate

clear

blur

Low Motion Blur

Improve Blur by Using SLC-IPS

Short-range Lurch Control In-Plane Switching (SLC-IPS) T. Matsushima et al,

“New Fast Response Time In-Plane Switching Liquid Crystal Mode”,

SID 2015 digest papers, 43.2, pp 648-651, 2015

19 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Synchronized Capturing and Playback Low Latency

4.536 ms Latency from Imager to Panel

1frame

System processing

Image Sensor

VSYNC

Image sensor

exposure

Panel driving

Panel VSYNC

Image sensor

data out

1frame

Line

0

768

Line

0

768

Line

0

768

Line

0

768

Exposure time

4ms

Output delay

5.05us

System delay

179us

1H time

5.37us

2frame 3frame

2frame 3frame time

time

Output delay (1H)

5.05us

1line

2line

3line

System delay (1H)

179us

1line

2line

3line

1Horizontal time

5.37us

1line

2line

3line

4.356 ms

20 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Under 100,000 Lx

HSV color space for RGB display

New HSV color space for RGBW display

Saturation S

Hue H

Low power mode brightness

outdoor mode brightness

Brightness (V)

Excellent Visibility under Brighter Environment

“Outdoor Mode” of RGBW Technology

Original picture

RGBW Outdoor mode

Conventional RGB panel

High Luminance

21 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

R G B W

WM IP

LED Dr. ・ ・ ・

Backlight

RGBW Display

Backlight illumination Backlight illumination

Local Dimming with RGBW Technologies

1. New Image Processing IP for local dimming

2. Special Light Guide Design

for edge-lit backlight

Improve Contrast Ratio by Using Local Dimming

High Contrast Ratio

22 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Over View of Our Prototype CMS

Specification

Imager size 1/2.3 inch

Display size 6.5 inch diagonal

Pixel number 1,280 x 768

Resolution 231 ppi

Pixel arraignment RGBW

Luminance 2,000 cd/m2 @OutdoorMode

Contrast ratio 20,000:1 with local dimming

Color gamut 76.5%

Frame rate 240 Hz

Note IPS-NEO

Low Motion Blur

Low Latency

High Luminance and High Contrast

23 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

1. Introduction

• Background

• Primary experiment

2. Subject-Evaluation Results of the Latency

• Latency time test

3. Technologies of Our Prototype

4. Summary

24 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Comparison

R G B R G W

R G W R G B

R G B R G W

R G W R G B

Features Improvement Conventional RGB Display

Prototype

1

• Quick Response Time

Tr+Tf (25deg.) = 20 ms Tr+Tf = 8.3 ms

Tr+Tf (-20deg.) = 146 ms Tr+Tf = 51 ms

• High Frame Rate 60 Hz 240 Hz (4times faster)

2 • Low Latency • Direct Synchronization

from Imager to Display Over 100 ms 4.356 ms

3

• • High Luminance

1000 cd/m2 2000 cd/m2 (6.5inch-demo)

• Local Dimming • Dynamic Contrast

improvement 1000：1 20000 : 1

25 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Conclusion

The ultra-low latency CMS has been developed, by adopting

Low Motion Blur

SLC-IPS technology

High speed scan panel

Low Latency

Synchronization of imager and display panel

High Luminance and High Contrast Ratio

RGBW technology with outdoor mode

Local dimming technology

These technologies becomes very important for safe driving.

By adding functions to CMS, the safety far beyond conventional optical mirrors will be provided in the near future.

26 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

Thank you for your kind attention.

27 Copyright © 2017-18 Japan Display Inc. All Rights Reserved.

本資料は、『電子ディスプレイの人間工学シンポジウム2018』のために作成されたものであり、

当社の発行する株式その他の有価証券への勧誘を構成するものではありません。本資料に記載される業界又は市場動向に関する情報は、現時点で入手可能な情報に基づいて作成しているものであり、当社がその真実性、正確性、合理性及び網羅性について保証するものではありません。また、本資料に記載される当社グループの計画、見積もり、予測、予想その他の将来情報については、現時点における当社の判断又は考えにすぎず、実際の結果は、国内外の個人消費その他の経済情勢、為替動向、スマートフォンその他の電子機器の市場動向、主要取引先の経営方針、原材料価格の変動等により、本資料記載の内容と大きく異なることがあります。