測量虛擬實境在行動裝置上的時間遲滯及所造成的影響

歐陽明 (Ming Ouhyoung) 教授,

台大網路與多媒體研究所, 資工系(前系所主任, 電資學院副院長)

2017/5 VR 教育論壇

大綱

• VR/AR 理論上的四大難題:人才培育

•測量虛擬實境在行動裝置上的時間遲滯

•眩暈的症狀及相關問題: 時間遲滯可能造成的影響

•娛樂及醫療科技結合之應用機會—人才培育

VR/AR 定義• Virtual Reality: 虛擬實境

a joke: 1992 我開創的翻譯, 碩士論文需要, 當時中國時報翻譯為虛擬現實, 虛幻XX, 擬真xx,後來兩岸採用.

虛擬實境（Virtual Reality），簡稱VR技術，也稱人工環境，是利用電腦模擬產生一個三度空間的虛擬世界，提供使用者關於視覺、聽覺、觸覺等感官的模擬，讓使用者如同身歷其境一般，可以及時、沒有限制地觀察三度空間內的事物。

擴增實境（Augmented Reality，簡稱AR），是一種實時地計算攝影機影像的位置及角度並加上相應圖像的技術，這種技術的目標是在螢幕上把虛擬世界套在現實世界並進行互動。

It seems that VR/AR is just going through the Cambrian explosion!(寒武紀大爆發)

•Google glass (failed product)

•Oculus Rift/Rift 2 (purchased by Facebook)

•Google Cardboard VR project

•Magic Leap secures 542M by Google Inc.

•Microsoft HoloLens project

all these happened in 2014/2015!

我的研究: 虛擬實境 (VR)

(1) 3D 跳舞機: video demo gesture_demo_2 (游名揚, 雷亞科技Rayark Inc.)

(2) 繽紛魔鏡(穿衣鏡): mp4 demo demo2

(3) 夢幻鼓: video demo

(4) 外科手術: Da Vinci (22 sets in Taiwan now), 白內障手術 (demo)

(5)逛虛擬新屋: video demo 台大360 街景服務(Google)

(6) 有表情並說話的人頭: video demo 分解動作Digital_emily_prpoject

VR/AR 理論上的4大難題

•1. 視覺解析度 display resolution:•2. 時間遲滯 Display Latency: should be less than 50ms

; >= 100 ms 會造成不真實感, 甚至暈眩,噁心等副作用•3. 視覺近端與遠端變焦 (睫状肌紧张與鬆弛)(fixed focus vs. variable focus): 眼睛可以在近與遠的物體定焦, 例如看山景與茫茫大草原

•4. 實景與電腦產生的虛景對齊 (Registration of real and virtual environments)

Extremely cheap HMD: US$6-20

30 years from UNC HMD: much cheaper! (Display, Sensors)

Polhemus position/orientation sensor: US$2000, 25 years ago.Display: LCD TV screens: US$400 per set, and need 2 of them!

Android Apps

For HTC, Samsung, Xiaomi Inc. etc.: best VR Apps,

(前4個很棒!)

• 1. Rollercoaster VR

• 2. Cardboard VR app

• 3. Cosmic Rollercoaster VR app

• 4. Jurassic Dino VR

• 5. Crazy Swing VR

• 6. Space Terror (need Bluetooth controller to play)

3D Video from Youtube

Key word: “3D” or “3D trailers” “360 vr” for 3D glasses

Example titles:

Air Racers, real airplane stunt show

Samsung 3D demo

Avatar 3D 阿凡達 Titanic 3D 鐵達尼號WoW, World of Warcraft

Finding Nimos, is one good example,

NYT vrse, 紐約時報難民報導 360 video, New York marathon 360

Google Street View (VR mode) 街景服務建議景點

• Lower falls of the Yellowstone,Old Faithful, YellowstoneMusee d'Orsay, ParisMusee du Lourve, Paris,Niagara falls, On, Canada, (Google, 坐船游覽, move forward)Himalayas (ABC， Annapurna Base Camp， by Jonathan Jenkins, clear view of Everest，喜馬拉雅山）Avila Spain (城牆，Avila 古城，馬德里附近，選Google 第四五張，有三角型箭墩的道路,效果最佳）Segovia Spain, Sevilla, SpainToledo, Spain

好玩景點

• Yosemite National Park,日月潭（選第5段左右，Google 提供的湖邊木板棧道，一步步走到纜車站）天壇，北京，北京故宮，黃山，九寨溝，張家界( Key word: 就是這三字，前面幾張都很美）玉山，（第二張，民間第一張，清楚的主峰）玉山，找到Google 提供的北峰，可ㄧ步一步爬到頂。台大校門到椰林大道（Key word: 台大地址，羅斯福路4段1號）

Catarata de Iguazu, (伊瓜蘇瀑布, Devil’s throat, Catarata del Iguazu, key word: 伊瓜蘇巴西，順著木板棧道走，非常壯觀）Politvice Lakes National Park, Croatia, （十六湖，南歐觀光勝地，很美的湖泊, key word: Politvice, )Zion National Park, Us

新的人機介面(User Interfaces)

• 為什麼有新需求?

戴上HMD 手就摸不到螢幕了, Touch Interface 失效!

. 有的還看不見外面世界! (VR vs. AR, 寶可夢Pokemon Go 的例子)

Oculus Rift 2 vs. HTC Vive Pre

Oculus Rift HTC Vive

Display OLED OLED

Resolution 2160 x 1200 2160 x 1200

Refresh Rate 90Hz 90Hz

PlatformOculus Home SteamVR

Field of view 110 degrees 110 degrees

Tracking area TBA 15 x 15 feet

Built-in audioYes Promised, not yet available

Built-in mic Yes TBA

Controller Oculus Touch, Xbox One controller

SteamVR controller, any PC compatible gamepad

Sensors Accelerometer, gyroscope, magnetometer, 360-degree positional tracking

Accelerometer, gyroscope, laser position sensor, front-facing camera

One of the best apps in AR in the future, if combined with machine learning, (台灣有22台)

• DaVinci surgery machine

VR/AR 理論上的4大難題

• 1. 視覺解析度 display resolution: (看到120角的場景)

• 2. 時間遲滯 Display Latency: should be less than 50ms

; >= 100 ms 會造成不真實感, 甚至暈眩,噁心等副作用

• 3. 視覺近端與遠端變焦 (睫状肌紧张與鬆弛)(fixed focus vs. variable focus): 眼睛可以在近與遠的物體定焦, 例如看山景與茫茫大草原

• 4. 實景與電腦產生的虛景對齊 (Registration of real and virtual environments)

A simple method: video demo

• Video from Totoro’s 1994 recording (DEMO) , the HMD motion, and its’ recording

New Instrument designed to measure latency (DEMO) (Sharp Z2)

New device to measure motion to photons latency

測量虛擬實境在行動裝置上的時間遲滯(Latency)手機種類 時間遲滯時間（毫秒） 時間遲滯時間（毫秒）

PsViewer Google Street View

HTC U Play 79.2 85.4

Sharp Z2 60.4 100

Sony XZ 100 97.9

Redmi pro (with OLED) 58.3 ms 58.3 ms

Google Pixel Phone 58.3-80 ms 50 ms

iPhone 6s 58.3 ms 58.3 ms

Latency Compensation

• Jiann-Rong Wu and Ming Ouhyoung, "On Latency Compensation and its Effects for Head Motion Trajectories in Virtual Environments," The Visual Computer, vol. 16, no. 2, pp. 79—90, 2000.

• Azuma R, Bishop G (1994) Improving static and dynamic registration in an optical see-through HMD. SIGGRAPH’94 Conference Proceedings,

• Jiann-Rong Wu and Ming Ouhyoung, "A 3D Tracking Experiment on Latency and Its Compensation Methods in Virtual Environments", Proc. of UIST'95 (User Interface and Software Technoilogy 1995), pp. 41-49, ACM Press Pittsburgh, USA

• From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, 2015 IEEE Trans. On Visulization & CG

by Peter Lincoln, Alex Blate, Montek Singh, Turner Whitted, Andrei State, Anselmo Lastra, and Henry Fuchs

Reducing the latency: Prediction (by Kalmann Filter, for example, for 20-40 ms prediction of head/hand motion)

• Occulus as well as HTC Vive

VR 真實可用嗎?How to verify if VR/AR is useful? (How Real?)

• Fred Brooks, ACM Siggraph 2002

Observation: Skin conductance, respiration, heart beat rate.

觀察緊張時的:

心跳率, 呼吸, 皮膚導電率 (出汗, 可增加導電)

Meehan, M., B. Insko, M.C. Whitton, F.P. Brooks Jr., 2002: "Physiological Measures of Presence in Stressful Virtual Environments," ACM Transactions on Graphics, 21, 3: 645-652. (Proc. of ACM SIGGRAPH 2002, San Antonio, TX).

實驗裝置: 夾板墊高, 4公分厚(高度)而已.

Physiological Measures of Presence:我在 UNC-CH, 2015, 實測中

心跳變異: 有無增加夾板 (下圖左側無, 右側有)

實驗結果:1. 心跳變化 2. 電腦(銀幕)顯示速率, 有影響

• We found that change in heart rate satisfied our requirements for a measure of presence, change in skin conductance did to a lesser extent, and that change in skin temperature did not. Moreover, the results showed that inclusion of a passive haptic element in the VE significantly increased presence and that for presence evoked: 30FPS > 20FPS > 15FPS.

VR/AR 理論上的4大難題

• 1. 視覺解析度 display resolution: (看到120度角的場景)

• 2. 時間遲滯 Display Latency: should be less than 50ms

; >= 100 ms 會造成不真實感, 甚至暈眩,噁心等副作用

• 3. 視覺近端與遠端變焦 (睫状肌紧张與鬆弛)(fixed focus vs. variable focus): 眼睛可以在近與遠的物體定焦, 例如看山景與茫茫大草原

• 4. 實景與電腦產生的虛景對齊 (Registration of real and virtual environments)

• 上述第一與第三, 最近的未來才有解,

• 變焦 (睫状肌紧张與鬆弛)(fixed focus vs. variable focus):

• 黃輔中(Fu-Chung Huang, NTU BA)• "The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-Eye Light

Field Display with Focus Cues",Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein,Youtube LinkACM SIGGRAPH 2015, Los Angeles (see some press coverage below)

• "Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays",Fu-Chung Huang, Gordon Wietzstein, Brian Barsky, and Ramesh Raskar,- 10 World Changing Ideas 2014, Scientific AmericanACM SIGGRAPH 2014, Vancouver (see some press coverage below)

Innovation: vergence-accommodation conflict inherent to

all stereoscopic displays

近,中,遠, 三段聚焦

VR/AR 理論上的4大難題

• 1. 視覺解析度 display resolution: (看到120度角的場景)

• 2. 時間遲滯 Display Latency: should be less than 50ms

; >= 100 ms 會造成不真實感, 甚至暈眩,噁心等副作用

• 3. 視覺近端與遠端變焦 (睫状肌紧张與鬆弛)(fixed focus vs. variable focus): 眼睛可以在近與遠的物體定焦, 例如看山景與茫茫大草原

• 4. 實景與電腦產生的虛景對齊 (Registration of real and virtual environments)

What is the best resolution in display? Problem 1:

視覺解析度 display resolution:

•For example, for mobile device display:

For Google Cardboard VR HMD, if the field of view (angle) is 60 degrees, the required screen resolution is around 5700x5700,

As in Oculus Gear VR, or HTC Vive, the filed of view is 90 degrees or above, the required resolution should be 10Kx10K !

(Because in visual acuity: eye’s ability to distinguish

two points of light is limited to 1.5 – 2.0 mm at a distance of 10 meters. (or 2 microns on the retina)).

New type of display, other than LCD and OLED: microLED

• Playnitrade Inc. (李允立)

can possibly provide a display of resolution 10Kx10K in a few years.

• Semiconductor components for laser diodes (LD) , more density, brighter, and 85% efficiency in energy consumption.

(UV-LED)

• LCD has a energy efficiency of 5% only, while OLED is about 35%.

眩暈的症狀及相關問題(I)A. 時間遲滯(Latency)：

在使用過程中的時間遲滯(Latency)如果超過50ms之後，使用者對於眩暈的反應可能會更為快速，使得使用者可以進行虛擬現實體驗的時間快速減少。

B. 視角 Field Of View (FoV)：

平常人類眼睛視野範圍大概是單眼120度左右，而在虛擬現實的環境當中，如果同時將在虛擬環境中可見的FoV加大，一方面會增加真實感，不過在固定的時間遲滯之下，對於眩暈的感受也會漸漸變得明顯，我們猜測是同一時間所要處理的資訊量也相對應的增加，所以加速了使用者的疲倦感和眩暈症狀。

C. 螢幕解析度 (Display Resolution)：

現在硬體上對於虛擬現實所使用的螢幕，它的解析度對於真實體驗來說還是不夠的，而目前硬體上的發展也慢慢將雙眼螢幕的解析度從水平像素1920進步到水平像素4K的程度，雖然真實感會增加，但是在固定的時間遲滯之下，反而眩暈也漸漸變得明顯，我們認為發生原因應該和FoV增加所導致眩暈的原因一樣。

眩暈的症狀及相關問題(II)

D. 轉頭速度 (Motion Velocity)：

在體驗虛擬現實的過程中，使用者自身的頭部轉動速度也會影響眩暈症狀的發生，可以想見的是如果頭部轉動速度越快，會讓眩暈的症狀越快出現，所以通常會建議不要轉動太快，以減少眩暈的症狀。

E.運動模糊 (Display/Motion Blur)：

在一般情況之下，人類快速轉動頭部時，所看到的影像不會是清楚的，而在虛擬現實當中畫面基本上一直都會是清晰的狀態，這樣對於人類視覺來說是一種不正常的現象，也會讓眩暈的症狀比

其他應用與機會: new sensors

•機會: App 軟體微型創業, 與健身,銀髮, 及醫療科技結合

• position sensors, camera-based tracking device (using Church’s algorithm): problem 2: 時間遲滯 Display Latency

• Super-resolution display (eg. UV-LED), up to 10k by 10k

• Image processing and SOC for super-resolution.

• new type HCI (Finer pad, gesture control, new HCI, game pad with joystick using USB)

• form factors for HMD

Application and opportunity:Apps, new sensors, higher resolution display

•Apps development, including health and medical care

•position sensors, or camera-based tracking device (using Church’s algorithm): problem 2: Latency reduction

•Super-resolution display (eg. UV-LED), up to 10k by 10k

• Image processing and SOC for super-resolution.

•new type HCI (Finer pad, gesture control, new HCI, game pad with joystick using USB)

• form factors for HMD

END

•SCOPE+: 歐陽 (白內障手術AR訓練模擬器)

• Docomo Vision 2020

• MicroSoft’s Concept: Future Vision 2020

• Samsung’s vision: display

Appendix:

A Stereoscopic Video See-ThroughAugmented Reality Microscope

Yu-Hsuan Huang Tzu-Chieh Yu Pei-Hsuan Tsai

Yu-Xiang Wang Wan-Ling Yang 歐陽明(Ming Ouhyoung)

National Taiwan University

Scope+ 在 ACM UIST 2015 Conference Demo 中脫穎而出獲選為今年的 Best Demo！

Scope+ : A Stereoscopic Video See-Through Augmented Reality Microscope

在 ACM UIST 2015 及 ACM Siggraph Asia 2015 Emerging Tech 中即時展示。二者均為圖學、人機介面領域頂尖會議

2016/4/14 今周刊封面故事之一SCOPE+ 報導

43

白內障手術步驟示意圖

44

眼科: 白內障手術AR/VR模擬台(DEMO 2)

45

46

47

短的VIDEO, 電路板https://www.youtube.com/watch?v=vV83pWR4Dko&feature=youtu.be長的VIDEO, SCOPE+https://www.youtube.com/watch?v=rmohFEAreUs&feature=youtu.be

https://01.org/zh/intel-deep-learning-framework?langredirect=1

Reference

AR + VR

• Microsoft Inc. HoloLens Project, Windows 10, Holographics Live Demo

• https://www.youtube.com/watch?v=kCMxBw-utEY&feature=youtu.be

• https://www.youtube.com/watch?v=vsKMi1377cU