1

Sensor-based Body Gesture Tracking System for Natural User Interface in E-learning

Sunghyun Song

SungKyunKwan Univ.

Department of Human ICT Convergence

OUTLINE

• Introduction

• Related Work

• Proposed System

• Pilot Study

• Improvement

• Experimental Condition

• Result & Discussion (Data log, TLX, SUS, and interview)

• Conclusion

2

Introduction

• E-learning is a rapidly growing educational sector as much as 8% a year through 2016

T.Navio, “Global E-learning Market 2013-2016”, Infiniti Reserch Limited, (2013)

• The main advantage of audio-video class material is that students can flexibly play, pause,

rewind or skip around material according to their needs.

Bassili, J. N. and Joordens, S.: Media player tool use, satisfaction with online lectures and examination performance,(2008)

• According to these observations, researchers have suggested systems adjusting video

playback properties automatically. For example, reducing playback speed if a student is

taking notes.

Park, M., Kang, J., Park, S., Cho, K.: A Natural User Interface for E-learning Learners, (2014)

• However, there is no current video-playback system that can tracks user activities to

adjust dynamically video playback in order to support eLearning. A novel natural user

interface that tracks a user’s head movements to infer playback speed without explicit user

input is needed.

3

Related Work (1)

• Tracking head position to control aspects of computer use

- Lean and zoom: A system based on a camera calculates the distance between head

and screen in order to adjust the magnification of displayed contents.

Harrison, C., and Dey, A.K.: Lean and Zoom: proximity-aware user interface and content magnification (2008)

- Based on head movement, a system can adjust not only the zoom but also the location

and orientation of contents.

Yamaguchi, K., Komuro, T., Ishikawa, M.: Ptz control with head tracking for video chat (2009)

• Head-trackers used to extend the capabilities of pointer input.

- How camera-based system that tracks the user’s face can support pointing or selecting.

Kjeldsen, R.: Head Gestures for Computer Control (2001)

- Face-tracking system that allows users to quickly locate the cursor across multiple display.

Ashdown, M., Oka, K., Sato, Y.: Combining head tracking and mouse input for a GUI on multiple monitors (2005)

4

Related Work (2)

5

• Video browsing techniques

- System that lets users navigate the timeline of a video by directly dragging objects in the

displayed scene along their past or future visual path.

Dragiceive, P., Ramos, G., Bibliowitcz, J., Nowrouzezahrai, ..:Video browsing by direct manipulation (2008)

- SmartPlayer semi-automatically adjusts the video playback speed depending on the

complexity of the scene presented or user defined events.

Cheng, K., Luo, S., Chen, B., Chu, H.:SmartPlayer: user-centric video fast-forwarding (2009)

- Video control through gestures in an interface powered by the Microsoft Kinect.

Chu, T., Su, C.: A Kinect-based Handwritten Digit Recognition for TV Remote Controller (2012)

- Playful physical interface that lets children control video content through squeezing and

stretching bubble-like objects.

Ryokai, K. Raffle, H., Horii, H., Mann, Y.: Tangible video bubbles (2010)

Related Work (3)

6

• Natural User Interface (NUI)

- NUI means that expand beyond the mouse and keyboard and start incorporating more

natural forms of interaction such as touch, speech, gestures, handwriting, and vision.

Ballmer, S.: CES 2010:A Transforming Trend-The Natural User Interface (2010)

- NUIs are those that enable users to interact with computers in the way we interact with the

world.

Jain, J., Lund, A., Wixon, D.: The future of natural user interfaces., In CHI’11

Proposed System

7

• State & control

- A custom head-tracker system can

reliably detect where the user is looking

and discern among four different situations.

The video playback speed is then adjusted

to conform with the current activity.

- This system doesn’t request any additional

manipulation compared to previous research.

Proposed System

8

• Prototype HW

- six axis gyro+accelerometer sensor (MPU-6050)

mounted on a pair of headphones, and

connected with wires to a controlling unit.

(This sensor is capable of sensing head angle)

- Arduino Leonardo is micro controller serially

connected to a PC through USB port. This receive the value from sensor and process

pitch and yaw angle for decision of states.

- Calibration button on the mini bread board for setting standards of each state’s angle.

Pilot Study and Result

9

• Evaluation for the proper video playback speed

- 6 participants ( graduate students, 1 female, age 27-33 )

- Each student was presented with two tasks in random order following within-subject design

- First task : watch a short lecture video and take notes with only keyboard

- Second task : watch another video and take notes with my system and the keyboard.

- Result t(5)=4.03 , p<0.01

: keyboard input for automatic system = avg 1.3 (SD=1.2)

: keyboard-only input = avg 3.6 (SD=2.3)

• Semi-structure interview

- 0.6x is too slow and hard to understand

- 0.8x is enough for taking notes and slow condition. 0.6x 0.8x

System can reduce direct manipulation of user.

Improvement

10

• Observation from the pilot, 3 states are added

- Focus (⑦)

Context: see contents in detail / full of interest

Posture: lean forward to monitor

Control: pause

- Skimming (⑨)

Context: contents are easy or not important

Situation: reclining position

Control: fast forwarding (1.2x)

- Dwell (⑥)

Context: Student lose attention for a long time

Situation: turn head + stay in that more than 5 sec

Control: pause + rewind 15 sec

Improvement (2)

11

• Final System overview

- The seven different postures. States are

sensed by combined angular data from a head-

mounted six-axis IMU and distance data from

an infrared proximity sensor placed at the

bottom of the screen and facing the user.

Experimental Condition

12

• Material

- Video: 8min for each video clip “cold and warm fronts”, “the solar system”

• Condition

- 12 participants (7male and 5 female )

- Control condition: only keyboard input

- Tracked condition: both keyboard and tracking system

- The presentation of the videos is fully balanced.

User

Ex 1 Ex 2

Prototype Video Prototype Video

p1 O V1 X V2

p2 X V1 O V2

p3 O V2 X V1

p4 X V2 O V1

Experimental Condition

13

• Evaluation List

- Log data (e.g. Time for staying each states, The number of transition, and so on)

- TLX ( Task Load Index ) to check mental, physical load and so on.

- SUS ( System Usability Scale ) to check usability and learnability

- Semi-structured interview to check pros, cons and additional points

• Experiment order

1. Introduce our experiment

2. Practice using prototype ( 2min )

3 (6). Watch the video

4 (7). Test for understanding ( also, induce taking down a lecture )

5 (8). TLX

9. SUS

10. Interview

Repeat one more

J, Brooke.: SUS: a ‘quick and dirty’ usability scale (1996)

Experiment

14

• Raw data

Log data TEST, TLX, SUS and so on

15

Result & Discussion_ System Log

Size of the circles = Total time was spent in each states

Line = State Transition

• In tracked, most common transition took place btw ‘Normal’ and ‘Others’

• Transition btw ‘Normal’ and ‘Notes’ is occurred more frequently in tracked C.

than in control C.

• In tracked C, transition btw Normal and Distract is also many.

16

Result & Discussion _ TLX

From initial t-test

• Significantly lower Overall Workload(p=0.016) in the tracked condition

• Improvement in

Mental Demand(p=0.011), Temporal Demand (p<0.002), and Effort (p<0.04)

Development of NASA-TLX(Task Load Index): Results of empirical and theoretical Research (1988)

SG. Hart.: NASA-task load index (NASA-TLX); 20 years later (2006)

17

Result & Discussion _ SUS

J, Brooke.: SUS: a ‘quick and dirty’ usability scale (1996)

High Score Rank

• Q4: “Can learn by self”

• Q7: “Easy to learn”

• Q3: “Easy to use”

• Question

• Result in Likart 5 scale

learnability

18

Result & Discussion _ Semi-Structured Interview

• Question

(1) (Pros) Useful or comfortable point?

(2) (Cons) Inconvenience points or things to be improved?

(3) Is there any function to be added?

(4) Any other things?

• Answer_ key points

(1) “I want to get cue of transition... I felt anxiety because I don’t know which states I’m in”

=> People need feedback when transition be occurred. However, it should not disrupt user.

(2) “The speed of changing playback speed is too fast. I felt abnormal.”

=> System need delay btw 0.1x differency

(3) “It will be better, if I can customize the transition degree of each states.”

(4) “If the system provide the way of manipulating deliberately, will be better.”

=> For example, turn left your head two time quickly = 15 seconds rewind”

19

Conclusion

• Proposed Natural User Interface can

- be easy to learn and use based on SUS.

- lower user’s cognitive workload specifically in Mental, Temporal, and Effort based on TLX.

- be dynamically used during the studying based on Data Log.

• This system could be highly beneficial in learning scenarios.

• For better usefulness, below things could be added.

- Customizing the degree of transition

- Smooth change of playback speed

- Manipulating intentionally for more various control such as rewinding or skipping

- Feedback for transition

Reference

20

[1] T.Navio, “Global E-learning Market 2013-2016”, Infiniti Reserch Limited, (2013)

[2] Bassili, J. N. and Joordens, S.: Media player tool use, satisfaction with online lectures and examination performance, in J.

Distance Education, 22(2), (2008)

[3] Park, M., Kang, J., Park, S., Cho, K.: A Natural User Interface for E-learning Learners, in I.J. Multimedia and Ubiquitous

Engineering 9(7), (2014)

[4] Harrison, C., and Dey, A.K.: Lean and Zoom: proximity-aware user interface and content magnification in Proc. of CHI '08,

507-510 (2008)

[5] Yamaguchi, K., Komuro, T., Ishikawa, M.: Ptz control with head tracking for video chat, in CHI EA '09, 3919-3924. (2009)

[6] Kjeldsen, R.: Head Gestures for Computer Control, in IEEE ICCV Workshop on RATFG-RTS'01, 61-67 (2001)

[7] Ashdown, M., Oka, K., Sato, Y.: Combining head tracking and mouse input for a GUI on multiple monitors, in CHI EA '05,

1188-1191. (2005)

[9] Dragiceive, P., Ramos, G., Bibliowitcz, J., Nowrouzezahrai, ..:Video browsing by direct manipulation, in Proc. of CHI '08.

(2008)

[10] Chu, T., Su, C.: A Kinect-based Handwritten Digit Recognition for TV Remote Controller, in ISPACS'12, 414-419. (2012)

[11] Ryokai, K. Raffle, H., Horii, H., Mann, Y.: Tangible video bubbles, in CHI EA'10, 2775-2784. (2010)

[12] Cheng, K., Luo, S., Chen, B., Chu, H.:SmartPlayer: user-centric video fast-forwarding, in Proc of CHI '09, 789-798. (2009)

Reference

21

[13] Kim, J., Guo, P.J., Cai, C.J., Li, S., Gajos,K.Z., Miller, R.C.: Data-driven interaction techniques for improving navigation of

educational videos, in Proc. UIST’14, 563-572 (2014)

[14] Crossan, A., McGill, M., Brewster, S., Murray-Smith, R.” Head tilting for interaction in mobile contexts, in MobileHCI’09,

article 6 (2009)

[15] Kurihara, K.: CinemaGazer: a system for watching videos at very high speed, in AVI’12, 108-115 (2012)

[16] Smith, J.D, Graham, T.C.:Use of eye movements for video game control, in ACE 2006

[17] Hart, S. G., Staveland, L. E.: Development of NASA-TLX(Task Load Index): Results of empirical and theoretical Research,

Advances in psychology, 52, 139-183 (1988)

[18] Hart, S. G.: NASA-task load index (NASA-TLX); 20 years later, In Proc of the human factors and ergonomics society

annual meeting, Vol. 50, No. 9 (2006)

[19] Brooke, J.: SUS: a ‘quick and dirty’ usability scale, Usability evaluation in industry, 189-194 (1996)

[20] Ballmer, S.: CES 2010:A Transforming Trend-The Natural User Interface (2010)

[21] Jain, J., Lund, A., Wixon, D.: The future of natural user interfaces., In CHI’11. 211-214 (2011)

[22] Schilit, B., Adams, N., Want, R.: Context-Aware Computing appllication, In IEEE WMCSA’94, 85-90 (1994)

[23] Asteriadis, S., Tzouveli, P., Karpouzis, K., Kollias, S. : Estimation of behavioral user state based on eye gaze and head

pose—application in an e-learning environment. Multimedia Tools and Applications, 41(3), 469-493. (2009)

22

Thank you!

Sunghyun Song

23

Appendix_TLX

http://pro.sagepub.com/content/50/9/904.full.pdf

* Paper

24

Appendix_SUS

http://www.tbistafftraining.info/smartphones/documents/b5_during_the_trial_usability_scale_v1_09aug11.pdf

https://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CBwQFjAAahUKEwjk86Lp443JAhWmJ6YKHeWuBcQ&url=http%3A%2F%2Fwww.measuringux.com%2FSUS_Calculation.xls&usg=AFQjCNGLIUugJzqJtVqOZTDwkg-dt8Iv_w&sig2=5XV_IGss8e3VmllCHoKyBw&bvm=bv.107467506,d.dGY

* Paper

* Excel Sheet download (Ctrl+C and V)

25

•Context Aware Computing

- Head pose could be used for tracking student’s

attention

Asteriadis, S., Tzouveli, P., Karpouzis, K., Kollias, S. : Estimation of

behavioral user state based on eye gaze and head

pose—application in an e-learning environment. Multimedia Tools and

Applications, 41(3), 469-493. (2009)

- Context-triggered actions are simple IF-THEN rules

used to specify how context-aware

systems should adapt.

Bill, S., Norman, A., Roy, W.: Context-Aware Computing Appllication (1994)

Appendix_SUS

Concept idea

26

System will

(1) Notice Context from the posture of Students

(2) Control video playback speed automatically according to their posture.

( e.g If posture1 then playback speed1. )

As a result, system will be gesture-based user interface