i

中臺科技大學醫療暨健康產業管理系碩士班

Central Taiwan University of Science and Technology

Graduate Institute of Healthcare Administration

碩士論文

Master Thesis

指導教授：陳永福

Adviser: Yung-Fu Chen, Ph. D

Microsoft Kinect虛擬復健系統設計

A Microsoft Kinect based virtual rehabilitation system

研究生:艾而帝

Graduate Student: Erdenetsogt Davaasambuu

中華民國 一０二 年 六 月

2013

ii

iii

iv

v

Acknowledgements

Many thanks to my advisor, Prof. Yung-Fu Chen, for providing me the

opportunity and financial support to study in CTUST. I also would like to express

my appreciation to Dr. Chung-Liang Lai, Chief of Department of Physical

Medicine and Rehabilitation, Taichung Hospital, for helping and giving me

professional advices on this study. During my stay in Taiwan, I have made a lot

of Taiwanese friends, who are always with me when I was in difficult situations.

Without the assistance and participation of the staffs and patients of National

Taichung Hospital, my thesis cannot be finished in time. I deeply appreciate Dr.

Chin-Tun Hung, Chairman of the Department, and Dr. Chin-Chih Ho, and other

faculty members and staffs of the Department of Healthcare Administration.

vi

ABSTRACT

Virtual reality technology is currently widely applied in physical

rehabilitation therapy. The ability to track joint positions for Microsoft Kinect

might be useful for rehabilitation, both in clinical setting and at home. Currently,

most systems developed for virtual rehabilitation and motor training require quite

complex and expensive hardware and can be used only in clinical settings. Now,

a low-cost rehabilitation game training system has been developed for patients

with movement disorders; it is suitable for home use under the distant supervision

of a therapist. This research explores the potential and the limitations of the

Kinect in the application of e-rehabilitation. Commercial Kinect games present

disadvantages that are mainly bounded to the fact that they are not developed

for rehabilitation purposes. We evaluated the tools that could be used to help

promote physical rehabilitation at home by reducing the frequency of hospital

visits, resulting in the reduction of healthcare cost. A prototypic system has been

developed for the evaluation of 5 different games, which are useful for e-

rehabilitation. As we see from the result, it indicated a slightly positive outcome

for the patients after got involved the treatment. Also the questionnaire outcome

reveals that the designed exercises were perceived as effective and easy in

operation (p<0.05).

Keywords: Virtual reality; E-rehabilitation; Kinect, Balance training;

vii

TABLE OF CONTENTS

ACKNOWLEDGEMENTS………………………................... v

ABSTRACT……………............................................................ vi

TABLE OF CONTENTS........................................................... vii

LIST OF TABLES..................................................................... ix

LIST OF FIGURES................................................................... x

CHAPTER ONE: INTRODUCTION...................................... 1

1.1 Virtual Reality……........................................................... 1

1.2 Virtual rehabilitation……................................................. 2

1.3 Motivations and Objectives.............................................. 6

1.4 Related works…………..................................................... 8

CHAPTER TWO: MICROSOFT KINECT

DEVELOPING ENVIRONMENT……………………….…

10

2.1 Microsoft Kinect..…...…................................................... 10

2.2 Design guidelines……………........................................... 11

2.3 Choosing Open Source Drivers and SDKs........................ 12

2.4 Kinect API…………………….......................................... 14

2.5 Skeleton stream……………...…..…………..................... 15

2.6 Math formulas implementation……………..................... 17

Smoothing skeleton data………………… 19

2.7 Software development….…………………..................... 20

Login and user management……………… 24

viii

CHAPTER THREE: METHODOLOGY................................ 26

3.1 Interface Design……............................................................ 26

3.2 Development of Kinect Virtual Rehabilitation System...... 28

Ping Pong…………………..……………………... 28

Balance shift……….……………………………... 29

Window cleaner…….……………………………... 31

Step on It………...………………………............... 32

Coin Collection……………….………................... 33

3.3 Experiment……………..….…......................................... 33

3.4 Subject and Statistic analysis………................................ 36

3.5 Rehabilitation Assessment………..................................... 37

Biodex Balance Machine…..................................... 39

Berg Balance Scale.................................................. 40

Get up and Go test……..……………..................... 40

Kinect Virtual Rehabilitation system……............... 41

CHAPTER FOUR: EXPERIMENTAL RESULTS…........... 43

CHAPTER FIVE: DISCUSSION.AND CONCLUSIONS..... 46

REFERENCES........................................................................... 50

APPENDICES............................................................................ 56

Appendix A: Form of Initial assessment …………………....... 56

Appendix B: Form of Final assessment …………………....... 57

ix

LIST OF TABLES

Table 2-1 Comparisons of Microsoft Kinect SDK and OpenNI S.. 13

Table 2-2 Kinect programming smoothing parameters……….. 20

Table 3-1 Demographic information of the recruited patients... 37

Table 4-1 Comparisons of balance ability for patients before

ddsfsdand after treatment with Kinect Virtual Rehabilitation

dfsdfc system………………………………………………. 43

Table 4-2 Assessment of user attitude…..................................... 45

x

LIST OF FIGURES

Figure 1-1 Benefit/challenges in VR rehabilitation application ... 4

Figure 2-1 Kinect components………………….......................... 11

Figure 2-2 Kinect SDK and OpenNI skeleton joints …….……... 14

Figure 2-3 Kinect application interface ………..……………...... 15

Figure 2-4 SDK skeleton detection processing ……………...... 16

Figure 2-5 Pytagorean Theorem ……….…………..………...... 17

Figure 2-6 Law of Cosines ………………………...………...... 18

Figure 2-7 Scale function……...…………….……...…….......... 18

Figure 2-8 System Architecture Design…….……...…….......... 22

Figure 2-9 Activity Diagram Balance shift….……...…….......... 23

Figure 2-10 System login page...…………….……...…….......... 24

Figure 2-11 User creation window…………….……...…….......... 25

Figure 3-1 Carousel styled system main menu ……...………..... 26

Figure 3-2 Metro styled menu.…………………………..…….... 27

Figure 3-3 Standard cursor and feedback graphics……..……...... 28

Figure 3-4 Ping Pong..……………………................................... 29

Figure 3-5 Game for training balance….…………………...….... 30

Figure 3-6 Window Cleaner……………….................................. 31

Figure 3-7 Step on it game……………………..……………….. 32

Figure 3-8 Coin collection…………………................................. 33

Figure 3-9 Human center of gravity and line of gravity…............ 35

xi

Figure 3-10 Balance shift……………………………….…............ 36

Figure 3-11 Experimental procedure..………………….…............ 38

Figure 3-12 Biodex balance machine………….……………......... 39

Figure 3-13 Photograph of a patient testing the system….……...... 41

1

CHAPTER ONE

INTRODUCTION

VR offers a possible solution, which allows the user to directly interact with

a computer-simulated environment. It can lead to new and exciting discoveries in

these areas which impact upon our day to day lives.

Modern input devices have been massively influenced by VR and may

become the corner stone of further virtual reality developments (Virtual reality

blog, 2009). Nintendo Wii and Microsoft Xbox Kinect are good examples. The

former uses a controller, which can be latched to the hand, to make movement

becomes a form of input, while the later uses a camera to track a player's

movements, which are then reflected in-game.

Our society is aging rapidly. Currently, a little under 8% of the world’s

population is 65 or older and this percentage is expected to reach 16% by 2050.

The growth in the elderly population is more accentuated in developed countries

where life expectancy continues to rise (World Population Ageing, 2009). As a

result, the number of patients with motor function disorders can drastically

increase while the ability to care for them will be limited by public expenditure

and human resources. Thus, there is high demand for computer-aided tools which

support in-home rehabilitation (Gonzalez, Hayashibe & Fraisse, 2012).

In this chapter, the background information about virtual reality, virtual

rehabilitation, and related works are described.

1.1 Virtual reality

Virtual reality (VR) is a simulated environment that is created with special

software and presented to the user in such a way that the user is able to recognize

himself and operate in that environment (Virtual reality, 2012). Generally, in VR,

the user wears a head-mounted display or glasses displaying three-dimensional

images as part of their experiences. Some systems enable the user to experience

additional inputs by providing various sensory stimulation, such as audios, videos.

and images, to form part of the VR environments. Many newer environments

2

embed touch or force feedback by integrating wearable devices such as data glove

and helmet, further enhancing the realistic experiences.

A virtual environment should provide the appropriate real-time responses

when the users explore their surroundings. If a delay incurred between the users’

actions and system responses, their feeling immersed in the VR environment may

be disrupted. The users become aware that they are not in a realistic but artificial

environment, which results in an adjustment of their behaviors accordingly by

responding with a stilted, mechanical form of interaction (Virtual reality blog,

2009).

The use of interfaces that require physical activity for an optimal user

experience becomes an increasing trend in gaming. Rehabilitation forms an

essential component of the therapeutic continuum in patients with multiple

injuries or motor disability. Effective rehabilitation programs help patients

optimize their level of physical, psychological, and social functions, as well as

reduce the length of patient stay, re-admission rates and use of primary care

resources.

1.2 Virtual rehabilitation

Virtual rehabilitation is one kind of therapy for treating patients based

entirely on VR simulated exercises. If there is no conventional therapy provided,

the rehabilitation is said to be virtual reality-based; otherwise, if virtual

rehabilitation is in addition to conventional therapy, the intervention is virtual

reality-augmented." (Burdea, 2002). The use of virtual reality systems may:

Allow for repeated and consistent practice of the same task (Rizzo and Kim,

2005; Rose, Brooks, & Rizzo, 2005)

Enable clinicians to progress difficulty and challenge levels (Rizzo and

Kim, 2005)

Enable clinicians and researchers to easily record and analyze performance

outcomes (Rizzo and Kim, 2005)

3

Provide a safe environment to undertake tasks which may be difficult or

unsafe in real life (e.g. crossing a street, operating a motorized wheelchair)

(Rizzo and Kim, 2005; Schultheis and Rizzo, 2001)

Offer appealing games that may make therapy tasks more fun and engaging,

which may increase compliance with therapy (Harris and Reid, 2005; Rizzo

and Kim, 2005)

Enhance motivation to practice which may lead to longer practice duration

or more practice repetitions (Rizzo and Kim, 2005)

Provide enhanced real-time feedback about task performance or task results

which may be beneficial for learning (Holden, 2005; Rizzo and Kim, 2005)

Be useful as home interventions involving independent practice (Rizzo,

Strickland, and Bouchard, 2004)

Provide potential for telerehabilitation, if the virtual reality system is

internet-deliverable (Levac and Missiuna, 2009) (as clinicians can monitor

clients' practice from afar)

VR is a technology that undoubtedly has great potential to be used in

various branches of medicine. Although the systems associated with the use of

virtual reality have been present on the market for many years, their use is not

very popular. This is a modern approach to rehabilitation, which is fun for the

patient (Boian, Lee, Deutsch, Burdea & Lewis, 2002).

According to WHO, over a billion people, about fifteen percent of the

world’s population have some form of physical disability, around 150 million

people significant difficulties in functioning of limbs (WHO, 2013). Virtual

rehabilitation systems have advanced significantly over the past decade. It offers

the possibility to be precisely adapted to the patient’s therapy and to be specific.

It can provide realistic training for patients in different scenarios and phases of

the rehabilitation. It is now conceivable that computer-based rehabilitation

programs could be developed using current, widely available, affordable virtual

reality platforms, such as the Microsoft Kinect. Many VR systems have been

developed specifically for use in rehabilitation to improve movement skills. These

4

are often expensive systems which are limited to use in research laboratories

(Deutsch, Borbely, Filler, Huhn, and Guarrera-Bowlby, 2008). Figure 1-1 shows

list of benefits and challenges of virtual rehabilitation (Morganti, 2006)

Figure 1-1. Benefit/challenges in VR rehabilitation application

(Source: Morganti, 2006)

There are many reasons why virtual reality applications are so effective for

rehabilitation. First, virtual reality is an interactive, experiential medium. In the

same way that children and teenagers intuitively grasp computers, virtual reality

users become directly engaged with the effects of the virtual experience. In

addition, virtual reality creates a safe setting where patients can explore and act

without feeling threatened. Patients can make mistakes without be afraid of

dangerous, real, or humiliating consequences (Wiederhold, 2006). Moreover,

unlike human trainers, computers are infinitely patient and consistent. In cognitive

rehabilitation, virtual reality can be manipulated in ways that the real world cannot.

For example, virtual reality can convey rules and abstract concepts without the

use of language or symbols for patients with little or no grasp of language. Indeed,

virtual reality has been used to support individuals with intellectual disabilities by

training them in many skills of independent living, such as grocery shopping, food

preparation, orientation, road crossing, and vocational training (Standen and

Brown, 2005).

5

Virtual rehabilitation have some benefits to motivate patients. Cycle going

like this:

• When patients will exercise harder and longer when “immersed” in a virtual

rehabilitation environment providing positive reinforcement.

• This phenomenon is called a “Virtuous Cycle,” where patients receiving

positive feedback feel an increased sense of motivation, which encourages

them to perform more repetitions and/or exercise longer to surpass their

previous performance.

• This motivation and encouragement is important for many aging adults,

particularly those who are reluctant to participate in the therapy process or

are fearful due to physical limitations.

• Mentioned before when virtual reality exercises are combined with

traditional therapy techniques, rehabilitation outcomes can exceed

outcomes generated by traditional therapies alone.

There are two ways to classify Virtual Rehabilitation (Burdea. 2002).

The first one is based on specific patient condition that subdivided by:

• Musculo skeletal Virtual Rehabilitation (Suffered a bone or muscle injury.)

• Post-stroke Virtual Rehabilitation. (Stroke survivors.)

• Cognitive Virtual Rehabilitation (Significant disability from individual

reasons.)

In second series classification is related to the rehabilitation protocol is

distinguished VR-augmented and VR based therapy. Before we mentioned about

these.

6

1.3 Motivations and Objectives

There are a variety of VR games developed under currently available

commercial game platforms such as Nintendo Wii and PlayStation Move to

provide people to entertain from the virtual environment. Different levels of

challenge options can be selected by people with different motor skills. However

these commercial games are mainly bounded to the fact that they are not

developed for rehabilitation purposes. Most of these videogames requires the

patient to hold a remote controller, which is difficult for some patients with

serious motor injuries.

After the prevalence of activity-promoting game systems, such as Nintendo

Wii and Sony PlayStation, injuries associated with handheld controller overuse

and repetitive motion were frequently found. Compared to the aforementioned

video game systems, Microsoft Xbox Kinect allows the users to interact with the

machine through body gesture without needing any handheld controllers, which

is believed to be able to significantly prevent the occurrence of musculoskeletal

injuries (Tanaka, Parker, Baradoy, Sheehan, Holash, and Katz, 2012).

Kinect is small and affordable, making it an excellent tool for use in

home-based rehabilitation (Chang, Chen and Huang, 2011). In particular, when

used at home, a Kinect-based system can encourage increased use of the upper

extremity for older adults who generally lack of movements. The system

presented here uses this device to track the patient’s hand when playing a game

designed specifically for upper limb rehabilitation in older adults. Furthermore,

no extra accessories, such as markers or hand supports, are required in the

developed system. The research objectives of this study are as follows:

• Develop specialized exercise games for balance shift, upper limb, and

lower limb trainings using Microsoft Kinect.

• Investigate the possibility of using the Microsoft Kinect to aid rehabilitation.

• Increase physical activity of patients, as well as improve satisfaction and

enjoyment of therapy.

7

• Demonstrate Kinect to be useful for rehabilitation, both in clinical and

home settings.

8

1.4 Related works

Even before the release of the Kinect, some of project using Nintendo’s Wii

video gaming system. Such gesture interaction technologies are not new; however,

their recent availability as interface means within affordable mass-market gaming

products can be seen as evidence of a broadening usage beyond solely

entertainment. Often, Nintendo’s Balance Board is used as an input device, for

example, in various games for balance training (LaBelle, 2011). The Kinect, on

the other hand, is small and affordable enough to be used in virtually any home

environment, and it does not require patients to wear anything that could limit

their movement (Chang, Chen and Huang, 2011). According to an investigation

of self-reported cases, they reported 9 types of injuries identified; among them,

hand lacerations related to overuse or incorrect use of handheld controller were

the most commonly observed injuries (Sparks, Coughlin, and Chase, 2009). Other

studies have also identified the Kinect’s potential for use in physical therapy.

Chang, Chen and Huang (2011) developed a Kinect-based rehabilitation system

to assist therapists in their work with students who had motor disabilities. The

program used the motion tracking data provided by the Kinect to determine

whether the patient’s movements reached the rehabilitation standards and to allow

the therapist to view rehabilitation progress. The Kinect has been used for medical

purposes outside of physical therapy as well (Chang, et al., 2011). Rizzo and

others at the University of Southern California studied how video games that

require player movement could motivate persons at risk for obesity to engage in

physical activity (Lange, Suma, Newman, Phan, Chang, Rizzo and Bolas, 2011).

To demonstrate the concept, they developed a system using the Kinect in which

the popular computer game World of Warcraft could be controlled with user

gestures instead of mouse and keyboard commands.

A motor rehabilitation system can be developed applying different tools for

interaction such as haptic sensors, markers and cameras. The use of haptic sensors

is common on upper limb rehabilitation, where it is used to interact with the

system and also to enable feedback for the user (Luo, Kline, Fischer, Stubblefield,

9

Kenyon, and Kamper, 2005). Techniques that apply markers are commonly used

due its low cost and easiness of use. These markers are used as references to

extract information about the scene orientation and the positioning of the objects

in order to provide guidance to the treatment (Sparks et al, 2009). Another option

is the use of accelerometers and gyroscopes provided by the Nintendo Wii remote.

Despite being simple to implement, these approaches have the disadvantage that

the patient needs to hold or attach objects to his/her body, which is not always

applicable (Da Gama, Chaves, Figueiredo and Techrireb, 2012). Nevertheless,

none body reference for interaction is used on them, turning difficult to analyze

movement carefully, which is a powerful tool not only for the current patient

evaluation but also for the storage and future analysis of his progress on the

rehabilitation treatment (Timmermans, Saini, Willmann, Lanfermann, teVrugt.J,

and Winter, 2007). Nintendo Wii introduced a new style of VR in 2005 using a

wireless controller that interacts with the player through a motion detection

system and its avatar representation in the video. The controllers use embedded

acceleration sensors that can respond to changes in direction, speed and

acceleration to enable participants to interact with the games. A sensor, mounted

on top of a TV, captures and reproduces on the screen the movement from the

controller as performed by the participants. The feedback provided by the TV

screen generates a positive reinforcement, thus facilitating training and task

improvement

10

CHAPTER TWO

MICROSOFT KINECT DEVELOPING

ENVIRONMENT

2.1 Microsoft Kinect

Kinect is a motion sensing input device by Microsoft for the Xbox 360

video game console and Windows PCs. Based on a webcam-style add-on

peripheral for the Xbox 360 console, it enables users to control and interact with

the Xbox 360 without the need to touch a game controller, through a natural user

interface using gestures and spoken commands. It does not require users to hold

or wear any specialized equipment for tracking. The reasonably high accuracy (xy

resolution = 3mm, z resolution = 1 cm ) and low price makes it a good tracking

alternative for a home based rehabilitation system. (Schönauer and Mossel, 2011)

compared the performance of this device with a Motion Capture system and found

Kinect cannot measure many parameters and has lower accuracy. However, their

study demonstrated a custom game that was nevertheless controlled well with

Kinect.

Kinect competes with the Wii Remote Plus and PlayStation Move with

PlayStation Eye motion controllers for the Wii and PlayStation 3 home consoles,

respectively. Normal cameras collect the light that bounces off of the objects in

front of them. They turn this light into an image that resembles what we see with

our own eyes (Webb and Ashley, 2012). On the other hand, Kinect is a depth

camera, which records the distance of the objects that are placed in front of it. It

uses infrared light to create an image (a depth image) that captures not what the

objects look like, but where they are in space.

The reference device created by PrimeSense includes an RGB camera, an

infrared sensor, and an infrared light source. Microsoft licensed PrimeSense’s

reference design and PS1080 chip design, which processes depth data at 30 frames

per second. The chip also automatically alignes the information for the RGB

camera and the infrared camera, providing RGBD data to higher systems.

11

Microsoft added a four-piece microphone array to this basic structure, effectively

providing a direction microphone for speech recognition that would be effective

in a large room. Figure 2-1 demonstrates the Kinect components.

Figure 2-1 Kinect components (Source: http://

http://praveenitech.wordpress.com/tag/rgb-camera/)

The Kinect’s infrared projector shines a grid of infrared dots over

everything in front of it. These dots are normally invisible to us, but it is possible

to capture a picture of them using an IR camera. The overarching goal of this

development was to determine whether the Kinect and the data it supplies are

useful in balance rehabilitation.

2.2 Design guidelines

The design should not only focus on the patients’ needs but also on the

rehabilitation needs. Our system fulfills the five features of a suitable game

described by Burke, Morrow, McNeill, McDonough, and Charles (2008), which

meets the requirement of a home-based system. These features include:

Level of difficulty

Direct feedback

Easy to configure

Low cost

Minimal therapist involvement

12

In this study, a physical therapist, computer specialist and a biomedical

engineering specialist involved in the study participated in the discussion. The

features considered during the design process were ranked to determine their

importance. We determined to adopt that the Microsoft metro for graphic

Interface design because it is simple and easy to operate for patients. Users

interacting with the application should be able to:

Perform therapy of lower extremities using their legs and feet, such as

Step on It game which will be mentioned in Chapter 3.

Perform therapy of upper extremities using their arms and

hands(Window Cleaner and Ping pong mentioned in Chapter 3)

Perform stretching exercises using their full body (Coin Collect

mentioned in Chapter 3)

Easy to use and highly customizable.

2.3 Choosing Open Source Drivers and SDKs

In December 2010, OpenNI and PrimeSense released their own Kinect

open source drivers and motion tracking middleware (NITE) for PCs running

Windows, Ubuntu and MacOSX.

In February 2012, Microsoft released a commercial Kinect Software

Development Kit (SDK) for Windows that includes Windows 7 compatible PC

drivers for the Kinect device that does not support older Windows versions or

other operating systems (Webb et al, 2012). Microsoft’s SDK allows developers

to build Kinect enabled applications in Microsoft Visual Studio using C++, C# or

Visual Basic. A comparison of the capabilities and advantages of the OpenNI tool

and the Microsoft Kinect SDK is shown in Table 2-1.

13

Table 2-1. Comparisons of Microsoft Kinect SDK and OpenNI.

Microsoft Kinect

SDK OpenNI SDK

Raw depth and

image data Yes Yes

Joint tracking

without calibration Yes* No

Easy installation Yes* No

Audio processing,

speech recognition

capability

Yes* No

Number of joints

available 20* 15

Platform Only Works in

Windows

MacOSX,

Ubuntu, Windows*

Resource, sample

and support Excellent* Adequate

The advantages of the Microsoft SDK are its abilities to track joints without

needing calibration, to track 20 joints at the same time, and for easy installation.

On the other hand, OpenNI skeleton tracking requires the user to hold a psi pose

for calibration, which might be difficult for many patients. Figure 2-2 shows the

joints which can be detected by Microsoft SDK (Fig. 2-2a) and OpenNI SDK (Fig

2-2b) . As compared in Table 3-2, Windows SDK seems to be a better choice by

considering its ability to track seated skeletons using only the top 10 joints.

14

(a) (b)

Figure 2-2. Kinect SDK and OpenNI skeleton joints (Modified

from: OpenNI and Kinect Official Website)

2.4 Kinect API

The Kinect for Windows SDK works as an interface between the Kinect

device and an application. When the application needs to access the sensors, it

sends an API call to the driver. The Kinect driver controls the access to sensor

data. Figure 2-3 illustrates the layer structure of the Kinect API and the sensors.

As shown in this figure, the installed drivers for the sensors sit with the

components of system device drivers and can talk to each other. The drivers help

stream the video and audio data from the sensors and return it to the application.

These drivers help detect the Kinect microphone array as a default audio device

and also help the array interact with the Windows default speech recognition

engine. Another part of the Kinect device driver controls the USB hubs on the

connected sensor.

15

The Kinect SDK libraries can be classified into the two following

categories:

The libraries used for controlling and accessing Kinect sensors

The libraries used for accessing microphones and controlling audio

The first category deals with the sensors by capturing the color stream,

infrared data stream, and depth stream, tracking human skeletons, and taking

control of sensor initialization. Audio APIs control the Kinect microphone array

and help capture the audio stream from the sensors, control the sound source, and

enable speech recognition. Figure 2-3 showed architecture of Kinect API.

Figure 2-3 Kinect application interface (source: Jana. 2011)

2.5 Skeleton stream

The Kinect sensor returns raw depth data from which we can easily identify

the pixels that represent the players. Skeleton tracking is not just about tracking

the joints by reading the player information; rather, it tracks the complete body

movement. The Kinect sensor can identify the pixel range of a player from the

depth data. In the initial steps of the process, the sensor identifies the human body

object, which is similar to another object captured by the sensor. In the absence

of any other logic, the sensor will not know if this is a human body or something

else. The tracking procedure is as follows (Jana, 2012):

16

1. Depth data is processed in the rendering pipeline process and matches with

decision forests labeled data and generates the inferred body segments.

2. Once all parts are identified based on the labeled data, the sensor identifies

the body joints.

3. The sensor then calculates the 3D view from the top, front, and the left of

the proposed joints.

4. Then the sensor starts tracking the human skeleton and body movement

based on the proposed joint points and the 3D view..

The Kinect for Windows SDK provides a set of APIs allowing us easy

access to the skeleton joints. Figure 2-4 shows the process flow in creating joint

points from raw depth data:

Figure 2-4 SDK skeleton detection processing.

The fundamental approach of gesture recognition is to deal with the

skeleton's joint points and apply basic logic to perform some action. If the

performed action is matched with predefined set of condition, application can

understand that the user has performed a certain gesture, otherwise not. Each joint

is measured in three dimensional (X, Y, Z) plane. X and Y coordinates actually

indicate the joint location in the plane, and Z indicates how far the joint is from

the sensor.

17

2.6 Math formulas implementation

Frequently, system code uses geometry and trigonometry. In general

mathematics, to calculate the distance between two points, we need to make use

of the Pythagorean Theorem(fig 2-5).

Figure 2-5. Pytagorean Theorem.

According to the rule of right triangle, squared hypotenuse is equal to the

sum of the squares of the other two sides. The same formula is applied for the

given two coordiantes (X1, Y1) and (X2, Y2) in the XY plane. The calculation for

the distance between two points on a horizontal line, then, may be generalized as

(x1- x2) and for two points on a vertical line the distance is (y1- y2). Then the

formula for finding the distance between any two points based on the Pythagorean

theorem as follows in equation 1.

𝑑2 = √(𝑥1 − x2)2 + (𝑦1 − y2)

2 (1)

Aforementioned The Pythagorean theorem works well for three

dimensional planes as well, and the distance between points (X1, Y1, Z1) and (X2,

Y2, Z2) can be calculated by the equation 2.

𝑑3 = √(𝑥1 − x2)2 + (𝑦1 − y2)

2 + (𝑧1 − z2)2 (2)

The Law of Cosines, as shown in Figure 2-6, calculates the angle of type

of triangle. This is useful when determining the angle between two joints. Appling

the formula for this purpose requires a third point, which can be another joint

position, but generally should be a point along the X-axis from the base point. The

largest angle calculable by the Law of Cosines is 180. When calculating the angles

18

between joints, this means additional calculating to determine angles from 180 to

360, but this is trivial.

Figure 2-6 Law of Cosines (modified from:

http://www.beginnerstaichi.com)

The Coding4Fun Kinect Toolkit is a set of extension methods and controls

to make developing applications for the Kinect using the Kinect for Windows

SDK easier.

The Coding4Fun Kinect Toolkit has a ScaleTo extension as part of the

library. This adds the ability to take a joint and scale it to any display resolution.

The scaling function is demonstrated in Fig. 2-7.

Figure 2-7 Scale function

In Fig. 2-8, maxPixel = width or height, depending on which coordinate

your scaling is; maxSkeleton = 1; and Position = the X or Y coordinate of the joint

you want to scale. When you want to move eclipse control by joint movement

then use function call like

Canvas.SetLeft(eclipse, Scale(640, 1, joint.Position.X));

19

Canvas.SetTop(eclipse, Scale(480, 1, -joint.Position.Y));

To depend on your needs change to 640 & 480 with a different scale.

If you include the Coding4Fun Kinect Toolkit, instead of re-writing code, you

could just call it like so:

scaledJoint = rawJoint.ScaleTo(640, 480);.

A lot of coding and customized function implemented in this system so let

explain about healthcare side.

Smoothing skeleton data

Kinect sensor does not have sufficient resolution to ensure consistent

accuracy of the skeleton tracking data over time. This problem manifests itself as

the data seeming to vibrate around their positions. The Kinect for Windows SDK

uses the Holt double exponential smoothing procedure to reduce the jitters from

skeletal joint data. The smoothing parameters solve the jittering problem by

filtering the skeleton data and applying a smoothing algorithm to it .In my case

developing in menu section, hand tracking have to be very smooth and jitterless.

Code shown in below.

SkeletonStream.Enable(new TransformSmoothParameters() { Correction = 0.5f, JitterRadius = 0.05f, MaxDeviationRadius = 0.04f, Smoothing = 0.8f };

20

Table 2-2. Kinect programming smoothing parameters

(Source: Microsoft MSDN. 2013)

2.7 Software development

Our software is written in visual studio 2012 and C# language and design

language XAML used Extensible application markup language XAML language

is used for initialization of User Interface (UI) elements in a structured and

manageable way. XAML elements map directly to Common Language Runtime

(CLR) 2 object instances. Anything created in Xaml can be expressed using

any .NET language, such as C# or Visual Basic. Visual Studio is an integrated

development environment by Microsoft and used be programmers to develop

console applications or applications with a graphical user interface. Patient’s

performing progress and data stored in MySQL database management system.

Our project consists of a native Windows Presentation Foundation (WPF)

application that will authenticated users to access our physical therapy system.

System have following feature attached.

Parameter Description Default

Value Comments

Smoothing

Specifies the

amount of

smoothing.

0.5

Determines the amount of smoothing

applied while processing. If increase

this value you will get smoother

skeleton data, however, it increases

the latency. With the smoothing

value as zero, you will get the raw

skeleton data

Correction

Specifies the

amount of

correction.

0.5

Specifies the amount of correction

needed for the raw data. With lower

values more correction is applied,

the raw data is corrected, and the

data looks smoother.

Jitter

Radius

Specifies the jitter-

reduction radius, in

meters.

0.05 The default value of 0.05 represents

5cm. Any jitter beyond the radius is

clamped to the radius.

Max

Deviation

Radius

Specifies the

maximum radius

that filter positions

can deviate from

raw data, in meters.

0.04 This is the max limit of the deviation

that is allowed to be considered for

determining a jitter.

21

The authentication feature. The system should allow authenticated users

to login and allow access to their information. No user in the same

category is allowed to access the other user’s information in the system.

Customizable. The system should change its functionality according to

the type of users in the system. The therapists are represented with

different functionalities within the same application in comparison to the

patients.

Database The system will include a database to store the information

about patients and therapists. We will separate the database access from

its implementation for security and portability reasons.

Figure 2-8 presents the architectural design of the whole system. As can be

seen in the diagram, the patient interacts with the Kinect which sends information

to the Software. Software controlled by patients hand movement to select module,

change module or exit the system. After choose module, the new window will

popup which contains exercise function. The software also contains the Main

Window in two forms, the XAML file and the Class file. XAML, which stands

for Extensible Application Markup Language, contains the Camera viewers and

the main page design. This is the page one will first see when starting up the kinect.

It displays whether the Kinect is connected or not. If it is, then it continues to show

the main background image and whatever the programmer has allowed to show

from the C# class on this page. I’ve used this page as the menu for my software.

22

Figure 2-8 System Architecture Design

The last thing included in the Game are the components such as the text,

images and sounds used in the application. After the software detects the motion

of the user it sends information to the Display unit and speaker, whether on

monitor or TV in which the user can interact with.

Here is also Balance shift module activation diagram provided because

evaluation focused on balance therapy. Module designed for following

requirements:

Easy to operate and understandable.

Can be customizable for each individual patient.

Detect balance shift accurately

23

Provide realtime audio visual feedback.

Figure 2-9 Activity Diagram Balance shift

Our system can track only one closest user. First you get all the skeletons

in the returned collection. Then we find all the skeletons that belong to the closest

player currently being tracked and find the joints. Skeleton stream provides up to

six skeleton hipcenter joint data and skeleton IDs. When choosing closest skeleton,

24

compare Z coordinate of all skeletons then return skeleton ID with minimum Z

distance.

Login and user management

Login page is the first page the user interacts with. If an unauthorized user

attempts to enter the system, an error message is shown under the login button. In

the view-model of the login page we use the Login service contract of the database

connection to query if the user exists in the database. Login service will return a

status Enum value that we can branch to identify where to navigate to. Figure 2-

10 shows system login page.

Figure 2-10 System login page

If the login status is returned as patient, user is redirected to the software

main menu. Otherwise login status returned as admin, user redirected to user

creation and user information management window (Figure 2-11).

25

Figure 2-11 User creation window

Database have designed some tables for portability reasons. First table

stored username, password and some extra information about patient. Activity

table created for user login date, duration and perform count.

26

CHAPTER THREE

METHODOLOGY

3.1 Interface Design

The main purpose of this software was to demonstrate that games

developed with the Microsoft Kinect is useful for rehabilitation. This work

presents a virtual rehabilitation system consisting of 5 games proposed and

developed by an interdisciplinary group including experts majoring in computer

science, computer engineering, biomedical engineering, physical therapy, and

rehabilitation.

As shown in Figure 3-1, the personalized gesture-based carousel spinning

menu provides the patients with easier interface to adjust the main program

according to individual needs. A patient can swipe the next item from right to left

or swipe back the item from the left to right side using one’s right hand. The Users

can raise up the right hands above their heads in order to enter and load a new

screen to initiate an exercise.

Figure 3-1 Carousel styled system main menu

27

One-handed gestures, such as swipe gestures, are also easier for users to

learn, remember, and operate than two-handed gestures. For the SwipeToRight

gesture, we will use the following requirements:

Each new position should be placed at the right of the previous one

Each position must not exceed the height the first one by more than a

given distance (20 cm)

The time between the first and last positions must be between 250 and

1500 ms

The gesture must be at least 40 cm in length

Swipe and pushing gestures are most frequently used in Kinect

programming. The metro styled menu, much like mouse cursor, is simpler to use

by providing visual cues to the patients to guide to scroll the icons. And then, they

can just press to push the selected exercise icon to load the game.

Figure 3-2 Metro styled menu

Figure 3-3 (a) and (b) show the icons with default targeting state and

actionable state, respectively. As a user extends his or her arm in a pressing motion,

a hand color fill goes up and down within the cursor (Fig. 3-3c) to indicate how

far the user has achieved the requirement of the posture. This feature was shown

to be able to improve pressing accuracy and learnability. It also makes it easier

28

for users to cancel before a press has been made, or retarget if a press is about to

happen over the wrong control. The cursor also has a visual state, Fig. 3-3 (d) that

indicates when the user has completed a press.

When Kinect detects that the user’s hand is in a closed fist state, as shown

in Fig. 3-3 (e), the cursor changes to a gripped visual and shows a color consistent

with the fully pressed state. This confirms to the user that the grip is detected, as

well as being a strong visual cue for how to make a recognizable gripped hand.

Figure 3-3 (f) compares the right hand icon with the left hand cursor.

(a) (b) (c) (d) (e) (f)

Figure 3-3 Standard cursor and feedback

3.2 Development of Kinect Virtual Rehabilitation System

The developed system contains five games, including Ping Pong, Balance

Shift, Window Cleaner, Step on It, and Coin Collection games, which can be used

for upper limb exercise and balance therapy. Plenty of parameters and different

levels of difficulty can be set to customize the user’s need. The operations of

individual games are described as follows.

Ping Pong

The aim of this game is to defeat the competitor with higher score. When

the game starts, the ball randomly bounces down to the ground slowly. The user

can move his/her hand and elbow to change the direction of the paddle. The score

will be recorded at the bottom left corner of the screen. Players who earn more

points will be awarded. This game was designed for upper body, especially for

training flexion and abduction of the elbows and hands. The moving speed of the

29

ball will increase if the exercise has been carried out well. Figure 3-4 shows the

graphic user interface of the Ping Pong game.

Figure 3-4 Ping Pong

Balance Shift

For Wii balance board compatible games, the users interacts with the game

by shifting their weight on the board. However, our software, instead of using Wii

balance board, detects balance shift by processing Kinect skeleton frames. Figure

3-5 shows the game developed for training patients how to make correct balance

shifts. The exercise can be divided into easy (Level 1) and difficult (Level 2)

modes providing different degrees of challenge to patients. The default mode is

Level 1 which asks the patient to stand in straight and do balance shift. Level 2 is

a little more difficult than Level 1. At level 2, patients have to bend their knees

before starting doing balance shift. The software can automatically detect the

patients’ knee angles to determine whether they are standing or knee-bending.

Real-time audio and visual feedbacks are also provided. As shown in the figure,

when the patient makes a shift, the gauge styled indicator shows which side the

patient has shifted. When a shift has been completed and the system has identified

that one has done a correct action, the system play a sound showing it is a

30

successful shift and the counter increases by 1. Because some severe patients

cannot make balance shifts with great angles, the software provides the function

for the therapist to adjust the left and right threshold angles to the neutral position

(90o) for determining whether a balance shift is counted as successful according

to the patient’s status. The adjustable angle ranges from -10o (right shift) to +10o

(left shift), resulting the shifting angle spanning from 80o (right shift) to 100o (left

shift). For example, some serious disabled patients can start with a small shifting

angle and increase to a larger angle when one’s health status is getting better.

Figure 3-5 Game for training balance

31

Window Cleaner

This game targeted to upper limb exercise using both shoulder extension,

shoulder abduction and elbow flexion. The objective of this game is supposed to

clean a series of mirrors as quickly as possible, by wiping off the virtual dirt. The

background image of the game can be changed and both hands can be switched at

any time. In addition, the game provide the function to select different levels of

difficulty to help motivate the players in playing the game. Figure 3-6

demonstrates the graphic user interface of the game.

Figure 3-6 Window Cleaner

32

Step on It

Provides training on lower limb and trunk balance of the player. The Game

scenario is coming from dance pad, also known as dance mat, dance platform, or

flitter deck, which is is a flat electronic game controller used for input in dance

games. As you can see from the Figure 3-7, the player should stand on the center

of 3×3 virtual matrix square panels on one foot and move another according to

the guide of the game. When one panel randomly changes the color, the player

has to step one foot on the colored panel while keeping the other foot on the central

panel of the matrix.

Figure 3-7 Step on It game

33

Coin Collection

Knowing that game will be based on an augmented reality scenario, this

exercise displays the tracked body and coins falling from the sky. Using the

player’s head, left hand, right hand, left foot, and right foot joints, one can catch

coins and thus get scores. As illustrated in Figure 3-8, the score, game time, and

moving angle of the player will be displayed to help the user understand and feel

comfortable with the game.

Figure 3-8 Coin Collection game

3.3 Experiment

In this pilot study, in order to test the usefulness and effectiveness of the

Kinect Virtual Rehabilitation System, the Balance Shift game was used to train

patients with stoke to recover their balance ability. Stroke is one of the most

common causes of death in the developed countries. Stroke patients often suffer

from hemiparesis, affecting their balance ability and consequently their self-

dependency and quality of life. Balance rehabilitation can be a long and tedious

process. Virtual rehabilitation systems have been reported to provide therapeutic

benefits to the balance recovery of stroke patients while increasing their

34

motivation (Gonzalez, Hayashibe, and Fraisse, 2013). Although therapeutic

advances have reduced the mortality during the acute stages of stroke, patients

still experience severe neurological disabilities. Deficits in motor control,

abnormal synergistic organization of movements, muscle weakness, sensory

deficits, and loss of range of motion can all reduce the quality of life in stroke

patients. The reluctance to bear weight on the paretic leg may persist and cause

gait disturbances, despite conventional physical therapy to correct asymmetric

standing posture (Ishii, Matsukawa, Horiba, Yamanaka, Hattori, Wada and Ojika,

2010) Moreover, an asymmetric posture could foster further disuse and hinder the

recovery of motor function in the lower limbs during motor convalescence. The

balance training game was developed with the following goals:

Improving balance function and safety

Improving visual motor control and tolerance of motion

Increasing activity levels

Reducing falls or risks of falls

Balance Retraining Therapy is a specialized form of physical therapy to

decrease primary symptoms of movement-related dizziness and imbalance

through a customized "hands-on" approach. Additional symptoms addressed in

therapy may include decreased strength, loss of range-of-motion, muscle tension,

anxiety, and fatigue.

To skillfully control the center of gravity (CoG) is important for keeping

one’s body in balance. CoG is defined as:

The balancing, equilibrium or pivoting point in the body.

The point where the sum of all the forces acting on the body is zero.

The point at which all the weight of the body may be considered to

be concentrated and about which all the parts exactly balance.

In the human body, when standing in the anatomical position the CoG is

located anterior to the second sacral vertebra.. The location of the CoG can be

outside the human body during activities depending on the relationship of body

35

segments. A human’s CoG trajectory is useful to evaluate the dynamic stability

during daily life activities such as walking and standing up (Center Of Gravity,

2013).

The CoG is an imaginary vertical line passing through the CoG down to a

point in the base of support. In the human body it passes from the vertex through

the body of the second sacral vertebra down to a point between the feet when

standing in the anatomical position. The gravitation pull acting at the CoG of any

segment is along the LoG. As shown in Fig. 3-9, the CoG is located at the position

which is slightly above the hip center along the line of gravity (McCreary. 1993).

In this study, the CoG is detected from the Kinect infrared image.

Figure 3-9 Human center of gravity and line of gravity (modified

from http://killergearz.wordpress.com/2010/05/11/facts-about-human-

body/)

The system was also designed for wheelchair users to support their balance

improvement. Moderate to severe strokes often cause a loss of sitting balance in

patients so balance is one of the first things that needs to be addressed in stroke

36

rehabilitation. If one cannot sit up, it is impossible to do activities of daily living

(ADLs) such as dressing, bathing, and toileting. One cannot learn to stand safely

until sitting balance is maintained first.

The experient was conducted at the Department of Rehabilitation of

National Taichung Hospital. In the first series, the participants were asked to stand

with a level, shoulder-width distance, by keeping their eyes open (Fig 3-10a), and

their weight-shifting was recorded and calculated.

(a) (b)

Figure 3-10 Balance shift a) Correct movement b) Wrong shift

(modified from http://www.beginnerstaichi.com)

3.4 Subjects and Statistic analysis

A total of 10 patients (5 males and 5 females) with age ranging from 47 to

87 years old were recruited to participate the study. After the stroke, they have

difficulties in balancing their limbs because of neurologic injures.

All tests were conducted in the Department of Rehabilitation, National

Taichung Hospital. Each participant underwent 10 sessions of treatments within

10 days, each lasting for 3 minutes. Among them, data of 2 patients were excluded

37

because they hadn’t finished all of the treatment sessions. Table 3-1 shows the

demographic information of the participants.

Table 3-1 Demographic information of the recruited patients.

Male Female Total

Sex (number) 4 4 8

Age (years) 52.5 ±4.7 55.7 ±4.3 54.1 (4.5)

In order to assess the effectiveness of the developed virtual rehabilitation

system, patient’s activity during individual treatments were logged into the

database for later analysis. A statistic software package (SPSS 18) was used for

descriptive and inferential statistical analyses. The outcomes were tested with

paired-sample t-test and one-sample t-test with significance level defined as

p<0.05.

3.5 Rehabilitation Assessment

All the participants were assessed with the Berg balance scale, Biodex

balance machine, Get up and Go test, and our developed Kinect rehabilitation

system, as well as the user satisfaction survey at the beginning of, ten days after,

and at the end of the treatment. The experiments were divided to two sections:

Initial assessment

Final assessment

The experimental procedure is shown in Figure 3-11. The tests used for

evaluating the patient status include:

Kinect 2 min Test conducted by our developed software

Biodex Balance Test conducted by a machine)

Berg Balance Scale (BBS) Test scaled by a questionnaire

User Satisfaction Survey scaled by a questionnaire

Get up and Go test

38

Figure 3-11 Experimental procedure

Ten days treatment

Final assessment:

1. BBS, Biodex, Kinect

rehab and Get up and Go

tests

2. Questionnaire survey

Data

Analysis

Recording duration of

making 20 left and

20 right balance

shifts

Recording special

events and

collecting patient

opinions

Initial assessment:

BBS, Biodex, Kinect

rehab and Get up and Go

tests

39

Biodex Balance Machine

The participants underwent balance ability examination before

participating in the study. First, participant’s balance ability was measured using

a Biodex balance machine. This system uses a multiaxial testing platform which

can be set at variable degrees of instability. None of the subjects had experienced

vestibular system dysfunction, cerebral concussions, or central or peripheral

neurological dysfunctions. Biodex Balance System is a widely used device that

quantifies dynamic balance performance. It evaluates the ability to maintain

equilibrium while standing on a movable support surface with varying degrees of

instability (Biodex Balance System. 2012). Like other clinical instruments, the

first prerequisite to use of BBS is to determine the degree to which its scores are

reproducible with repeated measurements in conditions where the response

variable (e.g., balance performance) is stable.

Figure 3-12 Biodex Balance machine

Establishing reliability is necessary for either situation, when using BBS to

discriminate balance among individuals or using it to evaluate balance changes

over time following an intervention program.

40

Berg Balance Scale

The Berg Balance Scale (BBS), named after Katherine Berg, is a widely

used clinical test of a person's static and dynamic balance abilities. For functional

balance tests, the BBS is generally considered to be the gold standard. It was

developed to measure balance among older people with impairment in balance

function by assessing the performance of functional tasks. It is a valid instrument

used for evaluating the effectiveness of interventions and for quantitatively

describing functions in clinical practice and research. A 14-item scale was

designed to measure the balance status of an older adult in a clinical setting.

The test takes around 15–20 minutes and comprises a set of 14 simple

balance-related tasks, ranging from standing up from a sitting position to standing

on one foot. The degree of success in achieving each task is given a score ranging

from zero (unable) to four (independent) with the final measure calculated by

summing all the scores. The BBS has been recently identified as the most

commonly used assessment tool across the continuum of stroke rehabilitation. It

is considered a sound measure of balance impairment.

Get up and Go test.

The scoring is based on the observation of a patient's movements with

deviations from a confident, normal performance, using the following scale:

1 = Normal

2 = Very slightly abnormal

3 = Mildly abnormal

4 = Moderately abnormal

5 = Severely abnormal

where "Normal" indicates that the patient gave no evidence of being at risk

of falling during the test or at any other time; and "Severely abnormal" represents

that the patient appeared at risk of falling during the test. Intermediate grades

reflect the presence of any of the following as indicators of the possibility of

falling: undue slowness, hesitancy, abnormal movements of the trunk or upper

limbs, staggering, stumbling. Patients with a score of 3 or more on the Get Up and

41

Go test indicates that they are at the risk of falling (Podsiadlo and Richardson,

1991).

Kinect Virtual Rehabilitation System

In the second series, the participants were asked to stand in shoulder-width

stance on a fall-protection platform by keeping their eyes open to conduct balance

shifts for two minutes to determine their balance ability. The patient was asked to

do exercise with the developed game for 2 minutes with the number of movements

to the right and left sides being recorded. Balance shifting gesture was adopted

from ancient Chinese Taichi movements, which emphasize on weight-shifting,

postural alignment, and coordinated movement skills. Figure 3-13 shows an

example of a patient doing exercise in the rehabilitation center of a hospital. The

experimental setup is equipped with a 32-inch monitor, hand-hold supporting rail,

computer, and the Kinect device.

Figure 3-13 Illustration of a patient testing the system

Usage attitudes have been assessed using the questionnaire after the

subjects have used the e-rehabilitation system. In this study, the nine recruited

42

volunteers were asked to answer the question items included in the questionnaire

after having used the system.

43

CHAPTER FOUR

EXPERIMENTAL RESULTS

Table 4-1 compares the treatment outcomes of the initial and final tests

using pair-sample t-test. As shown in the table, the Medial Lateral Index of the

Biodex assessment, Get Up and Go Scale, BBS Sum, Left Shift Count, and Right

Shift Count show significant improvement (pair-sample t-test, p<0.05) after

treatments with Kinect Virtual Rehabilitation System (KVRS). However, no

significant improvement (p>0.05) was observed for Overall Stability Index and

Anterior/Posterior Index of the Biodex assessment. In this study, the patients were

only asked to practice medial/lateral balance shifts, that’s why the

Anterior/Posterior Index was not significantly improved, The Overall Stability

Index was calculated based on both Anterior/Posterior Index and Medial/Lateral

Index.

Table 4-1 Comparisons of balance ability for patients before and after treatment

with Kinect Virtual Rehabilitation System.

Initial Test Final Test Mean Differ.

(SD)

t-

value

p-

value Mean(SD) Mean (SD)

Overall

Stability Index 0.46(0.14) 0.58(0.11) -0.075 (0.14) -1.42 0.197

Anterior/Poster

ior Index 0.38(0.12) 0.39(0.11) 0.005 (0.15) 0.09 0.929

Medial Lateral

Index 0.22(0.07) 0.30(0.92) -0.07(0.08) -2.39 0.048*

Get Up and Go

Score 23.78(11.86) 17.31(5.92) 6.46(6.74) 2.71 0.030*

Beg Balance

Scale 50.50(3.85) 52.50(2.82) -2.00(2.00) -2.82 0.025*

44

Usage attitudes were assessed using the questionnaire after the patients

have finished all the 10 sessions of treatments using the KVRS. The results were

compared with the neutral value (3) and tested with one-sample t-test with

significance defined as p<0.05 (Table 4-2).

The eight volunteers were asked to answer the question items in the

questionnaires after used the system. As indicated in Table 4-2, the patients

expressed that the KVRS is interesting (p<0.05) and easy to control by hand

(p<0.05); the GUI of KVRS is easy to use (p<0.001); the feedback message is

clear and easy to understand (p<0.05); the treatment is effective (p<0.01); and

they feel comfortable with the game (p<0.01) and can immerse in the virtual

environment (p<0.01). However, they don’t agree that KVRS can simulate the

realistic environment (p>0.05).

Left Shift

Count 14.00(6.39) 36.13(11.29) -22.12(11.23) -5.57 0.001*

Right Shift

Count 14.00(6.80) 35.88(11.58) -21.87(11.55) -5.35 0.001*

45

Table 4-2 Assessment of user attitude.

Question Mean

(SD) t p-value

Mean

Differenc

e

Lowest Highest

I think playing the KVRS

games is interesting*.

3.88(1.03) 2.42 0.046 0.88 2 5

I think I can immerse in the

designed environment when

playing the KVRS games**

4.13(0.83) 3.81 0.007 1.12 3 5

I think treatments with

KVRS games is effective**

4.25(0.70) 5.00 0.002 1.25 3 5

I think the KVRS games can

be easily controlled by

hands*

4.25(1.03) 3.41 0.011 1.25 2 5

I think the KVRS games can

simulate the realistic

environment.

3.25(0.88) 0.79 0.451 0.25 2 4

I can understand the

messages feedback by the

KVRS games*

4.13(0.99) 3.21 0.015 1.12 2 5

I am comfortable with the

KVRS games***

4.38(0.74) 5.22 0.001 1.37 3 5

I think the KVRS games are

easy to operate***

4.50(0.54) 7.93 0.000 1.50 4 5

46

CHAPTER FIVE

DISCUSSION AND CONCLUSIONS

Spark et al. (2011) categorized Wii-related injuries into 4 different types:

tendinopathy, bursitis, enthesitis, and epicondylitis (Sparks, Coughlin, and Chase,

2009). According to an investigation of self-reported cases, 9 types of injuries

have been identified; among them, hand lacerations related to overuse or incorrect

use of handheld controller were the most commonly observed injuries (Sparks et

al, 2009). Other injuries, such as Tendinitis occurred in the thumb (Macgregor,

2000; Koh, 2000; and Karim, 2005) and Wii-itis in the shoulder and upper arm

(Bonis 2007; Sperling, Nett, and Collins, 2008), related to intensive use or

improper use of handheld controller were also reported in the medical literature.

Little resistance offered by the light-weight handheld controller to the

aggressive user leading to awkward deceleration to the upper extremity might be

the reason causing such injuries (Sterling et al, 2008). In addition, for sport-related

video games, such as tennis, bowling, golf, baseball, soccer, boxing, fighting,

physical fitting, and other sporting games, currently 1available on popular video

game platforms, injuries related to forceful and prolonged movements of body

and extremity are frequently observed because, unlike authentic sports, physical

strength and endurance are generally not limiting factors to discontinue the

sporting activities (Bonis 2007; Sterling et al, 2008). Compared to other video

game platforms, Microsoft Xbox Kinect allows the users to interact with the

machine through body gesture without needing any handheld controllers, which

is believed to be able to significantly prevent the occurrence of musculoskeletal

injuries.

Health professionals are always searching for a more effective treatment

that focus not only at the elimination of the pathology symptoms but also hold the

patient involved to it during the entire treatment in order to achieve the cure. The

importance of taking into consideration related human factors, such as patient

satisfaction and motivation is the key to ensure patient involvement and to achieve

47

a successful treatment. These factors are even more important for the success of

physiotherapy treatments due to the fact that the patient’s recovery is directly

associated with his/her continuous effort, commitment and discipline during the

entire rehabilitation process. This process consists in a series of sessions where

the patient must perform therapeutic exercises.

Sedentary activities, such as watching television, play video game, and

conducting cognitive work, may risk people in overconsumption of food, resulting

in the acquisition of obesities, cardiovascular diseases, and cancers (Chaput,

Klingerberg, Astrup and Sjödin, 2011). A more recent study also reported that

compared with inactive individuals, those with low-volume physical activities (an

average activity of 92 min/week) can reduce the risk of all-cause mortality of 14%

and increase the life expectancy of 3 years. In addition, Short-term of vigorous-

intensity activities have the same effect as mid-term of moderate-intensity

activities and long-term of mild-intensity activities (Wen, Wai, Tsai, Yang, Cheng,

Lee, Chan, Tsao, and Wu, 2011). Graves and Stratton (2008) reported that playing

bowling, tennis, and boxing of the Wii Sports consumed at least 50% more energy

than sedentary gaming for adolescents. Compared to a traditional sedentary video

game, it was observed that the heart rate, oxygen uptake, and energy expenditure

of schoolchildren were significantly higher for 2 Kinect activity-promoting video

games (Smallwood, Morris, Fallows, and Buckley, 2012).

In motor rehabilitation, there are essentially three major advantages that

virtual reality offers over traditional therapy alone. First, virtual reality creates a

safe, controlled environment for repetitive practice, and repetitive practice is

crucial in learning motor tasks. Second, virtual reality provides immediate, real-

time feedback about performance. Finally, because of its interactive nature, virtual

reality can increase motivation by making the experience fun (Holden, Bettwiler,

Dyar, Niemann, and Bizzi, 2001)

This work presented a rehabilitation system based on markerless interaction

in a virtual reality environment. The Kinect has shown much potential for use in

balance therapy. The system developed in this project were primarily intended

48

for use by balance professionals, but as mentioned above the Kinect could also

prove to be a useful tool in at-home therapy software. This system uses the Kinect

device for interaction and it is based on games controlled by therapeutic

movements, which were set by physiotherapists, inducing the user to do exercises

correctly. It is capable of identifying whenever the patient is doing it correctly,

otherwise warning him/her and also taking down the statistics in a report for a

further professional analysis.

Statistical analyses of paired t-test effect showed significant difference

between the initial and final assessment in the Berg Balance Scale, Get Up and

Go test, and Kinect virtual rehabilitation test (p<0.05), but not in the Biodex

assessment. P value is less than 0.05 which it is we can conclude that the groups

had trained effectively. In fact the test is effective patients who took the training

did better after taking training. As we see from the result, it indicated a positive

outcome for the patients after got involved the treatment.

Usage attitudes were assessed using the questionnaire after the subjects

have used the e-rehabilitation system. The outcomes were tested with one-sample

t-test with significance defined as p<0.05. The eight volunteers were asked to

answer the question items in the questionnaires after used the system. The

experimental result showed that the virtual training had a significant time effect

in the balance recovery of patients. With regards to the Berg Balance Scale, Get

up Go Test and Kinect Rehab system, the participants conditions improved

significantly between the initial and the final assessment. Ten days training is very

short time for balance recovery but Berg Balance Scale test and Get up and Go

test showed slight positive difference.

The system reached the five features considered during the design

process. Each feature is listed below:

Level of difficulty: All patients were able to play the game and

improve their scores over the study. Furthermore, none of the patients felt

extremely tired after a session.

49

Direct audio and visual feedback:The game score was

displayed at all times and users were able to see it and comment about it.

Patients remembered the score from the previous day and tried to improve

it. Patients quickly understand that moving their arm caused the cursor on

the screen to move and were also able to control it.

Easy to configuration: No configuration required to play the

game. Before playing the game, only tracking of the hand had to be started.

Low cost: The total cost of this prototype system was 700

USD(kinect sensor 150$ computer 550$).

Minimal therapist involvement:There was no need for a

therapist to be present during the treatment. Only one person was with the

patient during the sessions and the only tasks performed were helping the

patient stand up in the front of kinect and start the game (Burke et al, 2008).

Compared to other video game platforms, Microsoft Xbox Kinect allows

the users to interact with the machine through body gesture without needing any

handheld controllers, which is believed to be able to significantly prevent

musculoskeletal injuries. In conclusion, the developed Kinect virtual

rehabilitation system is useful for rehabilitation either in clinical setting or at

home. Adopting the VR rehabilitation system at home is more flexible, cheap, and

convenient for the patients and allows for more frequently repetitive exercises.

50

REFERENCES

Biodex balance system. (2012). Machine instruction material. Retrieved

Apr 30, 2013, from http://www.Biodex.com/.

Boian, R. F., Lee, C. S., Deutch, J. E., Burdea, G. C., & Lewis, J. A.

(2002). Virtual reality-based system for ankle rehabilitation post stroke.

Proceedings of the First International Workshop on Virtual Reality

Rehabilitation , 77–86, Citeseer.

Bonis, J. (2007). Acute Wiiitis. New England Journal of Medicine,

2431-2432.

Deutsch J.E., Borbely M., Filler J., Huhn K., Guarrera-Bowlby P.

(2008). Use of a low-cost, commercially available gaming console (wii™) for

rehabilitation of an adolescent with cerebral palsy. Physical Therapy, 88(10),

1-12.

Burdea, G. C. (2002). Virtual rehabilitation--benefits and challenges.

Methods of Information in Medicine, 42(5), 519-523.

Buker, J. W. (2010). Augmented Reality Games for Upper-Limb Stroke

Rehabilitation. 2nd International Conference on Games and Virtual Worlds for

Serious Applications 2010, pp. 75 – 78.

Burke, J. W., Morrow, P. J., McNeill, M. D. J., McDonough, S. M., &

Charles, D. K. (2008). Vision Based Games for Upper-Limb Stroke

Rehabilitation. Machine Vision and Image Processing Conference, 159-164.

Center Of Gravity, (2013). Retrieved Mar 25, 2013, from

www.kau.edu.sa/Files/0052891/Subjects/CENTER-4.DOC.

Morganti, F. (2006). Virtual interaction in cognitive neuropsychology.

Cyberteraphy.

Chang, Y. J., Chen, S. F., & Huang, J. D. (2011). A Kinect-based system

for physical rehabilitation: A pilot study for young adults with motor

disabilities. Research in Developmental Disabilities, 2566-2570.

51

Chang, C, Y., Rizzo, A., Lange, B., Suma, E, A., & Bolas., & Newman,

B. (2011). Development and evaluation of low cost game-based balance

rehabilitation tool using the microsoft kinect sensor. International Conference of

the Engineering in Medicine and Biology Society, 1831–1834.

Chaput, J. P., Klingerberg, L., Astrup, & A., Sjödin, A. M. (2011).

Modern sedentary activities promote overconsumption of food in our current

obesogenic environment. Obesity Reviews, 12-20.

Da Gama, A., Chaves, T., Fugueiredo, L., & Teichrieb, V. (2012).

Guidance and Movement Correction Based on Therapeutics Movements for

Motor Rehabilitation Support Systems. IEEE Proceedings of XIV Symposium of

Virtual and Augmented Reality, 191-200.

De Mauro, A. (2011). Virtual Reality Based Reahabilitaion and Game

Technology. eHealth & Biomedical Applications Vicomtech, 48-52.

Freitas, D. Q., Da Gama, A., Figueiredo, L., Chaves, T. M., Marques-

Oliveira, D., Teichrieb, V., & Araújo, C. (2012). Development and Evaluation

of a Kinect Based Motor Rehabilitation Game. Proceedings of SBGames

international conference, 2012, 144-152.

Gerling, K. M., Schild, J., & Masuch, M. (2010). Exergame Design for

Elderly Users: The Case Study of SilverBalance. ACE '10 Proceedings of the

7th Int. Conf. on Advances in Computer Entertainment Technology, 66-69.

Gonzalez, A., Hayashibe, B., & Fraisse, P. (2012). Subject-specific Center

of Mass Estimation for In-home Rehabilitation- Kinect-Wii board vs. Vicon-

Force plate. International Conference on NeuroRehabilitation, 705-709, Spain.

Herbelin, B., Ciger, J., & Brooks, A. L. (2008). Customization of

gaming technology and prototyping of rehabilitation applications. Seventh

International Conference on Disability, Virtual Reality and Associated

Technologies with ArtAbilitation, 211-218.

Holden, M, K. (2005). Virtual environments for motor rehabilitation:

Review. Cyberpsychology and Behavior, 187-211.

52

Holden, M. K., Bettwiler, A., Dyar, T., Niemann, G., & Bizzi,E. (2001).

Retraining movement in patients with acquired brain injury using a virtual

environment. Stud Health Technol Inform, 81, 192-8.

Ishii, F., Matsukawa, N., Horiba, M., Yamanaka, T., Hattori, M., Wada,

I., & Ojika, K. (2010). Impaired ability to shift weight onto the non-paretic leg

in right-cortical brain-damaged patients. Clinical Neurology and Neurosurgery,

406-412.

Jana, A. (2012). Kinect for Windows SDK Programming Guide. Packtt

Pub, Birmingham.

Graves, L., & Stratton, G. (2008). Energy expenditure in adolescents

playing new generation computer games. Br J Sports Med, 592-594.

Harris K., & Reid. D. (2005). The influence of virtual reality play on

children's motivation. Can.J.Occup.Ther., 72, 21-29.

McCreary, M. (1993). Muscles: Testing and Function p.75. [Book]

Mirelman, A., Patritti, B. L., Bonato, P., & Deutsch, J. E. (2010). Effects

of virtual reality training on gait biomechanics of individuals post-stroke. Gait

& Posture, 31(4):433–437.

LaBelle, K. (2011). Evaluation of kinect joint tracking for clinical and

home stroke rehabilitation tools. [thesis]

Lange, B., Suma, E, A., Newman, B., Phan, T., Chang, C. Y., Rizzo, &

A., Bolas, M. (2011). Leveraging Unencumbered Full Body Control of

Animated Virtual Characters for Game-Based Rehabilitation. HCI (14) 2011,

243-252.

Levac, D., & Missiuna, C., (2009). An update on the use of virtual

reality technology to improve movement in children with physical

impairments. CanChild Online published.

Luo, X., Kline, T., Fischer, H., Stubblefield, K., Kenyon, R., & Kamper,

D. (2005). Integration of Augmented Reality and Assistive Devices for Post-

Stroke Hand Opening Rehabilitation. Proceedings 2005 of IEEE Engineering

in Medicine and Biology 27th Annual Conference, 6855-58.

53

Microsoft press. (2012) Human interface guidelines Kinect SDK 1.6.

Retrieved 24, May, 2012, from http://msdn.microsoft.com/en-

us/library/jj663791.aspx.

Sperling, J. W., Nett, M. P., Collins, M. S.(2008). Magnetic resonance

imaging of acute “wiiitis” of the upper extremity. Skeletal Radiol, 481-83.

Nintendo. (2012). Official website. Retrieved Nov 20, 2012, from

http://www.nintendo.com/wii/what-is-wii.

Podsiadlo, D., & Richardson, S. (1991). The timed "up & go": A Test of

Basic Functional Mobility for Frail Elderly Persons. Journal of the American

Geriatrics Society, 121-132.

Koh, T, H. (2000). Ulcerative “nintendinitis”: a new kind of repetitive

strain injury. Medical Journal Aust,. 671-673.

Macgregor, D, M. (2000). Nintendonitis? A case report of repetitive

strain injury in a child as a result playing computer games. Scottish Medical

Journal,. 150.

Rizzo, A., Lange, B., Suma, E, A., & Bolas, M. (2011). "Virtual Reality

and Interactive Digital Game Technology: New Tools to Address Obesity and

Diabetes," Journal of Diabetes Science and Technology, 258-264.

Rizzo, A., & Kim, G. J. (2005). A SWOT analysis of the field of virtual

reality rehabilitation and therapy. Presence, 14(2), 119-146.

Rose, F. D., Brooks, B. M., & Rizzo, A. (2005). Virtual reality in brain

damage rehabilitation: Review. Cyberpsychology & Behavior, 241-262.

Schonauer, C., Pintaric, T., Kaufmann, H., Jansen - Kosterink, S. &

Vollenbroek-Hutten, M. (2011). Chronic pain rehabilitation with a serious

game using multimodal input. 2011 International Conference on Virtual

Rehabilitation (ICVR), 1-8.

Schönauer, C., & Mossel, A. (2012). Distributed Interaction with Kinect

and Mobile Phone in Augmented Reality. IGDA Unity3D, Vienna.

Schultheis, M. T., & Rizzo, A. (2001). The application of virtual reality

technology in rehabilitation. Rehabilitation Psychology, Vol 46, 296-311.

54

Smallwood, S. R., Morris, M. M., Fallows, S. J., & Buckley, J. P.

(2012). Physiologic responses and energy expenditure of Kinect active video

game play in schoolchildren, Arch Pediatr Adolesc Med, 1005-1009.

Sparks, D., Coughlin, L., & Chase, D. (2009). 'Wii have a problem: a

review of self-reported Wii related injuries. Informatics in Primary Care, 55-57.

Standen, P. J., Brown, D. J., (2005). Virtual Reality in the Rehabilitation

of People with Intellectual Disabilities. Review, CyberPsychology & Behavior,

8(3), 272-282.

Stevens, J, A., Corso, P. S,. Finkelstein, E, A., & Miller, T, R. (2006). The

costs of fatal and non-fatal falls among older adults. Injury Prevention, 12(5),

290–295.

Timmermans, A., Saini, P., Willmann, R. D., Lanfermann, G., teVrugt, J.,

& Winter, S. (2007). Home Stroke Rehabilitation for the Upper Limbs. 29th

Annual International Conference of the IEEE Engineering in Medicine and

Biology Society, 4015-4018.

Webb.J & Ashley.J. (2012) Beginning Kinect Programming with the

Microsoft Kinect SDK. Apress, New York.

Wen, C. P., Wai, J. P., Tsai, M. K., Yang. Y. C., Cheng, Y. D., Lee, M.

C., Chan, H. T., Tsao, C. K., Tsai, S. P., & Wu, X. (2011) “Minimum amount

of physical activity for reduced mortality and extended life expectancy: a

prospective cohort study,” Lancet, 1244-1253.

Wiederhold, B. K. (2006). Evaluation of virtual reality therapy in

augmenting the physical and cognitive rehabilitation of war veterans.

Proceedings. 6th Disability, Virtual Reality & Assoc. Tech. 91-96 , Esbjerg,

Denmark.

Wikipedia BBS (2013). Berg Balance Scale. Retrieved April, 23, 2013,

from http://en.wikipedia.org/wiki/Berg_Balance_Scale.

World Population Ageing, (2009). Population Division of the

Department of Economic and Social Affairs of the United Nations.Population

Division of the Department of Economic and Social Affairs of the United

55

Nations. Retrieved at Feb 20, 2013, from

http://www.un.org/esa/population/publications/WPA2009/WPA2009_Working

Paper.pdf.

Tanaka, K., Parker, J, R., Baradoy, G., Sheehan, D., Holash, J, R., & Katz,

L. (2012). A Comparison of Exergaming Interfaces for Use in Rehabilitation

Programs and Research. Loading…, 69-81.

WHO. (2013). Children and Young People with Disabilities Fact Sheet.

Retrieved at May 20, 2013, from

www.unicef.org/disabilities/files/Factsheet_A5__Web_NEW.pdf.

Newquist, & Harvey, P. (1992). Virtual Reality's Commercial Reality.

Computerworld, 93-95.

56

APENDICES

APPENDIX A

FORM OF INITIAL ASSESMENT

57

APPENDIX B

FORM OF FINAL ASSESMENT