1

“Polisocial Award 2012-2013” Politecnico di Milano

Interazione gestuale multicanale a supporto delle attività educativo-terapeutiche per l'autismo

infantile

Progetto presentato da

Dipartimento di Elettronica, Informazione e Bioingegneria

WP2 – Progettazione

Deliverable D2.1: Progettazione della UX (mese 6)

Con il supporto di

2

Abstract

The body of knowledge acquired in WP1 (see D1) gave us the input to begin WP2. WP2 proposes to design a system integrated with games which respond to users’ therapeutic and educational objectives (specified in WP1) and offer children the possibility to:

• Be stimulated from visual, audio, tactile stimuli from two interfaces: one fixed (screen) and one mobile (robot)

• Interact using body gestures, voice, touch with virtual contents on the screen and a physical objects and agent.

This deliverable presents the initial design of the user experience (UX) for a set of games that meet the above requirements.

3

Contents 1. Preliminary concepts ................................................................................................................................................... 4

2. Game UX Design ............................................................................................................................................................. 6

2.1 Statues (1,2,3 Stella) ............................................................................................................................................ 6

2.2 Hide & Seek (Nascondino) ................................................................................................................................ 8

2.3 Colors (Colori) ..................................................................................................................................................... 10

2.4 Traffic Lights (Semaforo) ............................................................................................................................... 12

2.5 Witch Says Color (Strega Comanda Colore) .......................................................................................... 14

2.6 Stories (Storie) .................................................................................................................................................... 16

Annex 1: Strega comanda colore – detailed concept and sketches .................................................... 18

4

1. Preliminary concepts Distilling the know-how acquired during the work performed in WP1 (survey of the state of the art and requirements elicitation in real life contexts) we defined a taxonomy for the role of the robot in the context of motion based touchless games, organized at two levels of abstraction:

i) ROLE OF THE ROBOT IN THE RELATIONSHIP WITH THE CHILD

During the play, robot establishes a relationship with the player, which can be:

• Informative: acting as an actor or substituting a therapist to provide information • Interactive: acting as a playmate interacts with the child • Affective: acting as a friend provides emotive feelings (being stupid in doing something…)

i) FUNCTIONAL ROLE OF THE ROBOT

In the game experience, the robot can address different functional goals

• Feedback: the robot acts as a rewarding agent, it reacts to an action performed by the child • Facilitator: the robot suggests what to do and when to do it, facilitate the entire game play • Prompt: the robot acts as a behavior eliciting agent enhancing the entire game play. • Emulator: the robot can play as an emulator (acts as the child) or can be emulated (supports the

child’s imitative reaction) • Restrictor: the robot limits the child’s spatial possibilities (going around) or offers a choice (blue or

red) • Mediator: the robot mediates social behavior between the child and others (therapist, children?)

In addition, we have conceptualized, at a the high level of abstraction, the interaction space of the triadic relation that involve robot-child-virtual world on the large screen, as reported in the following picture.

5

The following image shows how the various actors interact together and who is in charge of doing what.

The child interacts with the screen prevalently using his body (movements and gestures). Robot, acting in many different ways, communicate with the child using different channels:

• Auditory: sounds and voices • Visual: lights, facial expressions or colors • Motor: movements or gestures

The gameplay can be entirely controlled by the therapists who, governing the pc (bottom left), regulates both robot and screen.

6

2. Game UX Design We defined a structure for the specification of the UX in each game, specifying:

• Game rules • Goals (the skills that should be improved by effect of game play, according to the analysis reported

in D1) • Multimedia contents (visual and audio effects played by the screen or available on the robot) • Robot role and behavior, with related formal and technological features • Child’s actions (gestures and movements to interact with the robot or with the screen) • Further instruments (e.g., additional objects involved in the UX or characteristics of the physical

space, e.g., lines drawn on the floor)

2.1 Statues (1,2,3 Stella)

Game rules The screen acts out as the "Curator". Everyone else (robot and child) playing stands at the far end (distance depends upon playing area selected). The object of the game is for a "Statue" to tag the Curator, thereby resetting the game. The Curator turns their back to the field, and the "Statues" attempt to race across and tag the Curator. Whenever the Curator turns around, the Statues must freeze in position and hold that for as long as the Curator looks at them.

7

If a Statue is caught moving, they are sent back to the starting line to begin again (or thrown out of that round, whichever way is preferred.) Usually, the honesty of the Curator isn't enforced, since being a Statue is more desirable. Goals Promotion of:

• Abstract ideas • Engagement and motivation • Awareness of body shape and schema (be steady) • Imitative capability (mirror neurons) (imitate the movements of the robot) • Posture balance (child should remain steady as soon as the curator looks, so he can be in an

uncommon position) • Gross and fine motor skills (small and big steps) • Expression and reception of a language (robot and screen will speak out)

Multimedia contents

• A face and/or a character Robot roles and behavior

• Emulator: Child can imitate robot’s step • Facilitator: Robot can suggest what to do • Feedback: Robot can reward the child

Child’s actions

• Walking back and forth

8

2.2 Hide & Seek (Nascondino)

Game rules Hide-and-seek is a children's game in which a number of players conceal themselves in the environment, to be found by one or more seekers. The game is played by the screen counting to a predetermined number while the other players hide (robot and child). After reaching the number, the screen attempts to locate all concealed players. Goals Promotion of:

• Abstract ideas • Concentration and attention spans • Engagement and motivation • Imitative capability (mirror neurons) • Environment perception • Expression and reception of a language • Emotion and interest sharing • Inability to generalize between environments (relatedness)

Multimedia contents

• Screen: two eyes looking around Robot roles and behavior

• Emulator: Child can imitate robot’s step

9

• Facilitator: Robot can suggest what to do • Feedback: Robot can reward the child

Child’s actions

• Moving around the play area • Hide him/herself

Further instruments

• Some boxes to let players (child and robot) hide behind

10

2.3 Colors (Colori)

Game rules The screen shows a colored figure (fruit, animal, object…). Verbal instructions can be given by the system or the therapist. The robot wears some buttons to click on. The child has to identify the main color of the figure from a restricted gamut of choices (6 in this example) and press the corresponding botton on the robot. The game proceeds either automatically when the selection ir correct, or with the therapist selecting the next figure to be shown. Goals Promotion of:

• Abstract ideas • Concentration and attention spans • Engagement and motivation • Tactile peculiarity • Expression and reception of a language • Emotion and interest sharing • Non-verbal communication • Joint attention and turn taking

Multimedia contents

• Screen: simple one-colored figures • Robot: buttons of choices

Robot roles and behavior

• Restrictor: the robot restricts the choice of the child

11

• Feedback: the robot provide rewards • Facilitator: the robot facilitate the choice phase (if needed)

Child’s actions

• Click a button on the robot

12

2.4 Traffic Lights (Semaforo)

Game rules Red light/Green light can be seen as a variation of statues but it is slightly different. The robot acts out as the curator (policeman) and stands at one end of the playing field, with the player at the other end. Robot calls "Green light" or "Red Light” while the corresponding light on its body turns on. The screen shows a car moving when the light is Red and zebra crossing when is green. As soon as robot light gets green, the player must walk towards it. At any time, the robot can face the players, calling out "Red light", and the child must freeze in place. If the child fails to stop, he must return to the starting line. Other variations include calling out "Yellow light" as a diversion. Goals Promotion of

• Abstract ideas • Engagement and motivation • Environment perception • Non-verbal communication • Ability to generalize between environments • Control of fear of unpredictability and unfamiliarity • Simultaneous processing of sounds and visual stimuli • Turn taking • Sense of danger

Multimedia contents

• Screen: street, car, zebra crossing and landscape • Robot: colored lights

13

Robot roles and behavior

• Facilitator: robot suggests what to do and when to do it • Restrictor: robot limits child choices • Feedback: robot rewards the child

Child’s actions

• Child goes back and forth and listen to the robot’s instructions Environment • Zebra crossing on the floor

14

2.5 Witch Says Color (Strega Comanda Colore)

Game rules The virtual avatar inside the game acts out as the robot in the physical world. 1. The virtual avatar goes on a colored circle and says the color name. The robot moves to the circle and the child imitates it. 2. The virtual avatar goes on a colored circle and says the color name. The child and the avatar run to the physical circle on the floor. 3. The virtual avatar says the color name. The child and the avatar run to the physical circle on the floor. Goals Promotion of different skills at various levels: Level 1:

• Facial expression • Non verbal

communication • Direct eye gaze • Mimic expression • Emotion and interest

sharing • Understand feelings • Gross motor skill

Level 2:

• Expression and reception of language

• Concentration and attention spans

• Movement coordination

Level 3 :

• Verbal communication • Joint attention and turn

taking • Abstract ideas

15

Multimedia contents

• Screen: an avatar, some colored circled on the virtual floor Robot roles and behavior

• Emulator: Child can imitate robot’s step • Facilitator: Robot can suggest what to do • Feedback: Robot can reward the child • Restrictor: Robot can offer or restrict a choice

Child’s actions

• Moving around the play area • Imitate the robot and the virtual avatar • Listen to the robot and the virtual avatar

Further instruments

• Some colored circles on the floor

16

2.6 Stories (Storie)

Game rules The child see him/herself inside the screen as a story character as well as the robot. The child and the robot, cooperating or not, should become “part of the story”, and execute simple instructions (e.g. say: “Hello”, wave your hand…) visually and verbally played on the screen. A story can be seen as an ensemble of simple predefined micro-activities. Goals Promotion of:

• Awareness of body shape and schema • Facial expression • Mimic expression (imitation and expressiveness) • Postural and gestural understanding • Verbal communication • Imitative capability • Capability to understand other’s feelings or mental states • Concentration and attention spans • Engagement and motivation • Expression and reception of language • Emotion and interest sharing • Figuring out future events (evolution of things)

Multimedia contents

• Screen: shows the story (characters and scene) and visual instructions (PCS) if required

17

• Robot: act as a companion, a friedly playmate Robot roles and behavior

• Mediator: robot acts out as the mediator of a broader interaction (with the therapist?) • Facilitator: robot facilitates child’s performance • Emulator: if needed, robot emulates or let being emulated by the child

Child’s actions

• Follow the story flow • Listen to the screen and robot instructions • Speak if needed • Act out and perform the right gesture when required • Collaborate with the robot if needed

Further instruments

• Playground can be augmented using real objects (a ball, a puppet, a big dice, …) • Imagination can be promoted using real dresses (shoes, pants, hats, …) worn by the child and/or the

robot See examples of stories in http://www.exodusbooks.com/details.aspx?id=2416

18

Annex 1: Strega comanda colore – detailed concept and sketches Fase del processo di apprendimento

Promuovere le abilità di relazione sociale e di comunicazione

Gioco

Strega comanda colore 1° livello

Scopo

Riconoscere emozioni Rispondere al saluto

Svolgimento

1. nel video si vedono un personaggio e un robot

2. il personaggio nel video saluta 3. il bambino risponde al saluto

=> inizia il gioco 4. il personaggio nel video presenta il

robot 5. il robot entra fisicamente in 6. il personaggio nel video si sposta su un

cerchio colorato 7. il robot si sposta e invita il bambino ad

andare su cerchio dello stesso colore 8. il bambino va sul cerchio dello stesso

colore 9. il robot sorride

Canali di comunicazione

Visuale (Audio) Motorio

Ambiente

Ci sono dei cerchi colorati per terra

19

ADS users’ lacks and constrains

Facial expression Non verbal communication Direct eye gaze

Mimic expression Emotion and interest sharing Understand feelings

Gross motor skill

Robot

Accompagna o si fa seguire dal bambino Sorride quando il bambino raggiunge il colore

Video

Immagine del personaggio che saluta e poi presenta il robot Immagine del personaggio che si muove su un cerchio colorato

Fase del processo di apprendimento

Promuovere abilità utili per favorire lo sviluppo

Gioco

Strega comanda colore 2° livello

20

Scopo

Riconoscere emozioni Rispondere al saluto Capacità cognitive e linguistiche

Svolgimento

1. nel video si vedono un personaggio e un robot

2. il personaggio nel video saluta 3. il bambino risponde al saluto

=> inizia il gioco 4. il personaggio nel video presenta il robot 5. il robot entra fisicamente in scena 6. il personaggio nel video dice un colore 7. il bambino e il robot fanno a gara per

raggiungere il cerchio dello stesso colore 8. il robot sorride se vince, è triste se perde

Canali di comunicazione

Visuale Audio

Motorio

Ambiente

Ci sono dei cerchi colorati per terra

ADS users’ lacks and constrains

Facial expression Non verbal communication Direct eye gaze

Mimic expression Emotion and interest sharing Understand feelings

Robot

Si muove verso il colore Sorride se vince, è triste se perde

21

Gross motor skill Expression and reception of language Concentration and attention spans

Movement coordination

Video

Immagine del personaggio che saluta e presenta il robot Immagine del personaggio che dice un colore

Fase del processo di apprendimento

Promuovere attività pratiche

Gioco

Strega comanda colore 3° livello

Scopo Riconoscere emozioni

Rispondere al saluto Capacità cognitive e linguistiche Imparare un gioco da fare con altri bambini

Svolgimento 1. nel video si vedono un personaggio e un

robot 2. il personaggio nel video saluta 3. il bambino risponde al saluto

=> inizia il gioco 4. il personaggio nel video presenta il robot 5. il robot entra fisicamente in scena 6. il personaggio nel video dice un colore 7. il bambino e il robot fanno a gara per

raggiungere il cerchio dello stesso colore 8. il robot sorride se vince, è triste se perde 9. il gioco continua per diverse partite 10. c’è un punteggio delle partite perse e

vinte

22

Canali di comunicazione Visuale

Audio Motorio Tattile?

Ambiente Ci sono dei cerchi colorati per terra

ADS users’ lacks and constrains

Facial expression Non verbal communication

Direct eye gaze Mimic expression Emotion and interest sharing

Understand feelings Gross motor skill Verbal communication

Expression and reception of language long concentration and attention spans Joint attention and turn taking

Movement coordination Abstract ideas

Robot

Si muove verso il colore Sorride se vince, è triste se perde

Video

Immagine del personaggio che saluta e poi presenta il robot Immagine del personaggio che dice un colore Conteggio delle partite Alla fine il personaggio dice chi ha vinto

Sintesi

23

24