automatically generating high-quality user interfaces for appliances

Automatically Generating High-Quality User Interfaces for Appliances

Jeffrey NicholsThesis Proposal Presentation

Thesis Committee:Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr.

3

Problem, cont.Appliances are complex and their user interfaces are often hard to use!

4

Problem, cont.

5

Problem, cont.

April 29, 1991

CNN – Jan 2004

May 2003

6

Thesis Statement

“ A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the user’s performance is better than with the manufacturers’ interfaces for the

appliances.”

7

Approach: Use Mobile Devices

Common, cheap, and often have ability to communicate10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers

8

Approach, cont.

Specifications

Control

Feedback

Appliances Mobile Devices

Use mobile devices to control all appliances in the environment

Key FeaturesTwo-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

9

Automatic Generation of UIs

Benefits• Multiple modalities (GUI + Speech UI)• All interfaces consistent for a user

• With conventions of the handheld• Even from multiple manufacturers

Addresses hotel alarm clock problem

• Create user interfaces that control multiple connected appliances

e.g. a home theater

10

Important Work By Others

INCITS/V2 Standardization EffortAlternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002]

Controlling Appliances• Xweb [Olsen Jr., UIST 2000]• Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001]• Speakeasy [Newman, UIST 2002]

Model-based User Interface Work• UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993]• MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995]• eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002]• User Interface Modeling Language (UIML) [Abrams, W3C 1999]

None of these systems have been validated to generate high-quality interfaces without designer intervention.

11

Research Approach

1. Hand-design remote control interfaces

2. Determine functional information needed from appliances to design user interfaces

3. Design language for describing appliance functions

4. Build interface generators for multiple platforms

5. Perform user studies to evaluate the interface generators

12

Outline

• Introduction

• Hand-Designed Interfaces and User Studies

• Personal Universal Controller (PUC) System

• Handling High-Level Conventions

• Interface Consistency

• Generating Interfaces for the “Experience”

• Validation

• Proposed Contributions & Schedule

13

Initial Approach

What information is needed about the appliance to automatically generate remote control interfaces?

Investigate via a design process

Create interfaces by hand• AIWA Shelf Stereo• AT&T Telephone/Answering Machine

Improve quality with heuristic analysis and think-aloud studies with several users

Compare interfaces with actual appliance interfaces to validate PUC concept

Analyze interfaces for functional information

14

Hand-Designed Interfaces

Interfaces for Microsoft’s PocketPC (simulated remote control)

telephone stereo

15

Comparison StudyCompared performance of first-time users (not experts)

Procedure • Each subject worked two sets of tasks

both stereo and phonecontrolled for order and interface

Performance Metrics• Time to complete all tasks

• Number of times a user manual was needed

• Number of missteps

Appliance Hand-design

Phone

Stereo

16

Comparison Study Results

Using our interfaces, users were twice as fastand made half as many errors

All differences are significant (p < 0.05)

17

Outline

• Introduction






• Validation


18

Architecture

PROTOCOL(two-way communicationof specification & state)

COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)

PUC DEVICES(automatic interface generation)



APPLIANCES(Stereo, Alarm Clock, etc.)

ADAPTOR(publishes description +

appliance state + controls appliance)

control

device specification & state feedback

19

Architecture









control


20

Architecture









control


21

Architecture









control


22

Architecture









control


23

Language Design

Informed by hand-designed interfaces

• What functional information was needed to create interfaces?

Additional Requirements• Support complete functionality of

appliance• No specific layout information• Only one way to specify anything

Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

24

Language Elements

Elements• State variables & commands• Labels• Group tree• Dependency information

Example media player specification

• Play, stop, pause, next track, previous track

• Play list

25

Language Elements, cont.

State Variables and Commands• Represent functions of

appliance

• State variables have types• Boolean, Enumeration,

Integer, String, etc.

• Variables sufficient for most functions but not all

• e.g. “seek” button on a Radio

26


Label InformationOne label not suitable

everywhere• The optimal label length

changes with screen size• Speech interfaces may benefit

from pronunciation and text-to-speech information

“Label Dictionary”• A group of semantically similar

labels• Different lengths• Information for different

modalities

27


Label InformationOne label not suitable

everywhere• The optimal label length

changes with screen size• Speech interfaces may benefit

from pronunciation and text-to-speech information

“Label Dictionary”• A group of semantically similar

labels• Different lengths• Information for different

modalities

28


Group Tree• Specify organization

of functions• We use n-ary tree

with variables or commands at leaves

• Also used for specifying complex types

ListsUnions

29


Group Tree• Specify organization

of functions• We use n-ary tree

with variables or commands at leaves

• Also used for specifying complex types

ListsUnions

30


Dependency Information• Formulas that specify

when a variable or command is active in terms of other state variables

Equals, Greater Than, Less Than, Is DefinedLinked with logical operators (AND, OR)

• Allows feedback to user when a function is not available

31

Interface Generators

Generators for Two Modalities

Graphical• Desktop, PocketPC, and

Microsoft Smartphone

Speech• Ph.D. work of Thomas

Harris• Built on top of the PUC

framework• Implemented using

Universal Speech Interface (USI) techniques [Rosenfeld 2001]

32

Graphical Interface Generator

Rule-based approach• Multiple phases that

iteratively transform a specification into a user interface

Focuses on panel structure of user interface

• Small groups of controls have basic layouts

• Complexity comes from structure of groups

• Structure can be inferred from dependency info!

33

Generation Process

1. Determine conceptual layout• Infer panel structure from

dependencies using “mutual exclusion” property

• Choose controls (decision tree)

• Choose row layout (one column, two column, etc.)

2. Allocate space• Examine panel contents and

choose sizes

3. Instantiate and place controls

4. Fix layout problems

34

Generation Process






choose sizes



35

Generation Process






choose sizes



36

Generation Process






choose sizes



37

Generation Process






choose sizes



38

Generation Process






choose sizes



Without layout fixing rules

With layout fixing rules

39

Outline

• Introduction






• Validation


40

High-Level Conventions

Problem• Human

designers rely partly on conventions when making an interface

• Users expect their appliances to use conventions they know about

41

Smart Templates

Need a way to specify high-level information to interface generators

Solution• Mark groups with tags that identify

high-level informationmedia-controls, phone-dialpad, time, date, etc.

• Restrict the contents of groups so that interface generators can interpret the high-level meaning

• Standardize the tags and restrictions in advance, so that designers know what interface generators expect

42

Smart Templates, cont.

Features• Parameterized• Specified using

primitive elements of specification language

43

Interfaces with Smart TemplatesPreliminary Implementation

A few templates: image, image-list, media-controls, time-duration

44

Continuing Work

Define and implement Smart Templates• date, mute, power, time-absolute, volume, etc.• Develop more as more appliances are specified

Combinations of templates• Less implementation cost than a new template• e.g. date and time-absolute

Use of templates with data already on controller device

• e.g. calendar and address information• Might allow user to enter address from contact list

into navigation system

45

Outline

• Introduction






• Validation


46

Interface Consistency

PUC devices have a unique opportunity to provide consistency for the user

• Personal device• Used for interacting with most appliances

Two ways that PUC UIs can be made consistent• With other applications on the same device• With past interfaces for similar appliances

47

Consistency with Past Interfaces

Two sub-problems to address:

• SimilarityWhich functions of a new appliance are similar to the functions of interfaces generated in the past?

• ConsistencyFor similar functions, what rules from previous interfaces can be applied to ensure consistency?

48

Similarity Problem

Difficult to conclusively know whether two functions from different appliances are the same

• Very little semantic information in the specification language

Can estimate similarity based on properties of state variables

• Smart Template• Name• Group Name• Labels• Type

Improve estimate by looking at relationships between multiple similar variables

49

Similarity & Consistency Problems

new previous

new previous

new previous

sparse similarity

branch similarity significant similarity

50

Outline

• Introduction






• Validation


51

The “Experience”

52

The “Experience”, cont. (MIT dorm)

How can a PUC device provide improved interfaces for connected systems?

53

Improving Interfaces with PUC

Generate an interface that aggregates all functions into one set of screens

Organized by task instead of appliance

Automatically create macros for frequently used functions

e.g. “Play DVD” would: 1. turn on television, DVD player, stereo2. turn off VCR3. set the TV and stereo sources to the DVD player4. instruct the user to insert a DVD (if necessary)5. play the DVD

How can appliance descriptions support these features?

54

Distributed Task Language

My plan:• Store task information within each

appliance description Use an existing task language

• Combine sub-tasks from each appliance in a connected system to create complete tasks

55

Outline

• Introduction






• Validation


56

Two Goals for PUC System

BreadthThe appliance specification language is capable of describing a wide variety of appliances

QualityInterfaces generated for specifications across that range beat the usability of the manufacturers’ interfaces for the same appliances

How do I validate that these goals are met?

57

Evaluation

Evaluation Method1. Develop a list of appliances that are

interesting• Complexity• Unique feature• Representative of a class of appliances

2. Specify each appliance3. Perform a comparative user study on

several appliances to test quality

Also need to explore expert usage of interfaces…

58

Appliance List (so far…comments?)

Already Specified & Generated Propose to Specify

Axis Pan & Tilt Camera GMC Denali Navigation System

Audiophase Stereo Phone/Answering Machine

GMC Denali Driver Information Center

Windows Media Player with playlists

GMC Denali Climate Control System

Automated Teller Machine (ATM)

Sony Camcorder Microsoft Windows XP Media Center PC

Elevator Simulation Photocopier

Lutron Home Lighting System TV Tuner

Windows Media Player without playlists

Personal Video Recorder (e.g. TiVO)

X10 Lighting Powerpoint

Alarm Clock

Projector

59

Outline

• Introduction






• Validation


60

Proposed Contributions

• An abstract appliance modeling language for describing the functionality of appliances

• Algorithms for automatically generating high quality interfaces from that language

• The Smart Templates technique for incorporating high-level conventions

• Algorithms for ensuring consistency across generated interfaces

and…

61

Proposed Contributions, cont.

• A distributed task-modeling language for describing the sub-tasks specific to an appliance in a multi-appliance system

• Algorithms for automatically generating a single user interface for multiple appliances using distributed task information

• Interface generation software on multiple platforms that is shown by user testing to be better than manufacturers’ interfaces for the same appliances

62

Proposed Schedule

March 2004

September

1

2

3

4

5

6

7

8

1 Interface consistency

2 Smart Templates

3 Preliminary user studies with GM vehicle interfaces

4 Improve robustness and quality of interface generators on all platforms

5 Develop distributed task modeling language and multi-appliance user interface generation

6 Implement appliances for final user study

7 Conduct final user study and iterate UI generation rules, if necessary

8 Write dissertation

JuneJune December March2005

September

December2005

63

Conclusion

Problem• Appliances are increasingly complex

• Appliance user interfaces are often hard to use

Solution• Move appliance interfaces to a mobile device the user is

already carrying

• Automatically generate interfaces so that:Interfaces are customized for the device and modality that the user prefersInterfaces for similar appliances can be made consistent

Interfaces for multiple appliances can be combined into a single interface

• Validate generated interfaces to prove high quality claim

64

AcknowledgementsThesis Committee

• Brad A. Myers (chair)• Scott Hudson• John Zimmerman• Dan Olsen Jr.

Funding• National Science

Foundation• Microsoft• General Motors• Intel• Pittsburgh Digital

Greenhouse

Equipment Grants• Mitsubishi (MERL)• VividLogic• Lucent• Lutron• Lantronix• Nokia

PUC Project Members• Kevin Litwack• Thomas K. Harris• Michael Higgins• Joseph Hughes • Roni Rosenfeld• Rajesh Seenichamy• Pegeen Shen• Htet Htet Aung• Mathilde Pignol• Suporn Pongnumkul• Stefanie Shriver• Jeffrey Stylos• Peter Lucas

Collaborators & Friends• Naomi Ramos• Desney Tan• Daniel Avrahami• Gaetano Borriello• Laura Dabbish• Andrew Faulring• James Fogarty• Krzysztof Gajos• Darren Gergle• Andy Ko• Amy Nichols• Mick Nichols• Sally Nichols• Trevor Pering• Fleming Seay• Irina Shklovski• Roy Want• Jake Wobbrock• and many others…

Thanks for listening!

For more information…http://www.cs.cmu.edu/~pebbles/puc/http://www.cs.cmu.edu/~jeffreyn/

Automatically Generating High-Quality User Interfaces for Appliances

Jeffrey NicholsThesis Proposal Presentation

Thesis Committee:Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr.

67

Outside the Scope

• Help systems for generated interfaces• Automated trouble-shooting for complex

systems• Service Discovery• Macros and End-User Programming• Security• Inter-operability with INCITS/V2

68

Palm Interfaces

Initially designed paper-prototype interfaces for Palm

telephone stereo

69

Extra Group Tree

70

Choosing Panel Structure

a)

b)

c)

full screen

tabbed

partial screen

71

Controlling AppliancesWe have built adaptors for many actual appliances

• Sony Digital Camcorder• Windows Media Player• Axis UPnP Pan & Tilt Camera• Lutron Lighting• X10 Lighting• Audiophase Shelf Stereo• AudioReQuest MP3 player• GM Vehicle Information System• GM Vehicle Climate Control

Written specifications for others• Elevator• Telephone/Answering Machine• GM Navigation System• Several Alarm Clocks

72

Smart Templates Example

73

Smart Templates, cont.

One template can be used across multiple appliances and can be generated on multiple platforms

74

Consistency Problem

Apply different consistency techniques depending on similarity

• Sparse similarityUse similar controls for similar functionse.g. use a scroll bar for new volume, even if this would not be the normal representation (might be a combo box)

• Branch similarityUse the previous interface structure of branch in the new interfacee.g. branch of previous interface was shown in a tabbed panel. In the new interface, separate functions of new interface with tabs and have similar functions in a tabbed panel.

• Significant similarityUse the group tree from the previously generated interfaceInterpolate extra functions into the tree

75

The “Experience”, cont.

How can a PUC device provide improved interfaces for these systems?

Users often think of multiple connected appliances as one system

• Home theater• CMU lecture room system• others…

automatically generating high-quality user interfaces for appliances

Documents