applying user experience metrics for optimizing library...

Xin Wang, Senior Lecturer (PhD), University of North Texas Brian Rennick, Associate Dean for Information Technology

(M.S.)., Brigham Young University Library

Applying User Experience Metrics for Optimizing Library Floor Map Application

使用用户计量数据优化图书馆地图软件设计

1

2

Outline

ØWhat are user experience (UX) metrics?

ØWhy are they important?

ØHow do you choose and apply them to inform the design decisions?

ØA real life example: Design floor map through incorporating user experience metrics

ØWhat are user experience research?

3

§ User-Centered Design and Development (UCD) Process for Information Architecture (IA)

Fig. 1: Three clouds of IA processDing, W. & X. Lin.(2010). IA research, design ,and evaluation. In Information Architecture: The design and integration of information spaces (pp.23-39). San Rafael: Morgan & Claypool Publishers.

Research

§Understand business goals and contexts§Employ user research methods to solicit user feedback§Analyze research results§Develop design strategies

Design

§Develop blueprints, wireframes, low/high-fidelity prototypes for user interfaces

Evaluation

§Perform log analysis or web usage analysis§Conduct usability evaluation

1. UX Research Framework

Research

§Understand business goals and contexts§Employ user research methods to solicit user feedback§Analyze research results§Develop design strategies

Design

§Develop blueprints, wireframes, low/high-fidelity prototypes for user interfaces

Evaluation

§Perform log analysis or web usage analysis§Conduct usability evaluation

4

§ UX Research Framework

Context

Content Users

Business goals, funding, technology, human resources

Document/data types, metadata, volume, existing structure

Audiences, tasks, needs, Information seeking behavior, vocabularies, experience

Table 1. UX Research Techniques

1. UX Research Framework

Source: Ding, W. & X. Lin.(2010). IA research, design ,and evaluation. In Information Architecture: The design and integration of information spaces (pp.23-39). San Rafael: Morgan & Claypool Publishers.

ContextBackground Research

Presentations and meetings

Stakeholder interviews

Technology assessment

ContentHeuristic Evaluation

Content analysis

Content mapping

Comparative Benchmarking

UsersCard Sorting User Interviews

and user testingContextual inquiry

Search log and clickstream analysis

Survey Focus group

Table 2. IA Research Methods

5

Personas

1. UX Research Techniques

6

2. What are User Experience Metrics?

ØA metric is a way of measuring a particular phenomenon or thing;

ØUX metrics are based on a reliable system of measurement;ØUX metrics reveal the interaction between a user and a

product; § UX metrics are observable and quantifiable;§ UX metrics are “People-oriented”§ Interaction § Attitude§ Behaviors

Source: Tullis, T. & Albert, B. (2013). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability metrics. 2nd Edition. MA: Elsevier.

Type of UX Research Task

Suc

cess

Task

Tim

e

Erro

rs

Effic

ienc

y

Lear

nabi

lity

Issu

e-Ba

sed

Met

rics

Self-

Repo

rted

Met

rics

Beha

vior

al &

Psy

chic

al

Met

rics

Com

bine

d &

Com

para

tive

Met

rics

Live

Web

site

Met

rics

Card

-Sor

ting

Met

rics

1.Completing a transaction X X X X X 2. Comparing products X X X X 3. Evaluating frequent use of the same product X X X X X 4. Evaluating navigation and/or information architecture X X X X5. Increasing awareness X X X 6. Problem discovery X X 7. Maximizing usability for a critical product 8. Creating an overall positive user experience X X 9. Evaluating the impact of subtitle changes X 10. Comparing alternative designs X X X X X

7

Table 3: Ten Types of Common Usability Study and the Metrics that Maybe Most Appropriate for each

Ø Commonly Used UX Metrics

2. What are User Experience Metrics?

Source: Tullis, T. & Albert, B. (2013). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability metrics. 2nd Edition. MA: Elsevier.

• Task Success• Task Time• Errors • Efficiency (Time)• Learnability• Issue-Based Metrics• Behavioral & Psychical Metrics• Combined & comparative

Metrics• Live Website Metrics• Card-sorting Metrics

8

§ Card sorting: a technique of organizing the elements of an information system in a way that makes sense to users

§ Implementing card sorting:1. Open/Closed sorts2. Cards are content items from a site or an information system3. Number of cards: 50-60 or less4. Number of participants: “15” is a good target5. Granularity: cards can be high-level or detailed (items exist

at a similar level is better for one-round sorting)6. Cross-listing: defining primary hierarchy or exploring

alternate navigation paths?7. Quantitative and qualitative data analysis

4. Apply UX Metrics into Design Decisions

9

Fig 1: BYU Library Responsive Website and the Floor Maps Application

5. A Real Life Example: a Floor Map Application

ØIn 2014, BYU libraries converted the website to one responsive design that performed equally well on all devicesØThe library wayfinding system was converted to a

responsive design as part of the overall redesign of the library website.

10

§ OptimalSort : a powerful tool to find out how users think your content should be organized


11

1. Open card sorts: define the application structure with users’ perspectives

2. Number of cards: 503. Number of participants: 164. Granularity: high-level (content topics of main-page categories)5. Cross-listing: were not allowed6. Data analysis: Both quantitative and qualitative data analyses


12

This sorting activity is like a fun game !

Fig.2 Instructions for the Card Sorting Activity

§ Participants can complete the card sort process from their own computers§ The intriguing process of card sorting increases the response rate

During the data analysis process, you probably have noticed that the

card sorting is not a completely objective and precise technique. Many times, you need to make

some subjective decisions. Different participants will sort cards in different ways, so the

cluster analysis and dendrogram won't produce the ideal

information architecture for you. However, OptimalSort can help you

start examining the patterns and the strength of the relationships. In order to pull together a suitable

grouping of items, you need to refer to the Cluster analysis, the

Dendrogram, the raw sort results plus participants provided grouping names. Of course, there might also

be business rules or real world contraints that mean certain items

have to go in certain places. Sometimes, unfortunately, politics

gets in the way of a good information architecture. Luckily,

you can also use the data from your sessions to help convince

management that it's time for a change. I like to print all the

information off, arrange it on my desk, and just absorb it for awhile.

Then, I try creating groups that seem to best match the majority view based on the sort results. I'll check the groups I create against the individual raw data, in case

there was some people who sorted an entirely different way. But my

aim is to make a hierarchy that will be acceptable to everyone who

participates in the sort. In the end, you need to apply a combination of

knowledge from the cluster analysis, dendrogram, and what

participants said during the sort, in order to create a good first pass at an information architecture. You should also use other data you have, such as usability studies,

customer support data, search logs, and web logs, to inform your

analysis. Don't blindly follow the statistical output. Think about what

participants said, and about real world implications. Back up your

decisions with this data. If you can't find data to back up your

decision, it indicates you might not have it right.

The next step, after we have our abstract hierarchy, is to refine it by testing it with a reverse sort. That's

the next topic.

13

§ Standardize similar categories

Fig. 3 The Category Page in OptimalSort


Standardized category name

Participants’ category names

14

§ Find out the frequency of a specific card being placed in a normalized category§ Pick out cards with higher frequency§ Form a user-centered category with corresponding subcategories

15 out of 16 people placedThe “Family History”under this category.

Fig. 4 A Standardized Category


Help Centers Count

Family History 15Accounting Lab 12

Teaching & Learning Lab 12

Microforms Reference 12

American Heritage Lab 11

Research & Writing Center 10

Social Science/Education 9

Business & Economics 9Science/Maps 8

15

Figure 5: Similarity Matrix of Card Sorting Data

Identifying obvious cluster “Library Services”


Similarity Matrix: shows the percentage of participants who agree with each card paring.

16


Figure 6: Standardization Grid

Ø Standardization Grid: shows the distribution of cards across the standardized categories you have defined. Each table cell shows the number of times a card was sorted into the corresponding scandalized category.

17


Dendrograms: are used to illustrate the clusters. A “X%” score tells % of participants agree with this grouping.

Figure 7 : Standardization Grid: Dendrograms

18

§ Proposed New Categories for Global & Secondary Navigation

Figure 8: Newly Proposed Navigation and Labeling Systems of the Floor Map Application


19

Ø The card sorting is not a completely objective and precise technique. Many times, you need to make some subjective decisions.

Ø OptimalSort can help you start examining the patterns and the strength of the relationships.

Ø There are also be business rules or real world constraints that mean certain items have to go in certain places. Sometimes, unfortunately, politics gets in the way of a good information architecture.

Ø To create a good first pass at an information architecture. You should also use other data you have, such as usability studies, customer support data, search logs, and web logs, to inform your analysis.

Ø Don't blindly follow the statistical output. Think about participants’ feedback, and about real world implications. Back up your decisions with this data

Ø After we have our abstract hierarchy, is to refine it by testing it with a reverse sort.

5. Conclusions

applying user experience metrics for optimizing library...

Documents