china hci symposium 2010 march: augmented social cognition research from parc, by ed h. chi

Image from: http://www.flickr.com/photos/ourcommon/480538715/

Enhancing the Social Web through Enhancing the Social Web through Augmented Social Cognition ResearchAugmented Social Cognition Research

Ed H. Chi 紀懷新 , Area ManagerPeter Pirolli, Lichan Hong, Bongwon Suh, Gregorio Convertino, Les Nelson, Rowan Nairn

Augmented Social Cognition AreaPalo Alto Research Center

Interns: Sanjay Kairam, Jilin Chen, Michael BernsteinAlumni: Raluca Budiu, Bryan Pendleton, Niki Kittur, Todd Mytkowicz, Terrell Russell, Brynn Evans, Bryan Chan, KMRC students

12010-03-15 Ed H. Chi ASC Overview

PARC OverviewPARC Overview Interdisciplinary research

center

Founded in 1970

Spun out of Xerox in 2002

Business model:– Contract research

– Licensing

– Joint ventures

– Spinoffs

2010-03-15 2Ed H. Chi ASC Overview

INNOVATIONINNOVATION

chartered to create the architecture of information & the office of the future- invented distributed personal computing

- established Xerox’s laser printing business

- created the foundation for the digital revolution

Graphical User Interface

Laser Printing

Ethernet

Bit-mapped Displays

Distributed File Systems

Page Description Languages

First Commercial Mouse

Object-oriented Programming

WYSIWYG Editing

Distributed Computing

VLSI Design Methodologies

Optical Storage

Client/Server Architecture

Device Independent Imaging

Cedar Programming Language

14 years of work in visualization and foraging Information Scent

– WUFIS / IUNIS (Basic scent modeling algorithms) [CHI2000,2001]

– Bloodhound (Simulation of web navigation) [CHI2003]

– LumberJack (Log analysis of user needs) [CHI2002] Information Foraging

– ScentTrails [TOCHI2003]– ScentIndex [CHI2004]– ScentHighlight [IUI2005]– Visual foraging of highlighted text [HCII]

Visualization and Sensemaking– Visualization of Web Ecologies [CHI98]– Visualization Spreadsheets [Infovis97, Infovis99]


V. Bush’s Vision of Augmented V. Bush’s Vision of Augmented CognitionCognition

Problem: – Intellectual over-specialization

The Memex Extend the powers of the human

mind with technology– Individuals could attend to greater

spans

– Facile command of all recorded knowledge

– Sharing of knowledge gained


Finding a Finding a RestaurantRestaurant

Appropriate for the occasion

2010-03-15 Ed H. Chi ASC Overview 6

Poor heuristic

Good heuristic

HeuristicsHeuristics


““Hints”Hints”

Solo

Cooperative (“good hints”)


Wikipedia Success is Wikipedia Success is counter-intuitivecounter-intuitive

“ Wikipedia is the best thing ever. Anyone in the world can write anything they want about any subject, so you know

you’re getting the best possible information.”

– Steve Carell, The Office


WikipediaWikipedia


Research VisionResearch Vision

Augmented Social CognitionAugmented Social Cognition Cognition: the ability to remember, think, and reason; the

faculty of knowing. Social Cognition: the ability of a group to remember, think,

and reason; the construction of knowledge structures by a group.– (not quite the same as in the branch of psychology that studies

the cognitive processes involved in social interaction, though included)

Augmented Social Cognition: Supported by systems, the enhancement of the ability of a group to remember, think, and reason; the system-supported construction of knowledge structures by a group.

Citation: Chi, IEEE Computer, Sept 2008


Research MethodologyResearch Methodology

Characterize activity on social systems with analytics Model interaction social and community dynamics and

variables Prototype tools to increase benefits or reduce cost Evaluate prototypes via Living Laboratories with real users

12Ed H. Chi ASC Overview2010-03-15 12

Characterization Models

PrototypesEvaluations

2010-03-15 14

Conflict/Coordination Effects in Conflict/Coordination Effects in WikipediaWikipedia[Kittur et al., CHI2007][Kittur et al., CHI2007]

60%

65%

70%

75%

80%

85%

90%

95%

100%

2001 2002 2003 2004 2005 2006

Percentage of total edits

Article

User

Article Talk

User Talk

Other

Maintenance

(joint work with Niki Kittur, Bongwon Suh, Bryan Pendleton)(joint work with Niki Kittur, Bongwon Suh, Bryan Pendleton)

Ed H. Chi ASC Overview

Conflict in WikipediaConflict in Wikipedia Conflict is growing at the global level, and we

have some idea about where it is. But what defines conflict inside Wikipedia? Build a characterization model of article

conflict– Identify metrics relevant to conflict– Automatically identify high-conflict articles


Measure of controversyMeasure of controversy “ Controversial” tag

Use # revisions tagged controversial


Performance: Cross-Performance: Cross-validationvalidation SVM Classifier, 5x cross-validation, R2 = 0.897

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Predicted controversial revisions 182010-03-15 Ed H. Chi ASC Overview

Performance: Cross-Performance: Cross-validationvalidation 5x cross-validation, R2 = 0.897

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Predicted controversial revisions


Revert Graph Revert Graph [Suh et al., IEEE VAST [Suh et al., IEEE VAST 2007]2007]

Research Goal– How can we identify point of views between users?– Group people share a common point of view

Revert: Undoing one or more edits– Restored to a version that existed sometime previously. – Often used to fight vandalism

Force directed layout– Node: user, Edge: revert relationship


Opinions on Opinions on Dokdo/TakeshimaDokdo/Takeshima

Group A

Group B Group C

Group D

Number of users in user group A B C Total

Users with Korean point of view 10 6 0 16

Users with Japanese point of view 1 8 7 16

Neutral or Unidentified 7 3 6 17


Mediator Pattern - Terri Mediator Pattern - Terri SchiavoSchiavo

Mediators

Sympathetic to parents

Sympathetic to husband

Anonymous (vandals/spammers)


Ratio of Reverted Ratio of Reverted Contribution Contribution

Monthly Ratio of Reverted EditsMonthly Ratio of Reverted Edits


Example: Modeling Example: Modeling Wikipedia GrowthWikipedia Growth

Bongwon Suh, Gregorio Convertino, Ed H. Chi, Peter Pirolli


Bongwon Suh, Gregorio Convertino, Ed H. Chi, Peter Pirolli. The Singularity is Not Near: Slowing Growth of Wikipedia. In Proc. of WikiSym 2009. Oct, 2009. Florida, USA

Something happened in early Something happened in early 20072007

Monthly EditsMonthly Edits


Slowing Growth in Global Slowing Growth in Global ActivityActivity

Monthly Active EditorsMonthly Active Editors


Earlier Exponential Growth Earlier Exponential Growth ModelModel Edits beget edits

– more number of previous edits, more number of new edits

€

N(t) = N0 ⋅ert

€

dN

dt= r ⋅N

Growth rate of populationGrowth rate of population

CurrentpopulationCurrent

population

Growth rate depends on current population size N and r = growth rate of the population


Logistic Growth ModelLogistic Growth Model Ecological population growth model

– r, growth rate of the population– K, carrying capacity (due to resource limitation)

€

dN

dt= r ⋅N ⋅(1−

N

K)

K


http://en.wikipedia.org/wiki/Wikipedia:Modelling_Wikipedia’s_growthhttp://en.wikipedia.org/wiki/Wikipedia:Modelling_Wikipedia’s_growth

Number of New ArticlesNumber of New Articles Follows a logistic growth curve

New ArticleNew Article


A Modified Logistic ModelA Modified Logistic Model Carrying Capacity as a function of time.

K(t)


Two Sides of TaggingTwo Sides of Tagging

Encoding Retrieval

33

http://edge.org

“science research cognition”

http://www.ted.com/index.php/speakers

“video people talks technology”


Using Information Theory to Model Social Using Information Theory to Model Social TaggingTagging[Ed H. Chi, Todd Mytkowicz, ACM Hypertext 2008][Ed H. Chi, Todd Mytkowicz, ACM Hypertext 2008]

TopicsConcepts

Users Documents

Tags

T1…TnEncodingDecoding

Noise


H(Tag) shows saturation in tag usage H(Tag) shows saturation in tag usage



II((DocDoc; ; TagTag) Mutual ) Mutual InformationInformation

Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)

Raise in avg. tag per bookmarkRaise in avg. tag per bookmark(note parallel the development in increasing # of (note parallel the development in increasing # of query words)query words)


Understanding a new area…Understanding a new area…

2010-03-15 39




MrTaggy.com: MrTaggy.com: social search browser with social social search browser with social bookmarksbookmarks

Joint work with Rowan Nairn, Lawrence Lee

Kammerer, Y., Nairn, R., Pirolli, P., and Chi, E. H. 2009. Signpost from the masses: learning effects in an exploratory social tag search browser. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 625-634.


• Synonyms• Misspellings• Morphologies

People use different tag words to express similar concepts.

Social Tagging Creates Noise


2010-03-15 43

Guide

Web

Howto

TipsHelp

Tools

Tip

Tricks

Tutorial

Tutorials

Reference

Semantic Similarity GraphSemantic Similarity Graph

TagSearch: TagSearch: Use Semantic Use Semantic Analysis to Reduce NoiseAnalysis to Reduce Noise http://mrtaggy.com


http://mrtaggy.com/

MapReduce ImplementationMapReduce Implementation

Spreading Activation in a bi-graph Computation over a very large data set

– 150 Million+ bookmarks

Tags URLs

P(URL|Tag)

P(Tag|URL)


Understanding a new area…Understanding a new area…

2010-03-15 46




Baseline Baseline InterfaceInterface

Experiment DesignExperiment Design 2 interface x 3 task domain design

– 2 Interface (between-subjects) Exploratory vs. Baseline

– 3 task domains (within-subjects) Future Architecture, Global Warming, Web Mashups

30 Subjects (22 male, 8 female)– Intermediate or advanced computer and web search skills– Half assigned Exploratory, half Baseline.

For each domain, single block with 3 task types:– Easy and Difficult Page Collection Task [6min each]– Summarization Task [12min]– Keyword Generation Task [2min]

Procedure [2 hours]Procedure [2 hours] Prior Knowledge Test 1st Task Domain

– With easy and difficult page collection tasks, summarization and keyword generation task.

– NASA cognitive load questionnaire 2nd Task Domain

– Same battery of tasks and cognitive load questionaire

3rd Task Domain Experimental Survey

Experimental Evauation Experimental Evauation [Kammerer et al, CHI2009][Kammerer et al, CHI2009]

Exploratory interface users:– performed more queries, – took more time, – wrote better summaries (in 2/3 domains), – generated more relevant keywords (in 2/3 domains),

and– had a higher cognitive load.

Suggestive of deeper engagement and better learning.

Some evidence of scaffolding for novices in the keyword generation and summarization tasks.


Model-Driven Research Model-Driven Research MethodologyMethodology

Characterize activity on social systems with analytics Model interaction social and community dynamics and

variables Prototype tools to increase benefits or reduce cost Evaluate prototypes via Living Laboratories with real users

51Ed H. Chi ASC Overview2010-03-15 51



Living Laboratory:Living Laboratory:Prototyping Social Applications Prototyping Social Applications on the Interneton the Internet

Create a Living Laboratory as a platform to develop, test, and market innovations

[HCIC workshop 2009, HCII 2009, IEEE Computer Sep/2008]


WikiDashboard for WikipediaWikiDashboard for Wikipedia


SparTag.us Social SparTag.us Social Reading ToolReading Tool


Topic-oriented browsing of your Twitter feedTopic-oriented browsing of your Twitter feed

Browse over arbitrary periods of timeBrowse over arbitrary periods of time

Dashboard identifies topics that might be of interestDashboard identifies topics that might be of interest

AugmenteAugmented Social d Social

CognitionCognition

Mail2Tag

Build on email practicesBuild on email practices

To promotesharingacross the organization

To promotesharingacross the organization

email-based news sharing system where people ‘CC’ news to keyword tags

Mail2Tag systemMail2Tag system

Research Platform StrategyResearch Platform Strategy

Extracts data in the form of tuples from applications, e.g.

(user, tag, URL)

(user, activity, object)

Hadoop MapReduce, Pig, MySQL, Django, Java

Social Data Mining Platform

Pattern Operators, e.g., Tag Normalization, LDA Clustering,

Summarization, Voting Techniques…

Recommendations

Dashboard

Expertise Identification

Topic Identification

ASC is creating a plug-and-play platform to enable a number of applications in support of the Open Web Applications

Combine with other applications to create full products

App ConnectorsApp Connectors




…

Core Advantage

Augmented Social Cognition Augmented Social Cognition questions:questions: Crowdsourcing [collaborative co-creation]

– Is there a wisdom of the crowd in Wikipedia? – How does conflict drive content creation?

Collective Intelligence [folksonomy]– Are social tags collectively gathered useful for organization of

a large document collection? Collective Averaging [social attention]

– Does voting systems identify the best quality and most interesting information for that community?

Participation Architecture [interaction] – Does lowering the interaction cost barrier increase

participation productively? Expertise finding [social networking]

– Does getting experts through social network gets you to better quality information sooner?


The TeamThe Team


Image from: http://www.flickr.com/photos/ourcommon/480538715/

Augmented Social Cognition:Augmented Social Cognition:From Social Foraging to Social From Social Foraging to Social SensemakingSensemaking

Research Vision: Understand how social computing systems can enhance the ability of a group of people to remember, think, and reason.

Living Laboratory: Create applications that harness collective intelligence to improve knowledge capture, transfer, and discovery.

http://asc-parc.blogspot.comhttp://[email protected]


Collective Intelligence

Augmented Social CognitionAugmented Social Cognition


Higher Productivity via Collective Intelligence

Intelligence that emerges from the collaboration and competition of many individuals

search

sharing

foraging

TagSearch: Mining social data for automatic data clustering and organization:

• Better organization via user-assigned tags

• Better UI for browsing interesting contents

• Recommendation instead of just search

Social Transparency create trust and attribution:

• Increase participation via attribution

• Increase credibility and trust with community feedback

• Reduce wiki risks

SparTag.us: sharing of interesting contents:

• A notebook that automatically organizes your reading

• Social sharing of important and interesting tidbits

• Viral sharing of highlighted and tagged paragraphs

Foundation:• Understanding of human

cognition and behavior• Data mining of social data• Modeling of consensus-

driven decision-making

Generic benefits:• Greater trust• Better decision-making• Useful sharing of info• Auto-organization thru

social data

High-end of the collaboration High-end of the collaboration spectrumspectrum Groups utilize systems

to make sense and share complex topics and materials.

Wikipedia (social status)

Slashdot (karma points)

WikiHow.com Lostpedia.com


Middle of the spectrumMiddle of the spectrum

Systems that evolve structures that can be used to organize information.

Del.icio.us Flickr YouTube Friendster


Lightweight social processesLightweight social processes Counting votes

– A way to increase signal-to-noise ratio– Information faddishness

Examples:– Digg.com– Most bookmarked items on del.icio.us

– Estimating the weight of an ox or temperature of a room

– The true value of a stock

– PageRank or Hub / Authority algorithms


A way to think about these A way to think about these systemssystems

Voting systems Collaborative Co-Creation

Col. Information Structures

Naver

Heavier collaboration

Digg.comWikipedia

Slashdot

eHow.com

Del.icio.us

IBM dogearPageRank

Flickr


Layers of Models NeededLayers of Models Needed

Voting systems Collaborative Co-Creation

Col. Information Structures

Naver

Heavier collaboration

Digg.comWikipedia

Slashdot

eHow.com

Del.icio.us

IBM dogearPageRank

Flickr

Understanding of micro-economics

• of foraging [PARC]

• Personal vs. group

[Huberman, Adamic]

• Wisdom of Crowd

[Surowieki]

• Information cascades

[Anderson and Holt]

Understanding of conflicts and coordination

• Wikipedia coordination costs [PARC]

• Invisible Colleges [Sandstrom]• Interference effects [Pirolli]• Co-laboratories [Olson and

Olson]• Community networks / Col.

Problem solving [Carroll]

Understanding of info and social networks

• Tag network analysis [PARC,

Golder, Yahoo]

• Structural holes (info brokerage)

[Burt]

• Network constraints and

structure [various]

• Semantic of semiotic structures /

words [IR, LSA]


WikiDashboard: WikiDashboard: Social Transparency for Social Transparency for WikipediaWikipedia

Joint work with Bongwon Suh, Aniket Kittur, Bryan Pendleton

Bongwon Suh, Ed H. Chi, Aniket Kittur, Bryan A. Pendleton. Lifting the Veil: Improving Accountability and Social Transparency in Wikipedia with WikiDashboard. In Proceedings of the ACM Conference on Human-factors in Computing Systems (CHI2008). ACM Press, 2008. Florence, Italy.


Social DashboardSocial Dashboard Social translucent for effective communication and

collaboration [Erickson and Kellogg 2002]– Make socially significant information visible and salient– Support awareness of the rules and constraints– Accountability for actions

Wikis can be a prime candidate– Every edit is logged and retrievable– WikiScanner.com: analyze anonymous IP edits– WikiRage.com: top edits


Top Editor - Wasted Time RTop Editor - Wasted Time R


Subprime Mortgage CrisisSubprime Mortgage Crisis


Experimental Evaluation DesignExperimental Evaluation Design 3 x 2 x 2 design

Abortion

George Bush

Volcano

Shark

Pro-life feminism

Scientology and celebrities

Disk defragmenter

Beeswax

Controversial Uncontroversial

High quality

Low quality

Visualization

• High stability

• Low stability

• Baseline (none)


Example: High trust Example: High trust visualizationvisualization


Example: Low trust visualizationExample: Low trust visualization


MethodMethod Users recruited via Amazon’s Mechanical Turk

– 253 participants– 673 ratings– 7 cents per rating– Kittur, Chi, & Suh, CHI 2008: Crowdsourcing user

studies To ensure salience and valid answers,

participants answered:– In what time period was this article the least stable?– How stable has this article been for the last month?– Who was the last editor? – How trustworthy do you consider the above editor?


ResultsResults

1. Significant effect of visualization– High > low, p < .001

2. Both positive and negative effects– High > baseline, p < .001– Low > baseline, p < .01

3. No effect of article uncertainty– No interaction of

visualization with either quality or controversy

– Robust across conditions

1

2

3

4

5

6

7

Low qual High qual Low qual High qual

Uncontroversial Controversial

Tru

stw

orth

ines

s ra

ting

High stability Baseline Low stability


TagSearch Exploratory FocusTagSearch Exploratory Focus

79

3 kinds of search

navigational transactional

28% 13%

You know what you want and where it is You know what you want to do

Existing search engines are OK

informational

59%

You roughly know what you want

but don’t know how to find it

Difficult for existing search engines

Opportunity


SparTag.us: SparTag.us: Social Paragraph-level TaggingSocial Paragraph-level Tagging

Joint work withLichan Hong, Raluca Budiu, Les Nelson, Peter Pirolli

Lichan Hong, Ed H. Chi, Raluca Budiu, Peter Pirolli, and Les Nelson. SparTag.us: A Low Cost Tagging System for Foraging of Web Content. In Proceedings of the Advanced Visual Interface (AVI2008), (to appear).

ACM Press, 2008.802010-03-15 Ed H. Chi ASC Overview

Lowering Participation / Lowering Participation / Interaction CostsInteraction Costs

Interaction costs determine number of people who participate

Surplus of attention & motivation at small transaction costs

Therefore… Important to keep

interaction costs low Cost of participation#

Peo

ple

wil

ling

to p

rodu

ce f

or “

free

”


SparTag.usSparTag.us

In situ tagging while reading– No new window– Clicking vs typing

Tagging + highlighting


Paragraph TaggingParagraph Tagging

Intuition: sub-doc nuggets useful– Entities, facts, concepts, paragraphs

Annotations attached to paragraphs Portable across pages and other contents

(e.g. Word documents)– Dynamic pages– Duplicate content


Duplicate Content via Duplicate Content via Paragraph Fingerprinting Paragraph Fingerprinting [Hong [Hong

and Chi, CHI2009]and Chi, CHI2009]


My Reading NotebookMy Reading Notebook


Social SharingSocial Sharing

friend’s tagsfriend’s tags

my tagsmy tags

my highlightsmy highlights friend’s highlightsfriend’s highlights


Importance IndicatorImportance Indicator

recallrecall first-visitfirst-visit


WithoutSparTag.us

(WS)

WithoutSparTag.us

(WS)

SparTag.usOnly (SO)SparTag.usOnly (SO)

SparTag.usWith A

Friend (SF)

SparTag.usWith A

Friend (SF)

Experimental Evaluation: Significant Experimental Evaluation: Significant Learning GainLearning Gain

N=18 SparTag.us + Friend superior to both individual conditionsNo difference between the two controls

SF group, M=0.46, SD=0.22

SO group, M=0.13, SD=0.32

WS group, M=0.27, SD=0.23

[Nelson et al., CHI2009]


china hci symposium 2010 march: augmented social cognition research from parc, by ed h. chi

Technology