human social and behavioral variability: scientific ... social and behavioral variability:...

Human Social and Behavioral Variability:Scientific Challenges and Opportunities

Adam RussellBill Regli

Discover DSO Day

June 15, 2017

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

2

“Imagine how much harder physics would be if electrons could think.”

- Murray Gell-Mann

“Imagine how much harder physics would be if electrons had feelings.”

- Richard Feynman

Why DARPA?

https://www.pinterest.com/explore/gary-larson/

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

3

Why DSO?

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited) 4

Why are social systems “hard”?

https://xkcd.com/793/

5

Humans may present additional (unique?) complexity

molecular cellular organ individual group/family community society

GROUP

Not SelectedSelected

Mea

n N

PYD

ELT

(Rec

over

y - B

asel

ine)

-2

-4

-6

-8

-10

-12

-14

-16

-18

Mrazek, Alissa J. et al. Culture and Brain 1.2-4 (2013): 100–117http://www.businessinsider.com/shaq-simone-biles-picture-super-bowl-2017-2

Van Dongen et al. Sleep (2004) May 1; 27(3):423-33https://sportsscientists.com/2011/08/talent-training-and-performance-the-secrets-of-success/

http://www.cam-can.org/highlights/index.htmlCourtesy of Dr. Andy Morgan, Univ New Haven

Shariff et al. Psych Sci (2007), 18(9); 803-809http://www.technologyreview.com/view/538866/the-social-network-illusion-that-tricks-your-mind/

Mao et al. Nature (2017) Jan 13;8:13800

Henrich et al. Behav Brain Sci (2010) Jun;33(2-3):61-83

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

6

Humans may present additional (unique?) complexity

GROUP

Not SelectedSelected

Mea

n N

PYD

ELT

(Rec

over

y - B

asel

ine)

-2

-4

-6

-8

-10

-12

-14

-16

-18

Mrazek, Alissa J. et al. Culture and Brain 1.2-4 (2013): 100–117http://www.businessinsider.com/shaq-simone-biles-picture-super-bowl-2017-2

Van Dongen et al. Sleep (2004) May 1; 27(3):423-33 https://sportsscientists.com/2011/08/talent-training-and-performance-the-secrets-of-success/

http://www.cam-can.org/highlights/index.htmlCourtesy of Dr. Andy Morgan, Univ New Haven

Shariff et al. Psych Sci (2007), 18(9); 803-809http://www.technologyreview.com/view/538866/the-social-network-illusion-that-tricks-your-mind/

Mao et al. Nature (2017) Jan 13;8:13800

Henrich et al. Behav Brain Sci (2010) Jun;33(2-3):61-83

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

7

http://psychsciencenotes.blogspot.com/2010/09/assume-cow-is-sphere.html

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

8

What do we need?

http://boldarc.com/produkter/vr-ar-och-plattskarm/Adapted from Evolution, 2007 Cold Spring Harbor Laboratory Press

Courtesy of Josh Plotkin, Univ Penn - 2017http://www.ugrad.cs.ubc.ca/~cs414/notes/sampling.html

https://www.pinterest.com/pin/509962357776759635/https://en.wikipedia.org/wiki/Open_science

Gieannetto, D. and Heydari, B. (2015) Nature: Scientific Reports | 6:21870 | DOI: 10.1038/srep21870

Slide shots courtesy of Ed Wack, MIT-LL

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

9

What do we have?

PREDICT:• Post-hoc • Little generalizability

DESIGN:• Under-powered• Single method

COLLECT:• Small lab studies• Big correlational data

ANALYZE:• Inappropriate

statistics• Overfitting

REPRODUCE:• Misaligned incentives• Barriers to sharing

MODEL:• Narrow modeling• Narrow teaming

Many challenges are technical, many are cultural, some are both

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited) 10

One result

http://www.winchestermysteryhouse.com/

11

Next Generation Social Sciences (NGS2)

Program Concept• Build a new capability (methods,

models, tools, and a community of researchers) for rigorous, reproducible experimental research at scales necessary to understand emergent properties of human social systems

Technical Objectives• Use the exemplar challenge of “collective

identity” (CI) to develop and demonstrate new approaches for predictive modeling, experimentation, and reproducibility in social science

• Develop and demonstrate the impact of these approaches for accelerating discovery and understanding of human social behavior

Model

Predict

DesignCollect

Analyze

Reproduce

http://www.fbicaaa-bham.org/page-1298796; Lada Adamic, personal webpage

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

12

The road to a program: Early “Exploration”

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

13

The road to a Program: Request for Information

http://blog.executivebiz.com/2016/09/darpa-seeks-info-on-simulation-platform-development-to-validate-social-science-research-methods/

http://www.psychologicalscience.org/publications/observer/obsonline/darpa-seeks-information-on-experimental-falsifiability.html#.WRmdvfnyvRY

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

14

The road to a Program: Meet George Heilmeier

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

- What are you trying to do? Articulate your objectives using absolutely no jargon.

- How is it done today, and what are the limits of current practice?

- What is new in your approach and why do you think it will be successful?

- Who cares? If you succeed, what difference will it make?

- What are the risks?

- How much will it cost?

- How long will it take?

- What are the mid-term and final “exams” to check for success?

15

The road to a Program: The BAA

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited) 16

More roads are needed…

http://www.darpa.mil/news-events/socialsim-proposers-day http://science.dodlive.mil/2014/09/28/math-the-key-to-compiling-science-data/http://www.darpa.mil/news-events/2017-04-06 http://www.darpa.mil/news-events/next-generation-social-science-proposers-day

http://www.darpa.mil/program/2016-11-28http://www.darpa.mil/news-events/2017-04-07

17

Challenges

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

18

Challenge 1: “Big” Ethnography

https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7

How do we get here?

“Humans are animals suspended in webs of meaning they themselves have spun.” - Clifford Geertz

Lada Adamic, personal webpage

Insert hand clasp and arm wrestle – which is

which?

http://basementrejects.com/review/west-side-story-1961/ Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

19

Challenge 2: AI Coaches

How can we use human variability to

design for/thrive in complexity?

What does a machine need to know about you to be a coach and not just a tool?

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

20

Challenge 3: Quantifying Human Entanglement in Complex Environments

Network of Facebook friendships, Emilio Ferrara 2015https://taraingalls.wordpress.com/2013/07/17/is-social-media-ruining-relationships/

https://en.wikipedia.org/wiki/Ecological_systems_theoryDistribution Statement “A” (Approved for Public Release, Distribution Unlimited)

21

Challenge 4: Designing Un-gameable Games

https://www.nap.edu/read/18622/chapter/5#81https://policytensor.com/2013/08/21/asymmetric-escalation-games-and-the-taiwan-dilemma/

From this:To this:

https://ourworld.unu.edu/en/systems-thinking-and-the-cobra-effect

“Neglect of the qualitative dimension often renders social-scientific research deeply problematic.”

– Crossley and Edwards, 2016

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited) 22

Moving towards “Solution-Oriented Social Sciences”

http://openeducationresearch.org/2009/01/pasteurs-and-edisons-quadrants/

23

From Topics to Goldilocks Problems

B. Clearly hard

Etsy, TheArtifulBookworm

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

A. Clearly important

C. Clearly measurable progress

24

Where are the Goldilocks Problems for the Noosphere?

How can we shape what happens?

What will happen next?

Micro-scale

What has happened?

What is happening?

Meso-scale

Macro-scale

Prediction/Correlation

Explanation/Causality

“If you do not know how to ask the right question, you discover nothing.”

- W. Edwards Deming

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)