mitsmr making better decisions collection
DESCRIPTION
MITSMR Making Better Decisions Collection, a collection of the MIT Sloan Management ReviewTRANSCRIPT
-
SPECIAL COLLECTIONWINTER 2015
sloanreview.mit.edu
Learn how asking the right questions can help you make smarter decisions, how to make decisions with data using simula-tions and how new strategies can improve your decision making.
Making BetterDecisions
A SPECIAL COLLECTION OF MIT SMR ARTICLES
Brought to you by:Brought to you by:
-
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW i
CONTENTSSPECIALCOLLECTIONWINTER 2015
Making Better Decisions
1 The Art of Asking Pivotal Questions By Paul J.H. Schoemaker and Steven Krupp
`10 Using Simulated Experience to Make Sense of Big Data By Robin M. Hogarth and Emre Soyer
16 Why You Decide the Way You Do By Bruce Posner
Please note that gray areas reflect artwork that has been intentionally removed. The substantive content of the article appears as originally published.
-
PLEASE NOTE THAT GRAY AREAS REFLECT ARTWORK THAT HAS BEEN INTENTIONALLY REMOVED. THE SUBSTANTIVE CONTENT OF THE ARTICLE APPEARS AS ORIGINALLY PUBLISHED.
WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 39
The Power of Asking Pivotal QuestionsIn a rapidly changing business landscape, executives need the ability to quickly spot both new opportunities and hidden risks. Asking the right questions can help you broaden your perspective and make smarter decisions.BY PAUL J.H. SCHOEMAKER AND STEVEN KRUPP
GOOD STRATEGIC THINKING and decision making often require a shift in perspective particularly in environments charac-
terized by significant uncertainty and change. What worked in the
past simply may not apply in the future. Asking what if questions
about the future may create discomfort, since answers are often not
obvious. But asking such questions also forces you to step back and
challenge current assumptions that prevent you from seeing break-
through solutions. This article builds on our new book, Winning the
Long Game: How Strategic Leaders Shape the Future,1 by focusing on
the art of asking pivotal questions to improve strategic decision
making. (See About the Research, p. 41.) By presenting six ques-
tions that challenge executives to incorporate broader perspectives,
our aim is to stimulate out-of-the-box dialogues that help leaders
make better choices and find innovative solutions sooner.
Are You Solving The Right Problem?Back in the 1960s, IBM Corp. had the opportunity to buy or license
Xerox Corp.s new reprographic photo process. IBM hired the con-
sulting firm Arthur D. Little to answer a key question: If a more
reliable, cheaper and faster process for photocopying were available,
how many more copies would people make in a given year? Since
copies in those days could only be made from an original specimen,
ADL set out to estimate the number. Both companies framed the
problem too narrowly as copies from originals, ignoring a new
segment of the market that turned out to be many times larger
D E C I S I O N M A K I N G : T H E R I G H T Q U E S T I O N S
THE LEADING QUESTIONHow can managers make better decisions?
FINDINGSExamine both broad market trends and less visible undercurrents.
Seek out diverse views to see multiple sides of complex issues.
Push back when consensus forms too quickly.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 1
-
40 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : T H E R I G H T Q U E S T I O N S
(namely, copies of copies of copies).2 This huge,
overlooked opportunity could only have been fore-
seen if different questions had been posed. IBM
might have owned this new revolutionary technol-
ogy if the key question had been framed as how
might the new Xerox process change when and how
people make copies, and what might this grow to in
total number of copies made in future years?
The questions leaders pose sometimes get in the
way of solving the right problem or seeing more
innovative solutions. They are often too narrow,
overly protective of the current business, or assume
that the old habits, business models and regula-
tions will remain largely intact. At Google Inc.,
CEO Larry Page challenges leaders to anticipate the
future better by not just asking what is or likely will
be true, but what might be true, even if unex-
pected.3 The matter of what is the right question
should be much more central when leaders tackle
complex and important decisions, especially in an
era of profound change.
Think Outside InQuestion One: How well do you understand the
implications of broad market trends and less vis-
ible undercurrents for your business and for
upcoming strategic choices? Entrepreneurs like
Elon Musk from Tesla Motors, Steve Jobs from
Apple and Jeff Bezos from Amazon became known
for spotting unmet market needs and figuring out
how to serve them profitably. The best entrepre-
neurs excel at peeking around the corner and seeing
the future sooner.4 Weve found that leaders can
learn to anticipate better by simply being more
curious, looking for superior information, con-
ducting smarter analyses and developing broader
touch points with those in the know.
In an interview on CNN, Musk was asked where
his forward-thinking, innovative ideas come from.
He replied, Just trying really hard the first order
of business is to try. You must try until your brain
hurts.5 Ever since he was in college in the early
1990s, Musk had a vision of commercializing elec-
tric vehicles for the mass market and was
questioning how this could be achieved, given the
historic pushback against this idea. He mused that
getting into the electric car business was probably
one of the stupidest things you could do.6 (Even
Toyota Motor Corp. chairman Takeshi Uchiya-
mada, known as the father of the Prius, had
reservations about electric cars: Because of its
shortcomings driving range, cost and recharg-
ing time the electric vehicle is not a viable
replacement for most conventional cars.7) Musk
saw electric vehicles as the future, but if their devel-
opment was left to traditional car companies, he
thought it would take a long time.
In Musks view, the industry was operating
under two false premises: One, that you could not
create a compelling electric car; and two, that no
one would buy it.8 The challenge was to demon-
strate that electric cars can be a mainstream
product and to reassure consumers that infrastruc-
ture can be developed to give them the freedom and
reliability of a regular car.9 Well before others,
Musk saw the possibilities and asked different ques-
tions. Although this story is far from over, Musks
vision has struck a chord with consumers and Wall
Street. He expanded his enterprise to include global
distribution and battery manufacturing shortly
after the Tesla Model S was rated the number one
car ever tested by Consumer Reports in 2013.
The Challenge Strategic leaders are focused on the future and are
masters at asking discerning questions and exploring
ideas and options that are outside the mainstream.
They are wary of status quo views and prefer honest,
transparent questions that focus on how much, or
how little, is really known about the issue at hand.
Many studies emphasize the importance of strategic
thinking and anticipation, while also lamenting the
shortage of leaders who do this well.10 Those who
miss the early signals often come late to the party
when customer tastes are changing or when nontra-
ditional competitors are preparing to disrupt or
blindside them.
To protect themselves, companies must keep an
eye on innovations from both existing companies
and startups. Some of the ideas could become game
changers, and you may have to team up with the in-
novators, as a number of big pharmaceutical
companies have done with biotech companies.
Tips and Pointers 1. Learn from startups. What are they doing and
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 2
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 41
why? What do they see that you dont? Examine
their moves to detect market shifts and emerging
opportunities from the outside in.
2. Go to conferences outside your function or in-
dustry. In its Connect + Develop innovation
program, Procter & Gamble Co. reaches out to
companies outside the consumer products indus-
try to share lessons and explore joint challenges.11
Follow events in other regions and sectors, even if
they seem unrelated to your business at first.
3. Leverage current networks and join new ones.
How might you engage your existing networks
more systematically to stay on top of new develop-
ments? Join interest groups in adjacent businesses
or areas to expand your worldview and examine
questions you dont typically consider.
Explore Future ScenariosQuestion Two: How thoroughly have you ana-
lyzed major external uncertainties and future
scenarios that could significantly impact your
business decisions? Leaders must not only under-
stand the deeper trends but also the key uncertainties
that can rock their world. One way to do this is
through scenario planning and war gaming.12 For
example, a pediatric hospital in the U.S. Midwest
was grappling with rapid consolidation in its mar-
ket. Larger hospitals focused mostly on adult
patients and were actively looking to merge or to
form strategic alliances. In anticipation, the CEO of
the pediatric hospital engaged his board in a simu-
lation, presenting them with a hypothetical
scenario: a merger between two particular adult-
patient hospitals. He asked board members to
identify potential alliance partners, decide on an ac-
tion relative to competitors and assess their
hospitals readiness to execute the plan.
Then, the CEO introduced a second scenario: a
disruptive technology coupled with onerous new leg-
islation. The exercise spurred new questions and
helped the CEO crystallize a plan. The CEO deter-
mined that, if certain adult hospitals merged, the
competing pediatric hospital would likely want to
merge as well. Shortly thereafter, when two adult hos-
pitals in the region announced a major consolidation,
the CEO and his board were prepared to act. They
proposed a partnering arrangement to the other pedi-
atric hospital and were able to stay ahead of the curve.
The Challenge Developing different views of how the external en-
vironment may change allows leaders to better
determine whether the organization has sufficient
strategic flexibility to succeed. Scenarios can pick up
early indicators about how emerging technologies
or social trends might disrupt your current business
model, how customers preferences may change or
why new regulations could alter your industry.13
Asking what could happen in the future involves
imagination and curiosity. It pays, for example, to
ponder how and where a well-armed rival could at-
tack your business.
Even though good tools exist to raise important
questions about future uncertainties, time-pressured
executives occupied with putting out fires or
exploiting short-term gains arent always receptive
to them. For example, for several years leading up
to the U.S. subprime mortgage crisis that began in
2007, the investment community overlooked or
largely ignored the possibility that the subprime
mortgage boom might go bust. In a congressional
hearing in the fall of 2008, Standard & Poors presi-
dent Deven Sharma claimed, Virtually no one
be they homeowners, financial institutions, rating
agencies, regulators, or investors anticipated
what is occurring.14 Yet leading economists, in-
cluding Paul Krugman and Robert Shiller, and
savvy investors, such as Steve Eisman and John
Paulson, had been sounding the alarm.15 The in-
triguing question is not why top executives at large
rating agencies failed to acknowledge the elephant
in the room but why some investors and analysts
spotted the elephant sooner than others.
ABOUT THE RESEARCHThis article draws on multiple sources, including our ongoing research at the Wharton Schools Mack Institute for Innovation Management and our consulting work at Decision Strategies International. The discussion of the six questions we examine here draws on our book Winning the Long Game: How Strategic Lead-ers Shape the Future (PublicAffairs, 2014). We pretested the conceptual model underlying the book with many executives and then collected data from more than 30,000 managers representing diverse companies, functions and back-grounds around the world. Using factor analysis and other standard tests of validity, we refined the survey questions and identified remedies. The full-length version of our survey contains 39 items and has been used for self-assessment, peer-assessment and 360-degree feedback (sample size of 278). A shorter ver-sion of the self-assessment was published at Inc.com in March 2012, with a link enabling readers to complete a 12-item assessment online at www.decisionstrat .com (sample size of more than 30,000 at present). These two data sets were used to test the statistical reliability and validity of the instrument.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 3
-
42 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : T H E R I G H T Q U E S T I O N S
Tips and Pointers1. Identify weak signals at the boundaries of your
business. Strategic leaders ask questions about
the external business environment that have far-
reaching implications and then ask team
members to scout the periphery for emerging
trends.
2. Conduct war games to assess the perspectives of
competitors and stakeholders. Gauge their likely
reactions to novel opportunities or threats.
3. Analyze rivals, especially nontraditional ones,
and examine which of their moves puzzle
you and why.
Be a ContrarianQuestion Three: Do you regularly seek out diverse
views to see multiple sides of complex issues, and
do you purposely explore important problems
from several angles? A persistent problem for many
teams is promoting diverse thinking and creative
friction. Leaders must always ask if the team has
sought sufficient contrarian input and been ex-
posed to all sides of an issue before reaching a
decision. This can counter the tendency of many
team members to go along to get along. Offering
contrarian views is particularly essential when tack-
ling major strategic decisions in an uncertain
environment.
To promote diverse thought, Hala Moddelmog,
former president of Atlanta, Georgia-based Arbys
Restaurant Group Inc., a fast-food chain with
about 3,400 locations, surrounded herself with
colleagues of different races, geographies, socioeco-
nomic classes and personality styles. You really
dont need another you, she said. Staying open to
different viewpoints helps ensure leaders are not
unduly hindered by decision traps and can instead
open their eyes to information or solutions that
they may not have previously considered.16
Research shows that creative tension promotes
better idea generation and group problem solv-
ing.17 Constructive dissent and debate encourages
people to reexamine current assumptions to make
room for creative thinking. John Lasseter, chief cre-
ative officer at Pixar, Walt Disney Animation
Studios and DisneyToon Studios, has practiced a
powerful form of team challenge. Each morning at
Pixar, the team working on a movie would review
their previous days output and explore how to im-
prove. They were asked to provide tough questions,
offer honest critique and put alternatives on the
table.18 This practice was based on the belief that
team decisions were superior to any individuals,
but only if you pushed people out of their comfort
zones. Some team members had to get used to
being challenged and critiqued, but most came to
see how the product and decision improved.
Author Malcolm Gladwell has noted that the
best entrepreneurs and innovators are usually quite
disagreeable they love debate. He has gone so far
as to argue recently that an important role of senior
management in creating an atmosphere of inno-
vation is allowing people to be disagreeable.19 This
echoes an idea philosopher John Dewey presented
in 1922: Conflict is the gadfly of thought. It stirs us
to observation and memory. It instigates to inven-
tion. It shocks us out of sheep-like passivity, and
sets us at noting and contriving. Conflict is a
sine qua non of reflection and ingenuity.20
The Challenge The opposite of using questions to promote diver-
gent thinking is to coalesce around shared viewpoints
or succumb to groupthink. Amazons Jeff Bezos
decries social cohesion as the cloying tendency of
people who like to agree with each other and find
consensus comfortable.21 In response, he says he
tries to create a culture at Amazon where leaders
challenge decisions they disagree with, even when
doing so is uncomfortable or exhausting.
Bezos isnt the first business leader to value dis-
sent. As chairman of General Motors Corp., Alfred
P. Sloan Jr. told senior executives at the end of a
board meeting, I take it we are all in complete
agreement on the decision here. Then I propose
we postpone further discussion of this matter until
our next meeting to give ourselves time to develop
disagreement and perhaps gain some understand-
ing of what the decision is all about.22 Of course,
how conflict is handled differs strongly by culture.
Finding the right balance between encouraging
people to express diverse views and not offending
others requires cultural sensitivity, especially in
multinational settings. The benefits of frank debate
can dissipate quickly if they trigger resentment or
backstabbing.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 4
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 43
Tips and Pointers 1. Foster constructive debate in meetings. Help
leaders and team members to get used to a more
candid dialogue with creative friction about ideas.
2. Keep teams small. Amazon forms task forces of
just five to seven people, which makes it easier to
test ideas and guard against groupthink.
3. Push back when consensus forms too quickly.
Insist on alternatives. Like GMs Sloan, challenge
teams if they agree too fast on a complex issue,
and ask them to reflect more deeply and develop
constructive disagreement.
4. Use devils advocates. Before meetings, ask
someone to prepare the case against the prevail-
ing view, and rotate this role. Train people to
question the status quo and get them to appreci-
ate the benefits of such questioning.
Look for PatternsQuestion Four: Do you deploy multiple lenses to
connect dots from diverse sources and stakehold-
ers, and do you delve deep to see important
connections that others miss? As the then-CEO of
DuPont, Charles O. Holliday Jr. picked up several
weak signals in the fall of 2008 that helped him pre-
pare his company for the deep recession that
followed. While visiting a major Japanese customer,
Holliday learned that the CEO had instructed his
staff to conserve cash, an indication that the
company was seeing or expecting a decline in prof-
itability. That got Hollidays attention, both in
terms of the potential for weaker economic condi-
tions and specific fears about DuPonts own cash
position. Upon his return, Holliday sought to get a
fix on DuPonts financial resilience. The leadership
team found that the initial signs of weakness were
spreading to the broader economy and beginning
to affect DuPonts business across the board.23
But how big a problem would it be? Holliday
learned that reservations at the Hotel du Pont,
located near the companys Wilmington, Delaware,
headquarters, had dropped 30% in 10 days, which
was highly unusual for the end of the year. He also
discovered that many corporate lawyers were set-
tling disputes rather than exposing clients to the
financial uncertainty of a trial. And several U.S.
automakers, huge DuPont paint customers, were
scaling back on production schedules. Holliday
wanted to know why. The answer wasnt compli-
cated: Orders for new cars were dropping as the
number of U.S. mortgage foreclosures increased,
and the economy was going downhill.
The Challenge What was impressive about Holliday was his ability
to amplify discrete data points, connect them and
take decisive action. Combining seasoned intuition
with vigilant questions, Holliday figured out that his
company was about to hit a wall. To test his fears, he
engaged his team and asked for candid feedback. His
team put a plan in place so DuPont would be ready if
financial markets hit rock bottom.
Leaders are often limited by selective percep-
tion and seek information that confirms what
they wish to believe. Unlike Holliday, most dont
ask tough questions because they filter out weak
signals that dont fit their mental models. When
faced with complex issues and conflicting infor-
mation, it is easy to fool yourself: If you torture
the data hard enough, it will confess to almost
anything! At Eastman Kodak Co., for example,
leaders failed to ask the right questions soon
enough to fully understand and act effectively on
the signs that photography was rapidly moving to
digital. This misperception reflected middle man-
agements belief that digital technology was
inferior to film and top executives belief that the
demands of Kodaks shareholders mattered more
than those of its consumers and engineers.24 These
flawed assumptions allowed Kodak to continue
When people feel pressed for time, they become less flexible and much prefer certainty to ambiguity. Ambiguity aversion is typically heightened in crisis situations and can lead to cognitive myopia, a narrow focus that can be counterproductive.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 5
-
44 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : T H E R I G H T Q U E S T I O N S
deluding itself about the urgency for change for
much too long.
Tips and Pointers1. Look for competing explanations to challenge
your observations. Engage a wide range of stake-
holders, customers and strategic partners to
weigh in.
2. When stuck trying to recognize patterns or
interpret complex data, step away, get some
distance and then try again. Sleep on the data,
since the mind continues to process information
when resting. Each time DuPonts Holliday took
a break, and then reengaged, he got a deeper
understanding and asked better questions.
3. Use visual graphs or flowcharts to juxtapose
the larger picture with the individual puzzle
pieces. Pattern recognition is easier when all the
information is clearly laid out and presented in
different ways. Try to leverage the power of visual
thinking.25
Create New Options Question Five: Do you generate and evaluate
multiple options when making a strategic deci-
sion, and do you consider the risks of each,
including unintended consequences? It may seem
obvious that leaders should examine multiple
options before making a big decision. Yet in the
heat of the battle, few leaders actually engage in
creative options thinking. A common refrain is:
We dont have time, weve got to move. Research
shows that when people feel pressed for time, they
become less flexible and will much prefer certainty
to ambiguity. Ambiguity aversion is typically
heightened in crisis situations and can lead to
cognitive myopia, a narrow focus that can be coun-
terproductive.26 Weathering storms, real or
metaphorical, requires strategic leaders to counter
this ambiguity aversion. Asking good questions
about alternatives or unintended consequences,
even if done quickly in a crunch, will provide a
wider-angle lens to include the less obvious and
potentially more strategic course of action.
When a devastating storm hit the annual Sydney
to Hobart Yacht Race in Australia in 1998, nearly all
of the more than100 yachts that started the race
were either trying to outrun the storm or heading
directly for the shore. A notable exception was the
crew of AFR Midnight Rambler. They asked a criti-
cal question in the midst of the life-threatening
storm: Are there other options? Rather than get-
ting ahead of the storm or racing to shore, the
Midnight Rambler saw a third possibility: sailing
directly into the storm. Although it was a noncon-
ventional choice, the Midnight Rambler crew
concluded that it would be the safest and the fastest
option. They also believed they had the skill to exe-
cute this bold plan. The Midnight Rambler not
only survived traumatic moments; it won the race.
Many boats were capsized and destroyed, few
finished and six sailors tragically died.27 The Mid-
night Rambler had the smallest boat and fewest
resources. But its crew was the only one to ask a
crucial question in the face of the storm: Are there
creative options?
The Challenge Major disruptions, such as the appearance of new or
unexpected competitors, often lead to quick
action with little reflection akin to the fight-or-
flight response of animals. When we are under the
gun, we frequently cut corners. This makes us prone
to the traps of narrow focus and inside-out thinking
that limit choices. We rely too much on ourselves or
on an inner circle. This can blind us to possibilities
that reflect outside perspectives and potential conse-
quences for customers or external stakeholders.
In 2011, Netflix Inc., which had been very suc-
cessful with its DVD rental-by-mail model, added a
Leaders are often limited by selective perception and seek information that confirms what they wish to believe. Most dont ask tough questions because they filter out weak signals that dont fit their mental models.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 6
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 45
second delivery system based on Web download-
ing. To be competitive, CEO Reed Hastings decided
to unbundle the streaming service from the tradi-
tional model and offer it at a lower price. However,
the combined fee for both subscriptions ended up
being 60% higher than the original service. This in-
furiated consumers. In the following year, Netflix
lost 800,000 customers, and its stock price fell 65%.
By not asking the right questions, Netflix failed to
fully explore options that might be more flexible
and user-friendly. Although Hastings quickly
owned up to the mistake and publicly apologized,
the episode caused a lot of grief for both customers
and the company.
Tips and Pointers1. Rather than presenting binary go/no-go deci-
sions, reframe a situation to always examine
several more options. Always ask, What else
might we do?
2. Use impromptu meetings when time is limited
to generate more options, including unconven-
tional choices. The Midnight Rambler crew did
this during a major crisis.
3. Review alternatives based on clear criteria and
rank options accordingly. Clearly define deci-
sion criteria, make them explicit, weigh them and
then score each option against the criteria to
identify the best choice. Be disciplined when it
comes to making tough trade-offs.
Learn From FailureQuestion Six: Do you encourage experiments
and failing fast as a source of innovation and
quick learning? David Ogilvy, the advertising ge-
nius, purposely ran ads that he and his team did
not believe would work as a way to test their own
theories about advertising.28 One of the experi-
ments they tried was the famous Hathaway shirt
advertisement featuring a man with an eye patch.
This version of the ad (there were 17 others) was
an impromptu experiment whose success took
Ogilvy by surprise.29 The ad, in fact, was a brilliant
success, ran for a long time and received several
industry prizes.
Biologist Max Delbrck, who received a Nobel
Prize in 1969, believed in the principle of limited
sloppiness. He advised his students to be sloppy
enough in their lab experiments to allow for the
unexpected, but not so sloppy that they could not
identify the reasons for their anomalous results.30
Case in point: the eccentric Scottish scientist, Sir
Alexander Fleming, who received a Nobel Prize in
1945. His peers considered him brilliant but some-
what sloppy. In 1928, after a long summer holiday,
Fleming returned to his lab and began gathering up
the contaminated petri dishes for a good scrub-
bing. Suddenly, he noticed something different
about one of them: There was a halo where a blue-
green mold appeared to have dissolved the bacteria.
Many biologists might have missed the small irreg-
ularities, but Fleming knew bacterial growths as an
artist knows the color spectrum; in fact, he had
occasionally shaped colonies of Staphylococcus
into portraits of his coworkers. His keen perception
and curiosity led to the first breakthrough in what
later became the wonder drug penicillin.31
Talking about and even celebrating failure has
become central to the folklore about entrepreneurs.
Sara Blakely, who founded Spanx Inc., a fast-grow-
ing maker of slimming undergarments and other
apparel based in Atlanta, Georgia, got the idea for
the companys products from her own dissatisfac-
tion with the undergarment options available on
the market. She credits her success in part to a ques-
tion her father used to pose at the dinner table
when she was growing up: What have you failed at
this week? The message she got was that failure
wasnt about making mistakes it was about not
trying new things.32
Learning from mistakes has much to do with a leaders mind-set and the questions that he or she asks both before and after an unexpected event occurs. Strategic decision makers abandon the pursuit of perfection, allow some room for well-intentioned mistakes, and examine what went wrong and why.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 7
-
46 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : T H E R I G H T Q U E S T I O N S
The Challenge Learning from mistakes has much to do with a
leaders mind-set and the questions that he or she
asks both before and after an unexpected event
occurs. Strategic decision makers abandon the
pursuit of perfection, allow some room for well-
intentioned mistakes, and examine what went
wrong and why. What matters is how well a team
learns from setbacks and what mode of inquiry it
allows. The best teams try to fail fast, often and
cheaply in search of innovation.33
Few leaders are willing to give more than lip
service to failure; most corporate cultures view
missteps as crippling rather than as sources of
innovation. Only in Silicon Valley, perhaps, do
people wear their failures with pride.34 The
blame culture that permeates most organiza-
tions paralyzes decision makers so much that they
dont take chances and they sweep missteps under
the rug. Many companies dont learn and adapt
fast enough from missteps, as we saw with Black-
berry, Nokia and Microsoft in their early responses
to the iPhone.
To help a team learn faster, leaders must (1) frame
mistakes as valuable learning opportunities; (2) re-
spond to failure as temporary, isolated and not
personal;35 and (3) emphasize that learning from a
decision is a goal in itself.
In the U.S. and Israeli militaries, after-action
debriefs have become the norm.36 Debriefing in the
Israeli military is so highly valued that everyone is
graded on this skill, including their ability to create
a climate that accepts mistakes as natural and a
source for learning. This background training has
spread beyond the military to Israels venture com-
munity. Nearly all entrepreneurs in Israel have
served in the military, since such service is manda-
tory for both men and women. These shared
experiences, especially tolerance for failure and
after-action reviews, have helped Israel become a
technological innovation hub.37
Tips and Pointers 1. Shine a light on mistakes as sources of new
learning. Blakely of Spanx grew up in a home
where her parents admired her for trying and
failing. She incorporates this view into her lead-
ership philosophy.
2. Conduct after-action reviews to extract in-
sights. Define mistakes and successes in terms of
process rather than outcomes. Teach team mem-
bers to ask questions that elicit learning rather
than defensiveness.
3. Publicize stories about failed projects that led
to innovative solutions. Praise those who
learned from their errors and try to extract learn-
ing from near misses.
Start With Questions Not AnswersTypically, we dont judge leaders on the quality of
their questions, nor do we design our educational
systems or corporate training to develop this cru-
cial skill. If anything, we do the opposite. Television
game shows reward contestants who know answers
to preset questions and usually very trivial ques-
tions at that. Having encyclopedic knowledge may
win you a million dollars on a TV game show or
yield good grades in school, but it wont necessarily
make you successful in todays complex business
world. In changing environments, the big prizes go
to those who ask better questions and learn faster.
In organizations, this comes down to leaders teach-
ing and coaching others to think more strategically
and ask deeper questions. If you think like everyone
else, you are likely to be average. The best strategic
thinkers, leaders and entrepreneurs distinguish
themselves by how they frame decisions, the kinds
of questions they ask and their mode of inquiry.38
Paul J.H. Schoemaker is research director of the William and Phyllis Mack Institute for Innovation Management at the Wharton School of the University of Pennsylvania in Philadelphia and the founder and executive chairman of Decision Strategies International, a consulting firm with offices in Conshohocken, Pennsylvania, and London. Steven Krupp is the CEO of DSI. This article builds on the authors new book, Winning the Long Game: How Strategic Leaders Shape the Future (PublicAffairs, 2014). Comment on this article at http://sloanreview.mit.edu/x/56214, or contact the authors at [email protected].
REFERENCES
1. S. Krupp and P.J.H. Schoemaker, Winning the Long Game: How Strategic Leaders Shape the Future (New York: PublicAffairs, 2014).
2. V.P. Barabba, Meeting of the Minds: Creating the
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 8
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 47
Market-Based Enterprise (Boston: Harvard Business Press, 1995).
3. E. Schmidt and J. Rosenberg, How Google Works (New York: Grand Central Publishing, 2014).
4. G.S. Day and P.J.H. Schoemaker, Are You a Vigilant Leader? MIT Sloan Management Review 49, no. 3 (spring 2008: 43-51).
5. Transcript of Fareed Zakaria Global Public Square, aired Jan. 5, 2014, http://transcripts.cnn.com.
6. Making a Mark With Rockets and Roadsters, NPR, Aug. 9, 2007, www.npr.org.
7. W. Hall, Father of the Prius Declares Electric Cars Not Viable, Feb. 4, 2013, www.breitbart.com.
8. R. Lawler, Tesla CEO Elon Musk Says He Got Into the Electric Car Business Because No One Else Would, May 29, 2013, http://techcrunch.com.
9. E. Musk, http://videos.huffingtonpost.com.
10. R. Kabacoff, Develop Strategic Thinkers Throughout Your Organization, Harvard Business Review, Feb. 7, 2014, https://hbr.org.
11. L. Huston and N. Sakkab, Connect and Develop: In-side Procter & Gambles New Model for Innovation, Harvard Business Review 84, no. 3 (March 2006): 58-66.
12. For scenario planning books, see K. van der Heijden, Scenarios: The Art of Strategic Conversation (New York: John Wiley, 1996); P.J.H. Schoemaker, Profiting From Uncertainty: Strategies for Succeeding No Matter What the Future Brings (New York: Free Press, 2002); and L. Fahey and R.M. Randall, eds., Learning from the Future: Competitive Foresight Scenarios (New York: John Wiley, 1998).
13. P.J.H. Schoemaker, Scenario Planning: A Tool for Strategic Thinking, Sloan Management Review 36, no. 2 (winter 1995): 25-40.
14. Testimony of Deven Sharma, president of Standard & Poors, before the Committee on Oversight and Government Reform, United States House of Representatives (Oct. 22, 2008), http://oversight-archive.waxman.house.gov.
15. P. Krugman, That Hissing Sound, The New York Times, Aug. 8, 2005.
16. J.E. Russo and P.J.H. Schoemaker, Winning Deci-sions: Getting It Right the First Time (New York: Doubleday, 2001).
17. C.J. Nemeth, B. Personnaz, M. Personnaz, and J.A. Goncalo, The Liberating Role of Conflict in Group Cre-ativity: A Study in Two Countries, European Journal of Social Psychology 34, no. 4 (July/August 2004): 365-374.
18. J. Lehrer, Imagine: How Creativity Works (Boston: Houghton Mifflin Harcourt, 2012).
19. R. King, Malcolm Gladwell: Disruptive Innovators Are Usually Disagreeable, CIO Journal, Sept. 17, 2013, http://blogs.wsj.com.
20. J. Dewey, Human Nature and Conduct: An Introduc-tion to Social Psychology (New York: Henry Holt, 1922).
21. D. Baer, 5 Brilliant Strategies Jeff Bezos Used to Build the Amazon Empire, March 17, 2014, www.businessinsider.com.
22. Quoted in P.F. Drucker, The Effective Executive(New York: Harper & Row, 1967), 148.
23. R. Charan. DuPonts Swift Response to the Financial Crisis, Bloomberg BusinessWeek, Jan. 7, 2009, www.businessweek.com.
24. A. Hill, Snapshot of a Humbled Giant, Financial Times, Apr. 2, 2012. Also see G. Gavetti, Kodak: Inter-view with Dr. George Fisher, Oct. 1, 2005, DVD (Boston: Harvard Business School Publishing, 2006), http://hbr.org.
25. E.R. Tufte, Visual and Statistical Thinking: Displays of Evidence for Making Decisions (Cheshire, Connecticut: Graphics Press, 1997).
26. See H.J. Einhorn and R.M. Hogarth, Decision Mak-ing Under Ambiguity, The Journal of Business 59, no. 4, pt. 2 (October 1986): S225-S250, or the original study by Daniel Ellsberg, which became a classic in the field: D. Ellsberg, Risk, Ambiguity, and the Savage Axioms, Quarterly Journal of Economics 75, no. 4 (November 1961): 643-669.
27. R. Mundle, Fatal Storm: The Inside Story of the Tragic Sydney-Hobart Race (Camden, Maine: Interna-tional Marine/McGraw-Hill, 1999).
28. P.J.H. Schoemaker and R.E. Gunther, The Wisdom of Deliberate Mistakes, Harvard Business Review 84, no. 6 (June 2006): 108-115.
29. D. Ogilvy, Confessions of an Advertising Man (London: Southbank Publishing, 2004). First published 1963 by Holiday House.
30. R.S. Root-Bernstein, How Scientists Really Think, Perspectives in Biology and Medicine 32, no. 4 (summer 1989): 472-488.
31. W.H. Hughes, Alexander Fleming and Penicillin (London: Priory, 1974).
32. R. Frank, Billionaire Sara Blakely Says Secret to Suc-cess Is Failure, October 16, 2013, www.cnbc.com.
33. S. Kirsner, Campbell Soup CEO Denise Morrison Talks Corporate Innovation in Boston, Boston Globe, May 8, 2013.
34. C. Martin, Wearing Your Failures On Your Sleeve, The New York Times, Nov. 8, 2014, www.nytimes.com.
35. M.E.P. Seligman, Authentic Happiness: Using the New Positive Psychology to Realize Your Potential for Lasting Fulfillment (New York: Free Press, 2002).
36. After-Action Review: http://en.wikipedia.org.
37. D. Senor and S. Singer, Start-Up Nation: The Story of Israels Economic Miracle (New York: Twelve, 2009).
38. The power of critical inquiry is highlighted in P. Thiel, Zero to One: Notes on Startups, or How to Build the Future (New York: Crown Publishing Group, 2014).
Reprint 56214. Copyright Massachusetts Institute of Technology, 2015. All rights reserved.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 9
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 49
Using Simulated Experience to Make Sense of Big DataAs data analyses get more complex, how can companies best communicate results to ensure that decision makers have a proper grasp of the datas implications?BY ROBIN M. HOGARTH AND EMRE SOYER
Many behavioral experiments have shown that when the same statistical in-formation is conveyed in different ways, people make very different decisions.
IN AN INCREASINGLY complex economic and social envi-ronment, access to vast amounts of data and information can help
organizations and governments make better policies, predictions
and decisions. Indeed, more and more decision makers rely on
statistical findings and data-based decision models when tackling
problems and forming strategies. Scientists, researchers, technol-
ogists and journalists have all been monitoring this tendency,
trying to understand when and how this approach is most useful
and effective.1
So far, discussions have centered mainly on analysis: data col-
lection, technological infrastructures and statistical methods. Yet
another vital issue receives far less scrutiny: how analytical results
are communicated to decision makers. As the amount of data gets
bigger and analyses grow more complex, how can analysts best
communicate results to ensure that decision makers have a proper
understanding of their implications?
Communicating Statistical InformationHowever well executed, the usefulness of an analysis depends on
how the results are understood by the intended audience. Con-
sider a patient visiting a doctor about an illness. Arguably, the
most important task is the diagnosis of the disease, as this can lead
to choosing an appropriate treatment. Yet even if the final deci-
sion lies with the patient, the chosen treatment may depend on
how the doctor communicates different options to the patient.
The same is true when an investor consults a financial expert or a
manager seeks the services of a consulting firm.
THE LEADING QUESTIONHow can com-panies best communicate analytical results to executives?
FINDINGSThere is often a large gap between conclusions reached by analysts and what decision makers understand.
Descriptions of complex statistical information can be misleading.
Interacting with a simulation model can help executives make better decisions.
D E C I S I O N M A K I N G : S I M U L AT I O N S
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 10
-
50 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : S I M U L AT I O N S
Data science, like medical diagnostics or scien-
tific research, lies in the hands of expert analysts
who must explain their findings to executive deci-
sion makers who are often less knowledgeable
about formal, statistical reasoning. Yet many
behavioral experiments have shown that when the
same statistical information is conveyed in differ-
ent ways, people make drastically different
decisions.2 Consequently, there is often a large gap
between conclusions reached by analysts and what
decision makers understand. Here, we address this
issue by first identifying strengths and weaknesses
of the two most common modes used for commu-
nicating results: description and illustration. We
then present a third method simulated experi-
ence that enables intuitive interpretation of
statistical information, thereby communicating
analytical results even to decision makers who are
nave about statistics.
Description Description is the default mode of presenting statistical information. This typically
involves a verbal statement or a written report,
which might feature one or more tables summariz-
ing the findings. The strength of this approach lies
in its speed in providing the decision maker with
the most essential and salient aspects of a given
analysis. But as problems get more complex, this
very strength turns into a major flaw. While
highlighting one issue, descriptions can end up
hiding details that have important decision-making
consequences. (See Describing an Investment
Problem.) The question then becomes: When it
comes to making decisions, are we able to differen-
tiate between good and bad descriptions?
Our own research suggests that descriptions can
mislead even the most knowledgeable decision
makers. In a recent experiment, we asked 257 econom-
ics scholars to make judgments and predictions
based on a simple regression analysis. (See About
the Research.) This is the most prevalent type of
analysis employed in applied economics in order to
identify and quantify a causal relationship between
two or more variables. To our surprise, most of
these experts had a hard time accurately decipher-
ing and acting on the results of the kind of analysis
they themselves frequently conduct. In particular,
we found that our description of the findings,
which mimicked the industry standard, led to an il-
lusion of predictability an erroneous belief that
the analyzed outcomes were more predictable than
they actually were.
The description obscured some sources of
uncertainty, and the decision makers became over-
confident about their prospects. Ultimately, we
managed to avert this illusion by substituting the
description with an illustration. This time, judg-
ments and decisions were accurate, suggesting that
the description of the results was indeed to blame
for the misperceptions.3
Illustration Illustrations in the form of a graph, figure, diagram or chart are also used regularly to
communicate statistical information. Unlike de-
scription, the primary objective is to give an
overview of the analysis and provide a bigger (al-
beit less precise) picture about the findings.
Consequently, decision makers are better able to
acknowledge the trends, effects and risks of their
prospective decisions.4 Using illustrations, it is
more difficult for crucial parts of the results to re-
main hidden. Hence, one benefit of visualizing data
is in making uncertainties more transparent. (See
Illustrating an Investment Problem, p. 52.) In
fact, a 2011 Science article that evaluates human
proficiency in visualizing data is aptly entitled
Visualizing Uncertainty About the Future.5
DESCRIBING AN INVESTMENT PROBLEMConsider a scenario where Y is a desirable variable such as wealth or health, and X is a valuable and scarce resource like money or time. You want to end up with more Y by making an investment in X. Analysis of past investments shows that, on average, increasing X by one unit leads to a one-unit increase in Y. Your current investment is X = 0, and you are considering increasing it to X = 5. With this amount of investment, what are the chances that you will actually end up with a negative Y? Alternatively, how many units of X would you need so that you can be 80% sure that you will end up with a positive Y?
The salient aspect of this description is the estimated 1-to-1 average effect. While highlighting this aspect, however, the description completely hides the uncertainties inherent in Y. In particular, the information provided gives no clues as to how random events, beyond the control of the decision maker, might affect the outcome. Such uncertainty could, in fact, result in someone with a large investment ending up worse off than someone with less or no investment at all. We could, for instance, agree that smoking is bad for health, but being a nonsmoker does not guarantee an individual a longer and healthier life relative to a heavy smoker.
Hence, armed only with the information about the average effect, it is impossible to answer accurately the particular questions posed in the problem or perceive the potential risks associated with the decision.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 11
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 51
Yet for all of its benefits, illustration is not always
an ideal way to communicate complexities. The
more variables, structural changes, connections and
patterns there are in the data, the harder it becomes
to condense them into one display. Moreover, like
description, illustration is typically static and not
interactive. Hans Rosling, a professor of interna-
tional health at Karolinska Institute in Solna,
Sweden, has attempted to counteract these short-
comings with the aid of visual technology.
Specifically, he creates innovative visualizations of
trends in global development using dynamic graphs
that show changes across time.6 In a similar fashion,
Betterment LLC, an investing service based in New
York, has recently created a tool that lets users visu-
alize the evolution of an investment of $100 in the
S&P 500 over a given holding period.7
Nonetheless, the shortcomings of description
and illustration remain a pervasive problem. Pro-
jecting a multifaceted, convoluted and complicated
process into static words, tables and graphs inexora-
bly means omitting crucial parts of the results.
Relevant information is inevitably lost in transla-
tion. Would it be possible, therefore, to develop a
dynamic alternative to these available approaches
one that would let decision makers more readily
grasp the complexities and uncertainties inherent in
the analyses?
In a series of studies, we designed and tested the
effectiveness of an alternate method of communi-
cation. Instead of describing or illustrating the
analysis, we let decision makers gain experience on
the outcomes of different possible actions by inter-
acting with simulations based on the same analysis.
Our findings showed that regardless of their level
of statistical sophistication, people relate well to
such an approach. Moreover, as analyses become
more complicated, decision makers tend to trust
experience more than their analytical intuitions.
Most importantly, their judgments and decisions
improve in the face of increasing uncertainty and
complexity. We call this approach simulated
experience.8
Simulated ExperienceFor tens of thousands of years, humans formed
judgments and made decisions exclusively through
experience. Formal statistical reasoning and tools
are comparatively recent innovations. In particular,
probability theory, which constitutes the founda-
tion of our current methods of analysis, was only
conceived in the 17th century.9 The problem we
face today is that our ability to communicate and
understand statistical outcomes has not advanced
as rapidly as our proficiency in handling data. In
fact, all nonhuman animals still depend solely on
experience to make choices and solve their prob-
lems. In which locations are sources of food
available and with what regularity? Where are
predators present? Which meteorological patterns
and trends exist in a particular environment? These
are vital issues for survival. It is therefore unsur-
prising that evolution has endowed both animals
and humans with remarkable capacities to encode
information about past occurrences.10 When it
comes to understanding and communicating sta-
tistical information, experience is a powerful yet
often underappreciated tool.
Simulated experience exploits our natural abil-
ity to transform complicated information into
actionable knowledge. Essentially, it lets the
ABOUT THE RESEARCHWe recently published a series of papers in experimental psychology on the effectiveness of simulated experience as a communication tool for statistical information. To research these papers, we conducted several experiments. For example, in one of the experiments, we asked 257 economics scholars to make judgments and predictions based on a simple regression analysis a type of analysis with which they are extremely familiar. Yet most of these experts had a hard time accurately deciphering and acting on the results. In fact, our descrip-tion of the findings, while mimicking the industry standard, obscured some sources of uncertainty. Ultimately, as detailed in this article, we managed to avert this illusion by substituting the description with an illustration.
In another series of studies, we designed and tested the effectiveness of an alternate method of communication. Instead of describing or illustrating the analysis, we let decision makers whom we grouped into different pools based on their levels of statistical sophistication gain experience about the outcomes of different possible actions by interacting with simulations based on the same analysis. Our findings showed that regardless of their level of statisti-cal sophistication, people relate well to such an approach. Specifically, their judgments and decisions improve in the face of increasing uncertainty and complexity. Other insights we reached include: Descriptions are easy to construct, but tend to hide uncertainties by focusing attention on average effects. Illustrations make uncertainties more visible. However, they do not cope well with complex analyses involving multiple variables. As the uncertainties and complexities of decision situations increase, people tend to trust their experiences more than their analytic abilities. Regardless of their level of statistical knowledge, simulated experience helps decision makers form an accurate understanding about possible outcomes of the underlying statistical analysis.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 12
-
52 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : S I M U L AT I O N S
decision maker live through the problem many
times with the aid of a simulation. (See Under-
standing an Investment Problem Through
Simulated Experience.) The implementation of
simulated experience involves four components:
Analysis Conduct an analysis using available data, from which an analyst might normally craft a de-
scription or illustration.
Simulation Instead of creating some type of descrip-tion or illustration, the analyst constructs a simulation
model based on the findings of the analysis.
Interaction Executive decision makers interact with the simulation model and can input potential
actions and observe the subsequent outcomes.
Experience Decision makers can experiment with changing their inputs. That way, they can experience
as many outcomes as they wish, given those inputs.
Executives gain experiential evidence about potential
consequences of their actions, based on the statistical
analysis.
Research attests to the effectiveness of this ap-
proach when the problem is both hard to handle
and rife with uncertainties. For example, one article
in the Journal of Consumer Research found that in-
dividuals create better retirement plans when they
interactively observe their potential future bene-
fits.11 Researchers at the University of Zurich have
found that simulated experience helps investors
perceive accurately the underlying risk-return pro-
file of their investments.12 A recent Management
Science article suggests that banks could employ
such a communication method to help their clients
accurately perceive the risks associated with differ-
ent investment products.13 John Sterman, the Jay
W. Forrester Professor of Management at the MIT
Sloan School of Management and director of MITs
System Dynamics Group, argues that climate
change debates involve crucial misperceptions
and he effectively removes them by simulating
viable scenarios.14
In our own research, we find that people have
difficulty assessing their chances of success in compe-
titions and market-entry decisions, but simulating
such situations leads to improved assessments and
decisions.15 Moreover, simulated experience has de-
monstrably helped to correct judgmental biases, such
as neglecting base rates in probabilistic statements.16
Accordingly, initiatives such as Probability Manage-
ment Inc. a nonprofit organization that aims at
improving communication of uncertainty through
open-source decision-support tools seek to put
such evidence to use, applying simulation-based
communication to improve actual managerial
decisions and public policies.17 (See Designing Sim-
ulations, p. 54.)
There are several software packages that provide
the necessary tools to create a wide range of
ILLUSTRATING AN INVESTMENT PROBLEMConsider the investment problem described in the previous exhibit. Here is the illustration of the X-Y relationship based on 250 individual observations, on which the analysis is conducted:
Seeing a graph of the past investments instead of a description that summarizes them helps a decision maker acknowledge the uncertainties inherent in the out-comes. The description featured previously only mentions the fitted line and disregards completely the cloud of data that surrounds it. Now it becomes clear that despite a 1-to-1 average effect, someone with a positive investment might end up with a negative outcome (see the data points below the horizontal line). Moreover, a larger investment would not always guarantee a larger return than that of someone who made a smaller investment.
Illustrating the relationship between two variables is easy. What if, however, there were multiple investment options instead of just the one? This is almost always the case in real-life analyses and situations. Illustrating many interrelated variables in one figure is unwieldy, if not unfeasible. In the face of such complexities, illustrations are bound to be less meaningful.
X
Y
0 5 10 15 20 25 30 35
-10
0
10
20
30
40
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 13
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 53
simulations for diverse decision-making situations.
Oracle Crystal Ball, Frontline Systems Risk Solver
Platform and Luminas Analytica are among the
most prominent. With a little bit of knowledge in
programming, one could also create simulations
from scratch using platforms such as MATLAB or
C++. Of course, Microsofts Excel is always handy if
the decision problem is relatively simple.
Simulated experiences are not without their
weaknesses and blind spots. For instance, if decision
makers simulate only a small set of experiences, the
resulting intuitions might be driven by sampling
variability. The remedy is to generate a large num-
ber of observations, given the same input, before
forming an idea about the uncertainties. Finally, like
all other communication methods, simulated expe-
rience is useful as long as the underlying analysis is
unbiased and accurate. If the analysis has blind
spots, so will the descriptions, illustrations and sim-
ulated experiences that stem from it.
Precisely Wrong Versus Approximately CorrectAdvances in computing technology allow us to
build simulations for virtually any scenario. How-
ever, this does not mean that we should employ this
method for all problems. For simple probabilistic
situations, a description or illustration may be a
wiser choice. But as complexity grows and uncer-
tainties arise, simulations can help managers better
understand statistical information and thus enable
them to make better decisions, regardless of their
levels of statistical expertise. We do not call for the
abandonment of descriptions or illustrations. In-
stead, we argue that these should sometimes be
augmented with add-on simulations.
Why is it, then, that we rarely encounter simula-
tions for critical decisions about medical treatments,
investment options, pension plans, insurance pro-
grams and so on? After all, simulation technology is
not new. In fact, it has always been essential to any en-
gineering process. The issue is that simulations are
still primarily seen as sophisticated tools for statistical
analysis as opposed to a means for communicat-
ing results.
There are two main reasons for this. The first is
technical: There is a cost to building simulations.
So it is easier for analysts to only craft descriptions
or build illustrations using the features in statistics
software. Second, and more importantly, simula-
tions are vague. An interface that lets you provide
your inputs, simulate the analysis, and sequentially
observe the related outcomes does not steer you
toward a precise answer. You have to make up your
own mind as you experience the simulated out-
comes. We typically do not appreciate such a fuzzy
approach, especially if the decision is important.
We seek perfect solutions exact maneuvers that
will lead to desired outcomes. As managers, politi-
cians and individual decision makers, we prefer
learning correct answers right away.
However, the very presence of uncertainty suggests
its wise to refrain from seeking fast solutions. No deci-
sion has a completely foreseeable set of outcomes. In
both business and life, chance has its say. Leaning too
heavily on likelihoods will inevitably lead to the belief
that we can predict outcomes more accurately than
we can. Such misperceptions can lead to precise but
wrong answers to important questions.
UNDERSTANDING AN INVESTMENT PROBLEM THROUGH SIMULATED EXPERIENCEThis time, consider a slightly more complex investment problem, where there are three possible investment options (X1, X2 and X3) affecting the outcome Y, instead of just one. The interface below lets decision makers enter their choices of X1, X2 and X3. When a user clicks the simulate button, the model simulates a corresponding Y based on the analysis conducted on the available data. Users are free to enter as many input sets and simulate as many outputs for each as they wish.
The simulation presented here records the previous entries and outcomes. This allows users to select a subsample of outcomes (the highlighted part of the simulated Y column) and obtain the average of that selected subsample. (The reset button clears all previously simulated data.) Hence, users not only gain insights by making decisions and experiencing the consequences, but they also can gather information about the average effects of their strategies. One could also construct a histogram based on the selected subsample an illus-tration of the frequencies of simulated outputs to visually display how potential outcomes are distributed.
Choice of X1
Simulate
Reset
X1 X2 X3 Simulated Y Selected YCount Average
101010
555555
333555555
000555555
5
Choice of X2
5
Choice of X3
5
0.5-0.33.24.7
10.84.24.4
-1.07.5
6 5.1
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 14
-
54 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : S I M U L AT I O N S
Simulated experience aims to counter this ten-
dency by letting decision makers feel the most likely
answers. In the words of a famous American math-
ematician, the late John W. Tukey, Far better an
approximate answer to the right question, which is
often vague, than an exact answer to the wrong
question, which can always be made precise.
Robin M. Hogarth is an emeritus professor in the department of economics and business at Universi-tat Pompeu Fabra in Barcelona, Spain. Emre Soyeris an assistant professor of judgment and decision making on the business faculty at zyegin Univer-sity in Istanbul, Turkey. Comment on this article at http://sloanreview.mit.edu/56215, or contact the authors at [email protected].
REFERENCES
1. P. Simon, Too Big to Ignore: The Business Case for Big Data (Hoboken, New Jersey: John Wiley & Sons, 2013) offers an overview of business applications of data science. See also R. Fildes and P. Goodwin, Against Your Better Judgment? How Organizations Can Improve Their Use of Management Judgment in Forecasting, Interfaces 37, no. 6 (November-December 2007): 570-576.
2. A. Tversky and D. Kahneman, The Framing of Deci-sions and the Psychology of Choice, Science 211, no. 4481 (January 30, 1981): 453-458; A. Tversky, P. Slovic and D. Kahneman, The Causes of Preference Rever-sal, American Economic Review 80, no. 1 (March 1990): 204-217; C.K. Hsee, G.F. Loewenstein, S. Blount and M.H. Bazerman, Preference Reversals Between Joint and Separate Evaluations of Options: A Review and Theoretical Analysis, Psychological Bulletin 125, no. 5 (September 1999): 576-590; A. Tversky and R.H. Thaler, Anomalies: Preference Reversals, Journal of Economic Perspectives 4, no. 2 (spring 1990): 201-211; and for a specific case study, see J. Koehler, Psychol-ogy of Numbers in the Courtroom: How to Make DNA-Match Statistics Seem Impressive or Insufficient, Southern California Law Review 74 (2001): 1275-1306.
3. E. Soyer and R.M. Hogarth, The Illusion of Predict-ability: How Regression Statistics Mislead Experts, International Journal of Forecasting 28, no. 3 (July-September 2012): 695-711.
4. R.M. Hogarth and E. Soyer, A Pictures Worth a Thousand Numbers, Harvard Business Review 91, no. 6 (June 2013): 26.
5. D. Spiegelhalter, M. Pearson and I. Short, Visualizing Uncertainty About the Future, Science 333, no. 6048 (September 9, 2011): 1393-1400.
6. H. Rosling, The Best Stats Youve Ever Seen, TED talk filmed February 2006, www.ted.com.
7. An interface shows the relationship between time and returns based on daily data. See: D. Egan, Its About Time in the Market, Not Market Timing. October 14, 2014, www.betterment.com.
8. R.M. Hogarth and E. Soyer, Sequentially Simulated Outcomes: Kind Experience Versus Nontransparent Description, Journal of Experimental Psychology: General 140, no. 3 (August 2011): 434-463; R.M. Hogarth and E. Soyer, Providing Information for Decision Making: Contrasting Description and Simula-tion, Journal of Applied Research in Memory and Cognition, in press, published online January 29, 2014; R.M. Hogarth and E. Soyer, Communicating Forecasts: The Simplicity of Simulated Experience, Journal of Business Research, in press.
9. G. Shafer, The Early Development of Mathematical Probability, in Companion Encyclopedia of the History and Philosophy of the Mathematical Sciences, Volume 2 ed. I. Grattan-Guinness (London and New York: Rout-ledge, 1993): 1293-1302.
10. L. Hasher and R.T. Zacks, Automatic and Effortful Processes in Memory, Journal of Experimental Psy-chology: General 108, no. 3 (September 1979): 356-388; L. Hasher and R.T. Zacks, Automatic Processing of Fundamental Information: The Case of Frequency of Occurrence, American Psychologist 39, no. 12 (De-cember 1984): 1372-1388; P. Sedlmeier and T. Betsch, Etc. Frequency Processing and Cognition (New York: Oxford University Press, 2002).
11. D.G. Goldstein, E.J. Johnson and W.F. Sharpe, Choosing Outcomes Versus Choosing Products: Consumer- Focused Retirement Investment Advice, Journal of Consumer Research 35, no. 3 (October 2008): 440-456.
12. M.A. Bradbury, T. Hens and S. Zeisberger, Improv-ing Investment Decisions With Simulated Experience, Review of Finance, published online June 6, 2014.
13. C. Kaufmann, M. Weber and E. Haisley, The Role of Experience Sampling and Graphical Displays on Ones Investment Risk Appetite, Management Science 59, no.2 (February 2013): 323-340.
14. J.D. Sterman, Communicating Climate Change Risks in a Skeptical World, Climatic Change 108, no. 4 (October 2011): 811-826.
15. R.M. Hogarth, K. Mukherjee and E. Soyer, Assess-ing the Chances of Success: Nave Statistics Versus Kind Experience, Journal of Experimental Psychology: Learning, Memory, and Cognition 39, no. 1 (January 2013): 14-32.
16. B.K. Hayes, B.R. Newell and G.E. Hawkins. Causal Model and Sampling Approaches to Reducing Base Rate Neglect, in Proceedings of the 35th Annual Conference of the Cognitive Science Society, eds. M. Knauff, M. Pauen, N. Sebanz and I. Wachsmuth (Austin, Texas: Cognitive Science Society, 2013.)
17. Probability Management is an organization that aims to improve communication of uncertainty through open-source decision support tools. More information can be found at www.probabilitymanagement.org.
Reprint 56215. Copyright Massachusetts Institute of Technology, 2015. All rights reserved.
DESIGNING SIMULATIONSHere are a few recommen-dations on how to design and use simulated experience:
The interface should be user-friendly. Interaction should not be costly.
Decision makers should be informed that simu-lated experience is a communication tool; it helps people understand the results of an analysis, but it does not prove the reliability of the analysis.
Decision makers should be advised about how the simulation works and how it calculates the outcomes given their inputs.
Decision makers should be given time to interact with the simulation and make up their minds at their own pace.
The simulation should incorporate uncertainties. For example, it should allow decision makers to experience different results given the same inputs when there is randomness in the underlying process.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 15
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 55
Why You Decide the Way You DoFor executives, making good decisions is essential. New research offers insights into factors that can affect the decision-making process.BY BRUCE POSNER
HOW DO PEOPLE process different inputs and make complicated decisions? Variations on this question have engaged researchers for many years, with broad implications for a variety of individu-
als. But the topic is of particular interest to business executives, who must frequently make decisions.
Researchers have long sought to shed light on the inner workings of the human brain and the way
people make decisions. In recent years, curiosity about the decision-making process has heated up,
attracting academics from fields as diverse as neuroscience, management, behavioral economics and
psychology. Here are highlights of a handful
of recent scholarly articles that offer intrigu-
ing insights into decision making from
several disciplines.
1. The Advantage of Psychological DistanceInformation overload is a fact of modern
life, making many common decisions
(such as choosing a cellphone plan) un-
bearably confusing. Although choice offers
options to consumers, too many choices or
too many features per choice can cause
people to delay decisions or make less-
than-optimal choices. Recent research into
how individuals process information of-
fers some promising suggestions for
dealing with information overload. The
key may involve psychological distancing
removing oneself from the morass of
details surrounding a decision and consid-
ering the choices on a more abstract level.
As authors Jun Fukukura, Melissa
J. Ferguson and Kentaro Fujita explain
in their article in the Journal of Experimen-
tal Psychology: General, such distancing
THE LEADING QUESTIONWhat strategies can improve decision making?
FINDINGSIf you face informa-tion overload, psychological distancing can be helpful.
A willingness to ask for advice on diffi-cult problems can increase your per-ceived competence.
For minor, day-to-day choices, you may not need to de-liberate carefully to achieve a satisfac-tory choice.
D E C I S I O N M A K I N G : N E W R E S E A R C H
Too many choices or too many features per choice can cause people to delay decisions or make less-than-optimal choices.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 16
-
56 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : N E W R E S E A R C H
(which can be either temporal or physical) can help
people to filter out the less-vital details and enable
them to focus on the gist of the matter. The authors
tested several aspects of how psychological distance
influences decision
making. In one study,
they asked some par-
ticipants, who were
students from Cor-
nell University in
Ithaca, New York, to
write about a car they
would buy next year,
and others to write
about a car they
would buy tomor-
row. (A control group
was not given a writing task.) Participants were then
given information to read about 48 individual fea-
tures (such as mileage, handling, year and trunk
capacity) of four different cars twelve features per
car and had only seven seconds to absorb each
piece of information before the next piece appeared
on a computer screen. Participants were then asked
to choose the car they thought was best. Those who
has written about the future before receiving infor-
mation chose the best car (the one whose features
were considered most important to people in an ear-
lier pilot) significantly more often than participants
who had written about a near-term purchase (69%
vs. 40%) or those in the control group (39%).
In another test of psychological distancing, the
researchers randomly assigned one group of indi-
viduals to write for three minutes about the
previous day and another group to write for three
minutes about a day about a year earlier. Then they
presented participants with sets of information
about the features of the four different cars; a com-
puter screen displayed information about the
features of one car at a time, and the participants
learned about the cars at their own pace. When the
participants were done reading, they were asked to
select the car they would buy and to characterize
the memory strategy they had used. Those who had
written about the past selected the best car at a
much higher rate than those who had written about
recent occurrences (59% versus 34%) or members
of the control group (29%), who had not done a
writing task. Whats more, those participants who
had written about the past reported relying on gist
memory in other words, memory about the gist
of a matter significantly more often than the
others. The researchers found that mind-sets
involving psychological distance enabled partici-
pants to organize related product features better.
To be sure, psychological distancing isnt appro-
priate for every situation. In instances where people
are expected to recall and piece together specific de-
tails (for example, jury trials or investigations), it may
be harmful. But in many circumstances involving in-
formation overload, it can result in better decisions.
2. Balancing Exploration and ExploitationScholars have argued that companies can develop
greater ambidexterity as they search for better ways to
balance practices supporting optimal exploitation
of existing opportunities and those promoting
exploration of new ones. Although much of the re-
search on corporate
ambidexterity has
been focused on how
companies can best
achieve ambidexter-
ity, less attention has
been paid to how the
cognitive processes of
individual managers
can shape perfor-
mance on a broader
level. New research by
Daniella Laureiro-
Martnez, Stefano
Brusoni, Nicola Can-
essa and Maurizio
Zollo shifts the discussion. In an article published in
Strategic Management Journal, the authors describe
how different regions of the brain control different
cognitive activities.
Exploitation, the authors explain, is behavior that
optimizes performance in current tasks, and explo-
ration is behavior leading to disengagement from
current tasks to search for alternatives. Exploitative
decisions take place in areas of the brain associated
with reward seeking and involve learning by doing.
Exploration choices, by contrast, activate the brains
REFERENCEJ. Fukukura, M.J. Ferguson and K. Fujita, Psycho-logical Distance Can Improve Decision Making Under Information Overload via Gist Memory, Journal of Experimental Psychology: General 142, no. 3 (August 2013): 658-665.
REFERENCED. Laureiro- Martnez, S. Brusoni, N. Canessa, and M. Zollo, Understanding the Exploration- Exploitation Dilemma: An MRI Study of Attention Control and Decision-Making Performance, Strategic Manage-ment Journal, in press, published electronically February 28, 2014.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 17
-
SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 57
attention control and executive functioning regions,
which are tasked with managing new situations.
The researchers studied the decision-making be-
haviors of 63 people who had at least four years of
experience making managerial decisions. Partici-
pants were asked to sit at computers and play a game,
the purpose of which was to accumulate points that
could be traded for cash. Following a brief warm-up,
they played the game while lying inside a functional
MRI scanner that took images of their brains. The
game featured four slot machines that awarded
points according to rules that changed from trial to
trial; each participant played a total of 300 trials.
However, the changing rules were never spelled out;
participants were expected to learn about them
through experimentation. Participants could choose
to pursue an option they were familiar with (exploi-
tation) or explore a new one (exploration).
The researchers compared the choices of study
participants (the number of exploration and exploi-
tation choices, and the number of times they
switched between the two) and their decision-
making performance. The authors found significant
links between greater activation of regions of the
brain associated with attention control and better
performance in the game, which supported their
hypothesis that increased attentional control is
linked to better decision-making performance. In
this study, participants who did less exploration gen-
erally performed better, but, more broadly, the
authors concluded that superior decision-making
performance relies on the ability to sequence exploi-
tation and exploration appropriately and to
recognize when to switch to exploration.
3. How to Tee Up ChoicesWhen does it make sense to let people make active
choices on their own, and when is it preferable to
design default rules that nudge people in a certain
direction (for example, to become an organ donor
or to use energy generated by wind)? In modern
societies, individuals face a barrage of complicated
choices: how to set up retirement accounts; how
much to save; whether to waive collision coverage
on rental car agreements, and so on. Decisions take
time and attention, and people are busy. Default
rules determine what happens if people choose to
do nothing.
Depending on what you are trying to achieve,
changing default rules can be a particularly power-
ful tool that institutions have, argues Harvard Law
School professor Cass R. Sunstein perhaps
more effective than significant economic incen-
tives. Writing in the University of Pennsylvania Law
Review, Sunstein ex-
amines the rationale
for default rules and
why and when orga-
nizations would use
blanket rules instead
of allowing individ-
uals to make their
own choices or establishing personalized rules
based on a persons individual profile (for example,
using demographic data). Default rules, he ex-
plains, dont impose mandates or bans. Rather, they
steer people in a particular direction (while offer-
ing opportunities to opt out), producing outcomes
that institutions want at costs that are lower than
economic incentives. By contrast, requiring indi-
viduals to make their own choices can impose high
costs in terms of the time it takes to learn about the
options. The job of choice architects, according to
Sunstein, is to understand decision costs (including
how confusing the decision is and how heteroge-
neous the pool of decision makers is) and the costs
of errors (what happens when people decide in a
way thats detrimental to them or to other members
of a group).
In Sunsteins view, the most desirable default
rules are informed chooser defaults, which align
with what most well-informed people would
The job of choice architects is to understand decision costs (including how confusing the decision is) and the costs of errors (what happens when people decide in a way thats detrimental to them or to other members of a group).
REFERENCEC.R. Sunstein, Deciding by De-fault, University of Pennsylvania Law Review 162, no. 1 (December 2013): 1-57.
SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 18
-
58 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU
D E C I S I O N M A K I N G : N E W R E S E A R C H
choose. Such defaults appeal to those interested in
efficiency, welfare, autonomy or fairness. (On the
other end of the spectrum are default rules that are
either badly designed or intentionally misleading;
with so-called negative option marketing, for
example, companies offer people free products,
then enroll them in programs with a monthly fee
unless they make the effort to opt out.) Even when
its possible to develop default rules that are geared
to individuals personal needs and tastes (as in
algorithms that use your past choices in books or
music to make recommendations), Sunstein
argues that there may be an argument for preserv-
ing a system based on active choosing. Why? In
some areas, he believes, active choosing promotes
learning in ways the defaults do not, which may
generate long-term benefits.
4. Going With the FlowWhen you have a decision to make, you may
assume that you should focus rationally on the
choices and select the best one. Legal and economic
decision-making theory generally argues for care-
fully considering each option and then picking the
one that delivers the highest expected value. The
advantage of this approach is that the decision will
reflect your intentions, and you will be less likely to
have post-decision
remorse or so the
theory goes.
But new research
suggest that people
who make decisions
more spontaneously
by allowing their
thoughts to wander
until they arrive at a
choice that they feel drawn to can be as satisfied
with their decisions as those who choose more
deliberately. Writing in Frontiers in Psychology, re-
searchers Colleen E. Giblin, Carey K. Morewedge
and Michael I. Norton describe research they con-
ducted that included co