international institute for analytics at the chief analytics officer forum, europe

LEADING ORGANIZATIONS THROUGH THE NEW RISKS OF ANALYTICS

Dan MagestroVP-Research Director, IIA

2

Tom DavenportCO-FOUNDER

Author of Competing on

Analytics,Keeping up with the

Quants

Jack PhillipsCEO

Editor of Enterprise

Analytics

Dan MagestroRESEARCH DIRECTOR

Led analytics teams in insurance,

banking, & healthcare

INTERNATIONAL INSTITUTE FOR ANALYTICSABOUT US

• IIA helps build and sustain 200+ enterprise analytics programs by creating a competitive advantage

• IIA’s flexible approach to supporting organizations ensures time- and cost-efficient insights and solutions

MATURITYASSESSMENT

CONSULTINGSERVICES

PEER--EXPERTNETWORK

Confidential – Copyright © 2016 IIA All Rights Reserved

A MARKETING EXAMPLEMEASURING AD EFFECTIVENESS

• Two print ads were placed in 1 of 2 national magazines with similar circulation

• Each ad ran for three consecutive months, exclusively in the same magazine

• Detailed point-of-sale datafrom all merchants is provided to manufacturer

• Ad effectiveness might be measured through regression analysis of magazine ad spend and sales

A B

Confidential – Copyright © 2016 IIA All Rights Reserved 3

“Campaign A drove $63.50 of incremental monthly sales, and Bdrove $58.20, per $1,000 spent.”

“Campaign A seems to perform directionally better than B, but it depends on assumptions.”

A MARKETING EXAMPLEWHICH STATEMENT IS MORE ACTIONABLE? ACCURATE?


5

ANALYTICS AND RISKSHIFTING THE FOCUS


Measure, quantify, and predict riskusing analytical methods

Analytics inRisk Management

6

Risk Managementin Analytics

vs.

Common theme Improve decision making

Measure, quantify, and predict riskusing analytical methods

Assess and mitigate risks in analytically driven decisions

Analytics inRisk Management

ANALYTICS AND RISKSHIFTING THE FOCUS


7

IMPROVING DECISIONS WITH DATA AND ANALYTICSIMPLIED ASSUMPTIONS

More data sources bring more, better

information to a business decision

More information leads to more

knowledge about factors affecting

the decision

More knowledge means reduced uncertainty (i.e. reduced risk) in

decision

More advanced algorithms, faster

platforms, and smarter analysts improve insights

Improved insights means deeper

understanding of factors affecting

the decision

Deeper understanding

reduces uncertainty and risk in decision


8

HOWEVER, NEW RISKS ARISEKEY SOURCES OF RISK IN ANALYTICS


• Scarcity of skilled talent• Analytics embedded in analysis tools

• Lack of accountability for results• Missing business context or knowledge• Lack of sufficient rigor or scrutiny

Errors and biases in analysis results and recommendations

9

HOWEVER, NEW RISKS ARISEKEY SOURCES OF RISK IN ANALYTICS

• Scarcity of skilled talent• Analytics embedded in analysis tools

• Lack of accountability for results• Missing business context or knowledge• Lack of sufficient rigor or scrutiny

Analytics shifts some risk from the gut to the analysis.

So what does this mean for analytics leaders?


Errors and biases in analysis results and recommendations

10

THE CAO AS RISK MANAGERSHIFTING RESPONSIBILITIES FOR RISK

DriverDecentralized

analytics teams Chief Analytics OfficerEnterprise risk managementfunction (ERM)

Analytics often lacking representation at risk leadership level

Represents analytics activities and needs on risk committee

Ownership of analytics risk

Risk ownership spread across teams and individuals; weakor accountability or junior

Consolidated under CAO; Potential accountability for results and reco’s

Modelgovernance

Either nonexistent or ad hoc, depending on ERM

Well-positioned to establish model governance

Data governance

Often driven by IT or data leader; difficult to capture requirements

Brings enterprise perspective on business needs for data


11

TYPES OF ERRORS IN ANALYTICS

• Incorrect assumptions, failing to test

• Lack of alternative scenarios

• Interpreting data incorrectly

• Not asking the right questions

Logic Errors

• Careless mistakes in analysis

• Selectively considering alternatives

• Incorrect or insufficient decision-making criteria

• Postponing decisions in quest for perfect analytics

Process Errors

Business Errors

• Misaligned incentives: Using analytics to justify action, or rigging the result

• Belief that analytics always makes decisions better


SCARCITY OF ANALYTICAL TALENT

12

McKinsey & Co., Big data: The next frontier for innovation, competition, and productivity. June, 2011


13

HOW CAN ANALYTICS LEADERS MITIGATE THE RISKS IN ANALYTICS?


14

Statistical Analysis

Visualization/Storytelling

DataSavvy

Programming Business Acumen

Critical Thinking

Creativity Art


THE ANALYTICS SKILL SET

1. RECRUIT, RETAIN, AND POSITION THE RIGHT PEOPLEEVOLVING ROLES OF ANALYTICAL TALENT

2016: Analytics professional(front office)

2006: Statistician/Modeler/Quant(back office)



DataSavvy

Programming

Business Acumen

Critical Thinking


16

1. RECRUIT/RETAIN THE RIGHT PEOPLEBROAD SKILLS WILL MITIGATE RISK

• Incorrect assumptions, failing to test

• Lack of alternative scenarios

• Interpreting data incorrectly

• Not asking the right questions

Logic Errors

• Careless mistakes in analysis

• Selectively considering alternatives

• Incorrect or insufficient decision-making criteria

• Postponing decisions in quest for perfect analytics

Process Errors

Business Errors

• Misaligned incentives: Using analytics to justify action, or rigging the result

• Belief that analytics always makes decisions better


Viz/Story-telling

DataSavvy

Program-ming

Business Acumen

Critical Thinking

Art/Creativity

2. DRIVE ANALYTICAL SKILLS OF BUSINESS LEADERS

2010: Business leader 2020: Business leader




DataSavvy

Business Acumen

Critical Thinking

18

3. CAO: INSTILL COMFORT WITH UNCERTAINTY ACROSS THE ORGANIZATION


“Campaign A drove $63.50 of incremental monthly sales, and Bdrove $58.20, per $1,000 spent.”

“Campaign A seems to perform directionally better than B, but it depends on assumptions.”

19

OTHER WAYS TO MITIGATE RISKSACTIONS FOR A CAO

• Facilitate and prioritize collaboration among analytics & data science teams

• Establish/influence data and model governance programs

• Engage Chief Risk Officer (or enterprise risk leaders)

• Emphasize critical thinking, problem solving, and skepticism

• Seek broad participation in progress- and readout sessions for analytics projects

• Align incentives and establish accountability among business and analytics teams


WHAT’S NEEDED TO MEASURE AD EFFECTIVENESS

Correlated response Sufficient activity Sufficient time

Effective activity Good dataEnough isolation

The activity needs to drive sales in a time-correlated way (i.e. not just drive awareness)

Sales

Marketingactivity

Marketingactivity

The activity needs to have sufficient spend to find the “signal” among sales drivers

Prolonged activity allows better mapping of s-curve and increased detection

The campaign must be effective at driving sales in the first place

Activities with similar execution might be difficult to separate

Appropriate and complete data is critical for measurementBroadcast Week DMA Net Spend TRPs

1/4/2010 214 121,371 7.01/11/2010 214 122,799 7.01/18/2010 214 111,264 7.01/25/2010 214 111,908 3.02/1/2010 214 53,924 2.02/8/2010 214 40,951 3.0

2/15/2010 214 48,793 3.02/22/2010 214 48,077 3.03/1/2010 214 50,598 2.03/8/2010 214 44,138 2.0

3/15/2010 214 39,914 2.0



NEXT STEPSLEARN MORE ABOUT HOW IIA CAN HELP

v

ANALYTICS MATURITY ASSESSMENT

Independent measurement for analytics maturity

v

CONSULTINGSERVICES

Custom solutions for analytics crossroads

Engagements

RESEARCH & ADVISORY NETWORK

Written Research

Peer Community

Subscription-based service for insights and expertise

ü Peer-to-Peer Exchangesü Community Eventsü Phone Discussions

ü DELTA scores across 33 competencies

ü Identification of gaps and opportunities

Progress Monitoring

ü Multiple reports and data updates to track progress

ü Latest trends & insightsü Extensive library of

foundational researchü Actionable Frameworks

Capability Evaluation

ü In depth team interviews, analysis and recommendations by IIA experts

Consultative engagements to navigate analytics roadblocks leveraging IIA’s unparalleled community of analytics experts and proven methodologies.

ü Solutions• Opportunity Portfolio Creation• ROI Prioritization • Change Management • Practitioner Workshops

ü Areas• Maturity, Organization,

Governance, Technology Selection, Architecture, Culture, Training

Baseline Report

Retained Expertise

ü Inquiry Serviceü Industry thought leadersü Personalized Advice

international institute for analytics at the chief analytics officer forum, europe

Data & Analytics