Risk, Options and Cost of Delay

Troy MagennisLKNA 2014 San Francisco. May 2014

risk events

1 2 3

Performance AND Vendor Delay

Performance OR Vendor Delay

Nothing Goes Wrong

Time

Prob

abili

ty

Definition: Risk

The impact of uncertainty on an outcome

Technical Risk

Financial Risk

Market Risk

• Real Options• Right Staff / liquidity• Dev Practices• Dependencies• Constraints

• Lean Startup• Agile Processes• Competitive

Awareness

• Having funding/cash

• Having a strategy

• Economic prioritization

• Real Options

“Aleatory Risk”Cannot be reduce by more info

Delay(Technical Risk)

Low Adoption(Market Risk)

Low Cashflow(Financial Risk)

Less Resources(Financial Risk)

RiskPositive

FeedbackLoop

Key Point

Occurrence of a risk Increases exposure to other

risks

Break the chain early

AKA: Early and meaningful contact with enemy – RISK

(source: quote from Reinertsen, but sources from US marines?)

Correlation != Causation

We can see average flight delay matches the shape of “Late Aircraft,” but don’t yet know why…

Key Point

Serialized dependencies cascade

delays, but are not the root cause –

Why was the aircraft late?

The later you are, the later you get.

Four people arrange a restaurant booking after work

Q. What is the chance they arrive on-time to be seated?

Commercial in confidence

Person 1 Person 2 Person 3 Person 41

in 1

6 EV

ERYO

NE

is O

N-T

IME

15 T

IMES

mor

e lik

ely

at le

ast o

n pe

rson

is la

te

1

2

3

4

5

6

7

Team Dependency Diagram

1 in 2n

or

1 in 27

or

1 in 128

7 dependencies1 chance in 128

6 dependencies1 chance in 64

5 dependencies1 chance in 32

Key Point

Risk of being impacted decreases by half for every risk vector/factor removed

But, not all risks have the same likelihood (or impact)…

Frequency

Recency

Impact

If you haven’t seen an event after testing for it n times, you can be 95% sure that its probability of

happening is less than

3/nReferences: Wikipedia: Statistical Rule of Three and Thanks to John Cook: Estimating the chances of something that hasn’t happened yet, http://www.johndcook.com/blog/2010/03/30/statistical-rule-of-three/

The Math: (1-p)n = 0.05 for p. Taking logs of both sides, n ln (1-p) = ln(0.05) ≈ -3. Since log(1-p) is approximately -p for small values of p, we have p ≈ 3/n.

Statistical Rule of Three

• Example: Proofreading a book, you find no grammatical errors in n pages

• Error decreases as a proportion to the number of independent test cases examined

• It hard to be independent!

n percentage

20 15% (3/20)

100 3% (3/100)

200 1.5% (3/200)

500 0.6% (3/500)

1000 0.3% (3/1000)

1 25 49 73 97 1211451691932172412652893133373613854094334574810.00000

0.10000

0.20000

0.30000

0.40000

0.50000

0.60000

0.70000

0.80000

p

‘s Absence of Evidence isn’t Evidence of Absence

But, it does demonstrate the occurrence is rare with growing certainty

Depends on consequence….

Ps. The most common Black Swan is project on-time delivery!

CONSEQUENCEMATTERS

Capture Actual

Impacts

Calculate “Impact”

Order from

highest to lowest

Discuss, Root

cause Top 10

Prioritize

Sum of Days impacted for 3

last monthsSum of Days

impacted for 3 last months

CategoryStartEnd

“Value” Cost of Delay

Product 1

Product 2

Product 3

Complete Order?

3

2

1

“Time” Remaining Time/Effort to solve

Economic Prioritization – same time, different value

Product 1

Product 2

Product 3

1

2

3

Economic Prioritization – same value different time

“Value” Cost of Delay

Complete Order?

“Time” Remaining Time/Effort to solve

W.S.R.F. =Prioritization Heuristic to optimize reward“Do Highest First”

Impact of risk

Time to resolve/mitigate

Weighted Shortest Risk First

Sum of delay time of same risk causes over the last 3 (?) months

Effort estimate of the resolution time of risk root cause

All Sheep in Scotland Are Black• A psychologist, a biologist, a mathematician, and a physicist were riding a

train through the Scottish countryside. Looking out the window, they all noticed a lone black sheep on a hill.

• The psychologist intoned, “Well, what do you know. I didn’t realize the sheep in Scotland were black.”

• The biologist corrected him, saying, “You don’t know that all the sheep in Scotland are black – just some of them.”

• Piping in, the mathematician retorted, “Tut, tut, tut, to be correct you must say, ‘At least one’ sheep in Scotland is black.”

• The physicist had the last word, though, stating, “Gentlemen, all we know with certainty based on our observations is that at least one sheep in Scotland is black on at least one side, at least part of the time.”

• Moral: There are hard and soft sciences, and extrapolation is not always justified.http://creationsafaris.com/humor.htm

Total Story Lead Time

30 days

Story / Feature Inception5 Days

Waiting in Backlog25 days

System Regression Testing & Staging 5 Days

Waiting for Release Window5 Days

“Active Development”30 days

Pre Work

30 days

Post Work

10 days

9 days (70 total)approx 13%

THE SHAPE OF CYCLE TIMEWhat distribution fits cycle time data and why…

If we understand how cycle time is statistically distributed, then an

initial guess of maximum allows an inference to be made

Alternatives -

• Borrow a similar project’s data• Borrow industry data• Fake it until you make it… (AKA guess range)

Why Weibull

• Now for some Math – I know, I’m excited too!

• Simple Model• All units of work between 1 and 3 days• A unit of work can be a task, story, feature, project• Base Scope of 50 units of work – Always Normal• 5 Delays / Risks, each with

– 25% Likelihood of occurring– 10 units of work (same as 20% scope increase each)

Normal, or it will be after a few

thousand more simulations

Base + 1 Delay

Base + 2 Delays

Base + 3 Delays

Base + 4 Delays

Base + 5 Delays

Exponential Distribution (Weibull shape = 1)The person who gets the work can complete the workTeams with no external dependenciesTeams doing repetitive work E.g. DevOps, Database teams,

Weibull Distribution (shape = 1.5)Typical dev team ranges between 1.2 and 1.8

Rayleigh Distribution (Weibull shape = 2)Teams with MANY external dependenciesTeams that have many delays and re-work. E.g. Test teams

What Distribution To Use...

• No Data at All, or Less than < 11 Samples (why 11?)– Uniform Range with Boundaries Guessed (safest)– Weibull Range with Boundaries Guessed (likely)

• 11 to 30 Samples– Uniform Range with Boundaries at 5th and 95th CI– Weibull Range with Boundaries at 5th and 95th CI

• More than 30 Samples– Use historical data as bootstrap reference– Curve Fitting software

Probability Density Function

Histogram Weibull

x1201101009080706050403020100

f(x)

0.28

0.24

0.2

0.16

0.12

0.08

0.04

0

Scale – How Wide in Range. Related to the

Upper Bound. *Rough* Guess: (High – Low) / 4

Shape – How Fat the distribution. 1.5 is a good starting point.

Location – The Lower Bound