applying 'kanban' in enterprise-class products sustaining engineering - an experience...
DESCRIPTION
My presentation at Agile-Scrum International Summit, Bangalore on a 'Kanban' implementation we did in 2004-05.TRANSCRIPT
1
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Applying ‘Kanban’ in Enterprise-Class Products Sustaining Engineering
~ An Experience Report ~
Tathagat Varma@tathagatvarma
http://managewell.net http://slideshare.net/managewell
2
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Discussion Topics
• Enterprise-class Software Products• Our Product • Old Process• Problems with Old Process• New Process• Did this move the needle?• What is Kanban?• Kanban in Software Engineering• What did we learn?
3
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
What characterizes Enterprise Software?
4
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Microsoft Windows timeline
https://en.wikipedia.org/wiki/Timeline_of_Microsoft_Windows
5
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Other major OS and tools timeline
https://en.wikipedia.org/wiki/Solaris_(operating_system) https://en.wikipedia.org/wiki/Bugzilla
6
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
As a contrast, what are consumer internet companies doing?
• Continuous Integration -> Continuous Delivery -> Continuous Deployment
• On ‘good days’, Flickr releases a new version every half an hour (Jun 20, 2005)
• IMVU pushes a revision of code to the website every nine minutes (Feb 10, 2009)
• The other day we passed product release number 25,000 for WordPress. That means we’ve averaged about 16 product releases a day, every day for the last four and a half years! (May 19, 2010)
• A new version of Google Chrome now due every six weeks (Jul 22, 2010)
• Facebook does code push twice a day (Aug 4, 2012)
7
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Desktop OS Market share (Nov 2012)
http://marketshare.hitslink.com/operating-system-market-share.aspx?qprid=10&qpcustomd=1
8
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Our Product
• Network Management domain• Windows-based specialized hardware (“Appliances”)• Installed in data centers for traffic monitoring,
analysis and network troubleshooting
– but not generally on production network• Typical users are technical folks – CIO, Network
Manager, Network Engineers• Selling cycles typically align with quarterly or annual
budget cycles• Many sales require implementing customer specials
9
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Old Process, circa 2003
• Customer Bugs prioritized based on multiple business parameters, including (partial list) -
– Severity
– Impact on Revenue, Volume, Competitive, etc.
– Case age• PMO would prepare Maintenance Release Plan of Record (MR
POR) and get buy-in for various types of MRs -
– Service Packs – bunch up ~50-60 bugs typically every quarter
– Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP
– Patches – workaround for customer-specific issues• SPs would have
– Above The Line (ATL) requirements – must fix
– Below The Line (BTL) requirements – fix if time permits
10
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
A typical support timeline
http://itconvergence.blogspot.in/2012/10/oow-12-elison-extradata-oracle-r122.html
11
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Problems with Old Process
• Dev team had no bandwidth to take on maintenance releases• Huge pile of customer escalations without “home”• Compounded by high incoming field rate• Low closure rate (largely due to no dedicated resources)• Large wait for customers to get bug fixes• Tech Support often tasked team directly and broke the process• Hot fixes not always available to all customers• Sometimes, a new bug fix might break a hot fix• If a hot fix failed in the field, rollbacks would be very difficult• Difficult to estimate time to resolve a bug and give an ETA• High-priority bugs could arrive at any time• Customer specials could arrive anytime with top priority• High internal rejection rate of bug fixes by Tech Support
12
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
New Process, 2004-05
13
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Product C
Product D
Product C
Product D
ProtocolsProtocols
Decide DriversDecide Drivers
Backend
Product B
Backend
Product B
GUI
Product A
GUI
Product A
Our Kanban Process in action…
PMO
CST Manager
QA TeamQA Team
Dev Team = 15
WIP = 15
WIP = 3
WIP = 4
WIP
= 2
WIP
= 3
WIP
= 3
WIP = 1wk
WIP = 1wk
Tech Support
Qu
eue
= 0
Queue = 0
14
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Process improvement…the beginnings…
15
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Cumulative Hot Fix Process
16
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Weekly Build Process
17
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
So, what is happening?
• Though not an originally stated vision or goal, the “Work in Progress” (WIP) is being limited to # of team members
• At any time, one developer is assigned only one piece of work, thereby achieving “One-Piece Flow”
• New work is only assigned when current work is completed (or cancelled/stalled), and a team member is available
• No wait state or switching costs at an individual level• Smaller lead time for bugs (in contract to lead time for SP)• The process is allowing ‘continuous deployment’ of each
of the hot fixes – even though it is only being practiced in a limited manner
• Finally, the flexibility gained is not a zero-sum game – there is no penalty on performance in rest of the process
18
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Did this move the needle?
• Bugs addressed each quarter• Quality of bug fixes• “Homes” for bugs• Total bugs open• Open days open• People motivation
19
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Shift from SPs to Cumulative Hot Fixes while maintaining High Quality
Maintenance Releases (Service Packs, Patches, Hot Fixes) Q3 2003 Through Q4 2006 (Fiscal Year)
0 2 3 4 4 4 42 2 2 1 2 3
15 2 1
53 1 0
0 1 0 0 01
0
28
11
1618 20
62
28 2726 26 24
28
21
100
92
87
96
88
92
85
80
97 9693
100
9491
7
12
15
25 25 25
30 3028 27 26
32
22
66
0
10
20
30
40
50
60
70
Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 06
0
10
20
30
40
50
60
70
80
90
100
Service Packs Patches Hot Fixes % Successful
Percentage of released Maintenance Releases (Service Packs, Patches, Hots Fixes) that addressed customer reported
Total Maintenance Releases (Service Packs, Patches, Hot Fixes) for this quarter.
ServicePacks
Patches
Hot Fixes
20
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Increase in bugs with “homes”
Customer Direct and Indirect % w/Homes - HistoricalWeek Ending February 03, 2006
55
46 45 4744 45 45 44
5552 54 56
35
4539 38
41
5146
31
40 40
5963
60 61 61
68 6974 72
69 69
55
73
61
535655
4854 55
52 51 51 51 50 4954 55
47 47
1115
18 16
4753 55
43 42 43
75
8276 76 75
82 83 82 81
71 72
65
8782
7580
55
4751 51 50 48 48 48 48 48
53 5451 51
2127 27 25
44
52 51
43 41 42
6772
68 69 68
75 76 78 7670 71
60
80
71
6468
4447
0
10
20
30
40
50
60
70
80
90
100
Direct Indirect Goal (80% w/Homes) Combined
Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)
21
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Total Bugs Open
Severity Across Products (Historical)
Week Ending March 31, 2006
222
191
243
213
224230
236
201199
210211
196197195
161
147146146146
132
122
131
121127
111
127125
148150
65605857
5357
626466676768717475756866
4947465049495052
606369707069
747368
61605555
4749514944
3538323332343536
0
10
20
30
40
50
60
70
80
90
100
110
120
130
0
20
40
60
80
100
120
140
160
180
200
220
240
260
Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
22
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Average Days Open
Severity: Average Days By Severity (Historical)
Week Ending March 31, 2006
203
155
136
244242246246244248
234235226223
232
244
260268
275275
299
232
201210207
179
165172
156162
180
168163
156152147
142
132135139143
148140
144147146144
155
135
145152
148157156156
149
133
144140
144147154155157
163166
177179183182
164162165
154146
141149
123117
134134
123
0
25
50
75
100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
475
500
0
25
50
75
100
125
150
175
200
225
250
275
300
Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
23
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
People motivation
• Started with 16 people dev team• We had zero attrition in the team• Once the backlog started coming down, engineers were
ramped off the team to do new features• Eventually dismantled the team and rolled-up engineers into
dev teams when backlog came down to single digits
24
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
What is Kanban?
• Kanban (literally signboard or billboard) is a scheduling system for lean and just-in-time (JIT) production. According to its creator, Taiichi Ohno, kanban is one means through which JIT is achieved.
• Kanban is not an inventory control system; it is a scheduling system that helps determine what to produce, when to produce it, and how much to produce.
• The need to maintain a high rate of improvement led Toyota to devise the kanban system. Kanban became an effective tool to support the running of the production system as a whole.
• In addition, it proved to be an excellent way for promoting improvements because reducing the number of kanban in circulation highlighted problem areas.
https://en.wikipedia.org/wiki/Kanban
25
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
A Kanban System at my Toyota dealership
https://twitpic.com/het3u
26
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
How does it work?
http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/just-in-time.html
27
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
How Kanban helps achieve “Just-in-Time”
• For example, to efficiently produce a large number of automobiles, which can consist of around 30,000 parts, it is necessary to create a detailed production plan that includes parts procurement. Supplying "what is needed, when it is needed, and in the amount needed" according to this production plan can eliminate waste, inconsistencies, and unreasonable requirements, resulting in improved productivity.
http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/just-in-time.html
28
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Kanban for Software
• Visualize the Workflow: Represent the work items and the workflow on a card wall or electronic board
• Limit Work-in-Progress (WIP): Set agreed upon limits on how many work items are in progress at a time
• Measure and Manage Flow: Track work items to see if they are proceeding at a steady, even pace
• Make Process Policies Explicit: Agree upon and post policies about how work will be handled
• Use Models to Evaluate Improvement Opportunities: Adapt the process using ideas from Systems Thinking, Deming, etc.
Kanban: Successful Evolutionary Change for your Technology Business – David Anderson
29
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Why Kanban in Software Engineering?
https://leanandkanban.files.wordpress.com/2009/04/kanban-for-software-engineering-apr-242.pdf
30
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
What did We learn?
• Process improvement should be
driven by business needs – and NOT because some process looks sexy!
• Don’t let a process limit your potential – think beyond gurus!
• Don’t let absence of a process limit your potential – do whatever it takes to
serve customer better!
31
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Thanks guys!
• To the wonderful Customer Sustaining Team (Dev and QA), Program Management and Tech Support at Network Associates / Network General in Bangalore and San Jose for some great stuff !!
32
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Resources
• Was that Kanban? - http://finance.groups.yahoo.com/group/kanbandev/message/4131 and http://finance.dir.groups.yahoo.com/group/kanbandev/message/4166
• http://refcardz.dzone.com/refcardz/getting-started-kanban• Ship early and ship twice as often, https://www.facebook.com/notes/facebook-
engineering/ship-early-and-ship-twice-as-often/10150985860363920• How we built Flickr,
http://www.plasticbag.org/archives/2005/06/cal_henderson_on_how_we_built_flickr/• Continuous Deployment at IMVU: Doing the impossible fifty times a day,
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
• A New Version of Google Chrome now due every six weeks, http://techcrunch.com/2010/07/22/google-chrome-versions/
• In praise of continuous deployment: The WordPress.com story, http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/
• CONWIP, https://en.wikipedia.org/wiki/CONWIP • Kanban applied to Software Development: from Agile to Lean,
http://www.infoq.com/articles/hiranabe-lean-agile-kanban
33
www.scrumindia.in
Date: 7-9 Dec 2012Name: TV
ScrumIndia.In Proprietary Information
Q & A
Questions and Answers?