A successful improvement process with measurable results
Yigal Cohen
(Ex) R&D Manager Deputy - NDS
March 2010 –ע "אדר תש
Chosen by the top TV operators
– Global company
– Complex
– industry leading system
– Changing matrix
Improvement plan goals
3
• What ?
– Increase business results by improving project management, process & methodologies
• How?
– Provide a set of principles, tools, metrics, and improvement framework
– Push Lines to lead and manage their actual improvement plan
– Accompany and mentor real projects
R&D Charter
32
10
4
Criteria
14 Ch.14 Every Build will have the following artifacts:1. Build in Release management system2. Should have both production as well as Debug enabled image3. Release note with CQ list and component details
5: NA
15 Ch.12 a Defects and issues are tracked 2 : somewhat
16 Ch.12 b Defects and issues undergo root-cause analysis
2 : somewhat
17 Communication plan is defined 3 : well into it
18 Ch.13 Artefacts are held within a configuration or document management system
5: NA
19 Ch.15 Project has end of milestone reviews 2 : somewhat
20 Communication about the completion of milestone
2 : somewhat
Self AssessmentsImprovement Plan
Toolkit
Continuous Improvement Framework
4
Self Assessment Analysis
The Community
5
Users
2200
Users
440
AI Holders35 SQAs
8 WW_QA +
R&D Management
1 32 4 6 7 8 10 12 135 9 11 14 15
2 4 7 10 1211 14
2 6 1311 14
1 32 4 6 7 8 10 12 135 9 14 15
R&D
Division
Line
Project
Every division and line is authorized and encouraged to prepare its own procedures for part or all of the R&D Charter.
The most specific procedures will be used for each project.
11
Local toolkits
6
SDP• Time & effort Estimation
• Scheduling
• WBS
• Design/Dev/Test done
iteratively
• Progress Monitoring
• Burn Down
• TOC
• Risk Management
Testing• Common Logging
• Automated regression tests
• TPI
Integration• Continuous Integration
• Release
• Release Criteria
Configuration• Branching Policy
• Building
• Automated Unit level tests
Requirements• Tests cover requirements
Design• Defensive Design & coding
• UML
• Pattern Design
Coding• Re-factpriong
• Complexity analysis
• Static Analysis
7
• Does it help ?
• To Whom ?
8
Done Successfully
Infrastructure
• Coding standards written for New Generation Smart Card, SimCAM QC Tools, and Non-Smart Card Products
• Migration to SVN source management
• Root cause analysis accepted part of standard quality procedure
• Smart Card development utilizing Visual Studio environment and associated tools (pilot)
Actual improvements
• New Smartcard code base developed using:
– Unit test on 80% of code
– Continuous Build on 100% of code
– ->New Smartcard ROM integration & testing in only 4 weeks!
• SVN configuration management (Most of the line)
• Root Cause Analysis being performed on all defects found in I&T and at the customer site
• Average R&D Charter compliance of 89%
One line - improvement
activities
9
Slide 9
AV4 Not sure where this fits in.Dr Alison Vincent, 02/07/2008
Impacts on the development process(Saves time, better quality and increase efficiency)
• Integration Plan
– Planning the integration saves a lot of time by making pre-integrations and preparing tools in advance (Saves time)
• Test Strategy
– Testing process was improved especially in multi-site projects (Saves time and improve quality)
• Architects started to increase their involvement along the development process (Quality and efficiency)
• Spreading of Coding Standards, Build, Root Cause Analysis and Lessons learned across R&D with Common global language and transparency (All)
• Contributed to increase profit by 25% in a year!
10
Customer defects reported in CQ
0
100
200
300
400
500
600
700
800
900
Qtr 1
'06
Qtr 2
'06
Qtr 3
'06
Qtr 4
'06
Qtr 1
'07
Qtr 2
'07
Qtr 3
'07
Qtr 4
'07
Qtr 1
'08
Qtr 2
'08
Qtr 3
'08
Qtr 4
'08
Qtr 1
'09
Customer Defects
0
2000
4000
6000
8000
10000
12000
14000
16000
Q1 '06
Q2 '06
Q3 '06
Q4 '06
Q1 '07
Q2 '07
Q3 '07
Q4 '07
Q1 '08
Q2 '08
Q3 '08
Q4 '08
Q1 '09
11
R&D Charter is the guideline for the process
1. For Planning, Tracking & Management projects will have: 1. Software Development Plan 2. Resource Allocation plan
3. Re-use strategy
4. Test Strategy and Integration Plan
2. Development phases1. Requirements will be Documented/Reviewed/Managed
2. Designs will be documented and Reviewed
3. Coding standard
4. Code review
5. Automatic tools and ATP for every API
3. Infrastructure1. Artefacts will be held within a configuration or document
management system
2. Defects and Issues will be tracked and will undergo root cause analysis
3. Projects will have end of project reviews
4. Builds will be automated and repeatable
5. Released builds will be controlled
NDS R&D Charter
12
What’s in the toolkit ?
Checklists
Templates
Tools
Support
Guidelines
Examples
Contacts
Other Refs
Training
!
13
R&D Charter Toolkit
14
15
3: TS compliant to the checklist
communicated and followed
2: TS reviewed and approved
1: Test Strategy is written
0: No Test Strategy
4: TS updated throughout
project
An Example Criteria for Self
Assessment
15
Self Assessments - ExampleCh.01 Projects will have a Software /
Hardware Development Plan that meets a defined schedule according to the customer's needs
2: Documented or presented at Kick Off meeting
2 http://shareatnds.in.nds.com/sites/Intranet/OpenTV/master_ui/Shared%20Documents/project_plan.aspx
Ch.02a Projects will have a Test strategy
2: Approved Test Strategy (SQSC-style) reviewed and approved by all stakeholders
2 http://doogle.uk.nds.com/worksitemp/dispatch?objectId=%21V3%21UKDMPRODUCTION1%21C%21PJ%2441860%21&returnUrl=121145625069359336&operation=zoom&datasource=project
Ch.02b Projects will have an Integration plan
0: No Integration Plan 0 Include link to plan
Ch.03 Projects will have clearly defined roles and responsibilities
4: All roles actively performed 4 http://shareatnds.in.nds.com/sites/Intranet/OpenTV/master_ui/default.aspx
Ch.04 Projects will agree on the strategy for re-use at Project Kick-Off (requirements, design, test and code)
1: Initial discussion on re-use of previous assetts
1
Ch.05 Projects will follow the R&D Resource allocation plan
2: resource plan done (available) but not reviewed against delivery requirements
2 http://shareatnds.in.nds.com/sites/Intranet/OpenTV/master_ui/Shared%20Documents/project_plan.aspx?RootFolder=%2fsites%2fIntranet%2fOpenTV%2fmaster%5fui%2fProject%20Plan%2fWorkPlan&FolderCTID=&View=%7b69BD6617%2d6CCB%2d46E0%2dA5CE%2dFCE21C885064%7d
Ch.06 Requirements will be documented / managed / reviewed
2: requirements and change requests documented (Document, ReqPro, ClearQuest).
2 http://doogle.uk.nds.com/worksitemp/dispatch?datasource=documentFolder&operation=zoom&objectId=!V3!UKDMPRODUCTION1!C!FD$44881!&returnUrl=12114569796644311
Ch.07 Designs will be documented & reviewed
3: design reviews done with tangible outcomes, key design choices documented and major issues fixed.Traceability and effectiveness checked in review
3 http://doogle.uk.nds.com/worksitemp/dispatch?datasource=documentFolder&operation=zoom&objectId=!V3!UKDMPRODUCTION1!C!FD$44875!&returnUrl=121145717958875383 16
Sample Analysis
Charter
Item #
Charter Item P1 P2 P3 P4 Average
Ch.01 Projects will have a Software / Hardware Development Plan that meets a defined schedule
according to the customer's needs1 2 2 2 1.8
Ch.02a Projects will have a Test strategy 3 1 2 0 1.5
Ch.02b Projects will have an Integration plan 4 3 3 3 3.3
Ch.03 Projects will have clearly defined roles and responsibilities 1 2 4 4 2.8
Ch.04 Projects will agree on the strategy for re-use at Project Kick-Off (requirements, design, test
and code)1 4 2 1 2.0
Ch.05 Projects will follow the R&D Resource allocation plan 4 3 4 1 3.0
Ch.06 Requirements will be documented / managed / reviewed 2 2 2 3 2.3
Ch.07 Designs will be documented & reviewed 3 1 4 2 2.5
Ch.08 Code will follow a coding standard 2 3 3 3 2.8
Ch.09 Code will be reviewed 2 2 1 1 1.5
Ch.10 Builds will be automated and repeatable 3 4 3 3 3.3
Ch.11 APIs will have a tool by which they can be tested 1 1 1 1 1.0
Ch.12a Defects and Issues will be tracked 3 3 3 3 3.0
Ch.12b Defects will undergo root cause analysis 2 2 2 1 1.8
Ch.13 Artefacts will be held within a configuration or a document management system 4 4 4 4 4.0
Ch.14 Released builds will be controlled 2 3 3 4 3.0
Ch.15 Projects will have end of project reviews 3 2 3 0 2.0
COLUMN AVERAGES 2.4 2.5 2.7 2.1 2.417
Improvement Plan – 3 Examples
18
• Faster and more reliable builder
– Learn Hudson and build infrastructure. 6 weeks. Danny
– Run a pilot. Next 8 weeks. Mike.
– Evaluate Results. Next 2 weeks. Rachel
• Deploy Scrum in all projects
– Use Scrum in projects X, Y, Z. Q1. Danny, Shani, Rafi
– 1 Day seminar for GL, including our lesson leaned. Week 3
– Bi-Weekly Scrum Masters meeting. Moshe.
– Write a Departemnet Scrum guide. Moshe.
– Run a full cycle of iteration 1 on a PC for evaluation
• Increase efficiency
– AI: Inquire Why after using Agile the number of bug was not reduced. 6 weeks. Danny
1200 action items from IL/FR/IN/UK/US
75% of the were done
All lines were reviewed by R&D Manager
14 Ch.14 Every Build will have the following artifacts:1. Build in Release management system2. Should have both production as well as Debug enabled image3. Release note with CQ list and component details
5: NA
15 Ch.12 a Defects and issues are tracked 2 : somewhat
16 Ch.12 b Defects and issues undergo root-cause
analysis
2 : somewhat
17 Communication plan is defined 3 : well into it
18 Ch.13 Artefacts are held within a configuration or document management system
5: NA
19 Ch.15 Project has end of milestone reviews 2 : somewhat
20 Communication about the completion of milestone
2 : somewhat
Self Assessments
QC Lead 3 4 4 4 4 4 4 4 3 3 3.70
Project has a software Development plan that defines the schedule according to the customer needs 3 2 3 3 3 4 4 4 4 4 3.40
I teration plan exists 3 3 3 3 3 1 1 1 1 1 2.00
Component Development Plan exists along with component QC plan 2 2 3 2 1 1 1 1 1 1 1.50
Configuration Management Plan exists 1 1 1 1 3 1 1 1 1 1 1.20
Project has a Resource allocation plan 3 2 3 3 3 2 2 1 1 2 2.20
Project has an Integration Plan 3 2 1 4 3 1 1 2 1 1 1.90
Project has a test strategy 3 2 2 3 3 3 3 1 3 3 2.60
Project has all the required documents managed and rev iewed 2 2 1 3 3 4 3 3 3 2.67
Test cases are prepared, Mapped to Requirements, reviewed and managed 3 3 2 2 3 3 4 4 3 3 3.00
APIs have a tool by which they can be tested 2 2 2 2 1 1 1 4 1 1 1.70
Code follows a coding standard 3 1 3 2 2 3 3 1 3 3 2.40
code is reviewed 4 2 4 2 3 3 3 3 3 3 3.00
Self Assessment AnalysisImprovement Plan
R&D Process Improvement
Root Cause
Analysis
Other
needsLessons
Learned
19
WW_QA Working Group
Thank you !
20
Metrics• Development progress
– Defects trends
– Code Coverage by tests
– Test Progress
– Requirements Coverage by tests
– Team velocity, Gantt Progress
• Quality prediction
– Counting Escaped Bugs by phases
21
Code Coverage Defect trends
Test Progress Static Analysis
22