A successful improvement process with measurable results

Yigal Cohen

(Ex) R&D Manager Deputy - NDS

March 2010 –ע "אדר תש

Chosen by the top TV operators

– Global company

– Complex

– industry leading system

– Changing matrix

Improvement plan goals

3

• What ?

– Increase business results by improving project management, process & methodologies

• How?

– Provide a set of principles, tools, metrics, and improvement framework

– Push Lines to lead and manage their actual improvement plan

– Accompany and mentor real projects

R&D Charter

32

10

4

Criteria

14 Ch.14 Every Build will have the following artifacts:1. Build in Release management system2. Should have both production as well as Debug enabled image3. Release note with CQ list and component details

5: NA

15 Ch.12 a Defects and issues are tracked 2 : somewhat

16 Ch.12 b Defects and issues undergo root-cause analysis

2 : somewhat

17 Communication plan is defined 3 : well into it

18 Ch.13 Artefacts are held within a configuration or document management system

5: NA

19 Ch.15 Project has end of milestone reviews 2 : somewhat

20 Communication about the completion of milestone

2 : somewhat

Self AssessmentsImprovement Plan

Toolkit

Continuous Improvement Framework

4

Self Assessment Analysis

The Community

5

Users

2200

Users

440

AI Holders35 SQAs

8 WW_QA +

R&D Management

1 32 4 6 7 8 10 12 135 9 11 14 15

2 4 7 10 1211 14

2 6 1311 14

1 32 4 6 7 8 10 12 135 9 14 15

R&D

Division

Line

Project

Every division and line is authorized and encouraged to prepare its own procedures for part or all of the R&D Charter.

The most specific procedures will be used for each project.

11

Local toolkits

6

SDP• Time & effort Estimation

• Scheduling

• WBS

• Design/Dev/Test done

iteratively

• Progress Monitoring

• Burn Down

• TOC

• Risk Management

Testing• Common Logging

• Automated regression tests

• TPI

Integration• Continuous Integration

• Release

• Release Criteria

Configuration• Branching Policy

• Building

• Automated Unit level tests

Requirements• Tests cover requirements

Design• Defensive Design & coding

• UML

• Pattern Design

Coding• Re-factpriong

• Complexity analysis

• Static Analysis

7

• Does it help ?

• To Whom ?

8

Done Successfully

Infrastructure

• Coding standards written for New Generation Smart Card, SimCAM QC Tools, and Non-Smart Card Products

• Migration to SVN source management

• Root cause analysis accepted part of standard quality procedure

• Smart Card development utilizing Visual Studio environment and associated tools (pilot)

Actual improvements

• New Smartcard code base developed using:

– Unit test on 80% of code

– Continuous Build on 100% of code

– ->New Smartcard ROM integration & testing in only 4 weeks!

• SVN configuration management (Most of the line)

• Root Cause Analysis being performed on all defects found in I&T and at the customer site

• Average R&D Charter compliance of 89%

One line - improvement

activities

9

Slide 9

AV4 Not sure where this fits in.Dr Alison Vincent, 02/07/2008

Impacts on the development process(Saves time, better quality and increase efficiency)

• Integration Plan

– Planning the integration saves a lot of time by making pre-integrations and preparing tools in advance (Saves time)

• Test Strategy

– Testing process was improved especially in multi-site projects (Saves time and improve quality)

• Architects started to increase their involvement along the development process (Quality and efficiency)

• Spreading of Coding Standards, Build, Root Cause Analysis and Lessons learned across R&D with Common global language and transparency (All)

• Contributed to increase profit by 25% in a year!

10

Customer defects reported in CQ

0

100

200

300

400

500

600

700

800

900

Qtr 1

'06

Qtr 2

'06

Qtr 3

'06

Qtr 4

'06

Qtr 1

'07

Qtr 2

'07

Qtr 3

'07

Qtr 4

'07

Qtr 1

'08

Qtr 2

'08

Qtr 3

'08

Qtr 4

'08

Qtr 1

'09

Customer Defects

0

2000

4000

6000

8000

10000

12000

14000

16000

Q1 '06

Q2 '06

Q3 '06

Q4 '06

Q1 '07

Q2 '07

Q3 '07

Q4 '07

Q1 '08

Q2 '08

Q3 '08

Q4 '08

Q1 '09

11

R&D Charter is the guideline for the process

1. For Planning, Tracking & Management projects will have: 1. Software Development Plan 2. Resource Allocation plan

3. Re-use strategy

4. Test Strategy and Integration Plan

2. Development phases1. Requirements will be Documented/Reviewed/Managed

2. Designs will be documented and Reviewed

3. Coding standard

4. Code review

5. Automatic tools and ATP for every API

3. Infrastructure1. Artefacts will be held within a configuration or document

management system

2. Defects and Issues will be tracked and will undergo root cause analysis

3. Projects will have end of project reviews

4. Builds will be automated and repeatable

5. Released builds will be controlled

NDS R&D Charter

12

What’s in the toolkit ?

Checklists

Templates

Tools

Support

Guidelines

Examples

Contacts

Other Refs

Training

!

13

R&D Charter Toolkit

14

15

3: TS compliant to the checklist

communicated and followed

2: TS reviewed and approved

1: Test Strategy is written

0: No Test Strategy

4: TS updated throughout

project

An Example Criteria for Self

Assessment

15

Self Assessments - ExampleCh.01 Projects will have a Software /

Hardware Development Plan that meets a defined schedule according to the customer's needs

2: Documented or presented at Kick Off meeting

2 http://shareatnds.in.nds.com/sites/Intranet/OpenTV/master_ui/Shared%20Documents/project_plan.aspx

Ch.02a Projects will have a Test strategy

2: Approved Test Strategy (SQSC-style) reviewed and approved by all stakeholders

2 http://doogle.uk.nds.com/worksitemp/dispatch?objectId=%21V3%21UKDMPRODUCTION1%21C%21PJ%2441860%21&returnUrl=121145625069359336&operation=zoom&datasource=project

Ch.02b Projects will have an Integration plan

0: No Integration Plan 0 Include link to plan

Ch.03 Projects will have clearly defined roles and responsibilities

4: All roles actively performed 4 http://shareatnds.in.nds.com/sites/Intranet/OpenTV/master_ui/default.aspx

Ch.04 Projects will agree on the strategy for re-use at Project Kick-Off (requirements, design, test and code)

1: Initial discussion on re-use of previous assetts

1

Ch.05 Projects will follow the R&D Resource allocation plan

2: resource plan done (available) but not reviewed against delivery requirements

2 http://shareatnds.in.nds.com/sites/Intranet/OpenTV/master_ui/Shared%20Documents/project_plan.aspx?RootFolder=%2fsites%2fIntranet%2fOpenTV%2fmaster%5fui%2fProject%20Plan%2fWorkPlan&FolderCTID=&View=%7b69BD6617%2d6CCB%2d46E0%2dA5CE%2dFCE21C885064%7d

Ch.06 Requirements will be documented / managed / reviewed

2: requirements and change requests documented (Document, ReqPro, ClearQuest).

2 http://doogle.uk.nds.com/worksitemp/dispatch?datasource=documentFolder&operation=zoom&objectId=!V3!UKDMPRODUCTION1!C!FD$44881!&returnUrl=12114569796644311

Ch.07 Designs will be documented & reviewed

3: design reviews done with tangible outcomes, key design choices documented and major issues fixed.Traceability and effectiveness checked in review

3 http://doogle.uk.nds.com/worksitemp/dispatch?datasource=documentFolder&operation=zoom&objectId=!V3!UKDMPRODUCTION1!C!FD$44875!&returnUrl=121145717958875383 16

Sample Analysis

Charter

Item #

Charter Item P1 P2 P3 P4 Average

Ch.01 Projects will have a Software / Hardware Development Plan that meets a defined schedule

according to the customer's needs1 2 2 2 1.8

Ch.02a Projects will have a Test strategy 3 1 2 0 1.5

Ch.02b Projects will have an Integration plan 4 3 3 3 3.3

Ch.03 Projects will have clearly defined roles and responsibilities 1 2 4 4 2.8

Ch.04 Projects will agree on the strategy for re-use at Project Kick-Off (requirements, design, test

and code)1 4 2 1 2.0

Ch.05 Projects will follow the R&D Resource allocation plan 4 3 4 1 3.0

Ch.06 Requirements will be documented / managed / reviewed 2 2 2 3 2.3

Ch.07 Designs will be documented & reviewed 3 1 4 2 2.5

Ch.08 Code will follow a coding standard 2 3 3 3 2.8

Ch.09 Code will be reviewed 2 2 1 1 1.5

Ch.10 Builds will be automated and repeatable 3 4 3 3 3.3

Ch.11 APIs will have a tool by which they can be tested 1 1 1 1 1.0

Ch.12a Defects and Issues will be tracked 3 3 3 3 3.0

Ch.12b Defects will undergo root cause analysis 2 2 2 1 1.8

Ch.13 Artefacts will be held within a configuration or a document management system 4 4 4 4 4.0

Ch.14 Released builds will be controlled 2 3 3 4 3.0

Ch.15 Projects will have end of project reviews 3 2 3 0 2.0

COLUMN AVERAGES 2.4 2.5 2.7 2.1 2.417

Improvement Plan – 3 Examples

18

• Faster and more reliable builder

– Learn Hudson and build infrastructure. 6 weeks. Danny

– Run a pilot. Next 8 weeks. Mike.

– Evaluate Results. Next 2 weeks. Rachel

• Deploy Scrum in all projects

– Use Scrum in projects X, Y, Z. Q1. Danny, Shani, Rafi

– 1 Day seminar for GL, including our lesson leaned. Week 3

– Bi-Weekly Scrum Masters meeting. Moshe.

– Write a Departemnet Scrum guide. Moshe.

– Run a full cycle of iteration 1 on a PC for evaluation

• Increase efficiency

– AI: Inquire Why after using Agile the number of bug was not reduced. 6 weeks. Danny

1200 action items from IL/FR/IN/UK/US

75% of the were done

All lines were reviewed by R&D Manager

14 Ch.14 Every Build will have the following artifacts:1. Build in Release management system2. Should have both production as well as Debug enabled image3. Release note with CQ list and component details

5: NA

15 Ch.12 a Defects and issues are tracked 2 : somewhat

16 Ch.12 b Defects and issues undergo root-cause

analysis

2 : somewhat

17 Communication plan is defined 3 : well into it

18 Ch.13 Artefacts are held within a configuration or document management system

5: NA

19 Ch.15 Project has end of milestone reviews 2 : somewhat

20 Communication about the completion of milestone

2 : somewhat

Self Assessments

QC Lead 3 4 4 4 4 4 4 4 3 3 3.70

Project has a software Development plan that defines the schedule according to the customer needs 3 2 3 3 3 4 4 4 4 4 3.40

I teration plan exists 3 3 3 3 3 1 1 1 1 1 2.00

Component Development Plan exists along with component QC plan 2 2 3 2 1 1 1 1 1 1 1.50

Configuration Management Plan exists 1 1 1 1 3 1 1 1 1 1 1.20

Project has a Resource allocation plan 3 2 3 3 3 2 2 1 1 2 2.20

Project has an Integration Plan 3 2 1 4 3 1 1 2 1 1 1.90

Project has a test strategy 3 2 2 3 3 3 3 1 3 3 2.60

Project has all the required documents managed and rev iewed 2 2 1 3 3 4 3 3 3 2.67

Test cases are prepared, Mapped to Requirements, reviewed and managed 3 3 2 2 3 3 4 4 3 3 3.00

APIs have a tool by which they can be tested 2 2 2 2 1 1 1 4 1 1 1.70

Code follows a coding standard 3 1 3 2 2 3 3 1 3 3 2.40

code is reviewed 4 2 4 2 3 3 3 3 3 3 3.00

Self Assessment AnalysisImprovement Plan

R&D Process Improvement

Root Cause

Analysis

Other

needsLessons

Learned

19

WW_QA Working Group

Thank you !

20

Metrics• Development progress

– Defects trends

– Code Coverage by tests

– Test Progress

– Requirements Coverage by tests

– Team velocity, Gantt Progress

• Quality prediction

– Counting Escaped Bugs by phases

21

Code Coverage Defect trends

Test Progress Static Analysis

22