tl 9000 measurements and requirements interactive workshop

Copyright 2009 QuEST Forum. All Rights Reserved.

11

TL 9000 Measurements and Requirements

Interactive Workshop

APAC Best Practices Conference

Tokyo, Japan

16 April, 2015

Ken Koffman – JDSU

John Wronka – Alcatel-Lucent

Topics for Today

• Questions were provided in advance in

the following Areas:

– General

– QuEST Forum Membership

– TL 9000 Certification

– TL 9000 “Adder” Applications

– Measurements

– Data Submissions

• Time Permitting:

– PDR Strategic Initiative

– Member Value Survey2

“GENERAL” QUESTIONS

Copyright 2009 QuEST Forum. All Rights ReservedCopyright

2009 QuEST Forum. All Rights Reserved.

3

Acquiring TL 9000 Certification

• Tell me about how my business can

become TL 9000 certified

• How long does certification take?

• How many personnel are required?



4

Copyright 2012 QuEST Forum. All rights reserved. 5

TL 9000 Implementation Path

6. Continual Improvement

5. External Assessment

4. Assess and review

3. Process Improvement

1.Management Commitment

• Understand TL 9000 and Customer Needs, Set Goals, Commit Resources

• Core Team Selection and Training, Gap Assessment, Project Planning

• Process Mapping, Data Collection and Analysis, Process Changes to Meet New Requirements

• Internal Audit, Management Review, Corrective and Preventive Action

• Certification Body Pre-Assessmentand Registration Audit

Measurements Reporting – Often a gating item. Plan at least 9 months to prepare and submit.

Start

Suggested Implementation Time Line

4-6 Months

3-5 Months

Ongoing

2. Organizational Preparation

2-3 Months

1-2 Months

6

I would like to have realistic examples of measures to meet TL 9000 Requirements:

Access Examples Utilizing “Links” ( tl9000.org/links.html).

Use “Contact Us” to Suggest New Examples for IGQ Examples Subteam

Scroll Down

What Kinds of Questions Does QuEST Forum Get on TL 9000 Measurements and Trends?



7

http://tl9000.org/alerts/qa/overview.html

General Questions –ISO 9001:2015 and TL 9000 Requirements R6

• The requirements for documentation and

records have been reduced version by version

in ISO 9001 QMS. On the other hand, in the TL

9000 system which include ISO 9001,

documentation is mandatory for many

requirements. In the future, will TL 9000 be

aligned to ISO 9001:2015 in which the

documentation requirements are supposed to

be much more reduced?



8

ISO 9001:2015 Status Update

9

CD available 2013

DIS available 05/2014

FDIS expected AFTER 07/2015

ISO 9001:2015 to be published

09/2015

3-Year Transition Proposed

ISO Stan

dard

s Develo

pm

ent P

rocess

IAF Informative Document - Transition Planning Guidance for ISO 9001:2015

Published

TC 176 SC 2/WG 23 Communication & Product Support – Ashok Dandekar

(QuEST Forum Representative)

International ballot on the DIS closed on Oct. 10, 2014 –Results were APPROVE DIS by 90%

ISO/NP TS 9002 “Guidelines for the application of ISO 9001” may be

published in 2015

http://www.iaf.nu/upFiles/IAFID9Transition9001PublicationVersion.pdf

QuEST Forum ISO 9001:2015 Position

What actions have QuEST Forum taken regarding ISO 9001:2015?

Solicited, reviewed and submitted comments on ISO 9001:2015 CD

QuEST Forum agreed continued alignment to ISO 9001

QuEST Forum Academy ISO 9001:2015 Webinar by Nigel Croft

Solicited, reviewed and submitted comments on ISO 9001:2015 DIS

Mapped R5.5 TL 9000 Adders to ISO 9001:2015 DIS clauses (now 10!)

08/2013

06/2014

07/2014

07/2014

11/2014

May REPEAT

upon release of the FDIS!

“Key” Detailed Changes:• New version of Figure 1; ISO 9001 still follows PDCA

• “documented information” remains

• “procedures” and “records” are not mentioned instead “maintain documented information” and “retain documented information”

• A quality manual and a management representative are no longer required

• “products and services” replaces products• ISO 9000 will be updated to address changes in terms & definitions

• “applicability” replaces “exclusions”

• risks are to be determined and addressed, but there is no requirement for formal risk management or risk approach

• New/different requirements from ISO 9001:2008 include organizational context and knowledge management

• Annex A to provide general guidance on implementation and other Annexes will be included such as a summarized analysis of key enhancements between ISO 9001:2008 and ISO 9001:2015

13

ISO 9001:2015 Revision Overview

General Questions –Third Party Process

• How does QuEST Forum take

action with all certification

bodies in order to have a

uniform level of TL 9000

certification audits?

• Or, is there no level difference of

the audit among all certification

bodies in the world?



14

Regional Findings / Audit Day

ABCB Sub-Team Review:Overview of Sub-Team Charter

• CB Statistics

• AB Approval / Re-approval

• Code of Practice

– Including TL 9000 Auditor

Qualification and Experience

Requirements for CBs

• Validation Audits

Copyright 2011 QuEST Forum. All Rights Reserved16

ABCB Sub-Team Review:Validation Audits

• Plan: 2008-9

– Decision to investigate the effectiveness of

CB audits in identifying major non-

conformities

• Do: 2010-11

– ABs conducted validation audits at randomly

selected sites

• Check: 2012-13

– ABCB Sub-team analyzed data and

developed recommendations

Copyright 201117

Action Thresholds

Copyright 2009 QuEST Forum. All Rights Reserved Copyright


18

90%Failures

Certs=800 Sites

Problem

Sites High Midpoint Low

Site=1700 12 0 22.1% 11.0% 0.0% 3rd Party Process Function Well

1 32.7% 16.5% 0.4% Individual Issue, 3rd Party Process OK

2 40.8% 21.9% 2.9% Investigation Required

3 47.6% 27.1% 6.6% 3rd Party Process Review Required

4 53.4% 32.1% 10.8%

5 58.4% 36.8% 15.1%

6 62.7% 41.1% 19.6%

7 66.6% 45.3% 23.9%

8 70.0% 49.1% 28.2%

9 73.0% 52.7% 32.4%

10 75.7% 56.0% 36.4%

QuEST Forum Validation Audit

Confidence Interval

SitesAudited

Problem site count from VA Audits

Problem site estrapolation estimates

5 of 12 audits Failed:

Team determines there

is a high risk of CB

audits failing to detect

some major

non-conformities to

TL 9000


• Act: 2014-15 – Approved

1. Code of Practice Changes

• Additional qualifications for new auditors

• Additional continuing education requirements

• CBs to identify and respond to soft grading

• CBs to collect additional pre-audit information from sites

• On site audits required for installation processes

2. QF to ensure validity of product category selections

3. Optional checklist for Measurements auditing

4. Optional webinars

– Auditor Rotation – REJECTED by OSWG Vote

19


• Act: 2014-15 – Proposals Under Discussion

1. Upgrade CB Technical Review of audit reports

2. Prepare a list of optional questions for CB auditors

3. MHB Checklist improvements

4. Anonymous feedback on CBs

5. Library of recorded webinars

6. Provide additional training for CB auditors

7. Reconsider AB oversight requirements

8. Perform further validation audits

20

General Section – Other Questions

• What is the difference between "Tier 1 Supplier" and "Tier 2

Supplier"?

– It depends!!!

• Does QuEST Forum have any plans to set standard objectives for

each measurement based on PDR Industry Statistics?

– No

• Thoughts on estimating future measurement trends based data

collected from all organizations. What is the most effective way?

– Depends on maturity level, data systems in place, connection

with improvement initiatives

• What part of the TL 9000 Adders should be emphasized or

enhanced in the future?

– Some of the newer “adders” such as security, sustainability,

risk management, supplier performance management



21

QUEST FORUM MEMBERSHIP QUESTIONS



22

TL 9000 Certification & QuESTForum Membership Questions

• TL 9000 Certification does not require

Forum membership

• QuEST Forum membership does not

require TL 9000 Certification



23

QuEST Forum Membership –Other Questions

• Can an Operator Receive IA/BIC/WIC for their

suppliers and the suppliers of other operators?

– Yes

• Can a Forum member or TL 9000 certified supplier

obtain industry statistics solely for the carrier which

it supplies products?

– No

• Does being a QuEST Forum member reduce the

personnel required to be TL 9000 certified?

– Not directly, but QuEST Forum membership provides access

to SMEs and best practices.



24

TL 9000 CERTIFICATIONQUESTIONS



25

TL 9000 Certification –Audit Days



26

I recognize that TL 9000 certification audit is conducted

in conjunction with ISO 9001:

• How long is the additional audit time for TL 9000 specific part?

Is there a case where a service provider requires its suppliers to require their suppliers to meet TL 9000?

• Yes! - See Examples– “The Supplier shall determine which of its subcontractors and vendors

(Quality Parties) should be registered to TL 9000 and ensure that each of

those timely registers to TL 9000”.

– “In the event Seller uses a contract manufacturer or subcontractor to

fulfill the terms of the Agreement, Seller shall….. Commit to contract

manufacturer or subcontractor registration to TL 9000 Quality

Management System Standard”

• 7.4.1.C.2 Supplier Performance Management –

– requires identified “key suppliers” to align towards TL 9000 OR

other appropriate QMS

– But does stops short of mandating TL 9000 for suppliers



27

http://tl9000.org/resources/resources.html

TL 9000 Certification - Scope Questions

• The clause 3.2 b) in the Requirement Handbook requires in the Scope

Statement that if not all products/product lines are included in the registration,

then either the included or excluded products/product lines shall be listed. I

would like to suggest for the list to be included in the Registration Profile.

– Suggestion has been discussed but becomes difficult for large

certifications

• Is it possible to limit the scope of certification to the products for a specific

customer who requires us to be TL 9000 registered?

– No, but you are required to submit only data for customers where you

have confidence in the data.

• Once an organization gets TL 9000 certified, must the organization supply all

products that meet TL 9000 requirement to customers? Or, can the

organization decide which products should meet TL 9000 requirement s or

not. (For an example, is it possible for product A to meet TL 9000 and Product

B to meet only ISO 9001)

– Yes-you can limit the scope of your products or organizations and

activities to either TL or ISO.Copyright 2009 QuEST Forum. All Rights ReservedCopyright


28

TL 9000 Certification - Other Questions

• An organization wants to limit its scope of registration to products for one

customer (a service provider). But, because the customer is new, it has not yet

supplied any products to the customer. In this case, is it possible to submit

preceding three month data as all "0"?

The organization has been supplying similar products to the other customer.

But the customer is out of scope, so the organization thinks it cannot use the

latter customer's data because it's out of scope.

- Data must be submitted before you can be TL 9000 certified

• Can we participate in the global telecommunications industry if we are not TL

9000 certified?

– Yes of course, but…

– Customer requirement’s for TL 9000 must also be considered



29

APPLICATION OF TL 9000 ADDERS

QUESTIONS



30

Inputs to TL 9000 from Other Standards

• What part of the items in TL 9000

Requirements and Measurements reflect

the standards in CMMI and Bellcore?



31


TL 9000 is based on ISO 9001

ISO 9001

Telcordia(Bellcore)

ISO/IEC12207

(Software)

ISO 9004

TL 9000

Later Versions Incorporated Concepts

From CMMI, Automotive and ITIL

33

CMMI and TL 9000

Full TL 9000 Certification Approximates CMMI Level 3. Standards

Comparisons including CMMI can be found here .

http://www.tl9000.org/about/tl9000/comparison.html

7.1.C.2 Disaster Recovery

• We have Business Continuity Plan for disaster

recovery planning. Could it be applied for this

requirement? The head office is responsible for the

Plan as a whole. Should the responsible function in the

head office be audited for investigation? The head

office will be out of scope for the certification at this

time. So, we would like to exclude this requirement

from the certification. Is it possible?

• Answer:

– BPC is a major part of disaster recovery

– Yes it should be audited

– This requirement can’t be excluded



34

7.1.C.3 Product Security

• As we are merely producing and supplying the

products based on the specification which the

Telecom Service Provider requires, the responsibility

for the product security should be on the Provider. So

we would like to exclude this requirement. Is it

possible?

• Answer:

– In this case, if the supplier is doing any form of development

work, even if to the service provider specification, this

requirement can’t be excluded.

– If there is no design work, and for example providing contract

manufacturing (product category 8 ) – it could be excluded



35

7.3.1HS.1 Migration Planning

• As we do not intentionally change the OS of the

product and just follow the customer requested

specification, and the network operation is completely

under control of the customer (Telecom Service

Provider), so we do not have relevant activities of

migration.

• Answer:

– If your company does any part of the design and the

development, then migration will be required, and this

requirement can’t be excluded.



36

7.5.5.S.1 Software Virus Protection

• As we merely produce/supply customer (service

provider) specified products, we take it for

granted that the responsibility for virus

protection is on the customer. Therefore, we

would like to exclude this requirement if

possible.

• Answer:

– This requirement is difficult to exclude if there is any

type of SW in the product. For example if you install any

of the SW or have to replicate it, this requirement would

apply. In these cases, procedures to prevent infections

would be required.



37

8.2.4.H.1 Periodic Retesting8.2.4.H.2 Content of Testing8.2.4.H.3 Frequency of Testing

• Although our production plant is out of

the certification scope, our quality control

department takes care of the product

quality. So we recognize that we should

apply these requirements into our QMS.

(This is just a confirmation.)

• Answer:

– Correct!



38

MEASUREMENTQUESTIONS



39

• What if the "units shipped" period does not actually

encompass the past full 12 months, because of a

newly introduced product which has become

general availability only within a few months ? And

also if the BRR basis shipping period does not

reach 18 months for a new product?

• Answer:

– In both cases, report only the data for the

months available up to and including the 12

month period



40

TL Measurements Questions –Partial Shipping Period

• As per the product category 6.2.1.2.4 Wireless Terminal Software Applications,

is it not necessary to register this product category for a company who

designs and manufactures complex wireless user terminals and also designs

& installs basic control software for the terminals when shipping the terminals,

but does not design nor supply application software after shipment?

• Answer:

– Based on Product Category 5.2, it is correct that a registration would be required,

but based on the question it appears that the proper category would be 6.2.1.2.2

(Smart Phone)

• MHB3.5.2n) says that the organization shall provide its suppliers all necessary

information it possesses to allow those organizations to generate their TL 9000

measurements. In this sentence, do "those organizations" stand for "the

organization's suppliers? I believe YES, otherwise, it will cause a confusion.

• Answer:

– Yes, "those organizations" means "those suppliers".



41

TL Measurements –Other Questions:

Measurements Questions –Linked Certifications

• My company and its subsidiary company are now on the way to get TL 9000

certifications in parallel, that means to get different certifications. And they have

already got different TL 9000 ID's for RMS registration. The main function of my

company is design and development of wireless user terminals and the subsidiary

company is the production function of the terminals. Some of the measurement data

of these two companies are overlapped. So, the subsidiary company would like to

submit Proxy-xxxx's for those measurement data using the TL ID of its parent

company(my company), for the preceding data submissions(including the three

month data submission) before successful completion of the certification. The

certification process of the parent company is still on-going(not completed). Will it

be allowed for the subsidiary to use Proxies during both company are in on-going

certification process? If it is not allowed, there would occur a discrepancy of TL

9000 measurement statistics because of the duplicated data submissions. Please

give me advice or opinion to this situation.

• Answer:

- If they are the same company, registrations can be linked. If there are multiple

registrations where one registration wants to use data from the other registration(s),

they can be linked. See section 4.2.9. The other registration(s) would need to

provide the receiving registration with DSRs that can be viewed during an audit.

This is a useful with common data systems.Copyright 2009 QuEST Forum. All Rights ReservedCopyright


42

DATA SUBMISSIONQUESTIONS



43

Data Submission Questions- Initial Certification

• Do the three consecutive months of data imply measurement

data as defined in the Measurement Handbook must be

submitted to the TL 9000 Measurement Administrator?

– Yes – Data must be submitted to achieve certification

• As for the three consecutive months data, should the data be

submitted month by month?, or submit at once for three

consecutive month?

– Either way is acceptable initially for certification. Going forward

must be monthly

• Shall an organization submit the Data Submission Receipts for

three consecutive month to the certification body before the

initiation of the certification audit?

– CB has access to them once set up in the registration profile



44

Data Submission Questions- Customer Data

• There is a concern about that we cannot submit data without

the permission of customer because some of the field data are

customer's own.

– There should be no concern. All data submissions are anonymous.

Reports also don’t show submissions.

• If a customer does not supply data for a specific measurement,

can it be exempted?

– You are obligated to report data for all customers where you have

confidence in their data. . If you do not have confidence in the

data from one or more customers you exclude those customers. If

you don't have confidence in data from all customers for a

particular month, you would report 0/0.

• How does an organization, who supply OEM products to the

specific customers, submit the measurement data?

– Report data in the organization’s product categories



45


Repository Systems

- Identities of Companies

- Contact Information

- Certification Body

- Scope of Registration and

Registration Status

- No identities (anonymous

reporting)

- Data segmented by product

category and reporting month

RMS MRS

Registration Management

System (RMS)

PUBLIC information used to manage TL

9000 Registrations

Measurement Repository

System (MRS)

PRIVATE - Database of TL 9000

Registered Organizations DATA

47

Secure Data Submission Process

Secure Network

Internet

SwitchFront-end

Web Server

Registration

Management

System (RMS)

Measurements

Repository

System (MRS)

Network Firewall

Encrypted Data

Processed by

Software with

No Human

Intervention

Raw Data Stored in

Encryption Format

Company Data

Encrypted Submission

HTTPS /

Encrypted Tunnel Connection

5/6/2015Copyright 2015 QuEST Forum. All Rights

Reserved 47

Data Submission Questions- Other

• As for the product delivery, the delivery to the end customer is

done through our sales company which is out of the

certification site. In this case, can we regard the sales company

as customer, because we control delivery time for the sales

company？– Yes

• If, for example, we have data for three models of a product

family, shall we submit the data to sum up all?, or separately?

– Either way. In the end all data is aggregated anyway

• How do you annualize data for the product that does not reach

12 months after the first shipment?

– Monthly and smoothed data is annualized by the Afactor.



48

Data Submission Questions- Other

• What is the reason why the field data and delivery data for the

products submission is required for the products before

certification is completed? The purpose of measurement data

submission under TL 9000 QMS system should be to monitor

the field data of the product and to improve the QMS by PDCA

cycle. So, this should be after completion of certification. Why

data submission needed before certification?

• Answer:

– The external assessor must verify that processes are in place to

collect, validate and submit the required data.



49

Data Submission Questions -Advisories

• What is the purpose of advisories?

• What shall an organization do if an advisory is found

in the DSR? Does it need to prepare evidences for

each Advisory to prove if any problem exists or not?

• How are the Advisories defined and how are the

conditions set to issue Advisories?



50

Copyright © 2014 QuEST Forum

Measurements Advisories (page 1)

QuEST Forum strives to continually improve the value of the data provided through TL 9000 Performance Data Reports, which include industry statistic trends (Industry Average, Monthly Average, Best-in-Class, Worst-in-Class) as applicable for each product category and TL 9000 measurement. The more accurate and complete the data, the more useful it is to TL 9000 certified organizations and QuEST Forum members.

A QuEST Forum initiative to further enhance data entry checking to identify and correct potential data errors before inclusion in the Measurements Repository System maintained by The University of Texas at Dallas, was completed in 2009. The result of this initiative was an “Advisor” software application that notifies TL 9000 data submitters with “advisories” indicating a potential data integrity issue that should be investigated. The existence of an advisory does not prevent the data from being successfully submitted. However, Organizations are expected to review any advisories identified and verify integrity of the submitted data. If errors are found, the Organization’s data is required to be corrected and resubmitted. Also, during registration and surveillance audits, the Certification Body will be expected to follow-up with the organization to insure data integrity, as appropriate, for any measurement flagged with an advisory.


Measurements Advisories (page 2)

The “Advisor” checks look at data across measurements within submissions and across submissions over time to better validate the integrity of the organization’s submission. Advisories will be highlighted on the summary status of Data Submission Receipts (DSRs). Each advisory is assigned an integer identifier, and is briefly described at the end of the DSR. Some advisories may be common to all measurements , while others may apply to certain measurements. After investigation of a potential problem indicated by an advisory, it is very possible that the organization may find the data isindeed correct and no resubmission is required. For example, when multiple measurements in the same submission have the same Normalization Unit (NU) defined in Table A-3 of the Product Category Table, if the measurement submission includes differing values for the denominators (i.e. NU’s) for those measurements, an advisory will be generated. However, this difference may be legitimate (e.g. for NPR and SONE where in a particular product category customers don’t report outage data, but do report problems). In cases where advisory investigation does identify a problem, the organization will correct it, resubmit and maintain records of this process for review in registration and surveillance audits.

More detailed information on the Advisories can be found in the paper referenced on page:

http://tl9000.org/alerts/data_submissions.html

http://tl9000.org/alerts/data_submissions.html


Advisory # 1 - The calculated Measurement over the smoothed period is perfect – no longer applied as of August 2011Advisory # 2 - This submission is the same as month, year

Advisory # 3 - Normalization Units differ across measurements that should be the same per Table A-2 – no longer applied as of July 2014Advisory # 4 - Normalization Units changed greater than 25% from the prior monthAdvisory # 5 - Data Element >150% of the highest value reported over the previous 11 monthsAdvisory # 6 - Data Element <50% of the lowest value reported over the previous 11 monthsAdvisory # 7 - Data Element >125% of the highest value reported over the previous 11 months

Advisory # 8 - Data Element <75% of the lowest value reported over the previous 11 months

Advisory # 9 - Not yet Assigned

Advisory # 10 - Data Element >120% of the highest value

reported over the previous 11 months

Advisory # 11 - Data Element <80% of the lowest value

reported over the previous 11 months


reported over the previous 11 months and is 3x the average value in the same 11 months – reduced from 4x as of August 2011


reported over the previous 11 months and is 3x the average value in the same 11 months – reduced from 4x as of August 2011

Advisory # 14 - Normalization Units are the same across measurements that should be different per Table A-2

Advisory # 15 - “Exempt” has been submitted for a measurement not listed as exempt on the organization’s TL 9000 profile - added August 2011

Measurements Advisories

This list of advisories is current as of July 28, 2014

For the current list see DSR Advisories referenced on page: http://tl9000.org/alerts/data_submissions.html


Advisory # 16 – Downtime reported less than minimum expected

Advisory # 17 – Downtime reported less than minimum expected

Advisory # 18 - Outage frequency reported is less than minimum expected


Advisory # 20 - NPRs should normally be greater than NEOs

Advisory # 21 - NEOs and SOs should normally be equal

Advisory # 22 - Downtown reported less than minimum expected


Advisory # 24 - Input value greater than 95% of maximum value expected

Measurements Advisories – Added July 2014

This list of advisories is current as of July 28, 2014

For the current list see DSR Advisories referenced on page: http://tl9000.org/alerts/data_submissions.html


What is expected of CB Auditors?

Adhere to the QuEST Forum Code of Practice for TL 9000 Registrars

Assure there is a documented system in place to collect, validate and report TL 9000 measurements

Investigate and validate that the system in place is implemented and effective

Ensure that the rational for all measurements exceptions are valid

Verify that Scope and Product Categories are appropriate for the Registration

Ensure that measurements submissions are made for all product categories under the scope of registration

Ensure that measurements submissions are made on time

Understand measurements advisories

Ensure that all measurements advisories are thoroughly investigated

PDR ImprovementExecutive BoardStrategic Initiative

Improve Member Satisfaction with Performance Data

And Performance Data Reports (PDRs).

EB SI #5 - PDR Improvement Project OverviewTasks

• Data Anomaly

Investigations

• Advisors/Error Checks

• Product Category

Verification Process

• Industry Reports

• Normalization Floors

• “How To” White Paper

Updates

• Continued Workshops

At BPC’s

• Regional Webinar

Delivery

• Agreement with EB #3

For Goal/Path Forward

•Identify Holes

•Identify Potential

Registrants

• Identify “Similar” PC’s

• PC Aggregation

Initiative #1

Improve PDR Accuracy,

Create Industry Reports

Initiative #2

Maximize PDR

Usage Value

Initiative #3

“Fill The Pipe”, i.e.

Expand PDR Scope

Su

rvey

Feed

back

Init

iati

ves

TL 9000 Industry Reports



58

• Network Field Maintenance Study

• Wireless Infrastructure Product Family Trends

• Then and Now: On-Time Delivery Measurement Study

2009 vs. 2013

• Repair Service Trends Study

• Network Operator Quality Improvement Trends Study

• Quality Improvements in Simple and Complex Wireless

Devices Study

• Edge Router Product Category Trends Study

• Wireless Return Rate Study

• Problem Report Fix Response Time Study

• On-Time Delivery Measurement Study

All Ten Industry Reports can be found on the

QuEST Forum Web Site

http://questforum.org/resources/industry_reports.html

Network Field Maintenance Study



59

7.3.1 Product Category

Measurements studied:

• NPR4 (Problem Reports)

• FRT4 (Fix Response Time)

• OFR4 (Overdue Fix Response Time)

• OTI (On Time Service)

• SQ (Service Quality)

• SSO (Support Service Caused Outages)

60

Problem Report Industry Average

Industry Average has Strong Performance

showing an improvement trend

0

0.0005

0.001

0.0015

0.002

0.0025

0.003

Nu

mb

er

Pro

ble

m R

ep

ort

s p

er

Se

rvic

e

Month

7.3.1 NPR4 Industry Average

7.3.1-IA

Linear (7.3.1-IA)

61

Fix Response Time Industry Average

Better than 95% in 2014

Showing an improvement trend

50

55

60

65

70

75

80

85

90

95

100

% F

ixe

d O

n T

ime

Month

7.3.1 FRT4 Industry Average

7.3.1-IA

Linear (7.3.1-IA)

62

OFR4 Industry Average

Consistent Performance

50

55

60

65

70

75

80

85

90

95

100

% O

verd

ue

Fix

Res

po

nse

Tim

e

Month

7.3.1 OFR4 Industry Average

7.3.1-IA

Linear (7.3.1-IA)

63

On Time Service Industry Average

Approaching 100% in 2014

50

55

60

65

70

75

80

85

90

95

100

% O

n T

ime

Se

rvic

e

Month

7.3.1 On Time Service Industry Average

7.3.1-IA

Linear (7.3.1-IA)

64

Service Quality Industry Average

Improving to less than 0.5%

0

0.5

1

1.5

2

2.5

% D

efe

ctiv

e S

erv

ice

Tra

nsa

ctio

ns

Month

7.3.1 Service Quality Industry Average

7.3.1-IA

Linear (7.3.1-IA)

65

Support Service Caused Outages Industry Average

SSO Improving in 2014

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

% S

up

po

rt S

erv

ice

Cau

sed

Ou

tage

s

Month

7.3.1. Support Service Caused Outages Industry Average

7.3.1-IA

Linear (7.3.1-IA)

Wireless Study



66

Three Wireless Product Categories

• 3.3.2.1-Basic (2G) Base Transceiver Systems

• 3.3.2.2-Advanced (3G) Base Transceiver Systems

• 3.3.2.3-(4G) LTE Base Transceiver Systems

Measurements studied:

• YRR (One Year Return Rate)

• SPR2 (Major Software Problem Reports)

• FRT2 (Major Fix Response Time)

• OTI (On Time Item Delivery)

67

Wireless Study

50% Decrease in 3G Return Rate

GO

OD

0

0.5

1

1.5

2

2.5

% R

etu

rn R

ate

Month

YRR Industry Average

3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G)

YRR Industry Average

68

Wireless Study

2G and 3G at a mature level. 4G Trend Improving

0

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

0.0007

0.0008

Maj

or

Soft

war

e P

rob

lem

Re

po

rts

Pe

r N

etw

ork

Ele

me

nt

pe

r Y

ear

Month

SPR2 Industry Average

3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G)

Major Software Problem Report Industry Average

69

Wireless Study

4G SPR2 IA Improving; 2G and 3G have mature IA

00.00010.00020.00030.00040.00050.00060.00070.00080.0009

Maj

or

Soft

war

e P

rob

lem

Re

po

rts

Pe

r N

etw

ork

Ele

me

nt

pe

r Y

ear

Month

SPR2 Industry Average Trends

Linear (3.3.2.1 (2G)) Linear (3.3.2.2 (3G)) Linear (3.3.2.3 (4G))

Major Software Problem Report Industry Average

70

Wireless Study

4G Approaching 2G and 3G Major Fix Response Time

GO

OD

0

10

20

30

40

50

60

70

80

90

100

% F

ix R

esp

on

se T

ime

Month

FRT2 Industry Average

3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G)

Major Software Fix Response Time Industry Average

71

Wireless Study

OTI Trend lines at about 70%. 3G & 4G Degrading

0102030405060708090

100

% O

n T

ime

Ite

ms

Month

OTI Industry Average Trends

Linear (3.3.2.1 (2G)) Linear (3.3.2.2 (3G)) Linear (3.3.2.3 (4G))

On Time Item Delivery Industry Average

72

Wireless Study

4G YRR, SPR2, FRT2 maturing rivaling 2G and 3G;

OTI degrading for 3G and 4G and not in line with customer expectations

• 2G – Shows most mature performance

as expected

• 3G and 4G:

•SPR2 improving

•FRT2 improving

•OTI degrading

Investigation Letter Trial Example (PC 1.2.2)

73

Dear TL 9000 Registration Administrator,

It has been brought to the attention of the QuEST Forum Sub-team charged with monitoring and improving the

TL 9000 data reports that there is a problem with the return rate reporting for Product Category 1.2.2 Access

Multi-Service. The specific issue is there is at least one organization reporting return rates that are extremely

high. This is evident in the Worst-In-Class values for all four of the return rates, but is most evident for ERI

where the value is running over 200% per year. This would mean that on average each new unit is returned

twice within six months of shipment. Since it is highly unlikely this could be true, we are asking all

organizations with TL 9000 Certification in this product category to verify the FR data that they are submitting

before your next data submittal.

Specifically, please verify the following:

1. Only returns received during the month being reported are being counted

2. Each return is included in one and only one basis population

3. The is no factor applied to the number of returns, the actual count is being reported

4. The denominator includes all units shipped that belong to the basis population

If you find there has been an error in your data reporting, you are required to correct and resubmit your data

for up to the last 24 months.

This letter is being sent to all organizations certified in Product Category 1.2.2 and to your Certification

Bodies. The Certification Bodies are asked to confirm that the organization has verified their TL 9000 return

rate data during their next audit. In the interim, the organization shall confirm to QuEST Forum that this

verification has been performed within 30 days of receipt of this e-mail. Failure to do so will result in the

suspension of your TL 9000 Certification until your confirmation has been received.

If you have any questions, please use the “Contact Us” function on the tl9000.org web site.

Thank you for your co-operation in this matter. Correcting this issue is essential in order to provide useful

comparable return rate data for the organizations in this product category.

QuEST Forum TL 9000 Administrator

1.2.2 Example Continued –Investigation Letters Resulted in ERIRates Returning to Typical Levels



7424 letters sent, plus reminders as needed.

A few acknowledged issues and resubmitted data.

0

20

40

60

80

100

120

5/2011 8/2011 11/2011 2/2012 5/2012 8/2012 11/2012

1.2.2 ERI

ERI Moved to typical

levels after

resubmissions

2014 Investigations Conducted

75

Product

Category Name Issue # of Certs. # Responded Suspended

Dropped

Cert

Still

Suspended

Reported

Corrected

Data

3.2.5

Fiber to the

User FR - WIC 18 17 0 1 0 1

7.6.1

Procurement

Services OTI - WIC 16 15 0 1 0 1

3.1.1.1.1

Metallic

Conductor

Cable NPRs 19 17 4 0 2 4

3.1.1.2.1

Fiber Optic

Cable NPRs 28 26 6 0 2 7

Tracking of Investigation Responses is Currently Done Manually.

The Team Is Considering Mechanized Methods Going Forward.

No Response Has

Resulted in

Suspensions!

Asking Companies to Check Their Data When Anomalies Occur, Can Improve Data Accuracy

76

0

10

20

30

40

50

60

70

80

90

100

4/2

012

5/2

012

6/2

012

7/2

012

8/2

012

9/2

012

10/2

012

11/2

012

12/2

012

1/2

013

2/2

013

3/2

013

4/2

013

5/2

013

6/2

013

7/2

013

8/2

013

9/2

013

10/2

013

11/2

013

12/2

013

1/2

014

2/2

014

3/2

014

Change in 7.6.1 OTI Worst in Class

NEW WorstInClass

OLD WorstInClass

While Anomalies can occur for valid reasons, if one seems completely

unreasonable, report it via “Contact Us” and the PDR Team will investigate.

77

Correct?

Category added to profile

QF approval noted in profile

Pop-up asks for info

Info Complete?

No

Yes

Verification team notified

Info Sufficient?

No

Yes

Request for more info

No

Yes

Feedback to organization

Info received

1) Description

2) Product name and/or model number

(hardware/software products –

Families 1-6 and 8)

3) Primary function (hardware/software

products – Families 1-6 and 8)

4) Competitors products name or

model numbers (hardware/software

products– Families 1-6 and 8)

5) Link(s) to material on web

6) Attach brochure(s)

A New Product

Category Approval

Process Has Been

Approved to Improve

Product Category

Selection Accuracy –

Useful PDR Whitepapers

• PDR Usage

• PDR Measurements Outputs and

Calculations

• Product Category Selection

• TL 9000 Data Submission

Methods



78

http://tl9000.org/pdrs/documents/PDR_Usage.pdf

http://tl9000.org/pdrs/documents/TL_9000_Measurement_Outputs_and_Calculations.pdf

http://tl9000.org/registration/documents/Product_Category_Selection_Validation_Guidelines.pdf

http://tl9000.org/registration/documents/TL_9000_Data_Submission_Methods.pdf

QuEST FORUMMember Value Study

2014 Results

© 2014 Walker Information - proprietary and confidential 80

2014 MEMBER VALUE SURVEY RESULTS

Response Rate

Invitations Sent1871

Completed Surveys273

Response Rate15%

Companies Identified205

CompaniesParticipated125

Company Response61%

2014 INDIVIDUAL RESPONSE

2014 COMPANY RESPONSE

1720 Invitations Sent 258 Completed Surveys 15% Response Rate

187 Companies Identified 112 Companies Participated 60% Company Response

2013 COMPARISON

2013 COMPARISON



Overall Membership Satisfaction

The higher the impact number, the more impact that measure

has on the overall question shown at the top of the graph.



Performance Data Reports

The higher the impact number, the more impact that measure

has on the overall question shown at the top of the graph.



Performance Data Reports

QuEST Forum Member Value Survey PDR Specific Data Trends

2008 2009 2010 2011 2012 2013 2014

Overall PDR Satisfaction 49 60 63 70 67 71 70

Useful/Relevant PDR Data 48 54 62 68 66 69 69

Identifying Improvement Areas 45 52 56 58 55 60 62

Easy To Use 54 58 60 71 63 68 69

Providing Benchmarking Data 53 56 64 73 67 67 65

Providing Accurate Data 47 52 61 67 67 64 71

Timely Data 65 66 73 84 79 79 82

Supporting Continual Improvement 70

Aiding in Supplier Management 51

Top 2 Box Results

2014 n=152

2013 n=161

2012 n=156

2011 n=172

Significant Improvement in “Providing Accurate Data” while the

2014 PDR Overall Satisfaction Score holds steady near all time highs.

Efforts are paying off - but more needs to be done

Contact Information:

Ken KoffmanJDSU+1 (240) [email protected]

John Wronka

Alcatel-Lucent

+1 (727) 204-9508

[email protected]

mailto:[email protected]

tl 9000 measurements and requirements interactive workshop

Technology