Quality Metrics

Presented by: Karen Ginsbury

For IFF, Denmark

November 2016

What is Quality

• Much more than the quality unit or QA department

• It is a mindset

What is a Quality System

• And who owns it

The Business Case for Metrics

Quality is…

• Meeting all the requirements all the time

• Therefore – not meeting the requirements at any time is not quality

• Don’t use “high” or “low” quality – it cheapens the term

Waivers

• the act of intentionally [and voluntarily] relinquishing or abandoning a known right, claim, or privilege

• So a “planned” deviation or “temporary” change or “special” release is a waiver of “quality” which is meeting requirements (all of them all the time)

Metrics are about

• Changing behavior

• Data overload vs Data Collection Plan

• Everything and nothing is important

• 100s of KPIs

• Nothing changes

Be selective

• Before you choose a KPI ask yourself:

– What do I want to change / improve

– What behavior do I dislike

– What can I measure

– What is the target and reward that will encourage the behavior I want without creating untoward consequences

What DI related behavior

• Would you like to change?

DI Guidance

• MHRA GMP

• WHO

• FDA Q&A

• MHRA GxP

• PIC/s

• EMA Q&A

• January 2015

• September 2015

• April 2016

• July, 2016

• August 10, 2016

• August 11, 2016

What are Metrics

• Measurements

• Monitoring

• A method of measuring something, or the results obtained from this

• Any type of measurement used to gauge some quantifiable component of a company's performance

Measurement is the basis of the Quality Process

• What do we measure

• What could we measure

• What should we measure

• What can we measure

How many metrics

• BE CAREFUL – if everything is important then nothing is important

About Measurement

• “Measurement is the first step that leads to control and eventually to improvement

• If you can’t measure something, you can’t understand it

• If you can’t understand it, you can’t control it

• If you can’t control it, you can’t improve it.”

H. James Harrington

Measureable…But Meaningful

• If a measurement matters at all, it is because it must have some conceivable effect on decisions and behavior

• If we can't identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value

Douglas W. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business

Write Down Three Metrics

Leading (Proactive / Prevent)1. Right first time2. Preventive actions – change requests

and Risk Management and control strategy

3. Product and Process performance monitoring and continuous process verificationcontrol parameters)Monitoring CPPs (critical

4. PQR5. Surveys – talking to operators,

feedback6. Quality metrics and yields7. Effectiveness checks8. Internal audits / self inspection

Lagging (Reactive)1. Complaints2. Corrective actions3. Deviations – don’t use numbers

measure lead time for handling and implementing corrective actions and recurrence

4. Overdue items5. Recalls6. Destructions7. Regulatory and customer audit

findings8. Regulatory actions9. Back orders and / or unfilled orders10. OOS, OOT, questionable and

laboratory error11. Reprocessing, rework12. (Change requests)

FAQs: How Many Metrics

• Three is good

• Is five better

• Is 25 better still?

• Should we use CPPs and CQAs – we have 100s of them?

FAQs: How Many Metrics

• Target high level indicators(stay out of the weeds)

• Write down some for a parenteral product

High Level Indicators for Sterile Products

• (Potency)

• Environmental Monitoring Data – TrendsSterility Assurance Levels – could be a summary of a whole lot of individual metrics which are weighted and give the overall picture

• Training / competency – by observation of personnel doing their jobs

• (Homogeneity of product)

Prerequisites – Quality Culture

Quality Non-Quality

Voting Time

• Does your company have:

– A quality culture

– A non-quality culture

– A mixture (we want to do the right thing but…)

PrerequisitesEngaging Most Senior Management

• How can you get management on board?

• Measurements (of what)

QS Maturity GridQUALITY MANAGEMENT MATURITY GRID

Rater _______________________________________ Job Title: __________________________________

Measurement Category

Stage 1Uncertainty

Stage IIAwakening

Stage IIIEnlightenment

Stage IVWisdom

Stage VCertainty

Management understanding and attitude

No comprehension of quality as a management tool. Tend to blame quality department for “quality problems.”

Recognize that quality management may be of value but not willing to provide money or time to make it happen.

While going through quality improvement program, learn more about quality management; becoming supportive and helpful.

Participating. Understand absolutes of quality management. Recognize personal role in continuing improvement.

Consider quality management an essential part of company system.

Quality organization status

Quality is hidden in manufacturing or engineering departments. Inspection probably not part of the organization. Emphasis is on appraisal and sorting.

A stronger quality leader is appointed but main emphasis is still on appraisal and moving the product.

Quality department reports to top management, all appraisal is incorporated and manager has role in management of company.

Quality manager is an officer of the company; effective status reporting; preventive action. Involved with consumer affairs and special assignments.

Quality Manager on board of directors. Prevention is main concern. Quality is a thought leader.

QS Maturity Grid

Measurement Category

Stage 1Uncertainty

Stage IIAwakening

Stage IIIEnlightenment

Stage IVWisdom

Stage VCertainty

Problem handling

Problems are fought as they occur; no resolution; inadequate definition; lots of yelling and accusations.

Teams are set up to attack major problems. Long-range solutions are not solicited.

Corrective action communication established. Problems are faced openly and resolved in an orderly way.

Problems are identified early in their development. Functions are open to suggestion and improvement.

Except in the most unusual cases problems are prevented.

Cost of quality as % of sales

Reported: unknownActual: 20%

Reported: 3%Actual: 18%

Reported: 8%Actual: 12%

Reported: 6.5%Actual: 8%

Reported: 2.5%Actual: 2.5%

Quality improvement actions

No organized activities. No understanding of such activities.

Trying obvious “motivational” short-range efforts.

Implementing the 14-step program with thorough understanding and establishment of each step.

Continuing the 14-step program and starting Make Certain.

Quality improvement is a normal and continued activity.

Summation of company quality posture

“We don’t know why we have problems with quality.”

“Is it absolutely necessary to always have problems with quality?”

“Through management commitment and quality improvement we are identifying & resolving our problems.”

“Defect prevention is a routine part of our operation”.

“We know why we do NOT have problems with quality.”

The Critical Few

• How many metrics

• How to present them

• How often to measure

• What should be considered “improvement”

Data Collection Plans

• Who decides what data to collect?

• How do they make that decision?

• How do they COMMUNICATE the decision?(what are the tools)

• How do they educate for compliance with the data collection plan?

Data Collection Plans

• What data NEEDS to be collected (if all data is important then none is important)

• HOW should it be collected?

• HOW should it be analyzed?

• HOW should it be presented? And by who and to whom?

The wrong metricsUnintended Consequences

• Give examples of KPIs(Key Performance Indicators) which might have unintended consequences…

The wrong metricsUnintended Consequences

• Drive wrong behavior to achieve a metrics target

• Establish too many, overly complex metrics taking resources from daily activities (e.g. walking the floor)

• Compare data that is not consistently defined or interpret single data values / limited data leading to “tampering” – inappropriate responses

• Using metrics as a quality surrogate out of context

Cost of QualityCost of conformance / Non conformance

• Does your company translate non-compliance to costs?

• Do Management Reviews include Euro / Dollar values of losses resulting from deviations, incidents, failure to meet schedules…

• Do you have a policy

• Is the CFO on board?

Scorecards

• Scorecards are applications that show progress toward a strategy, goal or objective using KPIs

• More meaningful than any single metric, they actually indicate the performance of the business

• Scorecards may be part of dashboards

Scorecard - Example

DI Scorecard - Example

Dashboards

• Dashboards are reporting tools that consolidate and arrange numbers, metrics and sometimes scorecards on a single screen

• Usually tailored for a specific role

• Display metrics targeted for a single point of view or department

Dashboards

Statistics for All

• SPC Tools For EveryoneJohn T Burr

Statistics for All

• Competitive market place requires efficiencies

• Reduce the cost of providing our products

• Cultural change in thinking

• Management are team members with workers at all levels

• Understand the basic tools of handling and controlling variation in mass production

• 100% conformance to realistic specifications and requirements as the normal way of doing a job

Statistics for All

• Every member of the workforce needs the basics of statistical process control for non technical people

• How to picture information – so that everyone in a group sees the same thing at the same time

• Competitors who use SPC will make product and give services faster and better

• SPC will allow the company to survive, grow and improve efficiencies (our 20%)

Product vs Process Control

Product

• Is controlled at the end of the process and you can do nothing about it – you have already made the quality –so you test and if it is ok –release and if it is not ok –you SHOULD reject but you probably get into an OOS fight

Process

• Is controlled through understanding where the variability is and controlling critical process parameters such that all three cakes come out the same even when baked by three different students in three different ovens

The toolbox – Fabulous 14 Tools

• Process flow diagram• Cause and effect (fishbone)• Pareto chart• Check sheets• Location diagrams• Tally count• Time plot• Scatter diagram• Histogram• Stem and Leaf• Box and Whisper plot• Regression analysis• Control Chart• Conformance chart

The Power of Walking the Floor

• Thoughts?

The Power of Walking the Floor

• “You see but you do not observe”

• Just stand and watch a production operation and ask questions about what you see

• Talk to operators and LISTEN to their comments

In Conclusion

• Keep it Simple

• Few KEY metrics

• Define data collection, analysis and presentation in a Data Collection Plan with standardized templates

• Management review and leadership of the program

• Follow up and feedback

• Walk the floor and LISTEN

THANK YOU FOR LISTENING!

Metrics

Leading

• First through cycle time• # of changes to production planning

schedule• Yield• No (major) / fewer than last time

observations internal audits• Process capability?• CAPA effectiveness rate ?• Successful changes ?• Equipment usage / planned downtime vs

repairs / malfunctions• Human error rates• Investigation free lots• Organizational health metric (%

temporary workforce, employee satisfaction, % safety, turnover rate)

• Recapitalization as % of asset value rate• Risk management profile changes

Lagging

• Complaints (confirmed?)

• OOS (confirmed?)

• Batch rejections

• Investigations close out rate? (or is it leading) –what about unintended consequences?

• Analytical invalid rate

Metrics for Parenterals

Environment• Media fill failures• EM recovery rateProduct• Once through cycle / no investigation• Batches started vs released• Potency assay trend and process capabilityMarket performance• Stability OOS results• Complaints? PV??

What is a Quality System

• All the processes and documents that ensure quality

• The sum total of the organized arrangements made with the intention of ensuring product quality

• Entropy – the investment of energy to do things right the first time

What is the Role of QA / Quality Unit?

• To set up, monitor, report and continually improve the quality system

• To ensure compliance with the quality system and that the quality system complies with regulatory requirements

• Ensure that there are no contradictions or resolve conflicts

Prerequisites – Quality Culture

Quality• Management meetings have

quality high on the agenda• Quality = schedules = costs• Openness / transparency• Lack of fear• System takes responsibility

rather than burdening individuals

• Quality perceived as a strategic / business advantage

• Respect the customer (internal and external)

Non-Quality• Lack of communication• Lack of openness / reporting• Quality system perceived as a burden /

overhead / “necessary evil”• “I am for quality”• Conflict• Departments don’t pull together with

others – each of each – no understanding of company responsibility

• Fear / irrational firings / yelling and accusations

• Customer is an idiot and a cheapskate and doesn’t know what they want

CultureQuality• Transparency• Honesty• Reporting• Listening• Open / lack of fear• Open, honest and regular

communication• Clear vision and belief in quality• Management leadership• Management sponsorship of

quality initiatives• Listening• Education• Team approach to continuous

improvement• Rewards initiative and prevention

Non Quality• Punishes• Blames• Doesn’t listen• Lack of communication• Hides / buries problems• Rewards fire-fighting• Lacks direction• Management expects

managers to “fight it out”• Trains but doesn’t educate• Cops and robbers