1248959559

8/7/2019 1248959559

1/11

Statistics & graphics for the laboratory 82

Method evaluation (validation) and method comparison

Introduction

The analytical quality triangle

Purpose of method evaluation

Performance standards

Performance characteristics of a method

Precision

Limit of detection

Working range

Experiments

Making decisions

Method evaluation strategies

References

Content

8/7/2019 1248959559

2/11


Objectives of the course

Be able to efficiently manage analytical quality by applying a concept that

integrates specification, creation, and control of analytical quality.

Understand that management of analytical quality needs communication with the

"outside" partners (note: this should be a two-way communication).

Accomplish means that allow you to anticipate future quality needs in an early

stage.

Analytical quality in the medical laboratory

An integrated approach

Introduction

Specification Creation

of quality of quality

Profession Labor- Manufacturer

Legislation atory

External quality-

assessment

Control

of quality

Specifica ion of quality Pa en

Pro e on Physician

Regua ionLabora ory

C eation of quality

Manu acturer

Laboratory

Control of quality

Interna: Laboratory

Externa: EQA

8/7/2019 1248959559

3/11


Method evaluation Place in the overall analytical quality

The analytical quality triangleFor valid measurements

the analytical quality triangle!

Purpose of method evaluation

J. Westgard

The inner, hidden, deeper, secret meaning of a method evaluation/validation

= ERROR ASSESSMENT

From: J.O. Westgard, Basic method validation, Westgard Quality Corporation

1999, pp 250. www.westgard.com

Carey et al$

Evaluate performance & make decisions about performance.

Apply a clinical perspective to the whole task!

Requirements

Experimental protocols to estimate performance reliably ("Error assessment")

Standards (specifications, claims) for acceptable performance

Criteria for comparing estimated performance with performance standards

$Carey RN, Garber CC, Koch DD. Concepts and practices in the evaluation of

laboratory methods. Workshop, AACC 48th Annual Meeting, Chicago (IL), July 28,

1996.

Introduction

Method evaluation/comparison

Improvement

ssuranceControl

Qualitycreation

Q tyspecification

P nning

Q itymanagement

Inst m ntChemist y

State f-a tEx e t

Bi gyC inical Meth d eval ati n/

c m a is n

Im vementAssu

anceC nt l

Qualitycreation

Qualityspecification

Planning

Qualitymanagement

Inst umentChemist y

State- f-a tEx e t

Bi l gyClinical

8/7/2019 1248959559

4/11


WHAT is validation?

Validation is the confirmation, through the provision of objective evidence, that

requirements for a specific intended use or application have been fulfilled (ISO

9000).

We see, from this definition, that we have to

specify the intended use of a method,

define performance requirements,

provide data from validation experiments (objective evidence), and

interprete the validation data (confirmation that requirements have been fulfilled).

WHICH type of performance requirements (specifications) exist?

Performance requirements can be statistical, analytical, or application-

driven/regulatory.

Statistical and analytical specifications are most useful for method evaluation.

Application-driven/regulatory specifications are used for validation. Some

examples are given in the table below.

WHICH performance characteristics exist?

We have seen that we have to specify performance requirements for a validation.

These requirements refer to the following performance charateristics of an

analytical method:

Imprecision

Limit of detection

Working range Linearity

Recovery

Interference/Specificity

Total error (method comparison)

[Robustness/Ruggedness]: will not be addressed in this book.

Introduction

Performance requirements (specifications)

Statistical

t-test: P 0.05

F-test: P 0.05

Analytical

Bias e

Calibration tolerance

CV e stable CV

Application-driven#

Bias e 3%

CV e 3%

#Cholesterol (National Cholesterol Education Program)

8/7/2019 1248959559

5/11


WHICH experiments do we have to perform?

The experiments we have to perform depend on the performance characteristic

we want to validate. For the estimation of method imprecision, for example, we

need to perform repeated measurements with a stable sample. However, there isno agreement over the various application fields of analytical methods about the

design of such experiments. In this book, we will mainly refer to the experimental

protocols from the Clinical and Laboratory Standards Institute (CLSI). The table

below gives an overview about typical experiments to be performed during a

method validation study.

Introduction

Performance

chracteristic

Samples

Measurements

Imprecision IQC-samples; no targetn = 20 (repetition over several days)

LoD Lo Blank; Low sample

n = 20 (repetition over several days)

Linearity 5 related samples/-calibrators (mix); no target

n = 4 (repetition within day)

Working range See: Imprecision/Linearity

Interference Samples: Interferent spike & control (no target)

n = 4 (repetition within day)

Recovery

(Accuracy Trueness)

Samples: Known analyte spike & control or

certified reference materials (CRM)

n = 4 - 5 (repetition over several days)

Total error

(method comparison

u40 samples (target by reference method)

n = 1 or 2 (measurement in one or several days)

IQC: Internal Quality Control; LoD: limit of detection; LoQ: limit of quantitation

8/7/2019 1248959559

6/11


HOW do we make decisions?

When we have created data, we have to decide whether they fulfill the

requirements that have been selected for the application of the method "for a

specific intended use". Currently, it is common practice to make decisions withoutconsidering confidence intervals or statistical significance testing. Modern

interpretation of analytical data, however, requires the use of confidence

intervals/statistical significance testing.These two approaches are compared in the

table below for the case of a recovery experiment.

In the old approach, we compare one naked number with the specification. This

approach misses the information on the number of measurements that have beenperformed and the imprecision of the method. If we would repeat the validation,

we easily could obtain a recovery estimate of 80%, for example. Therefore,

decision-making should be statistics-based. This is by applying a formal statistical

test or by interpreting the confidence interval of an experimental estimate.

Statistics-based decision Importance of the test-value

(= requirement, specification)

When we make statistics-based decisions, the selection of the test value will

depend on the type of requirement we apply (statist ical, analytical, validation).

Statistical Statistical test versus Null-hypothesis (F-test, t-test, 95% confidence-

intervals, ): Bias = 0; Slope = 1; Intercept = 0; etc.

Analytical

Statistical test versus estimate of stable performance (F-test, t-test, 95%

confidence-intervals, etc.): Bias e calibration tolerance; etc.

Validation case (application-driven; specific intended use)

Statistical test versus validation limit (F-test, t-test, 95% confidence-intervals,

etc.): CVexp e CVmax; Biasexp e Biasmax; etc.

Nevertheless, in all three situations, we apply the same type of statistical tests.

Introduction

Decision making approaches

Old

Experimental recovery: 90%

Limit: 85 115%

Decision: passed

Modern

Experimental recovery: 90%

Confidence interval: 11%(with n = 4 and CV = 7%)

Limit: 85 115%

Decision: fail

(90 11 = 79%, exceeds 85%)

Action: increase n or reduce CV

8/7/2019 1248959559

7/11


Evaluation strategies

Strategies

By comparison "with a reference"

By the method itself

Evaluation strategy: By comparison "with a reference"

"Traditional" external quality assessment

"Traditional" Certified Reference Materials

IQC materials with target

Method comparison with "true" reference method Preferred

="Complete picture"-type

Evaluation strategy: By the method itself

Imprecision

Limit of detection (LoD)

Working range

Linearity

Recovery

Interference

Specificity

Shift/drift/Carryover

Ruggedness

=

"Mosaic"-type


8/7/2019 1248959559

8/11



Evaluation strategy "complete picture"

Advantages

1 experiment Gives the complete picture

Beware

The interpretation heavily depends on the quality of the comparison method

& the samples used!

Disadvantages

The reason of errrors may remain unknown

Apply "mosaic-type"

Specific purposes of method comparisons

Sufficient quality of a test method?

Comparison method is a reference method

Are 2 methods equivalent?

The 2 methods are of the same hierarchy

Recalibration of the test method

The comparison method is of higher or of the same hierarchy

Note on the calibration (adaptation) of a method via method comparison studies:

Be aware that minimum criteria have to be fulfilled!

Reliable outcome Not possible Unreliable outcome


0

300

600

900

0 300 600 900

e

eren

e

n

ol

ou

tine

n

ol

= 0,97

+ 54r = 0,9

580

300

600

900

0 300 600 900

e

eren

e

n

ol

ou

tine

n

ol

ubgrou

= 1,17

7,4r = 0,9939

0

300

600

900

0 300 600 900

e

eren

e

n

ol

ou

tinen

ol

8/7/2019 1248959559

9/11



Evaluation strategy "mosaic"

Evaluate the performance characteristics of a method separately

Imprecision LoD

Interferences

etc

Try to put together the complete picture from these "mosaic stones".

Recommended reference

Westgard JO. Basic method validation. Madison (WI): Westgard Quality

Corporation, 1999, 250pp.

But be aware: Makes simplifications often confidence limits are missing!

Advantages

Detailed evaluation of the method

For a commercial test: manufacturers' task

Task of the lab: performance verification

Can be done with the method itself

Disadvantages

Time-consuming experiments

Are the results reliable?SD with IQC materials

LoD from SD of blank

Linearity/recovery = trueness

Interferences: all tested/effect of combinations

Matrix effects of investigated materials

Can we establish the complete picture from the mosaic stones?

May be unnecessary for the laboratory!


"Mosaic stones"

8/7/2019 1248959559

10/11


Evaluation strategy

Westgard terminology

The practice

We take performance standards

From "biology" (westgard.com/biodatabase1.htm)

Manufacturers' specifications

We use xperimental protocols

CLSI protocols

"Adapted" CLSI protocols (LoD, recovery)

We compare the experimental estimates with the performance standards

Statistics/Graphics

Note

Method evaluation/validation is detailed in the book:

Method validation with confidence.


et!

o"

#

o$

%

arison&

e#

over'

(

ro%

ortional

et!

o"

#

o$

%

arison)

nter0

eren#

e

stu"

ies1

onstant

2

u%

li#

ates3

native sa

$

%

les&

e%

eat#

ontrolsa

$

%

les&

an"

o$

E4

% eri$

ent5

6

ele#

tive 5 51

o$

%

le4

5

7

' %

e o0

error

et!

o"

#

o$

%

arison&

e#

over'

(

ro%

ortional

et!

o"

#

o$

%

arison)

nter0

eren#

e

stu"

ies1

onstant

2

u%

li#

ates3

native sa

$

%

les&

e%

eat#

ontrolsa

$

%

les&

an"

o$

E4

% eri$

ent5

6

ele#

tive 5 51

o$

%

le4

5

7

' %

e o0

error

5

osai#

5 5

1

o$

%

lete%

i#

ture 5

8/7/2019 1248959559

11/11


References

Book

J.O. Westgard, Basic method validation, Westgard Quality Corporation 1999, pp

250.

CLSI protocols

Evaluation of Precision Performance of Clinical Chemistry Devices; Approved

guideline. CLSI Document EP5-A. Wayne, PA: CLSI 1999.

Evaluation of the linearity of quantitative measurement procedures: A statistical

approach; Approved guideline. CLSI Document EP6-A. Wayne, PA: CLSI 2003.

Interference testing in clinical chemistry; Approved guideline. CLSI Document

EP7-A. Wayne, PA: CLSI 2002.

Method comparison and bias estimation using patient samples; Approved

guideline. CLSI Document EP9-A2. Wayne, PA: CLSI 2002.

Preliminary evaluation of quantitative clinical laboratory methods; Approved

guideline. CLSI Document EP10-A2. Wayne, PA: CLSI 2002.

Protocols for Determination of Limits of Quantitation. CLSI Document EP17.

Wayne, PA: CLSI in preparation.

Related CLSI protocols

Evaluation of Matrix Effects; Approved guideline. CLSI Document EP14-A.

Wayne, PA: CLSI 2001.

User Demonstration of Performance for Precision and Accuracy; Approved


Estimation of Total Analytical Error for Clinical Laboratory Methods; Approved


Other

Vassault A, et al. Socit Franaise de Biologie Clinique. Protocole de validation

de techniques. Ann Biol Clin 1986;44:686-719 (english: 720-45).

Vassault A, et al. Socit Franaise de Biologie Clinique. Analyses de biologie

mdicale: spcifications et normes dacceptabilit lusage de la validation de

techniques. Ann Biol Clin 1999;57:685-95.Dewitte K, Stckl D, Van de Velde M, Thienpont LM. Evaluation of intrinsic and

routine quality of serum total magnesium measurement. Clin Chim Acta

2000;292:55-68.

References

1248959559

Documents