Quality Controlwhat to compare against the validation

templates (see templates in course webpage:

http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/Validation%20Templates%20from%20Redbook%20QA%20Handbook%20vol%20II.pdf

)

1

2

April 7: QA Systems, EPA definitions, PQAOs and common sense – Mike Papp

April 14: Routine Quality Control and Data Management (1-pt QC, flow rate, and instrument stability checks) – Travis Maki

April 21: Audits Overview (NPAP, PEP, Annual PE, Flow Rate Audits) – Jeremy Howe

April 28: Calculating Bias and Precision and AQS reports – Angie Shatas

May 5: 40 CFR 58 App. A- Gaseous Pollutants – Glenn Gehring

May 12: 40 CFR 58 App. A- Ozone – Brenda Jarrell May 19: 40 CFR 58 App. A- PM filter and continuous

methods –Brandy Toft

Overall Course Overview:

Measuring Comparing with MQO Graphing it/ watching for

its drift Fixing it when needed

QC: An Ongoing System

Everything must be documented and, when significant, reported

3

Instruments operating stably Measure what you can, verify it is:

◦ Within limits◦ Not slowly drifting up or down

Verify you are meeting your QAPP specs QAPP specs assess bias and imprecision Ideal is to estimate bias and imprecision, so

that you can know and limit your overall error

Write it down!

Objectives of QC:

4

Preparing for the field Checking your equipment

◦ Analyzer working, QC within limits◦ Transfer standards also need to be working

Supporting stuff: ◦ Site, power, weather, frequency of your checks

ok? Look to the future Data recording system working

◦ Routine data◦ QC data

Downloading the data

Evaluate where “Things Can Go Wrong”—plan how to check and what to document

5

…the difference between your answer and the

“truth”Two components of total error (accuracy):

Bias (jump)Precision (wiggle)

Error

6

Minimized by calibrating your equipment against a standard◦ Make sure the standard has TRACEABILITY to a

NIST standard◦ Compare result of the standard(s) to what your

instruments read (pollutant, flow rate, temp, pressure)

Measures any bias in your

equipment at that time

Bias Error

7

Some imprecision is unavoidable

Sometimes up, sometimes down–“random”

Estimated by measuring the same thing several times

Or by side-by-side collocatedMinimized by carefully following procedures

Precision Error

8

The term “transfer” means that you are “transferring” the reliability of a measurement from a primary calibration standard to your field instrument

The primary calibration standard can be the place where you send your transfer standard to be recalibrated annually

Pressure, temperature, and flow rate of your instruments must be checked with your transfer standards

QC Checks are made using transfer standards:

9

For example, from CFR 50 Appendix L: “An authoritative flow rate standard shall be used for calibrating or verifying the sampler's flow rate measurement device with an accuracy of ±2 percent. The flow rate standard shall be a separate, stand-alone device ... This flow rate standard must have its own certification and be traceable to a National Institute of Standards and Technology (NIST) primary standard for volume or flow rate.

Transfer Standards:

10

Transfer standards needed:

11

Perform a check of your equipment:◦ flow rate for PM◦ concentration for gas◦ internal mass for TEOM

This may be single-point (one flow rate) or multi-point (several flow rates)

If this is within specifications, record this and continue

Example of QC check (verification)

13

If results are NOT within specs, then the instrument must be adjusted

This adjustment means that the response of the instrument is changed, which is defined as a CALIBRATION

Calibrations must be multi-point After calibrating, do QC check again

Calibrations (covered in more detail in webinars #4-7)

14

Corrective Action if the flow rate difference between the transfer standard and the sampler is > 4%

Check sampler for internal and external leaks

Ensure that temp. and pressure are within normal ranges

Run check a 2nd time If still > 4%, perform a multipoint

calibration followed by a single-point verification

Example of QC checks for PM2.5

15

1-point QC check must show within 7% for O3, 10% for others (if starting out, every week)

Zero must not drift more than 2% of full scale

Span concentration must not drift more than 7% since last check

QC checks for gaseous-every 2 weeks:

16

Gaseous Verifications Gaseous instrument verifications can be conducted:◦Through-the-probe (is what you will be

audited on—do this if possible)◦Through-the-manifold◦Through the back of the analyzer

To evaluate the sampling system it is recommended that at least some verifications be conducted through-the-probe

17

QC Checks (and audits):Method Pollutants Frequency MQO

One-Point QC SO2, NO2, O3, CO Every 2 Weeks

O3 :

Precision = 7% Bias = 7%

SO2, NO2, CO :

Precision = 10%Bias = 10%

Annual Performance Evaluation (Audit)

SO2, NO2, O3, CO Once per Year <= 15% for each audit concentration

Flow Rate Verification (QC check)

PM10 (lo-Vol), PM2.5

Once every 4 weeks

<= 4% of Standard & 5% of Design Value

Flow Rate Verification (QC check)

PM10 (high-Vol), TSP

Once per quarter

<= 7% of Standard & 10% of Design Value

Semi-Annual Flow Rate Audit

PM10, PM2.5 Every 6 Months

<= 4% of Standard & 5% of Design Value

Semi-Annual Flow Rate Audit

PM10 (high-Vol), TSP

Every 6 Months

<= 10% of Standard & 10% of Design Value

Collocated Sampling PM10, TSP, PM2.5 15% of Network Every 12 Days

PM2.5, TSP, PM10 :

Precision = 10%

PM2.5 PEP Program

NPAP

PM2.5

SO2, NO2, O3, CO

QuarterYear (see QA Requirements.xls)

Bias = 10%

18

Who can do these, and with what?Method Pollutants Frequency Who? With What?

One-Point QC SO2, NO2, O3, CO

Every 2 Weeks Site operator Annually calibration standard, owned by tribe

Annual Performance Evaluation

SO2, NO2, O3, CO

Once per Year Not site operator, no national certification required, not the same standard that is used for routine QC checks

Flow Rate Verification (not required to be reported to AQS)

PM10, PM2.5 Once every 4 weeks

Can be site operator, no national certification required, not the same standards that are used to calibrate your analyzer

Semi-Annual Flow Rate Audit

PM10, PM2.5 Every 6 Months Preferably not site operator, no national certification required, not the same standards that are used for monthly QC checks

PM PEP, NPAP PM2.5,

All gaseous

See QA Requirements.xls

OAQPS/TAMS certified auditor

OAQPS/TAMS equipment

19

Gaseous Standards All calibrations, verifications, and audits

should be conducted with certified gas standards or a certified transfer standard (ozone).

Gas standards are certified by the vendor and can be recertified (EPA Protocol 600/R-97/121)

Ozone transfer standards should be annually certified and traceable to a Standard Reference Photometer (SRP)

20

DASC (Data Assessment Statistical Calculator)◦In Excel, easy to use◦Calculates what EPA calculates in

AMP255◦Can be used with any parameters◦Cites equation #s from CFR

AQS: Data Quality Indicators Report (AMP255)◦Covers all of the Requirements◦http://www.epa.gov/ttnamti1/qareport.ht

ml

21

What do I do with QC results?

Consider the standard to be the “truth” for your equipment

Difference between the “truth” and your equipment is the instrument error (this may include bias [error either usually high or usually low], and imprecision [wiggle sometimes high and sometimes low])

How is instrument error quantified?

Calculations:

22

Percent Difference used for both Precision and Bias

Basis for All Calculations (divide by best estimate of the truth, to get a %-off-from-truth):

For standard gas

For Collocated Samples100

2)(

ii

iii YX

YXd

23

100

audit

auditmeasdi

If you are outside the limits: Review procedures & logs to identify

problem Go back and review data—may have to

throw out (invalidate—never delete) data to the last passing check (see validation templates for critical criteria that require invalidation—may not have to invalidate data if it’s a systematic or operational criterion)

Fix the problem, document it and report (when significant)

Next steps of QC

24

“Accuracy” for EPA means “total error” Comparison conducted with completely

different system can be used to estimate total error

For PM2.5, this is a performance evaluation conducted with a side-by-side FRM device, with the filter weighed by a different lab

Total error includes both precision and bias errors

Accuracy estimates are verified thru audits

Accuracy = Total Error

25

Resources AMTIC http://www.epa.gov/ttn/amtic/ Transfer Standards for Calibration of Air Monitoring

Analyzers for Ozone: Technical Assistance Document http://www.epa.gov/ttnamti1/files/ambient/qaqc/OzoneTransferStandardGuidance.pdf

U. S. EPA Ambient Air Monitoring Protocol Gas Verification Program: Implementation Plan 2nd DRAFT 7/2009

http://www.epa.gov/ttnamti1/files/ambient/qaqc/aapgvpimpplan.pdf

EPA NPAP TTP PE SOP COMPENDIUM (DRAFT FINAL, 1-12-05)

http://www.epa.gov/ttn/amtic/files/ambient/npapsop/npapttpsop.pdf

26

Resources Field Standard Operating Procedures for the Federal

PM2.5 Performance Evaluation Programhttp://www.epa.gov/ttn/amtic/files/ambient/pm25/qa/PEP_Field_SOP.pdf Pb Performance Evaluation Programhttp://www.epa.gov/ttn/amtic/files/ambient/pb/Pb-PEPImplementationPlan_7_09.pdf

Model Quality Assurance Project Plan For the National Air Toxics Trends Stations

http://www.epa.gov/ttn/amtic/files/ambient/airtox/NATTS_Model_QAPP.pdf

Resources—National QA Workgroup QA National Meeting Presentationshttp://www.epa.gov/ttn/amtic/qaqcrein.html

27

QC checks are necessary to both do, and Track results of Subsequent webinars will cover pollutant

specifics QC checks Course website:

http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/

Our emails:◦ Travis Maki: tmaki@itcmi.org◦ Melinda.ronca-battista@nau.edu◦ Christopher.lee@nau.edu

28

Conclusions: