ensuring continuous introduction monitoring relevance · ensuring continuous monitoring relevance...

introduction

Ensuring Continuous Monitoring Relevance

©2015 Ramaley Group, LLC

Using Control Charts

Ken Ramaley, CIA, CRMA Managing Director

Ramaley Group [email protected]

What is this all about?


• Continuous Monitoring (CM) programs have tremendous value to auditors • Early identification of potential control failures

• Dynamic re-allocation of resources • “Automated issues”

CM are High Potential Programs – But Only Relevant if they enable effective inferences

How to ensure Relevance?


Statistically-based continuous monitoring enables appropriate response to out-of-control conditions

• Observed control failures may not mean anything. Continuous monitoring is like performing an extended hypothesis test

• Plan to react when you see “signal” vs. “noise”

What does audit do next?

You have implemented a real-time continuous monitoring program. Last night’s sample came in this morning – yielding results on 100 units, divided equally between two sites. The control is expected to be functioning with 90% quality. Site A shows up with 10 exceptions (of the 50 tested). Site B has only 2 exceptions. Is the control functioning acceptably? Are your concerns limited to Site A, or are they across the board?


What does audit do next?

A regulatory agency reports that it has received double the volume of complaints about your firm this month that it received last month. How do you respond? Is this a problem?


In order to conclude on the relevance of an observation, you must understand context and statistical likelihood.

Testing for Significance

Statistical data allows you to perform tests of significance for exceptions Fundamentally: How unusual is the observation?

Solution: Hypothesis Testing Based on forming a “Null Hypothesis” – that the population has a particular property (e.g. mean of a certain value, etc.) - written as H0

Likelihood of what you have observed will allow you to reject H0 or NOT reject H0


Control Charts

A control chart is a visual display which focuses attention on process performance over time Key Features: • Center line – average performance • Data in time order • Upper/Lower Control Limits displayed (optionally also display Zones A,B,C at 1s,2s) • For variable data, top chart monitors

average (location), bottom chart monitors range (grouping)


Control Chart Example


Mean Perf.

Time

UCL

LCL

Spec Limits and Control Limits

Specification Limits Determined by customers of the process

(requirements!) Changed based on customer demands

Control Limits

Calculated based on historical process performance

Only changed when process has experienced a special-cause change – and new process has been in place for at least 20 data points


Control Charts as Hypothesis Tests


By mapping your CM data in a Control Chart, you can identify critical factors in ensuring good audit conclusions: Is the process capable of delivering to spec limits?

If capable, is the process continuing to function “in control”?

“Out of control” = “Would fail a hypothesis test”

Standard rules of thumb can help

Western Electric Rules for Interpreting Control Charts*

*Adapted from Western Electric Company (1956), Statistical Quality Control handbook. (1 ed.), Indianapolis, Indiana: Western Electric Co.

Limits of Simple Control Chart Rules

Rules exist as “guidance” not “gospel” for finding out-of-control signals

Judgment must be incorporated into control chart assessment

False Positives may occur (approx 1% due to chance if executing all 4 rules)

Control charts must be accompanied by control plans

A control chart is only as good as its underlying data ©2015 Ramaley Group, LLC

Additional Control Chart Considerations

Autocorrelation may be a factor – the data collection period should be long enough that each data point is truly independent (can test using a time series regression)

Control Limits should only be re-calculated when the process has experienced a known external change and some new data has been collected

Context for control chart data must be provided by people who are engaged in the process


±

How Do I Build a Control Chart?

1) Begin with a data set. For our example, lets assume it is in small batches.

2) Take the mean of each batch of data you have collected, then take the mean of the means (m).


DISCLAIMER: There are many different kinds of control charts – it may take assistance to determine the correct one for your situation. This is a simple illustrative example.

How Do I Build a Control Chart?

3) Calculate the standard deviation (s) of your data points

4) Define your Upper Control Limit (UCL) as m+3*s and your Lower Control Limit (LCL) as m-3*s

5) Draw your control chart with the time series data, drawing in the mean line, UCL, LCL, and lines showing m±s, m±2s


Practical Control Chart Implementation


Standard statistical functions can be executed through tools you already have: Excel SQL Minitab JMP

Practical Control Chart Implementation


As you establish your CM program, the outputs should populate control charts and automatically execute tests – triggering alerts based on Control Chart diagnostics!

Relevance flows from meaningful CM alerts. False alarms will cause results to be ignored. Integrate Control Chart diagnostics into your CM program to hit next-level audit efficiency and effectiveness

Control Chart ACTION

When your control chart suggests a concern: EXECUTE YOUR ACTION PLAN!!!


Consider responses to out-of-control conditions in advance of experiencing them. This drives audit consistency and relevance.

Questions?


ensuring continuous introduction monitoring relevance · ensuring continuous monitoring relevance...

Documents