mr. thomas j. chapel - measure that make a difference! why measure and what to measure!

Thinking Strategically About Antibiotic Resistance Efforts

2

6 Things We MUST Understand…1. Big “need” to which we are contributing

2. “What” vs. the “so what” of the program

3. Sequence of ST outcomes MT outcomes LT outcomes

4. “Accountable” outcome—the outcome that equals success

5. “Moderators”—cntextual factors that get in the way

6. “Outputs”—activities strong enough to get to the outcomes

33

A Simple “Program Roadmap” Helps

Activities Inputs Outputs

Intermediate Outcomes

Short-term Outcomes

Long-term Outcomes/

Impacts

What the program does…

Who or what will change because of the program…

What the program needs…

Context and Assumptions

External factors that influence getting to outcomes

4

Payoff: For Strategy

1. The key outcomes we “own”

2. How to implement our activities in way that most assures our outcomes

3. How to mitigate/own outside factors that may keep us from getting outcomes

5

Payoff: For Measurement

1. Accountable outcome is key one to measure

2. What ST/MT drive it, IF we don’t think we’ll get there?

3. Outputs=Process measures for our activities

4. Moderators=Ability to explain (lack of) progress

6

Today1. Presentation:

1. Creating and using a roadmap.

2. Some examples

2. Small groups:1. Creating/refining a roadmap for antibiotic

resistance effort

2. Identifying main outcomes of interest

3. Defining main “pathways” to get there

4. Identifying big outside factors

3. Quick debrief from small groups

7

Thursday1. Identifying main outcomes for our

“dashboard”

2. Defining some measures for outcomes

3. Identifying potential sources and potential challenges

8

REMEMBER!1. Strawman only

2. First, not last, discussion

3. Many steps post-session to get to clarity and consensus

99

(Simple) Logic Models are Roadmaps for ANY Effort: Lab Safety Example

Lab Safety Training

3: Behaviors sustained on the job

1: Happy trainees

4: Org change—Culture of lab safety

What the program does… Who or what will change

because of the program…

2: KAB and skills

1010

(Simple) Logic Models are Roadmaps for ANY Effort

Activities


Short-term Outcomes

Long-term Outcomes/

Impacts



Big “need” to which program is contributing

“What” vs. “So what”

1111


Lab Safety Training


1: Happy trainees




2: KAB and skills

Big “need”: Lab Safety Training EXAMPLE

“What” vs. “So what”

12

Case: Lead PoisoningLead poisoning is a widespread environmental hazard facing young children, especially in older inner-city areas. Lead poisoning effects can be ameliorated through medical interventions. But, ultimately, the source of lead in the environment must be contained/eliminated through renovation or removal of the lead-based paint by professionals. Short of that, families can reduce the bad effects on their children through intensive housekeeping practices and selected

nutritional interventions.

13

The ProgramCounty X, with a high number of lead-poisoned children, has received money from CDC to support its Childhood Lead Poisoning Prevention Program. The program aims to do outreach to identify screening sites, screen and identify those with elevated blood lead levels (EBLL), assess their environments for sources of lead, and case manage both their medical treatment and the correction of their environment. They will also train families of EBLL children in selected housekeeping and nutritional practices. While as a grantee they can assure medical treatment and reduction of lead in the home environment, the grant cannot directly pay for medical care or for renovation of homes.

14

Listing Activities and Outcomes: Lead Poisoning

Activities Outreach Screening Case management Referral for medical tx Identification of kids with

elevated lead (EBLL) Environmental assessment Referral for env clean-up Family training

Effects/Outcomes Lead source identified Families adopt in-home

techniques Providers treats EBLL kids Housing Authority

eliminates lead source EBLL reduced Developmental “slide”

stopped Q of L improved

15

Lead Poisoning: “Causal” Roadmap

ScreeningScreening

Do Environment

Assessment

Do Environment

AssessmentID Source and

Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Activities Outcomes

16

Antibiotic Resistance: National Plan Goals1. Slow the Emergence of Resistant Bacteria and

Prevent the Spread of Resistant Infections.

2. Strengthen National One-Health Surveillance Efforts to Combat Resistance.

3. Advance Development and Use of Rapid and Innovative Diagnostic Tests…

4. Accelerate Basic and Applied Research and Development for New Antibiotics, Other Therapeutics, and Vaccines.

5. Improve International Collaboration and Capacities…

17

The “What” of Antibiotic Resistance Monitor sales, use, and management practices Mobilize, advocate, engage across sectors Strengthens infrastructure for surveillance and

reporting Establish regional PH lab network with standard

platform for resistance testing Provide R&D funding and support Form public-private research partnerships Deliver education programs

18

The Big “So Whats” Less drug resistance Preserve the usefulness of antibiotics and

effective treatment of infection More effective prevention, control, and

treatment of infection/disease in humans and animals

Prevention of infection and disease Reduced spread of antibiotic resistant

pathogens Fewer resistant infections; Less disease in

humans and animals

19

Some “So Whats” That Drive Them Improved use of antibiotics in healthcare

settings Improved use of antibiotics in food production Increased trust and reduced blame Rapid recognition of resistance in humans,

animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics/non-antibiotic

prevention methods and therapies New and innovative animal management and

food production innovations

20

1 Activities

2 ST Outcomes

3 MT Outcomes

4 MT Outcomes

5 LT Outcomes

[WHICH PLAYER(S)] Monitors sales, use, and management practices [WHICH PLAYER(S)] Mobilizes, advocates, engages across sectors [WHICH PLAYER(S)] strengthens infrastructure for surveillance and reporting [WHICH PLAYER(S)] establishes regional PH lab network with standard platform for resistance testing [WHICH PLAYER(S)] provides R&D funding and support [WHICH PLAYER(S)] forms public-private research partnerships [WHICH PLAYER(S)] delivers education programs

Strong antibiotic stewardship in humans/healthcare Strong antibiotic stewardship/vet oversight in food production More regional cross-sectoral cooperation Consensus on strong and supportive policy Stronger lab capacity One health surveillance; Integrated surveillance Development of new technologies and innovations Increased private sector investment Increased private sector interest

Improved use of antibiotics in healthcare settings ; Optimal treatment decisions Improved use of antibiotics in food production Real-time info sharing on resistance and infections Increased trust and reduced blame Rapid recognition of resistance in humans, animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics New non-antibiotic prevention methods and therapies New and innovative animal management and food production innovations

Less drug resistance Preserve the usefulness of antibiotics and effective treatment of infection More effective prevention, control, and treatment of infection/disease in humans and animals Prevention of infection and disease

Reduced spread of antibiotic resistant pathogens Fewer resistant infections; Less disease in humans and animals

S A F E A D E Q U A T E F O O D S U P P L Y

21

Small Group Task1. Review the draft/strawman roadmap

2. Refine as needed:1. BIG activities missing?

2. BIG outcomes missing?

3. Some outcomes really activities that we do?

4. Sequence of outcomes right?

“Networked” Interventions

Agency A

Program A-n

Program A-1

Agency B Program B-1

Agency C

Program C-n

Program C-1

Agency D

Program D-n

Program D-1

OUTPUTS

SHORT-TERMOUTCOMES

LONG-TERMOUTCOMES

SYSTEMOUTCOME

23

Small Group Task1. Review the draft/strawman roadmap

2. Refine as needed:1. BIG activities missing?

2. BIG outcomes missing?

3. Some outcomes really activities that we do?

4. Sequence of outcomes right?

3. Think about YOU/your role. Can you trace a pathway thru the roadmap?

2424


Activities


Short-term Outcomes

Long-term Outcomes/

Impacts



“Accountable” outcome

25

Choosing Accountable Outcome— “Staking Our Claim”

Want to include outcome(s) that is (are):Relevant—sufficiently “downstream”

to matter to stakeholdersResponsive—sufficiently “upstream”

that program’s efforts can be expected to make a difference

2626


Lab Safety Training


1: Happy trainees




2: KAB and skills

“Accountable” outcome: Lab Safety Training EXAMPLE

27

Lab Safety Accountable Outcome: (Selected) Behaviors Sustained on Job Proper use of PPE Proper use of BSC Conduct formal risk assessments Ability to recognize hazards in any

situation Complete appropriate documentation Proper waste management

28

Lead Poisoning: “Causal” Roadmap—ACCOUNTABLE OUTCOME

ScreeningScreening

Do Environment

Assessment

Do Environment


Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

**ReducingEBLLs**

**ReducingEBLLs**

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Activities Outcomes

29

1 Activities

2 ST Outcomes

3 MT Outcomes

4 MT Outcomes

5 LT Outcomes







30

Small Group Task1. What is/are the accountable outcome(s)

for the antibiotic resistance effort?

2. REMEMBER:1. Thinking about THE WHOLE EFFORT—all

the pathways working jointly.

2. Looking to balance relevance with responsiveness. May/may not be outcomes on the far right!

3. Not naming ALL outcomes, just the ones furthest “east”

“Networked” Interventions

Agency A

Program A-n

Program A-1

Agency B Program B-1

Agency C

Program C-n

Program C-1

Agency D

Program D-n

Program D-1

OUTPUTS

SHORT-TERMOUTCOMES

LONG-TERMOUTCOMES

SYSTEMOUTCOME

32

Small Group Task1. Looking at the pathways, what

earlier outcomes seem most important to get to the accountable one/s?

3333

Simple Logic Models as Roadmaps



Short-term Outcomes

Long-term Outcomes/

Impacts




Context and Assumptions

External factors that influence getting to outcomes

5. What “strong” activities means

Killer assumptions: Outside factors that can

help or hobble

34

Lab Safety: (Selected) Moderating Factors, Even if Our Training is Great! Supervisors’ attitudes and expectations Lack of resources for quality and safety Lack of reinforcement (decay of KAB) Time/safety trade-off, particularly during

public health emergency (time constraint) Competing priorities

35


ScreeningScreening

Do Environment

Assessment

Do Environment


Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Activities Outcomes

36

Contextual Factors

PoliticalEconomicSocialTechnological

37

Moderators—Lead Poisoning

Political—“Hazard” politicsEconomic— Health insuranceTechnological— Availability of

hand-held technology

38


ScreeningScreening

Do Environment

Assessment

Do Environment

Assessment ID Source and

Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Outcomes Inputs

Funds

Staff

R’ships

Legal

Authority

Funds

Staff

R’ships

Legal

Authority

Activities

technology

insurance climate

“hazard politics”

39

(Selected) Actions We Can Take to Mitigate Moderating Factors

Supervisors’ attitudes and expectationsAdd consistent messages in supervisor

trainingPush others to create SOPs or policies to

induce supervisor cooperation.

Lack of reinforcement (decay of KAB)Refresher training; written materials

40

(Selected) Actions We Can Take to Mitigate Moderating Factors (cont’d) Time/safety trade-off, particularly during

public health emergency (time constraint) Competing priorities

Highlight consequences of not doing things right

Lack of resources for quality and safetyArray solutions in class from

cheapexpensivePush others for more budget for these efforts

41

1 Activities

2 ST Outcomes

3 MT Outcomes

4 MT Outcomes

5 LT Outcomes







42

Small Group Task1. What are some big outside factors that

might keep us from getting our outcomes?

2. Use the P-E-S-T taxonomy IF helpful

3. Consider at HIGH LEVEL: Which of the outside factors:

1. Are “laws of gravity” we must work around?

2. Can be influenced by adding activities to our roadmap or doing things differently.

4343

Simple Logic Models as Roadmaps



Short-term Outcomes

Long-term Outcomes/

Impacts




Outputs: What “strong” activities means

4444


Lab Safety Training


1: Happy trainees




2: KAB and skills

Outputs: What “strong” activities means: Lab

Safety Training EXAMPLE

45


ScreeningScreening

Do Environment

Assessment

Do Environment


Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Activities Outcomes

4646

The Plot Thickens

Screening so good it will lead to…

Behavior Change

ID People with Condition

Training in Self-Management so good it will lead to…

Referral for Medical Treatment so good it will lead to…

Quality Medical Management

4747

“Upgraded” Outputs: More than Simple Counts Screening: Pool (#) of screened kids

(meeting likely risk profile) Training: Pool (#) of clients trained

(using culturally-competent curriculum and with appropriate supports)

Referrals: Pool(#) of referrals to (qualified or willing) medical treatment providers

48

Outputs: (Selected) Key Attributes of Activities to Ensure Outcomes are Reached Content of training should explain why it

should be done Focus on actual work practices/evidence-

based work practices Real-time experience during the training Include reinforcing materials Personalizing content to the situation of

the person as opposed to general regs

49

1 Activities

2 ST Outcomes

3 MT Outcomes

4 MT Outcomes

5 LT Outcomes







50

Small Group Task—TIME ALLOWING1. Pick (one or more) activity in column 1.

2. Follow its pathway—what outcomes is it to generate?

3. Knowing the outcomes it is to achieve:1. How must we do this activity so it reaches

that outcome?

2. What are some attributes of doing that activity “well”? (quantity, quality, intensity, reach, etc.)

51

WRAP UP: In Your Groups—TODAY

Groups pre-assigned. Facilitator assigned to guide discussion Scribe assigned to take notes HIGH-LEVEL Topics:

Refine simple roadmap Set accountable outcome and some major

outcomes on pathways to get there“Moderator” discussion

52

In Your Groups—THURSDAY

Identify performance dashboard Set outcome measures

[TIME ALLOWING] “Output” discussion Set process measures

mr. thomas j. chapel - measure that make a difference! why measure and what to measure!

Health & Medicine