impact workshop slides - taylor newberry consulting
DESCRIPTION
2013 OVCN INNOVATION & ACTION! Conference 'If Demonstrating Impact Seems Boring, You're Doing it Wrong' facilitated by Andrew Taylor of Taylor Newberry Consulting Inc. http://taylornewberry.ca/ #OVCNactionTRANSCRIPT
A PRESENTATION FOR THE
OVCN INNOVATION & ACTION!
CONFERENCE
If Demonstrating Impact Seems Boring, You’re Doing it Wrong
Pre - Test
! When someone says “lets talk about demonstrating impact,” I: ! Get excited and energized. ! Get moderately enthusiastic. ! Picture my long to do list getting even longer, sigh, and say
“Ok, sure. I guess that’d probably be a good thing for me to think about.”
! Make a mental note to bring my phone to the meeting so I can get caught up on some emails.
! Think “just kill me now.”
A N D R E W @ T A Y L O R N E W B E R R Y . C A 5 1 9 5 4 6 4 7 8 9
@ T A Y L O R N E W B E R R Y C
Andrew Taylor Contact info
E v i d e n c e I n s i g h t A c t i o n
A Basic Overview of Outcomes Measurement
Acting
Listening Visioning and
Reflecting
Connecting
Program Planning; Strategic Planning;
Program Design and
Implementation
Consultation Partnership Building
Achieving Buy-in
Needs Assessment Outcome Measurement
Program Evaluation
Setting the stage for good program monitoring
1. Get buy-in first. 2. Make sure you and your partners
understand your program and its outcomes clearly and consistently.
3. Remember that the key is using the data, not gathering the data. You don’t necessarily need to gather more data.
E N G A G I N G W I T H S T A K E H O L D E R S
1. Get Buy-in First
Thinking About Use before You Start
! Who do you want to act on the basis of your outcome measurement findings?
! How do you want them to act?
Why are people not interested in outcome measurement?
We tend to assume:
But the real reasons may be:
They don’t have the training to understand
They know that the big decisions aren’t really based on outcome measurement findings (despite the rhetoric).
They aren’t interested in learning
They see measurement as complicated, dry and boring. They don’t see how it will help them to learn.
They’ve got something to hide
They worry that people won’t understand the context. They are afraid of being unfairly judged.
They are too busy
They see no tangible “payoff” for the significant time investment.
Exercise: What’s in it for them? Stakeholder How might evaluation
help them? How might evaluation make their lives difficult?
Board Member
Manager
Front-line Staff Member Participant in a Program Funder
Partner Organization Volunteer
C L A R I F Y I N G T H E O R I E S O F C H A N G E
Make Sure You Understand the Program Clearly and Consistently
Theories of Change
! The beliefs or assumptions that inform the design of a program or an intervention
! In particular, those beliefs and assumptions about how change happens, and how the intervention will lead to change.
! Sometimes theories of change are very explicit and structured, but sometimes they are implicit and emergent.
The importance of short-term outcomes
Train volunteer
coordinators Program Activities
Long-Term Outcome or Goal
Publicize volunteer positions
Create a more engaged, connected community
The importance of short-term outcomes
Program Activities
Long-Term Outcome or Goal Create a more engaged, connected community
More diversity in volunteers
Volunteers have more meaningful experiences
More people
volunteer
Members use other
best practices
Members manage
volunteers better
Members are Stronger Short-Term Outcomes
Train volunteer
coordinators
Publicize volunteer positions
Volunteers are more engaged
Breaking Down the Complexity
Did our program have an impact???
What were we trying to change? (Long-Term Outcome Objectives)
How were we planning to make that contribution? (Activities)
What sorts of things would we see if the expected change was happening? (Indicators)
What particular contribution were we going to make to that change?
(Short-Term Outcome Objectives)
How can we document those observations in a systematic way?
(Methods)
Problems with Logic Models
! They are dense ! They are abstract ! They talk about change in a very linear, mechanical
way ! They are good for getting your own thinking
organized, but not so good for communicating with outsiders, especially if those outsiders ALSO have a complex program logic model that they want to communicate with YOU.
Hello. My life goal is to get married and have kids. I am currently pursuing several
short term outcomes related to this goal, including meeting a nice person, getting in better physical shape, moving out of
my parents’ house, and choosing names for the kids.
Hello. I am new to town. I don’t know many people, or many places to go, and I’m
feeling a bit lonely. My outcome objectives are to
make some friends and get out more. My goal is to feel more connected to my new
community, although I recognize that this goal may
need to be made more measurable.
Yes! There’s a great place downtown. Want to go after
work?
Know anywhere good to go for coffee
around here?
Sounds great! I’ve been wanting to get
to know that neighbourhood. Me too, actually!
I’d love to live downtown someday.
What are the most important shared short-term outcomes for volunteer centres?
What are the short-term outcomes we are well positioned to achieve that “plug in” to the shared
“collective theory of change?”
What are the shared outcomes that the community has identified as important? What is the shape of the
“collective theory of change?”
The Really Useful Idea
Identify short-term outcome objectives that are
within your control, measurable and achievable,
and use a theory of change to explain how these
short-term achievements contribute to the
achievements of collective outcomes over time.
What’s the big, shared change that matters to
the community?
What do we do? What do we do?
What do we do? What do we do?
M A K I N G S M A R T U S E O F S I M P L E D A T A
3. Remember that the key is using the data, not gathering the
data.
What are Indicators?
! Bits of information that provide part of the answer to one of your questions
! Things you can see or touch or hear – things that are observable in the world and don’t involve ‘interpretation’
! Are often numbers but can also be (e.g.) stories, quotations, examples, pictures
Indicators are a Useful Idea Because...
! They help to break down the complex task of “outcome measurement” into manageable chunks
! They help others to understand what you mean in practical terms when you talk about a particular outcome
! They help you build up a strong meaurement plan by combining different kinds of information from several sources
The Notion of Triangulation
A Quick Way to See if You’ve got the Right Kind of Indicators (from Friedman, 2005)
Quantity Quality
How much work did we do? # of people who attended the workshop # of handouts created # of workshops run #
How well did we do it? % of invited people who attended % % %
# # # #
% of participants who are better prepared to complete outcome measurement plans % of measurement plans that are useful % of plans that are manageable and practical
Is anybody better off?
Different kinds of research on social programs
Measuring outcomes through ongoing program monitoring
! Carried out by program staff and volunteers on a routine basis
! Simple, easy to use methods ! Focused on short-term outcomes ! Usually measures a few basic things
well, rather than many different things
Measuring outcomes through ongoing program monitoring
! Ideally, becomes a seamless element of program delivery
! Periodic analysis of data ! Focuses on ongoing improvement of
programs
Outcome measurement planning sheet
What are the questions you need answered?
What indicators can you use from information you’ve already got?
What additional information do you need?
How can you get it?
These may be questions about outcomes, but they may also be questions about process, lessons learned, etc.
Be as specific as you can!
Think stories and examples and also quantitative data
What is the simplest, least intrusive method?
S H A R I N G F I N D I N G S
Making Your Evaluation“Worth It:”
Its all about “messaging”
! The key to powerful communication about evaluation findings is to know: ! Who your audience is ! What that audience is already passionate about, and how that
passion connects to your work ! How you want them to act with the knowledge you give them ! How you can present your information in the way that
maximizes the chances of them acting
Acting
Listening Visioning and
Reflecting
Connecting
Program Planning; Strategic Planning;
Program Design and
Implementation
Consultation Partnership Building
Achieving Buy-in
Needs Assessment Outcome Measurement
Program Evaluation
Outcomes Measurement: Why do it?
The challenges
! Steep learning curve ! Time consuming ! Different expectations from different funders
! Lack of control over what you measure and how you measure it ! Funders that demand certain indicators but don’t explain what
they think those indicators mean ! Differing jargon across funders ! Knowing how much measurement is “enough” and what type is
expected ! Poor quality of data ! Lack of obvious “payoff”
The (potential) benefits
! More mindful practice ! more engaged, satisfied staff and volunteers ! A better sense of “where you fit” in the system of services and
supports
! Improved programming; more benefits for the people you serve
! Improved capacity to explain the value of your work to donors, volunteers, board members, and the general public
Q U E S T I O N S ? C O M M E N T S ?
Yeah, but ….
Where do I start?
1. Get buy-in first. 2. Make sure you and your partners
understand your program and its outcomes clearly and consistently.
3. Remember that the key is using the data, not gathering the data. You don’t necessarily need to gather more data.
Post - Test
! When someone says “lets talk about demonstrating impact,” I: ! Get excited and energized. ! Get moderately enthusiastic. ! Picture my long to do list getting even longer, sigh, and say
“Ok, sure. I guess that’d probably be a good thing for me to think about.”
! Make a mental note to bring my phone to the meeting so I can get caught up on some emails.
! Think “just kill me now.”