Earmark Grant Evaluation: An Introduction and Overview

May 19, 2005

Presented by:Jeff Padden, President

Public Policy Associates, Inc.119 Pere Marquette Drive

Lansing, Michigan 48912-1231(517) 485-4477

www.publicpolicy.com

Presentation Topics The evaluation requirement for earmark

grants Evaluation overview – or – “Where’s the

upside?” Planning the evaluation The evaluation process for earmark grants Discussion

We’ll do clarifications on the fly, broader discussion at the end.

The Evaluation Requirement

Each grantee must …

Conduct or commission an evaluation

Submit evaluation plan

Use the evaluation template

Submit evaluation report shortly after completion of project activities

Evaluation Overview– or –

“Where’s the upside?”

Program evaluation is ...

The systematic collection of information about the subject of the evaluation

Used to make decisions about organization’s or program’s: Creation

Improvement

Effectiveness

Evaluation is a mindset …

We are all evaluators

Evaluation is continuous

Evaluation looks forward, not just backward

Involves organizational learning

Means people working together

Evaluation allows you to examine ...

What’s working well

What is not

How to improve

There is no bad news, only news!

Evaluation requires comparison ...

of the same group over time• pre- and post-tests

• trends in community-level data

of two comparable groups at one point in time over time

of your group to a larger group county compared to state

Our Approach: Utilization-Focused Evaluation

Focuses on intended uses and users

Is inherently participatory and collaborative by actively involving primary intended users in all aspects of the evaluation

Leads to ongoing, longer-term commitment to using evaluation logic and building a culture of learning in a program or organization

Symbiotic rather than parasitic

Benefits of Evaluation

Program/organizational improvement Accountability to funders and others Planning Program description for stakeholders Public relations Fund raising Policy decision making

Evaluation has lots of upside!

Planning the Evaluation

Elements of the Evaluation Plan Who conducts the evaluation?

Internal or external?

Experienced or novice?

When do they do it? Along the way or after the fact?

How much do they do? The level of intensity must fit the project

Too much diverts resources, too little leaves unanswered questions

What exactly do they do? Six major steps

Evaluation Steps

1. Clarify project & goals 2. Establish measures

3. Collect data

4. Analyze data

5. Prepare reports6. Improve project

Step 1: Clarify Project & Goals

Thinking about goals What are you trying to accomplish?

What would success look like?

What is the difference between the current state of affairs and what you are trying to create?

Example of a goal statement:“Increase incomes of low-income families in the region through training for entry-level jobs that have career ladders leading to good jobs.”

Does the Project Hang Together? Are the expected outcomes realistic?

Are there enough resources?

Do the customers like the product?

Does the organization have the right skills?

Logic models help answer these questions.

A Simple Logic Model

Things needed to run the project:

People, resources, money, etc.

What you do:

Market, recruit, design, train, place, etc.

Direct results of activities:

Training completers,credentialsawarded, etc.

Changes caused by the project:

Jobs, wages, promotions, etc.

Inputs Activities Outputs Outcomes

Step 2: Establish Measures

Determine performance measures Must be quantifiable

Data must be available, reliable, and valid

Examples of measures:

Activity: Number of training sessions

Output: Number of trainees

Outcome: Skill and credential gains

Impact: Stronger local workforce

Step 3: Collect Data

Identify data sources, such as: Administrative records Surveys, interviews, focus groups Observation

Gather dataDesign the instruments and procedures for collection.Conduct data collection periodically.

Record data Organize data. Create database. Verify data.

Remember the measures!

Step 4: Analyze and Interpret Data

Sort and sift: organize data for interpretation Cross-tabs

Modeling

Conduct data analysis to look for: Changes over time

Progress relative to goals or standards

Differences between groups

Test preliminary interpretation

This is the most creative step.

Step 5: Prepare Reports

Determine reporting schedule

Report preliminary findings to key stakeholders and other audiences

Gather reactions

Incorporate reactions

Finalize reporting products

Different audiences need different types of reports.

Step 6: Improve Project

Deliver reporting products internally.

Facilitate strategic and operational planning.

Improve processes and results.

A good evaluation will be more valuable to you than to DOL!

The Evaluation Process for Earmark Grants

Use the DOL Tools

“The Essential Guide for Writing an Earmark Grant Proposal”

“Evaluation Template for Earmark Grantees” (to be provided later)

Discussion

Thanks to …

… for the use of the “Demystifying Evaluation” materials.

Useful evaluation links:

• W.K. Kellogg Foundation: www.wkkf.org/Programming/Overview.aspx?CID=281

• American Evaluation Association: www.eval.org/EvaluationLinks/default.htm

• Western Michigan University Evaluation Checklists: www.wmich.edu/evalctr/checklists/checklistmenu.htm

Earmark Grant Evaluation:An Introduction and Overview

May 2005

Presented by:Jeff Padden, President

Public Policy Associates, Inc.119 Pere Marquette Drive

Lansing, Michigan 48912-1231(517) 485-4477

www.publicpolicy.com