Evaluating The California Endowment Clinic Consortia Policy and Advocacy Program:

Lessons for Evaluators

Annette L. Gardner, PhD, MPHSara Geierstanger, MPHClaire Brindis, DrPHPhilip R. Lee Institute for Health Policy StudiesUniversity of California, San FranciscoNovember 4, 2011

SF

University of California San Francisco

Presentation Objectives

UCSF evaluation design and findings by outcomeMonitoring advocacyAssessing impact

Relevance of findings for researchers/evaluators, advocates and funders

The California Endowment Clinic Consortia Policy and Advocacy

Program

Component of Community Clinics Initiative (CCI)

Launched in 2001 and refunded two more Rounds in 2004 and 2007, an investment of $28 million

14 local/regional clinic consortia and 5 statewide clinic associations representing 800+ primary care clinics

Goals:1) Increase services to underserved populations2) Increase access through improved clinic efficiencies

Theory of Change:“Resources and staffing dedicated to policy advocacy and TA will increase the collective influence of clinics and strengthen a broad base of long-term support of

clinic policy issues.”

Grantees – 18 State, Regional, Clinic Consortia

Alameda Health Consortium (AHC)Alliance for Rural Community Health (ARCH)California Family Health Council (CFHC)California Planned Parenthood Education Fund (CPPEF)California Primary Care Association (CPCA)California Rural Indian Health Board (CRIHB)Capitol Community Health Network (CCHN)Central Valley Health Network (CVHN)Coalition of Orange County Community Clinics (COCCC)Community Clinic Association of Los Angeles County (CCALAC)Community Clinic Consortium Serving Contra Costa and Solano Counties (3C)Community Health Partnership, Inc. (CHP)Council of Community Clinics (CCC)North Coast Clinics Network (NCCN)Northern Sierra Rural Health Network (NSRHN)Redwood Community Health Coalition (RCHC)San Francisco Community Clinic Consortium (SFCCC)Shasta Consortium of Community Health Centers (SCCHC)

Alameda Health Consortium (AHC)Alliance for Rural Community Health (ARCH)California Family Health Council (CFHC)California Planned Parenthood Education Fund (CPPEF)California Primary Care Association (CPCA)California Rural Indian Health Board (CRIHB)Capitol Community Health Network (CCHN)Central Valley Health Network (CVHN)Coalition of Orange County Community Clinics (COCCC)Community Clinic Association of Los Angeles County (CCALAC)Community Clinic Consortium Serving Contra Costa and Solano Counties (3C)Community Health Partnership, Inc. (CHP)Council of Community Clinics (CCC)North Coast Clinics Network (NCCN)Northern Sierra Rural Health Network (NSRHN)Redwood Community Health Coalition (RCHC)San Francisco Community Clinic Consortium (SFCCC)Shasta Consortium of Community Health Centers (SCCHC)

The California Endowment Clinic Consortia Policy and Advocacy

Program

Advocacy capacity

Technical Assistance

Partnerships

2001 - 03 2004 - 06 2007 - 09

Evaluation Design

Methods

EvaluationTA

Evaluation Partnership

2001 - 03 2004 - 06 2007 - 09

Policy Change Logic Model: Measures and Tools

Expandadvocacy capacity

Increase policymaker awareness And supportfor clinic policyIssues, funding

Strengthenedclinic operations

Increasedservices forunderservedanduninsured

Improvedhealth Outcomes

* Annual Advocacy Activities Worksheet* Annual Grantee Interviews* Sustainability Survey

* Decision Maker Awareness Surveys (2)* Interviews with Media Representatives (2)* Policy Outcomes Wrksht* Funding Secured Wrksht

* Clinic Focus Groups (2)* Grantee Case Studies (17)

* Policy Case Studies (3) * Grantee Case Studies (17)

* Policy Case Studies (3) * Grantee Case Studies (17)

Outcome 1 Outcomes 2, 3 Outcome 4 Outcome 5 Outcome 6

Outcome 1: Expand Advocacy Capacity

Organizational capacity gains:StaffingAbility to maintain advocacyTechnical acumen

Advocacy activities:Strategically deploy advocacy “toolkit”, e.g.,

Working with the mediaEducating/working with elected officials pre/post

passagePursuing/leveraging partnerships with stakeholdersMobilizing grassroots advocates

Tool: Annual Assessment of Advocacy Capacity and Outcomes

13 grant- funded

activities

Federal (Y/N)

State (Y/N)

Local (Y/N)

Effectiveness (1 to

4, 4= “very effective”

Outcome 1: Increased

clinic funding

(Y/N)

Outcome 2: Achieve a

policy change

(Y/N)

Outcome 3: Increased

policymaker awareness

(Y/N)

Media advocacy

Contact policymakers directly

Analysis:

1) Determine percent activities at the federal, state and local levels, and overall

2) Calculate effectiveness overall and by activity

3) Determine percent of activities that achieved each Outcome

4) Sort by grantee age (pre/post 1990 launch)

Informal contacts with elected officials

Hosting clinic tours for policymakers

Helping draft regulations, rules

Findings: Grantee Advocacy Capacity, 2002 to 2006

Number: engaged in more grant-funded advocacy activities--increase from 73% to 86% of surveyed activities.Target: consistently targeted most activities (80%) at the local and state levels and less (60%) at the federal level.Effectiveness: fairly constant or 3.28 to 3.36 (4 equaled “very effective” Grantee type: newer (post-1990) consortia were equally effective in achieving advocacy goals by 2006.

Outcome 2: IncreasePolicymaker Awareness

Whoever controls the flow of information dictates our perceptions and perspectives; whoever controls the news shapes our destiny.

- George ClintonMusician and Funk Philosopher

Evaluating Grantee Work With the Media

Effectiveness of media compared to other advocacy tactics

Advocacy Activities Worksheet: # of grantees using media at federal, state, local levels; achievement of three Program outcomes

Partnerships with media

# and type of media request for information;

Number and type of media outlets

Coverage secured # of stories in newspapers, radio

Placement of stories, e.g., page, column

Newspaper article type (op-ed, Letter to the Editor)

Presence of policy issues in the media

Inclusion of specific issues, e.g., role of clinics, patients served

Framing of issues - content analysis

Outreach to media # of press releases developed and distributed (grantee tracking)

Findings-- Media Perspective:Effectiveness of Grantee Strategies

to Obtain Media Coverage

5=very effective

Policymaker and Community Leader Familiarity with Grantee

and Clinic Activities

4 = high familiarity

Educating AND Working with Policymakers

Advocacy contributes to policymaker awareness. The challenge is to position yourself as a credible partner and work with decision makers

Watch for opportunities, e.g., bipartisan support

Leverage the partnership with decision makers pre/post passage of policies

Maintenance of effort required - ongoing education and visibility

Outcome 3: Increased Policymaker Support for Clinic for Clinic Funding

Policy targeted

Bill Number and/or title?

Grant-funded policy advocacy activities undertaken in 2007. Choose from list: a) media advocacy; b) in-person meetings with decision makers; c) drafting of rules, regulations; d) materials development; e) grass roots activities; f) Other: list

Target populations to be affected by policy. Choose from list: a) clinics; b) uninsured; c) low-income populations; d). clinic consortia; e) other special populations; f) Other: list

Outcomes as of 12/07. Choose from list: a) Passed - signed in to law; b) Vetoed; c) Pending - still being considered by decision makers; d) funding secured; e) Other: list.

Federal

Analysis:

1. Count number policies and of grantees that mentioned each policy

2. Code advocacy activities, target populations

3. Code and count outcomes, e.g., “passed”, “vetoed”, “pending”

State

Local

Tool: Annual Grantee Policy Outcomes Worksheet

Evidence of Policymaker Support – Funding Secured by Grantees (ROI)

Funding Secured, by Source

Outcomes 4 & 5: Strengthened Clinic Operations and Increased Services for The

Underserved and Uninsured

Findings: Evidence that more Californians have insurance coverage and are using the services to which they have access

OUTCOME 6: Improved Health Outcomes for Targeted Populations

Achievements in three areas:

Summary of Evaluation Findings

Advocacy as staffing and toolkit to be deployed strategically – gains after 2-3 years

Strategic involvement in policymaking process - desired policy change facilitated by maintenance of effort, technical acumen and credibility

Crafting working relationships with decision makers - “pays off”

Achieving a policy change – gains to member clinics and their target populations

Partnerships with member clinics – critical to achieving a workable solution

Policy Change Logic Model

Expand Advocacy Capacity

Increase policymaker awareness

and support

Strengthenedclinic

operations

Increasedservices forunderservedanduninsured

Improved health

outcomes

Suggestions for Evaluators

Address different information needs of advocates, funders, evaluators

Combine rigor with creativityMixed methodsTriangulateClarify what you’re evaluating re “advocacy”

and “advocacy capacity”

Create a working partnership with advocates, funder

Communications are key

Thank you! For More InformationAnnette Gardner, PhD, MPHPhilip R. Lee Institute for Health Policy StudiesUniversity of California, San Francisco3333 California St., Suite 265San Francisco, CA 94118Phone: (415) 514-1543Email: annette.gardner@ucsf.eduhttp://ihps.medschool.ucsf.edu/News/california_endowment.aspx

SF

University of California San Francisco