תכנית קרב יחידת הערכה

Evaluation Utilization:Evaluation Utilization:From theory to practiceFrom theory to practice

The 9th European Evaluation Society International ConferenceOctober 6 - 8, 2010, Prague

This study was carried out under the This study was carried out under the auspices of the Evaluation Unit of tauspices of the Evaluation Unit of the

Karev Educational Program

Neriah Shahor, Ilan Shina, Anat Sarid, Zahava Sa’ar,:Evaluation Unit , Neriah Shahor, Ilan Shina, Anat Sarid, Zahava Sa’ar,:Evaluation Unit , Karev Karev Educational ProgramEducational Program

Dr. Ari Neuman: Dr. Ari Neuman: Western Galilee CollegeWestern Galilee College

Prof. Riki Savaya: Tel Aviv UniversityProf. Riki Savaya: Tel Aviv University

22

This presentation includesThis presentation includes::

1.1. The mapping process – The mapping process – creating a creating a concept map of evaluation utilizations concept map of evaluation utilizations (EUs) used in the (EUs) used in the KarevKarev program program

• Bill Trochim * - CornellBill Trochim * - Cornell• Participatory process which involves people within Participatory process which involves people within

the organizationthe organization

2.2. Examination of the EUs actual and Examination of the EUs actual and preferable statespreferable states

3.3. Promoting evaluation utilizationPromoting evaluation utilization

** http://www.socialresearchmethods.net/

33

KarevKarev Educational Program Educational Program

Provides enrichment programs in Provides enrichment programs in elementary and pre-schoolselementary and pre-schools

Launched in1990Launched in1990 Serves over 260,000 childrenServes over 260,000 children Internal evaluation unitInternal evaluation unit

Data & informationData & information Work concept evaluation Work concept evaluation Coordinates & administers organizational informationCoordinates & administers organizational information Assimilates evaluation cultureAssimilates evaluation culture Promotes reflective processes Promotes reflective processes

44

Mapping ProcessMapping Process

a.a. Create a list of possible EUsCreate a list of possible EUs By brainstorming in the Evaluation Unit and EUs By brainstorming in the Evaluation Unit and EUs

suggested by stakeholders in the organizationsuggested by stakeholders in the organization This generated a list of more than 120 utilizations, This generated a list of more than 120 utilizations,

which was then reduced to 71!which was then reduced to 71! Some utilization examples are listed below:Some utilization examples are listed below:

Data collectionData collection Summarizing knowledgeSummarizing knowledge Confirming and refuting knowledgeConfirming and refuting knowledge Assistance with strategic planningAssistance with strategic planning Creating dialogueCreating dialogue Examining efficiencyExamining efficiency Developing an evaluation cultureDeveloping an evaluation culture

55


a.a. Create a list of possible EUsCreate a list of possible EUs

b. Group the EU list according to b. Group the EU list according to common denominatorscommon denominators

The grouping is done by the Evaluation Unit The grouping is done by the Evaluation Unit and other stakeholders in the organizationand other stakeholders in the organization

Each participant grouped the EUs as they Each participant grouped the EUs as they saw fit.saw fit.

The number of “divisions” was equivalent to The number of “divisions” was equivalent to the number of participants the number of participants

66


a.a. Create a list of possible EUsCreate a list of possible EUsb.b. Group the EU list according to common Group the EU list according to common

denominatorsdenominators

C. Create a concept mapC. Create a concept map Load data about the EU groupings into the Concept Load data about the EU groupings into the Concept

System softwareSystem software Process the data and create a points map where each Process the data and create a points map where each

point represents a utilizationpoint represents a utilization EUs grouped together by a large number of EUs grouped together by a large number of

participants will be located close to each other and participants will be located close to each other and vice versa vice versa

77

Points MapPoints Map

88

Grouping single EUs into clustersGrouping single EUs into clusters

99

Grouping single EUs into clustersGrouping single EUs into clusters

1010

Interpreting the findings – identifying the Interpreting the findings – identifying the common denominator of each cluster common denominator of each cluster

1111

Interpreting the findings – identifying the Interpreting the findings – identifying the common denominator of each cluster common denominator of each cluster

Evaluation Outcomes Mapping

Information & Data Knowledge management

Methodical Thinking

Evaluation Culture

Marketing Organizational preservation

discourse

Initiating Organizational Processes

Developing Organizational Strategy

Organizational reflection

1212

ExampleExample of utilizations grouped in clustersof utilizations grouped in clusters

1. Information & Data 1. Information & Data 11. Data collection. Data collection 77. Data supply. Data supply 2222. Building up a database. Building up a database 4242. Creating an archive. Creating an archive 7070. Documentation. Documentation

2. Knowledge management 2. Knowledge management 22. Confirming and refuting knowledge. Confirming and refuting knowledge 3131. Inclusion of knowledge. Inclusion of knowledge 2626. Building knowledge. Building knowledge 5353. Organizational knowledge management. Organizational knowledge management 6969. Sharing knowledge. Sharing knowledge 7171. Summarizing knowledge. Summarizing knowledge......

1313

Areas in which clusters have a Areas in which clusters have a common denominatorcommon denominator

1414 Organizational preservation Organizational influence

Discourse

Evaluation Theory Evaluation Practice

Areas in which clusters have a Areas in which clusters have a common denominatorcommon denominator

1515


Organizational preservation Organizational influence

Discourse

1616

discourse


Organizational influence

Organizationalpreservation

Evaluation PracticeEvaluation Theory

Organizational influence Organizational preservation

Discourse

1717

discourse




Evaluation

Organization

1818

discourse




1919

discourse




2020

This presentation includes:This presentation includes:

1.1. The mapping process – creating a The mapping process – creating a concept map of EUs used in the concept map of EUs used in the KarevKarev program program

2.2. Examination of the EUs actual and Examination of the EUs actual and preferable states preferable states


** http://www.socialresearchmethods.net/ http://www.socialresearchmethods.net/

2121

Review of actual state & Review of actual state & preferable statepreferable state

Setting up an Setting up an online review of evaluation review of evaluation utilizations for stakeholders in the utilizations for stakeholders in the organizationorganization

Developing the review Developing the review based on clustersbased on clusters found in the mapping processfound in the mapping process

Grading each cluster on a scale of 1-5 Grading each cluster on a scale of 1-5 according to the extent that the EU according to the extent that the EU actually occurs and the extent that it actually occurs and the extent that it would be preferable that it did occurwould be preferable that it did occur

2222

Review of actual & preferable Review of actual & preferable statesstates

actual preferable difference

Evaluation Theory 3.15 4.39 1.25

Evaluation Practice 3.72 4.49 0.84

Organizational Influences 3.12 4.18 1.10

Organizational Preservation 2.70 3.92 1.22

Discourse 2.90 4.03 1.21

2323

Positioning findings to generate a map with Positioning findings to generate a map with aa vertical dimension – the difference vertical dimension – the difference

between the actual and preferable states between the actual and preferable states

discourse




2424

Positioning findings to generate a map with Positioning findings to generate a map with aa vertical dimension – the difference vertical dimension – the difference

between the actual and preferable states between the actual and preferable states

2525

Summary – Summary – from theoretical concept to practicefrom theoretical concept to practice Theoretical conceptualization, including Theoretical conceptualization, including

identifying types of EU in the organization identifying types of EU in the organization and examination of the link between themand examination of the link between them

Examining the practical consequences of Examining the practical consequences of this conceptualization regarding the this conceptualization regarding the perception of actual and preferable status perception of actual and preferable status in the organization.in the organization.

2626

This presentation includes:This presentation includes:

1.1. The mapping process – The mapping process – creating a creating a concept map of EUs used in the concept map of EUs used in the KarevKarev programprogram

2.2. Examination of the EUs actual and Examination of the EUs actual and preferable states preferable states


** http://www.socialresearchmethods.net/ http://www.socialresearchmethods.net/

2727

Promoting Evaluation UtilizationPromoting Evaluation Utilization

The Evaluation Unit is responsible for The Evaluation Unit is responsible for encouraging and promoting the use of encouraging and promoting the use of evaluation findingsevaluation findings

The process described above contributed The process described above contributed to further understanding organization to further understanding organization needs, based on the mapping clusters, needs, based on the mapping clusters, and by demonstrating the difference and by demonstrating the difference between the actual state and the between the actual state and the preferable statepreferable state

2828

Promoting Evaluation UtilizationPromoting Evaluation Utilization To promote EU within the organization, we must:To promote EU within the organization, we must:

Draw up a work plan Draw up a work plan for the Evaluation Unit for the Evaluation Unit based on the Unit’s objectives and on the based on the Unit’s objectives and on the difference between the actual state and the difference between the actual state and the preferable statepreferable state

Select evaluation objects Select evaluation objects based on the theory based on the theory developed regarding the use of findingsdeveloped regarding the use of findings

Involve stakeholders Involve stakeholders in the process described in the process described aboveabove

Publicize the process Publicize the process described here within the described here within the organization to generate a dialogue around the organization to generate a dialogue around the subject of evaluation utilizationsubject of evaluation utilization

2929

Additional ContributionsAdditional Contributions

The process described above helps the The process described above helps the Evaluation Unit:Evaluation Unit: To reflect on its current work To reflect on its current work and also to and also to

reflect at different points in time when it is reflect at different points in time when it is exposed to similar processesexposed to similar processes

To live up to one’s principles To live up to one’s principles by making by making wise, data-based decisions about evaluation wise, data-based decisions about evaluation processesprocesses

To guide other units To guide other units in the organization in the organization through a similar mapping processthrough a similar mapping process

3030

SummarySummary

MappingMapping of EUs using participatory of EUs using participatory methodmethod

Developing a Developing a theorytheory regarding EUs regarding EUs Examining the actual and preferable stateExamining the actual and preferable state Promoting EU Promoting EU based on the studybased on the study Additional contributions (reflections, data-Additional contributions (reflections, data-

based decision making, guiding other based decision making, guiding other units)units)

3131

AppendixesAppendixes

3232

1. Information & Data 1. Data collection 7. Data supply 22. Building up a database 42. Creating an archive 70. Documentation

2. Knowledge management 2. Confirming and refuting knowledge 31. Inclusion of knowledge 26. Building knowledge 53. Organizational knowledge management 69. Sharing knowledge 71. Summarizing knowledge 14. Preliminary evaluation 58. Assist clients to become the best possible service

consumers 61. Developing programs 62. Developing an evaluation culture

4. Discourse35. Presentation of many different opinions41. Creating dialogue43. Creating dialogue between different position-holders44. Creating dialogue between different units within the organization

3333

5. Methodical Thinking 28. Defining concepts 48. Creating a theory 64. Promoting methodical thinking 63. Developing a data-based culture –

academization

7,9. Evaluation Outcomes 8. Testing training 11. Examining influence 16. Examining employees 20. Examining study outcomes 18. Examining changes as they occur in real time 9. Examining the relationship between planning

and execution 10. Examining attaining goals 12. Examining different units in the organization 13. Examining efficiency 17. Examining program activities 18. Examining changes as they occur in real time 21. Implementing controls 52. Client response

6. Evaluation Culture 30. Assimilation of evaluation culture32. Guiding position-holders for evaluation34. Providing meaning60. Developing evaluation tools

8. Mapping 49. Mapping49. Selecting programs

3434

10. Initiating Organizational Processes 3. Identifying opportunities 23. Policy backup 45. Creating motivation 47. Creating a change 68. Reflection / mirroring

11. Organizational reflection 4. Identifying points of strength 5. Identifying points of weakness and risks 15. Examining organization agenda 33. Expanding understanding of organizational

processes 6. Identifying expectations of different position holders 19. Examining internal processes within the organization 36. Raising dilemmas 54. Formulating standards

12. Developing Organizational Strategy 29. Defining targets and objectives37. Indicating directions of development59. Assistance with strategic planning55. Assistance with setting objectives40. Improving processes38. Identifying needs56. Assistance with decision making

3535

13. Organizational preservation 24. Fundraising 46. Creating a profit center Creating change 39. Reinforcing achievements 57. Assistance for position-holders dealing with clients 66. Promoting organizational agenda

14. Marketing 65. Promoting public relations 25. Emphasizing uniqueness, strength and prestige 27. Defining professional "standards marker" for marketing purposes 51. Branding 67. Promoting marketing processes

3636


Organizational preservation Organizational influence

Discourse

• Confirming and refuting knowledge• Promoting methodical thinking • Assimilation of evaluation culture

Creating dialogue between different

stake-holders

• Reinforcing achievements• Fundraising• Branding

• Identifying opportunities• Examining organization agenda• Defining targets and objectives

• Examining influence• Examining attaining goals• Mapping

תכנית קרב יחידת הערכה

Documents