results from the performance management survey

1

Results from the Performance Management Survey

Prepared by Janice Rouiller and Laura Stouffer, SAICand

Joe Wolski, OQM

9 May 2005

2

Table of Contents• Survey Background……….…………………………………………………………...3

• Respondent Information…………………………………………………………….…7

• Perceptions of Cycle 3 PM Implementation………………………………………..15

• Perceptions of Needs for Cycle 4…………………………………………………...52

• Organizational Climate Perceptions………………………………………………...68

• Organizational Climate Perceptions by Program Area……………………………78

• Organizational Climate Perceptions by Role………………………………………84

• Customer Scorecard Ratings………………………………………………………..88

• Summary of Comments..…………………………………………………………...101

• Summary………..……………………………..………..…………………………...105

• Appendix - Individual Comments from Survey…………………………….……..111

3

Survey Background

4

Purpose

• Administer a customized version of the ORS Customer Scorecard to:• Gather feedback on past Performance

Management (PM) implementation efforts• Collect needs assessment information to tailor the

on-going implementation of PM• Gather data on the required Customer Scorecard

dimensions• Collect organizational culture change information

for the Evaluation Study

5

Methodology

• Administer the survey at the completion of the third cycle of Performance Management implementation on an organization-wide basis

• Which occurred in February 2005

• Designed and tested web survey during February 2005

• Survey went “live” on 3 February, 2005• Email requests sent to all PM Team Leaders, Members, and

additional senior ORS/ORF managers• First reminder email sent 17 February 2005• Closed survey field period 24 February 2005

• Data file sent to contractor for analysis

6

Survey Distribution

Number of email recipients 186

Number of respondents 41

Response Rate 22%

7

Respondent Information

8

Respondents by Role

311

28

75

0

5

10

15

20

25

30

35

40

45

50

Office/Associate/Division Director

PM Team Leader PM Team Member PM Consultant ManagementCouncil

Other

N = 45

Note: Multiple responses allowed.

Nu

mb

er

of

Resp

on

den

ts

9

Respondents by Organization

30

47

0

5

10

15

20

25

30

35

40

45

50

55

60

ORS ORF Other

N = 41

Nu

mb

er

of

Resp

on

den

ts

10

Respondents by Program Area

7

4

88

10

4

0

3

6

9

12

15

Mgmt Services Prgm & Empl Svcs Security & EmergResponse

Scientific Resources Real Estate &Facilities

Not Part of theServices Hierarchy

N =41

Nu

mb

er

of

Resp

on

den

ts

11

Experience with the PM Process

49%

69%

58%

63%

100%

0% 20% 40% 60% 80% 100%

Part of ASA Team in FY02

Part of ASA Team in FY03

Part of a PMP Team in FY04

Attended FY04 Managing with MeasuresTraining

Attended FY04 PM Using BSC ApproachTraining

Percent “Yes”

N = 41

N = 40

N = 40

N = 36

N = 35

12

Approximately what percent of time did you spend on PM-related activities during FY04?

2

01

0 0

2

55

8

18

0

2

4

6

8

10

12

14

16

18

20

0-10% 11-20% 21-30% 31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100%

N = 41

Percent of time Spent

Nu

mb

er

of

Resp

on

den

ts

13

On average, how often did your PM team meet during FY04?

4

8

17

9

3

00

5

10

15

20

25

30

Several Timesa Week

Once a Week Biweekly Once a Month Once a Quarter Less thanOnce a Quarter

N = 41

How Often PM Team Met

Nu

mb

er

of

Resp

on

den

ts

14

0

1 1

0

7

8

7

3 3

4

5

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

To what extent did your team members work between meetings to implement PM?

To No Extent

To a Great Extent

N = 41

Mean = 6.71

Median = 6.5

15

Perceptions of Cycle 3 PM Implementation(January 2004 - January 2005)

16

How helpful did you find the following in implementing PM?

4.78

4.96

5.38

5.44

5.46

5.74

5.86

5.88

6

6.06

6.18

6.25

6.29

6.38

6.45

6.52

7.15

7.82

7.97

1 2 3 4 5 6 7 8 9 10

PM Website

Other PM Teams

Design of Data Collection

PM Conference

Services Hierarchy

PM Related Emails

Process Behavior

Measures Reporting Form

Data Collection

Graphing Data

Measures Roadmap

Initiative Roadmap Template

Strategy Mapping

OQM Staff

ORS Customer Scorecards

PM Presentation Template

PM Template

Your Supervisor

PM Consultants

Not at all Helpful

Extremely

HelpfulMean Ratings

17

How helpful did you find the following tools in implementing PM?

6.52

5.88

6.25

6.18

6.29

7.15

5.46

1 2 3 4 5 6 7 8 9 10

PM PresentationTemplate

Measures ReportingForm

Initiative RoadmapTemplate

Measures Roadmap

Strategy Mapping

PMP Template

Listings of theServices Hierarchy

N = 41

Not at all Helpful

Extremely

HelpfulMean Ratings

N = 41

N = 41

N = 41

N = 38

N = 40

N = 41

18

5

2 2 2

5

6

4

5

1

3

4

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Listings of the Services Hierarchy

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 5.46

Median = 6

19

2

0

3

0

2

3

6

15

2

6

2

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Performance Management Plan (PMP) Template

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 7.15

Median = 8

20

4

1

2

0

3

4

6

11

1

3

5

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Strategy Mapping

Not at all Helpful

Extremely

Helpful

N =40

Mean = 6.29

Median = 7

21

4

2 2

0

2

6

4

8

2

4

6

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Measures Roadmap Template

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 6.18

Median = 7

22

4

3 3

0 0

5

4

9

4 4 4

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Initiative Roadmap Template

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 6.25

Median = 7

23

5

2

1 1

3

4

5

6

4

2

7

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Measures Reporting Form

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 5.88

Median = 7

24

3 3

1

0

3

0

5

9

2

5

6

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Presentation Template for Conference

Not at all Helpful

Extremely

Helpful

N = 38

Mean = 6.52

Median = 8

25

How helpful did you find the following data analysis services in implementing PM?

6.45

5.86

6.06

6.00

5.38

1 2 3 4 5 6 7 8 9 10

ORS CustomerScorecards

Process Behavior

Graphing Data

Data Collection

Design of DataCollection

Not at all Helpful

Extremely

HelpfulMean Ratings

N = 41

N = 41

N = 40

N = 41

N = 41

26

5

4

2

1

5

3

2

8

1

3 3

4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

How helpful did you find the following data analysis services in implementing PM:

Design of Data Collection Forms

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 5.38

Median = 5.5

27

4 4

2

0

3 3

5

9

2

4

2

3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Establishing Data Collection Methodologies

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 6.00

Median = 7

28

4 4

01

2 2

8 8

23

2

4

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se


Graphing Data

Not at all Helpful

Extremely

Helpful

N = 40

Mean = 6.06

Median = 7

29

4 4

0

1 1

3 3

10

0

3

5

7

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Process Behavior Chart Analysis

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 5.86

Median = 7

30

2

3

1 1

4

2

5

9

1

5

4 4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


ORS Customer Scorecards

Not at all Helpful

Extremely

Helpful

N = 41

Mean = 6.45

Median = 7

31

How helpful did you find the following communication vehicles in implementing PM?

5.44

5.74

4.78

1 2 3 4 5 6 7 8 9 10

PM Conference

PM Related Emails

PM Website N = 40

Not at all Helpful

Extremely

HelpfulMean Ratings

N = 40

N = 40

32

6

2

5

4

3

2

1

5

2 2

5

3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

How helpful did you find the following communication vehicles in implementing PM:

Performance Management Website

Not at all Helpful

Extremely

Helpful

N = 40

Mean = 4.78

Median = 4

33

2

3

2

3

7

2

6 6

2 2

3

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Related Emails from OQM

Not at all Helpful

Extremely

Helpful

N = 40

Mean = 5.74

Median = 6

34

6

1

4

0

4

2

5

6

1

3

4 4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Conference

Not at all Helpful

Extremely

Helpful

N = 40

Mean = 5.44

Median = 6

35

How helpful did you find assistance from the following groups in implementing PM?

7.82

4.96

7.97

6.38

1 2 3 4 5 6 7 8 9 10

Your Supervisor

Other PM Teams

PM Consultants

OQM Staff N = 39

Not at all Helpful

Extremely

HelpfulMean Ratings

N = 40

N = 38

N = 38

36

3

0

4 4

1

3

2

9

3

5

4

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


OQM Staff

Not at all Helpful

Extremely

Helpful

N = 39

Mean = 6.38

Median = 8

37

0

21

01 1

8

5

7

11

2 2

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Consultants

Not at all Helpful

Extremely

Helpful

N = 40

Mean = 7.97

Median = 9

38

7

3

1 10 0

2

7

12

3

11

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Other PM Teams

Not at all Helpful

Extremely

Helpful

N = 38

Mean = 4.96

Median = 6

39

1

0

1 1

3

1

3

8

6

9

2

3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Your Supervisor

Not at all Helpful

Extremely

Helpful

N = 38

Mean = 7.82

Median = 8

40

Immediate Supervisor Involvement in PM

7.53

1 2 3 4 5 6 7 8 9 10

To what extent wasyour immediate

supervisorinvolved in your

PM process duringthis PM cycle?

To no Extent

To a Great Extent

7.60

1 2 3 4 5 6 7 8 9 10

How helpful wasthe involvement of

your immediatesupervisor in PMrelated matters

during this cycle?

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 38

N = 39

41

1

3

1

0

2

4

2

7

3

13

1 1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

What extent was your immediate supervisor involved in your PM process during this PM cycle?

To No Extent

To a Great Extent

N = 38

Mean = 7.53

Median = 8

42

1

2

1

0

1

4

5

6

4

11

2 2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

How helpful was the involvement of your immediate supervisor in PM related matters during this cycle?

Not at all Helpful

Extremely

Helpful

N = 39

Mean = 7.60

Median = 8

43

To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?

4.19

4.79

5.03

5.54

5.54

5.68

6.18

6.03

1 2 3 4 5 6 7 8 9 10

Using Visio

Using PowerPoint

Using Excel

Working in a Team

Presentation, Preparationand Delivery

Methods to Analyze Data

Methods to Gather Data

BSC approach to PM N = 38

To No Extent

To a Great

ExtentMean Ratings

N = 40

N = 40

N = 40

N = 39

N = 38

N = 39

N = 38

44

3

2

4

1

3

5 5

7

2

4

2

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Balanced Scorecard Approach to PM

To No Extent

To a Great Extent

N = 38

Mean = 6.03

Median = 7

45

4

2

4

1

3

2

7

8

3

5

1

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Methods to Gather Data

To No Extent

To a Great Extent

N = 40

Mean = 6.18

Median = 7

46

5

3 3

2

3

2

7

6

2

4

1

2

0

2

4

6

8

10

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Methods to Analyze Data

To No Extent

To a Great Extent

N = 40

Mean = 5.68

Median = 7

47

6

1

2

4

3

1

8

5

3

2 2

3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Presentation Preparation and Delivery

To No Extent

To a Great Extent

N = 40

Mean = 5.54

Median = 7

48

5

0

45

23

56

4

1 1

3

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Working in a Team

To No Extent

To a Great Extent

N = 39

Mean = 5.54

Median = 6

49

6

2

1

3

4

7

3

5

2

0

1

4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Using Excel

To No Extent

To a Great Extent

N = 38

Mean = 5.03

Median = 6

50

8

2

0

3

5 5

3

5

2

0

1

5

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Using PowerPoint

To No Extent

To a Great Extent

N = 39

Mean = 4.79

Median = 5

51

10

01

2

43

4

2

01 1

10

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Using Visio

To No Extent

To a Great Extent

N = 38

Mean = 4.19

Median = 5

52

Perceptions of Needs for Cycle 4

53

In planning for the next PM cycle, how helpful would each of the following be to your PM Team?

5.59

5.94

5.97

6.45

6.54

6.56

6.57

6.58

6.86

7.03

7.47

7.84

1 2 3 4 5 6 7 8 9 10

Yearly Performance Mgmt Conf

Rotating Quarterly Presentation Meetings

Training on the BSC

Training on data collection systems and forms

Training on Process Behavior Charts

Training on ORS ABC Cost Accounting Model

OQM Staff

Training on Analyzing Data through Graphs

Training on Customer Assessment

Regular Meetings with Mgmt to discuss PM data

IT Support to Establish Data Collection Systems

PM Consultants provided by OQM

Not at all Helpful

Extremely

HelpfulMean Ratings

54

In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team?

7.47

5.59

5.94

7.03

6.57

7.84

1 2 3 4 5 6 7 8 9 10

IT Support to Establish Data CollectionSystems

Yearly PM Conference


Regular Meetings with your Mgmt todiscuss PM Data

OQM Staff

PM Consultants provided by OQM N = 40

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 40

N = 37

N = 39

N = 38

N = 37

55

21

21

01

4

6

8

13

2

00

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:

PM Consultants Provided by OQM

Not at all Helpful

Extremely Helpful

N = 40

Mean = 7.84

Median = 9

56

2 2

4

1

3

2

4

7

3

7

5

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


OQM Staff

Not at all Helpful

Extremely Helpful

N = 40

Mean = 6.57

Median = 7

57

3

2

1 1

3

2

3

9

1

11

0

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Regular Meetings with your Management

Not at all Helpful

Extremely Helpful

N = 37

Mean = 7.03

Median = 8

58

7

2

0 0

3 3

5

6

1

6 6

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse



Not at all Helpful

Extremely Helpful

N = 39

Mean = 5.94

Median = 7

59

7

2

1 1

4

1

5 5

0

6

5

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Yearly Performance Management Conference

Not at all Helpful

Extremely Helpful

N = 38

Mean = 5.59

Median = 7

60

21

21

21

3

65

11

21

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


IT Support to Establish Data Collection Systems

Not at all Helpful

Extremely Helpful

N = 37

Mean = 7.47

Median = 8

61

In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?

6.56

6.86

6.54

6.58

6.45

5.97

1 2 3 4 5 6 7 8 9 10

ORS ABS Cost Accounting Model

Customer Assessment (interviews, focusgroups, surveys)

Analyzing Processes with Process BehaviorCharts

Analyzing Data through Graphs

Setting up Data Collection Systems andForms

Balanced Scorecard Methodology N = 38

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 39

N = 40

N = 39

N = 40

N = 40

62

6

1 1

5

1

4

2

7

1

7

3

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Balanced Scorecard Methodology

Not at all Helpful

Extremely Helpful

N = 38

Mean = 5.97

Median = 6

63

6

2

0 0

4

2

6

8

3

7

1

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Setting up Data Collection Systems and Forms

Not at all Helpful

Extremely Helpful

N = 39

Mean = 6.45

Median = 7

64

6

1

0

1

3 3

5

9

2

8

2

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Analyzing Data Through Graphs

Not at all Helpful

Extremely Helpful

N = 40

Mean = 6.58

Median = 8

65

5

2

1

0

3

1

5

8

3

7

4

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Analyzing Processes with Process Behavior Charts

Not at all Helpful

Extremely Helpful

N = 39

Mean = 6.54

Median = 8

66

3

2 2

0

4

1

3

10

3

8

4

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Customer Assessment through Interviews, Focus Groups and Surveys

Not at all Helpful

Extremely Helpful

N = 40

Mean = 6.86

Median = 8

67

4

2

0 0

6

1

2

8

2

7

8

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


ORS ABC Cost Accounting Model

Not at all Helpful

Extremely Helpful

N = 40

Mean = 6.56

Median = 8

68

Organizational Climate Perceptions

69

4.15

3.56

3.21

3.84

3.70

3.50

4.03

3.59

1 2 3 4 5

Actively Involved in Data Collection

PM Assist with A-76

PM Contributed to Improvements

Accountability is an Important Value

Results-Oriented Culture

I Understand Services Hierarchy

Committed to PM

See Value of PM

Organizational Climate PerceptionsSummary

Strongly Disagree

Strongly Agree

N = 40

N = 39

N = 40

N = 40

N = 39

N = 40

N = 40

N = 40

Neither Agree nor Disagree

70

4 4

7

13

11

10

0

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

I see the value of performance management for improving business operations in my area.

Strongly Disagree

Strongly Agree

N = 40

Mean = 3.59

Median = 4


71

2 2

5

14

16

0 00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

My organization (ORS or ORF) is committed to the performance management effort.

Strongly Disagree

Strongly Agree

N = 39

Mean = 4.03

Median = 4


72

7

3 3

1112

4

00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

I understand what the Services Hierarchy is and its purpose.

Strongly Disagree

Strongly Agree

N = 40

Mean = 3.50

Median = 4


73

4

2

8

10

13

3

00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

The culture of my organization (ORS or ORF) is changing to be more results-oriented.

Strongly Disagree

Strongly Agree

N = 40

Mean = 3.70

Median = 4


74

3 3

6

10

15

2

00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Managers in my organization (ORS or ORF) believe accountability is an important organizational value.

Strongly Disagree

Strongly Agree

N = 39

Mean = 3.84

Median = 4


75

7

2

8

18

32

00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

Performance management has contributed to improvements in my area.

Strongly Disagree

Strongly Agree

N = 40

Mean = 3.21

Median = 4


76

7

1

5

8

13

4

2

0

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Performance management will assist my organization if we have to go through an A-76 competition.

Strongly Disagree

Strongly Agree

N = 40

Mean = 3.56

Median = 4


77

32

1

14

20

0 00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

My PM Team (or the groups I am responsible for) are actively involved in data collection and analysis.

Strongly Disagree

Strongly Agree

N = 40

Mean = 4.15

Median = 5


78

Organizational Climate Perceptions by Program Area

79

Program and Employee Services Climate PerceptionsPositive Responses

57%

71%

57%

71%

71%

100%

57%

100%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM

Percent Agree or Strongly Agree

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

80

Scientific Resources Climate PerceptionsPositive Responses

100%

50%

50%

83%

17%

33%

67%

50%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 6

N = 6

N = 6

N = 6

N = 6

N = 6

N = 6

N = 6

81

Security and Emergency Response Services Climate PerceptionsPositive Responses

75%

50%

38%

38%

75%

63%

75%

50%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 8

N = 8

N = 8

N = 8

N = 8

N = 8

N = 8

N = 8

82

Management Services Climate PerceptionsPositive Responses

67%

33%

67%

67%

67%

67%

100%

67%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 3

N = 3

N = 3

N = 3

N = 3

N = 3

N = 3

N = 3

83

Real Estate and Facilities Climate PerceptionsPositive Responses

75%

50%

25%

75%

50%

75%

50%

75%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 4

N = 4

N = 4

N = 4

N = 4

N = 4

N = 4

N = 4

84

Organizational Climate Perceptions by Role

85

Division Directors’/Office Directors’ Climate Perceptions

80%

80%

80%

40%

80%

100%

100%

80%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 5

N = 5

N = 5

N = 5

N = 5

N = 5

N =5

N = 5

86

PM Team Leaders’ Climate Perceptions

100%

43%

57%

86%

57%

57%

71%

71%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

87

PM Team Members’ Perceptions

85%

52%

52%

65%

56%

52%

77%

56%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 27

N = 27

N = 27

N = 27

N = 26

N = 27

N = 27

N = 27

88

Customer Scorecard Ratings

89

Customer Scorecard Ratings for PMOverview for Cycle 3

7.18

7.71

7.23

7.16

7.16

6.82

6.67

6.77

5.92

1 2 3 4 5 6 7 8 9 10

Handling of Problems

Competence

Convenience

Responsiveness

Availability

Reliability

Timeliness

Quality

Cost N = 40

Unsatisfactory Outstanding

Mean Ratings

N = 40

N = 39

N = 38

N = 40

N = 40

N = 39

N = 40

N = 40

90

2

01 1 1 1

3

0

4

0

25

2

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Cost Satisfaction Ratings


N = 40

Mean = 5.92

Median = 7

91

2

0

1 1

3 3

5

9

6

0

10

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Quality Satisfaction Ratings

Unsatisfactory

Outstanding

N = 40

Mean = 6.77

Median = 8

92

3

0

1 1

3

2

6 6

7

1

9

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Timeliness Satisfaction Ratings

Unsatisfactory

Outstanding

N = 39

Mean = 6.67

Median = 7

93

2

0

2

1

0

2

7

8

6

0

10

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Reliability Satisfaction Ratings

Unsatisfactory

Outstanding

N = 38

Mean = 6.82

Median = 8

94

2

0

1

3

1 1

6

7 7

4

8

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Availability Satisfaction Ratings


N = 40

Mean = 7.16

Median = 8

95

2

0

2

3

0

2

4

6

9

4

8

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Responsiveness Satisfaction Ratings

Unsatisfactory

Outstanding

N = 40

Mean = 7.16

Median = 8

96

2

0

2 2

0 0

7 7 7

4

8

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Convenience Satisfaction Ratings


N = 39

Mean = 7.23

Median = 8

97

1

0

1 1

0

2

6

7

9

4

9

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Competence Satisfaction Ratings

Unsatisfactory

Outstanding

N = 40

Mean = 7.71

Median = 8

98

2

0

2

1 1 1

5

4

9

3

10

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

en

cy

of

Re

sp

on

se

Handling of Problems Satisfaction Ratings


N = 40

Mean = 7.18

Median = 8

99

Customer Scorecard Ratings for PMTrend Analysis

7.03

7.06

7.36

6.57

6.83

6.58

6.82

6.12

6.82

6.67

6.77

5.92

1 2 3 4 5 6 7 8 9 10

Reliability

Timeliness

Quality

Cost

Cycle 1 Cycle 2 Cycle 3Unsatisfactory Outstanding

Mean Ratings

N = 37

N = 70

N = 67

N = 67

N = 40

N = 40

N = 39

N = 38

N = 68

N = 68

N = 67

N = 68

100

Customer Scorecard Ratings for PMTrend Analysis (cont.)

6.33

7.33

7.50

7.50

7.17

6.33

7.06

6.78

7.14

7.03

7.18

7.71

7.23

7.16

7.16

1 2 3 4 5 6 7 8 9 10

Handling ofProblems

Competence

Convenience

Responsiveness

Availability

Cycle 1 Cycle 2 Cycle 3Unsatisfactory Outstanding

Mean Ratings

N = 68

N = 66

N = 69

N = 65

N = 40

N = 40

N = 39

N = 40

N = 40

N = 68N = 69

N = 68

N = 68

N = 68

N = 65

101

Summary of Comments

102

What did you value most about this cycle of performance management?

• Knowledge and Experience Gained (4)• Results show that we have improved since last year.• Surveys have provided an opportunity to find out how

customers perceive quality of service.• Data gathering has helped quantify quality of service.

• Assistance provided by consultants and OQM staff (3)• Interaction with OQM staff & consultants in analyzing

performance.• Knowledgeable and helpful consultants.

• Measurement Tools (3)• The use of data reporting template and the initiatives template.• The Measures Reporting Form allowed us to look at the

objectives and measures in a more relevant perspective.

Note: Comments were sorted into categories. Most frequently mentioned categories listed first.

103

What obstacles were most challenging to you during this cycle of performance management?

Note: Comments were sorted into categories. Most frequently mentioned categories listed first.

• Lack of Time and/or Resources (6)• Lack of time to devote to PMP and keep up with workload• Short notice• OQM website is out of date

• Miscellaneous (4)• Learning to use Visio and Excel• Preparing the report

• Lack of Management Involvement (3)• Not having discussions about PM in staff meetings• Managers need to use data in decision making

• Data Collection and Measurement (2)• Obtaining data

• Lack of commitment from team members (2)• Getting all team members to contribute

104

How can we help integrate PM into your daily business activities?

• Increase support currently provided (6)• Need help in learning how to distribute data collection

throughout the division.• Need help collecting and interpreting data.

• Miscellaneous (3)• Work to establish timelines so that we see the year at a glance.• Continue emphasis on the importance of performance

management.

105

Summary

106

SummarySurvey Respondents

• Response rate of 22%• Respondents were primarily from ORS (73%)• The majority of respondents were team members

(68%)• About 75% of respondents were experienced with

PM (i.e., participated in FY03)• About 60% of respondents attended FY04 training• About 63% of respondents spent 20% or less of their

time during the past year on PM• Team meetings typically occurred once a month

(41%), biweekly (22%), and quarterly (20%)

107

Summary (cont.)

Feedback on Past Cycle

• Most Helpful Tools/Resources• PM consultants• Service group supervisors• PM template• PM presentation template• ORS Customer Scorecards• OQM Staff. • These are the same tools/resources receiving the highest ratings in

FY03.

• KSA Improvements• Just above mid-scale improvements noted on all topics with the

following exceptions: PowerPoint and Visio. • Perhaps respondent familiarity with these tools has leveled off.

108

Summary (cont.)

Planning for Next PM Cycle

• Future needs• Most helpful to teams for the future would be

• PM consultants• IT support for data collection systems• Regular meetings with management to discuss PM data• Training on customer assessment • Training on analyzing data

• These requests are similar to previous years

109

Summary (cont.)

Organizational Climate

• Majority of respondents either agree or strongly agree that:

• PM Teams are actively involved in data collection (85%)• Their organization is committed to PM (77%)• Managers in their organization believe accountability is

important (64%)• They see the value of PM for improving business operations

(60%)• The culture of their organization is changing to be more

results-oriented (58%)

110

Summary (cont.)

Customer Scorecard Trend Analysis

• Ratings on dimensions show minor variance from cycle to cycle but none of the differences are statistically significant.

• Current ratings:• Highest on competence, convenience, and handling of

problems, • Lowest on cost

111

Appendix

Individual Comments From Survey

112

What did you value most about this cycle of performance management? Individual Comments

• The opportunity to find out how my customers perceive the quality of the job I'm doing.

• Data gathering enabled us to quantify that construction inspection was decreasing our submittal review response time.

• Our consultant Janice Rouiller.• The survey provided verification of the quality of service DMCS is

providing the NIH community.• The numbers show that last year's improvements have not only

been sustained, but improved upon.• Honestly, I must state NOTHING! This ploy that the PMP process is

management tool is hog-wash, the true intent of the PMP is to tie it to how much time staff spend in each function and to cut staff.

• Interaction with OQM staff & consultants in analyzing performance.

113

What did you value most about this cycle of performance management? (cont.) Individual Comments

• Patrick Patterson, is the consultant with whom our team works on PMP. His skills & knowledge of all aspect of the program are excellent. He ensures that we ‘see’ the process & purpose rather than ‘just do it’.

• The willingness of the ORS Scientific resources to be first in the rotation of PM conferences.

• The use of data reporting template and the initiatives template.• Meeting as a group to plan ahead.• the introduction of the use of the Business Framework and the

concept of the Balanced Scorecard.• It was helpful to focus just on the high impact objectives.• The Measures Reporting Form allowed us to look at the objectives

and measures in a more relevant perspective.

114

What obstacles were most challenging to you during this cycle of performance management?Individual Comments

• Preparing the report is painful.• Being taken away from my primary duties created undue stress and

hardship when OQM wanted their deadlines met, yet they couldn't seem to get their parts done in a timely fashion.

• We did not have the time to do the customer survey focus groups we had planned on doing.

• OQM website is out of date and needs updating.• The challenge of continuing performance gains without losing any

quality.• learning to use Excel and Visio.• Lack of time to devote to PMP AND perform ones own work. PMP

is almost a full time job. This is an OQM function and should be performed by that staff.

• Supervisors and managers should use data more often in their decision-making.

115

What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments

• Commitment from all the team members.• obtaining data.• Getting all team members to contribute to the tasks of identifying

relevant data and analyzing it.• The way the PMP was created in Excel with an entire page in one

row of cells made it extremely time consuming & tedious to update.• Switching the rules in the middle of the game! We went from PMP,

to score card, then to the dash board. These aberrations made staff wonder what value all of this has to our the ultimate duty to provide the service to the Institutes.

• Not having discussions about the performance of service groups/discrete services in staff meetings.

• Lack of a program area manager who accepts PM plans and manage their implementation.

116

What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments

• Concentration on the Measures Reporting Form throughout the year caused the PMP to be neglected. At the end of the cycle, it took a lot of time to update the PMP to match the Measures Reporting Form.

• The short notice and scheduling a report due for the week of January 17 was inane!

117

How can we help integrate PM into your daily business activities? Individual Comments

• We have always been collecting and analyzing data to improve our program--no value added from this formal process.

• In my opinion performance management might be applicable for some businesses in private industry but not for the US Gov,t.

• It already is, and was before I was tasked with being a part of this initiative.

• Work to establish timelines so that we see the year at a glance. Several times my consultant worked with me to make deadlines at the last minute.

• Implement the ORF design review and inspection processes• Already done.• Have the ORS Office of Quality Management (OQM) staff do the work.

In many agencies and in private sector companies this function is not put on the backs of the line staff but is performed by either consultants or an office such as OQM.

118

How can we help integrate PM into your daily business activities? (cont.) Individual Comments

• Need help in learning how to distribute data collection throughout the division.

• Follow up with Quarterly team meetings.• Continue emphasis on the importance of performance management to

efficient 7 effective business operations.• Need help collecting observation type data.• Need help interpreting data.

results from the performance management survey

Documents