research measurement for realizing quality & excellence in universities
DESCRIPTION
Deanship of Quality and Skills Development Seminar. 24 April 2012, Majmaah University, KSA. قياس النشاط البحثي خطوة أساسية لتحقيق الجودة والتميز في جامعاتنا. Research Measurement for Realizing Quality & Excellence in Universities. Tariq A. Aldowaisan, Ph.D., CSP, CQM/OE, CQE, CQA - PowerPoint PPT PresentationTRANSCRIPT
Research Measurement for Realizing Quality & Excellence in Universities
24 April 2012, Majmaah University, KSA
Deanship of Quality and Skills DevelopmentSeminar
Tariq A. Aldowaisan, Ph.D., CSP, CQM/OE, CQE, CQAAssociate Professor, Kuwait University (KU)
MD, Gulf Lead Consultants (GLC)
أساسية البحثيخطوة قياسالنشاطجامعاتنا في والتميز الجودة لتحقيق
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 2
1. Provide an overview of the meaning of quality; the relationship between quality and excellence; and their applications in the university context.
2. Recognize the importance of performance measurement in driving performance improvement.
3. Suggest the four-dimensional OECD performance measurement system, and give examples of its application in the research activity.
Seminar Objectives
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 3
1. The meaning of quality
2. Quality obligations of universities
3. From quality to excellence
4. Importance of performance measurement
5. Performance measurement systems: Stakeholders, BSC, & OECD.
6. Applications of the OECD Model in research activity
7. Concluding recommendations on performance measurement in research
8. Overcoming implementation challenges
Agenda
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 4
Quality is customer satisfaction through:
• Capture of customer requirements – Quality of Design
• Compliance with customer requirements– Quality of Conformance
What is Quality?
Who are our c
ustomers?
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 5
What is Quality?
Quality is to
• identify and encode customer requirements through STANDARD/SPECIFICATIONS, and then
• achieve customer satisfaction by complying with STANDARD.
Diameter 12 ± 0.30 mm11.7 mm 12.3 mm
Defective Defective
Payment not to exceed 30 days
30 days
Defective
CUSTOMER
REQ
UIR
EMEN
TSHard Standards in Production
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 6
What is Quality?Hard Standards in Service
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 7
What is Quality?
Continual Improvement of the QMSContinual Improvement of the QMS
Input Outputvalue adding
activities
info. flow
info. flow
CUST
OM
ER R
EQU
IREM
ENTS
CUST
OM
ER S
ATIS
FACT
IONmanagementresponsibility
managementresponsibility
productrealization
productrealization
resourcemanagement
resourcemanagement
meas, analysisimprovement
meas, analysisimprovement
productproduct
clause 4
clause 5clause 6 clause 8
clause 7
value adding activities
Management Systems – Soft Standards
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 8
• Examples from ISO 9001:2008 Quality Management System Standard:
– 8.2.1 Customer satisfactionAs one of the measurement of the performance of the quality management system, the organization shall monitor information relating to customer perception as to whether the organization has met customer requirement.
– 8.2.4 Monitoring and measurement of productThe organization shall monitor and measure the characteristics of the product to verify that product requirements have been met.
Management Standards
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 9
Hard & Soft Standards
Comparison Hard Soft
Type Technical Management
Language Specification Requirements/Criteria
Coding Numeral Shall/Must/Is/Are
Checking Inspection Audit
Difficulty of Checking Low High
Example BSI-999, CSE, ISO 17025
ISO 9001, CSE, ISO 17025, ABET
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 10
• Basic Quality Obligationgraduates’ abilities are aligned with the market’s current and potential needs and expectations.
• National Quality Obligationcontribute to the development of their nations.
Quality Obligations of Universities
Source: Aldowaisan, T. “Quality Obligations and Newly Established Universities”, Quality Beat, 1st Issue October 2011.
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 11
1. Alignment of the university’s strategic plan with the national plan.
2. Selection of a robust performance measurement system.
3. Choice and development of faculty members.
Quality Obligations of UniversitiesQuality Controls
Source: Aldowaisan, T. “Quality Obligations and Newly Established Universities”, Quality Beat, 1st Issue October 2011.
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 12
From Quality to Excellence
Customer Focus
Organization Focus
Meet Requirements
Exceed Requirements
Good operation(e.g. 3 )
Great operation(e.g. > 6)
CustomerResults
StakeholderResults
Employers
ResearchCommunity
SocietyFaculty& Staff Students
Ranking Bodies
Accredit Rank
AccreditationBodies
Satisfy Delight
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 13
BNQP – Performance Excellence Program
1Leadership
3Customer
Focus
7Results
4Measurement, Analysis, and Knowledge Management
2StrategicPlanning
6Operations
Focus
5Workforce
Focus
Organizational Profile:Environment, Relationships, and Strategic Situation
2011-2012 Education Criteria for Performance Excellencewww.nist.gov/baldrige/publications
120
85
85
90
85
85
450
Education Criteria for Performance Excellence Framework
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 14
BNQP – Performance Excellence Program
1Leadership
3Customer
Focus
7Results
4Measurement, Analysis, and Knowledge Management
2StrategicPlanning
6Operations
Focus
5Workforce
Focus
Organizational Profile:Environment, Relationships, and Strategic Situation
2011-2012 Education Criteria for Performance Excellencewww.nist.gov/baldrige/publications
120
85
85
90
85
85
450
Importance of Performance Measurement
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 15
ISO 9001 QMS
Continual Improvement of the QMSContinual Improvement of the QMS
Input Outputvalue adding
activities
info. flow
info. flow
CUST
OM
ER R
EQU
IREM
ENTS
CUST
OM
ER S
ATIS
FACT
IONmanagementresponsibility
managementresponsibility
productrealization
productrealization
resourcemanagement
resourcemanagement
meas, analysisimprovement
meas, analysisimprovement
productproduct
clause 4
clause 5clause 6 clause 8
clause 7
value adding activities
Importance of Performance Measurement
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 16
Excellence Drivers
1Leadership
3Customer
Focus
7Results
4Measurement, Analysis, and Knowledge Management
2StrategicPlanning
6Operations
Focus
5Workforce
Focus
Organizational Profile:Environment, Relationships, and Strategic Situation
Improvement Team
Education Criteria for Performance Excellence Framework
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 17
• Leadership – a priori, – 2nd weight (120)
• Results– clarity of target, – 1st weight (450)
• Improvement Team– Spearhead change– Link leadership & Results Triads
Determiners of Deployment Effectiveness
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 18
Results – Get it Right
An Encompassing Vision
• Times Higher Education• QS World University Ranking• Academic Ranking of World Universities• US News World Best Universities• etc.
Achieving success in our universities,www.glc-icm.com.
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 19
Results – Get it Right
Real Goals – Outcome based
Research
Teaching
Community Development
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 20
Performance Measurement SystemsStakeholders’ Perspectives
PMsPMs
PMs
PMs
PMs
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 21
Performance Measurement SystemsBalanced Scorecard (BSC)
Source: SmartDraw 2010
PMs
PMs
PMsPMs
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 22
Performance Measurement SystemsOECD Model – Activity Based
Source: Measuring Government Activity, OECD 2009
Inputs (Resources) Activities Intermediate
OutcomesFinal
OutcomesOutputs
organization structure andmanagement provisions
Context
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 23
Performance Measurement SystemsOECD Model
Inputs (Resources) Activities Intermediate
OutcomesFinal
OutcomesOutputs
Efficiency
Productivity Quality
Customer
National/ University Objectives
Effec
tiven
ess
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 24
OECD Performance Measures Dimensions
• Efficiency: measures how well resources are utilized to produce the desired output (resource/input-driven).
• Productivity: measures the capacity to produce the maximal output possible for given resources (output-driven).
• Quality: measures how well the produced outputs meet or exceed relevant quality dimensions standards (output-focus).
• Effectiveness: measures how well the outputs and outcomes agree with the planned/targeted outputs and outcomes (outcome-focus)
Doing Things Right Doing Right Things
Inputs (Resources) Activities Intermediate
OutcomesFinal
OutcomesOutputs
organization structure andmanagement provisions
Context
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 25
• Teaching – Undergraduate
• Teaching – Graduate
• Research
• Community Services; e.g. consultation, professional development, and testing.
Key University Activities
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 26
The Research Activity
• Projects• Papers• Patents• Spinoffs
ResearchActivity
• Budget• Faculty
• Facilities
Country Priorities
Research Community
University Ranking
Business Community
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 27
• Efficiency: the cost of funded published papers (journal & conferences). The cost may be expanded to include the administration of the research activity.
• Productivity: the number of published funded and non-funded papers (journal & conferences) per faculty (all faculty). Of course more specific measures of productivity could be developed for journals and conferences separately.
• Quality: the percentage of papers (funded and non-funded) that are published in the top quartile of the Journal Citation Reports (JCR)
• Effectiveness– the compounded annual growth rate (CAGR) of papers (journal & conference) over
a certain period (Immediate outcome).– the percentage of papers addressing national priorities (Intermediate outcome).– the percentage of papers utilized by market (Final outcome).
Suggested Performance Measures for Research Activity
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 28
Performance Measures & Objectives
• Research Investment• Faculty members
25% address Priorities
At least 20% published in top quartile of JCR
Quality1.5 paper per faculty member
productivity
2000 KD per paperEfficiency
10% utilized by market
Note: 1. Papers include conference and journal papers as well as funded and non-funded.2. Faculty members include all faculty members (active and non-active).3. For Research Investment Efficiency, we consider funded papers only.
25% address priorities
30% CAGR
Effec
tiven
ess
Research Activity Papers
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 29
Using the PM to Drive Improvement
Manage to achievetarget value
Set target value &attainment date
Determine basevalue of PM
NoUndertake
corrective actionProgress
OK?
Yes
Claim success
Target valueattained?
No
Yes
1
2
34
5
"You can't manage what you
can't measure "
__ Edwards Deming
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 30
• Process/activity based
• More appropriate for non-profit institutions
• Readily supports ratio performance metrics
• Simple, efficient, robust
• Supports doing things right and doing the right things
Advantages of OECD Model
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 31
• Focus more on reporting outcome-related performances and their development over the year.
• Include non-funded published papers as well as conference papers to calculate faculty productivity (i.e. papers per faculty); which is an important measure of immediate outcome (i.e. output).
• Include a measure of research cost to calculate research investment efficiency (i.e. KD/paper).
• Include a priority effectiveness index related to research relevance to the priority areas (i.e. KD/KD or Papers/papers).
• Include a marketability effectiveness index related to marketability of research output (i.e. KD/KD or Papers/papers).
Recommendations on Performance Measures in Research
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 32
• Include commercial effectiveness indices on spinoff businesses and patents.
• Include a leading indicator faculty research engagement (i.e. percentage of faculty involved in research).
• Include a leading indicator on faculty collaboration index (KD of researches that involves two or more faculty of total research investment).
• Include a leading indicator on interdepartmental collaboration index (i.e. % of collaboration research projects involving collaboration from different departments).
• Include a leading indicator on large-scale interdisciplinary research collaboration (i.e. % of collaboration large-scale research projects involving different disciplines).
• reflect on the impact of the level and quality of research on the university rank; through engaging credible ranking bodies.
Recommendations on Performance Measures in Research
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 33
No. Measure OECD Perspective
BSC Perspective
Stakeholder Perspective
1 Faculty productivity
Productivity Internal Process
faculty, research community, Academic Institution
2 Investment efficiency
Efficiency Financial Funding agencies
3 Publications quality
Quality Customer Research community, faculty, Academic Institution
4 Publications growth
Immediate outcome – Effectiveness
Academic Institution
5 Priority effectiveness index
Intermediate outcome – Effectiveness
National and local community
Cross-Check for Completeness
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 34
No. Measure OECD Perspective
BSC Perspective
Stakeholder Perspective
6 Marketability effectiveness index
Final outcome – Effectiveness
National and local community, Academic Institution
7 Faculty research engagement
Learning & Growth
Faculty, Academic Institution
8 Faculty collaboration index
Learning & Growth
Faculty, Academic Institution
9 Interdisciplinary collaboration index
Learning & Growth
Faculty, Academic Institution
Cross-Check for Completeness
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 35
• Ensure selected performance measures are collectively comprehensive.
• Ensure that the selected performance measures are mutually exclusive.
• Gain the benefits of each measurement framework.
Benefits of Cross-Checking
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 36
Definition of a Performance Measure
•Base Value •Assumptions
•Data Sources
•Benchmarks•Targets
•Beneficiaries
•Analyses
•Measurement Unit
•Frequency
•Formula
•Dimension
•Interpretation Reference•Goal/Objective
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 37
• Leadership Commitment
• Culture – Fear of accountability– Not meeting a target construed as a failure
• Competency building– General awareness starting at top– Deep knowledge & skills of improvement team to
cover both measurement and improvement methods and tools
• Cumbersome to monitor– Automation is a must
Implementation Challenges
Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 38
واالمتنان الشكر كل
الدويسان المحسن عبد طارق[email protected]