european research council │ 1 granting and evaluation of research in humanities and social...

European Research Council

│ 1

Granting and Evaluation of Research in Humanities and Social Sciences at the European Level – and especially at ERC and ESF

Alain PEYRAUBE

CNRS – EHESS

Membre of the ERC Scientific Council


│ 2

European Reports on Evaluation

• AUBR Report « Assessing Europe’s University-Based Research » by Mackiewicz (Summer 2009, 130 pages). Detailed lists on the evaluation system in Australia, Belgium, Finland, France, Germany, Hungary, Ireland, Italy, The Netherlands, Norway, Spain, Sweden, United Kingdom Suède, on Webometrics, on ARWU (Shanghai rankings, on THES, on PRSP (Taiwan), on CWTS (Leiden)

• SPRU report « Towards a Bibliometric Database for the Social Sciences and Humanities – A European Scoping Project » (ANR, DFG, ESRC, AHRC, NWO, ESF) – January 15, 2010


│ 3

European Reports on Evaluation (2)

• Article « Journal Base – A comparative international study of scientific journals in the social sciences and the humanities », M. Dassa, C. Kosmopoulos, D. Pumain Cybergeo, January 2010

• Article « Research Quality Assessment and the Metrication of Social Sciences » (Myriam E. David, European Political Science 2008.7, 13 pages)

• Bibliométrie et évaluation en SHS (Ghislaine Filliatreau, Revue d’histoire moderne et contemporaine 55-4, 2008, 4 pages)


│ 4

FP7 (EC)FP7 (EC)

7th Framework Programme (FP7) 50.52 Billion Euros, 4 Specific Programs

“Family” of FP7 Specific Programmes Co-operation 33 B Euros (8 priorities)

Ideas 7.5 B Euros [ERC]Ideas 7.5 B Euros [ERC]

People 5 B Euros (Marie Curie fellowships)

Capacities 4 B Euros (Infrastructures, etc.)

Ideas: complementary to other FP7 supportpolicy vs. science-driven, bottom-up vs. targeted research


│ 5

The Scientific CouncilThe Scientific CouncilMembers & RoleMembers & Role

22 most respected researchers reflecting the full scope of European research and scholarship

proposed by an independentidentification committee

appointed by the Commission(for 4 years, renewable once)

Role: Establishes overall scientific strategy

establishes annual work programmes (incl. calls for proposals, evaluation criteria); defines peer review methodology; ensures selection and accreditation of experts

Controls quality of operations and management Ensures communication with the scientific community


│ 6

The Executive AgencyThe Executive Agency

Executes annual work programme

as established by the Scientific Council

Implements calls for proposalsand provides information and support to applicants

Organises peer review evaluation

Establishes and manages grant agreements

Administers financial aspectsand follow-up of grant agreements


│ 7

Strategic AimsStrategic AimsOverviewOverview

• ERC : first pan-European funding agency for frontier research

• Funding directed to individual teams and projects selected solely on the criteria of excellence

• Operates according to the principles of scientific excellence, autonomy, efficiency and transparency Autonomous scientific governance Simple, user-friendly delivery


Strategic Aims (2)Strategic Aims (2)

• Reinforce excellence, dynamism and creativity in European research

• Improve the attractiveness of Europe for the best researchers from both European and third countries, as well as for industrial research investment

• ERC grants can complement the efforts of national funding bodies and host institutions to build or reinforce excellence across Europe


│ 9

ERC Grants: Open to individual teams anywhere in the world

• ERC is the only component of the FP7-family to widely open its programs and schemes to individual teams anywhere in the world,

• under only one condition, i.e. that funded projects will have to be located in EU or its associated countries

• Thus, PIs and team members can be of any nationality, working in almost any country

• Team may consist solely of the PI (mathematics,humanities)


│ 10

Two types of Grants

• ERC Starting Grants, supporting excellent early stage independent investigators (2-9 years since completion of PhD) – 2007 and 2009.

Starting Grant divided into two categories from 2010 on : Starters (PhD : +2, - 7) and Consolidators (PhD : + 7, - 12). Grants : up to 1.5 M euros for a period of 5 years (+ additional Euros 500,000 for eligible ‘start-up’ costs)

• ERC Advanced Grants, supporting investigators at all subsequent stages – 2008 and 2010. Grants up to 2.5 M euros (+ additional Euros 1 M for eligible ‘start-up’ costs)


│ 11

For operational reasons the ScC agreed on 3 main research domains:

─ Domain 1: Mathematics, Physical Sciences, Information and Communication, Engineering Sciences, Universe and Earth Sciences (PE)

─ Domain 2: Life Sciences (LS)

─ Domain 3: Social Sciences and Humanities (SH)

Pre-allocation of call budget for ERC Grants per domain currently as follows: 45% PE - 40% LS - 15% SH

2009 : 39% PE – 34% LS – 14% SH + 13% Interdisc.

2011 : 40% PE – 35 % LS – 15% SH + 10% ID

ERC Grant schemesERC Grant schemes Budget AllocationBudget Allocation


│ 12

Panels have one Panel Chair and 10-12 Panel Members (now between 12 and 15), all selected by the Scientific Council members

20 Panels for the 1st Call : 8 PE, 7 LS, 5 SH

25 Panels now: 10 PE, 9 LS, 6 SH

Panel Chair oversees evaluation process for the proposals assigned to his/her Panel in collaboration with ERC staff

Panel Chair gives high level stamp of credibility and visibility to the whole evaluation process

ERC Grant schemesERC Grant schemes Panel StructurePanel Structure


│ 13

Panels composition

• Panel Members and Chairs Make decisions and document the decisions

• Panel chairs Ensure quality of the evaluation process Chair panel meetings (steps 1 & 2) Participate in panel chair meetings

• ERC Panel Teams (2 Scientific Officers) Ensure that rules are respected Ensure comparable process is followed in all panels Transition: composed of ERC staff and RTD

colleagues


│ 14

Evaluation criteria

1. Principal Investigator

CV of 2 pages, and Early Achievements track-record

of 2 pages for Starting Grant; CV of 2 pages, and 10-year

track record of 2 pages for Advanced Grant

2. Research Project

Extended Synopsis of 5 pages, Full scientific

proposal of 15 pages maximum for both Starting Grant and

Advanced Grant


Evaluation criteria (2)

3. Research EnvironmentStep 1 of evaluation criteria 1 and 2

Starting Grant PIs invited for interview for the 2nd step of

evaluation

Step 2 of evaluation all criteria Referees and panels evaluate and score criteria

1 and 2 numerically, which will result in the ranking of the proposals

Criteria 3 is considered as "pass/fail" and commented upon but not scored.

│ 15


Evaluation criteria (3)

• Each proposal marked on a scale of 1 to 4 for each of the 2 evaluation criteria:

4 : Outstanding, 3 : Excellent, 2 : Very Good, 1 : Non-competitive

• A proposal marked below the quality threshold of 2 on either of the two headings after the 1st stage of evaluation not retained for the second step

• Proposal marked below the quality threshold of 2 at the end of the 2nd step not funded


│ 17

Co-investigators

• Exceptional

• For interdisciplinary proposals only

• PI can include 1 or more co-investigators (co-I)

• Co-I submitted to same resubmission rule as PI

• Higher budget (up to 3,5 M€ for Advanced Grant)

• Scientific added value of including co-investigators to be assessed by panel


│ 18

New Panel structure and descriptionSocial Sciences and Humanities (18 March 2008)

• SH1 Individuals, institutions and markets: economics, finance and management

• SH2 Institutions, values and beliefs and behaviour: sociology, social anthropology, political science, law, communication, social studies of science and technology

• SH3 Environment and society: environmental studies, demography, social geography, urban and regional studies

• SH4 The Human Mind and its complexity: cognition, psychology, linguistics, philosophy and education

• SH5 Cultures and cultural production: literature, visual and performing arts, music, cultural and comparative studies

• SH6 The study of the human past: archaeology, history and memory


│ 19

DomainNumber

of proposals

%of

proposals

% of budget

pre-allocated*

Life Sciences 3,396 37,0 40

Physical Sciences & Engineering 4,408 48,1 45

Social Sciences & Humanities 1,363 14,9 15

Total 9,167

ERC-2007-StGERC-2007-StG Submission (stage 1): Massive response!Submission (stage 1): Massive response!

*Indicative budget established by ScC (Work programme 2007)


│ 20

Country of host institutionCountry of host institution

Number of proposals by domain and country of host institution (head quarters)

0

200

400

600

800

1000

1200

1400

1600

IT UK DE FR ES NL SE EL BE FI CH IL PL HU PT DK TR AT IE RO NO CZ BG SI CY SK RS HR LT EE LU IS LV MT

SH

PE

LS

34 countries


│ 21

DomainNumber

of proposals

%of

proposals

% of budget

pre-allocated*

Life Sciences 108 36 40

Physical Sciences & Engineering 137 45 45

Social Sciences & Humanities 58 18 15

Total 303 3.3

ERC-2007-StGERC-2007-StG Granted projectsGranted projects

*Indicative budget (Scientific Council, ERC work programme 2007)


│ 22

Distribution of the 303 granted

• 303 proposals granted (3.30%): 137 PE (45%), 108 LS (36%), 58 SH (18%)

• Distribution of the 303 by host institution: 19% UK, 13% FR, 11% DE, 8.5% NL, 8.2% IT, 8% SP, 8% IL, 1 proposal from CZ

• The 58 SH: 18 UK, 10 NL, 6 FR, 4 DE, 4 ES, 4 IT, …


│ 23

ERC Advanced GrantERC Advanced Grant – 1 (2008) – 1 (2008)(ERC Advanced Investigator Researcher Grant)(ERC Advanced Investigator Researcher Grant)

Targeting researchers who have already established their independence as team leaders and are exceptional leaders in terms of significance of their research achievements (in the last 10 years)

up to 5 years, up to € 3,5 million per

grant

~2000 Starting Grants over 7 years of FP7

(2007-2013)


│ 24

ERC Advanced Grant 1ERC Advanced Grant 1

Evaluation Criteria: Excellence is the sole award criterionEvaluation Criteria: Excellence is the sole award criterion

1. Principal Investigator- Quality of research output/track-record ( 10-year track-record)- Intellectual capacity and creativity ( CV + leadership profile)

1. Research Project- Ground-breaking nature of the research- Potential impact- Methodology

1. Research Environment (assessed in Step 2 only)

- Contribution of the research environment to the project- Contribution of the project to research environment- Participation of other legal entities (if clear/ substantial added value)


│ 25

DomainNumber

of proposals

%of

proposals

% of budget

pre-allocated*

Life Sciences 766 35 34

Physical Sciences & Engineering 997 46 39

Social Sciences & Humanities 404 18 14

Total 2167

Distribution / Domain - ApplicationsDistribution / Domain - Applications


│ 26

DomainNumber

of Proposals

%of

proposals

% of budget

pre-allocated*

Life Sciences 84 33.6 34

Physical Sciences & Engineering 117 46.8 39

Social Sciences & Humanities 49 19.6 14

Total 250

Distribution / Domain - GrantedDistribution / Domain - Granted

+ 29 INTERDISC. (11 PE, 11 LS, 7 SH) = 13% of the budget = 279 (12.9%)


│ 27

Advanced Grant 1 – DistrIbution / Host Institution of the granted projects

• 63 UK, 37 FR, 28 CH, 28 DE, 20 IT, 1 NL, 15 IL, 13 SE, 13 ES, … 2 CZ

• PE Domain, out of the 117: 20 UK, 15 FR, 12 CH, 12 IT, 11 DE, 10 NL, 9 IL, 9 SE …

• LS Domain, out of the 84: 18 UK, 11 DE, 11 FR, 9 CH, 5 ES, 5 NL, 4 IT, 4 IL …

• SH Domain, out of the 49: 12 UK, 9 FR, 5 DE, 5 NL, 5 ES, 4 IT, 3 CH …


│ 28│ 28

ERC Starting Grant 2 - 2009ERC Starting Grant 2 - 2009Submitted and mainlist proposals by domain

Life Sciences

Physical Sciences & Engineering

Social Sciences & Humanities

927 (37%)

1112 (44.4%)

464 (18.6%)

69 (31.5%)

93 (42.5%)

41 (18.7%)

Submitted proposals Mainlist

∑ = 2503 ∑ = 219

16 (7.3%)Interdisciplinary Domain


│ 29│ 29

ERC Starting Grant 2009ERC Starting Grant 2009 The 238 granted proposals (9.50%)

• Distribution by host institution:43 UK, 31 FR, 28 DE, 18 ES, 17 CH, 17 NL, 15 BE, 15 IT, 15 NL, 14 BE, 14 IL, …

• In the SH domain : 12 UK, 6 DE, 6 NL, 5 BE, 5 ES, 5 IT, 2 DK, 2 FR, …


│ 30│ 30

Advanced Grant 2 - 2009Advanced Grant 2 - 2009Number of submitted and short-listed proposals

Life Sciences

Social Sciences & Humanities

512

736

335

198

247

109

Submitted proposals Short List

?Interdisciplinaryresearch

Physical Sciences & Engineering

∑ = 1583 ∑ = 554


│ 31

Main Lists : 236 (14.9%)

• PE : 99 proposals• SH : 42 proposals • LS : 81 proposals• ID : 14 proposals (6 PE, 8 LS, 0 SH)• Host Institution : 58 UK, 34 FR, 31 DE, 29 CH, 16

NL, 15 IT, 15 NL, 12 SE, 11 IL, 10 ES, …


│ 32

Distribution by domain

• 105 PE : 23 UK, 15 CH, 15 DE, 14 FR, 8 SE, 6 IL, 6 IT, 5 ES, 4 NL, …

• 89 LS : 21 UK, 15 FR, 13 CH, 12 DE, 6 NL, 5 IT, 4 ES, 3 IL, 3 SE, …

• 42 SH : 14 UK, 5 FR, 5 NL, 4 DE, 4 IT, 2 IL, 2 NO, 1 AT, 1 CH, 1 ES, 1 IE, 1 HU, 1 SE


│ 33

Research Assessment in the Humanities and the ESF-ERIH project


│ 34

ESF-SCH workshop in 2001

We can no longer allow ourselves the indulgence of attributing the lack of appropriate tools to a peculiarity of the Humanities and Social Sciences

Specialists in the Humanities should take into consideration its special characteristics and develop the corresponding tools

It is essential for the future of the Humanities to furnish itself with reliable reference tools, following the example of other scientific disciplines.


│ 35

ESF-SCH workshop in 2001 (2)

• AHCI (ISI-Thomson Scientific Web of Knowledge) not appropriate

• Urgent need for a European Reference Index for the Humanities (ERIH) as an additional tool for research evaluation, and not the exclusive means

• Request the ESF-SCH to go ahead and try to compile lists of reference journals (Stage 1)

• Develop methodology for including other formats: monographs, book chapters, edited volumes, etc. (Stage 2)


│ 36

ESF-SCH > ERIH Steering Committee

Overall responsibility with the ESF Standing Committee for the Humanities (SCH)

SCH nominates ERIH Steering Committee in Spring 2004: F. Kiefer (HU), A. Mustajoki (FI), A. Peyraube (FR, Chair), Gudrun Gersmann (DE), Marc Waelkens (BE), Michael Worton (UK).

ERIH Steering Committee responsible for: Identification of the disciplinary structure Definition of methodology including the definition of categories of

journals Approval of membership of Expert Panels (members suggested

by MO’s, SCH, ERIH StComm) Validation of journal lists proposed by Expert Panels


│ 3737

ERIH: current disciplinary structure

• Archaeology

• Art and Art History

• Classical Studies

• Anthropology

• Gender Studies

• History

• Hist. & Phil. of Science

• Linguistics

• Literarature

• Musicology

• Oriental & African Studies

• Pedagogical & Educ. Research

• Philosophy

• Psychology

• Religious Studies & Theology

Disciplines under considertation•Archives, Library & Museum Studies•Film, Media & Cultural Studies•Area Studies


│ 38

Peer review: the basis of methodology

• Peer review recognised as the only practicable method of evaluation in basic research (standard method used in evaluation of scientific production) nomination of panel members

• ESF Mos approached – with guidelines – to provide lists of reference journals discipline by discipline


│ 39

ERIH – Journal categories:

A category (expected: 5-20% of all titles):

High-ranking, international level publications;

Very strong reputation among researchers in the field;

Regularly cited all over the world

B category:

Standard, international level publications;

Good reputation among researchers in the field in different countries, occasionally cited all over the world


│ 40

ERIH – Journal categories (2):

• C category:• Important local / regional level publication;• Mainly local / regional readership, but

occasionally cited outside the publishing country

• Only European publications are considered in this group


│ 41

Process of accreditation

• Input: National panels / scientific communities provide lists of journals

• Selection: Expert Panels define scope, analyse and assess input, produce lists

• Consultation: MOs, subject associations (European level and some national), specialist research centres

• Calibration/harmonisation: ERIH Steering Committee; • After approval by ESF-SCH, publication of ‘initial lists’ took

place in 2007• Open feedback process via on-line questionnaire for

editors and publishers• ‘Revised lists’ to be published in 2010


│ 42

Some examples

• Anthropology: 752 journals proposed by Mos 242 journals selected: 40 A (16.5%), 112 B (46.3%), 90 C (37.2%)

• Archaeology: 1312 (Mos) 425: 91 A (21.4%), 170 B (40%), 164 C (38.6%)

• History: 1419 (Mos) 934: 135 A (14.45%), 380 B (40.7%), 419 C (44.9%)

• Linguistics: 1093 (Mos) 585: 96 A (16.4%), 92 B (36.5%), 275 C (47%)


│ 43

Some examples (2)

• Literature: 1453 (Mos) 976: 255 A (26.1%), 481 B (49.2%), 240 C (24.6%)

• Music and musicology: 300 (Mos) 166: 21 A (12.6%), 86 B (51.8%), 59 C (35.5%

• Philosophy: 658 (Mos) 305: 44A (14.4%), 130 B (42.6%), 131 C (42.9%)

• Relig. Stud. &Theology: 745 (Mos) 371: 76 A (20.5%), 203 B (54.7%), 92 C (24.8%)


│ 4444

ERIH: challenges and criticism

• Misunderstandings about the character

of the currently used A/B/C categories: ranking or assessment of audience, distribution and reach?

• Misunderstandings around category C seen as ‘low quality’ when the idea is to identify quality European journals with, mostly linguistically, limited circulation; identification of quality national journals is the main innovation of ERIH

• Following discussions in the research community, process of renaming ERIH categories is underway


│ 45

More misunderstandings

• Some research councils and research organisations have used ERIH as a tool for assessment of individual research production / productivity

• ERIH “initial lists” were used when they were still under revision


│ 46

New categorization of Journal categories

• National Journals (former category C): European publications with a recognised scholarly significance among researchers in their respective research domains in a particular (mostly linguistically circumscribed) readership group in Europe; occasionally cited outside the publishing country, though their main target group is the domestic academic community

• International Journals (former categories A and B): both European and non-European publications with an internationally recognised scholarly significance among researchers in their respective research domains, and which are regularly cited worldwide


│ 47

New Journal categories (2)

Differentiation between current categories A and B is based on a combination of two criteria: influence and scope:

Category A• international publications with high visibility and a very strong

reputation and influence among researchers in the various research domains in different countries

Category B• international publications with significant visibility and a good reputation

and influence in the various research domains in different countries.


│ 48

Next steps

• Reorganisation of the expert panels by using a panel rotation mechanism and inclusion of new experts

• Expert Panel meetings revising ‘initial lists’ based on received feedback (adding in new journals, eliminate some journals, changing and revising categories)

• Consolidation of the lists in 2009 (First update) > New lists (Revised lists) to be published in 2010

• Workplan for inclusion of other publications (monographs, edited books, proceedings, etc.)


│ 49

ERIH - contact

www.esf.org/erih


│ 50

Merci

Thank you

谢谢

european research council │ 1 granting and evaluation of research in humanities and social...

Documents