The future of research evaluation:Trends and prospects in research assessment

Syun TutiyaNational Institution for Academic Degrees and University Evaluation

土屋俊大学評価・学位授与機構

Novermber 2, 2012JAMJE

Outline

• Types of “research assessment”(研究評価の種類 )

• Currently in Japan(日本における状況 )

• Problems recognized(その問題点 )

• Institutional assessment of research(研究の機関別評価 )

• Alternative philosophies and methods(代替的評価 )

• Prepublication review and research assessment (査読と研究評価 )

Types of “research assessment”• By purpose and assessment agent(目的と評価主体 )

– POLISY MAKING => policy maker– EVECTIVENESS => funder– EMPOYMENT/PROMOTION/INCENTIVE => management – university, research institute

• By assessment unit(評価対象の単位 )– institution, department, research unit– project, researcher– “product” – article, protocol, database, dataset, software/program

• By transparency(客観性 )– self– external(peer/stakeholder)– third party(peer/stakeholder)

• By viewpoint(観点 )– process -- “how many articles?” (output)– quality -- “how many citations?” (outcomes)– impact -- “how much money?, what product?”

• By timing(時期 )– prior– interim– posterior

Current research assessment practices in Japan

• Project reviews– Big Grant-in-Aid projects(JSPS and others)– JST grant projects

• Institutional reviews– National University Corporation Evaluation by NIAD-UE => by third party

peers• Personal assessment

– employment/promotion– funding

• National trends– JST– NISTEP– Elsevier

Problems of current practices• Peer review is OK, but NO standards, NO benchmarks(研究評価はピアレビューで、しかし比較を全然しないのは ?)

– Typically, Nat Uni Corporation Evaluation(国立大学法人評価は、その典型 )

• “Quantitative indicators”– Conventional: citation, “Journal impact factor”– h-index(Hirsch) => researcher-level(productivity and impact) => group-level

(but co-authorship, fields, youth discriminated, etc, etc)– Altmetrics => article-level

• Commercialization– Thomson Reuters, Elsevier(Scival Spotlight)

• “World University Ranking”– ARWU by Shanghai Jiao Tong University– THE-WUR by Times Higher Education(Thomson-Reuters)– QS-WUR by Quacquarelli Symonds(Scopus) and more

What if RAE/REF in Japan?• REF: Research Excellence Framework(UK)

– For selective allocation of research funding, accountability, benchmarking information and reputational yardstick

– Assess the quality of research in UK HEI – By expert review, 36 units(4 main panel)– 5 levels(4 * to unclassified)– outputs(65%) in terms of “originality, significance and regour”,

impact(20%) in terms of “reach” and “significance”• Any use?

– Close to “research” category of Nat U C Assessment– bibliometric approach? => “citation information is not sufficiently

robust to be used formulaically or as a primary indicator of quality, but there is considerable scope for it to inform and enhance the process of expert review.”

Altmetrics

• Usually measuring article level impact – views, downloads– mentions in social media– correlation with citation counts

• I.e. post-publication impact/pre-citation– Light weight review by PLoS ONE => making it

sustainable, though– Scientific Reports review process growing heavier

• Any use for article-level measure?

eLife’s new approach“Media policy” Oct 29

1. Presenting and discussing the work prior to publicationa. Prior to publication authors are encouraged to present their findings to their

peersb. When there is media interest..., we encourage the author to deposit the accepted

version of the manuscript in an open-access repository, c. ...include a reference to eLife and/or elifesciences.org.

“NO Ingelfinger rule! NO embargo!”

2. Promotion of published contenta. Every published eLife paper will have a short, plain-language summary (the eLife

Digest).b. Papers in eLife will be promoted to the media and to interested readers on the

day of publication. c. We will issue press releases for some papers on the day of publication.