Peer Review Can Make Metrics Accountable同行评议可以赋予指标以责任

Kelli Barr凯利 · 巴尔

Department of Philosophy and Religion StudiesCenter for the Study of Interdisciplinarity

University of North Texas哲学与宗教学系跨学科研究中心北德克萨斯大学

Translation by ZHANG Wei

The Question at Hand目前的问题

Are metrics better tools for evaluating research than peer review?在评估研究方面，指标是比同行评议更好的工具吗？

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Are metrics better tools for evaluating research than peer review?

Increasingly, academics and administrators are answering with an unqualified “yes专家学者和管理者越来越多地倾向于给出肯定的回答，但这是有问题的。

– H-index, g-index, webometrics H 指数、 g 指数、网络计量学– US News and World Report, Shanghai Rankings 美国新闻与世界报道，上海（交通大学）大学排名– National research evaluations 国家研究评估– Hiring, promotion, and tenure decisions 人才招聘、晋升和终身教职决定

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Peer Review Bibliometrics同行评议 文献计量学

Subjective 主观性

Contingent 差异性

Non-transparent 不透明性

Unaccountable 不可解释性

Costly 高成本性

Objective 客观性

Replicable可重复性

Transparent透明性

Accountable可解释性

Low(er) Cost低成本性

Defining the dichotomy using these terms creates a distinction without much of a difference

用这些术语定义的二分法给出了一个其实并没有太多差别的区分

– Bibliometrics are ultimately based on peer review processes 文献计量方法最终要建立在同行评议的程序之上

All metrics instantiate specific values that affect the outcome of the measurements

所有的指标都是某种具体价值标准的体现，都会对评估结果产生影响

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

http://chronicle.com/article/30-Ways-to-Rate-a-College/124160/

Forbes – concerned with economic, financial, commercial, and business outcomes福布斯—涉及经济、金融、商业、贸易的结果Academic ranking of world universities – concerned with bibliometric (research) outcomes世界大学的学术排名—涉及文献计量学的（研究）结果

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Journal impact factor, calculated using Web of Knowledge期刊影响因子——基于 Web of Knowledge 数据库

Sciences Nature 34.48自然科学 《自然》

Cell 31.152《细胞》Science 29.747《科学》

Social Sciences J. of Economic Lit. 6.919社会科学 《经济文献杂志》

Political Analysis 3. 756《政治分析》Ann. Rev. of Sociology 3.702《社会学年评》

Philosophy Am. J. of Bioethics 4.000哲学 《美国生物伦理学杂志》

Phil. and Public Affairs 1.957 《哲学与公共事务》Environmental Values 1.250《环境价值》

Do social science and philosophy journals really have that much less “impact”?社会科学和哲学期刊真的是只有这么点儿影响力吗？

Journal impact factor, calculated using Web of Knowledge

Sciences Nature 34.48

Cell 31.152

Science 29.747

Social Sciences J. of Economic Lit. 6.919

Political Analysis 3. 756

Ann. Rev. of Sociology 3.702

Philosophy Am. J. of Bioethics 4.000

Phil. and Public Affairs 1.957

Environmental Values 1.250

Bibliometrics assume that all research conforms to the model of scientific research 文献计量学假定所有的研究符合自然科学的研究模式。

– Cumulative 累积的– Disciplinary 分科的

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Bibliometrics assume that all research conforms to the model of scientific research

– Cumulative – Disciplinary

Bibliometrics highlight certain kinds of research:文献计量学突出强调某些种类的研究：

– Journal publications 期刊发表– Quantitative 定量的– Methodologically focused 关注方法论的– Review papers 评议论文

Implications 启示

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Bias toward disciplinary, status quo research对学科的偏见，现状分析

Reinforcing the status quo is not morally neutral 对现状的加强在道德上不是中立的

Appeals to the supposed objectivity, transparency, etc. of metrics are used as evidence of their normative superiority to peer review

对指标的所谓的客观性、透明性等属性的诉求被用来作为它比同行评议具有规范的优越性的证据。

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Reflexive evaluation of activity toward goals

对面向目标的行动的自反性评价

Episteme知识

Rational calculation of goals对目标的理性计算

Phronesis实践智慧

Techne技艺

Accountability is about rational justification责任性涉及合理的辩护

• Definitions vary widely with context: 定义会因为语境的不同而产生很大的变化：

– Legal liability法律责任

– Return on investment对投资的回报

– Academic rigor学术的严格性

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012

Accountability is about rational justification责任性涉及合理的辩护

• Definitions vary widely with context:

• Common denominator: evaluating whether or not current tactics meet overall strategic goals

共同点：评估当前的策略是否能够达到总体的战略目标

Accountability is about rational justification责任性涉及合理的辩护

• Definitions vary widely with context: • Common denominator: evaluating whether or not current tactics meet

overall strategic goals

• Accountability is more than just rule following 责任性不仅仅是遵守规则

Conclusions:结论

Neither metrics nor peer review are an unqualified good指标和同行评议都不是最好的方法

The question of the meaning of accountability is also the question of the meaning of responsibility

责任性的意义问题同时也是一个义务的意义问题

If science funding is to be a goal-directed activity, then evaluations of the science funded cannot normatively privilege values such as objectivity for their own sake.

如果科学资助是一个以目标为导向的活动，那么对受资助的科学研究的评估就不能给予诸如为其自身着想的客观性这样的价值以特权。

‘Peer Review Can Make Metrics Accountable’International Workshop on Peer ReviewDalian University of TechnologyMonday, May 21 – Wednesday, May 23, 2012