A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy

and Public Health in Web of Science and Google Scholar

Lorna WildgaardRoyal School of Library and Information Science, Faculty of the Humanities, Copenhagen University,

Denmark, Birketinget 6, 2300 Copenhagen Spnm664@iva.ku.dk

Abstract Author-level bibliometric indicators are becoming a standard tool in research assessment. It is important to investigate what these indicators actually measure to assess their appropriateness in scholar ranking and benchmarking average individual levels of performance. 17 author-level indicators were calculated for 512 researchers in Astronomy, Environmental Science, Philosophy and Public Health. Indicator scores and scholar rankings calculated in Web of Science (WoS) and Google Scholar (GS) were analyzed. The indexing policies of WoS and GS were found to have a direct effect on the amount of available bibliometric data, thus indicator scores and rankings in WoS and GS were different, correlations between 0.24 and 0.99. High correlation could be caused by scholars in bottom rank positions with a low number of publications and citations in both databases. The hg indicator produced scholar rankings with the highest level of agreement between WoS and GS and rankings with the least amount of variance. Expected average performance benchmarks were influenced by how the mean indicator value was calculated. Empirical validation of the aggregate mean h-index values compared to previous studies resulted in a very poor fit of predicted average scores. Rankings based on author-level indicators are influenced by 1) the coverage of papers and citations in the database, 2) how the indicators are calculated and, 3) the assessed discipline and seniority. Indicator rankings display the visibility of the scholar in the database not their impact in the academic community compared to their peers. Extreme caution is advised when choosing indicators and benchmarks in scholar rankings.

Keywords: author-level bibliometrics; ranking; bibliometric evaluation; indicator properties; harmonic mean; arithmetic mean.

1

IntroductionPublication and citation rankings are interesting to study as they reflect only the countable aspects of “research quality” and scholarly performance, in this case the highly specialized activity of writing articles. Despite this, rankings are used in evaluations of career, and in funding and tenure decisions. The amount of publications and citations credited to a researcher is different in different citation databases. As a result, the calculation of author-level indicators using publication and citation data from different databases can produce different values (Bar-Ilan 2008). Logically the resulting indication of the impact of a scholar is highly database dependent and consequently the meaningfulness of author-level indicators as indications of researcher impact in the academic community is questionable. We know this because the consistency between citation databases such as Web of Science (WoS), Scopus and Google Scholar (GS) has previously been thoroughly investigated, i.a. (Yang et al 2006; Li et al 2010). In this paper expands upon what is already known. First, the paper investigates what different bibliometric counting methods on data from WoS and GS mean for our expectations of average scholar performance. Then, how each indicator mathematically combines both publication and citation counts is examined, followed by a study of where scholarly rankings from the two databases correlate.

The Thomson Reuters’ Web of Science (WoS) database is recognized as an authoritative citation index for citation analysis. WoS is a structured citation database that indexes selected publications from over 12,000 journals, covering the majority of significant scientific results, and linking to citing articles within these included and excluded journals. Through extensive evaluation of content, author-diversity, citedness and timeliness, journals are added or deleted each year. This means that the indexing policies of WoS have a direct effect on the value of author-level indicators (Testa 2012). Different versions of the WoS database permit access to different indexes within WoS, thus analyses can result in a fuller or less complete picture of the scholar’s work. Further, the limited coverage of some disciplines can result in only partial representation of a scholar’s body of work. Indexing policies limit the collection to 1) specific types of scholarly output (articles, reviews, some conference papers) and 2) cited references from every item in every journal covered in  WoS whether or not the cited work is also covered as a source publication. If however the cited work is not a WoS source publication, only citations for the first author are covered (Roediger 2006; Meho and Yang 2007). Other important scholarly outlets are ignored, such as reports, working papers, dissertations, encyclopedia articles, etc., and in this age of evaluation all research output has importance. There is also concern of the overrepresentation of American and UK-based journals to the detriment of non-English language publications (Archambault & Gagné 2004; Meho and Yang 2007).

GS is increasingly becoming a serious competitor as a citation database that can provide more transparency in scholar assessment. It is free, and allows the citation counts and computations of indicators to be replicated by anyone with internet access (Harzing 2008; Smith 2008). It is also reputedly better at providing coverage of non-English language publications from the Social Sciences and Humanities, but as coverage is uncertain, this might not be the case (Neuhaus 2006). Like WoS, GS has its own automatically generated author-level indicators, (Connor 2011). Unlike WoS, GS is a web-crawler that includes all types of articles, teaching materials, conference papers, slide presentations, technical reports, dissertations and other types of scholarly, scientific and non-academic output from across the Web. Thus the advantage of GS is that it links to publications and citing articles on the internet to give a more complete picture of the researcher. Yet because of this linking structure, GS also links to irrelevant documents and is still missing important publishers and top ranking journals, with partially indexed digital collections (Jacsó, 2008). There is a reported 15 week delay for newly published items, meaning that Google Scholar is not as regularly updated as WoS (Jacsó 2005; Neuhaus et al 2006; Harzing 2008). Further as GS does not have an open indexing

2

policy there have only been estimates about the extent of its disciplinary coverage (Wilson 2007; Harzing 2014; Kosmopoulos & Pumain 2014).

The lack of peer-review and quality assurance of publications in GS is a problem, and publication and citation data requires thorough cleaning before any indicator values can be calculated. Whichever database the publication and citation data comes from, WoS or GS, the data has to be verified and cleaned for typographical errors in the source papers and in the references, for name ambiguity, duplicates and missing data before bibliometric indicators are calculated. Meho and Yang (2007), reported that it took them 30 times longer to collect and process the data that they needed from GS as compared to data from WoS.

There are basically three ways to calculate author-level impact: count the publications, count the citations or combine the publication and citation counts to create a “hybrid indicator”. Publication and citation counts are traditionally used to indicate the influence or impact that a scholar has within the academic community. Yet if a scholar has a high citation count, it does not mean that his/her work is excellent or that all of the individual’s works are highly cited. Citations are not given equally to all publications, some papers can be highly cited, and some receive only a few citations, while others remain uncited or are too new to have been cited at all. Hybrid measures attempt to correct for this skewed distribution of citations across publications by calculating citation counts normalized to the number of papers that are performing well or by recomposing the mathematical properties of the indicator, to produce a measure of productivity and effect in a single number. The mathematical properties of these indicators are presented when the indicator is first introduced to the bibliometric community, often through an empirical example, i.a. (Zhang 2009). When different indicators are compared, i.a. (Schreiber, Malesios, and Psarakis 2012), tests are usually carried out to see if they are intuitively and mathematically reasonable and this includes assessing the degree to which an indicator has a strong correlation with other indicators (Waltman and van Eck 2012). As this paper uses the h-index, h, in an empirical validation a brief introduction of the characteristics of h follows: Hirsch’s h, is the best known example of a hybrid indicator (Hirsch 2005) though it is by no means a consistent indicator of a researcher’s overall scientific impact. h has in some tests of its convergent validity correlated well with peer judgments of research performance (Van Raan 2006; Bornmann and Marx 2011), while other tests have shown how the h can be inconsistent in certain forms of research assessment (Costas and Bordons 2007; van Leeuwen 2008). Fundamentally h behaves in a counterintuitive way in scholar ranking because of a mathematical inconsistency that cannot be remedied, which is also present in a derivate of h. As Waltman and van Eck (2012) explain, the way h aggregates publication and citation statistics in a single number, means that scholars who are ranked relative to each other, can have their ranked position reversed even when they have achieved the same relative or absolute performance improvement. Even though the robustness and appropriateness of h is questioned, the appeal of measuring the overall impact of a set of publications using one integer number has led to variations of h that attempt to correct for its flaws and still allow researchers to benefit from its advantages. h has also inspired new methods of mathematically manipulating the number of citations to papers in order to improve the estimation of a scholar’s impact. The advantages and disadvantages of h can be found in i.a. (Costas and Bordons 2007; Alonso, Cabreriazo, and Herrera-Viedma 2009; Schreiber, Malesios, and Psarakis 2012).

Author-level bibliometric indicators are dependent on the database used to identify a scholar’s publications and citations and this is because databases vary in scope. In this paper, specific ways of counting and combining publications and citations are examined, i.e. the mathematical properties of the indicators, affect indicator values and how they can also affect the correlation between scholar rankings when calculated using two fundamentally different databases. Differences in the numerical values of 17 simple author-level indicators in WoS and GS for 512 scholars across four disciplines are examined: Astronomy and Astrophysics, Environmental Science, Philosophy, and Public Health.

3

Previously these indicators have been examined in detail in Wildgaard, Schneider, and Larsen (2014) and in other extensive reviews of author-level indicators including i.a. (Alonso et al 2009; Panaretos and Malesios 2009). The disciplines were chosen from the Natural Sciences, Social Sciences and the Humanities because they represent a broad variation in publication types and citation traditions. The dataset allows us to study rankings of individual scholars from different fields and with different demographics using author-level indicators, and to subsequently analyze the relationship between scholar and indicator construction.

The present study poses three research questions:

1) Do rankings with author-level indicators in WoS and GS produce similar scholar ranks? 2) Do different counting methods affect our concept of the “average” scholar?

And consequently:3) Do discipline and seniority affect scholar rankings and our concept of the average scholar?

To answer these questions this paper studies the mathematical properties of the 17 indicators and examines how different counting methods can affect our concept of a benchmark “average” and help determine if rankings overall can produce useful information about individual-level performance. Scholars are assessed on the basis of field “averages”, or more locally on the basis of the “average” performance of their peers, thus it is important to consider how the average is calculated when it is used as a benchmark for distinguishing the “above average” from the “below average” scholars. The mathematical properties of indicators used in the rankings and likewise the statistical assumptions used in estimating the significance of the correlation between rankings must also be critically appraised. It might seem obvious that how something is counted and analyzed determines the result but often the significance tests we make in correlations are so habitual that we forget to question their appropriateness. The 17 indicators are presented in Table 1.

INSERT TABLE 1 HERE

The 17 indicators are categorized as follows:Publication-based indicators: indicate the productivity of the researcher and include P, Page, App.

Page is adjusted for the age of the publications, and App is adjusted for the number of authors written in the author byline of each paper.

Citation-based indicators: indicate the effect or reception of the scholar’s publications by their peers. Effect is counted as citations, as in C and CPP, AWCR, AWCRpa. AWCR is adjusted for the age of the publications, and AWCRpa is the number of citations normalized for the number of authors written on the author byline of each paper.

Hybrid indicators: indicate the productivity and effect of the scholar in a single number. h, ħ and h2 summarize the structure of citations to publications. AW normalizes for age, h-norm allows across field comparison for multidisciplinary researchers and supplements h by including the lower-cited papers h ignores, while the e-index supplements h by calculating the value of highly cited papers; g allows greater distinction between the order of researchers and hg allows a greater granularity in rankings of scholars with similar h- and g- indices. M-quotient and mg-quotient are the h and g indices divided by the Page indicator.

4

There are many different indicators that could have been included in the analyses in this paper. The indicators were chosen because: they are accessible for non-bibliometricians to use in scholar assessments, they are simple to calculate, they aim to measure concepts such as quality and quantity, excellence, the impact of the best papers, and they also enable cross field comparisons. These concepts are defined in the source papers listed in Table 1. The hybrid indicators are constructed to use just basic mathematical properties such as division, subtraction, the mode, geometric mean, square root or cube root of all or selected citations and publications. Once we understand how indicators combine both publications and citations using basic mathematical properties, our comparison of indicator performance across databases can move beyond discussions of the “scope” of database indexing policies.

The remainder of this article is structured as follows: in the next section the data and method are presented; followed by the result section, a discussion of the findings, their implications and a comparison to related literature that compiles the motivation for this study. A brief discussion of the methodological limitations of the paper follows, before the conclusions and recommendations are presented.

Data and Methods

1. Data sources, dataset and bibliometric indicatorsPublication and citation data was collected in Google Scholar1 (GS), using Harzing’s Publish or Perish (POP) 2 and Thomson Reuters’ Web of Science (WoS) database, accessed through Copenhagen University Library. Citations to duplicate publications in GS were combined in POP to remove duplicate citations, in WoS the relatively few duplicates and citations were sorted manually. Through a previous online questionnaire of 2544 European scholars, 741 scholars with online curriculum vitae and publication lists were identified as part of the ACUMEN FP7 project 3. These scholars represent different academic seniorities and nationalities within the fields of Astronomy (n=192), Environmental Science (n=195), Philosophy (n=222) and Public Health (n=132). The questionnaire provided demographic information for each scholar: age, gender, nationality, current affiliation, academic seniority; a brief description of the scholar’s specialty within the discipline and links to their online CVs, publication lists and other dissemination channels used. Each scholar’s publications and the citations to their works were collected in WoS and GS. These two datasets were combined, and only scholars represented in both databases were included in the final sample of researchers. Ultimately publication and citation data of 512 scholars: 190 from Astronomy, 99 from Environmental Science, 155 from Philosophy and 68 from Public Health, were used for the analysis. The sample produced 22,143 journal papers and received in total 423,371 citations from other journal papers in WoS. In GS it was possible to identify 52,227 publications and overall 746,985 citations, see Table 2. The version of WoS available to us does not include conference papers, which significantly reduced the amount of papers found for some of the scholars in the dataset, especially in Astronomy and Environmental Science where conferences are an important channel for presenting, discussing and developing current investigations. Philosophy was weakly represented in WoS.

1 http://scholar.google.com/2 http://www.harzing.com/pop.htm3 http://research-acumen.eu/

5

INSERT TABLE 2 HERE

2. Data collectionEven though both Curriculum Vitae (CV) and publication lists were included in the dataset, name ambiguity was still a problem, especially for scholars with common surnames within the same disciplines, and researchers with both a surname and a family name, which is common in Spanish and Scandinavian countries. The latter surname problem resulted in different variations in the registered order of the author’s name. To address the name problem, title of each publication was searched and verified to the corresponding scholar’s publication list. However, as not all of the publication lists were up-to-date or listed all publications, some scholars were perhaps not credited with all publications that belonged to them. Further, not all publication lists were written in English, especially in Philosophy and Environmental Science where mixed language CVs were common, as the importance of the subject matter on a national level determined the language of the publication. Google Translate was used to translate titles and abstracts to ensure foreign language publications were credited the correct scholar. Indicators were calculated using Excel.

3. Method

The first part of the investigation assessed whether or not different counting methods might affect our concept of the “average” scholar and if author-level indicators are indeed indicators of scholar performance or if they are indicators of the performance of the database. To this end Franceschet’s (2010) comparative study of indicators’ arithmetic mean values is developed by using instead the harmonic mean, which compensates for extreme outliers in the citation and publication data. Each scholar’s score for each indicator was calculated using data from WoS and GS. The harmonic and arithmetic mean value for each indicator across all scholars within the same discipline were calculated and then the ratio between WoS and GS mean indicator values were computed (Tables 4, 5, 6, and 7). The results were empirically validated to previous studies of average values. Due to the variability in the amount of publications and citations a scholar has indexed in WoS and GS, the concept of “average performance” can have a range of predictions. As seen in the above analysis, the harmonic and arithmetic mean provide very different representations of the term “average” thus making it difficult to establish the most correct representation. Furthermore, assessing if a scholar is performing above or below an average is highly dependent on where and how this average is calculated.To investigate the appropriateness of the calculated averages in WoS and GS, an example results from Astronomy was compared to previous studies that have investigated the average h-index. These studies use data from the NASA Astrophysics Data System (ADS), which is the dominant means by which astronomers search, access and read their technical literature. ADS is a database accepted by astronomers and astrophysicists as a comprehensive index; therefore in this comparison it is assumed that the estimates of average h-values in ADS-studies present more accurate disciplinary averages and will consequently allow comparison with this paper’s h-values calculated in WoS and GS. Overall, the two values will help us to assess how well the assumptions of harmonic and arithmetic distribution present a fair model of average performance. The h-index only adds information when authorship is accounted for, is normalized for the specialty and plotted as a function of time. The statistics presented here are used to investigate the h averages of scholars across databases and different counting methods, and hence this paper does not address the very important discussion of the actual

6

usefulness of the metric. The validation model is taken from Hagen’s study of bibliometric counting methods (Hagen 2010) where:

 n is the total number of empirical observations, O is the empirical observation, and E is the model prediction.

In the second part of the investigation scholar rankings produced using indicators calculated on data from WoS and GS are analyzed. The aim is to understand the extent to which the different indicator values calculated in the two data sources produce different rankings at the disciplinary and at the seniority level. The exact rank of the researcher and the standard deviation of the difference between ranks in each database are compared. It is particularly interesting to identify if there are indicators that produce the same ranking in both databases regardless of the different structure and content of the databases. For each discipline the indicators are correlated using Kendall’s tau rank correlation coefficient (τ), (Kendall 1955), as τ is an alternative to the typical correlation using Pearson’s r that uses the standard deviation as a measure of dispersion which would not be an appropriate analysis for the skewed dataset used in this paper. To understand where the correlation between scholar rankings in WoS and GS actually lies, and to investigate the effect the rankings have on the performance of the scholar, the change in rank positions of scholars in both databases are mapped.

Results

Distribution of publication and citation countsFigure 1 shows boxplots for the number of publications and the number of citations found in WoS and GS for the four disciplines. GS identifies more publications and more citations than WoS. In both WoS and GS the high value outliers in both publications and citations for Astronomy, Environmental Science and Public Health represents the same researchers. In Philosophy different researchers are the outliers in publication count compared to citation count, and these outliers are different scholars in WoS compared to GS.

INSERT FIGURE 1 HERE

In order to better comprehend the log-scales, Table 3 shows the minimum and maximum values for each discipline and in WoS and GS.

INSERT TABLE 3 HERE

Before analyzing the differences in scholar rank position, it is important to assess the composition of the data in WoS and GS. This will help in understanding the base composition of the indicators used

7

to rank the scholars. The harmonic, xh, and arithmetic mean, x, values for each indicator are presented in Tables 4 to 7. The ratio is calculated as the harmonic mean indicator value in GS divided by the harmonic mean indicator value in WoS. The ratio of the arithmetic means in WoS and GS is calculated in the same way. It is expected the WoS scores to be smaller than the GS scores because of the wider range of available data in GS than in WoS, and thus the ratio between indicator values in WoS and GS for all indicators greater than 1.

INSERT TABLE 4 HERE

INSERT TABLE 5 HERE

INSERT TABLE 6 HERE

INSERT TABLE 7 HERE

The immediate observations for both harmonic and arithmetic mean values are as follows:

Publication-based indicatorsThe numerical values of the publication-based indicators calculated in GS were higher than the values for the indicators calculated in WoS. GS identified about four times more publications than WoS for Environmental Science and Philosophy and nearly twice as many publications for Astronomy and Public Health. GS found older papers than WoS in all disciplines, and the average APP was noticeably higher in GS for Philosophy but little difference was observed for the other disciplines.

Citation-based indicatorsWoS produced on average a lower C than GS for scholars in Environmental Science, Philosophy and Public Health, while WoS produced higher C in Astronomy. However, as proportionally fewer publications to more citations were found in WoS and more publications to fewer citations in GS, the indicator CPP returns similar averages in Astronomy, Environmental Science and Public Health, ratio about 1. In Philosophy CPP is two times smaller in WoS than GS and the AWCR and AWCRpa indicator in this discipline is over 7 times smaller. In the other three disciplines, the AWCR is between 1.3 and 4 times smaller in WoS than GS and the AWCR two times smaller.

Hybrid indicatorsThere is little difference between the hybrid indicator ratios within Astronomy, Environmental Science and Public Health. Yet the ratios differ greatly within Philosophy, Table 6, particularly the g-index, xh 8.5 (x3.7¿ . The h-norm and the hg indices appear stable across both databases in Philosophy, ratio ≤1.5, and the results show that hg also provides very similar indicator values to Philosophy in the other disciplines, ratio WoS to GS, ≤ xh 1.1, ( x1.1). In Astronomy and Public Health the ratio for ħ, h, m_quot, h-norm, g, hg, mg, e and H2 are ≤ 1.5. In Environmental Science the hg, mg, e indicators return ratio values ≤ 1.4, and the ħ, h, g and H2 are up to two and a half times smaller in WoS than GS.

8

Empirical validationEmpirical data from bibliometric literature presented in the previous studies were obtained from tabulated typical h-index values of 65 Astronomers from 50 different universities in ESO (2011), 14 senior astrophysicists at the Raman Research Institute in Meera & Manjunath (2012), 621 astronomers from the Netherlands Research School for Astronomy, the American Astronomical Society and the International Astronomical Union in Kamphuis & van der Kruit (2010), 50 Italian Astronomers at the INAF (Istituto Nazionale di Astrofisica) in Gratton (2014) and 31 outstanding condensed matter and statistical physicists in North USA and Europe in Redner (2010). The scholars in these studies range from PhD students at the start of their careers to outstanding scholars with more than 30 years of research experience, resulting in a large distribution of h-values, thus comprising a scholar set very similar to the one used in this paper. The typical h-index values are compared in Figure 2.

INSERT FIGURE 2 HERE

The model of the arithmetic mean in GS captures the average performance of outstanding scholars with h-index values of x >22, while the arithmetic mean in WoS is x 18. Likewise the harmonic h mean in GS xh16 appears to capture a typical average h value whereas the harmonic h mean in WoS xh 9.4 appears to underestimate average h-index values. Overall the differences between reported h-index values and the models were large indicating that neither model is a good fit.

Lack of fit The fit to the harmonic and arithmetic h-index values was quantified by a standardized score that estimated the overall departure from the model’s prediction in WoS at xh 1.95, x 0.38 and in GS xh

1.36, x 0.37, Figure 3. A perfect fit would result in a value of 0.00.

INSERT FIGURE 3 HERE

Neither model fits the empirical data very well. The harmonic mean in WoS shows the greatest disagreement with model prediction and empirical data with a standardized departure score of 1.95, a five-fold increase over the arithmetic mean in WoS.

Correlation between scholar rankings in GS and WoSIn the following the differences and similarities in scholar rank position based on the indicators calculated on WoS and GS data are investigated. Table 8 presents the Kendall correlation coefficients. The correlation coefficients in Astronomy for Page, C, h, g, hg and h2 are very strong, >0.8, and also in Public Health the C, AWCR, AW, ħ ,h, g, hg, e and h2 a strong positive correlation in the rank position of scholars was indicated, >0.8. In Environmental Science and Philosophy only the hg index displayed a correlation, >0.8.

9

INSERT TABLE 8 HERE

The correlations are summarized as follows:

1. Scholar rankings based on publication indicators in WoS and GS, Page, App, P have a moderate to strong correlation in Astronomy, Environmental Science and Public Health (between 0.57 and 0.81); and a weak to moderate correlation in Philosophy (between 0.24 and 0.57).

2. The citation-based metrics, CPP, C, AWCR, and AWCRpa produce rankings in GS and WoS that have a moderate to strong correlation in Astronomy, Environmental Science and Public Health (between 0.63 and 0.80); and a moderate correlation in Philosophy (scoring 0.45 and 0.55 respectively).

3. The correlation of the hybrid indicators, AW, ħ, h, m_quot, h-norm, g, hg, mg, e and H2, varies from weak to strong. Generally the correlation between rankings is stronger in Astronomy (between 0.45 and 0.89) and Public Health (between 0.49 and 0.83) than in Environmental Science (between 0.37 and 0.90) and Philosophy (between 0.24 and 0.88).

4. The correlation tests between WoS and GS in all disciplines for the hg index is strong to very strong, between 0.88 and 0.91.

The scatterplots in Figures 4-5, illustrate the extent the relation between publication and citation count in WoS and GS are linear and the correlation coefficient is an estimation of how well pairs of values in both variables “fit” a straight line (Gorard 2003). The funnel shape of the data-points in the projection of the publication and citation indicators warn cautious interpretation and supplementing the statistical analysis with manual judgment is an appropriate solution.

INSERT FIGURE 4 HERE

INSERT FIGURE 5 HERE

The hg indicator displayed a stronger correlation coefficient than any other indicator in this analysis, and could be a possible indicator of scholar performance across both WoS and GS. The high correlation, illustrated in the projection of the hg index, see Figure 6, strongly indicates that the scholars in WoS and GS are probably ranked in the same position, as the τ values represent the probability that the observed data are in the same order versus the probability that the observed data are not in the same order.

INSERT FIGURE 6 HERE

However, the τ value still does not tell us anything about where the agreement lies in the rankings, i.e. if the same scholars are ranked in the middle, the bottom or randomly spread over the entire rank in WoS and GS. The scholars were divided into their seniorities and then in to top 25%, middle 50% and bottom 25% and compared the rank positions. The results are presented in Table 9. Each cell shows the Kendall correlation coefficient and the number of researchers that have the same rank position in both WoS and GS. The top ten scholars ranked using any of the indicators were generally the same in both WoS and GS, but ranked in a different order, moving about +/- 3 rank positions. It was not possible to detect a pattern in shift in rank position in the middle group of scholars and likewise for the scholars that were ranked at the bottom, since the strength of the agreement is dependent on the indicator used to rank the scholars. These observations are illustrated in Figure 7, for simplicity

10

illustrating only a portion of the data. WoS ranks the bottom scholars using publication indicators between 1 and 14 rank positions lower than in GS, but the rank uses the same scholars in both databases, whereas ranking using citation count includes different scholars in the bottom 10 in both databases. Most noticeable is that indicators that normalize for academic age, Page, m-quotient, mg-quotient, or the number of authors, App, or normalize citations across all publications, CPP, h-norm, produce very different scholar rankings in WoS and GS. Hybrid indicators that use only the productive papers and the geometric mean in their computations, h2, hg, ħ, AW, produce very similar scholar rankings in WoS and GS. The hg indicator produced the highest rank agreement, see Figure 7.

INSERT FIGURE 7 HERE

INSERT TABLE 9 HERE

The rankings are illustrated in Figure 7 using Associate Professors from Public Health, but the same disparity in rankings was observed in Astronomy, Environmental Science and Philosophy. Philosophy presents an interesting case study. The top, middle and bottom ranked scholars are different in WoS and GS across all indicators but when the hg indicator is used to rank scholars, see Figure 8, the same scholars are ranked top and middle (though in different rank positions) with a strong agreement between databases in regards to the bottom ranked scholars, hence the strong correlation shown in Table 9.

INSERT FIGURE 8 HERE

Variation between scholar rank position in WoS and GSMatching exact rank position was not informative about the rank position of 50% of “middle” scholars and unsuccessful in underpinning the stability of indicators in scholar rankings. The indicators variability was determined by calculating the standard deviation of the difference in scholar rank position (calculated from the matched pairs). Scholars were compared within seniority and in WoS and GS, Table 10.

The differences were normally distributed. Table 10 is interpreted as follows: the mean is 0 (no change), a standard deviation of 5.0 means that 50% of the scholars have gone up or down 5 rank positions around the mean, 95% of the scholars have gone up or down 10 rank positions, i.e. 2 standard deviations. The smaller the standard deviation, the smaller the change in the scholar’s rank position in the two databases, thus the more stable the indicator is for cross-database scholar rankings. Different indicators produce ranks with less variance for different seniorities within the four different disciplines. Generally, the hybrid indicators that have been designed specifically to rank scholars, i.e., the g and hg indicators produced the least change in scholar position between the two databases, outperforming the h-index in all seniorities apart from junior scholars in Astronomy. There is a possible effect of small sample sizes on the statistic and thus the results are cautiously interpreted. Supplementing the differences with significance testing will not help us conclude if the results provide evidence that the change in rank position helped or hurt scholar performance. It is not

11

possible to assume the sample values are sufficiently close to population values to calculate the standard error of estimate and informative confidence intervals, (Altman 2005).

DiscussionThis paper proposes that scholar rankings in WoS and GS are primarily caused by a combination of indexing policies, citation matching algorithms, and the mathematical properties of the different indicators not the academic performance of the scholar. Building on the work of Bar-Ilan (2008) and Fiorenzo (2013) this paper demonstrates, through the use of the harmonic and arithmetic mean scores, that WoS and GS provide quite different amounts of publications and citations and index a different number of authors per paper. This means that hybrid indicators that combine two or more of these three variables result in different numerical values and hence different scholar rankings. Extreme caution is advised when making comparisons of bibliometric indicators across fields and databases as variables influencing indicator values are clearly database dependent, (i.e. based deliberately on implemented indexing policies and software applications). Factors used to normalize indicators, such as the first publication registered for the scholar in the database, the amount of publications and citations to these included in the database, the proportion of within database citations and the availability of information on the number of co-authors indexed from the author by-line in an article are highly dependent on the depth of indexing policies in the databases. As a result very different values are used in calculating the normalized indicators in the two databases and apparently unsystematic scholar rankings are the result between databases as well as when compared to other indicator rankings in the same database. The AWCRpa indicator is a good example. One consequence for scholars who are ranked in evaluation studies, is that the same indicators calculated for the same scholar, but in two different databases, might provide a different picture of the scholar’s impact. This has also been investigated in studies by (Bar-Ilan 2008; De Battisti and Salini 2012; Farhadi, Salehi, Yunus, Chadegani, Farhabi, Fooladi, and Ebrahim 2013; Patel, Ashrafian, Almoudaris, Makanjuola, Bucciarelli-Ducci, Darzi, and Athanasiou 2013). De Battisti and Salini (2012) found that the different structures and indexing policies of four databases (Current Index to Statistics, WoS, Scopus and GS) identified different “situations” for the same scholar. Likewise, Patel et al (2013) found that the calculation discrepancy of the h-index scores in WoS, GS and Scopus databases were due to the scholar’s age, impact of time period covered by the database and the professional status of the scholar, who in Patel’s study were physicians. Similar to this present paper, Patel also found superior performance in hybrid indicators. The h-index was more consistently calculated between the databases than between raw publication counts or citation counts. They found a greater reliability between WoS and Scopus, (Cronbach’s alpha 0.91) than between the GS and Scopus (0.85), or WoS and GS (0.82). It is not the aim of this paper to provide an in depth investigation of database variables that can influence indicator values, please refer instead to Jacsó (2008) who provides an overview of the variables that effect indicator values. Below is a list of the database variables identified in Jascó’s study that were observed whilst calculating the indicators and ranking the samples in this paper:

1) the structure of the database (i.e., scope, completeness of disciplinary indexing, version or license, (retrospective capacity), composition, breadth, consistency of source coverage of the cited references, type of source documents and journal base).

2) software issues (i.e., the citation matching algorithm, methods of collecting, ranking and importing citation information, citation matching software, syntactic matching of author names, the visibility of references that can't be matched).

3) computational issues (i.e., the inclusion of self-citations, the scholars “academic age” and corresponding time period covered by the database).

12

The success of the database for locating citations is field-dependent as some sources provide better coverage of certain disciplines and publication-types than others. Lancho-Barrantes and Guerrero (2010) found that average values of indicators depend on citation habits of different disciplines, but cannot be separated from the proportion of with-in database references. The descriptive statistics of the dataset used in this present paper clearly illustrate similar citing patterns in Astronomy and Public Health, which are completely alien to Philosophy and Environmental Science, (see Figure 1). The results also show that WoS provided less publication and citation data for all four disciplines than GS, and even though the dataset had been thoroughly cleaned to remove duplicates and erroneous data, the raw publication count was between 2 and 4 times lower for all disciplines in WoS than GS, while the raw citation count was up to 13 times lower in WoS than in GS (see Tables 4 to 7). The profiles of the scholars ranked at the bottom of the disciplines when the P, C and CPP indicators were used as the ranking factors, were scholars affiliated with Southern or Eastern European institutions who published primarily in their national languages; they were amongst others Spanish, Italian, Polish, Bulgarian and Czech. Furthermore, there was a vast difference in the amount of bibliographic data for Philosophers in WoS compared to GS, resulting in a low correlation of the rankings between databases and large differences in scholar rankings. In Philosophy, GS identified twice as many citations per paper and the AWCR indicator in this discipline is 7 times larger than in WoS. This is because GS links to a broader set of publication types in addition to articles and reviews, including books and journals and works written in languages other than English, which are important for Philosophy.

Even though the amount of publications and citations found in each database was very different, it was somewhat surprising that the correlation of scholar rankings between the databases was so strong. Manual mapping of scholar rank position showed that the strong correlations were due to either agreement between scholar rankings in the top or in the bottom rank positions. Figures 7 and 8 illustrate this paper’s argument that a high correlation value is not enough to assume useful performance rankings across databases, hence one must look at where the correlation is. The commonality in Philosophy is caused by the lack of publication and citation information and thus the correlation exists mainly for scholars positioned at the bottom of the ranking where both WoS and GS have little or no bibliographic data registered for the scholar. These are not scholars who are lacking publications on their Curriculum Vitæ (CV), yet for a number of reasons they are not visible in either of the two databases. For publication-based indicators the correlation between the indicator rankings at the seniority-level was more moderate than at the disciplinary-level. It was expected senior researchers to have produced more publications and thus have a greater visibility than their younger colleagues in the databases, but this was not the case. The agreement between rankings based on publication indicators was low for all disciplines, and the agreement does not increase systematically with the seniority of the researchers. This paper concludes that the low agreements are an expression of the extent to which the scholar’s specialty and discipline are indexed differently in the two databases. Yet the correlation of the indicator values based on the publication and citation counts were generally strong, apart from in Philosophy where only the hg index scored a τ value > 0.80. It was found that this strong correlation coincided with scholars showing either a low presence or no presence in WoS and GS rather than the top performing scholars.

The hybrid indicators showed a greater agreement between ranks at the seniority level than publication or citation-based indicators. As WoS and GS have such highly different collection policies it was not expected indicator values and scholar rank position to be similar. Therefore the ability of

13

the g and hg to present less varied rankings in the two databases was impressive. Despite publication and citation counts varying between WoS and GS, the hg index both correlates strongly for all disciplines and seniorities, and produces the highest percentage of agreement between rankings; thus outperforming h. The correlation of g was weaker than the correlation of h, yet the percentage agreement is not noticeably improved by using either the h or g index. The result for hg is (Astronomy between τ 0.67 and 0.97, percent agreement between 13% and 76%; Environmental Science 0.39 and 0.90, percent agreement 20-42%; Philosophy 0.88 and 1.00, percent agreement 35-87%, and Public Health 0.85 and 1.00, percent agreement 52-61%). The result challenges the stability of the h index as an indicator unaffected by the long-tails in citation rank distribution and robustness of the h value, as a value that does not varying greatly if the number of documents included changes significantly (Vanclay 2007; Courtault and Hayek 2008). hg incorporates productive and highly cited papers by using the geometric mean as an estimate of the central point and appears to provide a better indication of researcher rankings across databases. The hg index avoids the problem of a big influence that a highly cited article can introduce on the g-index but still includes the highly cited papers that h ignores. Can this property be a possible reason for its consistency in ranks among the two databases, as it alleviates the higher number of citations received by more highly cited articles in GS compared to WoS? It was observed that the averages in GS were much higher than they were in WoS. However, caution is advised as the h and g values used in the computation of hg are by no means equivalent and cannot rationally be combined in this way to produce an indicator with improved discriminatory power (Franceschini and Maisano 2011). Nevertheless, this paper makes no assumptions about what the indicator says about a scholars’ excellence. The hg fulfills its original intention as a ranking indicator, (Alonso et al 2010), and the standard deviation of the differences between scholar rankings show that the g and hg indicators produced rankings with less variance than the other indicators derived from two very different citation databases.

Database structure, software, computational issues and disciplinary citing traditions have a direct effect on the indicator value, but at the author-level, another influencing variable warranting discussion is the mathematical property of the indicators. This study demonstrates the problems that one might encounter when calculating indicators on a small aggregate of data and the consequence this can have on the perceived effect of scholarly performance. In the computation of citation-based indicators not all papers and citations are used. They are limited only to those papers included in the database and further limited by the criteria of the indicators, for example, only papers published after a certain year, academic age, a cut off point for citations or normalizing for number of authors, which all invariably make interpreting the indicator-value even more complex. To choose appropriate indicators, we must understand the data we are working with, what they represent, and the distribution of citations to publications and hence consider the reliability of the inferences we can draw from it. Simple citation counts for each publication is only informative when put into context of the amount and type of publications a scholar has published over the length of his/her career. Average-based indicators attempt to summarize the typical performance of the scholar, resulting in a single number that can be used to compare scholars. However, if applied incorrectly, the averages can be misinformative of the importance of the scholar, as the scholar may be cited fewer times than the average seems to suggest. The arithmetic average, as in the CPP, standardizes the citation rate and approximates the typical when the shape of the distribution of citations to publications is symmetrical or normally distributed. But, this is hardly ever the case. Citation distributions are commonly skewed to the right; containing many publications with low citation values and relatively few publications with high citation values. Logically, as citation distributions are highly skewed they should not be arithmetically averaged, hence the appeal of h-type indicators that attempt to correct skewed data. Roediger (2006) points out in his column on the validity of the h-index, that h is similar to the mode

14

in that it is not sensitive to extreme values. But h is as previously discussed not robust. Transforming the data and using alpha and velocity levels to provide more robust indications of individual-level performance has been suggested as an effective improvement on h-type indicators, but these types of indicators become very complex and less reproducible. Moreover, the data used in this study was so skewed that it is not close enough to lognormal to be handled using log-transformations, however square roots, as in hg and AW, yielded a distribution that is close enough to normal to apply standard techniques. As illustrated in Figure 6, the geometric mean based hg index produced a more stable comparative estimate of scholar impact than the arithmetic mean or mode, which is what h approximates.

The difficulty of calculating effective individual-level bibliometric indicators is related to the skewed distribution of citations to publications. Because of this skew, the mathematical properties of indicators can fit some ratios of citations to papers better than others, and can inadvertently favour or disfavour scholars or disciplines. Regardless of their simplicity and the skewed distribution of citation averages (Haustein & Larivière, 2015), we must think carefully about how useful they are as indicators. In general, the arithmetic mean is not effective because aside from its presence in a normal distribution, it is not representative of the majority of documents. Neither is the median appropriate since as it disregards the most frequently cited document, and cannot fully represent the citation impact of a set of papers. They suggest providing the standard deviation with the mean and additional distribution-based indicators such as citation percentiles. Like h-type indicators, percentile indicators, Ptop10% or PPTop10% provide information about the productivity and impact of a scholar in a single figure, (Waltman and Schreiber 2013). In addition, percentile indicators eliminate the arbitrary cut-off points that are built into an indicator construction (see for example case studies of the h- and g-index in Schreiber 2013a; Schreiber 2013b). Because they are normalized for document type, field and publication year, percentiles can legitimately be used to verify if scholars are performing above or below expectation (Bornmann and Marx 2013). Nevertheless, percentile indicators can be adjusted to assess the top, middle or bottom percent, so their usefulness is still limited to how well the definition of the “field” is. Typically a “field” is based on journal categorization and is specified on the basis of the indexing policies of the database and/or related to the extent to which the scholar’s specialty is represented in the database. In light of this, percentiles add a layer of complication to individual-level bibliometrics and this paper suggests that the effectiveness of averages should not be disregarded in favour of more complicated approaches. Perhaps the solution is to develop percentile-like indicators similar to the CWTS internal coverage indicators but based instead on coverage relative to the scholar’s CV. One simply counts the eligible number of works which are visible in the index and divide that number with the total number of scholarly works in the CV. With this approach, one can get an indication of how relevant an assessment exercise becomes for the individual.

Alternatively, if integer values are required to benchmark performance rankings, the solution is to use the appropriate average instead of the most common one, i.e. the arithmetic. Using an inadequate average can lead to incorrect results and poor decisions. In this paper the aggregate arithmetic and harmonic mean indicator values are compared to investigate if different counting methods affect our concept of the “average” scholar in WoS and GS as well as the ratio between average indicator values that was used to illustrate the differences between the two databases. Due to the variability in the amount of publications scholars produce and the number of citations these publications receive, the concept of “average performance” can have a range of predictions and it is difficult to prove a correct, truthful representation. This means that assessing if a scholar is performing above or below an

15

average is highly dependent on where and how the average is calculated. The arithmetic mean, as used in a similar study by Franceschet (2010), is a parametric statistic that can be distorted if used to find the central tendency of very skewed data. To find the arithmetic mean of a set of n numbers, add the numbers in the set and divide the sum by n. This calculation assumes the distribution is not skewed and there are no outliers. Logically, this is not appropriate in bibliometrics. The harmonic mean is calculated by adding the reciprocals of a set of n numbers, dividing the sum by n, and then taking the reciprocal of the result. It is recommended in situations where extreme outliers exist in the population, as in citation distributions, and when comparing averages of different sized groups.

In this analysis, the accuracy of indicator scores could be improved by removing some of the input data that is creating a distorted bibliometric bias. In other words, fit the data to a more accurate model of average scholar performance, resulting in a more correct numerical representation of scholar performance (Hagen 2008; Tol 2009). Therefore as the dataset used in this paper contains different sized groups, extreme outliers and is highly skewed, it is assumed that the harmonic mean is a more suitable estimate of average indicator value and thus the appropriate benchmark of average expectations to scholar performance. The harmonic mean did however underestimate the average and tended to aggravate the impact of small outliers, specifically the lower performers rather than the higher ones. This stood in contrast to the arithmetic mean, which overestimated the average advantage of high performing scholars. One could argue that the large outliers are the really interesting objects in individual assessments and must not be ignored in scholar rankings. For this reason, this paper assumes an “above average” scholar would produce values higher than the aggregated mean. For example, the arithmetic mean number of publications for a scholar in Environmental Science using data from WoS is 32, compared to 75 in GS while the harmonic mean in WoS, 11.6 and in GS is 41.7. Seventy-five publications is a very high average benchmark, but this leads one to question the realistic nature of the other average estimates. To put the results in perspective aggregated h-indicator values in Astronomy were compared to previous bibliometric studies. As shown in Figures 2 and 3, both the arithmetic and harmonic predictions of a typical h-index value proved to be misleading. The difference in expectations to average performance in WoS, GS and ADS, and using different means gave conflicting results. If, as is assumed, data from the ADS database produces numerical values that are accepted as representative in the Astro-community, the indicator values calculated using data from WoS and GS produce a very distorted concept and expectation of what is considered useful as a measure of average performance. The type of mean matters in benchmarking the average performance of ranked scholars, and creates expectations related to performance, particularly with respect to our concept of a below or above average scholar. Serious consideration is warranted, as measuring expected performance with contrasting averages is not very accurate. The implementation of inappropriate averages in scholar assessment can result in disillusioned scholars and misinformed assessors. Bibliometric counting methods need to be validated against accurate notions of expected performance, accepted by the scholar community. In the above example bibliometric data for Astronomers was used, extracted from a database regarded as inclusive, and found that both the arithmetic and harmonic means poorly fit the empirical data (Hagen 2010).

The investigation in this paper is limited. First, although the results fit well with previous research findings, the possibility of selection bias due to the sampling process cannot be ruled out, and how it can influence the results in important ways (Schneider 2013). It is evident that the researcher selection and citation matching processes used to generate the dataset, under no circumstances can be considered as probability sampling procedures from known finite populations where each member has a known probability of selection, e.g.,(Freedman, Pisani, & Purves 2007). Yet this is the premise for

16

most non-experimental social science data sets. Like so many other social science data sets, the dataset used in this paper is a convenience sample. However, unlike most other studies, in the absence of a known stochastic data-generation mechanism, the data is not submitted to significance tests, or used to produce confidence intervals, as they are meaningless under such circumstances, e.g., (Berk & Freedman 2003; Schneider 2013; 2015). Without a known stochastic data-generation mechanism applied to real-world data, addressing sampling uncertainty is meaningless. Here, statistical inference based on sampling theory is absent. Second, the version of WoS this study had access to limits the representativeness of the investigation because access to conference papers was limited and the book citation index was not available. Candidates for tenure and promotion demonstrate performance in a number of different ways and in this paper the comparison of researchers is limited to just one scenario, that is, the rank of scholars to their peers. This is one possible way in which candidates can be compared in order to satisfy minimum scholarship expectations of productivity and readership.

The value of results must be seen in relation to previous comparable findings. Calculating bibliometric statistics at the individual level means the indicator values are very subjective and heavily influenced by the scholar’s specialty, publication history, age and language of publication (Jascó 2008). Ideally, the representation of each scholar in the citation databases and their resulting indicator values should be judged individually to assess if observations are due to chance or of real practical importance (Gorard 2006).

ConclusionsThis paper examines three research questions: 1) the extent rankings with author-level indicators produce similar ranks between researchers in GS and WoS; 2) if different counting methods affect our concept of the “average” scholar?, and 3) the effect of discipline and seniority on scholar rankings and our concept of the average scholar?

Correlation coefficients alone were not sufficient to analyze the similarities in rank; therefore the correlation analysis was supplemented by manually counting the agreement in rank positions to understand where the correlation actually was. It was found that a strong correlation between scholar rankings was primarily due to a lack of data in both databases rather than agreement in rankings of top performing scholars. Scholar rankings based on author-level indicators in WoS and GS put a different value on the perceived effect of the individual scholar’s performance, resulting in apparently haphazard ranking in the two databases that could be counterproductive for evaluation. Despite the problems with database dependence and missing data, the hg indicator provided a potential robust indication of scholar performance, comparable across both WoS and GS. The g indicator was also potentially robust, but to a lesser extent. The hg indicator produced the highest agreement in cross-database rankings and least variance in rank positions across disciplines and seniorities, possibly because of the use of the geometric mean of the h and g indices in the calculation. The geometric mean gives larger weight to smaller values than larger values of variables in a positively skewed distribution, which fits both the classic citation distribution and accommodates the differences in publications and citations in WoS and GS. In science it is the extremely valuable and influential papers that usually represent the great breakthroughs and hg captures this feature. It will be interesting in further studies to examine the across database stability of other author-level indicators based on the geometric mean such as the Q2 (Cabrerizo et al 2012) or the t-index (Tol 2009).

17

The differences in indicator values when calculated across different databases may have implications if used in assessing scholar rankings and in benchmarking average performance. It matters for the ranking of scholars which citation database is used to calculate the author-level indicators, which indicator is used as the ranking factor, and how the average is calculated. WoS produced good numbers but was noticeably weaker in its representation of the subject of Philosophy than it was in GS, which is why it is important to use the correct database and the correct indicator for the field/specialty. Informed choice of indicator and benchmark values and importantly how they are calculated is vital for a fair ranking assessment. Although the concept of an average may seem simple, it is important to consider which average to use and to communicate to the intended audience the method of deriving the average and the rationale for doing so. This paper recommends therefore, that end-users begin by exploring the distribution of their bibliographic data before calculating, if necessary, any bibliometric indicators. This way they can make an informed choice about the indicator that provides the best mathematical model for estimating the average effect of their publications.The hg and g indicators produced the least variance between scholar ranks and the stability of these indicators in cross-database rankings is worth investigating further. Yet it is important to reiterate, no one indicator fits all situations and to use common sense in application. The best option is to use a set of bibliometric indicators to compare the different aspects of scientific impact and to interpret what constitutes average performance in scholar in rankings.

Acknowledgements I would like to thank Dr. Jesper W Schneider and Dr. Kim Wildgaard, M.D. for their feedback on an earlier draft of this paper and the reviewers for their suggestions for revisions. I also acknowledge the ACUMEN collaboration for allowing the continued use of the dataset throughout the final year of my PhD-project. ACUMEN was a European FP7 project completed in Spring 2014, grant agreement 266632.

18

Reference List

Alonso, S., Cabreriazo, F., and Herrera-Viedma, F. (2009). H-index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3(4), 273-289.

Alonso, S., Cabreriazo, F., and Herrera-Viedma, F. (2010). hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics, 82(2), 391-400.

Altman, D. (2005). Standard deviations and standard errors. British Medical Journal, 331(7521), 903.Archambault, E., & Gagné, E. V. (2004). The Use of Bibliometrics in Social Sciences and Humanities, (Social Sciences and Humanities Research Council of Canada, SSHRCC), Montreal: Science Metrix. http://www.science-metrix.com/pdf/SM_2004_008_SSHRC_Bibliometrics_Social_Science.pdf. Accessed 30 March 2015.

Bar-Ilan, J. (2008). Which h-index? - A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257-271.

Berk, R. A., & Freedman, D. A. (2003). Statistical assumptions as empirical commitments. In T.G.Blomberg & S. Cohen (Eds.), Punishment and social control (pp. 235-254). New York: Walter de Gruyter.

Bornmann, L., and Marx, W. (2011). The h-index as a research performance indicator. European Science Editing, 37(3), 77-80.

Bornmann, L and Marx, W (2013). Evaluating individual researchers' performance. Euorpean Science Editing, 39(2), 30-40.

Cabrerizo, F. J., Alonso, S., Herrera-Viedmac, E., & Herrera, F. (2012). Q2-index: Quantitative and qualitative evaluation based on the number and impact of papers in the hirsch core. Journal of Informetrics, 4(1) 23-28.

Connor, J. (2011). Google Scholar Citations Open to All. Resource document. http://googlescholar.blogspot.dk/2011/11/google-scholar-citations-open-to-all.html . Accessed 30 March 2015

Costas, R., and Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193-203.

Courtault, J. M., and Hayek, N. (2008). On the robustness of the h-index: a mathematical approach. Economics Bulletin, 3(78), 1-9.

De Battisti, F., and Salini, S. (2012). Robust analysis of bibliometric data. Statistical Methods and Applications, 22(2), 269-283.

Egghe, L. (2006). Theory and practise of the g-index . Scientometrics, 69(1), 131-152.

ESO. (2011). Some statistics on recent job hires in Astronomy: Who is getting the jobs and how do you compare? Science News. Resource document.

19

https://www.eso.org/wiki/pub/Science/ScienceNewsLetter/ScienceNews-May-Jun2011.pdfAccessed 30 March 2015

Farhadi, F., Salehi, H., Yunus, M. M., Chadegani, A. A., Farhabi, M., Fooladi, M., et al. (2013). Does it matter which citation tool is used to compare the h-index of a group of highly cited researchers? Australian Journal of Basic and Applied Sciences, 7(4), 198-202.

Franceschet, M., (2010). A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar. Scientometrics, 83(1), 243-258.

Franceschini, F., and Maisano, D. (2011). Criticism on the hg.index. Scientometrics, 86(2), 339-346.

Freedman, D. A., Pisani, R., & Purves, R. (2007). Statistics. (4 ed.) New York, NY.: W.W. Norton & Company.

Gorard, S. (2003). Quantitative methods in social science: the role of numbers made easy. London: Continuum.

Gorard, S. (2006). Towards judgement-based statistical analysis. British Journal of Sociology of Education, 27(1), 67-80.

Gratton, R. (2014). Evaluation of Italian atronomical production: 2010-2012. Resource document.arXiv: arXiv:1402.4080v1 [astro-ph.IM]. Accessed 30 March 2015

Hagen, N.T. (2008). Harmonic allocation of authorship credit: Source-level correction of bibliometric bias assures accurate publication and citation analysis, PLoS ONE, doi:10.1371/journal.pone.0004021

Harzing, A. (2008). Google Scholar - a new data source for citation analysis. Resource document, http://www.harzing.com/pop_gs.htm . Accessed 30 March 2015

Harzing, A. (2012). Publish or Perish user's manual. Resource document, http://www.harzing.com/pop.htm#metrics. Accessed 30 March 2015.

Harzing, A. (2014). A longitudinal study of Google Scholar coverage between 2012 and 2013. Scientometrics, 98(1), 565-575.

Haustein, S., & Larivière, V. (2015). The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects. In I. M. Welpe, J. Wollersheim, S. Ringelhan and M. Osterloh (Eds.), Incentives and Performance:Governance of Research Organizations (pp. 121-139). Switzerland: Springer International Publishing.

Hirsch, J. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America (PNAS), 102(46), 16569-16572.

Jacsó, P. (2005). As we may search? Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science, 89(9), 1537-1547.

Jascó, P. (2008). The plausibility of computing the h-index of scholarly productivity and impact using reference-enhanced databases. Online Information Review, 32(2), 266-283.

20

Kendall, M. G., (1955). Rank Correlation Methods. New York: Hafner Publishing Co.

Kosmoloulos, C., and Pumain, D. (2014). Citation, citation, citation: Bibliometrics, the web and the Social Sciences and Humanities. Resource document, Cybergeo: European Journal of Geography. http://www.cybergeo.eu/index15463.html. Accessed 30 March 2015.

Kamphuis, P., and van der Kruit, P.C. (2010). Citations and impact of Dutch astronomy. Resource document arXiv:1011.5311v1 [astro-ph.IM]. Accessed 30 March 2015.

Kosmulski, M. (2006). A new type Hirsch-index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3), 4-6.

Lancho-Barrentes, B., and Guerrero, V. P. (2010). What lies behind the averages and significance of citation indicators in different disciplines? Journal of Information Science, 36(3), 371-382.

Li, J., Burnham, J.F., Lemley, T., and Britton, R.M. (2010) Citation Analysis: Comparison of Web of Science®, Scopus™, SciFinder®, and Google Scholar. Journal of Electronic Resources in Medical Libraries, 7, (3), 196-217.

Meera, B. M., and Manjunath, M., (2012). h-index of astrophysicists at raman research institute: performance of different calculators, Working Group on Libraries, IAU Division XII Commission 5, In L. Utley (Ed.) Special Session 15: Data-Intensive Astronomy, Beijing, China, 23-24 August 2012. http://iau-commission5.wikispaces.com/LibWG+Beijing+2012. Accessed 30 March 2015.

Meho, L.I., and Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105-2125.

Miller, C. W., (2006). Superiority of the h-index over the impact factor for physics.Resource document, arXiv:physics/0608183 [physics.soc-ph]. Accessed 30 March 2015.

Neuhaus, C., Neuhaus, E., Asher, A., & Wrede, C. (2006). The Depth and Breadth of Google Scholar: An Empirical Study. Portal: Libraries and the Academy, 6(2), 127‐141.

Patel, V. M., Ashrafian, H., Almoudaris, A., Makanjuola, J., Bucciarelli-Ducci, C., Darzi, A. et al. (2013). Measuring academic research performance for healthcare researchers with the H index: Which search tool should be used? Medical Principles and Practice, 22(2), 178-183.

Panaretos, J., & Malesios, C. (2009). Assessing scientific research performance and impact with single indicators. Scientometrics, doi: 10.1007/s11192-008-2174-9.

Redner, S. (2010). On the meaning of the h-index. Journal of Statistical Mechanics: Theory and Experiment, doi:10.1088/1742-5468/2010/03/L03005

Roediger, H. L. (2006). The h index in Science: A New Measure of Scholarly Contribution. Resource document, APS Observer: The Academic Observer. http://www.psychologicalscience.org/index.php/publications/observer/2006/april-06/the-h-index-in-science-a-new-measure-of-scholarly-contribution.html. Accessed 30 March 2015.

21

Schneider, J. W. (2013). Caveats for using statistical significance tests in research assessments. Journal of Informetrics, 7(1), 50-62.

Schneider, J. W. (2015). Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102(1), 411-432.

Schreiber, M. (2013a). A case study of the arbitrariness of the h-index and the highly-cited-publications indicator. Journal of Informetrics, 7(2), 379-387.

Schreiber, M. (2013b). How to derive an advantage from the arbitrariness of the g-index. Journal of Informetrics, 7(2), 555-561.

Schreiber, M., Malesios, C. C., and Psarakis, S. (2012). Exploratory factor analysis for the hirsch-index, 17 h-type variants, and some traditional bibliometric indicators. Journal of Informetrics, 6(3), 347-358.

Sidiropoulos, A., Katsaros, D., and Manolopoulos, Y. (2007). Generalized hirsch h-index for disclosing latent facts in citation networks. Scientometrics, 72(2), 253-280.

Smith, A. G. (2008). Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise. Scientometrics, 74(2), 309-316.

Testa, J. (2012). The Thomson Reuters Journal Selection Process. Resource document, http://wokinfo.com/essays/journal-selection-process/ . Accessed 30 March 2015.

Tol, R. S. J. (2009). Of the H-index and its alternatives: An application to the 100 most prolific economists. Scientometrics, 80(2), 317-324

van Leeuwen, T. (2008). Testing the validity of the hirsch-index for research assessment purposes. Research Evaluation, 17(2), 157-160.

Van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491-502.

Vanclay, J. K. (2007). On the robustness of the h-index. Journal of the American Society for Information Science and Technology, 58(10), 1547-1550.

Waltman, L., and Schreiber, M. (2013). On the calculation of percentile-based bibliometric indicators. Journal of the American Society for Information Science and Technology, 64(2), 372-379.

Waltman, L., and van Eck, N. J. (2012). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63(2), 406-415.

Wildgaard, L., Schneider, J. W., and Larsen, B. (2014). A review of the characteristics of 108 author-level bibliometric indicators. Scientometrics, 101(1), 125-158.

Wilson, V. (2007). A content analysis of Google Scholar: Coverage varies by discipline and by database. Evidence Based Library and Information Practice, 2(1), 134-136.

Yang, Kiduk and Meho, Lokman I (2006). Citation Analysis: A Comparison of Google Scholar, Scopus, and Web of Science., 2006 . In 69th Annual Meeting of the American Society for Information

22

Science and Technology (ASIST), Austin (US), 3-8 November 2006. http://eprints.rclis.org/8605/1/Yang_citation.pdf. Accessed 9 April 2015.

Zhang, C. (2009). The e-index, complementing the h-index for excess citations. PLoS ONE, 4(5), e5429.

23

Table 1: The seventeen indicators of individual scholar impact investigated in this study and the concepts they measure

Nr. Indicator Abbr. Definition Indicator Type Concept

1Number of publications

PTotal number of publications by the researcher

publication-based production

2Number of Citations

Ctotal number of citations received by publications of the researcher

citation-based times cited

3Citations per paper

CPPThe average number of citations per paper, C/P. citation-based

arithmetic mean number of citations

per paper

4Academic age

PageNumber of years since first publication by the researcher recorded in the database.

publication-based academic age

5

Age Weighted Citation Rate (Harzing 2012)

AWCR

(Citations to all papers, divided by age of paper)/number of publications. The sum over all papers in the AWCR. citation-based effect of all papers

6

Age Weighted h, (Harzing 2012)

AW

Square root of AWCR

hybrid effect of all papers

7Authors-per-paper

AppAverage number of authors per paper over all papers

publication-based contribution

8

Per-author AWCR, (Harzing 2012)

AWCRpa

Citations to a given paper divided by age of that paper. The result for each paper is divided by the number of authors per paper. Summed over all papers

citation-based independence

9h-index, (Hirsch 2005)

h

Publications are ranked in descending order after number of citations. Where number of citations and rank is the same, this is the h index

hybrid quality and quantity

10m-quotient, (Hirsch 2005)

m-quoth divided by academic age

hybrid effect of best papers

11g-index (g), (Egghe 2006)

g

Publications are ranked in descending order after number of citations. The cumulative sum of citations is calculated, and where the square root of the cumulative sum is equal to the rank this is g-index

hybrid rank of scholar

12 mg-quotient mg-quot g divided by academic age (Egghe 2006) hybrid effect of best papers

13

Hg, (Alonso, Cabreriazo, and Herrera-Viedma 2010)

hg

Compare researchers with similar h and g indices. Square-root of (h multiplied by g)

hybrid rank of scholar

14e-index, (Zhang 2009)

e

The e-index is the (square root) of the surplus of citations in the h-set beyond h2, i.e., beyond the theoretical minimum required to obtain an h-index of 'h'.

hybrid excellence

15 Normalized h,

(Sidiropoulos, Katsaros, and

h-norm Normalizes h to compare scientists across fields. Normalized h=h/np, if h of its np articles have received at least h citations each, and the rest (np-h) articles receive no more than h citations.

hybrid comparison across fields

24

Manolopoulos 2007)

16h2, (Kosmulski 2006)

h2Weights most productive papers by finding the cube root of all citations (not just citations to h articles).

hybrid excellence

17 ħ, (Miller 2006)

ħ Square root of half the total number of citations to all publications

hybrid citation distribution

Table 2. Description of dataset

Discipline Publications WoS Publications GS Citations WoS Citations GSAstronomy, n=190 12,318 28,044 320,971 513,611

PHD, n13 121 285 1,769 2,487Post Doc, n48 1,249 2,920 26,933 40,764Assis Prof, n26 1,328 2,978 29,084 41,923Assoc Prof, n66 5,130 12,548 130,756 206,576Full Professor, n37 4,490 9,313 132,429 221,861

Environmental Science, n=99 3,228 7,425 34,851 62,351PHD, n2 7 34 76 141Post Doc, n7 58 322 235 1,086Assis Prof, n21 337 777 2,569 4,281Assoc Prof, n44 1,413 3,424 13,996 26,045Full Professor, n25 1,413 2,868 17,975 30,798

Philosophy, n=155 2,223 9,538 7,108 66,386PHD, n5 6 19 9 48Post Doc, n16 91 323 182 865Assis Prof, n24 192 526 353 2,753Assoc Prof, n53 454 1,856 1,121 6,671Full Professor, n57 1,480 6,814 5,443 56,049

Public Health, n=68 4,374 7,220 60,441 104,637PHD, n4 52 76 172 468Post Doc, n3 25 53 164 526Assis Prof, n17 621 1,024 6,224 11,269Assoc Prof, n31 1,743 3,236 22,197 41,129Full Professor, n13 1,933 2,831 31,684 51,245

Total 22,143 52,227 423,371 746,985

25

Table 3. Minimum and minimum and maximum values for each discipline and in WoS and GS.

Discipline Publications WoS Publications GS Citations WoS Citations GSAstronomy 2-327 2-882 3-16481 0-25855Environmental Sci. 1-150 10-360 0-2749 4-4087Philosophy 1-140 1-540 0-1282 0-6807Public Health 3-661 3-943 6-13520 9-21879

26

Table 4. Astronomy: harmonic and arithmetic mean scores for indicators, ratio between scores in italics

27

  

Publication-based Citation-based Hybrid

Page App P CPP C AWCR AWCRpa AW ħ h m_quot h-norm g hg mg e H2

GS harmean 13.8 3.3 43.8 8.4 115.8 36.7 15.5 9.0 14.7 10.9 0.8 0.1 18.1 12.5 1.4 8.9 6.1

GSAmean 18.8 3.5 147.6 15.7 2703.2 349.3 93.9 15.5 30.0 21.8 1.2 0.3 39.5 25.6 2.2 35.2 11.5

WoS harmean 10.8 3.5 22.8 10.7 126.7 26.6 9.1 7.3 12.5 9.4 0.8 0.4 14.9 12 1.3 11.4 7.4

WoSAmean 16.1 4.6 64.8 20.2 1689.3 228.2 60.8 12.4 23.6 18.1 1.2 0.7 30.8 23.6 2.1 2.8 9.8

Ratio harmean 1.2 0.9 1.9 0.7 0.9 1.3 1.7 1.2 1.1 1.1 1.0 1.25 1.2 1.0 1.0 0.7 0.8

Ratioamean 1.2 0.8 2.3 0.7 1.6 1.5 1.5 1.2 1.2 1.2 1.0 0.4 1.2 1.0 1.0 12.5 1.1

Table 5. Environmental Science: harmonic and arithmetic mean scores for indicators, ratio between scores in italics

28

  

Publication-based Citation-based Hybrid

Page App P CPP C AWCR AWCRpa AW ħ h m_quot h-norm g hg mg e H2

GS harmean 16.2 3.1 41.7 3.0 101.7 17.7 5.9 5.2 9.4 6.8 0.4 0.1 11.3 4.1 0.6 4.5 5.9

GSAmean 20.2 3.2 75.0 7.6 629.8 76.1 24.0 7.5 14.9 11.9 1.2 0.2 18.4 12.0 0.9 13.2 7.3

WoS harmean 10.6 2.6 11.6 3.1 7.8 4.4 2.6 2.6 3.7 3.1 0.3 0.3 3.9 3.5 0.5 3.3 3.0

WoSAmean 16.2 3.1 32.6 7.8 352.0 42.4 16.6 5.4 10.6 8.5 0.5 0.5 13.1 10.5 0.8 9.1 5.7

Ratio harmean 1.5 1.1 3.5 0.9 13.0 4.0 2.2 2.0 2.5 2.1 1.3 0.3 2.8 1.1 1.2 1.3 1.9

RatioAmeans 1.2 1.0 2.3 0.9 1.8 1.8 1.4 1.3 1.4 1.4 2.4 2.8 1.4 1.1 1.1 1.4 1.3

Table 6. Philosophy: harmonic and arithmetic mean scores for indicators, ratio between scores in italics

29

  

Publication-based Citation-based Hybrid

Page App P CPP C AWCR AWCRpa AW ħ h m_quot h-norm g hg mg e H2

GS harmean 12.5 4.0 16.0 0.7 2.3 1.4 1.3 1.2 1.6 1.5 0.2 0.1 1.7 0.2 0.3 1.0 1.5

GSAmean 18.8 44.1 61.5 4.7 428.3 41.0 30.3 4.6 10.2 6.9 0.3 0.2 12.7 4.3 0.6 0.2 5.3

WoS harmean 7.3 1.1 4.0 0.3 0.4 0.2 0.2 0.3 0.4 0.4 0.1 0.1 0.2 0.2 0.1 0.2 0.4

WoSAmean 13.3 1.2 14.3 1.8 45.8 5.0 4.0 1.4 2.9 2.3 0.9 0.3 3.4 2.7 0.2 2.6 2.2

Ratio harmean 1.7 3.6 4.0 2.3 5.7 7.0 6.5 4.0 4.0 3.7 2.0 1.0 8.5 1.0 3.0 5.0 3.7

RatioAmeans 1.4 36.7 4.3 2.6 9.3 8.2 7.5 3.2 3.5 3.0 0.3 0.6 3.7 1.5 3.0 0.1 2.4

Table 7. Public Health: harmonic and arithmetic mean scores for indicators, ratio between scores in italics

30

  

Publication-based Citation-based Hybrid

Page App P CPP C AWCR AWCRpa AW ħ h m_quot h-norm g hg mg e H2

GS harmean 13.1 3.2 34.3 4.9 128.8 23.8 6.4 8.1 11.1 7.5 0.5 0.1 12.9 8.3 0.9 9.8 6.8

GSAmean 17.7 3.4 106.1 11.0 1538.7 204.5 59.2 11.2 21.5 15.2 0.9 0.2 27.6 17.2 1.5 20.4 9.1

WoS harmean 10.1 3.4 19.2 4.5 59.3 11.7 4.7 4.4 7.5 5.9 0.5 0.2 8.5 7.2 0.8 3.5 5.2

WoSAmean 15.0 4.1 64.3 9.6 888.8 116.2 32.8 8.1 15.5 11.9 0.7 0.4 19.7 15.3 1.3 14.1 7.3

Ratio harmean 1.2 0.9 1.7 1.0 2.1 2.0 1.3 1.8 1.4 1.2 1.0 0.5 1.5 1.1 1.1 2.8 1.3

RatioAmeans 1.2 0.8 1.6 1.1 1.7 1.7 1.8 1.3 1.4 1.2 1.2 0.5 1.4 1.1 1.1 1.4 1.2

Table 8. Kendall correlation of indicators in GS and WoS. Kendall’s tau (τ) takes values between -1 and +1, where perfect agreement τ = 1, no association τ = 0, with a positive correlation indicating that the ranks of both variables increase together whilst a negative correlation indicates that as the rank of one variable increases the other one decreases.

Type

Indicator Astronomy n190 Environmental Science n99 Philosophy n155 Public Health n68

Publ

icat

ion Page 0.81 0.67 0.57 0.78

App 0.63 0.64 0.24 0.62

P 0.69 0.57 0.48 0.72

Cita

tion

CPP 0.63 0.67 0.45 0.72

C 0.80 0.76 0.55 0.80

AWCR 0.79 0.74 0.53 0.80

AWCRpa 0.71 0.69 0.51 0.75

Hyb

rid

AW 0.79 0.74 0.53 0.80

ħ 0.80 0.76 0.55 0.80

h 0.79 0.79 0.60 0.82

m_quot 0.65 0.48 0.37 0.67

h-norm 0.44 0.37 0.24 0.49

g 0.80 0.79 0.56 0.83

hg 0.89 0.90 0.88 0.91

mg 0.71 0.62 0.42 0.67

e 0.68 0.75 0.50 0.80

h2 0.80 0.76 0.55 0.80

31

Table 9. Agreement between indicator rankings at the seniority-level Astronomy Publication-based Citation-based Hybrid

Page App P CPP C AWCR AWCRpa AW ħ h m-quot h-norm g hg mg e h2Phd, n13same rank GS, WoS

0.302

0.542

0.521

0.673

0.673

0.776

0.541

0.766

0.673

0.869

-0.331

0.351

0.553

0.9510

0.394

0.796

0.673

Post Doc, n48same rank GS, WoS

0.766

0.706

0.666

0.733

0.8710

0.847

0.711

0.846

0.877

0.877

0.810

0.333

0.8810

0.9216

0.834

0.713

0.879

Assis. Prof., n26same rank GS, WoS

0.201

0.152

0.2951

0.271

0.333

0.281

0.301

0.281

0.334

0.296

0.224

-.0111

0.331

0.674

0.151

0.303

0.334

Assoc. Prof., n66same rank GS, WoS

0.751

0.712

0.651

0.666

0.845

0.888

0.724

0.888

0.845

0.846

0.839

0.433

0.856

0.9213

0.846

0.8311

0.845

Professor, n37same rank GS, WoS

0.7411

0.720

0.726

0.592

0.8410

0.878

0.767

0.878

0.849

0.753

0.786

0.553

0.829

0.895

0.789

0.803

0.8410

Environmental Sci.Phd, n2 - - - - - - - - - - - - - - - - -Post Doc, n7same rank GS, WoS

-0.111

0.240

-0.260

0.142

0.000

-0.481

0.241

-0.480

0.000

-0.111

0.390

0.813

0.161

0.393

0.522

0.051

0.000

Assis. Prof., n21same rank GS, WoS

0.704

0.513

0.402

0.741

0.753

0.753

0.625

0.754

0.754

0.844

0.432

0.394

0.792

0.908

0.585

0.795

0.753

Assoc. Prof., n44same rank GS, WoS

0.535

0.767

0.462

0.692

0.734

0.763

0.686

0.763

0.734

0.764

0.614

0.333

0.784

0.909

0.714

0.771

0.733

Professor, n25same rank GS, WoS

0.542

0.703

0.701

0.531

0.777

0.757

0.696

0.7511

0.777

0.755

0.611

0.161

0.788

0.857

0.581

0.713

0.776

PhilosophyPhd, n5 - - - - - - - - - - - - - - - - -Post Doc, n16same rank GS, WoS

0.331

0.523

0.381

0.420

0.424

0.411

0.412

0.412

0.423

0.626

0.580

0.304

0.312

0.9314

0.354

0.261

0.423

Assis. Prof., n24same rank GS, WoS

0.341

0.221

0.101

0.453

0.404

0.375

0.392

0.354

0.405

0.423

0.252

0.421

0.399

0.8813

0.264

0.373

0.405

Assoc. Prof., n53same rank GS, WoS

0.372

-0.070

0.291

0.442

0.493

0.472

0.472

0.474

0.493

0.524

0.314

0.173

0.514

0.9022

0.325

0.434

0.492

Professor, n57same rank GS, WoS

0.431

0.250

0.562

0.480

0.662

0.683

0.632

0.684

0.665

0.710

0.430

0.200

0.664

0.8920

0.562

0.573

0.670

Public HealthPhd, n4 - - - - - - - - - - - - - - - - -Post Doc, n3 - - - - - - - - - - - - - - - - -Assis. Prof., n17same rank GS, WoS

0.696

0.551

0.502

0.721

0.734

0.704

0.662

0.714

0.734

0.775

0.603

0.352

0.836

0.889

0.564

0.835

0.733

Assoc. Prof., n31same rank GS, WoS

0.783

0.776

0.573

0.748

0.8213

0.8011

0.703

0.8111

0.8212

0.8710

0.734

0.551

0.846

0.9517

0.715

0.789

0.8211

Professor, n13same rank GS, WoS

0.663

0.610

0.773

0.775

0.678

0.797

0.695

0.797

0.678

0.848

0.583

0.592

0.726

0.858

0.675

0.723

0.678

32

Table 10: Standard deviation of the difference in scholar rank position in WoS and GS.

Astronomy Publication-based Citation-based HybridPag

eApp P CPP C AWCR AWCRpa AW ħ h m-quot h-norm g hg mg e h2

Phd, 13 5.7 5.0 5.0 5.9 5.0 2.7 4.9 2.7 5.0 1.2 5.2 5.6 5.4 1.8 4.8 3.5 5.1

Post Doc, n48 20.7 19.6 21.8 18.8 20.4 18.7 18.5 18.7 21.

5 18.5 21.9 20.0 19.9 19.3 20.

3 20.1 21.5

Assis. Prof., n26 10.7 11.3 10.4 12.1 11.2 11.5 11.9 11.5 11.

2 9.8 9.0 12.3 11.9 10.3 9.7 10.0 11.2

Assoc. Prof., n66 27.5 27.6 28.8 30.3 28.5 26.2 25.3 28.7 28.

5 27.9 24.4 29.8 24.3 25.5 28.

4 30.7 28.5

Professor, n37 13.2 15.7 13.4 15.5 14.6 13.2 13.2 16.2 14.

6 16.8 16.2 16.3 11.9 14.6 16.

0 15.0 14.6

Environmental Sci.Phd, n2 - - - - - - - - - - - - - - - - -Post Doc, n7 6.8 5.5 6.4 6.3 6.2 6.7 6.9 6.3 6.2 4.4 6.6 3.1 6.1 3.3 5.5 5.8 6.2

Assis. Prof., n21 21.7 11.7 16.4 22.5 20.6 18.5 15.9 17.9 20.

6 19.1 17.4 13.7 20.2 17.7 17.

1 16.5 20.6

Assoc. Prof., n44 30.2 32.1 34.2 34.0 37.0 31.8 30.4 32.4 36.

6 32.2 37.0 35.4 32.7 30.1 31.

8 40.4 35.8

Professor, n25 20.0 25.8 26.7 24.3 20.1 19.5 22.0 16.6 20.

1 21.9 25.3 19.4 18.3 18.2 21.

7 26.7 20.1

PhilosophyPhd, n5 3.6 2.7 1.8 1.4 1.4 1.2 1.4 1.4 1.4 1.4 4.3 4.4 1.4 1.4 4.9 4.4 1.4

Post Doc, n16 6.3 5.5 10.4 8.3 8.1 9.3 10.2 8.9 8.7 7.6 10.5 10.1 10.0 3.6 9.2 7.9 8.7

Assis. Prof., n24 20.7 23.0 22.9 19.2 18.9 19.5 21.0 16.5 21.

1 21.1 21.8 26.5 20.5 16.9 19.

1 22.8 19.9

Assoc. Prof., n53 29.5 30.1 29.3 31.2 33.1 29.9 27.5 29.3 31.

6 30.6 30.4 33.5 30.6 25.9 28.

4 27.8 30.3

Professor, n57 38.9 41.1 34.5 33.3 39.0 33.1 37.1 31.7 39.

7 38.0 36.5 37.8 36.9 34.3 38.

2 35.9 37.8

Public HealthPhd, n4 - - - - - - - - - - - - - - - - -Post Doc, n3 - - - - - - - - - - - - - - - - -Assis. Prof., n17 5.7 11.3 11.0 11.4 7.7 9.8 13.2 9.8 7.7 8.2 7.6 9.9 6.8 5.3 8.4 6.5 7.7

Assoc. Prof., n31 20.8 18.2 21.5 20.2 18.0 17.0 23.8 17.0 18.

5 14.5 26.0 23.6 20.7 11.0 25.

5 17.6 18.5

Professor, n13 8.1 12.3 13.3 9.5 8.0 12.3 12.0 12.3 8.0 11.6 9.3 10.5 11.3 12.7 10.

9 13.8 8.0

33

Figure 1. Box plots for number of publications and number of citations in WoS (blue) and GS (green). The rectangle represents 50% of the cases, with the whiskers going out to the smallest and largest values. The line inside the rectangle is the median value. Notice, frequencies of publications and citations are shown on log-scales.

34

Figure 2. Relationship between predicted harmonic and arithmetic h-index scores and previously published empirical data from Astronomy ESO (2011), Meera & Manjunath (2012), Kamphuis & van der Kruit (2010), Gratton (2014) and Redner (2010)

Nova

Redner

Gratton

Amean GS

Amean W

OSIAU

ESO

Harmean

GSAAS

Meera

Harmean

WOS

0

5

10

15

20

25

30

35

Studies investigating scholar performance

Typi

cal h

-inde

x va

lue

35

Figure 3. Lack of fit (summed) between predicted harmonic and arithmetic h-index scores and previously published empirical data from Astronomy ESO (2011), Meera & Manjunath (2012), Kamphuis & van der Kruit (2010), Gratton (2014) and Redner (2010).

Amean WOS Harmean WoS Amean GS Harmean GS0

0.5

1

1.5

2

2.5

Counting Model

Stan

dard

ized

dep

artu

re fr

om m

odel

pre

dicti

on

36

Figure 4. Correlation models for Publications (P) in Astronomy, Environmental Science, Philosophy and Public Health. The number of publications identified in GS is on the x axis and the number of publications identified in WoS on the y axis.

0 200 400 600 800 1000

0

50

100

150

200

250

300

350

P_GS

P

0 50 100 150 200 250 300 350 400

0

20

40

60

80

100

120

140

160

P_GS

P

Astronomy Environmental Sciences

0 100 200 300 400 500 600

0

20

40

60

80

100

120

140

P_GS

P

0 200 400 600 800 1000

0

100

200

300

400

500

600

700

P_GS

P

Philosophy Public Health

37

Figure 5. Correlation models for Citations (C) in Astronomy, Environmental Science, Philosophy and Public Health. The number of publications identified in GS is on the x axis and the number of publications identified in WoS on the y axis.

0 5000 10000 15000 20000 25000 30000

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

C_GS

C

0 1000 2000 3000 4000 5000

0

500

1000

1500

2000

2500

3000

C_GS

C

Astronomy Environmental Sciences

0 1000 2000 3000 4000 5000 6000 7000

0

200

400

600

800

1000

1200

1400

C_GS

C

0 200 400 600 800 1000

0

100

200

300

400

500

600

700

P_GS

P

Philosophy Public Health

38

Figure 6. Correlation models for hg in Astronomy, Environmental Science, Philosophy and Public Health. The number of publications identified in GS is on the x axis and the number of publications identified in WoS on the y axis.

0 20 40 60 80 100

0

10

20

30

40

50

60

70

80

90

hg_gs

hg

0 10 20 30 40 50

0

5

10

15

20

25

30

35

40

hg_gs

hg

Astronomy Environmental Sciences

0 5 10 15 20 25 30 35

0

5

10

15

20

25

30

hg_gs

hg

0 20 40 60 80 100

0

10

20

30

40

50

60

70

80

hg_gs

hg

Philosophy Public Health

39

Figure 7. Rank position of Associate Professors in Public Health by hybrid indicators in Wos and GS. The first column of the figure is the rank position. Each cell displays the identification number of the scholar and the name of the indicator is at the top of each column.

Rank AW AW_gs ħ ħ_gs h h_gs mquot mquot_gs hnorm hnorm_gs g g_gs hg hg_gs mg mg_gs e e_gs h2 h2_gs1 713 713 713 713 713 713 713 713 757 736 759 713 713 713 713 713 759 759 713 7132 759 759 740 740 740 740 719 717 725 726 740 759 740 740 719 759 740 713 740 7403 740 740 759 759 759 759 717 759 721 721 713 740 759 759 759 717 713 740 759 7594 752 752 752 752 752 752 759 730 754 730 752 752 752 752 717 730 752 752 752 7525 748 748 737 737 748 737 760 719 726 737 737 737 737 737 730 719 737 737 737 7376 717 717 748 748 717 717 720 760 730 724 717 743 717 717 712 743 746 743 748 7487 737 743 717 743 737 748 752 743 736 757 760 730 748 748 752 752 730 730 717 7438 760 737 760 717 760 760 712 752 724 712 746 717 760 760 737 760 714 714 760 7179 719 714 743 730 743 743 743 740 745 754 714 714 743 743 760 720 717 746 743 73010 743 711 714 714 736 736 730 718 737 717 743 760 736 736 740 740 726 717 714 71411 714 760 746 760 714 730 740 720 717 719 748 748 714 730 720 737 760 760 746 76012 720 730 730 711 726 714 726 712 743 714 736 746 726 714 726 711 743 711 730 71113 736 719 736 746 720 726 711 737 711 762 726 711 730 726 743 712 736 748 736 74614 726 720 726 736 730 711 737 711 714 759 730 736 720 711 711 745 711 736 726 73615 711 745 720 720 711 712 745 745 762 743 712 726 711 712 762 718 725 720 720 72016 730 736 719 712 712 720 748 748 716 760 719 712 746 720 745 757 719 726 719 71217 712 712 712 726 719 719 736 736 760 746 720 720 712 746 721 762 712 719 712 72618 746 746 711 745 724 745 718 762 712 725 711 719 719 719 725 748 748 757 711 74519 718 726 724 719 746 718 721 757 759 720 724 745 724 724 718 736 724 712 724 71920 725 718 725 757 745 757 754 726 720 752 725 757 725 745 746 746 720 745 725 75721 745 757 718 718 718 724 762 751 756 751 745 725 745 718 748 714 762 725 718 71822 721 725 745 749 725 746 725 749 719 711 718 724 718 725 736 754 721 754 745 74923 762 749 762 725 757 749 716 754 746 740 762 718 757 757 754 725 757 749 762 72524 724 754 757 724 754 725 714 714 752 713 757 754 762 754 714 726 745 724 757 72425 716 716 721 754 716 754 757 721 740 748 721 749 721 716 757 749 718 718 721 75426 754 762 716 716 721 716 756 716 751 745 754 716 754 762 716 751 754 721 716 71627 757 724 754 747 762 747 746 725 747 718 716 721 716 721 724 721 716 716 754 74728 747 747 747 721 747 721 724 747 718 716 747 762 747 747 751 716 747 751 747 72129 751 721 751 751 751 751 751 746 713 747 751 747 751 751 747 747 751 762 751 75130 749 751 749 762 749 762 747 756 748 749 749 751 749 749 749 724 749 747 749 76231 756 756 756 756 756 756 749 724 749 756 756 756 756 756 756 756 756 756 756 756

40

Figure 8. Rank position of Associate Professors in Philosophy by hybrid indicators in WoS and GS.

rank AW AW_gs ħ ħ_gs h h_gs mquot mquot_gs hnorm hnorm_gs g g_gs hg hg_gs mg mg_gs e e_gs h2 h2_gs1 531 544 531 544 531 544 507 531 553 499 531 544 531 531 507 531 531 544 531 5442 562 507 553 507 553 507 531 547 499 546 553 558 553 544 531 507 553 558 553 5073 553 531 516 558 516 531 547 507 517 527 526 507 516 516 547 558 526 507 516 5584 543 558 526 531 544 516 553 544 532 553 516 531 544 553 553 547 543 531 526 5315 544 547 537 532 547 504 497 499 510 551 547 504 547 526 526 544 516 519 537 5326 547 490 544 504 537 547 524 497 526 497 544 516 526 547 543 553 547 489 544 5047 537 519 547 516 548 532 492 526 516 489 537 489 537 537 497 497 544 553 547 5168 526 532 543 519 490 489 518 553 497 530 548 519 548 543 524 526 497 504 543 5199 490 504 548 489 526 490 526 541 495 537 543 532 497 489 518 519 537 532 548 48910 497 489 490 490 518 526 519 510 508 541 497 553 543 490 550 489 514 516 490 49011 518 526 497 547 497 519 541 519 547 531 489 526 518 497 537 510 489 526 497 54712 516 497 518 526 532 537 515 527 541 518 518 547 490 518 516 524 548 499 518 52613 519 553 498 553 504 518 537 530 504 542 514 518 519 532 491 518 530 497 498 55314 548 537 519 499 519 499 543 489 557 511 498 537 532 548 544 491 498 547 519 49915 524 516 514 518 543 553 510 504 531 504 490 497 524 519 514 499 491 514 514 51816 498 499 524 537 510 497 490 524 518 522 530 514 541 514 498 530 550 518 524 53717 492 543 489 543 524 543 511 491 550 550 541 490 508 498 548 493 519 493 489 54318 541 518 532 498 541 510 548 518 546 536 550 550 489 504 519 504 524 537 532 49819 558 498 530 514 508 541 516 516 562 510 491 510 514 541 541 550 518 549 530 51420 514 514 541 497 557 550 532 490 519 548 508 493 498 550 489 541 532 510 541 49721 557 530 508 510 492 530 544 542 537 491 524 543 504 530 490 527 490 524 508 51022 532 510 558 550 558 524 517 550 551 508 519 498 550 524 530 542 559 550 558 55023 510 524 504 493 489 508 495 517 544 526 532 530 510 508 511 516 508 498 504 49324 491 550 491 530 499 558 557 509 492 516 499 524 530 491 532 549 558 490 491 53025 489 493 550 524 498 498 550 548 529 496 546 549 491 499 495 509 499 543 550 52426 530 491 557 549 514 514 559 493 524 539 511 541 557 510 492 514 511 530 557 54927 504 541 492 517 550 493 491 537 530 524 504 491 492 558 559 532 493 491 492 51728 550 509 510 541 493 517 504 511 548 507 510 499 558 493 510 490 541 509 510 54129 559 527 559 491 530 491 562 532 514 493 493 509 493 511 558 496 546 517 559 49130 511 503 499 509 517 511 558 546 543 509 557 511 499 557 546 537 495 503 499 50931 508 549 511 508 491 562 546 503 489 547 495 527 511 495 499 517 507 496 511 50832 499 517 493 511 511 509 530 558 490 519 492 517 495 546 493 498 510 511 493 51133 493 557 495 503 562 503 499 562 491 544 559 503 546 559 557 506 504 541 495 50334 495 496 546 562 495 527 498 514 498 517 558 508 559 492 508 511 527 506 546 56235 546 562 517 527 551 548 514 522 559 503 527 522 507 507 504 503 542 542 517 52736 515 511 562 495 559 557 493 496 493 562 551 496 549 549 549 548 549 527 551 49537 529 495 551 522 546 495 529 498 558 514 542 562 509 509 509 522 509 522 515 52238 517 542 515 557 515 551 508 557 507 557 522 506 527 527 527 546 496 551 562 55739 551 522 529 496 529 522 489 506 527 498 536 548 517 517 517 515 517 515 529 49640 507 506 507 506 507 559 551 495 542 529 496 557 503 503 503 562 506 539 507 50641 503 508 509 548 549 546 509 492 509 532 539 495 522 522 522 557 503 548 549 54842 509 548 527 551 509 549 503 559 511 495 507 551 496 496 496 543 522 559 509 55143 527 546 503 542 503 496 527 529 503 492 509 542 562 562 562 495 515 495 503 54244 549 536 549 536 527 506 522 549 522 559 517 546 506 506 506 492 562 557 496 53645 542 551 496 539 522 542 549 515 496 490 503 559 551 551 551 559 557 536 506 53946 506 559 542 559 496 536 496 543 506 502 562 536 542 542 542 529 492 508 542 55947 496 539 522 502 506 539 506 536 549 558 529 539 536 536 536 536 529 546 527 50248 522 515 506 546 542 502 542 539 515 549 502 502 539 539 539 539 536 562 522 54649 536 492 536 492 536 492 536 508 536 515 549 515 502 502 502 508 539 492 539 49250 539 529 539 515 539 515 539 551 539 506 515 492 515 515 515 551 551 529 536 51551 502 502 502 529 502 529 502 502 502 543 506 529 529 529 529 502 502 502 502 52952 512 512 512 512 512 512 512 512 512 512 512 512 512 512 512 512 512 512 512 51253 545 545 545 545 545 545 545 545 545 545 545 545 545 545 545 545 545 545 545 545

41