multiple bibliometric indicators approach to croatian open access (oa) journals
DESCRIPTION
Talk given at LIBRARIES IN THE DIGITAL AGE (LIDA) 2014: ASSESSING LIBRARIES AND LIBRARY USERS AND USE (Part II: Altmetrics - new methods in assessing scholarly communication and libraries: issues applications, results) at Zadar, Croatia, 16 - 20 June 2014, University of Zadar, Zadar, Croatia (http://www.unizd.hr/) Web site: http://ozk.unizd.hr/lida/TRANSCRIPT
Multiple Bibliometric
Indicators Approach to
Croatian Open Access
(OA) Journals
Franjo Pehar, Jadranka Stojanovski
Department of Information Sciences, University of Zadar
Libraries in the Digital Age 2014
“Better to understand a little than to
misunderstand a lot.”
Anonymous writer
Rise of metrics
● importance and usage of citation metrics has
increased substantially in recent years
● the most popular metric indicator is still JIF
● ...initially intended as a way to help libraries
to select the journals to be acquired…
● ...and to help academics to ‘filter content’
● used to measure academic achievement o at individual, project (grant) and institutional level
...
● academic career
● research project assessment
● institution ranking
● funding decisions
● …
● new situation has drawn significant criticism o practices that encourage conformity in research
o “salami-slicing” or incremental publishing
o unequal treatment of researchers from low-cited
disciplines
Eugene Garfield at LIDA 2004
“JIF is not
created for all the
purposes that
some people
want to use them
for. A lot of
people who are
using JIF are
doing it for
advertising
purposes.”
http://www.garfield.library.upenn.edu/
papers/medicalwritingv8(1)1999.html
“It is crucial to
know how
citations are
connected, how
they are related.”
“The real problem is when you
start using JIF as the
substitute for citation
analysis of individuals. I have
always said that you should
not use JIF to evaluate a
person or department. If I
manage to get my paper
published in Nature, does that
guarantee that it is a great
paper? Even if the paper is
never cited?”
http://researchimpact.files.wordpress.com/2011/10/research-impact-cartoon.jpg
What is a citation?
● simple line in the reference list (not) pointing
to the paper
● paper which act as a foundation for our own
work
● paper which help us to broaden our
knowledge
● tool to confirm that we prepared well prior to
our own research
● tool to flatter someone
● tool to “self-cite” at different levels ...
Citation as a metric tool
● many factors can influence the rate of citation o age of article, type of research conducted, the subject
area
o objectivity of citations? real benefit? (erroneous
references with hundreds, thousand citations)
● tools for evaluation and assessment are
evolving constantly, giving rise to new metrics o different aspects of citation and publication behaviour
Journals
● fee and free journals
● „OA journal” as a synonym for APC model
● journals? - or more portals
● very prominent journals = high JIF journals =
very for-profit journals
● „international journals” and „local journals”
● „regional journals”
Journals
● journal as an old concept from the printed
world
● journal as (still) main channel of scholarly
communication
● journals slowing down a possible
development of science (?), not
implementing ICT advantages and
possibilities…
● ...or implementing it at very slow pace
What’s about journals from the
scientific semi-periphery?
Narrower context
● Croatia o 4.5M population, 11k scholars, 7 universities, 25
research institutes, 25k papers per year and 1300
islands
● Croatian journals o challenges for small language and semi-peripheral
science context
Croatian journals in numbers
Academic career policies in Croatia
“papers must be published in journals with JIF above average for the specific
subject category”
“at least two-thirds of the papers must be published in the journals listed in SCIE”
“paper is rated regarding ratio of the JIF and median of the subject category, and
the minimal value must be 0.5
“the value of the qualitative criteria Q is calculated as a sum of the ratios of JIF
and median JIF of the subject category, multiplied by the factor Fd of the
candidate”
“Example: Journal of Engineering Mechanics in 2011 has JIF 0.99, subject
category is Engineering – Mechanical with median JIF 0.743 …”
“in the A1 category are the papers published in journals with JIF in Q1, Q2, i Q3
of the subject category ...in the A2 category are papers from the journals in
Q4…”
“in the A1 category are the papers published in journals indexed by Web of
Knowledge...if the JIF falls in the first 50% of the subject category...then the
paper is counted as two papers”
Hmmmm….
are Croatian journals important?
is it reasonable just to transfer the Croatian
journal funds to APC or fee based prominent
and popular “international” journals?
what justification do we have to publish
Croatian scholarly journals?
HRCAK - improved visibility,
readability...popularity...citations
HRCAK by disciplines
Objective of the study
● Which aspects of journal usage metrics differ
from citations metrics
● In particular, following issues were be
addressed: o Croatian OA journals usage
o Relationship of full-text paper downloads and
citation counts
o Comparison between full paper downloads and
citations counts relative to different subject areas
Materials and methods
● HRCAK usage statistics o Cover journal's web page visits
o Issue web page visits
o Article (bibliographic records) web page visits
o Full-text (PDF) downloads
o Unique visits from the same IP address during the
period of 30 minutes
...
● For the purpose of the study o Usage data for 115,458 full-text papers
(documents) acquired from 340 unique OA
journal titles March 2007 to the May 1st 2014
Article page visits
Full-text downloads
674 downloads
1293 visits
Research sample
● 47 OA journals (28,121 papers) available in
HRCAK repository and indexed by
o Web of Science™ Core Collection (Thomson
Reuters) 28,725 indexed papers
o Scopus (Elsevier) 28,953 indexed papers
Preprocessing challenges
● Assembling a single data source based on three
data sets
o lack of (P)ID’s for matching documents from
different data sets
● Titles were reduced to the same form
● Levenshtein distance algorithm to determine
similarity between paper titles (2 iterations) o 15,023 (53%) papers in Web of Science and 16,592
(57%) papers in Scopus matched with HRCAK titles
o 13,127 (ca. 46%) unique papers merged from all
three data sets
● 2 journals dropped-out
...
● Initial cleaning of usage data o validity, reliability, transparency ...
● Missing data o HRCAK page visit & download dates/years, more
reader demo-geographic information
● Matching/merging procedure o two journal titles completely lost
o further improvements possible > expected matching
rate ca. 85% (cat(ti ... jn … is … vl … pg)
Results
● download and citation frequencies (March
2007 - May 2014)
Matched
articles
HRCAK
visits
HRCAK
Downloads
Scopus
citations
WOS
citations
13,127 2,084,174 4,174,888 38,106 30,324
Visits vs Downloads
Article page visits: 1-250=84%;
Downloads: 1-250=59%, 251-500=22%
cor(log(1+HRVisits, HRDown),
method=”spearman”)
Possible reasons …
...
...
Scopus vs Web of Science citations
Scopus: 0=41%, 1-5=44%;
Web of Science: 0=47%, 1-5=42%
cor(log(1+HRDown, ScopusCit,
WoSCit), method=”spearman”)
Visits, downloads and citations
relative to different subject fields
Subject area No. journals % No. articles %
Biomedicine and
health
10 22.2% 5856 44.6%
Biotechnical
sciences
5 11.1% 1420 10.8%
Humanities 7 15.6% 527 4.0%
Sciences 8 17.8% 2037 15.5%
Social sciences 6 13.3% 1365 10.4%
Technical sciences 9 20.0% 1922 14.6%
Total 45* 100.0% 13127 100.0%
Median number of visits and
downloads & Mean number of citations
Median number of HRCAK visits and
downloads relative to six subject areas
Mean number of Scopus and Web of
Science citations relative to six
subject areas
Key results
● Different usage (reader) and impact (author)
characteristics
● Very strong, strong and low correlations o Scopus and WoS citation counts
o article page visits and downloads
o downloads and citation counts
● But, with significant differences between and
within six subject areas
Conclusion
● Important to have different (MULTIPLE) but
complementary measures for better
reflecting the multifaceted and
multidimensional character of scientific work
(output, impact, activities …)
Acknowledgment
• HRCAK team (University Computing Centre)
• Tomislav Jagušt (Faculty of Electrical
Engineering and Computing, University of
Zagreb)