electronic resources evaluationofacademicevaluation of ... · u.s. arl collection budget 1992-2011...

Electronic Resources Evaluation of AcademicEvaluation of Academic

Libraries from the Perspective of Information Useof Information Use

學術圖書館電子資源評鑑

Mei-Ling Wang王梅玲Mei Ling Wang王梅玲Graduate Institute of Library, Information and Archival Studies

National ChengChi UniversityNovember 8, 2012

2012/11/15e-resources evaluation /ML Wang

Outline大綱Outline大綱

1. Growth of e-resources is transforming the1. Growth of e resources is transforming the library paradigm

2 Problems with e-resource acquisition and2.Problems with e-resource acquisition and management

3 I t f l ti3. Importance of e-resource evaluation4.Usage study and usage statistics are important5.Evaluation methods and models6.Performance evaluation of e-resources6.Performance evaluation of e resources 7.Conclusion:how to cultivate evaluation

lt f lib iculture of libraries2012/11/15 e-resources evaluation /ML Wang

1

Growth of e-resources

1970s Dialog, BRS1980 CD ROM d t b1980s CD-ROM database1990s Internet, WWW, Web based databasesWeb-based databases 1996- E-journals2000s Aggregated2000s Aggregated databases, Digital library, open access movementopen access movement2008 Amazon Kindle, E-book, ,2010s E-book databases

2012/11/15 e-resources evaluation /ML Wang2

Various e-resources collectionVarious e resources collection


Many e-journal databasesy j


Many E-book databasesy


Need to manage e-resources in digital era數位時代需要電子資源管理新觀念數位時代需要電子資源管理新觀念

A new model of digital collection development and managementmanagement.

E-resources selectionE reso rces acq isitionsE-resources acquisitionsE-resources organizationE iE-resources services E-resources management

U i kiUser is king.Usage statistics and the evaluation of e-resources and print holdings are making operation more difficultprint holdings are making operation more difficult.Use and evaluation of e-resources important


U.S. ARL collection budget 1992-2011

1992-93 2000-01 2010-11

a. Computer File Expenditures

Total 14,147,625 15,297,096 91,620,616

Average 172,532 159,345 825,411

b. Electronic Serial Expendituresb. Electronic Serial Expenditures

Total N/A 117,415,618 764,530,601

Average N/A 1,118,244 6,765,758

c Total e resources (Total a+b)c. Total e-resources (Total a+b)

Total 14,147,625 132,712,714 856,151,217

Average 172,532 1,252,007 7,576,559

Total Library Materials Expenditures

Total 393,271,073 828,778,808 1,370,337,208

Average 4,795,989 7,818,668 12,126,878

E- Resources Expenditures Percent of Total Materials

Average 3.6 16.25 65.03


University library collection budget in T i 我國大學圖書館館藏經費Taiwan我國大學圖書館館藏經費

In 2008,Taiwanes i iti &

2011, National Ch Chi U i ituniversities &

libraries, spent an average of NT

ChengChi University

g$33,754,843 per school on purchasing books and journalsbooks and journals, an average of NT$11,357,444 per , , pschool on electronic databases,accounting for about one-third of the books and journalsbooks and journals budget.


E-resource acquisition considerations complicated電子資源考量複雜complicated電子資源考量複雜

E-resources differ from print resources, computer ft d h d t k i t tsoftware and hardware , network equipment ,system

and user should be considered. I l di th f ll i tIncluding the following components:

(1)Technical infrastructure(2)E-resources content(3)Information services(4)Technical support(5)E-resources management( ) g

Some problems of ER collection development and managementg


Some questions on selecting e-resources電子資源評選問題電子資源評選問題

(1)Are the e-resources authoritative? content correct?(2)Do e-resources meet user needs?( )(3)Do e-resources fill lack of print collections?(4)Do e resources duplicate paper resources?(4)Do e-resources duplicate paper resources? (5) Can e-resources replace print collections? (6) Are e-resources licensed ? Are they be purchased by a consortia? Can they be obtained by interlibrary loan? Or can users use e-resources only in certain formats?


Problems of e-resources acquisition電子資源採購問題電子資源採購問題

Methods of e-resource acquisition are diverse, including purchase rent and subscription;including purchase, rent and subscription;Library purchasing and consortium

h ipurchasing.E-resources purchasing and print purchasing are considered at the same time. When purchasing e-journals and e-magazines, p g j g ,consider whether to:(1)buy print resources plus free e-resources;(2)buy print resources and pay ;( ) y p p ymore money to have electronic resources;(3)buy e-resources only;(4)big deal?y;( ) g


New problems e-resources bring to library t電子資源帶來管理新問題management電子資源帶來管理新問題

Management mechanism is needed to provide e-resources usage statistics and a cost analysis and management system to effectivelystatistics, and a cost analysis and management system to effectively track, manage and evaluate the benefits of e-resources purchasing.

(1)H ll i i l di l i(1)How can e-resources collection management including evaluation, selection , procurement and cancellation of e-resources be strengthened?

(2)How can collection development decisions through cost analysis of library be made?

(3)How can authorized limits be overcome in order to provide(3)How can authorized limits be overcome in order to provide users with the best access?

(4)How can new linked URL techniques be applied?( ) q pp(5)How can new retrieval tools or portals be used?(6)How can open architecture and industry standards be applied?


New challenges facing libraries圖書館使用報告統計要求與問題

E-resources budget cutting圖書館使用報告統計要求與問題

ChengChi University Library’s electronic database purchasing budget was reduced by 20%, about NT$120 000 00NT$120,000,00.

Many libraries collect usage statistics of electronic resources for purposes of performance evaluation;resources for purposes of performance evaluation; universities request reporting and proof of the value of electronic resources investment. Use study and evaluation of e-resources differ from print collectionsUsage statistics and evaluation of e-resources are important but difficult to implement.


OutlineOutline1. The growth of e-resources is transforming

th lib dithe library paradigm2.Problems with e-resource acquisition and

management3. The importance of e-resource evaluationp4.Usage study and usage statistics are

importantimportant5.Evaluation methods and models6 P f l ti f6.Performance evaluation of e-resources 7.Conclusion:how to cultivate evaluation

culture of libraries2012/11/15 e-resources evaluation /ML Wang

14

Problems from ChengChi University user survey政治大學使用者電子資源調查

Users love e-resources but raise more

政治大學使用者電子資源調查

questions.In 2010,users reported many problems with , p y pdatabases or e-resources retrieval such as they could not operate them, complex query p , p q yinterface, difficulties in using e-resources.In 2011,readers reported they needed moreIn 2011,readers reported they needed more databases. There are not enough to use databases, especially for electronic journals.databases, especially for electronic journals. Users have e-journals embargo problems.


Reflections on database services and e-resources from lib i 圖書館希望了解資料庫使用者libraries圖書館希望了解資料庫使用者

1. Libraries want to know users: Whether readers understand the available e-resources?Do readers understand how to find available e-resources and how to use them?and how to use them?Are readers satisfied wit the use of existing e-resources, and does their content meet the needs of learning and teaching g gand research?Why do readers not use electronic databases or e-resource services?

2. E-resources use studies will contribute to d t di th bl d diffi lti hi hunderstanding the problems and difficulties which

readers encounter in the use of e-resources for the basis of selection and extension of e-resources services.basis of selection and extension of e resources services.


E-resources evaluation is important電子資源評鑑十分重要電子資源評鑑十分重要

Library acquire many e-resources, but do not understand how users use e-resources and evaluate their quality.q yThrough network services , user use e-resources , such as online retrieval system, digital reference, interlibrary cooperation, DB promotion.Methods of traditional collection evaluation do not apply to e-resourcesresources.Usage statistics are provided by database vendors, the lack of consistent definition and different interpretation of usage p gstatistics make the evaluation of e-resources difficult.High frequency of using e-resources does not necessarily

h i li f i f i h h ’ll bguarantee their quality of information or that they’ll be popular with readers.Use and evaluation of library e-resources is an important issueUse and evaluation of library e-resources is an important issue worth exploring.


ARL New Measurement Initiatives

LibQual+ARL EmetricsARL Statistics™ARL StatisticsClimateQUAL®Di iQUALDigiQUALLibrary ValueMINES for Libraries®Organizational PerformancePerformance Assessment







19

University library database management survey report大學圖書館資料庫管理調查報告大學圖書館資料庫管理調查報告

Interlibrary Cooperation Association investigated university libraries database management conditions to promote the benefits of purchasing database, in 2009.Survey showed that it is difficult to obtain results yof use and evaluation: human resources, condition, working priority of each library is , g p y ydifferent, and the survey methodology used differs. Libraries should conduct e-resource use surveys and use online questionnaires.and use online questionnaires.


E-book use surveys of university libraries in Taiwan大學院校圖書館電子書使用調查Taiwan大學院校圖書館電子書使用調查

Taiwan Academic E-book Consortium investigated e-b k b f d ti i it lib ibook use by surveys of domestic university librariesusing an online survey in October 2010. 26,120 students and faculty members 91 schools respondedstudents and faculty members ,91 schools responded. (1) Only 40% of teachers, researchers and students prioritize using e-books, but most university students still prefer paper books.

(2) Users emphasize the characteristics of e-books: full text of the image is clear, search function is good, the retrieval network isimage is clear, search function is good, the retrieval network is easy to operate, convenient to browse. Full text articles reading meets users’ need?(3) S ti f ti f i b k 53 1%(3) Satisfaction of users using e-books 53.1% (4)Main reasons of dissatisfaction: users don’t know e-books are available, not used to using them, not know how to use them.


Difficulties of e-resources use studies電子資源使用研究重要但困難Difficulties of e resources use studies

Use studies of e-resources are divided into 5 types.e-resources use statistics, computer transaction log analysis, user study, use study of mixed survey / observation / statistical analysis, and return on investment evaluation.

Since the statistics provided by database vendor to use libraries are not consistent, a number of statistical differences exist, which affect the

i i i f icredibility of library usage data.


How to solve problems of e-resources usage inconsistency 使用統計數據不一致問題的解決inconsistency 使用統計數據不致問題的解決

Librarians should understand theLibrarians should understand the statistical nouns used by agents, such as

i h d f ll t t d l d dsession, search, and full-text download, and their significanceWe need to develop standards and guidelines, and ask DB vendors to provideguidelines, and ask DB vendors to provide statistics information according to those standardsstandards.


E-resources usage standards電子資源統計標準電子資源統計標準

E-resources usage standards: ICOLC, COUNTER, id li f th E t i j t NISO ISO CNSguidelines of the E-metrics project, NISO, ISO, CNS

library statistical standards and other standards.ICOLC /The evaluation guidelines for web use areICOLC /The evaluation guidelines for web-use are based on indexes, abstracts and full-text e-resources statistics.statistics.

Number of Sessions Number of Queries (Searches)Number of Queries (Searches)Number of Menu Selections Number of Full-Content Units examinedNumber of Full Content Units examined, downloaded, - Journal articles –- Reference materials – - Non-textualNumber of TurnAways,


Counter Code of Practice for e-resources

COUNTER (Counting Online Usage of Networked e-resources) was formally established in March 2002resources) was formally established in March 2002, participants included libraries, publishers and intermediaries.Release 4 released in 2012 is an integrated Code of PracticeRelease 4, released in 2012, is an integrated Code of Practice covering journals, databases, books, reference works and multimedia content. The organization continues to promote the development of technical/operational guidelines for practical standards of e-resources statistics and plays an important role in electronic resource usage statistics.(COUNTER,2012)







28

What is e-resources evaluation電子資源評鑑意義電子資源評鑑意義

Evaluation is a planned way of judging effectiveness and efficiency of some activities/McClureefficiency of some activities/McClureE-resources evaluation is a planned way of depicting and judging the effectiveness and efficiency of e-resourcesjudging the effectiveness and efficiency of e resources evaluation.E-resources evaluation needs to consider both print collection and e-resources, and also collect related statistical data for evaluation.

(1)A ti iti i f ti d I t t f(1)Activities, information access and Internet use of users(2)Computer equipment offered by library(3)e resources and cost of network services(3)e-resources and cost of network services(4)Usage statistics of websites and databases, sessions,number of browsing, query, print and download, g, q y, p


E-resources evaluation approaches電子資源評鑑種類與取向電子資源評鑑種類與取向

Macro evaluation (total databases) VS ( )micro evaluation(one DB, or E-book DB, e-journal DB)e journal DB)E-resources-oriented, system-oriented

d i d l iand user-oriented evaluation.e-resources evaluation criteria including gcost, efficiency, effectiveness, benefits, user satisfaction outcome library valueuser satisfaction, outcome, library value and impact .


E-resources evaluation methods電子資源評鑑方法電子資源評鑑方法

Framework for e-resources evaluation l i h l i h devaluation approaches, evaluation methods,

evaluation models , data collectionE-resources evaluation methods

(1)Use study, usage statistics and use evaluation ( ) y, g(2)Performance evaluation(3)E metrics evaluation(3)E-metrics evaluation(4)Satisfaction and service quality evaluation(5)Usability evaluation(6) Library value and impact evaluation( ) y p


E-resources evaluation diagram電子資源評鑑示意圖電子資源評鑑示意圖

2012/11/1532

e-resources evaluation /ML Wang

Output assessment and performance l i 產出評鑑與績效評鑑evaluation產出評鑑與績效評鑑

Output assessment of e-resources andOutput assessment of e resources and services refers to measuring the use, accessibility efficiency and validity ofaccessibility, efficiency and validity of services and resources. Performance Assessment combines measurement statistics of individualmeasurement statistics of individual library services with resources and benefitsbenefits


LibrQual+ /Georgia 2010服務品質LibrQual+ /Georgia 2010服務品質


E-resources evaluation model電子資源評鑑模式五構面

Evaluation of e-resources is complex, so we need a set of evaluation models to implement it

電子資源評鑑模式五構面

set of evaluation models to implement it.Construction of evaluation：what do we evaluate? To what does “e resources” refer? Including which elementsdoes e-resources refer? Including which elements, components and procedures?Background of evaluation：goals structures viewpoints ofBackground of evaluation：goals, structures, viewpoints of evaluation, or levels of selection, evaluationShould reflect the performance criteria for the selected targetShould reflect the performance criteria for the selected target Should reflect the measure used in related standards of performance records, such as amount of usage and cost. p , gMethods of evaluation: measuring tools, material sampling, material search and collection procedure, how materials are analyzed


E-resources and services evaluation process電子資源評鑑工作程序process電子資源評鑑工作程序

1. Understanding of resources or servicesg2. Evaluation goals and limits of analysis

of financial resourcesof financial resources3. Decide evaluation method and model4. Decide data collection method strategy5 Implement data collection5. Implement data collection 6. Analyze data and propose evaluation

report


Data collecting for e-resources evaluation電子資源評鑑的資料蒐集

Quantitative Qualitative

電子資源評鑑的資料蒐集

QuantitativeUsage statistics of e-resources

QualitativeFocus group interviewsresources

Questionnaire surveyUsage transaction

interviewsUser observation

Usage transaction log analysisWeb site usageWeb site usage statistics by vendorsInterlibrary ycooperation needs statistics







38

Definition of performance evaluation of e-電子資源績效評鑑意義resources電子資源績效評鑑意義

Performance evaluation is an act of measuring the input, process output and outcome of a library based on evaluation standards, or evaluation i di t hi h d t i th d tindicators ,which determine the advantage, effectiveness, and value of library resources, operational managementoperational management.Performance evaluation on of library e-resources is to analyze and understand the value andis to analyze and understand the value and applicability of various e-resources for decision-makingmaking. ER performance evaluation can strengthen management and improve service qualitymanagement and improve service quality.


General evaluation system of an academic libraryacademic library


Studies on performance evaluation indicators績效評鑑指標相關研究績效評鑑指標相關研究

Establishing evaluation indicators is an i k f f l iimportant task of performance evaluation. Some studies

(1) The ARL E-Metrics project proposed evaluation dimensions and indicators of digital library.g y

(2)Korea : Noh(2010)proposed evaluation indicators of e-resourcesof e resources.

(3)China : Ge Xu(2008)proposed performance evaluation indicator system of e resourcesevaluation indicator system of e-resources.


performance evaluation model績效評鑑指標模式績效評鑑指標模式

Van House et al.(1990)proposed performance evaluation indicators of academic libraries, divided into input, processing, and output.Noh(2012)

proposed the Performance Evaluation Modelproposed the Performance Evaluation Model, measuring input, processing, output and effectitveness of a library.effectitveness of a library. This evaluation judges library resources, operation management output effectivenessoperation management , output, effectiveness, benefits, value..


How to apply e-resources indicators?電子資源績效指標之應用電子資源績效指標之應用

Performance indicatorswhich are quantity, and efficiency data of e resources usage and service collected bydata of e-resources usage and service collected by computers, including:

(1)Shows how network services and resources are presented(1)Shows how network services and resources are presented and accepted

(2)Effectiveness and efficiency of the measurement of(2)Effectiveness and efficiency of the measurement of online services and resources

(3)Determine levels of user satisfaction(3)Determine levels of user satisfaction(4) Verify user needs(5) Identify problems of users of the network services and(5) Identify problems of users of the network services and

resources


Library e-resources evaluation indicators in China大陸大學圖書館電子資源績效評鑑指標

Ge Xu(2008): l i i

5 dimensions, 29 consult university library e-resources indicators ARL E

indicators(1)Database content

indicators ARL E-Metrics architecture to

( )(2)Database search

systemarchitecture to determine important factors

system(3)Database usage(4)D t b timportant factors

and indicator weights of

(4)Database cost evaluation

gperformance evaluation.

(5)Services provided by database

dvendors2012/11/15 e-resources evaluation /ML Wang

44

Performance study of e-resources in Korean academic libraries韓國學術圖書館電子資源績效研究academic libraries韓國學術圖書館電子資源績效研究

Noh(2012) provided an input-output analysis of e-resources ll ti i d i lib i b if i l ticollection in academic libraries by verifying evaluation.

Evaluation indicators were divided into inputs and outputs.Indicators: e-resource acquisition(40% weight) e-resourceIndicators: e-resource acquisition(40% weight) , e-resource use(45% weight) , e-resource use environment(15% weight) . efficiency=output/input * 100efficiency output/input 100Study procedures

(1)The evaluation indicators were finalized after having been ( ) gverified through a Delphi survey.

(2)A group of experts assigned weights to the evaluation ( ) g p p g gindicators.

(3)Performance of academic libraries was evaluated by conducting a pilot study.


E-resource performance measurement model(Noh)

Input 52% Output 48%


Input-Output effectInput Output effect


Findings 韓國電子資源績效研究發現g 資源績效研究發現

(1) A large share of academic library budgets is spent on e-resource purchases and e-resource environment for better use.

(2)Web DB subscription, annual e-resource development, and promotion achieve to higher p , p gefficiency.

(3) Input-output ratio is calculated in percentage(3) Input output ratio is calculated in percentage terms, the efficiency of e-resources in Korean academic libraries reaches 88.20 per cent, universityacademic libraries reaches 88.20 per cent, university libraries need to improve e-resources.


Electronic journals network service and evaluation models /王梅玲電子期刊評鑑研究and evaluation models /王梅玲電子期刊評鑑研究

電子期刊資源電子期刊使用1電子期刊title數量 6電子期刊連線Session(login)次數電子期刊數量2電子期刊article數量3電子期刊收錄範圍(年/月)4提供索引摘要

電子期刊連線 ( g )次數7目次8摘要9電子期刊查詢次數

5提供目次 10電子期刊下載格式11電子期刊紀錄下載次數12電子期刊虛擬造訪次數(OPAC、Web)13電子期刊連線時間14電子期刊連線被拒次數電子期刊參考諮詢次數15電子期刊參考諮詢次數

電子期刊成本電子期刊使用績效16電子期刊成本17對書目中心網路與聯盟支出

18每次連線平均下載全文論文篇數19連線被拒次數佔連線次數百分比17對書目中心、網路與聯盟支出 19連線被拒次數佔連線次數百分比20平均每次全文論文下載成本21平均每篇期刊論文成本22平均每篇全文論文被下載次數22平均每篇全文論文被下載次數23平均每種電子期刊成本


ARL Digital Library E-Metrics Projectg y j

This project was conducted by the Association of Research Libraries(ARL) ,December 2001.The E-Metrics Project suggests an evaluation structure of e-resources including five dimensions and nineteen indicators,19 indicators

(1)Patron access e-resources(2)Use of network resources and infrastructure(3)Expenditure of network resources(4)Library digitized activities(5)Performance measurement


Conclusion結論Conclusion結論

No evaluation , no management./Joseph Methews, g pUniversity libraries spend a lot of their budget on purchasing rich digital resources, up to 50% of library budget. User-centered collection development and services will make libraries more value and impact . As e-resources costly, libraries will face the issue of budget cutting in near future.Use study and evaluation of e-resoures are importantMany e-resources evaluation methods are introduced, such as

t d d l ti f l tiuse study and use evaluation ;performance evaluation; e-metrics evaluation ;satisfaction and service quality evaluation; usability evaluation; value and impact evaluationevaluation; usability evaluation; value and impact evaluation


Future suggestions未來建議1/2Future suggestions未來建議1/21.Libraries should continue to keep in touch on the

development of e-resources ,new media , digital publishing and the trends of new technologies .

2.Libraries should actively join consortium of e-resources actively and establish “e-resource usage t ti ti i l i t t " t hstatistics special interest groups " to share e-

resource usage survey methods, techniques and experienceexperience.

3. Libraries should pay attention to collecting e-resource usage statistics user needs informationresource usage statistics, user needs information and use cost-benefit analysis to conduct e-resources evaluationresources evaluation.


Future suggestions未來建議2/2Future suggestions未來建議2/24.Libraries should conduct e-resources collection

l ti d ti f ti fevaluation and user satisfaction surveys for adjustment of e-resources collection development and management operationsmanagement operations.

5. Libraries should pay attention to reading habits and needs of users on e-resources when conducting e-needs of users on e resources when conducting eresources collection development and management, and cooperate fully with the department of reader services and promotion services.

6. Libraries can apply performance evaluation in the ll l i f ll hoverall evaluation of e-resources, as well as the same

type of database performance evaluation.


Question and Answer問題討論

Thank you

王梅玲

政治大學圖書資訊與檔案學研究所

email:[email protected]


electronic resources evaluationofacademicevaluation of ... · u.s. arl collection budget 1992-2011...

Documents