इस ब्लॉग्स को सृजन करने में आप सभी से सादर सुझाव आमंत्रित हैं , कृपया अपने सुझाव और प्रविष्टियाँ प्रेषित करे , इसका संपूर्ण कार्य क्षेत्र विश्व ज्ञान समुदाय हैं , जो सभी प्रतियोगियों के कॅरिअर निर्माण महत्त्वपूर्ण योगदान देगा ,आप अपने सुझाव इस मेल पत्ते पर भेज सकते हैं - chandrashekhar.malav@yahoo.com
20. Evaluation of Digital Library
P- 01. Digital Libraries*
By :Jagdish Arora, Paper Coordinator
Multiple Choice Questions
1 / 1 Points
Question 1: Multiple Choice
COUNTER stands for
Counting Networked television Ratio- Counting Networked Electronic Resources
- Counting Electronic Resources
Counting Online Usage of Networked Electronic Resources
0 / 1 Points
Question 2: Multiple Choice
Which of following tool is not used for measuring usage statistics
- COUNTER
- SUSHI
- INFED
Shibboleth
0 / 1 Points
Question 3: Multiple Choice
Which of the following is not a evaluation model for digital library.
- DigiQUAL
- E-metrics
- Both (A) & (B)
- None of the above
1 / 3 PointsFinal Score:
...................................................................................................................................................................
Glossary
C
- Counting Online Usage of Networked Electronic Resources (COU
- An international initiative launched in March 2002 to serve librarians, publishers, and intermediaries by developing more consistent, reliable, and meaningful measures of online resource usage to facilitate the recording and exchange of e-usage statistics. The project initially focused on journals and bibliographic databases but will include other types of networked resources, such as e-books.
D
- Digital Elevation Model (DEM)
- A digital representation of cartographic information in the form of a raster grid of regularly spaced elevation values for ground positions derived primarily from the conversion of contours on printed topographic map series and from photogrammetric.
E
- Evaluation
- Evaluation is a systematic determination of a subject's merit, worthiness and significance, using criteria governed by a set of standards. It can assist an organization to assess any objective, realizable concept/proposal, or any alternative, to help in decision-making. The main purpose of evaluation, is in addition to gain clear perception into existing initiatives, to enable reflection and to assist in the identification of future change.
0. Objectives
Digital libraries are combination of state-of-the-art systems, collection and resources that are available in different formats. Evaluation of digital library is defined as systematic process of determining the merits, value and worthiness. Evaluation is a general term that includes several aspects of performance measurement and assessment and the activities that include experiments, surveys, time series analysis, online monitoring of user-system, observation of use, and other forms of data collection for the ultimate purpose of evaluation. Evaluation has many intentions ranging from highly focused and well-defined product testing to the highest form of cognitive reflection.
The objectives of this module is to impart knowledge on various aspects of evaluation of digital library includes:
- Basic concept behind evaluation of digital library
- Need and purpose of digital library evaluation
- Different approach to digital library evaluation
- Process and steps involved in digital library evaluation
- Crtieria for evaluation digital library and how to evaluate
- Different stages of evaluation
- Models of digital library evaluation.
1.0 Introduction
The world is changing very fast with increased application of information and telecommunication technology. With the advent of information and communication technology the concept of the library has gone under complete metamorphosis. Digital libraries extend and integrate approaches adopted in traditional libraries, as well as in distributed information systems, to concede high-end information systems, services, and institutions. Continuing research and innovations in various fields of studies led to a information explosion not only to preserve the flow of knowledge but also as solution for the problems that persist in the traditional library system. In last couple of years, the library and information centers have changed drastically in terms of its collection, organization and services. Digital library has the potentials to reach out to its users without geographical restrictions. Information and communication technology have changed the concept of the library as never before. Nowadays, libraries and information centers across the world are rapidly moving towards the digital format and collection. However, it requires sufficient fund to built-up a digital collection using known digital library management system. Development of digital libraries requires immense investment in terms of trained manpower, latest technology and sufficient funds. As such, it is essential to carry out timely evaluation for the digital library. The evaluation of the digital library helps in finding its integrity and necessity. The process allow managers to judge whether a digital library meets the objective for which was developed. Therefore, evaluation of a digital library is done for decision maker to take necessary policy decision on the development of digital library. Evaluation should be carried out on a regular interval to meet the user’s requirement and to keep the digital library up-to-date and relevant.
02. Need for Digital Library Evaluation
The first and foremost step in any evaluation process is to establish a strong and clear cut rationale for evaluation. In simple words, one should begin any evaluation by answering the question: “Why evaluation is necessary?”
Evaluations are carried out for many reasons. Often, they are conducted to comply with some sort of requirement laid down by a funding agency. It may also be done for political reasons, or simply because the people involved in an enterprise may try to find out whether vigorous exercise that is being done is on the right track or not. Various types of professionals organizations carry out evaluation process to help in taking decisions. For example, a doctor inquires about medical histories, conducts examinations, and may perform various medical tests before diagnosis and treatment. Likewise, those who are involved in making decisions about digital libraries, have to perform various tests called evaluations. There are certain advanced areas in digital libraries whose evaluation may require help of experts. Evaluation of the digital library is a fact finding process on the basis of collected evidences. It also includes value measuring which are integrated in the management process of digital libraries. It is an assessment of the performances, services, systems, products etc. Evaluation of digital library is often treated as a simple task and many times it is overlooked but, actually it is complex and scientific. Because of the complexity and multi dimensionality of the digital library it becomes quite technical to evaluate it.
In the evaluation of digital libraries, as in the evaluation of any other system or process, a difficult question arises that clearly affect the results: Where does a digital library under evaluation begin to be evaluated? Where does it end? What are the boundaries? What to include? What to exclude? According to Saracevic (2000) evaluation means an appraisal of the performance or functioning of a system, or part thereof, in relation to some objective The performance can be evaluated as:
a) Effectiveness: How well does a system (or any of its parts) perform for which it was designed?
b) Efficiency: At what cost (costs could be financial or involve time or effort)?
c) A combination of these two (i.e., cost-effectiveness).
According to Chawdhury & Chowdhury (2002) digital libraries are judged by their effectiveness in terms of cost and use. Marchionini (2000) has pointed out that evaluation is a research process that aims to understand the meaning of a specific phenomenon situated in the context.
The evaluation process for digital library needs to be carried out by the experts in the field, who can evaluate using various approaches. General framework of the evaluation of digital library includes approach systems. Digital libraries are different than traditional libraries in terms of its collection, execution and support and no evaluation can possibly address all of the aspects altogether. Thus, different approaches have to be utilized for evaluation. Important approaches used in evaluation of digital library are given below.
3.1 User and Content Centric Approach
The user centric approach involves assessment of features used by end users which are integrated in digital library portal. This approach is a bridge between human and system. This approach is used before developing a digital library.
3.2 System Centric Approach
The system centric approach involves study of software and its performances. It also assesses effectiveness of the system design and technological component. This approach is mostly used at the time of the system design and deployment.
3.3 Economic Approach
Economic approach is basically study of cost, it measures cost effectiveness of the digital library. This approach will help policy makers to take decision about worthiness of digital library.
4.0 How to evaluate DL: Process and Various Steps
Several complex evaluation models are proposed in scholarly papers, and evaluation specialists often have high qualifications in specific area. However, the simple evaluation process may have following steps:
- Identify the activities involved in the digital library;
- Identify the questions that are to be addressed to inform the decision makers;
- Identify the evaluation methods, models and tools, which will be used for data collection related to the questions.
- Bifurcate evaluation in a phased manner so that it becomes effective and efficient.
- Write up the finding of the evaluations in accurately so that it can provide meaningful data to make the best possible decisions.
Evaluation process for traditional library were established much before the invention of information and communication technology. The evaluation procedure for digital library is quite different and more advanced and it is carried out using latest technology.
The evaluation process for digital library is followed in a sequential manner starting from planning to evaluation. The evaluation process involves following steps:
- Surveys, including direct questionnaires and online surveys as well as structured interviews of users and developers
- Observations
- Transactional log analysis
- Experimentation
- Records analysis
- Usage analysis (COUNTER Statistics)
- Economic analysis
Fig. 1: Evaluation process of Digital Library
The evaluation process starts with to-do list of various activities to be performed at the time of evaluation. Once the activities are planned, methodology is selected based on data to be collected and analyzed by various analytical systems. The analyzed data is used for taking policy decision, and the decision makers set various criteria on the basis of results for funding the project. Finally, it is decided whether digital library achieves the objectives for which it was developed.
5. Criteria for evaluation of Digital Library
Evaluation is a combination between selected elements to be evaluated and selected elements of its performance. This leads to selection of stages and levels of evaluation. Digital libraries, like other systems, can be viewed, and evaluated, from a number of slants or levels. Each of these slants can be translated into a criteria for evaluation. An evaluation focuses on decisions about the types of services that the digital library should include. Some of the questions that might be answered are:
- Which types of services are offered by digital libraries?
- Which services the patrons have requested?
- For which services patrons are willing to pay?
- What services would require external funding?
Following are some of the important criteria that need to be used for the evaluation of digital library:
5.1 Usability
Usability is one of the important criterion for evaluation of digital library especially with respect to interface design and operability. This criterion evaluates navigation efficiency, search retrieval efficiency and informative entropy as well.
5.2 Quality of the Collection
One of the important constituent of a digital library is collection/content. The uploaded content should be evaluated on regular interval as there are many innovations are taking place in the same area of knowledge. The relevancy of the content should be maintained for which a digital library is developed.
5.3 Service Quality
One major criticism for the existing digital libraries is lack of a community service, which is one of the important functions for traditional library system. Community services refer to services that are provided to members of digital library community, such as allowing members to exchange ideas, make announcements etc.
5.4 Performance of the System
System performance is related to the relevance of retrieval of results and efficiency of the retrieval process. The main problem identified regarding system performance is how to increase precision ratio. In general, users care more about precision than recall since they only need enough information to solve their problems. In addition, precision affects the time a user needs to spend to evaluate and find relevant information to solve his/her problems in the digital library.
6.0 What to Evaluate?
According to Chawdhury & Chowdhury (2002), a digital library may be evaluated from a number of perspectives, such as system, access and usability, user interface, information retrieval, content, services, cost as well as overall benefits and impact.
Saracevic (2000) provided a list of elements to be evaluated at the time of evaluation of digital libraries, wherein he has clearly indicated what to include and what to exclude. He suggested that the following elements may be considered for evaluation:
- Digital collection and digital resources
- Organizational structure and storage
- Metadata and access management
- Physical network and distribution
- Security, privacy and legal aspects
- Licenses and cost
- User interface and retrieval etc.
7.1 Evaluation during System Design
In general, system for digital library is designed after consultation with users. The need to evaluate latest technology that is used for its development is one of the criterion for evaluation. The system needs to be designed on the basis of user’s requirements. At the time of designing system for digital library one has to conduct interviews, observations on user’s need and accordingly the system can be designed.
7.2 Evaluation during System Development
The second stage of evaluation is carried out during the development of the digital library. This includes logical work-flow, user friendly interface and necessary guidelines for the operation of the digital library. Since the development is being done by the technical professional, it needs to be evaluated at the time of the development to make it fruitful.
7.3 Evaluation during System Deployment
Deployment includes system execution, its use and the maintenance. Technical professional first release the beta version of the developed system that is tested on the basis of requirements. System deployment means that the software is ready for deployment.
8.2 System
The system is one of the complex aspects of the digital library. One has to evaluate hardware, software and network technology used in developing an infrastructure. During evaluation of system it is also expected to judge about the procedures, techniques, algorithms and operations performed by the digital library.
9. Models for Evaluation of Digital Library
To evaluate digital library applications, following two models are very much popular and used most often.
9.1 DigiQUAL
DigiQUAL is a model developed to measure service quality of digital library by the Association of Research Libraries with funding from the National Science Foundation. The DigiQUAL project is the new version of the existing LibQUAL+ protocol which was used to assess the services provided for the user communities of the National Science Digital Library (NSDL) program in US.
9.2 E-metrics
The e‐metrics is a tool for library professionals to evaluate the use of digital library resources and services which are to be measured for its worthiness. The e-metrics is a set of standardized tools and shared understandings about the value of the metrics for making data‐driven, managerial decisions in libraries. E‐metrics are applied to a number of library resources and service domains. It also provides statistics of usage of networked electronic resources collected by external vendors conforming to standards, such as COUNTER (Counting Online Usage of Networked Electronic Resources) and SUSHI (Standardized Usage Statistics Harvesting Initiative), also developed by external to the library organizations.
10. Summary
Digital libraries are now at the mature stage and well developed in terms of technology, but it is quite necessary to evaluate various strategies used at the time of its development. Service is one of the major parts of the digital library in view of users and therefore, it seems very much essential to evaluate digital libraries. Since there is less visible movements in evaluation digital libraries, the funding agencies and users are keen to evaluate digital libraries. Thus, it should become an imperative that efforts in formal evaluation of digital libraries be enlarged and become an integral part of all research and practice.
References
- Buttenfield, B. (1999). Usability Evaluation of Digital Libraries. Science & Technology Libraries, 17(3-4), 39–59. doi:10.1300/J122v17n03_04
- Chawdhury, G. G., & Chowdhury, S. (2002). Digital library evaluation. In Introduction to digital libraries (2nd ed., p. 304). London: Facet Publishing. Retrieved fromhttp://www.facetpublishing.co.uk/title.php?id=4653&category_code=301
- Iris Xie, H. (2006). Evaluation of digital libraries: Criteria and problems from users’ perspectives. Library & Information Science Research, 28(3), 433–452. doi:10.1016/j.lisr.2006.06.002
- Lesk, M. (2005). Usability and Retrieval Evaluation. In M. Lesk (Ed.), Understanding Digital Libraries (Second Edition) (pp. 185–215). San Francisco: Morgan Kaufmann. Retrieved fromhttp://www.sciencedirect.com/science/article/pii/B9781558609242500104
- Marchionini, G. (2000). Evaluating Digital Libraries: A Longitudinal and Multifaceted View. Retrieved from http://hdl.handle.net/2142/8336
- Nicholas, D., & Huntington, P. (2010). Evaluating the Use and Users of Digital Journal Libraries. In F. Papy (Ed.), Digital Libraries (pp. 211–222). ISTE. Retrieved fromhttp://onlinelibrary.wiley.com/doi/10.1002/9780470611302.ch13/summary
- Reevew, T. C., Apedoe, X., Woo, Y. H., & University of Georgia. (2005). Digital Libraries: A User- Friendly Guide. National Science Digital Library. Retrieved fromhttp://www.dpc.ucar.edu/projects/evalbook/EvaluatingDigitalLibraries.pdf
- Saracevic, T. (2000). Digital Library Evaluation: Toward Evolution of Concepts. Library Trends, 49(2). Retrieved fromhttp://alexia.lis.uiuc.edu/puboff/catalog/trends/49_2abs.html#saracevic
- Tsakonas, G., & Papatheodorou, C. (2009). Evaluation of Digital Libraries: An insight into Useful Applications and Methods. Elsevier.
- Zia, L. L. (2000). The NSF National Science, Mathematics, Engineering, and Technology Education Digital Library (NSDL) Program: A Progress Report. D-Lib Magazine, 6(10). doi:10.1045/october2000-zia
- https://liber.library.uu.nl/index.php/lq/article/view/URN%3ANBN%3ANL%3AUI%3A10-1-113589/8300 Accessed on Feb 27, 2014
- http://www.digiqual.org/about Accessed on Feb 07, 2014
Web links
- Tefko Saracevic, Kantor, P. (1997). Studying the value of library and information services. I. Establishing a theoretical framework. II. Methodology and Taxonomy. Journal of the American Society for Information Science, 48(6), 527-542, 543-563.
- Tefko Saracevic, 2000. "Digital library evaluation: Toward an evolution of concepts," Library Trends, volume 49, number 2 (Fall), pp. 350–369
- Marchionini, G.; Plaisant, C.; & Komlodi, A. (199...) The people in digital libraries: Multifaceted approaches to assessing needs and impact. Chapter in Bishop, A. Buttenfield, B. & VanHouse, N. (Eds.)
- Digital library use: Social practice in design and evaluation. MIT Press. (http://ils.unc.edu/~march/revision.pdf)
Glossary
Counting Online Usage of Networked Electronic Resources (COUNTER)
An international initiative launched in March 2002 to serve librarians, publishers, and intermediaries by developing more consistent, reliable, and meaningful measures of online resource usage to facilitate the recording and exchange of e-usage statistics. The project initially focused on journals and bibliographic databases but will include other types of networked resources, such as e-books.
Digital Elevation Model (DEM)
A digital representation of cartographic information in the form of a raster grid of regularly spaced elevation values for ground positions derived primarily from the conversion of contours on printed topographic map series and from photogrammetric.
Evaluation:
Evaluation is a systematic determination of a subject's merit, worthiness and significance, using criteria governed by a set of standards. It can assist an organization to assess any objective, realizable concept/proposal, or any alternative, to help in decision-making. The main purpose of evaluation, is in addition to gain clear perception into existing initiatives, to enable reflection and to assist in the identification of future change.
No comments:
Post a Comment