E-LIS, Eprints in Library and Information Science Homepage E-LIS, Eprints in Library and Information Science
   home   |   about   |   search   |   browse   |   register   |   registered users area   |   help   |   FAQ   |   JITA   

Developing Multi-Method, Iterative, and User-Centered Evaluation Strategies for Digital Libraries: Functionality, Usability, and Accessibility

Snead, John T. and Bertot, John Carlo and Jaeger, Paul T. and McClure, Charles R. (2005) Developing Multi-Method, Iterative, and User-Centered Evaluation Strategies for Digital Libraries: Functionality, Usability, and Accessibility. In Grove, Andrew, Eds. Proceedings 68th Annual Meeting of the American Society for Information Science and Technology (ASIST) 42, Charlotte (US).

Full text available as:
PDF - Requires Adobe Acrobat Reader or other PDF viewer.

View statistics for this eprint

Abstract

Usability, functionality, and accessibility testing of digital library information services and products are essential for providing high quality services to users. This paper details a long-term, evolving effort to develop meaningful evaluations for assessing digital libraries. The authors of this paper have been engaged in a multi-year study to determine appropriate evaluation techniques, tools, and methodologies for the Florida Electronic Library (FEL) and other digital libraries. The evaluation protocols and approaches have been designed over time and iteratively through assessment efforts of the research team of other digital library initiatives and with multiple versions of the FEL. As such, this paper examines the process of developing, applying, and refining appropriate evaluation methodologies for the networked environment of libraries, as well as the implications of these methodologies. The approach taken in the research described herein relies on a combination of evaluation strategies applied iteratively
to assess libraries from the perspective of patron needs. A number of specific methods, as shown in this paper, can be
readily developed to provide such evaluations. The following goals guide the presentation in this paper of the
development through time of the methods and instruments created, tested, refined, and operationalized in functionality, usability, and accessibility testing by the researchers. The goals of this paper are to: 1) demonstrate the potential roles of multiple, iterative evaluation strategies in the development and refinement of digital libraries; 2) detail the methodologies that focus on how the services meet the needs of users; and 3) encourage further discussion of the uses of these multiple evaluation approaches in assessing these libraries.

Keywords:assessment of digital libraries
Subjects:B. Information use and sociology of information. > BA. Use and impact of information.
H. Information sources, supports, channels. > HZ. None of these, but in this section.
ID Code:5268
Deposited By:Medeiros, Norm
Deposited On:10 March 2006
All fields:Show all fields

Battleson, B., Booth, A., & Weintrop, J. (2001). Usability testing of an academic library web site: A case study. Journal of Academic Librarianship, 27, 188-198.

Bertot, J. C. (2002). Project deliverable g: Assessment of ZLOT resource discovery service proof of concept prototypes. Austin, TX: Texas Center for Digital Knowledge. Available at http://www.unt.edu/zlot/poc/poc index.htm

Bertot, J. C. (2003). Libraries and networked information services: Issues and considerations in measurement. Proceedings of the 5th Northumbria International Conference on Performance Measurement in Libraries and Information Services, (pp. 15-25). University of Northumbria at Newcastle, Durham, England.

Bertot, J. C., & Davis, D. M. (2004). Planning and evaluating library networked services and resources. Westport, CT: Libraries Unlimited.

Bertot, J. C., & McClure, C. R. (2003a). Outcomes assessment in the networked environment: Research questions, issues, considerations, and moving forward. Library Trends, 51(4): 590-613.

Bertot, J. C. & McClure, C. R. (2003b). Assessing LSTA project outcomes: Methods and practice. Available at http://www.ii.fsu.edu/index.cfm

Bertot, J. C., McClure, C. R., Thompson, K. M., Jaeger, P. T., & Langa, L. A. (2003). Florida electronic library: Pilot project functionality assessment for the Florida Division of Library Services. Tallahassee, FL: Information Use Management and Policy Institute.

Bertot, J. C. & Snead, J. T. (2003). Social measurement in libraries. In K. Kempf-Leonard, (Ed.), Encyclopedia of Social Measurement. San Diego, CA: Academic Press.

Bishop, A. P., Mehgra, B., Bazzell, I., & Smith, C. (2003). Participatory action research and digital libraries: Reframing evaluation. In A. P. Bishop, N. A. Van

House, & P. Butterfield (Eds.). Digital library use: Social practice in design and evaluation (pp. 161-190), Cambridge, MA: MIT Press.

Borgman, C. L. (2003). Designing digital libraries for usability. In A. P. Bishop, N. A. Van House, & P. Butterfield (Eds.), Digital library use: Social practice in design and evaluation (pp. 85-118). Cambridge, MA: MIT Press.

Buckland, M. (2003). Five grand challenges for library research. Library Trends, 51(4), 675-86.

Campbell, N. (Ed.). (2001). Usability assessment of library-related websites: Methods and case studies. Chicago: LITA.

Carvalho, S. & White, H. (2004). Theory-based evaluation: The case of social funds. American Journal of Evaluation, 25, 141-160.

Dalrymple, P. W. & Zweizig, D. L. (1992). Users’ experiences of information retrieval systems: An exploration of the relationship between search experience and affective measures. Library & Information Science Research, 14, 167-181.

Feinstein, O. (2002). Use of evaluations and the evaluation of their use. Evaluation, 8, 433-439.

Hallfors, D., Khatapoush, S., Kadushin, C., Watson, K., & Saxe, L. (2000). A comparison of paper vs computer-assisted selfinterview for school alcohol, tobacco, and other drug surveys. Evaluation and Program Planning, 23, 149-155.

Hert, C. A. (2001). User-centered evaluation and its connection to design. In C. R. McClure & J. C. Bertot, Evaluating networked information services: Techniques, policy, and issues (pp. 155-174). Medford, NJ: Information Today.

Jaeger, P. T. (2002). Section 508 goes to the library: Complying with federal legal standards to produce accessible electronic and information technology in libraries. Information Technology and Disabilities, 8(2). Available at

http://www.rit.edu/~easi/itd/itdv08n2/jaeger.html

King, N., Ma, T. H-Y., Zaphris, P., Petrie, H., & Hamilton, F. (2004). An incremental usability and accessibility evaluation framework for digital libraries. In P. Brophy, S. Fisher, & J. Craven (Eds.), Libraries without walls 5: The distributed delivery if librarian and information services (pp. 123-131). London: Facet.

Marchionini, G., Plaisant, C., & Komlodi, A. (2003). The people in digital libraries: Multifaceted approaches to assessing needs and impacts. In A. P. Bishop, N. A. Van House, & P. Butterfield (Eds.), Digital library use: Social practice in design and evaluation (pp. 119-160). Cambridge, MA: MIT Press.

Mathison, S. (2001). What’s it like when the participatory evaluator is a “genuine” stakeholder? American Journal of Evaluation, 22, 29-35.

McClure, C. R. & Bertot, J. C. (2001). Introduction. In C. R. McClure & J. C. Bertot, Evaluating networked information services: Techniques, policy, and issues (pp. xiii-xx). Medford, NJ: Information Today.

McClure, C. R., Snead, J. T., Bertot, J. C. (2004). Public library geographical database final evaluation report. Tallahassee, FL: Information Use Management and Policy Institute.

McClure, C. R.; Snead, J. T.; Thompson, K. M.; & Bertot, J. C. (2004). Measures and Statistics to Assess the Florida Electronic Library. Tallahassee, FL: Division of Library and Information Services, The Florida State Library.

Moen, W. E. & Murray, K. R. (2002). Project deliverable c: Functional requirements for the Library of Texas. Austin, TX: Texas Center for Digital Knowledge. Available at http://www.unt.edu/zlot/projectdocs.htm

Nadler, D. M. & Furman, V. M. (2001). Access board issues final standards for disabled access under Section 508 of Rehabilitation Act. Government Contract Litigation Reporter, 14(19), 14.

Norlin, E. & Winters, C. M. (2002). Usability testing for library websites: A hands-on guide. Chicago: American Library Association.

Patton, M. Q. (1997). Utilization-focused evaluation: A new century text (3rd ed.). Thousand Oaks, CA: Sage.

Ryan, J., McClure, C. R., & Bertot, J. C. (2001). Choosing measures to evaluate networked information resources and services: Selected issues. In C. R. McClure & J. C. Bertot, Evaluating networked information services: Techniques, policy, and issues (pp. 111-136). Medford, NJ: Information Today.

Section 508 of the Rehabilitation Act, 29 U.S.C. § 794d. Available at http://www.section508.gov

Snead, J. T., McClure, C. R., Jorgensen, P., & Bertot, J. C. (2004). Evaluation of selected components of the Florida Electronic Library: Functionality, usability, and accessibility testing. Tallahassee, FL: Division of Library and Information Services, The Florida State Library

Sweeney, M., Maguire, M., & Shackel, B. (1993). Evaluating user-machine interaction: A framework. International Journal of Man-Machine Studies, 38, 689-711.

Thompson, K. M., McClure, C. R., & Jaeger, P. T. (2003). Evaluating federal websites: Improving e-government for the people. In J. F. George (Ed.), Computers in society: Privacy, ethics, and the Internet (pp. 400-412). Upper Saddle River, NJ: Prentice Hall.

Thompson, K. M., McClure, C. R., Ott, M., & Snead, J. T. (2004). Usability assessment of the Public Library Geographic Database (PLGDB). Tallahassee, FL: Information Use Management and Policy Institute.

Van den Haak, M. J., De Jong, M. D. T., & Schellens, P. J. (2003). Retrospective versus concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behavior & Information Technology, 22, 339-351.

Van den Haak, M. J., De Jong, M. D. T., & Schellens, P. J. (2004). Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers, 16, 1153-1170.

Usability.gov. (2004) [Homepage]. Available at http://www.usability.gov

Wallace, D. P. (2001). The nature of evaluation. In D. P. Wallace & C. V. Fleet (Eds.), Library evaluation: A casebook and can-do guide (pp. 209-220). Westport, CT: Libraries Unlimited.

World Wide Web Consortium. (1998). Web content accessibility guidelines. Available at http://www.w3c.org

Archive Staff Only: edit this record

   home   |   about   |   search   |   browse   |   register   |   registered users area   |   help   |   FAQ   |   JITA   

Send comments and questions to      |     E-LIS is hosted by AEPIC at CILEA      |     Powered by GNU Eprints software