Analysis of Search Failures in Document Retrieval Systems: a Review

1. M. E. Maron, "Probabilistic Retrieval Models," in Progress in Communication Sciences, vol. 5, ed. Brenda Dervin and Melvin J. Voigt. (Norwood, NJ: Ablex, 1984), 145-176.

2. Ibid., 155.

3. Ibid.

4. C. J. Van Rijsbergen, Information Retrieval, 2nd ed. (London: Butterworths, 1979), 10.

5. David C. Blair and M. E. Maron, "An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System," Communications of the ACM 28 (March 1985): 291.

6. S. E. Robertson, M. E. Maron, and W .S. Cooper, "Probability of Relevance: A Unification of Two Competing Models for Document Retrieval," Information Technology: Research and Development 1 (1982): 1.

7. Tefko Saracevic, "Relevance: A Review of and a Framework for the Thinking on the Notion in Information Science," Journal of the American Society for Information Science 26 (1975): 321-343. See also: Michael Eisenberg and Linda Schamber, "Relevance: The Search for a Definition," in ASIS '88: Proceedings of the 51st ASIS Annual Meeting, Atlanta, Georgia, October 23-27, 1988, ed. Christine L. Borgman and Edward Y .H. Pai. (Medford, NJ: Learned Information, 1988), 164-168.

8. An attempt has been made in Cranfield II to plot recall/fallout graphs. The size of the collection used in this experiment was relatively small (1,400 documents) and many tests were done with 200 documents. Nevertheless, no analysis has been performed to find out the causes of fallout failures. For details, see: Cyril Cleverdon, Jack Mills, and Michael Keen, Factors Determining the Performance of Indexing Systems, Volume 1, Design (Cranfield, England: Aslib, 1966); and Cyril Cleverdon and Michael Keen, Factors Determining the Performance of Indexing Systems, Volume 2, Test Results (Cranfield, England: Aslib, 1966).

9. Ray R. Larson, "Between Scylla and Charybdis: Subject Searching in the Online Catalog," in Advances in Librarianship, vol. 15, ed. Irene P. Godden (San Diego, CA: Academic Press, 1991), 188. See also: S. E. Wiberley and R. A. Dougherty, "Users' Persistence in Scanning Lists of References," College & Research Libraries 49 (1988): 149-156.

10. J. L. Kuhns implied that frustration usually occurs when a user reaches his or her "futility point" in a given search. The futility point is defined as "the number of retrieved documents the inquirer is willing to browse through before giving up his search in frustration." Source: David C. Blair, "Searching Biases in Large Interactive Document Retrieval Systems," Journal of the American Society for Information Science 31 (July 1980): 271.

11. Michael Buckland and Fredric Gey, personal communication, 1991.

12. Robert Wages, "Can Easy Searching be Good Searching? A Model for Easy Searching," Online 13 (May 1989): 80.

13. William S. Cooper, "On Selecting a Measure of Retrieval Effectiveness," Journal of the American Society for Information Science 24 (1973): 87-100, 413-424. Compare this with: Dagobert Soergel, "Is User Satisfaction a Hobgoblin?," Journal of the American Society for Information Science 27 (July-August 1976): 256-259.

14. Ibid., 88.

15. Judith A. Tessier, Wayne W. Crouch, and Pauline Atherton, "New Measures of User Satisfaction with Computer-Based Literature Searches," Special Libraries 68 (November 1977): 383-389.

16. Marcia J. Bates, "Factors Affecting Subject Catalog Search Success," Journal of the American Society for Information Science 28 (May 1977): 161-169.

17. Mark T. Kinnucan, "The Size of Retrieval Sets," Journal of the American Society for Information Science 43 (January 1992): 73.

18. Susan E. Hilchey and Jitka M. Hurych, "User Satisfaction or User Acceptance? Statistical Evaluation of an Online Reference service," RQ 24 (Summer 1985): 455.

19. Renata Tagliacozzo, "Estimating the Satisfaction of Information Users," Bulletin of the Medical Library Association 65 (April 1977): 248.

20. Ibid.

21. Melvon L. Ankeny, "Evaluating End-User Services: Success or Satisfaction," Journal of Academic Librarianship 16 (January 1991): 356.

22. Ibid., 354. See also: Ethel Auster and Stephen B. Lawton, "Search Interview Techniques and Information Gain as Antecedents of user satisfaction with Online Bibliographic Retrieval," Journal of the American Society for Information Science 35 (March 1984): 90-103.

23. Sandra R. Wilson, Norma Starr-Schneidkraut, and Michael D. Cooper, Use of the Critical Incident Technique to Evaluate the Impact of MEDLINE. (Palo Alto, CA: American Institutes for Research, 1989), AIR-64600-9/89-FR. For hypothetical examples as to the importance of unretrieved but relevant documents, see: Soergel, "Is User Satisfaction a Hobgoblin?," 258-259.

24. Ankeny, "Evaluating End-User Services," 356.

25. Debora Cheney, "Evaluation-Based Training: Improving the Quality of End-User Searching," Journal of Academic Librarianship 17 (July 1991): 155.

26. Tefko Saracevic and Paul Kantor, "A Study of Information Seeking and Retrieving. II. Users, Questions, and Effectiveness," Journal of the American Society for Information Science 39 (May 1988): 177-196.

27. Tefko Saracevic, Paul Kantor, Alice Y. Chamis, and Donna Trivison, "A Study of Information Seeking and Retrieving. I. Background and Methodology," Journal of the American Society for Information Science 39 (May 1988): 161-176. Note that it is not discussed in this paper how they calculated the precision/recall ratios and what figures (i.e., number of records (a) retrieved, (b) relevant, (c) not relevant) they obtained. As they stressed several times in their report, the recall figures they obtained were not absolute but comparative. For a more detailed account, see Part II of their article.

28. Saracevic and Kantor, "A Study of Information Seeking and Retrieving. Part II, 193.

29. Ibid.

30. Ray R. Larson, "The Decline of Subject Searching: Long Term Trends and Patterns of Index Use in an Online Catalog," Journal of American Society for Information Science 42 (April 1991): 198.

31. Charles W. Simpson, "OPAC Transaction Log Analysis: The First Decade," in Advances in Library Automation and Networking, vol. 3, ed. Joe A. Hewitt (Greenwich, Conn.: JAI Press, 1989), 35-67.

32. J. Dickson, "Analysis of User Errors in Searching an Online Catalog," Cataloging & Classification Quarterly 4 (Spring 1984): 19-38; Thomas A. Peters, "When Smart People Fail: An Analysis of the Transaction Log of an Online Public Access Catalog," Journal of Academic Librarianship 15 (November 1989): 267-273; Rhonda N. Hunter, "Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library: A Transaction Log Analysis," RQ 30 (Spring 1991): 395-402; and Steven D. Zink, "Monitoring User Search Success through Transaction Log Analysis: the WolfPac Example," Reference Services Review 19 (1991): 49-56.

33. Martha Kirby and Naomi Miller, "MEDLINE Searching on Colleague: Reasons for Failure or Success of Untrained End Users," Medical Reference Services Quarterly 5 (1986): 17-34; and Cynthia J. Walker et al., "Problems Encountered by Clinical End Users of MEDLINE and GRATEFUL MED," Bulletin of the Medical Library Association 79 (January 1991): 67-69.

34. Hunter, "Successes and Failures," 401.

35. Stephen Walker and Micheline Hancock-Beaulieu, Okapi at City: An Evaluation Facility for Interactive Information Retrieval (London: The British Library, 1991), British Library Research Report 6056; and Ray R. Larson, "Classification Clustering, Probabilistic Information Retrieval and the Online Catalog," Library Quarterly 61 (April 1991): 133-173.

36. Stephen Walker and Richard M. Jones, Improving Subject Retrieval in Online Catalogues, 1: Stemming, Automatic Spelling Correction and Cross-Reference Tables (London: The British Library, 1987), 139, British Library Research Paper 24. See also: R. Jones, "Improving Okapi: Transaction Log Analysis of Failed Searches in an Online Catalogue," Vine no. 62 (1986): 3-13.

37. Larson, "The Decline of Subject Searching," 198.

38. Sharon Seymour, "Online Public Access Catalog User Studies: A Review of Research Methodologies, March 1986-November 1989," Library and Information Science Research 13 (1991): 97.

39. Micheline Hancock-Beaulieu, Stephen Robertson and Colin Neilson, "Evaluation of Online Catalogues: Eliciting Information from the User," Information Processing & Management 27 (1991): 532.

40. John C. Flanagan, "The Critical Incident Technique," Psychological Bulletin 51 (1954): 327.

41. Wilson, Starr-Schneidkraut and Cooper, Use of the Critical Incident Technique to Evaluate the Impact of MEDLINE, 2.

42. Sammy R. Alzofon and Noelle Van Pulis, "Patterns of Searching and Success Rates in an Online Public Access Catalog," College & Research Libraries 45 (March 1984): 110-115; Marcia J. Bates, "Subject Access in Online Catalogs: a Design Model," Journal of American Society for Information Science 37 (1986): 357-376; Christine L. Borgman, "Why are Online Catalogs Hard to Use? Lessons Learned from Information-Retrieval Studies," Journal of American Society for Information Science 37 (1986): 387-400; Pauline A. Cochrane and Karen Markey, "Catalog Use Studies Since the Introduction of Online Interactive Catalogs: Impact on Design for Subject Access," Library and Information Science Research 5 (1983): 337-363; Mary Noel Gouke and Sue Pease, "Title Searches in an Online Catalog and a Card Catalog: A Comparative Study of Patron Success in Two Libraries," Journal of Academic Librarianship 8 (July 1982): 137-143; Charles R. Hildreth, Intelligent Interfaces and Retrieval Methods for Subject Searching in Bibliographic Retrieval Systems (Washington, DC: Cataloging Distribution Service, Library of Congress, 1989); Beverly Janosky, Philip J. Smith, and Charles Hildreth, "Online Library Catalog Systems: An Analysis of User Errors," International Journal of Man-Machine Studies 25 (1986): 573-592; Neal N. Kaske, A Comprehensive Study of Online Public Access Catalogs: an Overview and Application of Findings (Dublin, OH: OCLC, 1983), OCLC Research Report # OCLC/OPR/RR-83- 4; Cheryl Kern-Simirenko, "OPAC User Logs: Implications for Bibliographic Instruction," Library Hi Tech 1 (1983): 27-35; Ray R. Larson, "Workload Characteristics and Computer System Utilization in Online Library Catalogs" (Ph.D. diss., University of California at Berkeley, 1986); Gary S. Lawrence, V. Graham, and H. Presley, "University of California Users Look at MELVYL: Results of a Survey of Users of the University of California Prototype Online Union Catalog," Advances in Library Administration 3 (1984): 85-208; Karen Markey, Subject Searching in Library Catalogs: Before and After the Introduction of Online Catalogs (Dublin, OH: OCLC, 1984); Karen Markey, "Users and the Online Catalog: Subject Access Problems," in The Impact of Online Catalogs, ed. J.R. Matthews. (New York: Neal-Schuman, 1986), 35-69; Joseph K. Matthews, A Study of Six Public Access Catalogs: a Final Report Submitted to the Council on Library Resources, Inc. (Grass Valley, CA: J. Matthews and Assoc., Inc., 1982); Joseph Matthews, Gary S. Lawrence, and Douglas Ferguson, eds., Using Online Catalogs: a Nationwide Survey. (New York: Neal-Schuman, 1983); and Chih Wang, "The Online Catalogue, Subject Access and User Reactions: A Review," Library Review 34 (1985): 143-152.

43. Examples of such studies are (in chronological order): Cyril W. Cleverdon, Report on the Testing and Analysis of an Investigation into the Comparative Efficiency of Indexing Systems (Cranfield, England: Aslib, 1962); Cleverdon, Mills and Keen, Factors Determining the Performance of Indexing Systems, Volume 1, Design; Cleverdon and Keen, Factors Determining the Performance of Indexing Systems, Volume 2, Test Results; F. W. Lancaster, Evaluation of the MEDLARS Demand Search Service. (Washington, DC: US Department of Health, Education and Welfare, 1968); F. W. Lancaster, "MEDLARS: Report on the Evaluation of Its Operating Efficiency," American Documentation 20 (1969): 119-142; Dickson, "Analysis of User Errors in Searching an Online Catalog"; Blair and Maron, "An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System"; Jones, "Improving Okapi: Transaction Log Analysis of Failed Searches in an Online Catalogue"; Karen Markey and Anh N. Demeyer, Dewey Decimal Classification Online Project: Evaluation of a Library Schedule and Index Integrated into the Subject Searching Capabilities of an Online Catalog (Dublin, OH: OCLC, 1986), Report Number: OCLC/OPR/RR-86-1; Kirby and Miller, "MEDLINE Searching on Colleague"; S. Walker and Jones, Improving Subject Retrieval in Online Catalogues; Wilson, Starr-Schneidkraut, and Cooper, Use of the Critical Incident Technique to Evaluate the Impact of MEDLINE; Simone Klugman, "Failures in Subject Retrieval," Cataloging & Classification Quarterly 10 (1989): 9- 35; Peters, "When Smart People Fail"; Ankeny, "Evaluating End- User Services: Success or Satisfaction"; Hunter, "Successes and Failures"; C. Walker et al., "Problems Encountered by Clinical End Users of MEDLINE and GRATEFUL MED"; and Zink, "Monitoring User Search Success through Transaction Log Analysis: the WolfPac Example."

44. Cleverdon, Report on the Testing and Analysis; Cleverdon, Mills and Keen, Factors Determining the Performance of Indexing Systems, Volume 1, Design; and Cleverdon and Keen, Factors Determining the Performance of Indexing Systems, Volume 2, Test Results.

45. Cleverdon, Report on the Testing and Analysis, 1.

46. Ibid., 8-9.

47. Ibid., 89. The design and findings of the Cranfield I experiment have been criticized by many authors. For example, see: Don R. Swanson, "The Evidence Underlying the Cranfield Results," Library Quarterly 35 (1965): 1-20. For a review of the Cranfield tests, see: Karen Sparck Jones, "The Cranfield Tests," in Information Retrieval Experiment, ed. Karen Sparck Jones (London: Butterworths, 1981), 256-284.

48. Ibid., 11.

49. Swanson, "The Evidence Underlying the Cranfield Results,"

50. This percentage was obtained by averaging the figures given in the fifth column of Table 3.1 of Cleverdon, Report on the Testing and Analysis, 22.

51. This summary is based on Cleverdon, Report on the Testing and Analysis, Chapter 5. The report also includes the complete summary of the analysis of search failures (Appendix 5A) and "some examples of the complete analysis of the individual documents" (Appendix 5B).

52. Ibid., 88.

53. Cleverdon, Mills, and Keen, Factors Determining the Performance of Indexing Systems, Volume 1, Design; and Cleverdon and Keen, Factors Determining the Performance of Indexing Systems, Volume 2, Test Results.

54. Cleverdon and Keen, Factors Determining the Performance of Indexing Systems, Volume 2, Test Results, i ("Summary"). For the detailed performance figures along with recall/precision graphs, see volume 2 of the full report.

55. Lancaster, Evaluation of the MEDLARS Demand Search Service.

56. Ibid., 16, 19.

57. Ibid., 19-20.

58. Lancaster, "MEDLARS: Report on the Evaluation of Its Operating Efficiency," 123.

59. Ibid., 127.

60. Ibid., 131.

61. Blair and Maron, "An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System."

62. Ibid., 290-291.

63. Ibid., 291-293.

64. Ibid., 293.

65. Ibid., 295.

66. Markey and Demeyer, Dewey Decimal Classification Online Project, 1.

67. Ibid., 109.

68. Ibid., 162.

69. Ibid., 144.

70. Ibid., 146.

71. Ibid., 149.

72. Ibid., 165, Table 42.

73. Ibid., 162.

74. Ibid., 166.

75. Ibid., 182.

76. Ibid.; especially, see Chapter 8, 173-291.

77. Hilchey and Hurych, "User Satisfaction or User Acceptance?"

78. Ankeny, "Evaluating End-User Services," 352-354.

79. Ibid., 354.

80. Ibid.

81. Kirby and Miller, "MEDLINE Searching on Colleague."

82. Ibid., 20.

83. Ibid.

84. Dickson, "Analysis of User Errors in Searching an Online Catalog," 26.

85. Jones, "Improving Okapi: Transaction Log Analysis of Failed Searches in an Online Catalogue."

86. Ibid., 7-8.

87. S. Walker and Jones, Improving Subject Retrieval in Online Catalogues, 117-119.

88. S. Walker and Hancock-Beaulieu, Okapi at City, 30. The authors also surveyed the users to find out if they were satisfied with their search results using a five-point satisfaction scale. Ninety-five out of a total of 120 users (or 80%) indicated that they were satisfied with the search outcome (they marked 4 or 5 on the scale), 19 users (or 16%) "had some reservations" (i.e., they marked 3 on the scale), and 6 users (or 4%) "were negative" (i.e., they marked 1 or 2). Ibid., 24-25.

89. Ibid., 31.

90. Peters, "When Smart People Fail."

91. Ibid., 270.

92. Hunter, "Successes and Failures."

93. C. Walker, et al., "Problems Encountered by Clinical End Users of MEDLINE and GRATEFUL MED."

94. Ibid., 68.

95. Zink, "Monitoring User Search Success."

96. Ibid., 51

97. The following studies should be exempted from this as their analyses were not based on zero-hit searches only: Jones, "Improving Okapi: Transaction Log Analysis of Failed Searches in an Online Catalogue"; S. Walker and Jones, Improving Subject Retrieval in Online Catalogues; and S. Walker and Hancock-Beaulieu, Okapi at City.

98. Lancaster, Evaluation of the MEDLARS Demand Search Service.

99. Wilson, Starr-Schneidkraut and Cooper, Use of the Critical Incident Technique to Evaluate the Impact of MEDLINE.

100. Ibid., 5.

101. Ibid., 81.

102. Ibid., 83-84.

103. Gouke and Pease, "Title Searches in an Online Catalog and a Card Catalog," 139.

104. Alzofon and Van Pulis, "Patterns of Searching and Success Rates in an Online Public Access Catalog," 113.

105. Janosky, Smith and Hildreth, "Online Library Catalog Systems: An Analysis of User Errors."

106. Ibid., 576.

107. Ibid., 591.

108. Hildreth, Intelligent Interfaces and Retrieval Methods for Subject Searching in Bibliographic Retrieval Systems, 69.

109. Bates, "Subject Access in Online Catalogs: a Design Model"; Borgman, "Why are Online Catalogs Hard to Use? Lessons Learned from Information-Retrieval Studies"; David R. Gerhan, "LCSH in vivo: Subject Searching Performance and Strategy in the OPAC Era," Journal of Academic Librarianship 15 (1989): 83-89; Klugman, "Failures in Subject Retrieval"; David Lewis, "Research on the Use of Online Catalogs and Its Implications for Library Practice," Journal of Academic Librarianship 13 (1987): 152-157; Karen Markey, "Users and the Online Catalog: Subject Access Problems," in The Impact of Online Catalogs, ed. J.R. Matthews. (New York: Neal-Schuman, 1986), 35-69; Wang, "The Online Catalogue, Subject Access and User Reactions: A Review."

110. Larson, "Between Scylla and Charybdis: Subject Searching in the Online Catalog," 181.

111. Larson, "The Decline of Subject Searching," 208.

112. University of California Users Look at MELVYL: Results of a Survey of Users of the University of California Prototype Online Union Catalog. (Berkeley, CA: The University of California, 1983), 97.

113. Larson, "Classification Clustering, Probabilistic Information Retrieval and the Online Catalog," 136-144

114. Allyson Carlyle, "Matching LCSH and User Vocabulary in the Library Catalog," Cataloging & Classification Quarterly 10 (1989): 37.

115. Noelle Van Pulis, and L.E. Ludy, "Subject Searching in an Online Catalog with Authority Control," College & Research Libraries 49 (1988): 528-529.

116. Diane Vizine-Goetz and Karen Markey Drabenstott, "Computer and Manual Analysis of Subject Terms entered by Online Catalog Users," in ASIS '91: Proceedings of the 54th ASIS Annual Meeting. Washington, DC, October 27-31, 1991, ed. Jose-Marie Griffiths (Medford, NJ: Learned Information, 1991), 157.

117. Ibid., 161.

118. Michael K. Buckland and Doris Florian, "Expertise, Task Complexity, and Artificial Intelligence: A Conceptual Framework," Journal of American Society for Information Science 42 (October 1991): 635-643.