How the Overlap Between the Search Results of Different Retrieval Systems Correlates with Document Relevance

Spoerri, Anselm How the Overlap Between the Search Results of Different Retrieval Systems Correlates with Document Relevance., 2005 . In 68th Annual Meeting of the American Society for Information Science and Technology (ASIST), Charlotte (US), 28 October - 2 November 2005. [Conference paper]

[img]
Preview
PDF
Spoerri_How.pdf

Download (661kB) | Preview

English abstract

This paper provides an analysis of the overlap between the search results of retrieval systems that participated in TREC 3, 6, 7 and 8 to provide empirical support for some of the key assumptions guiding the design of data fusion methods. It shows that the potential relevance of a document increases exponentially as the number of search methods retrieving it increases – called the Authority Effect. It also shows that documents higher up in ranked lists and found by more systems are more likely to be relevant – called the Ranking Effect. Both effects can help explain why major data fusion methods perform well.

Item type: Conference paper
Keywords: information relevance ; search services
Subjects: I. Information treatment for information services > IC. Index languages, processes and schemes.
Depositing user: Norm Medeiros
Date deposited: 10 Mar 2006
Last modified: 02 Oct 2014 12:02
URI: http://hdl.handle.net/10760/7006

References

Aslam, J. & Savell, R. (2003). On the Effectiveness of Evaluating Retrieval Systems in the Absence of Relevance Judgments. In Proceedings of the 26th Annual ACM Conference on Research and Development in Information Retrieval (SIGIR-2003).

Belkin, N. J., Kantor, P., Fox, E. A., & Shaw, J. A. (1995). Combining Evidence of Multiple Query Representations for Information Retrieval. Information Processing & Management, 31(3), 431-448.

Callan. J. (2000). Distributed information retrieval. In Croft W.B. (Ed.), Advances in Information Retrieval (pp. 127-150). Kluwer Academic Publishers.

Foltz, P. & Dumais, S. (1992). Personalized information delivery: An analysis of information-filtering methods. Communications of the ACM, 35(12), 51-60.

Fox, E. & Shaw, J. (1994). Combination of Multiple Searches. In 2nd Annual Text Retrieval Conference (TREC-2), Gaithersburg, MD: National Institute of Standards and Technology.

Lee, J. H. (1997). Analyses of Multiple Evidence Combination. In Proceedings of the 20th Intl. Conference. on Research and Development in Information Retrieval

(SIGIR’97) (pp. 267–276).

Saracevic, T. & Kantor, P. (1988). A study of information seeking and retrieving. III. Searchers, searches and overlap. Journal of the American Society for Information Science. 39(3) 197-216.

Spoerri, A. (2004). Visual Search Editor for Composing Meta Searches. In Proceedings of the 67th Annual Meeting of the American Society for Information Science and Technology (ASIST 2004).

Spoerri, A. (2005). Using the Structure of Overlap Between Search Results to Rank Retrieval Systems Without Relevance Judgments. Unpublished manuscript.

Soboroff, I., Nicholas, C. and Cahan, P. (2001) Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual ACM Conference on Research and Development in Information Retrieval (SIGIR 2001) 66-73.

Turtle, H., Croft, B. (1991). Evaluation of an inference network-based retrieval model. ACM Transactions on Information Systems.

Vogt, C. & Cottrell, G. (1998). Predicting the performance of linearly combined IR systems. In Proceedings of the 21st ACM SIGIR conference on Research and

Development in Information Retrieval (pp. 190–196).

Voorhees, E. & Harman, D. (1994) Overview of the Eighth Text REtrieval Conference (TREC-3). In The Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: U.S. Government Printing Office.

Voorhees, E. & Harman, D. (1997) Overview of the Eighth Text REtrieval Conference (TREC-6). In The Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: U.S. Government Printing Office.

Voorhees, E. & Harman, D. (1998) Overview of the Eighth Text REtrieval Conference (TREC-7). In The Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: U.S. Government Printing Office.

Voorhees, E. & Harman, D. (1999) Overview of the Eighth Text REtrieval Conference (TREC-8). In The Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: U.S. Government Printing Office.

Wu, S. & Crestani, F. (2003). Methods for Ranking Information Retrieval Systems Without Relevance Judgements. In Proceedings of ACM SAC’03 (pp. 811- 816).


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item