Discriminating Meta-Search: A Framework for Evaluation

Chignell, Mark H., Gwizdka, Jacek and Bodner, Richard C. Discriminating Meta-Search: A Framework for Evaluation. Information Processing & Management, 1999, vol. 35, n. 3, pp. 339-364. [Journal article (Paginated)]

[thumbnail of DiscriminatingMetaSearch-ChignellGwizdkaBodner_IPM_1999.pdf]
Preview
PDF
DiscriminatingMetaSearch-ChignellGwizdkaBodner_IPM_1999.pdf

Download (225kB) | Preview

English abstract

There was a proliferation of electronic information sources and search engines in the 1990s. Many of these information sources became available through the ubiquitous interface of the Web browser. Diverse information sources became accessible to information professionals and casual end users alike. Much of the information was also hyperlinked, so that information could be explored by browsing as well as searching. While vast amounts of information were now just a few keystrokes and mouseclicks away, as the choices multiplied, so did the complexity of choosing where and how to look for the electronic information. Much of the complexity in information exploration at the turn of the twenty-first century arose because there was no common cataloguing and control system across the various electronic information sources. In addition, the many search engines available differed widely in terms of their domain coverage, query methods and efficiency.Meta-search engines were developed to improve search performance by querying multiple search engines at once. In principle, meta-search engines could greatly simplify the search for electronic information by selecting a subset of first-level search engines and digital libraries to submit a query to based on the characteristics of the user, the query/topic, and the search strategy. This selection would be guided by diagnostic knowledge about which of the first-level search engines works best under what circumstances. Programmatic research is required to develop this diagnostic knowledge about first-level search engine performance. This paper introduces an evaluative framework for this type of research and illustrates its use in two experiments. The experimental results obtained are used to characterize some properties of leading search engines (as of 1998). Significant interactions were observed between search engine and two other factors (time of day and Web domain). These findings supplement those of earlier studies, providing preliminary information about the complex relationship between search engine functionality and performance in different contexts. While the specific results obtained represent a time-dependent snapshot of search engine performance in 1998, the evaluative framework proposed should be generally applicable in the future.

Item type: Journal article (Paginated)
Keywords: search engine evaluation, meta-search engines, web metrics, information retrieval, hypertext and hypermedia
Subjects: L. Information technology and library technology > LC. Internet, including WWW.
C. Users, literacy and reading. > CB. User studies.
Depositing user: Jacek Gwizdka
Date deposited: 14 Nov 2009
Last modified: 02 Oct 2014 12:15
URI: http://hdl.handle.net/10760/13579

References

Abrams, D., Baecker, R. M., & Chignell, M. H. (1998). Information archiving with bookmarks: personal web space construction and organization. Proceedings of the Human Factors in Computing Systems Conference, 41-48.

Berners-Lee‚ T., Cailliau, R., Luotonen‚ A., Nielsen‚ H., & Secret, A. (1994) The World-Wide Web. Communications of the ACM, 37(8), 76-82.

Card, S.K., Mackinlay, J., & Shneiderman, B. (1998) Readings in Information Visualization. Los Altos, CA: Morgan Kaufman.

Chu, H., & Rosenthal, M. (1996, October) Search engines for the world wide web: a comparative study and evaluation methodology. Proceedings of the Annual Conference for the American Society for Information Science, 127-135.

Derry, T. K., & Williams, T. I. (1960) A short history of technology from the earliest times to A.D. 1900. New York: Dover Publications, Inc.

Ding, W., & Marchionini, G. (1996, October) A comparative study of web search service performance. Proceedings of the Annual Conference for the American Society for Information Science, 136-142.

Dreilinger, D., & Howe, A. E. (1997, July) Experiences with selecting search engines using metasearch. ACM Transactions on Information Systems, 15(3), 195-222.

Fox, E., & Marchionini, G. (1998, April). Special issue on digital libraries. Communications of the ACM.

Gauch, S., Wang, G., and Gomez, M. (1996). Profusion: Intelligent fusion from multiple different search engines. Journal of Universal Computer Science. 2(9).

Golovchinsky, G., Chignell, M. H., & Charoenkitkarn, N. (1997). Formal experiments in casual attire: Case studies in information exploration. New Review of Hypermedia, 3, 123-157.

Green, D. M., & Swets, J. A. (1988) Signal Detection Theory and Psychophysics. New York: Wiley.

Gwizdka, J. (1998). Measuring Information Retrieval in Hypermedia Systems. Unpublished Manuscript. Toronto, Ontario, Canada : Department of Mechanical and Industrial Engineering, University of Toronto.

Harman, D. (1995). Overview of the third text retrieval conference (TREC-3). In D.K. Harman (Ed.) Proceedings of the Third Text Retrieval Conference (TREC-3) (pp.1-19). Gaithersburg, MD: National Institute of Standards and Technology.

Harter, S.P. (1996). Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47, 37-49.

Hearst, M. A. & Pedersen, J. O. (1996). Re-examining the cluster hypothesis: scatter/gather on retrieval results. Proceedings of ACM/SIGIR ’96.

Hou, M. (1998). Comparison for three internet search tools (Yahoo, Alta Vista, Lycos). Unpublished manuscript. Department of Mechanical and Industrial Engineering, University of Toronto.

Kenk, F. (1997, August) Benchmarking www search services: a semantic portfolio analysis of popular Internet www search services. [On-line] Available: http://home.t-online.de/home/gerhard.kenk/hpgka00e.htm.

Leighton, H.V. (1995) Performance of four world wide web (WWW) index services: Infoseek, Lycos, WebCrawler, and WWWWorm. [On-line] Available: http://www.winona.msus.edu/is-f/library-f/webind.htm.

Leighton, H. V., & Srivastava, J. (1997, August) Precision among world wide web search services (search engines): Alta Vista, Excite, Hotbot, infoseek, Lycos. Unpublished master’s thesis, Department of Computer Science, University of Minnesota.

Liu, J. (1996, September) Understanding www search engines. [On-line] Available: http://www.indiana.edu/~librcsd/search.

Lynch, C. (1997, March). Searching the Internet. Scientific American, 52-56.

Meadow, C.T. (1992) Text information retrieval systems. Toronto: Academic Press.

Meghabghab, D. B., & Meghabghab, G. V. (1996, May) Information retrieval in cyberspace. Proceedings of ASIS Mid-Year Meeting, 224-237.

Notess, G. R. (1996, November) Internet "onesearch" with the mega search engines. Online, 20(6).

Over, P. (1997). TREC-5 interactive track report. Proceedings of the Fifth Text Retrieval Conference (TREC-5). NIST Special Publication 500-238, 29-56.

Salton, G. (1989). Automatic Text Processing. Reading, Massachussets : Addison-Wesley.

Schlichting, C. & Nilsen, E. (1997) Signal detection analysis of www search engines. Proceedings of the Second Human Factors on the Web Conference. [On-line] Available: http://www.microsoft.com/usability/webconf/schlichting/schlichting.htm

Search Engine Watch. (1998). [On-line]Available: http://searchenginewatch.com/.

Selberg, E. and Etzioni, O. (1995). Multi-service search and comparison using the MetaCrawler. Proceedings of the 4th International World Wide Web Conference.

Siegel, S., & Castellan, N. J. (1988). Non-parametric statistics for the behavioural sciences (2nd ed.). New York: McGraw-Hill.

Steinberg, S. G. (1996, May) Seek and ye shall find (maybe). Wired, 4(5), 108-182.

Stobart, S., & Kerridge, S. (1996, November) WWW search engine study. School of Computing and Information Systems, University of Sunderland. [On-line report] Available: http://osiris.sund/ac.uk/sst/se/.

Su, L. T., Chen, H., and Dong, X. (1998, October) Evaluation of Web-based search engines from the end-user’s perspective: a pilot study. Proceedings of the Annual Conference for the American Society for Information Science, 348-361.

Van Rijsbergen, C.J. (1979) Information Retrieval. Butterworths, London, England

Zorn, P., Emanoil, M., Marshall, L., & Panek, M. (1996, May) Advanced searching: tricks of the trade. Online, 20(3).


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item