The Retrieval Effectiveness of Web Search Engines: Considering Results Descriptions

Lewandowski, Dirk The Retrieval Effectiveness of Web Search Engines: Considering Results Descriptions. Journal of Documentation, 2008, vol. 64. (In Press) [Journal article (Unpaginated)]

[thumbnail of JDoc2008_Preprint.pdf]
Preview
 PDF
JDoc2008_Preprint.pdf
Download (1MB) | Preview

English abstract

Purpose: To compare five major Web search engines (Google, Yahoo, MSN, Ask.com, and Seekport) for their retrieval effectiveness, taking into account not only the results but also the results descriptions. Design/Methodology/Approach: The study uses real-life queries. Results are made anonymous and are randomised. Results are judged by the persons posing the original queries. Findings: The two major search engines, Google and Yahoo, perform best, and there are no significant differences between them. Google delivers significantly more relevant result descriptions than any other search engine. This could be one reason for users perceiving this engine as superior. Research Limitations: The study is based on a user model where the user takes into account a certain amount of results rather systematically. This may not be the case in real life. Practical Implications: Implies that search engines should focus on relevant descriptions. Searchers are advised to use other search engines in addition to Google. Originality/Value: This is the first major study comparing results and descriptions systematically and proposes new retrieval measures to take into account results descriptions. Article type: Research paper

Item type: Journal article (Unpaginated)
Keywords: Word Wide Web / search engines / retrieval effectiveness / results descriptions / retrieval measures
Subjects: L. Information technology and library technology > LS. Search engines.
Depositing user: Dirk Lewandowski
Date deposited: 11 Mar 2008
Last modified: 02 Oct 2014 12:11
URI: http://hdl.handle.net/10760/11258

References

Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, Vol. 54, No. 10, 913-925.

Broder, A. (2002). A taxonomy of web search. SIGIR Forum, Vol. 36, No. 2, 3-10.

Buckley, C. & Voorhees, E.M. (2000). Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, Athens, Greece, ACM Press, New York, 33-40.

Burns, E. (2007). U.S. Search Engine Rankings, April 2007. Search Engine Watch. http://searchenginewatch.com/showPage.html?page=3626021

Chu, H. & Rosenthal, M. (1996). Search Engines for the World Wide Web: A Comparative Study and Evaluation Methodology. Proceedings of the 59th ASIS Annual Meeting. http://www.asis.org/annual-96/ElectronicProceedings/chu.html

Crystal, A. & Greenberg, J. (2006). Relevance Criteria Identified by Health Information Users During Web Searches. Journal of the American Society for Information Science and Technology, Vol. 57, No. 10, 1368-1382.

Della Mea, V. & Mizzaro, S. (2004). Measuring Retrieval Effectiveness: A New Proposal and a First Experimental Validation. Journal of the American Society for Information Science and Technology, Vol. 55, No. 6, 530-543.

Ding, W. & Marchionini, G. (1996). A comparative study of web search service performance. Proceedings of the 59th ASIS Annual Meeting, 136-142.

Dresel, R., Hörnig, D., Kaluza, H., Peter, A., Roßmann, N. & Sieber, W. (2001). Evaluation deutscher Web-Suchwerkzeuge. Nachrichten für Dokumentation, Vol. 52, No. 7, 381-392.

Gordon, M. & Pathak, P. (1999). Finding information on the World Wide Web: the retrieval effectiveness of search engines. Information Processing & Management, Vol. 35, No. 2, 141-180.

Granka, L.A., Joachims, T., & Gay, G. (2004). Eye-tracking analysis of user behavior in WWW search. Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 478-479.

Griesbaum, J. (2004). Evaluation of three German search engines: Altavista.de, Google.de and Lycos.de. Information Research, Vol. 9, No. 4. http://informationr.net/ir/9-4/paper189.html

Griesbaum, J., Rittberger, M. & Bekavac, B. (2002). Deutsche Suchmaschinen im Vergleich: AltaVista.de, Fireball.de und Lycos.de. Procedings of the 8. Internationales Symposium für Informationswissenschaft, 201-223.

Guggenheim, E. & Bar-Ilan, J. (2005). Tauglichkeit von Suchmaschinen für deutschsprachige Anfragen. Information Wissenschaft und Praxis, Vol. 56, No. 1, 35-40.

Gulli, A. & Signorini, A. (2005). The indexable Web is more than 11.5 billion pages. Proceedings of the 14th International Conference on World Wide Web, Chiba, Japan, 902-903.

Hawking, D. & Craswell, N. (2005). The Very Large Collection and Web Tracks. In TREC: Experiment and Evaluation in Information Retrieval (S. 199-231). Cambridge, Mass.: MIT Press.

Hawking, D., Craswell, N., Bailey, P. & Griffiths, K. (2001). Measuring Search Engine Quality. Information Retrieval, Vol. 4, No. 1, 33-59.

Jacsó, P. (2005). Options for presenting search results. Online Information Review, Vol. 29, No. 3, 311-319.

Järvelin, K. & Kekäiläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, Vol. 20, No. 4, 422-446.

Lazarinis, F. (2007). Web retrieval systems and the Greek language: do they have an understanding? Journal of Information Science, in press.

Leighton, H.V. & Srivastava, J. (1999). First 20 Precision among World Wide Web Search Services (Search Engines). Journal of the American Society for Information Science, Vol. 50, No. 10, 870-881.

Lewandowski, D. (2006). Query types and search topics of German Web search engine users. Information Services & Use, Vol. 26, No. 4, 261-269.

Lewandowski, D. (2007a). Mit welchen Kennzahlen lässt sich die Qualität von Suchmaschinen messen? In Die Macht der Suchmaschinen / The Power of Search Engines (S. 243-258). Köln: von Halem.

Lewandowski, D. (2007b). Trefferpräsentation in Web-Suchmaschinen. Proceedings of Information in Wissenschaft, Bildung und Wirtschaft; 29. Online-Tagung der DGI 2007, DGI, Frankfurt am Main, in press.

Lewandowski, D. & Höchstötter, N. (2007). Web Searching: A Quality Measurement Perspective. In Web Searching: Interdisciplinary Perspectives (in press). Dordrecht: Springer.

Lewandowski, D., Wahlig, H. & Meyer-Bautor, G. (2006). The Freshness of Web search engine databases. Journal of Information Science, Vol. 32, No. 2, 133-150.

Machill, M., Neuberger, C., Schweiger, W. & Wirth, W. (2003). Wegweiser im Netz: Qualität und Nutzung von Suchmaschinen. In Wegweiser im Netz (S. 17-490). Gütersloh: Bertelsmann Stiftung.

Mizzaro, S. (1997). Relevance: The Whole History. Journal of the American Society for Information Science, Vol. 48, No. 9, 810-832.

Rose, D.E. & Levinson, D. (2004). Understanding user goals in Web search. Proceedings of the thirteenth International World Wide Web Conference, ACM Press, New York, 13-19.

Schmidt-Maenz, N. & Koch, M. (2006). A General Classification of (Search) Queries and Terms. Proceedings of the 3rd International Conference on Information Technologies: Next Generations, Las Vegas, Nevada, USA, 375-381.

Spink, A., Wolfram, D., Jansen, M. & Saracevic, T. (2001). Searching the Web: The public and their queries. Journal of the American Society for Information Science and Technology, Vol. 53, No. 2, 226-234.

Stock, M. & Stock, W.G. (2000). Internet-Suchwerkzeuge im Vergleich, Teil 1: Retrievaltest mit Known Item Searches. Password, Vol. 15, No. 11, 23-31.

Sullivan, D. (2006). Hitwise Search Engine Ratings. Search Engine Watch. http://searchenginewatch.com/showPage.html?page=3099931

Sullivan, D. (2007). Major Search Engines and Directories. Search Engine Watch. http://searchenginewatch.com/showPage.html?page=2156221

Tague-Sucliffe, J. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing & Management, Vol. 28, No. 4, 467-490.

Vaughan, L. (2004). New Measurements for Search Engine Evaluation Proposed and Tested. Information Processing & Management, Vol. 40, No. 4, 677-691.

Vaughan, L. & Thelwall, M. (2004). Search Engine Coverage Bias: Evidence and Possible Causes. Information Processing & Management, Vol. 40, No. 4, 693-707.

Véronis, J. (2006). A comparative study of six search engines. http://www.up.univ-mrs.fr/veronis/pdf/2006-comparative-study.pdf

Webhits Web-Barometer (2006). http://www.webhits.de/deutsch/index.shtml?webstats.html

Wolff, C. (2000). Vergleichende Evaluierung von Such- und Metasuchmaschinen. Proceedings of 7. Internationales Symposium für Informationswissenschaft, Darmstadt, Germany, Universitätsverlag, Konstanz, 31-48.


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item
Contact us
Send email Sponsors
Sponsor: AIMS Sponsor: D&P Sponsor: UniNa CAB Get connected
Facebook Facebook Twitter Twitter Linkedin LinkedIn RSS RSS