A Framework for Evaluating the Retrieval Effectiveness of Search Engines

Lewandowski, Dirk A Framework for Evaluating the Retrieval Effectiveness of Search Engines., 2012 [Preprint]

[img]
Preview
PDF
Evaluating_Next_Gen_Search_Engines_preprint.pdf

Download (1MB) | Preview

English abstract

This chapter presents a theoretical framework for evaluating next generation search engines. We focuson search engines whose results presentation is enriched with additional information and does notmerely present the usual list of “10 blue links”, that is, of ten links to results, accompanied by a shortdescription. While Web search is used as an example here, the framework can easily be applied tosearch engines in any other area.The framework not only addresses the results presentation, but also takes into account an extension ofthe general design of retrieval effectiveness tests. The chapter examines the ways in which this designmight influence the results of such studies and how a reliable test is best designed.

Item type: Preprint
Keywords: theoretical framework, evaluating, search engines, retrieval effectiveness, framework, information retrieval systems, retrieval effectiveness tests
Subjects: L. Information technology and library technology
L. Information technology and library technology > LC. Internet, including WWW.
L. Information technology and library technology > LS. Search engines.
Depositing user: Dirk Lewandowski
Date deposited: 30 Jun 2012
Last modified: 02 Oct 2014 12:22
URI: http://hdl.handle.net/10760/17244

References

Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10), 913-925.

Broder, A. (2002). A taxonomy of web search. SIGIR Forum, 36(2), 3-10.

Buckley, C., & Voorhees, E. M. (2000). Evaluating evaluation measure stability. SIGIR Forum (ACM Special Interest Group on Information Retrieval), 33-40.

Calderon-Benavides, L., Gonzalez-Caro, C., & Baeza-Yates, R. (2010). Towards a Deeper Understanding of the User's Query Intent. In B. Croft, M. Bendersky, H. Li, & G. Xu (Eds.): SIGIR 2010 Workshop on Query Representation and Understanding. (pp. 21-24). Geneva, Switzerland.

Chu, H., & Rosenthal, M. (1996). Search Engines for the World Wide Web: A Comparative Study and Evaluation Methodology, ASIS Annual Meeting.

Cutrell, E., & Guan, Z. (2007). Eye tracking in MSN Search: Investigating snippet length, target position and task types. Retrieved October 19, 2010, from http://research.microsoft.com/pubs/70395/tr-2007-01.pdf

Demartini, G., & Mizzaro, S. (2006). A Classification of IR Effectiveness Metrics. In M. Lalmas (Ed.), European Conference on IR Research (pp. 488-491). London, UK: Springer.

Ding, W., & Marchionini, G. (1996). A comparative study of web search service performance, Proceedings of the 59th American Society for Information Science Annual Meeting (pp. 136-142): Learned Information.

Dresel, R., Hörnig, D., Kaluza, H., Peter, A., Roßmann, N., & Sieber, W. (2001). Evaluation deutscher Web-Suchwerkzeuge. Nachrichten für Dokumentation, 52(7), 381-392.

Gordon, M., & Pathak, P. (1999). Finding information on the World Wide Web: the retrieval effectiveness of search engines. Information Processing & Management, 35(2), 141-180.

Granka, L., Hembrooke, H., & Gay, G. (2005). Location location location: Viewing patterns on WWW pages. Paper presented at the Eye Tracking Research and Applications Symposium (ETRA).

Granka, L. A., Joachims, T., & Gay, G. (2004). Eye-tracking analysis of user behavior in WWW search. Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 478-479.

Greisdorf, H., & Spink, A. (2001). Median measure: an approach to IR systems evaluation. Information Processing & Management, 37(6), 843-857.

Griesbaum, J. (2004). Evaluation of three German search engines: Altavista.de, Google.de and Lycos.de. Information Research, 9(4).

Griesbaum, J., Rittberger, M., & Bekavac, B. (2002). Deutsche Suchmaschinen im Vergleich: AltaVista.de, Fireball.de, Google.de und Lycos.de In R. Hammwöhner, C. Wolff, & C.

Womser-Hacker (Eds.), Paper presented at the Information und Mobilität. Optimierung und Vermeidung von Mobilität durch Information. 8. Internationales Symposium für Informationswissenschaft, (pp. 201-223). UVK.

Gulli, A., & Signorini, A. (2005). The indexable Web is more than 11.5 billion pages. In: Special Interest Tracks and Posters of the 14th International Conference on World Wide Web, (pp. 902-903). Chiba, Japan.

Harman, D. K., & Voorhees, E. M. (2006). TREC: An Overview. In B. Cronin (Ed.), Annual Review of Information Science and Technology: Vol. 40. Medford, New Jersey: Information Today.

Hawking, D., & Craswell, N. (2005). The Very Large Collection and Web Tracks. In E. M. Voorhees & D. K. Harman (Eds.), TREC Experiment and Evaluation in Information Retrieval (pp. 199-231). Cambridge, Massachusetts: MIT Press.

Hawking, D., Craswell, N., Bailey, P., & Griffiths, K. (2001). Measuring Search Engine Quality. Information Retrieval, 4(1), 33-59.

Hennies, M., & Dressler, J. (2006). Clients Information Seeking Behaviour: An OPAC Transaction Log Analysis, CLICK06, ALIA 2006 Biennial Conference.

Hjorland, B. (2010). The Foundation of the Concept of Relevance. Journal of the American Society for Information Science and Technology, 61(2), 217-237.

Hotchkiss, G. (2007). Search In The Year 2010. Retrieved October 19, 2010, from http://searchengineland.com/search-in-the-year-2010-11917

Hotchkiss, G., Garrison, M., & Jensen, S. (2004). Search Engine Usage in North America, A Research Initiative by Enquiro. Retrieved March 16, 2006, from www.enquiro.com

Huffman, S. B., & Hochster, M. (2007). How well does Result Relevance Predict Session Satisfaction? In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 567-573).

Höchstötter, N., & Koch, M. (2008). Standard parameters for searching behaviour in search engines and their empirical evaluation. Journal of Information Science, 34(1), 45-65.

Höchstötter, N., & Lewandowski, D. (2009). What Users See – Structures in Search Engine Results Pages. Information Sciences, 179(12), 1796-1812.

Jansen, B. J., & Spink, A. (2006). How are we searching the World Wide Web? A comparison of nine search engine transaction logs. Information Processing & Management, 42(1), 248-263.

Joachims, T. (2002). Optimizing Search Engines using Clickthrough Data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 133-142).

Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005). In Accurately Interpreting Clickthrough Data as Implicit Feedback (pp. 154-161). Paper presented at the Conference on Research and Development in Information Retrieval, Salvador, Brazil.

Kang, I. H., & Kim, G. (2003). Query Type Classification for Web Document Retrieval. SIGIR Forum (ACM Special Interest Group on Information Retrieval) (SPEC. ISS.), 64-71.

Keane, M. T., O'Brien, M., & Smyth, B. (2008). Are people biased in their use of search engines? Communications of the ACM, 51(2), 49-52.

Korfhage, R. R. (1997). Information Storage and Retrieval. New York, NY: Wiley.

Leighton, H. V., & Srivastava, J. (1999). First 20 Precision among World Wide Web Search Services (Search Engines). Journal of the American Society for Information Science, 50(10), 870-881.

Lewandowski, D. (2004). Date-restricted queries in web search engines. Online Information Review, 28(6), 420-427.

Lewandowski, D. (2006). Query types and search topics of German Web search engine users. Information Services & Use, 26(4), 261-269.

Lewandowski, D. (2008a). A three-year study on the freshness of Web search engine databases. Journal of Information Science, 34(6), 817-831.

Lewandowski, D. (2008b). Problems with the use of Web search engines to find results in foreign languages. Online Information Review, 32(5), 668-672.

Lewandowski, D. (2008c). Search engine user behaviour: How can users be guided to quality content? Information Services & Use, 28(3-4), 261-268.

Lewandowski, D. (2008d). The Retrieval Effectiveness of Web Search Engines: Considering Results Descriptions. Journal of Documentation, 64(6), 915-937.

Lewandowski, D. (2011a). The retrieval effectiveness of search engines on navigational queries. ASLIB Proceedings, 63(4) (accepted for publication).

Lewandowski, D. (2011b). The Influence of Commercial Intent of Search Results on Their Perceived Relevance. Proceedings of the 2011 iConference (pp. 452-458). ACM, New York, NY, USA.

Lewandowski, D., & Höchstötter, N. (2008). Web Searching: A Quality Measurement Perspective. In A. Spink & M. Zimmer (Eds.), Web Search: Multidisciplinary Perspectives (pp. 309-340). Berlin, Heidelberg: Springer.

Lewandowski, D., Wahlig, H., & Meyer-Bautor, G. (2006). The Freshness of Web search engine databases. Journal of Information Science, 32(2), 133-150.

Lorigo, L., Haridasan, M., Brynjarsdóttir, H., Xia, L., Joachims, T., Gay, G., et al. (2008). Eye tracking and online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science and Technology, 59(7), 1041-1052.

MacCall, S. L., & Cleveland, A. D. (1999). A Relevance-based Quantitative Measure for Internet Information Retrieval Evaluation. In Proceedings of the American Society for Information Science Annual Meeting (pp. 763-768).

Machill, M., Neuberger, C., Schweiger, W., & Wirth, W. (2003). Wegweiser im Netz: Qualität und Nutzung von Suchmaschinen. In M. Machill, & C. Welp (Eds.), Wegweiser im Netz (pp. 13-490). Gütersloh: Bertelsmann Stiftung.

Machill, M., Neuberger, C., Schweiger, W., & Wirth, W. (2004). Navigating the Internet: A Study of German-Language Search Engines. European Journal of Communication, 19(3), 321-347.

Marchionini, G. (2006). Exploratory search: From finding to understanding. Communications of the Acm, 49(4), 41-46.

Mizzaro, S. (1997). Relevance: The Whole History. Journal of the American Society for Information Science, 48(9), 810-832.

Nicholson, S., Sierra, T., Eseryel, U. Y., Park, J.-H., Barkow, P., Pozo, E. J., et al. (2006). How Much of It Is Real? Analysis of Paid Placement in Web Search Engine Results. Journal of the American Society for Information Science and Technology, 57(4), 448-461.

Nielsen, J., & Tahir, M. (2002). Homepage Usability: 50 Websites Deconstructed. Indianapolis, IN: New Riders Publishing.

Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google we trust: Users' decisions on rank, position, and relevance. Journal of Computer-Mediated Communication, 12(3), 801-823.

Rowlands, I., Nicholas, D., Williams, P., Huntington, P., Fieldhouse, M., Gunter, B., et al. (2008). The Google generation: The information behaviour of the researcher of the future. Aslib Proceedings: New Information Perspectives, 60(4), 290-310.

Saracevic, T. (2007). Relevance : a review of the literature and a framework for thinking on the notion in Information Science. Part III : behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13), 2126-2144.

Schaer, P., Mayr, P., & Mutschke, P. (2010). Implications of Inter-Rater Agreement on a Student Information Retrieval Evaluation. Paper presented at the LWA 2010, Kassel, Germany.

Singer, G., Norbisrath, U., Vainikko, E., Kikkas, H., & Lewandowski, D., (2011). Search-Logger -- Tool Support for Exploratory Search Task Studies. Paper presented at the SAC 2011, TaiChung, Taiwan. ACM.

Spink, A., & Jansen, B. J. (2004). Web Search: Public Searching of the Web (Vol. 6). Dordrecht, Boston, London: Kluwer Academic Publishers.

Su, L. T. (1998). Value of Search Results as a Whole as the Best Single Measure of Information Retrieval Performance. Information Processing & Management, 34(5), 557-579.

Sullivan, D. (2003). Searching with Invisible Tabs. Search Engine Watch. Retrieved October 19, 2010, from http://searchenginewatch.com/3115131.

Tague-Sucliffe, J. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing & Management, 28(4), 467-490.

Vaughan, L. (2004). New Measurements for Search Engine Evaluation Proposed and Tested. Information Processing & Management, 40(4), 677-691.

Vaughan, L., & Thelwall, M. (2004). Search Engine Coverage Bias: Evidence and Possible Causes. Information Processing & Management, 40(4), 693-707.

Vaughan, L., & Zhang, Y. (2007). Equal representation by search engines? A comparison of websites across countries and domains. Journal of Computer-Mediated Communication, 12(3), article 7.

Véronis, J. (2006). A comparative study of six search engines. Retrieved October 19, 2010, from http://www.up.univ-mrs.fr/veronis/pdf/2006-comparative-study.pdfA


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item