Evaluación del rendimiento de los sistemas de búsqueda de respuestas de dominio general

Olvera-Lobo, María-Dolores and Gutiérrez-Artacho, Juncal Evaluación del rendimiento de los sistemas de búsqueda de respuestas de dominio general. Revista española de Documentación Científica, 2013, vol. 36, n. 2. [Journal article (Unpaginated)]

Evaluación del rendimiento de los sistemas de búsqueda de respuestas de dominio general.pdf - Published version
Available under License Creative Commons Attribution.

Download (848kB) | Preview

English abstract

Information overload is felt more strongly on the Web than elsewhere. Question-answering systems (QA systems) are considered as an alternative to traditional information retrieval systems, because they give correct and understandable answers rather than just offering a list of documents. Four answer search systems available online have been analyzed: START, QuALiM, SEMOTE, and TrueKnowledge. They were analyzed through a wide range of questions that prompted responses of definitions, facts, and closed lists pertaining to different thematic areas. The answers were analyzed using several specific measurements (MRR, TRR, FHS, MAP and precision). The results are encouraging and they show that these systems, although each one different, are potentially valid for precise information retrieval of diverse types and thematic areas.

Spanish abstract

Los sistemas de búsqueda de respuestas (SBR) son una alternativa a los tradicionales sistemas de recuperación de información tratando de ofrecer respuestas precisas y comprensibles a preguntas factuales, en lugar de presentar al usuario una lista de documentos relacionados con su búsqueda. Se ha evaluado la eficacia de cuatro SBR disponibles en la Web —QuALiM, SEMOTE, START, y TrueKnowledge—, mediante una amplia muestra de preguntas de definición, factuales y de lista, pertenecientes a distintos dominios temáticos. Se utilizó una colección de 500 preguntas cuyas respuestas fueron valoradas por los usuarios y, posteriormente, se aplicaron varias medidas para su evaluación (MRR, TRR, FHS, MAP y precisión). Se observa que START y TrueKnowledge presentan un nivel aceptable de respuestas correctas, precisas y en una secuencia bien ordenada. Los resultados obtenidos revelan el potencial de esta clase de herramientas en el ámbito del acceso y la recuperación de información de dominio general.

Item type: Journal article (Unpaginated)
Keywords: Question-Answering Systems; performance analysis; definitional questions; factoid question; list questions; evaluation; Recuperación de información; sistemas de búsqueda de respuestas; evaluación; medidas de evaluación; World Wide Web.
Subjects: L. Information technology and library technology
Depositing user: Maria Dolores/ M.D. Olvera Lobo
Date deposited: 23 Jul 2018 13:05
Last modified: 23 Jul 2018 13:05
URI: http://hdl.handle.net/10760/32882


"SEEK" links will first look for possible matches inside E-LIS and query Google Scholar if no results are found.

Abdou, S.; Savoy, J.; Ruch P. (2006) Dépister efficacement de l’information dans une banque documentaire: L’exemple de MEDLINE. En Actes du XXIVè-me Congrès INFORSID, 129-143.

Belkin, N.J.; Vickery, A. (1985). Interaction in Information Systems: A Review of Research from Document Retrieval to Knowledge-based systems (LIR Report No 35). Londres: The British Library.

Blair-Goldensohn, S.; McKeown, K.; Schlaikjer, A. H. (2004). Answering Definitional Questions: A Hybrid Approach. En: Maybury, M.T. (ed.). New Directions in Question Answering, Palo Alto: AAAI Press, 47-58.

Buckley, C.; Voorhees, E. M. (2000). Evaluating evaluation measure stability. SIGIR ‘00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 33-40.

Cleverdon, C. (1997). The Cranfield tests on index languages devices. En Sparck Jones, K. y Willett, P. (eds.), Readings in information retrieval. San Francisco: Morgan Kaufmann, 47-59.

Cui, H.; Kan, M. Y.; Cua, T. S.; Xiao, J. (2004). A Comparative Study on Sentence Retrieval for Definitional Question Answering. SIGIR Workshop on Information retrieval for Question Answering (IR4QA), Sheffield.

Crouch, D.; Saurí, R.; Fowler, A. (2005). AQUAINT Pilot Knowledge-Based Evaluation: Annotation Guidelines. Palo Alto Research Center.


Fukumoto, J.; Kato, T.; Masui, F. (2003). Question Answering Challenge (QAC-1) an evaluation of question answering tasks at the NTCIRWorkshop 3. Proceedings of AAAI Spring Symposium on New Directions in Question Answering, 122-133.

Green, B. F.; Wolf, A. K.; Chomsky, C.; Laughery, K. (1961). Baseball: An Automatic Question Answerer. En: Proceedings of the Western Joint Computer Conference, v.19, pp. 219–224.

Greenwood, M. A.; Saggion, H. (2004). A Pattern Based Approach to Answering Factoid, List and Definition Questions. Proceedings of the 7th RIAO Conference (RIAO 2004), 232–243.

Harman, D. K. (1998). Text retrieval conferences (TRECs): providing a test-bed for information retrieval systems. Bulletin of the American Society for Information Science, vol. 24 (4), 11-13.

Jackson, P.; Schilder, F. (2005). Natural Language Processing: Overview. En: Brown (ed.), Encyclopedia of Language & Linguistics, 2. Amsterdam, Elsevier Press, 503-518.

Kaisser, M. (2008). The QuALiM question answering demo: supplementing answers with paragraphs drawn from Wikipedia. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. Stroudsburg: Association for Computational Linguistics, 32–35.

Kangavari, M.R.; Ghandchi, S.; Golpour, M. (2008). A New Model for Question Answering Systems. World Academy of Science, Engineering and Technology, vol. 42, 506-513.

Katz, B.; Borchardt, G.; Felshin, S.; Shen, Y.; Zaccak, G. (2007). Answering English questions using foreign-language, semistructured sources. Proceedings of the First IEEE International Conference on Semantic Computing (ICSC 2007). Irvine: IEEE Computer Society, 439-445.

Kolomityets, O.; Moens, M. F. (2011). A Survey on Question Answering Technology from an Information Retrieval Perspective. Information Sciences (en prensa), 2011.

Olvera-Lobo, M. D.; Gutiérrez-Artacho, J. (2010). Question-Answering Systems as Efficient Sources of Terminological Information: Evaluation. Health Information and Library Journal, vol. 27 (4), 268-276.

Olvera-Lobo, M. D.; Gutiérrez-Artacho, J. (2011a). Evaluation of Open -vs. Restricted- Domain Question Answering Systems in the Biomedical Field. Journal of Information Science, vol. 37 (2), 152-162.

Olvera-Lobo, M. D.; Gutiérrez-Artacho, J. (2011b). Language resources used in multilingual Question Answering Systems. Online Information Review, vol. 35 (4), 543-557.

Olvera-Lobo, M. D.; Gutiérrez-Artacho, J. (2011c). Multilingual Question-Answering System in Biomedical Domain on the Web: An Evaluation. Multilingual and Multimodal Information Access Evaluation, Lecture Notes in Computer Science, vol. 6941, 83-88.

Pérez-Coutiño, M.; Solorio, T.; Montes y Gómez, M.; López López, A.; Villaseñor Pineda, L. (2004). The Use of Lexical Context in Question Answering for Spanish. Workshop of the Cross-Language Evaluation Forum (CLEF 2004), pp. 377-384


Peters, C. (2009). What Happened in CLEF 2009: Introduction to the Working Notes. Working Notes for the CLEF 2009 Workshop.


Radev, D. R.; Qi, H.; Wu, H.; Fan, W. (2001). Evaluating Web-based Question Answering Systems. Informe técnico, University of Michigan.

Rodrigo, A.; Pérez-Iglesias, J.; Peñas, A.; Garrido, G.; Araujo, L. (2010). A Question Answering System based on Information Retrieval and Validation, Notebook Papers/LABs/Workshops (CLEF 2010)


Salton, G.; McGill, J. (1983). Introduction to modern information retrieval. New York: McGraw-Hill.

Sultan, M. (2006). Multiple Choice Question Answering. Tesis doctoral. Sheffield: University of Sheffield.

Tsur, O. (2003). Definitional Question-Answering Using Trainable Text Classifiers. Tesis doctoral.

Ámsterdam: University of Amsterdam.

Voorhees, E. M. (1999). The TREC 8 Question Answering Track Report. Proceedings of the 8th Text Retrieval Conference.


Voorhees, E. M. (2002). Overview of the TREC 2002 Question Answering Track. Proceedings of the Eleventh Text Retrieval Conference.


Voorhees, E. M.; Tice, D. (1999). The TREC-8 question answering track evaluation. En: Voorhees, E. y Harman, D., Proceedings of the Eighth Text Retrieval Conference. Gaithersburg, MD: NIST Publicación Especial.


Warren, D. (1981). Efficient Processing of Interactive Relational Database Queries Expressed in Logic. Proceedings Seventh International Conference on Very Large Data Bases, v.7. Cannes, VLDB Endowment, v.7., pp. 272-283.

Weizenbaum, J. (1966). Eliza: A computer program for the study of natural language communication between man and machine. Communications of the ACM. v.9, n.1, pp.36-45.

Woods, W. A.; Kaplan, R. M.; Nash-Webber, B. (1972). The Lunar Sciences Natural Language Information System. En: BBN Final Report 2378. Cambridge: Bolt, Beranek and Newman.

Zweigenbaum, P. (2005). Question answering in biomedicine. Proceedings Workshop on Natural Language Processing for answering. Budapest: ACL, EACL 2003, 1-4.


Downloads per month over past year

Actions (login required)

View Item View Item