Open- vs. Restricted-Domain QA Systems in the Biomedical Field

Olvera-Lobo, María-Dolores and Gutiérrez-Artacho, Juncal Open- vs. Restricted-Domain QA Systems in the Biomedical Field. Journal of Information Science, 2011, n. 37, pp. 152-162. [Journal article (Paginated)]

Open- vs. Restricted-Domain QA Systems in the Biomedical Field.pdf

Download (362kB) | Preview

English abstract

Question Answering Systems (hereinafter QA systems) stand as a new alternative for Information Retrieval Systems. We conducted a study to evaluate the efficiency of QA systems as terminological sources for physicians, specialized translators, and users in general. To this end we analysed the performance of two open-domain and two restricted domain QA systems. The research entailed a collection of one hundred fifty definitional questions from WebMed. We studied the sources that QA systems used to retrieve the answers, and later applied a range of evaluation measures to mark the quality of answers. Through analysing the results obtained by asking the 150 questions in the QA systems MedQA, START, QuALiM and HONqa, it was possible to evaluate the systems’ operation through applying specific metrics. Despite the limitations demonstrated by these systems, as they are not accessible to everyone and they are not always completely developed, it has been confirmed that these four QA systems are valid and useful for obtaining definitional medical information in that they offer coherent and precise answers. The results are encouraging because they present this type of tool as a new possibility for gathering precise, reliable and specific information in a short period of time.

Item type: Journal article (Paginated)
Keywords: Question answering systems; evaluation; biomedical information; Open-Domain Question answering systems, Restricted-Domain Question answering systems.
Subjects: L. Information technology and library technology
Depositing user: Maria Dolores/ M.D. Olvera Lobo
Date deposited: 23 Jul 2018 16:12
Last modified: 23 Jul 2018 16:12


R. Baeza-Yates and B. Ribeiro-Nieto, Modern Information Retrieval (ACM Press, New York, 1999).

R. R. Korfhage, Information Storage and Retrieval (Wiley Computer Publishing, New York, 1997).

G. Salton, Automatic processing of foreign language documents, Journal of American Society for Information Sciences 21 (1970) 187–194.

G. Salton and M.J. Mc Gill, Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).

C.J. Rijsbergen, Information Retrieval, 2nd Edition (Butterworths, London, 1979).

C.T. Meadow, Text Information Retrieval Systems (Academic Press, San Diego, 1993).

M. Pérez-Coutiño, T. Solorio, M. Montes-y-Gómez, A. López-López and L. Villaseñor-Pineda, Toward a document model for question answering systems, Proceedings of the Second International Atlantic Web Intelligence Conference (2004)

D. Crouch, R. Saurí and A. Fowler, AQUAINT Pilot Knowledge-Based Evaluation: Annotation Guidelines (Palo Alto Research Center, Palo Alto, 2005). Available at: (accessed 25 October 2010).

M. Lee, J. Cimino, H.R. Zhu, C. Sable, V. Shanker, J. Ely and H. Yu, Beyond information retrieval – medical question answering, AMIA 2006 Annual Symposium Proceedings (2006) 469–473.

H. Yu, M. Lee, D. Kaufman, J. Ely, J.A. Osheroff, G. Hripcsak and J. Cimino, Development, implementation, and a cogni- tive evaluation of a definitional question answering system for physicians, Journal of Biomedicine Informatics 4 (2007) 236–251.

E.M. Voorhees, The TREC 8 question answering track report, Proceedings of the 8th Text REtrieval Conference (1999) 107–130.

S.B. Blair-Goldensohn, K.R. McKeow and A.H Schlaikjer, A hybrid approach for QA track definitional questions, Proceedings of TREC 2003 (Gaithersburg, MD, 2003) 336–343.

M.R. Kangavari, S. Ghandchi and M. Golpour, A new model for question answering systems, World Academy of Science, Engineering and Technology 42 (2008).

L.F. Costa and D. Santos, Question answering systems: a partial answer (SINTEF, Oslo, 2007).

M.D. Olvera-Lobo and J. Gutiérrez-Artacho, Question-answering systems as efficient sources of terminological information: evaluation, Health Information and Library Journal 27(4) (2010) 268–276.

H. Cui, M.Y. Kan, T.S. Chua and J. Xiao, A comparative study on sentence retrieval for definitional question answering, SIGIR Workshop on Information retrieval for Question Answering (Sheffield, 2004).

O. Tsur, Definitional question-answering using trainable text classifiers (Institute of Logic Language and Computation (ILLC), University of Amsterdam, 2003).

G.O. Sing, C. Ardil, W. Wong and S. Sahib, Response quality evaluation in heterogeneous question answering system: a black- box approach, Proceedings of World Academy of Science, Engineering and Technology, 9 (Lisbon, 2005).

E. Alfonseca, M. De Boni, J.L. Jara, S. Manandhar, A prototype question answering system using syntactic and semantic informa- tion for answer retrieval, Proceedings of the 10th Text Retrieval Conference (TREC-10) (Gaithersburg, MD, 2002)

P. Jacquemart and P. Zweigenbaum, Towards a medical question-answering system: a feasibility study, Proceedings of Medical Informatics Europe (MIE ‘03), Volume 95 of Studies in Health Technology and Informatics (IOS Press, San Palo, CA, 2003) 463–468.

P. Zweigenbaum, Question answering in biomedicine, Proceedings Workshop on Natural Language Processing for Question Answering, EACL 2003 (2005) 1–4.

B. Katz, Using English for indexing and retrieving. In: P.H. Winston and S.A. Shellard (eds), Artificial intelligence at MIT: expanding frontiers, volume 1 (MIT Press, Cambridge, MA, 1990).

M. Kaisser, Acquiring syntactic and semantic transformations in question answering, (PhD thesis, University of Edinburgh, Edinburgh, 2009).

M. Kaisser, The QuALiM question answering demo: supplementing answers with paragraphs drawn from Wikipedia, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session (Columbia, OH, 2008) 32–35.

I. Fahmi. Automatic term and relation extraction for medical question answering system (PhD thesis, University of Gronigen, Gronigen, Netherlands, 2009).

J.W. Ely, J. Osheroff, P.N. Gorman, M.H. Ebell, M.L. Chambliss, E.A. Pifer and P.Z. Stavri, A taxonomy of generic clinical questions: classification study, BMJ 321 (2000) 429–432.

S. Cruchet, A. Gaudinat, T. Rindflesch and C. Boyer, What about trust in the question answering world? AMIA 2009 Annual Symposium (San Francisco, CA, 2009).

S.B. Blair-Goldensohn, K.R. McKeow and A.H Schlaikjer, A hybrid approach for QA track definitional questions, Proceedings of TREC 2003 (Gaithersburg, MD, 2003) 336–343.

H. Yu and D. Kaufman, A cognitive evaluation of four online search engines for answering definitional questions posed by physi- cians, Pacific Symposium on Biocomputing 12 (2007) 328–339.

B. Katz, S. Felshin, D. Yuret, A. Ibrahim, J. Lin, G. Martion, A.J. McFarland and B. Temelkuran, Omnibase: uniform access to heterogeneous data for question answering, Proceedings of the 7th International Workshop on Applications of Natural Language to Information Systems (Estocolmo, NLDB, 2002) 230−234.

J.R. Taylor, An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements, (University Science Books, 1999), 128–129.

T. Saracevic, Evaluation of evaluation in information retrieval, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Special issue of SIGIR Forum (1995) 138–146.

P. Buitelaar, P. Cimiano, P. Frank, M. Hartung and S. Racioppa, Ontology-based information extraction and integration from heterogeneous data sources, International Journal of Human–Computer Studies 66 (2008) 759−788.

M.T. Maybury, New Directions in Question Answering (AAAI/MIT Press, Cambridge, MA, 2004)


Downloads per month over past year

Actions (login required)

View Item View Item