Toward a Unified Retrieval Outcome Analysis Framework for Cross-Language Information Retrieval

Chen, Jiangping Toward a Unified Retrieval Outcome Analysis Framework for Cross-Language Information Retrieval., 2005 . In 68th Annual Meeting of the American Society for Information Science and Technology (ASIST), Charlotte (US), 28 October - 2 November 2005. [Conference paper]

[img]
Preview
 PDF
Chen_Toward.pdf
Download (469kB) | Preview

English abstract

This paper proposes a Retrieval Outcome Analysis Framework, or ROA Framework, to systematically evaluate retrieval performance of Cross-Language Information Retrieval systems. The ROA framework goes beyond TREC-type retrieval evaluation methodology by including procedures focusing on individual queries, especially difficult queries. The framework is comprised of four interrelated components: (1) Overall System Performance Evaluation, (2) Query Categorization, (3) Translation Analysis, and (4) Individual Query Analysis. An example of applying the framework is discussed in detail. The author believes the proposed framework would be especially useful for the development of real world Cross-Language Information Retrieval systems because the evaluation guided by the framework has the potential to discover causes behind poor retrieval performance.

Item type: Conference paper
Keywords: cross-language search and retrieval systems ; online language translators ; retrieval performance evaluation
Subjects: I. Information treatment for information services > IC. Index languages, processes and schemes.
Depositing user: Norm Medeiros
Date deposited: 08 Mar 2006
Last modified: 02 Oct 2014 12:02
URI: http://hdl.handle.net/10760/6865

References

Blair, D. C. (2002). Some thoughts on the reported results of TREC. Information Processing and management, 38(4), 445-451.

Chen, J. (2003). The construction, use, and evaluation of a lexical knowledge base for English-Chinese cross language information retrieval. PhD dissertation, Syracuse University.

Chen, J., Ge, H., Wu, Y., and Jiang, S. (2004). UNT at TREC 2004: question answering combining multiple evidences. TREC 2004 Conference Note Book, p. 695-702.

Conover, W. J. (1999). Practical Nonparametric Statistics. John Wiley and Sons, 3rd edition.

Dalrymple, P.W & Roderer, N. K. ( 1994), Database access systems. In Williams, M. E. (Ed.) Annual Review of Information Science and Technology, vol. 29, (pp. 137-178).

Diekema, A. (2003). Translation events in cross-language information retrieval: lexical ambiguity, lexical holes, vocabulary mismatch, and correct translations. PhD dissertation, Syracuse University.

Hu, X., Bandhakavi, S., and Zhai, C. (2003). Error analysis of difficult TREC topics. Proceedings of ACM SIGIR 2003 (poster).

Hull, D. (1993). Using statistical testing in the evaluation of retrieval experiments. Proceedings of the 16th ACM SIGIR, p. 329-338.

Liu, S., Sun, C., & Yu, C. (2004). UIC at TREC-2004: Robust Track. TREC 2004 Conference Note Book, p. 625 – 634.

Moldovan, D., Harabagiu, S., Clark, C., Bowden, M., Lehmann, J. & Williams, J. (2004). Experiments and Analysis of LCC's two QA Systems Over TREC 2004. TREC 2004 Conference Note Book, p. 21- 30.

Papineni, K., Roukos, S., Ward, T., & Zhu (2002). BLEU: a method for automatic evaluation of machine translation. Proceedings of 40th ACL Annual Conference, p.311-318. Available at: http://www.ldc.upenn.edu/acl/P/P02/P02-1040.pdf

Saracevic, T. (1995). Evaluation of evaluation in information retrieval. Proceedings of the 18th ACM SIGIR. p. 138-146.


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item
Contact us
Send email Sponsors
Sponsor: AIMS Sponsor: D&P Sponsor: UniNa CAB Get connected
Facebook Facebook Twitter Twitter Linkedin LinkedIn RSS RSS