Evaluation of a Language Identification System for Mono- and Multi-lingual Text Documents

Artemenko, Olga and Mandl, Thomas and Shramko, Margaryta and Womser-Hacker, Christa Evaluation of a Language Identification System for Mono- and Multi-lingual Text Documents., 2006 . In 2006 ACM SAC Symposium on Applied Computing (SAC). Document Engineering Track (DE), Dijon, France, April, 23-27, 2006. (Unpublished) [Presentation]

Preview

PDF sac2006Poster.PDF Download (398kB) | Preview

English abstract

Language identification an important task for web information retrieval. This paper presents the implementation of a tool for language identification in mono- and multilingual documents. The tool implements four algorithms for language identification. Furthermore, we present a n-gram approach for the identification of languages in multi-lingual documents. An evaluation for monolingual texts of varied length is presented. Results for eight languages including Ukrainian and Russian are shown. It could be shown that n-gram-based approaches outperform word-based algorithms for short texts. For longer texts, the performance is comparable. The evaluation for multilingual documents is based on both short synthetic documents and real world web documents. Our tool is able to recognize the languages present as well as the location of the language change with reasonable accuracy.

Item type:	Presentation
Keywords:	language identification
Subjects:	L. Information technology and library technology > LL. Automated language processing. L. Information technology and library technology > LM. Automatic text retrieval. L. Information technology and library technology > LS. Search engines.
Depositing user:	Thomas Mandl
Date deposited:	28 Aug 2006
Last modified:	02 Oct 2014 12:04
URI:	http://hdl.handle.net/10760/8034

Downloads

Actions (login required)

View Item