Evaluation of controlled vocabularies by inter-indexer consistency

Soler-Monreal, Concha and Gil-Leiva, Isidoro Evaluation of controlled vocabularies by inter-indexer consistency. Information Research: An International Electronic Journal, 2011, vol. 16, n. 4. [Journal article (Paginated)]

Text (Text in English)
Evaluation of controlled vocabularies for INFORMATION RESEARCH julio 2010 DEFINITIVO.pdf - Published version
Available under License Creative Commons Attribution Non-commercial No Derivatives.

Download (149kB) | Preview

English abstract

Introduction. Several controlled vocabularies are used for indexing three journal articles to check if with a list of descriptors are achieved better or equals of consistency rates that with a standard thesaurus and augmented thesaurus. Method. A set of terminology of Library and Information Science was used to build a list of descriptors with equivalence relations (USE and UF), a standard thesaurus and a augmented thesaurus (all the descriptors have scope notes). Subsequently, three articles were indexed by selected indexers who had varying degrees of experience – on the one hand Library and Information Science students and on the other, professionals from various documentation centres. Hooper’s measure to find the consistency between pairs of novice indexers and experts has been applied. Analysis. Data were tabulated and analysed systematically according pairs of novice indexers and experts has been applied. Results. The tool with the best results is the list of descriptors (39.5% consistency), followed by the augmented thesaurus (29.8%) and, with an almost identical value, the standard thesaurus (27.5%). Conclusion. It is concluded that the list of descriptors in both groups returns better indexing consistency but we need more research.

Spanish abstract

Se usan varios vocabularios controlados en la indización de tres artículos de revista para conocer qué lenguaje de indización consigue los mejores índices de consistencia entre indizadores. Para ello, con un conjunto de terminología de Biblioteconomía y Documentación, se construyó un listado de descriptores, con relaciones de equivalencia, un tesauro y un tesauro en el que todos sus descriptores cuentan con notas de aplicación. Posteriormente se indizaron tres artículos por los indizadores seleccionados que contaban con distinto grado de experiencia, por un lado estudiantes de Biblioteconomía y Documentación, y por otro, profesionales de diferentes centros de documentación. Una vez aplicada la fórmula para hallar los índices de consistencia entre los pares de indizadores noveles y expertos se concluye que en ambos grupos el listado de descriptores obtiene mejores resultados de consistencia en la indización, seguido por el tesauro aumentado y por el tesauro.

Item type: Journal article (Paginated)
Keywords: controlled vocabularies evaluation, thesaurus evaluation, list of descriptors evaluation, indexing, indexing consistency, interindexer consistency
Subjects: I. Information treatment for information services > ID. Knowledge representation.
Depositing user: Isidoro Gil Leiva
Date deposited: 24 Apr 2018 09:22
Last modified: 24 Apr 2018 09:22
URI: http://hdl.handle.net/10760/32637


BERTRAND, A. y CELLIER, J.M. Psychological approach to indexing: effects of the operartor's expertise upon indexing behaviour. Journal of Information Science, 1995, vol. 21, nº 6, p. 459-472.

BETTS, R. y MARRABLE, D. Free text vs. controlled vocabulary – retrieval precision and recall over large databases. Online Information 91:Proceedings of the Fifteenth International Online Information Meeting, London, 10-12 December 1991, p. 153-165.

CLEVERDON, C.W. ASLIB Cranfield Research Project: report on the first stage o fan investigation into the comparative efficiency of indexing systems, sept. 1960. Texto disponible en http://hdl.handle.net/1826/1122 [Consultado: 02/12/2009].

CLEVERDON, C.W. Proposals for an investigation into the efficiency of various retrieval systems, 1956. Disponible en: http://hdl.handle.net/1826/1367 [Consultado: 02/12/2009].

CHEN, X. Indexing consistency between online catalogues. Berlín: Universidad

Humboldt, 2008, PhD.

FIDEL, R. Searchers' selection of search keys: 2. Controlled vocabulary or free-text searching. Journal of the American Society for Information Science, 1991, vol. 42, nº 7, p. 501-514.

FIDEL, R. Who needs controlled vocabulary? Special Libraries, 1992, vol. 83, nº 1, p. 1-9.

FUNK, M.E., REID. C.A. Indexing consistency in MEDLINE. Bulletin of the Medical Library Association, 1983, vol. 71, p. 176-183.

GIL LEIVA, I. La automatización de la indización. Propuesta teórico-metodologícia: Aplicación al área de Biblioteconomía y Documentación. Murcia: Universidad de Murcia, 1997 (PhD Thesis).

GIL LEIVA, I. Consistencia en la asignación de materias en las Bibliotecas Públicas del Estado. Boletín de la Asociación Andaluza de Bibliotecarios, 2001, nº 63, p. 69-86.

GIL LEIVA, I. Consistencia en la indización de documentos entre indizadores noveles. Anales de Documentación, 2002, vol. 5, p. 99-111. Retrieved 15 juin, 2010 from http://webs.um.es/isgil/

GIL LEIVA, I. Manual de indización: teoría y práctica. Gijón: Trea, 2008.


na indexaçao em bibliotecas universitárias brasilerias. TransInformação, 2008, vol. 20, nº 3, p. 233-253.

GIL URDICIAÍN, B. Evaluación del rendimiento de los tesauros españoles en sistemas de recuperación de información. Revista Española de Documentación Científica, 1998, vol. 21, nº 3, p. 286-302.

GIL URDICIAÍN, B. Manual de lenguajes documentales. Gijón: Trea, 2004.

GROSS, T. y TAYLOR, A.G. What have we got to lose? The effect of controlled

vocabulary on keyword searching results. College & Research Libraries, 2005, vol. 66, nº 3, p. 212-230.

HOOPER. R.S. Indexer consistency test: origin, mesaruments, results and utilization. Bethesda, MD: IBM Corp., 1965.

HENZLER, R.G. Free or controlled vocabularies: some statistical user-oriented evaluations of biomedical information systems. International Classification, 1978, vol. 5, nº 1, p.21-26.

HUDON, M. An assessment of the usefulness of standardized definitions in a thesaurus through interindexer terminological consistency measurements. University of Toronto, 1998 (PhD Thesis).

HUDON, M. The usefulness of standardized definitions in thesauri: An assessment through interindexer consistency measurements. In TOMS, E.G., CAMPBELL, D.G. y DUNN, J. (ed.). 26th Annual Conference of the Canadian-Association-for-Information-Science. 1998, p. 221-230. Retrieved 15 juin, 2010 from http://www.caisacsi. ca/proceedings/1998/Hudon_1998.pdf.

IIVONEN, M. The impact of the indexing environment on interindexer consistency. In FUGMANN, R. (ed.). Tools for Knowledge organization and the human interface, 1990, vol. 1, p. 259-266.

IIVONEN, M. y KIVIMAKI, K. Common entibies and missing properties: similarities and differences in the indexing of concepts. Knowledge Organization, 1998, vol. 25, nº 3, p. 90-102.

KISHIDA, K. et al. A comparative evaluation of thesauri: performance of 'presentation of a body of knowledge' [in Japanese]. Joho no kagaku to gijutsu, 1988, vol. 38, nº 10, p. 565-572.

LANCASTER, F.W. Evaluation of the MEDLARS demand search service. Washington,

D.C. National Library of Medicine, 1968.

LANCASTER, F.W. El control del vocabulario en la recuperación de la información. Valencia: Universidad de Valencia, 2002.

LARSEN, M.S. Indexering med eudised thesaurusen (Indexing with the EUDISED

thesaurus). Biblioteksarbedje, 1988, vol. 9, nº 23/24, p. 95-114.

LEININGER, K. Interindexer consistency in PsycINFO. Journal of Librarianship and Information Science, 2000, vol. 32, nº 1, p. 4-8.

LEONARD, LE. Inter-indexer consistency and retrieval effectiveness: measurement of relations. Champaign: University of Illinois, 1975 (PhD Thesis).

LEONARD, L.E. Inter-indexer consistency studes, 1954-1975: A review of the

literature and summary of study results. University of Illinois, 1977.

MARKEY, K. Interindexer consistency tests: A literature review and report of a test of consistency in indexing visual materials. Library and Information Science Research, 1984, vol. 6, nº. 2, p. 155-177.

MIDDLETON, M.R. A comparison of indexing consistency and coverage in the AEI, ERIC, and APAIS databases. Behavioral & Social Sciences Librarian, 1984, vol. 3, nº 4, p. 33-43.

NESHAT, N. y HORRI, A. A study of subject indexing consistency between the

National Library of Iran and Humanities Libraries in the Area of Iranian Studies. Cataloging & Classification Quarterly, 2006, vol. 43, nº 1, p. 67-76.

POZHARISKII, I.F. Opredelenie semanticheskoi sily informatsionnopoiskogo

tezaurusa (Determining the semantic strength of an information-retrieval thesaurus).

Nauchno- Tekhnicheskaya Informatsiya, Series, 1982, vol. 2, nº 6, p. 21-25.

RIBEIRO, F. Subject indexing and authority control in archives: the need for subject indexing in archives and for an indexing policy using controlled language. Journal of the Society of Archivists, 1996, vol. 17, nº 1, p. 27-54.

ROLLING, L. Indexing consistency, quality and efficiency. Information Processing and Management, 1981, vol. 17, nº 2, p. 69-76.

SAARTI, J. Consistency of subject indexing of novels by public library professionals and patrons. Journal of Documentation, 2002, vol. 58, nº 1, p. 49-65.

SIEVERT, M.E. y ANDREWS, M.J. Indexing consistency in Information Science

Abstracts. Journal of the American Society for Information Science, 1991, vol. 42, nº1, p. 1-6.

SLAMECKA, V., and JACOBY, J.J. Consistency of human indexing. In “The Coming

Age of Information Technology”. Bethesda, MD: Documentation Inc., 1965, p. 32-56.

SOLER MONREAL, M.C. Evaluación de vocabularios controlados en la indización de documentos mediante índices de consistencia entre indizadores. Universidad Politécnica de Valencia, 2009 (PhD Thesis).

TARR, D. & BORKO, H. Factors influencing inter- indexer consistency. IN Ingår i Zunde, Pranas, (eds.). In Proceedings of the 37th annual meeting of the American Society for Information Science, 13-17 October, Atlanta, Georgia. Washington, D.C.: Asis, 1974, p. 50-55.

TONTA, Y. A Study of indexing consistency between Library of Congress and British Library Catalogers. Library Resources and Technical Services, 1991, vol. 35, nº. 2, p. 177-185.

WEINBERG, B.H. y CUNNINGHAM, J.A. The relationship between term specificity

in MeSh and online postings in MEDLINE. Bulletin of the Medical Library Association, 1985, vol. 73, nº 4, p. 365-372.


Downloads per month over past year

Actions (login required)

View Item View Item