Uso de robots.txt y sitemaps en la administración pública española

Martín-Galán, Bonifacio, Hernández-Pérez, Tony, Rodríguez-Mateos, David and Peña-Gil, Daniel Uso de robots.txt y sitemaps en la administración pública española. El profesional de la información, 2009, vol. 18, n. 6, pp. 625-630. [Journal article (Paginated)]

Text 05.pdf - Published version Available under License Creative Commons Attribution Non-commercial Share Alike. Download (1MB)

English abstract

The use of robots.txt and sitemaps in the Spanish public administration. Robots.txt and sitemaps files are the main methods to regulate search engine crawler access to its content. This article explain the importance of such files and analyze robots.txt and sitemaps from more than 4,000 web sites belonging to Spanish public administration to determine the use of these files as a medium of optimization for crawlers.

Spanish abstract

Se explica la importancia que tienen los ficheros robots.txt y los sitemaps para los sitios web. Se realiza un estudio sobre más de 4.000 webs de la administración pública española para analizar el uso de ficheros robots.txt y sitemaps como medio de optimización para los crawlers o arañas de los motores de búsqueda.

Item type:	Journal article (Paginated)
Keywords:	Robots; Crawlers; Sitemaps; Motores de búsqueda; Recuperación de información; Visibilidad; Sitios web; Search engines; Information retrieval; Visibility; Web sites
Subjects:	H. Information sources, supports, channels. > HQ. Web pages. I. Information treatment for information services > IC. Index languages, processes and schemes. L. Information technology and library technology > LS. Search engines.
Depositing user:	Ana Ribaguda
Date deposited:	06 Feb 2016 10:14
Last modified:	06 Feb 2016 10:14
URI:	http://hdl.handle.net/10760/28919

References

Ajay, S.; Ekanayake, J. “Analysis of the usage statistics of robots exclusion standard”. En: Iadis Intl Conf WWW/Internet, 2006.

http://grids.ucs.indiana.edu/ptliupages/publications/IADISConferenceRobtoExclusion.pdf

Kolay, S.; D’Alberto, P.; Dasdan, A.; Bhattacharjee, A. “A larger scale study of robots.txt”. En: Intl Conf on World Wide Web, 2008, pp. 21-25.

Schonfeld, U.; Shivakumar, N. “Sitemaps: above and beyond the crawl of duty”. En: Intl Conf on World Wide Web, 2009, pp. 991-1000.

Sun, Y.; Councill, I. G.; Giles, C. L. “A large scale study of robots.txt”. En: Intl Conf on World Wide Web, 2007a, pp. 1123-1124.

Sun, Y.; Zhuang, Z.; Councill, I. G.; Giles, C. L. “Determining bias to search engines from robots.txt”. En: Proc of Intl Conf on Web, 2007b, pp. 149–155.

Thelwall, M.; Stuart, D. “Web crawling ethics revisited: cost, privacy, and denial of service”. Journal of the American Society for Information Science and Technology, 2005, v. 57, n. 13, pp. 1771–1779.

Wilde, E.; Roy, A. Web site metadata: UCB ISchool report 2009-028.

http://dret.net/netdret/publications#wil09b

Downloads

Actions (login required)

View Item