Citation ranking versus peer evaluation of senior faculty research performance: a case study of Kurdish Scholarship

Meho, Lokman I. and Sonnenwald, D. H. Citation ranking versus peer evaluation of senior faculty research performance: a case study of Kurdish Scholarship. Journal of the American Society for Information Science, 2000, vol. 51, n. 2, pp. 123-138. [Journal article (Paginated)]

[img]
Preview
 PDF
jasist-2000-lokman-sonnenwald.pdf
Download (171kB) | Preview

English abstract

The purpose of this study is to analyze the relationship between citation ranking and peer evaluation in assessing senior faculty research performance. Other studies typically derive their peer evaluation data directly from referees often in the form of ranking. This study uses two additional sources of peer evaluation data: citation content analysis and book review content analysis. Two main questions are investigated: (a) To what degree does citation ranking correlate with data from citation content analysis, book reviews, and peer ranking? (b) Is citation ranking a valid evaluative indicator of research performance of senior faculty members? Citation data, book reviews, and peer ranking were compiled and examined for faculty members specializing in Kurdish studies. Analysis shows that normalized citation ranking and citation content analysis data yield identical ranking results. Analysis also shows that normalized citation ranking and citation content analysis, book reviews, and peer ranking perform similarly (i.e., are highly correlated) for high-ranked and low-ranked senior scholars. Additional evaluation methods and measures that take into account the context and content of research appear to be needed to effectively evaluate senior scholars whose performance ranks relatively in the middle. Citation content analysis data did appear to give some specific and important insights into the quality of research of these middle performers, however, further analysis and research is needed to validate this finding. This study shows that citation ranking can provide a valid indicator for comparative evaluation of senior faculty research performance.

Item type: Journal article (Paginated)
Keywords: bibliometrics
Subjects: B. Information use and sociology of information > BB. Bibliometric methods
Depositing user: Diane Sonnenwald
Date deposited: 14 Aug 2006
Last modified: 02 Oct 2014 12:04
URI: http://hdl.handle.net/10760/7972

References

Allen, B. (1989). Propositional analysis: A tool for library and information science research. Library and Information Science Research, 11(3), 235–246.

Allen, B., & Reser, D. (1990). Content analysis in library and information science research. Library and Information Science Research, 12(3), 251–262.

Arts & Humanities Citation Index (1980 –1998). Philadelphia: Institute for Scientific Information.

Baird, L.M., & Oppenheim, C. (1994). Do citations matter? Journal of Information Science, 20(1), 2–15.

Berelson, B. (1952). Content analysis in communication research. Glencoe, IL: Free Press.

Bilhartz, T.D. (1984). In 500 words or less: Academic book reviewing in American history. The History Teacher, 17, 525–536.

Book Review Digest (1905–1998). New York, NY: The H.W. Wilson Company.

Book Review Index (1965–1998). Detroit, MI: Gale Research Co.

Books in Print (1948 –1998). New York, NY: R.R. Bowker Co.

Bornstein, R.F. (1991). Manuscript review in psychology: psychometrics, demand characteristics, and an alternative model. The Journal of Mind & Behavior, 12, 429–468.

Busha, C.H., & Harter, S.P. (1980). Research Methods in Librarianship: Techniques and Interpretation. New York, NY: Academic Press.

Casey, J.B. (1985). Assessment of Quality in Book Selection: An Evaluation of the Effectiveness of Opinions Rendered by Peer Reviews in American History Journals. PhD Thesis, Case Western Reserve University, Cleveland, OH.

Champion, D.J., & Morris, M.F. (1973). A content analysis of book reviews in the AJS, ASR, and Social Forces. American Sociological Review, 78, 1256–1265.

Chubin, D.E., & Hackett, E.J. (1990). Peerless science: Peer review and U.S. science policy. Albany, NY: State University of New York Press.

Clark, K.E. (1957). America’s psychologists: A survey of a growing profession. Washington, DC: American Psychological Association.

Cole, S., & Cole, J.R. (1967). Scientific output and recognition. American Sociological Review, 32, 377–390.

Cole, S., & Cole, J.R. (1968). Visibility and the structural bases of awareness of scientific research. American Sociological Review, 33, 397–413.

Cronin, B., & Overfelt, K. (1994). Citation-based auditing of academic performance. Journal of the American Society for Information Science, 45(2), 61–72.

Garfield, E. (1979). Citation indexing: Its theory and application in science, technology, and humanities. New York, NY: John Wiley & Sons, Inc.

Garfield, E. (1983). How to use citation analysis for faculty evaluations, and when is it relevant (Pts. 1 & 2). Current Contents, 44, 5–13; 45, 5–14.

Gilbert, G.N. (1978). Measuring the growth of science: A review of indicators of scientific growth. Scientometrics, 1, 9–34.

Glanzel, W. (1996). The needs for standards in bibliometric research and technology. Scientometrics, 35, 167–176.

Glenn, N.D. (1978). On the misuse of book reviews. Contemporary Sociology, 7, 254–255.

Hicks, C.E., Rush, J.E., & Strong, S.M. (1985). Content analysis. In E.D. Dym (Ed.), Subject and information analysis (pp. 57–109). New York, NY: Marcel Dekker.

Hirsch, W., Kulley, A.M., & Efron, R.T. (1974). The gatekeeping process in scientific communication: Norms, practices and content of book reviews in professional journals. Working Paper Number 81, Purdue University, Institute for the Study of Social Change, West Lafayette, IN. Historical Abstracts (1955–1998). Santa Barbara, CA: American Bibliographical Center of ABC-Clio.

Holsti, O.R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison-Wesley Pub. Co. Index Islamicus (1906 –1998). East Grinstead, West Sussex, UK: Bowker– Saur.

Ingram, H.M., & Mills, P.B. (1989). Reviewing the book reviews. PS: Political Science and Politics, 22, 627–634. International Political Science Abstracts (1951–1998). Oxford: Basil Blackwell; Paris: Presses Universitaires de France.

International Society Kurdistan (1968). ISK’s Kurdish bibliography. S. Van Rooy, & K. Tamboer (Eds.), Amsterdam.

Irvine, J., & Martin, B.R. (1983). Assessing basic research: The case of the Isaac-Newton telescope. Social Studies of Science, 13, 49–86.

King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of Information Science, 13, 261–276.

Koenig, M.E.D. (1982). Determinants of expert judgment of research performance. Scientometrics, 4, 361–378.

Koenig, M.E.D. (1983). Bibliometric indicators versus expert opinion in assessing research performance. Journal of the American Society for Information Science, 34, 136–145.

Kostoff, R.N. (1996). Performance measures for government-sponsored research: overview and background. Scientometrics, 36, 281–292.

Lawani, S.M., & Bayer, A.E. (1983). Validity of citation criteria for assessing the influence of scientific publications: new evidence with peer assessment. Journal of the American Society for Information Science, 34, 59–66.

Lindholm-Romantschuk, Y. (1998). Scholarly book reviewing in the social sciences and humanities: The flow of ideas within and among disciplines. Westport, CT: Greenwood Press.

Liu, M. (1993). The complexities of citation practice: A review of citation studies. Journal of Documentation, 49, 370–408.

MacRoberts, M.H., & MacRoberts, B.R. (1984). The negational references: Or the art of dissembling. Social Studies of Science, 14, 91–94.

MacRoberts, M.H., & MacRoberts, B.R. (1986). Quantitative measures of communication in science: A study of the formal level. Social Studies of Science, 16, 151–187.

MacRoberts, M.H., & MacRoberts, B.R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40, 342–349.

MacRoberts, M.H., & MacRoberts, B.R. (1996). Problems of citation analysis. Scientometrics, 36, 435–444.

Martin, B.R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36, 343–362.

Meho, L.I. (1997). The Kurds and Kurdistan: A selective and annotated bibliography. Westport, CT: Greenwood Press.

Meho, L.I., & Haas, S.W. (in preparation). The importance, use, and nonuse of government publications by social scientists studying stateless nations.

Moxley, J.M. (1992). Publish, don’t perish: The scholar’s guide to academic writing and publishing. Westport, CT: Greenwood Press.

Mulkay, M.J. (1974). Methodology in the sociology of science. Sociology of Science, 13, 107–119.

Myers, R.C. (1970). Journal citations and scientific eminence in contemporary psychology. American Psychologist, 25, 1041–1048.

Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Cherry Hill, NJ: Computer Horizons.

Narin, F., & Hamilton, K.S. (1996). Bibliometric performance measures. Scientometrics, 36, 293–310.

Natowitz, A., & Carlo, P.W. (1997). Evaluating review content for book selection: An analysis of American history reviews in Choice, American Historical Review, and Journal of American History. College & Research Libraries, 58, 323–336.

PAIS International (1915–1998). New York, NY: Public Affairs Information Service.

Peritz, B.C. (1983). A classification of citation roles for the social sciences and related fields. Scientometrics, 5, 303–312.

Political Science Abstracts (1975–1998). New York, NY: IFI/Plenum.

Riley, L.E., & Spreitzer, E.A. (1970). Book reviewing in the social sciences. The American Sociologist, 5, 358–363.

Schwartz, C.A. (1989). Book selection, collection development, and bounded rationality. College & Research Libraries, 50, 328–343.

Seglen, P.O. (1992). The skewness of science. Journal of the American Society for Information Science, 43, 628–638.

Seglen, P.O. (1998). Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica, 69, 224–229.

Smith, L.C. (1981). Citation analysis. Library Trends, 30, 83–106.

Snizek, W.E., & Furham, E.R. (1979). Some factors affecting the evaluative content of book reviews in sociology. The American Sociologist, 14, 108–14.

Social Sciences Citation Index (1972–1998). Philadelphia: Institute for Scientific Information.

Sociological Abstracts (1963–1998). San Diego: Sociological Abstracts, Inc.

Spiegel–Rosing, I. (1977). Science studies: Bibliometric and content analysis. Social Studies of Science, 7, 97–113.

Stieg Dalton, M. (1995). Refereeing of scholarly works for primary publishing. Annual Review of Information Science and Technology, 30, 213–250.

Taylor, W.R. (1967). The influence of professional status differences upon book reviewing in sociology. Ph.D. Thesis. Purdue University, West Lafayette, IN.

Thomas, P.R., & Watkins, D.S. (1998). Institutional research rankings via bibliometric analysis and direct peer review: a comparative case study with policy implications. Scientometrics, 41, 335–355.

van Raan, A.F.J. (1996). Advanced bibliometric methods as quantitative core of peer-review based evaluation and foresight exercises. Scientometrics, 36, 397–420.

van Raan, A.F.J. (1997). Scientometrics: State of the art. Scientometrics, 38, 205–218.

Virgo, J.A. (1977). A statistical procedure for evaluating the importance of scientific papers. Library Quarterly, 47(4), 415–430.

White, H.D. (1990). Author co-citation analysis: Overview and defense. In C.L. Borgman (Ed.), Scholarly communication and bibliometrics (pp. 84–106). Newbury Park, CA: Sage. WorldCat [computer file, 1200–1999]. Dublin, OH: OCLC.


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item
Contact us
Send email Sponsors
Sponsor: AIMS Sponsor: D&P Sponsor: UniNa CAB Get connected
Facebook Facebook Twitter Twitter Linkedin LinkedIn RSS RSS