A framework for assessing the impact of units of scholarly communication based on OAI-PMH harvesting of usage information

Abstract

The wide-spread implementation of institutional repositories (IR), digital libraries, preprint services, and open access journals has dramatically changed the communication options that are available to scholars. At the same time, scholarship itself is becoming digital, thereby fundamentally extending the notion of a unit of scholarly communication beyond journal papers to include multimedia files, data sets, simulations, visualizations, etc. Meanwhile, the evaluation of scholarly performance remains bound to the use of citation data derived from a subset of all available communication channels (pre-selected journals), and an ever decreasing subset of all communicated units (journal papers). Clearly, there is a need for frameworks that allow measuring scholarly activity and its impact in the context of this new reality. We discuss the architecture of a system that is being developed at the Los Alamos National Laboratory that aims at determining impact and prestige rankings on the basis of aggregated usage data. This system relies on two key components. First, an architecture that allows to OAI-PMH harvest, and hence aggregate, usage logs from various scholarly communication venues. For interoperability, usage logs are expressed as XML documents that are compliant with the ContextObject of the OpenURL Standard. Second, a set of social network methods to determine impact and prestige from the temporal patterns detected in the aggregated usage data. The proposed solution can be deployed on top of any type of scholarly communication channel, and can take into account the use of scholarly communication units of all types. We discuss recent results which indicate that, when applied to articles and journals, the resulting impact rankings correlate significantly with the Institute for Scientific Information's Impact Factor, but highlight different aspects of publication status and can thus form the basis of a more comprehensive assessment of scholarly impact. We speculate on how an open, freely accessible system for the evaluation of science relying on widely aggregated usage data can be applied to a wider range of scholarly communication processes then is presently the case, and can ultimately liberate the scientific community from the limitations and distortions caused by the existing singular focus on proprietary, citation based science evaluation mechanisms.