"epub@mimas"

Early Integrators and the Passive Majority: An evaluation study of a large web-based bibliographic reference database

Ken Eason ¹, Ross MacIntyre², Ann Apps² and Martin Ashby^3
1The Bayswater Insitute, London, UK
²Manchester Computing, University of Manchester, UK
³Department of Human Sciences, Loughborough University, UK

Publication information.

Abstract

An evaluation is reported of user responses to zetoc, a service that provides access to the British Library electronic table of contents database and an email alerting service. Two questionnaire surveys and an interview programme gave data on over 800 users. The results show that about 17% are active users who are achieving a personal information management routine by integrating zetoc with other services (the early integrators). The other 83%, the `passive majority', make limited use of zetoc and do not achieve integration with other services. The interviews reveal that some of this group have a stable but restricted usage routine that limits information overload. The usage by others is less controlled and they make little use of the information they receive. The paper examines the impact of enhancements to the zetoc service to support integration and it discusses the barriers that prevent many users exploiting the potential of services available to them.

1. The zetoc Service

In September 2000, the British Library, in partnership with the University of Manchester, made its Electronic Table of Contents data available to UK Higher and Further Education institutions. The Electronic Table of Contents database lists the details of approximately 20 million journal articles and conference papers. The British Library contracted Manchester Computing to develop and mount a Z39.50-compliant version of the ETOC database, a service now live entitled zetoc (http://zetoc.mimas.ac.uk). The service is free to UK institutions supported by the Joint Information Systems Committee (JISC) of the UK Higher Education Funding Councils and by subscription to UK Research Councils, Irish higher education institutions and, since 2001, to National Health Service staff in England and Scotland.

zetoc aims to provide researchers with a means to find and access published research material to aid in the furtherance of their own research, thus assisting in the advancement of knowledge. Within an internet cross-referencing paradigm of `discover - locate - request - deliver', the initial zetoc service provided discovery of research articles in a timely fashion. Enhancements to the zetoc service have provided `request and deliver' through: direct linking, where it can be ascertained an article is available electronically; document supply directly from the British Library and indirectly through traditional inter-library loan routes.

An orthogonal purpose of zetoc is to provide a current awareness service through its Alert function, which emails, in plain text, tables of contents of new journal issues to requesting users. Users may also set up an alert search request based on keywords in the article title or an author's name. These saved searches are performed on new data when it is loaded each night, users being emailed with the records of articles that matched. Along with each article listed is a URL, which provides direct entry into the web service and enables the user to take advantage of the linking and document delivery functionality.

The service rapidly became popular in UK higher education institutions with over 16,000 users registered for email alerts in February 2003 setting 23,000 journal alerts which typically resulted in over 8,000 emails being sent out per night. On the same date zetoc use was around 30,000 sessions and 60,000 searches per month via the web interface. zetoc has received many very positive reviews include the accolade of `800lb Gorilla of UK email alerts' (Carnall 2002).

2. User Behaviour

The zetoc service is a valued and popular service and is playing a significant role in the `discover-locate-request-process' by which users move from an interesting title to a full text article. In order to understand what role it was playing for users and, if possible, improve it a formal evaluation study of zetoc was started in 2002 involving the Department of Human Sciences at Loughborough University. Manchester and Loughborough had worked together on a previous electronic journal project, SuperJournal (Pullinger and Baldwin 2002) and had discovered a wide variety of user behaviour with the service. In the SuperJournal project it had been possible to log the usage made by over 3,000 users and to classify individual behaviour patterns. A cluster analysis of 1,882 users, for example, identified 1,112 users (59%) who only visited the service once in contrast to a smaller group of repeat users for whom the service became a fundamental part of their academic strategy. In total eight separate clusters of users were identified who were using the SuperJournal service in quite different ways (Eason et al 2000a, 2000b). One of the reasons for the variation of usage was that users were utilising other electronic services as well as SuperJournal and they were trying to find the best way of getting advantages from the services that were available to them.

It seemed likely that, within the large population of zetoc users there was a similar range of types of usage. The evaluation study set out to determine the different types of usage, and the way zetoc was contributing alongside other electronic services to the `join up' process of getting from discovery to full text.

3. The Evaluation Process

The evaluation study was undertaken in three stages. The initial stage was an electronic questionnaire publicised on the website and in email alerts sent to users. The second stage was a brief electronic questionnaire publicised on the website after enhancements to the service had been introduced and finally an interview programme undertaken at a small number of institutions.

3.1 The First Electronic Questionnaire.

The first questionnaire asked respondents about their usage of zetoc, their satisfaction and problems with the service, other electronic services they used and their hopes for the future. It also collected information about the institution, the role of the user, the discipline etc. It attracted 655 responses. The overwhelming response was that zetoc was providing a very valuable service. The majority of users (75%) felt that it offered them a means of keeping on top of current developments in their field. zetoc offers two separate services, the alert service and access to the database. 75% of the users had set up alert services and saw zetoc as a very good way of keeping up to date with developments in their field and they had set up an average of 13 journal alerts each. About 50% of the users searched the database and valued the wide range of material that was available. The majority of the users wanted an integrated service that would enable them to go from search or alert to full text electronic copies of articles of interest. There were very few negative comments about the usability of the service. Most users also made use of other electronic services, especially the cataloguing and electronic journal services of their host institution. Where there was criticism of the usability of services it was in the interaction between one service and another, e.g. knowing whether you could obtain an electronic copy of an article listed in a zetoc alert. 28% (181) reported being aware of the Z39.50 capabilities of zetoc but only 10% (64) reported using these facilities.

What was clear from the responses was that users varied a great deal in the degree of use they made of the service. In order to analyse this a usage metric was constructed consisting of four topics examined in the questionnaire;-

• parts of the service used - e.g. search, alerts, document ordering (8 items)
• website support service - e.g. training materials, helpline (3 items)
• adaptation - e.g. refining alerts, storing and processing searches, sharing with colleagues etc to render the service in a configuration appropriate tot he user's needs (8 items)
• linking - using the Z39.50 capability, for example, to access zetoc from bibliographic software such as EndNote (3 items).

In total this gave a possible total score of 22 which was a measure of the degree to which the user was utilising the breadth of facilities in the zetoc service. Table 1 below provides the overall results for the 655 respondents and also the results for different user roles.

Table 1The Usage of zetoc

As table one shows a large majority of the users made fairly limited use of the facilities available. There was a break in the distribution at a score of 6; 83% scored 6 or less and then there was a thin spread of scores from 7 to 16. 34% of the users scored only 1 or 2. There are also notably different profiles of scores for the different user roles. The average score for the sample was 4.08 and the librarians were the most prolific scorers with an average of 5.19 and nearly 30% scoring 7 or more. The lowest average was for faculty members (university research and teaching staff with tenured positions). Their average was 3.22 with 45% scoring 1 or 2. The `other' category included national health workers, administrators and managers and they too had a low average (3.51) and a large number (50.6%) scoring 1 or 2. Research staff and postgraduate research students occupy positions closer to the average for the sample.

We examined this data further by dividing the sample into two user types, the `passive majority', the 82.9% scoring 6 or less, and the `active integrators', the 17.1% scoring 7 or more. Examining the questionnaire data for each group enables us to offer the following accounts of these two user types

• The active integrators This group of 112 users appear to understand the range of services zetoc offers and the way it relates to other services. They know, for example, that they are less likely to get an abstract with the zetoc service than they are with some other services but may be able to reach a different range of journals. They also know which of the journals in which they have an interest can be obtained as full text electronic versions in their host institution, which are in their local library and which they need to seek by other means. Most of these users have developed a stable routine by which they keep abreast of new developments and by which they store and process articles of interest. They make use of many other services and are the users who make use of Z39.50 services. 41% of this group are librarians and could be said to have a professional interest in keeping abreast of zetoc and related services. Another 37% are research staff and post-graduate students who, by their accounts, are expected to sustain a good awareness of particular parts of the literature and have the dedicated time to achieve this objective.
• The passive majority This group of 543 users tend to see zetoc as an alert service that provides them with regular emails. We have called this the `passive' group because many report creating alerts and `letting the service do its job' as one respondent remarked. Very few of these users had a stable routine for managing the information they received from zetoc or of following up articles of interest. They tended to let the alert messages accumulate in the hope they would find the time to check them and process them later. Many of them made occasional use of the database when they had to search an area outside of that covered by their alerts. Very few had received or sought any training in zetoc and when asked why they did not use other services it provides said they were unaware of them. Indeed, as a result of responding to the questionnaire, many said there was obviously more they could use and they would now look again at what zetoc had to offer. They did not make a great deal of use of other services and often found it difficult to track down articles when they had to switch from one system to another to do it. There was good representation of all work roles in this group but the very restricted scores were primarily faculty members and `others' such as managers and administrators, in all cases work roles that include a wide range of responsibilities

One issue united both user types; they each wanted an integrated, seamless service to enable them to get from discovery to the delivery of a full text electronic version of articles of interest. We may then ask why there is such a large difference. If the passive majority want the same integrated service as their colleagues what can be done to help them exploit the capability and get nearer to a more joined up service?

3.2 System Enhancements

During 2002, partly as a result of the first evaluation study, a series of enhancements were made to the service and at the same time many universities were enhancing and integrating the electronic services available to their users. One of the major changes to the zetoc service was the addition of further links to other services through the Z39-50 protocol. Z39-50 is a network protocol that allows searching of remote heterogeneous databases and retrieval of data from an individual user interface. As a result of the enhancements many users were able to access the zetoc database from other services, for example from portals, from bibliographic software and from linking software. Similarly, if they found an article of interest using zetoc, users could gain direct access to facilities in their institution by which they may be able best to obtain further information, perhaps electronic full text of the article.

The introduction of these enhancements had a dramatic effect on the usage of zetoc. In September 2002 there were 2,317 Z39.50 target sessions across UK academic institutions. By March 2003 it had risen to 11, 881. A related development was that eleven institutions had introduced `resolvers' that meant that their users could go from a zetoc search directly to a service which could check the availability of selected items in the resources of the institution. In March 2003 two of these institutions had achieved over 20,000 zetoc sessions in the first quarter of the year.

The pattern of usage was changing as a result of these enhancements and the next stage of evaluation sought to find out why. Was it now possible for users to achieve a more integrated service and, in particular were the passive majority now beginning to behave more like the active integrators?

3.3 The Second Questionnaire; the Effect of the Enhancements

The second questionnaire had two objectives; to examine the impact of the enhancements and to validate the conclusions of the first study about the benefits users sought and the problems they experienced.

In this case the questionnaire was made available from the zetoc website which affected the response rate. Users who primarily use the alerting service would not, for example, have been aware of the questionnaire. 146 responses were obtained of which 98 were Librarians. On the basis of the first questionnaire it can be assumed that this sample is biased in the direction of the `active integrator' users and the interpretation of the results takes this into account.

The overall usage scores for the user group and the whole sample are reported in table 3. The sample in this case was 146 in total with 141 usage scores.

Table 3 Usage Measures for the User Roles after Service Enhancements

The distribution of user groups in the sample was influenced by how people reached the questionnaire. 67% of the respondents were librarians and there are small numbers of respondents in the categories which were predominately the `passive majority' e.g. members of faculty. Compared with the results from the first questionnaire (Table 1) there is an increase in the overall score from an average of 4.08 to 5.95. However, there are very few low scores in this sample and more valid comparison may be made between the librarian groups in the two samples because they were the most active users in the first study. The average for the librarians in study one was 5.19 and in study two it is 6.10. Because of the enhancements the highest possible usage score has risen from 22 to 29 and the average has risen in line with this increase. There is some evidence therefore that the range of usage is increasing and it may be a product of the enhancements. In passing it is it should be noted that the scores for all user roles are greater, especially that of faculty members where a group of eight users have achieved high scores. However, the numbers in these categories are too small to draw any general conclusions. It is possible to examine the impact of the enhancements more closely by looking at the results of a question asking who had used them and these are presented in table 4.

Table 2 The Effect of the Enhancements

The usage of the enhancements contributes about 33% of the overall increase is usage scores from sample 1 to sample 2. The respondents who had made use of the enhanced services were mostly positive about their value. The biggest response was to the opportunity to launch an online article search. There was also a considerable response to `request a copy' but for most it made no difference; it still depends on whether a copy is available.

From the evidence of the overall usage statistics of zetoc and the results of the second survey it appears that offering Z39.50 links to and from zetoc is increasing the range of usage of the service. These opportunities are beginning to be exploited by librarians and other active users. Whether they are reaching the passive majority remains to be explored.

3.4 The Benefits and the Challenges.

The second questionnaire was also used to validate some conclusions from the first questionnaire in which users had expressed a variety of benefits they were seeking from electronic services in general and zetoc in particular. They also commented on the major problems they met and challenges to be overcome both in general and for zetoc if they as users were to achieve the benefits they were looking for. The benefits and the challenges were summarised in the second questionnaire and respondents were asked to rank which were the most important. The overall rankings for benefits are summarised in table 4 and for challenges in table 5.

Table 4 The Benefits Users Seek

The most important benefit being sought is keeping on top of current developments and this is what zetoc helps users to achieve. The second benefit that is sought is to be able to establish a routine for keeping up to date and again this is supported by zetoc. It is important to note that this sample of users probably do achieve an organised current awareness routine. The more restricted users who are less able to sustain a routine are not well represented in this sample. The third priority is to get from search to full text electronic publications. These users give greater priority to obtaining electronic rather than hardcopy full text publications. The correlation between sought benefits and the goals that zetoc helps achieve is high. Where there is a variation it suggests zetoc is seen primarily as a means of achieving a current awareness service.

Table 5 The Challenges to be Overcome

Making it easier to get from search to full text is the top challenge users wish to be addressed and it is the top challenge for zetoc. The second priority is improving the usability of services although in many cases users are referring to the ease of moving from one service to another. This is reflected in the third priority which is to achieve better integration of the different services. Two sources of aggravation to users as they seek an awareness routine and a `joined up service' are avoiding getting information overload and avoiding the promise of electronic full text only to be refused access. The degree to which users felt able to achieve these benefits and the extent to which the problems were inhibiting their usage were explored in the interview study.

3.5 The Interview Study

The third stage of the evaluation study is to conduct interviews with individual zetoc users in six universities. There are three aims. The first is to obtain an account of the strategies users employ to manage information search and the role of zetoc in their strategies, the second is to understand the barriers and difficulties they experience and the third is to explore the implications of the local institutional arrangements on the usage pattern that develops. A subsidiary aim is to improve our understanding of the `passive majority' because they are under-represented in the second questionnaire study. The interview sample is therefore skewed in the direction of user roles more likely to be part of this user type.

The interview study is on-going. 12 interviews have been conducted at three universities and, because of the sample size, the conclusions below must be tentative. However, the two questionnaire studies provided opportunities for respondents to offer comments and these have been used to validate the strategies that have been identified and provide additional evidence. Table 6 summarises the initial results of the study.

Table 6 Initial Results from Interview Study

Of the three universities, one (University A) has introduced a portal which gives users direct access to the zetoc database and they can use the portal to identify how best to follow up any references of interest. The other two universities have no equivalent system.

The majority of the interviewees have been members of faculty with a small representation of researchers and postgraduate students. The usage scores have been derived from the scale used in the second questionnaire study. The overall average (6.0) is higher than the overall average from the first questionnaire (4.18) even with no librarians in the sample. However, the difference is mostly attributable to the results from University A and is closely associated with the introduction of the new portal.

A major part of the interview was devoted to obtaining an account of how the user wished to manage information search and how they managed it in practice. The initial results suggest there are three major strategies in practice. These three strategies define in greater detail the behaviour of what we have previously described as the passive majority. Together with the user group defined previously as the active integrators, there are now four user types, as described below.

The Uncontrolled Restricted User (UR) These users are characterised by a tendency to start using zetoc services and then finding, usually because of other priorities on their time, that they are unable to follow-up the results of the service as they would like. They set alerts, for example, and then as one faculty member put it "come to hate zetoc because the emails keep coming and there is not enough time to deal with them". The emails become a constant reminder of not managing to keep up-to-date. Another user asked "how do I turn the alerts off?"not because they were not of interest but because they added to the guilt feeling of not coping. The same process occurred with searching the database; many items of interest may be found but then there is insufficient time to follow them up. When they followed up references these users became particularly irritated if they had to change services, re-enter data, find passwords etc and were particularly aggravated when a trail ended in disappointment either because the document could not be obtained or it turned out to be of little interest. These users tended to know little of the range of services zetoc provides or to use many of them. Similarly they knew little of other services available and could not find the time to study them. These users spoke of wanting a seamless, integrated service but there was little sign that they were attempting to integrate their use of services. The defining characteristic of this type of user is that they would like to achieve a certain type of service but they are unable to sustain their efforts towards achieving it. Their usage scores tended to be five or less and the majority of the users were faculty members.

The Controlled Restricted User (CR) This user type has established a routine for using zetoc which they do manage to sustain. It is usually quite limited in scope which renders it manageable. The majority of this user type only use zetoc for alerts and they set a limited number (5 to 10) which are the journals in which they have an enduring interest. When emails arrive they have a well established routine for checking them and identifying any worth pursuing. They tend to file the emails to provide a reference store of the contents of recent journals. The users tend to know what opportunities for access exist in their institution for each of the journals and, when they find a interesting reference, they know how to pursue it, for example, because electronic full text is available for that journal, it is kept in the library or it will have to be ordered. These users tended to pursue quite a lot of leads and to store valuable papers as their local store of significant contemporary papers. Many users wanted abstracts available with the TOC email but this user group was particularly keen on this provision because they needed to check the likelihood of the paper being of interest before they devoted time to the follow up. The defining characteristic of this type of user appears to be the hunt for a successful balance between getting a good and relevant service and not spending an excessive amount of time doing it. They were often aware that there were other aspects to the service and other services they might use but were guarded about the time they could commit to training or experimenting. They tended to be pleased with what they were receiving and would only expand their usage if the benefit looked really worth while. Researchers tended to be in this category as were some of the younger members of faculty. Their usage scores also tended to be relatively low (4 to 8) and they tended not to use integrating facilities either because they did not need to and or they had not taken the time to study what opportunities there were to achieve integration.

The Controlled, Restricted and Integrated User (CRI) This user type is similar to the controlled restricted user type in having a fairly ordered routine for setting and processing alerts and for making searches. They are also fairly restricted in the parts of the service they tend to use. Where they differ from their counterparts is that they are actively exploring the opportunities for an integrated service. These users are characterised by a slightly more experimental attitude to new opportunities than their colleagues although they give many examples of services they tried that led nowhere and wasted their time. they have since rejected many of these services. Their focus remains the creation of a good, sustainable, easy-to-use and relevant service for themselves and there are strict limits on how much experimentation takes place before real benefits are expected. There appear to be three main reasons why users might fall into this category. They may, as in University A, be in an institution that has Metalib and SFX and may therefore be in a position to develop an integrated route to and from the zetoc database. They may also use bibliographic software such as Reference Manager or EndNote and use zetoc in relation to them. Another reason may be that they are developing a research career and see the development of an understanding of information services as part of their training. Many post-graduate students fall into this category as do researchers; it is a priority for them to create a contemporary store of information relevant to their work and to develop a good understanding of electronic information services as part of their career training. A third reason users may be in this category is that their fields of interest are diverse or are changing and they have to keep revisiting what they need. As a consequence they experiment and identify the new integrating opportunities that exist. The usage scores of this user type are likely to be in the range 8 to 14.

The Active Integrated User (AI) Although there were no users of this type in the interview sample, the characteristics of this type can be stated as a result of the two questionnaire studies. This user sustains a personal information service for his or her own professional purpose but also has an active interest in new forms of service that emerge and will seek to understand and to use them. This is often because part of their work role may involve an understanding of the services and the way they are developing. The large number of librarians in the study and the fact that they dominate the high scoring category is testament to this requirement to stay in touch with new service developments. The work role may also carry with it a requirement to teach others about new services and otherwise support others in creating the services they need. In some cases the role of serving others may be more informal, e.g. a member of a research team may take on the role of establishing an information service for the team. These users tend to know what is possible within their institution and to have achieved an integrated service with respect to their discipline to the degree that is possible. They can speak with authority about the best way of obtaining information of different kinds and of gaining access to full text documents from different journals and other sources. They are likely to use a wide variety of services and parts of zetoc and will have usage scores in the range 10 to 20.

This characterisation of user types has to be tentative at present and will be validated in the remainder of the interview programme. Attempts will also be made to establish the user role composition of each user type and perhaps to establish the likely size of each type within the entire population of zetoc users. It is likely that the three kinds of restricted users constitute at least the 80% of users we have called `the passive majority'. It is also likely that this is an underestimate in relation to the full population; users who respond by completing questionnaires are likely to be more active than their colleagues. It is quite probable that the uncontrolled restricted user is the largest category of zetoc user.

4. Discussion; The Barriers to Integrated Service

zetoc is a very successful service and many of its users now consider it an essential part of their information management strategy. As it becomes possible to use it as part of an integrated joined up service so its potential value is increasing. And yet many of its users make use of it in a very restricted way and are not exploiting it fully to achieve their stated aim of an integrated service. We know from previous research that restricted usage was also the case with the SuperJournal electronic journal service and we strongly suspect that it is also the case with many other services. This is because the reasons are only partly to do with the service; they are also to do with the work role and professional context of the user and the institutions in which they operate. The evaluation studies undertaken of the zetoc service have provided an opportunity to understand the user and the context and to examine how these factors influence usage patterns. To conclude this paper we have constructed a model of three barriers to the achievement of a fully integrated service.

• The Aim. There are two aims that are very widely shared by users; to create a sustainable current awareness routine and to be able to move, where relevant, easily from an article title to a full text version, available in electronic form at the workplace if possible.
• The Technical Possibility. zetoc and Z39.50 opportunities are increasingly making both the current awareness goals and the `joined up' service a technical possibility for a greater range of materials.
• The Institutional Context. Although the service is technically possible it may not be available to users in all universities. zetoc may be available across the universities in the UK but whether the users can reach full text depends on the technical services available to users in each university and the commercial arrangements made with publishers etc. There is growing evidence that users in some universities are achieving a much more integrated service than others because they have the facilities to do so.
• The User's Role, Characteristics and Context of Use Even in the same university there is a wide range of user behaviour. Librarians and others who's role gives them a professional interest in electronic services usually keep up to date with what is available and exploit what is possible in their institution. The majority of users, however, are passive in their use of services they do use and probably do not achieve all they could given the resources of their institution. They also find it difficult to keep up with what is available and may not exploit new services and new links between services that have become available. It is difficult to measure the proportion of users who are part of the passive majority but it is likely to be under-represented in the data in this survey.

The implications of this examination of user behaviour for the providers of electronic services and for those delivering services in institutions are first that current awareness and `joined up' services are very much valued by users. However, it is not sufficient that these are technically possible or that institutions make them available. It is not even sufficient to provide regular training events because many users will not take advantage of them. The problem is how to help people who are so busy with other priorities that they cannot know about or take advantage of what is available to them. The requirement for seamless join-up perhaps provides the clue to how to help these users. If new services and enhancements are organically linked to the services with which users are already familiar the extension of usage can be evolutionary and can occur without users having to make a step function in their understanding of services. Although the evidence at this stage is slight the exciting part of the zetoc story may yet be the take off in usage caused by the Z39.50 enhancements because they provide a linkage mechanism for users of great value to their `joined up' service aspirations which is relatively undemanding to adopt.

References

• Carnall D. (2002) `Website of the week: email alerting services' British Medical Journal, January, 324:56
• Eason K.D., Yu L. and Harker S.D (2000a).`The use and usefulness of functions in electronic journals: the experience of the SuperJournal Project' Program, 34.1 1-28
• Eason K.D., Yu L. and Richardson S.M. (2000b) `Patterns of use of Electronic Journals', Journal of Documentation, Vol 56(5) pp 477-504.
• Pullinger D. and Baldwin C. (2002) 'Electronic Journals and User Behaviour' Cambridge, Deedot Press

25 July 2003, epub@manchester.ac.uk

Electronic Publishing          Home Page