Transferring evidence into practice: what evidence summaries of library and information studies research tell practitioners

Lorie A. Kloda
McGill University, Life Sciences Library, 3655 Promenade Sir William Osler, Montreal, Quebec, Canada, H3G 1Y6
Denise Koufogiannakis
University of Alberta, 5-25 Cameron Library, Edmonton, Alberta, Canada, T6G 2J8
Katrine Mallan
Jacobs University – IRC, Campus Ring 1, 28759 Bremen, Germany

Introduction

Critical appraisal is a crucial aspect of evidence-based practice. In order to determine whether research is valid, reliable and applicable, the evidence-based practice process advocates that published research be critically appraised. Critical appraisal in library and information studies is seldom published, but since 2006, the journal Evidence Based Library and Information Practice has published an average of eight evidence summaries in each issue. Evidence summaries provide a synopsis and critical appraisal of published research in order to facilitate the transfer of research into practice. This study aims to determine strengths and weaknesses of library and information studies research, as identified in the journal's evidence summaries, so that as a profession we might improve our research, and as readers we may consider the strengths and weaknesses of research publications as well as their utility.

Background

Evidence-based practice in library and information studies

This definition stresses three main areas that contribute to evidence-based practice: 1) research evidence; 2) user needs and preferences; 3) professional judgments. All three of these components are vital to evidence-based practice. Research is not conducted in isolation, but rather, is integrated with our professional judgment and within our local context.

The third step, appraise, refers to the critical appraisal of original research in order to determine its validity, reliability and applicability. Appraisal is a rather onerous task, requiring expertise in a myriad of research methods depending on the study's research design. It is this step that the evidence summaries published in the journal Evidence Based Library and Information Practice attempt to accomplish, saving the practitioners who read them time and providing them with the benefit of others' expertise.

Evidence Based Library and Information Practice is an open access journal featuring original research articles, evidence summaries, and commentaries. The journal is available entirely online and is hosted by the University of Alberta using the Open Journal Systems platform, part of the Public Knowledge Project. It reaches an international audience and as of 2009, has over 2000 registered readers. The journal launched in January 2006, with issues published quarterly. Evidence Based Library and Information Practice is the only current library and information studies journal that publishes evidence summaries.

Evidence summaries

The evidence summaries published in Evidence Based Library and Information Practice are modelled on synopses found in health sciences journals such as ACP Journal Club, Evidence Based Nursing and POEMs (patient-oriented evidence that matters) in Essential Evidence Plus. In these publications, clinicians can read structured abstracts of clinical research with commentaries, and sometimes a bottom line: a statement that answers the question: What does this change (or not change) in my practice?

Each evidence summary in the journal consists of a structured abstract summarizing the original research article as well as a commentary of 300 to 400 words appraising the quality of the research. Approximately eight evidence summaries are published in every issue, summarizing research in all areas of librarianship. The previously published research that undergoes appraisal is culled from reputable peer reviewed journals, mainly in library and information studies, and occasionally in related fields. The Associate Editor for Evidence Summaries regularly reviews approximately ninety journals that publish peer reviewed research articles, and chooses research articles that appear relevant for library and information studies practitioners. The topics and research settings are chosen in order to appeal to a wide ranging readership.

Evidence summaries are written by a team of writers selected by the Associate Editor on a rotational basis, each normally writing four evidence summaries within a two-year period. Team members apply to be on the team, and are selected by the Editorial Team based on their areas of expertise and their experience with evidence-based practice and critical appraisal skills. Evidence summary writers must follow established guidelines (Appendix A) detailing what should be included in an evidence summary, and are encouraged to use existing critical appraisal tools to aid them in their appraisal. Each evidence summary submitted by a member of the team goes through the journal's regular double-blind peer review process with at least two peer reviewers providing feedback before the submission is considered for acceptance by the Associate Editor.

The structured abstract of the evidence summary allows practitioners to obtain the essential elements of the study without having to read the original, while the commentary portion offers an evaluation of its quality and usefulness. The commentary provides the evidence summary writer with the chance to comment on the original study's method, the reporting, the research's significance within librarianship and, more importantly, how the research may impact professional practice.

Although several publications and electronic resources in the health sciences publish critical appraisals of original research, we were unable to find any published reports of research analysing their content. Several studies have investigated the use of evidence summaries for impact on knowledge and practice (e.g., Grad et al. 2008) but these focused on practitioners' (in this case, health professionals') perceptions.

Purpose and objectives

The purpose of this research study was to identify criticisms, both positive and negative, of the library and information studies research literature. The study's objectives were twofold:

Methods

We undertook a directed qualitative content analysis of the commentary portion of evidence summaries published in Evidence Based Library and Information Practice from 2006-2008. 'Qualitative content analysis goes beyond merely counting words or extracting objective content from text to examine meanings, themes, and patterns that may be manifest or latent in a particular text' (Zhang and Wildemuth 2009: 309). In this way, we were able to identify the nature of the criticisms described in the evidence summaries and the tone (positive, negative, neutral) of these statements. The emphasis was not on frequencies, but on the importance placed on certain characteristics, and on uncovering new aspects of research considered important by the evidence summary writers.

All twelve issues of the journal were included, for a total of 101 evidence summaries. A data extraction form was created based upon evidence summary guidelines for writers along with elements relating to validity, reliability and applicability from critical appraisal worksheets such as CriSTAL (Booth and Brice 2003), RELIANT (Koufogiannakis et al. 2006) and the critical appraisal tool created by Glynn (2006). The data extraction form (Appendix B) was designed to facilitate both inductive and deductive analysis. The form went through two rounds of pre-testing, whereby all three researchers independently coded and compared results. Researchers discussed areas of confusion or omission, refining the extraction tool until we felt confident that the form was clear and straightforward.

Each commentary was read through and scrutinized independently by two of the researchers after which any discrepancies were resolved by the third researcher. Additionally, we looked for possible emerging categories, that is, new concepts that arose from our reading of the evidence summaries during the analysis but that were not already accounted for in our data extraction form. We kept track of these new categories as well.

The data extraction form focused on areas of validity, reliability and applicability. While these terms might seem to reflect a quantitative or positivist approach, the criteria were also applicable for research employing interpretive, qualitative inquiry.

The data extraction form examined five areas that indicate validity. Validity is 'the extent to which the results of the research are likely to be free from bias' (Reynolds 2000: 25). In the commentary portion of each evidence summary, we looked to see if the evidence summary writer noted:

For each of these, we coded for whether the writer noted these elements as strengths of the original study, weaknesses, or if the element was described neutrally. While it was possible for the same element to be coded both as a strength and a weakness: this was rare.

Seven areas touching on reliability were also examined. Reliability is 'the likelihood that this study reports something that is reproducible, as opposed to being a "fluke" or chance result' (Booth and Brice 2004: 105). We looked to see if the evidence summary writer noted:

Finally, we coded for three aspects relating to applicability, 'the extent to which the results are likely to impact on practice' (Booth and Brice 2004: 105). We looked to see if the evidence summary writer noted:

Findings

General attributes

To put the evidence summaries into context we noted some general attributes of the 101 summaries we examined, by grouping them by aspects such as domain, setting, source, and length.

In evidence-based library and information studies, domains are used to assist with identifying the most applicable resources to search for evidence and to focus a given question. Based on the research of Koufogiannakis, Slater and Crumley (2004), there are six domains into which research literature can generally be grouped.

The evidence summaries were generally representative of the breadth of domains represented, with most articles appraised falling in the categories of information access and retrieval or collections (see Table 1). Domains are not mutually exclusive, so an article may belong to more than one domain.

The setting for the research was also noted, and is represented in Table 2. The highest number of research articles that were critically appraised in an evidence summary came from academic settings. Again, the setting within which research takes place is not necessarily mutually exclusive, as research could be in both a health and academic setting, for example.

Commentaries within evidence summaries are supposed to be between 300 and 400 words. We tabulated word counts for the 101 commentaries included in this study and found that the mean length of commentaries was 517 words. These range between 140 and 1220 words, and the majority of commentaries (n=39) were between 400 and 499 words.

Validity

**Table 1: Evidence summaries by domain**
Information access and retrieval	26
Collections	25
Education	21
Management	16
Reference	14
Professional issues	12

**Table 2: Evidence summaries by setting**
Academic	45
Health	27
Public	12
School	8
Any and/or All	7
Scholarly publishing	6
Special	2

For the validity aspect of the evidence summaries, the analysis revealed some trends. Elements of validity that were most commonly referred to by the evidence summary writers were: 'the suitability of the population and the sampling method' (n=80) and 'the suitability and replicability of the research method' (n=76). In both cases, more than two-thirds of the time, these were identified by the evidence summary writers as weaknesses of the original research. An example of a comment regarding a weak choice of method: 'one wonders if there are other variables in the studies that may have also had an impact on the study results'. Similarly, for the suitability of the population, another evidence summary writer commented, 'Participants were randomly contacted but it is unclear how randomization was done or whether there was a self-selection bias in the type of respondent who agreed to participate (response rates were not provided)'.

The only category relating to validity that was commonly viewed as a strength was 'the presence of a focused issue or research question'. This was referred to a total of fifty-one times, most commonly (n=24) as a strength of the original study. An example of a comment in the evidence summary that was coded as a strength is, 'The aims of the study were clear'. This variable was noted as a weakness eight times, and for the remainder, comments were neutral.

The other two aspects of validity were less often mentioned by the evidence summary writers. 'the data collection instrument's validity' was mentioned in thirty-eight of the evidence summaries and as a weakness on nineteen occasions, while 'the possibility of any conflict of interest' was mentioned in merely ten (and as a weakness, for seven of these).

Reliability

We coded for the seven areas relating to reliability. For all seven areas, these elements were noted by the evidence summary writers as weaknesses of the original research. In general, reliability was not an area very heavily noted by the evidence summary writers. The area most commonly noted (n=57) was 'limitations of the study', with the majority of these limitations (n=33) being looked upon negatively. Only in three did evidence summary writers remark on this issue as strength, and twenty-three were neutral in their appraisal. 'The utility and clarity of the analysis' was noted in forty-one evidence summaries, mostly as a weakness (n=26), and 'the compatibility of the analysis to the research design' was also seen as a weakness in the research being appraised (n=32 out of 37 mentions). We present an illustrative comment from one evidence summary discussing the utility of analysis method:

The other areas related to reliability were remarked upon much less by evidence summary writers, but when they were noted, were generally presented as weaknesses of the original study.

Applicability

In contrast to aspects of validity and reliability, applicability was an area many evidence summary writers noted as a strength of the original research articles. Most commonly noted in the commentary section were 'the implications for practice reported in the original study' (n=70) with forty-five studies noted as having this as a strength. Some reported the applicability as weakness of the study (n=14) or neutrally (n=10). The other two areas, 'applicability to populations beyond those investigated in the original study' and 'any local information required to implement the research results' were mentioned less frequently, but positively. An example of a positive comment by one evidence summary writer was: 'suggests that Web-based tutorials are at least as effective as face to face teaching sessions and that these may be successfully delivered either in the classroom or via the Web'. Occasionally, the writer noted applicability of the study as a weakness, for example, 'it is not immediately clear how this information could be utilized by library practitioners'.

Other findings of note

As part of the content analysis, we coded for other elements beyond validity, reliability and applicability. First, we verified if the evidence summary writer situated the research article they were appraising within a wider context. Forty-eight of the 101 summaries did situate the research they were appraising within a wider context, either commenting on research in this area in general, noting societal or political aspects affecting the topic, or directly citing relevant research (n=30). In addition, fifty-seven out of the 101 evidence summaries made statements noting the overall significance of the original research to library and information studies.

One category that emerged from the content analysis was the quality of the literature review. Eleven commentaries included a statement about the literature review's contribution to the research, with seven evidence summaries noting the literature as a strength, three as a weakness, and one neutrally. For example, one evidence summary writer noted,

Another emerging category dealt with ethical issues. In one evidence summary the lack of participant consent was noted, and in another the original article was noted as a duplicate publication. These two examples point to weaknesses in research that readers should be aware of and that impact the value of research results. One evidence summary writer consistently cited the critical appraisal tool that was employed in writing the evidence summaries.

Finally, for each evidence summary, the length of the commentary section and corresponding number of categories were calculated to determine if there might be a relationship between the length of the commentary and the number of elements noted. No relationship was apparent, that is, evidence summary commentaries that were longer did not contain more elements of appraisal than those that were brief. The average number of categories discussed in an evidence summary commentary was 7.8 (range between 4 and 14 elements noted).

Discussion of findings

This research has several limitations. Firstly, while we decided to include all evidence summaries from the journal Evidence Based Library and Information Practice, there were 101 summaries, written by twenty-eight different writers, some writing up to six or seven within the three year period. This may have led to findings that may not be as applicable to those evidence summaries published since 2008, as the team has changed over time.

The evidence summaries included appraisal of articles that were selected mainly by the Associate Editor at the time, and were not randomly selected from the library and information studies literature. Findings from this research are therefore not representative of strengths and weaknesses of the library and information studies literature as a whole, but reflect the emphasis placed on the selected research publications by the evidence summary writing team. As other scholarly publications begin to produce evidence summaries, and more writers create them, more variability among summaries may appear. The evidence summary writers from this journal may exhibit similar biases and preferences that may have skewed our findings. In addition, despite the structured element of evidence summaries, the writing style of authors varied greatly, thus making the content analysis difficult in some cases. For example, one writer might be clearly pointing out a study's weakness, while another might be more indirect. This results in difficulties in the analysis. This might explain why, in several cases, we had to judge a term as neutral. Perhaps the writer thought their statement was obviously negative, but if it was not overt, it was coded as neutral.

One of the researchers (LK) was also an evidence summary writer, and contributed six publications to the set of 101. Potential bias was avoided by assigning the other two researchers to analyse these documents, and allowed them to resolve any disagreements.

We believe this analysis will provide insights for library and information studies professionals regarding aspects of validity, reliability and applicability that should be considered when conducting or reading research. The analysis highlights several areas where overt weaknesses were noted by the evidence summary writers. These areas of weakness are ones that those conducting research should be aware of and proactively address when conducting their research and reporting results, hence strengthening the body of research within our field. While no research is perfect, being aware of weaknesses and potential pitfalls should help library and information studies professionals address these elements in their own research endeavours. Likewise, noted areas of strength provide examples of good practice that new researchers can model.

This study also raised many new questions related to critical appraisal and library and information studies research. Aspects of validity and reliability in studies that were critically appraised in Evidence Based Library and Information Practice were more often noted as weaknesses of the study. Whether this was due to general poor study design or the focus of the writer in trying to point out faults rather than positives, is unknown. A follow-up to this research could be to interview those who write evidence summaries to determine their goals and motivations, and the degree to which they note negative and positive aspects of previously published research. As well, we were left to wonder what we could conclude from areas that were not noted very frequently by the evidence summary writers. Are these areas not of concern because they were dealt with appropriately by the original author, not important enough to mention, or simply overlooked?

Finally, despite the evidence summary writers' overall negative view towards elements relating to validity and reliability, applicability was still widely viewed as positive: Why? It seems logical that if validity and reliability are weak in a study, that study's applicability would therefore be weak as well. Is there a lack of connection between the parts of a paper? Were the evidence summary writers focused on weaknesses that put findings into question (validity and reliability) and not as concerned with aspects of applicability, or were they trying to end their appraisal on a positive note as generally applicability is discussed last in the commentary? Alternatively, the research articles may have yielded elements that were relevant to practice, even if the actual results came from weak methodology. These questions could be looked at more closely by subsequent research that combined interviews of the evidence summary writers with closer inspection of the text within a paper, using a sample of the total summaries.

We also suggest that further study investigating library and information studies practitioners' perceptions of evidence summaries and their impact on knowledge and practice would be an important contribution to this area of research. Doing similar work to what was done in medicine by Grad et al. with their information impact scale would relay a sense of the importance of the evidence summaries to practitioners who wish to integrate research into their learning and decision making.

Conclusion

This study shows that evidence summaries published in Evidence Based Library and Information Practice between 2006 and 2008 reveal more weaknesses than strengths in the library and information studies research that was critically appraised. In general, evidence summary writers tend to remark on weaknesses in reports of research relating to validity and reliability, yet paradoxically point out strengths with respect to their applicability to practice. The exact reasons for this require further qualitative research, to reach a more complete understanding of the reasons why the results were as noted.

These results have implications for the future of evidence summaries published in Evidence Based Library and Information Practice. The Editorial Team will need to consider revisions to evidence summary guidelines, possibly adding more structure to the commentary section, and adding elements such as comment on the literature review and what critical appraisal tool was used, a better balance between strengths and weaknesses, and stricter adherence to word length. The Editorial Team has already implemented a checklist of items to consider for peer reviewers containing more detailed directions on what they should be looking for and commenting on in respect to their review of the evidence summary. They have also implemented an application procedure for new evidence summary writers that includes writing a sample evidence summary before they joined the team.

Acknowledgements

This paper is based on presentations given at the 5th International Evidence Based Library and Information Practice Conference in Stockholm, Sweden (June 30 to July 3, 2009) and the Workshop on Instruction in Library Use in Montreal, Canada (May 25 to 27, 2009).

About the authors

Lorie Kloda is an Associate Librarian in the Life Sciences Library at McGill University and a doctoral candidate at the School of Information Studies. She may be contacted at lorie.kloda@mcgill.ca
Denise Koufogiannakis is Collections and Acquisitions Coordinator at the University of Alberta Libraries, Edmonton, Alberta, Canada. She received her Master of Library and Information Studies degree from the University of Alberta, and is currently working on a PhD in Information Studies at Aberystwyth University, UK. She can be contacted at: denise.koufogiannakis@Ualberta.ca
Katrine Mallan is a Technical Services Librarian at Jacobs University, Bremen, Germany. She received a Bachelor's degree in Arts and a Master's degree in Library and Information Studies from McGill University, Montreal, Canada. She can be contacted at: k.mallan@jacobs-university.de

Appendix A: Guidelines for evidence summary writers

Evidence Summaries are brief critical appraisal reviews of current research articles. The summary follows a standardized format to ensure consistency and ease of use for readers. All evidence summaries will undergo peer-review to ensure quality. For each summary, the following components must be present:

1. Descriptive title of the review, that is indicative of what we can learn from this research.

3. Reviewer's name and contact information (this needs to be left out when submitting for peer review)

5. Commentary - provide 300-400 words of commentary on the quality of the research article. What was good about it; what are areas of concern; what could have been improved? Commentary should address the strength of the evidence presented, methodology used, issues of reliability, validity and applicability. What is the significance of the article to library and information practice; what are the practice implications for librarians in the target environment? Try to place this research article in the wider context of research available on this topic. How can this research be applied to everyday practice? Reviewers should refer to a critical appraisal checklist to ensure all important elements of assessment have been taken into consideration.

Appendix B: Data extraction form

For elements of commentary, indicate, if the item is present in the commentary, if it is mentioned as a STRENGTH, or WEAKNESS of the original study, or if it is mentioned neutrally. If it is not present, leave the item blank.

Highlight the corresponding segment(s) in the commentary, and use the code to identify these (e.g., v3, a0)

* Required

ID * (enter unique ID for Evidence Summary Commentary from document ) __________

Name of coder _____________________________

Title of Evidence Summary (cut and paste full title of Evidence Summary here)
_________________________________________________________________________

Research situated in a wider context the ES writer mentions other research specifically, or generally
Yes, the ES writer situates the study in a wider context
AND the ES writer names other researchers or other studies specifically

VALIDITY (v1) focused issue/research question The ES writer discusses the issue/research question''s focus
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

VALIDITY (v2) conflict data collection and service providers The ES writer addresses potential conflict of interest: those involved in the collection of data also delivering a service to the user group.
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

VALIDITY (v3) appropriate and replicable methods The ES writer addresses the research methods (appropriateness of methods and level of description)
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

VALIDITY (v4) appropriate population, representative sample The ES writer discusses the appropriateness of the population and sampling technique, and/or the level of description of the sampling/population
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

VALIDITY (v5) tested or piloted methods/instruments The ES writer discusses the data collection instruments' validation, appropriateness, or usage
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

VALIDITY (v0) other validity, but does not fit well into existing categories (please explain / provide emerging category) __________________________________________________

RELIABILITY (r1) results clearly explained The ES writer discusses the clarity of the results description/explanation of the original author
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

RELIABILITY (r2) response rate The ES writer discusses the original study's response rate (as a % or proportion)
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)
The ES writer includes the number of respondents, but not as a % or proportion

RELIABILITY (r3) analysis useful and easy to understand The ES writer discusss the usefulness of the analysis, and the presentation/organization (e.g., in table, with numbers rather than just %s)
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

RELIABILITY (r4) analysis appropriate and useful (stats or other) The ES writer discusses the appropriateness of the analysis method employed
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

RELIABILITY (r5) results address original research question The ES writer discusses whether or not the study's results address the original research question
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

RELIABILITY (r6) limitations The ES writer discusses the limitations of the study (even if restating limitations mentioned by the original author)
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

RELIABILITY (r7) conclusions based on results and supported by data The ES writer discusses to what extent the conclusion are based on the actual results of the study
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

RELIABILITY (r0) other reliability, but does fit well into existing categories (please explain / provide emerging category) ___________________________________________________

APPLICABILITY (a1) implications for practice reported in original study The ES writer discusses the original author's statement(s) about implications for practice
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

APPLICABILITY (a2) to other populations The ES writer discusses the applicability of this resaerch to other populations beyond what was discussed by the original author
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

APPLICABILITY (a3) additional research needed The ES writer addresses additional local information required to move forward
+ Appraised positively by ES writer (STRENGTH of original study)
- Appraised negatively by ES writer (WEAKNESS of original study)
~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal)

APPLICABILITY (a0) other applicability, but does fit well into existing categories (please explain / provide emerging category) ___________________________________________

Overall siginificance of the original study The ES writer addresses the theoretical/overall significance of the research (not just practical implications), e.g., using words like "important," "contribution to the field,"
The ES writer includes a statement about the overall significance of the research study
The ES writer includes future research are OTHER emerging category in addition to context, validity, reliability, applicability, significance
_____________________________________________________________________________

Quotable segment(s)? indicate if there is a representative example of one or more of the above categories in the commentary
(copy & pastas or questions to consider

e here)

Remarks by the coder ____________________________________________________________

vol. 16 no. 1, March, 2011 Contents \| Author index \| Subject index \| Search \| Home