Children’s Evaluation Criteria of Search Engines

Dipl.-Bibl. Eva Hornung, MLIS
Dublin, Ireland

Abstract

Search engines on the Internet are widely used as a starting point for Information Retrieval. Children are increasingly users of the Internet and therefore also of search engines. Some companies and institutions have responded to that demand and have created search engines with a young target group in mind. Still, there is no study to date on how children themselves judge these search features with regards to design and functionality without having given them pre-defined "adult" criteria. This presentation, which derived from a master’s thesis completed in 2000, tries to shed some light into the dark.
This article starts with an introduction to related studies in the fields of Web Design and Evaluation of Information Resources in general. The next part exploits evaluation criteria set by adults, which were published in the form of evaluation sheets. After the research questions, an introductory to the research project is provided, which was conducted in two public libraries in Dublin using the Critical Incident Technique and observational methods. The data collected shows that, when compared with adults’ criteria, there are significant differences, but also remarkable resemblance in the judgements. The last part sees a summary and recommendations on interface design issues of search engines for children.

Introduction

The purpose of this study is to understand children’s evaluation criteria concerning the design and navigation of children’s search engines. Moreover, their criteria will be compared with criteria set by adults in order to see if there are any similarities or differences.

This research topic required a review of literature in the areas of web design in general, web design for children and the evaluation of electronic information sources. Furthermore, the research fields of Information Retrieval and Information Literacy were examined.

Knupfer (1994) recommends the following several characteristics for a good screen design in general:

Clear, consistent frame protocol: the user should not be forced to search for functions.
Status lines should always available: user should always know where he or she is and how to come back.
Menus should be clear, concise, uncluttered, consistent and with easy-to-understand icons.
Text should be easy to read: left justified, limited to about 65 characters per line; centering for diagrams, lists and graphics.
Screen background should have enough contrast and be in grey or blue, colours which enhance readability.

I am convinced that an attractive design of the search interface is important for children. As De Vries (Brouwer-Janse, et al., 1997) has summarised it, talking about products for children in general,

designing for children means designing for fun.

The opening page of Yahooligans!, for example, offers lots of graphics in order to attract young users. In this case, the use of graphics adds value to the source. Sometimes, however, this habit only slows access speed down (Cooke, 1999, p. 75).

Looking at the so-called ‘media generation’, De Vries (Brouwer-Janse, et al., 1997) states that

it is relatively easy to gain access to the information that is available to children. It is not easy, however, to know how they perceive the stimuli, unless we enter their world and discuss topics with them.

This is the idea behind this project.

Design of web sites for children

Leh (1999) provides a key article on design of web sites by children. She observed 12 children (8-13 years) while they were searching the Internet and creating their own web sites within an extracurricular Internet course. She looked at how children were evaluating web sites, including their own.

In short, these were her findings with regards to outside web sites children have used for their own creations:

web sites with more visual input were more attractive for the children
web sites with colourful pictures, music, and animation were explored longer

Navigation is also a very important issue. There are two means of navigation: first the navigation bars provided by the browser and second all navigation aids within a web page or search engine. In our case, the navigation means of the search engines is of special interest.

Nielsen (2000) distinguishes between ‘structural’ and ‘local navigation’. The first way of navigation should provide links to all levels within a hierarchical order of related pages of a web site. This is very important in cases where users come to one page through a search engine and need a link to an overview page. The second way consists of local links to related content which should enable users to find similar or related pages. One area of investigation in this study will be navigation facilities in search engines for children.

Much research has been conducted to elucidate interface design principles, although rarely with children in mind as users (Large, et al., 1998).

The need for evaluation

There has been research on the comparison of search engines for adults. An area of interest in a study by Chu and Rosenthal (1996) was ‘user effort’, which means interface and documentation of a search engine. The authors pointed out that

users will not use a search engine unless they are comfortable with its interface, and able to read and comprehend its documentation when consulted.

Not only the user, however, profited from this evaluation. One of the hopes of the authors was to

help Web search engine developers design even better ones for the Internet community.

Hirsh (1999) draws the attention of further investigators on the fact that children should receive more training in evaluating electronic resources. They

need to understand how information is placed on the Internet and how to verify the accuracy of the information they find there.

She states a deficiency in age-appropriate interfaces for children which, if available, could

provide greater assistance to children in identifying the authority of the information

Large and Beheshti (2000) found out that despite provided training students in their project still encountered severe problems while searching the Internet. One suggestion was

to develop interfaces that offer much more help in dealing with such problems.

Finally, looking at the quality of web search tools, schools should not hesitate to put pressure on search engines designers. Only few search engines engage people to review web sites. It is high time to change that since

schools become more sophisticated with their Internet use, they will demand more than filtering, and they will put their subscription dollars behind that demand (Soloway, et al., 2000, p.20).

Evaluation Criteria - What adults think

Exploring adult criteria, I came across two different types of means of evaluation: evaluation sheets, which had been developed with children in mind and evaluation guides for adults. Both were examined and sets of criteria were singled out.

Criteria Of Evaluation Sheets For Children

In the following, I exploited some evaluation sheets for children developed by adults. All these evaluation forms were available online, could be printed and used in a school or library. This enumeration, however, was not by means complete, but a selection.

These guides should help children judging Internet information, but were not set for search engines in particular. In order to establish a 'hit list', I created a table and put all criteria into categories.

Table 1 continued

As the next step, I wanted to find the more important criteria by looking at the matches. Which were the criteria all evaluation sheets agree upon (four matches)? Which were the ones most agree upon (three matches)?

Ranking of evaluation criteria of table 1

4 matches:

Speed of loading
Graphics helpful/appropriate

3 matches:

Indication of authorship
Contact information provided
Indication of purpose
Design appealing to and/or appropriate for children
Navigation within site is clear and easy
Additional links (to other web sites)
Organisation of information
Workability
Content appropriate for audience
Quality of information
Indication of updates

This was the final set of criteria of the evaluation sheets for children. They will finally be compared with the evaluation criteria of children (see table 4).

Evaluation Criteria of Adults

The last paragraph saw a listing of evaluation sheets developed by adults and aimed at a young target group. The following table brings together the ideas of the adult information specialists which are experts in the field of web site evaluation. I found their articles recommended and referred to during my literature review. Again, this listing offers only a selection. It shows adult evaluation criteria, which are supposed to be used by adults.

Table 2 continued

Looking at these criteria, the ranking shows which ones are mentioned most:

Ranking of evaluation criteria of table 2

6 matches:

Indication of updates/revisions

5 matches:

Indication of authority
Reliability of author(s)
Contact information

4 matches:

Information about author(s)/sponsor(s)
Indication of purpose
Quality of links
Organisation of information
Topics covered
Bias?
Quality of information
Source of information indicated

This set of adults’ criteria will also be compared with children’s criteria (see table 4).

Research questions

A review of literature in the fields of Web Design and Adult’s Evaluation Criteria leaves the following research questions to be answered:

Does the outward appearance of a search engine made for children affect the searching behaviour of these children?
What are children’s evaluation criteria of these search engines with regards to design?
Do these criteria differ from those criteria adults consider for being important?

Two more aims of this project are:

to establish a set of children’s criteria for use by other children, parents, teachers and/or librarians
to make recommendations for search engine improvements (should the project show system failings)

The Search Engines

In the following, I will introduce the search engines for children I have picked for this study. For my selection, I rely on Bilal’s (2000) statement that there are only three search engines designed for children currently available on the Web: ‘Yahooligans!’, ‘Ask Jeeves for Kids’ and ‘Super Snooper’. Additionally, I will include a web search tool for children I found on the Internet. KidsClick! is designed by librarians and could therefore be interesting to compare with the above-mentioned search engines and their commercial background.

KidsClick! web search for kids by librarians

This search engine or ‘web guide’ was created by librarians of Ramapo Catskill Library System in 1997. Their intention was to guide young users to what they consider for being valuable and age appropriate web sites for entertainment or enlightenment.

They have a catalogue of criteria for inclusion of web sites, but insist on the fact that KidsClick! is not an Internet filter. The target group are children of grades K-7, which means children from kindergarten up to 7^th grade.

Figure 1: Snapshot of ‘KidsClick!

The interface of KidsClick! was designed by Zelacom, using graphic design elements (KidsClick! Web search for kids by librarians, 1999 ). Zelacom themselves described the outcome of the project, which was completed by April 1998, as a creation of

kid-friendly graphics [which] enhance navigation and contribute to the inquisitive, adventurous tone of the site (Zelacom Electronic Publishing, 2000).

The graphical interface of this search engine was the result of teamwork of graphic designers and librarians.

The indexing system lying behind KidsClick! is called ‘SWISH-Enhanced’, which is an enhanced version of SWISH (Berkeley Digital Library SunSITE, 2000). The most important feature of this software is that it allows limited searches to different HTML tags, such as the META tag, the TITLE tag or comments.

KidsClick! gives three search options: the search engine, a possibility of searching a subject by its first letter or a web directory. The search engine is divided into a simple and an advanced search. The simple search let users look for one or more words in ‘all fields’ (which, in my understanding, means all HTML tags) or in ‘web address only’. Obviously, the advanced search modus allows the search to be more specific. Here, the user can decide to search in all fields, only for subjects, only for titles or descriptions.

He or she can limit results by choosing the reading level he or she has (all reading levels, up to grade six or over grade seven) or by indicating how many pictures should be included in relevant sites (some or no pictures, some pictures, many pictures). Another option in advanced search is the possibility to search for URLs (Uniform Resource Locator). The user does only have to know parts of the address he or she is looking for, e.g. typing in ‘sunsite’ would lead to the same result as ‘sunsite.berkeley.edu’ or ‘http://sunsite.berkeley.edu/’ would.

Searching subjects by letter is another feature of KidsClick!. Here, if the user clicks on the letter of his or her choice, a window opens up where subjects are listed in alphabetical order. After clicking on one link, the user is led to the data pool.

The web directory is divided into fields of interests, such as ‘Facts/Current Events’, ‘Science & Math’, ‘Weird & Mysterious’ or ‘The Arts’. Each category has sub-categories. ‘Computers/The Internet’, for example, is divided into ‘Software’, ‘Programming’, ‘Internet’, ‘Key-Pals’, ‘Computer Games’, ‘Online Games’ and ‘more’. Clicking on one of these sub-categories leads the user to the data pool. This consists of recommended links together with small abstracts, a note of the amount of illustrations this site has, the reading level which is necessary here and the subject category.

Yahooligans! the Web guide for Kids

Yahooligans! is a Internet directory for children aged 7-12 and a trademark of Yahoo! Inc., just as Yahoo!, the search engine for adults. Its categories are searchable by using a keyword or browsable. According to Yahooligans!, each site has been checked by an experienced educator who evaluates and categorises it. Yahooligans! is updated daily. Sites were only removed when they go down, or the content is not longer appropriate for children (Yahooligans!, 2000).

Its privacy policy includes several statements. One point is not to ask for any personal information in order to allow access to its site. Yahooligans! is not responsible, however, for the policy of web sites it links to. The same can be said about the use of cookies. Because of its commercial background, Yahooligans! collects IP addresses for its advertisers (Yahooligans!, 2000).

Figure 2: Snapshot of ‘Yahooligans!’

Yahooligans! categories consist of sites belonging to the same topic. These categories are displayed in bold. Clicking on one category leads the user to another listing of sites, which are listed in plain text. If a user clicks on one of these sites, he or she is outside Yahooligans!. The category path on top of the page shows the hierarchical tree and should facilitate navigation around the web directory. To go back to the main category, for example, a user only has to click on the very first link in the category path (Yahooligans!, 2000).

Examples of main categories would be: ‘Around the world’, which includes sub-categories such as ‘Countries’, ‘Food’ or ‘Holidays’, or ‘School Bell’ with its sub-categories ‘Language Arts’, ‘Math’ and ‘Social Studies’.

Yahooligans! has a rating system that shows users sites, that are "particularly cool" (Yahooligans!, 2000), by putting symbol consisting of stylised sunglasses next to the link.

Ask Jeeves for Kids

Ask Jeeves was incorporated in 1996 and is based in Berkeley, California, named after the idea of a British servant in literature. The founders also developed Ask Jeeves for Kids. The illustrations are made by Marcos Sorensen.

This search engine uses natural language processing. Queries were matched against thousands of question templates in the knowledge base, which is created by research staff. User can enter questions in plain English and choose then from a list of matched questions. After having selected the closest match, the user will be directed to a site that was pre-selected by the staff of Ask Jeeves for Kids. Using this system is, according to them,

like having the world’s most knowledgeable internet [sic] librarian at your side. (Ask Jeeves for Kids, no year)

Unfortunately, there are no indications of how often the database will be updated or for which target group it is created. According to Yahoo! (2000), Ask Jeeves for Kids is a

natural language answering site for [bold font by Yahoo!, E. H.] children 7-14.

Figure 3: Snapshot of ‘Ask Jeeves for Kids’

On the one hand, Ask Jeeves for Kids sees itself as an educational web site that offers help for homework and school projects. On the other hand, it provides fun games and activities. Selection of sites is very strict and has to meet certain requirements: quality and depth of content and safety. This search engine, however, has no in-built filtering software. Ask Jeeves for Kids offers metasearches of other sites for children (Ask Jeeves for Kids, no year).

Ask Jeeves for Kids collects some data from its users by placing cookies in order to understand user visits and behaviour. According to the company, they want to improve site navigation and content. No personally identifiable information, however, will be required for access and use of this search engine. The company is not responsible for the privacy policy of other connected web sites (Ask Jeeves for Kids, no year).

Since Ask Jeeves for Kids has a commercial background, it displays advertising banners. They can be connected to specific categories of questions. There is a standard format for ads: 468 x 60, in GIF or JPEG format (Ask Jeeves for Kids, no year).

Super Snooper

Super Snooper was officially announced in October 1997 and is been recognised widely since January 1998 (Super Snooper, no year). It is not said who the provider is. This free-of-charge search engine for children claims to provide only material that does not

contain information of pornographic, violent or hatred-related nature.

The database, which lies behind Super Snooper, is, according to the provider, based on

an automated screening of sites that we find on the Internet, cataloging them based upon the type of information that they claim to provide, as contained in their site names, keywords and descriptions in the body of their home pages.

The idea behind that is to catalogue and review every Internet site which has a registered domain name; with over 1.5 million sites being included so far. Super Snooper favours listing sites from English-speaking countries.

The search options are simple: the user can enter one or more words, separating them by commas or spaces. To narrow a search, he or she has to type in more words. The providers claim to reduce the number of duplications of hits, since sites with the same description and title will not be displayed. Also, they try to present relevant sites in a sequence of importance. So far, Super Snooper has only one advertising banner on its site.

Figure 4: Snapshot of ‘Super Snooper’

Methodology

Strategy

Since the best strategy for exploratory purposes is, according to Robson (1993, Reprint 1999), a case study, I conducted two for this research project in two public libraries. The intention was to explore children’s criteria for search engines.

At the same time, I wanted to apply the so-called ‘Critical Incident Technique’ (CIT), which

is a well-tried method of qualitative research...[, that]...allows researchers to investigate a wide range of topics and generate theories using a variety of data collection tools with quite small samples, at least at an exploratory level. (Fisher and Oulton, 1999)

The practical implications were described by Fisher and Oulton (1999; italic by me, E. H.):

social scientific techniques are for example interviews or record forms
data collection can take place while the activity is going
data can be collected either verbally or in written form
sample size can range (here they are referring to Flanagan, 1954) from only 50 to some some thousand incidents

Flanagan (1954) recommends, in order to determine whether or not additional incidents are needed, to:

[keep] a running account on the number of new critical behaviors added to the classification system with each additional 100 incidents. For most purposes, it can be considered that adequate coverage has been achieved when the addition of 100 critical incidents to the sample adds only two or three critical behaviors.

He defines ‘critical’ and ‘incident’:

By an incident is meant any observable human activity that is sufficiently complete in itself to permit inferences and predictions in itself to be made about the person performing the act. To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.

For my study, a 'critical incident' was a statement a child made or an action he or she showed on the screen. I then set up a classification system by putting these incidents into categories. These categories were the actual evaluation criteria.

Since I was only working with a small number of participants, I put the number of required CIs on a lower level. Looking at how many CIs would be needed in order to form a category (a criterion), I considered ten CIs for being sufficient.

Verbal protocols

Observing the children, I asked a few introductory questions in order to stimulate their talking about what they were doing and their comments on the search engines they were using. Data collection included recording verbal protocols of them. This technique is also known as ‘Think Aloud’, widely used in cognitive psychology and education.

Problems could occur where

the task involves a high cognitive load, when the information is difficult to verbalize because of its form, i.e., visual data, or when the process are automatic for the participants. (Branch, 2000)

This could have been the case in this study since children might have had problems in talking about their actions while doing them. Furthermore, visual data was here examined. Also, some movements might have been automatic for the children. Therefore, I used a video camera to film children’s actions and the screens.

Observation

Observational methods are best suited for describing and understanding behaviour as it occurs. (Powell, 1997)

I wanted to compare the actions of children while using the search engines with what they told me about their evaluation criteria: why did they use this function? How did they judge them? Did they like the colours? etc.

Unstructured observation or participant observation,

an essentially qualitative style originally rooted in the work of anthropologists (Robson, 1993, Reprint 1999)

however, has some drawbacks. One is that

the primary data are the interpretations by the observer of what is going on around her [, which means that] the observer is [italic by author, E.H.] the research instrument, and hence great sensitivity and personal skills are necessary for worthwhile data. (Robson, 1993, Reprint 1999)

Another problem concerning ‘data’ is here that

to speak of ‘data’ in qualitative research may be misleading: the word ‘evidence’ may be more appropriate since the relevance of the evidence to the research objectives involves a greater exercise of judgement than in quantitative research (Wilson, 2000)

To overcome some of these limitations, however, I used an observation sheet in order to collect some precise quantitative data.

Limitations

Still, there were some limitations to this study and the collected data. First, since these were case studies, I was only looking at a limited number of people. I had to conduct research in two libraries in order to get enough participants (summer holidays). Second, I tried to be neutral, but there is always a danger of bias. The CIT had also drawbacks, because

the catagorization phase of the technique should be considered very subjective and difficult. (Shirey, 1971)

Data collection

Location and participants

Data collection took place in two public libraries, since schools were closed for summer. There were not enough participants in the first library, so I had to go to the second. One of the participating libraries was

first library in Europe to be involved in the Libraries Online! initiative. (Dublin City Public Libraries, no year)

Each observation session lasted 20-30 minutes per child. The starting point for all children was the start page I had had created, which included links to all four search engines. They were free to choose, which search engine they wanted to use and in which order.

Figure 5: Web site as starting point

Participants were fifteen children aged between eight and twelve, five boys and ten girls. I was aware of the fact that the age difference was very big and probably meant a big difference in cognitive abilities and Internet expertise. The examined search engines, however, had been created for this target group. So they should have been convenient for all children of this study.

Figure 6: Age of participants

All participants in both libraries were novice users of these search engines, except for one child, who had used Ask Jeeves for Kids before.

Parents of all children had given their permission in advance. Also, in order to protect children’s privacy, all names have been changed.

Children’s criteria

It is not easy to put qualitative data into a classification scheme. In order to satisfy scientific standards, I coded the gathered material into entities, which I put into symmetric matrices. First, I used a version of a time-ordered matrix called event listing for transcribing the videotapes. Here

concrete events, sorted into categories, are arranged into time sequence (Robson, 1993, Reprint 1999)

These events were actions children showed on screen or statements they made and therefore CIs. Second, I combined the outcome of the observation sheets with these CIs in a conceptually clustered matrix, that brought

together items ‘belonging together’ (e.g. relating to same theme) (Robson, 1993, Reprint 1999)

These ‘items’ were the CIs; the ‘themes’ the categories of the classification scheme, the actual criteria.

For want of more descriptive terms, I used more or less the same terms I have used in the observation sheet. These are nevertheless children’s criteria.

Evaluation of the observation sheets and the transcribed videotapes brought the following catalogue of criteria for all examined search engines:

Table 3 Children’s evaluation criteria of search engines

Criterion	CIs in total	Examples of comments/actions
First impression	19	"Looks easy to use" (John)
Speed	10	"I like it, it’s very fast" (John)
Graphics	6	"People would take...[this search engine], because of the dog" (John); "The dog is very funny" (Anna)
Interactive features	19	"I like ‘favourite places’, they help me finding stuff" (Liam)
Links	36	"They’re not nice to look at...I don’t understand what they mean." (John)
Organisation of information on page	26	"Easy to find information here" (Rob)
Quality of results	43	"That’s not what I’ve needed...the results look strange" (Patrick)
Search option: search box	38	"’Advanced search’ is not that difficult to use" (John)
Search option: web directory	13	Fiona clicks on category ‘music’: "These results are better"
Icons	10	"I know what they mean" (Liam)
Colours	7	"I like the blue here" (Anna)
Readability	3	"The text is easy to read" (Anna)
Alternative text	5	Sean reads alternative text provided by search engine by moving mouse over image

Criteria in bold would have the required amount of CIs and be therefore valid.

One child (Tara) mentioned ‘security’ as a criterion, but only because her mother has told her that she should never give her address to anyone on the Internet. Many children asked me for help when entering search terms, which means the location of letters on the keyboard as well as spelling. One example would be Aileen: "Can I type in...like...’Nickelodeon’? Do you know how to spell that?" One child used the spell checker provided by Super Snooper accidentally, but did not understand the options given by the system. Ask Jeeves for Kids does also offer spell checking, but no one has used it in this study.

As next step I ranked these criteria after their importance to children.

Ranking of evaluation criteria of table 3

Quality of results (43)
Search option: search box (38)
Links (36)
Organisation of information on page (26)
Interactive features and First impression (both 19)
Search option: web directory (13)
Icons and Speed (both 10)

The comparison below showed if these criteria had the same significance to adults.

Comparison of criteria - children and adults

To find out the differences and similarities between children and adults in their evaluation criteria, I compared the ranking list of table 1 and the ranking list of table 2 with the ranking list of table 3. All criteria were set in order of importance with the most important ones in the first row.

I tried to highlight all similarities I had found in this comparison. Quality of results, which was children’s most important criteria, has had the same meaning as quality of information had for adults. Both the search box and the web directory should be easy to use and work, just as workability is an important criterion for adults. Links are for children an evaluation criterion as well as for adults (quality of links). Organisation of information on page is identical and therefore important for both children and adults. The speed, with which a page loads, is as important to children as it is to adults (speed of loading).

Recommendations

One intended outcome of this research project was to make some recommendations in the end. The following bar chart presents children’s choice of their favourite search engine. Each child 'voted' for one search engine and gave me an explanation why he or she has chosen it.

Five girls and one boy had chosen Yahooligans!, which was therefore the preferred search engine. The reasons were manifold:

"Shows links to click on" (web directory)
"Was working!" (site was not down)
"Good, because I could find the information I wanted here"
"Can easily find information here"
"Is very organised and has ‘favourites’ feature"
"More interesting results than in the other one"

Organisation, workability and the quality of information retrieved seemed to be most important.

Looking at navigation, this study draws the following picture:

Navigation within site took exclusively place by ‘scrolling’, i.e. using the side bar and/or ‘up arrow’ and ‘down arrow’.

Navigation within pages, however, was problematic for most children. They did not understand that using the ‘back’ button of the browser would bring them to the previously visited page.

Navigation buttons, such as ‘ask’ or ‘search’ were used frequently, but provided some problems at the beginning, too. Children simply forgot to hit the ‘search’ button, and therefore no action took place. These were, however, only minor irritations and probably a result of children’s first interaction with search engines.

Nevertheless, there is room for improvement.

The following list shows ideas children mentioned during the sessions. They hold by no means scientifically grounded data, but are only suggestions. I am nevertheless convinced that they could be valuable for designers since these statements were made ‘in action’. This means suggestions from members of the intended target group who do actually use these search engines.

The following were some ideas of improvement from the participants:

‘Check your email’ feature for Yahooligans!, just as Yahoo provides for its adult clientele (John)
Easy to find information: no need to type much into search box (Sean)

[Remark: Both KidsClick! and Yahooligans! do now provide a pull-down menu of recently used search terms, which can be clicked on and transferred into search box for own search]

Navigation in categories of Yahooligans! was confusing for a girl. Claire got lost when she wanted to go back to the Yahooligans! home page and did not hit ‘back’ (browser), but ‘back’ provided by the search engine and placed close by the categories. She ended 'back' somewhere within the subject categories. Her idea was renaming this ‘back’ button by adding some more information to it, such as ‘back to categories’.

One phenomenon I observed during these sessions was that nearly all of the participants had difficulties in understanding how a search box works. The moment the search engine appeared on screen, they started typing without having clicked first into the white space of the search box. Obviously, the search box remained blank, and no results were retrieved. All of them, however, used it correctly the second time. This could be improved by explaining the meaning in simple words: e.g. 'click into this box and then type in'.

With regards to suggestions on how to include search engines for children into the Internet service of both public libraries and schools, I would recommend to provide text links as well as image links. Although it seemed that children preferred images over text links, both options should be available. Children could then distinguish between different search engines by reading the name and, at the same time, choose the way of connection they like more.

Moreover, children should be involved in the design of search engines. Other research projects, such as Start with the Child by CILIP Council, agree on the importance of integrating children's ideas into library services. They have

identified the creative use of ICT, and the extent to which libraries use it to deliver services, as benchmarks of libraries' effectiveness and responsivity to young people's needs. (Douglas, 2002)

An involvement of children in the design and the deliverance of services, especially of young people with special educational needs, would be a powerful strategy, which recognises and values diversity and promotes social inclusion.

Summary

Research in two public libraries with 15 participants aged between eight and twelve showed that children have their own evaluation criteria with regards to search engines for children. Observation of children’s interaction with these search engines provided data which was coded into matrices and evaluated. Comparison of children’s and adults’ criteria showed similarities and differences. Generally speaking, criteria on both sides did not differ much, but the order of importance did. Whereas adults considered ‘speed of loading’ respectively ‘indication of updates/revisions’ for being most important, children in this study favoured ‘quality of results’. It would be interesting to see if research projects with children who have experience in using search engines provided similar results. This is certainly a field for further investigation.

Furthermore, several recommendations were made, such as how to improve the examined search engines for children.

References

Ask Jeeves for Kids (no year) Ask Jeeves for Kids. http://www.ajkids.com (11 Jan. 2000 and 22 June 2000).
Berkeley Digital Library SunSITE (2000) Simple web indexing system for humans - enhanced. SWISH-Enhanced. http://sunsite.berkeley.edu/SWISH-E/ (22 June 2000)
Bilal, D. (2000) "Children’s use of the Yahooligans! web search engine: 1. cognitive, physical, and affective behaviors on fact-based search tasks" Journal of the American Society for Information Science, 51 (7), 646-665.
Branch, J. L. (2000) Th trouble with think alouds: generating data using concurrent verbal protocols. CAIS 2000: Dimensions of a Global Information Science; Canadian Association for Information Science, Proceedings of the 28^th Annual Conference. http://www.slis.ualberta.ca/cais2000/branch.htm (8 June 2000).
Brouwer-Janse, M. D. et al. (1997) "User interfaces for young and old" Interactions: New visions of human-computer interaction, IV (2), 34-46.
Chu, H. and Rosenthal, M. (1996) Search engines for the world wide web: a comparative study and evaluation methodology. ASIS 1996 Annual Conference Proceedings, October 19 - 24, 1996. http://www.asis.org/annual-96/ElectronicProceedings/chu.html (27 Jan. 2000).
Cooke, A. (1999) A guide to finding quality information on the Internet: selection and evaluation strategies. London: Library Association Publishing.
Douglas, J. (2002) "Start with the child" Library and Information Update, 1 (9), 36-37.
Dublin City Public Libraries (no year) Ballyfermot branch. http://www.iol.ie/resource/dubcitylib/xx.htm (22 Aug. 2000).
Fisher, S. and Oulton, T. (1999) "The critical incident technique in library and information management research" Education for Information, 17, 113-125.
Flanagan, J.C. (1954) "The critical incident technique" Psychological Bulletin, 51 (4), 327-358.
Hirsh, S. G. (1999) "Children’s relevance criteria and information seeking on electronic resources" Journal of the American Society for Information Science, 50 (14), 1265-1283.
KidsClick! Web search for kids by librarians (1999) KidsClick! web search for kids by librarians. KidsClick! Web Search. http://sunsite.berkeley.edu/KidsClick!/search.html (11 Jan. 2000 and 22 June 2000).
Knupfer, N. N. (1994) "Computers and visual learning" In Visual literacy: a spectrum of visual learning, edited by D. M. Moore and F. M. Dwyer, pp. 208-231. Englewood Cliffs, New Jersey: Educational Technology Publications.
Large, A. and Beheshti, J. (2000) Primary school students’ reaction to the web as a classroom resource. CAIS 2000: Dimensions of a Global Information Science; Canadian Association for Information Science, Proceedings of the 28^th Annual Conference. http://www.slis.ualberta.ca/cais2000/large.htm (8 June 2000).
Large, A., Beheshti, J. & Breuleux, A. (1998) "Information seeking in a multimedia environment by primary school students" Library and Information Science Research, 20 (4), 343-376.
Leh, A. S. C. (1999) "Understanding children through visual images on their webpages", 8p. Research Report. Available from ERIC (ED 428744), in microfiche.
Nielsen, J. (2000) Is navigation useful? Jakob Nielson’s Alertbox, January 9, 2000. http://www.useit.com/alertbox/20000109.html (14 Apr. 2000).
Powell, R. R. (1997) Basic research methods for librarians. 3rd. edition. Greenwich: Ablex.
Robson, C. (1993) Real world research: a resource for social scientists and practitioner-researchers. Reprint 1999. Oxford: Blackwell.
Shirey, D. (1971) "Critical incident technique" In Encyclopedia of Library and Information Science, edited by A. Kent and H. Lancour, pp. 286-291. New York: Dekker.
Soloway, E. et al. (2000) "K-12 and the Internet" Communications of the ACM, 43 (1), 19-23.
Super Snooper (no year) Super Snooper. http://supersnooper.com (11 Jan. 2000 and 22 June 2000). Please note that this search engine is not available anymore (December 2002)!
Wilson, T. D. (2000) "Recent trends in user studies: action research and qualitative methods". Information Research, 5(3). http://www.shef.ac.uk/~is/publications/infres/paper76.html (19 Apr. 2002).
Yahoo! (2000) Yahoo!. Found there 'site match'. http://www.yahoo.com (9 Sept. 2000).
Yahooligans! the Web guide for Kids (2000) Yahooligans!. http://www.yahooligans.com (11 Jan. 2000 and 22 June 2000)
Zelacom Electronic Publishing (2000) Zelacom electronic publishing. http://www.zelacom.com (22 June 2000).