Children’s Evaluation Criteria of Search Engines
Dipl.-Bibl. Eva Hornung, MLIS
Search engines on the Internet are widely used as a starting point for Information Retrieval. Children are increasingly users of the Internet and therefore also of search engines. Some companies and institutions have responded to that demand and have created search engines with a young target group in mind. Still, there is no study to date on how children themselves judge these search features with regards to design and functionality without having given them pre-defined "adult" criteria. This presentation, which derived from a master’s thesis completed in 2000, tries to shed some light into the dark.
This article starts with an introduction to related studies in the fields of Web Design and Evaluation of Information Resources in general. The next part exploits evaluation criteria set by adults, which were published in the form of evaluation sheets. After the research questions, an introductory to the research project is provided, which was conducted in two public libraries in Dublin using the Critical Incident Technique and observational methods. The data collected shows that, when compared with adults’ criteria, there are significant differences, but also remarkable resemblance in the judgements. The last part sees a summary and recommendations on interface design issues of search engines for children.
The purpose of this study is to understand children’s evaluation criteria concerning the design and navigation of children’s search engines. Moreover, their criteria will be compared with criteria set by adults in order to see if there are any similarities or differences.
This research topic required a review of literature in the areas of web design in general, web design for children and the evaluation of electronic information sources. Furthermore, the research fields of Information Retrieval and Information Literacy were examined.
Knupfer (1994) recommends the following several characteristics for a good screen design in general:
I am convinced that an attractive design of the search interface is important for children. As De Vries (Brouwer-Janse, et al., 1997) has summarised it, talking about products for children in general,
designing for children means designing for fun.The opening page of Yahooligans!, for example, offers lots of graphics in order to attract young users. In this case, the use of graphics adds value to the source. Sometimes, however, this habit only slows access speed down (Cooke, 1999, p. 75).
Looking at the so-called ‘media generation’, De Vries (Brouwer-Janse, et al., 1997) states that
it is relatively easy to gain access to the information that is available to children. It is not easy, however, to know how they perceive the stimuli, unless we enter their world and discuss topics with them.This is the idea behind this project.
Design of web sites for children
Leh (1999) provides a key article on design of web sites by children. She observed 12 children (8-13 years) while they were searching the Internet and creating their own web sites within an extracurricular Internet course. She looked at how children were evaluating web sites, including their own.
In short, these were her findings with regards to outside web sites children have used for their own creations:
Navigation is also a very important issue. There are two means of navigation: first the navigation bars provided by the browser and second all navigation aids within a web page or search engine. In our case, the navigation means of the search engines is of special interest.
Nielsen (2000) distinguishes between ‘structural’ and ‘local navigation’. The first way of navigation should provide links to all levels within a hierarchical order of related pages of a web site. This is very important in cases where users come to one page through a search engine and need a link to an overview page. The second way consists of local links to related content which should enable users to find similar or related pages. One area of investigation in this study will be navigation facilities in search engines for children.
Much research has been conducted to elucidate interface design principles, although rarely with children in mind as users (Large, et al., 1998).
The need for evaluation
There has been research on the comparison of search engines for adults. An area of interest in a study by Chu and Rosenthal (1996) was ‘user effort’, which means interface and documentation of a search engine. The authors pointed out that
users will not use a search engine unless they are comfortable with its interface, and able to read and comprehend its documentation when consulted.Not only the user, however, profited from this evaluation. One of the hopes of the authors was to
help Web search engine developers design even better ones for the Internet community.
Hirsh (1999) draws the attention of further investigators on the fact that children should receive more training in evaluating electronic resources. They
need to understand how information is placed on the Internet and how to verify the accuracy of the information they find there.She states a deficiency in age-appropriate interfaces for children which, if available, could
provide greater assistance to children in identifying the authority of the information.
Large and Beheshti (2000) found out that despite provided training students in their project still encountered severe problems while searching the Internet. One suggestion was
to develop interfaces that offer much more help in dealing with such problems.
Finally, looking at the quality of web search tools, schools should not hesitate to put pressure on search engines designers. Only few search engines engage people to review web sites. It is high time to change that since
schools become more sophisticated with their Internet use, they will demand more than filtering, and they will put their subscription dollars behind that demand (Soloway, et al., 2000, p.20).
Evaluation Criteria - What adults think
Exploring adult criteria, I came across two different types of means of evaluation: evaluation sheets, which had been developed with children in mind and evaluation guides for adults. Both were examined and sets of criteria were singled out.
Criteria Of Evaluation Sheets For Children
In the following, I exploited some evaluation sheets for children developed by adults. All these evaluation forms were available online, could be printed and used in a school or library. This enumeration, however, was not by means complete, but a selection.
These guides should help children judging Internet information, but were not set for search engines in particular. In order to establish a 'hit list', I created a table and put all criteria into categories.
Table 1 continued
As the next step, I wanted to find the more important criteria by looking at the matches. Which were the criteria all evaluation sheets agree upon (four matches)? Which were the ones most agree upon (three matches)?
Ranking of evaluation criteria of table 1
This was the final set of criteria of the evaluation sheets for children. They will finally be compared with the evaluation criteria of children (see table 4).
Evaluation Criteria of Adults
The last paragraph saw a listing of evaluation sheets developed by adults and aimed at a young target group. The following table brings together the ideas of the adult information specialists which are experts in the field of web site evaluation. I found their articles recommended and referred to during my literature review. Again, this listing offers only a selection. It shows adult evaluation criteria, which are supposed to be used by adults.
Table 2 continued
Table 2 continued
Table 2 continued
Looking at these criteria, the ranking shows which ones are mentioned most:
Ranking of evaluation criteria of table 2
Indication of updates/revisions
This set of adults’ criteria will also be compared with children’s criteria (see table 4).
A review of literature in the fields of Web Design and Adult’s Evaluation Criteria leaves the following research questions to be answered:
Two more aims of this project are:
The Search Engines
In the following, I will introduce the search engines for children I have picked for this study. For my selection, I rely on Bilal’s (2000) statement that there are only three search engines designed for children currently available on the Web: ‘Yahooligans!’, ‘Ask Jeeves for Kids’ and ‘Super Snooper’. Additionally, I will include a web search tool for children I found on the Internet. KidsClick! is designed by librarians and could therefore be interesting to compare with the above-mentioned search engines and their commercial background.
KidsClick! web search for kids by librarians
This search engine or ‘web guide’ was created by librarians of Ramapo Catskill Library System in 1997. Their intention was to guide young users to what they consider for being valuable and age appropriate web sites for entertainment or enlightenment.
They have a catalogue of criteria for inclusion of web sites, but insist on the fact that KidsClick! is not an Internet filter. The target group are children of grades K-7, which means children from kindergarten up to 7th grade.
Figure 1: Snapshot of ‘KidsClick!
The interface of KidsClick! was designed by Zelacom, using graphic design elements (KidsClick! Web search for kids by librarians, 1999). Zelacom themselves described the outcome of the project, which was completed by April 1998, as a creation of
kid-friendly graphics [which] enhance navigation and contribute to the inquisitive, adventurous tone of the site (Zelacom Electronic Publishing, 2000). The graphical interface of this search engine was the result of teamwork of graphic designers and librarians.
The indexing system lying behind KidsClick! is called ‘SWISH-Enhanced’, which is an enhanced version of SWISH (Berkeley Digital Library SunSITE, 2000). The most important feature of this software is that it allows limited searches to different HTML tags, such as the META tag, the TITLE tag or comments.
KidsClick!gives three search options: the search engine, a possibility of searching a subject by its first letter or a web directory. The search engine is divided into a simple and an advanced search. The simple search let users look for one or more words in ‘all fields’ (which, in my understanding, means all HTML tags) or in ‘web address only’. Obviously, the advanced search modus allows the search to be more specific. Here, the user can decide to search in all fields, only for subjects, only for titles or descriptions.
He or she can limit results by choosing the reading level he or she has (all reading levels, up to grade six or over grade seven) or by indicating how many pictures should be included in relevant sites (some or no pictures, some pictures, many pictures). Another option in advanced search is the possibility to search for URLs (Uniform Resource Locator). The user does only have to know parts of the address he or she is looking for, e.g. typing in ‘sunsite’ would lead to the same result as ‘sunsite.berkeley.edu’ or ‘http://sunsite.berkeley.edu/’ would.
Searching subjects by letter is another feature of KidsClick!. Here, if the user clicks on the letter of his or her choice, a window opens up where subjects are listed in alphabetical order. After clicking on one link, the user is led to the data pool.
The web directory is divided into fields of interests, such as ‘Facts/Current Events’, ‘Science & Math’, ‘Weird & Mysterious’ or ‘The Arts’. Each category has sub-categories. ‘Computers/The Internet’, for example, is divided into ‘Software’, ‘Programming’, ‘Internet’, ‘Key-Pals’, ‘Computer Games’, ‘Online Games’ and ‘more’. Clicking on one of these sub-categories leads the user to the data pool. This consists of recommended links together with small abstracts, a note of the amount of illustrations this site has, the reading level which is necessary here and the subject category.
Yahooligans! the Web guide for Kids
Yahooligans!is a Internet directory for children aged 7-12 and a trademark of Yahoo! Inc., just as Yahoo!, the search engine for adults. Its categories are searchable by using a keyword or browsable. According to Yahooligans!, each site has been checked by an experienced educator who evaluates and categorises it. Yahooligans! is updated daily. Sites were only removed when they go down, or the content is not longer appropriate for children (Yahooligans!, 2000).
Figure 2: Snapshot of ‘Yahooligans!’
Yahooligans!categories consist of sites belonging to the same topic. These categories are displayed in bold. Clicking on one category leads the user to another listing of sites, which are listed in plain text. If a user clicks on one of these sites, he or she is outside Yahooligans!. The category path on top of the page shows the hierarchical tree and should facilitate navigation around the web directory. To go back to the main category, for example, a user only has to click on the very first link in the category path (Yahooligans!, 2000).
Examples of main categories would be: ‘Around the world’, which includes sub-categories such as ‘Countries’, ‘Food’ or ‘Holidays’, or ‘School Bell’ with its sub-categories ‘Language Arts’, ‘Math’ and ‘Social Studies’.
Yahooligans! has a rating system that shows users sites, that are "particularly cool" (Yahooligans!, 2000), by putting symbol consisting of stylised sunglasses next to the link.
Ask Jeeves for Kids
Ask Jeeves was incorporated in 1996 and is based in Berkeley, California, named after the idea of a British servant in literature. The founders also developed Ask Jeeves for Kids. The illustrations are made by Marcos Sorensen.
This search engine uses natural language processing. Queries were matched against thousands of question templates in the knowledge base, which is created by research staff. User can enter questions in plain English and choose then from a list of matched questions. After having selected the closest match, the user will be directed to a site that was pre-selected by the staff of Ask Jeeves for Kids. Using this system is, according to them,
like having the world’s most knowledgeable internet [sic] librarian at your side. (Ask Jeeves for Kids, no year)Unfortunately, there are no indications of how often the database will be updated or for which target group it is created. According to Yahoo! (2000), Ask Jeeves for Kids is a
natural language answering site for [bold font by Yahoo!, E. H.] children 7-14.
Figure 3: Snapshot of ‘Ask Jeeves for Kids’
On the one hand, Ask Jeeves for Kids sees itself as an educational web site that offers help for homework and school projects. On the other hand, it provides fun games and activities. Selection of sites is very strict and has to meet certain requirements: quality and depth of content and safety. This search engine, however, has no in-built filtering software. Ask Jeeves for Kids offers metasearches of other sites for children (Ask Jeeves for Kids, no year).
Since Ask Jeeves for Kids has a commercial background, it displays advertising banners. They can be connected to specific categories of questions. There is a standard format for ads: 468 x 60, in GIF or JPEG format (Ask Jeeves for Kids, no year).
Super Snooper was officially announced in October 1997 and is been recognised widely since January 1998 (Super Snooper, no year). It is not said who the provider is. This free-of-charge search engine for children claims to provide only material that does not
contain information of pornographic, violent or hatred-related nature.
The database, which lies behind Super Snooper, is, according to the provider, based on
an automated screening of sites that we find on the Internet, cataloging them based upon the type of information that they claim to provide, as contained in their site names, keywords and descriptions in the body of their home pages.
The idea behind that is to catalogue and review every Internet site which has a registered domain name; with over 1.5 million sites being included so far. Super Snooper favours listing sites from English-speaking countries.
The search options are simple: the user can enter one or more words, separating them by commas or spaces. To narrow a search, he or she has to type in more words. The providers claim to reduce the number of duplications of hits, since sites with the same description and title will not be displayed. Also, they try to present relevant sites in a sequence of importance. So far, Super Snooper has only one advertising banner on its site.
Figure 4: Snapshot of ‘Super Snooper’
Since the best strategy for exploratory purposes is, according to Robson (1993, Reprint 1999), a case study, I conducted two for this research project in two public libraries. The intention was to explore children’s criteria for search engines.
At the same time, I wanted to apply the so-called ‘Critical Incident Technique’ (CIT), which
is a well-tried method of qualitative research...[, that]...allows researchers to investigate a wide range of topics and generate theories using a variety of data collection tools with quite small samples, at least at an exploratory level. (Fisher and Oulton, 1999)The practical implications were described by Fisher and Oulton (1999; italic by me, E. H.):
Flanagan (1954) recommends, in order to determine whether or not additional incidents are needed, to:
[keep] a running account on the number of new critical behaviors added to the classification system with each additional 100 incidents. For most purposes, it can be considered that adequate coverage has been achieved when the addition of 100 critical incidents to the sample adds only two or three critical behaviors.
He defines ‘critical’ and ‘incident’:
By an incident is meant any observable human activity that is sufficiently complete in itself to permit inferences and predictions in itself to be made about the person performing the act. To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.
For my study, a 'critical incident' was a statement a child made or an action he or she showed on the screen. I then set up a classification system by putting these incidents into categories. These categories were the actual evaluation criteria.
Since I was only working with a small number of participants, I put the number of required CIs on a lower level. Looking at how many CIs would be needed in order to form a category (a criterion), I considered ten CIs for being sufficient.
Observing the children, I asked a few introductory questions in order to stimulate their talking about what they were doing and their comments on the search engines they were using. Data collection included recording verbal protocols of them. This technique is also known as ‘Think Aloud’, widely used in cognitive psychology and education.
Problems could occur where
the task involves a high cognitive load, when the information is difficult to verbalize because of its form, i.e., visual data, or when the process are automatic for the participants. (Branch, 2000)This could have been the case in this study since children might have had problems in talking about their actions while doing them. Furthermore, visual data was here examined. Also, some movements might have been automatic for the children. Therefore, I used a video camera to film children’s actions and the screens.
Observational methods are best suited for describing and understanding behaviour as it occurs. (Powell, 1997)I wanted to compare the actions of children while using the search engines with what they told me about their evaluation criteria: why did they use this function? How did they judge them? Did they like the colours? etc.
Unstructured observation or participant observation,
an essentially qualitative style originally rooted in the work of anthropologists (Robson, 1993, Reprint 1999)however, has some drawbacks. One is that
the primary data are the interpretations by the observer of what is going on around her [, which means that] the observer is [italic by author, E.H.] the research instrument, and hence great sensitivity and personal skills are necessary for worthwhile data. (Robson, 1993, Reprint 1999)
Another problem concerning ‘data’ is here that
to speak of ‘data’ in qualitative research may be misleading: the word ‘evidence’ may be more appropriate since the relevance of the evidence to the research objectives involves a greater exercise of judgement than in quantitative research (Wilson, 2000)To overcome some of these limitations, however, I used an observation sheet in order to collect some precise quantitative data.
Still, there were some limitations to this study and the collected data. First, since these were case studies, I was only looking at a limited number of people. I had to conduct research in two libraries in order to get enough participants (summer holidays). Second, I tried to be neutral, but there is always a danger of bias. The CIT had also drawbacks, because
the catagorization phase of the technique should be considered very subjective and difficult. (Shirey, 1971)
Location and participants
Data collection took place in two public libraries, since schools were closed for summer. There were not enough participants in the first library, so I had to go to the second. One of the participating libraries was
first library in Europe to be involved in the Libraries Online! initiative. (Dublin City Public Libraries, no year)
Each observation session lasted 20-30 minutes per child. The starting point for all children was the start page I had had created, which included links to all four search engines. They were free to choose, which search engine they wanted to use and in which order.
Figure 5: Web site as starting point
Participants were fifteen children aged between eight and twelve, five boys and ten girls. I was aware of the fact that the age difference was very big and probably meant a big difference in cognitive abilities and Internet expertise. The examined search engines, however, had been created for this target group. So they should have been convenient for all children of this study.
Figure 6: Age of participants
All participants in both libraries were novice users of these search engines, except for one child, who had used Ask Jeeves for Kids before.
Parents of all children had given their permission in advance. Also, in order to protect children’s privacy, all names have been changed.
It is not easy to put qualitative data into a classification scheme. In order to satisfy scientific standards, I coded the gathered material into entities, which I put into symmetric matrices. First, I used a version of a time-ordered matrix called event listing for transcribing the videotapes. Here
concrete events, sorted into categories, are arranged into time sequence (Robson, 1993, Reprint 1999)These events were actions children showed on screen or statements they made and therefore CIs. Second, I combined the outcome of the observation sheets with these CIs in a conceptually clustered matrix, that brought
together items ‘belonging together’ (e.g. relating to same theme) (Robson, 1993, Reprint 1999)These ‘items’ were the CIs; the ‘themes’ the categories of the classification scheme, the actual criteria.
For want of more descriptive terms, I used more or less the same terms I have used in the observation sheet. These are nevertheless children’s criteria.
Evaluation of the observation sheets and the transcribed videotapes brought the following catalogue of criteria for all examined search engines:
Table 3 Children’s evaluation criteria of search engines
CIs in total
Examples of comments/actions
"Looks easy to use" (John)
"I like it, it’s very fast" (John)
"People would take...[this search engine], because of the dog" (John); "The dog is very funny" (Anna)
"I like ‘favourite places’, they help me finding stuff" (Liam)
"They’re not nice to look at...I don’t understand what they mean." (John)
Organisation of information on page
"Easy to find information here" (Rob)
Quality of results
"That’s not what I’ve needed...the results look strange" (Patrick)
Search option: search box
"’Advanced search’ is not that difficult to use" (John)
Search option: web directory
Fiona clicks on category ‘music’: "These results are better"
"I know what they mean" (Liam)
"I like the blue here" (Anna)
"The text is easy to read" (Anna)
Sean reads alternative text provided by search engine by moving mouse over image
Criteria in bold would have the required amount of CIs and be therefore valid.
One child (Tara) mentioned ‘security’ as a criterion, but only because her mother has told her that she should never give her address to anyone on the Internet. Many children asked me for help when entering search terms, which means the location of letters on the keyboard as well as spelling. One example would be Aileen: "Can I type in...like...’Nickelodeon’? Do you know how to spell that?" One child used the spell checker provided by Super Snooper accidentally, but did not understand the options given by the system. Ask Jeeves for Kids does also offer spell checking, but no one has used it in this study.
As next step I ranked these criteria after their importance to children.
Ranking of evaluation criteria of table 3
The comparison below showed if these criteria had the same significance to adults.
Comparison of criteria - children and adults
To find out the differences and similarities between children and adults in their evaluation criteria, I compared the ranking list of table 1 and the ranking list of table 2 with the ranking list of table 3. All criteria were set in order of importance with the most important ones in the first row.
I tried to highlight all similarities I had found in this comparison. Quality of results, which was children’s most important criteria, has had the same meaning as quality of information had for adults. Both the search box and the web directory should be easy to use and work, just as workability is an important criterion for adults. Links are for children an evaluation criterion as well as for adults (quality of links). Organisation of information on page is identical and therefore important for both children and adults. The speed, with which a page loads, is as important to children as it is to adults (speed of loading).
One intended outcome of this research project was to make some recommendations in the end. The following bar chart presents children’s choice of their favourite search engine. Each child 'voted' for one search engine and gave me an explanation why he or she has chosen it.
Five girls and one boy had chosen Yahooligans!, which was therefore the preferred search engine. The reasons were manifold:
Organisation, workability and the quality of information retrieved seemed to be most important.
Looking at navigation, this study draws the following picture:
Navigation within sitetook exclusively place by ‘scrolling’, i.e. using the side bar and/or ‘up arrow’ and ‘down arrow’.
Navigation within pages, however, was problematic for most children. They did not understand that using the ‘back’ button of the browser would bring them to the previously visited page.
Navigation buttons, such as ‘ask’ or ‘search’ were used frequently, but provided some problems at the beginning, too. Children simply forgot to hit the ‘search’ button, and therefore no action took place. These were, however, only minor irritations and probably a result of children’s first interaction with search engines.
Nevertheless, there is room for improvement.
The following list shows ideas children mentioned during the sessions. They hold by no means scientifically grounded data, but are only suggestions. I am nevertheless convinced that they could be valuable for designers since these statements were made ‘in action’. This means suggestions from members of the intended target group who do actually use these search engines.
The following were some ideas of improvement from the participants:
[Remark: Both KidsClick! and Yahooligans! do now provide a pull-down menu of recently used search terms, which can be clicked on and transferred into search box for own search]
One phenomenon I observed during these sessions was that nearly all of the participants had difficulties in understanding how a search box works. The moment the search engine appeared on screen, they started typing without having clicked first into the white space of the search box. Obviously, the search box remained blank, and no results were retrieved. All of them, however, used it correctly the second time. This could be improved by explaining the meaning in simple words: e.g. 'click into this box and then type in'.
With regards to suggestions on how to include search engines for children into the Internet service of both public libraries and schools, I would recommend to provide text links as well as image links. Although it seemed that children preferred images over text links, both options should be available. Children could then distinguish between different search engines by reading the name and, at the same time, choose the way of connection they like more.
Moreover, children should be involved in the design of search engines. Other research projects, such as Start with the Child by CILIP Council, agree on the importance of integrating children's ideas into library services. They have
identified the creative use of ICT, and the extent to which libraries use it to deliver services, as benchmarks of libraries' effectiveness and responsivity to young people's needs. (Douglas, 2002)
An involvement of children in the design and the deliverance of services, especially of young people with special educational needs, would be a powerful strategy, which recognises and values diversity and promotes social inclusion.
Research in two public libraries with 15 participants aged between eight and twelve showed that children have their own evaluation criteria with regards to search engines for children. Observation of children’s interaction with these search engines provided data which was coded into matrices and evaluated. Comparison of children’s and adults’ criteria showed similarities and differences. Generally speaking, criteria on both sides did not differ much, but the order of importance did. Whereas adults considered ‘speed of loading’ respectively ‘indication of updates/revisions’ for being most important, children in this study favoured ‘quality of results’. It would be interesting to see if research projects with children who have experience in using search engines provided similar results. This is certainly a field for further investigation.
Furthermore, several recommendations were made, such as how to improve the examined search engines for children.