Directories can eclipse search engines (New searching tricks)
Philadelphia Inquirer 2/24/00
Joyce Kasman Valenza
What would happen if, just for a day or two, Yahoo really disappeared from the face of the Web?
Students all over would be forced to try a new tool.
Yahoo-dependency, Excite-dependency or AOL Search-dependency, are frustrating syndromes, but they are curable. These are fine tools, but kids need to see the world beyond the one URL they remember.
Though many of us depend on just one search tool, there are many worthy new and underused tools to try.
Since we last visited search tools together the field has exploded. I sought out the best search tips of three searching experts.
They all agreed. When you are searching a general topic, start with a subject directory rather than a search engine.
“Subject directories have always had as their great strength the judgment and intellectual input of humans,” said Laura Cohen, Network Services Librarian at the University at Albany. Subject directories are collections of links organized into subject categories. Though the degree of selectivity varies, they offer selective, sometimes annotated and evaluated lists of sites. Search engines, on the other hand, rely on computer programs, often called robots or spiders, to match search terms. Cohen and I discussed the impressive directories hardly used by people outside the library world. Librarian's Index to the Internet, BUBL, WWW Virtual Library, Argus and Infomine are excellent places to begin research. And for a commercial effort Cohen recommends Open Directory, which uses around 2200 volunteer editors in 217,000 categories, and About.com, a site offering the advice of designated experts who do the “intellectual work of gleaning annotating and organizing,” in their own special topical areas. Cohen says there's a big difference between the directories maintained by libraries and universities and large commercial directories like Yahoo and Snap, which cater to general public. “Yahoo does no resource gathering,” says Cohen. “It is the passive recipient of sites that users send to it. Yahoo editors do not have time to put on everything that is sent. Wonderful sites may never get submitted or included.”
“The reason people do go to Yahoo is that humans do the work,” said Danny Sullivan, editor of SearchEngineWatch.com. “What is new is that a lot of the others are going in the same direction.” Sullivan noted many search tools have added strong directories--AOL Search, Netscape Search, and MSN Search. In fact, Lycos recently changed to become a directory. “Directories are especially good for finding an official site,” said Sullivan.
Joe Barker ,Web Search Program Coordinator in the Teaching Library at UC Berkeley, also advises that students who are just beginning their research look for guides to their subjects. He suggests they start with highly selective directories, like Argus and Scout, which specialize in collecting gateways to the subject areas. Barker also notes that search engine spiders are not particularly effective in indexing what he calls “the invisible Web,” for instance, the information maintained within databases, directories of lawyers, job postings, latest news stories. “A good directory will link to these,” said Barker.
Selected Subject Directories
About.com http://about.com
Argus Clearinghouse http://www.clearinghouse.net/
BUBL LINK http://bubl.ac.uk/link/
Infomine http://infomine.ucr.edu/
Librarian's Index to the Internet http://lii.org/
Open Directory Project http://dmoz.org/
WWW Virtual Library http://vlib.org/Overview.html
“The good news,” according to Cohen, is that “the human element has finally come to search engines.” Unlike directories, search engines are searchable databases of sites, collected by computer programs often called robots or spiders. No humans are involved in the selection, sorting, evaluating or organized of these sites. Until about a year and a half ago, search engines basically used simple term relevance ranking. They looked for the numbers of occurrences of search terms in titles, keywords, in URLs, in the document itself and how close those occurrences were. This relevance ranking approach was objective—completely unrelated to meaning. If you typed in the word “turkey,” the search engine would not distinguish among the various meanings of the word. Returns might include information about the animal, the country, a bad movie or a jerk. Cohen described a “second generation” of search engines, tools that address this problem by offering meaning-based searches.
AskJeeves allows users to pose questions in natural language and to select results among a database of similar questions. “AskJeeves is not the only search tool that will accept a natural language search,” said Sullivan. “Many other will now accept a query phrased as a question. In fact, for the inexperienced searcher, natural language works better. When people write out their questions, they generally include more words.” The search engine can perform a more specific search. “One of the biggest problems,” said Sullivan, “is that people are not specific. They'll type in `travel' when they mean `I want to book an airplane ticket online.'”
Oingo, one of my personal favorites, sorts meanings by using a lexicon described as “a rich database of words, meanings, and relationships.” My search on “cats,” would have been disastrous in a normal search engine. But Oingo's pull-down menu prompted me to select among a variety of possible meanings--domestic cats, Cats (the musical) or Computer Aided Translation software. Similarly the new Simpli http://simpli.com asks users to specify, or refine meanings, when there are confusions in meaning. Type in "java" and you are asked to choose among Java as a program language, term for coffee or Indonesian island.
Northern Light organizes results into “little blue folders” on the left side of the screen.
All the government sites, news sites, and related subject themes are parceled out into convenient and meaningful categories. In addition Northern Light indexes the over 6000 full-text sources, not otherwise available to Web searchers.
Google, another of my current favorites, is a citation search engine. Joe Barker is also a fan. “What we teach here (at Berkeley) is when to use Google and when not to use Google,” said Barker. Barker feels that students need to understand is how a search engine's ranking system works. Google is a relatively new search engine with a refreshingly clean interface and a unique ranking system. It ranks hits by link popularity. “It is incredibly accurate in bringing up valuable sites, because other people have linked to them,” said Barker. Google exploits the natural structure of the Web with its “PageRank technology.” It ranks results according to number of highly ranked pages that link to those results. Unlike many other search tools, Google prioritizes its results according to the proximity of search terms within a page and returns only those results that contain all search terms. Result lists offer a relevant preview of each hit, by excerpting the text that matches the query and highlighting the search terms.
But according to Barker, Google is far better for “turkey recipes than news about a recent earthquake in Turkey.” He advises that if students need a truly comprehensive search, which includes very current sites, they should give Fast Search, also known as All The Web, a try. Fast Search is not only fast, “it boasts a database of over 300,000,000 sites, one and a half times as big as any of the others,” said Barker.
Cohen feels that Direct Hit complements Google. Both tools “make use of the collective judgement and the behavior of other searchers,” said Cohen. Direct Hit ranks its results based on how many other searchers choose that search from their search results and how long they stay there, and scores relevance with a ranking system of one to five “people icons.”
Inference Find is a metasearch engine which ranks results by concept. It searches what it considers “all the best search engines on the Internet, merges the results, removes redundancies, and clusters the results into neat understandable groupings.” It is this clustering that is particularly helpful. SurfWax, another of Cohen's second generation search tools, calls itself “Your Source for Internet Satisfaction,” and offers visual tools for composing search criteria.” It incorporates searchers' recognition of “meaningful alternative words” to help focus a search.
Cohen noted a trend that will make the average computer using very happy. “What's happening is that the perfection of the search is beginning to be less important, and the processing is more prominent.” Search engines, according to Cohen, are using technology that mimics the indexing and conceptualizing elements of human judgement. “Before, users had to come up with the proper search terms; now the intellectual work of searching is being taken on by the search engines. The search tools are conceptualizing the search for you. They are doing more of the intellectual work.”
Second Generation Tools
AskJeeves http://www.askjeeves.com
Direct Hit http://www.directhit.com
Fast Search/Alltheweb http://www.alltheweb.com
Google http://www.google.com
Inference Find http://www.infind.com/
Northern Light http://www.northernlight.com
Oingo http://www.oingo.com
Simpli http://simpli.com
SurfWax http://www.surfwax.com
It is easy to become a better searcher.
For a clickable list of search options try my search tools page at: http://mciu.org/~spjvweb/searchtip.html
Search Tips and Tutorials
About.com Web Search http://websearch.about.com
BCK2SKOL http://kelim.jct.ac.il/b2s/
Choosing the Best Search Tool (Debbie Abilock) http://nuevaschool.org/~debbie/library/research/adviceengine.html
Finding Information on the Internet (Berkeley) http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/FindInfo.html
How to Teach the Internet http://www.unl.edu/websat/teach.html
How to Choose a Search Engine or Directory http://www.albany.edu/library/internet/choose.html
ICONnect Advanced Courses Archive http://www.ala.org/ICONN/advancedcoursesarchive.html
Information Searcher's CyberTours http://www.infosearcher.com/cybertours/
KidsClick! World of Searching http://www.worldsofsearching.org/
Learn the Net http://www.learnthenet.com/
OSLIS (Oregon Public Education Network) Search Strategies http://www.open.k12.or.us/oslis/tutorials/strategies/index.html
Scout Toolkit http://scout.cs.wisc.edu/toolkit/index.html
SearchEngineShowdown http://searchengineshowdown.com/
SearchEngineWatch http://www.searchenginewatch.com/
Search IQ http://www.infozoid.com/
Seven Steps Toward Better Searching
http://edweb.sdsu.edu/WebQuest/searching/sevensteps.html
Sink or Swim Internet Search Tools and Techniques
http://www.ouc.bc.ca/libr/connect96/search.htm
Spider's Apprentice: A Helpful Guide to Web Search Engines
http://www.monash.com/spidap.html
University of Albany How to Choose a Search Engine or Directory http://www.albany.edu/library/internet/choose.html
University of Albany Internet Tutorials http://www.albany.edu/library/internet/
Web Searching Tutorial http://www.askscott.com/tindex.html
WebTeacher Tutorial http://www.webteacher.org/macnet/indextc.html
Choose Your Search Tool! (Advice on when to use which)
Nueva Library Choose Your Best Search http://NuevaSchool.org/~debbie/library/research/adviceengine.html
How to Choose a Search Engine or Directory http://www.albany.edu/library/internet/choose.html
Subject Directory of Search Engines http://www.searchiq.com/subjects/
Fossick.com (specialist search engines and topical guides) http://www.fossick.com
Search Engines and News http://www.internets.com/
Search.com (topic-based databases) http://www.search.com
WebData http://www.webdata.com/
Lycos Searchable Databases http://dir.lycos.com/Reference/Searchable_Databases/
Back to Neverending Search