The purpose of this document is to provide you with background information on search engine technology and some Tips on how to get your Web site to appear on the result pages of search engines and directories. We are by no means promising any miracles. However, this information will help you better understand search engines and directories and will hopefully serve to put you in a better position than you currently are in now.
What is a search engine?
Search engines utilize indexing software agents often called robots or spiders. These agents are programmed to constantly “crawl” the Web in search of new or updated pages. They will essentially go from URL to URL until they have visited every Web site on the Internet.
When visiting a Web site, an agent will record the full text of every page (home and sub-pages) within the site. It will then continue on to visit all external links. Following these external links is how search engines are able to find your site regardless of whether or not you register your URL with them. Submitting your URL, however, does speed up the process. It notifies an agent to visit and index your site instead of waiting for it to eventually locate you through one of your external links.
Robots will then revisit your site periodically to refresh the recorded information. The revisiting of links is the reason why some search engines don’t require you to inform them of dead links. Eventually, their robot would try unsuccessfully to update the information on a dead link and realize it no longer exists.
Finally, an easy way to tell whether a Web index is a search engine as opposed to another type of directory is by the information it requires when adding your URL. A true search engine will only need the Web address. The indexing agent takes care of the rest.
How do search engines differ from directories, announcement sites, and guides on the Web?
The main difference between a search engine and a general directory is that a directory will not list your URL if you do not register it with them. They do not make use of indexing software agents and so have no way of knowing it’s out there. As a result, their registration form will be considerably longer than just your URL. Directories are usually subdivided into categories and you have to submit your URL under the most appropriate heading.
The explosion of sites being added daily to the Web has created a need for announcement sites that track all of the new sites that join the Internet. Announcement sites are not only useful for Webmasters and marketers to kick-off their online promotion campaign, but also for users to keep current on what’s happening with the WWW. Depending on the announcement site, different Internet documents can be announced — new web pages, new articles as well as new resources. The time period policy also differs from announcement site to announcement site, but all “announcements” are posted for a temporary period of time. Once removed from the What’s New section, most announcement sites archive these pages so users can continue to access them.
Guides & Cool Sites:
Guides are quickly becoming an important source for finding interesting and useful sites on the Web. In general, Guides review and rate only a small percentage of all sites submitted. Therefore, make sure your site is “rate-worthy” before posting. Most of the guides allow reviewed sites to use their special icons as a sign of quality. Cool sites usually select one new Web site every day. Getting selected as a cool site will attract high traffic, but that traffic will usually only be experienced for a temporary period of time.
General Tips for Getting Listed in Search Engines
Each search engine looks at different elements of your page, therefore we highly recommend implementing as many of these Tips as possible.
A. Use keywords in the <TITLE> of your document making it as descriptive as possible. When visiting your site, an agent will go first to the <TITLE> tag. For clarification purposes, the <TITLE> tag is what a browser will display in its title bar and is not simply the first line of HTML that shows up on your page. (Although your first words of introductory text should be descriptive as well). Search engines will display the text located between the <TITLE> tags when your web page is listed in a search. By making your <TITLE> descriptive, you’ll be better off than those who only have keywords within the text of their page. It will also be helpful when people bookmark your web site. If a more descriptive name appears in a person’s hotlist, it will be easier to find your site at a later date.
For example, instead of using <TITLE> Suncorp </TITLE> as the title of Suncorp’s home page, <TITLE> Suncorp: Tanning Supplier </TITLE> would be much more descriptive. It would also place greater emphasis or relevancy on “Tanning Supplier” when calculating keywords.
B. Descriptive Page Text Search engines assign greater relevancy to text located at the top of a page than to text located in the middle or at the bottom of the page. The search engines assume that web page authors will present their most important information first. If your page has a main graphic at the top, you should place some descriptive text either underneath or beside the image. The search engines will index this text and assign it a high level of relevancy.
C. Use <META> tags which allow you to provide even more detail about your Web pages and thereby gain greater control over how your pages are indexed. Not all search engines make use of <META> tags, but adding these tags to your pages will make them more accessible to the search engines that do.
<META> tag codes are inserted within the <HEAD> </HEAD> tags. The basic syntax is:
<META name="description" content="a health and fitness center located in Atlanta">
This will control what appears as the summary of your Web page and will be displayed after the title of your document in the index listing. The content of the description should clearly convey what one can expect to find when linking to your site.
<META name="keywords" content="running, weight control, nutrition, aerobics, cholesterol, Georgia">
This will allow you to provide extra information about your page to the search engines without it being visible to the reader. While search engines do take these keywords into account when indexing your page, they are still going to index the entire contents of your page as many sites do not include <META> tags. Since this is the case, there is no need to be redundant. Include keywords that will not necessarily be derived when a robot visits your site. In other words, “health” and “fitness” need not be included in your list of keywords as it is part of your <TITLE>. Robots index both the description and keyword <META> tag contents as searchable words. Hence, your site will come up in a search if someone typed in “nutrition” or “health center” from your description. One way to maximize the usefulness of keywords is to incorporate singular and plural cases of words as well as active and passive verbs. For example, diet, diets, and dieting will yield similar but somewhat varying results in a search. Since you’re able through tags, why not guarantee you come up on all of them.
Do not, however, excessively repeat keywords in a keyword <META> tag as search engines may penalize you for this. At present, InfoSeek and Lycos are two such examples and others may adopt similar policies in the future. The penalty will most likely be the spider disregarding the <META> tag and extracting keywords from the content of your page- as is usually the case. However, some Submit It! users have reported being dropped from a search engine’s database and felt keyword repetition played a role in the removal of their listing.
Who should definitely make use of <META> tags?
Sites using Netscape frames:
The main HTML file contains the