Web Design Manual
Robots Exclusion Standard to disallow web pages and
folders from being indexed by spiders.
If you don't want your entire site to be indexed, we
strongly advise that you take advantage of the Robots Exclusion Standard by setting up a
/robots.txt file. It only takes a minute, and gives you complete control over what
fraction of your site is indexed.
The file looks like:
User-agent: * # (directed to all spiders)
Disallow: /cgi-bin/sources
Disallow: /access_stats
Disallow: /cafeteria/lunch_menus/
Any URL matching one of these patterns will be ignored by robots visiting your site.
Instructions:
Place a text file named robots.txt by the index (home) .html page. (robots.txt)
Write the following (without a blank line at top of file).
Example:
User-agent: * #
Disallow: /account/
Disallow: /admin/
Disallow: /client/
Disallow: /collector/
Disallow: /email.asp
Disallow: /emailsent.asp
Disallow: /log_back.asp
Disallow: /log_in.asp
Disallow: /log_back.asp
Disallow: /pass.asp
Disallow: /pass_back.asp
Disallow: /reg.asp
Disallow: /regsent.asp
Web Design Manual
|
