Receive exclusive offers, coupons,
tips, and weekly updates.
Search Library:
Getting good search-engine placement
is a major goal for most webmasters.
After all, if people can’t find your site,
they can’t buy your products, support
your cause, or marvel at your innovative
page designs. But sometimes your site
isn’t ready for the world to see yet.
Maybe you’re still testing the site, or
perhaps you have a certain directory
that you don’t need to catalog. A robotexclusion
file (robots.txt) ensures that
you don’t reveal your site until you’re
good and ready.
A cautionary note: Well-behaved robots
check for a robot-exclusion file before
cataloging a site. Not all robots are well
behaved, however, so excluded pages or
directories can still appear in search results.
You should never publish content that you
don’t want to be made public unless it’s in a
secure directory.
A text file for robots?
As the name implies, robots.txt is a
plain text file—not an HTML file. You
must save it to the top level of your
website (for example, www.myURL
.com), and you must name it robots.txt.
To create the file, launch Notepad
or another word processor. Type your
commands, as we’ll describe a bit later
in this article, and save the file. If you use
a program such as Microsoft Word, be
sure to save the file in plain-text format.
The way in which you get the file
to your website’s main location differs
depending on what program you use.
For instance, you can save the file to
your hard drive and then import it into
your HTML editor. Or, if you use an
online application to create a website,
you can use its tools to separately
upload or import the file to your web’s
root directory.
...
