Getting good search-engine placement
is a major goal for most webmasters.
After all, if people can’t find your site,
they can’t buy your products, support
your cause, or marvel at your innovative
page designs. But sometimes your site
isn’t ready for the world to see yet.
Maybe you’re still testing the site, or
perhaps you have a certain directory
that you don’t need to catalog. A robotexclusion
file (robots.txt) ensures that
you don’t reveal your site until you’re
good and ready.
A cautionary note: Well-behaved robots check for a robot-exclusion file before cataloging a site. Not all robots are well behaved, however, so excluded pages or directories can still appear in search results. You should never publish content that you don’t want to be made public unless it’s in a secure directory.
A text file for robots?
As the name implies, robots.txt is a plain text file—not an HTML file. You must save it to the top level of your website (for example, www.myURL .com), and you must name it robots.txt. To create the file, launch Notepad or another word processor. Type your commands, as we’ll describe a bit later in this article, and save the file. If you use a program such as Microsoft Word, be sure to save the file in plain-text format.
The way in which you get the file to your website’s main location differs depending on what program you use. For instance, you can save the file to your hard drive and then import it into your HTML editor. Or, if you use an online application to create a website, you can use its tools to separately upload or import the file to your web’s root directory. ...