How to hide web pages, folders, or your entire site from search engines: robots.txt

Good search engines (Yahoo, Google, etc.) will leave designated parts of your web site alone if you ask them nicely.

WARNING

This DOES NOT HIDE your web content from BAD searches - all of your content is still publicly-accessible on your site. In fact, by including this file, you could be pointing bad searchers in the direction of your private content.

robots.txt

Create a text file in the root folder of the site where you want to "hide" content and use robots.txt for its filename.
Within the robots.txt file, place the code that tells good searchers (A.K.A. spiders) where NOT to look.

CODE

To prevent good spiders from viewing your entire web site, use this code:

User-agent: *
Disallow: /

To prevent good spiders from viewing a specific folder on your web site, use this code:

User-agent: *
Disallow: /foldername/

User-agent: *
Disallow: /toplevelfolder/foldername/

To prevent good spiders from viewing a file on your web site, use this code:

User-agent: *
Disallow: /foldername/filename.html

For more details, see the Web Robots page.