Description: The robots.txt file is the first thing search engine "spiders" look for when indexing a website. The robots.txt file tells search engine spiders (robots) which files and/or directories they are NOT allowed to index. This helps to prevent incomplete site indexing as well as prevent exposing the files and directories that you don't want "all over the world wide web". It's fast, easy and FREE.
Resource Specification
Platform:
Linux - Unix - Windows
Price:
Free
Date Added:
31/10/2006
Last Updated:
31/10/2006
Author:
John Kline
Average Visitor Rating:5 (out of 5) Number of Ratings: 1 Votes
Visit Easy Robots.txt Generator `s Website
Refer
Write a Review!
Report This Link
Description: The robots.txt file is the first thing search engine "spiders" look for when indexing a website. The robots.txt file tells search engine spiders (robots) which files and/or directories they are NOT allowed to index. This helps to prevent incomplete site indexing as well as prevent exposing the files and directories that you don't want "all over the world wide web". It's fast, easy and FREE.
5 (out of 5)
