web robots and robots.txt : beneficial?

[mnkeybsness]

i heard a little bit about web robots and noticed that robots.txt has been attempted to be visited on my servers quite a number of times.

i'm wondering if anyone has some insight as to whether or not i would benefit from a robots.txt file... and a short overview of how and why... i'm kinda lazy today and don't feel like reading a whole lot.

 

[rainman::|:|]

it's an access policy for the web bots that crawl through pages, looking for data or indexing (like google). you can specify anything that you don't wish the robot to view or index.

http://www.robotstxt.org/wc/norobots.html

Go down to Examples to start with, it's a clearer understanding--

paul

 

[Rower_CPU]

I've done some research on them but never felt compelled enough to do any 'bot filtering on my sites.

Now, if bandwidth or a rogue 'bot hitting your site too often were more of a concern, then I'd implement something.

 

[arson]

I use one on a site I maintain because I wanted some pages indexed that were only accessible through a javascript "jump-menu". They work, but don't count on them for security purposes (keeping bots out of cerrtain directories). The bots need to be written to read and follow the robots.txt file's instructions. Any malicious crawler/bot will just ignore its instructions.