View Full Version : web robots and robots.txt : beneficial?
Jan 11, 2004, 08:56 PM
i heard a little bit about web robots and noticed that robots.txt has been attempted to be visited on my servers quite a number of times.
i'm wondering if anyone has some insight as to whether or not i would benefit from a robots.txt file... and a short overview of how and why... i'm kinda lazy today and don't feel like reading a whole lot.
Jan 11, 2004, 09:05 PM
it's an access policy for the web bots that crawl through pages, looking for data or indexing (like google). you can specify anything that you don't wish the robot to view or index.
Go down to Examples to start with, it's a clearer understanding--
Jan 12, 2004, 12:35 AM
I've done some research on them but never felt compelled enough to do any 'bot filtering on my sites.
Now, if bandwidth or a rogue 'bot hitting your site too often were more of a concern, then I'd implement something.
Jan 12, 2004, 04:45 PM