web robots and robots.txt : beneficial?

Discussion in 'Web Design and Development (archive)' started by mnkeybsness, Jan 11, 2004.

  1. macrumors 68030

    mnkeybsness

    Joined:
    Jun 25, 2001
    Location:
    Moneyapolis, Minnesota
    #1
    i heard a little bit about web robots and noticed that robots.txt has been attempted to be visited on my servers quite a number of times.

    i'm wondering if anyone has some insight as to whether or not i would benefit from a robots.txt file... and a short overview of how and why... i'm kinda lazy today and don't feel like reading a whole lot.
     
  2. macrumors 603

    rainman::|:|

    Joined:
    Feb 2, 2002
    Location:
    iowa
    #2
    it's an access policy for the web bots that crawl through pages, looking for data or indexing (like google). you can specify anything that you don't wish the robot to view or index.

    http://www.robotstxt.org/wc/norobots.html

    Go down to Examples to start with, it's a clearer understanding--

    paul
     
  3. Moderator emeritus

    Rower_CPU

    Joined:
    Oct 5, 2001
    Location:
    San Diego, CA
    #3
    I've done some research on them but never felt compelled enough to do any 'bot filtering on my sites.

    Now, if bandwidth or a rogue 'bot hitting your site too often were more of a concern, then I'd implement something.
     
  4. macrumors member

    Joined:
    Jan 12, 2004
    Location:
    Minnesota
    #4
    I use one on a site I maintain because I wanted some pages indexed that were only accessible through a javascript "jump-menu". They work, but don't count on them for security purposes (keeping bots out of cerrtain directories). The bots need to be written to read and follow the robots.txt file's instructions. Any malicious crawler/bot will just ignore its instructions.
     

Share This Page