How do you prevent google crawlers from seeing a folder of webpages?

Discussion in 'Web Design and Development' started by whitedragon101, Jan 31, 2014.

  1. whitedragon101 macrumors 65816

    Sep 11, 2008
    I have a main website for advertising my services for web development and one of the links is to a complete site I have made. However these pages would be confusing to a crawler as they are nothing to do with my business. The format is this

    Is there a way to make this work and have good web crawling and SEO for the main pages. Or should I get a new domain name for the myexamplesite part?

    (I am a backend php/MySQL developer and SEO is new to me)
  2. Tomorrow macrumors 604


    Mar 2, 2008
    Always a day away
  3. whitedragon101 thread starter macrumors 65816

    Sep 11, 2008
    Thanks. Done. All working :)
  4. 960design macrumors 68030

    Apr 17, 2012
    Destin, FL
    Yep. robots.txt will work for the good guys. Just keep in mind that robots.txt doesn't have to be followed, spammers malware will skip it as well as page strippers, so don't use robots.txt to 'hide' important stuff.

    When I build sites for customers I put them in a subdomain of my mainsite; something like this:

    My main site would still look like:

    It's half a dozen one way and four score another.
  5. digitalField macrumors regular

    May 21, 2004
    ok so a couple things going on here that we should address that will kill a couple birds with a few stones.

    1. you do want google crawling the sites that you made.
    2. you do not want to host a complete duplicate site on your site as an example of work you have done.
    3. you do want to create a case study of this site on your site
    4. you do want to link to the live site.

    In short, create a single html page that shows a couple screen shots from the website and lists out some of the things that you did in creation of that site. Link to that from your main page or portfolio or whatever. On that page you can link to the live full version.

    This is good for a few reasons. One, if a human is visiting your site seeing a summary of the site you made is MUCH more useful to them then the full site itself. And then if they are still interested they can click deeper. Two, if a crawler is visiting your site it will connect more relevant information to you and the project and ultimately to person goggling.

Share This Page