How do you prevent google crawlers from seeing a folder of webpages?

[whitedragon101]

I have a main website for advertising my services for web development and one of the links is to a complete site I have made. However these pages would be confusing to a crawler as they are nothing to do with my business. The format is this

www.mymainsite.co.uk/index.html
www.mymainsite.co.uk/other_main_pages.html


www.mymainsite.co.uk/myexamplesite/index.html
www.mymainsite.co.uk/myexamplesite/lotsofotherpages_etc.html

Is there a way to make this work and have good web crawling and SEO for the main pages. Or should I get a new domain name for the myexamplesite part?

(I am a backend php/MySQL developer and SEO is new to me)

 

[Tomorrow]

Make a robots.txt file and place it in your HTML directory.

 

[whitedragon101]

Make a robots.txt file and place it in your HTML directory.

Thanks. Done. All working

 

[960design]

Yep. robots.txt will work for the good guys. Just keep in mind that robots.txt doesn't have to be followed, spammers malware will skip it as well as page strippers, so don't use robots.txt to 'hide' important stuff.

When I build sites for customers I put them in a subdomain of my mainsite; something like this:

http://siteimworkingon.mymainsite.co.uk
http://siteimworkingon.mymainsite.co.uk/anotherclientpage

My main site would still look like:
http://mymainsite.co.uk

It's half a dozen one way and four score another.

 

[digitalField]

I have a main website for advertising my services for web development and one of the links is to a complete site I have made. However these pages would be confusing to a crawler as they are nothing to do with my business.

ok so a couple things going on here that we should address that will kill a couple birds with a few stones.

1. you do want google crawling the sites that you made.
2. you do not want to host a complete duplicate site on your site as an example of work you have done.
3. you do want to create a case study of this site on your site
4. you do want to link to the live site.

In short, create a single html page that shows a couple screen shots from the website and lists out some of the things that you did in creation of that site. Link to that from your main page or portfolio or whatever. On that page you can link to the live full version.

This is good for a few reasons. One, if a human is visiting your site seeing a summary of the site you made is MUCH more useful to them then the full site itself. And then if they are still interested they can click deeper. Two, if a crawler is visiting your site it will connect more relevant information to you and the project and ultimately to person goggling.