Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
There are two possibilities:

One: it could run the ram at a reduced speed.
Two: it could run the ram at the increased speed, but with a higher chance of random errors

I think the mac opts for the first possibility.
Hold down the Option key and select System Information from the Apple Menu.

Then check out the memory section.

PC3-8500 is 1066 Mhz; PC3-10600 is 1333 Mhz

What you can't do is swap PC3 with PC4, or PC2 ram.
 
10600 (1333mhz) is just the maximum speed its capable of running memory at. It will still work fine with any slower memory and just run at that reduced speed (1066mhz in this case).
You could even put in 800mhz and it should work, but, as others have pointed out, not sure why you would want to.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.