Hi,
I have some input on this topic since it frustrated me a long time ago with my core duo macbook pro. 1080p .mkv files especially those "blu-ray" ones will generally play through a software codec only first of all. The second thing is after testing these files on a lot of different setups, the processors needs to be at least 2.4ghz speed minimum to play them acceptably. This is because its rendered in software only so its processor speed dependent, your 3D card doesn't matter but for battery life I'd go 9400. I do not have a technical explanation, but I can guess its because nobody has written optimized codecs for the x264 inside the mkv incap(this includes vlc and mplayer). Interestingly it doesn't matter if its a core 2 duo or hell a amd processor(I've done this on a lot of osx setups). 2.4ghz minimum. The 2.4ghz number comes from playing them on comp's with that speed or overclocking them to that speed) now the unibody even at 2.53 will only barely be fast enough. Yeah I know it sounds silly but were not talking apple h.264 here so its different. I assume the stuttering happens during high action or fast scenes with a lot of detail only, and in vlc when it stutters it may come up with a message saying its too slow or missing frames. Again this is the processor trying to keep up.
(what you can do)
1)
re-encode with visual hub or something to apple h.264 this will solve your problem with a slight degrade in quality and it'll probably be hardware accelerated then... or so the rumor has it with the unibodies =).
2)
when I checked 2 years ago the people that made the coreavc codec where talking about making one for osx. If this has happened you need this, since this codec is alot more efficient then the ones in vlc, mplayer, perian,etc.
This will likely be the only way for you to play 1080p .mkv unconverted stutter free. When I used this in windows it made some of the files playable or more playable.
3) As the others have said xbmc, I tried the first public release and it did make them slightly better, but probably not perfectly playable however.(that was for me 1 year ago so ymmv. )
4)Play them in windows. Get coreavc and the community codec pack and change the x264 codec to coreavc in ccp options .Then play the 1080p files in windows only.
I suggest just converting them, I hate going over to windows.
(oh yeah keep in mind this information is kinda old I haven't looked into it since then. I just play them on my macpro.)