There are Windows apps that support GPU decoding of x264 mkv's (mpc + dxva for example) - OS X Quicktime only decodes apples own h264 profiles with the GPU afaik, and that's not a codec thats very widespread in certain scenes.
Also there is CoreCodecs CoreAVC (=quick multithreaded h.264 decoder) that is a codec supported by multiple windows apps because it is a codec component - the OS-X Version is a standalone player only (havent tried it yet, $$).
Personally i use "Plex" for 1080p mkvs since it has multithreaded h.264 decoding, has a nice media library and is free. Its decoding speed not as fast as CoreAVC but its definitive faster than VLC.
I have my hopes high that SnowLeopard/Grand Central/OpenCL/QuicktimeX changes this all but we'll see...