System Preferences > Mission Control > (uncheck) Automatically Rearrange Spaces...
And yes, I agree with you, very annoying. OSX uses some kind of usage based algorithm, that has never been helpful for me. Also wish I could make App Expose automatically arrange itself alphabetically without having to press the Cmd-1 keys.