Why do my spaces get messed up?

I've been wondering this for a while now. I have 4 spaces setup, and sometimes I don't switch between spaces, but select safari or some other application from the dock because sometimes it just seems easier to click on the dock rather than change spaces. Usually when clicking on the application in the dock, the spaces will change to the space that is currently running the application I selected, so no problem.

BUT!!! There have been many many many times when I'm using an app, I look up at the top of the screen at the spaces numerical indicator and I see that my program is in the wrong space! For some reason, programs wander out of the spaces I assigned them to and get into another space. If I go into the sys pref and look at the spaces assignment, it still shows that I assigned the programs to the space I originally wanted, but in reality they're in a different space.

For example, I always put my browser in space 2, but it often winds up in any other space. Then I have to show all spaces, and drag it back over to space 2, where it belongs. Before long, it will wander away again.

I know I'm not accidentally dragging them to the wrong space. But just while working quickly with my computer and switching between programs and spaces, etc. they get assigned to other spaces!!

Anybody have any feedback on this?
Thanks