If you're doing some heavy duty things like rendering multiple images in photoshop, it can freeze and creates a queue of actions and once it gets a chance, it can run all of these at once and send it crazy.
If not, go into the "Keyboard & Mouse" in System Preferences and see if anything is ridiculously high/low
No the trackpad isn't dirty and multiple programs aren't running. I can just have finder open and it just freaks out, starts running all over the place. Then, 10 seconds later, its fine and dandy for an unspecified time...