Check the operating shock and vibration spec for your hard drive. If you're exceeding that, then I would suggest changing the operating environment.
If you have an iPhone/iPodT, there are several apps which can give you a close measurement of the shock/vibration. Below are some example specs which most current drives can survive. I imagine it would be pretty hard to generate those kind of accelerations by hand, but sudden movements can cause more than you might think. Also, these are absolute max ratings. Lesser levels might be survivable, but could weaken and cause premature failure of a component down the line.
Seagate Momentus XT:
vibration: 5500 Hz 1.0 Gs (0 to peak). Max displacement may apply below 10 Hz.
Shock: 350 Gs based on half-sine shock pulses of 2 ms
(This was one of the odder thread topics - my reaction was to treat it as Sheldon Cooper might.. 😉 )