It almost always goes up when I swap videos.
I have been having the exact same issue for the past two months now on my iPhone 4, iPad and even iPod nano 4th and 6th generation, and it does seem to happen more frequently when I add/remove videos, but not always. I search this forum (as well as others) and I was never able to come across anyone else with the same issue, so this past friday I made an appointment to the Genius Bar to see if they can help me find a solution. At the time, my 'other' was only at 2.5gb, but I had several screenshots showing up-to 6gb like the OP.
After explaining all of this to the Genius, he was stumped (and so where the two to other employees he asked to step in to help). His first guess was that their could be a corrupted file due to a 3rd party app (not jaiilbroken, from the App Store) that is causing this issue, but because this is also happening with my iPhone 4 and iPod nanos, he believes that the corrupted file could be in iTunes.
Not knowing how to pin-point the issue, I took the extreme steps to find a solution myself and I:
1. Deleted iTunes from my iMac (
http://support.apple.com/kb/ht1224)
2. Deleted Back-Ups of all of my devices.
3. Deleted Restore Files of all of my Devices
4. Re-installed a new copy of iTunes directly from Apple.com
5. Restored my iPad/iPhone not from a back-up and from a new copy of iOS 4.2.1 Restore File.
6. Only installed 20 essential apps (down from 45) and did not Log-in to GameCenter (this was a possible issue the genius brought up).
Now, I'm not sure which solution worked the best, but since Friday, my 'Other' on both my iPad and iPhone has stayed at 350mb or less and hasn't moved (no matter how much media I add/remove).