I've been having this problem for a while now on my iMac (OSX 10.6.3) where a file would just rename itself into something like "FULLM~20.MKV" after I rename a different file within the same folder.
It's occurred in several folders and is really annoying because I don't know what the original filenames are and messes up my organization.
I took note of some occurrences below. If I recall correctly, trying to fix the messed up filename sometimes works but it eventually gets messed up again. It's not limited to video files only, I've seen it happen to a jpg at least once.
Renaming
"✘ Hetalia Axis Powers 33v2 [Webcast].mkv"
to
"✘ Hetalia Axis Powers 33 [Webcast].mkv"
automatically renames
"✘ Hetalia Axis Powers 37 [Webcast].mkv"
to
"_HETA~34.MKV"
Renaming
"_HETA~34.MKV"
to
"✘ Hetalia Axis Powers 37 [Webcast].mkv"
fails with error message
"The name ✘ Hetalia Axis Powers 37 [Webcast].mkv is already taken. Please choose a different name."
Renaming
"FULLM~20.MKV"
to
"Fullmetal Alchemist Brotherhood 20.mkv"
fails with error message
"The name Fullmetal Alchemist Brotherhood 20.mkv is already taken. Please choose a different name."
Everything changed back to normal after renaming a few other files within the same folder
Renaming
"Oha Suta 20100331 [+, Koharu Kusumi].mp4"
to
"[+] Oha Suta 20100331 [Koharu Kusumi].mp4"
automatically renames
"Oha Suta 20090526 [Koharu Kusumi].mp4"
to
"OHASU~16.MP4"
I undid that and added another file into the folder called "Ogawa Saki - Oha Star 100521.mp4" and renamed it to "✘ Oha Suta 20100521 [Saki Ogawa].mp4". Doing the same renaming thing above then worked fine.

The weirdly named files work fine though.
It's occurred in several folders and is really annoying because I don't know what the original filenames are and messes up my organization.
I took note of some occurrences below. If I recall correctly, trying to fix the messed up filename sometimes works but it eventually gets messed up again. It's not limited to video files only, I've seen it happen to a jpg at least once.
Renaming
"✘ Hetalia Axis Powers 33v2 [Webcast].mkv"
to
"✘ Hetalia Axis Powers 33 [Webcast].mkv"
automatically renames
"✘ Hetalia Axis Powers 37 [Webcast].mkv"
to
"_HETA~34.MKV"
Renaming
"_HETA~34.MKV"
to
"✘ Hetalia Axis Powers 37 [Webcast].mkv"
fails with error message
"The name ✘ Hetalia Axis Powers 37 [Webcast].mkv is already taken. Please choose a different name."
Renaming
"FULLM~20.MKV"
to
"Fullmetal Alchemist Brotherhood 20.mkv"
fails with error message
"The name Fullmetal Alchemist Brotherhood 20.mkv is already taken. Please choose a different name."
Everything changed back to normal after renaming a few other files within the same folder
Renaming
"Oha Suta 20100331 [+, Koharu Kusumi].mp4"
to
"[+] Oha Suta 20100331 [Koharu Kusumi].mp4"
automatically renames
"Oha Suta 20090526 [Koharu Kusumi].mp4"
to
"OHASU~16.MP4"
I undid that and added another file into the folder called "Ogawa Saki - Oha Star 100521.mp4" and renamed it to "✘ Oha Suta 20100521 [Saki Ogawa].mp4". Doing the same renaming thing above then worked fine.
The weirdly named files work fine though.