Ok, the Finder behavior on this is very confusing, but as far as I can tell this is how it works:
If the file does have a type and creator set, and the file extension matches what the Finder "expects" (for example, .mp3 and iTunes, .doc and Word) then it will let you delete it without complaint. This sort of makes sense, since the file is still properly identified.
If the file does not have a type and creator that match the extension, or it has no type and creator at all (for example, a Word doc labeled .mp3 or an .avi downloaded from the internet), then deleting the extension will merely set the "Hide Extension" checkbox without actually deleting it. Furthermore, if you go back and type in a new extension at this point, the old one is still there, you just can't see it, so you end up with, for example, document.doc.mp3. Confusing.
Lastly, if you try to change an existing and visible extension, whether it's "correct" or not, the Finder will ask you if you're sure.
So, in this particular case, I assume the type and creator are set properly on those files, so deleting the .mp3 does nothing. Were they "unmarked" files, it would instead set the "Hide" flag.
Background: The OSX Finder (don't think they've changed the priorities recently) checks the filetype and creator first, then falls back on the visible .something extension if there are none or it doesn't recognize them.
This explains why you can have some .pdf files that open with Preview and others that open with Acrobat--either the types and creators reflect this, or the Acrobat ones are labeled while the ones that open in Preview have no type/cretator at all and the default app is Preview).
The metadata system is really a better system for the most part, but the internet (not to mention Windows) and its lack of obvious metadata (though a similar system does actually exist on the web, MIME types) is the reason for Apple's push toward in-filename extensions. The fact that it's invisible and hard to change is also a bit annoying, and potentially confusing for novice users (the "open with" settings help, though).