Modifying file extensions within iPhoto library

I don't think there's a quick solution for you. Yes, the database is finicky.

You could probably go into the iPhoto library file (right-click), copy the files you want to convert to somewhere else, convert the files, open iPhoto app, delete the movies, then re-add the new versions.

If you haven't already, you should test one movie to see if you're satisfied with the way it looks after conversion. Then, test the replacement method.

 

I'll give it a try and report back

 

And before you do that, check what codecs and settings the original files use.
In order to find that out, you can use the following three applications to analyse a sample video file (one .avi and one .mov).

• VideoSpec (free)
• MediaInfo (free)
• Media Inspector (2.99 USD € in the Mac App Store (MAS))

When done, you can use the "Report" (VideoSpec) and "Export" (Media Inspector) button to export a report,
attachable to your next post via the button.
In MediaInfo you have to go to the Menu Bar, select View > Text and copy the text via CMD+A > CMD+C and paste it via CMD+V into your next post.​

Maybe the original files are already using a highly compressive codec for which further compression would be detrimental.