I have a fix - or rather, a workaround.
WARNING: It is lengthy, but still far better than starting a new project from scratch.
I had this same issue and none of my transitions/clips were at custom speeds, nor did I have anything else weird going on. Unfortunately there is nothing you can do within iMovie to fix this, at least there wasn't for me. iMovie is BY FAR the buggiest software I have ever used and if you haven't already spent hours slaving away on a project, you are so much better off using just about anything else to edit your video.
...But let's assume you have already spent many hours working on your project and you don't want to start all over. Well then, you have at least one option.
It's a pain in the neck, but it's infinitely better than restarting your project from scratch.
This will require:
A software called Soundflower (which is free) and some kind of audio editing software (I use Logic, but really any should work).
1.) You're going to need to export your video with the messed up audio. Typically if you're audio is messing up, you're video is at least usually working just fine (at least it always does for me).
2.) You're going to need to open that video in your audio software and import only the audio. If your software doesn't give you this option, then you're going to need to record the audio from your video using the audio software and Soundflower. You can find tutorials on how to do this, but it's honestly pretty straightforward once you start playing around with you Mac's audio I/O settings. Remember to have your computer's volume all the way up when doing this.
3.) Watch/listen to your video and make notes of where the audio errors occur. I HIGHLY RECOMMEND writing down times too, as they will be extremely helpful later.
4.) Now were going to rip some working audio from the iMovie project itself. Go back into your iMovie project and only play back audio while in the edit window for this step. Do NOT play your project back in full screen.
This part is key: if you try to play back your project from the beginning the audio will usually still mess up. So, what you're gonna have to do is go back on your edit window timeline right before these errors occur and play that specific clip back while recording this working audio with Soundflower and your audio software. Essentially we're only recording the parts of the audio where these errors occur, so try to do this in order if you can. Try to leave several seconds of audio in before the error - that way we can use it as reference when we construct our new audio track.
5.) Now in your audio software you should have your original error-filled audio track and the new clips that we just captured. This part can be painstaking if you're not familiar with audio editing softwares, but as you'll learn, it's really not that hard.
You're going to look at your original audio track and compare it to your new working audio clips. You should be able to notice where those new clips are supposed to go by looking at the identical (or close to identical) waveform shapes in your original audio track. The two should correspond with one another. If you don't understand what I mean right now, you will when you start playing around with the audio. This is why marking times is very important.
Take your new clips and split them up if you haven't already, that way you can drag them around and edit them separately from one another. Essentially your going to line these new clips up with original and then replace the original parts with new ones.
We need to get this as precise as possible, so start off by getting them as close as you can with your naked eye, and then use a zoom tool to zoom in as much as possible so that you can get them lined up PERFECTLY, or at least extremely close. In some softwares you can just drag these new clips on top of the old track, but most of the time you will need to actually cut out the old piece and replace it with the new one. In doing this, zoom in and make sure that you are NOT cutting out a bigger space than your new clip will fill - otherwise you will have small gaps in your audio.
VERY IMPORTANT: Also make sure that this new audio track starts at the EXACT SAME time as your original - otherwise your new audio will be out of sync.
I know this part can really suck if you're not used to audio editing, but at this point, there is little other choice.
6.) Once you are done with this, export your newly pieced together audio track.
7.) Now open iMovie and create a NEW project. Re-import the video that you exported earlier that has the messed up audio. This might take a while, so just be patient.
8.) Then right click the clip and select "Detach Audio". This will do just that - allowing you to edit the audio separately from the video. Now delete this audio.
9.) Now, import the new, perfect audio track that we just made and make sure that it starts at the start of the project. Make sure it's in sync and that everything works.
Now just export your new video with the working audio and you should be done. Since we have given iMovie the audio with no editing actually being done in iMovie, it does not have to generate its own audio now and you should have absolutely no errors now - unless there is some that you forgot to take out earlier.
YOU'RE FINALLY DONE!!!
I know this entire process sucks, but if you're that far into your project, there is little else that you can do. Now you've learned your lesson:
DON'T USE IMOVIE EVER AGAIN.
I hope this helped!
