If you're AVI is playable in Quicktime (and if not get your codecs updated), then open the movie and in the QT menu go Window < Show Movie Properties. Click on the audio track and select extract. You can then import/move/edit/export the audio track as needed for whatever your purpose is.
If the above suits your purposes, go that route, as it will take a fraction of the time versus waiting for iMovie to drag its ass on the import step.