OK, I've just done a test and Audacity works perfectly under Windows, so there's no reason the Mac version shouldn't also work.
1. Install Audacity 1.3.13.
2. Install the FFmpeg import/export library, from the Audacity site. This will allow you to import/export AC3 files.
Now you can open your mkv/mp4 files directly in Audacity and select the AC3 track, but for some reason it downmixes it to stereo

So.....
3. Extract your AC3 track from your movie file.
4. Run Audacity and goto Edit->Preferences, this box will popup:-
5. Goto the Import/Export page and under the "When exporting tracks to an audio file" section select "Use custom mix", if you don't do this when you export your file all the tracks will be mixed into one file.
6. Goto File->Open and load your AC3 file, it should load as 6 mono tracks:-
7. Goto File->Export Multiple.
8. Select the export format and location then press Export.
9. You'll get an edit metadata page for each of the 6 tracks just press ok on each.
10. Mission accomplished
Just a quick side note, Audacity doesn't label the 6 mono audio tracks but the ordering is as follows:-
Track 1 = Left front
Track 2 = Right front
Track 3 = Center
Track 4 = LFE(Sub)
Track 5 = Left Surround
Track 6 = Right Surround