If the file really has been converted to AAC, then you can just change the extension to "m4r". Instead of trying to "open" it, try dragging it into your iTunes library (with iTunes open, drag from Finder to the iTunes window).
You have to push the "c" key and drag the yellow bar to the portion of the song in Garage Band you want to be the ringtone and then go to Share>Send ringtone to iTunes.