You're kidding right. I need a permanent solution.
I run with an iPod crowd, and everyone has a open input just waiting for some sounds, and I've been denied too many times with the iPhone. I have the adapter... silly little thing. It stays in my car, not in the phone (works great in my car btw, but I travel way too much). I actually 'whittled' the mini phono jack in the company van. (shhh...)
Well, no, I don't think I was kidding... maybe... no.
Take a close look at the housing - there's a barrel which meets with the jack itself. (some pictures here)
I'm too lazy to go get my caliper, but my "calibrated" eye says the barrel would end up paper thin. I obviously don't know your experience with metal working, but I've done some as an engineer, aircraft maintenance, various hobby and Amateur Radio projects (housings, antenna, etc.), scale aircraft, etc. It just doesn't look like a good idea - it'll most likely end up tearing the back up, or, like I said, leaving it weak and flimsy.
You seem to care how it'll look, as you don't really want something hanging out of the jack. But, an adapter is bound to be nicer (especially the Apple one) than what could happen to the back.
I just ordered one of those adapters off of eBay with that link. Even though it doesn't solve the whole deal with having a mic to talk to people, it's great that I can finally use my Etymotic ER-6i headphones with my iPhone. Those stock Apple iPhone headphones included with it just don't cut it for me sound quality wise.