I read this discussion with interest. I certainly don't pretend to know anymore than anyone else, but from what I can surmise, (and this is just my opinion) it's probably Apple that has prevented this.
First, Apple could have updated (or even first introduced the iPhone with) the iPhone's Mail App to work with standard IMAP Idle e-mail. I've used phones in the past that have had this and it works perfectly fine with any e-mail service that supports the IMAP Idle standard. For instance, Gmail already supports this standard and the past few Treo's I've used, were able to have e-mail instantly delivered with Gmail. Other providers like Fastmail also support IMAP Idle.
So far, Apple has not supported this standard, instead preferring to strike a deal with Yahoo to provide instant (at least that's the way it's supposed to work) e-mail through a proprietary standard. This way, Apple controls and influences the way we receive our e-mail. They are well aware of the allure that "instant" email has, and (because they have chosen not to support standard IMAP Idle) they are able to control which services are able to provide it for the iPhone.
Of course, you could use Exchange, but they are pretty well aware that most consumers will not go this route unless their work place provides or demands it. I'm also betting that they will not allow another third party developer to provide a competing mail app that does support the standard that many e-mail services (like gmail) use, so we are pretty much left at the mercy of Apple.
