You're wildly misrepresenting the agreement between Apple and the publishers. Apple doesn't "have a say on what minimal price they have to sell at". They just ask that whatever that lower price is, Apple gets it. And if publishers had a satisfactory alternative access to market, they wouldn't have cut a deal with Apple in the first place. The problem here is not about models, it's that the players are engaging in abusive practices in the implementation of those models. Amazon was abusing the wholesale model when it was the only game in town and then publishers, when they got into the agency model, priced their ebooks abusively. Had the publishers priced books reasonably after cutting the iBooks deal, there wouldn't have been DOJ/EU investigations to begin with.
I wish it was that simple. The fact is that nothing never prevented any author to forego publishers, be it in print or in ebooks. Yet, only a tiny fraction of authors decide to go for self-publishing and among those, few do so successfully. Despite all their shortcomings, publishers offer something crucial to authors: advertisement and promotion budgets.