What's not fair? He knew that prices would probably drop, and features added. He made a "never" prediction - knowing all the facts or not, that's pretty boneheaded.
Maybe he words things badly, but go back and look at his quote. He specifically pointed out the price as the main issue. His point was that the price of the iPhone was going to be a hindrance to its sales. Given that Apple dropped the price so soon after the iPhone's announcement, I would say Ballmer was exactly right.
Again, I'm not a fan of the guy, but the way some Apple fans seem so eager to hang that quote around his neck like an albatross is embarrassing. He was right. The price was a problem.
To me, that quote is embarrassing for him for entirely different reasons. The way he said it sounded really dismissive of a competitor that they have learned is not to be underestimated anymore. Secondly, that fakey humor he exhibits when he talks about competition gives off a vaguely desperate and defensive tone--not a very becoming quality for the CEO of the biggest software company on the planet.
I always feel sorry for MS employees. Ballmer must be like that crazy uncle every family has to suffer through on every holiday, the one you tolerate because he's your uncle, but still, you wish he would just shut up.