I also think we should have both up and down voting, or no voting at all. Yes, there are some abuses of down voting, where someone down votes a post simply because they disagree with the opinion in that post, but those are usually counterbalanced by people up voting posts they think have been unfairly down voted. As for posts accumuating extreme down votes, they are usually those that are seen as trolling or are extremely disorganized and incoherent -- posts that espouse an unpopular opinion but are not trolling and are coherent and well-stated generally accumulate very few down votes.
Incidentally, this very thread is an example of how having only up voting can skwer perception. I've up voted every post I agree with, but because no one can down vote those posts, people who disagree with them have no direct way of expressing their disagreement. Their only recourse is to post their disagreement in separate posts, then hope others up vote those posts. This strikes me as unfair -- so I think we should have votes in both directions, or no voting at all.