Generally speaking, it seems to me that proponents of keeping the downvote buttons are phrasing their discontent in terms of how it has affected them and their interaction with other people, whereas those advocating for and supporting the removal of the downvoting buttons are taking a wider view of the effect on the community and the quality of discussion.
Let me be clear, though. I never supported their introduction in the first place, noticed that the rationale for introducing them in the first place was never clearly explained, was suspicious of the potential implications that the voting data could be used in far more ways than just flagging content for the front page... and so after trying them out for a day or two, just stopped using them, ignoring them for the most part.
In all my years here: as forum member, contributor of many hundreds of dollars, and as moderator... there has never been a groundswell of public opinion clamouring for their introduction in the first place. Fundamentally, from the very start, the entire exercise has come across, to me at least, as a top-down initiative with a cloudy purpose and hoped-for outcomes that were never publicly specified.