Thoughts on rating poster quality (vs. post quantity)

[smithrh]

This is a follow-up to Doctor Q's "Top 50 Posters" thread, but I like specific thread titles so I created a new thread... (hopefully this time I won't time out and lose this post - I have "Remember Me" checked, crossing fingers!)

In that thread, Doctor Q wrote: "Now if I could just measure post quality, that would be really be something to study!"

I hope to show that yes, there are some ways of doing this, but not necessarily for individual posts, but for the population of posts that a poster has posted. That is, to understand if a poster is writing (on average) quality posts, you can't just look at a few and make that determination - you need to look at the universe of posts that person has made - or a predetermined subset.

Now, before I start, a few background items:

* I did take a look at vBulletin documentation, but after a while I gave up, so much is possible even without customization, but I may be suggesting things that are difficult or CPU-intensive to do. Likely the key here would be post-processing.

* I didn't consider whether or not people who post often are valuable to the site or not. e.g., it could be that having people post a lot - even if the posts are not good ones - is still somehow good for Arn/MacRumors. My personal judgement is that people that just post for the sake of posting are not helpful - but I don't sit in Arn's chair either.


OK, let's get to a few ideas.

* Post length/word count per post:
There are two sides to the coin, the "Yeah!" contentless posts, and also the TL;DR variety (e.g., this post). If someone consistently makes very short posts, then that's a warning signal. While I don't see a lot of TL;DR posts, if someone were to be doing that on a consistent basis then that would likely be an area of concern as well.

Adjustments would have to be made for people quoting others and for code snippets. I'm also unsure about how signatures might need to be included or discarded.

* Spelling/grammar:
This is much harder to do - not everyone here is a native English speaker, people make innocent typos, and code is simply going to throw a spellchecker/grammar checker for a loop. Nor would you want to punish someone for quoting someone who made errors. But technically this can be done, perhaps getting to a misspelling rate by word count.

* Hot words usage:
Instead of a full spell check, one could look for words that are intended to inflame or incite: fandroid, fanboy, fanboi, sheep, sheeple and so on. These terms wouldn't necessarily trip the censoring filter, but it would catch people that consistently use these terms. Again you would want to get to a rate of usage, not an absolute number. What would you think of a poster who uses the word "sheeple" in 97% of his 432 posts?

* Use of quotes in a post:
I actually have no idea of what this might show; I see many considerate posters using quotes very nicely. On the other hand, I also see quite a few arguments using quotes. But what I don't like is when someone posts something like "That is SO true" in a 10-page thread with no quote - with no context, it's useless (it was probably useless in any case, this is just for illustrative purposes). So perhaps this would be a rate of non-usage of quotes - but I really don't have a prediction here, other than I think it could be useful. Funny thing about analytics, things you think would be useful wind up on the floor quite a bit, and things that you'd think are dumb wind up being powerful... until you actually take a look, you don't know.

* Up-votes per post rate:
So, here we go. While I'm sure people have thought of this before, the devil is in the details. First off, it's probably best to view an up-vote as a sign of popularity, not quality. I've seen (back in the day when down votes were available) high-quality, accurate and timely posts be down voted massively, and inaccurate contentless posts get voted up. But overall I'd wager that posts with higher upvote counts tend to be the better posts.

Now, there are challenges here besides what was already said: older posts still have negative vote totals, while more recent posts can't - so you couldn't mix the populations of posts before Arn's "no down voting" decision was made. Also, posts with zero upvotes can be really horrid posts now, but they'd be regarded as the same as neutral or average posts - meaning a post that was certainly OK but just wasn't interesting or insightful enough to vote up.

Even given that, there's still good data there - but the best measure would be the rate of upvoting per post views. That is, if someone posts something popular in the very beginning of a widely-viewed thread, they're going to get a lot of up votes simply because there are a lot of people viewing that thread. Contrast that to a thread that only had 100 unique readers, yet a post there gets 8 up votes.


I have more, but this is probably overload for most folks. Oh yeah, I'd toss in length of membership into a lot of these as well.


Mind exercise: what would happen to posting behavior if some user titles were replaced with "Blowhard," "Village Idiot," "Know it all," "Self-Proclaimed Legend," "Blah Blah Blah" and the like?

<ducks and runs>

 

[GoCubsGo]

The flaw in using the voting button(s) as your guide is that, to your point, quality posts have been down voted frequently just and those lacking any real quality are up-voted. Perhaps because they're funny or people just generally felt the need to down vote a varied opinion. While this site likes to suggest that no content posts (or posts just posting a meme) are removed, this is not always the case. As members, we are often encouraged to report these posts. I'll spare the mod that comes here to write that.

I have seen quality posts overlooked by members, including the original author of the post. I find a quality post is absolutely subjective on this site. Just today some member suggested I was being sarcastic. His feelers were obviously tickled and he read my post wrong. I'm not about to state intent or tone, if you're here you should possess at least average intelligence and be able to understand that how something is read may not be how it was intended by the author.

Either way, members on this site aren't mature enough to really be the dictator of quality posts. I've seen absolute trash posted by members and mods. That doesn't mean people agree with me nor do I expect them to.

If you insist on measuring quality with a voting button then so be it. It simply doesn't mean that the posts are unquestionably quality posts. Some of the more prolific posters today are quite helpful and others are not. These posters may not be at the top of any one "50 most" list either. In my case, I happen to have it on good authority that this site, it's mods, and it's members do not find me personally helpful nor would they consider me a contributing member outside the occasional subscription payment. I make zero excuses but I can say I do my best.

As for the new titles, it would go against everything this site stands for ... or seems to want to believe they stand for. It would be a form of name calling and the mods, even the ones who have engaged in some unsavory activity off the site so you would think they'd be okay with it, would simply not have it.

But if you have a solid way of measuring quality posts then have at it. There are some members on this site who do quite a bit to help others and they're just not seen as much because they stick to certain forums. Like the MacPro forum for example, there are some shining stars there and even one that I think was banned previously (most unfortunately).

 

[MacNut]

If we started to judge quality posts here more than half would have to get deleted.

 

[macs4nw]

This is a follow-up to Doctor Q's "Top 50 Posters" thread, but I like specific thread titles so I created a new thread... (hopefully this time I won't time out and lose this post - I have "Remember Me" checked, crossing fingers!)

In that thread, Doctor Q wrote: "Now if I could just measure post quality, that would be really be something to study!"

I hope to show that yes, there are some ways of doing this, but not necessarily for individual posts, but for the population of posts that a poster has posted. That is, to understand if a poster is writing (on average) quality posts, you can't just look at a few and make that determination - you need to look at the universe of posts that person has made - or a predetermined subset.

Now, before I start, a few background items:

* I did take a look at vBulletin documentation, but after a while I gave up, so much is possible even without customization, but I may be suggesting things that are difficult or CPU-intensive to do. Likely the key here would be post-processing.

* I didn't consider whether or not people who post often are valuable to the site or not. e.g., it could be that having people post a lot - even if the posts are not good ones - is still somehow good for Arn/MacRumors. My personal judgement is that people that just post for the sake of posting are not helpful - but I don't sit in Arn's chair either.


OK, let's get to a few ideas.

* Post length/word count per post:
There are two sides to the coin, the "Yeah!" contentless posts, and also the TL;DR variety (e.g., this post). If someone consistently makes very short posts, then that's a warning signal. While I don't see a lot of TL;DR posts, if someone were to be doing that on a consistent basis then that would likely be an area of concern as well.

Adjustments would have to be made for people quoting others and for code snippets. I'm also unsure about how signatures might need to be included or discarded.

* Spelling/grammar:
This is much harder to do - not everyone here is a native English speaker, people make innocent typos, and code is simply going to throw a spellchecker/grammar checker for a loop. Nor would you want to punish someone for quoting someone who made errors. But technically this can be done, perhaps getting to a misspelling rate by word count.

* Hot words usage:
Instead of a full spell check, one could look for words that are intended to inflame or incite: fandroid, fanboy, fanboi, sheep, sheeple and so on. These terms wouldn't necessarily trip the censoring filter, but it would catch people that consistently use these terms. Again you would want to get to a rate of usage, not an absolute number. What would you think of a poster who uses the word "sheeple" in 97% of his 432 posts?

* Use of quotes in a post:
I actually have no idea of what this might show; I see many considerate posters using quotes very nicely. On the other hand, I also see quite a few arguments using quotes. But what I don't like is when someone posts something like "That is SO true" in a 10-page thread with no quote - with no context, it's useless (it was probably useless in any case, this is just for illustrative purposes). So perhaps this would be a rate of non-usage of quotes - but I really don't have a prediction here, other than I think it could be useful. Funny thing about analytics, things you think would be useful wind up on the floor quite a bit, and things that you'd think are dumb wind up being powerful... until you actually take a look, you don't know.

* Up-votes per post rate:
So, here we go. While I'm sure people have thought of this before, the devil is in the details. First off, it's probably best to view an up-vote as a sign of popularity, not quality. I've seen (back in the day when down votes were available) high-quality, accurate and timely posts be down voted massively, and inaccurate contentless posts get voted up. But overall I'd wager that posts with higher upvote counts tend to be the better posts.

Now, there are challenges here besides what was already said: older posts still have negative vote totals, while more recent posts can't - so you couldn't mix the populations of posts before Arn's "no down voting" decision was made. Also, posts with zero upvotes can be really horrid posts now, but they'd be regarded as the same as neutral or average posts - meaning a post that was certainly OK but just wasn't interesting or insightful enough to vote up.

Even given that, there's still good data there - but the best measure would be the rate of upvoting per post views. That is, if someone posts something popular in the very beginning of a widely-viewed thread, they're going to get a lot of up votes simply because there are a lot of people viewing that thread. Contrast that to a thread that only had 100 unique readers, yet a post there gets 8 up votes.


I have more, but this is probably overload for most folks. Oh yeah, I'd toss in length of membership into a lot of these as well.


Mind exercise: what would happen to posting behavior if some user titles were replaced with "Blowhard," "Village Idiot," "Know it all," "Self-Proclaimed Legend," "Blah Blah Blah" and the like?

<ducks and runs>

Aside from the obviously flippant posts, many posts are merely a personal opinion of the poster, as in when someone asks for, and gets advice, or when someone just wants to spout their unabashed opinion about a certain product or company. Then of course there are the highly speculative posts, as in: company x is going to do this or that. All these are highly subjective posts, that are mostly written by well-meaning individuals, but by their very nature, are difficult to quantify.

But I have also read, on more than one occasion, very long-winded, and upon closer scrutiny, very well researched posts, that try to address all angles of a topic/question, and appear to be very objective. Those are the posts that deserve the highest ratings, obviously.

But in all practicality, who would be the arbiter of this? An intensive and time consuming process, even for an expert panel of 'knowledgeable' judges. I think most regular MR members/visitors, after having been here for a while, know instinctively, how to judge 'post quality'.

 

[annk]

I understand the idea behind the wish, but the fact is that quality is going to be, to a great extent, in the eye of the beholder.

If you disagree with a post, you're more likely to call it "trash". That doesn't necessarily mean that the other 800,000+ members would call it trash. (The exception is of course spam or other rules violations, and we aim to remove them).

A post that only expresses an opinion can be helpful, for example in cases where a solution has been offered and a lot of members chime in to say that they agree for whatever reason.

On a big site like this one, spelling/grammar errors don't usually mean that a post isn't good (unless of course it's spam, and those are often reported and removed).

Post length is not an indication of quality. One-word or frivolous posts are removed whenever we see them, and a short answer can be every bit as valuable as a long one.

Length of membership doesn't mean any more on a group basis as far as quality of answers; it still comes down to the individual. A long-term member will be better acquainted with the dynamic of the forum, but not necessarily say anything you find worthwhile. By the same token, a member can have lurked for years, and finally register to post a great answer in his/her first post.

I could go on, but basically I feel that while some of the ideas could be incorporated onto a smaller site with good results, their mostly not feasible on a big site, where there's naturally going to be a larger span of answer types, and where manual actions after a post is made just aren't reasonable as far as time or resources.

It's in the nature of the Internet to leave the final evaluation of the source and the final decision up to the reader. Large-access sites like this one aren't meant to be paid expert advice, so it's up to the individual. I do get your idea, and I certainly don't mean to shoot you down here. You've written a lot of things that can give food for thought. I just don't see them translating into the kind of thing you do, at least not here.

 

[vrDrew]

It is an intriguing concept. I'm sure that the coders at Facebook or Google probably could cobble together some sort of algorithm that would rate and/or rank individual posters on a forum. But then again, even the assigning a weights to various criteria is a subjective decision. And I'm pretty certain that it would be impossible for such an algorithm to detect (and score) things like irony and humor.

 

[Carlanga]

The answer is simple, you can't quantify quality when the word in a social environment can't be counted by a defined set of universally accepted metrics IMO. I guess you can try to get all those things you mentioned together and try to calculate based on a formula for all of those things, but at the end you probably would still be missing stuff.

If all you want is to have a way to make some people have more MR 'street-creed' so that people recognize trolls and such, then there could a voting system under our avatars and depending on count you can fall in various categories from troll, dumb, normal, amazing, superhuman, mac, jobs, 1010, etc.

 

[ucfgrad93]

If we started to judge quality posts here more than half would have to get deleted.

Agreed. I don't think that it is realistic to objectively rate the quality of a post.