/    Sign up×
Community /Pin to ProfileBookmark

Fair rating system?

Copy linkTweet thisAlerts:
Oct 28.2011

Been trying to come up with a fair way to rate stuff.

If one person rates an item a 5, it’s total rate is 5, but if 999 people rate a second item 5 and one rates it 4, the 4 brings down the median rate a little. The first item has a higher average rate, but with one vote, it really shouldn’t be rated higher than the second.

Another problem scenario. Again with one item getting one vote of five. The second has 100 5 stars and 200 1 stars. Who would rate higher as the second has so many 5 stars yet twice as many 1 stars.

If you calculate a median then 1 vote of 5 will always rate higher. :/

Anyone put any thought into this problem or has an opinion I’d be interested to hear views on how to rate fairly.

Thanks

to post a comment
PHP

4 Comments(s) ↴

Copy linkTweet thisAlerts:
@tracknutOct 28.2011 — You might double check your definition of "median", because ironically the median is intended to help you out here.

The median of 999 "5" votes and 1 "4" vote is "5". It is the "middle" vote, not the "mean" (what we typically think of the average vote, which would be more like 4.999 in your example. The purpose of the median as an alternative way of "averaging" is that it will throw out these oddball outliers and often result in an "average" that makes more sense.

In your second example, the median in the first scenario is 5, and in the second scenario the median is 1 - which could be deemed a reasonable "average" given the skewed data.

But this is the tip of the statistical iceberg... Publishing the mean, population and standard deviation would also be a good way of looking at the true ratings, though you start to wonder how many of your customers really will know what that all means.

Dave
Copy linkTweet thisAlerts:
@SyCoauthorOct 29.2011 — Yea, you're right I did get that wrong. Should have stayed awake in statistics class. Glad you did though ?

I was trying to think of a weighting system where the number of votes can increase to overall rating. I the first example the median of both would be a tie at 5 which still isn't fair. The second item deserves top spot. So you could fall back to number of votes when there's a tie, and the right one wins.

But in the second who does deserve first place? I was hoping to derive a formula and find out, lol. I don't think it would be fair to rate an item with so many 5 stars as a 1.

I suppose one way around the issue is to not allow any item to be considered top until a minimum number of votes has been cast. At least then there would be more data to work with.

Can you explain what Publishing the mean, population and standard deviation means or point me to a resource a stats dumbo like me might understand. Thanks
Copy linkTweet thisAlerts:
@tracknutOct 29.2011 — This is why statistics gets a bad rap.... there is no "right answer" to what you're asking, at least that I can imagine. Your gut is telling you "this one should rate higher than that one", but the numbers aren't helping you state that definitively. I certainly don't have a silver bullet to solve it...

Publishing mean, population and standard deviation would be like this. Just to be clear, the mean is the average, Ie, add up all the ratings and divide by the count of votes. The population is the number of votes cast, and the standard deviation is a mathematical calculation (which is irrelevant for purposes of this discussion) that gives you the "spread" of the votes. I.e. two votes, 4 and 5, will have a lower standard deviation than two votes of 1 and 5.

So once you've got those definitions in your head, it becomes pretty easy to get a feel of the data. For example:

Pop: 100, mean: 4, Std Dev: 0.05 => Cool, lots of votes, everyone agrees on 4 basically.

Pop: 8, mean: 3, Std Dev: 1 => Very few votes, average of 3, but raging generally from 2 to 4

Pop: 8, mean: 3, Std Dev: 2 => Very few votes, average of 3, but ranging from 1 to 5 (well that basically means the same as "votes are random" since the whole range is 1 thru 5

Pop: 100, mean:4, Std Dev: 1 => Large number of votes, generally between 3 to 5, seems like kind of a normal standard deviation I'd expect when rating movies or something.

Pop:1, mean 5, Std Dev:0 => Ok, only one person voted, hard to say whether the vote is worth much or not.

Hope that helps...

Dave
Copy linkTweet thisAlerts:
@SyCoauthorOct 29.2011 — Very cool, I understood that. ?

I'll keep researching then. Maybe someone has a system that is a good real world workable system.

Thanks for your help and confirming this isn't as easy I thought it should be lol.
×

Success!

Help @SyCo spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.18,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...