Tad Milbourn
posted this on July 06, 2010 14:03
As you rate ideas in Brainstorm using the 5-star scale, you may notice that the average rating sometimes looks strange. For example, you may be the first person to rate an idea. Let's say you rate the idea 5 stars. When you mouse over the stars to find the "average rating" you may see that the average is calculated as less than 5. How is that even possible?

The answer lies in how we treat our ratings calculation. We do some funky math to keep our rating system doing what it's supposed to -- helping you identify top ideas.
Brainstorm employs a Bayesian algorithm which adds more nuance to the rating than just "the most votes win." Brainstorm looks not only at the strength of the rating (on a scale of 1-5), but also the number of ratings on that idea, and the number of ratings across the entire system to better determine which ideas truly stand out. This is in contrast to a rating system using a simple average. The Bayesian approach helps mitigate a key problem with rating systems that rely on simple averages -- ideas with one 5-star rating instantly shoot to the top of the charts
Example: If a rating system uses simple averages, then there is a bias towards ideas with one vote of 5 stars. The simple average is 5.0. For ideas that receive lots of ratings, there is a very high chance that one of those ratings is less than 5. As an example, you can imagine an idea with 10 ratings -- nine 5-stars and one 4-star (simple average is 4.9). In this case, the idea with one rating of 5 stars would be displayed as "higher rated" than the idea with 10 ratings. However, you can make a very strong argument that the idea with 10 ratings is likely better because a larger sample is saying this is a high quality idea. Brainstorm prefers ideas with more high ratings over time to ideas with just a couple of high ratings.
So, that's why we use the Bayesian approach. Across large numbers of ideas, this approach leads to a better understanding of which ideas stand out from the rest. However, at the individual idea level, when there are a low number of ratings, its effects can seem strange. We apologize for any confusion.