Recently I came across an article in the Wall Street Journal about online ratings. The article, which surveys a number of online properties, cites the tendency to 4.3: On the Internet, Everyone’s a Critic But They’re Not Very Critical. The article’s authors pretty much capture what many of us get intuitively about why online ratings really don’t work, but I thought I’d break this down from a social interaction design perspective to get at some of the causes of this. First and foremost is the fact that most online systems built to capture user tastes, preferences, and interests engender bias. And online media amplify bias, for a number of reasons.
This bias originates with the user’s intention, which goes unknown and is not captured in the rating system itself. The reasons a user may have for rating something can be many: a mood, attitude, a personal interest, a habit of use, interest in getting attention, building a profile, promoting a product, and so on.
Social media, because they provide indirect visibility in front of a mediated public, amplify any distortion baked into the selection itself (a selection being the act of rating something). This amplification is explained in part by the de-coupling of selective acts (rating) from consequences and outcomes.
Selections are de-coupled from personal consequences, which excuses a certain lack of accountability and responsibility. Selections are de-coupled from their context of use, which range from personal utility to social promotion. And selections are de-coupled from social implications, which removes the user from his or her contribution to a social outcome (eg, highly-rated items look popular).
Consider the reasons a user may have for making a selection (rating something). They include:
- personal recollection (like favoriting)
- to inform a recommendation engine (so that it can make better personal recommendations)
- because the item is a favorite (sharing favorites)
- because the social system has no accountability
- because it always creates the possibility of recognition for the user
- because it promotes the item
- because it’s nice (socially; possibly karmic)
- because it’s a gesture about how the user felt
Social selections are thus encumbered by ambiguity: of intent, of meaning, of relevance, and of use.
Can these be addressed and resolved by better system design? Or can they only be resolved by social means?
It might be possible to couple ratings with outcomes. This would involve new sets of selections and activities made available to other users and used to create consequences. Users would then consider these consequences when making a rating selection.
Contexts of use could be distinguished, so that users rate with greater purpose. This would involve creating new views of rated content, such as “rate your favorite item this wk,” “rate your favorite genre,” “rate your personal favorite,” “rate which you think is the best,” and so on. Each of these distinctions, if followed by users (!) would specify the selection by means of a different social purpose.
It might be possible to reduce ambiguity by means of some cross-referencing achieved by algorithms and relationships set up in the data structure. Without detailing these, they would probably include means by which to distinguish:
- the bias of the user him or herself, measured in terms of personal tastes
- the domain expertise of the user, as demonstrated by ratings provided by the user on other items and in which categories/genres/domains
- the social communication and signaling style of the user, which would reveal some of his/her relation to the social space
- use by other users and the public, as a measure of relevance
Cross references could then be applied when aggregating ratings, used to filter and sort the ratings sourced for averaged results. Theoretically, the system would be able to identify experts, promoters, favoriters, and others by their practices.
Social solutions might be created to supply distinctions among the different kinds of social capital involved in ratings. Such as:
- the user’s expertise (domain knowledge)
- trust capital, or the user’s standing within his/her social graph
- credibility capital, or the user’s believability, as measured in loyalty perhaps
- reputation capital, or the tendency of the user’s ratings to be referred to and cited beyond his/her immediate social graph
Finally, ratings systems can diversify possibilities for making selections, and separate communication from ratings selections so that ratings are used less for visibility and attention-seeking reasons (eg users who rate a lot).
There are too many kinds of socially-themed activities and practices in which ratings play a part for me to delve into this here. But each theme could be examined for the social benefits of ratings, for how they attribute value to the user, add value to content, and distinguish social content items to result in shared social and cultural resources. Those distinctions could be used to isolate different rating and qualification systems so that they are tighter and less biased.