As we move to the Web 2.0 world of user-created content and new notions of who should count as an “authority”, we all are bound to wonder about the believability and credibility of user reviews we come across on Amazon, Epinions, and all the other sites where anyone can rate information or tangible products.

I was recently alerted to a fascinating scholarly study of the credibility of Internet based user reviews that was published last month in First Monday, the outstanding peer reviewed journal of Internet technology and society. In that piece, which was titled, Six Degrees of Reputation, the authors used observation and special software to analyzed over 50,000 user reviews on over 10,000 books and CDs from Amazon.com

It’s worth reading the entire piece, but authors Shay David and Trevor Pinch noted at the outset of the piece that: “A preliminary literature search revealed that beyond our own experience, recent evidence suggested that many reviews are not authentic, that users are using various techniques to game the system, and that this phenomena might be widespread.”

A key conclusion of the author’s study was that:

“users who write product reviews are engaged in a variety of activities: promoting agendas, carrying out personal attacks, boosting their own and others reputations, building their own identities as reviewers, experiencing for the first time the empowerment of publication and so on”

Interestingly, the authors found one particularly common process was duplication of reviews or what they termed “self plagiarism.” Noted the authors:

“Our findings allow us to estimate that about one percent of all review data is duplicated, verbatim or with variations. The similar patterns observed across different genres of books and CD suggests that our findings will be corroborated with larger datasets”

There’s nothing completely surprising in this study, but it’s important to read. As we move into a world where business sources with increasing value also come to be rated by ordinary people on the Web—research reports, databases, new journals, it is important to be able to spot those who try to “game” the system, and tell the credible reviewers from those who are primarily only trying to achieve narrower self-interested aims.

Technorati Tag: Reputation Systems