Valve - Positive "Review Bombs"

[Chairmanchuck (另一个我)]

Valve published a quite long article about how to deal with positive "review bombs":

https://steamcommunity.com/games/593110/announcements/detail/1621770561051427036

It is about the positive reviews that appeared on Steam after Ubisoft gave away AC:Unity for free on uPlay and whether those reviews should be counted as reviews or be excluded with their new "review bomb" policy.
TL;DR: Valve decided that they will not exclude those reviews. Reasons are listed in the article. But the article ends with an invitation to discuss how to deal with it.

Valve also talks about how important reviews are for the sales of a game:

A Mixed game receives over 500% more boost than a game in Mostly Negative. That might seem scary, but we're still talking about a boost that's small relative to many other store factors, and it's the minority case - 71.7% of titles on the Store are Mixed or above.

Click to shrink...

Mod Edit: Further context in the post below.

The OP leaves out everything related to why leaving positive reviews up is ok like after an update or a giveaway or context change such as in this case it's AC Unity in light of the Notre Dame fire. Most people don't read the source, so here's an excerpt and the whole article is worth reading first.

A few weeks ago, in response to the Notre Dame tragedy, Ubisoft did something great for their fans by making Assassin's Creed: Unity available for free on Uplay, and committing funds towards rebuilding the monument. This led to a significant spike in players of AC: Unity on Steam, and a large number of positive reviews for the game. This led us, and members of the community, to wonder if this was finally a positive review bomb, and whether it should be considered off-topic.​

​

​

Data-wise, it doesn't quite fit the pattern of negative review bombs: in the case of AC:Unity there was a significant increase in actual players alongside the increase in reviews. That isn't necessarily the case with a typical off-topic review bomb (but, to be clear, we have seen some negative review bombs with that characteristic).​

​

Without reading the actual reviews, the data here all looks very much like a game that's gone on sale, or received an update. It's seen a spike in players, and as many people have come to realize, there's a fairly good correlation between player count and user reviews - if you get more players, you're going to get more reviews.​

​

But we also went and read a large chunk of the reviews. Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago. Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more.​

​

So it's not clear it's a review bomb. It certainly doesn't fit our original definition in the "aimed at lowering the Review Score" section, but back in 2017 the community's terminology around "review bombs" was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community.​

Click to shrink...

 

[Crayon]

Hmm. Never thought about that. That is sort of a little conundrum. It would feel good to leave them up, on the other hand it's definitely a review bomb that has to do with reasons other than whether or not you might like the game.

 

[olag]

I'm not sure they are sending the correct message by suppressing one end of the spectrum here.

Ideally both should be isolated as positive review bombs can be abused as well.

 

[LiK]

Can we call them Review Parties instead

 

[OnionPowder]

how long until people positive review bomb with only negative comments to try and keep their reviews up?

 

[NoblesseOblige]

Positive review bombs results in more sales. They have an inherent interest in letting them remain (more money for Valve). Nevermind that they're exploited by shitty people the same way negative are (ie certain games having received mass positive reviews to 'stick it to the SJW').

 

[Massicot]

Looks like one of the factors in their decision to not mark them as off-topic was the increase in players that coincided with the increase in number of review scores, and that it's at the moment impossible to say whether that's a characteristic of 'positive review bombs' or just this specific case because they're so rare (pretty much unheard of until this).

 

[Deleted member 249]

I'm not sure they are sending the correct message by suppressing one end of the spectrum here.

Ideally both should be isolated as positive review bombs can be abused as well.

Click to shrink...

Maybe they can do it on a case by case basis.
In the case of Unity, I think they made the right call.

 

[aerozombie]

For the reasons behind AC Unity it will definitely need to be case by case on positive ones. Especially since many games improve drastically via updates over time

 

[finalflame]

It's always good to see that Valve continues to be basically the only company thinking critically about things like this.

 

[BeI]

I don't really oppose positive review bombs as long as it's not abused. It's better than the alternative that could drive devs away.

 

[Parsnip]

Judging a situation on a case by case basis is the right move here.
And leaving AC: Unity score untouched is also the right move.


Also interesting tidbit I didn't know in there about how mixed and above get a similar boost in the algorithm, but also that the review score boost is small relative to other factors.

 

[Messofanego]

The OP leaves out everything related to why leaving positive reviews up is ok like after an update or a giveaway or context change such as in this case it's AC Unity in light of the Notre Dame fire. Most people don't read the source, so here's an excerpt and the whole article is worth reading first.

A few weeks ago, in response to the Notre Dame tragedy, Ubisoft did something great for their fans by making Assassin's Creed: Unity available for free on Uplay, and committing funds towards rebuilding the monument. This led to a significant spike in players of AC: Unity on Steam, and a large number of positive reviews for the game. This led us, and members of the community, to wonder if this was finally a positive review bomb, and whether it should be considered off-topic.​

​

​

Data-wise, it doesn't quite fit the pattern of negative review bombs: in the case of AC:Unity there was a significant increase in actual players alongside the increase in reviews. That isn't necessarily the case with a typical off-topic review bomb (but, to be clear, we have seen some negative review bombs with that characteristic).​

​

Without reading the actual reviews, the data here all looks very much like a game that's gone on sale, or received an update. It's seen a spike in players, and as many people have come to realize, there's a fairly good correlation between player count and user reviews - if you get more players, you're going to get more reviews.​

​

But we also went and read a large chunk of the reviews. Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago. Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more.​

​

So it's not clear it's a review bomb. It certainly doesn't fit our original definition in the "aimed at lowering the Review Score" section, but back in 2017 the community's terminology around "review bombs" was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community.​

 

[Deleted member 249]

It's always good to see that Valve continues to be basically the only company thinking critically about things like this.

Click to shrink...

In general, I would hardly credit them for thinking "critically" on issues like this, especially given their approach to selecting what they do and do not allow on their store is "lol fuck it".

In this case, I agree they have done well.

 

[olag]

Maybe they can do it on a case by case basis.
In the case of Unity, I think they made the right call.

Click to shrink...

This would be the best case scenario, but it seems valve are looking for a large scale solution to perhaps avoid additional labour costs ?

Either way looking through the source, I can see why they'd be persuaded to do nothing.

 

[Deleted member 249]

This would be the best case scenario, but it seems valve are looking for a large scale solution to avoid perhaps avoid additional labour costs ?

Click to shrink...

While I sympathize with that, I do think positive review bombs are so rare that it won't necessitate too much effort on their end to stay on top of things.
Of course, even positive review bombs can be weaponized by determined trolls, and I get that they want to stay ahead of that.

 

[Delusibeta]

Looks like one of the factors in their decision to not mark them as off-topic was the increase in players that coincided with the increase in number of review scores, and that it's at the moment impossible to say whether that's a characteristic of 'positive review bombs' or just this specific case because they're so rare (pretty much unheard of until this).

Click to shrink...

Reading the article, it appears that from Valve's perspective, it looked closer to what would be expected from a deep sale (which is basically what happened, albeit outside of Steam) than a review bomb. The review bomb policy also stated that they would take everything on a case-by-case basis, and here they concluded that they didn't have enough evidence to take action. What's unusual is that they publicly posted both the conclusion and the reasoning behind it. This is still Valve, they aren't known for communicating.

 

[Chairmanchuck (另一个我)]

The OP leaves out everything related to why leaving positive reviews up is ok like after an update or a giveaway or context change such as in this case it's AC Unity in light of the Notre Dame fire. Most people don't read the source, so here's an excerpt and the whole article is worth reading first.

A few weeks ago, in response to the Notre Dame tragedy, Ubisoft did something great for their fans by making Assassin's Creed: Unity available for free on Uplay, and committing funds towards rebuilding the monument. This led to a significant spike in players of AC: Unity on Steam, and a large number of positive reviews for the game. This led us, and members of the community, to wonder if this was finally a positive review bomb, and whether it should be considered off-topic.​

​

​

Data-wise, it doesn't quite fit the pattern of negative review bombs: in the case of AC:Unity there was a significant increase in actual players alongside the increase in reviews. That isn't necessarily the case with a typical off-topic review bomb (but, to be clear, we have seen some negative review bombs with that characteristic).​

​

Without reading the actual reviews, the data here all looks very much like a game that's gone on sale, or received an update. It's seen a spike in players, and as many people have come to realize, there's a fairly good correlation between player count and user reviews - if you get more players, you're going to get more reviews.​

​

But we also went and read a large chunk of the reviews. Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago. Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more.​

​

So it's not clear it's a review bomb. It certainly doesn't fit our original definition in the "aimed at lowering the Review Score" section, but back in 2017 the community's terminology around "review bombs" was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community.​

Click to shrink...

I left it out so people can read the source before commenting just based on the thread title and my short "summary".

 

[finalflame]

In general, I would hardly credit them for thinking "critically" on issues like this, especially given their approach to selecting what they do and do not allow on their store is "lol fuck it".

In this case, I agree they have done well.

Click to shrink...

This is a gross mischaracterization of how Valve handles Steam. They moved to an open publishing model deliberately, and have since been building self-service and community led tools that let people dictate their own experience. You're free to use the Steam Curators feature for a curated selection of games.

I see no other companies nor storefronts build your as many user-centric features around their platform as Valve does. I'd be happy to be proven wrong.

The OP leaves out everything related to why leaving positive reviews up is ok like after an update or a giveaway or context change such as in this case it's AC Unity in light of the Notre Dame fire. Most people don't read the source, so here's an excerpt and the whole article is worth reading first.

A few weeks ago, in response to the Notre Dame tragedy, Ubisoft did something great for their fans by making Assassin's Creed: Unity available for free on Uplay, and committing funds towards rebuilding the monument. This led to a significant spike in players of AC: Unity on Steam, and a large number of positive reviews for the game. This led us, and members of the community, to wonder if this was finally a positive review bomb, and whether it should be considered off-topic.​

​

​

Data-wise, it doesn't quite fit the pattern of negative review bombs: in the case of AC:Unity there was a significant increase in actual players alongside the increase in reviews. That isn't necessarily the case with a typical off-topic review bomb (but, to be clear, we have seen some negative review bombs with that characteristic).​

​

Without reading the actual reviews, the data here all looks very much like a game that's gone on sale, or received an update. It's seen a spike in players, and as many people have come to realize, there's a fairly good correlation between player count and user reviews - if you get more players, you're going to get more reviews.​

​

But we also went and read a large chunk of the reviews. Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago. Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more.​

​

So it's not clear it's a review bomb. It certainly doesn't fit our original definition in the "aimed at lowering the Review Score" section, but back in 2017 the community's terminology around "review bombs" was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community.​

Click to shrink...

Thanks for posting. Asking ERA to read things and think with nuance seems like a lot these days. Seems basically every discussion has to be painted in broad strokes.

 

[Nome]

Great insight into Valve's data-informed approach.

 

[Neil98]

Reviews made by customers are bad, leave that to press. Customers should only be able to comment, not review (just my opinion, though)

 

[SteveWinwood]

Reviews made by customers are bad, leave that to press. Customers should only be able to comment, not review (just my opinion, though)

Click to shrink...

Yeah why don't the press just review all of the hundreds of games that come out each month.

 

[papertowel]

Reviews made by customers are bad, leave that to press. Customers should only be able to comment, not review (just my opinion, though)

Click to shrink...

What about all the games that don't get reviewed by professionals

 

[eonden]

Great insight into Valve's data-informed approach.

Click to shrink...

You might be interested on this post.

It was also right on the reviews having no effect until being Negative:

Info: your user review rating doesn't affect visibility, unless it's 'mostly negative' or worse
This is straight from Valve, confirmed twice. User review % has no effect on when or where the Steam store promotes your game, unless it's 'mostly negative' or worse. That's not to say it won't affect sales, of course: if I see something's 'mixed' I at least scroll down to find out why. Steam also surfaces your user review rating in many places other than your store page, so while it won't affect when it gets shown, it might affect how many people click through.

Still, not worth stressing over dropping from 'overwhelmingly positive' to 'very'.

Click to shrink...

 

[Detail]

I disagree with them, if you are going to suppress review bombs it shouldn't matter if they are positive or negative.

Why is it ok to suppress consumer opinion that is critical of a product or company but it's ok to allow for opinion that speaks of a company in a positive manner?

It should be both ok or none ok imo.

 

[LewieP]

Yeah I don't think I agree with the decision to allow these reviews when they protect games from negative reviews related to DRM or a publisher's conduct outside of the game.

I think reviews should be moderated for things like hate speech and abuse, but outside of that I think pretty much everything else is relevant. I don't like this inconsistency.

 

[¡ B 0 0 P !]

Reviews made by customers are bad, leave that to press. Customers should only be able to comment, not review (just my opinion, though)

Click to shrink...

Of the 30,000 games on Steam how many do you think have been reviewed by the press?

And for the ones who have reviews how many of those do you think have more than 10 reviews each?

 

[ramoisdead]

Reviews made by customers are bad, leave that to press. Customers should only be able to comment, not review (just my opinion, though)

Click to shrink...

I think your opinion lacks any merit (just my opinion, though!).

 

[Detail]

Reviews made by customers are bad, leave that to press. Customers should only be able to comment, not review (just my opinion, though)

Click to shrink...

You don't work for Epic do you? :-D.

Jokes aside, I can't agree with you here, consumer reviews are imperative to a fair and trustworthy pro-consumer marketplace imo.

 

[collige]

I always wonder which specific employee writes these blog posts.

Yeah I don't think I agree with this decision, much like I don't agree with their decision to protect games from negative reviews related to DRM or a publisher's conduct outside of the game.

I think reviews should be moderated for things like hate speech and abuse, but outside of that I think pretty much everything else is relevant.

Click to shrink...

I'm not sure I follow this logic. I'm also in the camp that their new position on review bombs is dumb, but since they determined that this isn't a review bomb, wouldn't doing nothing be the correct decision?

 

[Deleted member 249]

This is a gross mischaracterization of how Valve handles Steam. They moved to an open publishing model deliberately, and have since been building self-service and community led tools that let people dictate their own experience. You're free to use the Steam Curators feature for a curated selection of games.

Click to shrink...

Please DM me if you wish to continue this further, if only to keep this thread on track, but I feel like Valve's laissez faire is lacking and harmful. They selectively decide to ban games (such as visual novels) from their storefront, citing sexually provocative content- which I absolutely agree with them removing. But allow games like Hatred or Agony or Postal to continue to exist on their store.

So no, I don't think Valve is as oriented towards a community led experience as they would like people to believe at all.

 

[MegaXZero]

I disagree with them, if you are going to suppress review bombs it shouldn't matter if they are positive or negative.

Why is it ok to suppress consumer opinion that is critical of a product or company but it's ok to allow for opinion that speaks of a company in a positive manner?

It should be both ok or none ok imo.

Click to shrink...

Read the post by valve instead of just the headline. Because part of their reasoning is this isn't a review bomb by definition and it had an increase in players, which is typical of a sale/update. There is also the matter of they only removed off topic review bombs which this doesn't quite fit as the game has added value now in having a recreation of a destroyed monument.

 

[Pottuvoi]

Solution is ignoring binary voters in scoring.
When distribution curve shows average at 6 and huge spikes at 1 and 10, we know what to ignore.

 

[SteveWinwood]

Solution is ignoring binary voters in scoring.
When distribution curve shows average at 6 and huge spikes at 1 and 10, we know what to ignore.

Click to shrink...

you don't use numbers in steam review scores

 

[collige]

Solution is ignoring binary voters in scoring.
When distribution curve shows average at 6 and huge spikes at 1 and 10, we know what to ignore.

Click to shrink...

Steam reviews are already binary. Data has shown that people tend to vote for extremes anyway, so a lot of other platforms have change to just using thumbs up/down (Youtube and Netflix being notable examples)

 

[LewieP]

I'm not sure I follow this logic. I'm also in the camp that their new position on review bombs is dumb, but since they determined that this isn't a review bomb, wouldn't doing nothing be the correct decision?

Click to shrink...

I don't understand what you mean.

I think they should let these types of reviews stand, and I think they should let complaints about DRM stand. If you disagree that's fine, but I'm not sure what you are suggesting is illogical about that.

 

[Jay Shadow]

I disagree with them, if you are going to suppress review bombs it shouldn't matter if they are positive or negative.

Why is it ok to suppress consumer opinion that is critical of a product or company but it's ok to allow for opinion that speaks of a company in a positive manner?

It should be both ok or none ok imo.

Click to shrink...

Did you read the entire article? They go over that in pretty heavy detail. Basically it boils down to:

1. It fit the pattern of a Steam sale more than it did a review bomb. There was a big influx of players.
2. This included a lot of players returning to the game since they last played before patches.
3. Because of those two factors, most of the influx of positive reviews actually were reviewing the game (either for the first time, or for the changes made after they last played), rather than being hype posts.
4. The reviews in the end effected the overall score less than 2% and it still hasn't risen above "mixed"

This was a specific case study for this one instance, which was the first example they've had. And in this case it fit the Sale pattern more than the Review Bomb pattern. They'll re-evaluate if the situation arises again.

 

[collige]

I don't understand what you mean.

I think they should let these types of reviews stand, and I think they should let complaints about DRM stand. If you disagree that's fine, but I'm not sure what you are suggesting is illogical about that.

Click to shrink...

I agree with your stance. The decision here was to let the reviews for Unity stand, which is why I was confused.

 

[LewieP]

I agree with your stance. The decision here was to let the reviews for Unity stand, which is why I was confused.

Click to shrink...

I disagree with the inconsistency. I think they should let both positive and negative reviews stand.

Edit: maybe my original post was a little unclear. I've reworded it.

 

[Pottuvoi]

you don't use numbers in steam review scores

Click to shrink...

Really?
Thought they did some year.

Steam reviews are already binary. Data has shown that people tend to vote for extremes anyway, so a lot of other platforms have change to just using thumbs up/down (Youtube and Netflix being notable examples)

Click to shrink...

Making voting system worse and reduce usable data may not be best solution.
Let people vote how they please and use it to strengthen the analysis.

Ie.
Someone uses good gradient in voting, yet follows every bombing trend can be a lot easier to identify than from binary vote.

Someone only votes extremes so use weighted scale for them etc.

 

[Neil98]

I think your opinion lacks any merit (just my opinion, though!).

Click to shrink...

Well I respect your opinion. Have a good day! :)

 

[collige]

Really?
Thought they did some year.

Making voting system worse and reduce usable data may not be best solution.
Let people vote how they please and use it to strengthen the analysis.

Ie.
Someone uses good gradient in voting, yet follows every bombing trend can be a lot easier to identify than from binary vote.

Someone only votes extremes so use weighted scale for them etc.

Click to shrink...

I don't think having an invisible system that fucks with people's review scores is going to inspire more customer confidence in the system.

 

[Pottuvoi]

I don't think having an invisible system that fucks with people's review scores is going to inspire more customer confidence in the system.

Click to shrink...

There is no need for it to be invisible.
Pretty sure you can show the weights and how they are calculated.

Anyway, if they have working system for making binary work better, excellent.

 

[Neil98]

Of the 30,000 games on Steam how many do you think have been reviewed by the press?

And for the ones who have reviews how many of those do you think have more than 10 reviews each?

Click to shrink...

I didn't take that into account, my apologies. But anyways, how many of those games are actual games and not some flash game-like experiences? Or don't deserve to get reviewed for pedo content? I feel as though a comment system would be much better for Steam. That's how you would actually have to do some research and see other people's opinions on games, and not just look at how many likes or dislikes a game has.

 

[Neil98]

You don't work for Epic do you? :-D.

Jokes aside, I can't agree with you here, consumer reviews are imperative to a fair and trustworthy pro-consumer marketplace imo.

Click to shrink...

Tell that to Metacritic's User Score.

 

[eonden]

Tell that to Metacritic's User Score.

Click to shrink...

Steam requires owning the game to review it, they are not the same as Metacritic.

I didn't take that into account, my apologies. But anyways, how many of those games are actual games and not some flash game-like experiences? Or don't deserve to get reviewed for pedo content? I feel as though a comment system would be much better for Steam. That's how you would actually have to do some research and see other people's opinions on games, and not just look at how many likes or dislikes a game has.

Click to shrink...

It says in the Steam post that 70% of the games have a mixed or better ratings. The myth of "Steam is full of trash there arent that many good games" is just that, a myth.

 

[xch1n]

That was a really detailed, really informative, thoughtful blog post. And yet, some comments here and on that article seem to be posted without reading, or about a completely different topic/sub-topic. Why is this so hard for people? Why is getting a one-liner opinion more important that thoughtfully reading what took someone a long time not only to write, but also to research?

The Unity situation is really unique given its poor initial reception and its long road of updates. I especially liked where they called out how context affects people's interest in a game, as that's either forgotten or ignored in a lot of discussion around critical reception. Usually the token "right game, wrong time," but in this particular case, it's really fascinating.

 

[SapientWolf]

In general, I would hardly credit them for thinking "critically" on issues like this, especially given their approach to selecting what they do and do not allow on their store is "lol fuck it".

In this case, I agree they have done well.

Click to shrink...

There was actually a ton of internal debate which lead to Valve starting, and then dropping Steam Greenlight. But I think they're still in the process of learning what content their own community will and will not tolerate. They have been reactionary rather than proactive, but they're at least using the feedback in their decision process. Whereas before Greenlight the community really had little to no say in what Valve allowed on Steam.

In this "review bomb" case, people are actually playing the game so I think their reasoning for leaving them up is valid.

 

[Deleted member 249]

There was actually a ton of internal debate which lead to Valve starting, and then dropping Steam Greenlight. But I think they're still in the process of learning what content their own community will and will not tolerate. They have been reactionary rather than proactive, but they're at least using the feedback in their decision process. Whereas before Greenlight the community really had little to no say in what Valve allowed on Steam.

In this "review bomb" case, people are actually playing the game so I think their reasoning for leaving them up is valid.

Click to shrink...

Oh yeah, in this situation I think Valve acted really well, and I think they deserve nothing but credit for that. I was speaking to the more general "Valve thinks critically" sentiment.

 

[SteveWinwood]

Oh yeah, in this situation I think Valve acted really well, and I think they deserve nothing but credit for that. I was speaking to the more general "Valve thinks critically" sentiment.

Click to shrink...

well the person who used that phrase initially worked there so I'd imagine they might have some idea about it