In Defence of Metacritic for Video Games

Metacritic gets lots of negative press within the video games area. Kotaku has the most recent of many articles that decries the impact of Metacritic on video games, going so far as to say that review scores (and by extension, the aggregator sites that collate such information) harm critics, developers, publishers and gamers. IGN had a very similar article last year, asking if Metacritic was ruining the games industry.

But if Metacritic and its ilk are so terrible to video games on the whole, why does it continue to exist? Why does it hold such power? There are some pretty big reasons why and I’m going to argue that Metacritic does have a very important role to play within the industry. It’s not a perfect mechanism, sure, but a lot of the complaints about it are overblown.

MetaMechanics

Video games have to achieve much higher scores than other entertainment media to reach the same level of subjective quality.

Metacritic works by collecting the first review scores delivered by a site, weighting them in some kind of black box manner, then providing an aggregate metascore that shows a title’s overall performance. You can see the overview here, and the first thing that pops out is that Metacritic treats games score categories differently to movies, TV and music. It’s tougher for video games to get into the most positive ‘green’ score band because that’s the way most video game reviewers work – a 4.5 out of 10 doesn’t mean a title of slightly below average quality, it generally means a terrible game that is flawed in many ways. For a video title to be received positively, it must average 75%+ across its reviews.

You can clearly see the skew in review scores for both Gamespot and IGN… and this was in 2006.

Video game scores traditionally skew high on a 0 to 10 (or equivalent) scale. It’s an industry issue that hasn’t been dealt with and I’m not going to point fingers at who is to blame for this here – that’s just the facts. But it is something to keep in mind when people complain about Metacritic scores being set at certain points.

Everybody Hates The Money People

The most often leveled criticism against Metacritic is its use by publishers as a key performance indicator – if you want a contract and / or to receive a development bonus, you must receive a certain Metacritic score or better. “Why are publishers so in love with Metacritic?”, is a question often asked.

That’s simple: it’s a quantitative measure of game quality, popularity and helps forecast sales.

Working backwards to unpack that – Metacritic scores are a convenient way of estimating how well a game will sell. A title that gets 90%+ in aggregate will sell three times the number of units as a title that scored in the 80% to 89% band, and nearly eight times the number of units as a title that scored in the 70% to 79% range, within the crucial first three months post-launch period. The flawed Full Sail study that looked at how Metacritic weights its meta-score indicates there is a 0.72 correlation between Metacritic’s score and sales figures, while “[s]ales rise sharply for games listed as 83 or higher on Metacritic“. (Sidenote: given that there is extremely limited context to that correlation figure, I’d recommend using it only as a guiding figure to the relationship and not an absolute hard number.)

Yes, correlation isn’t causation, but it points to an existing relationship – as review scores go up, so do launch period sales. Which makes sense – people only want to play good games and with all the choice gamers have available, it’s very easy to get picky. The age of digital distribution means that scarcity no longer exists, so if you want a title, you can find it online and much cheaper than it was in its launch window. It’s only the ‘must have’ titles – the ones the reviewers go nuts for, provide big scores for – that gamers go out and buy at full price in launch week 1 to 12. Good reviews can help make a title immediately popular. Mixed reviews mean “wait until the sales”.

Apparently the Wii was the first console who’s game sales didn’t rise in tandem with its review scores. Yet another reason for the gaming industry to dislike this particular platform.

Other factors come into play, of course. Extensive promotions can overcome weaker review scores, but that costs a lot of extra money. There will be times where highly reviewed games didn’t sell high numbers and poorly reviewed games did, but they are the exceptions.

And yes, Metacritic stands as a surrogate measure for game quality. That’s what games reviewers do – they judge a game and provide a verdict to the public about how good it is and provide a recommendation about whether people should buy a title or not. It’s a subjective number, but that’s the nature of reviewing anything. A high Metacritic score is a proxy for saying a game is of high quality. Studios with historically high Metacritic scores have shown they can produce the kind of games that sell.

So, taking us back to publishers: Metacritic gives them a lot of information. It’s easily available. So it’s used.

There has been comment that it’s wrong to tie performance bonuses to certain minimum Metacritic scores and that it is just a way of denying developer royalties if a game sells units but doesn’t get high review scores. This may be true in some cases, but as pointed out above, Metacritic is being used as a surrogate figure for studio quality / popularity years before the game being discussed is on the shelves. In signing that contract, a developer is saying they agree to meet a certain quality / popularity target in order to get their bonus. On the surface that doesn’t seem unreasonable, although I’ll certainly accept that any studio signing into a clause requiring them to get 90%+ on Metacritic for their bonus is either over-ambitious, naive, wilfully mislead or a combination of all three.

But why not just use sales? I think this is partly because such things are often hidden – publishers only know how well their own titles have sold and not the competitions’. NPD’s game sales figures are increasingly unreliable since it can only really track retail sales and appears to take an educated guess at digital sales (and with the news that Gama Sutra is dropping their coverage of NPD sales charts, it’s pretty clear how little respect NPD now has with the industry). The company is the best place to talk about how many online units are being shifted overall is Valve and they aren’t talking. Game studios can fudge the number of units they sold during discussions – maybe they did shift 1m+ units on their last title, but that figure took 12 months and a Steam sale. Metacritic provides public information that’s a lot harder to dodge.

From Square Enix’s briefing notes – their Metacritic-based forecast indicated that Sleeping Dogs would sell roughly 2m – 2.5m units, Hitman: Absolution would sell 4.5m – 5m units and Tomb Raider would sell 5m – 6m units worldwide. One of their key points is that the US market didn’t behave as expected in purchasing these titles.

Metacritic results aren’t perfect predictors of these things, of course – the Square-Enix announcement (that’s a PDF, but only 12 pages and easy to read) showed despite high Metacritic scores for Tomb Raider, Hitman: Absolution and Sleeping Dogs these titles didn’t reach their associated sales forecasts.

Square-Enix point to a number of issues why they failed to meet the 80% – 90% sales forecast based on their Metacritic score (although Sleeping Dogs does at the lower end, just) but whatever model they are plugging Metacritic scores into (along with other factors, such as the assumed popularity of the IP, “game content, genre“ and the like) resulted in some very high sales figures that weren’t met.

(I’ve seen comments that equate this failure to reach these sales targets must have meant that these titles were unprofitable, which isn’t true – all these games could have been profitable, but the income they generated was nowhere near what was forecast.)

There’s Some Swear Words In This Next Bit

One of the most incredibly bizarre claims about Metacritic ‘ruining’ the games industry was the idea that it impacted on games reviewers. Jason Schreier, author of the Kotaku article against Metacritic, cited “the guilt of taking money out of peoples’ pockets” as a reason that Metacritic harms games reviews – he (and others like Adam Sessler) don’t like that a reviewer’s score can financially impact on a game studio, such as the case of a publisher holding back bonus payments when Metacritic targets aren’t met.

Which raises the question: do they understand what the role of ‘games reviewer’ actually means? Do they understand the fucking job?

A video game reviewer’s role is to experience a game for themselves and to recount that subjective experience back to a wider audience, generally providing a recommendation (or not) to others about playing / buying the title. For reviewers at some sites – particularly at the larger end of the paid games site spectrum, such as GameSpot, IGN and Kotaku – this recommendation can reach millions of people. It directly influences whether or not people purchase a game – as shown above, good reviews encourage purchase.

Game reviewers directly impact on the financial success of a studio with or without Metacritic. If you can’t live with the idea that what your gaming reviews could cost people their jobs, don’t be a fucking games reviewer. “It’s just, like, my opinion, man,” doesn’t hold up when you are being paid to give that opinion and other decisions are going to be based on what you say.

Negative (and even mixed) review scores are an anchor on game sales, which means that games reviewers collectively have a lot of power in the gaming industry. (Image source: Wikimedia Commons)

If games reviews didn’t influence public opinion, then there would be no “PR pressure”, no free swag and no exclusive launch events for games reviewers. Games reviews wouldn’t form one of the ‘Big Three’ for game site content, along side previews and news. But good reviews are critical to the success of most video games, so they have power. This is without Metacritic ever becoming involved. Metacritic’s importance exists ONLY because video game reviewers have such a big impact on sales through their reviews, with a review score being an easy thing to compare and evaluate.

(One interesting bit of evidence here showing that good reviews make players more interested in certain titles is the positive correlation between review scores and game piracy rates – the better the review scores, the more times a game is likely to be pirated.)

Metacritic just takes all of those review scores and distills them into a single weighted average. Review scores remain a subjective figure, but it’s a quantitative subjective figure that’s easier to understand than an 800-word treatise of gameplay that may still leave the reader unsure if the title is being recommended or not. I’ve certainly read reviews where the writer has been vague about whether a title has been a good experience, or has been quite critical about game elements, only to provide a high review score.

There have been calls for video games to do away with the review score, but the replacement appears to be the pure text of the review itself… which just so happens to be a way of drawing people to multiple game sites rather than an aggregator.

Professional Jealousy

In an era where something like 1 in 2 visitors to major gaming websites block their revenue-generating ads while consuming the content at an ever increasing pace, every unique site visitor counts. Given this problem for gaming sites, aggregators like Metacritic (and there is also GameRankings) are parasitic in that they potentially steal site visitors by scraping and republishing key content – reviews. If a gamer wants to know how a game has been rated, they could search several game sites for a review, or they could just visit one site like Metacritic and get all the information they want at a glance.

Of course, if that gamer wants more information, they can always click on a link within Metacritic’s pages to read each review individually… but they probably won’t. A review score and single paragraph recap per site is going to be enough for a lot of people. Metacritic has the impact of taking potential readers from other sites.

Every game takes from their source, adds a little bit of something to make their article ‘original’, then hands it down to the next layer who do the same thing again. Everyone steals from each other. Review scores aren’t anything different. (Image Source: Wikimedia Commons)

However, it’s hard for gaming sites to complain about Metacritic taking their material since that’s what EVERY games site does to some extent – they take stories reported elsewhere and rewrite them as content for their site. It’s just the way the ecosystem works. What makes review scores different is that by collating them and providing a weighted average, Metacritic arguably adds value to what they take from other sites. If all you want is a score (perhaps on the way to confirming that you should / shouldn’t buy a particular game, or a quick recap on what a large number of sites said about a title e.g. Bioshock: Infinite on Metacritic lists over 100 reviews across all platforms and the PC version alone has 64 reviews to form the Metascore) then Metacritic is one of the most convenient sources for such information.

Reviewers may not like the idea that gamers will focus on the review score, and by extension the Metacritic score, when they’ve spent so long crafting a written review. That’s understandable. But games reviews are ultimately incredibly disposable things, most useful when a title launches and often quickly forgotten. The score at the end of the review is the shortest path to knowing how a games reviewer rates a title, with Metacritic serving as a convenient summary across a wide range of sites.

Let’s face it – for all the criticism of Metacritic, it isn’t doing much more than summarising (through weighted average, which I’ll get to) what other games reviews say. When games reviewers criticise Metacritic for somehow distorting the industry, I can’t help but see these reviewers criticising the end result of their own activities.

So Why Is Metacritic So Powerful for Video Games?

As pointed out in other articles, Metacritic is uniquely powerful within video games compared to other entertainment industries. Sure, it hurts if Rotten Tomatoes (another review aggregator) rates a movie as ‘rotten’ or a Pitchfork review dismisses an album, but it doesn’t necessarily have the same impact as a low Metacritic score. I think there’s a couple of reasons for this power.

• Video games (at the AAA level) cost end consumers a lot more than blockbuster movies or top-of-the-charts albums to buy. Amazon lists a number of top selling albums right now for less than $15 and just released movies for less than $25. New video games are costed closer to the $50 to $60 range. With more money involved in the purchase, people are likely more hesitant in buying a title with mixed reviews for full price.
• 

  Although well done, I’m not sure that huge dollars are made on the sale of these kind of things, at least at the individual studio level.

  Both movies and music have multiple potential channels to make money from. Movies can make money through cinema distribution, international licensing, licensing rights to television networks, merchandising, DVD releases, collection re-releases, and so on. Music can sell either in album or single song form, can also benefit from licensing deals with radio or movies or advertising or digital services like Spotify, merchandising to fans, plus the artist / band can tour. Video games really only have direct sales (e.g. through box sales and / or microtransactions), with limited merchandising opportunities outside of niche markets (e.g. children around the world aren’t clamouring for a Boy of Silence figurine). Video games traditionally have one, big, front-loaded opportunity to make their development budget back. If the sales aren’t there within that 3 months immediately post launch, it is fairly likely that those sales aren’t ever going to come through in most instances (at least until the heavily discounted Steam sale, anyway).

• People buy ‘old’ (in entertainment terms, 12 months can be a very long time) movies and music from ‘old’ artists that they think they’ll like, or because they know they liked it. In contrast to these other media, video games date very quickly and often became very hard to find (at least in a way where the creators get paid for a consumer obtaining the title). Even if you could find a copy of a title you liked, there was no guarantee it would still work on newer operating systems. This is another reason that the video game industry is almost entirely focused on short-term launch period sales – there wasn’t seen to be much value in having a large gaming back catalogue because it wouldn’t generate much revenue except through using the IP to develop sequels / offshoots. This may be changing with the increasing popularity of digital sales, but it will be a rare video game title that will remain popular after 20+ years as is the case with movies and music.
• Movies and video games can both cost hundreds of millions of dollars to develop, but movies have (as indicated in the point above) a long history of deals and licensing arrangements that help reduce their risk and guarantee some income. Video games don’t have that. As far as music goes, the most expensive album to every produce is believed to be Guns’N’Roses’ “Chinese Democracy” at a cost of over US$13m; a movie or video game with a development budget at that level would be considered small (or to sound nicer, “indie”).
• People will buy ‘bad’ movies to make fun of them (e.g. “The Room”) and get some kind of kitsch joy out of bad songs (e.g. the career of William Hung) but bad video games are, for the most part, completely unenjoyable. A negative review for a video game is an indication of multiple failings within a title, especially within its gameplay. A bad video game quickly disappears into the mists of history, and with it all chance of earning back its development budget.
• As indicated above, video games sales data is limited for those wanting a ‘full market’ picture. In contrast, how financially successful a movie was is easily available, while music sales are tracked by Nielsen Soundscan and available for a paid subscription. The only common source of information about how well a video game has performed is from its review scores (unless the studio / publisher behind the title reveals their sales figures, which they usually don’t).

In short, most AAA video games are expensive productions that have one real shot at making back their money, where music and movies have more ways of generating revenue. For a video game publisher, this means a big roll of the dice at launch, making it all the more important to have good reviews that encourage immediate purchase. Metacritic has ended up at the pointy end of that particular situation, acting as a barometer of sorts for how the gaming market is likely to receive a title.

Complaints About Metacritic

Three complaints I’ve seen about how Metacritic develops its metascore are:

1. Review scores are weighted according to some secret black box formula, so they can’t be trusted;
2. Review scales are adjusted to fit a 0 to 100 range, which isn’t always appropriate to the original scale; and
3. Metacritic takes the first review published by a site and then refuses to adjust if that score is changed.

I understand the complaints raised with Metacritic, but I don’t think there is anything strong enough that should stop people from using this resource. (Image source: Flickr)

I’ve looked in detail at how Metacritic weights its data when creating the metascore and found that it isn’t notably different than the unweighted average of the same scores. So that first point isn’t really an issue, even if the weightings are kept secret.

Scale conversion issues is a valid concern – changing an A – F grade score into a 0 to 100 score is a tricky process. It should also be recognised that the same review score at one site may not mean the same thing as another site – one site may try to make a score of 50 the ‘average’ game score where another might use a review score of 70 to indicate the same thing. There should be some consideration of this when Metacritic creates the metascore – Metacritic says it takes care with these conversions and tries to use a consistent approach, which is definitely a positive in its favour.

On the third point, I agree with Metacritic: game review sites should be leading with their most accurate review score up front. On Day 1 Polygon was out there telling players how great SimCity was, recommending to players that they buy it. Some players may have followed that recommendation. For them it’s no good that Polygon later walks back that review score and says things have changed. Review sites should be willing to lead with their best efforts, especially if they are going to try to also be the first out of the gate with a Day 1 review. (And as I’ve shown, Day 1 reviews tend to score higher than non-Day 1 reviews, meaning that these first reviews are often among the most likely to recommend games buy / play a title. Don’t trust Day 1 reviews!)

None of these complaints invalidate Metacritic as a source of information in my opinion.

So Why The Criticism of Metacritic?

In summary, I think the major criticism of Metacritic isn’t even Metacritic’s fault: it is that publishers use it for commercial purposes. Gamers love to complain about publishers (as do a number of game studios, for that matter), so anything that publishers use is fair game to attack.

Game reviewers don’t like Metacritic because it takes their work and aggregates it on a separate site, thus potentially reducing unique visitor numbers to their site. There’s some harping about how Metacritic’s single aggregate score shouldn’t be trusted, but this seems to be a narrow argument from any game site that use a review score system themselves – if you criticise one site’s scores, you certainly open the door to having your own scoring system questioned about why it is better.

After all, if you make the argument that Metacritic scores are often inconsistent and arbitrary, the same problem exists for stand alone game review scores, and even game reviews that don’t contain a score at the end. It’s just the nature of the beast when it come to reviewing an entertainment product.

Metacritic may not be absolutely perfect every time, but there’s little use making perfect the target when something is good enough most of the time. (Image Source: DeargRuadher on DeviantArt)

Game developers don’t like Metacritic because it can be tied to performance bonuses, but given that Metacritic is just ‘aggregate review scores’ this shouldn’t be considered unfair in principle – it’s a performance metric. Publishers want games with high sales (and so do game studios, to be honest), and high review scores show a link to this end goal. These kind of performance goals can certainly be set too high, but then it is up to the studio to fight for more realistic targets (or: find another publisher) before signing the contract.

Metacritic has such power because overall the games industry is information poor, highly focused on a limited sales period and collectively game reviews have a lot of influence over game buyers. Until a better source of information comes along, Metacritic – or something like it – is going to continue to be at the nexus of gamers, publishers, studios and game reviewers. Metacritic may not be a perfect source of information for all these groups, but it is good enough in being accessible, convenient and easily understandable.

The thing that will replace Metacritic isn’t going to be something that relies on review scores less (unless review scores and sales stop correlating); it will probably be something that takes everything on Metacritic and adds in social media monitoring plus sentiment analysis. (If you happen to create this service and make millions from it, you know where to reach me so we can talk stock options.)