The ESA Gaming Report has Serious Problems

We made a video to talk about the serious issues involved in the new Entertainment Software Association report. Following the video is a written transcript. Please note that the video is, obviously, a visual form, so you don’t get the full context with only the transcription.

Transcription

The Entertainment Software Association – the ESA – came out with their annual report on Wednesday, April 19. Essential Facts about the computer and video game industry: sales, demographic, and usage data is supposed to provide us with information about the American public, like how many of us play video games, for example. But as a former survey professional myself I immediately noticed a major problem with this survey and it calls into question the trustworthiness of the survey. Let’s talk about one of my favorite topics – data – in this Game Objective feature video.

First, a little background. The annual ESA survey is widely reported and cited. Not only does the media jump all over its well presented infographics and easy-to-access data, but academia cites the report all the time as well. Whenever people refer to half of gamers being women, they are almost always referring to ESA statistics. Whenever you see an article on the survey it almost always presents the data with no context and without question. That’s not what I do here at Game Objective so I decided to take a closer look.

On the 2nd page of the report I immediately noticed a problem. Take a look and see. The ESA is reporting that 97% of American households have a personal computer. 97%. There’s no way that’s accurate. The United States is wealthy, yes, but there’s a lot of poverty as well. So I looked for any possible corroborative data and came upon Pew Research. Pew is a respected research “fact tank” and they have indeed looked at computer ownership. And guess what? Their numbers don’t match the ESA’s.

Since 2008 Pew has reported between 73% and 78% of Americans own a computer. That’s a good 20% difference with the ESA survey, which is certainly outside the margin of error. That leaves us with three possibilities here: the ESA survey is wrong, the Pew survey is wrong, or both are wrong.

The next step into looking at what’s going on here is to look at the methodology – how the survey was conducted. If you know how the survey is conducted you can repeat the process and look at changes over time. That’s why you see the Pew having a small range difference over time. And that’s what you’d expect with computer ownership since it’s largely a stable product in a stable market.

So here we have the Pew methodology and we have 5 or 6 paragraphs talking about how it’s a phone survey done with both cell phones and landlines from a national sample. They talk about how it’s weighted. This is how it should be done.

Then we have the ESA survey. The only methodology that they’ll go on the record with is what they presented in their press release. “Essential Facts About the Computer and Video Game Industry is the most in-depth and targeted annual survey of its kind. Ipsos MediaCT conducted the research for ESA and surveyed more than 4,000 American households.”

That doesn’t help at all.

I also talked to Pew Research and other industry insiders but, unfortunately, it’svery difficult to get someone to go on the record to criticize someone else’s survey.

So where does that leave us? Well, we have Pew telling us around ¾ of Americans have a personal computer and they tell us exactly how they came to that number. Then we have the ESA telling us very nearly every single household in the United States has a computer and they won’t tell us how they arrived at that figure.

To put it plainly, I don’t trust the ESA numbers. If 97% of their respondents own a personal computer then their sample is biased in favor of people who are more technologically inclined. If that’s the case then it’s likely that these people are more likely to be gamers, which means that the ESA estimate of the number of gamers may be too high. This calls into question whether or not the survey accurately represents Americans. People of lower socio-economic status are less likely to own a computer. Are they fairly represented in the survey? Do they account for older females being more likely to opt-in to a survey and young males being less likely? We don’t know because they won’t provide their methodology. And that’s the problem with a lack of transparency. If you can’t trust one number in a survey, how do you trust the rest of the numbers?

The fact is their numbers are screwy and their transparency is non-existent. I have to question whether the ESA survey is actually helping the game industry or if it’s just passing along biased information.

Now, there are a couple more problems with the survey I need to address because I know my audience has talked about at least one of them. First, you’ll notice in their infographic that they list virtual reality devices at 11% market share. That is, of course, somewhat misleading. The vast majority of those devices are cell phones. And I noticed that PC Gamer actually picked up on that and questioned the ESA report. Well done, PC Gamer. Good job.

The second issue is the always fun talk of gender politics. In the ESA report they use their gender statistics in an unusual way. They say, “Women age 18 and older represent a significantly greater portion of the game-playing population than boys under age 18.” It looks like they’re trying to contradict the notion that games are for young boys but this is an oddly presented statistic. If you think about it, is it really worth mentioning that there are more women between the ages of 18 and 118 playing games than boys age 0 to 18 playing games?

Those are certainly two problems with the survey, but the thing is, if the survey is not representative and the sample has bias issues, then all of that data is called into question anyways. That’s how important proper sampling is. It’s the foundation of everything. So, the moral of the story here is that this survey is, unfortunately, not a very useful tool for understanding the American gaming population. And that’s too bad. The ESA needs to get good and do better. That’s what I think, what do you think?