Editorial: Everything you know about boys, video games, and surveys might be wrong

This article originally appeared on GamePolitics on July 24, 2015.

Everything you know about boys and video games is wrong,” is the bold title of a Time article written by author, teacher, and parenting educator Rosalind Wiseman. The claim is the result of a survey conducted by Wiseman in a collaborative effort with colleague Charlie Kuhn and YouTube gaming celebrity Ashly Burch to determine attitudes towards sexism in video games.

Given such a controversial topic it is only natural that the survey has become controversial as well, and gamers have indeed taken to Twitter and Reddit to voice their criticisms. With my background in academic and market research I thought it important to speak with a survey design expert to get to the heart of the matter.

“This all matters because gaming has become an important part of our culture, and it’s sending the wrong message onto our boys’ and girls’ screens. Our kids deserve better. And it’s what they want.” — Rosalind Wiseman, Time, July 8

Wiseman conducted the survey through SurveyMonkey, a popular polling website used by both academic and market research professionals. Along with Ashly Burch, Wiseman originally presented the data at the Game Developers Conference in early March of 2015. You can view a PDF of the slides with the survey data here. Unfortunately, that is nearly all the information we have about the survey data and its methodology. Wiseman, Burch, and Kuhn have ignored repeated requests for exact wording on the questionnaire as well as further details about the methodology they used.

Gamers looking to discredit the survey looked back through Wiseman’s Twitter feed and discovered her promoting the survey through Twitter and Facebook (archive of tweet here) in Nov. of last year.

This was evidence of biased sampling, they argued. What’s more, how could we tell if the people taking the survey were who they said they were when there was apparently no verification process?

Wiseman responded to this criticism on her webpage where she stated, “We have never claimed that this is a rigorous academic survey, nor that it should be treated as such.” She went on to say, “this survey was exploratory or a ‘convenience’ sample that was meant to generate conversations and encourage others in the field to continue this research in more thorough ways.”

I reached out to the American Association for Public Opinion Research (AAPOR), “a professional organization dedicated to advancing the science and practice of survey and opinion research to give people a voice in the decisions that affect their daily lives.” Among their goals is “promoting best practices in collecting, analyzing, and interpreting survey data,” according to their website. They put me in touch with Professor Allyson Holbrook.

An Associate Professor at the University of Illinois at Chicago, Holbrook earned a PhD in Social Psychology from the Ohio State University and has been teaching graduate-level survey design and data collection methods for the past 13 years. She also serves as AAPOR’s associate secretary and treasurer.

Prior to my phone interview, Professor Holbrook read the Time article and the data slides from the GDC presentation. She immediately mentioned a significant problem with the representativeness of the sample, apart from the problems that gamers had mentioned. The data slides report that a whopping 39 percent of respondents were from private schools. According to Department of Education statistics, the total percentage of children in private schools is around 10 percent, a number that has been declining for years. What’s more, the majority of private schools are religious, so opinions about sexism in video games may vary from public school children depending on religious values. But we don’t know if the survey polled religious private schools.

“The goal of a survey is to generalize to a representative population,” Professor Holbrook explained. “We don’t know who participated in [Wiseman’s] survey.”

Because Wiseman already admitted to problems with the sampling, I asked Professor Holbrook about possible bias within the survey. Of particular interest to me was the question that seemed to garner the most attention, “Are females too often treated as sex objects?” That is how the question is listed on the data slide, but it is important to note that we do not know the exact wording because the survey has not been released. The possible answers fell on an agree scale, from Strongly Agree to Agree, Neutral, Disagree, and Strongly Disagree.

“There are at least two problems with this question,” said Professor Holbrook. “One is it seems it has pretty strong social desirability implications. These are boys between [ages] 11 to 18. Kids in that age know that they’re not supposed to stereotype.” The kids could have answered in a way that they felt was expected of them, she surmised. “Just as there is evidence that surveys show decreasing racial prejudice, because people know how to respond,” she said. In other words, if you want to find out how many people are racists, you are not going to come to a scientifically accurate answer by asking them directly if they are racist. Most of us have a natural tendency to want to portray ourselves in the best light possible.

“The other problem is that this is framed as an agree/disagree question.” She explained a problem in survey research called acquiescence response bias. “More people are likely to say agree than disagree,” she said. “Depending on how you word the statement, the majority opinion changes. The current thinking is that agree/disagree questions don’t really ask about the key underlining dimensions,” of an issue. “The underlining dimensions here are how often women are treated as sex objects. The agree options here are framed in the most socially desirable way.”

What about Wiseman’s defense that this was an exploratory survey? I asked Professor Holbrook if the survey was still useful as exploratory research given these problems.

“To say there’s anything too much that you can draw from what she did, I’m not sure you could say that.” She said that the lack of transparency in methodology, which is one of AAPOR’s key values, severely limits the survey’s usefulness. “She should release more specifics about the methodology. You need to tell people how you did it so it can be evaluated.”

Professor Holbrook pointed out that these criticisms do not mean the results of the survey are not true. There may very well be a majority of boys who believe that females are too often treated as sex objects in video games. Wiseman’s survey does not appear to be helpful in answering that question, however.

What about Kate Upton?

“Kids are fed up with Kate Upton,” Wiseman writes in the first sentence of her Time article. She claims that many of her students were annoyed at the Kate Upton Game of War advertisements. They cited the lack of ability to close the ad, “but what really irritated them was Ms. Upton, in a full-cleavage-baring white flowing dress.” Again, her account may very well be true, and her students may indeed be annoyed by a sexualized swimsuit model in game advertising, but there is no mention of Kate Upton in any of her survey data.

Media coverage of the survey has been significant. “The survey does a great job at dispelling the myth that all gamers are just horny boys,” concludes one of the online publications covering Wiseman’s piece in Time. Despite what I found to be significant problems with sampling, question design, and transparency, at least 16 different websites reported the results, including Engadget, The Guardian, Wired (UK) The Mary Sue, and The Toronto Star. While The Guardian did question Wiseman’s assessment of the Kate Upton ads given their effectiveness, the only article I could find questioning the survey methodology was from Inquisitr.

I made multiple attempts to contact the writers who covered Wiseman’s article to see why they did not question the results or the survey, but, as you might expect, I have not heard back from any of them.

So what are we to take from this survey and its coverage by the media? I believe a healthy dose of skepticism is usually a good thing. On AAPOR’s website there is a list of questions for journalists to ask when writing about polls, from “who paid for the poll and why was it done?” to basic methodology questions. These are good questions to keep in mind for anyone reading about a survey.

Note: I retained publishing rights to all my articles published on GamePolitics.com. This article did go through the full GamePolitics editorial process. Thank you to James Fudge and Andrew Eisen for their work on that late, great website and their help with my foray into national publishing!