FAQ: Updated Norms

This FAQ was published on 3/16/2024 to coincide with the implementation of the new motivation norms. The updated norms are applied to all new and existing motivation profiles collected via the Gamer Motivation Profile.

What are “motivation norms”?

When gamers take the Gamer Motivation Profile survey, we score their results based on a set of norms—average scores on the survey items based on past respondents. This allows us to assess where a gamer’s motivations rank against other gamers using this population-based estimate. In the Dashboard and the Gamer Motivation Profile results, motivations are listed as percentiles based on this ranking. Thus, a gamer may see in their profile result that they score a 70%-tile on Design—i.e., they care more about customization and expressing their individuality in games more so than 70% of the gamers in the population baseline.

How often are the motivation norms updated?

We’ve updated the norms twice since we first launched the Gamer Motivation Profile in 2015: first in mid 2016 when we had accumulated a larger sample size after the initial web app was developed, and then again in late 2018 when minor drifts from the established norms were observed.

What spurred the current norms update?

Starting in 2021, we began observing a gradual, downward trend in Strategy. By mid-2023, the continued downward trend had become substantial. To put the magnitude of the change into perspective, the average new respondent was scoring at the 36%-tile on Strategy (based on the population norm established in late 2018).

Was this due to a change in the gamers being sampled in the Gamer Motivation Profile?

We considered this possibility and examined the data in detail, but no substantial changes were observed in terms of demographic proportions or identification as casual/core/hardcore. Of course, there were some minor changes in terms of gender–~5.6% of gamers now identified as non-binary/other compared to ~1% in 2015, but any sample-based bias would likely have shifted multiple motivations at the same time (e.g., see this blog post on the motivation profile of non-binary gamers). Remarkably, Strategy was the only motivation where we observed a substantial drift.

If we noticed the downward trend in Strategy in 2021, why didn’t we update the norms earlier?

Because this downward shift in Strategy coincided with COVID, we were reluctant to adjust the norms since it was entirely unclear whether we were observing a transitory blip caused by the pandemic—e.g., a global pandemic might have made it stressful for gamers to do long-term thinking and planning. Whatever the underlying cause, adjusting (potentially) to COVID seemed unwise. But then once the pandemic started to end in mid-2022 (around the time when bigger companies started transitioning back to offices), it was unclear whether the changes in Strategy would rebound now that the pandemic was over. We debated internally and made a decision to wait through the end of 2023 to make a decision. When we reviewed the data again at the beginning of 2024, it was clear that there was no rebound and that this was the new normal.

How were the motivation norms updated?

While it was the substantial change in Strategy that drove the norms update, we also took the opportunity to take into account minor shifts in some other motivations. We used data collected from the entirety of 2022-2023, excluding gamers from China, to develop the new norms. This data sample was comprised of 525,000 gamers from the Gamer Motivation Profile survey.

Why were gamers from China excluded from the norms update?

Gamers from China present a unique analytic conundrum in terms of norms. Primarily, their gaming motivations are outliers compared to everyone else (likely due to their historically more isolated gaming ecosystem). Since adding a Chinese localization in 2019, we have begun accruing a growing sample of Chinese gamers.

To be clear, what we’ve described in all the above sections is true whether or not Chinese gamers are included in the analysis. The collected data, even with Chinese gamers excluded, shows a strong downward trend in Strategy. In other words, the change in Strategy wasn’t due to the increase in Chinese gamers.

Including Chinese gamers in the calculation of the updated norms would create distortions for everywhere else. Because our data set is still predominantly populated by gamers in North America and Western Europe, we made the decision to update the norms with the exclusion of Chinese gamers.

Why aren’t the motivation norms just updated automatically?

From a psychometric perspective, norms adjustment should be very deliberate and not left to automation as this would make invisible any important changes in the sample/population. And for example, if Strategy had dipped and rebounded from COVID, then the motivation scores would have been in flux for really no good reason. Similarly, as our sample of gamers from China increased, an automated process would have started distorting everyone else’s scores. In our case, we needed to make sure we could rule out some potential causes to the change in Strategy before deciding how to adjust the norms.

What’s our current understanding of what caused this change in Strategy?

Our current understanding is that the slow downward shift actually started before COVID. The trend is detectable in 2019 although not yet robust. This implies that COVID wasn’t the initial causal factor, but could certainly have exacerbated it. Beyond that, we have some hunches, such as the broader effect of social media on our collective attention spans, but these are difficult to test/connect empirically.

How do the updated norms impact the audience profiles from the Dashboard?

In our analysis of profiles from before and after the norms update, most games will see an upward shift in the Strategy motivation by about 8-10 percentile points. Slightly more gamers will fall into the “Architect” gamer type segment. This update will have no impact on the demographics and the related games metrics. To recap everything, gamers have become much less interested in thinking and planning, and this is now the new norm.