How the Game Industry sets Data Science up for Failure.
Working between Games, Simulation, Retail, and Social Media groups has always been an interesting exposure to the world and how we all look at Data, Data Science, Machine Learning, and AI. Having been in this area of work for almost two decades, I have become more obsessed with the actual operational impact of workflow with decision-makers. Each industry has its own culture, practices, nomenclature, and pains. But one topic resonates, Data Science is F%$@ing hard, it's expensive, and it's set up for failure more often than people will talk about.
When speaking with different members of any game studio, I find that their views of how they are doing on the analytics front are GROSSLY different. An exec may be satisfied with some KPIs. The Marketing and User Acquisition teams tend to have shifting ideas on predictive LTVs. But the Product and Design teams are stuck with rudimentary KPIs that are rarely predictive, let alone have the capacity to break down Churn in a way that articulates the actual pains of an experience.
In games, we have the most complex, dense, and fast-moving data compared to any other industry I have been involved in. This is both an amazing opportunity and an incredible challenge. We have the opportunity to understand the very nature of people and how they interact with others and the experiences we create in games. But in truth, we don’t really know much about our players aside from how much they spend.
Why is this happening? There are a lot of reasons, but we can target the root of it around cultural divides and practices that are hard to change. As an example, setting up buckets, or cohorts by spending, means we are not comparing behaviors of users that WILL spend or WILL SPEND MORE. This also means a LOT of money is being left on the table.
The gut response to this is “we cluster”, but machine learning isn't a silver bullet, and the time it takes to do that project, the work will already become invalid when the next game update hits because changes in data will lead to machine learning models “drifting” or “degrading”, which are a couple of fancy ways of saying they lose accuracy. And “psychographics” are still just manual observations from a VERY limited biased process that have been disproven countless times in games when it comes to the actual impact of the product experience.
It's unreasonable to have Data Science come in and do ALL the things, much as it's unreasonable to make a game that has ALL the things. Other industries have entire teams of analysts and data scientists that are focused on a product's feature, or a type of merchandise. They are building tools, working with engineers to develop infrastructure, and automating a majority of their tasks to ensure they have support to get a model into production with support from ML engineers and DevOps teams. We are not even close to this sophistication in games.
Getting a model into production is a VERY different topic than generalizing a report. This is the difference between a personalized system for a game, a “one-click” task for data teams, and an email once a week that says “this thing did well this week by this percent and people like these other things also”. I have seen very few groups have the support to make these topics a reality. Meanwhile, a dashboard for monitoring tells teams that use them very little.
So, how does this change? Culture. This will always be the barrier. Take the time to understand the practical differences between AI, Machine Learning, and Data Science. While some lean into the other they are not mutually exclusive, they exist for different reasons and solve different problems. STOP bringing data late to the game, get them at the table with the designers and engineers when concepts are getting on paper.
To me, what is most important, is enabling your data teams to focus, and having reasonable expectations of them. Even if you only can support a small team that does one thing very well, understanding that one thing with enough depth is the best opportunity to demonstrate the impact of your data team for the entire studio and offer a quantifiable demonstration of why it's worth investing further into the team and their needs.
