Listen to users when you’re identifying problems. Do not listen to them when identifying solutions.
In the 14 years since this was written we’ve seen what happens when you both stop listening to your users and also stop using your intuition for what a good product would be and start relying solely on these telemetry-oriented data-driven methods where PMs’ bonuses are driven by how big they make their personal data number. Look at what’s happened to the modern web since this was written: it’s a KPI gradient descent into full-page email-harvesting coverups, websites that don’t work unless you download their app and log in, algorithm-driven feeds that prioritise viral rage-porn content over all else, feature removal in the name of UI minimalism, regular needless rearranging of whatever UI that’s being run by a PM gunning for a promotion right now. Maximising the number that goes onto the first slide of your board meeting presentation over all else ruins user experiences. Plus now it’s invisible to the people with the power to change it and it’s DATA-DRIVEN so it’s unassailable by common sense.
I’m not saying telemetry isn’t valuable, their example of the Medic class being underpicked points to a real game design problem and solving it is important to the users. Surely if you asked the users what they wanted, since they’re all playing Scout they’ve have asked for Scout buffs. But users not using feature X isn’t a problem to anybody except for the people whose job depends on feature X doing well. If that’s not a profit centre then maybe that feature just sucks and it doesn’t help anybody to make the button bigger and prompt me to use it in a non-dismissable tutorial. Instead we tell people that they get paid more if people use their feature more, so the ruin the product in service of it.
An implication is that one cannot use that technique without implementing telemetry, other than in a few cases where the data is collected “naturally” (like in online games with centralized, proprietary servers).