1. 31

  2. 10

    User research is hard. I have my own story from working on the XBox One team and how the Kinect was made the huge bet it was, before the launch. In this article, the author writes

    The participants’ mistake was one of “affective forecasting,” guessing how they’d feel in a hypothetical situation.

    Well, on XBox One, this was the opposite: we took the user research results on how people controlled apps via voice and gestures seriously, shipping what worked well as per the user research and the employee betas.

    Working on the Skype for XBox One team, we ran many user research tests to see how people would use the app via controller, voice controls (“XBox, call Joe”) or gestures (using your hand to scroll and call someone, Minotiry report style). The results for voice were promising, for gesture, they were neutral. So we shipped it and after similar results all-round, XBox leadership decided voice control will be mandatory for all XBox apps, with gesture strongly suggested (for first-party apps, it was also mandatory). And no XBox will be sold without a Kinect.

    It later turned out that voice worked well… initially. But for all users but the most power users, usage steadily dropped after the first few months, until people stopped using it completely. This did not come out on the user tests, as they were designed to be short. And it did also not come out on the company takehome beta - about 3,000 people using pre-release XBoxes in their homes, with strict confidentiality in place - as people were early adopters, continously showing off voice and gesture control.

    By the time the data was clear, it was too late: Kinect was bundled with XBox. And it took years to de-bundle it. As both this article and my story shows, user research is not straightfoward.

    1. 4

      Lesson 5: People paying for researches don’t necessarily want to find the right answers, they might just want to support theirs.

      1. 3

        Interesting insight into a part of games development I didn’t know existed.

        Anecdata: in the run-up to auto-focus for still cameras in the 1980s, Nikon supposedly relied on feedback from experienced pros who dissed the current state of primitive AF as much inferior to manual focus, and didn’t push forward developing it.

        Canon forged ahead, broke mount compatibility and introduced faster and more reliable AF, and captured the pro market by delivering a better system.

        1. 2

          I remember seeing a bunch of articles about ux experiments in Halo on gamasutra five or ten ish years ago. They were talking about doing things like making heat maps showing where players spent time in each level during play testing. This would let them find all the places where players were getting stuck in the levels so they could improve the signposting at those points.

          1. 1

            I think Halo 2 marked the transition from LAN-centric gaming towards more centralized but wider networks (Xbox Live exemplifies this). Game designers simply didn’t have the tools to see what the vast majority of gamers did. The article hints at that - the old lobby system could not scale to the hundred of thousands to millions of players expected for Halo 2.

            1. 1

              Did Counter-Strike 1 not hit the same kinds of numbers? That had a system more like a lobby.

        2. 1

          It’s funny but I had pretty much the reaction the researchers originally anticipated. It wasn’t with Halo 2 but I remember seeing my first games without lobbies and wondering what the heck happened. How come I couldn’t filter through a list of different server offerings and choose one that matched my style/mood.

          1. 4

            Think about it from the perspective of someone logging in to an online shooter along with millions of other people for the first time after a midnight launch. Do they have any way to figure out which lobby or server will lead to a game they enjoy? I’ve been gaming online for decades and picking “Big Team Battle” or “Solo Deathmatch” is honestly a more enjoyable choice than reading and deciphering the lobby names Xbox Live users would come up with.

            1. 1

              That said, I ended up with way more matches that I didn’t want to play vs BF2 where I could play exact style I wanted based on the server list. Plus, some games have a server list and a feature to drop you in a game. A game can have both to some degree.

              1. 1

                My FPS evolution roughly went Quake -> TF -> TFC -> TF2 (for a brief time). I stumbled into a group called Team Play First from their server’s description. They actually had some level of moderating on their servers — if you were playing a capture the flag map you’d be kicked if you weren’t playing as a team. This actually made the game really fun for me, and as new games came out I’d play on their servers.

                Are there similar “clans/guilds” on modern FPSes? I’m not sure I’d have ever met this group of people and had such an enjoyable time with the modern lobby system.

                1. 2

                  While I can’t argue against the magic of finding weird little communities (fond memories of “tfc.cfl.rr.com” and “Houston Vehicles”), this is literally something from the article:

                  What we as researchers should have done is be more discriminating in how we presented the results. Our data was true when looked at from a specific angle: “Here’s what some players will say when they hear about the system for the first time.” From there, we could have worked with the team to test different ways of presenting the system to improve that first impression and increase the speed at which players realized the true value of the system.

                  In a game with thousands of active players, and relatively low limits on game sizes (Halo 2 “Big Team Battle” was 8v8, Splatoon 2 is 4v4), lobby selection doesn’t seem like a productive way for players to get to an exciting game, and persistent community-owned servers are really bad for cheat prevention.

                  With matchmaking, you never have to make a choice if your server’s full or empty, or if it’s been taken over by a bunch of clanners using voice chat. After a few placing matches, most of the games with matchmaking I’ve played (Halo 2 & 3, Splatoon 1 & 2) do a pretty good job putting you in exciting games.

                  I don’t disagree that modern shooters do a terrible job at sub-community building, but I don’t think that’s what the massive player base is really there for.

              2. 1

                Same! But then playing a couple of quick matches on a couch with a friend is so much smoother without them.