©mitinteractivetechdesign

Our frequent partner in crime Natalie Golub, user researcher extraordinaire, wrote a terrific post here about a year ago on What Homeland Can Teach Us About User Research. Natalie outlined user testing from the point of view of the testing professional that she is, expounding on how to get good intel as a tester and how to present it to the product owners in the most useful way possible.

Recently, I’ve found myself in the midst of user testing from a different vantage point, out in schools testing a preliminary prototype of the new game I’ve been working on. While all of Natalie’s excellent points have been on my mind, I’ve also had to face the different circumstance of testing something that I myself am deeply involved with creatively. My objectivity is compromised to say the least, but it doesn’t make testing and getting good results any less important.

This presents specific challenges, as I endeavor to separate my emotional stake in the game that I’ve been working on for the past several months from the critical need to get objective, actionable information about the things I can do with my team to make the game better. It also introduces some logistical hurdles, as we’re not operating in a formal testing environment, but going into schools with all of the logistical uncertainties that entails (Hello, fire drills!). This has me thinking a lot about how to plan and execute informal testing in a way that will get the best results. Here’s what’s on my mind.

Get Over the Challenges and Test

In the middle of a hectic production process — and they’re nearly always hectic — the temptation to postpone or even skip user testing altogether can be enormous. We make the point frequently on this blog that it’s critically important to test early and test often, but even I found myself thinking, “Gosh, I wish we could just skip this testing.” The deadline was tight, we were planning on testing in multiple cities, and the idea of flying my very small production staff hither and yon for user testing was overwhelming. Beyond the practicalities, I should admit, I was scared. Testing is scary. On the one hand, it’s really exciting to see your product in front of exactly the kinds of users who will be playing it someday. On the other hand, what if they don’t like it? The idea of skipping testing all together, sticking one’s fingers in one’s ears, humming, “Mary Had a Little Lamb,” and hoping for the best starts to seem really appealing.

Thankfully, I remembered how critical user testing has been to my past development projects, and soldiered on. I altered the scope of our original plans, sticking to just a couple of schools in the NYC area to avoid travel logistics taking over, and hiring an outside producer to make sure the trains were running on time. This allowed me to focus on getting the game ready for testing and thinking about the kinds of questions I wanted the testing process to answer for us, rather than spending hours calling school principals or setting up online surveys. And with no logistical hurdles to point at, I had to admit that my nervousness was just about getting feedback, which is exactly why I needed to do it! So my advice on this point is, if you have obstacles to getting testing done, figure out what they are and how to get around them. Skipping testing really isn’t an option.

Plan Ahead

©andybardill

Time is at a premium when you’re testing, especially in an environment that you don’t control, and coming in with a clear plan and set of objectives is key to getting the most out of your experience. Because we were testing in schools, we went in ahead of testing to individually meet with each teacher and principal involved, ran the software on school computers to be sure that it wouldn’t crash or encounter strange firewalls, said hello to the kids so we weren’t total strangers on the first testing day, and did tons of other little things to make sure the details we all in place. Being rigorous about your testing plan and logistics will help you use the precious time that you have with testers doing what you’re there to do, rather than tying up messes.

Consider the Gatekeepers

Whether you’re testing with little kids or middle schoolers, you need to have a point of view about how you want to engage the gatekeepers who are giving you access to them. Teachers, parents, administrators, daycare providers, and even professional testers will all have their own points of view about how you should interact with your subjects, and it’s often a delicate line to walk between making sure that these gatekeepers are comfortable and achieving your own objectives.

Being as upfront as possible about the needs and expectations of the testing project can go a long way. If you’re going to need help observing the kids or taking notes, you need to communicate clearly to the people who’ll be helping you – and these are often those same gatekeepers – what kinds of questions you want answered, and what kinds of questions you want to avoid. Parents and teachers will often want to jump in and help kids figure out what to do, which can subvert the point if you’re trying to test if kids know what to do, so politely explain to them before testing begins that you would prefer they just observe. (If need be, you can always show them to another room!) And when all is said and done, be appreciative and express thanks for the work they’ve done to help you out.

Be Prepared to be Elated… and Depressed

©psd

It is the best of times, it is the worst of times. Testing is frequently a roller coaster of “Awesome! That works just the way we thought it would!” and “Omigosh! They hate that feature that I love!” It can be hard to keep perspective when something you are passionate about isn’t hitting home the way you thought it would, but often those setbacks will be offset by enthusiasm about something you hadn’t thought was a big deal.

The important thing is to LISTEN. Listen hard, and listen especially hard to the things you don’t want to hear. There’s no point in user testing if you’re doing it only to confirm your own ideas about the game you’re making. Frequently, if you keep your eyes and ears open, you’ll find not only unexpected things to fix, but also unexpected successful elements that you can build on and make a larger part of the experience.

Don’t Make Changes Based on the First Two Testers

In every user testing experience I’ve ever had, including television testing going back over a decade, there always comes a moment after the second kid has watched/played when someone says, “Well, everyone hates that character, let’s cut it,” or “Clearly, the best moment in the game is X, let’s make more parts of the game like that.” One of the largest dangers of testing is over-responding to the first five voices you hear at the expense of the many voices you hear throughout the rest of the testing process. It’s human nature, but it’s worth arming yourself against.

There are many philosophies of when and how to test, far too many to go into in the post, but the kind of testing you’re doing may influence when the best point is to start responding to the feedback you’re getting. If you’re testing continually throughout the development process, this might mean waiting for the full results of a day of testing before deciding what to test next. If you’re testing more intermittently, it might mean waiting through a cycle of testing and building a period to review and respond to that information into your development cycle. Whatever your production plan, just beware overreacting to any very small subset of feedback, and make sure you have a reasonable plan for how and when to respond to the information you’re gathering.

Help Them Give You Feedback

If you’re doing fast and loose testing with a larger group, as we have been, it can be hard to gather feedback from the players at the end of a session. Even in focus groups, one or two voices can dominate and shyer kids may not ever tell you what they think. We approached this in our recent testing with a combination of in-person observation by members of our team, private surveys for each child and teacher to fill out after a play session (SurveyMonkey is great for these), large class-wide focus groups, and smaller breakout sessions with just a handful of kids in each. Providing a variety of methods for the users to give feedback helps to ensure that everyone has a way to communicate their thoughts that is appropriate for them. Once again, finish up with a big round of “Thank yous.”

So, there you have it, I’ve gone into the user testing trenches and lived to tell the tale. It’s always an amazing experience and I look forward to being equally scared, interested, elated, and depressed over the course of many user tests to come. In the meantime, you’re welcome as always to reach us at KidsGotGame@NoCrusts.com, follow us @NoCrusts on Twitter, or sign-up to receive email updates.