This week we have a guest post on user testing with kids by Barbara Chamberlin, PhD.

Barbara Chamberlin directs game development and research at New Mexico State University’s Learning Games Lab. The development team at NMSU includes professionals in animation, game design, app development, video production and other media. Most recently, they are completing research showing the impact of their “Math Snacks” suite of animations and games on students (mathsnacks.com). More information about the games lab products and research is at learninggameslab.org.

Summary: User testing with kids can be a challenge. Barbara Chamberlin of NMSU’s Learning Games Lab shares specific strategies they’ve used to make testing easy, enjoyable and effective.

As a developer at a small studio, in my early years, I always (1) valued what I thought I could learn from user testing; (2) felt like user testing was something I should do; and (3) hated having to do any user testing.

This sentiment may feel familiar. I think many developers agree that there are lots of poor ways to do user testing. We can make errors in how we set up sessions, in the methods we use to find out how our users feel about our product, and in how we document how users feel about our product. At its worst, user testing can be a huge waste of time and can be used to inaccurately confirm what the developers already think about their own product.

Allow me to approach this topic with the zeal of the converted: I now love user testing. At the Learning Games Lab at New Mexico State University, we’ve spent the past ten years refining methods for testing our products as we develop them. It turns out that once you’ve got robust methods in place, testing can feel routine and indispensable. We test a lot, and we learn something every time. In talking with other developers who have come to value it as well, I’ve noticed some important trends.

1. Make access to testers easy, frequent and ongoing

With a busy development team, the largest barrier to user testing can be getting access to kids. We’ve found great success in setting up a standing summer camp program. Youth “Game Lab Consultants” participate in one or two-week programs in which they learn basics of game design, practice giving input on something every day (even if not one of our games), and usually learn a specific skill (such as coding, board game design, animation, or digital photography). The kids benefit from a creative, media-rich experience and get practice providing feedback daily (whether writing about a game they’ve played, answering a question in our Video Closet, or engaging in group discussion). “Expert Consultants” – kids who have completed a summer session – are invited back throughout the school year for different after-school or evening activities.

This ease of access certainly facilitates more frequent testing: at any point in development, we are able to “take it to the kids”. Some of our Math Snacks games went through about 40 user testing sessions — from quick 15 minute checks to longer gameplay sessions. We’ve found this type of access also improves the quality of the kids’ feedback.  Because they are engaged in constant review, the youth consultants build their skills in articulating what they like and don’t, and in communicating things that are unclear. The consultants feel comfortable in the space and are used to the presence of reviewers, which mitigates stress from being observed.

Developers benefit too: we improve our design intuition by having such constant access to the target audience; we are able to test, make a small change, and immediately test again; and we are able to refine our own processes because we can test so frequently.

The cost of this program is equal to the cost of providing a camp or other program. We do not provide stipends to the testers, but we cover the cost of the teacher running the program and the second adult who is always in the room whenever we have kids. As university researchers, our Institutional Review Board reviews all of our research standards, which includes having each consultant and a parent or guardian complete a consent form at the beginning of each session which outlines risks and benefits.

I know other developers have had similar success in creating ongoing access. They may have a standing play day where parents and kids are invited to play games together, an after-school program with students who return every week, or a summer camp for teachers and kids. These types of programs work because parents and kids enjoy being included in design. We make sure to communicate the important role testers play in helping us create quality products.  For the very small developer, however, a more feasible strategy may be taking an active role in an existing program, such as volunteering with a classroom once a month or establishing a group of parent peers in a club or community center. You may not need to test your product all that often, but by making yourself regularly accessible to parents and teachers as a volunteer, guest speaker or activity leader, you establish a relationship and build trust. You’ll find how much more frequently you test your product, when you can easily weave it into sessions already scheduled. If you already find yourself surrounded by kids at swim lessons, scout clubs, school play rehearsals or piano recitals, get to know parents and let them know what you do. Often, they are glad to let their kids work with you to review products.

While creating a recurring activity for your testers costs more in time and other resources than setting up a few sessions, we’ve found it provides an important service to our community, while significantly changing the way we approach user testing.

2. Involve developers in the testing

When developers are actively involved in user testing, they better understand why users act like they do, rather than having to address a list of action items.

Accurately observing and understanding users is tricky: accurately communicating that to a design team is even harder. We require our developers to serve as testers: they observe, ask, document, and discuss.

Ideally, we have at least three testers in every test session. The liaison works with the kids, sets up the day’s activities, and organizes the room. She also observes the testers, noting the methods they use. Usually two to four observers then interact with the testers, observe a play session, or engage in discussion or interviews. Immediately following any test, the liaison and the testers talk together, sharing notes and agreeing on important takeaways. The liaison writes those down, sharing them with the entire team. We also document which kids were involved, their ages, and what methods we used. While we don’t often share all the details from our user testing with clients, we know that they often appreciate seeing how we learned what we did.

It’s always difficult to hear that our products don’t work. When we see and hear testers using our products, we are better able to understand why a problem occurs. Not every developer in our shop is involved in every session, but each developer participates in at least one.

3. Never ask the question you want the answer to

The question we always want to ask is, “Do you like it?” We have lots of variations – What do you like about it? Is it too hard? Are the characters appealing? – but these are really versions of that most common question. The problem with user testing is that that question is really the hardest to answer. It assumes that the user is able to understand if she really likes it, free of her own biases, and doesn’t feel pressured to give you the right answer. Regardless of the question we want answered, realize that there are a lot of different pressures on a user tester keeping him or her from being able to answer it.

We work around this conundrum with a few different strategies. First, define the question you want answered, then establish several different ways to find the answer. Realizing that we never want to ask yes or no questions, our method yields something like this:

In offering the user three different ways to think about our specific concern, we give them several opportunities to process how they feel about the game. Additionally, this helps them keep from trying to answer it in the way they think we expect them to.

The second major strategy we deploy is using multiple methods in every test session. We have several strategies at our disposal: observation, one-on-one interview, one-on-two interview, focus group, digital recording, video closet, journals, creative activity. We have found that when we give users multiple ways to express themselves, they are better able to process their own thinking and avoid groupthink. It also gives us the ability to confirm what we think we hear them saying.

In the “Video Closet” at NMSU’s Learning Games Lab, youth consultants give feedback on games they’ve played… avoiding the ‘group think’ that often happens in focus groups, and giving kids more of a chance to reflect

Imagine that we are testing art styles for a game for kids ages 8–10. We may start with three distinct styles, and we want to know which the kids prefer and (especially) if any of the three are definitely not going to work. To approach this, we’d use three different strategies over a 45-minute session.

–          Small group observation (10 minutes): We’d put 12 kids into groups of 4 each, and give each the three graphics. We’d ask them to just talk with each other about the images (for example, “how would these fit into a game?”
“What should their names should be?” or “How they could be changed?”) and we would observe the discussion, without asking direct questions.  The observation notes would be helpful for hearing rich descriptions, but this activity particularly helps the kids think about how they feel about each character.

–          Video closet: (3 minutes, each kid): We’d post a photo of each character in our small video closet and ask each tester to describe their least favorite and why they dislike it. The video closet is a powerful antidote to groupthink, and users can really be honest and descriptive when talking to a camera.

–          Creative design activity (while each tester is waiting to use the closet): the group would be given blank paper and crayons and encouraged to draw a comic panel with their favorite character. They would be encouraged to give each a name and dress them in different ways, or change them if they’d like. While they draw, we’d interact asking questions: “I see you put a hat on this one… why?” or “What made you decide to draw this one?”

When I finally learned to embrace user testing, I saw how valuable it could be. I can now take a game to final testing fully confident that it works, appeals to the target audience, and is easy to use. Our summative testing rarely provides any big surprises, because we’ve been able to test so frequently along the way. Now, the testing days I used to dread have become my favorite development days.

Additional resources:

–          I’ve placed several short movies online about user testing.

Successful Strategies for User Testing
40-minute presentation shared at Colombia3.0
https://www.youtube.com/watch?feature=player_embedded&v=bK_RHsuWP50

 Trying Very Hard to Make Games that Don’t Stink:

21-minute presentation with specifics on how the Learning Games Lab works
https://www.youtube.com/watch?feature=player_embedded&v=bK_RHsuWP50

What I’ve Learned About Testing as Design
12-minute video on how we design our test sessions
https://www.youtube.com/watch?feature=player_embedded&v=ZwiPXfhvF8Y