Reaping the rewards of user testing

Montreal, Canada-based app publisher Budge Studios talks about the progress of its new worldwide user-testing program and gets into what's working and why.
May 28, 2015

In a new venture that’s taking user testing to the next level, Montreal, Canada-based app publisher Budge Studios is looking to enlist kids worldwide to participate in its app creation process.

The five-year-old company, which counts 108 million downloads to its name, unveiled its groundbreaking program Budge Playgroup earlier this week.

The Playgroup lets kids and parents test and critique the studio’s iOS apps before they hit the App Store. Their direct feedback is then incorporated into the next iterations of the product.

According to Budge Studios co-CEO and founding partner Michael Elman, the idea for the app-testing program had been in the works for some time, but the company was waiting for the right technology to come to market.

“We’ve been looking over the last few years at how we could do this, what tools we’d need, what’s the process and the legal ramifications,” says Elman. “There were some solutions that we saw for getting our work in progress to people, but they were all really complicated. Then Apple announced its TestFlight service and we were able to jump on this tool.”

Budge also wanted the ability to test its apps with a more global audience.

“If we’re only testing our apps with a very local audience, we’re not really getting the perspective of our actual audience,” Elman contends. “We may have the right age demo in our office when we do playgroups, but what we’re not getting is feedback on the localization into different languages, and also cultural differences from country-to-country. You may get feedback on apps and certain brands that we wouldn’t get if we were just playing it with kids here in Canada.”

The app-testing program is open to anyone with kids ages two to 10. Beta tests are sent out roughly once a month, depending on where the studio is in the development process. The group gets the app via TestFlight for about a week, during which time they fill out questionnaires and then return them to Budge.

“We think this is really innovative,” says Elman. “Traditional video game testing is often kids and their parents in this small, enclosed room with an interviewer. It’s a bit intimidating—they’re often being videotaped or watched through a window, and they’re also only seeing the game for that first time.”

Whereas with Playgroup, tests are carried out at home or wherever kids would naturally play, which Elman notes is key for more quality feedback.

“With Playgroup, you have the days at home. You can pick it up when you’re in the mood to play and you can come back a few days later. That’s much more natural to how you would actually be playing this app and you get better feedback.”

In terms of survey questions, queries are along the lines of who’s your favorite character, how much fun did you have playing the game, which parts of the game did you like and how challenging was the game.

From parents, the survey looks to determine whether or not there were unclear sections or if kids became disengaged at any point.

“Depending on where we are in the development process, we have the time to take all that feedback, read through it and then adjust the difficulty or usability,” explains Elman. “Or, if one character showed to be more popular than others, we’ll make sure that they’re a bigger part of the experience.”

The first Budge app to go through Budge Playgroup was Caillou Search & Count. Users got the first build on March 21, and sent in their responses by March 30. After a Playgroup LIVE test (an in-person extension of the program), the final version launched on the App Store on May 13.

“[For Caillou], they came up with great ideas,” says Elman, noting users were not shy about sharing what they liked and did not. For instance, the app uses a magnifying tool to locate objects. Kids reported that the initial placement of the magnifying glass got in the way of gameplay. “We looked at that again and changed the way the in which the magnifying glass appears on the screen and where it’s located.”

In addition, the program is also useful for testing out new characters. Budge is set to launch a new app next week, Miss Hollywood Showtime. The property, which follows Chihuahua Miss Hollywood and her fellow animal strays, is one of the studio’s original IPs. Budge was able to use the app-testing program to see which characters resonated with kids.

Going forward, the plan is to use Playgroup for all of Budge’s upcoming apps.

“There’s no reason for us not to do this on every single app that we do,” says Elman.  “As the Playgroup grows, we can have more choice as to who we send different apps. If it’s targeted more to preschoolers, for example, we can make sure we’re getting enough data from the right demographic, or from specific fans of the brand.”

So far, Budge has had no problem in getting volunteers for the program, both inside and outside of North America. Its users are currently split 50% in the US and 50% in 21 other countries, including Australia, Brazil, Czech Republic, the UK, Hong Kong, Hungary, Italy, Mexico, the Netherlands, New Zealand, Singapore and South Africa.

“Kids love to create. A lot of our apps are about creating things, whether it’s making a food item or dressing up characters, and now kids get to do that as game developers,” says Elman.  “They’re waiting for the app to come out to see if their feedback is involved and also to play the final version. For us, it gives us a chance to get feedback not only on the game but on new characters that we’re building and develop a little bit of buzz around them too.”

About The Author


Brand Menu