In a recent interview with Community Business Manager Alan Smoot, Jeff Brown, a quality assurance lead for the Church, talked about his experience using a group of volunteers to beta test version 2.0 of the LDS.org calendar application released back in fall 2011. You can watch the interview as well as read a summary below.
“We wanted to get the community involved,” Jeff said, “to give us some better assurance.” Jeff and his team realized that they could not test for every scenario and in every environment. “As QA testers, we don’t understand all the scenarios; we don’t understand what people are going to do . . . and that leads to a lot of uncertainty when it comes to release time.”
Cue the volunteers. In came LDSTech forum-goers from the United States, Australia, Sweden, the United Kingdom, Brazil, Argentina, and more. The forum, Jeff said, has been there for a while—as have its loyal participants. “What we found is that these people are just excited to contribute. They’ve been using our software for years in their wards and stakes and they’ve seen all the bugs we’ve had out there before. They’re excited to . . . have an opportunity to make this better.” These users recognize the importance of LDS applications; they know they will inevitably be used in wards and stakes worldwide to further the work of the Lord.
Mobile devices and foreign technology restrictions offer particularly weighty challenges to those testing new programs and applications. Every scenario cannot be predicted. Members are no longer simply using applications on their personal computers; rather they often turn to the latest or most available mobile platform. Android, Palm, Apple, Windows, Blackberry. Smartphone, tablet, music player, PDA, e-reader. Wi-Fi, dial-up, broadband, 4G.
With so many possible combinations—and we’re not even getting into non-standard operating systems or jail broken devices—it’s next to impossible for quality assurers to test for every circumstance. This is where volunteer, “real” users make the most difference. They can use the application in whatever environment they have.
“It turns out that real users of the calendar are better testers than we are,” Jeff said. In the case of the calendar project, many high-priority bugs were discovered only because of volunteers.
Volunteers didn’t simply test, either. When they found a problem, they took time to troubleshoot and communicate with other users to verify whether or not the problem was universal or exclusive. “It was great to see how they figured things out,” Jeff commented. “[The users] did a lot of the legwork to validate bugs and see where the real problems were. We were able to take their findings and . . . make some really great improvements.”
The volunteer testers turned out to be the difference that identified countless critical issues that could have lingered in the application for months, not to mention the resources saved. Jeff estimates around 400 hours were saved because of user contributions. Because of the time saved and the impact the community made on the project, Jeff’s team delivered a much more thoroughly tested application.
It’s clear that we’ve come to a point where community help is necessary if we desire to hasten the work. Particularly when software applications are used on a plethora of devices and platforms, active community involvement in testing can help identify bugs and other issues much more rapidly than internal resources.