The Church calendar is being updated with a new and improved back-end architecture and a simplified work flow, easing the scheduling of resources and events and introducing a more intuitive design. The latest version, Calendar 2.0, is due to come out in a couple of weeks. As part of the development of this product, the calendar team engaged the community using a new model for beta testing. It was a great success.
How It Worked
In July 2011, the calendar team asked for testing volunteers through the LDSTech Forum. Next, an article published on the LDSTech blog (Upcoming Calendar 2.0 Beta Test) advertised the opportunity for community members to conduct application testing. The article called for at least 30 participants, preferably from various geographical areas and having callings as stake or ward administrators, clerks, and building schedulers. These simple recruiting tactics encouraged over 200 community members to respond, and about 30 participated actively.
Once the testing period opened up, the calendar team created a private forum on the LDS Tech website to collect feedback from volunteer beta testers. Volunteers identified bugs, quirks, and other issues, and evaluated how the new version of the Church calendar met (or didn’t meet) their expectations. They identified dozens of high-priority items and sometimes reported 15 to 20 issues in a day, according to Jeff Brown, the engineer who managed the volunteer feedback. Brown also said that nearly all the feedback was useful, and there was very little “noise” that had to be filtered out of the responses.
Most testers would report bugs to the forum in the evenings, so each morning the team had a lot of posts to work with. They were careful to quickly respond to each posting so none would go “stale” without acknowledgement. Indeed, the team found it was not difficult to solicit quality feedback, but the real challenge was keeping up with the daily postings—a good kind of problem to have, engineer Kyle Quist said.
The team is very pleased with the outcome of this community beta-testing model and plans to continue using it for future products. Indeed, the only limitation of this model, as Brown sees it, was somewhat circumstantial—an individual’s level of participation could be limited by his or her role in the community.
For example, some were enthusiastic to help, but did not have callings with the administrative rights that allowed them to complete all the necessary tasks. These individuals had limited opportunity to participate because they had a limited amount of access to the calendar. To overcome this problem, the team encouraged participants to work together. Those with the willingness and know-how could help their friends who had the appropriate callings.
Ammen Harper, solutions manager for the project, noted that this community testing model allowed for a “broad distribution of testing in different locations around the world” with a mix of different testers. The model ensured the ability to test scheduling features across time zones, an important test element for this product. Quist added that using this model sped up the process of getting the calendar ready for release, and he appreciated the value of receiving bug reports specific to the users’ concerns.
The volunteers were an excellent resource. Quist explained that the testers were thorough, focused, and invested in the success of the product. Additionally, Harper said these members have technical knowledge that makes their contributions even more valuable. He suggested other teams take advantage of community forums for testing as well.
The success of this model demonstrates a new testing resource with great potential. The community is ready to be involved. “There is a hunger to participate in these changes,” Quist said. Volunteers jumped at the opportunity to help and proved invaluable to the project. This testing model is an exciting development. It shows that engaging the community is an attainable goal.
If you would like to engage community volunteers to test your applications, contact Kevin Fitzpatrick,
, with the LDSTech team. Kevin's group is actually working on a mobile application that will help facilitate and organize community testing efforts. Details about this application will be published as development progresses.