When I was a kid, I would use a hammer to set my tent stakes or to pound nails into two-by-fours to build tree forts. I thought I could use the hammer for anything. If I hit a board hard enough, it would break, and I wouldn’t need to use the saw. If I hit the plywood just right, I could punch a hole through the board without using the drill.
Obviously, using one tool to do it all may get quick results, but it won’t look pretty. Many in the testing industry use a similar single-tool approach, attempting to solve every problem using automated testing. No time? Slap some automation on your software and ship it out the door. No money? Cut down on testers and have an automation engineer do it all.
While I’m a proponent of automation in its proper context, you still need the interactive tester to dig deep using exploratory testing and find bugs. Let’s look into what exploratory testing is and a method to find great exploratory testers.
Exploratory testing is an art—know the requirements, understand the features, seek out their weaknesses, all while building the test plan. Cem Kaner, defines exploratory testing as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”
In other words, exploratory testing combines ad hoc testing with intelligent randomized testing. This testing approach requires a skilled tester with knowledge of the product and well-honed testing instincts. James Bach, another expert on exploratory testing, defines exploratory testing as “test design and test execution at the same time.”
I still don’t shun the automation engineer—we must have automation to reduce repetitive tasks and give us more time to dig deeper with exploratory testing. But sometimes people overemphasize automation. One team I used to manage had a few expert automation engineers that even refused to log bugs. They said their job wasn’t to find bugs but to write automation. Huh?
How, then, do you find an exploratory tester? You need to know how to recognize the attributes of a successful exploratory tester. Over the years, I have seen the following attributes continue to bubble up in great candidates:
- Natural test aptitude
- Current technical knowledge
- Desire to continually learn
- Deep curiosity of how things work
- Strong problem solver
- Good communicator
- An ability to handle ambiguity
I needed to find a way to help filter out the best potential candidates. A number of years ago, in conjunction with another senior tester, I developed a sample application with built-in bugs. It is a simple application, with GUI defects, memory leaks, functional quirkiness, documentation errors, etc.
I give an interviewee a 15 minute limit to find as many defects as they can. I look for somebody who enthusiastically dives into this activity and tries to uncover many bugs. This model has consistently guided my search for testers with great aptitude and has helped me identify strong candidates. The best candidates apply a structured methodology to find bugs rather than spending too much time running redundant tests.
I don’t want to give away all my secrets, since I may be interviewing some of you in the future, but I do want to point out that after I’ve verified that the candidate has tested the basic features, he or she should also hit boundaries, accessibility, error cases, usability, and edge cases.
The success rate I’ve had using this tool, while not scientifically documented, has been very high—many of the candidates testing high have proven themselves to be great exploratory testers.
Automation will always be part of testing, but an exploratory tester brings insight and intelligence to the testing process. Finding the right tester will improve the testing process and ultimately create a better product for the end user.
David Kosorok is a QA Team Manager for the Church.