The common metaphor for building software systems has traditionally been the construction industry. People knew a lot about constructing buildings. They had been doing it successfully for thousands of years. Anything bigger than a simple shed required a complete blueprint before a single shovelful of earth was turned or a single nail was driven.
The implementation of the construction metaphor in the software industry was the waterfall methodology. The waterfall approach called for a complete and detailed documentation of system requirements, followed by a set of program designs and specifications from which programs were written. When all the programs were written, they would be tested. The final product would then be turned over to the customer.
However, in the 1980s industry experts began to see a trend – software projects were failing more often than they were succeeding. While the development tools, languages, and technologies were getting better, most of our projects were not seeing the promised benefits. Software systems took too long to build, cost too much money, and did not deliver what the customer needed
What is Agile?
Through the 1990s, while the rest of us were trying to simply get better at executing the next great waterfall method, discerning industry leaders and pundits began to suspect that the whole metaphor of the construction industry was fatally flawed. It became obvious that if we continued to do what we had always done, we would always get what we had always gotten.
In 2001, a group of people with experience, insight, and ideas got together and created the Manifesto for Agile Software Development. The Manifesto was not designed to define methodologies and specific practices, but to articulate a set of principles that could lead to successful software development projects.
Proponents of many other lightweight methodologies converged on this idea of agile development. Representatives of Extreme Programming, SCRUM, DSDM, Feature-driven Development, Pragmatic Programming, and others saw a common thread in the Manifesto. Though there was no proof that agile software development would work, there was ample proof that waterfall development was not working. Anything different was worth a try.
Adoption of Agile Principles
In the spring of 2002, I was working on a software development project for the Missionary Department of the Church. When we started the requirements gathering and documentation effort, the methodology at the Church was still very much a waterfall approach. We planned a nearly two-year effort to write a comprehensive set of requirements and specifications that would be the blueprint for what we assumed would be a five-year development project. Our intention was to write the most comprehensive, detailed, well organized and bulletproof requirements document ever seen in the Church.
Three of us had worked for over a year and had produced over three hundred pages of carefully detailed documentation when I was called to a meeting with the managers of the ICS (Information and Communication Systems) Department. The statistical probability of success on our massive project was slim. ICS really wanted to hit a home run with this project. The Agile Manifesto had appeared a couple of months earlier. Several key managers were eager to try this new approach. My project, along with several others, was targeted as a likely candidate.
The analyst team quickly wrapped up the requirements document and published it. A team of user interface designers, developers and quality assurance engineers was organized, and we went to work on the first phase of the new system – the Internet version of the Missionary Recommendation Form, an on-line version of the papers that missionaries fill out prior to receiving their mission call.
The transformation from traditional waterfall to new-age agile was a bold move. A couple of members of the team had recently been hired by ICS from other companies and they had prior experience with early forms of agile methodologies. The rest of us simply had the Manifesto on which to rely for our guidelines. We learned about user stories, iterations, daily stand-ups (which we referred to as scrums), test-driven development, paired programming, refactoring, and incremental releases. For most of us, these concepts were completely foreign and more than a little frightening.
We were making up the process as we went along. We made many mistakes and were sometimes slow to learn from them.
At the same time, the direction for development technologies was changing as well. ICS wanted to move from the licensing and support problems of single-vendor solutions to the freedom and reduced cost of open source technologies. All new systems were to be developed in Java and were preferably to be web-based rather than thick-client applications. Eventually, ICS wanted to replace their entire portfolio of Windows-based applications with browser-based Java applications. The need for new frameworks, tools, and skill sets added to the challenge of becoming agile developers.
Getting started in this environment was challenging. The developers were trying to generate velocity – the measurable rate at which they could anticipate completing programming tasks – while experimenting with technologies and inventing new methods and processes. It took the team six months to finally produce the first few demonstrable pieces of the new Recommendation Form module, but eventually they got traction, and development began to speed up.
Less than a year after the Recommendation Form project was started, the fledgling Internet Missionary Recommendation Form was ready for beta testing. It was very rough around the edges, not a polished product like we were used to releasing. However, several beta test stakes along the Wasatch Front helped us shake out the bugs and make needed refinements. Several months later, a wider range of beta stakes across the United States were added. More feedback came, and more refactoring was done.
By the spring of 2004, about two years after the project was started, the Internet Missionary Recommendation Form was gaining wide acceptance throughout the United States. Today, virtually all recommendation forms submitted by prospective missionaries in the United States and Canada come through the on-line system. Usage is growing in other English-speaking areas, such as Australia and New Zealand. The system has been translated into Spanish and is being adopted gradually in Mexico. It is soon to be released in other South American countries. A Portuguese version is soon to begin beta testing in Brazil.
With the Recommendation Forms project under our belt, we moved on to other modules of the missionary department’s legacy system. The eventual goal has been, and continues to be, the total replacement of the legacy system. Working in phases, we have been gradually rebuilding outdated Oracle Forms, Oracle Reports, Visual Basic, and other programs while producing more scalable, flexible, and robust Java programs that utilize Spring, Hybernate, Java Server Faces, and various Stack technologies. The users have been highly pleased with the results.
At the same time, we have been maturing as an agile shop. Iterative, incremental development has become a way of life. While we do not fully practice test-driven development, we are much more quality-centered than we once were. Test cases are written before coding begins, whenever possible. Code coverage is measured and reported regularly. Designers, developers, and testers work closely together as mini-teams assigned to each user story to flesh out the requirements, specify the tests, write the code, and implement the features. Each team works with the customers throughout the development process to ensure that what is produced is what the users need.
This is a far cry from the old days where only the business analysts talked to the customer, the designers talked to the analysts, the programmers talked to the designers, the testers talked to the programmers, and projects seemed to go on forever. Today our environment is vibrant, and exciting. Not only is it more satisfying for the team members, but we are producing software that the users can actually use. Not everything is perfect and bug free, but we have produced a substantial amount of software in a reasonable period of time. Today we are on budget and on schedule to complete the last phase of the replacement of the legacy missionary system.
Embracing the Agile Manifesto
No one on the project today is yearning for the good old days of waterfall planning and endless requirements documents. We still do not know everything there is to know about agile development and agile project management, but we are convinced that traditional waterfall projects are dinosaurs for which extinction could not have come at a better time. Like the Agile Alliance, we are beginning to truly value people over processes, software over documentation, collaboration over negotiation, and embracing change over enslaving ourselves to a plan.
David Armstrong is a technical program manager for the Church.