Updating the Worldwide Application

[McDanielCA]

Updating the Worldwide Application was originally posted on the main page of LDSTech. It was written by Devon Gibson.
---------------------------------------------------

It has been my experience that homegrown applications can present a number of challenges as they expand and evolve. One of these challenges is the absence of a defined build and release schedule. A homegrown application that I currently manage has evolved over the years into a massive system that is accessed around the clock by users worldwide. Because it has evolved so much over the years, there has been a great deal of interpretation left to different teams as to how builds and releases should be handled.

To add a little more flavor to the situation, this application is not written in Java, the long-standing development platform of choice at the Church. It is written in a mixture of classic ASP and ASP.NET, something that has only recently been adopted as an acceptable platform here. Due to the efforts of strong development and QA teams, we’ve seen an increase in application stability and improvements in overall functionality as we’ve worked to get this homegrown behemoth under control. We have now come to another fork in the improvement road: an acceptable release process.

The primary objective of a solid release should be to get a fully functional application into the hands of users as quickly as possible while keeping the impact and inconveniences to a minimum. Our current release process is fairly simple: after QA signs off on the code changes a manager notifies the primary business user and the development team pushes the changes to production. Even though this method allows for a very quick turnaround going from development to production, there are a few areas in need of change to ease the impact of a release:

1. Production code should be getting pushed from a clean environment instead of the development repository. We have code freezes and bug scrubs, but pulling the production-ready code from development leaves room for errors like accidentally making changes to a file that has already been flagged as clean or grabbing the wrong file completely. Leveraging the staging environment for this application would serve three purposes: 1) Allow for a test deployment of the code; 2) Provide a clean environment identical to production for regression testing; 3) Serve as the code repository for the production release. Access to both the staging and production environments should be restricted to those involved in the deployment to insure the integrity of the code.
2. Allow the developers to turn production ready code over to someone once it’s approved. We have people responsible for promoting code to the staging environment and then to production (and then taking the heat if something goes wrong). There is no need to take up development time with deployments. Ideally, once the code changes have been approved, development shouldn’t need to worry about it anymore.
3. All users should be notified of changes gracefully. We do notify primary users and have a set release schedule, but it may or may not get communicated down to the thousands of users worldwide. Currently, access to the system is disabled for releases by disabling logins to the database. This is very effective in keeping users out but causes a browser timeout for the Web-based application. A change as simple as redirecting the load balancer to a page stating the system is undergoing maintenance will ease user frustrations and eliminate calls to the Global Service Center.

As always, whenever there is a change in policies or procedures, there is a tradeoff that will take place. In this case, we will lose some of the quick turnaround that comes with development releasing straight into production, but we will be able to deliver a stable release in a shorter maintenance window with fewer service desk and helpdesk calls.

Devon Gibson is an applications systems engineer for the Church.