YouthMaster v3.0: The Technology Behind

Discussions around miscellaneous technologies and projects for the general membership.
Post Reply
mattfarley
New Member
Posts: 31
Joined: Sun Feb 18, 2007 9:20 am

YouthMaster v3.0: The Technology Behind

#1

Post by mattfarley »

After recently announcing the release of YouthMaster v3.0 as a [General Discussion] thread, I received a PM asking me if I could share with the community some of the technology and tools that were used to create it.

In response I'll happily attempt to outline the details below.

First let me explain some of the background behind the development of v1.0 and v2.0.

v1.0 was very much a quickly-thrown together mess of spaghetti cut-n-pastes to meet its sole purpose: 1 ward, DTG-only. There was no vision for Scouting, YWPP or extending the app to other wards. Thus a very "quick'n'dirty" approach was taken.

v2.0 was a rush to redo a bit of plumbing to facilitate multiple wards and add Scouting + YWPP. The result was an even larger, even messier codebase. It was very procedural, repetitive, and maintenance was a nightmare.

As a vision formed for all the new features of v3.0 and beyond I knew that it was time to hunker down and do things properly from the ground up, building a solid foundational architecture on which to grow and function. I knew the up front investment in terms of time and effort would be significant, but the payoff down the road in terms of elegance and maintainability would be worth it.

Thus, with v3.0 I scrapped most of the codebase and started over with the goal of incorporating many industry standard software development best practices such as unit tests, proper environments, continuous integration, selenium testing, object-oriented design patterns, etc.

The diagram below illustrates the end result:
Image

Development Environment:
I opted to go with a virtualized development server/environment which is becoming more and more of a standard practice as virtualization has become so pervasive.

From anywhere in the house (desktop, laptop, etc) I can open an RDP session to the development server and am greeted with an Ubuntu Desktop with Eclipse (3.4), MySQL 5, PHP5, Apache2. From the development server I iterate through a cycle of develop, unit test, manual test, check-in to svn, and then let the build server take it from there.

As I'm developing on the dev server, I often use a virtualized WinXP box to test the dev environment with several browsers (IE6/7, Firefox, Safari, etc). Most often I'm developing from my Ubuntu workstation, and I'll often use the local Firefox there to test as a I develop.

Build Server:
In the Java world CruiseControl and Apache Ant dominate the build and deployment scene. Luckily, there exists an application called phpUnderControl which wraps CC and Ant in more of a "PHP-flavor".

Once the build server detects a new checkin to source control (SVN) it kicks off a new build (the process of downloading the latest version of the code and deploying it to the remote host).

My current build script performs the following:
  1. Downloads latest version of source code
  2. Checks the code for style compliance (PHP_Codesniffer)
  3. Checks for proper syntax on all .php files (php -l)
  4. Combines all .css and .js files into a single .css and a single .js
  5. Compresses the .css and .js, stripping them of comments and whitespace
  6. Uploads the resulting copy of the source to the remote web server
  7. Configures miscellaneous environment settings on the server (php.ini, .htaccess, creates several symlinks)
  8. Strips down and recreates the test instance database
  9. Performs all Unit Tests
  10. Performs all Selenium Tests
Selenium tests are a great way to test web applications -- they actually spawn a browser of your choice and click through the website as a pseudo-user while validating (assertions) functionality. For example, I have a Selenium test that registers itself as a new user using the forms on the site, attempts to login and edit its profile, then logs out. If anything goes wrong in the process, the test fails, and I'm notified. It even works well with Ajax-powered UI's (as most of YouthMaster is).

Test Environment:

After the above, the code is now deployed and tested in the "test environment" on Dreamhost's web servers. I've been using Dreamhost for all my web hosting needs for the past 3 years and they've been excellent. Through several promotions and "thanks-for-being-a-loyal-customer" my account has been upgraded to "unlimited everything" (space, bandwidth, databases, etc) and I still only pay $8 / mo. I host about 15 non-youthmaster sites on Dreamhost and the performance has always met or exceeded my needs. They keep incremental backups of files and databases. They give me nearly free reign in the Linux shell on my account. They have a very slick web-based control panel. I could go on and on.

Live Environment:
Once a build has been fully tested and is deemed as worthy for release, "going live" is just a matter of changing a single symlink to point from the old copy of the code to the new. Users will immediately be using the new version. It's important to note that from an infrastructure perspective, the test and live environments are identical -- this is key to a smooth transition from test to live.

So let's walk through how I might "fix a bug" and get it deployed:
  1. Sit down at my thin-client of choice and open a remote desktop to the virtualized development machine
  2. Open Eclipse, synchronize with SVN
  3. Make the change
  4. Run the Unit Tests
  5. Manually test (if necessary)
  6. Check in to SVN
  7. Build server detects change and kicks off a build
  8. The "build" performs the 10 steps listed above
  9. Assuming all the build tests pass, I can then just change the symlink on Dreamhost to point all "Live" url's to the new build
Now that may seem like a lot of work just to make a simple change, but the reality of software development requires an extremely robust test and deployment process to minimize the risk of your new changes introducing errors (there's some philosophical metric that states for every bug you fix, you introduce x# of additional bugs).

There's great comfort in being able to make significant changes and knowing that all your unit tests and selenium tests got you "covered" and will alert you to errors. (Code Coverage is a software development metric that measures the percentage of your code which is "covered" (executed) by Unit Tests)

Code being built->tested->deployed as its being checked in, in an automated fashion, is the gist of Continuous Integration (see link and essay at the bottom).

Other Notes:

Object-Oriented Design/Patterns -- v3.0 does a fairly decent job of leveraging the OOP features that were introduced with PHP5. I don't profess to be a OOP guru, but for the most part the application logic has been moved to objects/methods/properties. I've also done a much better job of separating the presentation layer from the logic layer (and database layer) -- MVC. I also used the PHP5 PDO data abstraction layer, and thus theoretically could switch from MySQL to Oracle / SQL Server / etc, with a 1-line code change.

The jQuery Powered Ajax UI -- in v3.0 the jQuery library has replaced my previous home-grown library. The power of jQuery and the ease with which you can do everything from Ajax forms to fluid animations is intoxicating. I went wild with jQuery, using many components from the UI library (http://ui.jquery.com). The Ajax form validation and submission is also note worthy.

External APIs -- v3.0 uses 3 APIs from Google: Maps, Geocoding, and Charts. I love Google.

According to your diagram, each ward has their own database? So if you have 100 wards that's 100 databases? Why not just use 1 large database and use the ward as a primary key? -- I've often pondered the pro's and con's of my architecture here. On one hand I like the fact that the wards are segregated. I get a warm fuzzy feeling knowing that if a database gets lost, corrupted, hacked, etc, it's only that 1 ward, not everyone. Also in terms of performance and design, it's nice not to have to carry that extra primary key around and filter out everyone else's data. Given those thoughts and the fact that Dreamhost provides me with unlimited databases, I've settled on the current approach. I also talked it over with a co-worker who is a Microsoft MVP for SQL Server and he agreed that there were both pro's and con's to either approach and stated there is no right or wrong answer, it's just whatever works best / makes the most sense for my app. Someday I may find drivers to consolidate everyone to a large db.

Can I help the project, as a graphic designer? -- I would like to a bit of a visual refresh for v4.0. If you're interested in drafting some mockups or playing with the css, please feel free to contact me. I must warn you though, I'm very picky when it comes to UI :)

Can I help the project, as a tester? -- If you're interested in writing Unit Tests or Selenium Tests (these can be 'recorded' instead of hand-written) or manual testing, please let me know.

Can I help the project, as a documenter? -- If you're interested in updating / adding to the FAQ's or writing some user documentation, yes.

Can I help the project, as a developer? -- Theoretically I could email you a clone of my dev vm, create your svn account, and you'd be ready to contribute. However, I'm currently hesitant to grow beyond myself for development. The main reason being that I have been entrusted with the personal information of hundreds of users, and it would be irresponsible of me to give strangers access to that information (which they would have if they had access to the source code). Another driver is that I would have little means to validate your development skills. I'd only want the best PHP coders anywhere near my baby :)

So that about covers it. Please let me know if you have any questions about the technology or ideas for future versions. I'll leave with some reference links below and a couple screenshots: phpUnderControl (dashboard view), and what my workstation looks like when I'm developing (Dev VM, WinXP VM, Build Server VM).

Links:
Eclipse PDT
phpUnderControl
CruiseControl
Apache Ant
PHPUnit
PHP_CodeSniffer
Selenium
Continuous Integration -- essay by Martin Fowler
Sun VirtualBox
Dreamhost
YouthMaster Demo Videos (screencasts)

Image

Edit: Here's the workstation pic: Ubuntu Desktop 9.04 RDP'd to 3 VM's

(It's a bit large (high resolution))
togiles
New Member
Posts: 15
Joined: Fri Aug 22, 2008 9:41 am
Location: Beijing, China
Contact:

#2

Post by togiles »

Thanks for the details... nice development flow / setup.
Post Reply

Return to “Other Member Technologies”