Page 1 of 1

Emergency Response Tool

Posted: Thu Sep 11, 2008 5:00 pm
by douglas.a.hatch-p40
I was recently asked to implement and emergency response plan in our ward. Specifically, the Bishopric asked me to divide the ward into proximate geographic regions, each consisting of a roughly-equal number of ward members. Within each region a leader was assigned and given a list of all members in his region. In the event of an emergency (e.g. earthquake), the leaders would be responsible for contacting each member in their region (on foot if necessary), discovering needs, and reporting back to the Bishopric.

I accomplished this task very manually via the following method:
  1. Downloaded membership data from MLS
  2. Imported membership data into Google earth
  3. Manually subdivided the ward by eyeballing clusters of members on the map. Used custom markers to assign members to different regions.
  4. Exported the KML file from Google Earth into Excel
  5. Performed filtering to break out each district
  6. For each district, batch geocoded the addresses of all the members
  7. Ran TSP Solver to produce a least-distance route
  8. Finally, sent each district leader a sheet with a list of families and a map (see attached for example)

This process took about 6-8 hours the first time I did it. I've been re-performing this process every 3 months to incorporate move-ins and move-outs. This maintenance takes about 3hrs per.

I think there is an opportunity here to build and awesome tool that would streamline this process and which could be used by wards all over the World. Here is my vision of what this tool would look like:

Ideally, the tool would be built into MLS so that it could pull membership data automatically. More likely, someone would need to do a dump each week and upload into the tool.

INITIAL SETUP
The tool would show a map of the ward with all members' home plotted. The tool would then have markers labeled 1-X that could be assigned to each member representing district 1-X. There would also be markers 1.1-X.X representing district leaders. When all members and district leaders have been assigned a marker, the tool automatically generates the district lists. There should be an option to print these off or email them to the district leaders.

MAINTENANCE
When updated membership data is provided to the tool (either auto or via manual upload) the tool should determine move-ins and move-outs. Move-outs should automatically disappear from the map. New families should be plotted, but left unassigned to a district. Someone will manually need to make the assignment using a marker. The tool will then show which district have been changed and provide and option for printing or emailing.

Originally, I was trying to come up with a fancy algorithm to automatically assign families to districts as a function of distance, load balancing, etc. But this became overwhelmingly complex. So whatever the solution ends up being, there will probably need to be a manual component where someone makes the actual assignment. This tool would just make the process a whole lot easier.

On a side note, it would be nice to incorporate into this tool a DB that tracks other information that could be pertinent during an emergency (e.g. special needs, inventory of equipment, tools, etc).

Someday I hope to have the time to code this myself. Until that time, I guess I'll stick with our manual process.
Emergency Response.pdf
(210.11 KiB) Downloaded 763 times

Posted: Thu Sep 11, 2008 8:52 pm
by RossEvans
Impressive work, espiecially the optimized route maps.

Emergency preparedness is one of several uses for geography-based technology in the Church. As civill authorities can attest, such emergency uses are particularly sensitive to accurate geocoding.

I have added this thread to several others I subscribe to in this forum on the general topic. (Search on "maps" or "geocoding" and you will find a large body of work here.) In addition, Church developers are known to be working on some centralized solutions for release sooner rather than later. Like you, I also hope they will integrate such functionality into MLS.

How are you doing your batch geocoding? In my experience, getting that done right and keeping it updated is the hard part -- especially free and legal at the same time. The crux of the problem is dealing with the errors of omission and commission, and I have not found a general solution that permits that to be done for free. I published some scripts on this forum that I was using to update our ward maps weekly, but I had to un-release them when I realized that I was running afoul of intellectual-property requirements. I am just now about to launch a whole new geocoding method based on very accurate public-domain data obtained directly from my city instead, but I think most localities are not so progressive about producing and releasing such data.

The optimized routes you publish could be quite useful, and of course depend upon accurate geocoding. I also have been experimenting with TSP Solver(also called OptiMap) for other purposes -- fast-offering routes. It is elegant technology to deal with a famous problem of computing.

As for the method of dividing the ward into small routes or districts, that is essentially the same computing problem stakes face in developing balanced boundaries. For my smaller-scale purposes I use commercial GIS software to gerrymander the ward into such small pieces. Census block groups work well as jigsaw pieces in our ward; others with different scales would need to find optimal areas for their situations. In some cases, if wards or stakes populate the legacy "GEO Code" fields in MLS reasonably, those might be useful for this purpose.

Posted: Fri Sep 12, 2008 10:19 am
by douglas.a.hatch-p40
I've been using gpsvisualizer, which is a batch geocoder using the Yahoo Maps API. I've done spot checking of output and it appears to be fairly accurate. There are some issues. Like it pin-points a single spot for all addresses residing in trailer parks and military bases. It seems like most Map APIs are getting pretty robust in terms of interpreting rougly-formatted address strings. On only two occasions have I had to clean up the address before submitting.

After I've got a list of geocodes, I dump them into the TSP Solver and then use optimal-route map that is generated.

Posted: Fri Sep 12, 2008 1:25 pm
by RossEvans
douglas.a.hatch wrote:I've been using gpsvisualizer, which is a batch geocoder using the Yahoo Maps API. I've done spot checking of output and it appears to be fairly accurate. There are some issues. ....

The problem with the gpsvisualizer site, which it shares with others such as batchgeocode.com, is that these third-party implementers submit the address transactions to Yahoo's geocoder -- or someone else's geocoder -- but then just accept each lat/lon output as if all were correct, and discard the error/warning messages that the geocoder returns along with each transaction. Only by testing each transaction and dealing with the errors and ambiguities can the user be sure of getting accurate locations. The gpsvisualizer site does provide an interactive, one-at-a-time interface called Quick Geocoder that does display the accuracy/error for each transaction, and GPSVisualizer's batch-geocoding page warns about this inherent anomaly:

Verifying strange results

Sometimes the geocoder returns coordinates that don't seem right. Unfortunately, there's no way to get Yahoo's precision information using this form -- but you can see Yahoo's precision estimate for individual addresses if you run them through the Quick Geocoder.

For example. I threw some of our problem addresses as a batch at GPSVisualzer, and sure enough it plotted them in the wrong place without so much as a warning. Only the QuickGeocoder revealed that the addresses were being plotted by the unacceptable (to me) level of zip or city centroids.

It is not that hard to write a script that accesses Yahoo's geocoder directly and returns the results along with the extra fields describing errors and warnings. I did that, and used as as the basis for my ward's geocoded maps. That allowed for a manual step between geocoding and mapping for the clerk to resolve the errors and ambiguities, and manually geocode some exceptions. I also published those scripts on this forum. But after I belatedly read the fine print in Yahoo's Terms of Use for the Maps API, I scrapped that method because the TOU clearly allows Yahoo's geocoder to be used only for purposes of plotting maps on Yahoo.

As an aside, I still wonder why Yahoo allows sites like GPSVisualizer.com and BatchGeocode.com to operate, since they also seem to violate the TOU. Yahoo and its licensed map provider(s) seem to tolerate the free batch gocoding in practice, so long as the errors and warnings are suppressed. Giving users access to all this for free would more seriously threaten the intellectual property interests they are trying to protect.

As I said above, I have not found a generalized method of geocoding all of our addresses accurately, freely and legally. The legal complexities are almost worse than the technical complexities. There are commercial solutions that can be licensed up-front -- which tools I hope the Church will license and make available to local units -- but they are not within easy reach of wards and stakes directly. I have found a local solution for my own ward using very precise public-domain data for my city, but at this point I cannot build a general solution for others.

Posted: Fri Sep 12, 2008 4:26 pm
by russellhltn
boomerbubba wrote:As an aside, I still wonder why Yahoo allows sites like GPSVisualizer.com and BatchGeocode.com to operate, since they also seem to violate the TOU.
Maybe it would be worth asking Yahoo if what you wish to do is OK. The worst that could happen is your email goes in the bit bucket.

Posted: Fri Sep 12, 2008 8:45 pm
by RossEvans
RussellHltn wrote:Maybe it would be worth asking Yahoo if what you wish to do is OK. The worst that could happen is your email goes in the bit bucket.

Actually, an explicit "No" might be even worse. I thought it better just to cease the activity that I formerly had performed out of non-malicious ignorance.

Besides, I have learned, big companies typically do not engage such questions about interpreting their terms. They want the language of their legal documents to stand for itself, which is surely what their lawyers would advise. Especially if I am not a paying customer, they have too much to lose and little to gain by appearing to modify their carefully crafted terms in such over-the-transom correspondence with a stranger. Even vendors offering services for sale have not responded to such inquiries, I presume because my potential revenue is trivial to them but a conciliatory email that gets into the wrong hands theoretically could cost them a lot down the road.

In my parochial case I have moved beyond trying to use such a proprietary geocoder, because I found public-domain data for my city that is qualitatively equal or superior to any company's geocoding, free or otherwise. It is absolutely superior to Yahoo's because while that geocoder is based on interpolated address ranges, I have discrete coordinates for each parcel of land from the city/county database.

(That interpolation error is a whole different category of error than address-parsing errors, BTW. If the intended use of the geocoding is for emergency preparedness, one might reasonably decide not to use interpolated coordinates because they are too fuzzy for such high-stakes purposes. That is why Yahoo and its competitors typically disclaim that their geocoding is not suitable for uses such as public-safety dispatch.)

I did have to develop some fuzzy-match SQL queries to resolve my address matches against the city database in basically the same way a geocoding engine's parser would: certain matches, probable matches to be examined manually, and no-hit mismatches. A professional geocoding engine typically would categorize and code more gradations than that, but I as an implementer/user would need to translate their larger set of categories into those three simple buckets by setting the threshholds for my application. (Someone else's application might have different requirements and thus different rules.)