Page 1 of 3

Tools for Making Ward Directories and Maps

Posted: Mon Oct 20, 2008 6:43 am
by davebailey
Hello,

I wanted to let folks know about some resources for making ward directories and maps.

One of the challenges is ways of converting information into a usable format and keeping it tight. I try to distribute our ward directory on a single page 8.5x11 sheet on one side and other information, such as leadership, building numbers, schedules on the other.

I've released a command-line tool for converting MLS Membership exports into various directory formats.

http://ldsadmin.org

The resulting information can easily be put into word processor or spreadsheet documents for formatting and printing.

Also, using the above tool to pre-process the data (or not), you can easily put together a Google Earth map utilizing a method I detailed on the excellent LDS Clerks wiki. Google Earth is free software for easily visualizing geographic information in 3D.

Take care,

David Bailey

Posted: Mon Oct 20, 2008 8:42 am
by RossEvans
davebailey wrote:
Also, using the above tool to pre-process the data (or not), you can easily put together a Google Earth map utilizing a method I detailed on the excellent LDS Clerks wiki. Google Earth is free software for easily visualizing geographic information in 3D.

Thanks, David. I have used the method you describe -- having batchgeocode.com geocode the records and create a KML file for Google Earth. It works, but with one large caveat: it does not handle geocoding errors.

The batchgeocode.com web site internally uses the geocoder at Yahoo, submitting each address and getting the result as an XML transaction. But batchgeocode.com discards the errors and warnings that Yahoo sends back. Rather, it just takes Yahoo's best guess and plots it. Usually Yahoo guesses right in parsing and mathcing an address, but sometimes it guesses quite wrong. The batchgeocode.com site will plot those errors -- families mapped in grossly wrong locations, such as Zip code centroids or incorrect streets -- without so much as a warning to the user.

Posted: Mon Oct 20, 2008 9:00 am
by davebailey
Well, that's true in the import/export process, but you can fix points in Google Earth if you are aware of errors before distributing the KML file. Also, I have found the process to be very accurate, typically within a block of its actual location, at least where I am in the Southeastern United States.

If you know of an error free geocoding method, I'm all ears!

Posted: Mon Oct 20, 2008 10:57 am
by RossEvans
davebailey wrote:Well, that's true in the import/export process, but you can fix points in Google Earth if you are aware of errors before distributing the KML file. Also, I have found the process to be very accurate, typically within a block of its actual location, at least where I am in the Southeastern United States.

If you know of an error free geocoding method, I'm all ears!

Actually, I don't know of an error-free geogcoded method, because geocoding is inherently error-prone. The best that can be done is to trap the flagged errors and precision qualifiers and deal with those as special cases. That is why quality geocoders -- even Yahoo's free API -- flag such errors and warnings. But users ignore those specific warnings at their own peril.

Unfortunately there does not seem to be a batch-geocoding method that reports the error flags and is also legal and free for general use. It is even problematical to license commercial solutions on a scale useful to local units. (Search terms such as "geocode" on this forum and you will find lots of discussion about that problem.) The best hope for a general solution for local units is that Church developers are known to be working on a central tool that we hope will be released sooner rather than later. Such a tool, presumably, will engage local clerks to correct errors.

Your suggested method for correcting the errors in the KML file assumes that the user can detect the errors somehow. That can happen if addresses are miscoded so far away to be obvious outliers, but finding other errors witihout benefit of the underlying geocoder's precision flags is not easy. It's a matter of not knowing what you don't know.

In the case of batchgeocoding,com, Yahoo's geocoder does flag precision warnings, but batchgeocode.com ignores the flags. (I wrote some geocoding scripts that accessed Yahoo's API directly, and retrieved the results along with the precision fields, and created KML for Google Earth after a process of manual review by clerks. But I withdrew those scriptswhen I discovered this method seems to violate Yahoo's terms of use.)

There are other third-party tools, such as this one from GPSVisualizer, that do show the warnings about precision from Yahoo or Google -- but only one address at a time. The online providers seem to tolerate this use of their geocoding services -- even though the third-party sites such as batchgeocode.com and gpsvisualiser.com may violate the terms of use of theiir APIs -- so long as batch geocoding suppresses the precision flags. Presumably that is to protect the intellectual property and markets of the original providers such as Tele Atlas, which market professional-level geocoding. Much of the pro stuff categorizes the precision of its output with even more granularity. But at a minimum, we need to see the precision categories reported by free online geocoders such as those at Yahoo or Google.

Posted: Mon Oct 20, 2008 5:52 pm
by davebailey
Considering that I have never heard about flagging precision warnings (I'm assuming you mean when it can't find the street or the address on that street) I bow to your superior knowledge on the subject.

I'll think about ways to resolve this, but I think you're right. When the church offers a Google Map mashup for clerk use that can be edited and corrected is when we'll have a good answer for this.

Posted: Tue Oct 21, 2008 8:04 am
by RossEvans
davebailey wrote:I'll think about ways to resolve this, but I think you're right. When the church offers a Google Map mashup for clerk use that can be edited and corrected is when we'll have a good answer for this.

I have high hopes that the Church will develop a robust geocoding tool that will engage clerks to fix the inevitable errors. Since most of those errors are ultimately caused by dirty data in the MLS address fields, I hope that tool also includes help with validation and standardization of those addresses and capturing the corrected addresses in MLS. It is a classic GIGO (garbage in, garbage out) problem.

As far as the precision flags go, there are many possible categories of granularity that different professional geocoding engines might use. (I assume that the Church is able to license such enterprise solutions.) But it seems that most applications, including this one, ultimately have to employ those categories to define breakpoints for three major quality groupings:
  1. Geocoded addresses that are automatically accepted because they meet a certain standard. In the case of Yahoo's output, this would be data that was classified as geocoded at the "address" level without warnings. (Actually, I think the Church likely will even adopt a higher standard, with the objective of capturing so-called "rooftop" or parcel coordinates without the range-interpolation errors present in Yahoo and similar tools. That might entail a combination of premium commercial geocoding and manual correction by clerks.)
  2. Geocoded addresses that are automatically rejected because they flunk the quality standard, or addresses that could not be geocoded at all. That would include addresses where the street number is invalid, or the street could not be found at all, etc. What Yahoo typically does in such cases is fall back to centroid coordinates of a whole street, Zip code or even city -- which typically will be far away from the real location -- and flag that in a field that describes the precision.
  3. Geocoded addresses that are ambiguous, and need human verification. This would include addresses where the geocoder cannot find a sure match for the street address, but finds something that is close in spelling, etc.. Or the geocoder might not find "Montana Street" but substitutes "Montana Ave," or assumes "123 Elm St" when the input data is just "123 Elm." Geocoding software is usually right about such guesses, but sometimes it is wrong. Good software will provide the warnings along with the corrected address, so the user can verify it or override it manually.
In both of the latter two cases, clerks need the ability to fix underlying address problems and geocode again, or manually supply coordinates by some other means. Those means might include a visual tool such as maps.lds.org now employs to correct meetinghouse locations. They should also provide latitude and longitude fields that could be manually entered into a screen form. Some small percentage of addresses simply will not geocode automatically, and that percentage grows to be large in rural areas. In developing countries, manual geocoding may be the only way to capture the data.

Posted: Tue Oct 21, 2008 11:28 am
by russellhltn
boomerbubba wrote:three major quality groupings
Wouldn't there be another where the address is accepted without problems, but the software isn't sure exactly where 123 is, but is plotting a position between the 100 and 200 block?

Posted: Tue Oct 21, 2008 12:44 pm
by RossEvans
RussellHltn wrote:Wouldn't there be another where the address is accepted without problems, but the software isn't sure exactly where 123 is, but is plotting a position between the 100 and 200 block?

That is actually what Yahoo's geocoding engine does as its best case, in my experience. Those records geocoded at its "address" level are just interpolated within such a range, and Yahoo's API has no higher category of precision. Until recently that has been state-of-the-art in the geocoding industry.

But higher-end providers now are folding in parcel-level data gleaned from local property tax records. Somehwat misleadingly, this is often called "rooftop level" geocoding, although "parcel level" would be a more accurate label. Even within the United States, in high-end geocoding services, the coverage of this level of precision is far from complete. However, it is increasingly possible to achieve this precision by filling in the gaps by manual intervention.

I rather think the Church will be aiming for this level of precision somehow. Not only is it more convenient for general purposes such as viewing my home-teaching route, but precise geocoding is very important for emergency-response purposes. Since emergency preparedness is one of the many uses for geocoding in the Church, and we do possess an unparalleled reservoir of free labor, I would expect the Church to aim for that high standard in capturing the data.

Google Maps Problem

Posted: Wed Oct 22, 2008 9:17 am
by zaneclark
After much trial and error, I finally succeeded in uploading our membership onto Google Earth. I am still working out all the possibilities but have one problem. When I first managed to load it, I was able to also see the locations on Google Maps. But now when I go to "Show in Google Maps" i get only a map of the area with none of the locations noted. Google Earth is wonderful, but I can't figure out how to show the street names that you get with Maps. I am very new at this type of thing, so I have probably missed something very simple...

Posted: Wed Oct 22, 2008 9:40 am
by RossEvans
zaneclark wrote:After much trial and error, I finally succeeded in uploading our membership onto Google Earth. I am still working out all the possibilities but have one problem. When I first managed to load it, I was able to also see the locations on Google Maps. But now when I go to "Show in Google Maps" i get only a map of the area with none of the locations noted. Google Earth is wonderful, but I can't figure out how to show the street names that you get with Maps. I am very new at this type of thing, so I have probably missed something very simple...

This is a Google shortcoming. The "View in Google Maps" function in Google Earth is a misleading label. The function does not actually export to the Google Maps web site the placemarks and data you created and displayed in your local Google Earth application. It just opens the web site and moves the standard map to the same location.

(That's a good thing for us, since uploading both names and addresses to the Google Maps website would probably put you in violation of Church policy guidance. It may be possible to upload the same KML file to Google Maps by other means, but I do not recommend it for that reason. In our ward, when we upload addresses to Google Maps to get a better printable rendering, we omit the names. And we do not rely on the batchgeocode.com geocoding.)

To see streets and street names within Google Earth, in the "Layers" sidebar on the left of the screen, make sure the item labeled "Roads" is checked. Then when you zoom in, the roads and their names should become visible.