Apparent bugs in MLS export files (v 3.3.0 and 3.3.1)

Discussions around using and interfacing with the Church MLS program.
RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Apparent bugs in MLS export files (v 3.3.0 and 3.3.1)

Postby RossEvans » Fri Mar 04, 2011 3:18 pm

For users of the ward MLS export files, I have discovered and reported two apparent bugs in the HomeTeaching.csv and VisitingTeaching.csv files, which I believe began with MLS v 3.3.0:

HomeTeaching.csv

This file normally contains one record for each household in the ward, together with its assigned home teachers, if any, and everyone's contact info.

Now it also contains spurious records, with only the Household column populated in the form "Name Unknown, William James" where "Wiliam James" is the given name of a recent move-out.

Such records have long been maintained internally in MLS as hidden placeholders for computing the ward's quarterly HT statistics. But they did not use to appear in the HomeTeaching.csv export file. If this is by design -- I can't imagine why -- the release notes need to say so.

VisitingTeaching.csv

This file contains one record for each sister in the ward, together with her assigned visiting teachers, if any, and everyone's contact info.

The phone and email data for the Sister Taught, Visiting Teacher 1, Visiting Teacher 2 and Supervisor heretofore has followed a standard rule:

If the sister has an Individual Phone in Membership.csv, then use that
Else use the Household Phone if she has one
Else null string.

If the sister has an Individual E-mail in Membership.csv, then use that
Else use the Household E-mail if she has one
Else null string.

The maddening thing is that these business rules still seem to be followed for almost every sister in my ward. But for a handful, either the phone, email address or both in for Sister Taught, Visiting Teacher 1 or Visiting Teacher 2 are wrong. (I suspect that Supervisor might also be affected in some cases someday, but there are so few supervisors that we just didn't happen to trigger the bug in my ward.)

Other MLS users with whom I correspond have reported similar problems. There were no such problems found in MLS 3.2.4.

I can discern no pattern to why this bug occurs only for a handful of records, so I don't think it was just a matter of an unannounced design change in MLS. I recall that there was a serious bug reported in the content of VisitingTeaching.csv in an early beta build of 3.3.0 that was fixed in a later build. So it may be that this subtle bug was introduced in the process of patching that one.

This is a serious problem for some programs consuming this data. Because of the lack of a proper Individual ID or Household ID as appropriate to uniquely identify the persons in these records, the contact info is often used to minimize ambiguity. So when the phone or email data is wrong, the assignment might be missed altogether. I first found out about this issue from a user of Ward Tools.

russellhltn
Community Administrator
Posts: 20734
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Postby russellhltn » Fri Mar 04, 2011 4:44 pm

RossEvans wrote:But for a handful, either the phone, email address or both in for Sister Taught, Visiting Teacher 1 or Visiting Teacher 2 are wrong.


When you say "wrong" what happens? Did it not follow the business rule and select the household email rather then the individual? Did it garble it? Or did it pull someone else's email?
Have you searched the Wiki?
Try using a Google search by adding "site:tech.lds.org/wiki" to the search criteria.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Fri Mar 04, 2011 6:28 pm

RussellHltn wrote:When you say "wrong" what happens? Did it not follow the business rule and select the household email rather then the individual? Did it garble it? Or did it pull someone else's email?


It did not follow the business rules for either phone, email or both. In some cases that resulted in blank data when under the rules there was a valid phone or email. Even within the small set of sisters with errors, there was not consistency. There were more email errors than phone errors, but one within the small set of sisters with email errors had a phone error and others did not. And, as I mentioned above, for the overwhelming majority of all sisters' records the results seemed to conform to the same business rules that have been in effect since the individual contact fields were introduced 15 months ago.

I did not discern a complete pattern, which I suspect might have something to do with other data elements such as the spouse's data, or might have some other root cause entirely. I will let the MLS professsionals finish the analysis with benefit of their source code and debug tools. I offered to send my query results showing the errors, my full export files, and a full MLS backup snapshot from the day of those exports, but I was not going to send that confidential stuff as email attachments.

I have not yet heard a response on the visiting teaching issue after a day, so I don't know how far any troubleshooting or triage might have proceeded. On the home teaching issue I reported to Salt Lake earlier, I got a response that was off-point. Rather than address the HomeTeaching.csv export file, the response -- which sort of seemed like a canned issue reply from level 1 support to close the ticket -- concerned a different MLS function entirely (the ""Households not Being Home Taught" report) and said the inclusion of the spurious records there was a longstanding feature. Maybe that response is correct about that report, but I replied pointing out that I was describing the HomeTeaching.csv export file, which is a different MLS function. So I still consider this issue open, too.

russellhltn
Community Administrator
Posts: 20734
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Postby russellhltn » Fri Mar 04, 2011 6:52 pm

RossEvans wrote:It did not follow the business rules for either phone, email or both. In some cases that resulted in blank data when under the rules there was a valid phone or email.


Just for kicks and grins, what happens if you re-key the missing phone number/email? (Or re-clear the "empty" fields?) I'm wondering if there's a stray character in there that's messing up the rules.

RossEvans wrote:I did not discern a complete pattern,


How about move-in date? I wonder if it has something to do with things being entered in a certain version of MLS that messed things up?
Have you searched the Wiki?

Try using a Google search by adding "site:tech.lds.org/wiki" to the search criteria.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Fri Mar 04, 2011 8:01 pm

RussellHltn wrote:Just for kicks and grins, what happens if you re-key the missing phone number/email? (Or re-clear the "empty" fields?) I'm wondering if there's a stray character in there that's messing up the rules.


Good hypothesis. I thought of that myself and neglected to include that possibility in my email to CHQ. With an MLS backup, the devs surely can look at the actual data captured under the hood with a hex editor and see. I did that within the export files themselves and found nothing, not even an extra space character, but those files are a step removed from the internal data store. One more reason the pros need to take it from here.

RussellHltn wrote:How about move-in date? I wonder if it has something to do with things being entered in a certain version of MLS that messed things up?


I discounted that immediately because I recognized some old names among the error records, and some relative newbies. Of course, this data can be rewritten whenever a clerk edits it in MLS. And lately it can even be changed by data entry on lds.org by members themselves, which is pushed down to MLS.

The only date-related clue I have is that this seems to have started with MLS 3.3.0 exports, and I recall a specific report of a hard-stop bug that got reported and fixed in the VisitingTeaching.csv output in an early 3.3.0 beta build. So this code got touched then. My information on that was secondhand, as my unit is not a beta site. According to the release notes for 3.3.0. that version did expand the number of MLS reports showing the individual phone and email data. Maybe there was some side-effect from shared code. Again, that's something for the pros to explore.

From the point of view of a consumer of this export data, the really maddening thing is that the worst effects of this bug would never arise if the HT/VT files had proper IDs in the first place, as do Membership.csv and Organization.csv. That basic design flaw has been the cause of much pain over the years for folks doing non-trivial work with this data. If that flaw weren't here, this latest subtle issue would not be such a problem because it would not be significant for joining tables.

As for the other problem in HomeTeaching.csv, by serendipity it has not damaged my work, nor I believe that of volunteer colleagues. But I report it here in case it affects others. The users of these export files range from fairly sophisticated developers to intrepid clerks who pop the CSVs into a spreadsheet and crunch it for some purpose.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sat Mar 05, 2011 4:20 pm

I now have a theory about how this is happening, or at least a pattern: It seems to have something to do with the interaction between the new lds.org directory and MLS.

The sisters with the missing email addresses have no Individual E-Mail in MLS, but do have a Household E-Email, so under the business rules we would expect their email address in the VisitingTeaching file to default to the Household E-Mail, just like other sisters.

However, these sisters' households have no email address on the website directory. It's not that they each have a household email but have suppressed it from general display; there is no household email address present. (As an assistant clerk I have privilege to see their profile.)

Since members now are allowed to edit their own contact information, I suspect that these few households have chosen not to list a Household E-mail at all, so they deleted it online. Then, when the changes were pushed down to MLS, they were applied inconsistently. They did not change the main membership table that feeds Membership.csv, but they did affect the VisitingTeaching.csv export file somehow. Meanwhile, further testing shows that this also affects the HomeTeaching.csv file, and my spot-checking shows the same pattern online.

Also, the mysterious sister with a missing phone number does not appear in the online directory at all.

There is a timing difference of six days between the day I took the MLS export snapshots and the day (today) when I checked the online directory, but it is highly unlikely that all these changes were made online this week.

If the objective is to allow members to edit and control their own contact information directly in Church records -- which I understand to be the designers' intent -- I think those edits must be applied consistently and actually edit the member's record in MLS. If that is not the intent, then the changes should not affect the data in the VisitingTeaching.csv file and HomeTeaching.csv files. They certainly should be consistent with Membership.csv.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sun Mar 06, 2011 2:42 pm

The plot thickens. After taking a fresh set of exports at church today and and viewing the records within the MLS user interface (in the Household Contacts dialog) it now appears that it is the Membership.csv file that is wrong.

The MLS screen (like the website) shows the Household Email to be blank, but Membership.csv shows it to be populated. The HomeTeaching.csv and VisitingTeaching.csv files both show the Email to be blank.

I am only assuming that this data may have been edited online.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Tue Mar 08, 2011 5:06 pm

I have replicated a case that triggers this bug in Membership.csv.

I am a household of one. I first cleared both my Household E-mail and my Individual E-mail via the website. After allowing time and doing a batch send/receive, the edits appeared correctly in MLS: Both fields were blank in the user interface. Both fields were also blank in the Membership.csv export. So far so good.

Then online I added a value to only my Individual E-Mail. After enough time and another send/receive, that value was correctly added to the Individual E-mail field in the user interface, and my Household E-mail correctly remained blank. But in Membership.csv, the new value populated both the Individual E-Mail column and (incorrectly) the Household E-Mail column.

This may not be the only case that triggers such an error, and I have not experimented with the phone fields. (Maybe there is even a case that can replicate the behavior directly within MLS without an update from the web service.) But there is a bug, and a scenario that can replicate it.

Meanwhile, I did receive a reply from my bug report indicating that at least it has been forwarded to "the MLS programmers." So hopefully it will get found and fixed.

russellhltn
Community Administrator
Posts: 20734
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Postby russellhltn » Tue Mar 08, 2011 10:05 pm

RossEvans wrote:The MLS screen (like the website) shows the Household Email to be blank, but Membership.csv shows it to be populated.


Interesting. Clearly, despite the fact that it displays as blank, MLS still has that information tucked away somewhere.
Have you searched the Wiki?

Try using a Google search by adding "site:tech.lds.org/wiki" to the search criteria.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Wed Mar 09, 2011 2:38 pm

Further progress. I received this acknowledgement from MLS Support:


This is more or less a bug that occurred in a previous version and appears to be recurring. We are working on addressing it, so it is not in the release notes as it is not intended.

That is good news. But as I sort out the emails, I see this one apparently applies to the "Name Unklnown" bug in Hometeaching.csv. The Membership.csv bug is a separate case.


Return to “MLS Support, Help, and Feedback”

Who is online

Users browsing this forum: No registered users and 1 guest