Test data for MLS export files?

Discussions around using and interfacing with the Church MLS program.
RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Test data for MLS export files?

Postby RossEvans » Fri Sep 04, 2009 8:02 am

In the process of trying to help some users of third-party MLS helper apps, such as Ward Tools, I and they have been frustrated by the lack of standard test data. And when users have undiagnosed problems, they can't share their own files for troubleshooting because the content is confidential.

This is also a hindrance for the developers themselves, who lack access to such data unless their callings happen to grant them access to live data. For example, I think one such third-party application (MLS Companion) was developed by a bishop and made available as a free resource to others. But what will happen when he is released?

I know that fictional data exists in the test version of MLS, which is available to MLS administrators and stake technical specialists. (I read that it may need updating, however.)

Would it be possible to post the export files from the current test versions of MLS -- both ward- and stake-level -- to the wiki to facilitate such testing?

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Fri Sep 04, 2009 8:46 pm

boomerbubba wrote:Would it be possible to post the export files from the current test versions of MLS -- both ward- and stake-level -- to the wiki to facilitate such testing?


I see no reason why this couldn't be done. But there are some issues:

  1. As you well know, there are subtle but important changes between some MLS versions. We could post data from the most current version, but I wouldn't particularly like to go back to all previous versions.
  2. The Preferred Name/Full Name setting in System Options also affects the export files. I suppose we would use the Preferred Name setting, since the difference that happens with the Full Name setting is already documented in the wiki.
  3. The test data contains no HT/VT data nor any organizational data. So some of the export files would be nothing more than the headers, unless someone first created such data before doing the exports.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Fri Sep 04, 2009 9:03 pm

Alan_Brown wrote:I see no reason why this couldn't be done. But there are some issues: ...



Thanks. I can think of one more issue. As I now recall from an earlier thread, you said there was no way in the test version of stake MLS to populate WardOrganization.csv, which by default has no data. Do I understand that right?

BTW, I know there is a project started on the wikito create a body of test data. But I understand the scope of that project is just to support new online applications. Perhaps in the long run, populating MLS test versions could also be included.

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Sat Sep 05, 2009 4:14 am

boomerbubba wrote:Thanks. I can think of one more issue. As I now recall from an earlier thread, you said there was no way in the test version of stake MLS to populate WardOrganization.csv, which by default has no data. Do I understand that right?


That is correct.

boomerbubba wrote: BTW, I know there is a project started on the wikito create a body of test data. But I understand the scope of that project is just to support new online applications. Perhaps in the long run, populating MLS test versions could also be included.


On that project's talk page, I made that very suggestion, but was told that it was not at all in scope for that project. On the MLS Test Data talk page, a discussion of some of the limitations and suggestions for improvement took place a couple of months ago.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Tue Oct 27, 2009 9:34 am

Still thinking about how to build test data somehow.

What if someone installs the test/training version of ward or stake MLS, and populates the fictional ward with enhanced data for HT/VT assignments and callings insofar as that is possible? Are my fears justified that with the next MLS release, none of that work could be preserved because an installation of the new version would start from scratch with just its basic membership data?

russellhltn
Community Administrator
Posts: 20762
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Postby russellhltn » Tue Oct 27, 2009 10:04 am

boomerbubba wrote:Are my fears justified that with the next MLS release, none of that work could be preserved because an installation of the new version would start from scratch with just its basic membership data?


I'd expect that there would be a way to update the data. Such as installing the new version on top of the old one and letting it do the upgrade - and then doing a backup.

My concern is that HT/VT data would be "aging out".
Have you searched the Wiki?
Try using a Google search by adding "site:tech.lds.org/wiki" to the search criteria.

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Tue Oct 27, 2009 10:08 am

boomerbubba wrote:Still thinking about how to build test data somehow.

What if someone installs the test/training version of ward or stake MLS, and populates the fictional ward with enhanced data for HT/VT assignments and callings insofar as that is possible? Are my fears justified that with the next MLS release, none of that work could be preserved because an installation of the new version would start from scratch with just its basic membership data?


I doubt that your fears are justified. MLS always has to be able to upgrade a database from an earlier version. The same process that is used for all the units throughout the Church should work just fine with an installation of the test database.

When I get some spare time (after I get our stake's 2010 calendar done in the next week), I'd be happy to create a more complete set of test data. I've taken a rather haphazard approach in the past. The release of MLS 3.1 is probably a good time to do this. I haven't seen 3.1 yet, but my guess is that the test data doesn't populate the new individual phone & e-mail fields, so that's a new area that would need to be addressed to fill out the data, in addition to the calling and HT/VT holes that already need to be filled.

The HT/VT visit statistics are time-dependent, so that data will not always be extremely current. Also, I'm not sure if I want to embark on doing much with the financial side of things. It would be helpful for some testing, but it has no effect on the exports, and it's quite time-dependent as well.

When I do this, perhaps I could upload the .bak files to the wiki so that others could save some time in similar efforts. Does anyone see any problem with doing that?

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Tue Oct 27, 2009 11:30 am

I'm wondering if we can set up some sort of source control repository where the Community could update the test data (and even write migration scripts if needed) and the Church development team would just grab the latest test db and put it in each MLS release.
Many questions are already answered on the LDSTech wiki. Check it out!

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Tue Oct 27, 2009 12:01 pm

mkmurray wrote:I'm wondering if we can set up some sort of source control repository where the Community could update the test data (and even write migration scripts if needed) and the Church development team would just grab the latest test db and put it in each MLS release.


I think that is a good idea. What I don't know is whether Church IT is willing to expend any significant effort supporting the MLS export files, which are not even documented by their creators.

Another alternative I have been toying with is to base a set of test data on actual unit exports -- from ward and stake MLS -- by anonymizing everything with transformation scripts. Those scripts might break if there were any structural change in the export files between releases, so there is a chicken-and-egg problem. But otherwise I think it might be feasible.

However, I am encouraged by Alan_Brown's comments above. Maybe the best approach would be to use the test DB to originate most of the test data, and script what could not be done for such files as WardOrganization.csv.

It is too bad that the aging problem and other deficiencies are not cured within the MLS test database, because they also affect other purposes such as MLS training. But if they are not cured there, some bandaid fixes could be made in the export files in scripts. For example, dates could be incremented by years, and names could be converted to mixed case.


Return to “MLS Support, Help, and Feedback”

Who is online

Users browsing this forum: Google [Bot] and 1 guest