MLS Copy to File totally broken

Discussions around using and interfacing with the Church MLS program.
TechnoBabel-p40
New Member
Posts: 46
Joined: Tue Apr 22, 2008 11:14 am

MLS Copy to File totally broken

Postby TechnoBabel-p40 » Sun Nov 22, 2009 10:24 pm

Guys, this latest MLS that was released this weekend totally screwed up the Copy to File feature. The data isn't in any remotely readable form when I copy it into Excel now.

When looking at the file, there are newlines everywhere (after almost every field) and there is some \tX\tX code in there.

Can we get this fixed quick? My Android application relies on this as well as other automated Home Teaching applications that I use on my Windows 7 machine.

I don't care if you need to change the format, but at least make it a format that is reasonable for me to read with my applications.

Thanks,

TB

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Mon Nov 23, 2009 5:20 am

TechnoBabel wrote:Guys, this latest MLS that was released this weekend totally screwed up the Copy to File feature. The data isn't in any remotely readable form when I copy it into Excel now.

When looking at the file, there are newlines everywhere (after almost every field) and there is some \tX\tX code in there.


It does appear that the "Copy to file" feature (accessible in most reports/lists via Edit > Copy to file or Ctrl+Shift+S) in the MLS 3.1 release is broken. The file that is generated has a CSV extension, but it is a tab-delimited file, with a simple newline (ASCII 10) at the end of each record, rather than the Windows standard CR-LF (ASCII 13 and ASCII 10).

If you bring such a file into Excel, it gets confused, since it trusts the CSV extension and uses the commas that might appear in names as field delimiters.

Clearly this is a bug; even if it was desirable to switch to tab-delimited format, such a file should not have a CSV extension. And given that all the exports are still in CSV format, I'm sure that Copy to File is still supposed to be in CSV format. I have reported this to Clerk Support.

In the meantime, it's not that hard to convert a tab-delimited file to a CSV if you have applications that require this format. Excel can import tab-delimited files just fine, as long as you change the extension to something like ".txt". Then you can open it and save it as a CSV. With this technique, note that Excel will change date formats, so if the receiving application relies on the original date format, you may need to reformat date fields before you save as a CSV.

TechnoBabel-p40
New Member
Posts: 46
Joined: Tue Apr 22, 2008 11:14 am

Postby TechnoBabel-p40 » Mon Nov 23, 2009 6:55 am

Alan_Brown wrote:It does appear that the "Copy to file" feature (accessible in most reports/lists via Edit > Copy to file or Ctrl+Shift+S) in the MLS 3.1 release is broken. The file that is generated has a CSV extension, but it is a tab-delimited file, with a simple newline (ASCII 10) at the end of each record, rather than the Windows standard CR-LF (ASCII 13 and ASCII 10).

If you bring such a file into Excel, it gets confused, since it trusts the CSV extension and uses the commas that might appear in names as field delimiters.

Clearly this is a bug; even if it was desirable to switch to tab-delimited format, such a file should not have a CSV extension. And given that all the exports are still in CSV format, I'm sure that Copy to File is still supposed to be in CSV format. I have reported this to Clerk Support.

In the meantime, it's not that hard to convert a tab-delimited file to a CSV if you have applications that require this format. Excel can import tab-delimited files just fine, as long as you change the extension to something like ".txt". Then you can open it and save it as a CSV. With this technique, note that Excel will change date formats, so if the receiving application relies on the original date format, you may need to reformat date fields before you save as a CSV.


Hey Alan,

The file generated by the "Copy to File" option has always been tab delimited. The problem that I'm seeing is that there is no rhyme or reason to the structure of the data now. You'll find that there are newlines in random places, which makes it impossible to parse through. If you (or anyone else) can come up with a decent way to parse through and identify the data in this file, please let me know, because I really do not see any reasonable pattern to the file now. The tab delimited file that was generated before yesterday's update was just fine.

This issue is causing me major problems right now, so I'm extremely anxious to get a solution or work-around. I'd like to avoid making changes to all my applications so that they support this screwed-up format, only to have the format change on me again in the future.

If there is an alternate way for non-administrators to export data for excel in the latest MLS, I'd love to hear it. An XML export would be just fine also.

TB

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Mon Nov 23, 2009 9:31 am

TechnoBabel wrote:The file generated by the "Copy to File" option has always been tab delimited. The problem that I'm seeing is that there is no rhyme or reason to the structure of the data now. You'll find that there are newlines in random places, which makes it impossible to parse through.


Obviously I don't use that feature much, or I would have known that MLS saves such files in a completely non-standard way. A tab-delimited file should never be saved with a .csv file extension.

However, at least on the test data, I see no difference between MLS 2.9.4, MLS 3.0.3, and the released MLS 3.1; the result for all three versions of doing Copy to File is a file with a tab between fields and a single newline at the end of each row. I did my testing on the Abbreviated Directory and Telephone Directory. Perhaps it matters what report you are trying to copy to a file. If so, please let me know which one you are using so that I can try to duplicate it.

Personally, I always copy to clipboard and then paste into a spreadsheet. That's something you could still do; then if you really want CSV format or tab-delimited format, you can use your spreadsheet application to do a Save As and save it in that format. I would think that should work, regardless of the problems you are seeing with Copy to File.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Mon Nov 23, 2009 9:54 am

Alan_Brown wrote:Obviously I don't use that feature much, or I would have known that MLS saves such files in a completely non-standard way. A tab-delimited file should never be saved with a .csv file extension.

However, at least on the test data, I see no difference between MLS 2.9.4, MLS 3.0.3, and the released MLS 3.1; the result for all three versions of doing Copy to File is a file with a tab between fields and a single newline at the end of each row. I did my testing on the Abbreviated Directory and Telephone Directory. Perhaps it matters what report you are trying to copy to a file. If so, please let me know which one you are using so that I can try to duplicate it.


I did replicate the problem TechnoBabel describes last night, but I am not at the MLS computer right now. As I recall, the problem occured in one or more of the reports that are in the Bishop's Action and Interview List. As I recall, there was a column called "Phone and Email," or something similar, that had a newline at the end instead of a tab. So I suspect this problem may be related to the changes made for the new phone and email data elements.

As for whether tab-delimited files should ever have a csv extension, I happen to agree with you. It certainly causes problems in Excel's default behaviour, for example.

However, MLS is not the only offender there. I encounter other such examples in other systems. Although "CSV" originally meant "comma separated values," maddeningly there is no standard, and the term has become bastardized to mean just about any kind of delimited file. Several parsers for "CSV" include options for changing the delimiter to be tab, pipe, or something else, as well as other extended options. For example, some idiosyncratic applications (including MLS exports) also include a trailing delimiter at the end of each line, in addition to the delimiters that separate columns. Likewise, there is no standard for when to include quote marks.

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Mon Nov 23, 2009 10:18 am

boomerbubba wrote:As I recall, the problem occured in one or more of the reports that are in the Bishop's Action and Interview List. As I recall, there was a column called "Phone and Email," or something similar, that had a newline at the end instead of a tab. So I suspect this problem may be related to the changes made for the new phone and email data elements.


Thanks for the lead; I can now duplicate the problem. Indeed, it appears that any report that has a combined phone/e-mail column now puts in extra newlines when you do Copy to File. It adds one newline after the field, no matter what, and if a person has both phone and e-mail, then it adds an additional newline within the field.

This is obviously not the correct behavior for a tab-delimited file; a newline should only be at the end of a record. Short of writing some sort of script or application to clean up such a file, I don't know how a user would clean up such a file to be properly formatted.

TechnoBabel-p40
New Member
Posts: 46
Joined: Tue Apr 22, 2008 11:14 am

Postby TechnoBabel-p40 » Mon Nov 23, 2009 10:53 pm

Could someone give me a feel for how long it takes for them to fix major bugs like this? Am I going to have to wait weeks? Months? Years? It would just help me if some level of expectations are set so that I can seek an alternate solution if necessary. I prefer not to waste my time on this if it is going to be fixed in a few days, but I will if I have to.

TB

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Mon Nov 23, 2009 11:05 pm

TechnoBabel wrote:Could someone give me a feel for how long it takes for them to fix major bugs like this?

No offense intended, but I'm not sure I would consider this a "major bug." Don't get me wrong, it should definitely be fixed as soon as it can. But Alan_Brown did mention a usable workaround that you could be using:
Alan_Brown wrote:Personally, I always copy to clipboard and then paste into a spreadsheet. That's something you could still do; then if you really want CSV format or tab-delimited format, you can use your spreadsheet application to do a Save As and save it in that format. I would think that should work, regardless of the problems you are seeing with Copy to File.
Many questions are already answered on the LDSTech wiki. Check it out!

lajackson
Community Moderators
Posts: 6139
Joined: Mon Mar 17, 2008 9:27 pm
Location: US

Postby lajackson » Mon Nov 23, 2009 11:05 pm

TechnoBabel wrote:Could someone give me a feel for how long it takes for them to fix major bugs like this? . . . It would just help me if some level of expectations are set . . .


I would be surprised (but I could very well be wrong) if it were to be fixed in a few days.

My father used to say, "Blessed is he who expecteth nothing, for he shall not be disappointed." I never really liked that philosophy, but it does come in handy in situations such as this one.

I would guess (total speculation here) that, if it is fixed, it might be fixed in the next release, which I expect would be 3.1.1 and which I would not expect before the end of the year. Then again, there might be other things that must be fixed before year-end, so something like this bug might get worked in. Otherwise, I would not expect a fix until sometime next year.

Again, I cannot and do not speak for the programmers. The most I think I would do would be to make sure they are aware of the bug. I suspect they are by now.

User avatar
aebrown
Community Administrator
Posts: 14693
Joined: Tue Nov 27, 2007 8:48 pm
Location: Sandy, Utah

Postby aebrown » Mon Nov 23, 2009 11:17 pm

mkmurray wrote:No offense intended, but I'm not sure I would consider this a "major bug." Don't get me wrong, it should definitely be fixed as soon as it can. But Alan_Brown did mention a usable workaround that you could be using:


Actually, now that Boomerbubba has pointed me to the problem scenario (the new combination phone/email fields in MLS 3.1), I must report that the workaround of using Copy to Clipboard doesn't help at all in those cases. For any report or list containing those fields, I don't know of a reasonable workaround,


Return to “MLS Support, Help, and Feedback”

Who is online

Users browsing this forum: No registered users and 1 guest