MLS Copy to File totally broken

[aebrown]

The most I think I would do would be to make sure they are aware of the bug. I suspect they are by now.

I have reported the bug with quite a bit of detail, so they are now aware of it. I can't imagine it being fixed before Christmas at this point.

[TechnoBabel-p40]

It is a major bug to me and the thousands of people who use the Windows software I have written. This issue causes a major technical support headache for me. While I appreciate Alan's advice, it is not so simple for me to explain it to all the people who are using my apps. They just get frustrated and think my apps are defective. Anyhow, I am really in a bad, stressed position right now because of this and I hope that whoever injected this bug can appreciate that.

TB

[aebrown]

It is a major bug to me and the thousands of people who use the Windows software I have written. This issue causes a major technical support headache for me. While I appreciate Alan's advice, it is not so simple for me to explain it to all the people who are using my apps. They just get frustrated and think my apps are defective. Anyhow, I am really in a bad, stressed position right now because of this and I hope that whoever injected this bug can appreciate that.

I don't know what your applications are assuming, but it seems like the new fields would require at least some modification to your application anyway, even if the tab-delimited format had not been broken.

If you have written Windows applications, then I would assume that you are in control of the parsing of the input files. It would not be all that difficult to enhance the parsing of fields such as this. The column headers show just how many fields there are. As you are processing a field, if you encounter a newline before a tab, convert it to a space (or simply strip it if it precedes a tab). It's a bit of a pain, but I think you could design an enhanced parsing algorithm that could reliably handle MLS 3.1's current output format for Copy to File.

[TechnoBabel-p40]

I don't know what your applications are assuming, but it seems like the new fields would require at least some modification to your application anyway, even if the tab-delimited format had not been broken.

If you have written Windows applications, then I would assume that you are in control of the parsing of the input files. It would not be all that difficult to enhance the parsing of fields such as this. The column headers show just how many fields there are. As you are processing a field, if you encounter a newline before a tab, convert it to a space (or simply strip it if it precedes a tab). It's a bit of a pain, but I think you could design an enhanced parsing algorithm that could reliably handle MLS 3.1's current output format for Copy to File.

Thanks for the encouragement, Alan. I will see if there is a reliable way to do that. My fear is making the change only to have it break on me once the problem is fixed. Perhaps if I detected the funky file-type and handled it separately, that might be a low-risk way to do it. This file format is terrible for importing in the first place. The only reason I use it is that it is the only way to do it without super user access in MLS.

By the way, I'm not trying to be secretive about the project I'm working on. I just prefer not to discuss it or advertise it here. You can read all about it elsewhere.

TB

[russellhltn]

If you have written Windows applications, then I would assume that you are in control of the parsing of the input files. It would not be all that difficult to enhance the parsing of fields such as this. The column headers show just how many fields there are. As you are processing a field, if you encounter a newline before a tab, convert it to a space (or simply strip it if it precedes a tab). It's a bit of a pain, but I think you could design an enhanced parsing algorithm that could reliably handle MLS 3.1's current output format for Copy to File.

It kinda sounds like the new line is the separtator for a multi-value field. While the delimiters may change, I suspect that mult-value is here to stay.

[aebrown]

It kinda sounds like the new line is the separtator for a multi-value field. While the delimiters may change, I suspect that mult-value is here to stay.

That may be, but it's not a very good practice to put the record delimiter inside a field, with no quoting to prevent interpretation of that character as a record delimiter. Admittedly there is no official standard for delimited files, but this seems to stretch any reasonable interpretation of the de facto standards too far.

On the other hand, there is already a precedent for multi-value fields in MLS. It is in the Palm CSV export files. They contain a notes field with multiple lines. But in my opinion, this was done reasonably:

1. It is an actual comma-separated values file with a .csv extension
2. All fields are surrounded by double quotes, and separated by commas
3. All records end with a CR-LF combination, as is standard on Windows
4. The internal separators within the Notes field are simple newlines
5. Such a file can be imported directly into Excel with no preprocessing required

So the MLS developers could redeem themselves from the poor choice of saving a tab-delimited file with a .csv extension by making the output of Copy to File consistent with the Palm export files.

[RossEvans]

So the MLS developers could redeem themselves from the poor choice of saving a tab-delimited file with a .csv extension by making the output of Copy to File consistent with the Palm export files.

Those Palm export files are devilish to parse. I have used them, but only as a last resort. They display decently inside Excel, and I guess in some legacy Palm Pilot app, but are not readily parsed as CSV files without some nasty custom processing.

The offending file format here seems to be an unfortunate side effect of embedding a hard line break in the phone/email data for purposes of the interactive display. The designer did not want to use the horizontal real estate, so this was the compromise.

It does ruin the file for decent use by a consuming program. It would be much better in the saved file, where real estate is not a problem, just to output two separate fields for processing. If the user just wants to save a formatted output file, that is easily done with a print-to-PDF action.

Overall, the prevailing design philosophy for MLS seems to disrespect the common-sense needs of programs that read exported files, which by design are supposed to be read primarily by software processes rather than just human eyeballs. I have serious grievances with the core "Export" CSV files, which in many ways are just human-readable reports misadvertised as true data exports. The design of the HT/VT Export files would flunk an exam in Database Programming 101. And they just got worse in that regard with a new layer of needless complexity introduced in MLS 3.1.0. That harm was easily avoidable.

[TechnoBabel-p40]

I have fixed the problem in my applications. PLEASE do not make me go through that again... it was not pleasant.

string[] lines starts out as being the screwy format exported with the "Copy To File" option in MLS. At the end, it is in the format as it was before the latest MLS update.

Can anyone say "NIGHTMARE?"

TB

string stream = sReader.ReadToEnd();

string[] lines = stream.Split(newchar[] { '\n' });
string[] secondLine = lines[1].Split(newchar[] { '\t' });
List<string> formattedLines = newList<string>();

// check if the file received is in the funky screwed up format 11/25/09
if (secondLine.Length == 2)
{
string[] unformattedLines = stream.Split('\n');

StringBuilder sb = newStringBuilder();

foreach (string unformattedLine in unformattedLines)
{
string[] fields = unformattedLine.Split('\t');

// we can identify the fields we are working with by the number of tabs present in that field
switch (fields.Length)
{
case 2:
// companionship or not assigned first part of a line
sb.Append(string.Format("{0}\t{1}\t", fields[0], fields[1]));
break;

case 3:
// companionship alternate first part of line
sb.Append(string.Format("{0}\t{1}\t", fields[1], fields[2]));
break;

case 4:
// companionship second part of a line
sb.Append(string.Format("{0}\t{1}\t{2}\t{3}\t", fields[0], fields[1], fields[2], fields[3]));
break;

case 5:
// companionship end of line
sb.Append(string.Format("{0}\t{1}\t{2}\t{3}", fields[1], fields[2], fields[3], fields[4]));
formattedLines.Add(sb.ToString());
sb = newStringBuilder();
break;

case 7:
// not assigned end of line
sb.Append(string.Format("{0}\t{1}\t{2}\t{3}\t{4}\t{5}", fields[1], fields[2], fields[3], fields[4], fields[5], fields[6]));
formattedLines.Add(sb.ToString());
sb = newStringBuilder();
break;

case 8:
// companionship header
if (fields[0] == "Household" || fields[0] == "Sister Being Taught")
{
formattedLines.Add(unformattedLine);
continue;
}

// not assigned header
if (fields[0] == "Home Teacher" || fields[0] == "Visiting Teacher")
{
formattedLines.Add(unformattedLine);
continue;
}

break;
}
}

lines = formattedLines.ToArray();