Why so many duplicate ordinances performed?

[RonaldF-p40]

Pack1970 mentioned that the church did the checking for duplication prior to 1991. He did not mention that when names were submitted, they were ALL sent to Salt Lake and it took many months before the submitter heard bach that their names had been cleared for ordinances.

In 1991, TempleReady was introduced. Although the church knew that there were inherent weaknesses in the system, the hope was to get names cleared more efficiently. Here is the problem: At that time the IGI on CDs was updated once every four years, which meant that if you checked for duplicates, the data being checked against could be as much as 4 years old. You can see that a name could be easily submitted many times during that four years, and TempleReady would not catch it, because it wasn't on the CD yet. A second issue was that if a member was so anxious to have a name to take to the temple, and TempleReady identified a possible match, even if the data was identical, the member could click on the option "Not a Match" and do the temple work again. By thetime new CDs were released every four years, the data actually on the CD was already nearly a year old. I don't have the information at hand, but I believe that the last time CDs were released was in the year 2000 containing data up through 1999. Could someone with access to an FHC confirm or correct that statement?

That is correct. 2000 was the last update to TempleReady. The on line IGI is the current line of defense.

[scion-p40]

Under the current system, the IGI "corrects" places to the current place (as of the submission, from my observation) and removes the historically accurate data already provided. Not only does the submitter search to find this immediately after known work is done, it causes a problem in the future, assuming that political boundaries will change again from time to time. With this practice, future researchers will need to research more than the political boundaries where their family lived in a particular place and time, they'll need to research where that place falls in the 2000s and all of the years up to their current search.

[JamesAnderson]

I think that could cause a problem when people go to check their work on even the present online IGI, especially if they are now looking for work they left off at the temple to do.

Got some names at Jordan River in July that were from what we today would call 'Germany' but in fact were in political subdivisions that were not considered part of Germany until a couple hundred years later. Other researchers will not find that.

Maybe one idea is to add a line to the XML for the individual entry both on the pedigree and individual and other records indicating the place is in modern-day (name country) now rather than 'correct' the place names. That will help both camps of researchers.

That could be added into the existing code for the entry display formats for both versions of familysearch.org that are out there. Then the name submission checking software could recognize both names, and believe me, that could help out on the duplication front exactly.

An example of what I mean. This is by no means historically accurate, but I am using this to illustrate how somehting like this would work.

John DOE
4 June 1825
Beckley, {county name} Virginia

Location notes
County name was {old county name) from xxxx to yyyy
State name was Virginia until 1863 when West Virginia was formed.

All text other than names and dates in the above two lines would read exactly, the software would simply pull the matches from a table, then insert them in the page view the user sees. Any locality field could be set to display as follows:

IF locality name changed from X to Y before DATE in year format, show this line of text and fill in needed info.

I'm not sure if I'm clear enough to any programmers, but you should at least get the idea I'm looking at.

[scion-p40]

Given the IGI propensity to "correct" places, I like your idea of somehow keeping the original submission viewable in addition to the "corrected" data. Both would also need to be searchable to be effective.

[thedqs]

You can continue to try to get correct info but when do you know you have the best. It was probably submitted before those records were available and the only document was the immigration record. Understandably then they get the name wrong but how do you know when you have "verified" your data?

[JamesAnderson]

Your example of someone just finding only the one item of evidence, the immigration record, also indicates that alot of times the knowledge about how to find the naturalization records and intent to naturalize papers elsewhere may not have been known.

That does bring up another matter, what about the planned knowledge base and other helps that were supposed to be tested? Someone had mentioned looking at various forum software for that on this site, and also there was some talk in the genealogical community at large that mentioned there would be all kinds of informational helps on almost anything out there, but no details were given as to when they would finally begin the beta testing of that, only to say they had found some software they wanted to integrate into that.

Once they can get over those issues, and test it, that might go a long way to helping people finding 'lost' records containing the right evidence, even if it is not really 'lost'.

[blackrg]

Short of direct revelation, I don't think there is ever a point where you're "positive" your data is actually "correct". There is no source in the genealogical world that is completely infallible. Now with some careful research, you can be fairly certain. But even after that is done, your mind should always be open to the possibility that you may one day find out your info is wrong.

In one case I can think of from my own family history, I had a (several greats) grandmother that was wrong. The info I had had been done by my father many years earlier (no internet, no computers, etc.) and included some on site research. The first tip I received was someone who posted data on the internet that showed different parents for her. This person was unfortunately unresponsive when I asked for sources. Despite the possibility that this was just more poorly researched data though, I decided to pursue it. What I found is I had to women with the same name in the same town with only a few years between their births. At that point it became impossible in several cases to tell who's information I was looking at when I found a record of one type or another. The answer to my delima thankfully did come one day and came quite by accident (or maybe not...). I was doing some on site research in the local genealogical society for that area and a guy a few rows down suddenly popped his head up and said "Hey, you're doing research for Black's right?", I responded yes and he proceeded to show me an obit for my (several greats) grandfather this woman (whichever one was the right one) had married. The obit showed me that she was still alive when he died. By chance, the duplicate person had died before his death date. Now I could spot the correct death date, and from her age at the time of death find the correct birth date, etc. and begin to put the puzzle together. Though we do have some nice software to help us, and I only expect it to get better, at the end of the day this will probably still be the sort of problem that requires a human (and maybe some divine intervention) to sort out.

Another instance in my family history that can really throw things off are that some researchers have placed the birth of some of my ancestors at Connellsville, PA. While this may be the correct physical location, it didn't exist at the time of their birth. Best I can tell, the correct designation then is probably a small township to the south called Bullskin - but even with a designation of Bullskin, it's possible that Connellsville may be more physically accurate (I haven't pulled the land records to do some guessing about where in the actual area they may have been when birth occurred. Connellsville was founded by the grandfather of the infants in question a few years after their birth, and hence there's some reason to believe it might be more geographically correct.) A computer with little info to go on could easily miss this match.

There are also name spellings. I have one line today where the living descendants who are closely related have about 3 different spellings in play. Then there is the americanization of names as was mentioned earlier - should I use John Roush or Johannes Rausch? I don't think any of the Church's current software is going to catch that match.

Sometimes there just isn't enough info for a good match either. If you've got a full name, a birth date, the birth location, and some other event to let you know the person lived beyond the age of 8, and someone else has a full name a marriage date, and a marriage location that's in a different state, you're going to be hard pressed to say it's the same person unless one of the two of you can find something to make the connection - yet you've got a perfect setup to duplicate ordinances.

In the end, I personally feel the best way to avoid dupes (currently - and this being said without having yet seen the new website) is to simply check for dupes by hand. The computer inside my head is going to catch far more than the church website or any program running at the local family history center.

[huffkw]

I don't understand technology that well....but, it seems to me that avoiding duplications should be a simple database, computer issue.

You are right, but no one seems to know it yet. Here is a small part of my proposal to kill the duplication beast once and for all.

(I would prefer to send this lengthy response as an email to the person who started this thread and raised the question of why so many duplicates, but email does not seem to be an option, so I will try it here.)

The topic of duplicate names has been a very frustrating one for me, as apparently it has been for you. The fact that a simple, complete, efficient, straightforward solution to the problem is possible makes it even more frustrating.

You have inspired me to write A PAINFULLY DETAILED VERSION of the basic concept needed to solve the problem, in hopes that someone will actually read it and grasp my point.

I have been trying for years to introduce a new computer concept to put a complete end to this problem of duplication, but without any success. I cannot say that a single person, programmer or layman, has grasped the simple concept I offer. Maybe you will be the first. If I had the money, I would have done it all myself, so that I would not need to explain it in advance to anyone; I could just SHOW them how it works. But I don’t personally have the $1 to $2 million it would take to do it right.

Here is this brain-bending stumper. Turn up your thinking cap to full power or you will miss it. Suppose that we had a computer system that contained all past and current completed genealogy research online, in a format so that you could almost instantly determine what research work had already been done that relates to you, BEFORE YOU DID ANY RESEARCH OF YOUR OWN. This could be done on the first day a person felt an interest in doing genealogy research and temple work. It might take that person an hour or two to learn about everything of importance that had been done before that was related to that person, so that person need not repeat ANY RESEARCH in those areas. If you wished, you could also spend a few moments longer and examine some of the areas that had NOT been researched, so that you could choose a good place to add something new which might be of value to yourself and others.

Now, let me make it clear that I am not talking only about temple work that has been done. The research in this database would be ALL the research, regardless of whether it had been submitted for temple work or not. This would allow people outside of the Church to use the system just as well as any member, so that members need not duplicate the work of other US genealogists.

One more time: The names in the temple can only get there if people do research work and submit any names found. BUT, if you can tell, at step 1, what research has already been done, YOU NEED NEVER GET TO THE POINT OF HAVING TO CHECK TO SEE IF ANY NEW NAMES PRODUCED BY YOUR RESEARCH HAVE ALREADY HAD TEMPLE WORK DONE FOR THEM. You just avoid the whole question, and you can do it because you can check centrally all completed genealogy RESEARCH, not merely all completed TEMPLE WORK. There would be no need for a Temple Ready system, or anything like it, because no one would be doing any duplicate research which needed to be checked for duplicate names.

Now to try some contrast: In today’s (old) system concept, which goes back at least 60 years, long before computers or the Internet were even thought of, we imagine, and remain convinced, that the only thing we can possibly do to stop duplicates from going to the temple and then into the Church ordinance records, is to expect people to do all their research, taking however many years it may take, and only then, with all that work done and names identified, do we try to see if those names have been processed before. That is a terribly wasteful use of member research effort, with 95% of the effort wasted. That was indeed all one could expect to be able to do using a manual check in a manual system, 60 years ago, or even 20 years ago. There never was the slightest chance, back then, that the many researchers, working independently, outside of a big, well-managed genealogy research firm, could coordinate their efforts to avoid duplicate research effort in identifying their ancestors. But today, such full, advance coordination of work could be a very simple matter. Perhaps all that is required is that the mere possibility of it be recognized by those with the power to implement it.

===================================================
Old Versus New System Sequence -- diagram
We need to reverse the sequence and change the method of checking for duplicate work

Old System sequence:
Phase 1 -- Research -- may last many years.
--only minimal and unpredictable coordination of effort is possible during research.
--95% of the effort is normally wasted in duplicate research.
Phase 2 – Name check phase
--lasts a few hours or days.
--95% of work is typically found to be duplication, causing disappointment of researchers. After doing all that otherwise wasted work, they tend to send the names in anyway, mostly defeating the purpose of the Temple Ready and similar systems.
Phases:
|--------------------------------------------1----------------------------------------------------|-2-|


New System sequence:
Phase 1 -- Check status of all research related to you.
--Spend a few hours on this task. A non-duplicating research plan can be formulated.
Phase 2 -- Research -- may last many years, but all of it is productive.
--Do only research that has not been done before. 98% to 99% will be new data.
--There is no name check at the end. There is no need to check for duplicate names submitted for temple work, because very few, if any, will slip through or arrive at that point. A last check might be done just to make sure the system is working correctly, but the duplication rate should be in the range of an insignificant 1% to 2%.
Phases:
|-1-|--------------------------------------------------- 2 -------------------------------------------->

===================================================

Now I have to assume that you still do not get it, because it seems every Church member’s brain is so conditioned by 150 years of manual, paper-based system thinking that those brains simply cannot form the new concept, regardless of the number of times the concept is repeated in words. But maybe you can break out of that mold and grasp it. If so, I do hope you will let me know how you cracked the code, and that you will pass your insight along to others.

Why can no one grasp this simple new idea? I don’t know, but I have to assume that there are very few who have any real idea of what the Internet makes possible. And of those few people who do understand the mathematical and cooperative power of the Internet, there are few or none who have made any deep study of genealogy processes and possibilities. In any event, so far the concept has never popped up where there might be a person who has the power to act on it.

I have to assume for most people there is an instant wall of complete denial that any such “magical” database as I have described could possibly exist in the real world. Surely it could be no more that a foolish flight of fancy, they might say, since nothing like that now exists.

But let me begin to explain what this new database might look like. In one good beginning example, it might consist of a central collection of all the current contents of everyone’s PAF files on their home computers. (That is where all the best data is kept. That data is usually degraded as it goes into current public systems. This needs to be fixed so that data going into central systems goes UP in quality and usability.) With all serious PAF files online, everyone could see what everyone else had done and was working on.

But wouldn’t there still be massive duplication and confusion in those combined PAF files, even more than in the temple ordinance files? Isn’t the Pedigree Resource File exactly that -- a copy of everyone’s PAF file? Perhaps several copies of PAF files were submitted to PRF at different stages of development? Yes, to all those questions. That is what makes using the PRF a confusing task.

So here is where a tiny amount of new logic can make a huge difference. There is no need to put up with this confusing duplication, and no need to try to “merge” the duplication out of existence. Some simple rules for entering data, and for showing whether data is complete enough, possibly including source references, is all that is needed to stop essentially all duplication. I won’t try to get more detailed than that here, but it is not very tricky, mostly just requiring a change in attitude and sequence. But, of course, changing attitudes is itself a very big challenge.

In the past, the standard “solution” for duplication, going back 30 to 40 years to the Four Generation project and the resulting Ancestral File, was to try to take that mountain of heavily-duplicated data and mash it all together. People have invented innumerable machine algorithms to try to do this massive merge automatically. The results are usually pretty bad. This can become the classic Garbage In/Garbage Out situation, including a new Garbage-Squared multiplier. The New Family Search system used a machine algorithm to more carefully merge a portion of the data, but then has assigned Church members to do the many tens of millions of hours of manual merging it will take to finish the job of reducing the 1.5 billion names down to perhaps 100 million unique names.

But when all that huge amount of NFS merging work is done, I believe there will still be no efficient way to prevent discouraging and wasteful duplicate RESEARCH effort done before ever getting to the NFS duplicate check stage. There will still only be some restraint on duplicate names, produced by unnecessary duplicate research, passing into the temple name processing channels. (The NFS system might conceivably be modified to let people put in all their new research-in-progress, far ahead of, and on top of, and eventually many times larger than, the temple work actually completed. But this is mixing apples and oranges, and would likely cause a whole host of problems, for programmers and for users. It would be better to keep the different fruits in two different baskets – two databases.)

In my view, it is hard to say whether the New Family Search system will be better or worse than today’s Temple Ready system in stopping duplicate names. If it uses only the merged names, the “Summary View,” it will logically catch less duplicates than the current Temple Ready system. If it uses ALL names, merged or not, it will hardly be any different from today’s TR system.

To summarize, I argue that the concept underlying the Temple Ready process, and its replacement, the New Family Search system, is now technologically obsolete. The basic concept is at least 50 years old. I consider the methods used to be about 20 years out of date because they have not fully embraced the new opportunities made possible by the Internet. (This is true even though the NFS uses the Internet intensively.)

Consequences
I predict that, with the NFS system in place, the number of duplicate entries that get into the temple ordinance files each year will likely be higher than ever before. Our directly attacking the SYMPTOM even harder (of duplicate names being sent to the temples), rather than recognizing the real PROBLEM -- the unnecessarily duplicated research effort -- will actually make things worse.

There will likely be less really new research (and thus fewer new, unduplicated temple names), and even more reliance on existing names in Church systems. (Nearly all names in Church ordinance files are only there because their temple work has been done. So they would automatically be duplicates if they were reintroduced into Church temple work systems).

The reduction in new research, caused by the large new NFS member workload, means that even more duplicates will be resubmitted than in the past. The temples need new names, and duplicates are the easiest to find, as any local genealogy librarian can attest. This unintended consequence of more duplicates will probably be deplored, but also mostly ignored, because everyone assumes there can be no better solution.

As you saw in an earlier post on this thread, when things began 150 years ago, it was solely the member’s duty to avoid duplicates. Apparently, that eventually proved to be ineffective as the volume of names grew, so it was accepted as a corporate responsibility to try to minimize the flow of duplicate names into the temples and the temple ordinance files. I believe the Temple Index Bureau was the name of that original central processing group. Apparently, that TIB method also proved to be ineffective, presumably because the number of members, temples, and submitted names mushroomed further.

So then the responsibility was put back on the members. The Temple Ready system automated the prior centralized manual process and was offered as a helpful tool. But in my opinion, that tool was not sufficient. Today’s 10 or 15 times average duplication rate in ordinance files shows the problem was only partially solved. The New Family Search system is an attempt to improve upon the TR system, but I believe it still falls short of what is needed. At least conceptually, it puts the burden on members to correct 150 years of duplication of name submissions before they can move on to new areas of research. And even then, the NFS system will probably do little to curtail the continued duplication of the research itself.

Apparently, the old ways of thinking are hard to shake, even as the Internet offers new options. A better central solution holds out the possibility of a 20-times increase in productivity and accuracy. Until that opportunity is grasped, I expect the behavior of genealogists will remain unchanged, and the duplicates will keep rolling in.

[MarianJohnson]

I have additional insight into your suggestion. My experience has been that LDS genealogists fall into 2 camps - those who do research into records, and those who look for research someone else has alreaady done to copy and submit to the temple, i.e. books, internet submissions etc. Those in the second camp have no idea if the information they submit is correct, so any effort they make to combine names in new FamilySearch is pure guesswork on their part.

The answer is to start a fresh new database in which those in the first camp can enter their information, complete with copies of their uploaded sources, so that the whole world can see where they got their information. There would still need to be room for more than one version of dates and places, since even original documents are not infallible. That way, when a new individual decides to do genealogy, he can SEE why the previous submitter entered the information as he did.

The huge expense of doing genealogy is in obtaining those documents. Once the document was uploaded, no one would have to spend money to order that document again. In the case of microfilmed records from the Family History Library, a link to that record would suffice (once those records become available). But many records, especially the more recent vital records, are not in the FHL's collection. The expense of obtaining vital records is what slows down the REAL genealogical researcher. I would like to see the church focus their indexing on what vital records they have been able to microfilm, rather than reindexing the census records which are now available from other sources.

The second source category which is difficult to obtain is records that are privately held, like family Bibles. If those who own family Bibles and other private documents had a place where they could upload copies of the "family pages" from their Bibles, the rest of the world would be able to use this information in its reaserch.

In summary, I think the answer to your suggestion lies in the concept that a new database be started, in which only those who have documents to prove their data would be allowed to enter what is known to be true, along with copies of the documents as sources. By sharing the documents, this would decrease the expense of everyone re-researching the same people repeatedly, and reduce the chance of errors significantly. The temple work, as deduced from the combined records in new FamilySearch could then be added to this correct database, to make a record "worthy of all acceptation." The challenge to the members would then be to share their documents in the new database, and locate new documents containing new information to add to the database.

[huffkw]

The answer is to start a fresh new database in which those in the first camp can enter their information, complete with copies of their uploaded sources, so that the whole world can see where they got their information. There would still need to be room for more than one version of dates and places, since even original documents are not infallible. That way, when a new individual decides to do genealogy, he can SEE why the previous submitter entered the information as he did.

I think you are exactly right. I would add one little extra feature for flexibility which would let people put in whatever they want, but the public would only see the best, sourced data. That way, the data suppliers could do their homework online, assembling and correcting all the data, and when it is good enough for public viewing, it is made visible.