Send/Receive Errors Today

Discussions around using and interfacing with the Church MLS program.
gpetersen-p40
New Member
Posts: 30
Joined: Mon Oct 22, 2007 8:47 pm
Location: Sacramento, California, USA

Postby gpetersen-p40 » Sun Sep 28, 2008 3:36 pm

Thanks goes to the HQ staff for at least giving us an updated recording acknowledging the problem this week. But I would also like to see emails go out to the STS notifying them of the problems.

crislapi
Senior Member
Posts: 1265
Joined: Mon Jul 07, 2008 3:05 pm
Location: USA

Postby crislapi » Sun Sep 28, 2008 3:46 pm

I just transmitted (4:45 pm on Sunday) without any errors.

If you're waiting, do it quickly!!!

lajackson
Community Moderators
Posts: 8578
Joined: Mon Mar 17, 2008 9:27 pm
Location: US

Postby lajackson » Sun Sep 28, 2008 3:48 pm

gpetersen wrote:Thanks goes to the HQ staff for at least giving us an updated recording acknowledging the problem this week. But I would also like to see emails go out to the STS notifying them of the problems.


Yes. When the calls started coming in this afternoon, the recording was the first thing I checked. The e-mail would not really help me, since I am at Church in the afternoon.

Because of the vast distances we have to travel, even within wards, we instruct our clerks to have someone try and transmit again Sunday evening if they are having a fireside, or to transmit on activity night, either Tuesday or Wednesday.

Our clerks are not expected to make a special trip the meetinghouse on Mondays under any circumstances. They are to be with their families. We allow exceptions only if the meetinghouse is directly on the way to or from work for someone with S/R privileges.

User avatar
childsdj
Community Moderators
Posts: 258
Joined: Wed Feb 07, 2007 9:51 am

Postby childsdj » Sun Sep 28, 2008 4:05 pm

The system is back up and functional. You should be able to transmit data as of right now. No guarantees for the future.

jmorrey-p40
New Member
Posts: 4
Joined: Thu Feb 21, 2008 8:36 pm
Location: Port Orchard, Washington

Postby jmorrey-p40 » Sun Sep 28, 2008 4:11 pm

lajackson wrote:Our clerks are not expected to make a special trip the meetinghouse on Mondays under any circumstances. They are to be with their families. We allow exceptions only if the meetinghouse is directly on the way to or from work for someone with S/R privileges.


I agree with this. I live 20 minutes each way from my meeting house, so it's not a quick trip either. It seems like these reoccurring Sunday availability problems should be a high priority for Salt Lake. Is this an issue of scaling to meet the traffic needs?

crislapi
Senior Member
Posts: 1265
Joined: Mon Jul 07, 2008 3:05 pm
Location: USA

Postby crislapi » Sun Sep 28, 2008 4:46 pm

DJC wrote:The system is back up and functional. You should be able to transmit data as of right now. No guarantees for the future.


I had to restart a couple of the wards' computers before the transmission finally went through (do they get locked into a particular socket?), but finally got all my wards transmitted. We're a student stake, so almost all the computers are in the same building. I do have one lingering worry - half the computers never printed out a financial transmission report after the successful transmission. I'm pretty sure this isn't the end of the world, but that means there is no confirmation or record of the batch/checks being transmitted. I did check under expenses and all the checks said "Sent", so I guess that means it transmitted successfully but I'll always wonder until the September statements come out...

techgy
Community Moderators
Posts: 3183
Joined: Sun Jan 13, 2008 6:48 pm
Location: California

Postby techgy » Sun Sep 28, 2008 8:55 pm

I had a ward clerk call me about an hour ago and tell me that the servers were back up.
Any report on what the problem was and what is being done to aleviate it? This has occured on two sundays in a row and it's making it difficult to maintain members as well as the finance transmissions.

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Mon Sep 29, 2008 10:31 am

I just want to reiterate what we know from DJC and then add my 2 cents...

DJC has noted that the likely cause of the problem (for both weeks) was with the clustered databases on the back-end servers.

My personal opinion is that I'm sure this got ICS's attention when it happened two Sundays in a row now. I imagine there are putting every resource available on keeping it from happening again. Remember, that's 2 weeks in a row that Church employees probably had to travel to the office on the Sabbath to fix the problem for us.

I seriously doubt it is a scaling or any other architectural issue. I don't have any really good reasons, but it just wouldn't make sense. They've been using and monitoring these systems for years, with every Sunday being jam packed with traffic. Think of how many Sundays have gone by without a hitch (granted the actual communication may or may not have been slower, but always functional). I just imagine this being some configurational problem that is more temporal in nature than many of you are suggesting. It's probably just necessary to get to the root of the cause, which may not have been blatantly apparent last Sunday.

MorettiDP
Member
Posts: 302
Joined: Wed Jan 31, 2007 8:15 pm
Location: São Carlos, São Paulo, Brazil
Contact:

Postby MorettiDP » Mon Sep 29, 2008 12:12 pm

We have had these same problem in our stake, for tree wards, before 01:00PM of Brasilia Brazil Standard Time tomorrow.
MorettiDP
Vila Prado Ward Technology Specialist

User avatar
childsdj
Community Moderators
Posts: 258
Joined: Wed Feb 07, 2007 9:51 am

Postby childsdj » Mon Sep 29, 2008 12:34 pm

mkmurray wrote:

My personal opinion is that I'm sure this got ICS's attention when it happened two Sundays in a row now. I imagine there are putting every resource available on keeping it from happening again. Remember, that's 2 weeks in a row that Church employees probably had to travel to the office on the Sabbath to fix the problem for us.

I seriously doubt it is a scaling or any other architectural issue. I don't have any really good reasons, but it just wouldn't make sense. They've been using and monitoring these systems for years, with every Sunday being jam packed with traffic. Think of how many Sundays have gone by without a hitch (granted the actual communication may or may not have been slower, but always functional). I just imagine this being some configurational problem that is more temporal in nature than many of you are suggesting. It's probably just necessary to get to the root of the cause, which may not have been blatantly apparent last Sunday.


You are right on. Architecturally, the Church is in a good place as far as scale to handle the number of transactions that we see. It is also very redundant. While I am not involved with the specifics of the issues right now, I know there were very many people involved in resolving the issue yesterday. I would guarantee that this is a very high priority for the Church. When I receive word on any specifics of why the outage took place two weeks in a row, I will pass them on. We rarely have seen an outage of this magnitude.


Return to “MLS Support, Help, and Feedback”

Who is online

Users browsing this forum: No registered users and 2 guests