Page 1 of 1

Problems with firewalls (YMMV)

Posted: Sun Jul 06, 2014 10:10 am
by aclawson
One of our units took a weird power surge that affected the fire alarm, one port on the cable model and the WAN port on the firewall. A replacement 881W was supplied by FM and the firewall was activated. The image that came down during the process however was oddly corrupted in that tm.lds.org was reporting that it it was either an ASA5505 or a PIX501 and SLC could not get it to respond (it was serving up the internet, but nobody could control or configure it). The device was given a hard reset and a re-activation attempted but the device is already in the database so the imaging script won't run. firewall.lds.org recognizes that this is a re-deploy but the script won't all the device to be reactivated. GSC can't resolve this issue, it has to go to "engineering" who will get to it eventually, meanwhile the building will have no internet service because the new machines no longer have a modem for backup.

I have another building which working - unreliably as the firewall has required four power cycles in the past two weeks because the internet keeps dying - but tm.lds.org shows some issues as well. Again, SLC is unable to remotely access the firewall so it will have to be re-scripted but I need to make sure that "engineering" in SLC will be available so the stake offices, two units and an FHC won't be offline indefinitely because they are too busy to fix it if the issue arises. (And I'm going to have to remap all of the static IP devices when it gets reimaged because there are no known methods of backing up and restoring the scopes).

Are these two firewall glitches just some wild coincidences or are other STSs noticing a recent uptick in firewall appliance glitches?

Re: Problems with firewalls (YMMV)

Posted: Sun Jul 06, 2014 1:09 pm
by lajackson
Our wireless goes out from time to time on an 881W and GSC has to reflash it. Other than that, we have not had any regular problems.

When one of our buildings took a lightning strike a few years ago, everything had to be replaced. Cable modem, 881W, some of the wiring, cards in the administrative computers, and some of the phone lines.

Re: Problems with firewalls (YMMV)

Posted: Sun Jul 06, 2014 6:39 pm
by rolandc
I have two firewalls that don't show in TM, one of them is working normally. the other....not at all.

2nd level problem with TM.lds.org maybe

its been a bad year for lightning here in Fl.

Re: Problems with firewalls (YMMV)

Posted: Sun Jul 06, 2014 6:47 pm
by russellhltn
rolandc wrote:I have two firewalls that don't show in TM, one of them is working normally.
Appears to be working normally. I'd guess there's a script that's not running and that's why it doesn't show.

Re: Problems with firewalls (YMMV)

Posted: Mon Jul 07, 2014 2:58 pm
by aclawson
This has been escalated to "known issue" status - apparently I am not the only person running into an apparent bug with reactivating firewalls. The engineers are working on it. In the meantime, I personally am holding off resetting another problematic firewall out of fear that the same thing will happen again.

Re: Problems with firewalls (YMMV)

Posted: Mon Jul 07, 2014 3:38 pm
by danpass
We have 4 881W's in our stake. We have been having issues with one of them. Same as the second one you describe. Internet access is lost. Power cycling the firewall is the only thing that resolves it. It has been happening with increasing frequency. I concluded from my own troubleshooting that the 881W is failing intermittently. Global support had me do a hard reset and reactivation, which did not resolve the issue. A couple of weeks later, they loaded "new tar files" on the device. I don't know if that process updates the firmware or what. We went several weeks without further problems, but then a week ago Sunday, it looked like we had a hard failure of the device, but after power cycling 4 times it came up. Support finally agreed that it needs to be replaced and said to request a replacement from FM.

Re: Problems with firewalls (YMMV)

Posted: Mon Jul 07, 2014 7:07 pm
by johnshaw
This experience is similar to one I had in a meetinghouse where we positively identified a lightning strike on the device. Let's all make sure these things are protected.

Re: Problems with firewalls (YMMV)

Posted: Tue Jul 08, 2014 9:00 am
by aclawson
In an ideal world each meetinghouse would have good lightning protection (I've lost an organ, various components of sound systems and other electrical thingies in the meetinghouses) that would include whole-building surge protection, and verified good grounding (with all grounds bonded together) but such things are low priority in the budgeting things. In this case the equipment was plugged into protection (real protection, not just an outlet multiplier) but the surge got in somehow (telephone lines, possibly) and appeared to cause trouble through induction rather than a direct surge down the line.

At some point there are going to be major problems - I haven't seen a single building hardened against the known risk of another Carrington Event, with said event being inevitable (though the current risk window is rapidly closing).

Re: Problems with firewalls (YMMV)

Posted: Sun Jul 13, 2014 9:51 am
by aclawson
The engineering people have reportedly fixed the scripting bug that was preventing firewalls from being re-registered (I was apparently not the only one with this issue) so everything should be working now.

Re: Problems with firewalls (YMMV)

Posted: Sun Jul 13, 2014 11:28 am
by rolandc
I received the same phone call about the scripting being fixed for re-registering.