BUG: AP missing from TM

[russellhltn]

I've got two buildings in my stake that's missing a AP in TM. I'm not talking about the historically problematic AP that's part of the firewall, I'm talking a 1041. When I contacted GSD, they could see the right number of APs, but on closer examination, one AP was repeated twice - same IP, same ID. The AP itself is functioning normally. I can connect to it and use it to connect to the web. The AP does appear in the list of "Connected devices" on the main page for the firewall.

The effects of this bug:

• One AP is missing from the list for the building.
• Attempting to disable the WiFi for the duplicated AP may result in shutting down the missing AP instead of the indicated AP.
• Likewise, "Disable All" fails to shut down all of them.
• In retrieving the list of devices connected to a AP, the devices connected to the missing AP may appear under other APs.

I'm not sure when they bug appeared. I'm sure it wasn't there when TM 1.0 came out. I only noticed it now while trying to label the APs using the new feature found in TM 2.0. I don't think it's TM itself, but something upstream of it.

Running scripts, complete power cycle and even hard resets of the affected APs did nothing. The ticket has been kicked upstairs. At this point, I think the bug is in the management software. Networking should have an Monday morning.

[tlhackett]

We need more specifics. Which firewall are you having issues with? When you mentioned that the GSC saw the right amount of APs but noticed that one was repeated, does that mean that they didn't see the missing AP either? If the firewall can't "see" the AP, TM will never know about it and you won't be able to enable/disable the Wifi on it.

If the GSC couldn't see the missing AP on the firewall, that would indicate some kind of infrastructure issue. The AP may still be able to communicate with a controller and be able to function correctly without the firewall being able to "see" it. In most cases this is caused by a managed switch being used on the network. Managed switches, even if being used as a dumb switch, prevent the firewall from "seeing" the AP.

[russellhltn]

does that mean that they didn't see the missing AP either?

Correct.

If the GSC couldn't see the missing AP on the firewall, that would indicate some kind of infrastructure issue.

Probably. The person I talked to indicated that their software was written in-house. So it could be a bug with the custom software.

Between the fact that in both cases one AP is showing up twice and the fact that the missing one can be shut down, I think some software package isn't parsing the IDs correctly. In fact, I double checked:

Building A:
AP10f3.119a.278c - Doubled
AP10f3.119a.2780 - Missing

Building B:
AP10f3.119a.259c - Doubled
AP10f3.119a.254f - Missing

Now, Building A has a number of APs that are identical except for the last two, and they seem to be fine. This could be just coincidence.

The AP may still be able to communicate with a controller and be able to function correctly without the firewall being able to "see" it. In most cases this is caused by a managed switch being used on the network. Managed switches, even if being used as a dumb switch, prevent the firewall from "seeing" the AP.

No managed switches. Just regular ones. I don't remember the model. It's a 16 port with POE on the first 8 ports. FM ordered it from their catalog when we set up the system.

[tlhackett]

I took a look at your buildings. Both buildings have the same issue below. The AP information is specific to the KAHALA, KAIMUKI building.

This is what the the firewall sees:

Code: Select all

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
AP4c4e.35b3.e439 Fas 0              156            T B I  AIR-LAP10 Gig 0
APbc16.6537.49de Gig 0              129             T B   AP802GN-A Gig 0
AP10f3.119a.259c Fas 0              168            T B I  AIR-LAP10 Gig 0
AP10f3.119a.259c Fas 0              121            T B I  AIR-LAP10 Gig 0
AP10f3.119a.27f9 Fas 0              168            T B I  AIR-LAP10 Gig 0

Notice how AP10f3.119a.259c shows up twice on port Fas 0 (FE 0). Normally this would indicate a network loop.

On closer examination we can see that AP10f3.119a.259c is showing up on two different IP addresses:

Code: Select all

Device ID: AP10f3.119a.259c
Entry address(es):
  IP address: 192.168.108.35
  IPv6 address: FE80::12F3:11FF:FE9A:259C  (link-local)
Platform: cisco AIR-LAP1041N-A-K9,  Capabilities: Trans-Bridge Source-Route-Bridge IGMP

....

Device ID: AP10f3.119a.259c
Entry address(es):
  IP address: 192.168.108.34
  IPv6 address: FE80::12F3:11FF:FE9A:254F  (link-local)
Platform: cisco AIR-LAP1041N-A-K9,  Capabilities: Trans-Bridge Source-Route-Bridge IGMP

...

But when you look at the arp table, You can see that your missing AP is using one of those IP addresses:

Code: Select all

Internet  192.168.108.34          0   10f3.119a.254f  ARPA   Vlan1
Internet  192.168.108.35          0   10f3.119a.259c  ARPA   Vlan1

I cannot say whether this is a network, firewall, or an AP issue. What I suggest is that you move that AP and connect it directly to the firewall and see if your AP shows up after it boots up and starts working. You might have to wait for the firewall to "check in" after the AP boots up.

If it starts working and shows up in TM, it is a network issue. If not, it's still up in the air whether it is the firewall (less likely since other APs are working properly) or an issue with the AP.

[russellhltn]

I cannot say whether this is a network, firewall, or an AP issue. What I suggest is that you move that AP and connect it directly to the firewall and see if your AP shows up after it boots up and starts working. You might have to wait for the firewall to "check in" after the AP boots up.

I'll give that a shot. Currently, Port 0 of the firewall feeds the switch. The switch feeds the hardwired ports and the POE boxes. There's nothing unique or different about how this AP (or the double) is connected.

This evening, I'll take the cable from the POE box for the missing AP and connect it directly to the firewall (port 1).

I can't think of any way that a network loop could come about - unless one of the hardwired clerk machines has a active WiFi card - but they were both powered off during this test and I had already power cycled the whole network. Can a loop be caused by cross-talk in a defective cable?

[russellhltn]

I put both affected APs on their own firewall ports and power cycled the whole thing to force rebuilding the IP table - no change observed.

Since the AP is on it's own port, I'm guessing you can isolate it and see if it keeps showing up on two different IPs. I'm wondering if the AP is defective and responding to a MAC address close to it's own. It's either that or some kind of interesting software bug.

[tlhackett]

Found and solved the issue. Somehow, both missing APs had their names, which should be unique, set to AP10f3.119a.278c and AP10f3.119a.259c, respectively. This caused them to show up twice on the firewall and prevented TM from "seeing" and finding the APs. Since it looks APs up by their names, the first one found (since they had the same name) was the first one to be shown. This could potentially have caused the wrong AP to enable/disable. I've changed the AP names for the two missing APs back to what they should be. Both APs should now be showing up in TM.

It's still unclear how they ended up with the same name because the AP defaults to "AP[macaddress]" which ensures that it is a unique name. The APs may have been unintentionally renamed.

[russellhltn]

Good work! Confirmed that both buildings are fixed. Yeah, duplicate IDs would sure cause a problem. I wonder how that happened? I'd guess it would be during some kind of update. I'd imagine after a AP firmware update, the device would need it's name reset.

Perhaps a better question - since my stake had it happen twice, how many other meetinghouses are affected?

[russellhltn]

Question: Would a hard reset reset the name? Because I know I did a hard reset of the doubled AP at the stake center.

[tlhackett]

Possibly. But you wouldn't want to do it on the doubled AP but instead the missing AP. It was the missing APs that had the incorrect name associated with them.