Problems with latest PXHID (AP9361) firmware, DCE alarms and beacon

DCIM_Support · ‎2020-07-03

Recently there was an update to the AP9361 firmware that finally corrected the issue of priority scanning and PX HID controllers. Good news....

However we have 2 racks that have a beacon connected to them so that when either the front or rear door is opened the beacon will begin to flash.

By chance I ended up upgrading just one of the PXHIDs and not the other. I then noticed that after a while the Beacon was flashing on the rack that had been given the latest PXHID firmware. The PXHID event logs show that no one entered the rack despite the beacon flashing.

The web page on the PXHID shows that there are no alarms, but the Beacon is in an abnormal state.

I can turn off the beacon but it will just start flashing again some time later. I have not determined how long this time is.

DCE does not receive anything to say the beacon is flashing at all. DCE nor the PXHID recorded any doors being opened at all.

How do I now make the beacon part of this setup work without downgrading to the previous version of the PXHID firmware on which priority scanning does not work?

Thanks

DCIM_Support · ‎2020-07-03

Hi Garry,

The screenshot you provided shows the unit is in an abnormal state (in yellow). Check the device logs on the unit itself to see what this abnormal state is. We'll want to verify what is happening on the device before we look to see what might be showing or not showing on DCE.

Perhaps you might even want to pull the config.ini from both devices and compare them. Perhaps there is a configuration listed there that will explain it. You may also want to back up the config and restore it to defaults except for TCP to see if maybe something strange got reconfigured or corrupt.

If not, please provide exact detail on your setup as well as the firmware that was originally on the appliance as well as what you have now. You may also want to add the logs and config.ini so any potential configuration issues can be seen.

I know you said you don't have a time-frame for the unit to change state but could you give an estimate? I don't need to know that it was specifically 5 minutes or 20 minutes but if I am to try to replicate the issue, knowing that an hour or a day or a week will give me a clue as to how long I may have to run such a test.

Steve

DCIM_Support · ‎2020-07-03

Hi Steve. There appears to be no entry for anything in the event logs on the device itself. Just to be sure I have cleared the logs, reset the beacon and will wait until the beacon starts flashing again. I should get a clear view of the event logs then. I have already done the reset to defaults on both racks before. I also meant to say that on the second rack the problem was not apparent until I also upgraded that one - so points strongly towards a firmware issue. If I was to guess I would say it is a few hours between resetting the and it starting to flash again. I have not been able to tell properly as I have found nothing in the event logs to tell me when it started flashing again. Hopefully clearing the logs will help to find out the length of time between the beacon reset and then flashing again. I will update when I know.

DCIM_Support · ‎2020-07-03

Garry, Can you pass on the exact config?....maybe the config.ini from that device? Feel free to delete specific IP or e-mail configs . Steve

DCIM_Support · ‎2020-07-03

Hi Steve. I left the 2 racks over the weekend after clearing the logs. Someone entered one of the racks and so triggered the beacon. The other rack was not entered and the beacon was flashing on that one too. I would guess 3 days for this to happen. Nothing appeared in the event logs other than DHCP lease updates:- (This is not the whole event log) 03/09/2016 23:35:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 23:35:11 System: Configuration change. DHCP lease date. 03/09/2016 21:05:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 21:05:11 System: Configuration change. DHCP lease date. 03/09/2016 20:35:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 20:35:11 System: Configuration change. DHCP lease date. 03/09/2016 20:05:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 20:05:11 System: Configuration change. DHCP lease date. 03/09/2016 19:35:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 19:35:11 System: Configuration change. DHCP lease date. 03/09/2016 19:05:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 16:35:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 16:35:11 System: Configuration change. DHCP lease date. 03/09/2016 16:05:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 16:05:11 System: Configuration change. DHCP lease date. 03/09/2016 15:35:11 System: Network service started. System IP is 192.168.16.106 from DHCP server 192.168.16.1 with 3600 second lease. 03/09/2016 15:35:11 System: Configuration change. DHCP lease date. 03/09/2016 15:12:16 System: Web user 'apc' logged out from 192.168.16.1. 03/09/2016 15:11:12 Access PX: Beacon turned off. I'll get the config off the devices when I am next able to connect directly to them with FTP

DCIM_Support · ‎2020-07-03

Thanks Garry, Unless there's something not being logged like an upload of a config.ini, I don't see what might be happening. Please send over the version info on the unit. Also, are you monitoring it with DCE or ISX manager? Once you get me the config.ini, I'll try to replicate your config and let it run it's course to see if I can replicate the issue. Steve

DCIM_Support · ‎2020-07-03

Hi Steve. Version information of the HID's:- Model Number: AP9361 Serial Number: QA1046380101 & QA0937180014 Hardware Revision: 04 Manufacture Date: 11/08/2010 & 09/10/2009 Application Module Name: pxhid Version: v3.9.1 Date: 11/05/2015 Time: 20:58:13 APC OS (AOS) Name: aos Version: v3.9.1 Date: 11/05/2015 Time: 19:25:21 We are using DCE 7.2.7 to monitor the HID's. One of the HID's is configured to flash the beacon when either the front or rear door alarms occur. The other does not map the beacon to any of the alarms at all. However both HID's will flash the beacon after a period of time even if the doors are not opened. The time it takes for the beacon to start to flash is seemingly random, maybe a couple of hours, maybe a couple of days... The config.ini file of both HID's is attached. Device alerts do not show that the beacon is ON in DCE, however the DCE logging / graphing does appear to show when the beacon is on or off. Regards

00 C0 B7 22 5B 7E config.ini

00 C0 B7 23 73 6D config.ini

DCIM_Support · ‎2020-07-03

Also just remembered, but not sure if it is related. Sometimes on other sites we used the APC config.ini upload tool to change user access on a number of PX HID's in one go. On some of the PX HID's this would trigger a 'Beacon disconnected' alarm even though on these devices a beacon had never been connected.

DCIM_Support · ‎2020-07-03

Hi Steven. Is there any news on this one at all? Had any chance to test this yet?

DCIM_Support · ‎2020-07-03

HI Garry, I've had my rack access unit configured with a beacon since this issue started and I have been unable to reproduce the issue. One of my platform engineers also had the same config (different firmware) and they too have failed to replicate the issue. I haven't checked with them in a few days and I have an e-mail into them to verify this. Do you have any info on how things are physically routed? Could it be that something is running too close to electrical cables and a signal is potentially induced? Could you, for a test, find one of the units that is doing this most often and remove it from DCE? If it's happening every 3 days or so and this changes when removed from DCE, perhaps that's another aspect we need to look at. There is no specific option to turn the beacon on this way but perhaps something is being pushed (although there's no record of a file transfer). Perhaps even change logins and SNMP access just in case. Again, there's nothing logged that I can see nor is there a configuration that explains this. Steve

DCIM_Support · ‎2020-07-03

Hi Steve. We now have 1 of the device on the old firmware and the other on the new firmware. The one with new firmware is configured so that nothing will trigger the beacon. See attached image. However on the device with the new firmware the beacon is now flashing even with the settings like this. The rear door WAS opened on this rack but the beacon started flashing a couple of days later. The device with the old firmware has not started flashing the beacon (yet). I will probably factory reset both devices an started all over again when I get chance. Regards

DCIM_Support · ‎2020-07-03

Hi Garry, I just wanted to let you know that although I was unable to replicate your issue, someone in our engineering team was finally able to replicate it. As with you, there are no configurations that are set to cause this and nothing in the logs showing why this happened. We're still stumped as to why this is happening but at least now with our systems being able to replicate it, there may be a way to determine what's happening and why. Since it does take a while to happen, it will still be quite a while before anything can be done to fix it and if you need an immediate resolution, downgrading may be the only answer at this time, sorry. I'll lstill let you know if we find anything more. Steve

DCIM_Support · ‎2020-07-03

Hi Steve. I performed a "reset to default including TCPIP" on both cards and then did no other configuration whatsoever on them. They were still connected to the network though. After a few days the HID with the new firmware started to flash the Beacon. The one with the old firmware has not yet started to flash the firmware. Seems pretty repeatable to me. Have you seen anything on your test setup yet? Garry

DCIM_Support · ‎2020-07-03

Hi Gary, I noted that on the 2nd that engineering had replicated it. I only have one in tech support so I was unable to set it up myself with 2 different firmwared (I used older fw) but one of the engineers set one of theirs up as well (newest fw). As I noted, they were able to replicate and are looking into it. I felt it best for them to test the fw that may have an issue so they see it first hand.

DCIM_Support · ‎2020-07-03

Hi Steve - is there any update on a resolution to this at all?

DCIM_Support · ‎2020-07-03

Hi Garry,

It was put in the system as a bug but I do not see where any progress was made as to why it occurs. Since it doesn't always seem to happen and it takes a long time to happen, I can only assume that's part of why it's taking so long.

I saw that you had posted this comment a few days ago stating that you had removed the beacons and recently re-added them. I'm assuming you removed the post as you wanted to verify over time that the issue was / was not still occurring and that today's post is inferring that it is.

Steve

DCIM_Support · ‎2020-07-03

I did delete a comment a couple of days ago - I have plugged in the beacons back in recently and had seen no errors seen no errors. Turns out the beacons were plugged into the wrong ports! I plugged them into the correct port and the issue came back in a day on one of them.

I have also noticed more issues on the PX HID controller - would it be best to start a new thread on them or add them to here? I don't think they are related...

DCIM_Support · ‎2020-07-03

Hi Garry,

I'd suggest creating a new thread if it's not related.

Steve

DCIM_Support · ‎2020-07-03

Hi Gary,

Sorry for the delay. Of the 2 INI files you provided, Have they both seen this same issue? If not, what was different? I noticed there is a different configuration between the 2 as you mentioned:

00 C0 B7 22 5B 7E: AlarmBeaconMap=FDTimeout, RDForcedOpen

00 C0 B7 23 73 6D AlarmBeaconMap=""

Did you get these ini files while the issue was occurring?

Do you know what version of the config.ini tool that you used? If you're using DCE on these devices too, that should have been what was used for the configuration. I'm not sure how DCE would react or if it may try to reconfigure (Don't think it should but I haven't tested) if it's got a config for the device and then you use the config.ini tool. What changes were made using the tool? If all of these devices are monitored by DCE, Can you remove and re-add one? I would remove the one that this issue occurs with the most and see if it repeats.

If you push an INI to the device and this happens, what exact settings did you push? If you can repeat it, can you also send me the INI that was pushed (or tell me if it was one of the ones already attached.

You also noted:

Device alerts do not show that the beacon is ON in DCE, however the DCE logging / graphing does appear to show when the beacon is on or off.

That simply means that it was not an alarm. You can turn the beacon on or off and not tie it to an alarm. If that's the case, you won't see it in DCE under active alarms, you'll just see it in the data under a state change.

I don't see the issue I'm sorry and I've got similar configurations and I have not seen this on my system. I'm going to speak with engineering to see if there's any way to get more info. Maybe we could even replace one just to get the worst offender back here. Not sure but I'll se ewhat engineering suggests>

Steve

DCIM_Support · ‎2020-07-03

Hi Steve. Originally both devices were set up the same (AlarmBeaconMap=FDTimeout, RDForcedOpen) and I got the spurious beacon flashing on them both even when the doors were not opened. I changed one of them to try to make sure that nothing could trigger the beacon (AlarmBeaconMap=""). However the beacon will still flash on this one too after a period of time. The beacon settings were changed via the web interface, not by uploading a config file. The configs were downloaded from the devices using command line FTP and are the configs that were 'active' when the beacon flashes. There does not seem to be one worse than the other. This only seems to have started after the last FW upgrade. Garry

DCIM_Support · ‎2020-07-03

Hi Gary, I know you had passed along a snippet of the event log in the text but could you please download and attach the actual file please? Engineering is asking for it. Also, is it possible to downgrade one of these units to the previous firmware? Engineering wants to verify it's happening just on the newer firmware. Steve.

DCIM_Support · ‎2020-07-03

I have downgraded one of the devices to the previous version. I have reset the Beacon to off, 'unmapped' any alerts from the beacon and cleared the event logs. Attached are the config and event files from both devices, before I changed the firmware etc Just a matter of waiting for the beacons to start flashing now... Could take a few hours - could take a few days. I'll let you know when they start again. Regards, Garry

00 C0 B7 22 5B 7E config.ini

00 C0 B7 22 5B 7E event.txt

00 C0 B7 23 73 6D config.ini

00 C0 B7 23 73 6D event.txt

DCIM_Support · ‎2020-07-03

Thanks Gary, Engineering is running one unit at 3.9.1 and I'm running one at 3.5.3/3.5.4 and we're waiting for something to happen. I'm not sure what else to do for testing. I am monitoring mine with DCE and I have created a threshold so if the state does change and DCE sees it, DCE will alert for it so I can more easily see if the issue occurred. Steve

DCIM_Support · ‎2020-07-03

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.