PX500 UPS confusing alerts on DCE during battery test

DCIM_Support · ‎2020-07-03

Hi all. We have site of several PX500 UPS's and every 2 months or so they each do a battery test, which is as expected but the results from the DCE server are unexpected.

Firstly the UPS goes into battery test and DCE displays a warning message of

"The load is drawing more power than the UPS can draw from the input, and therefore adding power from the batteries. Located on battery enclosures."
This seems reasonable if not a long winded way of saying the load is on batteries.

When the test stops this warning message clears. Again to be expected.

The DCE server is setup to email on warning alerts and indeed we get the email alert stating "The load is drawing more power than the UPS can draw from the input, and therefore adding power from the batteries. Located on battery enclosures."

Now the confusing bit - looking the DCE alarm history we see 2 alerts during the test period:-

"The UPS is charging the batteries, or batteries are resting. Located on the Battery enclosures"
Not really sure what this is saying - The batteries are being charge - maybe - but might just be resting!

Shortly after that message an informational message of
"The load is supplied from inverters of the unit. The unit is drawing power from the utilities."
Is this an obscure message to say we are no longer on battery?

Then about 3 - 4 minutes later both of these alerts clear with 10 seconds of each other.

There is no sign of the emailed alert in the DCE alarm history.

The UPS event logs just add to the confusion:-

Battery test running - informational - DCE does not pick this up at all

Started a self test - informational - DCE does not pick this up at all

Batteries are discharging - Warning - DCE does not pick this up at all

Battery test complete - informational - DCE does not pick this up at all

Battery condition is OK - informational - DCE does not pick this up at all

UPS operation mode - Normal - informational - DCE does not pick this up at all

Batteries are charging or resting - Informatinal - DCE shows this alarm in the alarm history but as a warning alarm not informational. This alarm never appears in the active alarms in DCE during the battery test.

The alarm "The load is drawing more power than the UPS can draw from the input, and therefore adding power from the batteries. Located on battery enclosures." that appears on Active Alarms on DCE never appears in the UPS event logs.

I end up with a few questions from this sequence of events.

Is this confusing array of alarms / text really what is expected to happen during a battery test?

Why does the alarm "The load is drawing more power than the UPS can draw from the input, and therefore adding power from the batteries. Located on battery enclosures." never appear in the DCE alarm history?

Why does the alarm "The UPS is charging the batteries, or batteries are resting. Located on the Battery enclosures" appear in the alarm history but not ever in the Active Alarms on DCE

DCE never says at any point that the UPS is performing a battery test. Why not?

Any answers please?

Thanks

DCIM_Support · ‎2020-07-03

Hi Garry,

I'm going to assume none of these are manually created thresholds and only the automatic ones already existing on the DCE server. For the most part, I wouldn't expect to see most of these messages and I'm going to try to explain why.

Here's the way the automatically generated alarms on APC devices work. When we discover an APC device, we have a scanner DDF already on the DCE system that defines the points and sensors to be scanned. When discovered, DCE also downloads an alerts DDF file from that device. These "essential alerts" that are defined in the DDF show up in the Alarm Configuration perspective in DCE under Device Alarm Configurations.

When the device has an event (assuming you checked the priority scanning option in the discovery), the device sends DCE a special trap. This trap simply tells DCE there is an issue. DCE then polls the device for a specific OID that has a number of bits set that define the alert on the device. With that set of values, DCE then checks that essential alerts DDF it downloaded and looks to see if the event that is in the list. If it is not, it does not send any alert. If it does find the alert, it then alarms based on the configuration of the configurations of the thresholds, policies, and alarm actions.

For the alarms, let me first comment on the alerts you are not seeing:

Battery test running - informational - DCE does not pick this up at all
Started a self test - informational - DCE does not pick this up at all
Batteries are discharging - Warning - DCE does not pick this up at all
Battery test complete - informational - DCE does not pick this up at all
Battery condition is OK - informational - DCE does not pick this up at all
UPS operation mode - Normal - informational - DCE does not pick this up at all

These events are all based on the test. The NMC coders felt that you needn't be alerted for these events so they were omitted from the list as many informational alarms are. You don't necessarily need to be alerted that the unit is being tested was I'm assuming the feeling. If the test fails, another alert will trigger and the DDF and DCE are configured to alert based on this alarm. The test failure could be a bad battery, a bad power module, etc.

If you do need to be alerted for a test in progress, you can always create a manual threshold on the device which will cause the unit to alarm. Again, I'm not sure you want that as it is only informational. Potentially, any informational alert the device may see will be left out of what the DDF considers unnecessary.

The level of alarm is also defined in the DDF. For instance, an alarm in the device may be warning however if the DDF is written so that it shows as informational, that is what DCE will show and how it will alert. Any alert that is in the Device Alarm Configurations can also be edited so as to show a different alarm severity level.

As for alerts that are not in the alarm history, I'm not sure why they may not be there. I'm not familiar with the specific events so please bear with me on that. First the obvious, make sure it's coming from the DCE server and not the device. Also, make sure you're actually in the alarm history window and not the active alarms window. Something else maybe not so obvious, check the dates. If there is a time sync issue between a device and DCE, it is possible the alerts may now show up properly. I've seen NetBotz units for instance now show surveillance data unless you choose a wider range to search from. Choose today and tomorrow instead of just today:

Maybe change the second date to include 9/10/2016.

This synchronization issue could also be why the message does not immediately show up in the active alarms window. Additionally, it may take a minute or 2 to actually show in the window based on the UI performance so since the message is only active for 2-3 minutes, that window may simply have needed refreshing.

Thanks,

Steve

DCIM_Support · ‎2020-07-03

Thanks for the thorough answer.

The alerts and thresholds are all the standard alerts from the UPS.

With regards to the alarm history results I had already looked way back in the history to see if it was missed and have both tried to refresh the alarm history pages and it definitely does not appear there.

One thing I though about though is if a UPS is discovered it then downloads this 'alerts DDF file' from the UPS. If the UPS has had firmware updates (not the NMC but the UPS) does the DDF file get downloaded from the UPS again? Or is the DDF file regularly checked and downloaded by DCE if necessary? If not then this could cause some issues I guess.

I have tried to rerun the device discovery for these UPS's but not sure if that would help

Also 3 of the 11? UPS's do not have any Comms Lost threshold in DCE (Refer to ) which I thought was not possible. Could this be related?

DCIM_Support · ‎2020-07-03

Hi Garry,

I do understand they are standard alarms, I'm simply not familiar with them as I've never supported the 3 phase units. Again, they would appear to have simply been omitted from the list again as it was felt they were not necessary. If you feel the need to be notified when a test is occurring, as I mentioned you can create a manual threshold or I can put an enhancement request in the system. Please keep in mind that adding such a request does not ensure the enhancement is added.

As for the alarm history, I can check to see if we have a PX that I could test and look to see if I get those alerts and if they show up or not. I know I don't have a 500 available.

The DDF is based on the NMC. Updating the UPS firmware will have no effect on this. If the NMC is updated however, we should be downloading a new DDF from that device.

As for the threshold Alarm Configuration alert for Communications Status (lost comm), I see you're already working with John on that and I don't want to duplicate efforts. I think we'd have to figure out why each is happening before we could say with any confidence that they are connected. I have looked at this and noticed that I can't manually even delete them.

I can say that I've not seen this threshold just disappear before. I have them listed for all my devices and I'm running DCE 7.3.1, 7.2.7, and a few other older versions (what version are you running?). The only devices that I can find on my system that do not have these are virtual sensors. When clicking on devices, those are the only ones that don't show it. Is it possible that these 2 issues are somehow related? I can not say, I've not seen these events just disappear or not appear to begin with. I've also not seen where alerts don't (eventually) show up in the active alarm window. Do any other alerts do this? Can you create a manual threshold and make it alert to see how long it takes to appear in that window? Have you tried installing the client on a different system? Maybe that would effect both of these issues.

Steve

DCIM_Support · ‎2020-07-03

Thanks again Steve. I don't think the customer wants alerting that the UPS is undergoing a battery test. It was just causing confusion as to what was really going on. They get one email saying one thing, and an alarm on DCE saying another thing. I just couldn't explain why so asked the question. I have already tried using the DCE client on multiple machines. The server is at 7.2.5 and it is unlikely I can upgrade it any time soon, but will suggest (again) that it is upgraded)

DCIM_Support · ‎2020-07-03

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.