APC NMC2 email notifications inconsistently sent

[seuser3]

I'm having issues with inconsistent email notifications from 2 different NMC2 devices installed in 2 different SMX3000RMLV2U units. Sometimes notifications are sent to all recipients, but often times some notifications are not sent to some recipients. I first noticed this when I would get a notification about an input power restoration event for which I had not previously been notified about the preceding input power disruption event. Or I would receive a certain notification to 1 address but not to another configured in the same way for the same event. This happens relatively frequently and I've confirmed on the configured SMTP server it is never asked to deliver the missing messages so it seems to be an issue on the NMC units themselves.

Since then, I've also added a 5 second notification delay to some recipients to prevent emails to SMS aliases from going out for short events/blips that clear without the need to wake anyone up. However, I continue to receive messages for events that last < 5 seconds.

I am attaching (redacted) screenshots showing the configuration of the notification in question & also an even log showing the length of the most recent event that only 2 out of the 4 recipients received notifications for. Also, one of those who did receive it was an SMS email address which should not have received it since the event cleared in < 5 seconds.

UPS Model: Smart-UPS X 3000
SKU: SMX3000RMLV2U
Firmware Revision: UPS 08.1 (ID1003)
Manufacture Date: 12/10/2022 (unit 1) & 11/20/2022 (unit 2)

NMC Model Number: AP9631CH (unit 1) & AP9631 (unit 2)
Hardware Revision: 05
Manufacture Date: 07/03/2011 (unit 1) 07/10/2013 (unit 2)
Application Module
Name: sumx
Version: v7.1.2
Date: Mar 21 2023
Time: 17:44:20
APC OS (AOS)
Name: aos
Version: v7.1.2
Date: Mar 21 2023
Time: 17:34:00
APC Boot Monitor
Name: bootmon
Version: v1.0.8
Date: Apr 8 2014
Time: 10:59:40

[Teken]

I would suggest you start by reviewing the Syslogs & NMS. Next show me the About tab for the NMC.

As a next step I would update one NMC to the latest v7.1.8 firmware and report back it’s behaviour.

On another NMC reboot the card and report back have it behaves afterwards.

Questions Ask . . . 👍

[seuser3]

What logs/NMS are you referring to? I have reviewed the event logs of both units (+ some screenshots included showing events in question) and the logs of my SMTP server. I have also included the details from About -> Network in my OP, which I believe is what you mean by the about tab for NMC.

I looked at updating NMC, but if I've read the documentation correctly (https://www.apc.com/us/en/faqs/FA170679/ & https://www.apc.com/us/en/faqs/FA279197/ & https://www.apc.com/us/en/faqs/FA164737/) I cannot do so without first shutting down loads on the units so that cannot be done immediately and will require change planning which is disappointing.

Both cards were rebooted multiple times last year due to power failures that outlasted the battery capacity. I can reboot them again, but don't think this will have any fresh effect unless the suggestion/implication that  I need to regularly reboot the cards for them to operate as documented which would be beyond disappointing and not in line with my previous experience with APC and products of this nature.

[Teken]

NMC: You can update the NMC firmware with no impact to the connected loads on the UPS as rebooting it affirms there’s no linkage to the main system.

The links you referenced are specifically for updating the (UPS Firmware) where (IF) your specific model UPS does not support an upgrade while the outputs are still On and connected.

NMC About: I want to see the uptime for the system. As doing so would provide insight as to if the NMC is self booting itself whether from the watch dog timer or due to hardware / firmware corruption.

You obviously need to know if anyone has rebooted the NMC and ignore that data point.

NMS / Syslog: If the UPS is being monitored by an external monitor which it should be. It would inform you of the historical behaviour prior, during, and after things have changed.

Your OP does not call out when this problem appeared or what events led to this behaviour. If this problem has been seen since day one that’s a important data point vs it started two days ago.

If it just started happening the question is why? What has changed? As it relates to email / SMS messages being received, not received, or timely has been a problem since the Internet has existed.

Keep this simple and identify what method works with the highest degree of uptime and reliability.

Delete all aliases except the working one and validate over a short / long period of time. Once validated add in one problem alias / SMS.

Your going to affirm or negate something is wrong with the delivery method or there are barriers at play. 

Do not rely on the (Test) function as the end all be all from the NMC! ☝️ The only validation is receiving an email / SMS when the system is in full production and you remove AC Mains to see the results.

Questions Ask . . .  👍