APDU10x/11x: Outlets Randomly Turn Off While NMC3 Still Reports Them as On

[stormwatching]

In February, we worked with Schneider on a support case after several brand new APDU11452SM outlets at our primary datacenter were unknowingly powered off. The web UI showed the affected outlets as "on," despite the outlet LEDs being off and no power actually being delivered.

Schneider's recommended fix was to reset all six circuit breakers on the affected PDU and upgrade the firmware. We did both, resetting the faulty PDU and upgrading firmware on all PDUs at our primary datacenter to v3.4.0.2.

Recurrence in June

In early June, we saw the same issue resurface on a different set of unused outlets, again at our primary datacenter. A few days later, a separate PDU at that same site experienced a partial outage affecting two portions of physically neighboring banks.*

*Just for reference, it’s important to understand how the PDU is organized. Each of the six banks are identified by a color, but the outlets belonging to a bank are not all grouped together in one spot. Instead, each bank is divided into several smaller groups of outlets that are then distributed across the length of the PDU and interleaved with one of the other six bank portions.

Shortly after, we found a set of outlets turned off at our secondary datacenter (APDU10452SM), with identical symptoms: UI shows the outlet as on, LED is off, no power delivered. The only workaround we've found is cycling the outlet through the web UI, which brings it back online.

Separately in April, we also hit a delivery delay on APDU11151SM PDUs destined for our tertiary datacenter apparently due to an unresolved issue affecting the entire SM PDU line, which had put that model on hold.

Our current environment

• Primary DC: APDU11452SM
• Secondary DC: APDU10452SM
• Tertiary DC: APDU11151SM

All NMC3 controllers have been updated to the latest firmware (v3.4.0.9), and we're continuing to monitor for recurrence.

Questions for the community:

1. Has anyone else run into this issue with their PDUs?
2. Any additional insight from Schneider or otherwise on root cause?
3. Are you running 3-phase switched and monitored PDUs, or single-phase switched and metered? Curious if this is correlated with a specific configuration.

We appreciate any input trying to figure out if anyone else has encountered this issue with the SM series.

[Teken]

What do the event / data logs indicate from the following: PDU, SNMP, SysLog, Email???

Questions Ask . . .  👍

[stormwatching]

There was not a single entry in the logs related to the issue. The only reason we knew something happened was because one of our servers alerted us that power wasn't being supplied to one of its PSUs.

We have a time series dashboard showing power consumption for each outlet, and for the outlets with any workload connected, the power consumption drops to zero. No failures, no warnings, nothing. For outlets with no workload connected, we have to periodically check all the LED indicator lights next to the outlets.

[Teken]

You haven’t answered my questions with specificity. 

As such I’m going to provide you with some action items to complete:

Enable all of the following methods of logging / notification: SNMP, SysLog, Email.

Provide the NMC logs in their entirety minus any (PII) related information. Of special interest is login activity, event logs, data, cold boot, warm boot, watch dog timer.

The diagnostic logs will pool everything together for viewing. The email notification will send you an email (IF) outlets are turned on, off, cycled.

You’ll obviously want to test and validate each of those actions generates an email notification! ☝️

If it doesn’t stop and report that information here and with APC Technical Support.

If the PDU is managed / controlled via API or some other mechanism call this out such as BACNET, ModBus, etc.

Confirm which SNMP is in use and verify none of them are set to a write privilege. Take one problem (PDU) and reset it to a factory default with the default APC user and not networked to anything else.

Monitor if this defaulted PDU shows any problems in this factory state vs the others that have been user defined.

The facility should place a power quality monitor on all problem target PDU’s. To capture the line input vs what the PDU is outputting.

Buy any cheap 120 VAC smart plug and connected it to one of the problem outlets that turned off.

All of the newer ones will track the voltage, frequency, current, KWH, and On/Off state.

Having this in place will speed up the process of being notified as to the PDU Output.

Questions Ask . . . 👍

[jchiglo]

I need tech support to call me. I just had outlets 17-24 turn off on in production data center. Same exact symptoms as described above. I am using IT Expert and IT Expert monitoring shows all outlets on and the power trending drops on affected outlets. I logged into the PDU and pushed turn on outlets immediately and I was a video call with the onsite employee with video, watching the outlets turning on.

This is a major problem for Fastenal.

[jchiglo]

I was advised to update pdus to 3.4.0.9 but IT Expert doesn't have update available yet to push it out. SNMC Tool is the only way to deploy the firmware.