SmartConnect - Communications Lost

[Fookoff]

I'm hoping someone else has had a similar experience and has found a solution to the following:

We've a number (45) of SmartUPS. Primarily SMT1500C, but a couple of one-off SMT750C and a SmartUPS 1000. The issue affects all the units.

They are spread across three different geographical regions, but live on the same network infrastructure in terms of firewall/switches/operating systems/etc.

The bulk of the devices live on one network and have the same sort of issue:

1. The location loses power for a prolonged period. It is usually planned and the role of the UPS is really just to allow time to either gracefully close network connections and/or buy time to switch to a backup power source. As always, that is the plan, but reality tends to have other ideas, and we end up in an unplanned outage that exceeds the runtime available resulting with the UPS battery depleted and the UPS shutting down. Still not an issue as we have usually prepared for this and just wait for power to be restored.

2. The actual issue is on power restoration. The UPS comes back online and connectivity is restored. Except for the UPS's connection to SmartConnect. For reasons unknown, the UPS is unable to connect back to Schneider and the only resolution that works is to perform a network reset on the UPS.

3. In case this was an issue of the UPS not having internet connectivity on boot, we confirmed that such connectivity existed BEFORE the UPS power was restored. We tested by unplugging a UPS and replicating a complete outage like it would experience. However, the switch and internet connection never went offline, just the UPS. When we re-energized it, stayed on "Communications Lost". We left it for 15 days to see if it was some timing/DHCP/DNS issue, but the UPS never changed on the SmartConnect portal. We did a network reset on the UPS control panel and it re-appeared back in the portal within a minute.

4. We repeated the test again and monitored the traffic via Wireshark. We could see the UPS SmartConnect ethernet begin chatting away and noticed it doing DNS queries for the SmartConnect internet portal, which were sent back to the UPS. However, the UPS made NO ATTEMPTS to actually connect to the portal. It just did a lookup and then ceased all network communications. Forever. Until we did the network reset. Then we saw traffic to/from the SmartConnect portal.

5. To isolate if this was a static/bad IP entry, we switched the UPS to DHCP. The results were the same. Except on DHCP, we could see the UPS requesting, and getting, an IP. Doing the same DNS lookup for the SmartConnect portal. But likewise doing nothing else. The only difference was that UPS diligently honoured the DHCP lease time and there was traffic for those and similar DNS queries, but beyond the first DNS lookup, it never made any attempt to connect to SmartConnect until we intervened with a network reset.

I'm hoping someone has some ideas as to where to go, as I'm out of ideas. Performing a network reset for the UPS's when this happens is not practical. I've been moving towards NMC enabled UPS as they simply don't seem to have the issues the SmartConnect units do. It is just that the UPS's that support NMC interfaces tend to be more UPS than is really warranted. 

A note that this seems to be a recent development (within the last six months). Most of the units having these issues have been in service for one-three years. There have been no core changes in that time, so I'm thinking it is either a firmware or portal change that is causing this behavior. I'm hoping someone from Schneider can look at the backend logs if I give them some devices to compare what they see versus what I see.

[Teken]

What is the UPS ID / Firmware on these units?!?

Questions Ask . . . 👍

[BillP]

@Fookoff 

I recommend you contact local Schneider Electric Technical Support. They will assist you. They will need to know the serial numbers of the UPSs and the firmware version(s). 

I also recommend not allowing the UPS to drain completely. You can use the load sheading feature via the UPS LCD panel to configure the UPS to power off before the battery(s) drain completely. You will find information concerning load sheading in the UPS user's guide.

As for NMCs, the SMT1500C and SMT750C Smart-UPS accept NMCs.