Our Smart-UPS Ultra 5000 doesnt come online when commercial power returns

[Pete89]

Hello,

We recently acquired aSmart-UPS Ultra 5000 for an office in Brazil. We have lost power in the building twice in the last month.

Each time we were forced to go into the LED screen of the Ultra 5000 and bring the UPS back online manually.

Is there a way to configure the 5000 to automatically come online when commercial power is restored?

Thank you for any help you can provide.

The only relevant logs are the following:

10/11/2024	09:46:01	Device	UPS: The remaining runtime is no longer below the NMC's low runtime setting while not on battery.
10/11/2024	09:44:48	Device	UPS: The remaining runtime is below the NMC's low runtime setting; if power fails, the UPS will be shut down immediately.

[Teken]

Based only on the limited information you have provided.

You may want to consider reviewing what the sensitivity and upper / lower transfer voltage settings are defined to.

Please upload the data & event logs leading up, during, and after each of these grid down events.

If you have more questions please ask. 👍

[Pete89]

Thanks for reaching out!

Here are then events logs for the days we lost and then regained power. 

In parallel, I have opened a case with Schneider who wants me to cut power, let the batteries drain, bring commercial power back on and see if the UPS comes up and supplies power to the devices. They want me to video the whole process.

Past events would say that when commercial power is restored, the UPS will power on, but not come online and power the devices that run off it.  We will need to go through the menus on the LED screen and bring it online manually. The helpdesk person did mention that the unit is very sensitive and if the power it is receiving is dirty it might not come online alone.

I appreciate any help/advice from the group.

Pete

[Teken]

I’m not sure why the logs you provided show no power interruption?!? Is this system setup to send emails and also to a NMS via SNMP?!?

If so can you provide that information while filtering out any private information?!?

Lastly, based on the information you provided that the system may be sensitive. It would make sense to adjust the power quality sensitivity to see how the device behaves and reacts.

Let me know . . . 👍

[Pete89]

My apologies. I sent the event logs for the Network Card not the UPS. Here are the UPS logs for the days we lost power.

Your suggestion to adjust the power quality sensitivity to see how the device behaves and reacts. Where are those adjustments made? I cant seem to find that.

I really appreciate your input!

Pete

[Teken]

Hi Pete,

Please confirm a few things for me as it relates to this specific unit. Appreciate the detailed logs as it offers lots of insight as such the following questions.

- EPO: Is there an EPO circuit present and has it been activated by you or anyone else as seen in the logs?

- Internal Faults: The system has declared many internal faults in the history logs. Some are expected as the unit is rebooted while others are not and considered hard faults such as the battery module failure. As such the unit should be submitted for a RMA under warranty.

Brain Dead: Did you or the APC Technical Support initiate a Brain Dead Procedure?!? If yes fine, if no that is a serious issue which adds to the credibility of requesting an RMA replacement. It’s possible this error was captured during the battery drain step they had you perform.

Sensitivity (Power Quality): It appears this specific model UPS no longer has or uses the term sensitivity to define if and when the system will go into battery only mode. 🤦‍♂️

I have appended the user manual and on page 18. You may consider adjusting these values as outlined to see how the unit behaves or reacts.

https://download.schneider-electric.com/files?p_enDocType=User+guide&p_File_Name=SUO_UM_990-6458A_EN...

Please let me know 👍

[Pete89]

Hey Teken,

There is no EPO circuit in that room where we have the UPS.

I am attaching a couple of screenshots. One of which is the power levels which are all set to the default values. I am a network engineer and most of those numbers are unfamiliar to me. Does anything stick out?

We have not done any Brain Dead procedures with Technical Support. When I opened the case, I was asked for the UPS event logs and they had me generate some debug files. The person who looked over the files was saying nothing would indicate an issue, but at the same time, he asked for the address of the office. Then he told me at some time in the future to:

1.Cut commercial power to the UPS
2.Let the batteries drain down to zero and everything should go quiet and turn off (UPS reports is has a running time of 49 minutes)
3.Restore commercial power to the UPS
4.If UPS does not come back online, use procedure to bring back online. (LED Screen->Control->UPS->Turn on - No Delay)

All these events need to be captured on video.

He said if after draining the batteries, restoring commercial power, and the unit does not come back online by itself, they will start the RMA process.

I already know what is going to happen because we have lost power twice and had to manually bring the unit online, but I guess they need to do their own troubleshooting matrix. Still, there was something in the logs you saw that made you think the unit was BAD, and I would have thought APC tech support would have seen the same thing.

One thing I will say about Brazil, is that there doesn't seem to be any standard on output voltage. While I was there, I stayed at two Hotels. In one all outlets were labeled 110v, and in the other everything was 220v. In the venue where we did an event, I saw outlets in the same room with labels of 110v and 220v.

[Teken]

Hi Pete,

The event logs were pretty clear that you provided. There were serious issues captured by the system.

EPO: There have been many instances (cases) where different series of APC UPS declared a EPO activation / Fault. Even with hardware that don’t incorporate this EPO feature! 🤦‍♂️

As such this was something that stood out to me and had to ask you (IF) the circuit / feature is actually in place and available to use. 

Please validate there isn’t a phoenix connector installed on the back of the unit. There will also be a twin cable attached and that cable will lead to a N.O. EPO Switch mounted somewhere in the room to activate the same.

The Emergency Power Off (EPO) is intended to rapidly terminate power to the system in the event of an emergency.

Unless this circuit was manually / remotely activated there is zero reason for this error code to be ever declared or captured in the event logs.

The event logs also show hard faults as it relates to the battery module. Given this is a lithium based system there is an internal BCM (Battery Control Module) think of it as the brain that manages all limits and safety aspects of the charging / discharging system.

Brain Dead: This procedure is intended to clear and reset the internal MCU of any possible errors / soft faults. This is a very specific procedure that can’t simply be declared unless manual steps are performed and complete. 

It’s also possible APC has incorporated a flag to capture this event when a similar event happens which is probably related to your two separate grid down / lights out event! 

If so that would be very silly because that would mean every time the system was drained this specific and (special procedure) would be captured and declared! 🤦‍♂️👎

Have you sent in the recording of the battery drain if so what happened?!? Did the system recover as expected or required you to place the system back on line?!?

[Pete89]

Hello Teken,

Thanks so much for all the valuable information you have shared.

I will work to get you photos of the back on the UPS. All I remember is we has to get an electrician to hard wire the UPS to mains power. I don't recall seeing any type of device attached to the UPS. We only used what was provided with the UPS in the box in came in. We replaced an older APC UPS which was connected to mains power the same way and which worked as expected.

I need to open a change ticket to do the work APC wants us to do as part of the troubleshooting. That means having someone come in on a weekend. No one wants that job. 

It sure seems from everything you said that the logs indicate an issue. But I have experience with RMA's and networking equipment. They always make you work to before they send you a new box.

I will come back with more information as soon as possible.

Again many thanks for all the input!!

[Pete89]

Hey Teken,

As promised, here are some photos of the UPS. 

And as chance would have it, there was a brown out on Sat night. From what I can gather, the UPS did not go to battery at all, and all devices lost power. I have a video showing the work around to get things up and running again. I have also uploaded the logs for the day of the brown out.

https://drive.google.com/file/d/1FQ7o3SMn-Dpm8hyVmb4pi8nUDQFD3LTg/view?usp=sharing

We are going to have an electrician check on the work done to see if it was properly installed, but the error messages are not could, but could they be the result of a poorly installed UPS?

Thanks again for all your help,

Pete

[Teken]

Hi Pete,

Please update your service ticket with the new logs. They can easily see the internal fault codes declared and can reference what they actually mean.

This shouldn’t be a hard decision for them to make that the unit needs to be RMA! 🤦‍♂️

Please reference this APC Forum Thread. Ask to have this service request to be escalated to next level support.

Let me know what happens 👍

[Pete89]

Hey Teken,

I called in again and they said that they wanted to do a firmware upgrade before doing an RMA. I told them no, I cant have any more downtime, I really need to get this RMAed. They agreed. 

Remember I told you that on the first call they asked for the address where the unit was? In Brazil. Well, they saw the address again, and said this is the Americas support line, you need to open a ticket with the Brazil support team to do the RMA.

So, I am back at square one.

They said they can transfer the case to Brazil but it will be a few days before they contact me.  

I will let you know how it goes.

Thanks again for everything.

[Teken]

Hi Pete,

That’s great to hear and look forward to the final outcome when the new unit arrives. 👍