Symmetra LX 16000 RM - UPS: Turned off - during power outage - No on battery event.

[Anonymous user]

As the title states I have a Symmetra LX 16000 RM.  The last self test was may 8th and came back as fine.   The unit says it has 15 minutes of run time.

We had a Momentary power failure and the unit just shut off, there was no on battery notification.

08/02/2017

17:32:00

UPS: Turned off.

The UPS turned back on 7 seconds later and it was stated that it had 100% battery until  the following about an hour later this was in the logs:

08/02/2017

19:03:57

UPS: Runtime below alarm threshold cleared.

08/02/2017 19:02:38 UPS: Battery discharge condition cleared.

08/02/2017 18:56:57 UPS: Batteries discharged.

08/02/2017 18:56:57 UPS: Runtime below alarm threshold.

The batteries still say they are 100% and are only about 18 months old.  I am trying to figure out what might have happened.

Thanks,

Alex

[BillP]

Hi Alex,

It will be extremely helpful if you can provide the full event.txt and data.txt files for review from the onboard management card which you've provided snippets from. Or, depending on your Network Mgmt Card model/firmware, the .tar file would be good. More explanation and help for downloading log files here -> http://www.apc.com/us/en/faqs/FA156131

If you don't feel comfortable with posting them here, you can also ask for help from esupport@apcc.com or this link (http://www.apc.com/us/en/support/contact/ask_apc.cfm) to have support review them.

[Anonymous user]

I added files, but the data file does not go back far enough.

[BillP]

Thanks Alex. First thing I noticed is that I think you have really old firmware on the Network Mgmt Card (NMC).

Can you confirm the interface is blue and white with navigation on the side? If there is a Help->About system option, can you tell me what you see in there for AOS and APP versions? Also the model number? AP9619 maybe?

I would like to make sure you get upgraded to the latest rev for many years worth of bug fixes including logging capability, etc moving forward.

Looking at the log, to confirm, the power outage began on Aug 2nd sometime? Do you know what time it started by chance, maybe based on another device's log or your own knowledge? 

Secondly, you said the power came on 7 seconds later - that was automatically or by human intervention? If by human intervention, what specific steps? Did anyone catch the LEDs on the front panel on the off chance someone was physically there?

I also noted the following messages in your log which may be key here:

08/02/2017	17:32:38	UPS: Overload condition cleared.	0x0104
08/02/2017	17:32:36	UPS: Sensed a load greater than 100 percent of rated capacity.	0x0103

I looked at the data log (knowing it did not go back to the 2nd as you said) and I see your phase 2 output wattage % hovers around ~68-75%. My guess is that it is possible you had a temporary overload condition that the UPS was not able to sustain. There is a setting I think on the UPS/management card relating to if the UPS should go to bypass during an overload or if it should drop the load. You can check that setting too. Read more here on that -> http://www.apc.com/us/en/faqs/FA157500

Is there anything by chance downstream you know of that is connected on phase 2 somewhere, possible 120V load, that could have had in-rush current or a high load (air conditioner, vacuum, etc) that may have led to this temporary overload condition that was logged?

The temporary overload/bypass/drop load may have happened all so quickly that the NMC log didn't catch it. Upgrading the NMC firmware will be good to make sure we have all of the latest enhancements/robustness on logging as possible. Once I confirm your version info, model number, etc I'll help you get the latest version and the card updated.

[Anonymous user]

Thank you for your response, to expand:

AOS:  v2.5.3

APP:  v2.5.3

Model: AP9619

Hardware Rev: A10

The power was out for a half a second, a flicker.   The only items attached the UPS are servers.  When the UPS Shut down so did all the servers, when it came back on 7 seconds later all the servers did to which cause the load issue.   I am getting 7 seconds from the APC logs,  which i was wrong about, it was 27 seconds.   The events as I understand it  Power fluctuated (Storm) --> UPS Immediately failed -->UPS Comes right back on.   Power issue was only momentary. 

The loads on the two phases are not balanced,  my belief is it happens because all the servers are 2 PSU servers, so whichever rail comes back first, gets all the load.

I do not have an exact time for the power failure. I was not onsite at the the time.  But I was told the lights flickered and all the servers went off at the same time.

[BillP]

Thanks.

As far as getting the NMC updated, you can use the firmware attached to this post and then use the following procedure to upgrade (refer to NMC1 instructions): http://www.apc.com/us/en/faqs/FA156047

Please note, I had to .zip the .exe file attachment of firmware for the forum here to accept it. So, make sure when you try to apply the firmware, you have apc_hw02_aos373_sy372.exe.

Thanks for your explanation on the other questions I had. I'd like to ask my colleague if he could check further into this from the Symmetra side. My expertise is more the network management side beyond what I already observed 

[BillP]

One other thing actually I just thought of - you can compare what is in the NMC log to what is in the PowerView menu log that is on the front of the UPS. It usually starts at event #64 which is the most recent and you can walk backwards through it to see what it logged.

[BillP]

Hi Alex,

On top of what Angela asked for (powerview logs)could you also provide the serial numbers on the batteries as well as the raw status on one of them? You can get the raw status by going to "Diags" -> "Batts". If you hit enter on the second line of this screen (Status) it should bring up the first battery. If you hit enter 3 more times you will notice the upper right hand corner says page 4/5. The data on this page and on page 5/5 is what we are looking for. 

[Anonymous user]

On the unit itself, it does say on battery and then shutdown due to low battery.   I did not know the display on the unit was different from the web interface logs.

I am working on the update now.

[Anonymous user]

Battery Info:

cV7Zsbua1scI1i94%2FXy2vg%3D%3D.png

[BillP]

Thanks Alex,

Did you replace the entire SYBT5 battery module, or did you replace the smaller individual batteries inside each module? 

[Anonymous user]

The battery modules were shipped to us with the unit 18 months ago.

[BillP]

Would you be able to provide page 5/5 for the battery information as well as the serial numbers on the batteries and the LX itself? You can send it here or via a private chat. 

Thanks. 

[Anonymous user]

Thanks David, here is the information.

.

[Anonymous user]

If I attempt to do a Battery Calibration;

it starts saying 15 minutes.

30 seconds later it stops the calibration with 28% battery left and 9 minutes run time.

10 seconds later it is at 99% battery

10 minutes later it gets to 100% battery.

Is that a battery problem or a calibration issue?

[BillP]

Based on the data you provided it would appear that the batteries are in rough shape. There doesn't appear to be much power left in them and some of the data is not what we would typically see. Was this LX purchased from a reseller or secondhand? If this is a new unit I would call tech support so they can determine the cause of the batteries discharging and issue replacement batteries. 

[Anonymous user]

I have one final question:

Is possible for the UPS to state 15 minutes, but only have 1 minute?

[BillP · Accepted solution]

The Intelligence modules estimate the runtime  based off of what a new, fresh battery would provide. From this point the runtime  will be updated based on the performance of the batteries. If a unit never goes on battery then the Intelligence module would not be able to accurately provide the best runtime. This is why we recommend doing a "runtime  calibration" yearly or bi-yearly for units that never get to experience power failures and run off battery. This would be the only time you could have some discrepancy between the runtime  shown and actual runtime  you get. It is usually not as dramatic as you are experiencing.

The issue with your batteries is the data is not reading correctly. It appears as if the batteries inside the modules have been replaced, as opposed to the entire SYBT5 module being replaced. The batteries inside each module are programmed to a board inside that module which the Intelligence module uses to access them. Swapping out the batteries inside the module would result in invalid data as the new batteries are not programmed/calibrated to the board.