Symmetra LX 8000 - Output Voltage Out of Range

[Anonymous user]

We have a Symmetra LX 8000 setup with 10 Batteries (9 in an expansion cabinet).   We inherited this UPS via merger/acquisition.  It has been quite stable for us, and we've replaced 7 of the batteries over the last 5 or so months.   Two weeks ago, we got a message that a power module had failed.  

Replaced the power module, and over the course of the next day, the batteries charged, and the Vbat values climbed and have been holding steady at 138.3 (which is consistent with pre-power module failure). 

The power module shows as "OFF and OK", does not show a fault.  

Critical Alarm: Output Voltage Out of Range

We can not return from bypass, do self test, nor a run-time calibration -- evidently because of out the "out of range" condition.

The new power module is in the same position (3) as the previous.  Have re-seated it, and made sure that the thumb-screw-switch is fully raised before tightening. 

Any thoughts?  I appreciate any help you might be able to give or thoughts you might have.

[BillP · Accepted solution]

What is the model number of the power module you installed -- the one showing OFF and OK? You can see the sticker on the front face of the module and compare to the other power modules you have.

The choices are SYPM4KP (for 208/240 split phase models that can output 120v) or SYPM4KI for 230v models used elsewhere around the world (they output only 230v phase-neutral).

OFF and OK means the power module's inverter is OFF, but the module is otherwise OK and contributing towards battery charging, etc.

Another useful bit if data would be the raw status codes for the suspect OFF and OK module. Please post those if you can. They are available via the Network Management Card or directly off the main PowerView display.

[Anonymous user]

Hi Matt;

Thanks so much for your reply.  I hope we bought the correct module.  For clarity, there was only one Power Module in the UPS that failed.  We replaced it, in the same slot (3).  I tried putting the failed module back into slot two, but the management interface immediately detected it as faulty so I removed it.  We have one (new) power module installed and it is in "OFF and OK".

Here's the RAW data:

APC | UPS Network Management Card

Power Modules
Power Module 1:	NA
Power Module 2:	NA
Power Module 3:	0020 5DE1 0000 2307 0002 09 A1 7B 05 04 00 00 4E C4 BD 01 01

Here are two images, the first is from the failed unit, second from the New.  Also has S/N on the photos.  Both appear to be the same SYPM4KP.   

Thanks again for your reply.   Look forward to hearing what else you have to say / Suggest.

Matt

[BillP]

Thank you for the confirmation of the module part number. You have the correct one.

The suspect module in L3 indeed has its inverter off, but is contributing to charging along with your other modules. I don't see any other obvious issues in the module status other than the fact that the module is reporting that is has not been commanded to enable its inverter. Having the inverters off and charger ON is a valid state for a power module in some bypass scenarios, but not for normal Online mode.

Can you collect the raw status from the Intelligence modules please?

Also, what is the list of the faults/alarms in addition to the Output Voltage out of Range?

I'm thinking that depending on what failure mode the previous module had, and given you have just one power module (and thus no redundancy), we may be seeing the lingering effect of the original bad power module. Following analysis of the intelligence module raw status I think we will have a procedure you can perform on the UPS to get it back online.

[BillP]

I confirmed that OFF and OK is a power module state set by rule when the UPS' maintenance bypass breaker is closed (Active, pushed to right). Is your Symmetra LX still in this manually locked bypass state? That would explain the OFF and OK, and your inability to command the UPS online or perform any load bearing tests (self-test, runtime test, etc).

Your intelligence module raw status will confirm this (as will a visual confirmation of the maintenance bypass switch state), but perhaps I incorrectly assumed that you would have mentioned that in the first post. Please let us know.

[Anonymous user]

Hi Matt, I work with Matt who is out of the office at the moment. I am not on-site with the Symmetra unit but I am waiting for photos of the bypass breaker current state. In the interim, below is raw status output. Again, many thanks for your assist!

Intelligence Module (MIM): M 08 0000 0000 0000 0000 0000 E0FB 0030 00039DBB 0007F952
Redundant Module (RIM): R 08 0000 0000 0000 0000 0000 E0FB 8030 21E8E0FB 0087D852

Power Modules
Power Module 1: NA
Power Module 2: NA
Power Module 3: 0020 5DE1 0000 2307 0002 09 A2 7B 05 04 00 00 54 C4 BD 01 01

Battery Frame
Main Frame: 00 0000 01/02 0002 0000 0000 0000 00 00 000.0 00.0 0000 0000
Battery Frame 1: 00 0000 01/02 0002 0000 0000 0000 00 00 000.0 00.0 0000 0000

[Anonymous user]

Hi Matt,  the photos indicated a "to the left" or off position. To confirm, I had the person on site cycle the maintenance bypass breaker. This resolved the Off & OK state. I just ran a self test and we are green across the screen. I think the issue is resolved! Thanks again for your time and assist!!

[Anonymous user]

Thanks so much for your help.   As I said, we inherited this UPS, and I never thought to look at the bypass breaker.  I'll presume that the fault or event that caused the "original" power module to fail was enough overcurrent to trip the breaker.   Lesson learned, but again, many thanks for your assist.   We appreciate it.

Matt

[BillP]

Great news. The cycling into and then out of the maintenance bypass state sent a fresh command to the power module to engage its inverters and thus allowed the ups to transition gracefully online. 

[Anonymous user]

Hi Matt,

we have a similar problem.

After replacing all 4 batteries in Symmetra LX, we found out that

1) a red light blinks on one of IM,

2) a fan does not rotate at a full speed in one power module. 

We tried to resolve the situation and

1) removed the fault PM

2) replaced the fault IM with RIM when in Maintenance Bypass mode.

Of course, we got the UPS turned off.

After that, we can not return from the bypass mode (due to overload or fault).

Attached are files dumps and logs.

Please, help.

[BillP]

Hi,

I have a similar problem.

After I put two new battery pack from APC, and remove Power Module in Manual Bypass for cleaning, only one Power Module is working.

Power Module 3: 0820 5DE1 0000 2007 0002 09 48 7C 06 00 00 00 57 AF AE 00 00     ok and working
Power Module 4: 0820 04A0 8100 0000 0002 08 48 7C 0D 00 00 00 5D 8E 8F 00 00     off and ok
Power Module 5: 0820 04A0 8100 2000 0002 08 48 7C 0C 00 00 00 5F 90 90 00 00     off and ok

Any help?

[BillP]

Hi Pedro, 

Are you sure those power modules say "off and ok" and not "off and failed"? It appears there might actually be an issue with a capacitor in them based on the information you provided.