Testing a SURT5000XLI without batteries

[Malvineous_apc]

Hi all,

I've picked up a 5 kW Smart-UPS with dead batteries, and I'd like to test it to see if it actually works before I shell out on the 16 batteries I'll need to make the 192 VDC it runs on.

The unit won't switch on when I plug in the mains (although it does draw about 80 watts).  I am assuming that like the smaller Smart-UPS units, it won't switch on unless there is some power left in the batteries/capacitors, and since it's been apparently sitting for two years the batteries are thoroughly dead (the packs are reporting 5 V each instead of the expected 96 V).

So, my question is, what's the easiest way to charge up the unit enough that it can be switched on?  With the 24 V units I have connected a 24 VDC bench power supply to the battery connector for a few seconds, then after disconnecting it I was able to plug in the mains and switch the unit on without any batteries connected.

Since coming up with 200 VDC to do this is a little bit trickier, is there any other way I can charge up the unit to get it to switch on?  It looks like all the internal relays run at 24 VDC, so I am wondering whether there's a point at which I can inject 24 V instead, rather than sticking 200 V onto the battery connector.

Any tips?

[MarkC_apc]

The SURT units will try to start when the mains are applied, but when the battery test fails, they shutdown with a bad battery indication on the panel.  Not sure if lack of battery packs (either connected or totally dead) will result in what you describe.  

On another note, I have been able to "rejuvenate" battery packs by individually charging the 12 volt batteries - usually having to throw out and replacing the totally dead ones (this is for testing only - not for backup use!).  If each battery pack shows much less voltage than 80-90 VDC, likely they are all "dead" or a significant number are.  

[Malvineous_apc]

Thanks for your reply!  This one appears completely dead.  Upon connecting the mains there is no indication that anything has happened (no relay clicks, no LEDs, front panel buttons do nothing.)  The only indication I get is that powering it via an ammeter says it's drawing 87 watts.

However my experience with non-RT SU units is that this is normal when the batteries are dead, so I'm not willing to concede defeat just yet!  However this is the first unit I've had that has a 'bypass' switch - ostensibly to keep the load powered if there's something wrong with the unit - but weirdly it still won't switch on in bypass mode and there's no AC output regardless of the state of the bypass switch.  Not sure what the point is of having a bypass switch if it doesn't actually bypass the unit, though.

I have started charging the batteries individually but it's going to be a long process as they are well and truly dead.  I'm not sure whether a desulfator and possibly even water replacement will help, but either way it will be many weeks to attempt to restore this many batteries.

I was thinking if I could get hold of 20-30 old 9V batteries (from old smoke alarms) I might be able to get 200 VDC for long enough to get the thing to switch on, so that's another possibility if I can find a source for them.

[MarkC_apc]

I agree with your concerns of the unit being dead.  I tried starting my 10kSRT with the batteries disconnected and it did go to an alarm state - did at least respond.  Not sure that proves anything with a SURT.  My experience in "reviving" these SLAs is not good - doing everything you have stated.  "watering" did essentially no good and long desulfation did little to nothing - especially batteries under 9-10 volts.  let us know about the 9v try - doubtful, but a useful experiment!

There are many SUAs, and some SUs around with 8 battery packs that might be available really cheap if you could find some in your area - and often end up being great emergency units.

Good luck.

[Malvineous_apc]

Thanks for the info!  It would be interesting to know whether your 10kSRT unit would still power on if it was discharged - i.e., switch off, disconnect both mains and batteries, hold power button to switch on (will click on for only a moment while the capacitors discharge), *then* reconnect only the mains and see if it switches on again.  I suspect the unit will appear dead, and it won't be until you reconnect the batteries that you will be able to switch it back on.  But if you have a unit you can test this with, it'd be really helpful to confirm this theory!

I do have a 24V and two 48V units but they aren't RT ones and the switchover delay is enough that it occasionally causes some of the connected devices to reboot, so I'm looking at RT units for that reason.  This is the first one I've found at pocket-money prices but I will certainly keep my eye out for others.  I do however like the fact that at 192V it only needs wiring rated at 50A so there should be less loss than with the 3kW 48V unit that looks like it has 100A wiring.

[MarkC_apc]

The SRTs use the LCD screen similar to SMT/SMX units - It seems they do not do the "click" for "brain dead" restarts similar to the SURTs.  Anyway, using the same complete discharge technique on my SRT did result in a restart of the unit - with an error message of no battery connections.

The SU,SUA,SMTs all use the same batteries as the SURT - only 8 with 4/4 parallel hookup.  Two units with decent batteries could be used by switching battery cages - a fairly easy job.  The SMX uses 10 of the same batteries, but not so simple to change out!

Good luck

[Malvineous_apc]

Interesting, so it looks like the newer units have been improved so that they don't appear dead when the batteries are flat.  Thanks for testing that out, that's very helpful!

I've only had experience with older units (and this one seems no exception - there's no LCD on the front panel) so I'm still unsure whether it should be behaving like yours or not.

I got hold of a variac and bridge rectifier to generate 200 volts (peak) of pulsed DC (don't have any capacitors handy that can do more than 200 V, so couldn't get constant DC) but I figure since the unit has capacitors anyway it probably won't matter too much so I hooked it up.  I did get a few watts being pulled through on the battery connectors once I got to around 140 V peak, with the current quickly dropping as (presumably) the internal capacitors charged up.  I wound it up to 200 V peak and left the power on for a few seconds until the current dropped right back, suggesting the capacitors were fully charged.

Unfortunately I didn't have any luck powering the unit on at this point, with or without the mains input connected.  I also didn't hear any relay clicks or anything so I guess that's not surprising.  I'm a bit hesitant to bump the voltage up any further given that the units aren't designed to get pulsed DC on the battery connectors so I might go back to the battery rejuvenation attempts.  Maybe having a constant DC voltage on the battery connectors will give me more success.

It'll probably take a few weeks though, but I'll report back how I go if I can get the batteries restored just enough to reach the magical 200 V!

The 24 V units I have take 9 Ah batteries and the 48 V units take 7.2 Ah batteries, so none of them are physically interchangeable with the 5.5 Ah batteries in the SURT unit, but either way I don't have the requisite 16 of any type of battery in working order at the moment, so swapping batteries around is unfortunately not possible!

As a side note, do you or anyone else know what the name of the battery connector is on these APC units?  All the old ones I've used had Anderson connectors but this one has similar but slightly different APC-branded connectors.  The 96V internal ones are brown, and the rear 192V ones (of which there are two for some reason, any ideas?) are white.  They aren't Anderson connectors as far as I can tell, so does anyone know whether you can buy compatible/mating plugs?  Or is it easier to replace them with proper Anderson connectors?

[MarkC_apc]

The connector issues:

Schneider uses proprietary connectors.  The two "whites" in the back are for two extended battery packs and are direct connects to the 192 VDC "rail".  I've used one of them to make an external connection cable and substituted a "standard" white Anderson in its place (that then can be used to connect any "standard" based cable).  I labeled both to prevent confusion.  Of course, don't leave any of the spade connections exposed!

Understand about the batteries - RM style smart UPSs (non XL) above 2000 VA use a battery cage of 8 - 5.5 ahr batteries - easily interchanged with the SURT/SRT cages.

As for the SURT unit - I've done extensive work trying to "recover" units from various failures - none with any success to date unless it was related to battery issues.  Usually, these units will throw a fault code that can then be used to identify areas of failure, but unless you have an electronics background, fixing is a rather difficult reverse engineering job!  If you are seeing any fault codes on the LED panel (any combination of the green battery charge LEDs), I can help with the "areas of failure".

[Malvineous_apc]

I see, handy that they use common batteries with the newer units.

No error codes on the 5 kVA RT unit as I can't get it to switch on at all, not so much as a click or a blink.  I'm still convinced I need to get a decent battery voltage first.  Fingers crossed that's all it is.  I didn't know they could show error codes on the battery level LEDs, that's handy to know.  Is there any sort of reference anywhere for those codes?

The only repair I've ever done on one of these units was replacing a faulty 24 V relay that was arcing when the unit switched between mains and the inverter.  Certainly didn't need to do any reverse engineering on that one! 🙂

[MarkC_apc]

Try searching for "SURT 3kVA-10kVA Service Manual" .  I believe there is a .pdf that has the fault codes - and more.

[Malvineous_apc]

Wow that's really helpful, I never even thought to look for a service manual online as it's so rare to find one.  Many thanks!

[MarkC_apc]

Please keep me appraised of any "repair" progress.  I have a SURT 6000 that has fault problems that I believe are fixable as it can operate on batteries consistently and sometimes on AC (??).  The NMC works, so I can obtain additional fault codes also.  I've done substantial 'tracing" of circuits, but have not be yet successful.  My skills are not the best in this area!

Let me know,

Mark

[Malvineous_apc]

Interesting.  The SU1400XL I repaired by replacing a relay was an easy fix as you could hear an arc buzzing for a few seconds as it tried to switch over to the inverter, and the relay had a transparent case so the blackened interior was an obvious giveaway.  But it sounds like yours might be a bit more complex.  What are some of the fault codes you get?  What behaviour do you observe (e.g. does it fail to switch back to the mains and continue to run on the battery, go into bypass, shut down with a fault, etc?)

[MarkC_apc]

Fault code was related to PFC unit failure.  I checked most of the recommended components and followed the "logic" check section, but was unable to resolve.  I've been working with a local amateur electronics group to attempt a repair, but have not make much progress to date - been much too busy.  It's still on the back burner.  Will be a great learning experience even if it does not work out.

[Malvineous_apc]

Sounds interesting.  Just to provide an update on this I ended up getting old of some cheap 96 V battery packs (~US$25ea) so I've been able to power on the UPS and it works perfectly.  I've also confirmed that removing the batteries immediately prevents the unit from being powered on - there's no residual charge that can be used to switch the unit on like in the older models.

On another note, I also picked up another 5 kVA UPS of a slightly newer revision (now there's a 'D' in the model number) and this one seems to have a fault.  It will run fine off the batteries, but it shows fully charged batteries with only 2/5 LEDs on the indicator.  Plugging the mains in works for about 10 seconds but then the unit clicks into bypass mode with the battery LEDs showing the 01010 pattern.  The extremely useful service manual says this means it's a charger failure.

I worked my way through all the diode checks in the service manual and after using the original unit to find the B+ test point (in the 'D' unit it no longer exists but +VIN seems to be the same) and all the checks pass, so I'm not quite sure what the problem is or where to look next.  The fact that the charger fails *and* the batteries are showing a low charge suggests that there's a problem with the battery voltage, either for real or when it is being measured.  Only 300 mA or so is drawn from the batteries while the inverter is running without a load, so it doesn't look like it's actual voltage sag as I would have expected that to require quite a bit of current draw.

The service manual suggests there could be diagnostic info on the serial port when the unit is cold booted so I will see if I can find a cable for the RJ45/8P8C serial port on the back (another change on the 'D' model) to see if I can get any additional info out of it.

[Malvineous_apc]

Well I managed to find the pinouts for the cable and rewire an old Cisco RJ45/8P8C connector so that I can talk to the UPS via RS232.

Everything seems fine except for the fault codes:

-------------------------------------------------------------------------------
        Factory Data
-------------------------------------------------------------------------------
Fault Log: 0001 002C 002C 002C
Status  Cause
0010    0002
0009    0000
0228    0010
0002    000B
0228    0010
0002    000B
0228    0010
0002    0008
0244    000B
0010    000B


        - Previous Menu, - Refresh

None of these seem to match the service manual, I guess because this is the newer 'D' model.  I don't suppose anyone from APC would be willing to shed any light on what those fault codes are for?  I think the last two (224/10) are when I disconnected the battery, so the three 228/2 ones seem to correspond to the three times I tested it on the mains and had it click over into bypass.  I'm guessing the first two (10/9) are the factory overload test.

So the question is, what does status 228/fault 10, and status 2/fault 8 mean?

[MarkC_apc]

Adam;

What is the full model number of your 5kva unit?  Installing a network management card will give you a convenient communications method to log events, but also to download error information for APC to help analyze.  The SURT units support the older cards that are very cheap - AP9617 card.

[Malvineous_apc]

I thought the same thing, but sadly this probably isn't going to be a (cheap) option.

The original working unit is a SURT5000RMXLI, and this new one with a fault is a SURTD5000RMXLI.  From what I understand, the addition of the D means completely new firmware.  I believe this is the firmware APC moved to when the models with LCD screens came out, although mine doesn't have an LCD.  The DB9 serial port has been replaced by an RJ45 connector (with an undocumented pinout of course), and apparently the presence of the RJ45 instead of DB9 is the main way to identify units with the new firmware.

Sadly one other thing that was changed is that the SmartSlot is now different, and it has two protruding plastic tabs to prevent the insertion of any older SmartSlot cards.  So the network management cards I have for the older SmartUPS units going back many years are not compatible with the new units and can't be fully pushed into the slot.

The only option here would be to buy one of the new ones, but from the quick look I had, they haven't yet dropped in price enough to warrant the expense in my case.

It looks like the new serial communication protocol introduced with the new firmware hasn't been reverse engineered yet either, so something like NUT (Network UPS Tools) can't be used to further diagnose the issue sadly.

It sure makes it easier for us hobbyists when a company won't even release information about its discontinued products!

[Malvineous_apc]

Well after saying that, I found this APC FAQ page which says that the old AP9617 is compatible with the SURTD units!  But it requires a new faceplate so it can fit in the slot, and these are supplied with the SURTD units.  Well unfortunately I didn't get one with my 10 year old unit, so I removed the plug from the back of the SmartSlot and connected it directly to the AP9617 sitting outside the UPS.

Sure enough it powered up but couldn't find a UPS.  The firmware was from 2004 so I downloaded AOS v3.7.3 (w/ SUMX 3.7.2) - the latest version I could find, dated 2010 - and reflashed the AP9617.

Now all it says is "there was a problem loading the application" and to look at telnet for more info.  Telnet suggests there is nothing wrong, except that it is again unable to locate a UPS.  So I'm not sure what the issue is - the card seems to be supported by the UPS and it's running the latest firmware, so why can't it talk to the UPS?  I've disconnected the serial cable from the SURTD just in case so it's not that.

EDIT: In the time it took to write that reply, the card detected UPS, it just seemed to take a few minutes!

I can't see any further detailed specs, other than the status showing as "Off, Battery Charger Fault".  So looks like it is definitely an issue with the charger.  Shows the batteries as 201 V, 36% which doesn't seem right as the other unit fully charged them...

[MarkC_apc]

Great tenacity!  As you likely know by now, the "tabs" where put so that the NMC gen 2 card was the only one that could be used (AP963X) - planned obsolescence as we know it!  I'm amazed you could get the AP961X to work!  It's been a while, but you should be able to download a "debug" file to send to APC for their review using the NMC - is that possible?

I do have continued interest in this and will get my ailing SURT6000 back out and send you the similar error codes/messages to see if there are some common solutions.  

Out of curiosity, have you done a battery recalibration on the two used battery cages with your good unit - just to completely ensure it's not something to do with the battery conditions - one bad battery will manifest itself in really strange ways with these units!

[Malvineous_apc]

Do you recall how to generate the debug log?  If I FTP to the card and download the .txt files I only get the normal event log and some config files, and nothing with more detail than I can see through the UI.

I haven't done a battery recalibration but I did run these particular batteries until they went flat when I was testing the first unit and at ~15% load they went for about 20 mins, so they aren't new condition but they're not too bad either.  Perhaps the new firmware is now including the capacity loss as the batteries age in the percentage calculation?  I don't mind if it says 34% and that's accurate, I only mention it because it is different to what I see on the other units (100%) and because of that I'm not sure whether it is an additional indicator of a failure somewhere in the UPS.

I would like to try hooking up the working UPS' 192 VDC bus to the failing UPS's bus and power up the failing UPS with no mains connected, to see if it reports the batteries in better condition with the higher DC voltage (should be about ~215 VDC from the other UPS as opposed to 201 VDC from the batteries.)

I should also open up the cages and check each battery now you mention it, as a matter of course!  Maybe even top them up with distilled water to see if that will extend their life at all.

Let me know if you manage to get the codes out of your unit to see if we can decipher any of them.

[voidstar_apc]

Some comments

 - The debug log is only on AP9630/31/35. It's mostly a more convenient way of collecting all those txt files you can get through FTP.

 - The tabs are to prevent using an AP9617 on newer UPSes it doesn't support. Odd that they changed to the new tab arrangement on the SURTD... maybe it was part of shipping them with the new NMC? Anyways, you can always break off the tabs.

- Getting 34% is odd... capacity loss is calculated but 100% state-of-charge takes that into account (ie, 100% is less and less charge as the battery ages).

[Malvineous_apc · Accepted solution]

Interesting, thanks for the info!  It never occurred to me to break off the tabs jutting out of the SmartSlot, but then I didn't know for sure that the old AP6917 was compatible.  Now that it is I think I will probably do just that.  I have no plans to use the old relay/environmental cards that are no longer compatible.

Thanks for the confirmation that the capacity loss is taken into account.  Perhaps if the charger was last used with full-capacity batteries, then it broke, and now I am connecting older reduced-capacity batteries, it hasn't had a chance to recalibrate?  And with the failed charger preventing a charge and subsequent capacity adjustment, perhaps that's why it's only showing 34%.  If that's the case then I guess once I hook the working unit's 192 VDC bus to the failed unit, it should go up to 100%.

EDIT: Connecting the DC buses together and connecting the good unit to the mains causes the SURTD with the failed charger to correctly report the battery voltage as 218 VDC, however it still only reports the battery capacity at 36% (up from 34%).  Very strange!

It also looks like the firmware has a bug in it and never expected the unit to be off with a failure, as telnet shows this:

------- Smart-UPS RT 5000 XL --------------------------------------------------

        Status of UPS : Off, Bypass Out of Range
        Input Status : "HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST""HashForm2" METHOD="POST"
        Last Transfer : None

EDIT2: Now after being on for a while the charge is up to 47%. It must be based on a timer? It doesn't increase when the unit is on and running off "battery" (the other unit's DC bus) but when it's off it increases by 1% every minute or two. I guess if I leave it sitting here for a while it will get to 100%!  Unfortunately it clicks off after a few minutes as presumably with no mains input it doesn't want to flatten the battery.