UPS shut down outlets for no obvious reason

[BillP]

We just had a power event with our new rack/servers/UPS setup. The APC UPS is a 4000 KVA, 208V model with 2 external battery packs.

A storm caused a minor power interruption - maybe a few seconds of power loss. But all the servers in this new rack immediately shut down. Servers in another rack, backed up by a similar APC UPS device, kept running fine.

On review of the logs, we saw these messages (reverse chron):

***********07/27/2008 15:12:33 UPS: The output power is turned off.
07/27/2008 15:12:31 UPS: No longer in bypass.
07/27/2008 15:12:28 UPS: In bypass in response to the bypass switch at the UPS, typically for maintenance.

Now, there was no one in the facility at the time so no one manually did anything. Can anyone explain why a momentary power glitch would cause this?


Later entries in the log (12 minutes later) show this:
***********07/27/2008 15:24:45 UPS: The output power is turned off.
07/27/2008 15:24:45 UPS: Restored the local network management interface-to-UPS communication.
07/27/2008 15:24:18 System: Network service started. System IP is 10.10.1.51 from manually configured settings.
07/27/2008 15:24:18 System: Warmstart.

and a minute later:
07/27/2008 15:25:46 Environment: Restored the local network management interface-to-integrated Environmental Monitor Input T Sensor (Int Sensor at Int Sensor Loc) communication.

Finally, we had someone go to the facility and maually power up the UPS device. This is the log entry for that:

07/27/2008 16:01:25 UPS: Passed a self-test.
07/27/2008 16:01:10 UPS: The output power is now turned on.

Can anyone explain what could have caused this? This was a major embarrassment as we just spent a bundle on the UPS and batteries, etc!

[BillP]

We just had a power event with our new rack/servers/UPS setup. The APC UPS is a 4000 KVA, 208V model with 2 external battery packs.

A storm caused a minor power interruption - maybe a few seconds of power loss. But all the servers in this new rack immediately shut down. Servers in another rack, backed up by a similar APC UPS device, kept running fine.

On review of the logs, we saw these messages (reverse chron):

***********07/27/2008 15:12:33 UPS: The output power is turned off.
07/27/2008 15:12:31 UPS: No longer in bypass.
07/27/2008 15:12:28 UPS: In bypass in response to the bypass switch at the UPS, typically for maintenance.

Now, there was no one in the facility at the time so no one manually did anything. Can anyone explain why a momentary power glitch would cause this?


Later entries in the log (12 minutes later) show this:
***********07/27/2008 15:24:45 UPS: The output power is turned off.
07/27/2008 15:24:45 UPS: Restored the local network management interface-to-UPS communication.
07/27/2008 15:24:18 System: Network service started. System IP is 10.10.1.51 from manually configured settings.
07/27/2008 15:24:18 System: Warmstart.

and a minute later:
07/27/2008 15:25:46 Environment: Restored the local network management interface-to-integrated Environmental Monitor Input T Sensor (Int Sensor at Int Sensor Loc) communication.

Finally, we had someone go to the facility and maually power up the UPS device. This is the log entry for that:

07/27/2008 16:01:25 UPS: Passed a self-test.
07/27/2008 16:01:10 UPS: The output power is now turned on.

Can anyone explain what could have caused this? This was a major embarrassment as we just spent a bundle on the UPS and batteries, etc!

[BillP]

We did call tech support, and they wanted the serial number. We asked where, in the web interface, we might find the serial number (since we are remote) ... the person did not know. Is the s/n visible anywhere through the UI? I'm trying to get it from my vendor now but it's taking time.

Thanks!

[BillP]

hi..

what model of UPS is this? it sounds like a symmetra.

going by this:

***********07/27/2008 15:12:33 UPS: The output power is turned off.
07/27/2008 15:12:31 UPS: No longer in bypass.
07/27/2008 15:12:28 UPS: In bypass in response to the bypass switch at the UPS, typically for maintenance.

It seems to me that the UPS was in maintenance bypass (assuming none of these messages are erroneous), thus in bypass, it has no battery backup power which would explain why the servers went down. if nobody touched it today, it may have been sitting in bypass for a little while. can you check the data log on the card?

what are the events right before that bunch also?

the other events you posted are pretty normal - there should be warmstarts when the network management card turns on, they dont apply to the UPS. the environmental event is just the temperature probe on your network management card being recognized again.

now when you got to the UPS to power it back up, was it completely off, or just the servers? more importantly, what exactly did you do to power on. what position was the maintenance bypass switch in? depending on the UPS you have, which i am guessing is a symmetra since we dont have anything else i can think of at 4kVA, this will be on the back or front of the UPS.

also, if this IS a symmetra UPS, you may want to cross check this event log from the web interface to the powerview LCD menu on the front of the UPS. it has a log of its own that we can look at to figure out what happened.

hope this helps! please post back with any other details.

[BillP]

Also note - The UPS is very lightly loaded - 4 typical servers and a few miscellaneous devices.

[BillP]

ok so this is a Smart UPS RT. There is a small maintenance bypass switch on the back of the UPS that you can move which where that message would have been generated from normally. this also has an automatic bypass function which i believe may return that same specific message.

dealing with your stepdown transformer, did everything plugged into it turn off? do you have anything plugged directly into the UPS that DIDNT turn off?

the fact that a self test passed means that the UPS was online, so not in bypass.

there is a way to get a diagnostic fault code from this unit if you had time to do so. also, i wouldnt recommend doing this unless you are comfortable. it isnt that difficult but it would involve putting the UPS into that maintenance bypass mode (so if we arent sure whats going on with this yet, may want to do this at a non production time if possible) and then remove that network management card from the UPS

you need to have the serial cable that came with the UPS - 940-0024 or 940-1524, then you can set the port settings to 2400, 8, none, 1, none for flow control, then pick your com port. attaching any other serial cable could cause the UPS to turn off

DO not hit enter after you establish a terminal session! press shift+Y which will return "SM>" meaning the UPS is in smart signaling mode. after that press 1 wait two seconds, then hit 1 again, this should return "prog." make sure not to hit enter after pressing the '1' command. then press Ctrl+U. this should return 3 or 4 pairs of numbers - something like 010211

we can look that up to see if the UPS recorded some type of fault that isnt in any of the logs for some reason. maybe. it needed to go to automatic bypass during the outage for some reason..

let me know if you arent able to do this. but normally, this wouldnt be a risk. we aren't sure whats going on with the UPS right now..even though it may seem fine, maybe there is an issue with bypass operation.

[BillP]

We don't have anyone onsite at the moment; I'll be back there in 3-4 weeks, and we have an IT consultant there the week after next - so we could start looking at these things.

Since this is almost brand-new, do we have any support options such as an on-site visit or similar? It's literally one month old.

Nothing is plugged into the main unit; just the transformer. Everything plugged into the transformer died, to the best of our knowledge.

Thanks!

[BillP]

the units only come with a factory warranty that covers parts. on-site service is a special service that you'd need to purchase separately.

let us know when someone is able to be in front of it or it may be easier to check out the items i have posted then call into tech support to have them go over them with you if thats easier.

look forward to seeing the results.

[BillP]

if you have the newest management card firmware:

go to the "UPS" tab at the top and then click on "about" at the left hand side. it should be displayed there.

if you have the older management card firmware:

click on the UPS name on the left hand side and make sure "status" is selected under it. it should be reported at the bottom. it should be in the format of XXNNNNNNNNNN - X's being letters and N's being numbers.

[BillP]

What are the steps/requirements for popping the network card? This UPS is in live use at the moment. Thanks!

[BillP]

Hi there --

I just tried doing the dump command from my RT 5000 XL (I'm testing local so I can walk someone through the process over the phone) and using its' included cable I had to use 9600bps to get anything at all. Once connected all I can do is log into the management card. The cable i'm using is a 940-1525A (db-9 on one end rj-45 on the ups end). I believe the 6000 in question uses this same cable.

Once I log out of the management interface it does say "You are now in passthru mode." but Shift-Y doesn't do anything.

Is there a different process for this line of UPSs?

[Cap1_apc]

Hey,

I'll step in here if ipickedawinna isn't monitoring the forums right now...

You need to have the Network card removed for this to work, otherwise, you are correct, you will only be able to log into the card.

You've got the right cable.

All of the previous instructions remain the same, just pull that card first.

Message was edited by: Cap1

[BillP]

i had noted this in my above post:

there is a way to get a diagnostic fault code from this unit if you had time to do so. also, i wouldn't recommend doing this unless you are comfortable. it isnt that difficult but it would involve putting the UPS into that maintenance bypass mode (so if we aren't sure whats going on with this yet, may want to do this at a non production time if possible) and then remove that network management card from the UPS

[BillP]

ugh i apologize. now that i think about it, i dont think the RTD's (versus just the older RTs) can return this diagnostic code since they use a different type of communication.

so if you havent done this yet, hold off and let me see if there is any type of diagnostic for this model. if you have already tried this, there is no harm, it just wouldnt work.

again, i apologize about that and i'll post back. i think that all we will be able to go by is the management card though.

the 6000 RT should use the 940-0024 or 940-1524 cable and be able to do this diagnostic procedure after removing the management card as we were talking about. the 6000 should also use the 2400 baud rate.

[BillP]

ok, on the RTD model that you have, we will need to rely on any LEDs that were lit when it was turned back on, which i know you said someone else turned it off. other than that, we need to go by this network management card log.

have you seen anything funny going on since this issue?

[BillP]

To clarify - we have a unit in New Jersey that is the problematic one, and we have a similar unit in California that we are hoping to do a 'dry run' of any diagnostic test, since CA is where we are and CA is also less critical. The models are:

Smart-UPS RT 5000 XL - CA
Smart-UPS RT 6000 RM XL - NJ

So other than size (5k vs. 6k), the only other difference is 'RM' - whatever that means.

Does that help clarify the situation, and determine what we can/cannot do in terms of diagnostics?

Thanks!

[BillP]

oh ok!

the RT6000 CAN do this diagnostic procedure, the RTD5000 cant.

so unfortunately, you wont be able to do a true dry run of this. you can get it set up but the 5000 wont return anything in hyperterminal. also the 6000 will use a port speed of 2400 rather than 9600 like the 5000 does. the 6000 also uses a different serial to serial cable rather than RJ-45 to serial. the cable for the 6000 should be 940-0024 or 940-1524.

everything else applies.

let me know if you have any questions.

[BillP]

Again, to clarify - you say "the RT6000 CAN do this diagnostic procedure, the RTD5000 cant"

But according to what I know, we have an RT 5000, not an RTD 5000 in CA. I just double checked and our CA model is "Smart-UPS RT 5000 XL", as stated above - so no 'D' there.

[BillP]

well the problem is it RTD is only reflected in the model number off of the actual sticker on the UPS - SURT D 5000XLT or SURT5000XLT. the software/network management card will say the same thing for both so you will need to read the sticker.

but if the cable you are using is 940-1525A then it is definitely an SURTD5000XLT. are you able to check the back of it so you can see the difference?

so, lets refer to the actual white stickers on the UPS that list the models and serial numbers since that is what reflects the D or no D.

SURT6000XLT can do the diagnostic fault code prodcedure mentioned. this has an actual DB9 connector on the UPS and uses a serial to serial cable.
SURT D 5000XLT can't. this has an RJ45 com port. it has a 940-1525A part number on it and is RJ45 to serial.
SURT5000XLT can. it has an actual serial port on the UPS. it uses a serial to serial cable.

again, you'd have to check the physical UPS to see the model numbers. let me know if you have any other questions.

by the way, the D in the model number indicates a UPS with a new communication protocol which is why it uses a different cable. the D models that we currently have are only SURTD3000XLT and SURT5000XLT.

[BillP]

OK - in California, we have a SURTD 5000 XLT

There is no sticker on the back, but there is a sticker on the front under the plastic bezel.

So what we now know is, the model in CA is fundamentally different from the model in NJ. And in NJ, since it is not a 'D' model, we should be able to do the more detailed diagnostic.

We are now trying to locate a tech to visit that location for us so we can do that test.

And the hope is - the more detailed diagnostic may reveal something more than the browser log,

Because at the moment, nothing whatsoever reveals why the UPS went into bypass mode the instant the source power went down - and why it did not recover the instant the power returned (seconds after).

We are seriously considering pushing our vendor to give us a replacement model at this point in time;

Do you have an honest expectation that something meaningful will be revealed by the console?

Thanks!

[BillP]

i am hoping so!

i do expect something to be there and i am eager to see what..

but anyway,

yes what you stated is correct. the actual UPS that exhibited the problem will be able to do this diagnostic and at least give us a better explanation.

please let me know the results as soon as i can and i will let you know what i can about the code that is returned.

since we arent sure what happened yet, and got a message about bypass, we may want to do this at a non critical time at possible. since we arent sure of what happened, then i mean the UPS may have an issue coming out of bypass or something.

Just as a reminder:

•UPS should be put into maintenance bypass and then the network management card removed.
•The tech will need to get hyperterminal going, or some type of emulation program.
•we need the 940-0024 or 940-1524 cable - port speed is 2400, 8 data bits, parity none, stop bits 1, none on the flow control.
•once the connection is started, DO not hit enter. this will cause the UPS control menu to come up and we will need to start over.
•Upon starting the session:
-Enter Shift and Y - this will return SM>
-Enter 1, wait about a second, press 1 again, making sure not to hit enter between the 1's. This returns "prog"
-Enter Control and U. This will return the fault code in the format of xxyyzz or wwxxyyzz.

Then we can see what the deal is!

[BillP]

Well, some news. We have not had anyone on site long enough to run the diagnostics, but - we DID find that the bypass switch was in bypass mode!

So it would appear it shipped that way from the factory, and ... all the self-testing, and front-panel lamps, were oblivious to this fact. We moved the switch from 'bypass' to normal, and got NO log entry. We then pulled the plug, and DID get a log entry for 'running on battery', and then returned the power, and got another log event for return to normal.

We now want to have someone put the switch BACK in bypass mode, and then back to normal, and see if a) front panel laps reflect this, and b) we get log events for this. We'll have to wait till next week for this.

Does this make any sense to you guys?

[BillP · Accepted solution]

hmm very interesting. the UPS shouldnt be shipped from in bypass, but upon start up sometimes you put it into bypass and then you need to take it out.

if the UPS has been in bypass for this long, the batteries sound like they'd be dead and have a low voltage.

can you find in the network management card interface what the battery voltage is? i am also hoping that someone will still be able to go on site and get the fault code..?