SU2200 occasionally powers off, taking everything down with it.

[Anonymous user]

Hi,

Over the last two to three years, we've seen two particular SmartUPS at one client exhibit a really nasty problem. This happened again today and we don't know what to do.

Here's what happens: The SU2200 simply powers off. Power coming from mains is lost. So, subsequently, all computers and servers connected go down quite ferociously. I am fairly certain all LEDs blink. It would be beautiful if it weren't so deadly.

About two years ago, this happened. I opened a case and we tried to troubleshoot it. We got nowhere. We powered the unit back up and hoped for the best. (ha!) It happened some time later. I spoke to APC as the unit was under warranty. They could not determine why it happened but since the unit was still under warranty, they sent a replacement. (Interestingly, we had to ship the old one back at our expense. Not exactly low-cost...)

The problem has followed and has happened two more times with the replacement unit. I tried to talk to APC about it again but they were not helpful troubleshooting it. They chalked it up to "dirty power" and suggested we hire an electrician.

(To be sure, this place may have internal electrical problems. I don't know. I can say that they do have an awful lot of brownouts and power loss events. I might lose power at my home once a year. They seem to experience a brown out or outage with every soaking rain. We have the UPS so that the computers can survive these largely brief events ... Alas...)

Other symptoms:
The unit frequently seems to lose connection with PCBE. I get alerts of lost communication and reestablished communication irregularly. Sometimes when communication is lost, it's lost for good. To my knowledge, there is no physical indication that anything is amiss onsite.

While researching a UPS for another customer, I ran across this site which made me wonder if we weren't experiencing this problem.

[http://www.ingenuityredefined.com/APC/#vgf]

Has anyone here tried the change?

Otherwise, I think we're going to have to purchase something entirely new. It'll likely be APC as this issue seems to be an edge case. We have a lot of UPS and PDUs all over our client base, nearly all of it APC. Maybe APC hates golf courses...

[BillP]

no there is not. the UPS has surge protection built into it already.

[Anonymous user]

Good news.

NIC driver update seems to have squashed the occasional PCNS disconnects. None since last week.

Huzzah, as they say. Huzzah.

[Anonymous user]

Hmmm....

This site experiences a lot of sags and surges. Is there any sense in placing a surge protector in between UPS and wall?

[BillP]

when all of the LEDs flash on the front, that is generally not good. do you hear any tones? does it always fail in this same way with the LEDs flashing? i ask because the LEDs can tell us why its failing - meaning i just want to be sure its ALL of them and not like only the center 6 LEDs which means something different.

i assume you analyzed what logs you had with tech support?

if i were investigating this, these are the questions i'd ask myself, some of which you may have already gone through with tech support:

-how do you have to turn the UPS back on?
-confirm specific LED combination flashing/solid and any audible tones when unit is found off (when it goes into sleep mode it can also be sequentially flashing
-what is the normal load percentage on the unit
-if you are using powerchute with this, by default it can only gracefully shutdown one client. it sounds like there are multiple clients attached to this UPS - how is that set up?
-is it definitely powerchute business edition you are using and do you happen to also have a network management card installed?
-have you done any testing with a non critical load and tripping the UPS upstream breaker or pulling the plug to simulate an outage to see what happens?
-what is the normal runtime this UPS provides to the equipment?

based on these, i may have more questions but some of these could help diagnose the issue as well.

[Anonymous user]

Regrettably, the site is far away and I'm not confident that I have all of the information I need to really answer your questions. My recollection from the one time I personally spoke with someone onsite after this occurred some time ago is that all LEDs flash.

As for logs, tech support specifically told me at the time there would be no logs to view. I can't recall if this was because were weren't running PBE at the time or it was because communications between PBE and the UPS was not solid. (For example, I'm looking at the logs in PBE right now. Communications with the unit was lost on 9/23 but the unit didn't actually fail until 10/20. In the past, the only thing that has consistently brought back communications is a completely power down of both the APC and servers it protects, although I have seen it wake up on its own from time to time. (Rebooting the server has not proven to re-awaken communications.))

Here's some sample log output:

8/12/12 09:22:23 Communication Established
8/12/12 09:22:23 Monitoring Started
8/12/12 09:19:54 Monitoring Stopped
8/12/12 09:19:54 Monitoring Started
7/22/12 13:40:19 Monitoring Stopped
7/14/12 01:05:13 Communication Lost
7/11/12 15:57:48 Self Test Passed
7/11/12 15:57:45 Self Test Initiated
7/2/12 08:55:37 Communication Established

Now, I am going to lean a bit on an old saw: "I inherited this." What this means is that I didn't up the configuration though I did physically install the replacement unit. (More on that below.) I've had to learn as I go. I don't believe there is a NMC, but when I started working with this company I was not at all familiar with larger UPSs such as the SmartUPS and it all is likely not configured properly.

Since then I've done a lot more work with the SmartUPS series including setting up environment monitoring and clean shutdowns, etc., etc. This site deserves a new, fresh look.

However, I'll try to answer some of your questions:

1. I have to hold the power button until the lights pop off. Then I turn it back on.

2. I'll have to wait for it to happen again, which I don't think is a good idea. I'd rather replace the unit with one that has network management and can shut down all servers clean.

3. 39%

4. Yes, we are using PBE, however in looking at this now, it's not at all set up properly. The software is installed and can, evidently, speak to the UPS most of the time, but it would only shut down one server and that ain't good.

5. To my knowledge, there is no NMC.

6. I have personally pulled the plug on it, yes, but that was a long time ago. The UPS behaved as it should at the time.

7. I honestly don't know.

The last thing I'll mention is this: I installed the replacement unit. Coming in green like I was, I was not at all sure what to do when I realized that the plug on the replacement unit was a NEMA-5/20 while the wall was a NEMA-5/15. (heh. I'm not even sure I'm using those terms right.) I called my boss and he asked me to take the plug off of the unit we were replacing and put it on the new unit. I mention all this because I had never done anything of the type before. My boss walked me through it and it seemed okay once I was done. The UPS charged, held power after AC loss, etc. Also: The reason we were replacing the unit was because we had seen the problem in the previous unit, so I doubt I exacerbated the problem by replacing the plug. Still, it may be useful information.

Thank you for the response!

[BillP]

yes, sounds like you do not have a network management card if the slot on the back of the UPS is empty. the log you gave me is definitely from PCBE.

with changing the plug, all you've done is de-rated the UPS as explained in FA156517 @ http://www.apc.com/site/support/index.cfm/faq/

all LEDs flashing is a failure mode indicating there is a problem with the DC bus on the UPS.

at this point, i agree maybe things are not set up properly or ideally. if it were me, you could pursue looking at an AP9630 or AP9631 network management card with powerchute network shutdown on each client (and removing business edition and serial or USB cable on the one server). i cant explain just at this point what is going on with the business edition logging since it seems you are virtually getting no info. do you know what version it is? do you want to pursue updating that and setting it up properly? seems like there are communication problems as well that should be ironed out if you continue with PCBE.

at this point, i don't feel like i can provide a concrete resolution. we can at least try to set it up the best we can with what you have now but basically, all LEDs flashing is not good on a UPS and neither is a hard shutdown on those other devices not being gracefully shutdown.

what do you think your next step is that i can help with? a network management card at least will not rely on the OS and any serial or USB communication issues.

[Anonymous user]

I don't know that there's anything else you can help with at this point. I think that my company is going to purchase a new unit for them with network management etc. At that point, we'll revisit if the problem continues. 😕

[BillP]

alright well i'll be here 🙂

[BillP]

glad to hear you got it sorted out.

[Anonymous user]

Well, the deed is done at the moment. My company purchased an SMT2200. Initially this new UPS did not have a management card. (!!) Fortunately, we were able to add it. I have the card here and the UPS is at the site.

I'll be on-site tomorrow doing the preliminary setup. I have to swap out the plug like in the previous. I also have to install the NM and assign it a static IP address. The rest can be done off-site. On Monday, I'll be back for a scheduled server shutdown where we'll move everything to the SMT.

Wheeeee!

[Anonymous user]

No, unfortunately.

It seems to disconnects last less than a second. Other than a messy event log, I'm not sure that anything is really going on...

I attached a screenshot.

[BillP]

ok, well i hope it works well for you. let me know if you have any questions on setting it up. is your management card at 5.1.7?

[Anonymous user]

I'll have to check the firmware rev. on the NMC.

Lots of goings on today. I went on-site to do some preliminary work. Primarily I wanted to get the batteries installed, install the NMC and maybe power it up. On Monday morning, I'll cleanly shut down the servers and move power to the UPS.

But I ran into an issue. While moving the unit out of the box, my finger touched the power button. I couldn't figure out how to turn the thing off. The most I could do was make the display dark. The fan continued to spin though.

I called APC and the tech was very helpful. He was pretty surprised that it should power on at all since the batteries should be disconnected. (That had been my experience as well.) Sure enough, they were connected. I have no idea why. I pulled the batteries out of the UPS and continued setup.

While on the phone with APC, I mentioned the trouble we were having and that I had planned to swap the plug for NEMA 5/15. Long story short, he convinced me I need to have an electrician come and make sure that the power in the wall is NOT contributing to power problems in that room. He also convinced me that derating the unit, while ok, could be contributing to poor battery runtime and calibrations. So, I talked to the folks on-site and did a little arm twisting. (Really light arm twisting, mind you.) In the end, we found that 20 Amp is already coming into the room and terminating at a 5/20 outlet. (They ran it a bit ago to run an A/C.) There is a free receptacle and the course electrician is going to dedicate a 20Amp circuit and terminate it at the upper receptacle. (Hope I'm getting the terminology right.)

So, by Monday, we'll have to a dedicated 20amp circuit run on new wiring that will power the UPS. No need to swap plugs.

Aside from that, well, one of the workers threw out my NMC box which still had the console cable and CD inside. We found it, though. Whew.

[Anonymous user]

You wanted to know the NMC firmware versions?

5.1.7 for both the AOS and APP

[BillP]

ok, good. those are the latest.

[Anonymous user]

would you pls let us know series No.?

[Anonymous user]

Begging your pardon again. I am setting up PCNS and thought I might ask you a question:

Is it possible to stagger-step server shutdown with a single outlet group? I'd like to pull down the servers in a certain order. For instance:

1. After 15 minutes of On Battery, begin legacy DC shutdown.
2. After 25 minutes of On Battery, begin SQL Server shutdown.
3. After 30 minutes of On Battery, begin Exchange Server shutdown, primary DC shutdown.

My runtime is 43 minutes at average load.

I see how I can configure events to measure the time On Battery and then begin Shutdown procedure. And then there's a hard limit before the UPS sends the shutdown signal, but I am unsure if I can configure the above scenario with the tools provided.

Cheers,

m

[BillP]

so you mean everything is connected to one outlet group but you want to stagger it within that one outlet group? i believe then you'd just use the UPS On battery event and plug in the right amount of seconds on each PCNS client versus if you wanted to stagger outlet groups as a whole, you can configure the outlet groups via the management card to shutdown after X seconds or minutes and other load shedding conditions.

if i understood you correctly, my recommendation is to use the UPS: on battery event within PCNS and enter in the minutes (which you actually have to use seconds i think).

[Anonymous user]

You understood.

But here's what I am a little uncertain about.

Let's say I set the "On Battery" event at 300 seconds (5 minutes) for Server1 but I have the to shutdown when there is 10 minutes of battery left.

Does this mean that, at five minutes of on-battery, Server1 will begin to shutdown? Or will it wait until there is 10 minutes of battery left as set in the Management card?

Looking at the graphic in the manual, it seems like the five minutes elapsed and the 10 minute remaining would need to be reached.

Thank you much!

[Anonymous user]

Series number?

SMT2200...?

[BillP]

here is what will happen. you'll set the value (5 mins) on PCNS. that particular client will shut down as you specify - meaning after the UPS has been on battery for a continuous 5 minutes, it shuts down and applies any specific command file delays for that machine. When the UPS has 10 minutes (or whatever the low battery duration is set to). it will send out a low battery signal and force a shutdown to anything else you didnt specifically configure or anything that is still up and running.

on your last client to shutdown, you probably want to leave only that one selected to shutdown the UPS when it is done shutting off so that the UPS does not continue to deplete its battery while its not even supporting the servers anymore (if they all turned off).

[Anonymous user]

Ah! Wonderful. Thanks for the explanation. 🙂

[Anonymous user]

Ok, here's what I go:

UPS goes on Battery:
PCNS configured to shut down DC1 at 20 minutes in.
PCNS configured to shut down EXCH1 at 25 minutes in.
PCNS configured to shut down DC2 at 30 minutes in.

Low Battery Duration is 10 minutes.
Basic Low Battery Duration is 120 Seconds.

Runtime at average load is 43 minutes.

So, if I read you right:

DC1 will shutdown 20 minutes after UPS has gone on battery.
EXCH1 will shutdown 25 minutes after UPS has gone on battery.
DC2 will shutdown 30 minutes after UPS has gone on battery.

At 10 minutes battery left, UPS will signal to anyone else listening to shut down now.

Right?

The only other issue I seem to be dealing with is that my 2008 Server PCNS client is routinely losing connectivity to the NMC... several times a day. Grrr...

[BillP]

yes, you are correct.

communication is maintained on UDP port 3052. have you checked that out at all within any firewalls? you'll need bidirectional communication enabled for that UDP port. TCP 3052 is also required and then 6547.

[Anonymous user]

Yep, all ports necessary are open. And yet PCNS logs a communication loss 2x an hour.

Windows Events show no hangs or errors logged. The firewall rules allow outbound connects completely. Inbound connections are blocked if a rule is not matches. Pretty standard stuff.

As for the rules individually defined:
PCNS Group1 HTTP Web Interface Port (TCP 3052) -- Any Local Address / Any remote Address
PCNS Group1 HTTPS Web Interface Port (TCP 6547) -- Any Local Address / Any remote Address
PCNS NMC Communication Port (UDP 3052) -- Any Local Address / Any remote Address

I've turned on Firewall Logging to see if I can capture something. None of the other PCNS installs are complaining, so it seems to be an issue at this one box. Could Java be crashing?

I also just opened up the communication on all profiles. That would cover Domain, Public, and Private. (No harm since we're behind an edge firewall.)

[BillP]

see any change?

[Anonymous user]

I should really proofread more often.

[BillP]

ok, i noticed something from your log.

its telling you the network management card lost communication to the UPS, NOT powerchute network shutdown to the NMC which is what the firewall would help with. it tells you this because the NMC cannot report UPS events (such as on battery, etc).

i need the event.txt from your network management card please. also, what is the first few digits of your NMC serial number? Something like JA1012XXXXXX

to get event.txt, you can download that file via FTP - instructions here -> www.apc.com/site/support/index.cfm/faq/ -> FA156131

[Anonymous user]

Uh Oh.

Maybe there's a larger issue.

The FTP server is working, but I can't download anything. (I'm versed enough with FTP to know a few things. I've tried manually via the FTP client in Windows Server and via FileZilla which is way more robust.) When I use Filezilla, I can get further in that I can get a directory listing, but when I attempt to download any logs the server responds with "421 Service Not Available."

I've rebooted the NMC. No good.

I have the serial hooked up to one of my servers and I can login that way. I tried to download the event file that way using XFERINI but I can't seem to get a proper XMODEM download going. I'm still poking around.

As for your note about the event logs indicating a problem between NMC and UPS, I wondered that as well. None of the other PCNS clients are showing these events, however, so it focused me on the one server.

FWIW: I am calling into APC now. I'm going to ask for "i picKed a winna".

[Anonymous user]

Well...

FTP won't work from the 2008 server, the same server where these events show up in PCNS.

I was able to download the event.txt via FTP from one of the other servers.

This would seem to indicate some other kind of network problem with the 2008 server.

S/N: ZA1218020747

Event.txt is attached.

[BillP]

weird..and what did they say when you called in? lol.

my first question is - are the times matched up on the NMC and server?

i can't seem to match the two events to the NMC event logs showing UPS: Restored the local network management interface-to-UPS communication. or the opposite event which is basically UPS: Lost the local network management interface-to-UPS communication.

with my FTP experience, i had a similar issue. The FTP data port was blocked (could not do a directory listing or transfer files) and it turned out to be the firewall in my Cisco VPN service. the VPN service was started even though i was not connected VPN or did not have the VPN client running anywhere. once i turned it off, it started working.

and OK, it is a newer NMC. some of the newer ones are susceptible to EFI/RFI from the UPSs so on the newer ones we had made a change to avoid that.

i don't see anything abnormal in the NMC log. i see a lot of restarts for normal reasons (web user request, configuration changes, etc) which makes sense. the NMC reboots so it temporarily loses comm to the UPS until it has finished its reboot process. i expected to maybe see more unexplained lost comm events but it looks normal.

[Anonymous user · Accepted solution]

Yeah, it is weird, but it does seem to be something specific to the server. The FTP issue is, I think, symptomatic of what I'm seeing in PCNS' logs on that server. Since we're not seeing corresponding disconnects every hour in the event log on the NMC is even more telling.

The restarts you see are ones I did with APC Support today.

As for FTP: Normally when I see something like this, especially with the MS FTP client, the issue is that the server wants to establish active vs. passive connections. Narrowed out that possibility.

Tried turning off the firewall. No difference. Hmmmm...