Smart-UPS VT 40 kVA - false "The output power is turned off" ?

[BillP]

Hi,

I got several alarm emails from our UPS, saing "The output power is turned off", but the servers didn't turn off. Could be the card playing up?

Iim%2B%2FcU3csGf6igaQ51n4Q%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.jpeg 

I don't see this message is the UPS screen and in the logs I don't see nothing relevant. The problem is the message don't go away.

[Terry_Kennedy_apc]

The "restored the local management interface to UPS connection" message would indicate that something is going wrong between the management card and the UPS hardware. It could be on either side. APC will likely need the event log from the unit to diagnose this, so you may want to go ahead and attach it here as a reply. What model NMC is this (it should say on the faceplate of the card, or in one of the "About" pages via the web interface)?

The 81.2% battery capacity also seems unusual, since the last thing that would have used the batteries is the self-test that ran about an hour before the communication restored message. I would have expected the batteries to "top up" and be at 100%. You might want to check and see if they are still reporting less than 100% capacity.

[BillP]

See the zip file attached to my post.

The problem happen earlier than the print screen attached (7:59) with no previous events.

We did a ups seft test, disconnected/connected the cable to the display and changed the IP configuration, that's why in the image you see that kind of events.

[BillP]

Hi Joao,

I don't see a zip file attached. Could you see if you can edit your post or try it again and make sure it attaches/uploads for review?

And yes, we would need to see event.txt and data.txt at a minimum I think. (Depending on what you intended to be in the .zip, instructions on gathering log files are here -> http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA156131)

[BillP]

I've re attached a zip file, see if can open it

[BillP]

Hmm, no. See if you can email it here to my Box folder and I'll see if I can attach it for you or at least see the files to answer some questions.

upload.APC_For.obfq3m3rb5@u.box.com 

[BillP]

OK, got the files. Let me see if they attach for me. Then I'll review them shortly after a meeting.

[BillP]

Hi again,

I reviewed the log files and also asked my colleague who is a little more experienced with this specific model of UPS. My expertise is with the Network Management Card (NMC).

First idea was to ask if you had a service bypass panel installed which could be powering your load somehow while the UPS was "off" but then the UPS wouldn't report load. That kind of rules that idea out since your UPS reports a load.

Our best idea at this point is to reseat the NMC and see what you get after, assuming the PowerView display on the front of the UPS shows everything as normal. Like, is the load on LED showing there? And does the PowerView event log show anything about the output turning off. (To "reseat" the NMC, just pull it out of the UPS SmartSlot for 5-10 seconds, then stick it back in, let it boot up for a minute, and log in and check the UPS status. You may need to undo two screws holding it in.) I assume there is also no other accessories installed/connected to the UPS in an expansion chassis or to the UPS DB-9 port? If there are, it could be interfering with the NMC reading the correct UPS status.

In addition, it was recommended to double check the status of the bypass switches Q1, Q2 & Q3 (just for the record). If there is no external bypass and the UPS is not in bypass from the PowerView display or from the bypass switch from the front of the UPS, then we would focus on the NMC as the culprit first.

We also noticed at the at the time the UPS is reporting that the output is turned off the temperature in the data log is not recorded.

Let's go from there and see what you find. Let me know if you have any questions before you get started.

[BillP]

nd5MWyQPa%2BiDKBUYJw9jhQ%3D%3D.jpeg

The only thing I have is conected to the card and it is the external temperature sensor.

The UPS only have the front bypass and nobody touch there - It's Off

Should I proceed with the card reseat?

[BillP]

Thanks for the info and picture! Yeah, proceed with the card reseat.

[BillP]

I got the same error after reseating the nmc:

OyfS%2B8f7qq8hFIuhm2D1%2FQ%3D%3D.jpeg

[BillP]

Hmm, ok. Let me research more. In the meantime, can you give me the firmware details on the UPS itself from under the UPS->About menu? Screenshot or copy and paste is ok. Thanks!

[BillP]

Ok, here goes:

Model:	Smart-UPS VT 40 kVA
Serial Number:	PQ0640140161
Firmware Revision:	912.0901.I
Manufacture Date:	01/27/10

[voidstar_apc]

Could you check two SNMP OID's?

1) upsAdvOutputVoltage (.1.3.6.1.4.1.318.1.1.1.4.2.1.0)

.iso.org.dod.internet.private.enterprises.apc.products.hardware.ups.upsOutput.upsAdvOutput.upsAdvOutputVoltage

2) upsHighPrecOutputVoltage (.1.3.6.1.4.1.318.1.1.1.4.3.1.0)

.iso.org.dod.internet.private.enterprises.apc.products.hardware.ups.upsOutput.upsHighPrecOutput.upsHighPrecOutputVoltage

If unfamiliar with SNMP, you can download a SNMP browser called GetIF from here. You'll also want the PowerNET MIB from here. Put the mib into the "mibs" folder in the GetIF program directory. In GetIF's parameters tab, enter the card's IP address in the "Host name" field and click "Start." It should display "SysInfo variables OK". Change to the MBrowser tab, navigate to and click on each of the OID's above, and click the "Start" button for each one. You should get back a response.

[BillP]

Here goes the 1st one:

mBAsBMdW%2Fg2KCRNGy6wKBw%3D%3D.jpeg

Them the second value. I added two prints because I got 2 diferent values over a short period of time:

i8ev2pyg4hgS7ANPWoZZBw%3D%3D.jpeg 

KC0opwssXKi1nTaQo1vlbA%3D%3D.jpeg

Thank you

[BillP · Accepted solution]

Here goes the 1st one:

mBAsBMdW%2Fg2KCRNGy6wKBw%3D%3D.jpeg

Them the second value. I added two prints because I got 2 diferent values over a short period of time:

i8ev2pyg4hgS7ANPWoZZBw%3D%3D.jpeg 

KC0opwssXKi1nTaQo1vlbA%3D%3D.jpeg

Thank you