Smart-UPS 15kVA doesn't finish self-test - what to do?

[Anonymous user]

Hi all,

after I simulate power faul, I got email and see logs on the UPS: "UPS:Started a self-test"

and nothing happend.

What to do? should I wait until the UPS ends the Self-Test?

thanks.

[Anonymous user]

Hi all,

after I simulate power faul, I got email and see logs on the UPS: "UPS:Started a self-test"

and nothing happend.

What to do? should I wait until the UPS ends the Self-Test?

thanks.

[BillP]

Was the other self test the automatic one that happens every 7 or 14 days? Let's see if it does not log it on the next go around for that since we know it logs appropriately on the manual self test.

[BillP]

Hello,

Does the UPS Network Management Card status report self test in progress? I am just wondering if the self test is starting and passing but the event for self test passed is not being logged.

To help me better understand too, could you provide the event.txt and config.ini files? Instructions here-> How can I download Event, Data, Configuration, and Debug files from my Network Management Card? | FA...

[Anonymous user]

logs are attached.

and here what I'm speaking about:

02/25/2014    12:02:51    Device    UPS: Started a self-test.    0x0137

[BillP]

I am going to check the logs. I know what you're seeing in the log but what does the status show in the interface right now - does it still say self test in progress or does it report the UPS is online?

We need to determine if:

• The UPS still thinks the self test is in progress, thus hasn't reported it has passed or failed.
• Or, did the self test pass and complete but just did not log the self test passed (or failed) event

[Anonymous user]

it reports "UPS is online"

I think: Or, did the self test pass and complete but just did not log the self test passed (or failed) event.. BUT I'm not sure.. that's why I'm asking you, what to do?

[BillP]

Ok, good. This means just the event didn't log rather than it being stuck in self test mode.

I've seen this happen before and what happens is that the NMC looks for a particular "response" from the UPS after the self test is completed and if it does not see it, it does not log the self test passed event.

What I would do is run a manual self test under Tests-> UPS and see if it logs a started/passed. If it does, then I'd just keep an eye on it. If it continues to do it, we can try to reboot the UPS or even the NMC and go from there (which I can provide instructions for) but this would require a power down of the UPS since it does not have a mechanical bypass.

To me, this is more of a cosmetic issue so I wouldn't suggest we do it immediately, especially if we need to reboot the UPS as I suspect you need to plan for downtime - if not, we can do it right away.

[Anonymous user]

now it works...

[Anonymous user]

I checked it again. I started "Simulate Power Fail" directly on the UPS, and got this log message: "UPS:started a self-test"

[BillP]

Simulate power fail isn't really useful unfortunately. It just puts the UPS on battery very briefly and it seems like the Network Management Card is erroneously interpreting it as a started self test even but you don't see that it passed because it is not really a self test and shouldn't be logging that to begin.

[Anonymous user]

Does it mean, the NetworkCard is broken? and should be replaced? I still have the warranty.

Because I need this test. I want to make sure, that my ESXi servers get this notifications and shutting properly down, without this test, I can't be sure. And if I do this test, my servers don't get any notifications from UPS.

And before Updating to the newest Firmware version, it worked fine.

[BillP]

No, I don't believe this is a problem with the network card. If anything, it'd be a firmware issue if you're saying the behavior was different in a different version.

We don't really have a true "PowerChute Network Shutdown (PCNS)" test which I have been requesting for a little while now. It was entered as an enhancement request. Some of the larger UPSs like this one offer "Simulate Power Fail" but it only puts the UPS on battery for a few seconds and that's it. It doesn't test anything further in my mind and I never suggest to people to use it. That is why I suggested a PCNS test function for a future Network Management Card release to help assist with this concern.

If you want to downgrade if you believe "Simulate Power Fail" worked better in a previous firmware version, then I can help you with that.

voidstar - do you have any feedback on this by chance from your view point?

[BillP]

Hi,

You can test communications between the UPS and PCNS by putting the UPS on battery. PCNS should record the event. If you want to test shutdown you can configure PCNS to shutdown after UPS has been on battery for 60 seconds and then put the UPS on battery and wait for PCNS to shutdown the ESXi host.

[Anonymous user]

I configured PCNS to shutdown after UPS has been on battery for 1 second, and run "Simulate Power Fail" test, the UPS has been putted on battery for 3-4 seconds, but the PCNS didn't shutdown the servers, and even didn't write anything in logs.

Angela N. - I called the APC technical support today, and described the issue, they suggest me to hard reset the network card and then update the firmware with teh same version again. If it'll not help, I want you to help me downgrade the firmware.

[BillP]

Hi Andrey - I don't agree with what technical support told you to be honest. I don't think we should waste our time doing that as I don't even really think there is an issue here beyond maybe a firmware bug with the "Simulate Power Fail" logged as "started a self test" in v6.0.6. Please note I am the global technical support escalation leader on this product (Network Management Card) so I'll let you decide what to try next I just want you to understand the level of expertise that I have with this and why I am suggesting what I suggest.

If you want to downgrade, you'll need to reconfigure your Network Management Card too since I'd suggest "formatting" the network management card to ensure integrity of the system and formatting wipes out flash memory, including TCP/IP settings.

I would also suggest what Bill said as this "Simulate Power Fail" is really not a true simulation of the entire shutdown process, especially if you're using PowerChute. I think it is very misleading..so if you don't use this, it also may be worthwhile to stay at v6.0.6.

Let me know what you think or if you have further questions.

[Anonymous user]

I'm pretty sure that it's a network card bug. I've already reset the card and update again, but the issue still exist.

The APC technical, who installed the APC system in our office, told me this Test function exactly to do the test. And you can look in documentation, this function should work. And it worked as long as firmware was old on the network card!

I also checked the logs on the Smart-UPS, and it logs !!! it logs "the ups on the battery" . please see pictures: ... but the logs on the Network Card is different!! how can it be???

I want APC software developers to check software before uploading to customers.

And I don't want workarounds. I need working solution! If you mean, the one possible solution is to downgrade the firmware on the card, then I will do it (let me know how, step by step please), but I want to make sure, that I don't lose the warranty. Because the APC system is new, and I would prefer to get a replacement card, or even better, that the APC technical should do it.

[voidstar_apc]

I asked one of the designers of the UPS about this. What it comes down to is:

1) The only differences between self-test and simulate power failure is the duration (self test is slightly longer) and that simulate power failure does not check the health of the batteries.

2) When a simulate power failure is commanded through the front panel display, the NMC is informed that a self-test has been commanded.

3) When a simulate power failure is commanded through the NMC or PowerChute, the NMC can tell the difference.

I will try to get a 15kVA SmartUPS RT online to verify point 3.

From a big-picture perspective, the value of simulating a power failure would be to test the system's response to a power failure without putting the load at risk. That requires the ability to turn computers back on without cutting power to them. Right now we have no such capability, so our more current UPS products don't pretend to offer a "simulate power failure" option. But such a thing would be very useful and I hope we can offer it in the future.

If you're okay with needing to manually start your servers back up, I've seen people set PowerChute to shutdown on a temperature violation, and then configure the temperature thresholds on the NMC to trigger such a violation. And if you're okay with putting the load at risk, you can turn the UPS off from the NMC.

[BillP]

Hi Andrey,

I don't disagree perhaps it is a Network Management Card firmware bug with how it is interpreting "Simulate Power Fail" but a new Network Management Card will not change this if the issue exists in the firmware, not the hardware. I also don't see how this would be a Network Management Card hardware problem that deems a replacement necessary.

As I noted, I did not think resetting the management card or re-upgrading would affect how this issue is handled.

Also, downgrading the network management card firmware does not void any warranties or anything like that. We try our best in verifying issues before releasing firmware and when there are known issues, then we provide release notes and work on them for the next release. Once I discuss on this end, I can log this as a bug to be addressed and prioritized but as I've said, I don't think this is necessarily the best possible option to verify your configuration even when we do fix it. I understand what you're saying how this is an option, APC Germany told you to use it, and it's listed in the user's guide and I won't argue with that but I am just trying to be honest here on what I'd suggest to any customer to test their solution and configuration 100%. I feel this is more of a legacy option that just has not been removed from the powerview and it probably should be but I don't have much control over the UPS side of things versus the Network Management Card.

To downgrade, please review this article -> Things To Consider When Upgrading or Downgrading a Network Management Card 2 (NMC2) Device between v...

Then, do you have the v5.1.7 files? I will attach them to this post if you don't. Then, you need to use one of the manual methods (like FTP) to downgrade which is outlined in the article here ->How do I upgrade the firmware on an APC Network Management Card? | FAQs | Schneider Electric US

[Anonymous user]

voidstar I think you missunderstood my issue. This function "simulate power failure" worked fine on previous Firmware. And this function listed in the APC manual as a function to trigger alarm.

Angela N. Yes, I will downgrade FW on NMC, then I will sure, that the Network card works properly and will send shutdown signal to the PowerChute clients.

But have one question: what does it mean? "Please note that v6.X.X is a major revision of firmware. v6.X.X of the firmware contains major changes. For instance, user types are different. Downgrading is unsupported, and will without a doubt cause unexpected results or be prevented altogether without noting certain items." ??? Does it mean no warranty after downgrade, in case the card will broken by downgrading??? Or just no support for the downgrading  process???

How can we get APC developer to resolve the issue that I found? and get a new release for FW 6.x.x?

Angela N. now about downgrade process: unfortunatly I don't see the step by step downgrade process for my card AP9631. I found only the first step, that will remove all data on my card, but what next? Please Help!

1. Press and hold the NMC's pinhole reset button for 20+ seconds, ensuring the NMC's status LED is pulsing green during this time. When it changes to amber or orange, let go of the reset button to let the format function complete and for the NMC to complete its reboot process.

[BillP · Accepted solution]

Hi Andrey,

As far as downgrading, we don't technically support it but it can be done and to ensure integrity after the downgrade, then we suggest formatting the network management card to wipe any data from the card. The typical task customers do is upgrading and we cannot support every single downgrading scenario so that is why we state it is unsupported. Also, you'll lose your configuration data so that is another reason why we say is unsupported because we cannot carry it over for a downgrade and ensure the system operates properly.

For what you're doing, as I mentioned earlier, you won't break anything (assuming you follow the instructions and even so, I have not seen any problems with this) and anything you do to the network management card firmware does not affect your warranty in this scenario.

You should follow the instructions for a manual upgrade (or in this case downgrade) for NMC2 (which is what an AP9631 is) according to this article (how do i upgrade my NMC) -> http://www.apc.com/support/index?page=content&country=US〈=en&id=FA156047 and pick either FTP, SCP, or XMODEM as the method. The article I think you reviewed is the "things to consider" kbase which is good to review and understand what happens when you downgrade and how you should do the formatting function to erase all data after you downgrade.

During the downgrade (after you click on the link I provided again), you'll copy over the bootmon file (which you can actually skip since that file is not changing), let the card reboot, copy over the AOS file per the instructions, let the card reboot, and then copy over the sumx file last and let the card reboot. This is all outlined in the article above for AP9631/NMC2 so did you click on that article when I posted it originally? After this part is all done, then you should do the "format" by holding the reset button or if you can log into it via a serial cable or after a DHCP address is assigned (for example), then you can also issue the format command via the command line interface, type yes to confirm, then type reboot, and yes to confirm that.

As far as fixing this, I will discuss with voidstar what we will suggest or do in the long term if you absolutely need this function and cannot utilize any of the more reliable options Bill P or voidstar as mentioned. While I understand you're telling us Simulate Power Fail is there and it's in the user's guide, I am just saying that it probably should not be there anymore and I personally feel they should remove it as a function. Lastly, as voidstar said, our newer style UPSs don't even offer it anymore and I already noted earlier, I have requested a "PCNS test function" enhancement for a future revision that will be a lot better than Simulate Power Fail and how that works and how confusing this is. My thought is my enhancement request would also remove the requirement of the user (you) having to set the PCNS clients to turn off after one second.