PCNS Runtime Exceeded

[knowlesy_apc]

Hey all

Last night I rolled out a windows update to our PCNS Server that controls our virtual environment (VMWare) following the server coming up a run time exceeded was stated see log below this then proceeded to shut our entire infrastructure down.

28/08/2016 19:16:10 PowerChute successfully opened UDP port 3052.
28/08/2016 19:16:12 PowerChute successfully opened TCP port 6547.
28/08/2016 19:16:12 PowerChute successfully opened TCP port 3052.
28/08/2016 19:16:38 Network Management Card 123456789  communication has been established.
28/08/2016 19:16:38 Network Management Card 123456789  Communication established with Environmental Monitoring Card.
28/08/2016 19:16:38 PowerChute Network Shutdown version 4.1.0 monitoring started.
28/08/2016 19:16:39 Network Management Card 2222222222  communication has been established.
28/08/2016 19:16:39 Network Management Card 2222222222  Communication established with Environmental Monitoring Card.
28/08/2016 19:16:39 Communications have been established.
28/08/2016 19:16:43 Available runtime has been exceeded.
28/08/2016 19:21:37 PowerChute successfully opened UDP port 3052.
28/08/2016 19:21:39 PowerChute successfully opened TCP port 6547.
28/08/2016 19:21:39 PowerChute successfully opened TCP port 3052.
28/08/2016 19:22:05 Network Management Card 123456789 communication has been established.
28/08/2016 19:22:05 Network Management Card 123456789 Communication established with Environmental Monitoring Card.
28/08/2016 19:22:05 PowerChute Network Shutdown version 4.1.0 monitoring started.
28/08/2016 19:22:05 Network Management Card 2222222222 communication has been established.
28/08/2016 19:22:05 Network Management Card 2222222222 Communication established with Environmental Monitoring Card.
28/08/2016 19:22:05 Communications have been established.
28/08/2016 19:22:08 Available runtime has been exceeded.
28/08/2016 19:27:08 UPS critical event: Runtime Exceeded

Runtime exceeded http://www.apc.com/us/en/faqs/FA241713/ which as it stands we have no config or low battery to suggest that our runtime and shutdown time will overlap as we have 2 hours and 4 hours on either UPS

From the FAQ to explain what runtime exceeded means…..

Runtime exceeded meets on these conditions

Cause:

1 & 2 for every but vmware
Condition 3 for VMware:
When the total shutdown time required by PowerChute is greater than the Low Battery Duration minus two minutes configured for the UPS. In the event of a low battery condition, PowerChute will not have enough time to complete the shutdown sequence before the UPS powers off. For example, if the total shutdown time required is 3 minutes and Low Battery Duration is 4 minutes, the Available Runtime has been Exceeded event will be triggered.

Condition 4 for VMware:
When the shutdown duration configured for the UPS On Battery event plus the total shutdown time required by PowerChute is greater than the Runtime Remaining on the UPS minus two minutes. This condition can be caused by having too great a load on the UPS when the battery is fully charged.

5&6 are hyper-v only

None of these seem to apply to us, total shutdown time is 27 minutes (high 6 min/ med 6 min / low 10 min / un prioritised 5 min) even if we included the start up timers and migration along with file execution (which is only applicable to hyper-v) we are at 38 minutes with the two ups at 248 minutes and 160 minutes run time well above the required amount to shutdown even if you throw in the extra 2 minutes for PCNS.

Checking the loads of the UPS's they are under 3kw combined and these are 2 x 10kw (APC Smart-ups SRT 10000) units with battery extenders and set in redundant mode in pcns so either one can handle both. I have since disabled the runtime exceeded check and pulled the network cable on the server as I simply cant risk this happening until I  can fully understand whats happened if Im miss reading or there is a bug ? 

PCNS is on its latest version 4.1.0

If there are any other dev logs hidden away etc please let me know and I will supply.

Thanks

[BillP]

Peter,

1 - What model network management card?

2 - What firmware revision is on the network management card (NMC)?

3 - Please attach the pcnsconfig.ini file and the config.ini file from the NMC. The pcnsconfig.ini will be found in C:\Program Files\APC\PowerChute\group1 if you installed to the default path. You can access the NMC config.ini file via FTP, or if the NMC is running firmware 6.? via About - Support and select generate logs the download them. 

The reason the system is shutting down is because someone configured PCNS to power down the systems when runtime has been exceeded. To prevent the unwanted shut down, stop the pcns service, open pcnsconfig.ini and change the line event_RunTimeExceeded_enableShutdown = true to event_RunTimeExceeded_enableShutdown = false then restart the service. 

To stop the service open a command prompt as an admin and enter net stop pcns1. To restart enter net start pcns1

[knowlesy_apc]

Hey Bill,

we meet again. Thanks for the quick reply, I have since disabled it in the config. please see the two files attached. for the model and firmware however the logs Im a little worried about posting (as I haven't sanitised them) is there any way to send to you directly ? 

I also have only an export of the log from the PCNS web ui and not the file in question would that do  ? I currently cant get the pcns file as it is offline (no network connection) and I am away until Thursday

Cheers

[BillP]

Peter,

What is the low battery threshold set for on the NMC? To find the go to Configuration - Shutdown. The low battery threshold should be a minimum of 41 minutes based on the info provided. Once you change that the error should clear. 

[knowlesy_apc]

Hey Bill both ups's are the same, which i dont understand why its only one of them reporting this ? not only that I can only have a max of 30 see below

[BillP]

Pete,

Adjust the outlet groups power off delay which in turn will increase the maximum required delay. To adjust the delay go to Configuration - Outlet Group You will need to increase the delay for each outlet group to 2460 seconds. 

Here is a diagram of how the delays work

I think increasing the outlet off delay will disarm the alarm. The only thing I am not sure about is the low battery threshold. Unfortunately I do not have a SRT available to test this. The low battery settings are coded into the firmware and as noted cannot be increase passed 30 minutes. I will check with a few of our hardware engineers concerning this.

[knowlesy_apc]

Cheers bill, I will look into that, if you can confirm the change to the 30 minutes or how we would go about proceeding on that, that would be great.

However what I dont understand is why only 1 ups reported this (since they are configured the same) as well as that we have 2 ups's which either one can handle the full load and are in redundant on the PCNS, so I would of thought that the pcns could see this and be able to handle it ?

If we can pm each other or somthing I can send all the logs direct just not publicly.

Cheers 

[knowlesy_apc]

Hey bill any luck ? 

cheers 

[BillP]

Peter,

I was able to get access to a SRT 5000 with AP9537SUM. I found that changing to Outlet Group Power off Delays did change the Maximum Required Delay but it does not clear the Available Runtime Remaining error. Unfortunately the 0 - 30 minute maximum is coded into the UPS firmware and since your configuration requires 41 minutes the error message will remain.

What I recommend is

1 - Do not configure shut down for Available Runtime Remaining (as you have all ready done).

2 - Either configure PCNS to state the shut down when the UPS have been on battery for x amount of minutes or when available run-time falls below x. Just keep in mind that the system/s need a minimum of 41 minutes of battery life.

I have entered an enhancement request to allow the Low Battery threshold to be a floating number similar to the Outlet Group Power off Delay.  

[knowlesy_apc]

Cheers Bill,

what I still dont get is that only one of the UPS's were effected however both are configured the same and they are in redundant mode? 

[BillP · Accepted solution]

Peter,

I am not sure why only one would trigger the event. When I tested with 2 SRT5000 I can trigger the event on each by enter a low battery threshold that is less than the needed shutdown time required. 

What are the settings on the NMCs for the low battery, maximum required delay, and the outlet shutdown delays?