PCNS shutdown timing help

[BillP]

Hey everyone, I've been kind of confused as to why this isn't functioning correctly. Any help you could provide would be greatly appreciated.

Here's my setup:
!http://i.imgur.com/eXELh.jpg!

Goal:
The goal of this setup is to provide redundant backup power to the five devices above. Each device has
its own place within a specific shutdown and startup order that needs to be followed:
Starting up: Fiber switch, SAN, DNS, SRV1 and SRV2
Shutting down: Virtual machines, SRV1 and SRV2, DNS, SAN, Fiber

Overview of shutdown sequence:
Building loses power. An event delay timer in each instance of PCNS starts counting down (PCNS is
installed on the DNS server and each VM). For testing, this was set to five minutes. The DNS server’s
delay is set to 7 minutes to ensure that the other servers have already started their shutdown sequence
before it starts its own. Once the event delay timer for the virtual machines has counted down, it
initiates a shutdown sequence. The virtual machines shutdown. By this time, the DNS server’s event
delay timer has counted down. The instance of PCNS running on the DNS server is set to run a script
before it shuts down. This script tells SRV1 and SRV2 to shutdown (This is safe to do since the VMs have
already shutdown). The script execution time is set to 240 seconds. After this time, the DNS server
should start its shutdown procedure. The instance of PCNS on the DNS server has been set to shutdown
the UPS units.

What happens correctly:
The virtual machines shutdown and the script run by the DNS PCNS runs and shuts down SRV1 and
SRV2.

What doesn’t happen correctly:
It seems that as soon as the virtual machines shutdown, a timer starts counting down on the LCD of the
UPS. The timer starts at around 300 seconds. When this timer reaches a time of around 150s, the DNS
server says PowerChute will start shutting the system down. The problem is that the UPS timer reaches
zero and turns off the power before the DNS server shuts down gracefully. I'm confused by this because
I've set the DNS server to shutdown the UPS units once it has been shutdown, but that is not what is happening.

I've worked on figuring this out for a few hours, read and reread the documentation regarding PCNS installation, setup for redundant power supplies, etc, but haven't come across anything that has helped me solve this.

Again, any help is greatly appreciated.

Thanks.

[voidstar_apc]

>
What doesn’t happen correctly:
It seems that as soon as the virtual machines shutdown, a timer starts counting down on the LCD of the
UPS. The timer starts at around 300 seconds. When this timer reaches a time of around 150s, the DNS
server says PowerChute will start shutting the system down. The problem is that the UPS timer reaches
zero and turns off the power before the DNS server shuts down gracefully. I'm confused by this because
I've set the DNS server to shutdown the UPS units once it has been shutdown, but that is not what is happening.
>

Two suggestions:
- The VMs appear configured to shutdown the UPS once they have been shutdown. If that's the case, you can disable it. However, if the UPS doesn't shut down and power returns, your VMs will need to be started again some other way.
- The DNS server doesn't have enough time to shut down. If you're running PCNS 3.0, you probably want to add 120s to the script execution time for the DNS server and the VMs.

[BillP]

Hi, thanks for your feedback. I have checked the configuration of PCNS on the virtual machines and it is not set to power off either the outlet group or the UPS.

If I extend the run time of the script, won't that further delay the shutdown of the DNS server? Looking at the PCNS shutdown timeline in the documentation, it shows that the PCNS operating system should shut down after the script has executed. Is this not the case?

[voidstar_apc]

>
Hi, thanks for your feedback. I have checked the configuration of PCNS on the virtual machines and it is not set to power off either the outlet group or the UPS.
>

Okay. Two other possibilities:
1) Runtime has dropped below the low battery duration (shown under configure->shutdown on the NMC).
2) Load shedding has been configured for the outlet group.

>
If I extend the run time of the script, won't that further delay the shutdown of the DNS server? Looking at the PCNS shutdown timeline in the documentation, it shows that the PCNS operating system should shut down after the script has executed. Is this not the case?
>

The script execution time is part of the calculation of how long the computer needs to shut down. The UPS waits that long before turning off. I'm suggesting making it longer than the time the script actually needs to complete so that OS shutdown occurs well before the UPS turns off.

[BillP]

I have the low-battery duration set to 10 minutes, but the battery life on each of my units is around 4 hours (they each have an extra battery pack), so I don't think the low-battery event is being initiated.

I've also checked on each NMC to see if load shedding has been enabled for any of the outlets, that doesn't appear to be the case.

Any other suggestions?

[voidstar_apc · Accepted solution]

Does the DNS server shuts down gracefully when you increase the script execution delay by 120s or so? I'm curious if we can consider that particular problem fixed.

>
I have the low-battery duration set to 10 minutes, but the battery life on each of my units is around 4 hours (they each have an extra battery pack), so I don't think the low-battery event is being initiated.

I've also checked on each NMC to see if load shedding has been enabled for any of the outlets, that doesn't appear to be the case.

Any other suggestions?
>

That's strange because if it's not the NMC and not PCNS... you wouldn't happen to have a device attached to the UPS via USB or Serial?

The NMC's event log might contain information on what initiated the shutdown. You could use wireshark on the NMC side examining traffic on the web port (by default 80) to determine if any PCNS devices are asking the NMC to start a shutdown. If you find anything on wireshark that you want to post here, attach the .pcap file.