Symmetra 40K & Agent Config Questions -> Massive Issues

[Mgregory2K_apc]

I've posted a couple questions, but I'm still having big time issues and I'm going to try and throw this all in one thread and hopefully get it figured out. I thank you in advance for your input.

I have:
Symmetra 40K w/ 16 batteries
~20 Windows Server 2003 boxes
1 ESX box
PowerChute Network Shutdown Installed on all servers
...and I'm learning all of this from scratch with no help from anyone except the forums

Goals:
-When we lose power to the building have all the servers stay up for the max amount of time, except for 4 non-production servers which we'd like to shut down after about 5 minutes.
-If power returns before the batteries are exhausted then there is no OS interruption and the 4 non-prod boxes reboot
-If the batteries exhaust before power returns then have all hosts shut down gracefully and then reboot when power returns

Reality:
-Only 4 hosts gracefully shutdown (3 windows, 1 ESX server)
-When the batteries exhaust the others shut down unexpectedly and don't register anything in their event logs
-Also, the UPS shuts off and won't come back on until I flip the bypass switch, then back
-My boss is upset but wants me to learn all this with zero resources
-One of the hosts doesn't come back on until I push the power button (which is coincidentally one of the gracefully shut down servers)
-Finally, we used to receive email alerts from the UPS but that has stopped and we receive DNS errors
-Below are some log snippets and some of the configuration settings

**A note about the configs: The ESX box is the only one that's actually configured to shut down before battery exhaustion. We get about 25 - 30 minutes and I've got it set to shut down after 900 seconds (15 mins) which works perfectly. Also, all other servers have no events configured (including those that work as intended and those who do not). I have registered all servers with the NMC and PCNS is installed on all servers.

Loss of Power Log Events and UPS Shutdown, but never came back up
Apparently "Shutdown UPS" was checked on 192.168.26.39, but the server didn't shut down gracefully
12/07/2009 20:14:23 UPS: Communications established.
12/07/2009 20:14:10 System: Network service started. System IP is 192.168.26.89 from manually configured settings.
12/07/2009 20:14:10 System: Coldstart.
12/07/2009 19:34:09 UPS: Returned from sleep mode.
12/07/2009 19:31:13 UPS: Entered sleep mode.
12/07/2009 19:31:12 UPS: Communications established.
12/07/2009 19:30:59 System: Network service started. System IP is 192.168.26.89 from manually configured settings.
12/07/2009 19:30:59 System: Warmstart.
12/07/2009 19:19:08 UPS: Turned off.
12/07/2009 19:19:08 UPS: Returned from a Low-Battery condition.
12/07/2009 19:19:08 UPS: Bypass not in range cleared; either frequency or voltage.
12/07/2009 19:18:48 UPS: Started reboot sequence.
12/07/2009 19:17:55 UPS: Graceful shutdown by PowerChute ns from 192.168.26.39.
12/07/2009 19:17:54 UPS: Battery power is low and will soon be exhausted.
12/07/2009 18:51:39 UPS: Bypass not in range; either frequency or voltage.
12/07/2009 18:51:37 UPS: Switched to battery backup power.

Unable to email out, but used to. I CAN send a test email successfully when everything is working.
+(I changed our domain name to protect the innocent)+
12/07/2009 20:15:10 System: DNS Network Error, Socket call failed.
12/07/2009 20:14:37 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 20:14:36 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 20:14:23 UPS: Communications established.
12/07/2009 19:31:26 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 19:31:26 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 19:20:12 System: Email: Could not mail 'orion@domain.com'.

Symmetra Shutdown Parameters
Return Battery Capacity: 10%
Low-Battery Duration: 2 minutes
Maximum Shutdown Time: 2 minutes
Shutdown Delay: 20 seconds
Return Delay: 0 seconds
Sleep Time: 0.0 hours

Utility Line Settings
Output Frequency Range: 60 +/- 10 Hz
Shutdown Mode: Normal
If UPS fails, and frequency or voltage is out of range: Go to Bypass

A Few Questions
1. With non-configed servers why isn't it shutting them down with low battery signal?
2. If 1 server sends a UPS Shutdown signal to the UPS will that shut it down before all the hosts have gracefully shutdown?
3. Why isn't the UPS coming back on when power returns?
4. Who shot JFK?
5. Why do bad things happen to good people?
6. Does it just make sense to copy the config from the working servers over to the non-working servers?
7. I've read that as long as the hosts are on the same network (and they are) that PCNS doesn't even need to be installed on all of them. Is one way better? It's less than 50 hosts.
8. What would cause the emails to start failing when they worked before? No network changes have been made.
9. Am I making this far more difficult than I need to?
10. I've never actually done an off-hours test. What's the best/fastest way? I don't want to have to sit there for 30+ minutes waiting for the battery to discharge to test, especially if something doesn't go right.

I know this is a lot to ask of you guys in just one thread, but this is getting to be pretty crucial because we have been having power outages about twice a month for a few months now. Also, my boss is riding me like Seabiscuit to get this done, tested, and 100% working.

[Mgregory2K_apc · Accepted solution]

I've posted a couple questions, but I'm still having big time issues and I'm going to try and throw this all in one thread and hopefully get it figured out. I thank you in advance for your input.

I have:
Symmetra 40K w/ 16 batteries
~20 Windows Server 2003 boxes
1 ESX box
PowerChute Network Shutdown Installed on all servers
...and I'm learning all of this from scratch with no help from anyone except the forums

Goals:
-When we lose power to the building have all the servers stay up for the max amount of time, except for 4 non-production servers which we'd like to shut down after about 5 minutes.
-If power returns before the batteries are exhausted then there is no OS interruption and the 4 non-prod boxes reboot
-If the batteries exhaust before power returns then have all hosts shut down gracefully and then reboot when power returns

Reality:
-Only 4 hosts gracefully shutdown (3 windows, 1 ESX server)
-When the batteries exhaust the others shut down unexpectedly and don't register anything in their event logs
-Also, the UPS shuts off and won't come back on until I flip the bypass switch, then back
-My boss is upset but wants me to learn all this with zero resources
-One of the hosts doesn't come back on until I push the power button (which is coincidentally one of the gracefully shut down servers)
-Finally, we used to receive email alerts from the UPS but that has stopped and we receive DNS errors
-Below are some log snippets and some of the configuration settings

**A note about the configs: The ESX box is the only one that's actually configured to shut down before battery exhaustion. We get about 25 - 30 minutes and I've got it set to shut down after 900 seconds (15 mins) which works perfectly. Also, all other servers have no events configured (including those that work as intended and those who do not). I have registered all servers with the NMC and PCNS is installed on all servers.

Loss of Power Log Events and UPS Shutdown, but never came back up
Apparently "Shutdown UPS" was checked on 192.168.26.39, but the server didn't shut down gracefully
12/07/2009 20:14:23 UPS: Communications established.
12/07/2009 20:14:10 System: Network service started. System IP is 192.168.26.89 from manually configured settings.
12/07/2009 20:14:10 System: Coldstart.
12/07/2009 19:34:09 UPS: Returned from sleep mode.
12/07/2009 19:31:13 UPS: Entered sleep mode.
12/07/2009 19:31:12 UPS: Communications established.
12/07/2009 19:30:59 System: Network service started. System IP is 192.168.26.89 from manually configured settings.
12/07/2009 19:30:59 System: Warmstart.
12/07/2009 19:19:08 UPS: Turned off.
12/07/2009 19:19:08 UPS: Returned from a Low-Battery condition.
12/07/2009 19:19:08 UPS: Bypass not in range cleared; either frequency or voltage.
12/07/2009 19:18:48 UPS: Started reboot sequence.
12/07/2009 19:17:55 UPS: Graceful shutdown by PowerChute ns from 192.168.26.39.
12/07/2009 19:17:54 UPS: Battery power is low and will soon be exhausted.
12/07/2009 18:51:39 UPS: Bypass not in range; either frequency or voltage.
12/07/2009 18:51:37 UPS: Switched to battery backup power.

Unable to email out, but used to. I CAN send a test email successfully when everything is working.
+(I changed our domain name to protect the innocent)+
12/07/2009 20:15:10 System: DNS Network Error, Socket call failed.
12/07/2009 20:14:37 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 20:14:36 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 20:14:23 UPS: Communications established.
12/07/2009 19:31:26 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 19:31:26 System: Email: Could not resolve 'mail.domain.com'.
12/07/2009 19:20:12 System: Email: Could not mail 'orion@domain.com'.

Symmetra Shutdown Parameters
Return Battery Capacity: 10%
Low-Battery Duration: 2 minutes
Maximum Shutdown Time: 2 minutes
Shutdown Delay: 20 seconds
Return Delay: 0 seconds
Sleep Time: 0.0 hours

Utility Line Settings
Output Frequency Range: 60 +/- 10 Hz
Shutdown Mode: Normal
If UPS fails, and frequency or voltage is out of range: Go to Bypass

A Few Questions
1. With non-configed servers why isn't it shutting them down with low battery signal?
2. If 1 server sends a UPS Shutdown signal to the UPS will that shut it down before all the hosts have gracefully shutdown?
3. Why isn't the UPS coming back on when power returns?
4. Who shot JFK?
5. Why do bad things happen to good people?
6. Does it just make sense to copy the config from the working servers over to the non-working servers?
7. I've read that as long as the hosts are on the same network (and they are) that PCNS doesn't even need to be installed on all of them. Is one way better? It's less than 50 hosts.
8. What would cause the emails to start failing when they worked before? No network changes have been made.
9. Am I making this far more difficult than I need to?
10. I've never actually done an off-hours test. What's the best/fastest way? I don't want to have to sit there for 30+ minutes waiting for the battery to discharge to test, especially if something doesn't go right.

I know this is a lot to ask of you guys in just one thread, but this is getting to be pretty crucial because we have been having power outages about twice a month for a few months now. Also, my boss is riding me like Seabiscuit to get this done, tested, and 100% working.