PCNS constantly shutting down server

[Anonymous user]

Hello,
My boss restarted a network card on one of our servers yesterday, which resulted in PCNS shutting down the server the "Communication: PowerChute cannot communicate with the Management Card" event was set to shutdown after 180 seconds. I can understand this happening the first time as it may be that it genuinely took that long and the network card had not re-initialised.

The event log reported:
03/15/2011 13:23:23 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 13:25:23 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 13:25:23 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 13:25:23 PowerChute cannot communicate with the Management Cards
03/15/2011 13:28:23 Shutdown process started. Windows will shut down soon

Repeatedly (about 30 times)

The problem was however that the server kept on shutting down every time it was switched back on even though the network card was back up (my colleague was connecting the the server remotely using terminal services using the same LAN card so it definately initialised for that to happen). However he was only able to spend a few seconds on the server to fault find before it shut down again and in the end it took several hours before they realised that PCNS was to blame. The issue eventually resolved itself but is this something that others have experienced? The server in question is a primary backup server and as such it is important that downtime is minimal and we cannot really have this same scenario happen again. (I have increased the shutdown delay to 600 seconds in any case to give us more time to react if this does happen in future).

Thanks in advance for any advise.
Allan

[BillP]

hi allan,

can you check the event.txt from the NMC at the time that this happened?

you can get it via FTP via these instructions -> http://nam-en.apc.com/app/answers/detail/a_id/9321/

also, can you post any more of the PCNS log besides what you posted?

thanks.

[Anonymous user]

Thanks for your responce, apart from showing me that the date/SMTP server name were configured incorrectly (all events are out by a year and several reports of SMTP server resolution failed) the event.txt that I downloaded does not show much, Only events to happen on that day are - Started Self Test, failed to resolve SMTPservername, Passed Self Test.

Below are all of the events from the PCNS log that occured on the day:

03/15/2011 13:51:25 Communications have been established
03/15/2011 13:51:25 Management Card birmups3.crawco.co.uk communication has been established
03/15/2011 13:51:08 Management Card birmups4.crawco.co.uk communication has been established
03/15/2011 13:50:38 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 13:37:46 Communications have been established
03/15/2011 13:37:46 Management Card birmups3.crawco.co.uk communication has been established
03/15/2011 13:37:30 Management Card birmups4.crawco.co.uk communication has been established
03/15/2011 13:37:12 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 13:28:23 Shutdown process started. Windows will shut down soon.
03/15/2011 13:25:23 PowerChute cannot communicate with the Management Cards
03/15/2011 13:25:23 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 13:25:23 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 13:23:23 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 13:10:09 Shutdown process started. Windows will shut down soon.
03/15/2011 13:07:09 PowerChute cannot communicate with the Management Cards
03/15/2011 13:07:09 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 13:07:09 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 13:05:09 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 12:56:54 PowerChute cannot communicate with the Management Cards
03/15/2011 12:56:54 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 12:56:54 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 12:54:55 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 12:45:44 Shutdown process started. Windows will shut down soon.
03/15/2011 12:42:44 PowerChute cannot communicate with the Management Cards
03/15/2011 12:42:44 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 12:42:44 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 12:40:44 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 12:32:07 Shutdown process started. Windows will shut down soon.
03/15/2011 12:29:07 PowerChute cannot communicate with the Management Cards
03/15/2011 12:29:07 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 12:29:07 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 12:27:07 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 12:18:28 Shutdown process started. Windows will shut down soon.
03/15/2011 12:15:27 PowerChute cannot communicate with the Management Cards
03/15/2011 12:15:27 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 12:15:27 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 12:13:27 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 12:02:35 Shutdown process started. Windows will shut down soon.
03/15/2011 11:59:35 PowerChute cannot communicate with the Management Cards
03/15/2011 11:59:35 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 11:59:35 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 11:57:35 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 11:48:15 Shutdown process started. Windows will shut down soon.
03/15/2011 11:45:15 PowerChute cannot communicate with the Management Cards
03/15/2011 11:45:15 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 11:45:15 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 11:43:15 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 11:22:57 Shutdown process started. Windows will shut down soon.
03/15/2011 11:19:57 PowerChute cannot communicate with the Management Cards
03/15/2011 11:19:57 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 11:19:57 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 11:17:57 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 11:07:34 Shutdown process started. Windows will shut down soon.
03/15/2011 11:04:34 PowerChute cannot communicate with the Management Cards
03/15/2011 11:04:34 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 11:04:34 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 11:02:34 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 10:52:01 Shutdown process started. Windows will shut down soon.
03/15/2011 10:49:01 PowerChute cannot communicate with the Management Cards
03/15/2011 10:49:01 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 10:49:01 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 10:47:01 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 10:38:35 Shutdown process started. Windows will shut down soon.
03/15/2011 10:35:35 PowerChute cannot communicate with the Management Cards
03/15/2011 10:35:35 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 10:35:35 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 10:33:35 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 10:23:26 Shutdown process started. Windows will shut down soon.
03/15/2011 10:20:26 PowerChute cannot communicate with the Management Cards
03/15/2011 10:20:26 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 10:20:26 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 10:18:26 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 10:10:06 Shutdown process started. Windows will shut down soon.
03/15/2011 10:07:06 PowerChute cannot communicate with the Management Cards
03/15/2011 10:07:06 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 10:07:06 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 10:05:06 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 09:57:24 Shutdown process started. Windows will shut down soon.
03/15/2011 09:54:24 PowerChute cannot communicate with the Management Cards
03/15/2011 09:54:24 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 09:54:24 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 09:52:24 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 09:42:21 Shutdown process started. Windows will shut down soon.
03/15/2011 09:39:21 PowerChute cannot communicate with the Management Cards
03/15/2011 09:39:21 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 09:39:21 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 09:37:22 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 09:21:26 Shutdown process started. Windows will shut down soon.
03/15/2011 09:18:27 PowerChute cannot communicate with the Management Cards
03/15/2011 09:18:26 Network Communication with address birmups4.crawco.co.uk has been lost
03/15/2011 09:18:26 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 09:16:27 PowerChute Network Shutdown version 2.2.4 started
03/15/2011 08:36:45 Shutdown process started. Windows will shut down soon.
03/15/2011 08:33:44 PowerChute cannot communicate with the Management Cards
03/15/2011 08:33:44 Network Communication with address birmups3.crawco.co.uk has been lost
03/15/2011 08:33:28 Network Communication with address birmups4.crawco.co.uk has been lost

At 08.30 is when my manager re-initialised the LAN card on the server, so we can accept that it would have lost connection and possibly rebooted at that time.

Hope this helps - thanks, Allan

[BillP · Accepted solution]

well this definitely just shows that PCNS could not communicate with the NMCs. my only thought is somehow the NMC got stuck in a certain state or lost communication with the UPS or something that caused the issue you saw. i cannot really say much more without seeing an event.txt from the NMC. if it is spammed out with the events you mentioned, then perhaps we will just have to guess at what happened.

we definitely dont have any known issues of this sort so if it happens again for some reason, ensure that you get the PCNS logs and the event.txt from the NMC to review.