Network Management Card notification issue with PCNS clients

[Anonymous user]

I have a multiple UPS configuration where only 1 of my 2 Symmetras is notifying PCNS clients of events. The NMCs are both AP9619 running the latest firmware (SFSY372). One of my Symmetras is sending notification fine to my clients as verified by the Eventlog.txt files on the servers (I have about 10 server running PCNS and all are seeing the same symptoms) and the other one is not. The initial communication seems fine. I get the standard "Network Management Card x.x.x.x communication has been established." for both Symmetras but when we lose power I only get events like "UPS x.x.x.x is running on battery power." from the one and not the other. It is always the same one that I get the event/notification from and not the other.

I believe I have them both configured identically as far as the event actions. Anyone have any ideas as to what is going on or anything I could try to resolve this? Thanks,

-john

[BillP]

i was more looking for the logs on the PCNS client itself - if you login to that interface and see what's in that event log.

[BillP]

can you paste the entire event log on the unit that is not working properly? also, what version of PCNS are you using? you can find that under help->about on PCNS itself.

usually things that cause a lack of communication are an authentication phrase mismatch or monitoring the wrong IP. you could double check those but the event log from PCNS may give me some more ideas.

[Anonymous user]

I'm using a mix of 2.2.4 and 3.0.0 of PCNS. Here is all of the event logs I have from the problem UPS. It must be recycling after a couple weeks.

09/20/2011 10:25:25 System: Web user 'apc' logged in from 100.100.100.76.
09/20/2011 10:17:43 System: Web user 'apc' logged out from 100.100.100.76.
09/20/2011 10:07:43 System: Web user 'apc' logged in from 100.100.100.76.
09/20/2011 10:07:33 System: Web user 'apc' logged out from 100.100.100.76.
09/20/2011 09:55:06 System: Web user 'apc' logged in from 100.100.100.76.
09/19/2011 23:51:27 UPS: Passed a self-test.
09/19/2011 23:50:34 UPS: Started a self-test.
09/19/2011 15:21:19 System: Web user 'apc' logged out from 100.100.100.76.
09/19/2011 15:06:53 System: Web user 'apc' logged in from 100.100.100.76.
09/19/2011 15:01:41 System: Web user 'apc' logged out from 100.100.100.76.
09/19/2011 14:42:26 System: Web user 'apc' logged in from 100.100.100.76.
09/19/2011 14:30:22 UPS: No longer on battery power.
09/19/2011 14:23:52 System: Set Daylight Saving Time.
09/19/2011 14:23:52 System: Set Date or Time.
09/19/2011 14:04:29 UPS: On battery power in response to an input power problem.
09/16/2011 14:23:36 System: Set Daylight Saving Time.
09/16/2011 14:23:36 System: Set Date or Time.
09/13/2011 14:23:17 System: Set Daylight Saving Time.
09/13/2011 14:23:17 System: Set Date or Time.
09/10/2011 14:22:59 System: Set Daylight Saving Time.
09/10/2011 14:22:59 System: Set Date or Time.
09/07/2011 14:22:42 System: Set Daylight Saving Time.
09/07/2011 14:22:42 System: Set Date or Time.

[Anonymous user]

I grabbed a copy of the EventLog.txt from one of my Windows boxes and attached it rather than paste since it is rather long. Looking through it reminded me that it was working ok as recently as June of this year. But during our last 3 power outages it has failed to send "UPS 100.100.100.147 is running on battery power." events.

-john

[BillP]

ohh ok.

so let me make sure i understand this correctly. you have all of your PCNS clients looking at two UPSs for redundancy? i see you did mention two UPSs but did not think you had the clients looking at them. thought they were separate. my mistake.

i misunderstood and thought you had one UPS with a few clients and one client was not talking with your one UPS.

i did see this:

09/13/2011 13:49:04 Management Card 100.100.100.147 communication has been established
09/13/2011 13:49:04 Communications have been established
09/19/2011 14:04:26 UPS 100.100.100.150 is running on battery power.
09/19/2011 14:04:26 UPSs have switched to battery power
09/19/2011 14:30:23 Input power has been restored: One or more UPS(s) has switched to On Line operation
09/19/2011 14:30:23 UPS 100.100.100.150 is running on battery power.
09/19/2011 14:30:23 The On Battery UPS has switched to On Line operation

to me, it seems like this is working? it indicates the UPSs have switched to battery. it seems to not be using the IP address for some reason like it was earlier. at the same time on your UPS log, .147 was on battery when we see *09/19/2011 14:04:26 UPSs have switched to battery power* on your PCNS client log.

have you tried generating any other specific events to see if they pop up - such as going to manual bypass or temporary bypass to see if that registers? perhaps try removing a battery module which might be a little less risky if you are not sure of your power situation or UPS condition.

if that does not work, i will ask around and see if we have any other ideas.

[Anonymous user]

Yes, all PCNS clients look to both Symmetras (100.100.100.150 and 100.100.100.147) as configured per the "Multiple-UPS configuration" described in the documentation.

I tried your idea of removing a battery and reinserting it. I tried it on both Symmetras and I received events from both on my server. You're right, there is some communication occuring there. It is very strange though that it doesn't list the 100.100.100.147 ip address of the 2nd Symmetra and instead gives the more generic "UPs have switched to battery power". The power outage the other night was about 5 minutes too short to hit my thresholds for initiating shutdowns on the servers so I can't use that to verify.

Could you verify that it not listing the ip addresses for both Symmetras is normal? Also, do you know of a simple test I could do to see if my servers are really going to be able to shutdown when needed. Is there some type of shutdown action that I can trigger by something benign such as removing a battery? I don't want to just drop the network connection to the Symmetras as a test since that doesn't test the notifcation event procedure. That would just be the PCNS software deciding on its own to take action since it can't communicate with the Symmetras.

[BillP]

hi john. i did a little research. can you send me a log from the 3.0.0 PCNS system that displays the same issue? i found a similar issue noted but it said it was fixed in 3.0. i'd like to see if there is any difference in the reporting to see if this is the same issue or a different one than you have. that also does not explain why it worked properly in 2.2.4 earlier in the log 😞

also, to do some testing, if you go to configure events within PCNS on the left hand side, these are the events you can initiate shutdowns on (if you see an option in the shutdown column). one of those might work. also, it is possible to reboot a UPS or turn off a UPS and signal a powerchute client at the same time. we are working on a possible test feature to make this easier for a future NMC release on NMC2 devices.

hope that helps.

[Anonymous user]

Thanks for replicating my environment and doing that testing! It is very good to know that the behavior I'm seeing is normal operation. I agree that it is confusing wording and would like it to list each UPS separately with its ip address.

Thanks also for clearing up the mystery of the missing events on my 3.0.0 installs.

I noticed that 2.2.4 still has a configurable event for "battery removed". I think I will try using that event to test out my auto shutdown on a 2.2.4 server. I may have to use something like "Environment: Temperature Probe 1 Out Of Range" or something on my 3.0.0 servers to test them.

Thanks again for all your help!!

[Anonymous user]

Here are the logs from one of my 3.0.0 systems. Seems to be the same in regards to the on battery messages. Another strange thing is this one doesn't have the battery removed/battery installed messages from the other day like the 2.2.4 one did.

09/06/2011 17:49:32 PowerChute Network Shutdown version 3.0.0 started .3.5.1.5.6.10
09/06/2011 17:53:27 Network Management Card 100.100.100.150 communication has been established. .3.5.1.5.6.13
09/06/2011 17:53:28 Communication established with Environmental Monitoring Card .3.5.1.7.20
09/06/2011 17:53:41 Network Management Card 100.100.100.147 communication has been established. .3.5.1.5.6.13
09/06/2011 17:53:41 Communications have been established. .3.5.1.5.6.1
09/06/2011 17:53:41 Communication established with Environmental Monitoring Card .3.5.1.7.20
09/19/2011 14:04:31 UPS 100.100.100.150 is running on battery power. .3.5.1.5.12.3
09/19/2011 14:04:31 UPSs have switched to battery power. .3.5.1.5.4.1
09/19/2011 14:30:24 Input power has been restored: One or more UPS(s) has switched to On Line operation. .3.5.1.5.4.2
09/20/2011 18:11:09 Network Management Card 100.100.100.150 communication has been established. .3.5.1.5.6.13
09/20/2011 18:11:09 Communication established with Environmental Monitoring Card .3.5.1.7.20
09/20/2011 18:11:26 PowerChute Network Shutdown version 3.0.0 started .3.5.1.5.6.10
09/20/2011 18:11:29 Network Management Card 100.100.100.147 communication has been established. .3.5.1.5.6.13
09/20/2011 18:11:29 Communications have been established. .3.5.1.5.6.1
09/20/2011 18:11:29 Communication established with Environmental Monitoring Card .3.5.1.7.20

[BillP · Accepted solution]

hmm ok. i was able to have someone do some testing on this for us. it appears this is normal operation. we tried cutting power to the units, separately and at the same time and it seems to log the IP of which NMC/UPS that it gets an on battery event from first over the network. so, which ever client gets the on battery packet first shows its IP in the log and then you get the event of UPSs on battery when the other one is also on battery too. when you bring them back online, it sort of does the reverse order in the log as far as which unit came online, where it just says one unit came online and then it indicates the opposite unit's IP is still on battery and then comes back online, pretty much at the same second.

this is very confusing so i will see what we can do about this in a future release as far as the logging goes.

did i make sense though?

also, i found out that they decreased the amount of logged events (like the battery modules inserted/removed) in PCNS 3.0.0. it is listed in the help file which events you can expect.

[BillP]

no problem with the testing and yes, thats a good idea - with the temp probe OR you can configure it to shutdown while the UPS is on battery for 60 seconds. but, with the environmental monitoring, you should be good to go per -> http://nam-en.apc.com/app/answers/detail/a_id/3143