timezone reset issue with network management cards ap9630/1

[BillP]

I am having a problem with my ap9630 and ap9631 network management cards resetting their timezone settings automatically back to -12:00 hours Eniwetok, Kwajalein after I switch them to -7:00 hours Arizona, Mountain Time after less than a week. I am using NTP settings (updates every 168 hours), but, since the timezone appears to be independent of NTP, all NTP does is end up setting the time and day based on the timezone of the card, not of where I am. The cards always seem to reconfigure themselves before an NTP update, but, as I said, that doesn't seem to matter since NTP doesn't affect the timezone, just the current date and time. The setting isn't configurable through my Infrastruxure Central server, so I can't use it to periodically correct the timezone, and, since I have over 30 of these cards, I'm not about to go into each one all the time to fix this. This is very frustrating since it messes up the date and time stamps of the logs, which makes troubleshooting difficult when I'm trying to reconcile the time of an email notification and/or notification from my server to figure out problems. With this issue, the "default" timezone makes it two days and one hour into the future from my actual time!

The firmware of the cards is as follows:

Application Module
Name: sumx
Version: v5.1.7
Date: Dec 1 2011
Time: 13:01:45

APC OS (AOS)
Name: aos
Version: v5.1.7
Date: Nov 22 2011
Time: 09:53:57

APC Boot Monitor
Name: bootmon
Version: v1.0.2
Date: Jan 21 2010
Time: 13:35:57

Is anyone else experiencing this and/or have any idea how to fix this? My older ap9617/8/9 cards don't have this issue (over 20 of them; same NTP settings); firmware as follows:

Application Module
Name: sumx
Version: v3.7.2
Date: 02/02/2010
Time: 19:08:16

APC OS (AOS)
Name: aos
Version: v3.7.3
Date: 02/02/2010
Time: 17:05:14

Any help would be appreciated!

Message was edited by: vhinckle

[BillP]

I am having a problem with my ap9630 and ap9631 network management cards resetting their timezone settings automatically back to -12:00 hours Eniwetok, Kwajalein after I switch them to -7:00 hours Arizona, Mountain Time after less than a week. I am using NTP settings (updates every 168 hours), but, since the timezone appears to be independent of NTP, all NTP does is end up setting the time and day based on the timezone of the card, not of where I am. The cards always seem to reconfigure themselves before an NTP update, but, as I said, that doesn't seem to matter since NTP doesn't affect the timezone, just the current date and time. The setting isn't configurable through my Infrastruxure Central server, so I can't use it to periodically correct the timezone, and, since I have over 30 of these cards, I'm not about to go into each one all the time to fix this. This is very frustrating since it messes up the date and time stamps of the logs, which makes troubleshooting difficult when I'm trying to reconcile the time of an email notification and/or notification from my server to figure out problems. With this issue, the "default" timezone makes it two days and one hour into the future from my actual time!

The firmware of the cards is as follows:

Application Module
Name: sumx
Version: v5.1.7
Date: Dec 1 2011
Time: 13:01:45

APC OS (AOS)
Name: aos
Version: v5.1.7
Date: Nov 22 2011
Time: 09:53:57

APC Boot Monitor
Name: bootmon
Version: v1.0.2
Date: Jan 21 2010
Time: 13:35:57

Is anyone else experiencing this and/or have any idea how to fix this? My older ap9617/8/9 cards don't have this issue (over 20 of them; same NTP settings); firmware as follows:

Application Module
Name: sumx
Version: v3.7.2
Date: 02/02/2010
Time: 19:08:16

APC OS (AOS)
Name: aos
Version: v3.7.3
Date: 02/02/2010
Time: 17:05:14

Any help would be appreciated!

Message was edited by: vhinckle

[BillP]

Just to close this out if anyone else comes across it, we were able to figure out that an incorrect timezone offset being provided by DHCP option 2 on the customer's end was the root problem. The value from the server was incorrect and invalid and causing the management card to react to it and basically recalculate it the best we could. Once fixed on the DHCP server end, everything time-wise is working properly. Because of differences in how our older AP9617/18/19 works compared to newer AP9630/31/35 cards work, that is why behavior was different between the two.

[BillP]

still no dhcp lease renewal emails...

if I reboot a card, I still get those false "outlet group restarted" messages, but they don't post to the card's log...the renewal message posted in the log, but I never received an email for that message...

[BillP]

still the same regarding dhcp lease renewal emails...

fully migrated to win 2012 dhcp servers, and still the same problem 😞

I now have two cards that I cannot assign snmpv3 passwords to...seems if I reset a card, use of snmpv3 passwords fails...:(

still working towards getting my server upgraded...

[BillP]

Looks like using the ISX server didn't help...the unit still did a config change before the NTP update occured (see screenshots of the email received and the test unit's time settings).

[BillP]

ok, keep us updated. i will be on vacation next week before Thanksgiving holiday but I'll try to check back. but it seems its just too early to tell if anything is different.

if it does change again, we can easily do some testing to verify a problem.

[BillP]

can you provide event.txt and config.ini files from one or two of these cards? instructions are @ http://www.apc.com/site/support/index.cfm/faq/ and refer to FA156131.

thanks!

[BillP]

The two zip files contain the files from the two of the types of cards. When you look at the event.txt file from the ap9631.zip file, note the date and time stamps from 10/28/2012 back to 10/26/2012 and from 10/31/2012 to 11/02/2012 - the Configuration change on 10/31/2012 was when it flipped back to the default timezone after I had changed it on 10/26/2012.

if you want files from more cards, let me know; I have plenty I can get files from!

Message was edited by: vhinckle

[BillP]

just curious - is it at all possible to do a packet capture during an NTP update? I assume it does it (turns back the time) if you force an update versus the automatic update? also, by what you said, the NTP server is an ISX Central?

FYI, i will be traveling all day tomorrow so i probably cannot post back for at least a day or two. i have not seen this type of issue on 5.1.7 before..

[BillP]

I am not using the ISX server as an NTP server; the devices get their NTP server settings from the network and are two other servers (both are Windows DHCP/DNS servers; I think either 2008 R2 and/or 2012). I could try setting up the ISX server as an NTP server and set one of the cards to manually get it's NTP update from the ISX server as a test, if you think it will help.

If I force a timezone change followed by (or at the same time) an NTP update, the result is I get the correct day and time. This does reset the NTP update timer. However, the cards usually do a configuration change back to the wrong timezone before the next NTP update occurs, after which the cards consistently update using the wrong timezone.

I'm not certain if I'll be able to do a packet sniff on the network - the network here is heavily routed and secured. I'll talk with my colleagues about the possibility of doing so and post the results if I'm able.

[BillP]

ok. just curious. also, are these NMCs on the public or private side of central? I am not going to rule out some type of something coming from the ISX central either. i think it might be a good test though if you could temporarily reset the NTP server to the ISX central.

[BillP]

All of the units are on the Public side of the ISX server and none of them are in the same network subnet/Vlan as the server - almost all of them are in separate buildings from where the server sits.

I have setup the ISX server as an NTP server now, and manually set one of the ap9631 cards up to receive NTP information from the ISX server first. So far, I have not received a "configuration change" email from the unit indicating a reset of the timezone, but it may be early yet - the next NTP update won't be until next Monday.

PS. Something that I noticed when applying the changes: with the ISX server, before I changed the timezone, when I manually forced an update, I at least got the correct date to apply. With the network NTP servers, if the timezone was wrong, the date always ended up two days in the future. I still had to change the timezone, however, to get the correct time to apply.

Message was edited by: vhinckle

[BillP]

ok, i will see about getting it set up while i am out on vacation this coming week or i would just do it. hope you can bare with me. i find it odd we have not seen this before when it seems like a clear issue. i searched for bug logs and one question I have is - what OS is the NTP server being provided by? just curious. I know ISX Central is Linux on the backend and then our own client on top of it but i figured i'd ask.

just curious - are you able to try and do a format on the NMC (for one of these cards?) I am not sure what has happened previously or any upgrades you've done but to rule out any corruption, i might try it one of the cards if it were me. a format wipes EVERYTHING, including TCP/IP settings so it'd require a reconfiguration. you could back up the config.ini file at least first and then configure the IP and just push that back out to it if you wanted. you can do a format via telnet/ssh 'format' command or holding the NMC reset button for 20+ seconds (which will pulse green then change to orange when its time to let go).

in the meantime, i will see about getting it set up while i'm out.

[BillP]

what version of ISX Central are you using? let me see what i can find out about that. I am not a Central expert but I will see what we need to check to find the root cause.

edit all i found out so far is that we would make sure the community strings are set up properly but i am figuring you double checked that. the only other thing I see is SNMPv1 is enabled now and SNMPv3 was disabled previously. does it work if you disable SNMPv1? (the card will need to reboot after that). does the card have the same IP address that it used to?

[BillP]

the version of ISX Central is 6.0.2 (yes, I know it's old)

in the config files I attached, you're seeing that SNMPv3 is disabled? that doesn't make sense. at the moment, when I log into the card, I see both versions enabled. I did have version 1 disabled prior to the format since, for security purposes, I shouldn't be using ver 1; I only have it enabled now since I can't get ver 3 to stay connected to my server. it is really odd, though...when I try to connect using ver 3, the server will connect long enough to register the device, but then can't communicate after...

the device does have the same IP address as before. If I disable snmpv1, my server can't communicate at all with the device (the server doesn't allow two types of connections to one device - with snmpv3 not working, I end up without a means of communication when snmpv1 is turned off)

Message was edited by: vhinckle

[BillP]

sorry, no, i meant in one config I saw SNMPv1 disabled and one SNMPv1 enabled (which i believe is the current one).

anyway, you clarified my question anyway. so i guess i misunderstood previously. the device does get discovered but then loses communication. just curious, are you able to query the management card outside of central via SNMPv3 when this happens (like with a MIB browser)? can you try SNMPv3 without authentication/privacy to see if that works?

i have to check on the card we had set up for NTP while i was away and see what's going on with that as well.

[BillP]

so...no auth/no privacy results in no discovery at all with the server. With another MIB browser, no auth/no privacy snmpv3 gets me in; auth and privacy fails - discovery times out.



also just noticed timezone reset on it's own again...

Message was edited by: vhinckle

[BillP]

ok, still looking into that central issue.

Edit - on this problem, did you change the discovery/polling options to match what you used on the card? i am guessing there is some type of setting there that may require changing if you tried to discover the device with no auth/privacy and maybe central requires that? just have to make sure that of course the SNMP settings match on the card and the central. and yes, what i got told is "that is old!" lol.

on the NTP problem, my card has been set up to update NTP every hour and it was set to -7 hours as yours was. i have not seen any issues. now i am going to change the interval to the same as yours (168 hours).

edit just noticed something. in your config file, I see:

NTPManualOverride=enabled

Selection of Override Manual NTP Settings will result in configuration data from other sources (typically DHCP) taking precedence over the manual configurations set here (on the date and time page).

if it was being changed by DHCP, i would suspect that a.) you'd see the IP address it changed to and b.) see this event in your log: System: Configuration change. Primary NTP server address.

i am just going to ask though - does your DHCP server provide NTP server settings at all in addition to the TCP/IP settings? can you try changing this to disabled and see if it makes a difference? i noticed that is one thing different between our two configurations.

[BillP]

For the device discovery,I tried the simplest testing I could. I just added a Smart-UPS running sumx 5.1.7 to my 7.2 Central. first, I disabled SNMP v1 enabled SNMP v3.

I used apcapcapcapcapcapc for all 3 User Name, Authentication Passphrase, and Privacy Passphrase.

Both Authentication Protocol and Privacy Protocol are set to none.

Under access control, specifically enabled the apcapcapcapcapcapc user with 0.0.0.0 for nms IP.

Created a discovery with those settings and ran the discovery. The unit discovered in seconds.

[BillP]

With one of my other systems (not the one that was formatted), I did try it with both manual NTP settings (ISX server only) and the settings received via DHCP. I have no control over the DHCP servers sending out NTP settings (many of our servers and switches rely on that feature). Whether I used manual override or not, it didn't make any difference - the unit still reset the timezone on its own. The length of the NTP renew time, I believe, comes from the DHCP servers since I've never specifically chosen the time out. I don't think the DHCP server provides timezone information, however, since I do have switches that still use their default timezone settings.

For the formatted card...the server's device discovery does request that I choose the settings that will match my unit. Each initial discovery can be customized per unit, if I wanted - it doesn't force me to use the same settings for each device. Usernames and/or password settings can be changed after the device has been discovered, but, a change between snmpv1 and snmpv3 does require I "remove" the device from the server and do a rediscovery with the change in version and applicable security settings. The attached image shows the device discovery screen for an snmpv3 discovery - the authentication and encryption type drop-down lists each include an option for "None".

[BillP]

you're right - the NTP protocol provides time in GMT so then the device should be using a timezone offset to calculate and display the correct time. i believe the default NTP update interval is 168 hours from my experience.

i forget (and also double checked above) have you tried to change the auto update to every hour (like we did) and see if there was a problem? I know when you forced an NTP update, you said it did not cause a problem. that would be the last thing i'd check.

once i know what the days, i will see if anyone can look at it from the firmware side since i cannot replicate.

also, let me go back to the central team to see what they suggest for this device that will not maintain communication since I am just not sure what is going on there.

[BillP]

a shot in the dark about the formatted card...is it possible, since it worked before formatting, that the software and/or firmware on the card has become corrupted?

I will try reducing the NTP refresh on the other card and let you know.

edit: in checking the other snmp program again, I got the attached error message...the settings for no auth/no privacy seemed to work yesterday with the program...

Message was edited by: vhinckle

[BillP]

I've tried simple settings....the only set of simple settings that work with this particular unit are SNMPv1 settings...

I've tried the following combinations of SNMPv3 settings:

username no auth/no privacy - fails to discover
username auth/no privacy - discovers briefly, then loses communication
username auth/privacy - discovers briefly, then loses communication

Weird point - if the device is discovered by my server, if I go to the device scan settings screen to try and test other settings, I cannot get the "NONE" option to stick to the Authentication type - it fails back to whichever type I had used to obtain discovery
Another weird thing...I only get the partial success if the username and password settings I'm testing are applied to the first access control option. It doesn't seem to matter whether I assign the server's address or not.

Thing is, the card worked fine before I formatted the card. The other 60+ units work fine with SNMPv3 settings of username with auth and privacy on, SNMPv1 turned off.

edit: I don't know if it matters, but, when I change the access control settings, the events logged look a little...odd:

11/29/201217:35:25System: Configuration change. SNMP access control *%$1ld* access address.
11/29/2012 17:35:24 System: Configuration change. SNMP access control *%$1ld* user name.

Message was edited by: vhinckle

[BillP]

i asked Steve (smarchet) about this just a few minutes ago. is it possible to try adding this one device to the private side of central to see if that works?

another suggestion was to try rebooting the server if you had not tried that yet. we only had newer versions of central to try or 5.1.1 and could not replicate any of it using the configurations you have tried.

[BillP]

so, rebooted the server (had to in order to turn on the private LAN anyway), and still no success with the device. nmc2's are backwards compatible with older smart-upses, right? if so, I do have one unit near the server I could put the card in temporarily to test the private LAN (though, the unit it belongs in will end up unmonitored during the test since I don't have a spare nmc2).

[BillP]

okay....tried again with passwords...failure....tried no passwords and....SUCCESS

I have no answer to why no passwords is working now.

...if my passwords are, somehow, bad, shouldn't I have been getting invalid access attempt messages from the unit?


EDIT: are there any restrictions regarding length and/or characters for MD5 and DES passwords?

Message was edited by: vhinckle

[BillP]

I don't have the same version of Central that you're running (I believe that was 6.0.2) but I found an old 5.1.1 box, I have a 6.3 box, and my initial testing was on a 7.2 box. With the settings I outlined previously, I got the same unit to be discovered by all 3 systems. Before testing on the other systems, I was going to suggest upgrading as you're a number of revisions behind (still wouldn't be a bad idea) but since it seems to work everywhere I tried it, I would think the version is not part of the issue.

Have you tried the exact same things I did step by step? Disabling SNMP v1...using a very simple user and passwords (maybe the exact same one)?

Another idea may be to take your network out of the picture. Put this device on the private LAN of Central. Leave the NTP issue out of the equation for this test.

I would also suggest discovering the unit each time you make changes. Don't make a change on the card and then change the configuration in Central, simply delete the unit and re-add it with a new discovery.

Message was edited by: smarchet

Did not see your last post stating that you got it working. I have seen anomalies where if you change the settings instead of totally recreating the entries (especially on older Central versions) they may not stick. This may have been part of the issue.

[BillP]

There are limits to the lengths. Here's from the card's help about the Authentication passphrade:
A phrase of 15 to 32 ASCII characters that verifies that the Network Management System (NMS) communicating with this device through SNMPv3 is the NMS it claims to be, that the message has not been changed during transmission, and that the message was communicated in a timely manner, indicating that it was not delayed and that it was not copied and sent again later at an inappropriate time.


This for the privacy passphrase:
A phrase of 15 to 32 ASCII characters that ensures the privacy of the data (by means of encryption) that a Network Management System (NMS) is sending to this device or receiving from this device through SNMPv3.

[BillP]

so, you're right about settings not sticking when using the device settings screen (it is a 6.0.2 server - really going to have to talk to boss about upgrading :P)

turned off snmpv1

retested strictly with new discovery each time....even with the simplest password for both auth and privacy (0123456789012345), if I have auth on (either MD5 or SHA), I only get the momentary connection when using snmpv3....no auth/no privacy, it connects.

if a nmc2 will work in an older smart-ups, I can move the card to connect it to the server's private LAN for the test; if not, I may have to get really creative in order to get a connection between the server and the card. I presume, on the server side, I would want the DHCP auto-discovery via snmpv1 left off for the test? Will the server assign an address in order of available IPs? I currently don't use the private LAN, so I'm not certain what to expect.

[BillP]

the private LAN can do DHCP or static addressing so its up to you. also, yes, NMC2 will work in any older style Smart UPS that begins with a model prefix SU or SUA or SUM. if it has a smart slot, it should work.

[BillP]

okay, moved the card and plugged it into the private LAN of the server. Pinged the private address from the server successfully (192.168.1.2). Created a new discovery using the same SNMPv3 settings that were successful before moving it (no auth/no privacy) and could not get even a glimmer of discovery from the server.

there isn't a special way to discover devices using snmpv3 on the private LAN, is there?

Message was edited by: vhinckle

[BillP]

i'm still investigating here..my card is indeed using a manual address so it might explain why i haven't seen anything. BUT if the setting for NTP override is DISABLED, then this should not occur (if its the problem). AP9617/18/19 don't have that option since we introduced it in NMC2.

if i use a DHCP server as well, i don't have access to it either to do any testing. someone else here though might so i will see about that as well.

how often does the lease renew again? i am still interested in a packet capture 😉

Edit - two more questions - 1.) are the DHCP servers located in the same timezone -7? and 2.) are your DHCP servers utilizing option 2 (NTP offset) for the leases it gives out? we have information on the DHCP options listed in FA156110 for reference in our knowledge base @ www.apc.com/site/support/index.cfm/faq/ and option 2 relates to time offset.

it is possible NMC1's are ignoring or not accepting this for some reason but NMC2's are using it.

[BillP]

looks like setting NTP updates to one hour didn't help:

12/02/2012 09:52:47 System: NTP update successful.
*12/02/2012 09:52:18 System: Configuration change.*
11/30/2012 08:05:22 System: NTP update successful.

There was approximately half an hour between the last "good" update and the configuration change. That the second NTP update occurred immediately after the configuration change, even though it hadn't been an hour since the previous update, seems odd to me, unless the configuration change forced an update...?

[BillP]

...the dhcp lease acquired time corresponds to the time of the timezone configuration change...coincidence? If it isn't a coincidence, any ideas as to how to solve this? I don't control the DHCP servers, so I can't make changes to them. If the DHCP servers are the problem, why don't they affect my older cards (ap9617/8/9 v3.7.2) the same way? I'll take a look at some additional cards to see if this is just a coincidence for the one card, or if it is the "smoking gun" we've been looking for.

[BillP]

the lease renews are variable - for the six I am watching, the renews range from 8 hours to 7 days. Most of the units are getting their addresses from one server (we have two). Both servers are local; I will need to ask about the actual settings from the server admin, unless that is something I can check from a Windows Management Console...? When I check the scope options, I do see a 002 Time Offset option with a value of 0x25200 - the setting is apparently coming from the server, since it isn't editable from the scope options screen.

I have yet to try a strict Manual system time config...at least I have plenty of cards to test with 😛

if dhcp is the cause, at least a few of my test cards will be renewing their leases over the weekend, so I should be able to provide more results (also, the strictly manual card has a short lease time, so I should be able to provide a report on it as well).

I'll see if I can talk with our security person about a packet capture next week 🙂

[BillP]

thanks for that information.

i was googling and came across http://www.cisco.com/en/US/tech/tk86/tk804/technologies_tech_note09186a0080093d76.shtml which seems to indicate that a +/- 7 timezone is is +/-25200 seconds..

[BillP]

so, email notification didn't occur as expected, so I couldn't confirm changes from the weekend's updates :S hopefully have the email working this time, so, by end of today or tomorrow should have some confirmable results. out of the seven test systems, the one I could confirm did reset the timezone when it's dhcp lease renewed. it also looked like the one I set to use manual settings reset both it's timezone and reverted to using NTP settings at the time of lease renewal, but I am testing again to confirm that I didn't set it back to ntp myself.

[BillP]

so you're thinking maybe the DHCP server provided NTP server settings and switched the time mode somehow? i wasn't 100% sure if DHCP could switch the time mode versus just providing server IPs for the DHCP clients.

[BillP]

apparently I wasn't imagining things, but, the timing was off - the one with the manual settings reverted to ntp ~4 hours before the lease was renewed. The other two that had really short DHCP lease times also changed timezones before their leases expired (both 4 hours before expiration), so it kind of looks like I'm back to square one on this; I guess the timing on the one was just a coincidence after all...

The DHCP servers do provide time server addresses (oddly, only one, but I get two...need to figure out where the second address is coming from...), though how they could affect the settings of a client device doesn't make sense (beginning to think my idea was a red herring :S ).

[BillP]

did a little more digging...the ntp servers that the cards are getting assigned appear to be coming from the dhcp servers' option 042 NTP Servers (4 servers listed), not option 004 Time Server (1 server listed). I'm going to redo a couple of tests, and, I'm going to test a system with manually assigning an IP (no dhcp) with manual ntp time settings and see what happens.

[BillP]

bleh...not even an hour later...see attachments

one of my systems that I re-tested manual settings with...the "pre" is how it was set before it auto-configured itself to the settings in the other image.

[BillP]

interesting - and it indicates update from "secondary server" even though one is not listed..

[BillP]

we are trying to investigate further..did this ever work correct? we looked at the logs you had posted and it looked like this worked correctly for almost a year and then maybe some configuration changes were made in October of this year and then I know you started posting here shortly after. just curious when the issue started.

[BillP]

so far, it is not working correctly on any of the dhcp enabled cards. the one I set to manual ip settings is behaving so far, but I have seen the configuration change not occur for nearly a week before, so it may still be too soon to tell. I'm hoping that not using dhcp isn't the solution since it is frowned up on our network to have static ips. we are planning to move to windows 2012 servers during the winter break....

I'm not certain, but I don't think they ever worked correctly. Unfortunately, shortly after we got the equipment, I had changed positions and wasn't monitoring the units as closely as I do now. I do have emails from as far back as April of this year that show the discrepancy between the real day and time and what the cards thought was the real day and time (see email header info below). I do, vaguely, recall resetting the time on the nmc2 cards more than once soon after they were put into production, though, with over 30 cards, I definitely did not make any sort of connection between the behavior and any events at that time.

Received: from snowwhite.uccs.edu (128.198.1.150) by UCCS-EX2.uccs.edu (128.198.1.102) with Microsoft SMTP Server id 14.2.283.3; Wed, 18 Apr 2012 09:29:28 -0600 <- real day/time
Received: from apc-uh-c (apc-uh-c.uccs.edu [128.198.155.9]) by snowwhite.uccs.edu (8.13.8/8.13.8) with ESMTP id q3IFTR30031788 for ; Wed, 18 Apr 2012 09:29:27 -0600
Message-ID: <201204181529.q3IFTR30031788@snowwhite.uccs.edu>
From:
Reply-To:
To:
Subject: UPS: On battery power in response to distorted input.
Date: Fri, 20 Apr 2012 10:43:51 +0000 <- day/time unit was claiming it was
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="--TRECK-MIME-MULTIPART-BOUNDARY"
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.6.7580,1.0.260,0.0.0000 definitions=2012-04-18_07:2012-04-18,2012-04-18,1970-01-01 signatures=0
Return-Path: apc-uh-c@uccs.edu
X-MS-Exchange-Organization-AuthSource: UCCS-EX2.uccs.edu
X-MS-Exchange-Organization-AuthAs: Anonymous
X-Auto-Response-Suppress: DR, OOF, AutoReply

[BillP]

another failure....unit that is manually only using the ISX server as its ntp servers changed timezone on its own...again 😞 it is possible that it just renewed its dhcp lease, but the date/time stamp on the IPv4 page makes it difficult to tell. also, since nmc2 cards no longer list dhcp lease renewals as an event that could generate an email (nmc cards listed dhcp lease renewals as "configuration change" events), I don't have any way to obtain a time stamp outside of the IPv4 page. since we use reservations for the units, going to the dhcp servers doesn't help (doesn't show expiration date/time for reservations); going to the switch and converting the lease time into days does seem to indicate a recent lease renewal given the lease time listed for the vlan on the dhcp server.

....so, might still be the dhcp servers causing the problem...???

may have to wait and see how things behave after the upgrade from window 2008 r2 servers to windows 2012 later this month here...

[BillP]

the NMC2 devices should be logging an event when the lease is renewed..unless it has been disabled. you can check that if you go to Administration->General->Notification->by event and look at the list of system level events.

previously, we did have an issue with email timestamps in the headers as well so i don't know if that caused part of the problems with the emails you were looking at. that was fixed fully in 5.1.6 i believe.

i do feel like there is something going on there on the card's side that shouldnt be happening, especially seeing those last screenshots. i will investigate more monday now that i have more background from you.

[BillP]

yes, i believe it will be network service started. the second sentence in the alert indicates "manual settings in use" or "received address from DHCP with XXX second lease." i found one in your log:

08/30/2012 10:04:03 System: Network service started. System IP is 128.198.155.9 from DHCP server 128.198.4.73 with 14390 second lease. *0x0007*

the "outlet group turned on" messages after a reboot of the management card was an issue we addressed in 5.1.7 because it was misleading that the outlets had turned off so you could go ahead and turn it back on and it should not continue with 5.1.7 and higher.

let me take what you've shared here and see what else we might need to discover where we think the problem is, if not DHCP. anything new happen by the way with switching to server 2012? i know you thought you might not know right away either.

[BillP]

we have not moved to 2012, yet; it should be within the next couple of weeks that we will be doing so.

turned email notification back on for that event...still don't understand why the event doesn't show regularly in the logs, though... 😞

once I can confirm that I am getting dhcp lease renewal emails, I'll try that set of tests again (I'll also let you know if I don't see any; should get emails anywhere from within a few hours to seven days at most from all of the units in question)

so far, the one unit I set to use manual IP settings is still behaving...ntp will be updating in about 23 hours from now...

[BillP · Accepted solution]

ntp update succeeded on my manually addressed card - no timezone change so far...

no dhcp renewal emails yet from the other cards...