SMT3000 self tests coming in pairs is confusing.

[BillP]

When my SMT3000 does its 14-day self test, I get "two for the price of one" as evidenced by the log information below. Can I assume one is initiated by the UPS and the other one is initiated by the AP9630?

02/20/2013 12:53:08 UPS: passed via internal operation a self-test.
02/20/2013 12:53:01 UPS: Started a self-test.
02/20/2013 11:54:30 UPS: passed via management device a self-test.
02/20/2013 11:54:23 UPS: Started a self-test.
02/20/2013 11:23:00 System: NTP update successful.

Does disabling the test via the web interface affect only one or both of the tests? Do I need to connect via the serial or USB interface to stop the UPS from running its own test? What's the preferred method of initiating the 14-day self test?

Thank you.

[BillP]

Hi Angela, It's been two weeks and once again the SMT3000 has posted a self test in and of its own accord.  Again, there is nothing configured either via the NMC or via the front panel and there is nothing connected to the UPS other than the ethernet cable.  I know you've repeatedly asked Liam what to do and I hate to keep bothering you if the ball's in Liam's court as to what to do at this point.  I remember you emailing me that there may be something going on with my particular version or revision level of SMT3000 but I'll ask again if you know if this is a one-off issue with my particular UPS or is there something systemic within the SMT3000 series?

Thanks in advance!

[BillP · Accepted solution]

Hi Joe,

I understand the issue and I will follow up. But, will the self test stay disabled forever on your UPS (which we would not recommend)?

[BillP]

Hey there.  No, the self test will be enabled on a 14-day basis.as it is on my other two APC boxes that included and SMX1500RM2U and an older SU2200 .  I just want the UPS to function as it should.  Thanks again.

[BillP]

Yes, I understand that. I am just wondering what behavior you see now if you put it back to every two weeks. I am not saying we won't fix the problem but we know there is an issue so I am just asking to see if you will be re-enabling the self test now.

[BillP]

I understand now.  What I will do is enable the self test every two weeks via the NMC initiate a baseline self test via NMC and watch what happens.  Additionally, I will try to be on hand near the UPS around the time the scheduled 14-day test is to be run to make sure the test is actually happening and not some fluke with the logging application or something.

[BillP]

Hi Angela.  The latest version of the NMC firmware did not stop the SMT 3000 self tests.  As a matter of fact, I have had the internal self tests set to "never" and I got one today

05/01/2013	12:47:31	UPS: passed via internal operation a self-test.
05/01/2013	12:47:24	UPS: Started a self-test.

We can pick this back up when you are back off holiday.  Hope it was a good one!

[BillP]

Ok so that one sounds like it is the self test the UPS did for itself, not issued via the management device. Can you send me the new logs from 6.0.6 so I can look at them tomorrow (when I am back from holiday) and review? Also, what firmware is that UPS at again? 01.9?

[BillP]

Hi Angela and thanks for the response.  The files are attached (using 6.0.6 in the NMC) and the UPS firmware revision is UPS 07.1 (ID17).  The three files you requested are attached as well.  Let me know what you find and "sorry" to see you back off vacation....not!  Hope it was a good one

[BillP]

Hi Joe Keifer - I had a great vacation, thanks for asking. Too short!

Back to our issue - just one clarification. I was re-reading everything and I noticed initially you said firmware 09.1 and ID20 - are their multiple units doing this? I just want to sort it out and what units we have and what firmware.

Also, have you verified at all via the UPS front display (if you can physically check the UPS) that it says the self test is set to never? The management card definitely says: SelfTestSchedule=Never so I just want to make sure there is no discrepancy some place between the UPS and the management device (AKA the NMC).

If the UPS says never on the display, then I will bring Liam G. back in the loop here because me and the NMC team believe that to be a UPS problem.

The other thing is on the new firmware, we MIGHT need the debug logs from the UPS (under About->Support) in the web where you can generate logs and then download them and it basically gives you every single log the card and UPS can generate (including those you already provided) with some more info. Depending on your answers above, I may ask for that.

[BillP]

Just to confirm, the SMT3000 that I am having problems with has the front panel self test set to "no" and the UPS firmware is UPS 07.1 (ID17) as confirmed through the NMC interface's "About" pull down menu.  I do have another UPS in the system, an SMX1500RM2U at firmware UPS 09.2 (ID20) that IS NOT having any issues.  Sorry about the confusion.

[BillP]

Thanks. One last thing I forgot before - has anyone seen the self test happen? Like is it actually going to battery and doing the self test or do we think it could be an erroneous message. My assumption is its actually happening but that might not be the case. Either way, I will be reviewing here with applicable parties.

[BillP]

No I have not seen or heard it go onto battery during one of these weird events. Thanks for the help!

[BillP]

Hi Angela.  The NMC reported another self test today.  Any feedback from Liam on what the next step is?  Thank you.

[BillP]

Okay, thanks Angela.  I will wait for the next testing cycle and, if it occurs again, I'll let you know.  It's good to see the forum back up and the new look!

[BillP]

The UPS initiated a self test yesterday and there was only one occurrance of the test.  The two tests that ran that one day must have been a quirk.  But, then again, I've always heard that there is no such thing as "intermittent" software so who knows.  If it happens again, I'll let you know.  Ciao!

[BillP]

Hi Angela.  The self tests are still coming at odd intervals (this time a day apart) and not per the configuration I have set so disabling SNMPv1 didn't do the trick.  I will either try reflashing the NMC firmware or wait until you have released the next version which, I recall, you said had something to do with a correction or fix to the NTP functionality.  Have a good weekend!

[BillP]

Yes, that is very strange..6.0.6 is due around April 19th as it stands today. Conveniently I will be on vacation the following week or so Hope you have a nice weekend too.

[BillP]

Well enjoy your vacation you lucky dog!  I decided to format the NMC2 and will wait and see how it performs as I wait for 6.0.6 to be released.

[BillP]

you should not be seeing two self tests. the UPS is the only one who initiates its own self test and the management card (AP9630) just reports it back to you. it cannot initiate a self test without you commanding it to which then tells the UPS to do it.

can you tell me what the UPS firmware is under UPS->About section of the web UI in the management card? Also, what is the management card firmware under Administration->General->About.

the other thing I would ask is if you can check the back of the UPS and let me know what you see plugged in back that. i suspect that there could be something connected back there to a port that it shouldn't be and it could cause a problem like this.

lastly, how long as this been happening? can you provide the full event, data, and config files for review? instructions are in the knowledge base @ http://www.apc.com/site/support/index.cfm/faq/ and look for article ID FA156141.

[BillP]

Hi Angela (I believe that's your name) and thanks for the response.

The firmware is: UPS 09.1 / MCU 09.1 / UBL 06.9 / MBL 06.0 (ID20). If I am correct, I think there is is a version 9.2 out there but haven't attempted to install it.

My UPS has been installed for a couple of weeks now so this is the first occurrance of the 14-day self test, I believe, and the log is not very "deep" with data..

There is nothing plugged into the back except for a couple of AC cables and the ethernet cable that I use to gain access to the unit via the web interface.

[BillP]

you're right - 09.2 is the latest version. was just curious.

i just looked at your logs again. i noticed the difference in events (excuse the terrible wording):

02/20/2013 12:53:08 UPS: passed via internal operation a self-test.
02/20/2013 12:53:01 UPS: Started a self-test.
02/20/2013 11:54:30 UPS: passed via management device a self-test.
02/20/2013 11:54:23 UPS: Started a self-test.
02/20/2013 11:23:00 System: NTP update successful.

internal operation is the normal automatic self test. can you look at the very beginning of the event log? was there an event dealing with self test 14 days earlier? management device tells me this was issued via the management card but I do not see a user log in previous to this event. this is why I asked for the full log so i could look at these types of things and what happened 14 days ago.

i do not suggest disabling the automatic self test because then you may not know you have a bad battery until its too late.

can you please confirm to me the setting name that is selected for automatic self test according to the management card? for UPS, I imagine you see this or similar:

• Never
• Every 7th Day
• Every 14th Day
• UPS Startup and every 7 Days since last test
• UPS Startup and every 14 Days since last test
• UPS Startup

[BillP]

thanks for the feedback.

i would see if it does it again after making the change you mentioned. i've never seen this before..if it happens again, i will inquire further with the Smart UPS team and indicate it has happened multiple times.

[BillP]

Hi Angela,

The spurious self test has occurred once again.  From the logs, it can be seen that the self test occurred on 3/12 at about 05:58 and earlier on 3/6 at 11:52.  I have the UPS configured to produce a test every 14 days.  The requested event, data and configuration files are attached.  Please see if there is a fix or if hardware/software engineering is aware of this.  Thank you.

[BillP]

I believe it is a problem with the UPS configuration (internally - not what you've done). I consulted with liamg and he suggested resetting the UPS to defaults as well as re-flashing the firmware on the UPS. I personally would try the defaults first because it is easier and does not require the downtime that a re-flash would. I have not seen this before besides with your situation but he did mention he had seen it before recently.

[BillP]

Thanks for bearing with me on this.  I reset to defaults via the front panel, set the test schedule to "UPS Startup and every 7 Days since last test", initiated a test via the NMC, and will wait.  I will let you know what I find.

[BillP]

03/21/2013	05:46:26	System: Web user 'apc' logged in from 192.168.1.43.
03/21/2013	05:34:47	System: NTP update successful.
03/20/2013	12:51:42	UPS: passed via management device a self-test.
03/20/2013	12:51:34	UPS: Started a self-test.
03/20/2013	11:28:15	System: Web user 'apc' logged out from 192.168.1.30.
03/20/2013	11:24:51	System: Web user 'apc' logged in from 192.168.1.30.
03/20/2013	11:23:56	System: Web user 'apc' logged out from 192.168.1.30.
03/20/2013	11:23:17	UPS: passed via management device a self-test.
03/20/2013	11:23:10	UPS: Started a self-test.
03/20/2013	11:22:14	System: Event Log cleared.

Angela,

As you can see from the events copied from my short event log, the "rogue" self test occurred again at 12:51:34 on 03/20/2013.  This was after I reset the unit to default from the front panel and I was not logged into the user interface.  I guess I will try and reflash the firmware in the UPS and see what that does.

As a point of clarification, the information from Liamg recommends reflashing the UPS firmware and not the firmware in the NMC, correct?  Is there any chance the firmware in the NMC is bad?

Thank you!

[BillP]

Yes, that's the UPS firmware we were talking about, not NMC.

Is there any possibility that there is anything attached to the serial port of this UPS that should not be? I assume also there is no script executing via the management card some how, like via SNMP? I will just through it out there even though I don't think so. I would hope the log could provide that detail (like if an SNMP command was sent or any command and where it came from).

the only other thing I would try if it were me at this point would be to format that the management card itself, which wipes the entire configuration out, including TCP/IP settings. It wipes flash memory on the card (and does not upgrade/downgrade the firmware) but if there is something strange in this specific configuration, it should get deleted. you may want to consider backing up the config.ini file before doing that or preferably, reconfigure after format incase there is something strange in that config.ini file.

in the meantime, i will check further on this with liamg.

[BillP]

Thanks for the information and suggestions. No, there is nothing connected to the serial port and there is no script executing unless there is a ghost in the machine. I am not sure how to “format” the NMC itself. I will look through the manual that I have for the card and see if I can come up with anything.

I also have an SMX1500RM2U with another AP9630 in it. Would it be worthwhile swapping the cards between the two UPS to see if the issue moves to the other UPS?

Thanks again!

[BillP]

The "UPS: passed via management device a self-test." message would indicate that it is being initiated by the NMC or Software, so I think that formatting the card would be the most logical next step. If that did not resolve it I would then upgrade the UPS FW, as that is what controls the logic of the unit itself. Ultimately I think we have an unintended configuration issue.

Also: It would be very helpful to know if SNMP is configured, or if there is anything attached to the Serial or USB ports on the UPS itself.

[BillP]

Sorry Joe Keifer - I intended to provide you with instructions on formatting the NMC.

The easiest is to issue the format command via the telnet/ssh/local console command line. You'll have to type yes to confirm then issue the reboot command for it to happen. Again, it will require reconfiguration of the TCP/IP settings.

The other way is holding down the pinhole reset button for 20+ seconds which can be more frustrating. When you are doing it, the status LED should pulse green on the NMC and when its done and time to let go, it changes to pulsing amber or orange. Then it will reboot on its own.

As we have been saying, I have only seen this type of thing when some type of command via SNMP was occurring. If you're not using SNMP, we could try disabling it just incase as SNMPv1 is enabled by default. I have brought up to the NMC team in that instance that then the NMC event log should be more specific in saying what NMC interface the command came from (web, SNMP, console, etc) so it would make it easier to pinpoint.

Also - if you want to swap the cards, that's fine. I might try the format first but up to you.

*EDIT* - I just did some testing. When I do a self test via SNMP, the only thing my NMC says is:

03/21/2013	10:45:08	UPS: Passed a self-test.
03/21/2013	10:44:52	UPS: Started a self-test.

Let me look into the wording more and where it comes from depending on how its worded.

[BillP]

Thanks Angela N. and liamg.  I decided to take the easiest of the suggestions and I disabled SNMPv1 and will watch it for a week or two.  If that doesn't clear it, I will go ahead and format the NMC via one of the two methods you described.  Take care and have a good spring!

[BillP]

You too

I am still researching a little. My UPS was an older UPS-Link communications model. I tried an SMX UPS and it logged the same events you had. So, it could be something via SNMP logging in somehow and issuing a command.

We have also found potential that this is related to an NTP issue messing up the logging mechanism which will be fixed in the next firmware release, hopefully within the next month or so. That could also be an option to try depending on what we've found then.

Anyways, I don't think we've ruled out the UPS or NMC just yet but we'll see what happens in the next few weeks here per your post.

[BillP]

Hi Angela.  One other question about UPS and NMC firmare updates:  How is the end user made aware of these updates?  Is there a service one can sign up for, a link of the website's home page, or is it by serendipity alone that an end user like me finds out?  Thanks in advance!

[BillP]

I can speak for NMC firmware updates - I believe the major ones are put out in product announcements on the apc.com web page just as any "new product" is as it is a new firmware product. I also believe there is a software & firmware newsletter you can sign up for on the Software & Firmware Downloads page.

For the UPS, my current understanding of it is that the tool is stored in the knowledge base for now and referenced as needed but I feel like it will eventually be treated like an NMC firmware update.