SMX1500RM2U switching to battery

[Anonymous user]

We have a brand new SMX1500RM2U with a AP9630 V2 card that we have installed in the last couple of weeks. Since installation we have been seeing random switches to battery with the following warning:

Date : 03/30/2010
Time : 20:05:11
Code : 0x0109

Warning Events - UPS: On battery power in response to unknown.


Within 1 second it switches back to normal power with this warning:

Date : 03/30/2010
Time : 20:05:12
Code : 0x010A

Informational Events - UPS: No longer on battery power.

We have other equipment installed in this closet on the same circuit but NOT connected to the UPS and it isn't indicating that the AC power has been lost. We have both the most current SUMX code and UPS OS code of 5.11 loaded.
I have searched through the forums and haven't seen anything related to this. However, this is fairly new firmware. are there any known issues out there that would be causing this? At this point it doesn't appear to be facility power as mentioned above. It does appear to be a problem with the UPS.
I should also mention that we have ran both self tests and calibrations without issues.
Any help would be greatly appreciated.

[BillP]

hello everyone!

h2. I have some great news. we have the firmware available to address this issue!

it will be UPS firmware 03.8 for SMX units to address frequent clicking and transferring back and forth to battery/AVR. also, if you haven't upgraded to address "on battery due to unknown," it will fix that too.

unfortunately, i cant manage every single upgrade here so i would like to have everyone via some point contact APC tech support via phone or email and reference APC Knowledgebase # 11322 - this will have instructions for tech support on this particular issue. [alternatively, take a look here|http://nam-en.apc.com/app/answers/detail/a_id/11333].

please have the model of your UPS available along with the firmware versions listed in the NMC or the LCD menu on the front of the UPS. ensure your menu is set to "advanced" mode so that you will see all four firmware versions: UPS, COM, CBL, UBL. we need to know this first to give you the proper update instructions.

as i mentioned, this upgrade unfortunately will require the load to be removed from the UPS so please plan accordingly.

thanks again to everyone for their patience and understanding.

[BillP]

thank you very much. i see that you are not at the latest firmware revision - which even if you were, i cannot guarantee that it would solve the issue.

is there a reason you have not upgraded?

also, can you FTP in and see if there is a debug.txt inside of the /dbg directory?

[BillP]

No I will try updating the network management cards firmware and see if that helps and I will look for the debug file

[BillP]

ok thanks. will look for your reply.

[BillP]

got it - makes sense.

even so, if we dont find anything on the scope, i would change the sensitivity to see if it stops doing this.

[BillP]

Alright I have updated both network cards to 5.1.3 but I do not see the debug.txt file. It will take a few days to see if the firmware update helped out or not so I will probably let you know on Monday

[Anonymous user]

After upgrading the Firmware last week in response to these posts i am also still seeing the power switching and the error message is still "On battery power in response to unknown".

So, nothing appears to be resolved at this point.

Thanks.

[BillP]

jwilder - did you upgrade the firmware on your UPS via tech support or NMC firmware? i don't think firmware for the UPS is released yet, but will eventually be upgradable. the problem is not with the NMC firmware and cannot be resolved through an NMC firmware upgrade.

ubuntu17 - the latest logs will help and then i will need to email you to get some more information in order to set up a tech support case. also, please confirm there is still no debug.txt file in the /dbg directory.

[Anonymous user]

I upgraded the NMC on my own. i did not do a firmware upgrade via tech support.

[BillP]

okay, so yes, to be clear the NMC and UPS itself both have firmware. the on battery due to unknown is a UPS firmware issue. upgrades on UPS firmware will be available via tech support shortly but an NMC firmware upgrade, like you did, will not fix the issue since the issue is not with the NMC.

[Anonymous user]

I understand. Thank you.

Will the UPS firmware upgrade also fix the false notifications about switching to battery?

[BillP]

i cant say for sure. Smart UPS isnt my realm so I havent been following updates on whats fixed besides this issue that sort of relates to the message seen in the NMC. i can't say for sure it is even an issue indeed even though i know you have a lot of evidence saying it is/

[Squuiid_apc]

Angela, please could you take up my case? I am trying to get the new firmware but am dealing with an utterly incompetent APC support tech.
Please assist...

Case ref: 110423-000658

[BillP]

hi,

i will send you an email to the address you were using for the incident you submitted and email you what you need.

sorry for the troubles.

[Anonymous user]

Angela,

Sorry to bring up an old post, but I'm currently working on getting this issue resolved. What's the "special procedure" for upgrading CBL 1.0? Tech support just links me to the following article: How do I update the firmware of my SMT, SMX, or SMC series Smart-UPS? | FAQs | Schneider Electric IT...

Thank you,

Matt

[BillP]

If you're unit is still on CBL 1.0 firmware it should be upgraded first to CBL 2.0 before you can use the Firmware Upgrade Wizard.  It would be best that you call our support hotline to walk you through the steps and check your UPS as well if it needs to go through the CBL update first.

[BillP]

Do you have your case number? I can send you the CBL 1.0 steps. It is a script to run basically. I am surprised tech support did not provide it unless they did not know you required it.

*EDIT* I think I found some. I will just send it to you myself but normally, tech support is supposed to walk you through it.

[Anonymous user]

C1-4852301971 - I asked about the procedure for 1.0, they referred me to that link above. Please send the script, thank you!

[Anonymous user]

Refer to case: C1-4852301971. I've tried contacting them about it, but didn't get anywhere with it.

[BillP]

Alrighty. I found that case and updated it - check your email

[Anonymous user]

Thank you!

[BillP]

Hi Squuiid- please try increasing your high transfer threshold to maximum allowed value. That should reduce the incidence of the problem.

[Squuiid_apc]

What value is 'safe'?

[Anonymous user]

Just heard back from our monitor team. We did experience a 3 volt drop during this time and they are investigating as to why this is happening on facility power.

My question now is what is the sensitivity setting for this UPS? Is 3VAC enough to force a cutover to battery? The range was set to "normal" but i don't know what "normal" means.

The output threshold is 107 to 126. If i am normally running at 114 then 3 volts only takes me down to 111. Is the Input threshold monitor different?

I think we are getting to the root cause here......

Thanks!

[Anonymous user]

Did you find the issue? My new SMX1500 RM2U is having the same issue? Can the input sensitivity be adjusted?

[BillP]

there is an option either via the NMC web interface or the Smart UPS LCD to adjust transfer voltages/sensitivity. please understand that this is not necessarily a problem with the reporting but you may have a legitimate power problem that you should look into rather than covering it up with adjusting the settings.

lastly, if it says "on battery due to unknown" - that will be adjusted with a future rev of the UPS firmware (versus the NMC)

[Anonymous user]

Not so sure its a site power issue because this happens as well.

Date : 09/22/2010
Time : 13:56:12
Code : 0x0002

Critical Events - System: Warmstart.
*****
Date : 09/22/2010
Time : 13:56:19
Code : 0x0101

Informational Events - UPS: Restored the local network management interface-to-UPS communication.
******
Date : 09/22/2010
Time : 13:56:20
Code : 0x0344

Informational Events - Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication.
***
Date : 09/22/2010
Time : 13:56:23
Code : 0x0134

Informational Events - UPS: The power for switched outlet group 3, Outlet Group 3, is now turned on.
Informational Events - UPS: The power for switched outlet group 2, Outlet Group 2, is now turned on.
Informational Events - UPS: The power for switched outlet group 1, Outlet Group 1, is now turned on.
***
No power outages for the site equipment.
Sounds like an issue similar to jwilder.
I also can't seem to fix the time stamp inside the message. It appears to be 8 hours back from the time that the email was sent.

[BillP]

i picKed a winna wrote:

if your UPS has CBL 1.0 currently, you will need to do an extra little step to get this upgrade started.

What's the trick for those of us with CBL 1.0?

Model: Smart-UPS X 1500
Firmware Revision: UPS 02.2 / COM 02.3 / UBL 05.0 / CBL 01.0
Manufacture Date: 04/28/2010

The "Firmware Upgrade Wizard v3.0" indicates that my "SMX1500 with firmware UPS 02.x / COM 02.x" is not supported.

When I run the firmware upgrade wizard v3.0.0.14, using the DB-9 to RJ-45 cable (940-0625A), it appears to find my UPS, but then reports that the "Firmware Upgrade Not Supported", which is consistent with what's stated in the manual.

[BillP]

if you contact technical support, there is a special procedure you need to run through to update the CBL 1.0 then you can re-do the procedure you just did that did not work while at CBL 1.0.

if you contact technical support, just tell them you need the SMX firmware update procedure for units running CBL 1.0 so that you can update your UPS to firmware 3.8 (which would be the latest I think).

[Anonymous user]

I want to add myself to the list of users experiencing this issue with a newly installed SMX1500RM2UNC (came with the AP9631). Version 5.1.3 was installed via NMC. The regular temperature probe installed as well. This appears to happen randomly every few days and I am able to access the NMC after.

Firmware Revision: UPS 02.2 / COM 02.3 / UBL 05.0 / CBL 01.0
Manufacture Date: 02/24/2010

Model Number: AP9631
Serial Number: BA0917014754
Hardware Revision: 05
Manufacture Date: 04/24/2009

10/04/2010 09:19:07 UPS: The power for switched outlet group 3, Outlet Group 3, is now turned on. 0x0134
10/04/2010 09:19:07 UPS: The power for switched outlet group 2, Outlet Group 2, is now turned on. 0x0134
10/04/2010 09:19:07 UPS: The power for switched outlet group 1, Outlet Group 1, is now turned on. 0x0134
10/04/2010 09:19:06 Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication. 0x0344
10/04/2010 09:19:06 UPS: Restored the local network management interface-to-UPS communication. 0x0101
10/04/2010 09:18:40 System: Network service started. System IP is 192.168.xx.xx from manually configured settings. 0x0007
10/04/2010 09:18:34 System: Warmstart. 0x0002
10/04/2010 09:15:36 UPS: On battery power in response to unknown. 0x0109

Has anyone here anything more about this issue from APC? I plan to open an official ticket referencing this thread.

Message was edited by: mcctech

[BillP]

hi

opening a ticket wont get you any additional help at this point based on my response below but you can do it if you'd like.

here is what i can tell you - the issue affects only smaller VA models that begin with SMX. it has already been addressed in a future revision of firmware that has already been put into production. i believe that those of you that already own units will be able to update your unit shortly to address the concern.

to address WHY the unit is unexpectedly going on battery has also been fixed - assuming you have validated that there is no power outages, electrical noise, etc. if the unit is still frequently transferring to battery, you can try changing the power quality to fair to override this.

[Squuiid_apc]

I'm having this exact same issue too on my SMX1500RM2U.
This is definitely an issue that APC must fix ASAP. It is unacceptable for a product to reach the market with evidently very little testing having been done by APC before release.
APC not what they used to be?

Here are my details:

Model Number: AP9631
Serial Number: xxxxxxxxxxxx
Hardware Revision: 05
Manufacture Date: 08/13/2010
MAC Address: xxxxxxxxxxxx
Management Uptime: 2 Days 19 Hours 56 Minutes

Application Module

Name: sumx
Version: v5.1.3
Date: May 4 2010
Time: 14:06:35

APC OS (AOS)
Name: aos
Version: v5.1.3
Date: May 4 2010
Time: 13:55:26

APC Boot Monitor
Name: bootmon
Version: v1.0.1
Date: Aug 3 2009
Time: 16:30:40

Model: Smart-UPS X 1500
Serial Number: xxxxxxxxx
Firmware Revision: UPS 03.6 / COM 03.6 / UBL 05.0 / CBL 02.1
Manufacture Date: 09/11/2010
Apparent Power Rating: 1440 VA
Real Power Rating: 1200 W

01/16/2011 14:05:33 UPS: No longer compensating for a high input voltage.
01/16/2011 14:05:25 UPS: Compensating for a high input voltage.
01/16/2011 14:05:20 UPS: No longer compensating for a high input voltage.
01/16/2011 14:05:10 UPS: Compensating for a high input voltage.
01/16/2011 14:04:56 UPS: No longer compensating for a high input voltage.
01/16/2011 14:04:46 UPS: Compensating for a high input voltage.

Sensitivity is set to low and I am not having this issue with any of the other (older) UPS units.

[BillP]

Took a few steps, and waiting (legacy serial communication), but I'm now upgraded. Thanks.

Firmware Revision: UPS 03.8 / COM 03.8 / UBL 05.0 / CBL 02.2

[Squuiid_apc]

Log:
01/16/2011 14:05:20 UPS: No longer compensating for a high input voltage.
01/16/2011 14:05:10 UPS: Compensating for a high input voltage.
01/16/2011 14:04:56 UPS: No longer compensating for a high input voltage.
01/16/2011 14:04:46 UPS: Compensating for a high input voltage.
01/16/2011 14:04:09 UPS: No longer compensating for a high input voltage.
01/16/2011 14:04:09 UPS: Compensating for a high input voltage.

Message was edited by: Squuiid

[Anonymous user]

Hi winna,

Thanks for the info. I assume that this soon to be released production firmware will be a download from the usual downloads section? Will it utilize a special installer app like that used with the NMC firmware?

Thanks.

[Anonymous user]

I called APC Support and established ticket. After looking at logs with no real big issues, he said to reduce the sensitivity to "Reduced" and we changed the lower limit to 100 versus default of 107..
Up to this past Sunday, I have had nothing but false positives email notices as mentioned but as I mentioned, on Sunday, the UPS lost its power completely. (All the equipment that was plugged into it rebooted) Other equipment (not plugged into the Smart UPS) in the facility stayed on.

We shall see how this goes..

[Anonymous user]

Winna - it's now mid Feb 2011, any word on updates? I just checked at there still are not any updates for the 9631! The consistent warmstart and "UPS: On battery power in response to unknown." errors have become so annoying that I will likely have no choice but to turn off all notifications. Unfortunately I will be avoiding APC until this gets resolved. Very disappointed.

[BillP]

mcctech - i believe that if you have an NMC, you will be able to do this in the next rev of the NMC firmware. it will support upgrading the UPS directly.

there is a "wizard" in beta currently to be able to flash the UPS, if you don't have an NMC, but I am not sure when that will be available. i think the NMC feature will be available sooner. it will indeed be available on apc.com. if i remember, which i hope i do, i'll post back here with a link.

[BillP]

Squuiid, the issue you are showing us has nothing to do with what this thread is about. your logs indicate the UPS is using its internal AVR transformer to trim your voltage since it seems to be a bit high.

nominal voltage in the USA is 120v. if it goes above a certain voltage, the AVR trim kicks in to get it closer to 120v.

perhaps you can post your data.txt and full event.txt so we can examine the logs further. you could also maybe do it on a different thread since i think this is a different issue. you can use [these instructions|http://nam-en.apc.com/app/answers/detail/a_id/9321] to get the logs.

[Anonymous user]

winna - Thanks for the information. Updating this thread with a link would be greatly appreciated.

[Anonymous user]

I agree - we need a solution to this problem. I don't like the idea of turning the notifications off, since that nearly defeats the purpose of buying the network management cards in the first place.

[BillP]

Glad I found this post. I have 3 SMX1500 and was going to buy a 4th. I have the same issues everyone else is having. Any ETA on the fix??




Mike

[BillP]

hi everyone. sorry - just saw this post.

here is what i know. keep in mind, my responsibility here relates to the network management cards but i investigated an update for us.

"on battery due to unknown" is really a UPS problem but it presents itself via the NMC. i am told this is fixed in a revision of the UPS firmware itself (UPS 03.6). UPS upgrades can be requested via APC tech support for SMX units. the upgrade tool is not available publicly yet. if your UPS is not upgradeable, then we will have to pursue a different route that tech support will go if that is the case.

in regards to the warmstarts, i am just finishing some research and compiling it today. the new revision of the NMC firmware (AOS 5.1.5/sumx 5.1.4) does not appear to have fixed the warmstart issues with even the latest version of UPS firmware. again, i am working even more closely with engineering on UPS and NMC side so we can take care of this problem since as we know, it has been going on for some time and the new firmwares did not fix it as i was originally told.

i can provide the latest version of NMC firmware for you try if you'd like, and even if it doesn't, the "warmstart" event is now an Informational alert (versus critical) and is now called "network interface restarted" to reflect the true nature of what is exactly happening here. thus, in your event log, for any warmstart, you will now see "System: Network Interface Restarted" and it will be listed as informational.

just so we are clear, i understand the warmstart is a nuisance but should not be a major concern functionality wise. it is definitely something that needs to be fixed and addressed, believe me, but i don't want anyone to think the UPS is rebooting or that anything else is going on that is really CRITICAL. it is similar to an OS reboot when it hangs or freezes, except the NMC does it automatically and quickly to ensure that nothing else happens to the UPS and NMC to ensure that it is monitoring the UPS and itself 24/7, as it is designed to do.

i hope that this helps tell you where we are at. i suggest we at least get everyone's UPS updated to address the "on battery due to unknown" issue. contact tech support to have them see if your UPS is upgradeable then tell them you need the latest UPS firmware upgrade if so. If we can't upgrade it, they will give you an alternative option.

any issues with doing that, PM me and i'll help you address it.

thanks!
Angela

[Squuiid_apc]

Angela,

I am running firmware 03.6 and I can assure you that the issue persists. The ONLY difference is that it is now not 'unknown', it is listed as:

02/21/2011 12:52:24 UPS: Compensating for a high input voltage.
02/21/2011 12:51:27 UPS: No longer compensating for a high input voltage.
02/21/2011 12:51:27 UPS: Compensating for a high input voltage.
02/21/2011 12:51:10 UPS: No longer compensating for a high input voltage.
02/21/2011 12:51:04 UPS: Compensating for a high input voltage.
02/21/2011 12:50:42 UPS: No longer compensating for a high input voltage.
02/21/2011 12:50:38 UPS: Compensating for a high input voltage.
02/21/2011 12:47:24 UPS: No longer compensating for a high input voltage.
02/21/2011 12:47:19 UPS: Compensating for a high input voltage.
02/21/2011 12:47:06 UPS: No longer compensating for a high input voltage.
02/21/2011 12:47:00 UPS: Compensating for a high input voltage.
02/21/2011 12:39:23 UPS: No longer compensating for a high input voltage.

The SMX1500RM2U is inherently defective and we all deserve a solution to this serious problem. It has been months and months now of no fix from APC.

Message was edited by: Squuiid

[Squuiid_apc]

This is what the error looks like prior to 03.6 firmware:

03/24/2010 10:38:01 UPS: On battery power in response to unknown. 0x0109
03/24/2010 04:26:26 UPS: No longer on battery power. 0x010A
03/24/2010 04:26:25 UPS: On battery power in response to unknown. 0x0109
03/24/2010 04:11:39 UPS: No longer on battery power. 0x010A
03/24/2010 04:11:39 UPS: On battery power in response to unknown. 0x0109
03/23/2010 18:05:52 UPS: No longer on battery power. 0x010A
03/23/2010 18:05:51 UPS: On battery power in response to unknown. 0x0109
03/23/2010 04:26:35 UPS: No longer on battery power. 0x010A
03/23/2010 04:26:33 UPS: On battery power in response to unknown. 0x0109
03/23/2010 02:50:50 UPS: No longer on battery power. 0x010A
03/23/2010 02:50:49 UPS: On battery power in response to unknown. 0x0109
03/22/2010 13:26:23 UPS: No longer on battery power. 0x010A
03/22/2010 13:26:21 UPS: On battery power in response to unknown. 0x0109
03/19/2010 10:51:04 UPS: No longer on battery power. 0x010A
03/19/2010 10:51:02 UPS: On battery power in response to unknown. 0x0109
03/19/2010 08:20:32 UPS: No longer on battery power. 0x010A
03/19/2010 08:20:31 UPS: On battery power in response to unknown. 0x0109
03/19/2010 04:27:38 UPS: No longer on battery power. 0x010A
03/19/2010 04:27:36 UPS: On battery power in response to unknown. 0x0109
03/18/2010 17:05:57 UPS: No longer on battery power. 0x010A
03/18/2010 17:05:55 UPS: On battery power in response to unknown. 0x0109


NOTICE ANY SIMILARITIES?!

Message was edited by: Squuiid

[BillP]

you're welcome. i'll keep on top of it.

[Anonymous user]

Are there any updates about this issue? We bought three of these models (same as mcctech) back in August, and they are all doing the same thing as mcctech described. We have several other UPSs of a different model with AP9630s, and they have never done this before.

The most recent version of firmware is loaded on all three.

Thanks.

[BillP]

The fix was to make it so the UPS does not go to battery power "due to unknown" so yes, it is fixed. I am not aware of an issue that causes to go to battery when it shouldn't so that may be a separate issue. The only thing being fixed here, since it was verified to be a problem, was the actual alert being "on battery in response to unknown" and now it being properly worded as "in response to in an input power problem" or due to high or low input voltage - which is when the UPS should boost or trim the voltage or go to batter power.

If there is another issue where you don't believe the UPS should actually be going to battery or boosting/trimming, we need to analyze it in a separate thread to see what's causing it, if it is not a legit alert. Is in the input voltage rising and falling or teetering on a threshold, etc

Either way, I suggest that you start another thread so we can keep track of it

[Anonymous user]

I agree, my original post was that we were seeing the UPS switch to battery for no apparent reason. We did a ton of troubleshooting and determined that the UPS is seeing minor power fluctations but the sensitivty is to high on these devices causing a bunch of unneccessary alerts. As i mentioned in my previous posts i have multiple UPS's in the same facility and my older model UPS's are not kicking these messages. ONLY the new 1500's that we have installed. So, if after 6 months we are going back to ground zero then i for one am going to be very upset. i have tried (unsuccessfully mind you) to adjust the thresholds on the management interface and it does not appear to be working. We need resolution to this yesterday. Can you please tell me whom i need to call at APC to get this fixed now.

[BillP]

no problem. any additional issues, PM me. i am the tech support lead here on NMCs.