UPS: An internal UPS communication fault exists.

Anonymous user · ‎2021-07-01

Updated two SMX750's in two different physical locations this morning to 6.2 firmware. After the update, both UPS's are throwing this error:

Critical - UPS: An internal UPS communication fault exists.

Error code is Code : 0x0172. Haven't tried to downgrade yet, but I haven't seen anyone else online or in these forums with a similar problem. Has anyone else had issues upgrading to 6.2? Once location has NMC2 with environmental and the other location has no environment, both with the same error.

BillP · ‎2021-07-01

Hi Gregory, yes, I still see the updates. I informed the case owner how it should've been handled. Sorry about that.

See Answer In Context

Anonymous user · ‎2021-07-01

Scratch that, can't downgrade.

10/24/14: 22:56:30 172.16.0.8: Testing device connection

10/24/14: 22:56:30 172.16.0.8: Device connection passed

10/24/14: 22:56:31 172.16.0.8: Testing FTP Log-in

10/24/14: 22:56:34 172.16.0.8: FTP Log-in passed

10/24/14: 22:56:35 172.16.0.8: Saving data file

10/24/14: 22:56:38 172.16.0.8: ERROR --> Disallow AOS downgrade

10/24/14: 22:56:38 172.16.0.8: AOS Downgrade not allowed: 620-->606

Anonymous user · ‎2021-07-01

I am having this problem as well after upgrading firmware to 6.20 on AP9630 NMCs on a number of different Smart-UPS models including the SMX750. Seems all software these days is suspect. The problem occurs at certain times of the day only, not 24 hours a day. Very strange.

BillP · ‎2021-07-01

Let me see if I can investigate what this event is triggered by and if it has changed recently.

In the meantime, if anyone is interested in downgrading, you can use a manual method like FTP (explained here ->How do I upgrade the firmware on an APC Network Management Card (NMC) or NMC embedded device (Rack P...) . The older wizard/upgrade utility versions blocked downgrading but I had them adjust that for future versions of the wizard/upgrade utility, especially for users who purchase a standalone Network Mgmt Card and it comes with Smart-UPS firmware and need to put it in a different device - it was treating that as a "downgrade." So, post v6.2.0, if you want to downgrade back to v6.2.0, the wizard/upgrade utility will let you.

Also, I noticed both of you are saying you're seeing this on SMX750. Can you tell me the firmware versions on the UPS? It will be under About->UPS in the v6 firmware web UI of the Network Management Card. I need something like UPS 03.8 (ID12) or whatever it says.

Thanks!

Edit/Update: Based on what I learned, I essentially need to see some additional files from the UPS/Network Management Card to look at this further. Ideally, please provide the "single file download/dump" available via About->Support or logzip CLI command because it will have the file I need (Instructions here-> How can I download Event, Data, Configuration, and Debug files from my Network Management Card? | FA...) No major changes were made to this event and when it appears in v6.2.0 so the problem may be with the particular UPS and it reporting a particular flag or something.

BillP · ‎2021-07-01

Thank you!

The UPS firmware is a bit older. My first thought is that we should try upgrading it which I believe the latest version for ID11 is UPS 03.8. I have a feeling the UPS firmware is the issue here. The only downfall to getting these log files is that the fault appears to be "cleared" right now when you extracted the files for me so there is a file that can tell us information about the fault when it is currently active. I am using an SMX1500 ID11 at my desk coincidently with v6.2.0 and have never seen this alarm. I highly doubt all your UPSs have an internal fault either based on what you've described.

Anyway, I think the best thing to do on at least this one UPS is upgrade it to UPS 03.8 and see if it still throws the alarm periodically. If it does, then I'll need this dump of files when the Internal Communication Fault is active.

Details on upgrading the firmware are here - the only caveat is that the UPS output must be turned off -> How do I update the firmware of my SMT, SMX, SMC, or SRT series Smart-UPS? | FAQs | Schneider Electr... explains how to do it via a serial cable (or USB if supported) and then How do I upgrade my SMT/SMX/SRT Smart-UPS firmware using my AP9630/AP9631 Network Management Card? |... explains how to do it via your AP9630 and have it push the right file to your UPS - I believe this UPS should support upgrading via the NMC (as opposed to the Firmware Upgrade Wizard via serial cable) but the UPS I have of that type is already at UPS 03.8 but the UPS appears to support upgrading via NMC (since you already have one). You need to put SMX11UPS_03-8.enc file on any ID11 device, or least this one to start and test to see what happens.

This event I think is only supported by SMX/SMT/SRT type UPSs so you wouldn't see it on older SU, SUA, or SUM type.

Let me know if you have any questions before proceeding.

Anonymous user · ‎2021-07-01

Thanks - I will try that. This will take me a little time since I cannot shut the UPS outputs off most of the time. But I'll report back once I am able to do this.

Anonymous user · ‎2021-07-01

Angela - you assisted me in getting a replacement NMC in this thread (Cannot Upgrade AP9631 with SUMX/AOS 5.0.3 and BootMon 1.0.0) - got the replacement card, upgraded to SUMX/AOS 6.2.0 & Bootmon 1.0.8 successfully and now I'm experiencing the same errors in the logs as well. I'm also at UPS 02.2 (ID11). It's interesting to note that the logs show this error only occurs between 7pm and Midnight - see log below

In attempting to use the NMC to update the UPS firmware following the link you posted ( Frequently Asked Questions ), I was able to FTP the correct enc file to the upsfw directory, but when I get to step 3, there is no Firmware update page in the web UI in 6.2.0 as described here:

Check the firmware update page with the NMC web UI to see that the update file is loaded onto the card.

• For v5.1.X firmware, this page is under UPS tab->Configuration->Firmware Update
• For v6.X.X firmware, this page is located under Configuration->Firmware Update

Not sure how to proceed with updating the UPS firmware - see screenshot below showing Firmware Update option not listed.

.

Network Management Card AOS v6.2.0

Smart-UPS & Matrix-UPS APP v6.2.0

Date Time Name Contact Location System IP

12/10/2014 03:02:46 apc88DB05 Unknown Unknown 192.168.1.61

Date Time User Event Code

12/10/2014 00:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/09/2014 23:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/09/2014 23:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/09/2014 22:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/09/2014 22:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/09/2014 21:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/09/2014 21:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/09/2014 20:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/09/2014 20:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/09/2014 19:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/09/2014 00:00:17 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/08/2014 23:01:22 Device UPS: An internal UPS communication fault exists. 0x0172

12/08/2014 23:00:17 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/08/2014 22:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/08/2014 22:00:17 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/08/2014 21:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/08/2014 21:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/08/2014 20:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/08/2014 20:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/08/2014 19:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/08/2014 00:00:08 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/07/2014 23:01:22 Device UPS: An internal UPS communication fault exists. 0x0172

12/07/2014 23:00:07 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/07/2014 22:01:22 Device UPS: An internal UPS communication fault exists. 0x0172

12/07/2014 22:00:07 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/07/2014 21:01:22 Device UPS: An internal UPS communication fault exists. 0x0172

12/07/2014 21:00:07 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/07/2014 20:01:22 Device UPS: An internal UPS communication fault exists. 0x0172

12/07/2014 20:00:07 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/07/2014 19:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/07/2014 00:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/06/2014 23:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/06/2014 23:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/06/2014 22:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/06/2014 22:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/06/2014 21:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/06/2014 21:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/06/2014 20:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/06/2014 20:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/06/2014 19:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/06/2014 00:00:08 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/05/2014 23:01:24 Device UPS: An internal UPS communication fault exists. 0x0172

12/05/2014 23:00:09 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/05/2014 22:01:15 Device UPS: An internal UPS communication fault exists. 0x0172

12/05/2014 22:00:10 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/05/2014 21:01:17 Device UPS: An internal UPS communication fault exists. 0x0172

12/05/2014 21:00:12 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/05/2014 20:01:17 Device UPS: An internal UPS communication fault exists. 0x0172

12/05/2014 20:00:12 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/05/2014 19:01:18 Device UPS: An internal UPS communication fault exists. 0x0172

12/05/2014 00:00:14 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/04/2014 23:01:19 Device UPS: An internal UPS communication fault exists. 0x0172

12/04/2014 23:00:14 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/04/2014 22:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/04/2014 22:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/04/2014 21:01:21 Device UPS: An internal UPS communication fault exists. 0x0172

12/04/2014 21:00:16 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/04/2014 20:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

12/04/2014 20:00:15 Device UPS: An internal UPS communication fault no longer exists. 0x0173

12/04/2014 19:01:20 Device UPS: An internal UPS communication fault exists. 0x0172

BillP · ‎2021-07-01

Hi apoulias - can you give me the first six characters of your UPS serial number? Something like AS0937? I also am interested in the CBL version but I have an ID11 at my desk with 6.2.0 and I cannot see it via the NMC. Can you make sure the LCD menu on the UPS is in Advanced mode and check the About menu there and confirm you see the CBL version as 1.something?

With ID11 and UPS 02.2, I have a feeling you'll need to run through a special procedure to update the bootloader (CBL) on this UPS to get it to UPS 03.8. I just want to confirm and if that is the case, then I will send you those instructions to get the UPS upgraded as I believe the bootloader is older so it does not support upgrading the UPS via the NMC.

Anonymous user · ‎2021-07-01

SMX1500RM2UNC

First 6 of the UPS SN =AS101513

From the LCD

UPS Firmware 1 UPS 02.2

UPS Firmware 2 COM 02.3

UPS Firmware 3 UBL 05.0

UPS Firmware 4 CBL 01.0

BillP · ‎2021-07-01

Ok, I suspected correctly. I'll send you a private message with a special procedure you'll need to do. Please read carefully and let me know if you have any questions before you get started but to get this particular UPS updated, you'll need to upgrade the CBL with a special script (via serial connection), get it to firmware UPS 03.7, and then you can use the NMC (if you want) or continue via serial to get to the latest rev for ID11 UPS, UPS 03.8.

BillP · ‎2021-07-01

I also have this same problem on a bunch of id11's I just upgraded the nmc's on to 6.2 to fix a radius problem and am now getting the connection problems messages. I have the same firmware levels, could you send me the special procedure needed to upgrade the firmware to 3.8?

Anonymous user · ‎2021-07-01

I am also having the same issue with a ID11 UPS. I have the same firmware levels. Could you please send me the special procedure needed to upgrade the firmware to 3.8?

BillP · ‎2021-07-01

Hi,

Tech support is up to speed on this issue. I created a knowledge base article -> http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA241461

At this time, Peter anyone else with this issue, please contact technical support via phone or email (http://www.apc.com/site/support/index.cfm/contact-information/) and just reference this knowledge base article and let them know you require the procedure mentioned in the article and they can easily send it to you and provide guidance. We have just streamlined how we can provide support for this issue as it is difficult for me to be the only one sending out instructions to everyone.

Thanks.

Anonymous user · ‎2021-07-01

Angela - Can you please send me this special procedure. I have been unable to get it from APC support. They keep sending me the normal NMC / Wizard procedure which does not work for this case. They tell me there is no other procedure.

Thanks - Paul

BillP · ‎2021-07-01

Hi Paul, do you have a case or reference number I can get someone to look at for the disconnect in providing the procedure, please? I am trying to avoid being the only one that can figure out how to send this procedure, especially when my presence here on the forum is dwindling due to a new role. I also want to inform whoever it is what the process is.

I'll send you the procedure via a private message here but it is important to me that I follow up with tech support.

Anonymous user · ‎2021-07-01

Angela - I've tried a few times recently. Here is the most recent case - #2788-499424333-5645

Thanks - Paul

BillP · ‎2021-07-01

I have the same issue, can i get the procedure?

thanks

BillP · ‎2021-07-01

On 10/26/2015 1:11 PM, Angela said:
I'll send you the procedure via a private message here but it is important to me that I follow up with tech support.

I am having the same issue. I have contacted support, and they know nothing about a special upgrade procedure. I did share your knowledgebase reference with the support agent. I can send you the case number for reference if needed. Please PM the special procedure.

BillP · ‎2021-07-01

They sent this:

Jason

I am attaching a file called ""SMX_cbl_Updater.7z"". This file contains the script to update the unit to CBL 2.1 as well as the Firmware Upgrade Wizard and SMX11UPS_03-7.enc firmware file. After unzipping the file follow the directions in the Read_ME.txt file. It is imperative that the you follow directions exactly to avoid bricking the UPS. It is strongly advised that you call us on the phone and we will walk you through the upgrade process. The final step in the CBL updater script will launch the Firmware Upgrade Wizard and allow you to apply the SMX11UPS_03-7.enc firmware file. You can reach us on toll free:800-555-2725

Thank you!

One thing i noticed.. the have the file SMX11UPS_03-8.enc in the outside directory of the zip file, but SMX11UPS_03-7.enc inside. i used 3.8 without issue. Its a slow process. I had to make my own serial cable as i didn't have the factory cable. I would read, re-read, then slowly read the directions as you go. seems one bad click or windows error mid-upgrade could brick the UPS. I also used an app called Free Device Monitoring Studio so that i could watch the data over the serial interface. that way i knew data was being passed even when the upgrade screen didnt show it. Good Luck

BillP · ‎2021-07-01

ohh. and there seems to be a bug that if you run the Runtime Calibration, it seems to work, but it will always display that it is still Running even after its done. Seems the only way to clear that is to reset the UPS. and its a PIA to power off all my devices to do so. 🙂

BillP · ‎2021-07-01

Thank you very much Jason! I will go through the procedure in detail before proceeding. It seems that I am getting about a half-dozen "UPS: An internal UPS communication fault exists" errors between the hours of 7pm and midnight every day. For now, I have just turned off the alert notification of that error. I will have to wait until I can power down my loads before attempting the upgrade. I'm wondering if there are any issues with the "UPS: An internal UPS communication fault exists" error, other than cluttering up the log.

BillP · ‎2021-07-01

Prior to the update the runtime calibration would always fail with an error about the load level (which was constant), and it showed i had only a 30-45min runtime. I have the internal battery as well as the external battery chassis (with two new battery strings). My load level was only 15-20%. Right now, at 17.5% load it shows runtime of 3hr and 50min 🙂

I personally think it was causing an issue, and i always seemed to get the alerts at night as well. strange. Its been up 63 days now since updating and the only thing in the logs are normal items, NTP sync, occasional High voltage compensation from Mains..but no UPS fault logs at all.

Anonymous user · ‎2021-07-01

I too have the error, and firmware...

Firmware Revision

UPS 02.2 (ID11)

Can you PLEASE send the procedure ?

This error has been driving me crazy for a while, and I just started to ignore it.

THRILLED to see there is a fix!!
🙂

Thanks!

Greg

BillP · ‎2021-07-01

A few posts up i attached the files along with the docs APC sent with the step by step procedure.

Anonymous user · ‎2021-07-01

LOL...I see that now.

The link was very small, and I totally missed it.

Thanks!

Anonymous user · ‎2021-07-01

Jason -

Any idea if I can use the Network card to upgrade the firmware?

Thanks.

BillP · ‎2021-07-01

You need to use the serial interface for the initial upgrade steps. The text file shows the steps. I did some googling and found the serial pin out online. Was able to make my own cable as I did not have the factory cable.

After it’s up to date with the firmwares I attached I think further updates may be supported for Network mgmt card.

Anonymous user · ‎2021-07-01

Thanks Jason!

Anonymous user · ‎2021-07-01

Hey Angela -

I'm not sure if you are still monitoring this thread, but it seems like support techs still don't know about your FAQ.

I referenced it in the request I opened, and they just told me it's too old, and can't be updated.

48910935

Thank you for finding a way to do it.
🙂

BillP · ‎2021-07-01

Hi Gregory, yes, I still see the updates. I informed the case owner how it should've been handled. Sorry about that.

angelo_79 · ‎2023-05-07

Please send me the procedure as well. I need to update mine from 2.2 to 3.7/3.8. Thank you

BurgerTime · ‎2023-05-18

I'm having this same issue with eleven SMX1500RM2U units and one SU1400RMXLB3U unit. A of them have an AP9631 card running the latest version firmare. Downgrading is not an option for me either.

MHagreis · ‎2024-08-12

i have the same error see related post of my self SMX3000HV ap9631 - Schneider Electric Community (se.com)
can you support me with this error?