SMX3000 Internal fault message

[MarkC_apc]

Trying to track down the cause of the LCD message "internal fault"

The UPS works under full control of the NMC2.  All menu items work perfectly EXCEPT when I try to do a battery calibration test, it "refuses" and gives a message that the "source" is the Management Device and the "cause" is SOC not acceptable.  All batteries have been load tested and where fully charged to do the calibration (according to the data log).  Output voltages and load capabilities are all OK.  Again, batteries have expected load capabilities (fits load/time curve well) - however are NOT new.

I also have a UPS fault log number 154 "interprocessor communications fault"

I am not able to use the front control panel - it is non- responsive except for a single beep when the power button is pressed (power is not affected).  Again all functions work from the web interface of the NMC2

I've turned "green mode" off

NMC2 firmware is v6.2.1

UPS firmware is 01.8 - ID 10

Have tried to update the firmware to UPS 02.0 using the "wizard" and also ftp, both of which have failed.

Any suggestions?  Would like to be able to use full functions of this very efficient inverter with extended battery packs, but would like to be able to fix the internal fault message if at all possible and use the front control panel rather than depending upon the NMC/computer interface.

Mark

[voidstar_apc]

There are three processors in the system: one running the NMC (this is fine), one running the LCD display (this is fine), and a third doing the actual UPS tasks (it sounds like this one isn't talking). In this condition, none of the controls will work even if everything looks okay. I've only seen this after a failed firmware upgrade

Hopefully tech support on this forum will have some tips; my only suggestion is to try downgrading to SMX10UPS 01.8 which may fix it if the UPS processor got upgraded but the one for the LCD display did not.

[MarkC_apc]

On 5/14/2015 10:09 PM, voidstar said:

Hopefully tech support on this forum will have some tips; my only suggestion is to try downgrading to SMX10UPS 01.8 which may fix it if the UPS processor got upgraded but the one for the LCD display did not.

By downgrading, do you mean to attemp a reload the file "SMX10UPS_01-8" as the UPS firmware is currently 01.8?  I tried to upgrade to SMX10UPS_02-0.enc, however it shutdown the NMC at the end of the wizard and would not finish the upgrade.  Using the (dos) ftp method, at the end of the transfer the unit shut down,  beeped continuously and I had to cold start it back to operation.  Again, I can change all the variables of operation with from the NMC.

Thanks,

Mark

[BillP]

I was thinking that I'd like to see the log files from the NMC and see if anything jumps out in there. event.txt, data.txt, and config.ini would be required or the .tar file dump since you're on v6.2.1 and can retrieve that though About->Support menu. Instructions here on how to download if you're not familiar -> http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA156131 

I also would be interested in a screen capture of Logs->UPS menu that shows the last 10 transfers and last 10 faults from the UPS through the NMC. This is not included in the .tar dump.

[voidstar_apc]

Yes, load "SMX10UPS_01-8.enc" back in via the Firmware Update Wizard. While the display is currently at 01.8, the micro running the UPS may not be.

I agree with Angela, grab the NMC's debug logs while you're at it.

Also, the first time you tried to do a firmware update and it failed, were you using the Firmware Update Wizard or the NMC? I assume it was the FUW from your statement "it shutdown the NMC at the end of the wizard," but you might also be referring to the NMC's web UI.

[MarkC_apc]

On 5/16/2015 4:15 AM, voidstar said:

Yes, load "SMX10UPS_01-8.enc" back in via the Firmware Update Wizard. While the display is currently at 01.8, the micro running the UPS may not be.

I'm having trouble locating the legacy file - any idea where it can be found?

Thx,

Mark

[MarkC_apc]

I've been traveling - will try to get the info to you tomorrow.

Mark

[MarkC_apc]

On 5/18/2015 9:47 AM, Angela said:

Anyway, I'll attach the UPS's 1.8 file for you here if you want to try it. I haven't really encountered this issue before so I'll defer to voidstar if he has any further suggestions but yes it definitely seems to be a UPS issue here so hopefully we can address this firmware issue and get it working. I also too just wanted to clarify specifically what tool you used to try and upgrade the UPS firmware originally.

Thanks for the file - I'll try the upgrade first thru the NMC wizard and let you know the results. 

I used the NMC firmware upgrade menu for the first attempt to upgrade the UPS firmware to 2.0.  The original NMC firmware did not have that option, so I upgraded the NMC firmware from a direct file load "wizard".  I also (with help from APC tech) tried a dos based ftp method for 2.0, which seemed successful until the end when the whole unit went into a continuous alarm - had to "pull the plugs" and cold start.  None of this affected the operation of the unit when restarted and monitered thru the NMC.  BTW, I do like the new NMC-2 firmware/software interface.

[MarkC_apc]

Sequence of events:

1.  Cold start unit - Immediately the LCD reads "internal fault"  Turned all outlet off - confirmed with light bulb.

2.  Initiated firmware update thru NMC - LCD screen reads "Firmware Update in Progress"

3.  File transfer completes in a few minutes - the NMC is powered off (no lights) and the web connection is lost with message "UPS NMC2 page unresponsive".   LCD same message - no buttons respond.

4.  I "kill" the page as waiting does no good

5. I attempted to use hyperterminal to access thru the "console" connection on the NMC - no response - NMC has definitely been powered off

6.  Pulled AC plug - no change

7.  Pulled DC plug - unit shutsdown

8.  Cold start - all conditions same as before

9.  Tried battery calibration - "refused", "management device", "Invalid state"

10. The UPS fault log had an additional "154" fault for a total of 3 (the screen shot sent earlier had 2).

11. Nothing of note in the event nor the data logs

Hopes this helps in any analysis,

Mark

[MarkC_apc]

Just noticed that I replied to myself, which may not alert you.  The test results are previous message.

Thanks

[BillP]

Hi Mark,

Sorry for the delay. I asked one of our UPS firmware managers and he confirmed that there is an internal problem that was not caused by a firmware update or anything like that. Unfortunately, it is broken and there is nothing the user will be able to do in order to fix it as one of the processors is not talking properly.

I presume this unit is under warranty so that tech support can process a replacement for you?

[MarkC_apc]

It is out of warranty (2011 model).  What a shame as it seems to function essentially 100% as long as i am using the NMC interface to determine status.  Battery calibration is not such a big deal for me as I use it with external batteries with substantial backup capabilities during extended outages.  What I like about this unit is it's high power factor and very good efficiency levels at all loads.

Thanks for all your help,

Mark

ANYONE ELSE HAVE SUGGESTIONS??  Sending it to Coast Tech is simply too expensive.

[BillP]

Hi Mark,

No problem. Of course you could still use it the way it is. Did it ever work properly for you since 2011 or did you obtain it recently?

[MarkC_apc]

I obtained it recently (early 2015) with three out of the 10 batteries getting weak - probably going bad, which seems to be the case so often - get "discarded" because the APC battery "test" (appears to be a spike load) fails and they get rid of them.  But this one has an internal fault of some kind.  I do mainly renewable energy type work - so it will make a nice back up DC/AC inverter for solar panels - just not as dependable without the LCD warning of problems.

Mark

[MarkC_apc]

Angela - can you tell me what management cards will work with the SMX3000?  I might try a an older "spare" (AP9617) rather than keeping the much more valuable AP9631 in the unit - as long as I can moniter/control thru the NMC.

[BillP]

Mark,

Only the Network Management Card 2 (AP9630/AP9631) would work on the SMT/SMX units.

Jon

[MarkC_apc]

Voidstar;

Seems to hit another dead end with this relatively valuable and functional unit.  Any further suggestions or avenues to pursue?  Without wiring schemas, it is improbable that I can do any meaningful hardware checks (outside of tightening all ribbon connectors!).  I'm going to look for a "for parts" duplicate unit and try replacing each separate circuit board - with the idea of not a lot to lose!

Thanks,

Mark 

[Terry_Kennedy_apc]

One thing to be careful of is that APC would make design changes without changing the model number. So random combinations of boards may not work together properly. The closer the two serial numbers (your unit and the parts unit) are, the better your chances.

Also, very old APC designs (not your UPS) performed the factory setup procedure by installing different values of components in various spots on the boards. Newer units do most of this setup via software. You probably want to adjust (at least) the battery float voltage after building a unit out of parts.

Of course, none of this is supported / recommended by APC.

[MarkC_apc]

Thanks for the advice.  Since my ultimate goal is to use the DC/AC inverter for off-grid power, with Li technology batteries, the charging will be handled by a MPPT CC anyway.  However, I'd be more comfortable with a working LCD display - even though the battery alarms will have to be silenced - especially for long term testing in this off-grid mode. 

If Anyone is intereted in helping (with enough working knowledge of these units to do this type of repair), i'd certainly be willing to share any results of such off-grid useage/testing.

[voidstar_apc]

Hi Mark,

Friend me and send me a private message.

[MarkC_apc · Accepted solution]

Hit a dead end on this particular unit (LCD display issues) - which is such a shame with the condition (great) and capabilities of this SMX3000!  It became apparent that under careful scrutiny, that the main processor on the motherboard (not sister board) had an attempted repair in it's past - with bent pins and what appeared to be possible bridging solder.  I was able to remove the processor successfully with "chip quick" (very easy to use) resulting in a clean surface.  One of the pin was severely bent and actually fell off when I tried to straighten it.

So - last gasp at repair!  Any suggestions on a path to repair?  I'd be willing to try scavenging parts from another failed motherboard if it makes sense - the "chip quick" used with a temperature controlled iron would seem to allow removal and reuse of a good processor?