Symmetra E7 error help

[Anonymous user]

Hello all,

I have a Symmetra RM with eight Power Array XR's. Recently my XRs are displaying the E7 error code and my Network module is indicating that it has lost communication with the UPS.
I have updated the firmware on the Network Management module (have an AP9617) as suggested here:
http://nam-en.apc.com/app/answers/detail/a_id/615/session/L2F2LzEvdGltZS8xMzM3Nzc5OTM4L3NpZC85U29WRF...

I have tried reseating all of the cards. Both the SYCC, the AP9617, and the SYXRCC in each of the XRs. 6 out of the 8 XRs are blinking with the E7 comminication error code. 2 are not. I do not understand why those 2 are not blinking. If I put one of the non-bliking SYXRCC cards into a XR unit that is blinking, it the XR still blinks with the Error, which suggests to me that it is the SYCC causing the problem, or at least I think so. Of course an SYCC card apparently takes a couple weeks to get.

So... my question is. Is my Symmetra UPS enviroment still active(i.e will still provide batter backup), even though I am unable to view their status? Is the Powerview and Remote management simply for viewing the batteries and providing notifications? or if the communication isn't working then the batteries won't either.

also if anyone has any more troubleshooting to try please let me know. Thanks.

Jeff

[Anonymous user]

I finally recieved the new SYCC unit today. I plugged it in. And.... no change. My units still read E7.
What else should I try?

[Anonymous user]

Ah yes we only have 7. I keep counting them wrong since the XR Frame starts at 2, and the 7th XR Frame says 8 on the display.

I have tried just adding the XR frames that are not displaying the E7 error to the SYCC, however the SYCC still doesn't see them. The 'working' XR Frames are set at XR frame 3 and 4. If I add them to the SYCC and only those two to the SYCC do I need to set them to 2 and 3? i.e. does the first XR frame always have to start with 2?

Can a communication cable simply just go bad?

We have had a couple unexpected power outages lately, and at least one one case the batteries had been completely drained. All the XRs seem to come back online after the the outage, however the first XR in the series (XR Frame setting of 2) started showing the E7 error. Then a couple days later 5 out of the 7 show'd the error. Is that a sign of bad SYXRCCs or one bad SYCC?

thanks for your help

[Anonymous user]

I'm also interested in the answer to this. I have a Symmetra RM with 2 XR frames which freaks out every now and again (1 or 2 months between random occurrences). It will report a flurry of bogus messages (NMC loss of communication with UPS, over temperature, number of battery packs decreased, and so on). While this is happening the management interface reports "UPS not discovered". This is accompanied by a flashing E error on one or both XR frames. I'm not sure what the error is as I haven't really caught it - I have to run into the server room when I hear the RM beeping. After anywhere from 1 minute to 15 minutes or so, the management card will log a set of "returned to normal" messages.

The MIM/RIM and all of the main frame and XR communication cards and cables have been swapped to a different RM, and the problem remains in the original RM.

I suspect that all communications are happening over an I2C bus or similar and when one device has a problem it jams up the bus and prevents all communication. The only parts that haven't been swapped are the power modules and the batteries (and the batteries aren't on the serial bus, as far as I know).

Your problem seems different as you have a constant E7 error and (presumably) the RM front panel / web interface operate normally aside from not showing the correct number of XR frames / batteries.

If there isn't a live load on your UPS (or you can temporarily shift it) you could try configuring the bare minimum Symmetra RM - one PM, one battery, MIM only and cold starting it (System Enable off, main input breaker off). Switch those back on, then add components one by one - you can start with a single #2 XR frame and single communication cable with no batteries, then add batteries one at a time, adding on the next XR, and so on. Then the RIM, then the PM's. Since you're having a constant E7 it should be easier to pin down than my very intermittent one. You're going to get complaints and asked to confirm turning on the load due to missing RIM / insufficient runtime / missing batteries when you do a cold start in the minimum configuration. IMPORTANT: This is definitely going to disconnect the power to anything powered from the UPS.

I'm just a customer and this is the procedure I follow when tracking down Symmetra faults. APC may say "don't do that" or have an alternate methodology. You should listen to them and not me in that case.

[BillP]

hi terry - i wanted to comment on one of your points/questions:

I'm also interested in the answer to this. I have a Symmetra RM with 2 XR frames which freaks out every now and again (1 or 2 months between random occurrences). It will report a flurry of bogus messages (NMC loss of communication with UPS, over temperature, number of battery packs decreased, and so on). While this is happening the management interface reports "UPS not discovered". This is accompanied by a flashing E error on one or both XR frames. I'm not sure what the error is as I haven't really caught it - I have to run into the server room when I hear the RM beeping. After anywhere from 1 minute to 15 minutes or so, the management card will log a set of "returned to normal" messages.

NMC loss of communication to the UPS sort of makes sense to me to see along with the E7. E7 is saying the battery frames cannot talk to the UPS and if the NMC cannot talk to the UPS, it tells me there is maybe an issue on the UPS side in your scenario. since the NMC cant talk to the UPS at this time either.

what firmware is on your management card for AOS and sy app? we'd gladly take a look at these logs. the NMC loss of communication to the UPS message seems legit but with the other messages, maybe it is just an older firmware with an older UPS/MIM?

[Anonymous user]

Yes, it seems to be on the UPS side as the front panel reports the same things. Also, the management card has been swapped (I forgot to mention that in my prior post) and there's no way for the NMC to trigger an E7 on the XR frames.

NMC AOS version: 3.7.3 SY version 3.7.2
MIM/RIM master version: 137 slave version: 223
Power modules version: 310

[BillP]

hmm ok..MIM is a bit older but NMC is up to date..if it were me, i might look at a new MIM and trying it alone with no RIM but i also would look at any event logs before actually suggesting a purchase. i'd also see if clarkson agreed.

[Anonymous user]

I've swapped the MIM/RIM between this unit and another Symmetra and the problem stayed with this one, so I'm pretty confident it isn't the IMs.

Since the power modules are on the same data bus and they haven't been swapped, I'll try them next. The PM's that are in there now came as APC warranty exchanges way-back-when because the original ones made a noise like an angry swarm of bees whenever more than one was installed. I think there was a production change to address this about a year after my purchase.

[BillP · Accepted solution]

Does the powerview appear to be working properly? You can try disconnecting the powerview or the NMC and see if this affects communication on the NMC or with the XR's. You have ruled out the XRs, and the SYCC so it's likely the MIM. The powerview and NMC itself could theoretically be causing this too which is why i suggested the above steps.