"System ID card" removed errors on Symmetra PX

[ojfsdi]

We often see this fault:

Symmetra 80K: APC01 at Datacenter

1 Critical Alarm Present

System ID Card Removed

Why do System ID cards exist?

What information is stored on them?

Can they be ejected and replaced without interrupting load?

What about management cards?  (Referring to the side by side dual/twin modules in slot L1 at the very top of the Symmetra frame) If I removed both management cards would that 'freeze' the ups in it's current state, or shut it down completely?

If the management cards are ejected, would that create a safe time during which the System ID card can be removed and reinstalled without interruption?

I kind of think surely it's okay for me to pull it, because if the UPS is already telling me it's been removed and the load hasn't dropped, then there's no "difference" between an ID card that isn't detecting and one that is physically missing.  Is that logical?

[al_buffington]

Hi,

I've tried to address each of your questions below. 

Why do System ID cards exist?

The ID cards are embedded in the static switch module and are used to store various system settings.

What information is stored on them?

ID information about the system that is set during the manufacturing process, statistic information related to UPS operation (time spent on battery etc.), user entered UPS parameters that were set on either the display or network card (battery test interval, load thresholds, etc)

Can they be ejected and replaced without interrupting load?

Yes

What about management cards?  If I removed both management cards would that 'freeze' the ups in it's current state, or shut it down completely?

Are you referring to the network management cards? If so, they can be removed while the UPS is running without affecting the UPS. If you are referring to something else, please clarify.

If the management cards are ejected, would that create a safe time during which the System ID card can be removed and reinstalled without interruption?

The management cards being present/not present have no bearing on the ID card

I kind of think surely it's okay for me to pull it, because if the UPS is already telling me it's been removed and the load hasn't dropped, then there's no "difference" between an ID card that isn't detecting and one that is physically missing.  Is that logical?

Correct. You must keep in mind that a UPS cannot start without the ID card as it will not be able to load the pre-programmed manufacturing settings. If the card was missing, or removed, and you powered down the UPS you would not be able to bring it back online if it was not being recognized as installed.

Some other things to consider about this alarm:

• The fact that it is reporting as removed, and not failed, may mean that the card is poorly seated.
• Because the card sits within the static switch module there is a chance that the ID card is not the root cause of the problem, and that an issue exists with the static switch module.  This module can only be serviced with the UPS in maintenance bypass by a trained FSR. You should never remove the static switch while the UPS is running.
• If the ID card needs to be replaced, the replacement must be specially programmed for your UPS. You cannot simply swap one from another UPS or acquire one from some other source.
• If you have other alarms that are active it may alter the above assessment. 

[ojfsdi · Accepted solution]

I mis-spoke.  When I said management cards,  I should have said 'Intelligence Modules'.  I'm kind of surprised that the ID card would exist, as a single point of failure.  Why wouldn't they keep that data replicated between the two intelligence modules?

What happens if both intelligence modules are removed at the same time?  sorry this is somewhat off topic from my main question.

I think I solved the ID card problem, and I think you were right to suspect thinking it wasn't seated fully/properly.

When I ejected it and particularly when I re-inserted it, it did not feel like a particularly positive engagement of the connector.  It felt like there was no contact at all until the "lid" of the ID card was half of a milimeter from the surface of the static switch, perhaps closer.

My intuition was that it should engage over a longer distance.  So I took the card out again and examined and noticed that the lid is connected to the circuit board by two screws.  The holes in the circuit board are larger than the screws of course, so there's some play to it....  I loosened them, and then pulled the circuit board away from the metal lid piece, and re-tightened them under tension.

This makes the card stick into the static switch further. I re-inserted it, and felt more engagement of the pins this time.  And I've been monitoring the syslog for 48 hours now and haven't seen a single event, whereas before it would happen a couple times a day.  So. I think I fixed it, and hopefully this hack helps someone else too.

My other fear was that the receptacle for this card was dislodged inside the static switch, or bent or loose in there, and I doubt I can take the static switch out without dropping load, so if that was the case I'd need to fashion a custom bracket for the ID card or insert it with needle nose pliers and no bracket. (with much caution)

[al_buffington]

I'm glad you have resolved the problem. If there have not been any events over the past couple of days then I am optimistic that the issue will not return. While not impossible, I believe it's unlikely that the problem is with the receptacle inside the static switch. The connector that mates to the ID is on a PCB inside the module. You are correct that the static switch cannot be removed while in operation. This is primarily due to safety concerns. Regardless, the module is not considered user-serviceable anyway.

I’m not sure I would call the ID card a single point of failure. Even if it “fails”, or is removed, the UPS will keep running on-line. Theoretically speaking, having the information that is on the ID stored on the IMs would introduce some complexity if the Intelligence Modules ever needed to be replaced.  Many times both IMs are replaced together  (firmware update purposes, etc.).  

To answer your other question, if you remove both IMs the UPS will transfer to bypass operation.