"Unplugged" sensors following DCE upgrade

[DCIM_Support]

Following updates of DCE on several servers recently I have noticed a LOT of sensors on device are now reading unplugged.  The most effected devices are APC Rectifiers and Enlogic PDU's.  On the Enlogic PDU's all power related sensors have been duplicated with all the 'old' sensors now saying 'unplugged'.  I think this has caused some sort of knock on effect to DCO too.  This appears to have happened on all Enlogic Inline meters and PDU's on the DCE servers.

On the Rectifiers all but one of the sensors has been duplicated with all the old ones saying 'unplugged'. I know I can get rid of the 'unplugged' sensors by selecting them in sensor view and deleting them.  However many of the power sensors have been shown on maps to display the power usage of the PDU and have also been used in virtual sensors which therefore also now show unplugged. 

There are also the same effects to a lesser degree on UPS's (SURT's, PX series, SUVT series so far)

It looks to me as though the DDF files have been changed during the server updates causing the issues.

Given that there are 100's of effected sensors across many devices on multiple servers that now need to be corrected, is there a quick fix for this or will I have to manually update every single maps and sensor on every server? Is this expected behavior?   I could find nothing in the release notes to suggest this may happen.

If it helps I suspect the issues were caused by V7.3.1 upgrade and 7.4.2 upgrade but can not be 100% sure of this.

Urgent answers required please...

Regards

(CID:110795584)

[DCIM_Support]

Hi Garry,

This is a known DDF issue. Duplicate sensor IDs were created for interaction with DCO. I do not believe this was fully international however it is being looked at. We do have a newer DDF that can be added to the system so that the now unplugged sensors will revert to their original communications. I believe however that for each unplugged sensor, a new sensor was made. This would require that after upgrading the DDF, the new sensors would need to be removed...from each device.

If you have a recent backup, the quickest resolution may simply be that  yuo do a restore and import the backup. I'm looking to get an updated DDF and will post it as soon as I get it.

Steve

(CID:110795597)

[DCIM_Support]

if you still need to remove unplugged sensors, a quicker way to do it is to:

• select a group of the same devices,
• right click, click view device sensors
• search for "unplugged"
• select the unplugged sensors and delete

(CID:110795610)

[DCIM_Support]

Hi Garry, Here's the updated DDF:

apcsmartups_v75.zip

Steve

(CID:110795600)

[DCIM_Support]

Thanks for the response, but before I start any more changes, I just want to make sure I am not making more work for the future.

Please advise / comment / answer on the below:-

On the Enlogic devices, DDFs were changed to allow DCO to recognise some of the sensors (mostly power related) that it previously could not recognise and therefore allow it to report energy usage on.   I know this from a previous issue we had with this.

I assume based on the above, the same sort of changes have been made to some APC DDF's too suggesting that they too do not interact well with DCO.

Reverting back to the previous DDF will then show all the sensors that are now showing as unplugged as their actual previous values, but the 'new' sensors will then show unplugged?  These will then need to be deleted...
This also means that the sensors put back in place by the DDF you attached will not work with DCO properly?

On the basis that these changes were made to the DDF's to make the sensors work with DCO I can presume that at some point in the future the DDF's (like the one you attached)  will change again to make them work properly with DCO and I will need to change everything anyway...again...

Because you have created new sensors in the DDF's replacing all the old sensors - this means that if I delete the old 'unplugged' sensors I will also lose all historical data from these sensors too.

The DDF you provided is also only for one type of UPS,  I so far have seen this issue on all the Enlogic inline meters and PDU's that we have attached to DCE, It has also effected every single type of APC UPS we have connected to DCE, (PX, PX250/500, SUVT, SURT, SRT, SIlcon) and APC MAGNUM VS Rectifiers.  It does not at least appear to have effected the APC PDU's (yet).
That single DDF provided will not revert all the sensors on all these devices back to how they were. I need several 'correct' DDF's to get back to where we were before.

To be honest this all makes me very uncomfortable as to whether anyone has a handle on this.  Will more changes to DDF's come in the future for different devices that will cause similar issues?

This all seems like a pretty messy deployment error which has already cost me several hours. I need some sort of proper explanation for this and an explanation of how this will be fully resolved both now and for the future. I can't just start throwing patchy type fixes to multiple servers that are monitoring 1000's of devices for multiple customers.

It would be very helpful to know which server updates have introduced these changes too, so I at least have a chance to plan around the issues.

Regards

(CID:110795649)

[DCIM_Support]

Hi Garry,

First, I can't comment on the Enlogic devices. I am aware that especially in the past and especially with 3rd party devices, many DDFs had been created without thought of how to get the data over to DCO. I can only assume this same issue occurred with those DDFs however I know nothing about changes to those Enlogic systems specifically.

You also assumed that other DDFs had been similarly changed but in my conversations with engineering, I have not been made aware of any other APC DDF that has recently been changed. The initial issue of this type actually occurred in 2012 as noted in k-base FA158441.

Yes, the original sensors were indeed duplicated. If you add the DDF, you will have the original sensors again showing values and the new versions of the sensors will go unplugged. This is why I noted in my initial response the easiest and cleanest way to resolve this issue is to revert back to a previous version if you created a backup prior to the upgrade. Updating the DDF to the attached version was a second option but I felt I needed to provide you with that option.

As for future changes, I do not believe this will be done again at least not in this same way. I believe they are  looking to make changes for future systems to actually do a clean-up of the sensors by the system itself should the need to change the IDs happen again in the future for APC devices. I can't say that this is something that has been thoroughly worked through but since they are going back to the previous format with this DDF, I believe they do not want to inflict this pain on the customer base again.

The DDF provided is the SNMP APC UPS DDF. Symmetra units, Smart-UPS units, SURT units, and any unit that uses the 318.1.1.1  or apc.products.hardware.ups branch of the MIB is effected by this. There are potentially other additional DDFs that may be used but the "Smart-UPS" DDF is the base APC UPS DDF.

This issue occurs in 7.4.2. As far as I have been made aware, there have been no other recent updates that have caused similar issues. As for this being a messy fix, I totally agree. I am part of tech support and as such, we deal with most  customers that are affected by this issue. We must walk customers through this so although it does not affect our teams as much as it does you, we do have to assist and we do also feel the pain of increased effort.

This is the best explanation and suggested workarounds I have and what I can suggest. I will ask engineering / product management if they can offer an official statement or suggest a response to your comments directly.

Steve.

(CID:110795656)

[DCIM_Support]

Hi Steve thanks for the response.

So just so that I understand this...
 If I apply the DDF attached it should correct any issues on the UPS's (only)?
In future if the DDFs change again then the engineers may well have found a way apply these changes without the unplugged issue appearing (hopefully)...

Any tips on how to identify this in future?  Manually browsing through the sensors of thousands of devices is not very practical.  It is actually quite easy for this to go unnoticed and even worse, if any DCE thresholds / alerts have been set up on sensors that suddenly become 'unplugged', DCE will not (cannot) alert you to this, AND the alert will no longer work.
It is highly possible (probable?) that there are many DCE servers out there  with these problems that the users are unaware of...

Could you investigate the state of the DDF for the MAGNUM rectifiers too please? Almost every sensor on this device has changed and since device alarms are not provided, all the alert thresholds are created from within DCE, meaning that now all of those alert thresholds are incorrectly mapped to the old 'unplugged' sensors and need correcting.  We get no alerts at all from these devices now except Warning and Critical.  This is arguably enough, but not what we originally set up.

Can you confirm if the 'built in' DDF's changed or were altered by the 7.3.1 upgrade too please?  I am relatively sure on 7.3.1 for Enlogic PDU's we saw sensor changes.

Could we ask that in future, the DDFs that are updated in the server update are included in the release notes too?  At least we have a chance to know where to look for problems.  It would be good if someone at Schneider could compile that list now.

Regards

(CID:110795843)

[DCIM_Support]

Hi Garry,

The DDF will effect the APC UPS units only. I can't say it will "correct" the issue as although as we both noted, the original sensors will again start logging data and won't be shown as unplugged, the "new" sensors will be unplugged.

I have no real tips to identify this in the future. You can see it if you run reports or view sensors but there is no easy way to be notified and, would you want to. If a series of sensors goes off line for legitimate reasons, you don't want every one of them individually messaging you about it. I can say that this is something that is being addressed at a high level right now.

As for the Magnum and enlogic DDFs, I have asked the DDF team to look at this. Hopefully they can check out the post directly but I have also sent the info in a direct e-mail to one of the main players in that arena.

As for release notes, there are many DDFs that are changed in almost every revision. I can ask and it is possible that they can note which DDFs were updated but I doubt they will go into any detail.

Steve.

(CID:111413015)

[DCIM_Support]

Hi Steve.
I deployed that DDF file you posted onto our test server that had recently been upgraded to DCE 742.  It does not seem to have done what I expected.  I expected as you said 'original sensors will again start logging data and won't be shown as unplugged, the "new" sensors will be unplugged.'  That has not happened.  All I can see is that the original sensors are still showing unplugged but they are not logging data on the reports graphs, whereas before they were showing the last value that the unplugged sensor read.

What I did was select 2x UPS and choose a sensor on each that was showing unplugged (UPS operation mode) .
I deleted one of the unplugged sensors on one of the UPS so I am left with 2 'good' sensors and 1 unplugged sensor.
I uploaded the DDF file provided and after several hours I still have  2 'good' sensors and 1 unplugged sensor.  If the DDF had taken us back to before I should now have 2 good sensors and 2 unplugged sensors.
The values on the graph plots on these sensors should also have swapped over when the DDF was applied and this has not happened.

(CID:111413138)

[DCIM_Support]

I stated 'the original sensors are still showing unplugged but they are not logging data on the reports graphs, whereas before they were showing the last value that the unplugged sensor read.' 

After double checking this is incorrect. The behaviour is the same as before.  The DDF has been accepted by DCE as it now shows at version 75.  I can't see what this DDF has changed.

(CID:111413140)

[DCIM_Support]

Gerry,

I'm trying to get any detail on how this change should manifest itself. I have judged my statments on what has historically happened with this type of change. I don't have a system that is showing as many unplugged sensors so I'm looking to engineering to provide some info. I'm assuming the duplicate sensors are reporting values, correct? I had also suggested the best way to resolve this was to roll back to the previous version of DCE if you had a backup. Do you have such a backup?

Steve

(CID:111413563)

[DCIM_Support]

Hi Garry,

I work on the DCE Engineering team, and wanted to give you an update on this situation. First off, I wanted to apologize for the frustration and loss of time this has caused you. We are taking this very seriously and this is the top priority for everyone that can help out.

There are multiple efforts going on right now that I can outline:

• We've identified a breakdown in communication as to how DCO support should have been added to these DDFs. There is a more sophisticated way to add DCO support that does not require changing sensor IDs. This has already been communicated to the DDF team, and any additional training that needs to happen will begin ASAP.
• We have identified a gap in our automated test coverage with respect to changing sensor IDs across an upgrade. We are in the process of building a suite of tests to catch any sensor that goes unplugged after an upgrade in order to catch this before we release the product.
• We are looking into every DDF change made to DDFs that ship with DCE from 7.3.1 until now. After identifying all sensor ID changes we'll revert the sensor ID to what it originally was.
• We're building 7.4.3 which will have all of the fixed DDFs in it, and if possible it will automatically clean up the extra sensors that were created in any of the previous product upgrades. Customers under support with 7.3.1, 7.4.0, 7.4.1, or 7.4.2 will be able to upgrade directly to 7.4.3. I can't promise any dates, but we're hoping to have this release done within the next week and a half.
  
  

For customers wondering what to do about upgrading now, I would suggest waiting until 7.4.3 becomes available. If that is not an option, then I would upgrade to 7.4.1 now and then 7.4.3 once available.

-Scott

(CID:111414146)

[DCIM_Support]

As an update to my comment, it is turning out to be significantly more work than we originally anticipated to properly fix and cleanup the duplicate sensors. We are still working on the fix and evaluating all of our options. Future updates about the fix for this issue will be communicated on DCIM Support outside of this thread so that the information is in a more visible place.

(CID:111937519)

[DCIM_Support]

Thanks for the update.

Do you have any indication of timescales at the moment please?

(CID:111937527)

[DCIM_Support]

Any update to timescales on this yet please?

(CID:114233693)

[DCIM_Support]

Hi Gary,

The only update that I have to this point is that k-base FA310594 has been written to address this issue and the resolution as you're aware is to wait for DCE 7.4.3. I don't have an ETA at this time but I will see if I can get something for you.

Steve

(CID:114233852)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.