Devices lost in DCE after rebooting

[jzurera]

Hi team,

in a customer with DCE 7.9.0 we have lost all the devices that were monitored in the private network after a server reboot. It was a non controled reboot, the host where the DCE virtual machine was located was disconnected to the power supply by error. 

After the reboot all the devices dissapeared. The customer had to restore a backup to recover the configuration. 

This is the log of the error if you could check what could happen:

10/08/22 18:41:36.974 INFO - StruxureWare Data Center Expert server version: 7.9.0

10/08/22 18:41:36.975 INFO - StruxureWare Data Center Expert server model: VIRTUAL

10/08/22 18:41:36.975 INFO - StruxureWare Data Center Expert server build: 193

10/08/22 18:41:36.975 INFO - StruxureWare Data Center Expert GUI minimum version: 7.9.0

10/08/22 18:41:38.045 INFO - Starting StruxureWare Data Center Expert

10/08/22 18:41:43.757 INFO - Dynamic root password has been set.

10/08/22 18:41:44.645 INFO - Loading Operating System information

10/08/22 18:41:44.649 INFO - Loading Modules....

10/08/22 18:41:45.508 WARN - log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).

(com.netbotz)

10/08/22 18:41:45.508 WARN - log4j:WARN Please initialize the log4j system properly.

(com.netbotz)

10/08/22 18:41:56.187 INFO - Initializing connections to the database...

10/08/22 18:41:57.907 INFO - Database connections initialized

10/08/22 18:42:00.421 INFO - mxBeanName: IsxcQueues:name=EventServiceWorkQueue

10/08/22 18:42:01.029 INFO - mxBeanName: IsxcQueues:name=EventQueueThread

10/08/22 18:42:02.143 INFO - mxBeanName: IsxcQueues:name=ACULQueuedefault

10/08/22 18:42:02.152 INFO - mxBeanName: IsxcContextListeners:name=default

10/08/22 18:42:02.465 INFO - Varlib Initializer start

10/08/22 18:42:09.917 INFO - Varlib Initializer stop

10/08/22 18:42:09.942 INFO - mxBeanName: IsxcQueues:name=VarLibPersisterWorker

10/08/22 18:42:09.955 INFO - (ISXC) Initialized

10/08/22 18:42:09.981 INFO - mxBeanName: IsxcQueues:name=ACULQueueISXC

10/08/22 18:42:09.981 INFO - mxBeanName: IsxcContextListeners:name=ISXC

10/08/22 18:42:10.189 INFO - IP Interfaces loaded

10/08/22 18:42:10.199 INFO - private network settings loaded

10/08/22 18:42:10.221 INFO - NetworkService started

10/08/22 18:42:10.591 INFO - Enclosure Bb17b3a_nbSNMPEncA2CC634F appears to be orphaned, removing.

10/08/22 18:42:10.616 INFO - Enclosure Bb17b3a_nbSNMPEnc29730E84 appears to be orphaned, removing.

10/08/22 18:42:10.616 INFO - Enclosure Bb17b3a_nbSNMPEnc818BC6CA appears to be orphaned, removing.

10/08/22 18:42:10.616 INFO - Enclosure Bb17b3a_nbSNMPEnc0D89F975 appears to be orphaned, removing.

10/08/22 18:42:10.616 INFO - Enclosure Bb17b3a_nbSNMPEnc47661CC6 appears to be orphaned, removing.

10/08/22 18:42:10.617 INFO - Enclosure Bb17b3a_nbSNMPEnc42CEDE80 appears to be orphaned, removing.

10/08/22 18:42:10.617 INFO - Enclosure Bb17b3a_nbSNMPEnc978B5F84 appears to be orphaned, removing.

10/08/22 18:4

10/08/22 18:42:18.196 ERRR - Device Bb17b3a_nbSNMPEnc4EA7FA50 does not exist, removing from group SAIs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.196 ERRR - Device Bb17b3a_nbSNMPEncCE4387D2 does not exist, removing from group SAIs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEncBC051CD8 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc41620400 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc928CF61C does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc279D373E does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEncC8A1942B does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEncADD5E2F4 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc3438EAFB does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc5EAF3093 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEncB8A11CF1 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEncD239CD84 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc23AD1B49 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc7834FB32 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc0F675768 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc775CFDBE does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEncC5371D25 does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.217 ERRR - Device Bb17b3a_nbSNMPEnc7A574FAF does not exist, removing from group PDUs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.238 ERRR - Device Bb17b3a_nbSNMPEnc10CB1D4C does not exist, removing from group Netbotzs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

10/08/22 18:42:18.238 ERRR - Device Bb17b3a_nbSNMPEnc56E0E855 does not exist, removing from group Netbotzs

(com.netbotz.server.services.inventory.impl.DeviceGroupServiceImpl)

[Mike-Sutton]

Hi Jzurera,

Have you tried rolling back the VM to its last snapshot or backup point 

This should work fine

Mike

[APC_Steve]

Hi @jzurera ,

I would ask that you send this in to tech support directly where they can look at logs and potentially get more info. If they were to request info such as potentially a prior backup of the system etc we wouldn't want to do that through the forum.

Thanks,

Steve

[jzurera]

Done. Thanks Steve.

[MuhammedElMahy]

Hi Jzurera,

Did you find any solutions to this issue? I'm facing the same issue right now in a customer site.

Thanks.

[eslamsabry2230]

Dears,

I am facing the same issue on DCE version 7.9.1 as all discovered devices missed from DCE, Have that issue solved??

[APC_Steve · Accepted solution]

Hi folks,

Sadly once you've encountered this issue with DCE 7.9.0 or 7.9.1, there is no resolution other than re-deployment and bringing in a backup.

APC has determined the root cause of this issue however and a fix has been implemented in the new 7.9.2 release of DCE that should prevent this issue from recurring. Please contact your local tech support for links to 7.9.2. (and 7.9.1 if you're still on 7.9.0).

Thanks,

Steve

[jzurera]

Hi @APC_Steve ,

with this server (DCE 7.9.0) we have had another issue. A lot of devices have dissapeared from DCE suddenly.  We have tried to add them again and it is not possible, but from DCE web page the SNMPwalk works perfectly for each device, but when we try to discover them in the DCE desktop client, they don't appear. 

We have lost around 30 devices from a total of 350 devices, all are APC products (ATSs, Netbotzs, Inrows, PDUs...) 

The customer has tried to recover from a backup and it did not work. I have tried to do a new backup and it failed, but the repository mount test worked fine. 

I have tried to upgrade to 7.9.1, we have been waiting for 2 hours and the web page says "Espere mientras Data Center Expert se inicializa" (Wait while Date Center Expert restart). 

We have entered by the console in the vCenter to check if we could see any progress in the console, but it appears to be working still in version 7.9.0, we can logon with apcesetup user and check the network configuration, but we do not see any progress in the console abput the update.

We try to enter by the desktop client and it says that "DCE server is not working" 

What else can we check?

Thanks in advance.

[APC_Steve]

Hi @jzurera 

Once this issue occurs, you do not want to try to re-add the devices. You must redeploy. Do not try to upgrade the system as is.

You do not want to create a backup of this system at this point either. 

If you have a backup from before the issue, you should be able to import that.

If you can not import a backup from an earlier system, it is indeed a different and separate issue and that would be a whole new troubleshooting.

If you started at 7.9.0 or 7.9.1, you'll want to start over again.

If you had 7.8.1 backups and you imported that to 7.9.0 and then got this issue, you want to go that route again.

Steve

[jzurera]

Sorry Steve, but I do not understand exactly what you mean.

We started this project in 7.9.0 and we have a backup of this version. But the customer says that the restore did not work, so I tried to do a backup only to check if it worked because the scheduled backup was not working, and it failed. So if the customer is right, we cannot restore from the previous backup.

So, do we have to create a new virtual machine in 7.9.0 and restore the previous backup that we have? Do we have another alternative?

Thanks in advance.

[APC_Steve]

Hi @jzurera ,

That's correct. You have to deploy a brand new VM.

If you have a backup from the very beginning, of the original 7.9.0 deployment (before the issue), use that backup.

Try to deploy using the same mac address so that you don't have to contact support to release the licenses from the old mac and associate them with the new mac.

If you can't import that backup into a new VM, it's possible there was an issue even before the issue we're seeing and you may want to deploy 7.9.2 (fixed version) and start from scratch.

At minimum, importing of the backup failing is a new issue and needs to go through tech support. If support can't find a resolution, we'll be back at deploying 7.9.2 and manually reconfiguring.

Steve