ITA failed restore

[DCIM_Support]

I was trying to restore my ITA system from my DCO backup, it failed, it worked for me before though. Blow is a snippet of error messages from platform log. I confirm the backup image is good. 

03/01/20 11:25:51 ERROR Operations::OperationsLib::warnHandler Use of uninitialized value $host in concatenation (.) or string at /usr/sha
re/perl5/vendor_perl/Operations/ClusterManager.pm line 680.
at /usr/share/perl5/vendor_perl/Operations/ClusterManager.pm line 680.
Operations::ClusterManager::waitForOperationShutdown('Operations::ClusterManager') called at /usr/share/perl5/vendor_perl/Operatio
ns/BackupManager.pm line 589
Operations::BackupManager::stopOperation('Operations::BackupManager') called at /usr/share/perl5/vendor_perl/Operations/BackupMana
ger.pm line 395
Operations::BackupManager::doRestore('Operations::BackupManager', '/tmp/Final_EMEA_2019_7_10_Room_Power_2019-07-10_16.58.tar.gz')
called at /sbin/Platform.pm line 283

03/01/20 11:25:51 INFO Operations::ClusterManager::waitForOperationShutdown Waiting for Operation to shut down on host: for 300 s
03/01/20 11:25:52 INFO Operations::ClusterManager::waitForOperationShutdown .
03/01/20 11:25:52 ERROR Operations::OperationsLib::dieHandler Expected 'PeerHost' at /usr/share/perl5/vendor_perl/Operations/OperationsLib
.pm line 2318.
at /usr/share/perl5/vendor_perl/IO/Socket/IP.pm line 454.
IO::Socket::IP::_configure('IO::Socket::SSL=GLOB(0x41a04c8)', 'HASH(0x2512c30)') called at /usr/share/perl5/vendor_perl/IO/Socket/
IP.pm line 370
IO::Socket::IP::configure('IO::Socket::SSL=GLOB(0x41a04c8)', 'HASH(0x2512c30)') called at /usr/share/perl5/vendor_perl/IO/Socket/S
SL.pm line 279
IO::Socket::SSL::configure('IO::Socket::SSL=GLOB(0x41a04c8)', 'HASH(0x2512c30)') called at /usr/lib64/perl5/IO/Socket.pm line 49
IO::Socket::new('IO::Socket::SSL', 'Proto', 'tcp', 'SSL_cert_file', '/etc/isx-operations/ssl/certificate.crt', 'SSL_key_file', '/e
tc/isx-operations/ssl/certificate.key', 'PeerPort', 'https', ...) called at /usr/share/perl5/vendor_perl/IO/Socket/IP.pm line 337
IO::Socket::IP::new('IO::Socket::SSL', 'PeerAddr', undef, 'PeerPort', 'https', 'Proto', 'tcp', 'Timeout', 1, ...) called at /usr/s
hare/perl5/vendor_perl/Operations/OperationsLib.pm line 2318
Operations::OperationsLib::getHttpsContent('Operations::OperationsLib=HASH(0x2529d78)', undef, '/clients/alive', 1, undef) called
at /usr/share/perl5/vendor_perl/Operations/ClusterManager.pm line 492
Operations::ClusterManager::isWebsiteRunning('Operations::ClusterManager', undef, '/clients/alive') called at /usr/share/perl5/ven
dor_perl/Operations/ClusterManager.pm line 576
Operations::ClusterManager::jbossCli('Operations::ClusterManager', undef, '?operation=attribute&name=server-state') called at /usr
/share/perl5/vendor_perl/Operations/ClusterManager.pm line 557
Operations::ClusterManager::isOperationRunning('Operations::ClusterManager', undef) called at /usr/share/perl5/vendor_perl/Operati
ons/ClusterManager.pm line 688
Operations::ClusterManager::waitForOperationShutdown('Operations::ClusterManager') called at /usr/share/perl5/vendor_perl/Operatio
ns/BackupManager.pm line 589
Operations::BackupManager::stopOperation('Operations::BackupManager') called at /usr/share/perl5/vendor_perl/Operations/BackupMana
ger.pm line 395
Operations::BackupManager::doRestore('Operations::BackupManager', '/tmp/Final_EMEA_2019_7_10_Room_Power_2019-07-10_16.58.tar.gz')
called at /sbin/Platform.pm line 283

(CID:153551022)

[DCIM_Support]

Hi Yilai,

Apparently timeout expired waiting for operation to shut down (300 s). Can I ask for the complete server logs + the used backup file (via your shared site), so I can take a look.

I was wondering if rebooting the server and then restoring the backup file could have helped?

Kind regards

(CID:153551150)

[DCIM_Support]

Hi Jef, 

I tried your suggestion to reboot the system, but I forgot to change the parameter of our auto config system - the ITA packages got downgraded back to DCO. However, the log remains, and shouldn't affect our troubleshooting. I collected the platform and server log from OS, and zipped with the backup that was used, named as "ITA_restore_issue.zip". 

Thanks

(CID:153551329)

[DCIM_Support]

Thank Yilai,

I will get back to you asap,

Kind regards

(CID:153551606)

[DCIM_Support]

Hi Yilai,

Thanks again for providing the data.

I have tested restoring the provided backup file on my ITA 9.0.4 without any problem.

However, according to the provided log files, it seems the "Operations" is unable to shutdown, and therefore the restore job could not be continued (and fails). This might be code related issue, that may need to be investigated. I have therefore created a backlog for code investigations (based on your log entries).

Kind regards

(CID:153551732)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.