DCE 7.7.0 upgrade from 7.6.0 failed and stucked in endless reboot loop

DCIM_Support · ‎2020-07-05

I started DCE upgrade from 7.6.0 to 7.6.0 and followed the instruction. The progress went well: install set 1 and 2 and also a the migration step was success on the web UI.

But after the last successful step the server entered into an endless reboot loop which can be observed on the webUI and in desktop client as well. I am able to login to both interface but after a few seconds/mins it reports the restart/boot:

Please help me to fix it urgently without the clean install solution! The server is running on VMware but I have no direct access into that infrastructure.

Thank you.

DCIM_Support · ‎2020-07-05

One clarification: the application (DCE) itself is in endless loop because virtual machine and the OS is not rebooting. Please help!

DCIM_Support · ‎2020-07-05

Hi Mate,

For expedited support, I suggest contacting your local technical support by phone. This forum has no defined terms of service or specific service time so this type of issue may sit without answer for a time.

The first suggestion I'd make is to fully turn off the VM then manually boot it. If the issue continues, I'd suggest putting your devices in maintenance mode if you can get into the system for a long enough period of time. Possibly take them back out in small chunks if that works. Note the amount of time a reboot takes.

Other than that, you'd need to get logs. I would strongly suggest again contacting your local support as these logs may be too sensitive to put on a public forum such as this.

Open a Web Browser and Navigate to: http://DCE_IP/capturelogs
Example URL: http://10.11.12.13/capturelogs

If a log exists, delete it and create a new one. Provide logs to support.

Steve

DCIM_Support · ‎2020-07-05

Hi Steven,

Of course I opened case (ID 65120105) but the be honest with you: I really disappointed with that type of support. I have a valid 3 years contract without any SLA! for software support.

Usually that took days or rather weeks to get somebody who was really an expert and was able to help and solve the issue.

I always got faster and better help here from the community.

Back to your suggestion: I turned off the VM and started manually but did not solve the problem.

I turned off the email notification, so if your maintenance mode suggestion went into that direction then I don't worry about alerting.

Before I turned off the VM I checked and the application was UP for some hour without any reboot so I was able to create logs. Now I have 500MB file and happy to share it with a Schneider engineer.

Now I am open for a clean install but I want to discuss it with an engineer how the restore works (I have weekly full backups on NFS share, licenses will restored?, VM MAC address should be the same?, etc..).

Thanks,

Mate

DCIM_Support · ‎2020-07-05

Hi Mete,

The reason I suggest using direct support is because this is a public forum. There are no support people specifically assigned to answer these questions and a direct contact to support should yield better / more timely results. This was the SLA (or lack thereof) to which I was referring.

As for a restore, assuming the new server has a new mac address, you will have to deal with tech support directly to get them reset so you can associate them with the new mac in the digital entitlement portal.

For maintenance mode, I wanted to yes stop any attempted e-mails but also any ftp / scp pushes to the devices and I believe maintenance mode will do that as well (or at least it seems to have done so). I've seen this one or 2 other times and I've yet to see a resolution other than what I've stated or a restore as you've suggested.

Steve

DCIM_Support · ‎2020-07-05

Update: I did a clean install with version 7.6.0. I restored a full backup (took 4 hours). That version was stable for 2 hours and than I started again 7.7.0 upgrade. It went well but after a few minutes it started again to reboot the application. I created a log again but still not received any appbox from any Schneider support rep.

I opened some logs an I can see that apache error_log is full with:

[error] python_init: Python version mismatch, expected '2.6.5', found '2.6.6'.
[error] python_init: Python executable found '/usr/bin/python'.
[error] python_init: Python path being used '/usr/lib64/python26.zip:/usr/lib64/python2.6/:/usr/lib64/python2.6/plat-linux2:/usr/lib64/python2.6/lib-tk:/usr/lib64/py...'.
[notice] mod_python: Creating 4 session mutexes based on 512 max processes and 0 max threads.
[notice] mod_python: using mutex_directory /tmp 
[notice] Apache/2.2.15 (Unix) mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[notice] caught SIGTERM, shutting down

and it repeats again and again in the log

DCIM_Support · ‎2020-07-05

Waiting for full logs to be attached / linked to the case. I can not provide detail based on a single snippet. I will check the full logs when the case it updated and will escalate accordingly.

DCIM_Support · ‎2020-07-05

Hi Steven,

I uploaded the logs to our company exchange portal and provided access to service support rep. but I am quite frustrated that I had to force it and days just spent with this simple task to transfer necessary files to Schneider.

Support offered me to use a public file share service but for enterprise sensitive data is not really a professional way. Agree?

But turn it into professional technical area: now I have again version 7.7.0 because apache server is up for 12 hours. If I gracefully reboot the server it starts again the http service reboot and after a few hours it stop the crashes. I don't believe it is a stable system now therefore I have a backup related question to avoid any data loss.

Full backup which created under 7.6.0 is compatible and restorable only in the same DCE version. Now I am on 7.7.0 and scheduled full backups are running on every Sunday. So if I stay on this 7.7.0 I won't be able to revert back to 7.6.0 because of backup incompatibility.

What is your suggestion? Go back to 7.6.0 or stay 7.7.0?

Thanks

DCIM_Support · ‎2020-07-05

Hi Mate,

If you've gone to a new VM I understand but please don't have them close the case if you've not done so already. I'd like to continue this and let our engineers know it's an issue. On a side note, did we ever get the VM host info?

As for the logs, if you feel that our Schneider specific box folders are not to your standards, I understand however that is the current tool we have available, there's not much the people in tech support can do about that. I will comment to management about it but I can't promise anything will be done.

You mention : "now I have again version 7.7.0". Am I to assume you've restored to 7.6 and again upgraded to 7.7.0 and you're seeing issues (maybe fewer) or is this just a fresh 7.7.0 deployment and you're seeing these errors? I can deploy one and just let it run to see if I have similar errors but I do know that I have had plenty of fresh deployments of the 7.7.0 VM that do not regularly reboot, because there have been no issues, I've not checked the logs.

I have another similar issue going on right now and received a backup from the customer. I uploaded that backup to on of my VM deployments and I had the thick client open all day yesterday with not a single reboot. Again, this was why I was mentioning devices in maintenance mode, I know the customer's backup on MY VM can't reach his devices so there's no comm between the server and devices . The only difference? I don't know.

I understand you're seeing the Apache issue however just from the messages, I can't tell if it is an issue actually with that process or if another process or series of processes is causing this process to have issues. I have asked the rep yet again to work on getting the logs from you.

As for your last question about staying at 7.7.0, if you feel the deployment is unstable, that's one vote for staying at 7.6... probably the most important one. If you had not seen any issues with 7.6 there's another reason to stay there. If, when looking through the 7.7.0/7.7.1 release notes, if there's something specific in there that you require, that's a point for staying at 7.7.0.

You're right, if you begin having more reboot issues tomorrow or next week and feel the need to go back to 7.6, that backup will be from days ago and you will lose any data between that day and the time you decide you need to stick with 7.6. Personally, if I were running into such issues and felt 7.6 was more stable, I'd stay there, but that's more of a personal feeling than an official stance.

Steve

DCIM_Support · ‎2020-07-05

What cpu are you using in your hosts? AMD or Intel?

DCIM_Support · ‎2020-07-05

Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz.

The CentOS (maybe) doesn't have any issue. The apache server is crashing in almost every two mins.

DCIM_Support · ‎2020-07-05

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.