DCE CPU usage increase after upgrade 7.5.0

[DCIM_Support]

Hello support team,

My Customer is having some DCE issues after upgrade the server to version 7.5.0. After the upgrade, the DCE server randomly reboot itself, the server is working too slow and sometimes the users access is denied. Also, the CPU usage increased from 40% to 80-90% after the upgrade. See image for details:

The DCE has enough resources to run properly: CPU: 4 CORES // RAM: 4 GB // HD: 200GB.

Something important to highlight is that nothing change were done or additional nodes were added or DCE setup were modify after the DCE upgrade.

I request your support to clarify and understand this CPU behavior after the upgrade. Do you have some ideas how to fix this issue? If I restore the server to version 7.4.3 my customer wants to know if I update again to version 7.5.0 this behavior does not happen again and understand the root cause.

Attached you can find the nbc.xml taken from the web client.

 nbc.xml

thank you for your support

regards,

LUIS

(CID:134035301)

[DCIM_Support]

Hello support team,

Related to this issue and continuing with the troubleshooting, the amount of CPU belongs to virtual machine was increased from 4 Cores to 8 cores but it did not fix the error.

The server was rebooted watching the Linux command prompt and the below error message appear:

Please your support to find the root cause and understand the issue. If I go back temporally to version 7.4.3 I want to know how can upgrade the server without any issue.

Regards, 

(CID:134035418)

[DCIM_Support]

Dear Luis Lopez Borbon,

With a high degree of probability, I can assume that your problems are indirectly related to the vulnerabilities of Meltdown and Specter. The fact is that DCE-7.5.0 server already has security fixes for the above mentioned vulnerabilities (at least as of the end of January 2018). Read, for example, on this subject article Security fixes in StruxureWare Data Center Expert v7.5.0.

This can explain the increase in the percentage of CPU time usage of the VMware virtual machine. For example, as indicated in the special article Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715... on the RHEL website:

Red Hat has made updated kernels available to address these security vulnerabilities. These patches are enabled by default (detailed below) because Red Hat prioritizes out of the box security. Speculative execution is a performance optimization technique. Thus, these updates (both kernel and microcode) may result in workload-specific performance degradation...

And in order to avoid the consequences of performance degradation as much as possible, in the article Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715 and in the special detection script on the RHEL website it is said:

Red Hat recommends that you:

Note about virtualization
In virtualized environment, there are more steps to mitigate the issue, including:
* Host needs to have updated kernel and CPU microcode
* Host needs to have updated virtualization software
* Guest needs to have updated kernel
* Hypervisor needs to propagate new CPU features correctly
For more details about mitigations in virtualized environment see:
https://access.redhat.com/articles/3331571

For more information about the vulnerabilities see:
https://access.redhat.com/security/vulnerabilities/speculativeexecution

I recommend you a detailed talk with your VMware team on the above mentioned issues and their solutions. After all, the solution can be found on the side of the VMware virtualization system, and not in the DCE server.

I hope this helps you.

With respect.

P.S.: see also the similar topic .

(CID:134035652)

[DCIM_Support]

Hello support team,

According with your answer, I have some comments:

1)      you are talking about Linux Red Hat but DCE is running in Linux CentOS… all your comments related to these vulnerabilities apply to this OS?

2)      Reading the document “security fixes” the vulnerabilities that you are talking about were resolved in version 7.5.0… if these vulnerabilities were resolved, why the DCE has this behavior?

3)      The VMware is working in the last version (ISX 6.5)… and the DCE is installed in the central bank in our region with strong IT security policies… for this reason the vulnerabilities in the most of the cases are resolved and continuously they scan all the systems…

4)      Something important to highlight, after the upgrade to version 7.5.0 the DCE was working without any issue, but after 5 days, the DCE suddenly started to increase the resources, principally the CPU usage… no change was made in this server or additional nodes were added… if the server has problem with some vulnerabilities, the behavior must to appear immediately after the reboot…or not?

5)      If I talk with the IT department to scan again the server looking for these vulnerabilities, what are the next steps if these ones are not appear…?

thank you for your support

regards,

(CID:134036358)

[DCIM_Support]

Dear Luis Lopez Borbon,

From your questions in the order they follow.

1. Absolutely right, because in fact it is one and the same OS, but with different levels of technical support.
2. I can not say... I can guess that the latest version of DCE-7.5.0 based on CentOS-6.9 (kernel-2.6.32-696.18.7.el6.x86_64) has security updates for January 2018. During this time, a lot of critical security updates for this OS have been released (currently is CentOS-6.10 with kernel-2.6.32-754.3.5.el6.x86_64), as well as for VMware ESXi 5.x and 6.x versions. It is very possible, that your end customer regularly installs these critical security updates on VMware ESXi hosts. But it is important, that the above-mentioned security updates also regularly installed on virtual VMware guest machines, for example, as a DCE-server. This is strictly indicated by the Red Hat bulletin, which I mentioned above.
3. See paragraph 2 above.
4. Again, it's hard to say... Several variants of events are possible.
5. I do not know how your IT department will be able to scan the DCE-server for the above mentioned vulnerabilities. After all, according to the above-mentioned Red Hat bulletin, this requires console root access to the DCE-server and the launch of a corresponding detection script. But, as you know, DCE-server is a closed black box that does not provide for regular installation of any updates.

Based on the above, at the moment I see two options for solving your problem.

1. I hope you make a regular full backup of your DCE-server? Then create a clean virtual machine DCE-7.5.0 from the provided SE ova-file (the MAC address must be the same). And then deploy a full backup to this virtual machine. Then it will be possible to see if the above problem will again appear or not?
2. If possible, roll back to version DCE-7.4.3: this requires a full backup of the same version, that is, a full backup from DCE-7.4.3 server. Then you just need to wait a few months until SE makes a new release of DCE-7.x software. This new release will already contain the entire mass of critical updates for the entire 2018 year. This new release can be tried and see how this software will work on your VMware ESXi hosts.

In my opinion, the first option is more simple and practical, it must be tried.

If you have any further questions, please ask.

With respect.

(CID:134036737)

[DCIM_Support]

Hello support team… I really appreciate your support regarding this issue. I will do the steps according your recommendation… regards,

(CID:134037172)

[DCIM_Support]

Dear Luis Lopez Borbon,

Tell us, please, did you solve your problem or not?

Recently, a new version of DCE-7.6.0 has been released: 

With respect.

(CID:134679976)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.