Slow DCE Response times

DCIM_Support · ‎2020-07-04

This question was originally posted on DCIM Support by Mark Tucker on 2018-05-17

We are currently experiencing slow DCE server response time. After adding virtual sensors the application will hang (grey out) for anywhere between 5 and 10 minutes., it can take up to 5 minutes to move a PDU to a different zone/room/container this happens when a single user has the application open or when multiple users have it open. This is happening at two separate sites both are running their own instance of DCE with approx. 3500 nodes at each site. Both running DCE 7.4.3, network response times are generally under 20ms and users accessing the application are on the same network. I have restarted both servers and increased the polling time, neither has resolved the issue.

DCIM_Support · ‎2020-07-04

This answer was originally posted on DCIM Support by spezialist on 2018-05-17

Dear Mark Tucker,

Specify, please, your DCE-servers is physical servers or VMware virtual guests?
How long have you been having this problem?
Did this problem appear smoothly or immediately?

These my questions are related to the recently discovered vulnerabilities of Meltdown and Spectre. If your DCE servers are VMware virtual guests, then maybe your VMware command applied security patches for ESXi hypervisors or updated CPU microcode, BIOS and so on? All this could significantly affect the performance of guest machines, especially closed appliances (such as DCE software), which are not able to be often updated.

For example, as indicated in the special article Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715... on the RHEL website:

Red Hat has made updated kernels available to address these security vulnerabilities. These patches are enabled by default (detailed below) because Red Hat prioritizes out of the box security. Speculative execution is a performance optimization technique. Thus, these updates (both kernel and microcode) may result in workload-specific performance degradation...

And in order to avoid the consequences of performance degradation as much as possible, in the article Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715 and in the special detection script on the RHEL website it is said:

Red Hat recommends that you:

Note about virtualization
In virtualized environment, there are more steps to mitigate the issue, including:
* Host needs to have updated kernel and CPU microcode
* Host needs to have updated virtualization software
* Guest needs to have updated kernel
* Hypervisor needs to propagate new CPU features correctly
For more details about mitigations in virtualized environment see:
https://access.redhat.com/articles/3331571

For more information about the vulnerabilities see:
https://access.redhat.com/security/vulnerabilities/speculativeexecution

Additionally, as a preventive and verification method, I recommend that you upgrade at least one of your DCE servers to the latest version of DCE-7.5.0. Read, for example, on this subject article Security fixes in StruxureWare Data Center Expert v7.6.0.

I hope this helps you.

With respect.

DCIM_Support · ‎2020-07-04

This comment was originally posted on DCIM Support by Mark Tucker on 2018-05-17

Hi,

Thanks for your answer. Please see responses below

Specify, please, your DCE-servers is physical servers or VMware virtual guests?

Both are physical servers

How long have you been having this problem?

Approx 3 months

Did this problem appear smoothly or immediately?

Smoothly

Servers were upgraded to v7.4.3 at Schnieders recommendatiuon to protect against Spectre-Meltdown.

DCIM_Support · ‎2020-07-04

This comment was originally posted on DCIM Support by spezialist on 2018-05-18

Dear Mark Tucker,

Servers were upgraded to v7.4.3 at Schnieders recommendatiuon to protect against Spectre-Meltdown.

This is false, since if you read article Software Vulnerability Scan(s) StruxureWare Data Center Expert, you should be aware, that fixes for these vulnerabilities are only available starting with DCE-7.5.0.

Ok, then if it's a physical server and the problem is constantly present, it's very useful to perform an analysis of server system logs.

How to download server system logs is well told in the post .

What useful information can be learned by analyzing these server system logs can be found in my comment on post :

/data/logs/watchdog.log -- DCE server software related log;
/hs_err_files -- normal if this Java crash logfile directory is empty;
/var/log/messages -- Linux server related main logfile;
df_output -- status of free harddisk space on the server (output of "df" command);
pg_alerts -- dump of PostgreSQL tables "alerts" (what we see in the DCE client in the context menu Show Alarm History on your device);
pg_eventlog_message -- dump of PostgreSQL tables "eventlog_message" (what we see to the link https://you_server.domain/desktop/#eventlog (or http://you_server.domain/desktop/#eventlog);
ps_output -- list of all running processes on the Linux server (output of "ps" command);
top_output -- top 10 most resource-intensive processes on a Linux server (output of "top" command).

I am always glad to help.

DCIM_Support · ‎2020-07-04

This comment was originally posted on DCIM Support by spezialist on 2018-05-21

Dear Mark Tucker,

Tell us, please, could you solve your problem or not?

With respect.

DCIM_Support · ‎2020-07-04

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.