DCE slow response and high CPU usage

[DCIM_Support]

Hello

We had DCE slow responding due High CPU usage (It can be seen from VMWare side). And I was unable to connect via web or desktop client to the system.

Logs are attached

MISSING ATTACHMENT: nblogs_2-4-2019_17-11-4.zip

Is there a way to monitor what is causing High CPU usage to resolve the issues? 

Also we were not able to add additional CPU (currently 8 is maximum) because VM compatibility is out of date:

Is it safe to update VM settings? 

DCE version:

(CID:141270084)

[DCIM_Support]

Hi Olegas,

There isn't anything specific we can look at to tell us what is causing this. Additionally, I wanted to note that your link to the logs shows "Attachment File Not Found".

A few things I'd suggest you look at other than RAM however are:

1: Disk Space. How many hard drives have you provisioned for this system. If you only have the default 1 18 gig drive, I'd strongly suggest adding another. If you have attempted to increase the drive size, you should know that you can't. I know in VMware you are able to change the allocation but DCE will not see it. Additionally, once the drive gets full, you will end up with double duty as the DCE server must purge data to allow the new data a place to be stored.

2: Speaking about data, how much data comes in can cause issues. The biggest culprit is surveillance. If you're using surveillance, check out your camera settings. If you're trying to do surveillance at 10 FPS, try lowering it  to 2 fps or something along those lines. If you're storing at a high resolution, try lowering that. If you are using surveillance AND NetBotz alerts also have camera images, lower those resolutions / FPS or stop sending images at all if you're already getting them from surveillance. 

3: Another point to look at in the amount of data DCE is getting is scan rate. Assuming mostly SNMP devices, what kind of scan rate do you have? If you've set it to 1 minute, does lowering it to 5 minutes or 10 minutes change the response / CPU usage.

4: Check your VM for where storage exists. Is the storage for this VM local or is it on a NAS. Network storage is almost always slower and that will cause DCE to take more time in storing all the data.

5: Shared resources. If the VM has only 1 NIC but numerous VMs, they're all sharing the same throughput. The same can be said for RAM.

Steve.

(CID:141270153)

[DCIM_Support]

Hi Olegas,

Another thing I'd be curious about, how many 3rd party device and extra DDFs do you have? If there are any, have they been modified at all?

Steve

(CID:141270229)

[DCIM_Support]

Hello

Space is not an issue:

Surveillance is not in use.

Quantities are:

SNMP global scan time interval is set to 5 minutes

trying to attach logs again:

nblogs_2-4-2019_17-11-4.zip

VM is using shared resources. This is quite big VMWare cluster and resources are far from the limits.

(CID:141270338)

[DCIM_Support]

As for third party DDFs we have added for Eaton (~200 devices) and HP PDU (~15)

(CID:141270554)

[DCIM_Support]

Hi Olegas,

How many hard drives have you provisioned for this system in VMware?

For the 3rd party DDFs that have been added, have they been modified in any way?

As for the logs, the only thing I see (and it is consistent) is the following error:

thoughtworks.xstream.io.StreamException

It seems to be having an issue writing sensor data so again, I'd be curious to know about any DDF edits. It's also possible there is corruption in either the database or something potentially invalid about some of the data coming in but I can not directly associate it with a specific sensor or device.

If you were to create a config backup and import that into a freshly deployed VM, we may be able to see if it's something about the current system itself or something more related to the configuration, DDFs, etc.

Please note, this may take tim eto identify and more time over the forums. A phone call to your local support may be helpful in resolving the issue more quickly.

Steve

(CID:141270714)

[DCIM_Support]

We tested today:

from DCE device view window initiated delete device. 2 PDU have been selected, witch are being monitored for 3 month.

And client became unresponsive for few minutes. But there were no errors in the log

(CID:141271565)

[DCIM_Support]

I have also been noticing poor performance from DCE 7.6 desktop client.  It will consume around 40% CPU whilst idle and quite often not close using the red X button but only through File -> Exit.

(CID:141270645)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.