DCE Server freezing clients

[DCIM_Support]

Dear Team Members,

Can you Please support my below issue 

DCE server performance issue has been before and now we are facing this issue again as resources utilization of the DCE server is high. Below are our observations

Issue is raising when we run the reports and all our clients get freezing. When we check server performance during the report and after report is run, CPU is utilizing to 98% and then issue starts freeing of clients.

Note: We are not having admin access to the server to know what are the services inside the datacenter expert appliances is causing issue.

This is a very serious issue we are facing and need a resolution at earliest. 

We've have current configuration on VMWare for DCE: 

CPU  8 cores 
Memory 24GB 
Storage 1.7 GB allocated

We have latest Version DCE 7.4.3.We have total 804 Devices on DCE including (Netbotz 355,450,550, IRC, PDU, PDM, Virtual Sensor)

 Regards

Shaik Mahboob Ali

Data Center Analyst, Computing Infrastructure Team

R308-WS11| 3rd Floor | Building-14 (UN2200)

King Abdullah University of Science & Technology

Office: +966 (012) 8081269 |Mobile:  +966 (55) 919 9819 

Email:  mahboob.shaik@kaust.edu.sa

(CID:126170635)

[DCIM_Support]

Hi Shaik Mahboob Ali,

The first question I have is about the hard drive space. You mention 1.7GB. The default drive is 18 GB and it leaves 4 GB for storage, the rest being used for the OS. Did you mean 1.7TB? If more memory was added, was the primary drive increased or was a second drive added?

When you run the reports, how many devices and sensors are you including and for what period of time?

How many devices total are being monitored?

Are you using surveillance with the NetBotz appliances and if so, what are the settings?

If you go to the system menu and storage settings, what does it show for values (maybe a screenshot?)

If you need immediate assistance, I suggest contacting your local tech support by phone. Although we will try to assist on forums such as these, there is no immediate response available as there would be if you call in. It could take a day or two to respond to each set of questions where being on the phone won't have such a delay.

Thanks,

Steve.

(CID:126170930)

[DCIM_Support]

Hi Steve,

Thanks for the updated Please find below quires.

The first question I have is about the hard drive space. You mention 1.7GB. The default drive is 18 GB and it leaves 4 GB for storage, the rest being used for the OS. Did you mean 1.7TB? If more memory was added, was the primary drive increased or was a second drive added? Below Storage Property

When you run the reports, how many devices and sensors are you including and for what period of time? 48 IRC  or PDU or PDPM for 1 Year data or 6 Months data.

How many devices total are being monitored? 804 Including Virtual devices

Are you using surveillance with the NetBotz appliances and if so, what are the settings? We are using NetBotz 355 independently setting below. I have hidden DCE Server IP for security

If you go to the system menu and storage settings, what does it show for values (maybe a screenshot?) Screen shot below

(CID:126170976)

[DCIM_Support]

Hi Shaik Mahboob Ali,

Looking at the info you've posted, it appears that there is only a single drive on this system. As per K-base FA309639, the default drive must not be increased, instead a secondary drive must be added. If you've simply increased the drive size, that may be the issue. It also seems odd that it would allow you to have the amount of data it is suggesting with only the single drive unless you've somehow edited the OVA. 

Have you tried polling less information in your reports? You noted:

48 IRC  or PDU or PDPM for 1 Year data or 6 Months data.

If you're reporting on all of them at once, perhaps you can try polling only a few (5 or 10) for that period of time or perhaps choose a lesser time period...after all if you've already run this report a month ago, that data should be the same and you should only need 1 months worth of data to still have the full data set.

The surveillance settings are not extreme in any way however large numbers of cameras and consistent surveillance can cause higher throughput and more work for the system. How many cameras and are they running surveillance constantly?

If you go to the web page of the server and go to http:///capturelogs 

Delete any logs that already exist and create a new log. If you download that log, you can look at the top output when under normal load and this may tell us what's happening without the reports running. If you're able to run it again with the report running, perhaps we can see the difference.

In the end, unless there are specific errors when running the reports, it simply seems like you're over taxing the system.

Steve.

(CID:126171002)

[DCIM_Support]

HI Steve,

Thanks for the information so now we have done mistake by initialising  500GB as our primary and then as data grow we have added luns to the storage.

Now our data is on primary storage and as well on the added luns so kindly advise how we can now make our vm with standard storage and restore our data from current VM which is scattered on primary and secondary storage to new standard VM.

(CID:126173026)

[DCIM_Support]

Hi Shaik Mahboob Ali,

If you've increased the default drive size, the proper way to resolve it is to turn off the VM, reset the drive to it's original 18 gig, add an extra drive, then restart the server. This k-base FA309639 explains how to increase the disk space. I'm concerned though because it will normally show only the 4 gig available even after the apparent drive increase in VMware. DCE usually still shows the default. Was something done to the systemor the OVA specfiically to allow for the drive size to be increased other than just increasing it through the drive properties?

Steve

(CID:126173528)

[DCIM_Support]

Hi Steve,

We have verified with our storage team and windows team. Please find the attach pics of storage allocation configuration.The configuration is as per you mentioned above.

Please Advise

 

(CID:126174626)

[DCIM_Support]

Hi Shaik Mahboob Ali,,

I'm glad the storage question has been verified. 

This issue may end up taking some time and it may be quicker if you were to contact your local APC tech support.

In the meantime, have you attempted to limit the number of sensors or the time period that you're reporting on as suggested earlier?

How many clients are actually monitoring the system at any one time?

Are you using http or https?

If you can, go to the web page of the server http:///capturelogs as mentioned previously. Delete any existing logs. Start running your reports. If you see the issue of which you've been speaking, at that time create a new archive. This may slow the system a little more but that will get the logs and readings during the issue.

As for the VM, Are the drives connected directly to the VM itself or are the drives located on a different physical server / NAS?

Do you have any information on the IOPs for the disk(s)? What does the performance monitor in the host show? 

What version VMWare are you running?

Steve

(CID:126175974)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.