DCO cannot connect to ITO

DCIM_Support · ‎2020-07-03

I was doing device discovery on several subnets, perhaps too many at once...
Seems like it got hung because discovery stopped working and DCO stopped responding.

When I relaunched DCO, I could no longer run Server Discoveries. I get the following error message in DCO when I start it up:
"Unable to retrieve discoveries for all ITO servers. The following errors occurred: - : Internal Server Error"

I restarted the Windows VM that ITO runs on, no change.
I restarted the Linux VM that DCO runs on, no change.

When I go into DCO > System Setup > External Systems Config and click Test, all come back OK including ITO.
I deleted and re-added the ITO server, no change.

When I look in DCO > Planning > Data Center and go to the Server Discoveries tab, I don't see any of the Server Discoveries I've run; nor can I create new ones.

When I do netstat -an |grep 8090, I don't see connections established on either ITO or DCO server.

Is there something I am overlooking here?

DCIM_Support · ‎2020-07-03

Hi Mike Ryan,

It for you today I answered how to reboot of Linux server with ITO installed a question? I for some reason don't find the today's post on this matter...

Concerning your problem with ITO: You were precisely convinced, that all components of the ITO server normally work?

I am always glad to help.

DCIM_Support · ‎2020-07-03

Yes, I deleted the earlier question because ITO actually turned out to be on a Windows server.
I'm not sure how to check if all components of my ITO server are working normally.
All was working well until I started doing discovery on a few domains at once, Discover locked up and now I am in the situation I described above.

How shall I check whether ITO is working normally?

DCIM_Support · ‎2020-07-03

Hi Mike Ryan,

Ok, for a start, you can visit the your ITO server at the address https://ITO_server.domain:8643/ or https://ITO_server.domain:8643/DataCenterManager/ ?

On Windows server the firewall is enabled or not?

The PostgreSQL and Intel DCM system services are in a running state?

With best regards.

DCIM_Support · ‎2020-07-03

Hi Mike Ryan,

You have precisely fulfilled all requirements for installation of ITO software on Windows server?

For example from StruxureWare Data Center IT Optimize Server Requirements:

The hostname of the StruxureWare Data Center: IT Optimize/StruxureWare Data Center: Server Access server must be DNS resolvable from all StruxureWare Data Center Operation clients.

It is recommended that StruxureWare Data Center: IT Optimize servers connected to StruxureWare Data Center Operation match the version.

Windows specific requirements:

IT Optimize must be installed by an Administrator level user.

The IT Optimize services must run as a user which has been granted the "Logon as a Service" privilege. Instructions are available here: http://technet.microsoft.com/en-us/library/cc794944(v=ws.10).aspx

StruxureWare Data Center: IT Optimize license must be installed on the Data Center Operation server.

With best regards.

DCIM_Support · ‎2020-07-03

Yes, I can get to https://ITO_server.domain:8643/DataCenterManager

No, on the Windows ITO server the firewall is not enabled.

PostgreSQL is not installed on my system (but EnergyCenterMySQL is up and running.)
Intel DCM is not installed on my system (but EnergyCenter is up and running.)

As far as whether everything was installed properly, I believe it was as it was all installed under direction of Schneider techs and our initial install and support were done when Schneider engineers were here.
Also, everything worked just fine until the time when I ran too many searches at once and caused it to hang.

DCIM_Support · ‎2020-07-03

Ok, then Jef Faridi really is right (read his answer).

By the way, tell me exactly, what is your DCO version, StruxureWare ITO (with Server Access) version and Energy Suite version?

With best regards.

DCIM_Support · ‎2020-07-03

Adding to some of the comments previously mentioned.

On a windows server ITO should have two processes which are installed and running in the windows services panel. They are called "schneider electric energycenter" and "schneider electric energycenter database". Both should be present, and running.

To test whether ITO is running open a browser and browse to https://:8090/

If you get a ITO login window, then ITO services are running (at least the web service is running).

For the Intel DCM component, the previous comment post is correct in that you remote desktop to the ITO server you should be able to open a browser and browse to address http://localhost:8688/DataCenterManager/

If browsing to that address does not produce an Intel DCM window, check for the DCM services in the windows services control panel. There should be five services which start with "Intel(R)", they should all be started.

If none of the DCM services exist, then DCM was not installed.

The most common reason DCM is not installed when ITO is installed is when the ITO installer is executed directly from the ITO zip file which was downloaded from schneider. The proper procedure is to download the ITO zip, extract all of the contents of the zip file to a temp folder, then run the ITO installer from the temp folder.

If the ITO installer is executed from the zip file then it will not be able to find the DCM installers because they are also still in the zip file.

If DCM was not installed you can still install it manually by double clicking the "install_dcm.bat" file in the extracted temp folder. Wait for that to finish, then run the "install_vGateway.bat" file in the same temp folder. Do NOT run the DCM and vGateway exe files directly, if you do they will not be configured to talk to ITO.

Regards

Greg Sterling

DCIM_Support · ‎2020-07-03

Hello guys,

Thanks Greg Sterling for useful post.

And to verify from MS Windows cmd (for example, MS Windows 2012 Core or MS Windows 2016) status of all system services, belonging to the SxW ITO software, simple run this commands:

powershellquery_services.bat@echo off echo ======================================================= echo Schneider Electric StruxureWare ITO services status: sc query EnergyCenter sc query EnergyCenterMySQL echo ======================================================= echo Intel DCM and KVM services status: sc query VirtualGatewayProxy sc query DatacenterManagerServer sc query DatacenterManagerSnmp sc query DatacenterManager sc query "Intel(R) DCM Database Server"

With best regards.

DCIM_Support · ‎2020-07-03

Thanks for posting the status check script.

DCIM_Support · ‎2020-07-03

Hi Mike,

Based on the description, it seems you are trying to discover a very large number of networks at one go and/or the ITO server hardware configuration might be less than system requirements/expectations.That might be basically the reason for the ITO database stopped responding.

The minimum system requirements is mentioned here:StruxureWare Data Center IT Optimize Server Requirements

It might make sense to start with few discoveries at a time and see how it goes. The network bandwidth might also be taken into account and perhaps avoiding to run large discoveries at one go. I think one of engineers have tested (some time ago) discovery of 90 networks in one go, which seemed to be beyond the capability to handle.

To test/verify the ITO installation, as described in the following page: Installing IT Optimize

Browse to the ITO-Lite client, https://localhost:8090, (default port, might be different in your setup)

Browse to the DCM server GUI to test connectivity to the Intel DCM server, http://localhost:8688/DataCenterManager/

You might also want to check if DCM and IT Optimize services running on the Windows host.

PS. not related to this case, if you are running DCO 8.0.2 (potentiality vulnerable for memory leak), then I would suggest upgrading the product to the latest version (currently 8.1).

Kind regards

DCIM_Support · ‎2020-07-03

Thanks!

When I checked the ITO-Lite client at https://localhost:8090/ I saw everything there was still hourglassing, even after a reboot.

I restarted the EnergyCenterMySQL and EnergyCenter services and everything came back OK on the ITO Lite client.

I restarted DCO and all looks OK.

We do frequently have issues with slow network, I think you may be onto something with that suggestion.

As I move forward, I will only do one subnet at a time and see how it goes, thanks for your help!

DCIM_Support · ‎2020-07-03

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.