[500] An error has occured

[DCIM_Support]

Very often all users got this error in Web Console of  (see attached image).

And also in Logs there are errors like:

07.09.17 07:55:55.861 ERRR - Error handled: 500 null  (com.apc.isxc.common.integration.rest.RestExceptionResolver)

(CID:124524182)

[DCIM_Support]

Current version of DCE - 7.4.2

But also tried on latest DCE 7.4.3, error still happens.

These errors happen on different users, on with different browsers (IE/Chrome).

(CID:124524188)

[DCIM_Support]

Catch this error in Developers tools. So, that is not problem on client, HTTP 500 Internal Server Error comes from the DCE server 

(CID:124524195)

[DCIM_Support]

Hi Sergey,

I do not know if my post will help, but I often saw such an error and see if the user who logged into the web-GUI does not have sufficient rights to view the sensors of certain devices. Therefore, it will be useful to again check the access rights to devices and device groups for DCE web-GUI users. In addition, it is better to clarify: these are local (from DCE) or remote users (from LDAP, MS AD)?

Always glad to answer your questions.

(CID:124524283)

[DCIM_Support]

Hi spezialist. 

This error happens even on users with DCE Administrator rights.

Also I tried this situation: give user maximum permissions, add some sensors, remove permissions. And this worked as it should - there was error "no sensors available".

And what it comes to your second notice:

>> these are local (from DCE) or remote users (from LDAP, MS AD)?

We checked this situation. It seems, it is right way. 

We have designed the same page with the same sensors for MS AD user (DCE admin) and local user (DCE admin too). User with MS AD auth has the error, but local user DOESN'T have error.

Also, looked through logs on Domain Controller, there are no errors.

How can we fix this issue?

(CID:124524813)

[DCIM_Support]

Hi Sergey,

I.e., we can conclude that the above error in the web-GUI only occurs for remote users (from MS AD)? I understand you correctly?

Always glad to answer your questions.

(CID:124524888)

[DCIM_Support]

Yes, it seems, that you're correct. Issue happens with remote users from MS AD.

(CID:124524896)

[DCIM_Support]

Hi Sergey,

I looked at the list of documented Known Issues in DCE (from Release Notes) and have not yet found your bug. Therefore, we will wait for the answer from the employees of DCIM Support: what will they say about this?

I can not confirm this bug, since I do not use MS AD for authentication in DCE.

Additionally, I can recommend studying the DCE system logs for any anomalies or errors. How to get these system logs written in the topic , and how to understand them in my answer in the topic .

Always glad to answer your questions.

(CID:124524906)

[DCIM_Support]

Only /var/log/messages contains some errors. But I cannot confirm, that they are connected with my issue.

1. First, these errors happen a lot - 4-5 times a minute (PID is different each time): 
   

   Sep 8 13:42:14 dce-server init: ttyS0 main process (8835) terminated with status 1
   Sep 8 13:42:14 dce-server init: ttyS0 main process ended, respawning

2. Another thing, that happens sometimes:
   

   Sep 8 13:42:43 dce-server nbTranslateD: timeout_translators: translator for locale ru_ru, domain scannerDDFMsg, pid 8599 was inactive for 300 secs, timing out
   Sep 8 13:42:43 dce-server nbTranslateD: translator for ru_ru, scannerDDFMsg terminated

3. And another type of errors:
   

   Sep 8 13:42:45 dce-server nbSNMPScan: failScan: Marking scan of 10.??.??.?? with SCAN_FAILED resultcode.
   Sep 8 13:43:05 dce-server nbSNMPScan: failScan: Marking scan of 10.??.??.?? with SCAN_FAILED resultcode.
   Sep 8 13:43:23 dce-server nbSNMPScan: failScan: Marking scan of 10.??.??.?? with SCAN_FAILED resultcode.

But still, I'm not sure if it is connected.

(CID:124524942)

[DCIM_Support]

Hi Sergey,

All three of the above problems do not relate to this topic issue (the first part of the log is the process of the server's serial console, the second and third parts are the process of scanning SNMP devices, and some devices are not available, check this in the DCE client).

I would certainly recommend that you create a fresh VM for DCE and connect it to another MS AD server to test your problem. But this is if you have the time and resources for such a check.

Always glad to answer your questions.

(CID:124524959)

[DCIM_Support]

...In addition: what resources are allocated for your DCE-server, how many devices are under monitoring and how many simultaneously connected web-GUI clients?

And yet, show, please, the contents of the file top_output.

Always glad to answer your questions.

(CID:124524966)

[DCIM_Support]

Resources are: 2 vCPU, 2 GB RAM, 60 GB HDD

3-5 users connected at one time, approx. 200 devices, no cameras

Attached file: top_output

(CID:124524969)

[DCIM_Support]

Please, show the file top_output in the form of a screenshot, since your Attached file: [attachment deducted].

(CID:124524974)

[DCIM_Support]

Uploaded all three top-s from top_output

(CID:124524980)

[DCIM_Support]

And text format on pastebin:

https://pastebin.com/CK8GaYLa

(CID:124524985)

[DCIM_Support]

Hi Sergey,

I looked at your file: I have a similar configuration of the DCE-server in use and I highly recommend that you increase the RAM to at least 4GB. After increasing the RAM, restart your DCE-7.4.3 check whether your error 500 will appear and how often.

Always glad to answer your questions.

(CID:124525005)

[DCIM_Support]

spezialist

We have tested this earlier, nothing changed. Error happens with the same frequency.

(CID:124525010)

[DCIM_Support]

Hi Sergey,

Your problem is quite interesting, so next week I'll try to simulate your situation on my test DCE-server and report the result within a few days. To later understand what it is: a bug of the DCE-server or the problem of your configuration.

Always glad to answer your questions.

(CID:124525215)

[DCIM_Support · Accepted solution]

So. We tried some solutions. Did fresh install and restore from backup, upgrade to latest version and etc. Nothing helped. 

But, as always, helped old one solution - fresh clean install. It took a lot of time to re-add all devices, we lost all history, but an error does not happen anymore.

So, this is solution, but not a good one.

(CID:124526129)

[DCIM_Support]

Hi Sergey,

But, as always, helped old one solution - fresh clean install. It took a lot of time to re-add all devices, we lost all history, but an error does not happen anymore.

In this case, you made the authentication settings on MS AD servers exactly the same as originally?

Did you just try to completely delete the user and create anew? Once I talked about a similar problem in topic  and the customer found the solution himself:

Well yes it is resolved by my client himself, he removed & created the user again & everything goes fine.

Always glad to answer your questions.

(CID:124526141)

[DCIM_Support]

All settings were copy-pasted from old DCE to new one, including MS AD authentication. 

I could not delete and re-create user, as all permissions were through AD groups.

Anyway, it works fine now, so it is not connected with AD groups/users directly. 

Our old DCE installation was old enough, we did upgrades from 7.2.0 (approx. 5 years). It seems, that error appeared after one of upgrades.

(CID:124526200)

[DCIM_Support]

Hi Sergey,

Many thanks for the feedback: it is now clear that this is not a bug, but a local configuration problem.

With respect, glad to help 😀.

(CID:124526203)

[DCIM_Support]

spezialist

I completely disagree, it is definitely  a bug in DCE. It is hard to reproduce, but it's a real one. 

(CID:124526260)

[DCIM_Support]

Took a lot of google searching to find this page but I'm having the same 500 Error issue with remote user accounts on the thin client (Data Center Expert 7.5.0). Again using AD Groups for authentication. Local server accounts do not suffer this problem, only remote user accounts.

I'm in agreement that this must be a bug in DCE as it's hard to reproduce and happens randomly.

I've noticed that since Sep 12, 2017 no further responses have been made in regards to a solution or work around for this problem. I have a set of remote user accounts waiting to go live for monitoring over the Christmas period but I'm unwilling to release these accounts (which are high profile) without at least some kind of resolution to the bug.

(CID:137111046)

[DCIM_Support]

Dear Cassandra Phoenix,

In any case, as Steven Marchetti said in topic , you can report this bug so that you can check it on the resource Feature requests:

The Resources menu and "feature requests" can also be used in case of a bug. I looked at this but have not yet tested it so I've not entered anything in the system.

With respect.

(CID:137111063)

[DCIM_Support]

But its not an authentication error to a NetBotz device - its a communication error to DCE Thin Client. I have no issue at all with any of the NetBotz devices we host. If I post it there it will just get lost in the noise. Besides I'm not asking for a feature request but a solution to an existing issue with the thin client; or are you suggesting I have to make a feature request just to get a bug fixed??

(CID:137111082)

[DCIM_Support]

Dear Cassandra Phoenix,

I meant, that this bug can be reported on the above-indicated Feature requests resource. As far as I understand, this ?bug? (if it confirms =SE=) has not yet been fixed.

With respect.

(CID:137111185)

[DCIM_Support]

So you are saying I need to raise a feature request to get a bug fixed in the thin client.

Can you at least assure me that this issue has been addressed in EcoStuxure? We are looking to upgrade in the new year but I would hate to reassure our end users that it will be fixed in the upgrade only to find it still persists.

(CID:137111294)

[DCIM_Support]

As this topic has been risen up again, I will repeat my solution, as it was stated below.

It seems, that during one of the upgrade, some of sensors were duplicated. And that's the problem.

As it was almost impossible to detect duplicated sensors (I assume, there were a lot of them). we had decided to do clean install.

And that helped.

(CID:137111620)

[DCIM_Support]

Dear Sergey,

From your comment:

It seems, that during one of the upgrade, some of sensors were duplicated. And that's the problem.

As it was almost impossible to detect duplicated sensors (I assume, there were a lot of them). we had decided to do clean install.

That is, it conclusion from this that you had duplicate sensors in the DCE software, to which users did not have access (because of this, an error [500] An error has occured appears)?

With respect.

(CID:137111644)

[DCIM_Support]

Dear Cassandra Phoenix,

From your comments:

Can you at least assure me that this issue has been addressed in EcoStuxure? We are looking to upgrade in the new year but I would hate to reassure our end users that it will be fixed in the upgrade only to find it still persists.

Please tell us what you mean by EcoStuxure?

What =SE= software or service do you mean for which you want to upgrade?

With respect.

(CID:137111648)

[DCIM_Support]

spezialist, no, this is not connected with users rights to anything. This is internal bug in DCE. This error happened with full admin account too, which have all access to all devices and sensors.

(CID:137111682)

[DCIM_Support]

Dear Sergey,

Ok, I understood, although never for my many years of work with DCE software, I haven’t seen this.

Since it’s hard to figure out and reproduce this bug, Cassandra Phoenix can be advised to plan downtime to solve this problem. That is, perform a fresh installation of the DCE software and then deploy a previously created DCE-backup. In my opinion - this is the best and very real solution.

With respect.

(CID:137111730)

[DCIM_Support]

spezialist, Cassandra Phoenix

No-no-no. Backup is not solution. As I mentioned below, backup will restore this duplicates and you will get 500 error again.

Only fresh install, and re-add all devices manually.

(CID:137111743)

[DCIM_Support]

That what really helped us to get rid of this error.

(CID:137111744)

[DCIM_Support]

Ok, thanks for the important clarification.

(CID:137111757)

[DCIM_Support]

spezialist I meant EcoStruxure (I made a typo). We plan to upgrade to this from StruxureWare Data Center Expert 7.5.0 sometime in the new year. To clarify what Sergey commented I too am testing this with a full admin full access to everything remote account so it can't be an issue of device authentication. The account has no restrictions placed on it whatsoever.

As for a fresh install that is something that would need some major planning. We are looking at 300+ devices across several data centers impacting on the environmental monitoring of hundreds of critical systems belonging to many departments throughout the university I work for. If you had some kind of tool that can query the database for the rogue data that would make life much easier.

(CID:137111849)

[DCIM_Support]

Dear Cassandra Phoenix,

And again, what do you mean by "EcoStruxure"?

What upgrade of your infrastructure do you mean?

For example:

1. Stop the use of paid DCE software. Then, deploy the free EcoStruxure IT Gateway software with connect and purchase a subscription to one of two =SE= cloud services: EcoStruxure IT Expert and/or EcoStruxure Asset Advisor?
2. Continuing to use paid DCE software. Then connect and purchase a subscription only for =SE= cloud service EcoStruxure Asset Advisor?

With respect.

(CID:137111865)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.