Large number of emails being sent by DCE

DCIM_Support · ‎2020-07-03

Hi all.

Over the last few months I have had 4x DCE servers for different customers that have suddenly begun to send a very large number of email to email recipients. We are talking 10,000's of emails being sent to multiple recipients, at a rate of several per second. The emails show "Time Detected" on some of the events as being several days, weeks or even months before the Notification time in some cases. Also in each case, we see in the email - "Alert Level: Email - repeat 32651". On at least one of these servers there was not a single Alert Action that was set to repeat the sending of emails.

As far as I can see the emails are actually not regarding recent events but are of old events. It looks like the server may be resending multiple emails for every single alert that has ever occurred. On one occasion I tried to stop the emails by disabling every single device alarm and threshold which did not work. I then tired tom remove the Alert Actions from the Notification policies and still they did not stop. I could only stop them by deleting the email server settings from DCE. As soon as these were entered back in, the emails started again.

The way to fix it seems to be to reboot the server.

The servers were on various versions 7.2.1, 7.2.2, 7.2.5 and 7.2.7.

Obviously the main objective when this happens is to stop the emails being sent which means I cannot easily investigate the issue whilst it is happening.

I have only been able to capture logs from one of the servers too.

Any ideas what may be happening please?

DCIM_Support · ‎2020-07-03

Hi Garry,

Can you share a link to the logs that you did capture.
Did you capture these before or after you rebooted the server?

Thanks & Regards

John T

DCIM_Support · ‎2020-07-03

Same issue happening on my DCE server. The alarms are for current ones (not notifications in the past as Garry has stated). Hopefully the only answer isn't turn it off/on.

DCIM_Support · ‎2020-07-03

I know it doesn't really "help" but we do have this issue mentioned in the DCE 7.4.1 release notes which I think is likely what you're encountering:

Repeat Email Notifications are Sent for a Single Active Alarm

When the Data Center Expert server approaches approximately 500 days of uptime, it is possible to receive many repeat email notifications for a single active alarm. Reboot the Data Center Expert server to stop the repeat notifications.

For now unfortunately, the only fix is a reboot. It happens when a server uptime is relatively high - can you confirm that the servers you had were probably up close to the 500 days? I will add this post to our bug logged on this issue and continue to try and help drive this to resolution internally.

DCIM_Support · ‎2020-07-03

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.