Help
  • Get started
  • Ask the Community
  • How-To & Best Practices
  • Contact Support
Login / Register
Brand Logo
Help
  • Get started
  • Ask the Community
  • How-To & Best Practices
  • Contact Support
close
  • Community Home
  • Forums
    • By Topic
    • By Topic
      EcoStruxure Building
      • Field Devices Forum
      • SmartConnector Forum
      EcoStruxure Power & Grid
      • Gateways and Energy Servers
      • Metering & Power Quality
      • Protection & Control
      APC UPS, Critical Power, Cooling and Racks
      • APC UPS Data Center & Enterprise Solutions Forum
      • APC UPS for Home and Office Forum
      EcoStruxure IT
      • EcoStruxure IT forum
      • EcoStruxure IT™ Advisor CFD
      Remote Operations
      • EcoStruxure Geo SCADA Expert Forum
      • Remote Operations Forum
      Industrial Automation
      • Alliance System Integrators Forum
      • AVEVA Plant SCADA Forum
      • CPG Expert Forum DACH
      • EcoStruxure Automation Expert / IEC 61499 Forum
      • Fabrika ve Makina Otomasyonu Çözümleri
      • Harmony Control Customization Forum
      • Industrial Edge Computing Forum
      • Industry Automation and Control Forum
      • Korea Industrial Automation Forum
      • Machine Automation Forum
      • Modicon PAC Forum
      • PLC Club Indonesia
      Schneider Electric Wiser
      • Schneider Electric Wiser Forum
      Power Distribution IEC
      • Eldistribution & Fastighetsautomation
      • Elektrik Tasarım Dağıtım ve Uygulama Çözümleri
      • Paneelbouw & Energie Distributie
      • Power Distribution and Digital
      • Solutions for Motor Management
      • Specifiers Club ZA Forum
      • Електропроектанти България
      Power Distribution NEMA
      • Power Monitoring and Energy Automation NAM
      Power Distribution Software
      • EcoStruxure Power Design Forum
      • LayoutFAST User Group Forum
      Energy & Sustainability Services
      • Green Building Scoring and Certification Forum
      Light and Room Control
      • SpaceLogic C-Bus Forum
      Solutions for your Business
      • Solutions for your Business Forum
      Support
      • Ask the Community
  • Knowledge Center
    • Building Automation Knowledge Base
    • Remote Operations Devices Knowledge Base
    • Geo SCADA Knowledge Base
    • Industrial Automation How-to videos
    • Digital E-books
    • Success Stories Corner
    • EcoStruxure IT Help Center
  • Events & Webinars
    • All Events
    • Innovation Talks
    • Innovation Summit
    • Let's Exchange Series
    • Partner Success
    • Process Automation Talks
    • Technology Partners
  • Ideas
    • EcoStruxure Building
      • EcoStruxure Building Advisor Ideas
      Remote Operations
      • EcoStruxure Geo SCADA Expert Ideas
      • Remote Operations Devices Ideas
      Industrial Automation
      • Modicon Ideas & new features
  • Blogs
    • By Topic
    • By Topic
      EcoStruxure Power & Grid
      • Backstage Access Resources
      EcoStruxure IT
      • EcoStruxure IT™ Advisor CFD
      Remote Operations
      • Remote Operations Blog
      Industrial Automation
      • Industrie du Futur France
      • Industry 4.0 Blog
      Power Distribution NEMA
      • NEMA Power Foundations Blog
      Energy & Sustainability Services
      • Active Energy Management Blog
      Light and Room Control
      • KNX Blog
      Knowledge Center
      • Digital E-books
      • Geo SCADA Knowledge Base
      • Industrial Automation How-to videos
      • Remote Operations Devices Knowledge Base
      • Success Stories Corner
  • companyImpact

Important Announcement: Community Back to Full Functionality

Dear Members, we are thrilled to announce that our Community is back to full functionality and that posts publication is now enabled again! We appreciate your patience during the last weeks. Learn more about our Community Guidelines. Thank you, Schneider Electric Community Team.

DCE sending 1000's of alarms and var/ 100% used

EcoStruxure IT forum

Schneider Electric support forum about installation and configuration for DCIM including EcoStruxure IT Expert, IT Advisor, Data Center Expert, and NetBotz

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Home
  • Communities
  • EcoStruxure IT
  • EcoStruxure IT forum
  • DCE sending 1000's of alarms and var/ 100% used
Options
  • Subscribe to RSS Feed
  • Mark Topic as New
  • Mark Topic as Read
  • Float this Topic for Current User
  • Bookmark
  • Subscribe
  • Mute
  • Printer Friendly Page
Invite a Co-worker
Send a co-worker an invite to the portal.Just enter their email address and we'll connect them to register. After joining, they will belong to the same company.
You have entered an invalid email address. Please re-enter the email address.
This co-worker has already been invited to the Exchange portal. Please invite another co-worker.
Please enter email address
Send Invite Cancel
Invitation Sent
Your invitation was sent.Thanks for sharing Exchange with your co-worker.
Send New Invite Close
Top Experts
User Count
Jef
Captain Jef Captain
79
APC_Steve
Commander APC_Steve Commander
55
gsterling
Commander gsterling Commander
50
Cory_McDonald
Lt. Commander Cory_McDonald Lt. Commander
15
View All

Invite a Colleague

Found this content useful? Share it with a Colleague!

Invite a Colleague Invite
Solved Go to Solution
Back to EcoStruxure IT forum
Solved
DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

0 Likes
10
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

DCE sending 1000's of alarms and var/ 100% used

This question was originally posted on DCIM Support by Garry Priestland on 2018-10-05


We have had 2 DCE servers in the last 24 hours that suddenly started sending 1000's of alert emails out to all email recipients.  I personally got over 50000 from them

We have seen this before on servers with high uptime (both of these were reporting 497 days uptime).  Both are running on 7.4.3 of DCE.  I was under the impression this issue had been resolved after 7.2.5, but I guess not..  All the emails report that the email is repeat number 32xxx - the last 3 digits seem to be different on each server.

None of the alert action are set to repeat.  Due to our experience of seeing this issue before we set up the emails we receive to show the repeat number, since this seems to identify this specific issue is occurring

The emails are relating to historical events and it seems as though DCE is going through the whole history of alerts and resending emails for all of them.

In the past this has been resolved by rebooting the server.

However this did not work for one of them as the postgresql service would not start

The capture logs showed that the var folder was full and since I had previously (luckily) been provided with the root password for this server I went through the procedure provide by Schneider to empty the logs from var/ folder.  This was successful and the server restarted properly.

The second server did restart normally without needing the root password, which was lucky since I don't have the root password for that one!

So here come the questions:-

What causes the issue that makes DCE send out 1000's of emails?

Has this issue been resolved in 7.5.0?

Why does the var/ folder get filled up?

Is the issue regarding the var/ folder getting filled up resolved in 7.5.0?

Since the uptime was reporting as 497 days does this mean that when installing updates and the server says it is rebooting that it does ot really reboot the whole OS? I don't think it is 497 days since this server was updated to 8.4.3.

Most importantly:-

Can you provide the root password for the 2 other servers this customer has to allow us to get them back on line quicker?  (I know you don't like to give these out but please note we are an Elite partner not an end user, and we just want to support our customers...)

What do we do if we see this issue again on a server we do not have the root password for?  How do we get this issue resolved quickly?  As a partner can you provide quick access to this type of requirement?

 

 

If I had not got the root password from a previous issue the server would have been offline for several hours whilst we tried to either get the root password or asked Schneider to correct the var/ folder issue.  And at the end of all this we would have had an extremely unhappy customer..

If it helps I can provide the capturelogs etc for both servers.

 

 

(CID:134679635)

Labels
  • Labels:
  • Data Center Expert
  • Tags:
  • bug
Reply

Link copied. Please paste this link to share this article on your social media post.

  • All forum topics
  • Previous Topic
  • Next Topic

Accepted Solutions
DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

0 Likes
5
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

This answer was originally posted on DCIM Support by John Thompson on 2018-10-05


Hi Garry,

What I can tell you at the moment is that the issue with the /var folder getting filled up is resolved in DCE 7.5.0 so you should have those customers upgrade as soon as possible.

It would be good if you can share the capturelogs fro the two servers. I can contact you offline if need be.

Regards

(CID:134679683)

See Answer In Context

Reply

Link copied. Please paste this link to share this article on your social media post.

Replies 10
DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

0 Likes
5
849
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

This answer was originally posted on DCIM Support by John Thompson on 2018-10-05


Hi Garry,

What I can tell you at the moment is that the issue with the /var folder getting filled up is resolved in DCE 7.5.0 so you should have those customers upgrade as soon as possible.

It would be good if you can share the capturelogs fro the two servers. I can contact you offline if need be.

Regards

(CID:134679683)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

This comment was originally posted on DCIM Support by Garry Priestland on 2018-10-05


Thanks for the answers.

John, can you provide a link tot your Schneider box to uploads the logs files to please?

I believe you know this customer - Telstra, as the other Physical appliance had a similar issue recently that was resolved the same way.

I am being pressured now by the users of the servers to provide a report as to what went wrong...

 

(CID:134679738)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

This comment was originally posted on DCIM Support by Joshua Ellis on 2018-10-05


Hello,  is this the same issue that John Benedict Tayao is working on from Telstra?  They were reporting a performance issue under  BFO case #51942729?  If so, we mentioned the java process was high.  There also may be an issue with a high volume of apache requests that could be affecting this too.  If this is the same issue then I would work through John (Benedict Tayao) for updates.

(CID:134679792)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

This comment was originally posted on DCIM Support by Jim Davis on 2018-10-06


Josh

Not related. Didn't even know this was Telstra that Garry was talking about. I just responded on the /var full issue. Garry must be working on an issue that is at another part of Telstra I am not aware of. I work closely with the Telstra Australia DC team and they have upgraded all of their DCE's to 7.5.0 a couple of months ago.

Jim

(CID:134679852)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:29 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:29 AM

This comment was originally posted on DCIM Support by Garry Priestland on 2018-10-06


Yes correct these servers are located in the UK and not really related directly to Telstra AU.

(CID:134679864)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:30 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:30 AM

This comment was originally posted on DCIM Support by Joshua Ellis on 2018-10-09


Okay, thank you for confirming that.  Another issue from Telstra Australia had come up so just making sure that the two weren't connected.

(CID:134681219)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:30 AM

0 Likes
2
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:30 AM

This answer was originally posted on DCIM Support by Jim Davis on 2018-10-05


Garry

You will most likely find that the upgrades will fail after the upload of the file from the client requiring you to power cycle the server to allow it to restart. I have seen this time and time again. What you will need to do is to create some space on the /var partition to allow the upgrades to work. You will find that the /var/log/atop directory contains around 3GB of log files which are not used. You can safely delete all of them once you obtain the root password from support. Anything higher than 7.2.7 does not have this issue as the 'cleanup' processes worked much better from then. Take it all the way up to 7.6.0 (as of yesterday) and you will get all the other fixes sorted out.

FYI - atop is similar to the linux top program that also writes daily log files and so fills up the /var partition which is only 4GB in size.

 

The 1000+ email issue was finally corrected in 7.5.0. As a workaround if you rebooted the server, the issue will stop for about 6 months before starting again.

Good luck and keep fighting those bugs...

Jim

(CID:134679689)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:30 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:30 AM

This comment was originally posted on DCIM Support by Garry Priestland on 2018-10-05


Since both servers are at 7.4.3 should this cleanup process have prevented the issue in the first place?

 

(CID:134679741)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:30 AM

In response to DCIM_Support
0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:30 AM

This comment was originally posted on DCIM Support by Joshua Ellis on 2018-10-05


Hello,  version 7.4.3 was released in April 2017.  If they upgraded around that time then that would follow the 497 day (1 year ~ 5 months) pattern and would be expected.  If they had upgraded to 7.5.0 when it was released in January 2018, they would not have run into this bug related to the "1000's of emails" either.  FYI, this bug (497 day - 1000's of emails) is not fixed in 7.4.3, 7.5.0, or 7.6.0 (released this past week).  Trying to get this fixed as soon as possible since it comes up frequently.

(CID:134679791)

Reply

Link copied. Please paste this link to share this article on your social media post.

DCIM_Support
Picard DCIM_Support
Picard

Posted: ‎2020-07-05 12:30 AM

0 Likes
0
848
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Link copied. Please paste this link to share this article on your social media post.

Posted: ‎2020-07-05 12:30 AM

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.

Reply

Link copied. Please paste this link to share this article on your social media post.

To The Top!

Forums

  • APC UPS Data Center Backup Solutions
  • EcoStruxure IT
  • EcoStruxure Geo SCADA Expert
  • Metering & Power Quality
  • Schneider Electric Wiser

Knowledge Center

Events & webinars

Ideas

Blogs

Get Started

  • Ask the Community
  • Community Guidelines
  • Community User Guide
  • How-To & Best Practice
  • Experts Leaderboard
  • Contact Support
Brand-Logo
Subscribing is a smart move!
You can subscribe to this forum after you log in or create your free account.
Forum-Icon

Create your free account or log in to subscribe to the forum - and gain access to more than 10,000+ support articles along with insights from experts and peers.

Register today for FREE

Register Now

Already have an account? Login

Terms & Conditions Privacy Notice Change your Cookie Settings © 2023 Schneider Electric, Inc