Email notification problems

[robotvoice_apc]

This was originally posted on APC forums on 4/29/2014

---

Hi, I am having some trouble setting up email notifications on my NMC2 card (AP9630) running firmware v6.1.1. I currently have 2 older NMC v1 cards (AP9617) running firmware v3.7.3 on the same subnet which are able to send email notifications just fine. With the same exact settings entered into my NMC2 card I *cannot* get email notifications to send. HOWEVER, if I reboot the NMC2 card through the GUI, I *do* immediately get TWO email notifications that say the network interface has been rebooted. This tells me the device is able to send email just fine but for whatever reason it won't send any other kind of notifications. Test email notifications do not work either on the NMC2. Thanks in advance for any help you can provide!

[BillP]

This reply was originally posted by Angela on APC forums on 5/1/2014

---

The thing people usually do is telnet to the NMC server from a computer as explained in the knowledge base (so you cannot telnet to an SMTP server from the NMC - it only has a telnet server)-> How do I troubleshoot email failures on a Network Management Card? | FAQs | Schneider Electric US

On NMC1 devices, if the server requires a username and password (SMTP authentication), the card will not be able to send email. You can test this by connecting to the email server using telnet over SMTP port 25 and see if it gives a username/password prompt. The event log will contain entries if the connection could not be established.

Since you've been playing with this a little bit, what are your current settings set to? Can you give me an updated config.ini? The first thing I'd be doing is make sure everything is set normally and through configuration changes, something wasn't left incorrect. We can use the local or custom mail server option, just make sure the recipient is set to the correct one. And from your previous configuration, authentication is not used, right (which makes sense because NMC1 doesn't support it)?

If you SSH to the NMC, can it ping the domain name of the mail server you're using? (The CLI has a ping command) You just type ping mailserver.domain.com.

And also might not hurt to do a reboot to see if you're still getting the random emails still as well as just to reboot the system.

[robotvoice_apc]

This was originally posted on APC forums on 5/1/2014

---

(EDIT: The pinging from NMC2 seems to be fine again. Not sure what the issue was earlier)

Is there a possibility I have messed up the config in a way that I would need to wipe everything and start from scratch?

Also, the URL looks strange now when I log into the GUI and I am not sure if it always looked like this or if something has happened. Example, if I put *just* the IP of my card into the browser and hit Enter, it automattically changes the URL to "http://192.168.99.205/NMC/PeoLEYpqt5DS3M2m8bmhCw/logon.htm". I don't remember it doing that before when I first started configuring the card but I could be wrong and might just be paranoid now!

[BillP]

This reply was originally posted by Angela on APC forums on 5/1/2014

---

The login URL is normal. The letters and numbers represent a unique session. I double checked and it looks like firmware v5.1.X makes the session and changes the URL after you submit your user credentials and v6.X.X puts the session string in there right at the log in page as you described.

It may not hurt to start over and reconfigure the card if you can. If you can wipe the entire card, including TCP/IP settings (which means you'll be able to reconfigure it via one of the several methods), I recommend we do the format command via the SSH CLI. This will wipe everything, except the firmware, and is my favorite thing to do when you aren't sure what is going on.

If you cannot afford to wipe the TCP/IP settings because you won't be able to reconfigure it somehow, we'd want to reset to defaults and keep the TCP/IP settings. But then... -> v6.X.X Network Management Card 2 "Reset All" or "Reset to Defaults" Command Does Not Work | FAQs | S... The problem is I wrote the kbase and now I am seeing that I did not document if cfg2.db contains TCP/IP settings or not. So, as I was typing this, I did some researching and found that TCP/IP settings are stored in a different database and removing cfg2.db as the article mentions would not remove TCP/IP settings according to my research and testing. This entire paragraph only matters if you cannot format which is my first preferred recommendation to you

[robotvoice_apc]

This was originally posted on APC forums on 5/5/2014

---

Angela, I formatted the card and started over from scratch. I got the same exact results as before. As a sanity check I moved the NMC2 card over to another subnet to test and it worked flawlessly sending email notifications. The issue seems to be with some part of the configuration on my firewall on this subnet that I have my NMC cards on. I find it strange the the NMCv1 cards work without issue though (as well as some other brand cards that work without issue as well). The logic in your code behind the scenes is handling DNS or something with mail sending much differently between the cards it seems.

Would you be able to explain why I get erratic ping results when I am SSH'd into the NMC2 card? For example, I ping my mail server and get 'request timed out'. Then I ping loopback and get a response. Then I ping my mail server again and it responds back now. Then I ping the mail server again immediately after that and it times out. The NMCv1 card does not replicate this kind of ping behavior and gets responses back on everything I can ping with no time outs. Any thoughts?

[BillP]

This reply was originally posted by Angela on APC forums on 5/5/2014

---

Hi again,

Are the erratic ping times only from the NMC to something else? Or what about when you ping the NMC itself? And is this on the particular subnet that is giving you trouble with sending email or no matter where you put it? Just curious if all these troubles point back to that subnet where you have identified a certain issue with the NMC2s? Another thought was to look in the event log at the same time and make sure nothing else is going on. You can use the eventlog command right after you try to ping.

I would be trying to determine if this looks like it is the NMC or something else going on with the network connectivity as you've seen strange behavior so far on this one subnet for some reason. If anything, the NMC2 is more beefier hardware-wise so I'd be shocked if this had something to do with the load on the card at the time with the processing it is doing for various tasks when you're trying this pinging. I'd expect something like this, if from any of your devices, on NMC1 which is not as robust hardware-wise.

I cannot say I've encountered this particular thing since I don't use the ping tool in the NMC2 all that often but anything you can do to get a packet capture or log showing funky behavior anywhere I can always bring up to the development team to see if we can find anything out. Another option, depending on what you want to do, could be to try v5.1.7 NMC2 firmware and see if it shows any different behavior from v6.1.1. v5 to v6 was a major overhaul at the TCP/IP stack level and just in general. So, if you're interested in doing that, please note you'd want to format after the downgrade to ensure integrity of the system (as noted here -> Things To Consider When Upgrading or Downgrading a Network Management Card 2 (NMC2) Device between v...). I am also assuming you did not try to get anything to work on v5.1.7 before getting these cards to v6.1.1. If you want to try this, I can get you the older firmware.

[robotvoice_apc]

This was originally posted on APC forums on 5/6/2014

---

Can you send me or link me to the 5.1.7 firmware? I would like to test that out next. Thanks!

[BillP]

This reply was originally posted by Angela on APC forums on 5/6/2014

---

Sure - I attached it to this post for the sumx application. Let me know if you have any questions when you downgrade it.

[robotvoice_apc]

This was originally posted on APC forums on 4/29/2014

---

Hi, I am having some trouble setting up email notifications on my NMC2 card (AP9630) running firmware v6.1.1. I currently have 2 older NMC v1 cards (AP9617) running firmware v3.7.3 on the same subnet which are able to send email notifications just fine. With the same exact settings entered into my NMC2 card I *cannot* get email notifications to send. HOWEVER, if I reboot the NMC2 card through the GUI, I *do* immediately get TWO email notifications that say the network interface has been rebooted. This tells me the device is able to send email just fine but for whatever reason it won't send any other kind of notifications. Test email notifications do not work either on the NMC2. Thanks in advance for any help you can provide!

[BillP]

This reply was originally posted by Angela on APC forums on 4/29/2014

---

Hello,

To begin, can you get me the event.txt and data.txt from the NMC2? You can export these from the web UI on their respective pages (Configuration->general->User config file->Download and Logs->event-> "floppy disk icon"). Or, provide all log files from About->Support. If you want to provide a config.ini from an NMC1, I may also need that. Config.ini will help me understand to make sure events have not been specifically disabled for notification but ultimately the test email should still work either way when you perform the test email action.

When you go to do a test email in v6.1.1 and it does not work, what does that screen say for "Last Test Result" and "Last Server Response?" Is this what you meant by "cannot got email notifications to send" - that the test email fails? Based on the description and when you said "test email notifications don't work," I'll go with yes

I am assuming you checked your spam folder for the test emails...? If not, it can go into the spam folder sometimes for the test email.

[BillP]

This reply was originally posted by Angela on APC forums on 4/29/2014

---

I think what it looks for is an MX record in that query. What does the NMC1 respond with if you do the same test? They are using the same DNS server, correct? Depending on that, I need to perhaps look at the differences in the query function a little more in depth.

*Edit* I double checked, it looks for an A record.

[robotvoice_apc]

This was originally posted on APC forums on 4/29/2014

---

I have attached the requested output files for our NMC2. If you need the config from the NMC1 I will post that separately.

If I run a test email on the NMC2, it says:

Last Test Result:	xxxxxxxxxx@xxxxxxxxxxx.com) Test e-mail failed; SMTP configuration error.
Last Server Response:	No data.

Yes, the test email always fails on the NMC2, HOWEVER, if I manually reboot the card I *DO* get 2 email notifications sent to me telling me about this.

Nothing in the spam folder.

Thanks for your help so far!!

[BillP]

This reply was originally posted by Angela on APC forums on 4/29/2014

---

So it looks like you've changed your email server to an IP address per that config.ini? Was that to rule out a DNS problem with the normal FQDN or just how you normally do it - my guess is to rule out a DNS issue since I see lots of Email: Could not mail '10.1.0.26' but also saw  Email: Could not mail 'xxxxxxx.com'  perhaps.

I was wondering if you could try a DNS lookup on the domain name of the email address and also try by "IP" to see if the NMC can do a DNS resolution on that IP to the correct mail server name, assuming there is a DNS record for it.What I saw in the log makes me think they will fail.All of this is pointing to a connectivity problem to this server for some reason. In addition to a DNS test, there is a ping utility within telnet/SSH if you have access to that to see if the IP will ping.

No data usually also tells me it cannot connect to the mail server at all - to verify that, can you let me know if you're able to access the mail server? If so, I'd want to know if you even see any connection attempts from the NMC's IP to start an SMTP conversation. This could also mean a misconfigured NMC for your SMTP options (authentication, port, etc) but NMC1 doesn't offer any of that so that is likely not it.

I also saw your DNS was:

PrimaryDNSServerIP=192.168.99.1

SecondaryDNSServerIP=8.8.4.4

So since the mail server is a 10.X.X.X address, the NMC subnet is configured be able to access the DNS server's subnet without issue? And it looks like for the secondary DNS, you have a public server?

I am leaning towards this being a DNS issue so that is what I'd be looking at in the NMC1 config.ini 's that are working to compare..

[robotvoice_apc · Accepted solution]

This was originally posted on APC forums on 4/29/2014

---

Correct. The FQDN was not working either so putting the IP address in was part of my troubleshooting. However, the DNS test results do point to something being misconfigured.

Under DNS test in your GUI, if I do a FQDN query type against my mail server, it responds back with: DNS response packet received, did not contain answer to the query. However, if I do an IP query type against my mail server IP, it resolves the domain name correctly.

I do have access to our mail server and can look into the backend to see if I can rule anything out there in the meantime.

We utilize different VLANs here but to answer your question, the subnet that my NMC cards are on do have access to get to DNS and mail without problems. The google DNS was in there for troubleshooting, you can disregard that as well.

[robotvoice_apc]

This was originally posted on APC forums on 4/29/2014

---

On the NMC1;

FQDN query results - DNS server could not process query

IP query - successful

Email notifications and test email notifications work fine on NMC1 though. I have our mail server domain name in the SMTP settings on NMC1 as well.

[BillP]

This reply was originally posted by Angela on APC forums on 4/29/2014

---

Can you validate what type of record your DNS server has for this mail server IP? Is it something other than A? I am by no means a DNS expert but if it is not an A record, I was thinking, well perhaps that is why we are having a problem with the test since that is what it looks for at the code level but I'd have to look deeper into the resolution process for sending mail but I have to make an educated assumption it's the same as the test process.

I usually like doing packet captures at this level but we can't run from the NMC and I assume you cannot from the DNS server during the test. And assuming the mail server shows no connectivity requests from the NMC, a packet capture there won't be too helpful since again, we cannot run it from the NMC side unless you have a way to mirror ports or you happen to be using a hub on your network

Just for the heck of it too, I don't know if you want to try the "custom" mail server option with the server information as opposed to the 'local' option? "Custom" is intended for situations with multiple recipients using different mail servers but I tend to always use it anyway. I am just talking through what I'd do on my own to rule out issues with the 'local' mechanism.

Can I ask what mail server this is? Exchange, etc?

[robotvoice_apc]

This was originally posted on APC forums on 4/30/2014

---

We have an A record for our mail server in our DNS.

I tried the "custom" server option on the NMC2 and got the same results. We have an Exchange 2010 server here.

I have a feeling there is something strange going on with our DNS setup I need to look into. If you have any other suggestions for things I can test I would be more than happy to try. Thanks for your help so far.

[BillP]

This reply was originally posted by Angela on APC forums on 4/30/2014

---

Hmm, not beyond what I mentioned. To rule out a DNS issue, along with changing your mail server name to an IP, can you also change the recipient's to something like recipient@10.10.10.10 or whatever the IP is?

I tried playing around with mine and it at least does not reject that format (but I know I had trouble making sure I put the right IP in there). I eventually fiddled around and figured out the right IP for representing my email domain (or so I thought) and got it to work to at least send the mail according to the NMC to try and bypass DNS fully. But, I never received the email until I changed back to using my domain on the recipient.