AP9630 email notification issues

[Anonymous user]

I am having issues getting email notifications working. I understand the default USER with no password is for NO Authentication. I am trying to mail relay off of a SMTP gateway that does not require authentication. But when I enable an e-mail recipient and try to send a test message. The GUI interface hangs and I lose PINGS to the NMC.

About 1 minute later pings come back and I can login to the NMC again but 20 seconds later I lose PINGS and communication to the NMC card again. I have to finally login and disable all email recipients to correct the issue. So I cannot get email notifications working on the AP9630 with 5.1.5 smux/aos 1.0.2 bootmon.

Anyone else see this issue?

My ticket number is 110726-000377 if anyone can help.

Ronnie

[BillP]

Hi Damian,

What I would check first is your SMTP server's log and see if there is an incoming connection from the AP9630. If there is, I am interesting in seeing the SMTP commands and how far that the card is able to get.

There is an issue that comes to mind that we fixed in the next firmware version of this card (6.0.6) but 6.0.6 has a few issues of its own pertaining to email so I am not yet telling you to upgrade to fix this until we confirm it is the problem I think it could be.

Can you verify if your SMTP server is configured for SMTP or ESMTP (extended SMTP)? Here is an outline of the issue that comes to mind that maybe you are experiencing. The two ways to verify it are checking the log on the SMTP server end to see what commands are going back and forth or a packet capture.

When an NMC2 device wants to send email via SMTP, it sends the EHLO SMTP command to the configured mail server, indicating it would like to use the Extended SMTP (ESMTP) protocol. Most mail servers support ESMTP and will acknowledge the request, and allow the mail to be send or indicate it requires authentication or other options before forwarding the email. If the mail server does not support ESMTP, it will generally report a message back to the NMC along the lines of "Command Not Supported" and then proceed to offer standard SMTP using the command HELO.

When the NMC2 sees this, it will offer the HELO command and use standard SMTP to send the email, per the mail server's request. The mail server would support this, and in a normal situation, say something like Command OK, and proceed to send the email.

An issue was discovered where the NMC2 will send the EHLO SMTP command and the mail server will reject it - as expected, if the mail server does not support ESMTP or has it disabled. The NMC2 will try HELO next and the mail server will accept it but the NMC2 then gets stuck and potentially hung up on the previous error it received from the mail server on its initial EHLO command. This will cause the email message to not go though and ultimately fail.

Resolution

Investigate the capability of the mail server to see if it will support Exetended SMTP and the HELO command. The NMC2 will then be able to send email. Alternatively, NMC1 devices do not have this problem.

This will be addressed in a future revision of NMC2 firmware for most applications - AOS v5.1.9 or higher.

[damianm_apc]

Hi Angela,

Thanks for the reply.

Let's define the Remote Network as the one which has my client's Ap9630 on a SmartUPS and also a machine which we will call the

Remote-Mail-Gateway Also, my own network can access that Remote Network, and  the Remote Mail Gateway allows relaying from

any host on my own network., On a Linux host on my own network, I then type

     telnet Remote-Mail-Gateway 25

and feed it the SMTP speak

EHLO  hostname

MAIL FROM:<source-user-email>

RCPT TO:<target-user-email>

DATA

From:

To:

Subject: Test From Mail-Gateway

- System

.

QUIT

Everything works. The email gateway sends me that email. While I am not using any Extended features, I am sure the Remote Mail

Gateway supports ESMTP.

If I change the AP9606 on my own lovely Symmetra RM to use that Remote-Mail-Gateway as its mail host and force my Symmetra

to run an email test, everything runs as expected. I receive the email from  the correct user on the AP9606 which says.

Subject: Email Test

    [ The following text is in the "iso-8859-1" character set. ]

    [ Your display is set for the "ANSI_X3.4-1968" character set.  ]

    [ Some characters may be displayed incorrectly. ]

This is a test of the Email client.

Going back to the problematic Ap9630, with LogLevel = 90, I see the following traffic on the Remote Mail Gateway when talking to the AP9630

sm-mta[19788]: NOQUEUE: connect from apc981838.Source-Domain [Rem-UPS-UP]

sm-mta[19788]: rBA0ohAT019788: Milter: no active filter

sm-mta[19788]: rBA0ohAT019788: --- 220 ngmb.Source-Domain ESMTP Sendmail 8.14.6/8.14.6; Tue, 10 Dec 2013 11:50:43 +1100 (EST)

sm-mta[19788]: rBA0ohAT019788: <-- EHLO apc981838

sm-mta[19788]: rBA0ohAT019788: --- 250-ngbm.Source-Domain Hello apc981838.Source-Domain [Rem-UPS-UP], pleased to meet you

sm-mta[19788]: rBA0ohAT019788: --- 250-ENHANCEDSTATUSCODES

sm-mta[19788]: rBA0ohAT019788: --- 250-PIPELINING

sm-mta[19788]: rBA0ohAT019788: --- 250-8BITMIME

sm-mta[19788]: rBA0ohAT019788: --- 250-SIZE

sm-mta[19788]: rBA0ohAT019788: --- 250-DSN

sm-mta[19788]: rBA0ohAT019788: --- 250-ETRN

sm-mta[19788]: rBA0ohAT019788: --- 250-DELIVERBY

sm-mta[19788]: rBA0ohAT019788: --- 250 HELP

sm-mta[19788]: rBA0ohAT019788: <-- MAIL FROM:

sm-mta[19788]: rBA0ohAT019788: --- 250 2.1.0 ... Sender ok

sm-mta[19788]: rBA0ohAT019788: <-- RCPT TO:

sm-mta[19788]: rBA0ohAT019788: --- 250 2.1.5 ... Recipient ok

sm-mta[19788]: rBA0ohAT019788: --- 421 4.4.1 ngbm.Source-Domain Lost input channel from apc981838.Source-Domain [Rem-UPS-UP]

sm-mta[19788]: rBA0ohAT019788: lost input channel from apc981838.Source-Domain [Rem-UPS-UP] to MTA after rcpt

sm-mta[19788]: rBA0ohAT019788: from=, size=0, class="0", nrcpts=1, proto=ESMTP, daemon=MTA, relay=apc981838.Source Domain [Rem-UPS-UP]

What is up?

Thanks - Damian

[BillP]

Well, we've looked at the log and we're not sure yet. Is a packet capture an option from this mail server?

Can you provide the same log from when the AP9606 works successfully?

Can you tell me the application name you're using from your firmware? (Administration->General->About is where this is located. I need the application module name i.e. sy)

We may have an upgrade available to try just as an option to see if it behaves differently but there are a few caveats with it. So, I don't want to make you upgrade and go through it all and it still does not work.

[damianm_apc]

The problem still exists, i.e. when running 'Test' from the email message I get 'SMTP configuration error'.

However, the system ran a scheduled self-test today and the email worked. So the problem is actually in the 'Test' menu!!!!! The email configuration is correct. So the bug lies in either the testing code or the error messages.

In answer to your question, the Application name from the firmware is 'sumx'.

I will sent you a log from the AP9606 on the weekend, but not today.

I do not want to upgrade the firmware on it because the unit is 600miles away. And it is a new customer so I do not want to take any risks. Does the AP9630 work in my Symmetra? If so, I might get one here as a spare and try it within my office. Too bad APC do not have a Not-For-Resale program. While you are looking at fixes, the new web interface is awful. And so, so slow. The latency on keystrokes is woeful. The web page, or CSS, should be configured to expand to fill the available browser page. The way it is now, it has a very small font and is much harder to read than the older interfaces.

Thanks - Damian

[BillP]

hi again, well i am glad it worked. based on what we have so far, i am not necessarily convinced all the blame should go on the card but i understand what you're saying. And maybe something wasn't write at the time of the test. I just say this based on the fact that there is no problem with the test menu for the 99% of other customers using the product so while there might be a particular issue, it is very isolated to whatever configuration your mail rely might be using.

And yes, we are working on some enhancements to the interface already, some of which you mentioned.

Yes, we have 6.X.X firmware (6.1.1 coming soon to fix two specific issues) for symmetra single phase units (LX, RM, Power Array) or Smart UPS with sumx. What model is your Symmetra?

I agree it is probably not a good idea to update your customer, especially until 6.1.1 comes out with some issues addressed but it is good enough to test on your end if you wanted.

[Anonymous user]

I am having issues getting email notifications working. I understand the default USER with no password is for NO Authentication. I am trying to mail relay off of a SMTP gateway that does not require authentication. But when I enable an e-mail recipient and try to send a test message. The GUI interface hangs and I lose PINGS to the NMC.

About 1 minute later pings come back and I can login to the NMC again but 20 seconds later I lose PINGS and communication to the NMC card again. I have to finally login and disable all email recipients to correct the issue. So I cannot get email notifications working on the AP9630 with 5.1.5 smux/aos 1.0.2 bootmon.

Anyone else see this issue?

My ticket number is 110726-000377 if anyone can help.

Ronnie

[BillP]

yes, believe me I wish i could give it out now 😞

for anyone that has our newer PDUs that use rpdu2g app, versus 'sumx' for UPSs, i believe a fix for this will also be contained in AOS 5.1.8 and rpdu2g 5.1.5 when its released shortly but we cant load this on the UPSs unfortunately. and the UPS applications will be v 6.x when they are released.

[Anonymous user]

Hi.

I believe I have the exact same issue:
System: TCP/IP stack failure. Module: tfSheapFree-assertion error line 625, file(.\treck\source\trsheap.c)

I tried to use smtp-relay.ip.nianet.dk (aka mailserver3.dk) as SMTP-server. The initial response contains the following lines:
220-outgoing.mailserver3.dk ESMTP mailserver3.dk Hosting Center Aarhus; Wed, 19 Sep 2012 12:55:54 +0200
220-******************************************************************
220-* fuzion a/s Hosting Center Aarhus *
220-* *
220-* MAILSERVER3.DK (rr2.mailserver3.dk) *
220-* *
220-* All connections are logged! *
220-* This server uses antivirus and antispam technology *
220 ******************************************************************

This immediately crashes the card.

Version-info:
Bootmon: bootmon:v1.0.2 (Jan 21 2010)
AOS: aos:v5.1.7 (Nov 22 2011)
App: sumx:v5.1.7 (Dec 1 2011)

November 7th 2011 you wrote "this will be resolved with a future firmware update." Any idea when that will be?

Best regards,
Søren Thing.

PS: I believe the old AP9617 card has the same issue - I actually bought the new card to fix those reboots. Feeling kind of stupid now...

[BillP]

hello - for sumx, as of today we are looking at a Decemberish time frame for when I can provide it to this thread. i don't think it will be posted on the website until sometime after that but for an issue like this, i will post the firmware file once it is the final version.

[Anonymous user]

Hi "i picKed a winna".

Nice to hear that we are moving toward a solution 😉

/Thing

[BillP]

hi everyone,

we have discovered this to be an issue with the size of the SMTP server's 220 response after doing a network packet capture. the network management card cannot accept more than a 384 byte size (or around there) and in this particular scenario, the response was 400+ bytes, causing the NMC to not send email and get stuck in a reboot loop. this will be resolved with a future firmware update.

[BillP]

hi

i just helped someone else with this exact scenario the other day and got it working (the mail relay part).

first, we just came out with some new firmware, AOS 5.1.6/sumx 5.1.6. i couldn't tell from the incident if the update was done or not yet so please ensure you are at the rev i attached to this post.

next, can you please back up the config.ini again and go to the web on the Administration tab->General->Reset/Reboot and reset to defaults EXCLUDING TCP/IP settings so you don't lose your network settings. the config.in will save everything else when you reset the card.

after you reset it, you need to log off so it can reboot. log back in and configure the mail stuff manually (DNS, recipients, server, etc) and let me know if that works or not.

[Anonymous user]

AOS 5.1.6/sumx 5.1.6 installed now
Reset to defaults >>>>excluding IP configuration
entered mail relay information and recipients
tried to send test email

Still Hangs and loses PINGS.

[BillP]

ok, can you check for the following files via FTP for me to look at?

config.ini, event.txt, data.txt and check the /dbg directory for dump.txt or debug.txt?

instructions are here -> http://nam-en.apc.com/app/answers/detail/a_id/9321

thanks.

[Anonymous user]

Files you requested attached.....

Please review dump.txt

Also note the errors in the event.txt

"System: TCP/IP stack failure. Module: tfSheapFree-assertion error line 625, file(.\treck\source\trsheap.c)"

[Anonymous user]

Also just to provide you a little background here.

We have about 10 standalone Smart 1000 UPS's with the exact same issue. I have re flashed firmware 5.1.6 multiple times on my test UPS's.

I didn't have any issues with the previous version I had installed for many years.

; Network Management Card AOS v5.0.1
; Smart-UPS & Matrix-UPS APP v5.0.0

Before you guys introduced USER authentication feature. I had to upgrade the code to prevent cross site scripting. So I just installed the latest firmware and now you see my issue.

[BillP]

yes, i understand. the error in the log with the heap is important. this does not happen all the time so we need to figure out what is specifically causing it. give me a few days to look into it. probably around Tuesday-Wednesday (because of the labor day holiday) I will have some answers or better ideas.

[BillP]

alrighty. i will report back as soon as we have anything definitive on that dump you provided. still looking into it for you. thanks for your patience.

[BillP]

i did not see this in the email incident but let me know if I missed it..

did you try a format on the NMC? it wipes out flash completely beyond a reset to defaults. it WILL wipe out your TCP/IP settings though so i'd only recommend it if you can reconfigure the card with a serial cable or other method. i have had luck with it in other scenarios but we can hold this as a last resort.

you can format by doing one of these options (you don't need to do both):

1.) issue the format command via the telnet or local command line interface

2.) hold down the reset button for 20+ seconds until the status LED goes off.

[Anonymous user]

I had not tried that yet.....

I just FORMATTED the card command line and only entered Email receipents after the card came up DHCP.

Same issue

System: TCP/IP stack failure. Module: tfSheapFree-assertion error line 625, file(.\treck\source\trsheap.c)
0x001D

I have attached the debug files for your review.

[BillP]

i forgot to look into this earlier. we are looking at the dump and will see if we can track down what is causing this. very strange since we have never seen that and i have personally set up email hundreds of times i feel like with no issues.

i guess i do have one question.

how is this working with no DNS entered?

[NetworkDNS]
PrimaryDNSServerIP=0.0.0.0
SecondaryDNSServerIP=0.0.0.0

are you able to enter DNS servers in via the web UI which will be required since you are using FQDN's for your SMTP info? in your original file, I do see them:

PrimaryDNSServerIP=10.192.250.40
SecondaryDNSServerIP=10.193.250.40

doesnt seem to be the issue but something i just noticed.

[Anonymous user]

I noticed the same thing. I guess since the card defaults to DHCP and I didn't add anything. I did ping my mail relay server by hostname from the NMC test ping before I tried to send the test email. So DNS is working I assume just picking it up via dhcp and doesn't fill the NETWORK DNS section in the config.ini file for some reason.

I did also manaully enter those ip's below and it failed also.

[Anonymous user]

3 total AP9630 cards that this is happening on. 1 was new out of the box. I stopped buying this model until we get email's working again.

[BillP]

hello,

sorry for the delay..

i have a few suggestions here based on your dump.txt - can you please follow these specific instructions and see what happens so we can confirm or deny what the issue is and address it.

Let's start with a format again - where we wipe the enter flash out on the NMC. Manually configure the DNS servers, then do a DNS test from the card to the SMTP server: FQDN of the mail server to the server address, then query by IP on the address we get back. These both need to be working. Next, set the card's time to the current time and date. Then set up the SMTP server and email recipients.

please let me know the results when you get a chance. thanks!

[Anonymous user]

Results-


Same issue. Hangs as soon as I try to send Email notification. I tried both manually setting time and SYNCing with our internal NTP server. Same results with failures both times.

In our setup we have a CName alias setup for mailrelay that resolves to our exchange server hostname. All results attached along with NTP and manual time set dumps.

Ronnie

[BillP]

still researching - do you have more than one card with this problem or just the one?

[BillP]

this is a stumper. we have tried your config files, except DNS servers on an AP9630 and AP9631 and cannot replicate this - even while following the steps you have with a format, etc.

few other questions:

1.) what UPS(s) are these installed in? based on your config file, i took a guess that this is a SUA750, SUA1000 or SUA1500 UPS. we'd like to confirm though just incase.

2.) can you try this again but disable IPv6 on the NMC and then proceed with the rest of the email setup to see if it makes a difference?

[BillP]

if its ok with you, i'd like to take this offline and then hopefully once we reach a resolution, we can post back what we find.

i will send you an email.

[Anonymous user]

All UPS's are SAU1000 - Smart UPS 1000

Just tested all 3 just now same issue on all of them hangs on email test. Disable IPV6 as requested same issues.......

Ups 1000 with AP9630 firmware all have this is common

Name: sumx
Version: v5.1.6
Date: Aug 11 2011
Time: 10:01:53


APC OS (AOS)

Name: aos
Version: v5.1.6
Date: May 11 2011
Time: 11:35:16


APC Boot Monitor

Name: bootmon
Version: v1.0.2
Date: Jan 21 2010
Time: 13:35:57

[Anonymous user]

I hate to bring up an old thread but I've searched and haven't found a solution. 

We have three 9630's that are exhibiting the exact same issue. We're on v.6.2.1 connected to SmartUPS 5000. 

Every time I set the smtp server and do a test email, the card reboots and I lose connectivity. After about a dozen unresolved pings, the card comes up for about another dozen resolved pings before it repeats itself. 

Any ideas?

[Anonymous user · Accepted solution]

Hello, I am a new user with this same issue.  Did you find a resolution?

Dino