AP9630 SMTP Used to work with firmware 6.2.0, does not with 6.5.6 when connecting to EXIM4

[BillP]

I have an SMT1000 with firmware "UPS 09.2 (ID18)" and an AP9630 HW Revision 08 running firmware 6.5.6. I have a local SMTP relay to Amazon SES via EXIM 4. The configuration is pretty straightforward (https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-exim.html).

When I first got the card it worked perfectly with my local relay. The burn-in test says it shipped with firmware 6.2.0. When I upgraded to 6.5.6 SMTP notifications no longer function. The test result in the browser says "Test E-mail sent successfully" and the last server response is 221 but no email is ever received.

DNS is set correctly. I've tried using the FQDN and IP address of the SMTP server. Neither work. There's no auth and no SSL since it's an internal server.

This setup works perfectly with all the services on my network. In looking at the EXIM 4 logs, when I send an SMTP message with the AP9630 "Test" function, this is what comes back from AWS SES: SMTP error from remote mail server after end of data: 554 Transaction failed: Unknown encoding

It looks like there is a 2nd attempt which also gets rejected with a different error message: SMTP error from remote mail server after MAIL FROM:<> SIZE=3849: 501 Invalid MAIL FROM address provided (and indeed the from address appears blank in the logs).

Any thoughts?

[BillP]

Hi Dan,

There are a lot of changes between 6.2.0 and 6.5.6 so we'll have to do a little investigation here.

Is it possible to get a packet capture between the NMC and the mail relay? 

And correct me if I am wrong but the communication path is NMC->EXIM4 Mail Relay->Amazon SES ? 

The only thing that was done was upgrading the NMC? What is going through my mind is how do we know the communication between EXIM4 mail relay and Amazon SES is OK? I think you're saying it is because you don't have problems with any other items using this service, right?

We'd normally check into what is going on with a packet capture and ideally, if you can get a packet capture on the older firmware and newer firmware where it fails to compare. (I can provide the older firmware if you're willing to downgrade and check). We can also try a few other point revisions to see if we can see when it fails. I am not sure if you'll be able to take time to do this though but it'd be helpful for us to see if there is something we can change to fix this.

My only concern is if we only get a packet capture between NMC->EXIM4 Mail Relay, assuming you can do it, I'd hope we also can capture the data specific to this SMTP convo between EXIM4 and Amazon SES to see what is happening on that leg of the communication path.

[BillP]

• A packet capture is easy. I already created one with tcpdump. Just give me someplace to upload it.
• The communication path you cited is correct
• Yes, the only change made was upgrading the firmware
• I configured the relay with Amazon SES using Amazon's recommended configuration. It has worked fine for years. Getting a packet capture of this conversation might be tricky as it's encrypted but I'm sure I can get some more detail if desired using EXIM's logging.
• Sure I can downgrade firmware if you have a process to do that and can provide binaries. Downgrades are typically dicey. I'm going to assume I backup my configuration first, factory default it, downgrade the firmware and factory default it again. Or something like that.

[BillP]

In looking at the dump of the 6.5.6 firmware SMTP communication, I see this:

Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding:

The 2nd field has no value, which is exactly what AWS SES is complaining about.

[BillP]

Thanks Dan.

We have had a problem several years ago that I recall with Content-Transfer-Encoding not populating correctly causing some certain spam filters to reject it whereas other tools and email clients had no issue. I will see if my colleague could check to see if is getting populated or not at the code level in our modern 6.5.6 firmware and the 6.6.4 that is due to come out within a few days. Might need a few days on that but perhaps this problem has reappeared and is a problem in your setup too now. Do other devices have this populated that you're using and are working correctly?

Regarding downgrade, it shouldn't be too difficult or disruptive as you're staying within 6.X.X firmware revs (6.X to 5.X would be more concerning). You can back up your config.ini if you'd like prior but your settings should still stay and you won't see any major problems or have to reset your card or IP address. Settings that aren't available on the older rev may just go away and I am not 100% sure if they'll stick or get defaulted again when you upgrade. So, having the config.ini will be good so you could re-upload it when you re-upgrade to restore the "new" or different settings 6.5.6 offers.

If you do have the time and access, you can indeed downgrade, reset NMC to defaults, even upload your config.ini from 6.5.6 (some settings may not apply if they aren't present on the old firmware), then proceed to test. From there, reupgrade, reset to defaults, re-apply config.ini. This will ensure integrity of the system but to tell you the honest truth, I never do this type of thing between these point versions  

Try this link for the older firmware: ftp://ftp.apc.com/restricted/hardware/nmcard/firmware/sumx/620/apc_hw05_aos620_sumx620_bootmon108.ex...

If it prompts for a password, use jupiter for username and Pl@n3t#5 for password. I do know 6.2.0 has a bug with TLS (hence why we made 6.2.1) so if you use HTTPS for web access, it may not work on a modern browser since years later, now they've blocked SSL and may error out.

I assumed you were using a Smart-UPS based on 6.2.0 but let me know if you have another device/UPS type - Symmetra, etc.

If you have any questions or need clarification before you get started, let me know. But I guess the first thing you could check is if the Content-Transfer-Encoding is populating on the older rev?

[BillP]

Also, are you willing to share the dump or log files you took for reference with any info we should know? You can email them here which goes direct to my Box storage: Email_I.4ygpieocp6gtkedj@u.box.com 

If you have an alternate method you'd like to share via, let me know.

[BillP]

Moving between different firmware revisions was relatively painless. It was a simple process of downloading each one, letting the wizard do the upgrade or downgrade over the NMC, re-run tcpdump and send a test message. Below are the results. Looks like the problem started in 6.5.0. Since 6.4.6 works just fine I will run that until an update is available. Thanks.

6.2.0 (Works)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

6.4.6 (Works)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

6.5.0 (FAIL)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding:

6.5.6 (FAIL)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding:

[BillP]

Hi Dan,

Thanks for the great info and taking the time to test.

Can you send the pcap from 6.4.6 where it is working and shows the Content_Transfer_Encoding field populated properly to the Box folder? I want to show development and get a bug logged with all of your findings. They need to see what is populating there in Hex.

[BillP]

OK, I got the files, thanks again. I'll see if we can get this fixed ASAP, assuming it is just that field that needs to be fixed, but even if we do, you won't see it in the release coming tomorrow of course (6.6.4), but the release after that. 

To make sure that it is fixed with population of Content-Transfer-Encoding, are you able to test a beta assuming we could make one at some point?

[BillP]

Yes I can test a beta. I have a second set of identical hardware with the same problem.

[BillP]

OK, I won't get too ahead of myself but I will reach out when/if I can send something. I think it'd be more like a few weeks till I'd have something, just to set your expectations.

[BillP · Accepted solution]

Thanks for the ballpark. That good because I'm not going to be able to do anymore work on this for a while. Also I have a work around.