AP9630 v6.2.1 does not have internet connectivity but LAN works?

[Anonymous user]

Hello,

I have an AP9630 network card for my Smart-UPS, currently running firmware v6.2.1. I recently upgraded from v5.x.x in an effort to resolve this problem however it appears this did not resolve the issue.

Problem

• Email alerts using an external SMTP server do not work.
• It appears that this is due to the NMC not communicating with the internet.
• The NMC is configured with a static IP address and manual DNS settings.
• The NMC does have the ability to "reach" the internet, at least some of the time, because when I do a management reboot, I receive email alerts for the reboot, e.g. "network interface restarted".
• However, when try to send any "test" email via the GUI thereafter, the attempts fail.
• When I telnet into the NMC, I can ping anything on the LAN including the gateway however I cannot ping outside the LAN (I always try using 8.8.8.8 to rule out DNS issues.)
• Using an SMTP server on the LAN works without issue.

So in summary, the issue seems to be more about overall connectivity....but the strange thing is that email alerts (using an external SMTP server) do work immediately after the management reboot so obviously it's not as though the NMC can't route to the internet.

It's possible that this was an issue lurking all along but I never noticed before because I historically have used an SMTP server on the LAN....versus my recent change to an external server.

Help please.

[Terry_Kennedy_apc]

I would double-check the IP netmask and default gateway to make sure they are set to the expected values.

Other possibilities would be:

• Firewall or router access lists that deny access
• Using an external mail server that is not compatible with the NMC (IIRC, there were issues with GMail but I thought they were fixed)
• Blocking of SMTP by your ISP (common on residential service)
• Your IP addresses being in a range marked as "should not send email" blacklist and the SMTP server is therefore ignoring you

[Anonymous user]

Thanks Terry:

A few points

On 3/27/2015 5:15 AM, Terry said:

Firewall or router access lists that deny access

Reply: firewall/router rules checked and verified OK. And remember, alerts are sent when the management network is rebooted so traffic can pass.

On 3/27/2015 5:15 AM, Terry said:

Using an external mail server that is not compatible with the NMC (IIRC, there were issues with GMail but I thought they were fixed)

Reply: Not sure what else to say about this possibility but again, it would seem weird since alerts are sent when the management network is rebooted.

On 3/27/2015 5:15 AM, Terry said:

Blocking of SMTP by your ISP (common on residential service)

Reply: Not the case. All my other devices that use external SMTP function without issue.

On 3/27/2015 5:15 AM, Terry said:

Your IP addresses being in a range marked as "should not send email" blacklist and the SMTP server is therefore ignoring you

Reply: Same as above.

It is simply a bizarre situation as far as I'm concerned...hoping to get some feedback from a APC rep.

[BillP]

This is interesting. Can you share any details of your networking scheme? If you didn't want to publicly, you can open a tech support ticket to provide us with that information or you can message it to me privately (after we become "friends" on the site). I was also thinking VLAN configuration or similar type of issue but I don't know how that'd make sense if it temporarily works right after a reboot.

When the emails stop sending, what is the result of a DNS or email test as far as error messages go?

Are you able to move this device between different subnets to confirm the issue exists no matter what subnet/VLAN the device is located on?

Probably not the issue but this knowledge base came to mind to validate none of the TCP/IP settings show a problem, as explained here -> http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA171016 where the gateway will go to 0.0.0.0.

[BillP]

I was also thinking, maybe a traceroute would help from somewhere outside the local LAN if you can do that. I am not sure if the only thing "outside" is the SMTP server that you have the capability to do something like that with. Not sure if it'd show anything but the thought crossed my mind.

[Anonymous user]

On 3/30/2015 11:16 AM, Angela said:

This is interesting. Can you share any details of your networking scheme? If you didn't want to publicly, you can open a tech support ticket to provide us with that information or you can message it to me privately (after we become "friends" on the site). I was also thinking VLAN configuration or similar type of issue but I don't know how that'd make sense if it temporarily works right after a reboot.

Networking setup is pretty basic (10.0.1.0/24, default gateway: 10.0.1.1) but as you said, I am not sure how anything can be fundamentally wrong with the config given that the NMC does have internet connectivity for several minutes after each reboot.

Re the traceroute, I'm not sure I understand but I can't traceroute to the NMC from outside the LAN since that is not allowed.

DNS tests: Using the GUI, dns tests are successful, which makes sense since it is using DNS servers are on the LAN.

Email tests: Email tests result in "Test e-mail failed; SMTP configuration error" in the gui, and from the event log file: SystemEmail: Could not mail 'smtp-relay.gmail.com'.0x001A

I've also tried using DHCP to assign the NMC IP instead of static config. DHCP works as expected in terms of assigning the IP but same behaviour...loses connectivity to internet/WAN after a few minutes following a reboot.

[BillP]

Hmm, any particular reason why you're not using smtp.gmail.com? I wasn't sure if you were using this in attempt to relay something from an internal server?

And are you using this with SSL/TLS authentication on port 587, etc? Don't know if you want to share your config.ini file from your Network Management Card. Regardless, that would not explain why it works and then stops working.

I was wondering if you see the same behavior with smtp.gmail.com or gmail-smtp-in.l.google.com.

I am remembering a similar issue reported to us last year, specifically with Gmail. I am going to see if we can follow up with the individual to see what the outcome was but it was similar in the sense of it would work right after a reboot and stop shortly after. And then if the user kept doing email tests to "keep it alive" it continued to work. I'll report back on that if I can find out. We had repeated testing here on our network and did not experience the issue. And I know there are plenty of people out there using Gmail as well.

I sort of assumed this was a general access issue from outside the LAN but would you agree the only external connection in reality that fails shortly after reboot would be to external SMTP? And not access to the NMC's interfaces via a WAN or VPN? Let me know if I misunderstood.

[BillP]

I was doing some reading. Is it possible whatever domain you're on or this smtp-relay.gmail.com could be "greylisting" versus blacklisting? 

A mail transfer agent (MTA) using greylisting will "temporarily reject" any email from a sender it does not recognize.

[Anonymous user]

On 3/30/2015 4:34 PM, Angela said:

Hmm, any particular reason why you're not using smtp.gmail.com? I wasn't sure if you were using this in attempt to relay something from an internal server?

smtp-relay.gmail.com is the server used with Google Apps, which my company subscribes to. No user-based authentication is configured because IP authentication is configured at the Google Apps config level....so I can only relay via the SMTP server form an authenticated IP address.

Anyhow, given the fact that email works when "pinging" to the WAN works, I think we established that the issue was overall network connectivity and not SMTP issues.

On 3/30/2015 4:34 PM, Angela said:

I sort of assumed this was a general access issue from outside the LAN but would you agree the only external connection in reality that fails shortly after reboot would be to external SMTP? And not access to the NMC's interfaces via a WAN or VPN? Let me know if I misunderstood.

Not entirely sure I understood but yes I think you're right.

What I have now recognized, is that I add a default route pointing to my firewall at 10.0.1.1 for destination network 0.0.0.0 / 0.0.0.0 (i.e. a catch all), the NMC maintains network connectivity throughout.

So this is a workaround I suppose and I can consider this issue more or less resolved....but I don't understand why this route is needed for the NMC to maintain WAN connectivity given that the default gateway is configured on the NMC.

Fundamentally, I can't tell whether this is an NMC issue or broader network issue but again, I've only experiencing connectivity problems with the NMC....

[voidstar_apc]

It really sounds like the NMC doesn't have the right netmask/gateway for some reason. The "tcpip" command will show what it thinks is the gateway it is using:

apc>tcpip
E000: Success

Active IPv4 Settings
--------------------
  Active IPv4 Address:          10.169.121.145
  Active IPv4 Subnet Mask:      255.255.255.128
  Active IPv4 Gateway:          10.169.121.129

Then we can compare that to an IP where pinging the NMC doesn't work, and see if the IP is inside or outside the netmask.

If everything seems correct there, I'd use wireshark to capture the NMC's response and see where it's actually sending the gateway traffic.

[BillP]

Hi,

I want to let you know that I have the exact same issue as you.  The NMC works for the first 30-60 seconds after being rebooted.  It sends emails, notifications etç.  After that the NMC stops sending notifications.  I also did tests from other subnets, and noticed that for the first 30-60 seconds I can ping the device, after that it stops responding to pings.

I have a case opened with APC, for couple of weeks now.  I have tested 5x, 6x versions, they all have the same issue.

[Anonymous user]

Hi all,

Glad I found this post, as I'm having this issue as well and thought I was going crazy.  After seeing the suggestions above, I mirrored the switch port my NMC is plugged into and ran a WireShark scan while pinging 8.8.8.8 from the NMC, both after a fresh reboot and after waiting for a few minutes.  After the fresh reboot, the NMC correctly reaches out to the MAC address of my default gateway (a SonicWall firewall).  However, after a minute or two, the NMC reaches out to the MAC address of the management interface of the Cisco small business switch it's plugged into.  The switch has L3 functionality, but I don't use it for routing, so of course the switch responded with a Destination Unreachable error.

I confirmed the gateway settings in the NMC.  It's also worth noting that changing certain settings in the NMC fixes the issue momentarily, i.e. if I enable the firewall on the NMC with the default allow all rules and wait a minute, I'm able to ping externally again for a minute or two before the problem pops up again.

Julian, thank you for the route suggestion... I've added a 0.0.0.0 route to the Cisco switch and the NMC can now permanently access internet, but there seems to be some fundamental network issue going on here.  My NMC is on the same subnet as it's gateway so the route above shouldn't be necessary.

During my searching, I've also come accross some older posts that hint of the same issue:
http://forums.apc.com/spaces/11/netbotz-hardware-software/forums/general/9480/test-e-mail-failed-smt...
http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/447/email-n...

I haven't opened any support ticket on this, but if anyone from APC wants the WireShark info, let me know.

Hope this points other people in the right direction as well... thanks for reading!

[Anonymous user]

Hi,

We're experiencing the same issue. Network/SMTP is working fine after reboot. After a few minutes it's losing it's internet connectivity.

You've found a solution yet?

Cheers,

Egbert

[BillP]

Egbert,

Have you upgraded the NMC firmware to 6.4? That is the latest version and it has many upgrades and fixes.

http://www.apc.com/tools/download/index.cfm

[Anonymous user]

Hi,

NMC firmware is already upgraded to 6.4.0, this didn't resolve the issue. I've also tested a reset to factory defaults after the upgrade.

I've been in touch with Schneider support and they've sent a new AP9630. I'm waiting now till it arrives. Will update you in a few days.

Kind regards,

Egbert 

[Anonymous user]

Hi,

New AP9630 didn't solve the issue. When it arrived I firs upgraded firmware to 6.4.0 and edited the settings. After that it was sending mail for a few minutes by using our ISP mail server. After that the NMC is able to ping the gateway but is not able to ping any address outside our company's network.

😞

Kind regards,

Egbert

[BillP]

Hi Egbert,

I asked your local escalation team to step up and help on this case since you're already working with the local customer care center. I did not agree with replacing the hardware for an issue like this so I had looked up the RMA to try and see what happened. This thread is old and has a lot of contributors so I wanted someone to closely investigate your specific case and gather specific data about your setup.

I am still unclear what could be the problem in our system but I won't rule out an issue with it. Please let me know if you do not hear back from your local support team in the next day or so.

[Anonymous user · Accepted solution]

We've got our issue solved after some extensive packet sniffing on the network!

As it turns out we have a multilayer switch in our network setup, we do not use the routing features of this switch. However, the switch sends out ICMP router discovery packets and the UPS then starts forwarding packets destined for the Internet to the MAC address of the switch instead of using its default gateway and it keeps doing so even if the switch answers with a "Destination unreachable (network unreachable)" packet after which the UPS should have reverted to the default gateway as it's last resort.

After disabling all routing features on the switch the UPS can sent mail to an external SMTP server.

Important to notice; after a reset of the management interface the UPS could successfully send mail (one or two) until the first router advertisement was received.