AP9630 - web client no longer accessible

[mrfixit_apc]

Hi,

Just out of the blue the web client on my AP9630 card is no longer accessible. Also SNMP is down - my servers have lost conenction to the card, so they won't shut down if there is a power outage.

I can ping the IP of the card OK, but the client can't be found (http or https). The IP configuration tool also can't find it.

Is my only recourse to reset the card and connect with a serial cable to reconfigure access?

P.S. The card is in a Smart UPS 1000 (SMT1000I).

[mrfixit_apc]

Hi,

Just out of the blue the web client on my AP9630 card is no longer accessible. Also SNMP is down - my servers have lost conenction to the card, so they won't shut down if there is a power outage.

I can ping the IP of the card OK, but the client can't be found (http or https). The IP configuration tool also can't find it.

Is my only recourse to reset the card and connect with a serial cable to reconfigure access?

P.S. The card is in a Smart UPS 1000 (SMT1000I).

[BillP]

Hello,

The first thing I would do is go to the card and verify the status and link LEDs. The status LED is the one with the checkmark or X next to it and the link LED says 10/100 next to it. Let me know what the LED colors are and if they are solid or flashing.

This is also a good time to verify the network cable is attached properly and that for some reason if you disconnect the cable, your ping stops (like if somehow there is a duplicate IP on the network).

Depending on the status/link LED status, I can tell you how to proceed next.

[mrfixit_apc]

Angela,

Thanks. I will check the card LED light status / cabling / ping and then report back. Your help is much appreciated.

[mrfixit_apc]

Angela,

The Status LED is Green and solidly on - I guess that's normal then. The Link LED is Amber and flashing. When I disconnect the Ethernet cable I can no longer ping the card's IP - the Link LED light goes out (of course). When I reconnect the Ethernet cable I can ping the card's IP again and I can also ping it's FQDN (local DNS).

So, what next?

[BillP]

I know you tried multiple access methods - did you try either Telnet or SSH? Whichever you have configured?

I was thinking next, you could try just rebooting the card. You can either re-seat the card (which is ok to do "hot" as it won't affect the load on the UPS or NMC negatively) to power cycle or press the pinhole reset button. If you do the pinhole reset button, the reset button can be hard to find in the hole. I find it easier to use a paper clip and hold it to the top edge of the hole for best alignment. I would just use the ethernet cable to pull out the NMC to reseat it (if you don't have or remove screws securing the NMC in the SmartSlot).

That UPS model you have, I am not sure if it allows you to configure the NMC TCP/IP settings via the UPS LCD but it is worth a check (but make sure the menu is in advanced mode which I think you can configure under Configuration). I can't keep straight what UPS models and firmware versions allow you to adjust it. I was thinking maybe it could help verify the subnet/gateway has not been modified or corrupted. I would see about this if the reboot doesn't change anything.

My other thought was making sure nothing changed on our end, a new proxy configuration, etc that could cause this.

I think the next easiest thing is serial-ing in to verify the settings if possible. Another option would be to try a direct connect via a crossover connection between a PC and adjusting network settings on the PC to the same network as the NMC if not already. This won't help though if there is something on the computer, like a proxy, or the TCP/IP settings on the NMC have been corrupted somehow.

Hope this helps get us started.

[mrfixit_apc]

Angela,

Thanks for the info. I did try Telnet-ting but it got refused. I don't recall the exact message but I think I locked the interface down to reject Telnet. I might try SSH.

I will have a go at rebooting the card via the pinhole reset button and then try connecting serially if that doesn't do the trick.

I don't think my UPS model has the TCP/IP settings on the LCD interface. I will check though.

There is nothing I can think of that has changed in my network of late that might cause this behaviour. However, checking the logs I note that the SNMP connection seems to have dropped out a few times for just a few seconds. I guess this might have been an early signs of a problem.

[mrfixit_apc]

OK, I am in with SSH but I still can't access the management card via http or https, even though the command line command "web" tells me that both are up. I have rebooted several times already but with no change.

The "SNMP" command returns the following:

apc>snmp

E000: Success

  SNMPv1:     enabled

Access Control Summary:

  Access Control #:     1

  Community:            public

  Access Type:          read

  Address:              0.0.0.0

Access Control #:     2

  Community:            private

  Access Type:          write

  Address:              0.0.0.0

Access Control #:     3

  Community:            public2

  Access Type:          disabled

  Address:              0.0.0.0

Access Control #:     4

  Community:            private2

  Access Type:          disabled

  Address:              0.0.0.0

Are these IP addresses correct?

[BillP]

0.0.0.0 (or 255.255.255.255) means that anything with the correct SNMP community names will be able to access the device.

Can you show me the output of the web and console commands if you just type web and console? What firmware is on this NMC (should be listed right when you connect via SSH on the top right - an AOS and an APP version)? Something like v5.1.7 or 6.0.6..

I also would be interested to see the output of the firewall command if it is v6.0.6. Wondering if someone enabled the firewall on the device perhaps.

[mrfixit_apc]

login as: apc

apc@192.168.11.55's password:

American Power Conversion               Network Management Card AOS      v6.0.6

(c) Copyright 2012 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v6.0.6

-------------------------------------------------------------------------------

Name      : AP9630                                    Date : 03/25/2014

Contact   : Colin Fieldgate                           Time : 23:45:48

Location  : Loft                                      User : Super User

Up Time   : 0 Days 0 Hours 10 Minutes                 Stat : P+ N4+ N6- A+

Type ? for command listing

Use tcpip command for IP address(-i), subnet(-s), and gateway(-g)

apc>web

E000: Success

Http: enabled

Https: enabled

Http Port:      80

Https Port:     443

apc>console

E000: Success

Telnet:      disabled

SSH:         enabled

Telnet Port: 23

SSH Port:    22

Baud Rate:   9600

apc>firewall

E000: Success

Firewall:       disabled

File name:      example.fwl

apc>

[mrfixit_apc]

I think it may be a Daylight Saving time issue.I had my main switch to change to BST on the 4th Sunday of March (the 23rd - the day I lost SNMP). It should be the 5th Sunday. My router provides the LAN ntp and is correctly set.

However, although I have set the time/date correctly now on the card it hasn't restored connection.

[BillP]

Does FTP or Secure Copy work to retrieve the event.txt and config.ini files in the /logs directory?

If I had my way with this card , I'd probably format it but that wipes everything and will require reconfiguration from scratch and I usually have the capability to do it. (You type the format command at the SSH command line, and yes to confirm then reboot command). As this is in a production environment for you, I don't know what capability you have to do that/reconfigure it. I am skeptical that this is an issue with the card outside of configuration somehow but I could check the config.ini file to see what I can find if you can get it. Config.ini will also serve as a configuration back up for most items besides user accounts (for security reasons).

Another option is a crossover connection but again, it seems strange that SSH works over the network but HTTP or HTTPS do not. Might not hurt to verify with the tcpip command (to show manual settings, assuming you're using a static IP) the IP settings all look good. I don't know if it is possible for one network protocol to work but not another if the subnet is wrong (as an example) but my experience tells me no for the most part.

Let me clarify also, does the web page just go to something like "page cannot be displayed" or is there any message there that says "this object is protected?"

I was thinking you could try a crossover connection to the card next (which will require you to assign settings on the same network as the NMC onto your computer) if you don't want to format it or I can't get see those log files.

[mrfixit_apc]

tcpip

login as: apc

apc@192.168.11.55's password:

American Power Conversion               Network Management Card AOS      v6.0.6

(c) Copyright 2012 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v6.0.6

-------------------------------------------------------------------------------

Name      : AP9630                                    Date : 03/26/2014

Contact   : Colin Fieldgate                           Time : 00:32:45

Location  : Loft                                      User : Super User

Up Time   : 0 Days 0 Hours 57 Minutes                 Stat : P+ N4+ N6- A+

Type ? for command listing

Use tcpip command for IP address(-i), subnet(-s), and gateway(-g)

apc>tcpip

E000: Success

Active IPv4 Settings

--------------------

  Active IPv4 Address:          192.168.11.55

  Active IPv4 Subnet Mask:      255.255.255.0

  Active IPv4 Gateway:          192.168.11.1

Manually Configured IPv4 Settings

---------------------------------

  IPv4:                 enabled

  Manual Settings:      enabled

  IPv4 Address:         192.168.11.55

  Subnet Mask:          255.255.255.0

  Gateway:              192.168.11.1

  MAC Address:          00 C0 B7 4D 96 E7

  Domain Name:          fieldgate.DSmyNAS.com

  Host Name:            AP9630

apc>

[mrfixit_apc · Accepted solution]

Angela,

OK, so I took courage, formatted the card and rebooted. The card was picked up by DHCP (this did not happen on the older firmware that the card came with - I had to connect via serial cable / hyperterminal) and from there I could get to the web interface and reconfigure the card scratch. Thanks for your help - it's all working now .

P.S.I think that there are some differences with SSL certificates since this reformat / reboot - I don't recall there being a self-signed cert in place when I upgraded the firmware (which I did when I got the card). A cert was generated as a result of the reformat process. Perhaps the firmware upgrade didn't go quite right?

[BillP]

I am glad you got it working! I did not see anything in the logs to tell me there was any weird configuration that would cause this so I would've suggested you do this.

The format function wipes everything in flash memory. It does not modify the firmware in any way. The NMC after the format would default to HTTP only. When you enable HTTPS and do not upload your own certificate before actually enabling it, the NMC creates its own self generated certificate (which you can still replace at any time if you want to use your own).

So, you only will see the self generated cert upon enabling HTTPS, no matter what firmware you're using. It sounds like you formatted, then enabled HTTPS and that is when it was created? A format alone would not generate a cert and would actually remove any existing certs on the system. You might be able to tell by the date/time stamp in the cert when it was generated and align that with event log entry for enabling HTTPS and also there should be an event for when the SSL certificate generation completed.

(Also, I am going to edit your older posts to just put those log files into .txt attachments to avoid scrolling in the thread. Edit - Oh, I guess you removed your logs already fine with me.

I am really happy we resolved this!

[mrfixit_apc]

Angela,

Thanks for the format suggestion - that was what really helped - and DHCP working this time which meant I didn't have to play the serial game again (I had to use a serial connection to set the fixed IP originally when I first set up the card which was running older firmware).

I am really glad my UPS is now protecting my servers again - we have had several outages / voltage fluctuations of late. And yes, I did set up https before attempting to load my old cert (produced using the NMC Security Wizard) - which I see from the documentation is why a cert got automatically created.

The only remaining issue I have is that, whether from the web interface or via ftp, I just can't seem to load my cert correctly.The upload always fails (corrupt transfer). I guess I will figure it out eventually - it's not vital for me.

P.S. I removed the logs as they added nothing and just made the thread much longer!

[BillP]

If you want to send me your cert (offline if you want) I could see if I can load it.

Within the security wizard, did you import it and it was signed by a third party CA or did you make both the CA and SSL cert using the NMC Security Wizard?

Where do you see "corrupt transfer" specifically? (screenshot?)

[mrfixit_apc]

Angela,

I don't think it is a problem with the cert itself - I had it loaded before, but of course it was wiped with the recent format.

I have my own CA set up (OpenSSL based) and I used the NMC Security Wizard to generate a CSR for the card which I then signed with the CA.

The corrupt ftp transfer was confirmed with the xferstatus command.

[BillP]

Another user here recently had a problem with an OpenSSL CA (as well as a Microsoft CA). But if it worked before, then I suppose it should be OK..

I'd still like to try it to just verify it loads for me if you're willing to send it. If so, I can send you an email offline so you could send me the cert. I am sensitive to these SSL Cert issues because I want us to fix any issues so if there is a particular problem, then I could log it.

I checked on xferStatus. It only reports back on file transfers done via FTP or XMODEM protocols (so not for a web upload). This "corrupt transfer" refers to a Bad CRC.

When you do it via the web, does it report anything in the event log? Like Invalid Cert? When doing it via FTP, you put the cert in the /ssl folder?

[mrfixit_apc]

FTP - yes, I tried to put it in the /ssl folder. I checked xferstatus after an ftp transfer fail.

If you send me an e-mail I will respond.

[BillP]

Are you using Chrome for a web browser? We have an issue with Chrome going to a blank web page when uploading files.

Anyway, I'll send you an email to see if I can load it and then we'll figure out how you can get it loaded if I am successful.

Thanks!

[mrfixit_apc]

OK, I checked my FTP settings - I and using WinSCP connected via ftp (I can't get SFTP or SCP to work). If I set binary or text as the transfer setting then the cert transfers fine, but not at the default (what ever that uses) - but, hey, wait, even that now works. Odd.

[mrfixit_apc]

Yes. I am using Chrome. That will be it. I will try IE, even though I am now uploaded.

[BillP]

Sounds like we are headed in the right direction. Every file transfer we do, like firmware upgrades, we always say use binary transfer mode so this all is now starting to make sense. I rarely transfer SSL certs via FTP though so I did not even think of that. I am glad it is all getting sorted out now

[mrfixit_apc]

Angela,

OK, all working now using IE. Many thanks again.

[_eMA]

Hi,

i have same problem with my SMT750IC : i found IP of AP9630 (from display of ups )  but i can not access to web page!

Ping works but configuration IP not found AP9630

How should I proceed ? 

Thanks