Cannot connect over HTTPS to AP9630 with Chrome

mrfixit_apc · ‎2021-07-01

I have had my Smart-UPS 1000 (SMT-1000i) UPS with an AP9630 Management 2 Card configured and working for a while now and access it via HTTP or HTTPS (I have installed an SSL Certificate). However, using Chrome I now get a message that says:

The webpage at https://ap9630.xxxxxxxxx.com/logon.htm might be temporarily down or it may have moved permanently to a new web address.

Error code: ERR_SSL_FALLBACK_BEYOND_MINIMUM_VERSION

I take this to mean that Chrome is not allowing fallback to SSL v3 as a defence against the POODLE bug - which is good. However, the APC firmware v6.2 is not allowing the TLS protocol (I am guessing).

I can access the interface via IE 11, so I am assuming MS have yet to patch IE.

So, will there be a new firmware to handle this situation soon?

BillP · ‎2021-07-01

The knowledge base article I posted above mentions that there will only be a firmware update for NMC2 devices as it stands now. AP9617/18/19 UPS network management cards definitely will not have any updates as they were replaced by AP9630/31 NMC2 in 2009 but NMC2 will have a firmware update. Enabling SSLv3 will be the only option for older NMC1 cards at this time.

See Answer In Context

BillP · ‎2021-07-01

I have the same problem with an AP7900. APC firmware does not seem to support TLS

mrfixit_apc · ‎2021-07-01

Anyone?

BillP · ‎2021-07-01

Both generations of Network Management Cards support TLS 1.0 and will try that before SSL 3.0. AP9630/31 will support TLS 1.1 sometime next year.

I would double check your Chrome settings to see what versions of TLS are enabled.

Here is our knowledge base on POODLE, just in case you'd like to review it. Security Notification: "POODLE" vulnerability - impact to APC products | FAQs | Schneider Electric U...

mrfixit_apc · ‎2021-07-01

I am running the 64 bit version of Chrome: Version 39.0.2171.62 beta-m (64-bit). Could that be it?

With IE 11 I switched off the use of SSL v3.0 and it connected OK. So I am guessing this is a Chrome 64 bit beta specific issue.

mrfixit_apc · ‎2021-07-01

OK, so I have now tried Chrome 32 bit on another PC and I have the same problem. So I don't think it is a specific 64 bit beta issue.

It's odd as I have several other devices on my LAN that I access with HTTPS and for which I have generated my own certificates in the same way, and these work with all versions of Chrome and connect with TLS 1.0 or 1.2.

BillP · ‎2021-07-01

Oh, OK. I'm not sure. I think Chrome and IE use the same settings as far as "Internet Options" where SSL/TLS are configured. It could be something, no matter if 32 bit or 64 bit, Chrome decided to put in the beta before finalizing the release. I would say maybe research that Chrome error and maybe there is a way to disable the fallback thing but I don't understand why it would show if it should be trying TLS 1.0 first.

BillP · ‎2021-07-01

Yes, it does support TLS. I tested this recently but I will have to verify what version of Chrome I tried it on. I replied to your other post FYI. You may have multiple issues that apply to AP7900 and Network Management Card 1 devices.

Anonymous user · ‎2021-07-01

We can confirm that in the latest chrome version (tested with version 39.0.2171.65 m) due to removed SSL v3 support Chrome fails to connect to any APC AOS device with error code ERR_SSL_FALLBACK_BEYOND_MINIMUM_VERSION.

A couple of days ago it was working. All devices have 1024/2048 Bit certs/keys.

Tested with various APC devices running v3.7.4 up to v6.2.0.

Potentially there's a compatibility issue with your SSL implementation

Using openssl with explicit "use only tls1.0" flag set fails to connect too:

# openssl s_client -connect ups44.example.com:443 -tls1
CONNECTED(00000003)
140091215320736:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1260:SSL alert number 40
140091215320736:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1417007587
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

Using explicit SSLv3 flag it works as expected:

# openssl s_client -connect ups44.example.com:443 -ssl3
CONNECTED(00000003)
[CERT REMOVED]
---
No client certificate CA names sent
---
SSL handshake has read 1178 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC3-SHA
    Session-ID: 587372918C4B867C6A18D49F5FA5F131
    Session-ID-ctx:
    Master-Key: 72F4DA3F1E8953236BB1EA073CDE035672940AA6ECF9C7087ACDB7C7B1CFD4C791D24D50A7E94D6FDC9C10DAD3821326
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1417007688
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

OpenSSL without any explicit flags set fails to negotiate a protocol completely:

# openssl s_client -connect ups44.example.com:443

CONNECTED(00000003)

139922286679712:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:732:

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 7 bytes and written 213 bytes

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

---

Anonymous user · ‎2021-07-01

If SSLv3 is disabled (setting security.tls.version.min to 1) in Firefox v33.1 it fails to connect too.

Given that they intend to disable SSLv3 in v34 that will be a problem very soon.

Anonymous user · ‎2021-07-01

I looked at this using wireshark (using a v6.2 APC device).

Using Firefox with SSlv3 disabled it will send a TLSv1.1 handshake with support for downgrade.

The APC SSL stack can't handle this and sends a Handshake Failure alert.

At this time I'm already sure your implementation is broken as the hello send from Firefox is perfectly fine.The server should accept the message and return a server hello with tlsv1.0.

A successful TLSv1 handshake using IE:

Secure Sockets Layer

TLSv1 Record Layer: Handshake Protocol: Client Hello

Content Type: Handshake (22)

Version: TLS 1.0 (0x0301)

Length: 136

Handshake Protocol: Client Hello

Handshake Type: Client Hello (1)

Length: 132

Version: TLS 1.0 (0x0301)

Random

gmt_unix_time: Nov 26, 2014 14:55:30.000000000 W. Europe Standard Time

random_bytes: 26c225231e3ae5a82b0ca4e2f3b5b04e66aa8d6ef2db8f07...

Session ID Length: 16

Session ID: d63ac5f3268389cff3b5fa9ce18489bd

Cipher Suites Length: 20

Cipher Suites (10 suites)

Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)

Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)

Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)

Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)

Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)

Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)

Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)

Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)

Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)

Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)

Compression Methods Length: 1

Compression Methods (1 method)

Compression Method: null (0)

Extensions Length: 55

Extension: renegotiation_info

Type: renegotiation_info (0xff01)

Length: 1

Data (1 byte)

Extension: server_name

Type: server_name (0x0000)

Length: 21

Data (21 bytes)

Extension: status_request

Type: status_request (0x0005)

Length: 5

Data (5 bytes)

Extension: elliptic_curves

Type: elliptic_curves (0x000a)

Length: 6

Elliptic Curves Length: 4

Elliptic curves (2 curves)

Elliptic curve: secp256r1 (0x0017)

Elliptic curve: secp384r1 (0x0018)

Extension: ec_point_formats

Type: ec_point_formats (0x000b)

Length: 2

EC point formats Length: 1

Elliptic curves point formats (1)

EC point format: uncompressed (0)

Response:

Secure Sockets Layer

TLSv1 Record Layer: Handshake Protocol: Server Hello

Content Type: Handshake (22)

Version: TLS 1.0 (0x0301)

Length: 58

Handshake Protocol: Server Hello

Handshake Type: Server Hello (2)

Length: 54

Version: TLS 1.0 (0x0301)

Random

gmt_unix_time: Sep 24, 1998 09:40:38.000000000 W. Europe Daylight Time

random_bytes: 555953ebb2340b56ccdc9a291378581cbae1724b6625d320...

Session ID Length: 16

Session ID: d63ac5f3268389cff3b5fa9ce18489bd

Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)

Compression Method: null (0)

TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec

Content Type: Change Cipher Spec (20)

Version: TLS 1.0 (0x0301)

Length: 1

Change Cipher Spec Message

TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message

Content Type: Handshake (22)

Version: TLS 1.0 (0x0301)

Length: 40

Handshake Protocol: Encrypted Handshake Message

Chrome trys to start with tlsv1.1 too which fails just like firefox.

But apparently it's designed to handle broken SSL implementations and tries again with a tlsv1.0 hello

But it fails too:

Secure Sockets Layer

TLSv1 Record Layer: Handshake Protocol: Client Hello

Content Type: Handshake (22)

Version: TLS 1.0 (0x0301)

Length: 179

Handshake Protocol: Client Hello

Handshake Type: Client Hello (1)

Length: 175

Version: TLS 1.0 (0x0301)

Random

gmt_unix_time: Jun 20, 2021 12:00:21.000000000 W. Europe Daylight Time

random_bytes: 16dd5ffc045e7960e77bcc8938f8c3e53b4b8c6288548143...

Session ID Length: 0

Cipher Suites Length: 30

Cipher Suites (15 suites)

Cipher Suite: Unknown (0x5600)

Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)

Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)

Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)

Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)

Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007)

Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011)

Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)

Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)

Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)

Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)

Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)

Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)

Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)

Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)

Compression Methods Length: 1

Compression Methods (1 method)

Compression Method: null (0)

Extensions Length: 104

Extension: server_name

Type: server_name (0x0000)

Length: 21

Data (21 bytes)

Extension: renegotiation_info

Type: renegotiation_info (0xff01)

Length: 1

Data (1 byte)

Extension: elliptic_curves

Type: elliptic_curves (0x000a)

Length: 8

Elliptic Curves Length: 6

Elliptic curves (3 curves)

Elliptic curve: secp256r1 (0x0017)

Elliptic curve: secp384r1 (0x0018)

Elliptic curve: secp521r1 (0x0019)

Extension: ec_point_formats

Type: ec_point_formats (0x000b)

Length: 2

EC point formats Length: 1

Elliptic curves point formats (1)

EC point format: uncompressed (0)

Extension: SessionTicket TLS

Type: SessionTicket TLS (0x0023)

Length: 0

Data (0 bytes)

Extension: Unknown 13172

Type: Unknown (0x3374)

Length: 0

Data (0 bytes)

Extension: Unknown 16

Type: Unknown (0x0010)

Length: 27

Data (27 bytes)

Extension: Unknown 30032

Type: Unknown (0x7550)

Length: 0

Data (0 bytes)

Extension: status_request

Type: status_request (0x0005)

Length: 5

Data (5 bytes)

Extension: Unknown 18

Type: Unknown (0x0012)

Length: 0

Data (0 bytes)

Response:

Secure Sockets Layer

TLSv1 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)

Content Type: Alert (21)

Version: TLS 1.0 (0x0301)

Length: 2

Alert Message

Level: Fatal (2)

Description: Handshake Failure (40)

Chrome actually tries again with a SSLv3 handshake (which is successful) but chrome will show the ERR_SSL_FALLBACK_BEYOND_MINIMUM_VERSION error.

BillP · ‎2021-07-01

Thanks for the in depth detail freddy. Let me pass this to our engineering team and I hope to report back early next week. We are on holiday here in the US the rest of the week (and I am all week) but I don't want to leave anyone hanging here and let you know we see this information.

Anonymous user · ‎2021-07-01

I tracked it down further for you.

I identified two issues with your SSL TLS implementation (version 6.2):

- Sending no protocol extension at all will result in a handshake failure. Technically this is an incorrect implementation but in practice every client should send at least send the Heartbeat Extension.

- Sending a protocol extension with length=0 will result in a handshake failure. Extensions with length=0 are perfectly valid and usually used to signal support for a specific feature. This is a mayor issue issue as almost every modern client (Microsoft being the exception) will send the SessionTicket TLS (0x0023) Extension which has no data (This breaks support for Chrome/Firefox).

I also noticed some oddness with regards to the TLS version negotiation. I believe that the server server will always downgrade to the next older version (If the client requests v1.2, server will downgrade to v1.1 and if the client requests v1.1, server will downgrade to v1.0, v1.0 is not downgraded further),

First generation devices running v3.7.4 apparently do not support TLS at all.

This is the python scapy code I used to craft the corresponding packets:

#!/usr/bin/python
"""
test for broken APC SSL implementation in AOS 6.2
Dependencies:
apt-get install python-scapy 
wget 'https://raw.githubusercontent.com/tintinweb/scapy-ssl_tls/master/src/scapy/layers/ssl_tls.py'
"""
import scapy
from scapy.all import *
import socket
from ssl_tls import *
target = ('ups44.example.com',443)
#proto = "TLS_1_2" # server will return tlsv1.1
#proto = "TLS_1_1" # server will return tlsv1.0
proto = "TLS_1_0" # server will return tlsv1.0
#proto = "SSL_3_0" # server will return sslv3.0
#proto = "SSL_2_0" # server will return sslv2.0
# create tcp socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(target)
# create TLS Handhsake / Client Hello packet
p = TLSRecord(version=proto)/TLSHandshake()/TLSClientHello(
        version=proto,
        compression_methods=None,
        cipher_suites=[0x0a,0xff],
        extensions=[
            # at least one extension is required, sending no extension at all results in a handshake failure
#            TLSExtension(type=0x0023), # valid SessionTicket Extension with length = 0 (uncommenting this line will result in a handshake failure)
#            TLSExtension(type=0xaaaa), # dummy extension with length = 0 (uncommenting this line will result in a handshake failure)
            TLSExtension(type=0xaaaa)/chr(0), # dummy extension with length = 1 (no problem)
            TLSExtension(type=0x000f)/chr(1), # Heartbeat Extension (no problem)
#            TLSExtension()/TLSServerNameIndication(server_names=[TLSServerName(data="example.com")]) # SNI Extension (no problem)
            ]
        )
p.show()
print "sending TLS payload"
s.sendall(str(p))
resp = s.recv(1024)
print "received, %s"%repr(resp)
SSL(resp).show()
s.close()

The extensions with length=0 issue can be also tested with openssl:

openssl s_client -connect ups44.example.com:443 -tls1 # won't work (SessionTicket Extension with length = 0 is sent)

openssl s_client -connect ups44.example.com:443 -tls1 -no_ticket # should work (only the Heartbeat Extension is sent)

BillP · ‎2021-07-01

freddy wrote:

First generation devices running v3.7.4 apparently do not support TLS at all.

Thank you for confirming my report. I will test the key length hack suggested earlier, but I am not optimistic about it.

Anonymous user · ‎2021-07-01

This realy needs to be fixed ASAP as Chrome have completely removed support now for that version and other browsers will (if not already) remove support in upcoming months

BillP · ‎2021-07-01

I'm on it - well escalating it - immediately today.

Edit - see my post here -> Re: PDU AP7900 Firmware SSL Incompatible With Latest Web Browsers .. not claiming that is the long term fix but for any users in a pinch while we actively investigate. Making sure for any 3.X.X firmware devices you're using a 1024 bit cert is important and could be one of the roadblocks some users are seeing (since those devices generate a 768 bit cert by default), though most of you here I don't think have that issue. It would also not apply to v6.X.X devices since they generate a 2048 bit cert by default.

BillP · ‎2021-07-01

Looking for a solution for this also.

We have temporarily created 1024 bit certs for our AP96xx and AP7xxx. We're using Firefox with the suggested setting changes and we are able to get in and manage these devices for now.

BillP · ‎2021-07-01

I hope to have some details Monday in a more formal format. I don't want to tell anyone anything that I don't know to be true or accurate as far as an actual resolution. As far as I know, we have identified the problem with Firefox specifically with v34 and NMC2 (AP9630/31/35) and it being related to TLS extensions that aren't being processed properly because they are of 0 length.

theotang_apc · ‎2021-07-01

try this...

certutil -setreg chain\minRSAPubKeyBitLength 512

http://support.microsoft.com/kb/2661254

mrfixit_apc · ‎2021-07-01

Angela,

Is there still no fix for this?

BillP · ‎2021-07-01

I've been working on hard on creating a knowledge base outlining details. I will have it online today and post it here on the forum. Reviewing the technical details in it and resolution(s) with all appropriate parties has taken a while (which is frustrating to me to help support everyone out there having a problem) but since I'll be on vacation the next two weeks, it NEEDS to be online today.

The short answer is, NMC2 devices like AP9630/31 will have a firmware fix but it is not available yet. I don't have a concrete date when it will be but as a customer advocate, I have voiced loudly that it needs to be SOON!!!! (more like yesterday) Today, I can still only offer using another interface on the NMC2 or enabling/allowing SSLv3 within your web browser if permitted within the particular organization.

For most older devices that are no longer being sold (and a select few that are), they do not have plans to go back and fix it at this time and SSLv3 must be used.

I personally feel helpless since I cannot fix anything myself but if any of this does not suffice or is not acceptable once the detailed kbase goes online, there will be an escalation procedure that I will help facilitate.

BillP · ‎2021-07-01

OK, here is the response on this on behalf of the different product lines:

Title: Unable to access my APC Network Management Card (NMC) enabled device via HTTPS (SSL/TLS)

http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA238115

FYI, for some users, they may also see this issue or only have this issue and get confused so I'll put it here as an FYI too as a concern due to recent browser updates released by Firefox, Chrome, IE, etc.

Title: Network Management Card 1 (NMC1) Information Bulletin: Effects of Microsoft Internet Explorer and other web browsers blocking key lengths less than 1024 bits

http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA162031

Anonymous user · ‎2021-07-01

Hi Angela,

Thanks for the info.

Shall we expect a firmware update for Network Management Card 1 (NMC1)?

Thanks.

BillP · ‎2021-07-01

The knowledge base article I posted above mentions that there will only be a firmware update for NMC2 devices as it stands now. AP9617/18/19 UPS network management cards definitely will not have any updates as they were replaced by AP9630/31 NMC2 in 2009 but NMC2 will have a firmware update. Enabling SSLv3 will be the only option for older NMC1 cards at this time.

TimDeleon · ‎2024-04-01

We had to change TLS to 1.2 only and broke HTTPS:// so we just delete the SSL cert and recreated.

Deleting the certificate via Web UI

Navigate to Configuration --> Network --> Web --> SSL Certificate
Select "remove", then "apply"
Reboot the NMC

Recreated after boot and resolved issue