SSH from Fedora 41 to AP9631 fails: ssh_dispatch_run_fatal: error in libcrypto

[aswen]

Hi,

SSH to my AP9631 NMC fails with this error:

$ ssh -v apc-ups.my.net.work
OpenSSH_9.9p1, OpenSSL 3.2.2 4 Jun 2024
debug1: Reading configuration data /home/alex/.ssh/config
debug1: /home/alex/.ssh/config line 31: Applying options for apc-ups.my.net.work
debug1: /home/alex/.ssh/config line 108: Applying options for *.my.net.work
debug1: /home/alex/.ssh/config line 172: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config line 57: Applying options for *
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/alex/.ssh/config
debug1: /home/alex/.ssh/config line 31: Applying options for apc-ups.my.net.work
debug1: /home/alex/.ssh/config line 108: Applying options for *.my.net.work
debug1: /home/alex/.ssh/config line 172: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config line 57: Applying options for *
debug1: Connecting to apc-ups.my.net.work [172.17.255.240] port 22.
debug1: Connection established.
debug1: identity file /home/alex/.ssh/id_rsa type -1
(...)
debug1: Local version string SSH-2.0-OpenSSH_9.9
debug1: Remote protocol version 2.0, remote software version cryptlib
debug1: compat_banner: no match: cryptlib
debug1: Authenticating to apc-ups.my.net.work:22 as 'apc'
debug1: load_hostkeys: fopen /home/alex/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:5gSMQHqToW5REDACTEDt+vms
debug1: load_hostkeys: fopen /home/alex/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'apc-ups.barchem.de-eekhoorn.eu' is known and matches the RSA host key.
debug1: Found key in /home/alex/.ssh/known_hosts:54
ssh_dispatch_run_fatal: Connection to 172.17.255.240 port 22: error in libcrypto

My SSH Config:

### APC UPS ###
host apc-ups.my.net.work apc-ups 172.17.255.240
  Hostname apc-ups.my.net.work
  user apc
  HostKeyAlgorithms +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa
  PreferredAuthentications keyboard-interactive,password
  KexAlgorithms +diffie-hellman-group1-sha1
  Ciphers aes256-ctr
  MACs hmac-sha2-256
  PasswordAuthentication yes
  PubkeyAuthentication no
  VerifyHostKeyDNS no

#### DEFAULTS ####
Host *
  LogLevel QUIET
  ForwardAgent no
  SendEnv LANG LC_*
  HashKnownHosts no
  Port 22
  Protocol 2
  ServerAliveInterval 15
  ServerAliveCountMax 5
  PubkeyAuthentication yes
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  ForwardX11 no
  ForwardX11Trusted no
  HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa
  Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
  KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
  VerifyHostKeyDNS yes

Client

"Fedora Linux 41 (Workstation Edition)"
6.12.10-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jan 17 18:05:24 UTC 2025 x86_64 GNU/Linux
openssh.x86_64 9.9p1-1.fc41 updates
openssl.x86_64 1:3.2.2-11.fc41 updates

NMC

* Model: AP9631 hw revision 5

* Application module: v7.1.8

* AOS v7.1.8

* Boot monitor v1.0.9

I don't really know what's wrong?

[KarimEissa · Accepted solution]

Hello,

If you are using RedHat Linux or related distros ,you may have issues connecting to NMC's via SSH

Redhat has implemented changes to their crypto policies which can cause the SSH connection to fail with the following error message:
"ssh_dispatch_run_fatal: Connection to xxx.xxx.xxx.xxx port xx: error in libcrypto"

you can view the current policies by running the following command: "update-crypto-policies --show"
If the return output is "DEFAULT", you will need to enable legacy options.

This is done by running "update-crypto-policies --set LEGACY" with elevated privileges.

Once done, you will need to restart the linux machine in order for the policy changes to become active. 

Regards,

Karim

[aswen]

Thanks @KarimEissa for showing me this work around. I'm using SSH to login to many systems and need to do that as secure as possible. Switching my entire SSH config back to LEGACY is, from that perspective, not the safest solution.
How can I configure the host entry of my APC card in `.ssh/config` so that I can login to the NMC without lowering the standards for all my connections? (I know that SSH tries to negotiate the securest option, but it's not without reason that some ciphers and kex algorithms are removed from DEFAULT).

[KarimEissa]

Hello,

You may need to refer to the below link for SSH connections to NMC 

https://community.se.com/t5/APC-UPS-Data-Center-Enterprise/Scripting-SSH-connections-to-NMC-s-take-2...

 you may need to adjust the list of SSL/TLS ciphers in use for NMC web access on the NMC, to comply with local security policies, changes in browser compatibility, or to reflect ever-changing best practices.

Via the NMC command line:

Issue the “cipher” command to show the current enabled set, or “cipher help” for usage notes.
eg;

apc>cipher help
Usage: cipher --  Configuration Options
    Note: The minimal protocol setting is not considered when showing
           the available ciphers.

    cipher [-aes (enable | disable)] (AES)
           [-dh (enable | disable)] (DH)
           [-rsake (enable | disable)] (RSA Key Exchange)
           [-rsaau (enable | disable)] (RSA Authentication)
           [-sha1 (enable | disable)] (SHA)
           [-sha2 (enable | disable)] (SHA256)
           [-ecdhe (enable | disable)] (ECDHE)

Note:
Prior to 6.8.0, each option (eg -rc4) toggled the current state; these are now explicitly deterministic.
Reboot to commit changes.
 
Example:
List current settings, showing that all available are enabled (as default):
 

>cipher
E000: Success
Key Exchange Algorithms
-----------------------

        DH                   enabled
        RSA Key Exchange     enabled

Authentication Algorithms
-------------------------
(Warning: disabling the only algorithm in category
          will block all SSL/TLS sessions)

        RSA Authentication   enabled

Block Cipher Algorithms
-----------------------

        triple-DES           enabled
        RC4                  enabled
        AES                  enabled

MAC Algorithms
--------------

        MD5                  enabled
        SHA                  enabled
        SHA256               enabled

[...]
Disable RC4 cipher and RSA key-exchange:
 

>cipher -rc4 disable
E002: Success

>cipher -rsake disable
E002: Success

List new settings, confirming expected changes:
 

>cipher
E000: Success
Key Exchange Algorithms
-----------------------

        DH                   enabled
        RSA Key Exchange     disabled

Authentication Algorithms
-------------------------
(Warning: disabling the only algorithm in category
          will block all SSL/TLS sessions)

        RSA Authentication   enabled

Block Cipher Algorithms
-----------------------

        triple-DES           enabled
        RC4                  disabled
        AES                  enabled

MAC Algorithms
--------------

        MD5                  enabled
        SHA                  enabled
        SHA256               enabled

[...]

Using INI files (eg, for mass configuration):
 

[CryptographicAlgorithms]
;Warning: Changing these values can affect system access.
TripleDES=enabled
RC4=disabled
AES=enabled
DH=enabled
RSA_KE=disabled
RSA_Auth=enabled
MD5=enabled
SHA=enabled
SHA256=enabled

Using the web interface:

These settings are not yet exposed via the web UI.

Troubleshooting:

Be aware that disabling ciphers may affect browser compatibility; SSL/TLS will be unusable to the user unless their browser and the NMC have at least one cipher suite in common. Browser errors such as "ssl_error_no_cypher_overlap" or "err_ssl_version_or_cipher_mismatch" would indicate such an incompatibility.

Regards,

Karim

[Shaun]

Hi @aswen,

This can't really be solved in the ssh config - this is the operating principle for Red Hat's crypto policies.  Your ssh configuration is correct, but when it attempts to use a routine that's disallowed by the current policy, libcrypto throws an exception.

This means we can't solve this on a per-host basis, the system-wide crypto policy is the blocker.

The least invasive configuration I've found to work is

update-crypto-policies --set DEFAULT:SHA1

Instead of setting the whole policy to LEGACY, this will leave us on DEFAULT except with sha1 re-enabled so the group1-sha1 kex is no longer a policy violation.  It is still a step down from DEFAULT, however - the only real alternative I've found to that is to have an alternate toolchain in a container, so the containerized ssh isn't using the host's libcrypto.