Galaxy VS with NMC 4 - ssh/scp disabled

[Lincoln_Darran]

Just had 2 new Galaxy VS's installed, both with NMC 4 management cards.

https interface is working a fine.

No ssh access available.  config is reporting no key pair in place, so ssh disabled.

tried generating and uploading a new key pair from ssh-keygen -t rsa without success. complains about mismatch between key files and passphrase. doesn't allow a blank passphrase.

the option to reboot the nmc doesn't seem to exist in the new web interface, so rebooting to genereate a new key pait isn't available.  Both units are now up and running but remote.  Also geeting ddf errors because ssh is disabled which disables SCP.

[Teken]

You have several choices depending upon your level of access to the hardware.

In the NMC Webpage go to Control -> Network -> Reset / Reboot -> Reset All. If that doesn’t work you may consider the following action below.

NMC Format: Press and hold the reset button on the back of the NMC for 20+ seconds than release. The NMC will reset / erase and format the hardware to a factory state. 

You can than reenter all the network attributes that pertain to your environment. The only difference is you will enable all protocols & services - first!

NOTE: Ensure NTP is defined and the system clock reflects the correct date & time! Any DST / Time Zone obviously needs to set for your region. 

Once these basics are in place reboot the NMC. You can than use the webpage or CLI to make other changes as required.

[Lincoln_Darran]

Thanks @Teken,

Already tried your suggestions. Still the same issues NMC4 in a Galaxy VS.  First NMC4 I've seen. Now got 2 brand new ones exhibiting the same issues.

[Teken]

What firmware is on these NMC 4?!? When you go to Configuration -> Network -> Console Settings. SSH is enabled correct in the box?

Also when you go to Configuration -> Network -> Console -> SSH Host Key -> Remove.

Does the system prompt you to reboot the NMC?!?

[Lincoln_Darran]

Hi @Teken,

Version info attached.

SSH is enabled

When i choose remove, I get a confirmaiton popup, but no reboot dialogue.

[Teken]

To be clear you have deleted the SSH key and rebooted the NMC?!? Regardless of if the the reboot prompt is displayed?!?

Doing so should cause the NMC to self generate a new cypher key with a ten year certificate.

It goes without saying the web browser for that session must be closed. For good measure use another browser and clear all cache, cookies, history, etc.

Login using both HTTP vs HTTPS and report back what you see and observe. 

NOTE: When you delete the SSH key if there is no reboot prompt / option. Simply eject the NMC from the system for about ten seconds than reinstall.

The NMC is hot swappable . . . 

[Phlerm]

I have the exact same issue.  We have two 100kW Galaxy VS units.  As I type this, there is a Schneider Electric technician performing PM on the units.  One is finished and the other is in progress. 

The SSH issue exists in version 6.82.0.226 and 6.101.0.279. 

To address what Teken has said, I believe you are confusing a NMC3 and a NMC4.  There is no SSH key to delete.  The SSL key may be deleted, but it generates a 1-year self-signed cert for both versions of software (this is for web access only).  There is no way to SSH into either of these units (connection refused every time).  I was able to connect via SSH previously with some older version of software, but I have been unable since the upgrade a year or two ago.  There is no way to initiate a reboot remotely of the NMC.  There is no removable card.  NMC4 is the built-in card for this UPS.  The only way to reboot is to push the reset button located on the physical UPS where the NMC is located. 

I believe the SSH Host Key that you are speaking about is for authenticating via that key rather than username/password.  The SSH client holds a key that it authenticates with.

So, I guess the question is: Should we be able to connect to a NMC4 via SSH using username/password or do we need to authenticate via a shared key?

[Lincoln_Darran]

Thanks @Phlerm/@Teken 

Thought it was just me.  I haven't had a chance to stand in front of either of the new Galaxy's yet, so haven't been able to confirm if the NMC can be popped out, my gut feeling is "no".

For clarity, I've attached a screenshot of the SSH User Host Key Configuration screen.

Also noticed that neither of them launch to the web interface properly within DCE.  new window opens, but gets stuck on the Schneider logo.  Have to launch in an external browser window to view.

[Lincoln_Darran]

Just checked the hardware installation manual.  NMC 4 is definitely not hot swappable.

Manual scrape below...

[Teken]

If the NMC is embedded as the other 

member noted it can’t be removed / ejected.

Pressing the reset button would replicate the same behaviour. Also, just to be clear does this hardware have two Ethernet ports?!?

One for the NMC and another for the Smart Connect (Green Cloud Icon)?

[Phlerm]

I have a solution.  For whatever reason, the NMC4 will not generate an SSH key.  It may be deleted, but there is no way for the NMC4 to generate a new one on its own.  However, you can generate your own and upload the private and public keys to the NMC4 via the web interface.  This fixed it for us. 

However, this seems really silly.  Why in the world can't the NMC4 generate an SSH key on its own?  It seems like a bunch of hoops to jump through for no reason. 

[Phlerm]

Two more things I forgot to mention. 

- For those that do not know, there isn't a way to reboot the NMC4 via the web interface. 

- After updates to the NMC4 during the previous Preventative Maintenances (both times requiring a reboot), the NMC4 did not generate SSH keys.  I cannot see how this is desired behavior.  Can anyone from Schneider Electric jump in on this.  Why do we have to generate our own SSH keys and upload them to the NMC4?  Why doesn't the NMC4 have the ability to generate SSH keys via the web interface?  Why can't we reboot the NMC4 via the web interface?

[Lincoln_Darran]

@Phlerm @Teken Finally got back on site after some illness.

NMC only has one network port.

@Phlerm - I tried manually gereating a key set and the NMC whined about it, and kicked it out.  Any chance of a crib sheet on how you generated yours.

I can confirm that a physical reboot using the reset switch does not work to resolve this issue.

I've also discovered that the NMC4 console interface, isn't really a console interface, it's another network interface that lets you access the https engine.  I was hopeing for a quick local login over the console might offer a solution. Very misleading labelling it console.

Please can Schneider fix this as a matter of urgency.  We need a reboot option, and we need ssh/scp to work.

Arghhhh!

[Lincoln_Darran]

Anybody from the Galaxy or NMC team feel like commenting?

[Teken]

Please contact APC Technical Support to assist in this matter. Doing so will initiate a formal request for investigation from tier 2 support. 

Let us know the outcome good or bad. 👍

[Lincoln_Darran]

@Teken @Phlerm 

Have now got two fixed Galaxys.

Opened a support case, and got sent a new firmware.

So if you're still having the same issues open a support case.

Still no reboot option, but SSH/SCP now works a treat.

Thanks for all your help

[calmi]

Would you be willing to give instructions on how you generated your ssh keys?  I've tried myself with ssh-keygen -t rsa and the web interface just errors out on it without any explanation.

[Teken]

Appreciate the follow up and letting us all know the final outcome. Has this firmware been uploaded for the public?

Lastly, why is there still no method to reboot the NMC?!? 🤦‍♂️

[Lincoln_Darran]

Hi @Teken 

I ended up with firmware 6.102.0.  Can't see it as available to download.

There's still no reboot NMC option.  It is noticeable that the NMC 4 is a completely different OS build to the NMC 3, so that's maybe why.  

HTH

[Lincoln_Darran]

Hi @calmi 

Once I got the new firmware installed, the SSH key problem seems to have become irrelevent.  SSH is now working.  Previously I was getting "refused" when I hit up a SSH connection.

HTH

[Teken]

Thank You Sir! 👍

[calmi]

Found a procedure that worked for me since APC would not give me a newer firmware than 6.101
#generate a ssh key.  Passphrase is required
openssl genrsa -des3 -out private.pem 2048

#extract the public key
openssl rsa -in private.pem -outform PEM -pubout -out public.pem

Upload both the private.pem and the public.pem to the NMC interface.  Supply the Passphrase you used above and apply.  A new SSH fingerprint should then show once the upload is done.