SSH Implementation Issues with SmartUPS Network Management Card 2 (AP9630)

[netadmin_at_princessauto.com_apc]

  Due to limitations in mass remote configuration/management options for UPS NMCs, we explored trying to script commands to be sent to an AP9630's ssh console. We have specifically tried with OpenSSH from a Linux host and with PuTTY + PLink from a Windows host, and have discovered the following -- the NMC does not properly implement SSH. If we attempt to send SSH command line commands from either OpenSSH or Putty + Plink, the NMC logs show the following:

02/04/2014  11:24:26  System: SSH/SCP: File transfer failed.

02/04/2014  11:24:25  System: SSH/SCP: File transfer started.

  The APC SSH implementation is incorrectly interpreting SSH commands to be SCP file copies.

  Some specific command line examples:

[OpenSSH]

ssh -v upsadminaccount@ups.domain.internal eventlog

The relevant logging shows:

debug1: Authentication succeeded (password).

Authenticated to ups.domain.internal ([10.1.2.3]:22).

debug1: channel 0: new [client-session]

debug1: Entering interactive session.

debug1: Sending environment.

debug1: Sending env LANG = en_US.UTF-8

debug1: Sending command: eventlog

debug1: channel 0: free: client-session, nchannels 1

Transferred: sent 1624, received 1528 bytes, in 0.2 seconds

Bytes per second: sent 8925.9, received 8398.3

debug1: Exit status -1

[Putty + Plink]

plink -v -l upsadminaccount -pw upspassword ups.domain.internal eventlog

Looking up host "ups.domain.internal"

Connecting to 10.1.2.3 port 22

Server version: SSH-2.0-cryptlib

Using SSH protocol version 2

We claim version: SSH-2.0-PuTTY_Release_0.63

Doing Diffie-Hellman group exchange

Doing Diffie-Hellman key exchange with hash SHA-1

Host key fingerprint is:

ssh-rsa 2048

Initialised AES-256 CBC client->server encryption

Initialised HMAC-SHA1 client->server MAC algorithm

Initialised AES-256 CBC server->client encryption

Initialised HMAC-SHA1 server->client MAC algorithm

Using username "upsadminaccount".

Sent password

Access granted

Opening session as main channel

Opened main channel

Started a shell/command

Disconnected: All channels closed

The same entries about SSH/SCP file copies failing show up in the NMC event log.

Specific to plink on Windows, using "echo ftp -S disable | plink ..." to pipe text input into the SSH session will work to some extent for a single command. Chaining "echo" commands with "&&" has limited success, as does redirecting input from a text file, e.g. "plink ... < commands.txt". In no case does piping or redirecting ever seem to work for the reboot command, which requires a confirmation "YES". The reboot command can be entered, but the confirmation "YES" never works. That being said, none of the text redirection should be necessary as ssh natively support passing commands (typically through command line, though plink supports this from a file) though the NMC implementation does not.

Sending a reboot command from SSH console becomes necessary as either StruxureWare or the NMCs themselves cause a problem whereby the NMC thinks someone is logged into the web interface and won't allow anyone else to log on that way. Even well after the regular session timeout should have expired and when no one in fact is logged onto the web interface. Rebooting the NMC will at least clear that status and re-allow web logins once the reboot is complete.

Specific to the faulty SSH implementation, Putty support has actually confirmed that this appears to be the case based on logs that we provided to them.

As a last note, using telnet is not a "workaround" for this. It may be scriptable, however telnet is not secure. It is a security concern to send credentials in plain text to a telnet server, particularly when the credentials in question are tied into RADIUS and are actually accounts in a central account repository.

[BillP]

Hi,

Regarding your first sentence - what problems/limitations are you finding with mass configuration that you chose to move to SSH scripting? Maybe I can help there.

Can you confirm what version of firmware is on your AP9630? Based on what you said about having a problem with receiving messages about somebody else logged in, I'll assume its version v5.X.X which is not our latest. Based on my educated guess using what you described to assume you're on v5.X.X, then I'd suggest trying this newer firmware on one of your cards just to see if any behavior changes since that is what we are currently working on firmware platform wise. This firmware is available for sumx and sy UPS applications. We'd want to make sure you're using one of those applications. You can find this out via the about command in CLI or via the web.

But, when you upgrade, there are a lot of major changes in functionality (and admittedly a few considerations/issues). It is best to review this article first->Things To Consider When Upgrading or Downgrading a Network Management Card 2 (NMC2) Device between v... v6.X.X also has some limitations with StruxureWare Data Center Expert and maybe that is why you have also not upgraded? Or relates to the limitations I asked to clarify.

I do know we have identified an issue with OpenSSH v5 on v5.X.X that we addressed or will address (I can't say I specifically tested or verified it personally so that is why I word it this way) in v6.X.X - what I witnessed was it in relation to before was for transferring files via SCP. It is outlined here (https://bugzilla.mindrot.org/show_bug.cgi?id=1814) - are you using OpenSSH v5?

Furthermore, it's hard to support every single client with every single library. I do know we work with PuTTy by itself and I do know of that OpenSSH v5 bug I noted. Beyond that, we'll have to work through this if you're willing to with me. I am not sure by what you've described so far if we have a "faulty" implementation so I am asking all of these questions to help verify from our side.

Another thing I am noticing is that you're doing this (for example) - ssh -v upsadminaccount@ups.domain.internal eventlog where eventlog is the command for the NMC along with the login. I don't know if we have ever tested like that, nevermind with plink - putting everything on one line, including the command, (and if we should accept it) but does any of this work if you do everything separately (ssh to the IP as one step, enter the username as the next, enter the password, as the next, and then enter the command rather than doing it in one long command)? At least for testing? Alternatively, what about downloading the event.txt and parsing it locally vs using the eventlog command?

[netadmin_at_princessauto.com_apc]

Hello,

  The limitations we have encountered with mass configuration apply to most settings. Certainly RADIUS and SNMPv3 via StruxureWare. Even to add some secure passwords/secrets/etc "config.ini" and re-uploading, only some settings actually update. To be specific, I don't mean pending a reboot, they don't update at all. This seemed like it would be a good way to go, but after so much inconsistency in which settings worked and which didn't, we couldn't justify continuing to use this. In particular, it would be much better, if using this method, to accept INI files that only contain the settings changes to be made instead of downloading an entire config file, modifying it, and then re-uploading the entire modified config file.

  We have StruxureWare Data Center Expert 7.2.1 which really should be all that we need to deploy mass settings. Despite using a pre-configured NMC (AP9630, as all of current target ones are) to pull settings from in order to create a template for enabling SSH as the console protocol, we cannot get NMCs to change from default of telnet to SSH. Having tried all variations of "enabled, SSH v1/v2, SSH v1, SSH v2", none cause an actual update from telnet to SSH.

  In general, StruxureWare is not friendly to what we need to do. Our UPS units were initially deployed with fairly default NMC settings. snmpv1 is not particularly desirable, so when we wanted to change our existing discovery definitions from snmpv1 to snmpv3, we noticed that it wasn't an edit option. We thought it would be best to mass import settings anyway, but while StruxureWare supports import of snmpv1 discovery definitions, it does not appear to support import of snmpv3 discovery definitions. Even being able to clone existing snmpv3 discoveries such that we only have to change the IP range would have been a lot easier than changing the auth and privacy protocols from default for every new discovery and from also having to enter the same auth and privacy keys in each.

  RADIUS actually has been fully deployed manually (one unit at a time) as has SSH. RADIUS was a large pain to deploy that required access to the web interface. The settings within StruxureWare are not friendly to a mass amount of units. We have RADIUS providing a central credentials system allowing a single set of credentials to manage all of the UPS units via NMC. However, instead of offering (that we know) any way to set the same "Device Launch Settings" in StruxureWare for all devices or a selection of devices, we had to change every single one individually to HTTPS and provide the RADIUS username and password.

  In StruxureWare, even though all devices are now snmpv3-enabled and we have disabled snmpv1, "Device Scan Settings" shows most devices with Protocol snmpv1, even though we removed all previous devices and created new snmpv3 discoveries to rediscover them.

  With respect to updating our NMC firmware, we had a separate support case for low load values not showing up (i.e. showing as 0.0% load), and as part of that case we upgraded one to the latest firmware available. A couple of comments on that firmware:

1. The user interface is less user friendly. Particularly via StruxureWare where the cascading menus can end up out of the visible window and require scrolling. When doing things manually, at least the earlier versions of the NMC gui were quick to operate.

2. Functionality has changed. We were going about renaming the super user account from "apc" to a custom name on our "older" firmware units. In the newer firmware, there is no longer an ability to rename the super user account. The command line commands have changed as well (the user management command has different parameters now). No doubt the snmp templates from StruxureWare are rendered invalid because of changes like this.

  The last change that we tried to deploy has rendered our units unmanageable via http/https. We now have to reboot the management interface on every single one in order to allow them to be even manually configured via GUI. If we attempt to deploy another change from StruxureWare this may happen again. At least if SSH command support worked as expected we could automate the reboots of the management intefaces, but providing commands does not work. The "eventlog" command was used for reference -- we do not actually need to collect event logs this way. We really need to reboot the units at present, and then deploy some more settings via SSH. Another issue we have is with the reboot command and the "YES" confirmation that is required. Not allowing a "force" or "confirm" parameter/switch means that someone has to answer with "YES" and this second prompt is harder to script for. If a parameter was accepted, such as "reboot -FORCE" then there would be no user mistake as no one is going to accidentally put in a "-FORCE" command line switch.

  Most of our NMCs are on the following firmware:

American Power Conversion               Network Management Card AOS      v5.1.7

(c) Copyright 2010 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v5.1.7

  We have one using the following:

American Power Conversion               Network Management Card AOS      v6.0.6

(c) Copyright 2012 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v6.0.6

  As mentioned in a previous post, the certificate options are very difficult to deal with. It should as simple as making a request in StruxureWare to gather CSRs from all managed units, getting those units signed, and then deploying them all in automated fashion from StruxureWare. Instead it involves another utility, multiple steps, and manual installation on every NMC. Wildcard certificates are not appropriate for our implementation. Some other customers have noted that StruxureWare accesses managed devices by IP address even when a DNS name has been provided -- this causes a certificate mismatch as there is no Subject Alternative Name defined for the IP. IPs are subject to change -- there is good reason why most certificates are issued by DNS name without having their IP addresses as Subject Alternative Names. StruxureWare should be browsing to the HTTPS interfaces by the name or IP provided. If DNS name, then that is all it should use and the user should not receive certificate warnings. To that end it is also strange that StruxureWare wants to store a copy of each new certificate it comes across -- we have StruxureWare installed on a domain-joined server that trusts our PKI enterprise root CA certificate. Each UPS has a PKI-issued certificate, so there are no trust issues. StruxureWare shouldn't need to store a copy of each device certificate if the devices are trusted due to the root CA certificate being trusted on the server/computer.

  With respect to the SSH implementation, the SSH's ability to execute commands, execute subsystems, and run shells is detailed in section 6.5 of RFC 4254 (http://tools.ietf.org/html/rfc4254#page-13). Specifying the commands to execute at the end of the SSH command line is the accepted way to do so, and this syntax is accepted by both OpenSSH and PuTTY.

  Plink allows for PuTTY to accept a series of scripted commands from a text file (-m ) or a command/commands as part of the command line. We use PuTTY for such tasks, OpenSSH was included to show a verbose dump from a different client program.

  The ability for StruxureWare itself to send commands via SSH to the NMC shell would be far superior to the INI file transfer method. It could return accurate errors messages and even command output if desired. For now we just want to be able to script SSH ourselves to be able to mass configure / reboot NMCs reliably and with minimal effort.

[BillP]

StruxureWare Data Center Expert (DCE) configures the network management cards using FTP or SCP and sending the new configurations by sending them to the device as a new config.ini. Whatever settings are available to be set using the INI, DCE should be able to configure. If there are specific settings, this may be a limitation of the INI or potentially, the system has not been updated to match the changes in the INI. WIth 6.0.6 firmware. Some things such as passwords are no longer in the config.ini. As such DCE currently can not change them. A new configuration option needs to be created in the system and I have entered an enhancement request for just this issue just this fall.

On older units, the config.ini was the easiest way to configure the devices and remained a constant throughout different firmware revisions up until NMC 6.x. Telnet menus have changed over the years and it would have been more dificult to validate firmware revision and based on that, send a different series of commands to each device. Since the card itself is changing, DCE will need to change with it going forward.

To enable SCP on a card for the file transfer, on the AP9617 cards, this was available for configuration through the config.ini. I checked on an AP9630 with 6.0.6 firmware and this did not appear to be an option in DCE. Whatever is not available in the config.ini does not show up when configuring a device in DCE. With an AP9630 with 5.1.7 firmware, the options in DCE were the same. With 6.0.6 firmware however, there is no option in DCE. I believed this is another option that has been removed from the config.ini but upon a deeper look, it seems that the format for this option has changed in the ini. This gives the same result, DCE can't configure this device in the same way it can another device.

As for changing from SNMP version 1 to 3, the best option is to discover using version 3 to begin with. I have seen a few issues in changing over after discovery but it can be done. K-base FA158442 talks about this.

For device launch settings, you can multi-select using shift click or control click and select device launch settings. I tested this on 3 units at once and saw no issue.

For certs, I know of no plans to sign them for SNMP devices. This does sound like an interesting option and I can put in an enhancement request but it would be quire some time before something like this could be looked into and adding such a request is no promise that the enhancement will ever actually be added to the system.

[BillP]

I asked smarchet to comment on the StruxureWare stuff since that is outside of my realm but I'll comment inline on what I do know. My expertise is with the Network Management Cards themselves.

  The limitations we have encountered with mass configuration apply to most settings. Certainly RADIUS and SNMPv3 via StruxureWare. Even to add some secure passwords/secrets/etc "config.ini" and re-uploading, only some settings actually update. To be specific, I don't mean pending a reboot, they don't update at all. This seemed like it would be a good way to go, but after so much inconsistency in which settings worked and which didn't, we couldn't justify continuing to use this. In particular, it would be much better, if using this method, to accept INI files that only contain the settings changes to be made instead of downloading an entire config file, modifying it, and then re-uploading the entire modified config file.

I know for sure, outside of StruxureWare, you can upload pieces of a config.ini. I think StruxureWare always uploads the full file. If you're doing it outside of StruxureWare, then you only need at least the section name (the thing enclosed in square brackets, like [NetworkTCPIP]) and then at least one value under it and it will work. This could help you narrow down what's working and what's not too and I could always help you with this if you ever wanted to try again.

We have StruxureWare Data Center Expert 7.2.1 which really should be all that we need to deploy mass settings. Despite using a pre-configured NMC (AP9630, as all of current target ones are) to pull settings from in order to create a template for enabling SSH as the console protocol, we cannot get NMCs to change from default of telnet to SSH. Having tried all variations of "enabled, SSH v1/v2, SSH v1, SSH v2", none cause an actual update from telnet to SSH.

I agree - that is one of the main selling points is mass configuration. That is why I asked smarchet to look at it. v5.1.7 allows Telnet OR SSH and requires a reboot after. v6.X.X allows both telnet and SSH now to be on as they are independent since the introduction of multiple users now (if you have users that need telnet and SSH perhaps). I have a feeling that perhaps the wrong keywords may be used because I can say for sure between v5 and v6 firmware, the keywords/values have changed a little and also the older AP9617/18/19 had different keywords. Maybe we can revisit this again if you wanted to.

In general, StruxureWare is not friendly to what we need to do. Our UPS units were initially deployed with fairly default NMC settings. snmpv1 is not particularly desirable, so when we wanted to change our existing discovery definitions from snmpv1 to snmpv3, we noticed that it wasn't an edit option. We thought it would be best to mass import settings anyway, but while StruxureWare supports import of snmpv1 discovery definitions, it does not appear to support import of snmpv3 discovery definitions. Even being able to clone existing snmpv3 discoveries such that we only have to change the IP range would have been a lot easier than changing the auth and privacy protocols from default for every new discovery and from also having to enter the same auth and privacy keys in each.

I think smarchet commented a little bit on this..

RADIUS actually has been fully deployed manually (one unit at a time) as has SSH. RADIUS was a large pain to deploy that required access to the web interface. The settings within StruxureWare are not friendly to a mass amount of units. We have RADIUS providing a central credentials system allowing a single set of credentials to manage all of the UPS units via NMC. However, instead of offering (that we know) any way to set the same "Device Launch Settings" in StruxureWare for all devices or a selection of devices, we had to change every single one individually to HTTPS and provide the RADIUS username and password.

I think smarchet commented a little bit on this too..

In StruxureWare, even though all devices are now snmpv3-enabled and we have disabled snmpv1, "Device Scan Settings" shows most devices with Protocol snmpv1, even though we removed all previous devices and created new snmpv3 discoveries to rediscover them.

I also saw a comment on this from smarchet.

With respect to updating our NMC firmware, we had a separate support case for low load values not showing up (i.e. showing as 0.0% load), and as part of that case we upgraded one to the latest firmware available. A couple of comments on that firmware:

1. The user interface is less user friendly. Particularly via StruxureWare where the cascading menus can end up out of the visible window and require scrolling. When doing things manually, at least the earlier versions of the NMC gui were quick to operate.

2. Functionality has changed. We were going about renaming the super user account from "apc" to a custom name on our "older" firmware units. In the newer firmware, there is no longer an ability to rename the super user account. The command line commands have changed as well (the user management command has different parameters now). No doubt the snmp templates from StruxureWare are rendered invalid because of changes like this.

The last change that we tried to deploy has rendered our units unmanageable via http/https. We now have to reboot the management interface on every single one in order to allow them to be even manually configured via GUI. If we attempt to deploy another change from StruxureWare this may happen again. At least if SSH command support worked as expected we could automate the reboots of the management intefaces, but providing commands does not work. The "eventlog" command was used for reference -- we do not actually need to collect event logs this way. We really need to reboot the units at present, and then deploy some more settings via SSH. Another issue we have is with the reboot command and the "YES" confirmation that is required. Not allowing a "force" or "confirm" parameter/switch means that someone has to answer with "YES" and this second prompt is harder to script for. If a parameter was accepted, such as "reboot -FORCE" then there would be no user mistake as no one is going to accidentally put in a "-FORCE" command line switch. 

Most of our NMCs are on the following firmware:

American Power Conversion               Network Management Card AOS      v5.1.7

(c) Copyright 2010 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v5.1.7

We have one using the following:

American Power Conversion               Network Management Card AOS      v6.0.6

(c) Copyright 2012 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v6.0.6

Well, on the GUI for v6.X.X, we are working on making that better - by implementing bread crumbs, and potentially other options, especially to deal with the CSS menus. Super user is also new - we only have administrator, device, and read only user on v5.X.X. To emulate some Linux standards, we allow the user to disable the apc super user account  but it cannot be re-named or disabled, similar to root in Linux. And yes, because of multiple user names now and the way users are handled, the user command is different as this is a major overall change to the system. StruxureWare has not yet addressed the overall config.ini changes in v6 firmware, including this, yet in v7.2.1.

I am a bit unclear on the HTTP/HTTPS issue you're experiencing and what you mean by a change you made that rendered them inaccessible so I don't know if you can elaborate. I also don't know if SNMP is a viable option if we cannot get the SSH working properly just yet.

As mentioned in a previous post, the certificate options are very difficult to deal with. It should as simple as making a request in StruxureWare to gather CSRs from all managed units, getting those units signed, and then deploying them all in automated fashion from StruxureWare. Instead it involves another utility, multiple steps, and manual installation on every NMC. Wildcard certificates are not appropriate for our implementation. Some other customers have noted that StruxureWare accesses managed devices by IP address even when a DNS name has been provided -- this causes a certificate mismatch as there is no Subject Alternative Name defined for the IP. IPs are subject to change -- there is good reason why most certificates are issued by DNS name without having their IP addresses as Subject Alternative Names. StruxureWare should be browsing to the HTTPS interfaces by the name or IP provided. If DNS name, then that is all it should use and the user should not receive certificate warnings. To that end it is also strange that StruxureWare wants to store a copy of each new certificate it comes across -- we have StruxureWare installed on a domain-joined server that trusts our PKI enterprise root CA certificate. Each UPS has a PKI-issued certificate, so there are no trust issues. StruxureWare shouldn't need to store a copy of each device certificate if the devices are trusted due to the root CA certificate being trusted on the server/computer.

I think smarchet covered some of this as well as a potential enhancement.

With respect to the SSH implementation, the SSH's ability to execute commands, execute subsystems, and run shells is detailed in section 6.5 of RFC 4254 (http://tools.ietf.org/html/rfc4254#page-13). Specifying the commands to execute at the end of the SSH command line is the accepted way to do so, and this syntax is accepted by both OpenSSH and PuTTY.

I will have to research and discuss this more internally as per your reference to the RFC.

Plink allows for PuTTY to accept a series of scripted commands from a text file (-m ) or a command/commands as part of the command line. We use PuTTY for such tasks, OpenSSH was included to show a verbose dump from a different client program.

OK

The ability for StruxureWare itself to send commands via SSH to the NMC shell would be far superior to the INI file transfer method. It could return accurate errors messages and even command output if desired. For now we just want to be able to script SSH ourselves to be able to mass configure / reboot NMCs reliably and with minimal effort.

I like this idea too but as to what smarchet said, commands changes, are added, and I guess they chose to work with config.ini instead but it seems like there are loopholes with that too.

[voidstar_apc]

Angela N. wrote:

To emulate some Linux standards, we allow the user to disable the apc super user account  but it cannot be re-named or disabled, similar to root in Linux.

FYI, root can be renamed and/or disabled in Linux. Thankfully, AOS606 does give the option to disable the superuser account, so folks who want to rename it can instead add an administrator account with whatever name they like and disable the superuser account. It's too bad it can't be renamed directly.

The ability for StruxureWare itself to send commands via SSH to the NMC shell would be far superior to the INI file transfer method. It could return accurate errors messages and even command output if desired. For now we just want to be able to script SSH ourselves to be able to mass configure / reboot NMCs reliably and with minimal effort.

I like this idea too but as to what smarchet said, commands changes, are added, and I guess they chose to work with config.ini instead but it seems like there are loopholes with that too.

Agreed. I've always felt the INI interface was the best option for mass configuration because it covers all the settings (well, up until the multi-user business) and it remained constant over time at the expense of user friendliness. I'd be interested to learn what isn't being reliably set.

[BillP]

Thanks voidstar - I mis-spoke. I meant to indicate super user cannot be renamed, but it can be disabled which is what happens to root. I should know that after all of the trainings and documents I make saying that! I was not aware root could be renamed though. I am not an avid Linux user

[netadmin_at_princessauto.com_apc]

All but 1 of our NMCs runs with v5.1.7 and this has shown to have many problems with the CONFIG.INI method where settings just don't "take". As noted, converting from telnet to SSH (preferably v2) being a noted issue. In v5.1.7 SSH vs telnet is an either or. It just won't switch.

To what Angela had mentioned, can it be confirmed that NMCs will accept partial INIs? Is there a master copy of the configuration stored elsewhere? The concern would be overwriting the existing CONFIG.INI with a partial one and causing problems for the NMC. Be that as it may, if we can't get settings to "take" then it's rather a moot point whether partial INIs are accepted or not.

We will further investigate converting units over to snmpv3 as described in the KB article. Not being able to bulk import snmpv3 discovery settings is still a feature that would be desirable.

I have verified what you mentioned about the multi-select -- it seemed unintuitive given how mass settings were configured for device file transfer settings via global menu option, but the important thing is that it works. We have already manually (i.e. individually) converted all of ours to the correct credentials, however this will be handy if and when the credentials need to be updated.

Everything I communicated about certificates pertained to the HTTPS web interface. Since we will be entering credentials for logon, we want these communications to be SSL-encrypted and with a (Microsoft) PKI-issued certificate. I'm not sure what was meant by signing certs for SNMP.

Tools for managing other pools of devices (from other vendors) allow for easy discovery of all devices and then mass configuration. To this end, we see the functionality in StruxureWare as being lacking for UPS NMC management as described below :

• No ability to mass deploy snmpv3 settings (the settings did not take when attempting to deploy via StruxureWare SNMP configuration template)
• No ability to import discovery settings for snmpv3. The more devices to discover across different subnets, the more discoveries are needed. Configuring these manually is a pain.
• No ability to convert from telnet to SSH (v5.1.7). This may be different in v6.0.6, but it sounds like StruxureWare 7.2.1 has not been updated to support the new options
• ... which brings up compatibility as an issue. If we upgraded all of our NMCs to v6.0.6 then would we lose the ability to manage them (limited though it may be right now anyway)?
• No ability to centrally manage the HTTPS certificate process. Once all devices are discovered it should be as easy as selecting all of them and having an option to "Generate CSR" for each. Then once all are signed by the PKI root CA, mass deploying the signed certificates via StruxureWare instead of having to manually install them on every NMC.
• No ability to centrally configure RADIUS auth and privacy phrases. This is supposed to be possible via INI file but we have seen these settings not "take". We had to manually configure all of ours (using SHA for auth, AES for priv, for the record)

Further to what I mentioned to Angela, an issue we have that may be on the NMC side rather than StruxureWare is that when we deploy settings via StruxureWare (specifically we use SCP if possible), I'm not sure if it's because the logon sessions get jammed up/hung or if it's a bug, but the web interface will no longer be accessible in v5.1.7 because the NMC will report that someone is already logged into the management interface. This remains the case long after any session timeout should have expired. Perhaps SCP and/or FTP sessions are not subject to the session timeout because they involve file transfers? As seen, this can still cause some problems. There should always be session timeouts. Perhaps a file transfer timeout setting should be implemented separately with a higher timeout value?

Also, something akin to SSH console scripting would be much more robust as error messages can be communicated more easily that using a file transfer method. There can be immediate feedback about errors with individual settings being made, and it would be easier to dump configuration session output (perhaps include an ability to mask passwords in dumps) so that administrators could see what actually happened when trying to make configuration changes. Whether via this method or a custom APC management protocol, something is needed to robustly mass configured devices. We need certainty when deploying to devices -- right now it's a crap shoot whether a given setting will take, and the error/problem notifications are not specific enough.

[voidstar_apc]

netadmin@princessauto.com wrote:

All but 1 of our NMCs runs with v5.1.7 and this has shown to have many problems with the CONFIG.INI method where settings just don't "take". As noted, converting from telnet to SSH (preferably v2) being a noted issue. In v5.1.7 SSH vs telnet is an either or. It just won't switch.

Alright, I've tested toggling SSH and Telnet back and forth 10 times via INI on v6.0.6. Seems to work for me, but maybe there's a difference in our environment. I've attached the files I used during testing to this message. I FTP them directly in.

To what Angela had mentioned, can it be confirmed that NMCs will accept partial INIs? Is there a master copy of the configuration stored elsewhere? The concern would be overwriting the existing CONFIG.INI with a partial one and causing problems for the NMC.

That's a reasonable concern. The INI files are never actually stored on the NMC. They're generated on-the-fly from the current settings when you download, and the settings contained therein are applied on-the-fly when you upload. In short, it's basically a command interpreter. So uploading partial settings has no effect on any other settings.

Further to what I mentioned to Angela, an issue we have that may be on the NMC side rather than StruxureWare is that when we deploy settings via StruxureWare (specifically we use SCP if possible), I'm not sure if it's because the logon sessions get jammed up/hung or if it's a bug, but the web interface will no longer be accessible in v5.1.7 because the NMC will report that someone is already logged into the management interface. This remains the case long after any session timeout should have expired.

Interesting. If StruxureWare, some other automated system, or an NMC bug is keeping a session open, then it would explain why settings that require a reboot fail to apply. Won't automatically reboot until the sessions are closed.

Perhaps SCP and/or FTP sessions are not subject to the session timeout because they involve file transfers? As seen, this can still cause some problems. There should always be session timeouts. Perhaps a file transfer timeout setting should be implemented separately with a higher timeout value?

It's a thought. An absolute timeout won't work for some of the NMC's uses of FTP due to large transfers at slow speeds, however there's a timeout imposed by the TCP layer. Of course, by design, the other guy can sit there sending zero window ACKs.

[netadmin_at_princessauto.com_apc · Accepted solution]

There was a bug found when uploading an INI file to v6.0.6.

Specifically:

- Downloaded CONFIG.INI from NMC

- Modified only SSH console settings

- Uploaded (entire) CONFIG.INI to NMC

This broke RADIUS logins. Our first thought was that it reset to local authentication only or reset the RADIUS shared secret, but it was neither.

One of our RADIUS connection profile criteria matches on "NAS Identifier" as our UPS units all have internal registered domain names that we assign then and they can be pattern matched easily. The "NAS Identifier" offered is the fully qualified domain name of the UPS NMC unit. For reference, let's say this unit's hostname was "ups-1234-5678" and the domain name was "domain.internal", with a resulting FQDN of "ups-1234-5678.domain.internal", which is pattern matched as the "NAS Identifier" on the RADIUS server. When the CONFIG.INI was uploaded, the unit's hostname changed from "ups-1234-5678" to "ups-1234-5678.domain.internal". This in turn changed the FQDN to "ups-1234-5687.domain.internal.domain.internal" and broke the pattern matching of the "NAS Identifier" on the RADIUS server, thus the connection request didn't match any RADIUS connection profiles and was denied.

For reference, on this unit "Override Manual DNS Settings" is checked (we use DHCP for assigning IP addresses and DNS servers), as is "System Name Synchronization".