FYI: UPS NMC2 firmware v6.4.0 now available for Smart-UPS and Symmetra 1-phase apps

[BillP]

Hello everyone,

Applicable to AP9630 or AP9631 UPS Network Management Card 2 users...

I wanted to let everyone know that APC Operating System (AOS) v6.4.0 / Smart-UPS & Symmetra v6.4.0 firmware applications are now available for download @ http://www.apc.com/tools/download/index.cfm. To download, select Filter by Software/Firmware and select Firmware Upgrades - UPS Network Management Card. Look for part numbers SFSUMX640 (if you have a Smart-UPS) and SFSY640 (if you have a Symmetra 1-phase, such as Symmetra LX).

AOS v6.4.0 brings some exciting and welcomed changes such as:

• TLS v1.2 for HTTPS web access (as well as ability to select the minimum protocol supported - this will allow you to disable SSLv3 if you'd like)
• Removal of weak cipher suites (such as RC4)
• Fix to work with OpenSSH v6.7+
• Modbus TCP (and RTU supported with AP9635 coming soon!)
• Responsive web interface

For more details, take a look at the updated release notes and user's guides (which apply to both firmware applications) here: http://www.apc.com/tools/download/index.cfm?dl_method=display_manuals&part_num=SFSUMX640 

You'll see the same AOS v6.4.0 (or higher) coming with other firmware applications too, such as Rack PDU 2G AP8XXX series, in the next few months.

[arj_apc]

Please can you get the ap9630 web page sorted out ASAP so the documents and firmware downloads work again. At the moment it is showing V6.2.1 downloads but all the links result in "404 File or directory not found" error pages. You cannot download the latest 6.4 or even the old 6.2.1 files at the moment. I've tried raising a support request but get useless responses from someone who clearly does not understand what they are talking about.

Andrew

[BillP]

Hi, I see the problem. I'll ask the appropriate person to investigate. 

In the meantime, the method to download I wrote above in the article still works OK by using the main software/firmware download site @ http://www.apc.com/tools/download/index.cfm and also the docs show at http://www.apc.com/tools/download/index.cfm?dl_method=display_manuals&part_num=SFSUMX640 with working links to the latest documentation  

[arj_apc]

Angela, thanks. Could you perhaps add the Release Notes document as it is not currently in that list of downloadable documents. Andrew

[BillP]

Sure, see attachment to this post.

[Terry_Kennedy_apc]

The release notes mention "Symmetra LX Battery Frame Status total information displays incorrect values in the web UI." This is either not limited to the LX or there's a very similar bug in the Symmetra RM 6000. See the attached image - the total number of batteries in each chassis is off by a factor of 10.

Also, the "View raw status data" seems to have disappeared from the battery and intelligence module status pages. That's actually the only way to determine if the battery bus is in a charging, equilibrium, or discharging state (the "I-0.0" part of the raw status), although I do use other parts of the raw status.

[voidstar_apc]

Hi Terry,

Thank you for reporting this. Based on your report, we looked into it and fixed both problems for the next revision. One of the links to the raw status in the upmims.htm page was missing, although raw status for the battery seems to be there (are you seeing something different?). You probably already know this, but you can work around it by going directly to upraw.htm and uprawbatt.htm.

[Terry_Kennedy_apc]

Thanks!

I'm not seeing any links for raw status on the battery page, either the summary or when clicking on one of the batter frames. I've attached two screen captures.

[BillP]

I attempted to update an APC 9630 connected to a SmartUPS 1500 last week and the installation failed after the first part.

Starting Upgrade 01/06/16 13:23:54
-----------------
[01/06/16 13:23:54] 192.168.0.101: Pinging (via ICMP) device
[01/06/16 13:23:54] 192.168.0.101: Device connection passed
[01/06/16 13:23:55] 192.168.0.101: Testing FTP Log-in
[01/06/16 13:23:58] 192.168.0.101: FTP Log-in passed
[01/06/16 13:23:59] 192.168.0.101: Saving data file
[01/06/16 13:24:01] 192.168.0.101: AOS Prior to firmware transfer: Network Management Card AOS v6.2.0

[01/06/16 13:24:01] 192.168.0.101: APP Prior to firmware transfer: Smart-UPS & Matrix-UPS APP v6.2.0

[01/06/16 13:24:01] 192.168.0.101: Saving event & configuration files
[01/06/16 13:24:23] 192.168.0.101: Transferring firmware file (1/3): Boot Monitor
[01/06/16 13:24:30] 192.168.0.101: Boot Monitor transfer successful
[01/06/16 13:24:32] 192.168.0.101: Restarting network interface
[01/06/16 13:32:31] 192.168.0.101: ERROR --> Device connection failed

Now, the device is not known to the network, and I can't find anything in the APC documentation that tells me how to fix this.

Does anyone have any suggestions?

[Terry_Kennedy_apc]

You may be able to update the card using the USB port and a USB memory stick. APC has a knowledgebase article here (search for the text "devices with USB").

If that doesn't work, you'll need to connect the console cable (oddball cable with micro-headphone-style plug on one end and 9-pin serial on the other) to a PC (modern PCs will probably need a USB-to-serial adapter) and see what the card is saying.

If neither of those work, you'll probably need to call APC support. They may have another procedure, or if the card is in warranty they can arrange an RMA for you.

[BillP]

It is a 9630, so it doesn't have the USB port.

Support told me that holding the reset button in for 20+ seconds would also clear it out, but then I'd still need to get into the console mode to start communication.

I ordered that serial cable from an eBay seller.

We'll see what happens in about 10 days...

[Terry_Kennedy_apc]

Sorry, I didn't realize that the AP9630 was missing the USB ports. Hopefully you'll be able to get it going via the console cable.

[Anonymous user]

I think I'm going to wait a little bit before I apply this update to my only NMC2 series card. Just to be careful.

On 1/11/2016 2:57 PM, Angela said:

Removal of weak cipher suites (such as RC4)

I'd like to inquire about the possibility of something similar being done for the older AP9617/18/19 series network management cards. I realize these are older products and probably not actively supported any more, but there are quite a few of us (especially hobbyist users like myself) still using these cards with a need or desire for secured communications over HTTP. For us it's not cost effective to go out and buy all new network management cards and browsers removing (instead of deprecating) support for older SSL/TLS methods is problematic.

Also, since the AP9635 is mentioned...is it a product that actually exists? Pictures I've seen indicates that it has an onboard telephone modem, a feature for which it seemed there was no follow-on product after the AP9618. I've tried to see if I could scare an AP9635 up anywhere without success.

[Terry_Kennedy_apc]

I suspect the issue with the older cards is that they don't really have the CPU power to handle modern cipher suites. SSL on those cards is pretty slow already (but not nearly as bad as on the AP9210, which was based on the AP9605 card).

The AP9635 is a pretty rare card. As far as I know, it was only shipped with Symmetra PX and MGE Galaxy units. It is built on a different PCB than the AP9630/31. An AP9630 is an AP9631 with a bunch of parts not installed. The AP9617/18/19 all had the same base PCB, the difference was in the daughtercard and faceplate so upgrades were available.

There were some odd-looking AP9635 cards, probably fakes, on eBay a while ago which appear to have 4 Ethernet ports.

[BillP]

Hi Terry,

You are correct. These more complex cipher suites and protocols are not easy to implement on the older cards. Some investigations are in progress but I don't know what will end up happening. And for the most part, the investigations pertain to products that do not have an AP9630/31 based equivalent ready right this second. AP9617/18/19 have NMC2 replacements for the most part and the recommendation will be to purchase those since NMC2 has been available since early 2009. As you mentioned Terry, the experience there will be much more acceptable in regards to speed, overall features, etc.

AP9635 is not too rare actually... It is widely used in the 3 phase world and will shortly be compatible with Smart-UPS and Symmetra single phase.

What you saw Terry with the (4) ethernet ports were indeed VERY early prototypes someone stole and hence, legal action was taken against the seller(s).

You can look at the SKU at the below link (which AP9635 and AP9635CH are the same). Originally there was special Schneider Electric branding for China only (hence the CH) but then that part number became standard for AP9635.

http://www.apc.com/shop/us/en/products/Schneider-UPS-Network-Management-Card-2-with-Environment-Moni... 

[voidstar_apc]

Tech support should have access to a slightly newer version: sum/sy 391. It supports TLS 1.1, however it's still not enough to work with a modern browser.

[BillP]

*TLS 1.0 

[BillP]

Larry,

It looks like the first of the three firmware files (BootMonitor) transferred successfully. On subsequent reboot, the device couldn't be found.

What I would've done is tried to browse to the device over the network to see if it was there or not (web, telnet, FTP, etc). The utility may have tried to log in a few seconds too early or something and depending on your environment, then it through an error. You may have been able to still access the device via FTP and manually continue the upgrade.

But, tech support told you to "format" which is holding down the reset button for 20+ seconds (which you should've seen a pulse of green and then orange) which wipes ALL settings, including TCP/IP settings.

Now, a serial cable will be required or you can try the APC Device IP Configuration Wizard to search the network for unconfigured management cards. That tool is mentioned here. http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA156064 and can be downloaded from here http://www.apc.com/tools/download/index.cfm 

I very much doubt you need a replacement...

[Anonymous user]

On 1/17/2016 5:39 AM, Terry said:

I suspect the issue with the older cards is that they don't really have the CPU power to handle modern cipher suites. SSL on those cards is pretty slow already (but not nearly as bad as on the AP9210, which was based on the AP9605 card).

I went ahead just now and tried your procedure for unofficially upgrading an "unimportant" early production AP9617 to AOS 3.9.0 in order to gain TLS 1.0 support. From what I've seen so far, it's gone really well. I can't say as I timed it with a stopwatch, but there's little question in my mind that the card's CPU is responding faster than it ever did with SSLv3. All other functions seem to be working just fine.

I'd hope that APC and Schneider Electric might choose to officially release such updated firmware for the AP9617/18/19 series.

Someone gave me an old AP9210 PDU and I was shocked to learn that it had a web-based configuration interface. It's definitely a burden on the 13 MHz 80186 CPU!

[Terry_Kennedy_apc]

The AP9210 uses the same hardware as the AP9605 management card. That card has EPROM firmware. AP9606 has flashable firmware. I actually started with much earlier units - the first APC management device was a tiny little PC that booted from a floppy. After that came a pre-SmartSlot card for Matrix UPS. That was a deeper but narrower card, and if you got a warranty replacement of the Matrix upper half (EU) you needed to ask for a new management card, which was the AP9605.

History lesson aside, the AP9210 should never be used anywhere untrusted users or bad guys can get to it either via the network or the serial console. It has a hardcoded manufacturing password which is well-known. I don't remember if the AP9211/AP9606 got a fix for this (I think it was superseded at the time the problem was reported to APC, but the older firmware versions were definitely vulnerable.

[Anonymous user]

Hi,

just two bugs I noticed:

NMC sends mail to disabled accounts

1. Configure local mail server (Configuration -> Notification -> E-mail -> Server)
2. Add a recipient (Configuration -> Notification -> E-mail -> Recipients)
3. Make sure that mail is working and the recipient created in step 2 receives mails
4. Now you need to get a bad battery (that's how we discovered the bug)
5. When the NMC detects the bad battery ("UPS: At least one faulty battery exists." event) you will start getting mails:
   
6. Because you don't get your new battery on Saturday or Sunday, you want to turn of mail alerts (is that another bug, that the NMC will send that many warnings per minute?) so you can turn it on again when the problem is fixed: You go to Configuration-> Notification -> E-mail -> Recipients, uncheck the "Enable" checkbox and save.
7. But wait... the recipient will still receive these mails. You actual have to delete the recipient to get the mail bombing stopped.

On 2016-01-11 21:57, Angela said:

Removal of weak cipher suites (such as RC4)

That's not true for e-mail notification:

Jan 19 16:56:45 mailgw42 postfix/smtpd[26680]: connect from apc5DB4A3.exmaple.org[10.20.2.5]
Jan 19 16:56:45 mailgw42 postfix/smtpd[26680]: Anonymous TLS connection established from apc5DB4A3.example.org[10.20.2.5]: TLSv1 with cipher RC4-SHA (128/128 bits)

[BillP]

Hi Mike,

I just did a basic setup of your issue #1 with a Gmail account and a single recipient and did NOT see the issue. I set up the local email server and added my Gmail account. Test passed. I had some active alarms on my UPS and set those up to email me quickly (to expedite testing). I received alarms when my recipient was enabled and did not receive them when it was disabled. I enabled/disabled the recipient a few separate times and waited several minutes between each tests as the active alarms were set to email every 30 seconds (which is the minimum). I am not seeing emails when my recipient is disabled.

Also, when you add a recipient and don't do any further configuration, that recipient is set to receive ALL alarms every 1, 2, 5, or 10 minutes, by default. The user can go in and change the frequency of alarms, add delays, etc. So, receiving an alarm every 2 minutes is NOT a bug but rather the default. If the user doesn't like it, they can go in and change the frequency. We do it this way so that the user cannot complain they did not receive an alarm and then we allow them to customize it if they want a different configuration.

On the email issue, since this could be an isolated case or specific to a particular setup, I'd recommend you start a new dedicated thread or contact your local tech support to work through it. We'd want the log files/config file from your usual set up and then after you modify settings to see if we can replicate it or identify an issue. (Instructions on gathering log files are here - the .tar file is preferred -> http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA156131). For example, maybe it happens with more than one recipient, or a certain recipient # that I did not get into. We could use the config files to try and replicate it and match the setup. At this time, I cannot agree it is a bug affecting everyone.

On your second issue, to clarify, we removed RC4 ciphers from everywhere except email/SMTP authentication. We'll try to clarify that in the release notes and anywhere else it is mentioned. The SMTP authentication code is a separate code base from SSH, HTTPS, etc which is in progress of being updated to support TLS 1.1 and we'll see about disabling the RC4 ciphers at that time.

Please let me know if you have any questions specific to the release.

[BillP]

Thanks for that.

Tried to install 6.21 on another unit and it too failed - unable to reconnect.

Now, neither one responds to a ping, FTP, or telnet.

Still waiting for the cable to arrive.

Will try the IP Wizard and see what results.

[BillP]

The IP Wizard failed to find the two devices - they were completely off the network.

Using the cable and the Console, I found that the IP information was completely wiped out and I had to manually reset the IP of the card and reboot it.

Afterwards, the new firmware updated successfully on the two devices that were blown out by the 6.40 update.

Sure enough, attempting to install that update on the last UPS, the network dropped after the first part was installed and I had to manually set the IP.

Let's be clear; I am an MSP, and most of my work is done remotely.  I certainly do not want to have to perform a site visit for each client to ensure that APC firmware installs properly.  Somehow I think the update process should be more resilient than it was with this release. 

[Terry_Kennedy_apc]

6.4.6 exists in the FTP area. I believe the (new and definitely not improved) main APC site is still offering 6.4.0 (if you can find it). Release notes do not seem to be available. Could they be posted here?

[BillP · Accepted solution]

Terry, thanks for the point out!  I had no idea that v6.4.6 was available!  The site, as always, appears a work in progress.  A "nit" I have is that when you enter the site, because APC is a US company, I feel that the US should be the first country in the pick list regardless of the need to alphabetize the rest of them.

Has APC ever reached out to the user community directly to ask for their input into the design of the site?  In retrospect, that's probably a can of worms that should remain closed.

Anyway, thanks again and Happy Holidays to everyone!