Private CA SSL certificates

[MartBrooks_apc]

Hello

We have several hundred APC power bars, all of which are currently running with self-signed SSL certificates.  I have been asked to fix this.

We have an internal certificate authority from which we issue certificates for internal-facing management interfaces that can use SSL.  This is based on OpenSSL.

I've looked at the official "Security Wizard" software and it seems it's not capable of generating OpenSSL compatible key material.  Rather bizarrely it seems to want to use PKCS#15 which is something more usually associated with smartcard devices.

This would be less of a problem if the tool were capable of generating wildcard certificates - at least then I'd only have to deal with silliness once, but any attempt to do this results in (usually) an error due to restricted characters in the DN, or (sometimes) a crash.

OpenSSL itself doesn't deal with PKCS#15 (nor would I expect it to).   I've seen references to "pemtrans" (won't compile on any system I can find, relies on third-party proprietary library) and "apc_tools" (same) but cannot get either of them to work.

It seems like bulk deployment of SSL certificates (or support for a wildcard) would be an obvious and basic need for anyone buying these products.  Is anyone aware of a viable workaround for this issue?   Self-signed certs are (quite rightly) heavily frowned upon.

Regards

Martin.

[BillP]

Hi Martin,

Which model(s) of power bar(s) (Rack PDU) are you working with? Let's start there as that will determine how to best specifically proceed for what you need. Long story short, we do have some solutions for wildcard certificates and also using Subject Alternative Name (SAN) as wildcard in some instances.

The unfortunate part is the APC Security Wizard exists and that is what we have to work with based on the cryptography library in use on Network Management Card (NMC) based products as it stands now. I don't personally like it either but that is why it is the way it is.  The generation of NMC hardware you have though is important which is why I asked what model(s) of PDU we're working with.

The basic process will be to use the APC Security Wizard to make a .csr file and submit it to your OpenSSL CA. From there, you can hopefully get a .cer or .crt to import through the Security Wizard and import on to your devices.

[MartBrooks_apc]

Hello Angela,

The bulk of the powerbars I've been working with so far are model AP8953.

A wildcard approach would be preferable to using SANs simply because of the number of bits of kit involved.

Regards

Martin.

[BillP]

Hi Martin,

OK, thanks. So AP8953 uses an NMC2 which is good news for this. (If you encounter any other models, let me know so we can tell you how to best handle.)

Here are the options and FYI, both of involve a new "NMC Security Wizard CLI" utility we are in the process of making available for public download. But it can be shared with you via technical support or I can share it with you. Here is what you'll need to do.

Firstly, to ensure the best experience for this "department," I suggest to upgrade these AP8953 PDUs to AOS 6.6.4 (with rpdu2g app v6.6.4) which is the latest of the on-board APC OS where we have made several specific adjustments to SSL/TLS cert functionality to ensure compatibility with third party CA tools. That was recently released and available through StruxureWare Data Center Expert firmware catalog or technical support (as we are in process of transitioning our download site, I am not sure if it is posted yet for non-DCE users).

Secondly, you'll get the NMC Security Wizard CLI tool which is basically the APC Security Wizard we have had posted for a while but it is CLI based to aid customers who want to script and save time with the CSR, imports, cert creation, etc. This tool will allow you to create a wildcard certificate if you want. What I meant earlier is that since most web browsers want you to use SAN field now to determine if it is valid as opposed to CN, you can do something like *.company.com for both SAN and CN fields.

Actually, I will just attach the firmware here and the wizard tool for your reference.

Alternatively, you can hopefully use the CLI security tool to script creating CSR and importing certs a bit more easily than the existing GUI Security Wizard tool.

[MartBrooks_apc]

Hi Angela

That worked just fine, thankyou.  I have successfully created and installed the wildcard certificate on the first 30 or so PDUs on my list.

Just for the purposes of feedback, I found the CLI tool a little unintuitive to work with as it sometimes adds or requires file extensions and some times it does not.  As a system admin, I like the "least surprise" approach of just letting the user set the filename they want.

Also, I found a slight bug in the web interface where if the name of the certificate file to upload is too long then it will silently fail with no error. Shortening the file name to "apc.p15" worked.

We also have a block of older PDUs to deal with.   These are all model AP7920 (note, not AP7920B).  Do you know if it would be possible to install a wildcard certificate on these and what the minimum firmware is if so?

The few I've looked at are all running firmware versions:

* rpdu - 3.7.4
* aos - 3.7.4

I can't even find this model's firmware on the support website so I've no idea if this is current or not.

Regards

Martin.

[BillP]

Hi Martin,

Sorry for the delay as I was on holiday the past few days.

I think on the file extension thing, we require .p15 for some files and know it has to be that so maybe that is why you don't specify it in some examples. Either way, I understand what you're saying from a user perspective so I will surely mention it or our development team today.

On 11/9/2018 6:16 AM, Martin said:

Also, I found a slight bug in the web interface where if the name of the certificate file to upload is too long then it will silently fail with no error. Shortening the file name to "apc.p15" worked.

Can you confirm which AOS or device this was? The NMC2 devices at AOS 6.6.4 you mean? Can you give me an example filename I can replicate with this to log a bug? Thanks!

On 11/9/2018 6:16 AM, Martin said:

We also have a block of older PDUs to deal with.   These are all model AP7920 (note, not AP7920B).  Do you know if it would be possible to install a wildcard certificate on these and what the minimum firmware is if so?

The few I've looked at are all running firmware versions:

* rpdu - 3.7.4
* aos - 3.7.4

I can't even find this model's firmware on the support website so I've no idea if this is current or not.

Regards

Martin.

These older units are discontinued and have been replaced. They aren't designed to work with the new utility I gave you due to the differences in the cryptology stack. I will private message you another version of the APC Security Wizard (the older GUI based tool we've had for a while) we made for these older devices that will allow Wildcard certificates on them. I think it should work with AOS 3.7.4 but we do have an AOS 3.9.2/rpdu 3.9.2 for these devices I'd recommend since we did make a change in the newer rev to allow larger file sizes on the certs. I'll attach that firmware to the post for you.

[Anonymous user]

I still can't make it work with our units.

Model: AP9631

AOS: v6.6.4

I can generate CSR with GUI wizard (1.0.4), sign it with our CA, and then import the certificate back into GUI wizard, but the NMC does not accept it. It is stuck with "Loading certificate..." forever.

I also tried to use the CLI tool attached to this thread, but it does not start on Win10 at all. It complains about missing cl32.dll, which is there. Dependency analysis suggests that is not the cl32.dll library itself, but other Windows API libraries it depends on, which were removed in Win10.

This is pretty unfortunate to be honest. Especially since this issue exists since over 10 years according to the posts on this forum.

[BillP]

Hi Anton,

I have been using this utility on Windows 10 64 bit without issue. (I am currently on Win 10 build 1803.) I assume you extracted the files out of the .zip and then ran the .exe with the cl32.dll in the same folder (versus trying to execute from the .zip)?

7upBgY567JhEKQ9diqqn6g%3D%3D.png

For NMC2 devices, the new CLI Utility must be used is item #1 and with AOS 6.6.4, all of the issues we are aware of with MS CA and other third party CAs. The behavior you mentioned with "Loading certificate..." is "expected" on some level because you need the new CLI Wizard tool to make a proper certificate accepted by AOS 6.6.4 due to the crytopgraphy stack embedded in the Wizard tools and AOS firmware.

I agree it took us way too long to fix all of this but we do have fixes available now.

If you're still stuck after what I said here, we can work directly together on it - only problem is I am away most of next week on holiday.

[Anonymous user]

Hi Angela

First of all, your example of the CLI usage does not really show anything. It does run just fine without any arguments, because in this case it does not get to the DLL load. Here is sample output with empty use and real useK8zyqIVZpGEVS1VPmiyGSQ%3D%3D.png

As you can see the DLL is there, but the tool still throws an error. In fact, it is not the DLL that is missing, but one of its dependencies. It is yet another example of poor error reporting and documentation. Neither the real problem is displayed, nor the required dependency is documented.

The way the Web UI behaves with SSL certificates is terrible. It never reports any errors and never outputs any diagnostic data. If for some reason it does not like the certificate it just silently discards it and tells that everything is fine. In some cases it will fall to "Loading certificate...", which is kind of meaningless.

Finally, I managed to make the CLI tool work with Wine on Ubuntu virtual machine, which does not respect all the Win10 API changes yet. However, the certificate generated still triggers the "Loading certificate...", which is then stuck.

To be honest, this does not look like production-ready hardware/software. The "Loading certificate..." problem is known for years and I am fairly sure that downgrading to 6.2.1 would fix it as suggested in another topic on this forum, but I have already wasted too much time on this supposed to be simple task. Next time we extend our hardware I will strongly recommend to look for anything but APC.

[BillP]

Hi Anton,

I know it is frustrating and I lived similar frustrations working through this and fighting to get this fixed right to meet customer expectations. I definitely get it. We will still look into what you've shared even though I know you're sick of dealing with it. I also understand what you mean about how I checked earlier so my mistake there for not putting two and two together. I just re-tested it while doing a function that calls the DLL and I still don't see it on my machine/Win 10 build.

The only thing I can say is please know that there are a lot of people behind this NMC product that truly care about it and its customers. Official QA testing was done on both NMC firmware and the utility. Unofficial QA testing was also done by myself (not on actual QA team but have been involved in supporting this product in various roles over the past 11+ years).

So in summary, I appreciate your feedback here and I feel your pain and regardless of what you decide to do, I will still get it looked into and find out what is causing this issue on both Win 10 and Server 2012 in certain situations. The issue with firmware "Loading certificate.." I feel confident will be resolved with a new cert from the utility once that part works. Also can't disagree that it is a poor user experience when you get with no inkling of what to do next or clear error if there is a problem. While I recognize it, I will share your candied feedback here with our team for them to see and understand what customers are saying.

Just in case you do have further comment, I'll only be able to check back next week when I return from holiday so expect a delay.

[Anonymous user]

Hi Angela,

The proper thing for APC to do would be to make the tool part of the NMC card so you can upload a standard pfx certificate file and it converts and installs it automatically. That is what is expected and reasonable. No solution with command line conversion utilities or "Security Wizards", etc. is reasonable. This has gone on long enough.

[BillP]

Hi Ben,

I am totally with you. I have already formally requested this for the next gen product in development since it is a significant effort for the existing NMC based on how it all works under the hood. I don't like that answer either (that we likely can't change it for NMC2 type product) but that is what I think will end up being the reality - just to be transparent.

[Yannik_apc]

I am sorry for digging up this old thread once more, but I think it is asscociated with the previous activity in this thread.

Martin appeared to have had the same problem I am currently facing - the only difference is, that he had to deal with several hundred UPSes and I am just struggling to setup a private SSL cert, signed by an internal chain of trust, in a small lab environment (keys are all 2048 SHA256).

Personally (though I don't have any need to use it for scripting) I somewhat like the CLI tool.

However, it is raising an uncaught exception (sorry for the plain paste, but even the "code editor" --> plain HTML does not like code in

tags):</p> <p>F:\Downloads_extern\NMCSecurityWizardCLI\NMCSecurityWizardCLI>NMCSecurityWizardCLI --import -o apc -s sua3000csr.crt -p apc-csr</p> <p>NMC Security Wizard Command Line Utility v1.0.1 (c) Copyright 2018 Schneider Electric. All rights reserved.<br>-----------------------------------------------------------------------------<br>Uncaught Exception: cryptlib.CryptException: -3: Bad argument, parameter 3<br>at NMCSecurityWizardCLI.Program.ImportSignedCSR(String sCertFile, String sKeyFile, String sOutFile)<br>at NMCSecurityWizardCLI.Program.Main(String[] args);</p> <p>with SUA3000CSR.crt being the cert-file signed by my internal intermediate CA and apc-csr.p15 the PKCS#15 file from the previous certification request command in the Security Wizard CLI.</p> <p>I guess I'm missing something vital here :)<br><br>Best regards from Germany and stay safe in these times<br>Yannik</p></p>

[Anonymous user]

Hey all,
Because installing private SSL's on NMC is a reoccurring theme, I decided to create a discussion after some progress was made during a support chat.

https://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/95305/uplo...

[Anonymous user]

Good evening, Yannik.

Did you ever manage to figure out what the mysterious "-3: Bad argument, parameter 3" error meant and how to fix it?

I can get past the "-32: Bad/unrecognised data format" error by cutting down OpenSSL's certificate to just the cert, but I'm stumped with the bad argument error. Your post here is one of the few references to the issue I've found.

Any advice is appreciated. Best!

-Chris

[hendl_apc]

Hi all,

same problem here with Windows 10. I can generate a csr but the import fails.

C:\localdata\NMCSecurityWizardCLI>NMCSecurityWizardCLI.exe --import -o usv -s usv.cer -p usv

NMC Security Wizard Command Line Utility v1.0.1
(c) Copyright 2018 Schneider Electric. All rights reserved.
-----------------------------------------------------------------------------

Unbehandelte Ausnahme: cryptlib.CryptException: -3: Bad argument, parameter 3
bei NMCSecurityWizardCLI.Program.ImportSignedCSR(String sCertFile, String sKeyFile, String sOutFile)
bei NMCSecurityWizardCLI.Program.Main(String[] args)

Regards
Stephan

[Anonymous user]

Stephan  Chris 

You are not able to upload your own SSL Certificates because APC locks it down to their own proprietary keys (.p12). Please check out my form post and ask the APC team to develop an option for people like us that want to upload our own certs.

https://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/95305/uplo...

[Anonymous user]

My frustration is two-fold, really. It's a convoluted process that should be unnecessary, but even the tools provided that should work....don't, mysteriously, for no reason. To my read, it should be:

1) Create CSR on a Windows host with the NMCSecurityWizardCLI

2) Send the CSR to an appropriate signing host - in my case a Linux system with OpenSSL

3) Sign CSR, return signed cert to the windows host.

4) Use NMCSecurityWizardCLI --import -o certfornmc -s certfromopenssl.crt -k originalkeycreatedbysecuritywizard

If I use the OpenSSL-signed cert straight up, I get a "-32: Bad/unrecognized data format" error. Ooooookay. If I remove the header and just leave the bare cert, that's when I get "-3: Bad argument, parameter 3". Which is frustratingly vague, because that doesn't tell me what the parameter *is*.

It's bad enough to have a convoluted process that has annoying hoops to jump through, but at least they should work.

No idea who to turn to for any kind of ongoing support on this, since it's all pretty buried on APC's website. I know Angela is doing her best, but it's pretty sad for a company this size. Thankfully I'm in an environment where this is currently more of a 'nice to have to keep things proper and tidy' rather than a strict requirement, although that may change in the future.

[BillP]

Chris,

Sorry for the inconvenience. Please try using the following version of Security Wizard. It help another user so it may help you. 

https://schneider-electric.box.com/s/sxlkk4nljylwnyjzno3trr1ilvz46e1r

[Yannik_apc]

On 8/6/2020 1:30 PM, Bill said:

Please try using the following version of Security Wizard.

Hello Bill,

which version of the SecWiz is this? A pre-release candidate or is it already available on the APC Software Download page?

Sorry for my question - I am just a bit overly cautious, when it comes to security tools stored on cloud storage spaces rather than on an official site...

Kind regards

Yannik

[Anonymous user]

Ah-hah. Okay, so I threw caution to the wind and gave this a shot. It does appear to work. So 1.0.4 is broken and 1.0.0 is the right version to use?

[BillP]

HI,

Thanks for letting us know version 1 resolved your issue. 

As for 1.0.4 we are reviewing it. 

[hendl_apc · Accepted solution]

Hi Bill, hi Chris,

v1.0.0 works for me too.

Regards,
Stephan

[MrInOut]

Hi,

Can anyone provide me somehow the v1.0.0?

I cannot find it on the APC homepage. Only v1.0.4 is available.

Thanks in advance,

Mr. InOut

[Flowie]

Hi,

i've spent most of the day on searching a valid download link for v1.0.0 ... ... But didn't find any.

Can someone please share a download link for version 1.0.0?

Thanks in Advance!