Getting 802.1x to work

[BillP]

We tried to enable 802.1x on an NMC 9631 v6.7.2 without success. Firstly, we need to insert a CA certificate, which is strange because the NMC does not need to verify it's own certificate. Anyway, that certificate is accepted. So no problem there. Secondly, we need to enter a key file in either PEM or DER format which are both simply not accepted by the NMC, The former resulting in a format error and the latter simply not giving a message at all.

Any ideas?

[BillP]

Bump. Am I the only one having this issue?

[B-DoubleE_apc]

Hi Erwin,

Is the private key encrypted? I believe the private key needs to be RSA private un-encrypted - I don't think using an encrypted key is supported.

[BillP]

The private key is not encrypted and is readible with "openssl rsa -text -in ". Although the page does offer an option for encrypted keys which suggests that an encrypted key would be acceptable. Nevertheless, we have tried this with an unencrypted key.

[damonvaughn_apc]

The private key has to be encrypted.  I have been trying to get this working off and on for months, still without success.  At one point I was able to get the CA, key and host certs on the nmc but saw via packet capture that a decrypt error was tanking the TLS handshake, so starting fresh this week.  I will let you know if I can get it going.

[BillP · Accepted solution]

I'll try to bump this topic again. Just downloaded version 6.8.8 and still the same problem. Is the developer of this software also following this topic? This time the documentation says the key needs to be encrypted:

"Upload/replace or remove an encrypted private key. The supported file formats are PEM (Privacy Enhanced Mail) or the DER (Distinguished Encoding Rules) format with permitted file extensions .key or .KEY. NOTE: Unencrypted private keys are not accepted."

But being encrypted or unencrypted, being the format PEM or DER, being the extension .key or KEY. Being en 8.3-filename or not. Nothing changed since version 6.7.x. Could it be the encryption algorithm?

[Pete805]

Did you get any resolution to this?  I'm trying to implement it now on an isolated network and not having the best time at it.  I step-by-step document would help me out.  I was able to create a csr in NMCSecurityWizardCLI 1.0.1 and send it to my CA. My issue is, how do I use that and the tool to create a server certificate (.p15)?  A guide would be great.

Update:  I'm still struggling with this.  I am following along the Security Handbook: Network-Enabled Devices, firmware version 5.x as every document points back to this.  Unfortunately, it only references using the GUI APC Security Wizard--all that's available is the CLI tool.  With this being my first dive into this the CLI is a bit daunting.  I am able to follow the guide and attempt to perform the actions in the tool.  At this point, I'm trying to import the signed certificate using the --import command.  It seems I'm missing a file.  I'll have to go back and start over to find the disconnect.  Again, a step by step CLI instructions guide would be great, moreover, the GUI interface tool would be awesome for this beginner.  Anyways, thanks for reading and I hope someone reads this and provides a suggestion.  Any of you SE engineers out there have words of wisdom?

[RN-network]

We have just downloaded version 6.9.6 and it is clear that not a bit has changed since I first started this subject. We get stuck on exactly the same point as before.

[Shaun]

Hi @RN-network, @Pete805 

I've been working with this lately, trying to better document what's known to work and known not to work.  Nowhere near finished yet, but I can share some pointers to see if they help.  I'm still very much up to my neck in this topic, so would welcome any feedback on any issues you've found that I haven't (or indeed, if you do get it working, what fixed it for you)

First up, the nmccliwizard is a red herring, this system uses standard x509 certificates, not .p15 certificates, so the cli wizard isn't used (isn't needed, isn't helpful).  I've been using openssl for all examples as it's available everywhere.

Public certificate, CA certificate:

Certificate validation is done in both directions, so the UPS needs to trust the radius server's certificate, and the radius server needs to trust the UPS's CA. This is the big part to get right, as the CA-public cert relationship is how each device verifies the other.  Certificates should be either PEM-formatted (ie base64-armored) and named .pem or .PEM, or DER-formatted (eg binary ASN.1) and named .der or .DER.

* If the Server’s certificate was signed by an intermediate CA, this should be the public certificate of the same intermediate CA, not the root CA.
* Certificate chains (eg the root CA and intermediate CA bundled) are not supported.
* The same CA should be used to sign the device’s certificate.
* This cannot be the public certificate of the server itself; self-signed certificates are not supported for this role.
* SAN/CN/CommonName appear to be irrelevant - remember at this stage the UPS has no IP address, so names can't be resolved to it (and it can't resolve the server's name)

Private Key

This is the key to the public certificate that the UPS will use to identify itself to the radius server. It must be an RSA key, PKCS#8 format, and encrypted with TrippleDES.

To create an example with OpenSSL:

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -des3 -out client.key

The file should be PEM-formatted and named .key or .KEY

Known Issues

* Key Length - Private keys should be 2048-bit; 4096-bit keys are accepted but don't work correctly.

* Filenames - When matching filenames, ‘.key’, ‘.pem’, ‘.der’ etc is matched against the entire filename. For best results, ensure only one of these (and the one you intended) appears anywhere in the filename.

* Leading text (text in the certificate file before the BEGIN line or after the END line) may result in the file not being accepted; it’s strongly preferred to avoid this where possible.

* If the string ‘END’ appears anywhere in a base64-encoded portion of a certificate, the parser may end prematurely, and the file will not be accepted. If this cannot be avoided, it can be resolved by converting the key to DER format. (This obviously isn't the intended behaviour and has been logged as an issue.)

Example, for OpenSSL:

openssl x509 -inform pem -in example.pem -outform der -out example.der

[RN-network]

Finally it looks like we have a serious discussion. Thanks for that.

OpenSSL is fine. Documentation is ubiquitous and I am willing to provide the right commands if, and only if, I get this to work.

The part of mutual validation is new to me; it was not mentioned anywhere but it is ever so important. In this case is does make sense that a root CA needs to be uploaded, being it, however, the root that is at the top of the path that is involved in signing the RADIUS server's certificate, not the root that is involved in signing the NMC's certificate (although theoretically the same root can be involved in both cases of course).

Quote: "If the Server’s certificate was signed by an intermediate CA, this should be the public certificate of the same intermediate CA, not the root CA."

Reaction: I am sure it is my lack of interpretation skills but I don't understand the relation between "this" and "the public certificate" in your sentence. Would you please be so kind as to rephrase the sentence?

Quote: "Certificate chains (eg the root CA and intermediate CA bundled) are not supported."

Reaction: That is fine. But the client MUST send any intermediates along with its own certificate, in order for the validator (the RADIUS server) to reconstruct the chain. The same applies vice versa.

Quote "The same CA should be used to sign the device’s certificate."

Reaction: This is odd. Although this is very well possible in an internal PKI, it should not be obliged. Indeed, even in an internal PKI, if the server is signed by a different intermediate than the client, which is a common scenario, this is probably not going to work. It is completely irrelevant to the client who signed its own certificate; only the signer of the RADIUS server is to be verified. And vice versa for the RADIUS server (see picture).

Quote "SAN/CN/CommonName appear to be irrelevant - remember at this stage the UPS has no IP address, so names can't be resolved to it (and it can't resolve the server's name)"

Reaction: I disagree with you; getting 802.1x to work can be done on an open port of a switch (which probably will be the mostly applied scenario). So an NMC plugged into a switch, can be given both a DNS-name and an IP-address. Once it has a DNS name, a certificate can be created with the CN and the SAN fields filled. IMHO, conforming to the RFC, the SAN field MUST be used; the CN is not necessarily required but may be used. Not conforming to this may cause the RADIUS server to reject the request.

I will do some testing with the private key generation and let you know the results. Until now, not any private key was accepted so the may change things.

[RN-network]

OK, I did exactly as mentioned

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -des3 -out testups.key

Result: Bad Key/Certificate (file name: testups.key)

That is the same result we got before. I get the feeling that the format is not the issue but either the order in which files need to be uploaded or the algorithm that tries to link uploaded files.

[Shaun]

Hi,

So some of this I'm going to be quite light on - due to the Christmas holidays I'm currently away from my hardware, and as you've seen, documentation is quite light on the ground - so I've been strongly preferring what I can test hands-on. There are going to be thin patches here where I don't know and I can't test until the middle of next week.

I should also be fair and point out I'd never seen any of this before 6 weeks ago, so I'm not claiming to be an expert - I'm just comparing what has worked for me vs what hasn't. Essentially, in the first case I came across for this - the only documentation I could find said the private key must not be encrypted, and the card complained that the key must be encrypted.  So obviously we have a huge documentation gap here, and I went down the ensuing rabbit-hole.

First, I can share the scripts I've been using to reproduce this - https://schneider-electric.box.com/s/i1kl51rdju28k78qajx5b103urrw0jvu

The 'demo-certs' folder contains the scripts I use to generate CA, server & client certs, and the 'demo-config' folder contains the configuration I've been using for my switch & freeradius.  My "lab" for this purpose has been a Cisco 3750 (15.0) (for no specific reason other than no-one else was using it), and freeradius on a raspberry pi.  I offer these only as a test-case - please don't trust me to configure your switch, because I don't trust me to configure mine.  I've been testing these against firmware 6.9.6 & 7.0.4 on an AP9635, and firmware 1.5.something on an AP9641.  I expect all nmc2/nmc3 platforms to behave the same in this respect, but I haven't tested any older firmwares, there just aren't enough hours in the day.

I do need to experiment with what order things can be added in - I tend to wipe my card before I start a test, so I don't have to remember what strange things I did to it last week, so I've always been testing from a clean slate.  However, I suspect you're right that it's a combination of new & old - "bad key/certificate" sounds like it's complaining that the key doesn't match the certificate (I hope - it's the only reasonable reason I can think of why it doesn't know whether it's the key or the cert that's wrong).  If a previous certificate is still on the card, that's most likely true.  So most likely you need to create a public certificate from the private key, upload the new cert, and then the new key.

Now, with the caveat that I'd be much more comfortable with this in front of me, so I can't promise I'm not mis-remembering anything; the specific behaviour I've been seeing with CA - and specifically intermediate CA - is that the NMC has no native truststore - that is, it doesn't ship with a list of commercial root CA that it trusts.  And because it won't take a chained certificate, that means we have precisely one slot to insert a CA certificate.

So to follow the diagram you shared - in the last two steps where both systems share their certificates with the other; the server sends its own public cert, and the intermediate CA.  To verify this chain, the client would need either the same intermediate CA, or the root CA that signed the intermediate.  The client then sends it's own public cert, and the intermediate CA.  And this creates our catch - we must have the intermediate CA for this step, so that must be the CA that's loaded onto the card - and is then the only CA we have available to verify the server.

This creates the biggest friction point I've found so far - that when we verify the radius server's certificate, we still only have one possible CA to do so with. This is where we arrive at this odd restriction of having the same intermediate sign both client & server - not that eap-tls or radius requires it, but because the card only has one CA available to fill every role required.

for CN vs SAN - I've been using "Test Supplicant" as my CN, and no SAN / dnsnames. My switch doesn't show the line/protocol as up until after the port has been authenticated - so during the authentication, eapol is the only traffic allowed on the port.  So requiring them would create a catch-22.  I don't believe including them should hurt either, but I need to test this further - but for the scenario where 802.1x is being used to authenticate access to the network, requiring dns would be a circular dependency.

Again, this is going to be one of those points where I'm still learning, so if your system does require them - that'd be useful for me to learn.  All I know so far is that mine doesn't.

[RN-network]

Hi Shaun,

First of all: there is no hurry. Christmas fun comes first. I am already glad there is attention to this topic. Thanks again for that.

I'm currently studying your answers and the possibilities/limitations.

[000]

Anything new on this topic?  I know this is more than a year old but I am being tasked with setting up an NMC with NAC/802.1x.  I can upload the CA cert but key cert gives me an error.  I also tried the genpkey and get Bad Key/Certificate.  I'm stuck at installing the key part and any thing I try I get errors.