Cisco ACI: Change certificates

[BeniG]

So recently we needed to change some of our cert's for our APICs. Same root cert, same intermediate cert, just different CN (to reflect a load-balanced service for the cluster) as well as an additional SAN for the device. The same setup of a cert (CN = load-balanced service, SAN=all individual APIC in a cluster) works for other APIC clusters we have.

But for the once that we now had to change the web agent doesn't respond anymore. Error:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.

The error message is quite clear but I don't understand the root cause yet. As mentioned, same root and intermediate cert (checked by SNR) as we have on other clusters and had before the change.

Has anybody of you guys experience with changing the certs for APIC and the implementation on NNM? 

I added the new cert to the web-agent directly of the affected devices -> same error

I added it to the communication settings for the specific apic's and removed the device completley and readded it -> same error

Br

Beni

[Eugene Kopyshev]

Hi,

I guess, the web agent by device side faced and expiration of certificate. The new should be signed by a CN which is equal to FQDN of the device, if I remember that process right. All other details, such as DN should also met, of course.

https://docs.microfocus.com/itom/Network_Node_Manager_i:2020.08/WebAgentForm

If you have followed the guide above, try taking a traffic capture between nnmi and the device - https handshake shows the Cipher list and CN as plain text, so you can validate that.

BR,

[BeniG]

Interesting, maybe this rings a bell for someone of you all;
After loading the cert in to NNM, the shown SNR is correct, but the fingerprint changed.

[Eugene Kopyshev]

Just a guess from first try - have you downloaded the cer/crt file from/to device by TEXT mode of FTP?

[BeniG]

Downloaded from APIC web-gui (chrome) to Desktop -> Fingerprint still corresponds to each other
Upload from Desktop to NNM -> Fingerprints don't match anymore

[Eugene Kopyshev]

Is there a proxy  between your station and the NNMi?

[BeniG]

Very good idea. I checked it from a server in the same network and same behavior. Also the uploaded cert gets the same fingerprint if I upload from Desktop -> NNM as if from Server -> NNM.

[Misaq]

Might be redundant but removing the old cert would also do to less confuse
the cert validation in the store.

[BeniG]

I did remove the old ones.
Even went as far as checking the DB to make sure they are not stored anymore...

By the way if somebody is interested, those two tables seem to reflect the certs:
select * from nms_node_stored_comm_cert;
select * from nms_reg_stored_comm_cert;

[BeniG]

What seems to work:
1) Remove the full APIC cluster (including leafs & spines)
2) remove the specific node settings for this cluster (not sure if this one is necessary)
3) Restart NNM (ovstop && ovstart)
4) Add specific node settings
5) Add APIC as seed again

Since this is working it points for me to an issue on NNM side. I opened a support case for this.

[BeniG · Accepted solution]

Just to have a full history: I opened a case for this and it was confirmed by support that this is the process to follow:

1. Inventory > Web Agents > Select an agent
2. In “Trusted Certificates” delete the entry
3. Upload the new certificate
4. Once applied for all “Web Agents” restart NNM
5. Check the communication with the APICs.

Which is not really "user-firendly" in my opinion, just to replace a certificate of one monitored node to restart the whole NNM.

I have therefore added this idea to be voted on by the community:

https://community.microfocus.com/t5/NOM-Idea-Exchange/Dynamic-change-of-HTTPS-certificates/idi-p/2850315