eDirectory comes with PKI (Public Key Infrastructure) components built in. The most obvious manifestation of this is the Certificate Authority that is created on the first server in each tree. This is used to make X.509 Certificates that are used by eDirectory for a number of services. (LDAP with SSL/TLS, iMonitor's HTTP stack for SSL/TLS are the two obvious ones).
However, a CA is supposed to be treated better than we usually do in eDirectory. The honest truth is eDirectory admins do a really poor job of running their CA's. If you ever tried to buy a certificate from a well known CA like Verisign or GeoTrust a sub-CA certificate (My experience is from 10 years ago, so may have muted) they are super security conscious.
That is, they expected a specific hardware device to store the private key on, they had serious rules you had to follow. I recall we once received a device with a cert like this on it, and the security seal was broken, and it apparently was actually wired into the device and it refused to work with the broken seal. Now a days I am sure they are much less strict, but even so they are still stricter than we typically are in eDirectory.
A simple test, if you are an eDirectory Admin. Which server is your CA? Do you even know how to find out? (Look in the Security container of your tree, look for the Organizational CA object and look at the Host Server attribute). Do you back yours up? (If you made your CA in older NMAS/eDirectory versions you can't. Good news is, for maybe 10 years now you can. So only really old trees should have a problem).
Now calling out that Host Server attribute as the identifier, it turns out that often folk move eDirectory servers, replace them, upgrade them or whatever and lose the CA server. Now this is not initially catastrophic, but it has long term problems that need to be fixed.
If you call NTS for support, they can very simply help you recreate your CA and fix your certificates. However a customer of mine recently had this issue, where the server hosting the CA was replaced, but the CA was not properly replaced. They called into Support, and were helped very nicely and quickly in replacing the CA and fixing the certificates in the tree.
The problem is, the particular guy did mention, that anything that consumes these certificates needs to be updated, which is 100% correct and true, but requires insight into what actually consumes those certificates. I am not aware of a really good document on possible things that might be affected, so I figured I would therefore write it. (You know me, cannot stop talking).
So if your CA goes bad, (Who cares why. Recovery, that is a different issue, if you can recover it, great, do so) what can possibly be broken by fixing it?
First up, what certificates are affected by a change in the CA?
SSL CertificateDNS style server certs - for every single server
NMAS SAML certs (More on this later)
Custom Server certificates
Next let's consider what breaks when we do this. In the PKI model, an SSL/TLS (Even though SSL is now deprecated due to structural bugs in the protocol, and TLS only, I will use SSL or TLS interchangeably out of habit) connection actually starts with the asymmetric key exchange using the Public and Private key pairs, which is 'slow' in the grand scheme of things. This is used to secure a connection to decide on a symmetric secret to use for all future communications. This symmetric encryption is much faster.
But that asymmetric exchange requires some modicum of trust. That is, my Private key is signed by someone you trust, therefore you trust my private key. So your keystore of known trusted CA's needs to include my CA. If not, you get an error about no path to Certification or the like. There are many different looking errors possible.
When you change your CA because it was lost, the old certificates continue to work, since no one usually goes back to the CA to confirm the CA (Unless they are using a CRL (Certificate Revocation List) which is deprecated, or OCSP (Online Certificate STatus Protocol) to ensure that this specific cert was not revoked). Thus things will continue UNTIL you actually start using the new certificates.
Thus the fix is just importing the new CA's public key into every keystore that needs to trust it. Now we need to figure out from the list of possible certificates, who might need to trust it.
Let's think about what uses each of these cases, and then what downstream might be affected, as in who might need to trust it.
The SSL CertifcateDNS object is by default used by the HTTP stack for iMonitor, and LDAP for LDAPS services. Generally who cares about iMonitor, unless you have some automation for monitoring using it, in which case, you would have to check if that matters.
To get the HTTP stack for iMonitor to use the new certificate you would need to either reboot the server, or restart eDirectory. So you can prevent that change from happening until you are ready.
For LDAP however, all it takes is a reload or refresh of NLDAP to start using the new certificate. Of course an ndsd restart, or reboot of the server will do the trick. But it turns out that just going to the LDAP configuration pages in iManager and making a change can potentially cause it to refresh and reload. In which case the change will go live, so be careful!
For LDAP, downstream we can have many different things consuming it. All applications using LDAP need to be updated, but let's focus first on NetIQ products. If you use NetIQ Access Manager (NAM) you will likely have to import the new CA certificate, which you can do from a file, you exported in iManager (like you would for an AD remote loader) since you don't want to reload any LDAP servers until you are ready. NAM logs into eDirectory for user info via LDAPS and thus needs to trust. You can keep the old CA cert and the new CA cert in the keystore at the same time since they are really unconnected and do not interfere. This way when you are ready to cut over and restart your LDAP servers you should be good with either cert. There is a nice GUI to do this, so NAM is really the easiest one.
How about if you have any Identity Manager stuff in your environment? Well in the past, this would require that the Keystore used by User App's web server (BEA (Now Oracle) WebLogic, JBoss, WebSphere, Tomcat) would need the CA's public key imported. That is the easy part. Where else?
Well in IDM 4.5 it is not just the User App itself, it is a set of Identity Applications (Dashboard, Landing, Reporting, Catalog Access, Access Review, SSPR) that use OAuth to share authentication info. That OAuth front end is managed by OSP, which has its own keystore. Thus you need to import the CA's public key into the OSP keystore as well. (Location is defined in configupdate.sh on the first tab). This is important, since you would think, the main Java keystore has it, should suffice and not need it in OSP, but that is not quite correct, you need it in both.
But is that it for User App? Well turns out User App also needs to federate the logged in user, who came in via SAML, Kerberos, or username/password and then authenticate them to eDirectory as themselves. This is done by using the eDirectory based NMAS SAML method so that User App can make a SAML assertion to eDirectory itself instead of logging in with a username and password combination.
NetIQ supports login methods for NMAS (Novell Modular Authentication Services) that allow you to login to eDirectory in ways other than just a username password. For example a smart card, a token, or in this case SAML.
This gets confusing, but try to follow a sort of worst case model.
NAM is used with OSP and a user logging in will start with a protected URL, and then get redirected to the NAM login page. This is an eDirectory bind over LDAP (as we discussed above), done by NAM. Now they get their SAML Assertion responded to with success. This tells OSP (Who trusts NAM, and trusts the certificates in use so knows it trustworthy) to trust the user without ever seeing much more than the username.
Now the user wants to do some workflow stuff in User Application. It makes an OAuth request to OSP to ensure this user should be let in. OSP trusts the user because NAM trusts the user, and OSP trusts NAM.
User Application lets them in to its own web interface, but the user needs the ability to make queries to eDirectory as themselves so need a login there. Thus User App makes a SAML Assertion to eDirectory. This is done with security and only because they two sides are preconfigured to trust each other by sharing certificate information.
This last step, requires a couple of components that in IDM 4 needed to be manually configured, and in IDM 4.5 can be done by using Configupdate.sh, Advanced Options, third tab, and the last item in the RBPM section. You only need to do this a single time, which makes a Certificate in the Security container whose public key is stored in the XML-Data attribute of the Configuration object in the AppConfig container. That is, the eDirectory NMAS SAML method knows to use a particular certificate to sign its work. The User App knows to trust that certificate based on the configuration object.
When you change CAs you need to update that certificate and the value in the configuration object as well. Thankfully, with IDM 4.5 this is a piece of cake, and just change the setting from No Change to Auto and apply the change. If you are concerned, you can find the attribute in the object, check the info that the public key has on a cert validation web site, and compare to the certificate in the Security container it makes RBPMTrustedRootXXXXXXXXX where XXXXXXXXXXX is a timestamp is CTIME of when you created it to guarantee no naming collisions).
SSPR is another LDAP consumer that can import the trusted root in the configuration and would need to be updated but it is trickier, since it imports it from the tree live, as opposed to accepting a file. You would probably need to do a mixed cut over for this. Best bet would be to have one eDirectory server reloaded with the new CA based certificates, and use that to import into apps, and then keep the active LDAP servers on the old cert until you are ready to failover.
This is the same approach you would need for all your LDAP consumers, of which there are likely many.
Remote Loaders that are secured via LDAP usually need to know the trusted CA's public key, and usually have just that public key in a single file. These would need to be updated. The good news is you can change the file it uses to the new one, and on next restart of the RL it will read it. This just becomes a timing issue of restarting services for minimal outages.
Custom Server Certificates are often used for Remote Loaders, since the 2 year default span is too short, and it is nicer to make a 10 year certificate and just never worry about it again for a decade. Those do NOT have to be reminted, per se, since they will continue to work even if the CA is dead, unless you go and recreate them under the new CA. In general it would be better to recreate these as well to stay consistent. You might have switched your LDAP Server to use these instead, for the same reason of longer time span, in which case, same issues apply.
User Certificates will all need to eventually be reminted if you are using them, which is why it is so crucial not to lose the CA when using it to tasks like this.
All in all, what seems like a simple fix of:
Delete the broken CA object
Create a new one
Run Repair Default Certificates on all servers
is really just the very beginning of a long process of changes that are needed.