In the previous article on OSP I discussed the certificates needed, and the order of install. Since this is critically important to get OSP working let me repeat quickly here.

There are two keystores involved, the general Java stack (/opt/netiq/idm/apps/jre/lib/security/cacerts) and the OSP keystore. In principle you could have a third for Tomcat, but I like using just two. The private key that OSP and Tomcat use (Could be the same, see previous article for why you would decide either way) need to be in the OSP keystore. But it also needs the public key of eDirectory, (and if using SAML the IDP's public key). The OSP keystore can be wherever you want, but I like to store it in /opt/netiq/idm/apps/tomcat/conf personally.

The general cacerts keystore needs to have the public key of the certificate chain for the OSP and Tomcat certs. This is also important. It is not enough to trust just the CA that signed it, as a Browser would require, but also to trust explicitly any intermediate certificates that signed it as well.

In the good old days of yore, the well known CAs (Certificate Authorities) like Verisign had but a single CA. Then at some point when I was not looking they started using a CA that signed a sub CA (possibly many sub CA's) which then signed certificates. I am sure there is a very good security reason for this but I am not aware of it. If you happen to know, please leave me a comment, since I am curious as to why they do this.

If you look in your web browser at the certificate store, you will see many many CA's listed. In some ways, this is Mozilla's business model for Firefox. To get your certificate included in the default shipping Firefox used to cost many millions of dollars (I have no idea what it costs today). If you want to sell SSL certificates the major browsers had better recognize you or else there is no value in your certificates.

However you will notice that while the parent CA's are all listed, the many intermediate CAs are often not. Yet the browser trusts the parent of the chain and that suffices for web based SSL security. Java on the other hand is not so accommodating. The JRE that Oracle ships comes with 80 or so well known CA public keys set to be trusted out of the box in the cacerts file, however few if any of the intermediate certificates are in there.

Thus a common issue getting OSP to work is that you have the parent CA trusted in the OSP and cacerts keystores, but you are still get a Certification path issue. In that case, check and see if there is another certificate in the chain that you do not have trusted.

You can look at the contents of a keystore with a command something like this:
/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security -storepass changeit -list

In my JRE 18.0_73 shipping I see 93 entries that look like this:

affirmtrustpremiumeccca, Apr 14, 2014, trustedCertEntry,
Certificate fingerprint (SHA1): B8:23:6B:00:2F:1D:16:86:53:01:55:6C:11:A4:37:CA:EB:FF:C3:BB
starfieldservicesrootg2ca, Jul 18, 2014, trustedCertEntry,
Certificate fingerprint (SHA1): 92:5A:8F:8D:2C:6D:04:E0:66:5F:59:6A:FF:22:D8:63:E8:25:6F:3F

and so on.

This will show you the alias, and a checksum for each key stored. This is nice to see a quick list, but you really need verbose mode (-v switch) to see it. Use a command such as:

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security -storepass changeit -list

I recommend piping it to less as it generates dozens of pages of content.

This is much more verbose and looks like (grabbing just the last one):

*******************************************
*******************************************


Alias name: geotrustprimarycag2
Creation date: Dec 10, 2009
Entry type: trustedCertEntry

Owner: CN=GeoTrust Primary Certification Authority - G2, OU=(c) 2007 GeoTrust Inc. - For authorized use only, O=GeoTrust Inc., C=US
Issuer: CN=GeoTrust Primary Certification Authority - G2, OU=(c) 2007 GeoTrust Inc. - For authorized use only, O=GeoTrust Inc., C=US
Serial number: 3cb2f4480a00e2feeb243b5e603ec36b
Valid from: Sun Nov 04 19:00:00 EST 2007 until: Mon Jan 18 18:59:59 EST 2038
Certificate fingerprints:
         MD5:  01:5E:D8:6B:BD:6F:3D:8E:A1:31:F8:12:E0:98:73:6A
         SHA1: 8D:17:84:D5:37:F3:03:7D:EC:70:FE:57:8B:51:9A:99:E6:10:D7:B0
         SHA256: 5E:DB:7A:C4:3B:82:A0:6A:87:61:E8:D7:BE:49:79:EB:F2:61:1F:7D:D7:9B:F9:1C:1C:6B:56:6A:21:9E:D7:66
         Signature algorithm name: SHA384withECDSA
         Version: 3

Extensions:

#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#2: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
  Crl_Sign
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 15 5F 35 57 51 55 FB 25   B2 AD 03 69 FC 01 A3 FA  ._5WQU.%...i....
0010: BE 11 55 D5                                        ..U.
]
]

A quick primer on keytool first. Start it off this way:

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security -storepass changeit

That is, full path to keytool, so you do not get a keytool executable from Java 1.6 that is in the path instead of the Java 1.8 or 1.7 you desire, and then the -keystore switch followed by -storepass. Now you can reuse this command by changing the end of it, appending or changing the command. For odd reasons most examples I see do the command followed by the keystore info, which is inefficient if you are using command line caching (up arrow to get previous command as Windows and Linux provide).

The high level major commands of interest are:
-list (Show me the certs)
-import (bring in a cert)
-export (bring out a cert to import elsewhere)
-delete (remove a cert)

Use the -help option to see the copious options. (Like -certreq, -keygen, -storepasswd)

There are several helpful things here. The Alias name is how you address this entry with the keytool command, is the target of the -alias switch. (So to delete it would be the base keytool with keystore and password followed by -delete -alias geotrustprimarycag2)

Next you have an Entry Type - This will be trustedCertEntry (if imported with the -trustcacerts switch) or possibly PrivateKeyEntry for your actual Private Key to be used by Tomcat and OSP. I am sure there are more but these are the only two I ever use.

The Issuer line tells you who signed it. If there is a parent CA, that would be shown here. A self signed certificate will own itself. So the subject name and Issuer would be identical.

I usually use the Valid From/To dates, and the Serial Number to be sure I have the exact correct certificate since some of the names are mind numbingly close to each other. There is interesting information in the Extensions like Subject Alternate Names but that is the main one I know about. The rest I usually ignore. I find the MD5, SHA1, and SHA256 checksums not so helpful, since in a browser when you click on the lock icon on a SSL secured web page, I find the serial number much easier to find.

The Valid To date is important, since certificates expire, and when a CA or intermediate CA expires it can cause pain all over the place. In theory a nice SSL vendor would warn you when one of your certs issued by their CA is about to have its CA expire, but that is not always the case.

Having said all that, when you are done, you should be able to list your OSP/Tomcat keystore, and see at least the following entries:

1. trustedCertEntry for the eDirectory trees CA that is used for LDAP operations.

• trustedCertEntry for the primary CA that signed your private key for Tomcat/OSP.
  
  
  
  1. Possibly the Intermediate CA's as well.

• PrivateKeyEntry for the private key Tomcat/OSP is using. (One or two).

When you look at your cacerts keystore it should have in addition to the 90 it comes with the following:

1. trustedCertEntry for the eDirectory trees CA that is used for LDAP operations.

• trustedCertEntry for the primary CA that signed your private key for Tomcat/OSP.
  
  
  
  1. Possibly the Intermediate CA's as well.
  
  
• If your OSP cert is self signed, then the signer is the public key of the certificate.

One of the common mistakes I see people make when configuring LDAP applications to use SSL, is that they end up trusting the public key of the certificate itself (but that expires in usually 1-2 years) instead of trusting the chain of CA's which often are set to expire in 10-30 years. This means every year or two you need to redo the certificate import, which is probably not what you want. You might e tracking when your LDAP certificate expires in eDirectory and renewing as needed but your LDAP application which knows next to nothing about LDAP is likely not monitoring the expiration date so that when you, being the diligent admin you are, renew your certificate early, their LDAP application will suddenly stop working. They may not be aware of this issue to troubleshoot, and the blame casting will begin. Thus the simplest thing is to convince them to trust the chain, not the certificate itself.

To import one of these certificates you need to get it out somehow. You can start by having Tomcat use the private key, and then hit it with a web browser, so long as it renders an HTTPS connection, you can click on the lock icon in a web browser look at the certificate and then in the Details page usually export it. Firefox makes this very easy as in the Details, Certificate Hierarchy tab you can click on each step of the chain, and export them one by one. Often the CA will give you public keys of the CA as part of the certificate in a file.

There are two file formats for certificates. PEM and CER. One is binary and the other is the same thing, base64 encoded with a header line of:

-----BEGIN CERTIFICATE------
body of the base64 stuff
-----END CERTIFICATE------

Either works with keytool. To import, you can do can do a command like:

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security -storepass changeit -import -trustcacerts -alias osp-cert -file /path/to/file

You can break that up as the basic keytool specifying the target as:

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security -storepass changeit 

followed by the actual command:

-import -trustcacerts -alias osp-cert -file /path/to/file

Here we have the following switches:

-import

This tells keytool that we want add a key into the keystore.

-trustcacerts

This tells keytool to mark it as a Trusted CA certificate so that it can be used the way we desire.

-alias osp-cert

Alias is the nickname you use to address it to add it, export it, delete it and so forth. In terms of private keys this is name you tell Tomcat to point it at the proper private key to use. Oddly, even if you import it with case changes, it gets reset to all lower case which makes it harder to read some times.

-file /path/to/file

Finally you need a file to provide the input. You could of course dump from STDIN if you wanted to do it all in one command being the Unix maestro we all like to think we are.

One thing you will have to do, after you decide how to make your Tomcat and OSP certs, is possibly export their self signed public key, if you self signed either of them. In that case the command looks similar, but now instead of -import, you use -export.

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security -storepass changeit -export -alias osp-cert -file /path/to/file

This still uses -alias to tell keytool which of the 90 or more certificates you want, you specify a path to write it to, and you do not need the -trustcacerts switch anymore.

If this is all confusing the heck out of you, there are actually a couple of GUI tools to help. Oddly enough IBM has a really nice GUI tool I downloaded years ago and cannot find a good link for now. There is a nice open source one called portecle available here:
http://portecle.sourceforge.net/

So having said all that, the confusing parts are, you need:

• A private key for Tomcat and OSP. They can be the same or two different keys.

• Both keystores (OSP and cacerts) should trust all the certs involved.
  eDirectory. Tomcat HTTPS cert. OSP HTTPS Cert. SAML IDP cert.

• The URL is critical to OSP working properly so know your DNS in advance, make sure your Tomcat cert has the proper Subject Name to match DNS.

I keep writing about this topic in the hopes that one of my attempts is easier to understand for someone who is finding this complicated and confusing.