As I worked through a new Identity Manager 4.5 setup I was trying a number of things and ran across some interesting errors. This series of articles is meant to share what I found, hopefully with error codes for the Google cache. This way, if you run into this error on your own, you might never have read this article but I hope that Google directs you here. I have done a number of articles like this for a variety of drivers and other bits of the IDM infrastructure. Let me know if any of these errors happen to you and this helps out. Always nice to hear back that something has helped.

In my case, I was using Shibboleth as the IDP (Identity Provider) with OSP (One SSO Provider) that comes with IDM as the SP (Service Provider). That is, OSP would get credential info from the Shibboleth IDP, and then allow the approved user to connect.

But then to complicate things I was going to have several SSPR (Self Service Password Reset) servers and several Identity Application servers (User App, IDM Home, IDM Dash, Catalog Administrator, and Reporting). In that model, how many OSP installations do you need? What if you want to hide some of these behind a virtual IP load balanced? Lots of interesting questions to ask and answer there. I have this mostly done and will report back on what I found in a later series of articles. I have it working with 6 servers relying on OSP but only 2 running the OSP WAR file. Next up is to switch to using the virtual IPs and testing it behind the load balancer.

Along the way, I thought I got it all working and I went to test SSPR. I went to https://server.com:8443/sspr and the following process happened.

I got redirected from /sspr to the URL osp/a/idm/auth/ and saw the OSP splash page. Then I got redirected to the login page specified in the SAML Metadata. I entered my credentials, that generated a SAML request from the OSP SP to the Shibboleth IDP and then the IDP sent back a SAML response and I was redirected back to the /sspr URL on my originating server.

Now SSPR rendered a page, but showed an error about "peer not authenticated javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated"

Well off we go to the catalina.out (SSPR only runs in Tomcat I think, so only Tomcat log is relevant here), which if you did the default install should be /opt/netiq/idm/apps/tomcat/logs/catalina.out and we see the full error here:

2015-04-01 14:01:40,199 [http-bio-8443-exec-18] DEBUG password.pwm.AuthenticationFilter- {y} redirecting user to oauth id server, url: https://ssprSrv.acme.com:8443/osp/a/idm/auth/oauth2/grant?response_type=code&redirect_uri=https://ssprSrv.acme.com:8443/sspr/public/oauth&state=D9hIBGGYGDqQQ75OGT2T6xSHzSCJ4POs14c7625514f&client_id=sspr [10.110.38.159/vpnpool038159.vpn.acme.com]
2015-04-01 14:03:42,449 [http-bio-8443-exec-22] TRACE password.pwm.servlet.OAuthConsumerServlet- {y} received code from oauth server: eHwAIEvVae2pxHHhW2Q7FyiAZcAi8J9Nc76J@noQFsgDuPvIavqvkjDFuQGhTAHc1gF5pFtFK8eoUEP2zed@vCBHysIQyFZUvgz5tyXY@POs9YL1fQ9IHDTFGfOzYqQqgRTqExD39Nhx4lz7rvA4n6b9Jgz0e@ny60@mV3/FYuKOSd02JHdr6sUAJPqO@WPKT17BDWuPvTjBTK1njoJWrDvgPM2INBN1/Isv9bI142Fwj0qNN3bljKwDGtTvU9SiN7@WYfayDfbiNoGT/eeWJEGXqNS6twsYHovMbRhBK/GNpsYn8LcpBgW/Tsg29Y5X4rFmaI5rQT1GMenu5qU4vV5AvXw~ [10.110.38.159/vpnpool038159.vpn.acme.com]
2015-04-01 14:03:42,451 [http-bio-8443-exec-22] TRACE password.pwm.servlet.OAuthConsumerServlet- {y} beginning OAuth coderesolver request to https://ssprSrv.acme.com:8443/osp/a/idm/auth/oauth2/authcoderesolve, body:  [10.110.38.159/vpnpool038159.vpn.acme.com]grant_type=authorization_code&redirect_uri=https://ssprSrv.acme.com:8443/sspr/public/oauth&code=eHwAIEvVae2pxHHhW2Q7FyiAZcAi8J9Nc76J@noQFsgDuPvIavqvkjDFuQGhTAHc1gF5pFtFK8eoUEP2zed@vCBHysIQyFZUvgz5tyXY@POs9YL1fQ9IHDTFGfOzYqQqgRTqExD39Nhx4lz7rvA4n6b9Jgz0e@ny60@mV3/FYuKOSd02JHdr6sUAJPqO@WPKT17BDWuPvTjBTK1njoJWrDvgPM2INBN1/Isv9bI142Fwj0qNN3bljKwDGtTvU9SiN7@WYfayDfbiNoGT/eeWJEGXqNS6twsYHovMbRhBK/GNpsYn8LcpBgW/Tsg29Y5X4rFmaI5rQT1GMenu5qU4vV5AvXw~
2015-04-01T14:03:42Z, FATAL, servlet.TopServlet, 5015 ERROR_UNKNOWN (unexpected error processing request: peer not authenticated
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
        at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at password.pwm.servlet.OAuthConsumerServlet.makeOAuthResolveRequest(OAuthConsumerServlet.java:192)
        at password.pwm.servlet.OAuthConsumerServlet.processRequest(OAuthConsumerServlet.java:111)
        at password.pwm.servlet.TopServlet.handleRequest(TopServlet.java:104)
        at password.pwm.servlet.TopServlet.doGet(TopServlet.java:63)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at password.pwm.ApplicationModeFilter.doFilter(ApplicationModeFilter.java:71)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)
)

Well that is pretty clear that something is wrong with Certificates used to secure the connections with SSL. I decided to make my own keystore to hold the specific certificates.

This is worth discussing what you need in terms of certificates. You actually need one or two private keys. You can do the whole thing with the certificate that the OSP install makes for itself. Looking through the install logs I found that command and thought it was worth looking at:

/opt/netiq/idm/apps/jre/bin/keytool -genkey -keyalg RSA -keysize 2048 -keystore "/opt/netiq/idm/apps/osp_sspr/osp/osp.jks" -storepass $NETIQ_OSP_KEYSTORE_PWD$ -keypass $NETIQ_OSP_KEYSTORE_PWD$ -alias osp -validity 730 -dname "cn=idmServdev3.cc.acme.edu"

It uses the Java keytool program, which as you work on this will become your friend. Your very good friend! I must have run keytool literally one hundred times in the process of this. (I guess I could check on the boxes I worked on, via the history command... For the fun of it, I would execute 'history | grep keytool | wc -l' to see how many times I used it in my last 1000 commands).

Ok, so lets dissect that command since it looks like a mishmash of stuff. Let me simplify and reformat that to make it easier to read:

$PATH/keytool
	-genkey 
	-keyalg RSA 
	-keysize 2048 
	-keystore "/opt/netiq/idm/apps/osp_sspr/osp/osp.jks" 
	-storepass $NETIQ_OSP_KEYSTORE_PWD$ 
	-keypass $NETIQ_OSP_KEYSTORE_PWD$ 
	-alias osp 
	-validity 730 
	-dname "cn=idmServdev3.acme.com"

Well that is easier. Put it in key generation mode, use the RSA algorithm, make it a 2048 bit key in size, write this out to a specific keystore file path, pass in the password for the keystore. Next is keypass, which is for the Private key, as the key itself gets a password inside the keystore. Note that they use a variable to avoid storing it in the log file. The alias is important, since that is how you address a specific key. In principle you can name it anything you want, but I have heard hints (Not confirmed, but strongly suspected) that there may be hard coded spots where the key name is assumed. Who cares, it is easy enough to leave it as osp. Validity is interesting, and is in units of days, so that is 2 years. For a self signed certificate it seems like 2 years is a bit low. Finally, the specific distinguished name (-dname) of just the server name.

Now you could in principle use this in your Tomcat configuration as well as the SSL key. But this would be an untrusted certificate.

This is the certificate that the OSP SP will use, and publish the public key in it's metadata for communication with the IDP. The IDP has to be set to explicitly allow the federation request, so importing the public key of your self signed certificate is not even frowned upon, it is almost expected.

I am told by the IDP guys that they like using 20 year certificates, since they do not want to deal with this ever again. So you could steal that command line, and change validity to something like 7300 days instead to get what you need.

Next up you probably want a second certificate for use as the front end web page over SSL. In that case, it is worthwhile to get a proper certificate signed by a well known Certificate Authority. This way web pages will trust it and not give you a hard time. In that case, you could generate the private key as above, then use the -keyreq command to generate a CSR (Certificate Signing Request) which is sent to the Certificate Authority, returned, and imported back into the keystore. (See my article on getting OSP working with Shibboleth with some more details.)

Thus I had a keystore, I called mine tomcat.keystore instead of osp.jks, with two private keys. One aliased osp, basically as above but with a 20 year validity. The second I aliased as tomcat, and got signed by a well known CA and configured Tomcat to use.

In my JVM's cacerts file (/opt/netiq/idm/apps/jre/lib/security/cacerts) I imported the following certificates:

• Identity Vault CA's public key (Used the -trustcacerts switch, this is critical)

• IDP's Public key - since it was also a 20 year self signed cert

• Well known CA's public keys - turns out I was missing one out of the box with JVM.

That made sense, I was sure I had it all done correctly. So going back to the error (see, I did have a reason for that diversion) I was being told that the SSL connection from SSPR to something is not trusted. So I was clearly missing a certificate in a keystore somewhere.

I assumed, that since I had the private key for my osp and tomcat certs that I did not need to reimport them as -trustcacerts but I tried anyway. No joy. Finally I gave up and looked in the Google. What do you know? There is a TID on this issue:

https://www.netiq.com/support/kb/doc.php?id=7015979

Basically it says that I need the osp/tomcat certificates public keys in cacerts as trustcacerts. I had the right idea, make sure the private keys are trusted, but I had the wrong keystore.

This sort of makes sense as you think about how this works. OSP runs in Tomcat, as a WAR file, but it calls into its own keystore specifically to get the private keys it needs. Tomcat is running, using the JVM to provide the default set of trusted certificates in the cacerts file. It too reads its specific private key from the keystore we made, but I guess it does not add it to the list of keystores that it loads at startup in its entirety.

Well it was easy enough to do, using commands like:

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/tomcat/conf/tomcat.keystore -storepass password -export -alias osp -file /tmp/osp.public

This specifies the keystore as the one I talked about above, (put in a real password of course), using the -export switch. Then you need to specify the -alias you are interested in, and finally an output file.

Then import it back into the cacerts keystore:

/opt/netiq/idm/apps/jre/bin/keytool -keystore /opt/netiq/idm/apps/jre/lib/security/cacerts -storepass changeit -importcert -trustcacerts -alias osp-pub-key -file /tmp/osp.public

This specifies the cacerts keystore, which has a well known password of 'changeit', seriously. (If it is not changeit try default). Then we import the certificate we exported, but the key is, using the -trustcacerts switch, since that marks it as a trusted key.

Once I did that, restarted the Tomcat instance and all was good. That got me into SSPR again.

At one point in that process I wondered if I had a configuration setting I needed to change, so I was working through the SSPR configuration page. I saved some mostly cosmetic change and instead of the web page returning to the Administration page as it normally did, it stayed on the same page. That did not feel quite right. I did not think much of it, but I went out, came back, and my change was lost.

As usual, when in doubt look in the trace file for errors or messages. So off to catalina.out I went. Darn I thought I had recorded the error but I seem to have lost it. But basically it said that the file could not be written to, with a nice long Java trace, as it seems is always the case. Thus this looks like a permissions issue. But why would SSPR not have permissions to write to the configuration file? Don't you normally manage it via the web interface?

Well, that is how it is supposed to be done, but what had happened was that I tried to back up the SSPRConfiguration.xml before trying to patch and reinstall the SSPR war file. IDM 4.5 ships with SSPR 3.2 but since its release there have been two hot patches and you probably want to upgrade to it. Now you can use SSPR to back up the file via the web interface, but hey, who uses a web interface. We use command lines!

So I went to /opt/netiq/idm/apps/tomcat/webapps/sspr/WEB-INF and grabbed the SSPRConfiguration.xml file from there to back it up.

I deleted the /opt/netiq/idm/apps/tomcat/webapps/sspr.war file, and the /opt/netiq/idm/apps/tomcat/webapps/sspr directory. Then I copied in the new patched sspr.war and restarted Tomcat. This unpacks the WAR again, and actually starts WITHOUT a config file. You can then use the Web interface's wizard to configure SSPR. I thought I did it right, but I was wrong. So I restored the config file by copying it back. Of course, I was sudo'ed to root as I copied it, and Tomcat on IDM runs as the novlua user. So of course SSPR could not write to the file by default.

Easy enough to fix, went into the /opt/netiq/idm/apps/tomcat/webapps/sspr directory and issued the command 'chown novlua:novlua SSPRConfiguration.xml' which did the trick.

Of course what I SHOULD have done was used the Web interface to restore the config file so that Tomcat would have handled writing it back to the proper directory with proper permissions. Ah well, live and learn. I suppose I should have read the manual, but we don't need no steenkin manuals!

Anyway, hope you found these errors helpful. I have more, alas, there are always more. Stay tuned for more such articles. If you happen to find errors like this please consider writing an article like this explaining what the issue is and how you fixed it.