Recently I had occasion to spend a lot of time digging into the SSO (Single Sign On) configuration for User Application.

As it happened, I was starting with the IDM 4.5 version of the User Application, which uses the NetIQ OSP (One SSO Provider) framework. This opens an entire other can of worms, since OSP is a very interestingly complex product with all the foibles one would expect out a federation product. However, I will stand firm and say it is an improvement on the previous approach. Which is not a critique of the previous approach, just a comment on the relentless forward march of technology.

You can read about some of my adventures in OSP in the articles:

• 
  Configuring IDM 4.5s OSP to talk to a Shibboleth IDP
  

• 
  Series: Troubleshooting OSP in IDM 4.5
  

Lots of good stuff in there. The good news is, if you encounter an error, I hope if you paste the text into Google, these articles will show up to help you.

A little bit of history of what I am working on will make this article make a bit more sense and foreshadow the ultimate resolution to my problem.

We have a customer that uses IDM, and wants to upgrade to 4.5. But because of database, app server, and some other minimum versions is unwilling to upgrade to the 4.5 User App right now. However, we needed to demonstrate that the 4.5 User App and OSP could do the job.

Thus we needed to demonstrate how to do Shibboleth SSO with OSP, and you can see the results above in the article on that topic. To summarize: Turn off encryption of attributes in the returned SAML message, specify your attributes the way you want to get them, and edit the IDP's metadata since OSP has a bug and uses the Post URL instead of Redirect URL. So you just change the metadata to make them the same. In their defense, OSP and NAM use the same URL for both. Shibboleth uses two different URL's and they did not certify support for Shibboleth. Regardless that is actually an OSP bug that hopefully will be fixed, even though most people will never notice it.

In the end, I found out a lot about how OSP works and how it interacts with User App and SSO applications, so that was a great experience.

Unfortunately in the testing, it looks like something I did broke the SSO in the IDM 4.02 User App. I try to login, I get redirected to the IDP, I successfully authenticate and then I get stuck at a User App login page.

So what did I break, where did I break it, and who can I blame it on? The usual troubleshooting questions we all ask.

The first thing to discuss is how SSO works in 4.02 and 4.5, since the changes are the key here.

In 4.02, User App really only supports SAP Logon Tickets and Kerberos out of the box. That is, there are specific built in and configured connectors for those two authentication systems. Kerberos is meant to reuse the Windows based authentication that got you a Kerberos ticket for the duration of your login session, (Ok gets you a ticket that gets renewed constantly for the duration of your session) so that you get an SSO experience on a Windows workstation.

The documentation alludes to how you could define a custom SSO provider, and talks somewhat about the issue. However you need to dig deeper. I do not know exactly where these guys found it, but somewhere the way to build a custom SSO controller is defined. (PS: If you happen to know, send me the URL I would like to see it. I have only seen the result, not the source it came from).

The docs for 4.02 on the SSO configuration is here.
https://www.netiq.com/documentation/idm402/agpro/data/b6iiw3v.html#bl8utqh

In defense of the docs, if you do exactly what it says, it works most of the time. My complaint is that there is much more to be said here that is not. So let me try and fill in some gaps, and add some contrast, having seen how 4.5 does it now as well.

First up, in IDM 4.5, I noted in my second troubleshooting article ( https://www.netiq.com/communities/cool-solutions/troubleshooting-osp-idm-4-5-part-2/ ) that there was a bit of a red herring in the logs that sent me down a long path that was incorrect for troubleshooting an IDM 4.5 SSO issue.

The error message, just as the User App WAR (IDMProv.war) starts to deploy said in that case:

2015-03-10 11:35:16,804 [localhost-startStop-1] WARN  com.novell.common.auth.saml.AuthTokenGenerator- [RBPM] Failed to initialize SSO: Missing SAML signing cert/key.
2015-03-10 11:35:16,804 [localhost-startStop-1] INFO  com.novell.common.auth.saml.AuthTokenGenerator- [RBPM] SSO Framework is disabled.

At the time I noted that this was because the NMAS SAML method was not installed and configured. In the 4.5 world I concluded that this was only ever used for this functionality, as the osp.war is deployed much later in the process and it is the component that does the SAML authentication to things like NAM or Shibboleth.

However I was wrong. At least in the context of 4.02. This component is what USED to do the SSO for all the methods of all kinds, it would seem (SAP Ticket, Kerberos, Custom, and NMAS SAML) but in 4.5 with the stripping out of SSO and using OSP instead, it only ends up doing the NMAS SAML stuff.

See that article for an explanation of what that NMAS stuff is all about. (Short version: When you login to User App you also need to login to eDirectory for queries and stuff. When you login username/password, those are available to pass through to eDirectory. When you SSO in, no password, so you need to SSO via the NMAS SAML method to eDirectory).

But this was good to work through since it was clear what this error message was about. So I started looking at items I thought related to the NMAS SSO components. The first thing I noted, as in that article was that Trusted Root CA objects, in the Trusted Root Container object, in the Security container at the root of the tree had a name with a CTIME timestamp on it. I knew this was created by the configupdate.sh installer tool, if you turned on Advanced Options, in the Authentication tab, and told it to set it up Auto mode. (Default is No Change).

The file time stamp of the IDMProv.war changing matched when I ran configupdate.sh as well. But that was a 4.02 configupdate.sh which should not have configured this, since in 4.02 it was all manual and you create the objects yourself.

We decompressed the WAR, and yep, our Java class needed for Shib SSO was there. The WEB-INF\web.xml had the correct XML to call our class with the proper keystore and password. So everything looked good.

It looked like all I needed to do was fix the Trusted Root CA object and I should be good. You do that by going to the SAML Assertion.Authorized Login Methods.Security container, and find your authSamlAffiliate object, which has an attribute authsamlTrustedCertDN which is a DN reference to the key. This eDirectory's version of the cacerts file in a JVM, where you can import a Certificate Authority's public key as a trusted CA.

I did that, but I was still getting the issue with corrupted certificates in the server.log from JBoss as this all started with.

Clearly there are more certificates involved here. Where else are they entered? Well in the SSO configuration tab of the UA Admin interface, there are certificates used for the SSO Provider and SSO Consumer. Did these change? Did these need to change? Alas, the interface just says they are Valid, it does not offer any insight to what the specific certificate is about. You know, things that would help identify it, like NotBefore, NotAfter, serial number, fingerprint, or who signed it.

Before we started changing those, we really wanted to know that they were either good or bad. (If good, and we did not have a good copy to put back, we would be breaking something we could not fix). This got us thinking of where it could be stored.

The User App needs a database, but many times I wonder why exactly. As far as I can tell, mostly the state of running Workflows is stored in the DB. It seems like very little else is stored in there.

Surprising amounts of things are stored in eDirectory in the AppConfig container.

The User App Driver itself, has a container underneath it (Comes as part of the UA driver import or Packages) named AppConfig. I recommend spelunking in there with an LDAP browser. It is interesting what you will find.

The User App skinning customizations are stored in there in one of the many containers. The Roles and Resources, the requests for Role Assignments (Since they can have end dates, the request persists) and the Workflow definitions (The PRD's, Provisioning Request Definitions) are all under there.

If you find the object of class srvprvWebAppConfig, named configuration.AppDefs.Appconfig.YourUADriver.YourDriverSet.YourOU,YourO you can look at the XMLData attribute. It is obvious that a lot of the settings in the user interface and in Configupdate.sh are stored here. As I was writing this, I realized I had actually written about this object before.

I wrote about it in this article:
User App Roles Failing to Apply for Administrators

Before 4.01 or 4.02 (I forget where this was fixed) when you first installed the User Application, the first time a web browser would hit the web page it would complete unpacking the WAR which would trigger the assignment of the Admin Roles for UA permissions. If you had a mistake in your RRSD (Roles and Resources Service Driver) configuration you could fail to apply them, after which you cannot manage the UA. To fix it, you had to open this objects XMLData, find the <property> node that looks like this one:

   <property>
      <key>com.novell.idm.security.domain-admin.initialized</key>
      <value>20130408185839Z</value>
   </property>

And clear the value, which is the date it was initialized. In 4.02 and 4.5, it is much easier since in configudpate.sh, if you hit the Show Advanced Options there is a button to reset this value now. Much easier.

But in addition to that particular property, I noticed sever interesting other ones. Here are the set I thought were interesting:

<?xml version="1.0" encoding="UTF-8"?>
<properties>
   <property>
      <key>com.novell.saml.signingcert</key>
      <value>-----BEGIN CERTIFICATE----- 
      MIIEXTCCA0WgAwIBAgICEB0wDQYJKoZIhvcNAQEFBQAwgcUxCzAJBgNVBAYTAlVT
       9fajOSbl4ffQvSsRwyd9bDClwnCrkUaypw1W/kB4Y3NYFwSIL/FXkMsDE0ATB8g 
      enL5nSIMug8qJ2HwUTrXN3s= 
      -----END CERTIFICATE-----</value>
   </property>
   <property>
      <key>com.novell.sso.filters</key>
      <value>kerberos,sap,AcmeNetIQSSO</value>
   </property>
   <property>
   <property>
      <key>com.novell.sso.header.AcmeNetIQSSO</key>
      <value>AcmeNetIQSSO</value>
   </property>
   <property>
      <key>com.novell.sso.enabled.AcmeNetIQSSO</key>
      <value>true</value>
   </property>
   <property>
      <key>com.novell.saml.signingkey</key>
      <value>Nw1p6G9xOLpqwTrePRmGUw==:dGbyKavkZHOFaLcT2G13nf5tPUMrhaqsw55w8LgOpn1VMkJZzuRscvqT Hg58bTa9Vc1NgbWOHoW1L535RF1DNZY aZ5BStxdO4KdStwAehREeE/FoA sLZKruzYSV5y1MAlnYz8vWGPJyUW268H92KYhh9W8fHxmy8hnj5xhplRr2awbVw=</value>
   </property>
   <property>
      <key>com.novell.sso.signingcert.AcmeNetIQSSO</key>
      <value>-----BEGIN CERTIFICATE----- 
      MIIEXTCCA0WgAwIBAgICEB0wDQYJKoZIhvcNAQEFBQAwgcUxCzAJBgNVBAYTAlVT
       9fajOSbl4ffQvSsRwyd9bDClwnCrkUaypw1W/kB4Y3NYFwSIL/FXkMsDE0ATB8g 
      enL5nSIMug8qJ2HwUTrXN3s= 
      -----END CERTIFICATE-----</value>
   </property>
   <property>
      <key>com.novell.pwdmgmt.login.PREF_ENABLE_SSO</key>
      <value>true</value>
   </property>
   <property>
      <key>com.novell.sso.timeinterval.AcmeNetIQSSO</key>
      <value>60</value>
   </property>
   <property>
      <key>com.novell.sso.timeinterval.oauth</key>
      <value>30000</value>
   </property>
   <property>
      <key>com.novell.sso.signingkey.oauth</key>
      <value>09ZpmUg1YBsTfvF95 k6Tg==:uMVTuwDNSvF19QrzVxFggFF3t6DLC2clQDehoHEeT2KcK2k2wgppma3/Vs0aUqd jeW/UBOJbhjCyU8r3zwEEtnHieIvZU4nigZepmugjf3Rq0tgu4csdwG0fsMQjW9QaZ9fiKdouqy9NB2ESMnhOHWscqcrqiuabnjCQi8aZzfaMgKQ2NWnJyzVHovWnhz83UwmMaI6PSpQJQOTsLugGQzD0v4Rg4xxyez1x1gR dTbjLpCuEcDeQA2QDoXB9rhKzovfGinJiClpw8G 86X7pD8wDIHLQGuHxTznlnUWJ3Sg=</value>
   </property>
   <property>
      <key>com.novell.sso.signingcert.oauth</key>
      <value>-----BEGIN CERTIFICATE----- 
      MIIDMzCCAhugAwIBAgIEN0UUijANBgkqhkiG9w0BAQsFADBKMQ0wCwYDVQQGEwRO abcdDRmdS9twV4wiOMlXPtIQ/rQj5vjIyA8z2n9Z4YpuM8rDu5bH9asn7Rur92wl 
      TshxBGW9Iw== -----END CERTIFICATE-----</value>
   </property>
</properties>

The ones of interest to me were first off the three com.novell.saml.signing* values.

com.novell.saml.signingcert
com.novell.saml.signingkey
com.novell.sso.signingcert.AcmeNetIQSSO

These three were clearly where the three certificates are stored in the User App interface. There, you define the public key, com.novell.saml.signingcert, then the private key (with a password) in com.novell.saml.signingkey and then finally the SSO Controller needs its specific public key as well com.novell.sso.signingcert.AcmeNetIQSSO.

The signing key one is obvious when you look at the format of the value. It is 09ZpmUg1YBsTfvF95 k6Tg==:uMVTuwD== (Truncated for brevity). First up, the == signs padding the end indicates it is likely Base 64 encoded. But the colon is the give away that this is two values. The first short one (in real life, my sample is cut to pieces for brevity) is the encrypted password for decrypting the private key (which is in standard PKCS8 format). The rest is the private key itself, base 64 encoded.

The other two signing cert values are public key certificates.

We were able to grab the Base 64 content, open them in a certificate management tool, look at them and realize they were NOT the certificates we were looking for! (Darn you Jedi mind tricks!). This was key as it confirmed the values were wrong. Once we did that, we uploaded the proper ones in the Admin user interface in User App and it started working.

As a useful tip, what I did to see the certs was simply save the text into a file, then try to import it to an empty Java keystore with keytool:
/path/to/jre/keytool -keystore testing -storepass whatever -import -file CertText.b64 -alias signing

You get prompted with the certificate information before you add it to your keystore, which is all I wanted to see.

There are a couple of other settings that make sense as being involved in configuring the SSO which I included above, just to call out.

This one, com.novell.sso.filters, has a list of the filters. Two built in (SAP and Kerb) plus our custom one, AcmeNetIQSSO. Then a couple more that enable our specific Custom SSO provider:

com.novell.sso.header.AcmeNetIQSSO
com.novell.sso.enabled.AcmeNetIQSSO

Then one to enable SSO entirely:

com.novell.pwdmgmt.login.PREF_ENABLE_SSO

What is also interesting is that the oauth certs got added as well. I am pretty sure these are IDM 4.5 values put in by Configupdate.sh, since it is OSP that starts doing oauth:

com.novell.sso.timeinterval.oauth
com.novell.sso.signingkey.oauth
com.novell.sso.signingcert.oauth

All in all, that was very interesting to me, and enlightening as to how this all works under the covers. I wish there was more information on this in the documentation, but I understand how impossible a task that would be for a product as deep and wide as Identity Manager is.