Recently I was working on a SOAP driver that connects to Salesforce.com to synchronize objects on both channels.

You can read more about how you might go about building such a driver for yourself in this series of articles I am working on writing:

• Getting Started Building a SOAP Driver for IDM - Part 1

• Getting Started Building a SOAP Driver for IDM - Part 2

• Getting Started Building a SOAP Driver for IDM - Part 3

• Getting Started Building a SOAP Driver for IDM - Part 4

Well things were running fine, and our first development lab had been left alone for a week or three and when I went to go get a snippet of trace to show the guys building the next development version of the lab from scratch, I found the following error:

<nds dtdversion="2.0">
  <source>
    <product build="201006032211 Internal Novell build. Not for production use." instance="SOAP-SPML" version="3.5.5">Identity Manager Driver for SOAP</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <status level="error" type="app-general" value1="IOException">
      <description>sun.security.validator.ValidatorException: No trusted certificate found</description>
    </status>
  </output>
</nds>

Oh dear. That cannot be a good thing. As usual with SSL and certificates, the issue is probably more complex than it needs to be, and is confusing as all get out in general. So I took detailed notes as I went through the process of troubleshooting and resolving this issue, and let me walk through the process I took, so that while this may not solve the specific issue you have, the steps I took along the way might serve as a hint to get you pointed in the right direction to resolve your specific problem.

Well first thing to do when looking at such a problem, lets see if there is any more data in the remote loader trace. Oh ya, did I mention that the driver is running with a remote loader instance? Details, details.

If you were running the SOAP driver in the engine, then this specific trace and error would be in the engine trace, but since we are using a remote loader, it is in the remote loader trace.

If you do not know or understand how to read trace for Identity Manager, then you absolutely should read this amazing series on the topic by Fernando Frietas, who works in Novell Technical Support:

• Capturing and Reading Novell Identity Manager Traces

• Comprehending IDM Traces - Part 1

• Comprehending IDM Traces - Part 2

Remote loader trace shows:

DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML: HTTPSubscriberTransport.send()
DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML: Preparing HTTP POST connection to https://tapp0-api.salesforce.com/services/Soap/c/18.0/
DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML: Setting up SSL connection.
DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML: Setting the following HTTP request properties: 
DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML:   Authorization: <credentials suppressed>
DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML:   SOAPAction: #batchRequest
DirXML: [09/08/10 12:55:07.64]: TRACE:  SOAP-SPML:   Content-Type: text/xml; charset=utf-8
DirXML: [09/08/10 12:55:07.80]: TRACE:  Remote Loader: SubscriptionShim.execute() returned:
DirXML: [09/08/10 12:55:07.80]: TRACE:  <nds dtdversion="2.0">
        <source>
                <product build="201006032211 Internal Novell build. Not for production use." instance="SOAP-SPML" version="3.5.5">Identity Manager Driver for SOAP</product>
                <contact>Novell, Inc.</contact>
        </source>
        <output>
                <status level="error" type="app-general" value1="IOException">
                        <description>sun.security.validator.ValidatorException: No trusted certificate found</description>
                </status>
        </output>
</nds>

Hmm, no trusted certificate. Ok, so why would this be? First thing, lets go look at the certificates themselves.

SOAP is usually transported over HTTP, which means that I can look at the URL it is trying to connect too, https://tapp0-api.salesforce.com and see what my browser says about the certificates.

Looking at it in Firefox 3.6.x, and I see there are three certificates in the certification chain. There is a parent Verisign certificate, an intermediate certificate, and a newly minted site certificate for tapp0-api.salesforce.com

Ahah! SFDC Changed their tapp0-appi.salesforce.com cert on Aug 19.

Well found the root cause. Lets go get a coffee. No wait, how do we fix this?

Well if there is no trusted certificate, that means the JVM in use, does not trust the signing certificate. Well what keystore is the JVM using? Well I have one sitting in the engine directory, so let me try and look at that with keytool, the tool to examine keystore files. Now depending on your server, this might not be in path. You want to use the keytool from the same JVM instance that Identity Manager is using, so that there are no version or compatibility issues. Well the IDM engine stores it in a path that looks like this on a 64 bit eDirectory / 64 bit IDM install:
/opt/novell/eDirectory/lib64/nds-modules/jre/bin/keytool

On a 32 bit eDirectory / 32 bit IDM install the path would be more like:

/opt/novell/eDirectory/lib/nds-modules/jre/bin/keytool

Note the difference of lib and lib64 being the key difference.

Lets use the -list switch and -v for verbose and we see:

pesaidvv1m-acme:/opt/novell/eDirectory # lib64/nds-modules/jre/bin/keytool -list -v -keystore sslkey.keystore 
keytool error: java.io.IOException: Invalid keystore format
java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source)
        at java.security.KeyStore.load(Unknown Source)
        at sun.security.tools.KeyTool.doCommands(Unknown Source)
        at sun.security.tools.KeyTool.run(Unknown Source)
        at sun.security.tools.KeyTool.main(Unknown Source)
		
		

Ok, that keystore is clearly not the right one. Maybe it is the default Certificate store in the JVM? This is the cacerts file, in the JVM's parent directory then lib/security. Default password is usually 'changeit' or 'default' which is good to know.

On a 64 bit eDirectory box this would:

/opt/novell/eDirectory/lib64/jre/lib/security/cacerts

On a 32 bit eDirectory box it would be:

/opt/novell/eDirectory/lib64/jre/lib/security/cacerts

So the command on 64 bit would be:

/opt/novell/eDirectory/lib64/nds-modules/jre/bin/keytool -list -v -keystore
/opt/novell/eDirectory/lib64/jre/lib/security/cacerts

By now you should be able to figure out how to change that for 32 bit eDirectory.

Now there are 51 trusted root certificates in that file, so I outputted it to a file, and searched it for the serial number of the parent CA that I see in Firefox, and I see it there:

Here is the entry:

Alias name: verisignclass3ca
Creation date: Oct 27, 2003
Entry type: trustedCertEntry

Owner: OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
Issuer: OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
Serial number: 70bae41d10d92934b638ca7b03ccbabf
Valid from: Sun Jan 28 19:00:00 EST 1996 until: Tue Aug 01 19:59:59 EDT 2028
Certificate fingerprints:
         MD5:  10:FC:63:5D:F6:26:3E:0D:F3:25:BE:5F:79:CD:67:67
         SHA1: 74:2C:31:92:E6:07:E4:24:EB:45:49:54:2B:E1:BB:C5:3E:61:74:E2
         Signature algorithm name: MD2withRSA
         Version: 1
		 

From the certificate view in Firefox, I see the Serial Number is: 70:BA:E4:1D:10:D9:29:34:B6:38:CA:7B:03:CC:BA:BF which is the same as in the cacerts keystore file.

Ok, so we have the right parent root CA public key in our list of trusted stores. Which makes sense as we all trust VeriSign right?

So do we need the intermediate cert? Which is not currently in the cacerts file.

The intermediate cert is Verisign, Inc, with SN of: 25:4B:8A:85:38:42:CC:E3:58:F8:C5:DD:AE:22:6E:A4 and I do not see it in the cacerts file.

Before I go too much further, I noticed that my Remote Loader trace file had hit 2 GB in size, and I have run into issued where funny things happen when the log gets to full size, so I tried restarting the engine and the remote loader (after archiving the log file). But still no joy.

As long as I was on that external factor tack, I decided to test using soapUI, which I have setup with the WSDL from Salesforce.com and I used to test a lot of the SOAP stuff I needed to do. soapUI on a successful bind says this segment below for certificate info on a successful connection. I note that it does not have the three part chain, just the bottom 2, ignoring the parent Verisign cert that we already have in cacerts, rather it is the intermediate cert in the chain that Firefox shows

CipherSuite:
SSL_RSA_WITH_RC4_128_MD5
PeerPrincipal:
CN=test.salesforce.com, OU=Applications, O="Salesforce.com, Inc.", L=San Francisco, ST=California, C=US
Peer Certificate 1:
[
[
  Version: V3
  Subject: CN=test.salesforce.com, OU=Applications, O="Salesforce.com, Inc.", L=San Francisco, ST=California, C=US
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 148815233497639905907595162608493812428698497647343330754716554598875800947042848899507335288025469339414468361692946268172622245520434908773819231005966708315077298460673060882925226915809595744101295687822865656659367039780770370228217722826804480494167826426037300362636030164743330815161830290382513075353
  public exponent: 65537
  Validity: [From: Thu Aug 19 20:00:00 EDT 2010,
               To: Fri Sep 07 19:59:59 EDT 2012]
  Issuer: OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign, OU=VeriSign International Server CA - Class 3, OU="VeriSign, Inc.", O=VeriSign Trust Network
  SerialNumber: [    31df8a5a 0b2c1519 9a68f0ff 866926d5]

Certificate Extensions: 7
[1]: ObjectId: 1.3.6.1.5.5.7.1.12 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 62 30 60 A1 5E A0 5C   30 5A 30 58 30 56 16 09  .b0`.^.\0Z0X0V..
0010: 69 6D 61 67 65 2F 67 69   66 30 21 30 1F 30 07 06  image/gif0!0.0..
0020: 05 2B 0E 03 02 1A 04 14   4B 6B B9 28 96 06 0C BB  . ......Kk.(....
0030: D0 52 38 9B 29 AC 4B 07   8B 21 05 18 30 26 16 24  .R8.).K..!..0&.$
0040: 68 74 74 70 3A 2F 2F 6C   6F 67 6F 2E 76 65 72 69  http://logo.veri
0050: 73 69 67 6E 2E 63 6F 6D   2F 76 73 6C 6F 67 6F 31  sign.com/vslogo1
0060: 2E 67 69 66                                        .gif


[2]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://SVRIntl-crl.verisign.com/SVRIntl.crl]
]]

[3]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  2.16.840.1.113730.4.1
  serverAuth
  clientAuth
]

[4]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.16.840.1.113733.1.7.23.3]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 1C 68 74 74 70 73 3A   2F 2F 77 77 77 2E 76 65  ..https://www.ve
0010: 72 69 73 69 67 6E 2E 63   6F 6D 2F 72 70 61        risign.com/rpa

]]  ]
]

[5]: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

[6]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: 1.3.6.1.5.5.7.48.1
   accessLocation: URIName: http://ocsp.verisign.com, 
   accessMethod: 1.3.6.1.5.5.7.48.2
   accessLocation: URIName: http://SVRIntl-aia.verisign.com/SVRIntl-aia.cer]
]

[7]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 1A 99 BF 3F 63 67 68 F8   E7 FC 90 84 7C C6 75 EF  ...?cgh.......u.
0010: C1 6D 05 30 BC 4B 54 CA   5C A3 7F 97 3D FD B3 E7  .m.0.KT.\...=...
0020: 11 DB E6 9F CA A6 C3 76   3F 69 31 4D B9 F8 A6 0A  .......v?i1M....
0030: 7D A7 1A 1D 0A 68 74 F9   D4 49 93 E0 85 DF 08 EB  .....ht..I......
0040: 0E 77 8F B8 84 B8 C1 EC   6E 42 0C 2A 56 BD AC 68  .w......nB.*V..h
0050: 75 07 19 E8 53 86 99 1B   B4 8E F1 AD 50 60 AC 32  u...S.......P`.2
0060: 61 D8 A3 50 E2 6F 0C F9   BB BA D9 34 A5 C6 01 DF  a..P.o.....4....
0070: 7F 47 EF A0 91 F2 30 D6   FD 8C E7 7F 44 B2 52 D4  .G....0.....D.R.

]
Peer Certificate 2:
[
[
  Version: V3
  Subject: OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign, OU=VeriSign International Server CA - Class 3, OU="VeriSign, Inc.", O=VeriSign Trust Network
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 152038337595048255423847391294280316829513160608880590645748454678611296410614722508245144022405726502805452059694145417254341557335953427821740975082044924575579268245924346494324451976855998250066581618028401376050836623416546828558818294849045727347102003201845243865318616842453964918804144784127904873327
  public exponent: 65537
  Validity: [From: Wed Apr 16 20:00:00 EDT 1997,
               To: Mon Oct 24 19:59:59 EDT 2011]
  Issuer: OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
  SerialNumber: [    254b8a85 3842cce3 58f8c5dd ae226ea4]

Certificate Extensions: 6
[1]: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
NetscapeCertType [
   SSL CA
   S/MIME CA
]

[2]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://crl.verisign.com/pca3.crl]
]]

[3]: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  Key_CertSign
  Crl_Sign
]

[4]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
  2.16.840.1.113730.4.1
  2.16.840.1.113733.1.8.1
]

[5]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.16.840.1.113733.1.7.1.1]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 1C 68 74 74 70 73 3A   2F 2F 77 77 77 2E 76 65  ..https://www.ve
0010: 72 69 73 69 67 6E 2E 63   6F 6D 2F 43 50 53        risign.com/CPS

]]  ]
]

[6]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:0
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 08 01 EC E4 68 94 03 42   F1 73 F1 23 A2 3A DE E9  ....h..B.s.#.:..
0010: F1 DA C6 54 C4 23 3E 86   EA CF 6A 3A 33 AB EA 9C  ...T.#>...j:3...
0020: 04 14 07 36 06 0B F9 88   6F D5 13 EE 29 2B C3 E4  ...6....o...) ..
0030: 72 8D 44 ED D1 AC 20 09   2D E1 F6 E1 19 05 38 B0  r.D... .-.....8.
0040: 3D 0F 9F 7F F8 9E 02 DC   86 02 86 61 4E 26 5F 5E  =..........aN&_^
0050: 9F 92 1E 0C 24 A4 F5 D0   70 13 CF 26 C3 43 3D 49  ....$...p..&.C=I
0060: 1D 9E 82 2E 52 5F BC 3E   C6 66 29 01 8E 4E 92 2C  ....R_.>.f)..N.,
0070: BC 46 75 03 82 AC 73 E9   D9 7E 0B 67 EF 54 52 1A  .Fu...s....g.TR.

]

So it is looking more and more that I need the intermediate certificate to be trusted now.

I used Firefox, and clicked on the little button next to the https:// part of the URL bar to examine the certicate information. This allowed me to see the certificate chain, and for each of the three steps, I chose to export the public key information into a CRT file. This is the same as a PEM, which is a binary format that contains the public key. (The Active Directory driver wants it in Base 64 encoded format, which is the .der, .pem, or .crt file base64 encoded).

I was about to start importing it into the basic cacerts file, when I remembered that this is a Remote loader instance, not running in the driver instance, and therefore I was looking in the wrong place.

Well where is the Remote loader's keystore? Look at the Driver Configuration, Subscriber channel to see the following setting:

<definition display-name="Truststore file" name="subTrustStoreFile-1" type="string">
	<description>When the remote server is configured to provide server authentication, this is the path and the name of the keystore file which contains trusted certificates. For example: c:\security\truststore. Leave this field blank when server authentication is not used.</description>
	<value>/var/opt/novell/dirxml/rdxml/FirstKeystore</value>
</definition>

Lets go to the Remote Loader server, find the /var/opt/novell/dirxml/rdxml/FirstKeystore and look at it with keytool:

PESAIDPV1P-ACME:/var/opt/novell/log # /opt/novell/java/bin/keytool -list -keystore /var/opt/novell/dirxml/rdxml/FirstKeystore
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

mykey, Apr 19, 2010, trustedCertEntry,
Certificate fingerprint (MD5): B1:45:46:AC:E3:F8:85:F9:B9:06:E9:CE:BE:29:91:D9

With -v, you see it just expired (Sept 8). Oops:

PESAIDPV1P-EBSD:/var/opt/novell/dirxml/rdxml # /opt/novell/java/bin/keytool -list -v -keystore /var/opt/novell/dirxml/rdxml/FirstKeystore 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: mykey
Creation date: Apr 19, 2010
Entry type: trustedCertEntry

Owner: CN=tapp0-api.salesforce.com, OU=Applications, O="Salesforce.com, Inc.", L=San Francisco, ST=California, C=US
Issuer: OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign, OU=VeriSign International Server CA - Class 3, OU="VeriSign, Inc.", O=VeriSign Trust Network
Serial number: 60f2938887435101ea2dc5e7169d3a1c
Valid from: Sun Aug 24 20:00:00 EDT 2008 until: Wed Sep 08 19:59:59 EDT 2010
Certificate fingerprints:
         MD5:  B1:45:46:AC:E3:F8:85:F9:B9:06:E9:CE:BE:29:91:D9
         SHA1: E2:EA:F7:0F:AC:62:B4:5F:9E:63:6C:B9:6F:4C:89:E0:81:35:0F:A4
         Signature algorithm name: SHA1withRSA
         Version: 3
		 

I am not sure why the creation date of this certificate is Aug 19, 2010, which is when the site certificate expired and was recreated.

Regardless, it looks like this is the basic issue so lets copy that exported CRT (pem, der. whatever) file over to this SLES box, lets call it VeriSign.crt

Now where is the VM on the remote loader server? Not the same path as on the IDM Engine server. Well the there is a create_keystore script, in /opt/novell/dirxml/bin that if you read it you will see two interesting things.

First the path to keytool is thinking more in the case of an engine server, as you can see in this code block:

    if [ -f /usr/lib/nds-modules/jre/bin/keytool ]
       then
               KEYTOOL_PATH=/usr/lib/nds-modules/jre/bin/keytool
       else
               which keytool >/dev/null 2>&1
               if [ $? -ne 0 ]
               then
                       echo "  "Error: No Java keytool executable found.
                       echo "  "Please modify PATH or edit this script.
                       exit 2
               else
                       KEYTOOL_PATH=`which keytool`
               fi
       fi
	   
	   

You can see it want to use the path for the engine, as we discussed above. So when I tried running the script, it failed reporting the error that No Java keytool executable was found.

But the second part is most useful, where you see the keytool command they want you to run:

       echo Invoking keytool from $KEYTOOL_PATH ...
       $KEYTOOL_PATH -import -alias trustedroot -file "$1" -keystore "$storename" -storepass dirxml -noprompt

What is of import there is that the -storepass is not the usual 'changeit' or 'default' that most keystores use, rather it is using 'dirxml'.

Putting that all together the command I need to execute is:

/opt/novell/java/bin/keytool -import -alias tapp0 -file VeriSign.crt -keystore

/var/opt/novell/dirxml/rdxml/SecondKeystore -storepass dirxml

I used SecondKeystore, so I could keep a backup copy of FirstKeystore, and then I moved the files around and restart the remote loader, and whammo, it started to work!

Thus as usual for SSL, it was not a simple case of just one certificate, rather there were three involved. One was fine, and already included (Parent CA). Another was not relevant, the site certificate, since that can change and this is normal. Thus we do NOT want to import its public key, because then we would definitely have to change it when the certificate changed. Finally the issue was the third certificate, the intermediate certificate that had expired, and was changed.

To make it worse, it looks like Verisign used an intermediate CA that expires in 2011, but then minted certificates that expire in 2012. Which seems a bit odd. I imagine we will be doing this again in October of 2011 again. Oh well. Keeps me employed.