In part 1 of this series, Getting Started Building a SOAP Driver for IDM - Part 1 I discussed some of the things you need to get started building a SOAP driver. I was using the example of Salesforce.com (henceforth known as SFDC, since typing the full name is too much of a pain each time).

Anyway, I left off after explaining how to generate a <login> document for SFDC, and finally with a sample rule to parse the results of a SOAP <loginResponse> document into a useful <instance> document.

I thought it would be interesting to pick that rule apart, since it takes a somewhat different approach than the usual. Usually for this issue you would use XSLT to rebuild the document. But I wanted to do it in DirXML Script policy. This kind of policy is a little hard to read and understand so a walk through ought to be useful.

I think the trick is you need three things. First the input document to the rule, then the output you are trying to get to, and finally the rule to get from point A to point B.

What I have been doing on this project is actually pasting sample events into the Comments field of the rules. At one level that seems like overkill, but for the specific case of Input or Output Transform rules that are taking one XML document and converting it to another, it is really important. I agree this would be kind of silly overkill in say the Create rule of a driver.

Also, there is a great secondary benefit to having the sample document in the Comments field, is that when it comes time to test or debug later, you have a sample document you can copy and paste into the Simulator in Designer and work with. Otherwise you have to go track down an event to work from in fixing the issue. This came in very handy in the case of support for <query> documents since there are a lot of details to be managed in that case, and lots of exceptions and edge cases. It was very useful to have a sample document to start working from and to test with.

Here is what the <loginResponse> might look like after you succeed in connecting in your driver. (Or actually in soapUI as well).

<nds dtdversion="2.0">
  <source>
    <product build="201006032211 Internal Novell build. Not for production use." instance="SOAP-SPML" version="3.5.5">Identity Manager Driver for SOAP</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <soapenv:Envelope xmlns="urn:enterprise.soap.sforce.com" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <soapenv:Body>
        <loginResponse>
          <result>
            <metadataServerUrl>https://tapp0-api.salesforce.com/services/Soap/m/18.0/00DT0000000GV1Q</metadataServerUrl>
            <passwordExpired>false</passwordExpired>
            <sandbox>true</sandbox>
            <serverUrl>https://tapp0-api.salesforce.com/services/Soap/c/18.0/00DT0000000GV1Q</serverUrl>
            <sessionId>LongSessionIDStringGoesHere</sessionId>
            <userId>005T0000000opANCDCDE</userId>
            <userInfo>
              <accessibilityMode>false</accessibilityMode>
              <currencySymbol xsi:nil="true"/>
              <orgDefaultCurrencyIsoCode xsi:nil="true"/>
              <orgDisallowHtmlAttachments>false</orgDisallowHtmlAttachments>
              <orgHasPersonAccounts>false</orgHasPersonAccounts>
              <organizationId>00DT0000000GV1QMAW</organizationId>
              <organizationMultiCurrency>true</organizationMultiCurrency>
              <organizationName>ACME Kitten Wrangling</organizationName>
              <profileId>00e20000000h1j0AAA</profileId>
              <roleId>00E20000000hxc5EAA</roleId>
              <userDefaultCurrencyIsoCode>USD</userDefaultCurrencyIsoCode>
              <userEmail>ishkabibble@kittens.com</userEmail>
              <userFullName>IDM</userFullName>
              <userId>005T000000ABCDEFG</userId>
              <userLanguage>en_US</userLanguage>
              <userLocale>en_GB</userLocale>
              <userName>idm@ikittenscom</userName>
              <userTimeZone>America/New_York</userTimeZone>
              <userType>Standard</userType>
              <userUiSkin>Theme3</userUiSkin>
            </userInfo>
          </result>
        </loginResponse>
      </soapenv:Body>
    </soapenv:Envelope>
  </output>
</nds>

Now since we only really care about two values out of that entire document, the sessionId and the URL for the endpoint, we do not have to convert the entire thing into an <instance> doc, rather we just need two specific values. This is thus not the most general solution to the problem, but you should be able to see how you might do that from this more limited example.

The output document after our rule is finished processing, should look something like this:

<nds dtdversion="2.0">
  <source>
    <product build="201006032211 Internal Novell build. Not for production use." instance="SOAP-SPML" version="3.5.5">Identity Manager Driver for SOAP</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="login" src-dn="sfdc">
      <attr attr-name="sessionId">
        <value>LongSessionIDStringGoesHere</value>
      </attr>
      <attr attr-name="serverUrl">
        <value>https://tapp0-api.salesforce.com/services/Soap/c/18.0/00DT0000000GV1Q</value>
      </attr>
    </instance>
  </output>
</nds>

Finally we get to the piece that connects us from point A to point B, the DirXML Script policy that takes the <loginResponse> document, and converts it to the <instance> XDS compliant document that Identity Manager knows how to handle with no extra work.

<rule>
	<description>[icap] Handle loginResponse documents, with soapenv: header left behind</description>
	<comment xml:space="preserve">Lets see if we can do this in Policy instead of XSLT.  Well yes, yes we can!

Because the SOAP doc comes back with a namespace definition, this causes us pain, so the test is for self::urn:loginResponse.  This works because we use the urn: namespace to ask for it.

Next we build the instance doc, with the two values we care about, serverUrl and sessionId.  Step by step, element by element we build it.  

Note:
Append XML Element adds a <node>
Append XML text sticks a text string between <node>text</node>
Set XML attribute sets the class-name='User' in the <node class-name='User'>

Rinse and repeat for each attr.  

Set the driver scoped variables that we need to prove we are logged in.

Finally remove the SOAP doc from the XDS doc with a strip XPATH.

</comment>
	<comment name="author" xml:space="preserve">Geoffrey Carman</comment>
	<comment name="version" xml:space="preserve">2</comment>
	<comment name="lastchanged" xml:space="preserve">Apr 27, 2010</comment>
	<conditions>
		<or>
			<if-xpath op="true">self::soapenv:Envelope/soapenv:Body/urn:loginResponse</if-xpath>
		</or>
	</conditions>
	<actions>
		<do-append-xml-element expression=".." name="instance"/>
		<do-set-xml-attr expression="../instance" name="class-name">
			<arg-string>
				<token-text xml:space="preserve">login</token-text>
			</arg-string>
		</do-set-xml-attr>
		<do-set-xml-attr expression="../instance" name="src-dn">
			<arg-string>
				<token-text xml:space="preserve">sfdc</token-text>
			</arg-string>
		</do-set-xml-attr>
		<do-append-xml-element expression="../instance" name="attr"/>
		<do-set-xml-attr expression="../instance/attr[last()]" name="attr-name">
			<arg-string>
				<token-text xml:space="preserve">sessionId</token-text>
			</arg-string>
		</do-set-xml-attr>
		<do-append-xml-element expression="../instance/attr[@attr-name='sessionId']" name="value"/>
		<do-append-xml-text expression='../instance[@class-name="login"]/attr[@attr-name="sessionId"]/value'>
			<arg-string>
				<token-xpath expression="*//*[local-name()='sessionId']/text()"/>
			</arg-string>
		</do-append-xml-text>
		<do-append-xml-element expression="../instance" name="attr"/>
		<do-set-xml-attr expression="../instance/attr[last()]" name="attr-name">
			<arg-string>
				<token-text xml:space="preserve">serverUrl</token-text>
			</arg-string>
		</do-set-xml-attr>
		<do-append-xml-element expression="../instance/attr[@attr-name='serverUrl']" name="value"/>
		<do-append-xml-text expression='../instance[@class-name="login"]/attr[@attr-name="serverUrl"]/value'>
			<arg-string>
				<token-xpath expression="*//*[local-name()='serverUrl']/text()"/>
			</arg-string>
		</do-append-xml-text>
		<do-set-local-variable name="SfdcTargetURL" scope="driver">
			<arg-string>
				<token-xpath expression="*//*[local-name()='serverUrl']/text()"/>
			</arg-string>
		</do-set-local-variable>
		<do-set-local-variable name="SessionID" scope="driver">
			<arg-string>
				<token-xpath expression="*//*[local-name()='sessionId']/text()"/>
			</arg-string>
		</do-set-local-variable>
		<do-strip-xpath expression="self::soapenv:Envelope"/>
		<do-break/>
	</actions>
</rule>

Well that is quite the mouthful, so lets take that apart line by line.

So what we are doing is basically building a new XDS event document element by element from mostly scratch. Well not from total scratch. We keep the <nds> node, and the <input> or <output> node that came in with the SOAP XML doc from the driver. But everything else we need to build, and finally we remove the original document that we no longer need.

We use four DirXML Script tokens to do this. Three to add stuff, one to take away.

Append XML Element

Set XML attribute

Append XML Text

Strip by XPATH

In an XML document there are really only three components. Elements (what I keep calling nodes), like the <nds> element, or an <attr> node.

There are XML Attributes, which you might recognize more by how you select them... In the <attr> node you often see things you pick out with XPATH like @attr-name to get the value from a node like this:

<attr attr-name='nspmDistributionPassword'>

Finally, in between an open and close node, you might have text, which is what Add XML Text does.

So your <instance> doc, might have the above <attr> node, with a value set of nodes and some values.

<attr attr-name='nspmDistributionPassword'>
	<value>SomePasswordValue</value>
</attr>

So to build that above snippet, you would

Add XML Element: attr

Set XML Attribute: name of attr-name, with a value of nspmdDistributionPassword

Add XML Element: value

Add XML Text: SomePasswordValue

You can see why that gets a little hard to read, and I skipped the detail of having to provide the XPATH to tell the token WHERE to Add/set these XML thingies.

As usual, the devil is in the details.

So with that basic overview, lets work through the rules above. Note, I am going to use the XML view of the tokens, since getting this many little screen shots of all the tokens in Designer would be a total pain. (And you guys just aren't worth it! Kidding of course, but it would really be a LOT of screen shots to take and clean up!)

First off we look for a loginResponse document in the Condition block with an IF XPATH test of:

self::soapenv:Envelope/soapenv:Body/urn:loginResponse

The current context of XPATH in Identity Manager is the operation node level (usually <add>, <modify>, <query>, etc) so in this case, it is actually the <soapenv:Envelope> node.

I do not fully understand why the self:: axis needs to be used here in selecting/specifying this node, but it does seem to be needed. Because the SOAP stuff has several namespaces defined (soapenv:, urn:, sf:, and others) you need to address the nodes including the namespace names.

You could use the local-name() function to work with the local name (part after the namespace), but I try to avoid it as the XPATH gets really unwieldy!

Anyway this test means we only fire on the case of a loginResponse event in the urn: namespace.

First we add an <instance> node into the event:

<do-append-xml-element expression=".." name="instance"/>

Now the expression="" part of this token means, the XPATH that defines where to stick this node. There is a selector at the bottom of the token that says, Append to the end of XPATH expression, or else Insert before XPATH expression. I chose to do all mine as the default of Append to the end of XPATH expression.

Therefore an XPATH of ".." means just what you would think from the directory in a file system perspective, go up one level. That is an interesting twist on English. When did .. start to mean back up one level? I see that and almost treat it like a word in my head. Funny things you notice, eh?

That means, since the current context is the <soapenv:Envelope> node which is under an <output> node, it means insert this element after ".." which is right below the <output> node, and thus a sibling of the current context node.

Thus our current document looks something like this now (simplified to save space)

<nds>
	<output>
		<soapenv:Envelope>   (namespaces and everything collapsed for simplicity)
		</soapenv:Envelope>
		<instance/>
	</output>
</nds>

Well that was easy. Next we need to build up that <instance node with some XML attributes so we can use it in policy.

Lets set the class-name attribute to "login" so I can have a policy later that watches for Class Name = login events.

<do-set-xml-attr expression="../instance" name="class-name">
	<arg-string>
		<token-text xml:space="preserve">login</token-text>
	</arg-string>
</do-set-xml-attr>

In this token we specify the name of the XML attribute as class-name, set the value to login, and insert it via the XPATH expression of ../instance which is referencing our instance node we just added.

Our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope>
		</soapenv:Envelope>
		<instance class-name='login'/>
	</output>
</nds>

Better add a src-dn XML attribute, so we thus we do this token:

<do-set-xml-attr expression="../instance" name="src-dn">
	<arg-string>
		<token-text xml:space="preserve">sfdc</token-text>
	</arg-string>
</do-set-xml-attr>

I chose to use sfdc since there is no real meaningful value to indicate the actual source DN in the case of a login response event, but this way I could select the event by using this value as well, if I needed to.

Our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope>
		</soapenv:Envelope>
		<instance class-name='login' src-dn='sfdc'/>
	</output>
</nds>

Now we need an <attr> node for the first of our two values of interest.

<do-append-xml-element expression="../instance" name="attr"/>

Our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope>
		</soapenv:Envelope>
		<instance class-name='login' src-dn='sfdc'>
			<attr/>
		</instance>
	</output>
</nds>

Lets add an attr-name XML attribute, with the value sessionId.

<do-set-xml-attr expression="../instance/attr[last()]" name="attr-name">
	<arg-string>
		<token-text xml:space="preserve">sessionId</token-text>
	</arg-string>
</do-set-xml-attr>

Now we start to see something a little bit different. Note the XPATH expression we had to use to insert this looks different. Based on the previous examples you would think XPATH of ../instance/attr would have been enough. But in fact we used ../instance/attr[last()] so why the predicate of last()?

Well it turns out, for this first occasion, it would probably not matter, but when we do this for the next attr node, it would make a huge difference.

If you used the simpler XPATH of ../instance/attr it would set the XML element on ALL nodes in the document that match. So when we only have one attr node, not a big deal, but as the second and third are added, you would stick this XML attribute into each of those attr nodes, and it gets ugly fast. By using the predicate (I.e. the use of [] to set a condition, so that only when the stuff in the [] (square brackets) evaluates to true, do we do this) you limit it to the last attr node in the document. Thus you do need to do this in sequence, and be sure you complete a node in policy before coming back to it. Otherwise you would need to be specific enough in your XPATH expression to only affect that one node, because if you were not, you could append or set something on every node that matches.

Our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope> 
		</soapenv:Envelope>
		<instance class-name='login' src-dn='sfdc'>
			<attr attr-name='sessionId'/>
		</instance>
	</output>
</nds>

Next up lets add a value node, so we can then add some XML text into the value node, with the following two tokens.

<do-append-xml-element expression="../instance/attr[@attr-name='sessionId']" name="value"/>

Now when I added the <value> node, I probably could have used the predicate of [last()] here, but since I knew I could be more specific as this was a special case, I used the XPATH of:
../instance/attr[@attr-name='sessionId']

which uses a predicate of @attr-name='sessionId' to be sure I get the right node.

This is worth discussing for a moment. Why not use this approach all the time?

Well in a later article (if I get around to continuing this series that far, still not sure how far I will take this process) I might discuss how you might handle a query response, where you do not know in advance how many or which nodes need to be handled, so you might want to loop through all the nodes returned, and do something for each. In that case, it is much easier to just specify [last()] inside the loop and know it is going to work.

However, one of the side benefits of writing these sort of articles, is they make me explain my thinking after the fact, and a second review of something often reveals things I did not think of at the time. What this has made me realize, that if inside a for each loop, I append an <attr> node, set XML attribute a attr-name, I am probably calculating the value of attr-name by reading it out of the SOAP XML document. In which case, I do not have to use the last() predicate, I could just use the actual name I just set a line above to be most specific.

I would be interested in timing this to see which is more efficient XPATH, using the last() function as in:

../instance/attr[last()]

Or using a more specific predicate of:

../instance/attr[attr-name=$current-node/loacal-name()

Hmm... I do have a rule I wrote that was so slow, I stopped using it, that I could test this in. It was designed to read out schema from SFDC, and try and build the DirXML-ApplicatioSchema attribute for the driver object. I.e. I implemented the driver shim function getSchema() in policy.

Our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope> 
		</soapenv:Envelope>
		<instance class-name='login' src-dn='sfdc'>
			<attr attr-name='sessionId'>
				<value/>
			</attr>
		</instance>
	</output>
</nds>

So now we have our value node, and the second token will stuff some text into it.

<do-append-xml-text expression='../instance[@class-name="login"]/attr[@attr-name="sessionId"]/value'>
	<arg-string>
		<token-xpath expression="*//*[local-name()='sessionId']/text()"/>
	</arg-string>
</do-append-xml-text>

Here I got a little neater, and specified which <instance> node by the predicate [@class-name="login"] and which attr node by the predicate [@attr-name="sessionId"] which is a little bit of overkill, but this was the first rule of this type I worked on, and I was experimenting on possible solutions. As usual I never really went back and cleaned up, since it was working, and I had more pressing concerns on things that were NOT working.

After all, why fiddle with something that works, that is the most basic function required by everything else in the driver, when there is no need to, except for elegance. What am I saying, of course I need to go back and clean it up. He he.

Now the XPATH to select the text string is finally something different in this token. Previously I knew what value I was going to shove in, usually a static bit of text. This time I want to look at the <loginResponse> doc that SFDC sent back and get some values out of there. Here I needed to get a specific node, called sessionId, and I was having trouble selecting it correctly due to namespace info in the node, and ended up with this XPATH, which I am sure I could simplify.

*//*[local-name()='sessionId']/text()

* means all nodes below the current context, then // (slash slash) means find any occurrence of the next predicate [local-name()='sessionId'] the node whose local name (i.e. without namespace definition) is sessionId. Now the // is like a dir /s, a recursive search, which in XPATH land is considered a no no, since it is expensive as it traverses the entire document tree. However in this case, the document is fairly short so I was not too worried about it.

Finally, I wanted to return the text() value of that selected node.

I think (but do not have time to test) that a much traditional XPATH that is more direct and longer, should work for this task. Something more like:

self::soapenv:Envelope/soapenv:Body/urn:loginResponse/result/sessionId

Our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope> 
		</soapenv:Envelope>
		<instance class-name='login' src-dn='sfdc'>
			<attr attr-name='sessionId'>
				<value>SomeHexStringOfSessionIdValues</value>
			</attr>
		</instance>
	</output>
</nds>

Now we rinse and repeat the previous 4 tokens to add the <attr> node for serverURL.

<do-append-xml-element expression="../instance" name="attr"/>
<do-set-xml-attr expression="../instance/attr[last()]" name="attr-name">
	<arg-string>
		<token-text xml:space="preserve">serverUrl</token-text>
	</arg-string>
</do-set-xml-attr>
<do-append-xml-element expression="../instance/attr[@attr-name='serverUrl']" name="value"/>
<do-append-xml-text expression='../instance[@class-name="login"]/attr[@attr-name="serverUrl"]/value'>
	<arg-string>
		<token-xpath expression="*//*[local-name()='serverUrl']/text()"/>
	</arg-string>
</do-append-xml-text>

After that, our simplified view of the output doc now looks more like:

<nds>
	<output>
		<soapenv:Envelope> 
		</soapenv:Envelope>
		<instance class-name='login' src-dn='sfdc'>
			<attr attr-name='sessionId'>
				<value>SomeHexStringOfSessionIdValues</value>
			</attr>
			<attr attr-name='serverUrl'>
				<value>https://tapp-0.salesforce.com/services/soap/c/18.0/something</value>
			</attr>
		</instance>
	</output>
</nds>

Now at the same time, I set the driver scoped variables holding the same data, so it is available to all other rules in the driver.

<do-set-local-variable name="SfdcTargetURL" scope="driver">
	<arg-string>
		<token-xpath expression="*//*[local-name()='serverUrl']/text()"/>
	</arg-string>
</do-set-local-variable>
<do-set-local-variable name="SessionID" scope="driver">
	<arg-string>
		<token-xpath expression="*//*[local-name()='sessionId']/text()"/>
	</arg-string>
</do-set-local-variable>

Finally the coup de grace, as we remove the original SOAP document, via Strip by XPATH.

<do-strip-xpath expression="self::soapenv:Envelope"/>

This leaves us, finally with the following document, which looks very nice and XDS complaint.

<nds>
	<output>
		<instance class-name='login' src-dn='sfdc'>
			<attr attr-name='sessionId'>
				<value>SomeHexStringOfSessionIdValues</value>
			</attr>
			<attr attr-name='serverUrl'>
				<value>https://tapp-0.salesforce.com/services/soap/c/18.0/something</value>
			</attr>
		</instance>
	</output>
</nds>

This looks just like the sort of response you would get in any driver to a query response.

Cool eh? And zero XSLT involved. Ya, the DirXML Script is a little hard to read, but building is pretty easy if you have the input doc, the output doc you need to get too, sitting on your screen. It is just a matter of doing it step by step, element by element until you are done.

Using this approach, at least on the response side, will let you handle pretty much the entire set of API possibilities. The next step was to use this approach on query events, which has its own set of special requirements.

But once I had that done, I eventually realized I needed some other API calls, and this approach worked fine for those cases as well. Everything comes back to the engine from the shim as an <instance> doc, which makes handling it very easy.

Next to discuss is the Query handling. There are two sides to that problem, first the converting an XDS <query> event to a SFDC <query> SOAP document, which is harder than it seems, and then handling the response. Stay tuned for more in this series.