In part 1 of this series, Getting Started Building a SOAP Driver for IDM - Part 1 I discussed some of the things you need to get started building a SOAP driver. I was using the example of Salesforce.com (henceforth known as SFDC, since typing the full name is too much of a pain each time). In Part 1 I focused on how you might start connecting via SOAP to get a session ID.

In Part 2 of this series, Getting Started Building a SOAP Driver for IDM - Part 2 I discussed how you might process the results from SFDC after you submit a login request, and converting it into an <instance> document.

The next thing that needs to happen before pretty much anything will work in this driver, is to implement the ability to query into the application.

This seems like something that might not matter that much, but consider for a moment some of the basic things a driver needs to support to actually work, starting with how you start a driver up.

One of the first things you need to do is to migrate users from the Application into the Identity Vault. Well the way a Migrate from Application, when done from either iManager or from dxcmd works, is that a query event is sent in a way that the engine and shim recognize as special (alas, there seems to be no sign of that in trace, but there is a magical difference!) and all the results become <sync> events.

Now knowing this turns out to be really helpful, since my clients SFDC production system has 30,000 plus users in it, and for the purposes we wanted to use it, only about 800 were in scope. Additionally, we did NOT want to bring 30,000 users into Identity Manager when we only needed 800, especially when we did not want to pay licensing for 30 thousand, when we needed less than a thousand users.

It turns out we were able to construct a query event that got parsed correctly, and returned only the 800 of interest. So this bit of advice was helpful. The details of how that works is a little complicated at this point, and won't make a lot of sense till the end of this article anyway. So I will hold off on that thought.

Back to the point though, that Migrate event needs a query into the application, and then processing of the results to work. Well what happens next with a <sync> event? Well the engine queries the application for all the attributes in the filter for the object. It then uses the returned results to build the <modify> event that then goes through the regular process. It stays a <modify> if the object is associated, but gets converted to an <add> if it is not associated, (of course, if a match is found, then it gets converted back to a <modify>. Ah the life of a <sync> event, like a reality TV show, some days).

Anyway, you can see how much <query> events get used in Identity Manager. So it is really important, and really the number two thing you need to get working, where number one priority was logging in.

As it turns out there is a fair bit of complexity in building a query doc handler. Even worse, a lot of it will not become clear until you get much deeper into the process of building the driver.

The first thing that you need to consider is what a Query event looks like to SFDC. The easy way to do this is to open up the WSDL in soapUI. On a side note, you could potentially stay in Designer, open up a PRD (Provisioning Request Definition) and add in an Integration Activity.

You can read more about PRD's and what goes where, in these articles:

• Introduction to Workflows and PRDs in Identity Manager - Part 1

• Introduction to Workflows and PRDs in Identity Manager - Part 2

The Integration Activity is really a remnant of the old Composer that came from Silverstream, and was a really cool tool, which was really a driver builder. (With a horrible tool called Designer whose UI was enough to make you scream. Nonetheless it was usable.) Alas it has been deprecated and almost all the remains is the Integration Activity.

The guys at Omnibond, who wrote the Scripting, Unix/Linux, AS400, Mainframe, fanout and Bi-Directional drivers and mainframe Sentinel connectors are working on RaDD, Rapid Driver Development toolkit, which they were talking about at Brainshare. This is meant to do some of the things Composer could do like screen scraping a TN3270 session. I am really looking forward to seeing more about it, when it is ready. You can see more about it on their Products page: http://www.omnibond.com/products.html and read the sales page they have at: http://www.omnibond.com/RaDD_overview.pdf

Anyway, when you insert a Integration Activity into a PRD's Workflow, you get asked for a WSDL to work from, and then you get to pick the function you wish to use. Once you do that, you get a slightly crazy UI (this is the part of their old Designer tool I was referring too, but this is a STRIPPED down and upgraded version!! Imagine how it was before?) that shows the SOAP function, the response, and possible error messages.

Have fun figuring out the UI, that is topic for another article, but once you do it is quite powerful. So maybe soapUI is an easier choice.

So a Query event to SFDC basically takes a SOQL query statement. SOQL is Salesforce's custom query language. For the most part it is modeled after regular SQL, and with each release and update of SFDC's API set, it has gotten more powerful. For example the Apex Explorer tool I recommend using does not support the latest API, and a query I made to get my 800 users instead of thirty thousand is actually not supported in it.

You can learn and experiment with the SOQL language in Apex Explorer which I highly recommend if only to better understand the shape of the data.

The first thing you should learn is that "SELECT * from Users" will not work. There is no support for "SELECT *" this turns out to be a pain! Worse than that, most queries you need that are automated by the system are usually sent without any specific attributes specified. This usually implies all attributes (at least as defined by the filter) which would naturally translate into SELECT *. but of course that does not work.

Here is an example where first time through I did it one way, and in hindsight would redo it differently. But I have a working system, that does not really refactoring, so I probably won't do it.

What I should have done was used the data in the Schema Map and the Filter XML attributes and parsed them. It turns out digging into the Schema Map is pretty easy via XPATH as I discuss in this article: XPATH to do schema mapping rule

But I did not consider this in my first approach, and I ended up leveraging a fair bit of code around my actual solution, so it ended up working well.

What I ended up doing was using a Mapping table with more and more columns. The first column I ended up numbering for use in my getUpdated call later. (More on that later, probably much later several articles from now). Then I added in the SFDC object class name. Then the eDirectory Object Class name. Next up I added in pretty much all of the SELECT string, that * would imply (aka all the attributes in the filter). This is the part I would probably build using the filter and schema map via XPATH in a second version. I also added a column with the filtering stuff that would go in the WHERE part of the SOQL statement. This way, I could constrain an unconstrained query for all Accounts, to only handle Accounts of interest. (Again, we had ten thousand accounts but only twenty two hundred of interest). This makes everything more efficient.

Also it hides a complexity that I never could quite get to work perfectly, which relates to queries that return more than 2000 values. SFDC has a limit of returning a maximum of 2000 objects to any single query and supports a <queryMore> function that should map to Identity Managers query-ex function. I ALMOST got that working, but there are still some niggling little details, and it is not working on a migrate, which is where it would be most useful. In order for it to even work a little bit, you need the latest SOAP driver patch, where engineering quite nicely added a driver configuration option, called Query-ex support (Actually, queryExSupported), that needs to be added to the XML for the driver configuration.

<definition display-name="Query-ex Support" name="queryExSupported" type="enum">
	<description>Enable Query-ex support if the SOAP destination can support it, and policies are in place to manage it</description>
	<enum-choice display-name="Query-ex Supported">yes</enum-choice>
	<enum-choice display-name="Query-ex not supported">no</enum-choice>
	<value>yes</value>
</definition>

This tells the shim and engine to use Query-ex when it makes sense. This was needed, as when the driver starts up it looks for a class named: __driver_identification_class__ in a query that looks like this:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.6.10.4747">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query event-id="query-driver-ident" scope="entry">
      <search-class class-name="__driver_identification_class__"/>
      <read-attr/>
    </query>
  </input>
</nds>

This returns results that look like this snippet below:

<nds dtdversion="2.0">
  <source>
    <product build="201006032211 Internal Novell build. Not for production use." instance="SOAP-SPML" version="3.5.5">Identity Manager Driver for SOAP</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="__driver_identification_class__">
      <attr attr-name="driver-id">
        <value type="string">SOAP</value>
      </attr>
      <attr attr-name="driver-version">
        <value type="string">3.5.5</value>
      </attr>
      <attr attr-name="min-activation-version">
        <value type="int">3</value>
      </attr>
      <attr attr-name="query-ex-supported">
        <value type="state">true</value>
      </attr>
    </instance>
  </output>
</nds>

Do note that my example shows that the build="201006032211 Internal Novell build. Not for production use.". This is because I am using the patch they built for me, and did not upgrade yet to the released 3.5.5 SOAP driver in this environment. You would not see that part if you use the released version. On a side note, make sure you run the engine patch that came out as well, the 3.6.1 Engine Patch 2, (build 3.6.1.13) since it includes a fix in the nxsl.jar file that makes SOAP unusable. It was a crazy bug! If you had a node, with the xsi: namespace (as does almost all SOAP documents), and it was a xsi:type=nil, then the engine would hang. Designer as well if you tried the code in Simulator. But it is fixed now, patch and move on.

Anyway, prior to the 3.5.5 build of the driver, the node <attr attr-name="query-ex-supported"> would not be returned. Therefore the engine would never use it. I tried, but there seems like no way in policy to touch this event, which is probably a good thing, since if you could, you would modify the <attr attr-name="min-activation-version"> to try and get around licensing. Right? They were smart enough to block that avenue. Darn them!

So in order to keep queried results smaller than 2000 I wanted to include WHERE statements in every query, such that it limited the result set to be more manageable. Also to avoid bring in disabled stuff when migrating.

Thus I needed that column in my mapping table as well. As I got further into the subtleties of handling <query> and <queryResponse> docs I used it more and more so going back to refactor it would be less and less useful now.

Having said all that, your needs may differ than mine, and you may not care about all this at all.

Here is what a sample XDS query might look like:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.6.1.4427">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="Account" dest-dn="00120000000SACBOAAW" dest-entry-id="35702" scope="entry">
      <read-attr attr-name="Region__c"/>
      <read-attr attr-name="Account_Inactive__c"/>
      <read-attr attr-name="Trading_On__c"/>
    </query>
  </input>
</nds>

This is a nice query event that has a fair bit of stuff in it that needs to be converted into a reasonable SELECT statement.

My policy spits out this event, after it is done its dirty work:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.6.1.4427">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
      <soapenv:Header>
        <urn:SessionHeader xmlns:urn="urn:enterprise.soap.sforce.com">
      <urn:sessionId>whatever</urn:sessionId>
        </urn:SessionHeader>
      <urn:QueryOptions>
         <urn:batchSize>200</urn:batchSize>
      </urn:QueryOptions>
      </soapenv:Header>
      <soapenv:Body>
        <urn:query xmlns:urn="urn:enterprise.soap.sforce.com">
          <urn:queryString>Select  a.Region__c, a.Account_Inactive__c, a.Trading_On__c FROM Account a WHERE (a.isDeleted=false AND a.ID='00120000000SACBOAAW' )</urn:queryString>
        </urn:query>
      </soapenv:Body>
    </soapenv:Envelope>
  </input>
</nds>

So that nice XDS <query> doc becomes the following SELECT statement in SOQL:

Select a.Region__c, a.Account_Inactive__c, a.Trading_On__c FROM Account a WHERE (a.isDeleted=false AND a.ID='00120000000SACBOAAW' )

So lets look at that. First of all, custom attributes in SFDC all end in underbar, underbar, c. (__c) so you can tell that while Account as a class is part of SFDC, these guys extended it with Region__c, Account_Inactive__c, Trading_On__c and so on.

The WHERE clause picks up my mapping table item of isDeleted=false, since SFDC does not seem to delete anything, rather it flags them as deleted. Oh and as a fun side note, not every class in SFDC support isDeleted. At least one (RecordType I think) does not, so you need to handle that as well. Then, because the query doc had a target specified in the dest-dn XML attribute, we add the "AND a.ID='00120000000SACBOAAW'" part to make it specific. Of course remember you need a class specified and a dest-dn, or at least an association value. You might want to implement something to handle if src-dn is all that is available to go look back at that object for an Id value, or association, as without a real target, you cannot query.

The "FROM Account a" part is a way they use to alias class names, so that the SELECT part can say a.Region__c, instead of having to say Account.Region__c, and since there is a length limit on SFDC query statements, this is a big space saver! I could have used any letter I wanted, but 'a' was easy and there.

How you get from the XDS query to the output SOAP <urn:query> document I am afraid I am going to leave as an exercise for the reader, since the complexity of it is high, the implementation is pretty specific to my client, and you could probably do each step a number of different ways.

But you can start pretty easily, using just the two examples I have above, and whack away at it in Policy until you get close. There are a ton of minor potholes to avoid, so have fun, this one is a GREAT learning experience.

Then you will also need to handle some of the other API calls, that I found easiest to manage as <query> events, where the class-name specifies the API name you are calling, like getServerTimestamp(). The details truly are a killer on this driver. But thats good, it makes it more fun, and you are constantly revisiting the code to make it better!

In part 4 I think I will talk about some of the <queryResponse> handling things you need to do, and see if we have time for figuring out how to make this driver get events on the Publisher channel.