One of the nice things about the Cool Solutions site is that it allows various folk who have done neat things to share that information with everyone. Then they can add on a bit of functionality and so on, such that we end up with more than anyone on their own might develop. As an example of that Fernando Freitas wrote an awesome script that makes the User Application web service endpoint for Roles available as shell commands in bash. This is much more useful than you would think at first. After all, we can do them in a GUI, in SOAP UI, or other SOAP tools, what does bash give us?

Well if you are a shell command kind of guy or gal, it gives you a lot. Now you can use any other script framework, and just call out to these functions. As a simple example, I grabbed some sample scripts that take commands, processes them, and then executes a series of tasks, and then added in support to read input from a CSV file. Then adding in this bash shell extension, I was able to do User App role operations, in bulk, from a command line script in Linux. That turns out to be mighty useful and a great expansion upon what Fernando initially delivered.

For example, we had a customer who had thousands of roles, and needed to rename 900 of them in a hurry. Sure we could have done it in the GUI with enough time. In fact, SkyPro (http://skypro.ch/en/) sells a product called PowerRole to do this sort of bulk Role management task.

We thought we might have been able to do it by generating LDIF files to change the two attributes that seem to hold the information, but we were not sure if that would break anything. In this context, User Application is a bit of a black box, where some of it is clearly stored in the directory, and some of it is clearly stored in the database. The line between the two is very muddy and seemingly purposefully obfuscated.

Instead we needed to call the setRoleLocalizedString() API call twice, once to set the name, a second time to set the Description. Using the CSV reading script, we were able to call this 1800 times (twice each time) with one command. (As a side note, the setRoleLocalizedString API has a setting that is either 0 or 1, that determines if the value you are setting is the name or the description. That one took me a bit to figure out, figured I would save anyone else the headache. Sometimes the info is in the Web Services docs sometimes it is comments in the WSDL that give you the hint. You can see how to enumerate those comment values in this article: Enumerating values in a WSDL)

As you can see, this can be really useful. Now imagine you have 18,000 users you need to add to one or many roles. Lots of possible ways to do this, but having a script like this that can read input from a CSV sure makes it easy. (As noted earlier, there is a product you can buy to help you do it as well, PowerRole) You could for example use a Delimited Text driver (I recommend Stefaan Van Cauwenberge's Generic Text driver, it is free and way better than the NetIQ one) that has a policy to call the Add Role token for every line it reads in.

He has put a bunch of his plugins and tools available on his web site at, you should check it out there are great series of tools there.
http://vancauwenberge.info

You could do it in a Workflow that accepts text in a box, that is a list of DN's to add to a Role, and then call an Integration Activity to make the SOAP call. This has an issue that the User App will run out of memory as it loops pretty quickly, limiting the number of entries that can be processed in a single run. From a design perspective, if you tried this approach, call another workflow to do the actual Add Role, do not do it inside the loop. In this way, while you do use more memory starting additional workflows, they get queued up and do not eat memory until they run, whereas the loop persists some things in memory running you out faster.

However, using this script, you would be making many thousands of calls, but they queue up in order and get processed as fast as they can be processed since each is a standalone event/call. You can easily add a pause into the script to delay between operations if needed.

All this is nice to know, and if you read my series on how you might add a new function that Fernando missed, you will see how the script works and how you might extend it.

Series: Adding New Functions to the UA Bash extension
Enhancing the UA Bash extension for modifyResource

The User App webservices API has a lot of functionality. You can actually see the web services mature as you work through them, as the original services are very basic and have annoyances like calling all the variables arg0, arg1 and arg2 instead of useful names. (Provisioning side, which was first) Whereas in the Role and Resource endpoints you can see it switch to more meaningful names. After all, arg1 is such an informative name to help you guess at what the value is supposed to be provided.

Fernando only did the Role endpoint in his script. I extended it a bit in my articles and did some Resource function calls as well. But lately I had the need to do some bulk operations on running workflows.

I was helping my colleague with these, amusingly, while I was biking home and talking on the phone. I pointed him at my articles on how you might make a Stop Workflow, workflow. That is, there is a Start Workflow token in the IDM engine. But there is no Stop Workflow token. Instead I suggested in these articles that you could make a Workflow that takes an ID for the workflow, gets the proper ID for it, and then calls the Stop Workflow web services via an Integration Activity.
Using SOAP to terminate a running workflow – Part 1
Using SOAP to terminate a running workflow – Part 2

Well that Integration Activity is just a SOAP call, why not do it again here, in this scripting approach instead. In our case we had 300,000 running workflows, and were trying to do a User App upgrade. These two things do not usually mix well. Starting 300,000 workflows to kill 300,000 workflows did not seem like a good plan in this case either.

First things first, we need to go get the Provisioning endpoint WSDL. Same as the Role and Resource ones, if you are on older User App in the Roles or Resources tab, in the bottom left corner is a RBPM Security button, which if you click on it, has a bottom left panel showing the WSDL's available for download. Amusingly the Resource link was never really added, so you had to go to say the Role endpoint, edit the /service/role?wsdl to say /service/resource?wsdl instead to get the proper file.

Once you have the Provisioning WSDL you can open it in SOAP UI and take a look at what there is to see.

There are several functions we need. First up, we need the proper identifier to actually work with workflows, so that is getWorkEntries(). Then we need the terminate() function, which is so simply named it was easy to find.

Let's look at getWorkEntries() first. This one is actually pretty functional, which makes it somewhat complicated. You really do need to enable enumerating values in SOAP UI in order to understand this one though.

Here is the WSDL API call to start with:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://www.novell.com/provisioning/service">
   <soapenv:Header/>
   <soapenv:Body>
      <ser:getWorkEntriesRequest>
         <ser:WorkEntryQuery logic="?" order="?">
            <!--You have a CHOICE of the next 14 items at this level-->
            <!--type: string-->
            <ser:addressee>?</ser:addressee>
            <!--type: string-->
            <ser:processId>?</ser:processId>
            <!--type: string-->
            <ser:requestId>?</ser:requestId>
            <!--type: string-->
            <ser:activityId>?</ser:activityId>
            <!--type: int-->
            <ser:status>?</ser:status>
            <!--type: string-->
            <ser:owner>?</ser:owner>
            <!--type: int-->
            <ser:priority>?</ser:priority>
            <ser:creationTime>
               <!--type: long-->
               <ser:value>?</ser:value>
               <!--type: t_operator - enumeration: [EQ,LT,LE,GT,GE]-->
               <ser:operator>?</ser:operator>
            </ser:creationTime>
            <ser:expTime>
               <!--type: long-->
               <ser:value>?</ser:value>
               <!--type: t_operator - enumeration: [EQ,LT,LE,GT,GE]-->
               <ser:operator>?</ser:operator>
            </ser:expTime>
            <ser:completionTime>
               <!--type: long-->
               <ser:value>?</ser:value>
               <!--type: t_operator - enumeration: [EQ,LT,LE,GT,GE]-->
               <ser:operator>?</ser:operator>
            </ser:completionTime>
            <!--type: string-->
            <ser:recipient>?</ser:recipient>
            <!--type: string-->
            <ser:initiator>?</ser:initiator>
            <!--type: string-->
            <ser:proxyFor>?</ser:proxyFor>
            <!--type: string-->
            <ser:workTaskId>?</ser:workTaskId>
         </ser:WorkEntryQuery>
         <!--type: int-->
         <ser:arg1>?</ser:arg1>
      </ser:getWorkEntriesRequest>
   </soapenv:Body>
</soapenv:Envelope>

Some of the early lines in this give you a good idea of what is going on.

         <ser:WorkEntryQuery logic="?" order="?">
            <!--You have a CHOICE of the next 14 items at this level-->

This is a query (a WorkEntryQuery specifically) that has some ordering options (order="?") and some logic in how to use the search values. That is, of the next 14 possible ways to search, do you want them logically ANDed or ORed? There does not seem to be more clever ways to group them but this is quite powerful. What I mean by that is related to how IDM policies in the condition blocks could do simple groupings of AND vs OR of a set of conditions, but you could also then add a second condition group so that while the conditions inside each group are being treated like an AND, the two groups are being treated as an OR. This allows some fun flexibility that I do not quite see how to do here in this API call. However that is probably overkill, since you likely only ever need to look for a specific workflow by one, two, or at most three criteria.

You can see in my example that the WSDL enumeration helps because we get lines like this one:

               <!--type: t_operator - enumeration: [EQ,LT,LE,GT,GE]-->

and this one:

        <!--type: int-->

The first is really useful, since it is in the Completion Time section of:

            <ser:completionTime>
               <!--type: long-->
               <ser:value>?</ser:value>
               <!--type: t_operator - enumeration: [EQ,LT,LE,GT,GE]-->
               <ser:operator>?</ser:operator>
            </ser:completionTime>

So we need to provide an LONG data type (type: long), and the operator, which can be one of EQ,LT,LE,GT,GE. Without knowing these values, you would not know what kind of string they were expecting for the operator. However since the query option node is named completionTime we have to guess it is time related. This makes me think they want a CTIME value, since that is a 32 bit integer, sort of the classic LONG data type. But then again most things we do in User App seem to be LDAP time formatted, which has letters (Z for Zulu time zone) in it usually, and not integers.

Our 14 query options are:
addressee
processId
requestId
activityId
status
owner
priority
creationTime
expTime
completionTime
recipient
initiator
proxyFor
workTaskId

Now handling all these options in a single scripted SOAP call is probably more than I can think of how to handle. Instead what I would do is handle just a single query case, and then modify it whenever I needed to query by another kind. I just cannot think of a simple way to support 1, or as many as 14 options in this scripted approach.

In my specific case, I was looking to get ALL the workflows, because we had 300,000 stuck workflows and the database upgrade to go to IDM 4.6 just was NOT working. They were all incorrect workflows that did not have proper timeouts and never would complete or go away (yep bad design, but we came in after all this was done, not our plan) and thus needed to be terminated.

So again, our goal was to get all our workflow listed, so we could import them into a CSV file to run through the next function, TerminateRequest, to actually kill them.

Now as it turns out, my script as it sits does not do a very good job of returning output. So in the end, we just built the proper SOAP document in a text file, with the query we wanted, in our case a specific addressee, who was the 'uaadmin' user, and then sent it via 'curl' and grabbed the output for processing. If you look at any of the specific examples we have worked through so far, you will notice the final line is usually something like:

curl $_CURL_OPTIONS -k -u "$1:$SENHA" -H "$CTYPE" -H "$ACTION" -d "$POST" "$URL" -o "$4"

So really all the script does is build a doc in the variable $ACTION and the send it as a POST via curl.

Specifically because of the immensity of the 14 possible options, I only implemented a single query condition in the script which is where I determined it does not do the greatest job of returning output. You can see the example here so that I do not leave you hanging.

### Function: getWorkEntries
# Usage:
# getWorkEntries $username $password $rbpm_url $output_file $initiator $max_entries
#
getWorkEntries()
{

USAGE="Function Usage:

getWorkEntries username password rbpm_url output_file initiator max_entries
 The initiator should be in full ldap format, and if it has spaces need to be encased in quotes.
 for example:  cn=george,cn=users,o=data
 rbpm_url should be in the format:
  protocol://server:port/servicename
 for example:
  https://rbpm.lab.novell.com:8543/IDMProv";

if "X$_RBPM_SOAP_ROLE_DEBUG" = "Xtrue"
then
  dbgparams=$#
  dbgparam=1
  while [ "$dbgparam" -le "$dbgparams" ]
  do
    echo -n "Parameter "
    echo -n \$$dbgparam
    echo -n " = "
    eval echo \$$dbgparam
    (( dbgparam   ))
  done
fi

# Initial Parameters check
if | -z "$2" || -z "$3" || -z "$4" || -z "$5" || -z "$6"
then
  echo "$USAGE"
  return 1
fi

PARAMS=$#


if "X$2" = "X-W"
then
  read -sp "Please enter the password for user $1: " SENHA
  echo
else
  SENHA=$2
fi

# Setup for the SOAP call
URL="${3}/provisioning/service"
ACTION="SOAPAction: 'http://www.novell.com/provisioning/service/getWorkEntries'"
CTYPE='Content-Type: text/xml;charset=UTF-8'

# Build SOAP XML envelope and call to be issued
POST="<soapenv:Envelope xmlns:soapenv='http://schemas.xmlsoap.org/soap/envelope/' xmlns:ser='http://www.novell.com/provisioning/service'>\
<soapenv:Header/>\
<soapenv:Body>\
<ser:getWorkEntriesRequest>\
<ser:WorkEntryQuery>\
<ser:initiator>${5}</ser:initiator>\
</ser:WorkEntryQuery>\
<ser:arg1>${6}</ser:arg1>\
</ser:getWorkEntriesRequest>"

if "X$_RBPM_SOAP_ROLE_DEBUG" = "Xtrue"
then
  echo
  echo POST data:
  echo $POST
  echo
fi

# Issue the request
curl $_CURL_OPTIONS -k -u "$1:$SENHA" -H "$CTYPE" -H "$ACTION" -d "$POST" "$URL" -o "$4"
}

That runs me out of room, so I will continue explaining what I did in this example in the next article and then onwards to talk about terminate, and what is involved in that API call via SOAP.