Transaction consistency when using single-threaded nature of IDM to 'fan-out' to multiple end-points


As we all know, IDM is a 'single-threaded' technology, and as such, utilization of a generic IDM Connector (non 'fan-out') cannot really 'fan-out'. The problem exists that, whilst fanning-out events, if a particular end-system is unavailable, more-often-than-not then all events following the current event will be 'stuck'.

The problem arises that if an event A to Host A does not complete because Host A is unavailable, how does event event B to Host B complete in a timely manner, and at the same time event A to Host A is retried until it is processed successfully? What happens with all subsequent events to Host A while Host A is unavailable?

The following steps might make it easier to understand how one might solve the issue. By utilizing the Generic Loopback Connector and the WorkOrder Connector it might just be possible to build a transaction-oriented system.


The Generic Loopback Connector contains ECMAScript which does a 'port ping' to a host, with a port, and a timeout.

  1. On processing an event, the Connector makes a copy of the current XML document being processed

  • The Connector 'pings' the host-machine to determine if it is 'alive'

  • If the host-machine is alive, the code queries the WorkOrder container for 'pending' WorkOrders assigned to the host-machine (see naming convention referenced in 4a below)

  • If the host-machine is dead or if the number of pending WorkOrder Objects from the Query in (3) is greater than 0:

    1. Create a new WorkOrder Object with a naming convention to ensure a unique CN (contains host-machine name, timestamp, operation-type, source-name of Object)

  • WorkOrder Object 'DirXML-nwoContent' attribute is the text retrieved from (1) i.e. the entire XML document being processed

  • WorkOrder Object 'DirXML-nwoDueDate' attribute is set to a pre-defined value (30 mins for instance)

  • Drop the event

  • If (4) passes the test (i.e. the host-machine is alive and there are no WorkOrder Objects):

    1. The event is allowed to proceed

That's the first bit done

  1. Now, when the time comes and the WorkOrder Object is processed, the event is processed by the Connector

  • The content from the WorkToDo Object which contains the previous XML text 'DirXML-nwoContent' is converted to an actual XDS/XML document

  • Start at (1)

It is, therefore possible, the build a 'fan-out' scenario with built-in logic to ensure transaction consistency if an end-system is down...even with the single-threaded nature of IDM.


How To-Best Practice
Comment List