multiple ad driver is getting "Error retrieving the event: Padding is invalid and cannot be removed"

Hi,

We are setting-up a NetIQ IDM solution for a client for provisioning data into multiple applications using Multi AD Driver. We are having requirement to setup 11 multi AD driver to connect to 11 domain within same forest to provision same user across many domains.

 

We have two multi AD drivers in sandbox connecting to same forest where the 1st one is connected to Primary domain and second to the other child domain. When I am running these drivers alone its working fine like user is getting created in AD perfectly. But the moment I switch on both the Drivers and send user creation event to one of the driver(Say Driver 1 connected to Primary Domain) the other drivers gets some kind of event at publisher channel and gets shutdown and vice versa. Remote Loader is running on domain server. Below is the trace at level 5 of driver and remote loader. 

The connectivity between IDM engine/remote loader and AD is established at port 389.

Driver Trace:
<nds dtdversion="3.5">
<source>
<product build="201904100952" instance="" version="1.1.1.0">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<output>
<status level="error">Execption: Error retrieving the event: Padding is invalid and cannot be removed.
at DXMLMADDriver.MADEvents.MADEvent.GetXDSOperation()
at DXMLMADDriver.MADPublisherShim.start(IXmlCommandProcessor execute)</status>
</output>
</nds>
[04/13/21 07:01:28.027]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Remote Interface Driver: Received command: PUBLISHER EXECUTE(-124).
[04/13/21 07:01:28.028]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Receiving DOM document from application.
[04/13/21 07:01:28.028]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:
<nds dtdversion="3.5">
<source>
<product build="201904100952" instance="" version="1.1.1.0">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<output>
<status level="error">Execption: Error retrieving the event: Padding is invalid and cannot be removed.
at DXMLMADDriver.MADEvents.MADEvent.GetXDSOperation()
at DXMLMADDriver.MADPublisherShim.start(IXmlCommandProcessor execute)</status>
</output>
</nds>
[04/13/21 07:01:28.081]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Fixing up association references.
[04/13/21 07:01:28.085]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Applying schema mapping policies to output.
[04/13/21 07:01:28.088]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Applying policy: % CCSubSchemaP01-Schema Mapping%-C.
[04/13/21 07:01:28.091]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Applying output transformation policies.
[04/13/21 07:01:28.094]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Applying policy: % CCNETQMADDCFG-otp-FormatConversions%-C.
[04/13/21 07:01:28.094]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Applying to status #1.
[04/13/21 07:01:28.094]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Policy returned:
[04/13/21 07:01:28.099]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
<product edition="Advanced" version="4.8.0.0">DirXML</product>
<contact>NetIQ Corporation</contact>
</source>
<output>
<status level="error">Code(-9042) &lt;nds> element must contain a &lt;input> element.</status>
</output>
</nds>
[04/13/21 07:01:28.249]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Applying input transformation policies.
[04/13/21 07:01:28.251]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Applying policy: % CCNETQMADDCFG-itp-RestoreOperationData%-C.
[04/13/21 07:01:28.254]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Applying to status #1.
[04/13/21 07:01:28.257]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Evaluating selection criteria for rule 'restore operation data '.
[04/13/21 07:01:28.260]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: (if-operation not-match "query|instance") = TRUE.
[04/13/21 07:01:28.263]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: (if-xpath true "$current-op/driver-operation-data") = FALSE.
[04/13/21 07:01:28.265]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Rule rejected.
[04/13/21 07:01:28.267]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Applying to init-params #2.
[04/13/21 07:01:28.269]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Evaluating selection criteria for rule 'restore operation data '.
[04/13/21 07:01:28.272]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: (if-operation not-match "query|instance") = TRUE.
[04/13/21 07:01:28.276]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: (if-xpath true "$current-op/driver-operation-data") = FALSE.
[04/13/21 07:01:28.277]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT: Rule rejected.
[04/13/21 07:01:28.279]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:Policy returned:
[04/13/21 07:01:28.281]:CIMB-IDV_TO_MULTI-AD_Domain2_VM PT:
<nds dtdversion="3.5" ndsversion="8.x">
<input>
<status event-id="report status" level="fatal" type="remoteloader">Premature return from PublicationShim.start()</status>
<init-params event-id="write-state">
<publisher-state>
<ABC>
<cookie>TVNEUwMAAABxDYtiMjDXAQAAAAAAAAAAIAIAAK3xAQAAAAAAAAAAAAAAAACt8QEAAAAAAIC KDof6DJGkxdb3GQzaB0BAAAAAAAAABYAAAAAAAAAhq0iEGt700S80xk/xQ3TJwigAAAAAAAAoQs HpCqYki/hz0gppSNQhAgAQAAAAAAKhBFJBBlQEGKgIYj1sxnBhzgAQAAAAAAYFTVMIAuQU2pNFvw5 uSmgJAAAAAAAAAgL4oOh/oMkaTF1vcZDNoHa3xAQAAAAAAJYEERhNqWk /MA0wJzW IhvQAQAAAAAAQGBBT2pySUCm3rU EvgDRBEwAQAAAAAACEIBURmDQEaqbIQ/MrgFJQRgAAAAAAAAR/Q6U6o000OicPA7tK7akQNQAAAAAAAATifAVYdRpkOegggg0siW7Q8QAQAAAAAAm3 LVxDu kOZf2Xt34HTwQzgAAAAAAAAfo/ndqfYFk6ewJa1pMXjkx3wAQAAAAAA9REJll89p06aCl yAmyiTgvQAAAAAAAAjjkJnjU0FECa4E4LOBaTPAeQAAAAAAAA FDAvKtbLUeCqmxBAV2LhQVwAAAAAAAAJjzFw/7epkyYZv8oPhzhwwaAAAAAAAAAD/pCzeUTLkOFs1uM1Z5UFBaAAQAAAAAAtOZ84vtpekukw5/uztXwJQ4AAQAAAAAAiHVR488fl0eikRk6zvV2aw3wAAAAAAAAJ6Zd5iGdTU ITkCH6s5/WAmwAAAAAAAAFFik6NxgHUKgAYH0qcBD2xrAAQAAAAAA0Jm49B 6vEStkrqIgOMGIRmwAQAAAAAA</cookie>
</ABC>
<ABCA>
<cookie>TVNEUwMAAADV3PrNKDDXAQAAAAAAAAAAkAEAAEjQAQAAAAAAAAAAAAAAAABI0AEAAAAAACftVVwQ9a5Mk8CSZc2o/1oBAAAAAAAAABAAAAAAAAAAn2ckBBinE0yBOyHKigIt5wvQAAAAAAAA yk4D5dpb0SMBnaVOEcTBwVwAAAAAAAAfmHlIdMp7UinU3uV9BO9Ww4AAQAAAAAAPKCZIoXldESHpa8X96Te/grAAAAAAAAAVXjnJAZ4JEGeSXLddHF1cxrAAQAAAAAAHhFrMpd91UGUNVUAmuKp4hRgAQAAAAAAT1WnOUwb0UuZWT9kWN7M4BvQAQAAAAAABTMWSFP5oEKjr7XyCILvjQ8QAQAAAAAAJ 1VXBD1rkyTwJJlzaj/WtrQAQAAAAAAuzfSaRvZuEeuyRd6d60R BmwAQAAAAAAvc69gTv dk6B IyBiAvv0ReQAQAAAAAAn291p1cJzUyr7/9HjWA9Ug3wAAAAAAAAEEMBri4lxUimCe9pFMNubhJAAQAAAAAAmitMsDeLv0yr8BHVnnCT4gaAAAAAAAAAkJpixxV7sEiW2XtQ2HKMTQeQAAAAAAAAlcpb5GH/Okq/PkydM4dS0gmwAAAAAAAA</cookie>
</ABCA>
</publisher-state>
</init-params>
</input>
</nds>

Remote Loader Trace:

 

DirXML: [04/12/21 14:32:49.935]:
DirXML Log Event -------------------
Driver = \NIDM-IDV\system\driverset1\CIMB-IDV_TO_MULTI-AD
Level = success
DirXML: [04/12/21 14:32:58.146]: TRACE: ABC: ADDriver: Publisher Poll
DirXML: [04/12/21 14:32:58.146]: TRACE: ABC: ADDriver: get object changes - 0x0000
DirXML: [04/12/21 14:32:58.146]: TRACE: ABC: ADDriver: object changes complete
DirXML: [04/12/21 14:33:48.133]: TRACE: Event Processor:: RETRY Queue timed out: Timeout for the requested operation has expired.
DirXML: [04/12/21 14:33:48.133]: TRACE: ABC: Queue timed out: Timeout for the requested operation has expired.
DirXML: [04/12/21 14:33:48.352]: TRACE: Remote Loader: Received document from publicationShim
DirXML: [04/12/21 14:33:48.352]: TRACE:
<nds dtdversion="3.5">
<source>
<product instance="" version="1.1.1.0" build="201904100952">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<output>
<status level="error">Execption: Error retrieving the event: Padding is invalid and cannot be removed.
at DXMLMADDriver.MADEvents.MADEvent.GetXDSOperation()
at DXMLMADDriver.MADPublisherShim.start(IXmlCommandProcessor execute)</status>
</output>
</nds>
DirXML: [04/12/21 14:33:48.352]: TRACE: Remote Loader: Sending...
DirXML: [04/12/21 14:33:48.352]: TRACE:
<nds dtdversion="3.5">
<source>
<product instance="" version="1.1.1.0" build="201904100952">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<output>
<status level="error">Execption: Error retrieving the event: Padding is invalid and cannot be removed.
at DXMLMADDriver.MADEvents.MADEvent.GetXDSOperation()
at DXMLMADDriver.MADPublisherShim.start(IXmlCommandProcessor execute)</status>
</output>
</nds>
DirXML: [04/12/21 14:33:48.352]: TRACE: Remote Loader: Document sent.
DirXML: [04/12/21 14:33:48.664]: TRACE: Remote Loader: Received.
DirXML: [04/12/21 14:33:48.664]: TRACE:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
<product edition="Advanced" version="4.8.0.0">DirXML</product>
<contact>NetIQ Corporation</contact>
</source>
<output>
<status level="error">Code(-9042) &lt;nds&gt; element must contain a &lt;input&gt; element.</status>
</output>
</nds>
DirXML: [04/12/21 14:33:48.664]: TRACE: Remote Loader: Received document for publisher channel.
DirXML: [04/12/21 14:33:48.664]: TRACE: Remote Loader: Command : 138
DirXML: [04/12/21 14:33:48.664]: TRACE: Remote Loader: Waiting for receive...
DirXML: [04/12/21 14:33:48.664]: TRACE: Remote Loader: DirXML returned:
DirXML: [04/12/21 14:33:48.664]: TRACE:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
<product edition="Advanced" version="4.8.0.0">DirXML</product>
<contact>NetIQ Corporation</contact>
</source>
<output>
<status level="error">Code(-9042) &lt;nds&gt; element must contain a &lt;input&gt; element.</status>
</output>
</nds>
DirXML: [04/12/21 14:33:48.664]:
DirXML Log Event -------------------
Driver = \NIDM-IDV\system\driverset1\CIMB-IDV_TO_MULTI-AD
Thread = Publisher
Level = error
Message = Code(-9042) <nds> element must contain a <input> element.
DirXML: [04/12/21 14:33:48.664]: TRACE: Remote Loader: PublicationShim.start() returned:
DirXML: [04/12/21 14:33:48.664]: TRACE:
<nds dtdversion="3.5">
<source>
<product instance="" version="1.1.1.0" build="201904100952">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<output>
<status level="success" />
</output>
</nds>
DirXML: [04/12/21 14:33:48.664]:
DirXML Log Event -------------------
Driver = \NIDM-IDV\system\driverset1\CIMB-IDV_TO_MULTI-AD
Thread = Publisher
Level = error
Message = Premature return from PublicationShim.start()
DirXML: [04/12/21 14:33:48.664]:
DirXML Log Event -------------------
Driver = \NIDM-IDV\system\driverset1\CIMB-IDV_TO_MULTI-AD
Thread = Publisher
Level = fatal
Message = Premature return from PublicationShim.start()
DirXML: [04/12/21 14:33:48.679]: TRACE: Remote Loader: Sending...
DirXML: [04/12/21 14:33:48.679]: TRACE:
<nds ndsversion="8.x" dtdversion="3.5">
<input>
<status level="fatal" type="remoteloader" event-id="report status">Premature return from PublicationShim.start()</status>
<init-params event-id="write-state">
<publisher-state>
<ABC>
<cookie>TVNEUwMAAAANbrxvmi/XAQAAAAAAAAAACAIAAGrhAQAAAAAAAAAAAAAAAABq4QEAAAAAAH6P53an2BZOnsCWtaTF45MBAAAAAAAAABUAAAAAAAAAhq0iEGt700S80xk/xQ3TJwigAAAAAAAAoQs HpCqYki/hz0gppSNQhAgAQAAAAAAKhBFJBBlQEGKgIYj1sxnBhzgAQAAAAAAYFTVMIAuQU2pNFvw5 uSmgJAAAAAAAAAJYEERhNqWk /MA0wJzW IhvQAQAAAAAAQGBBT2pySUCm3rU EvgDRBEwAQAAAAAACEIBURmDQEaqbIQ/MrgFJQRgAAAAAAAAR/Q6U6o000OicPA7tK7akQNQAAAAAAAATifAVYdRpkOegggg0siW7Q8QAQAAAAAAm3 LVxDu kOZf2Xt34HTwQzgAAAAAAAAfo/ndqfYFk6ewJa1pMXjk2rhAQAAAAAA9REJll89p06aCl yAmyiTgvQAAAAAAAAjjkJnjU0FECa4E4LOBaTPAeQAAAAAAAA FDAvKtbLUeCqmxBAV2LhQVwAAAAAAAAJjzFw/7epkyYZv8oPhzhwwaAAAAAAAAAD/pCzeUTLkOFs1uM1Z5UFBaAAQAAAAAAtOZ84vtpekukw5/uztXwJQ4AAQAAAAAAiHVR488fl0eikRk6zvV2aw3wAAAAAAAAJ6Zd5iGdTU ITkCH6s5/WAmwAAAAAAAAFFik6NxgHUKgAYH0qcBD2xrAAQAAAAAA0Jm49B 6vEStkrqIgOMGIRmwAQAAAAAA</cookie>
</ABC>
</publisher-state>
</init-params>
</input>
</nds>
DirXML: [04/12/21 14:33:48.679]: TRACE: Remote Loader: Document sent.
DirXML: [04/12/21 14:33:48.679]: TRACE: Remote Loader: Received document for subscriber channel.
DirXML: [04/12/21 14:33:48.679]: TRACE: Remote Loader: Command : 96
DirXML: [04/12/21 14:33:48.679]: TRACE: Remote Loader: Calling DriverShim.shutdown()
DirXML: [04/12/21 14:33:48.679]: TRACE: Remote Loader: Remote Loader: null document
DirXML: [04/12/21 14:33:48.679]: TRACE: ABC: Waiting for Worker thread to exit ..
DirXML: [04/12/21 14:33:48.695]: TRACE: Remote Loader: Received document for subscriber channel.
DirXML: [04/12/21 14:33:48.695]: TRACE: Remote Loader: Command : 0
DirXML: [04/12/21 14:33:48.695]: TRACE: Remote Loader: Closing connection...
DirXML: [04/12/21 14:33:48.695]: TRACE: Remote Loader: Connection closed
DirXML: [04/12/21 14:33:48.695]: TRACE: Remote Loader: Leaving event loop for subscriber service thread.
DirXML: [04/12/21 14:33:49.944]: TRACE: Remote Loader: Received document from publicationShim
DirXML: [04/12/21 14:33:49.944]: TRACE:
<nds dtdversion="3.5">
<source>
<product instance="\NIDM-IDV\system\driverset1\CIMB-IDV_TO_MULTI-AD" version="1.1.1.0" build="201904100952">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<input>
<status level="success" type="heartbeat"></status>
</input>
</nds>
DirXML: [04/12/21 14:33:49.944]: TRACE: Remote Loader: Received document for subscriber channel.
DirXML: [04/12/21 14:33:49.944]: TRACE: Remote Loader: Command : 96
DirXML: [04/12/21 14:33:58.156]: TRACE: ABC: ADDriver: Publisher Poll
DirXML: [04/12/21 14:33:58.156]: TRACE: ABC: ADDriver: get object changes - 0x0000
DirXML: [04/12/21 14:33:58.156]: TRACE: ABC: ADDriver: object changes complete
DirXML: [04/12/21 14:34:18.685]: TRACE: Remote Loader: DriverShim->shutdown() returned:
DirXML: [04/12/21 14:34:18.685]: TRACE:
<nds dtdversion="3.5">
<source>
<product instance="" version="1.1.1.0" build="201904100952">Multi Domain Active Directory Driver</product>
<contact>NetIQ, Corporation.</contact>
</source>
<output>
<status level="success" />
</output>
</nds>
DirXML: [04/12/21 14:34:18.685]: TRACE: Remote Loader: Stopping driver.
DirXML: [04/12/21 14:34:18.685]: TRACE: Remote Loader: Received document for subscriber channel.
DirXML: [04/12/21 14:34:18.685]: TRACE: Remote Loader: Command : 96
DirXML: [04/12/21 14:34:18.685]: TRACE: Remote Loader: Waiting for DirXML to connect on 'TCP server socket, port 8094, address localhost'...
DirXML: [04/12/21 14:34:18.716]: TRACE: Remote Loader: Opening connection...
DirXML: [04/12/21 14:34:18.716]: TRACE: Remote Loader: Server socket parameters: address = <not-specified> port = 8094
DirXML: [04/12/21 14:34:18.716]: TRACE: Remote Loader: Creating a socket listener
DirXML: [04/12/21 14:34:18.716]: TRACE: Remote Loader: Entering listener accept()
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: Queue timed out: Timeout for the requested operation has expired.
DirXML: [04/12/21 14:34:48.149]: TRACE: Event Processor:: RETRY Queue timed out: Timeout for the requested operation has expired.
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: Worker thread Exiting ..
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: Driver::shutdown
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: shutdown subscriber
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: shutdown subscriber service
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: shutdown publisher
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: Shutdown 1

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: shutdown notification complete
DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] PassSyncRPC::StopServerRpc()

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] StopServerRpc() - signal the event to kill the RPC thread.

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] StopServerRpc() - wait for the RPC thread to exit...

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] InitializeRpcServer() - dwWait = 0x00000000

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] InitializeRpcServer() - exiting the RPC thread.

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] InitializeRpcServer() - release the RPC mutex.

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] PassSyncRPC::InitializeRpcServer() returned 0x00000000

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD] PassSyncRPC::StopServerRpc() returned 0x00000000

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD 6256] - PassSyncCache::StopInitDomainThread()

DirXML: [04/12/21 14:34:48.149]: TRACE: ABC: ADDriver: [PWD 6256] - waiting for InitializeDomainInfo thread to exit...

DirXML: [04/12/21 14:34:58.160]: TRACE: ABC: ADDriver: [PWD 6256] - PassSyncCache::StopInitDomainThread() returned 0x00000102

DirXML: [04/12/21 14:34:58.160]: TRACE: ABC: ADDriver: publisher shutdown complete
DirXML: [04/12/21 14:34:58.160]: TRACE: ABC: PublicationShim.start() returned

Can you please let me know what's going wrong. Thanks in Advance.

-Vineet