Highlighted
Micro Focus Expert
Micro Focus Expert
1295 views

IBM DataPower XML Gateway FlexConnector

This is a very rudimentry FlexConnector that I built for a Proof of Concept whereby the prospect wanted to search for trasaction IDs found in DataPower and another application called Aepona Causeway. There is no categorization as we where only interested in performing RegEx searches against the data in Logger.

I am waiting for DeathByWedgie's Flex for the same source as it is probably cleaner

This is a Syslog subagent parser that need to be added to the %connector%\current\user\agent\flexagent\syslog directory.

The submessages map to log messages such as:

Feb 06 00:05:17 1.1.1.1 [ssl][error] valcred(customercert): trans(3826290898)[2.2.2.2]: SSL Proxy Profile 'PreProcess': connection error: peer did not send a certificate

Feb 06 00:05:17 1.1.1.1 [mpgw][error] source-https(parlayx-8443-HTTPS): trans(3826290898)[2.2.2.2]: SSL error - could not establish SSL for incoming connection. Connection Refused.

Feb 06 06:28:33 1.1.1.1 [xslt][error] wsgw(parlayx-all-interfaces): trans(1506174369)[request][2.2.2.2]: Execution of 'local:///ProcessParlayX.xsl' aborted: <?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:m1="http://www.csapi.org/schema/parlayx/common/v2_1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:customerdp="http://www.customer.com/sdf/soap/header"><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode>SOAP-ENV:Server</faultcode><faultstring/><detail><m1:ServiceException xsi:type="m1:ServiceException"><messageId>SVC0001</messageId><text>DP009: After privacy considerations (group policy 'blacklist') plus eliminating any inactive subscribers, there are no addresses in this operation.</text></m1:ServiceException></detail></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>
Feb 06 06:28:33 1.1.1 [multistep][error] wsgw(parlayx-all-interfaces): trans(1506174369)[request][2.2.2.2]: request parlayx-all-interfaces_default_request-rule #3 xform: 'Transforming tempvar1 with local:///ProcessParlayX.xsl results stored in OUTPUT' failed: <?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:m1="http://www.csapi.org/schema/parlayx/common/v2_1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:customerdp="http://www.customer.com/sdf/soap/header"><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode>SOAP-ENV:Server</faultcode><faultstring/><detail><m1:ServiceException xsi:type="m1:ServiceException"><messageId>SVC0001</messageId><text>DP009: After privacy considerations (group policy 'blacklist') plus eliminating any inactive subscribers, there are no addresses in this operation.</text></m1:ServiceException></detail></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelo

0 Likes
9 Replies
deathbywedgie1 Frequent Contributor.
Frequent Contributor.

Re: IBM DataPower XML Gateway FlexConnector

I'm still hoping to get a release to share my content. In the meantime, let me suggest one thing for your first draft here... if you're wanting to use it as a syslog subparser, you need to remove the definitions for the timestamp and hostname/IP at the beginning. That's all going to be taken care of by the agent as it's consistent across all syslog events.

I appreciate you sharing sample events, too. Until today I'd never seen events from anyone else's environment, so I was anxious to find out whether my parser will work for everyone... those events confirm that the format will work. You do have a message in there that I don't, though, so I'll have to update it for another submessage. Mine is currently over 200 submessages long because I try to accommodate all the possible event formats to maximize how much information I get out of them and categorize them as appropriately as possible. It's not necessary to be that long if you're only looking for a simple parser, though, so with a few changes yours is usable for a quick parser.

My heart goes out to the unlucky individual(s) who had to write the Cisco Router subparser. By comparison this one's brief. lol

0 Likes
init1 Absent Member.
Absent Member.

Re: IBM DataPower XML Gateway FlexConnector (TTP# 62581)

Gary,

I would like to thank you for this code.  It will kick start my work I plan to start next week and here is some information I got form IBM as part of my research phase.  Also to let everyone know I opened a TTP request to have ArcSight code for this product.  The TTP # is 62581.

The format of the message section may vary slightly but generally following the following flow:

Domain/Service events:
Nov 30 17:41:23 LogTargetLocalIdentifier [domain][eventCategory][eventLogLevel] serviceType(serviceName): trans(transactionID): some message

System events:
Nov 30 17:47:55 LogTargetLocalIdentifier [eventCategory][eventLogLevel] trans(transactionID): some message

Then I asked them to define the variables:

domain - Name of the application domain containing the service which generated the event.
eventCategory - The event category. See Available option in the Log Target configuration.
eventLogLevel - The event's log level. e.g. debug/info/notice/warn/error etc.
serviceName - Name of the service. e.g. MyXMLFirewall
transactionID - Numeric transaction ID (tid).
some message - The message provided by the event
LogTargetLocalIdentifier - The value of Local Identifier as defined in the log target which sends the syslog data.
serviceType - The abbreviation of the service. e.g. wsgw/mgpw/xmlfirewall/rbm etc.

Sincerely,

Init 1

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: IBM DataPower XML Gateway FlexConnector (TTP# 62581)

Now if IBM could just generate everythying in CEF we'd be laughing

Thanks for the pointers guys, like I said, "rudimentry"... Basically just had to prove we could bring in the data source during the POC.

Between what deathbywedgie and I've given you you should be good to start working on this next week.

Cheers,

Gary

0 Likes
init1 Absent Member.
Absent Member.

Re: IBM DataPower XML Gateway FlexConnector

All,

My battle has begun and I'm not sure if this is normal stuff I'm looking at, since I'm new to the XML product.  As I write my regex to parse the common tokens I ran into a bump.  I have 2 types of outputs which almost makes "Time Stamp" the only thing common within all events as shown below:

Domain/Service events:
Nov 30 17:41:23 LogTargetLocalIdentifier [domain][eventCategory][eventLogLevel] serviceType(serviceName): trans(transactionID): some message

System events:
Nov 30 17:47:55 LogTargetLocalIdentifier [eventCategory][eventLogLevel] trans(transactionID): some message

The LogTargetLocalIdentifier seems to be some strang IP format or domain as shown:
192.168.45.23.234.65 or myserver.mydomain.com

and the other part the event can either have [domain][eventCategory][eventLogLevel] or [eventCategory][eventLogLevel].

How would someone regex the LogTargetLocalIdentifier for either IP or domainName? and then token the rest for subparsing?  Any thoughts on this would be greatly appreciated.

Init 1
0 Likes
deathbywedgie1 Frequent Contributor.
Frequent Contributor.

Re: IBM DataPower XML Gateway FlexConnector

For the IP vs domain, capture it as non-whitespace (\\S+). Then you can use the __oneOfHostName and __oneOfAddress operations to define the address/hostname fields in ArcSight.

For the optional domain, you can use regex to make some of it optional. wrap something with (?: in the front and )? at the end to make it optional. The ?: at the beginning identifies that you're wrapping something with parentheses without actually wanting to capture it, and the end question mark makes what you've just enclosed optional. So something like this for that domain definition that doesn't always appear.

(?:\\[\\S+?\\])?

0 Likes
init1 Absent Member.
Absent Member.

Re: IBM DataPower XML Gateway FlexConnector

deathbywedgie,

OK, that makes sense and will help me when I start defining the Event Mapping.  I really didn't absorb that part completely since I'm working on the regex parsing.  Either way  it has enlighten me greatly when I do get there and resolved 90% of my confusion.  The only problem I see with this is that the IP looks like this: 192.168.16.132.16.32 when it should be 192.168.16.132.

Have you ever seen this? or am I just missing something obvious?

Init 1

0 Likes
deathbywedgie1 Frequent Contributor.
Frequent Contributor.

Re: IBM DataPower XML Gateway FlexConnector

Eww, I missed that... IPv6? lol No, I've never observed that in any of my sample logs.
0 Likes
Trusted Contributor.. Dave Munger Trusted Contributor..
Trusted Contributor..

Re: IBM DataPower XML Gateway FlexConnector

Hi,

I also have created a flex connector for datapower.

My flexconnector is for all event sent from datapower appliance throught syslog protocole.

So if you have any question, comment please contact me.

Also, my flexconnector is currently in dev......i will publish a new release soon.

Thank you

Dave

0 Likes
Karl2 Honored Contributor.
Honored Contributor.

Re: IBM DataPower XML Gateway FlexConnector

Dear ,

Sorry by re-take an old post, but I am facing this integration and I can see that the logs are very flexible.

So, do you have available a flex connector for Datapower?

Would you mind to share it?

Best regards,

Karl.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.