Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..
714 views

Need Flexconnector Help - BIND named Logging

Jump to solution

Hi,

I have been tasked with offering some options for logging our internal DNS traffic.

Our DNS is hosted on BIND with logs currently being output to logfiles locally on each server.

We have the option of putting a smart-connector on each, so I will probably go with a FlexConnector setup.

The logs are called named_debug.log and roll over into named_debug.log.1-99 based on filesize.

The format of the log is as follows.

20-Jan-2015 09:19:47.835 queries: client 10.2.2.1#57812: query: some.other.host.name IN A +

20-Jan-2015 09:19:47.835 queries: client 10.2.2.1#50838: query: 76.11.13.10.in-addr.arpa IN PTR +

20-Jan-2015 09:19:47.836 queries: client 10.2.2.1#61921: query: some.host.name IN A +

20-Jan-2015 09:19:47.836 queries: client 10.2.2.1#63761: query: ctldl.windowsupdate.com IN A +

I tried writing a basic properties file (see at bottom), however it looks like it is not picking up the delimiter (Which is a space).

I get this error in the log file, leading me to believe its not splitting the string properly.

[2015-01-20 14:45:47,531][FATAL][default.com.arcsight.agent.parsers.l][constructAlertFromValues]

com.arcsight.agent.parsers.operation.WrongArgumentsException: Unable to create time stamp with Date Tue Jan 20 00:00:00 EST 2015 and time null

  at com.arcsight.agent.parsers.operation.createTimeStampOperation.getResult(createTimeStampOperation.java:80)

  at com.arcsight.agent.parsers.k$d_.a(k$d_.java:1395)

  at com.arcsight.agent.parsers.k.a(k.java:763)

  at com.arcsight.agent.parsers.k.a(k.java:640)

  at com.arcsight.agent.sdk.d.u.a(u.java:405)

  at com.arcsight.agent.sdk.d.u.a(u.java:313)

  at com.arcsight.agent.sdk.d.u.a(u.java:266)

  at com.arcsight.agent.parsers.j.b(j.java:549)

  at com.arcsight.agent.sdk.d.u.b(u.java:1196)

  at com.arcsight.agent.sdk.c.g.f.b(f.java:290)

  at com.arcsight.agent.baseagents.h.c.run(c.java:857)

  at java.lang.Thread.run(Thread.java:680)

Properties look like this:

regex=(.*)

comments.start.with=#

delimiter=\s

token.count=10

token[0].name=Date_of_the_event

token[0].type=Date

token[0].format=dd-MMM-yyyy

token[1].name=Time_of_the_event

token[1].type=Time

token[1].format=HH:mm:ss.SSS

token[2].name=Action_Type

token[2].type=String

token[3].name=Client_Type

token[3].type=String

token[4].name=Client_Source

token[4].type=String

token[5].name=Client_Action

token[5].type=String

token[6].name=Destination_Address

token[6].type=String

token[7].name=Destination_In_Out

token[7].type=String

token[8].name=Destination_Type

token[8].type=String

token[9].name=Destination_Ext

token[9].type=String

event.deviceReceiptTime=__createTimeStamp(Date_of_the_event,Time_of_the_event)

event.sourceAddress=__regexToken(Client_Source,"(.*)#.*")

event.sourcePort=__regexToken(Client_Source,".*#(.*):")

event.deviceSeverity="Info"

event.categoryObject=__stringConstant("/Network/DNS")

event.categoryDeviceGroup=__stringConstant("/DNS")

event.requestMethod=Client_Action

event.destinationDnsDomain=Destination_Address

event.deviceProduct=__stringConstant("BIND DNS")

event.deviceCustomString1=Destination_Type

event.deviceCustomString1Label=__stringConstant("Pointer Type")

event.deviceCustomString1=Destination_Ext

event.deviceCustomString1Label=__stringConstant("Pointer Ext")

Labels (2)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Attached is a regex file reader properties file based off your sample data and existing properties file above. n.b. this has had minimal testing only, you will need to ensure that it meets your needs.  As mentioned above I'm pretty sure BIND is supported by default by the syslog Smartconnector which would be a better choice if you have the option

View solution in original post

0 Likes
10 Replies
Highlighted
Absent Member.
Absent Member.

I think BIND is suppotred by the native syslog connector.

0 Likes
Highlighted

it looks like it might not be picking up the time correctly.  for troubleshooting, you might want to try mapping the time to a string field to make sure it comes back properly. 

0 Likes
Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..

Tried mapping to string.

It is reading the whole line at once.

I want to delimit it by spaces, but I don't know what character to define in the properties file.

I have tried:

delimiter=\s

delimiter=" "

delimiter=\\s


I guess I could try just reading the whole line in as one token, and then use a series of Regex queries to extract fields, but would prefer to get it delimited properly.





0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Are you creating a regex filereader (sdkrfilereader.properties) or a delimited filereader (sdkfilereader.properties)?  You either need to specify a regex that matches the whole log line with capturing groups for each of the fields (something like "(\\S+)\\s(\\S+) queries: client ([^#]+)#(\\d+): query: (\\S+) IN (.*)") OR a delimiter and then map the resulting fields to tokens

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Attached is a regex file reader properties file based off your sample data and existing properties file above. n.b. this has had minimal testing only, you will need to ensure that it meets your needs.  As mentioned above I'm pretty sure BIND is supported by default by the syslog Smartconnector which would be a better choice if you have the option

View solution in original post

0 Likes
Highlighted
Absent Member.
Absent Member.

Hi Dimiter. Try using this: .

0 Likes
Highlighted
Respected Contributor.
Respected Contributor.

Hi Dimiter,

The best approach to parse date and time is to put that in one token and apply directly to event.deviceReceiptTime.  There is no format conversion needed.

token[0].name=Date_of_the_event

token[0].type=TimeStamp

token[0].format=dd-MMM-yyyy HH:mm:ss.SSS

event.deviceReceiptTime=Date_of_the_event

If I were you, i would refine the regex and forget about delimiter.

Regards,

Martin

0 Likes
Highlighted
Absent Member.
Absent Member.

Hi Dimiter,

As is shown in your example it looks like the parsing issue originates in the "Time_of_the_event" token.

You are able to observe it in the following log line:

"Unable to create time stamp with Date Tue Jan 20 00:00:00 EST 2015 and time null"

As you can see the date is correct, but the is missing...


In the flexconn_devguide document it is stated that the data type format for Time data type is only: HH:mm:ss

while your Time format is: HH:mm:ss.SSS and I think that that's what basically causing your issue.


As a workaround I would advise you NOT to use the "Date" and "Time" token formats at all, but regard them as Strings and then use the __createOptionalTimeStampFromString operation to map the event.


For example:

comments.start.with=#

delimiter=\s

token.count=10

token[0].name=Date_of_the_event

token[0].type=String

token[1].name=Time_of_the_event

token[1].type=String

event.deviceReceiptTime=__createOptionalTimeStampFromString(Date_of_the_event,Time_of_the_event)


Or if that doesn't work because of the timestamp format you can also use the following function:

event.deviceReceiptTime=__parseMultipleTimeStamp(__createOptionalTimeStampFromString(Date_of_the_event,Time_of_the_event))

Good Luck!!

Tal.

0 Likes
Highlighted
Honored Contributor.. Honored Contributor..
Honored Contributor..

Richard's Solution worked.

Like others pointed out, dropping the delimiter and just using a custom Regex seems to work better for space-delimited files.

D

0 Likes
Highlighted
Absent Member.
Absent Member.

I pretty much always use regex, even when delimited. The main reason being is that you have extra functionality within the regex parser (for example, line.include.regex, line.ignore.regex).

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.