ALERT! The community will be read-only starting on April 19, 8am Pacific as the migration begins. Read more for important details.
ALERT! The community will be read-only starting on April 19, 8am Pacific as the migration begins.Read more for important details.
Absent Member.
Absent Member.
1734 views

Multiple directories with one FlexConnector

Jump to solution

I am writing a FlexConnector to parse a custom log files (whose file name changes daily).

These log files also exist across three different directories. For example:

/logs/logA/log_0620.log

/logs/logB/log_0620.log

/logs/logC/log_0620.log

So each day, there are three logs that need to be continually crunched by ArcSight, then they get rotated to:

/logs/logA/log_0621.log

/logs/logB/log_0621.log

/logs/logC/log_0621.log

and so on.

My question: do I need to run three FlexConnectors to achieve this? Or is there a way to do this with one FlexConnector? I really want to avoid running multiple connectors if possible.

I've looked at the doc and it's not clear to me how to do this.

Thanks!

Labels (2)
0 Likes
1 Solution

Accepted Solutions
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Look for agents[0].foldertable[x].wildcard= in agent.properties. There is one for every folder you configured. If you specify log_????.log that should read only the log files you specified.

View solution in original post

0 Likes
10 Replies
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Go with "ArcSight FlexConnector Multi-Folder File" as the flexconnector type. It allows you configure multiple folders and multiple parsers (or single parser for multiple folders if you wish)

0 Likes
Absent Member.
Absent Member.

Thank you for the response! I will give that a try.

0 Likes
Absent Member.
Absent Member.

Thank you again for your help. I've configured a Multi-Folder File FlexConnector. But what is not clear to me, and what I cannot find in the documentation, is how I tell it which log files to read.

I want it to read the following log files:

/logs/logA/log_0621.log

/logs/logB/log_0621.log

/logs/logC/log_0621.log

And those log file names change daily (those are dates in the file names). There are other log files in those directories (with different prefixes) that I do not want it to capture.

I know with a simple FlexConnector, I can just do something like:

agents[0].followexternalrotation

agents[0].rotationscheme=Daily

agents[0].rotationschemeparams=log_,MMdd,.log

Can I do that with a Multi-Folder File, however?

Thanks again!

0 Likes
Absent Member.
Absent Member.

Yeah, unfortunately, it went and tried to crunch ALL the log files in those folders, not just the specific ones I wanted. That's bad. Is there a way to get it read just specific [name-rotated] files in those folders?

Thanks!

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Look for agents[0].foldertable[x].wildcard= in agent.properties. There is one for every folder you configured. If you specify log_????.log that should read only the log files you specified.

View solution in original post

0 Likes
Absent Member.
Absent Member.

Gary,

   Thanks again!

---Branden

0 Likes
Absent Member.
Absent Member.

I'm almost there! Just one last tweak...

It's working beautifully except for one thing: every time I start the FlexConnector, it re-parses every log file, resulting in duplicate data. I thought this should only happen if agents[0].startatend was set to "false". I did not set this parameter, which means it should default to true (according to the doc).

Is there any other reason this would happen?

Thanks!

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

set startatend=true and latestlogonly=true (I don't trust defaults, which is why I set startatend just to be safe )

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

I believe startatend only applies if you are reading the file in realtime. I think for some reason your connector is defaulting to batch mode where it reads in all the files that are in the directory. You didn't mention which mode you are running in. Look for agents[0].foldertable[0].processingmode= If you are configured to run in batch mode then you need to do something with the files once you've processed them (agents[0].foldertable[0].mode=DeleteFile)

0 Likes
Absent Member.
Absent Member.

I'm definitely in realtime mode. But I did notice in agent.properties  that startatend was actually set to false. I never set it to that,  however; it did so itself at installation apparently. I [incorrectly]  assumed it took the default because I never specified it. chrisb was right in his response about not trusting defaults- I think  this is why!

Thanks, you and Chris, for your help!

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.