Highlighted
Absent Member.
Absent Member.
1161 views

Help with remote apache logs

Jump to solution

Hello,

I need to capture logs from a device producing apache access logs.  This is a Novell Access Gateway or Novell Access Manager (NAG/NAM).  Arcsight sees the logs as apache logs and imports a static file with no problem (thx chrisb!).  The problem is that the device is more or less a SUSE Linux black box and I am not sure if I will be able to install a connector directly on the box.  I do have ssh access.  SAMBA is not running on the device so I cannot "map" a drive to it to point to the log files.  Is there a way I can pull these files into a remote Windows collector?

The option I came up with is a script to copy the log file periodically and import it that way.  The log file does change names on a log rotate so the name may be a challenge.

Any other suggestions?

Thanks,

Mike

Labels (1)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hmmm...

It works perfectly for me:

parser.png

A few suggestions -

1.  Make sure there's no whitespace after the regex or any field in the parser for that matter.  It's bitten me more times than I can count

2.  Pull your parser up in the arcsight regex tool (see above) and test it with a number of log lines from the log.  Just don't use the generate button, it never works for me .  You can get to the tool by runnign $ARCSIGHT_HOME/current/bin/arcsight regex.  You just need to make sure your parser is sitting in the flexagent directory.  I personally use it by writing my regex in the Regex Coach, then copying that regex into the arcsight regex tool and letting it change it into arcsight-specific syntax (like escaping colons).

3.  If you have some time, "Mastering Regular Expressions" is a great read for learning regex, and I keep a regex cookbook and regex pocket guide with me at all times.  Then again, I don't get out much .

View solution in original post

0 Likes
10 Replies
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Can you do an NFS mount?  Alternatively, you can use syslog to send the logs and there is an Apache syslog connector already available - provided you can configure it on the box.

And you're welcome

0 Likes
Highlighted
Absent Member.
Absent Member.

Trying to just get the regex portion to work for Apache2 logs.  Using  your regex statement:

regex=(\\S+)\\s+(\\S+) (\\S+) \\[(.+?)\\] \\"((\\S+) (.*?)\\s*(?:HTTP/(.*?))?)\\" (\\d+) -?(\\d+)?\\s*(?:\\"(.*?)\\")?\\s*(?:\\"(.*?)\\")?\\s*(\\d+)?\\s*(\\S+)?

For straight up Apache2 logs:

127.0.0.1 - - [29/Mar/2011:13:51:58 -0400] "GET / HTTP/1.1" 403 1046 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.6) Gecko/20100626 SUSE/3.6.6-1.2 Firefox/3.6.6"

Not parsing at all.  The log is showing up in the Message line.

I am not a regex person at all.  Any suggestions?

Thanks,

mshannon

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Hmmm...

It works perfectly for me:

parser.png

A few suggestions -

1.  Make sure there's no whitespace after the regex or any field in the parser for that matter.  It's bitten me more times than I can count

2.  Pull your parser up in the arcsight regex tool (see above) and test it with a number of log lines from the log.  Just don't use the generate button, it never works for me .  You can get to the tool by runnign $ARCSIGHT_HOME/current/bin/arcsight regex.  You just need to make sure your parser is sitting in the flexagent directory.  I personally use it by writing my regex in the Regex Coach, then copying that regex into the arcsight regex tool and letting it change it into arcsight-specific syntax (like escaping colons).

3.  If you have some time, "Mastering Regular Expressions" is a great read for learning regex, and I keep a regex cookbook and regex pocket guide with me at all times.  Then again, I don't get out much .

View solution in original post

0 Likes
Highlighted
Absent Member.
Absent Member.

You are correct.  Your regex was perfect, my not naming my config file correctly was the fail.

Next will be the folder follower part.

Thanks again!

mshannon

0 Likes
Highlighted
Absent Member.
Absent Member.

I saw in another post a discussion of apache rotating logs using a multifolder follower.  When I look a the agent.properties there it looks like a file reader was used but the config is for a folder follower.  Is that possible or should I be using the multiple folder follower flexagent install?

mshannon

0 Likes
Highlighted
Absent Member.
Absent Member.

Just out of curiosity, you don't have SAMBA installed, that means no NFS, which leads to no reading of file, then why flexagent? since you won't be installing the connector on your box. Do you have syslog configured, that might be the easiest solution to send logs from access.log to arcsight connector, right. Let me know if I am missing something.

Gerra

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

It depends on how you're pulling your logs in.  If you're copying them from NAM to a server & parsing them there, I'd use a trigger file.  If you're pulling them directly from the NAM server, you can use this (it's how I'm pulling in mine):

agents[0].foldertable[0].badsubfolder=bad
agents[0].foldertable[0].configfile=apacheaccess
agents[0].foldertable[0].configfolder=/opt/arcsight/connector_1/current/user/agent/flexagent/
agents[0].foldertable[0].configtype=sdkrfilereader
agents[0].foldertable[0].delay=10000
agents[0].foldertable[0].encoding=
agents[0].foldertable[0].extractfieldnames=
agents[0].foldertable[0].extractregex=
agents[0].foldertable[0].extractsource=File Name
agents[0].foldertable[0].fixedlinelength=-1
agents[0].foldertable[0].fixedlinelengthcontains=Fixed Number Of Characters
agents[0].foldertable[0].folder=/mnt/idontthinksotim
agents[0].foldertable[0].followexternalrotation=false
agents[0].foldertable[0].maxretries=-1
agents[0].foldertable[0].minfilelenght=-1
agents[0].foldertable[0].mode=PersistFile
agents[0].foldertable[0].modeoptions=processed
agents[0].foldertable[0].monitoringinterval=-1
agents[0].foldertable[0].preservestate=false
agents[0].foldertable[0].processfoldersrecursively=false
agents[0].foldertable[0].processinglimit=256
agents[0].foldertable[0].processingmode=realtime
agents[0].foldertable[0].processingthreshold=-1
agents[0].foldertable[0].processingtimeout=-1
agents[0].foldertable[0].retryinterval=1000
agents[0].foldertable[0].sleeptime=5000
agents[0].foldertable[0].startatend=true
agents[0].foldertable[0].triggerextension=.done
agents[0].foldertable[0].usealternaterotationdetection=true
agents[0].foldertable[0].usefieldextractor=false
agents[0].foldertable[0].usenonlockingwindowsfilereader=false
agents[0].foldertable[0].usetriggerfile=false
agents[0].foldertable[0].wildcard=*.access.log

If you're doing the copy, change usetriggerfile to true, and add to your script "touch $DIRWITHFILES/apache.access.log.done" or some variant of that.  When doing a copy, I always use a trigger file because if the transfer is slow or the file is big, the connector can start processing it and get to the end before the transfer is done so the whole log file doesn't get parsed.

0 Likes
Highlighted
Absent Member.
Absent Member.

At this time I am going on the assumption I can actually put a connector install on the box.   The copying is going to be troublesome.

I will give this agent.properties a try.

Thanks,

mshannon

0 Likes
Highlighted
Absent Member.
Absent Member.

Yes no NFS or remote syslog.  This system is a Novell NAM box preconfigured with Novell logging and seems to not be friendly to modifications to almost all the ways I wanted to try.  The log rotation is not friendly either, or at least was unfamiliar to me.  Instead of renaming the old file and keeping the original file name, they generate a new file with a date like name and that becomes the current log file and continually changes. So I started looking at some form of flexconnector.  I was successful at installing a connector on a dev NAM device.  So with luck I wll be able to push right from the device to the manager.

Thanks,

mshannon

0 Likes
Highlighted
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

If NAM is doing it's own naming, you can actually use that foldertable entry almost verbatim just changing the wildcard option and keeping the PersistFile option set so that it knows what's already been read.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.