Apache Access Log in CEF

konrad.kaczkows1
4 Likes

I forward CEF only transaction log (access.log).

Since Apache version 2.4 Error.log also can be customized with CEF - I'll try to update that soon.

access log with CEF works file on Apache 1.X and 2.X

Log format definitions

HTTP

LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|Unknown|end=%{%b %d %Y %H:%M:%S}t app=HTTP cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=%U requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Responce Time cn1=%T out=%B cs4Label=Referer cs4=%{Referer}i dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i acs3Label=X-Forwarser-For cs3=%{X-Forwarded-For}i" CEF_HTTP

Sample output:

CEF:0|Apache|apache||200|GET /index.html|Unknown|end=Nov 07 2014 01:19:42 app=HTTPS cs2=HTTP/1.1 suser=- shost=192.168.0.26 src=192.168.0.26 dhost=www.pheo.net dpt=443 dproc=apache request=/index.html requestMethod=GET fname=/full/path/index.html cs1Label=Virtual Host cs1=toth.pheo.net cn1Label=Responce Time cn1=0 out=3479 cs4Label=Referer cs4=- cs5Label=SSL Protocol cs5=TLSv1.2 cs6Label=SSL CIPHER cs6=ECDHE-RSA-AES128-GCM-SHA256 dvchost=toth.pheo.net dvc=192.168.0.26 deviceProcessName=apache_access_log requestClientApplication=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 cs3Label=X-Forwarser-For cs3=-

HTTPS

LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|Unknown|end=%{%b %d %Y %H:%M:%S}t app=HTTPS cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=%U requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Responce Time cn1=%T out=%B cs4Label=Referer cs4=%{Referer}i cs5Label=SSL Protocol cs5=%{SSL_PROTOCOL}x cs6Label=SSL CIPHER cs6=%{SSL_CIPHER}x dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i cs3Label=X-Forwarser-For cs3=%{X-Forwarded-For}i" CEF_HTTPS

Sample output:

CEF:0|Apache|apache||200|GET /index.html|Unknown|end=Nov 07 2014 01:19:42 app=HTTPS cs2=HTTP/1.1 suser=- shost=192.168.0.26 src=192.168.0.26 dhost=www.pheo.net dpt=443 dproc=apache request=/index.html requestMethod=GET fname=/full/path/index.html cs1Label=Virtual Host cs1=toth.pheo.net cn1Label=Responce Time cn1=0 out=3479 cs4Label=Referer cs4=- cs5Label=SSL Protocol cs5=TLSv1.2 cs6Label=SSL CIPHER cs6=ECDHE-RSA-AES128-GCM-SHA256 dvchost=toth.pheo.net dvc=192.168.0.26 deviceProcessName=apache_access_log requestClientApplication=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 cs3Label=X-Forwarser-For cs3=-

Definition of log:

HTTP

CustomLog "CEF.log" CEF_HTTP

HTTPS

CustomLog "CEF.log" CEF_HTTPS

I use syslog-ng for read and forward file,

Mappings table with sample data.

Description source: mod_log_config - Apache HTTP Server Version 2.4

 

CEF COLUMN NAMEPARAMTERHTTPHTTPSDESCRIPTIONVALUE
HEADERCEF:VersionCEF:0XXCEF:0
Device VendorApacheXXApache
Device ProductapacheXXapache
Device VersionXX
Signature ID%>sXXStatus. For requests that got internally redirected, this is the status of the *original* request --- %>s for the last.200
Name%m %U%qXX%m - The request method
%U - The URL path requested, not including any query string.
%q - The query string (prepended with a ? if a query string exists, otherwise an empty string)		GET /index.html
SeverityUnknownXXUnknown
BODYend%{%b %d %Y %H:%M:%S}tXXTime in format: MMM dd YY HH:mm:ssOct 18 2014 20:37:09
appHTTP/HTTPSXXHTTP
cs2%HXXThe request protocolHTTP/1.1
suser%uXXRemote user (from auth; may be bogus if return status (%s) is 401)-
shost%hXXRemote host213.238.127.224
src%aXXRemote IP-address213.238.127.224
dhost%VXXThe server name according to the UseCanonicalName setting.www.pheo.net
dpt%pXXThe canonical port of the server serving the request443
dprocapacheXXapache
request%UXXThe URL path requested, not including any query string./
requestMethod%mXXThe request methodGET
fname%fXXFilename/full/path/index.html
cs1LabelVirtual HostXXVirtual Host
cs1%vXXThe canonical ServerName of the server serving the request.www.pheo.net
cn1LabelResponce TimeXXResponce Time
cn1%TXXThe time taken to serve the request, in seconds.1
out%BXXSize of response in bytes, excluding HTTP headers.3143
cs4LabelRefererXXReferer
cs4%{Referer}iXX-
dvchost%vXXThe canonical ServerName of the server serving the request.www.pheo.net
dvc%AXXLocal IP-address192.168.0.26
cs5LabelSSL ProtocolXSSL Protocol
cs5%{SSL_PROTOCOL}xXTLSv1.2
cs6LabelSSL CIPHERXSSL CIPHER
cs6%{SSL_CIPHER}xXECDHE-RSA-AES128-GCM-SHA256
deviceProcessNameapache_access_logXXapache_access_log
requestClientApplication%{User-Agent}iXXMozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
cs3LabelX-Forwarser-ForXXX-Forwarser-For
cs3%{X-Forwarded-For}iXX10.11.1.241
Comment List
Anonymous
Parents

  • Apologies if this thread is a little old, but a couple of questions:

    In the definition of the CEF_HTTP format there are a couple of spelling mistakes on the label field names and values. For instance:

    • cn1Label=Responce Time
    • acs3Label=X-Forwarser-For (the field name has an 'a' at the start and the value says 'Forwarser' instead of 'Forwarded')

    I just wanted to check and see if this was accidental or if you had found that 'Response' or 'X-Forwarded-For' were keywords in ArcSight and that using them broke the Connector somehow.

    Also, have you tried setting the 'requestUrl' field with %v:%p%U%q instead of just %U? ESM does expect you to put the URL and query string all together in one field and it automatically divides it up into requestUrl into the file name, host, port and query for free!

    I'm going to give it a go and I'll report back if I have any luck.

Comment

  • Apologies if this thread is a little old, but a couple of questions:

    In the definition of the CEF_HTTP format there are a couple of spelling mistakes on the label field names and values. For instance:

    • cn1Label=Responce Time
    • acs3Label=X-Forwarser-For (the field name has an 'a' at the start and the value says 'Forwarser' instead of 'Forwarded')

    I just wanted to check and see if this was accidental or if you had found that 'Response' or 'X-Forwarded-For' were keywords in ArcSight and that using them broke the Connector somehow.

    Also, have you tried setting the 'requestUrl' field with %v:%p%U%q instead of just %U? ESM does expect you to put the URL and query string all together in one field and it automatically divides it up into requestUrl into the file name, host, port and query for free!

    I'm going to give it a go and I'll report back if I have any luck.

Children
No Data
Related Discussions
Recommended

Resources

Support
Documentation
Training
CyberRes Academy
Partner Portal
Contact us
Compliance
Help
Company
Privacy Policy
Terms of Use
Accessibility
Anti-Slavery Statement
Support
How To Buy
Careers
Investor Relations
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.