Apache Access Log in CEF

4 Likes

I forward CEF only transaction log (access.log).

Since Apache version 2.4 Error.log also can be customized with CEF - I'll try to update that soon.

access log with CEF works file on Apache 1.X and 2.X

Log format definitions

HTTP

LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|Unknown|end=%{%b %d %Y %H:%M:%S}t app=HTTP cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=%U requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Responce Time cn1=%T out=%B cs4Label=Referer cs4=%{Referer}i dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i acs3Label=X-Forwarser-For cs3=%{X-Forwarded-For}i" CEF_HTTP

Sample output:

CEF:0|Apache|apache||200|GET /index.html|Unknown|end=Nov 07 2014 01:19:42 app=HTTPS cs2=HTTP/1.1 suser=- shost=192.168.0.26 src=192.168.0.26 dhost=www.pheo.net dpt=443 dproc=apache request=/index.html requestMethod=GET fname=/full/path/index.html cs1Label=Virtual Host cs1=toth.pheo.net cn1Label=Responce Time cn1=0 out=3479 cs4Label=Referer cs4=- cs5Label=SSL Protocol cs5=TLSv1.2 cs6Label=SSL CIPHER cs6=ECDHE-RSA-AES128-GCM-SHA256 dvchost=toth.pheo.net dvc=192.168.0.26 deviceProcessName=apache_access_log requestClientApplication=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 cs3Label=X-Forwarser-For cs3=-

HTTPS

LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|Unknown|end=%{%b %d %Y %H:%M:%S}t app=HTTPS cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=%U requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Responce Time cn1=%T out=%B cs4Label=Referer cs4=%{Referer}i cs5Label=SSL Protocol cs5=%{SSL_PROTOCOL}x cs6Label=SSL CIPHER cs6=%{SSL_CIPHER}x dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i cs3Label=X-Forwarser-For cs3=%{X-Forwarded-For}i" CEF_HTTPS

Sample output:

CEF:0|Apache|apache||200|GET /index.html|Unknown|end=Nov 07 2014 01:19:42 app=HTTPS cs2=HTTP/1.1 suser=- shost=192.168.0.26 src=192.168.0.26 dhost=www.pheo.net dpt=443 dproc=apache request=/index.html requestMethod=GET fname=/full/path/index.html cs1Label=Virtual Host cs1=toth.pheo.net cn1Label=Responce Time cn1=0 out=3479 cs4Label=Referer cs4=- cs5Label=SSL Protocol cs5=TLSv1.2 cs6Label=SSL CIPHER cs6=ECDHE-RSA-AES128-GCM-SHA256 dvchost=toth.pheo.net dvc=192.168.0.26 deviceProcessName=apache_access_log requestClientApplication=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 cs3Label=X-Forwarser-For cs3=-

Definition of log:

HTTP

CustomLog "CEF.log" CEF_HTTP

HTTPS

CustomLog "CEF.log" CEF_HTTPS

I use syslog-ng for read and forward file,

Mappings table with sample data.

Description source: mod_log_config - Apache HTTP Server Version 2.4

 

CEF COLUMN NAMEPARAMTERHTTPHTTPSDESCRIPTIONVALUE
HEADERCEF:VersionCEF:0XXCEF:0
Device VendorApacheXXApache
Device ProductapacheXXapache
Device VersionXX
Signature ID%>sXXStatus. For requests that got internally redirected, this is the status of the *original* request --- %>s for the last.200
Name%m %U%qXX%m - The request method
%U - The URL path requested, not including any query string.
%q - The query string (prepended with a ? if a query string exists, otherwise an empty string)
GET /index.html
SeverityUnknownXXUnknown
BODYend%{%b %d %Y %H:%M:%S}tXXTime in format: MMM dd YY HH:mm:ssOct 18 2014 20:37:09
appHTTP/HTTPSXXHTTP
cs2%HXXThe request protocolHTTP/1.1
suser%uXXRemote user (from auth; may be bogus if return status (%s) is 401)-
shost%hXXRemote host213.238.127.224
src%aXXRemote IP-address213.238.127.224
dhost%VXXThe server name according to the UseCanonicalName setting.www.pheo.net
dpt%pXXThe canonical port of the server serving the request443
dprocapacheXXapache
request%UXXThe URL path requested, not including any query string./
requestMethod%mXXThe request methodGET
fname%fXXFilename/full/path/index.html
cs1LabelVirtual HostXXVirtual Host
cs1%vXXThe canonical ServerName of the server serving the request.www.pheo.net
cn1LabelResponce TimeXXResponce Time
cn1%TXXThe time taken to serve the request, in seconds.1
out%BXXSize of response in bytes, excluding HTTP headers.3143
cs4LabelRefererXXReferer
cs4%{Referer}iXX-
dvchost%vXXThe canonical ServerName of the server serving the request.www.pheo.net
dvc%AXXLocal IP-address192.168.0.26
cs5LabelSSL ProtocolXSSL Protocol
cs5%{SSL_PROTOCOL}xXTLSv1.2
cs6LabelSSL CIPHERXSSL CIPHER
cs6%{SSL_CIPHER}xXECDHE-RSA-AES128-GCM-SHA256
deviceProcessNameapache_access_logXXapache_access_log
requestClientApplication%{User-Agent}iXXMozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
cs3LabelX-Forwarser-ForXXX-Forwarser-For
cs3%{X-Forwarded-For}iXX10.11.1.241
Comment List
Anonymous
Parents
  • Replying to myself...

    I implemented to following in the end and it works pretty well. ESM does its normal thing of chopping up the requestURL into its constituent parts.

    LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|%>s|end=%{%b %d %Y %H:%M:%S}t app=%H proto=TCP cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=https://%{HOST}i:%p%U%q requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Response Time cn1=%T in=%I out=%B cs4Label=Referer cs4=%{Referer}i cs5Label=SSL Protocol cs5=%{SSL_PROTOCOL}x cs6Label=SSL CIPHER cs6=%{SSL_CIPHER}x dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i cs3Label=X-Forwarded-For cs3=%{X-Forwarded-For}i" CEF_HTTPS

    Hope others find that useful. I also found myself using file-based connectors as bouncing messages through certain syslog daemons can cause truncation problems on messages that can be tricky to debug as the message is otherwise properly formed.

    If I didn't have that option I'd probably shunt the requestURL and user agent fields right to the end of the message as those have the most potential for being really long and causing the event to overflow some syslog byte-length restriction.

Comment
  • Replying to myself...

    I implemented to following in the end and it works pretty well. ESM does its normal thing of chopping up the requestURL into its constituent parts.

    LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|%>s|end=%{%b %d %Y %H:%M:%S}t app=%H proto=TCP cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=https://%{HOST}i:%p%U%q requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Response Time cn1=%T in=%I out=%B cs4Label=Referer cs4=%{Referer}i cs5Label=SSL Protocol cs5=%{SSL_PROTOCOL}x cs6Label=SSL CIPHER cs6=%{SSL_CIPHER}x dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i cs3Label=X-Forwarded-For cs3=%{X-Forwarded-For}i" CEF_HTTPS

    Hope others find that useful. I also found myself using file-based connectors as bouncing messages through certain syslog daemons can cause truncation problems on messages that can be tricky to debug as the message is otherwise properly formed.

    If I didn't have that option I'd probably shunt the requestURL and user agent fields right to the end of the message as those have the most potential for being really long and causing the event to overflow some syslog byte-length restriction.

Children
No Data
Related Discussions
Recommended