Apache Access Log in CEF

4 Likes
over 6 years ago

I forward CEF only transaction log (access.log).

Since Apache version 2.4 Error.log also can be customized with CEF - I'll try to update that soon.

access log with CEF works file on Apache 1.X and 2.X

Log format definitions

HTTP

LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|Unknown|end=%{%b %d %Y %H:%M:%S}t app=HTTP cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=%U requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Responce Time cn1=%T out=%B cs4Label=Referer cs4=%{Referer}i dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i acs3Label=X-Forwarser-For cs3=%{X-Forwarded-For}i" CEF_HTTP

Sample output:

CEF:0|Apache|apache||200|GET /index.html|Unknown|end=Nov 07 2014 01:19:42 app=HTTPS cs2=HTTP/1.1 suser=- shost=192.168.0.26 src=192.168.0.26 dhost=www.pheo.net dpt=443 dproc=apache request=/index.html requestMethod=GET fname=/full/path/index.html cs1Label=Virtual Host cs1=toth.pheo.net cn1Label=Responce Time cn1=0 out=3479 cs4Label=Referer cs4=- cs5Label=SSL Protocol cs5=TLSv1.2 cs6Label=SSL CIPHER cs6=ECDHE-RSA-AES128-GCM-SHA256 dvchost=toth.pheo.net dvc=192.168.0.26 deviceProcessName=apache_access_log requestClientApplication=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 cs3Label=X-Forwarser-For cs3=-

HTTPS

LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|Unknown|end=%{%b %d %Y %H:%M:%S}t app=HTTPS cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=%U requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Responce Time cn1=%T out=%B cs4Label=Referer cs4=%{Referer}i cs5Label=SSL Protocol cs5=%{SSL_PROTOCOL}x cs6Label=SSL CIPHER cs6=%{SSL_CIPHER}x dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i cs3Label=X-Forwarser-For cs3=%{X-Forwarded-For}i" CEF_HTTPS

Sample output:

CEF:0|Apache|apache||200|GET /index.html|Unknown|end=Nov 07 2014 01:19:42 app=HTTPS cs2=HTTP/1.1 suser=- shost=192.168.0.26 src=192.168.0.26 dhost=www.pheo.net dpt=443 dproc=apache request=/index.html requestMethod=GET fname=/full/path/index.html cs1Label=Virtual Host cs1=toth.pheo.net cn1Label=Responce Time cn1=0 out=3479 cs4Label=Referer cs4=- cs5Label=SSL Protocol cs5=TLSv1.2 cs6Label=SSL CIPHER cs6=ECDHE-RSA-AES128-GCM-SHA256 dvchost=toth.pheo.net dvc=192.168.0.26 deviceProcessName=apache_access_log requestClientApplication=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 cs3Label=X-Forwarser-For cs3=-

Definition of log:

HTTP

CustomLog "CEF.log" CEF_HTTP

HTTPS

CustomLog "CEF.log" CEF_HTTPS

I use syslog-ng for read and forward file,

Mappings table with sample data.

Description source: mod_log_config - Apache HTTP Server Version 2.4

 

CEF COLUMN NAMEPARAMTERHTTPHTTPSDESCRIPTIONVALUE
HEADERCEF:VersionCEF:0XXCEF:0
Device VendorApacheXXApache
Device ProductapacheXXapache
Device VersionXX
Signature ID%>sXXStatus. For requests that got internally redirected, this is the status of the *original* request --- %>s for the last.200
Name%m %U%qXX%m - The request method
%U - The URL path requested, not including any query string.
%q - The query string (prepended with a ? if a query string exists, otherwise an empty string)
GET /index.html
SeverityUnknownXXUnknown
BODYend%{%b %d %Y %H:%M:%S}tXXTime in format: MMM dd YY HH:mm:ssOct 18 2014 20:37:09
appHTTP/HTTPSXXHTTP
cs2%HXXThe request protocolHTTP/1.1
suser%uXXRemote user (from auth; may be bogus if return status (%s) is 401)-
shost%hXXRemote host213.238.127.224
src%aXXRemote IP-address213.238.127.224
dhost%VXXThe server name according to the UseCanonicalName setting.www.pheo.net
dpt%pXXThe canonical port of the server serving the request443
dprocapacheXXapache
request%UXXThe URL path requested, not including any query string./
requestMethod%mXXThe request methodGET
fname%fXXFilename/full/path/index.html
cs1LabelVirtual HostXXVirtual Host
cs1%vXXThe canonical ServerName of the server serving the request.www.pheo.net
cn1LabelResponce TimeXXResponce Time
cn1%TXXThe time taken to serve the request, in seconds.1
out%BXXSize of response in bytes, excluding HTTP headers.3143
cs4LabelRefererXXReferer
cs4%{Referer}iXX-
dvchost%vXXThe canonical ServerName of the server serving the request.www.pheo.net
dvc%AXXLocal IP-address192.168.0.26
cs5LabelSSL ProtocolXSSL Protocol
cs5%{SSL_PROTOCOL}xXTLSv1.2
cs6LabelSSL CIPHERXSSL CIPHER
cs6%{SSL_CIPHER}xXECDHE-RSA-AES128-GCM-SHA256
deviceProcessNameapache_access_logXXapache_access_log
requestClientApplication%{User-Agent}iXXMozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
cs3LabelX-Forwarser-ForXXX-Forwarser-For
cs3%{X-Forwarded-For}iXX10.11.1.241
Comment List
Anonymous
  • Replying to myself...

    I implemented to following in the end and it works pretty well. ESM does its normal thing of chopping up the requestURL into its constituent parts.

    LogFormat "CEF:0|Apache|apache||%>s|%m %U%q|%>s|end=%{%b %d %Y %H:%M:%S}t app=%H proto=TCP cs2=%H suser=%u shost=%h src=%a dhost=%V dpt=%p dproc=apache request=https://%{HOST}i:%p%U%q requestMethod=%m fname=%f cs1Label=Virtual Host cs1=%v cn1Label=Response Time cn1=%T in=%I out=%B cs4Label=Referer cs4=%{Referer}i cs5Label=SSL Protocol cs5=%{SSL_PROTOCOL}x cs6Label=SSL CIPHER cs6=%{SSL_CIPHER}x dvchost=%v dvc=%A deviceProcessName=apache_access_log requestClientApplication=%{User-Agent}i cs3Label=X-Forwarded-For cs3=%{X-Forwarded-For}i" CEF_HTTPS

    Hope others find that useful. I also found myself using file-based connectors as bouncing messages through certain syslog daemons can cause truncation problems on messages that can be tricky to debug as the message is otherwise properly formed.

    If I didn't have that option I'd probably shunt the requestURL and user agent fields right to the end of the message as those have the most potential for being really long and causing the event to overflow some syslog byte-length restriction.

  • Apologies if this thread is a little old, but a couple of questions:

    In the definition of the CEF_HTTP format there are a couple of spelling mistakes on the label field names and values. For instance:

    • cn1Label=Responce Time
    • acs3Label=X-Forwarser-For (the field name has an 'a' at the start and the value says 'Forwarser' instead of 'Forwarded')

    I just wanted to check and see if this was accidental or if you had found that 'Response' or 'X-Forwarded-For' were keywords in ArcSight and that using them broke the Connector somehow.

    Also, have you tried setting the 'requestUrl' field with %v:%p%U%q instead of just %U? ESM does expect you to put the URL and query string all together in one field and it automatically divides it up into requestUrl into the file name, host, port and query for free!

    I'm going to give it a go and I'll report back if I have any luck.

  • VirtualHost Custom Log:

    [ HTTP ]

    CustomLog "/var/apache2/logs/CEF.log" CEF_HTTP

    [ HTTPS ]

    CustomLog "/var/apache2/logs/CEF.log" CEF_HTTPS

    Simplest syslog-ng config:

    source s_apache_CEF {

      file("/var/apache2/logs/CEF.log"  follow_freq(1));

    };

    destination destination_CEF_SYSLOG{

      tcp("10.11.12.13" port (514));

    };

    log {

      source(s_apache_CEF); destination(destination_CEF_SYSLOG);

    };

  • mod_log_config is not availabe for all versions.

    Anyway - my idea is to most standard solution delivered by web server vendor as it is possible,

    K.

  • How did you configure the syslog-ng to achieve this purpose?

    Thanks in advance for your answer.

    BR

  • Very nice, thank you!

    Why do you say only starting in v2.4? v2.2 and v2.0 (not maintained) also have mod_log_config...

Related Discussions
Recommended