ScorpionSting Absent Member.
Absent Member.
604 views

Apache HTTP

I used to have this Connector running fine....then I decided it would be a good idea to update some plugins *shoot me now*. I've tried reverting to only "released" versions (Apache & Syslog), but still my events are mangled!

Raw:

{"i_Second":"11","s_Date":"Apr 09 08:10:11","i_milliseconds":"1554761411000","i_TrustDeviceTime":"","i_DayOfMonth":"9","s_raw_message2":"<133>Apr 9 08:10:11 xxxxxx APACHE_HTTPD: www.isag.melbourne x.x.x.x - - [09\/Apr\/2019:08:10:04 +1000] \"GET \/media\/images\/favicon_16x16.ico HTTP\/1.1\" 200 99678 \"-\" \"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko\/20100101 Firefox\/66.0\"","i_syslog_facility":"16","s_RV24":"B4F289F0-7F5F-1036-8B4A-000C294C00E8","s_RV25":"6349D9E9-3C75-1037-B6BD-000C294C00E8","s_RV22":"B4F289F0-7F5F-1036-8B40-000C294C00E8","s_RV23":"B4F289F0-7F5F-1036-8B48-000C294C00E8","s_RV21":"C76D2820-C395-1029-BB86-001321B5C0B3","CONNECTION_MODE":"map","s_SyslogRelayIp":"x:x:x:x:x:x:x:x","i_Hour":"8","sf":"","i_syslog_priority":"133","CONNECTION_METHOD":"SYSLOG","s_Version":"2019.1r1-201902270522-SNAPSHOT","s_Body":"APACHE_HTTPD: www.isag.melbourne x.x.x.x - - [09\/Apr\/2019:08:10:04 +1000] \"GET \/media\/images\/favicon_16x16.ico HTTP\/1.1\" 200 99678 \"-\" \"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko\/20100101 Firefox\/66.0\"","i_Minute":"10","s_AppId":"APACHE_HTTPD","i_Year":"2019","s_MessageOriginatorHost":"xxxxxx","s_chainId":"1554704691183","s_sha256Hash":"f180ec1d8212525bc2a99db5866fb940f3997f9a21a2e128b0ab0906c95a8f60","i_Month":"3","i_syslog_severity":"5","s_chainSequence":"1558","s_MessageOriginatorPort":"54686","i_RXBufferLength":"244","i_Type":"2","EventSourceManagerID":"C76D2820-C395-1029-BB86-001321B5C0B3","CollectorID":"B4F289F0-7F5F-1036-8B40-000C294C00E8","EventSourceGroupID":"B4F289F0-7F5F-1036-8B48-000C294C00E8","EventSourceID":"B4F289F0-7F5F-1036-8B4A-000C294C00E8","EventRecordID":"6349D9E9-3C75-1037-B6BD-000C294C00E8","ChainID":"1554704691183","ChainSequence":"1558","EventDate":"04\/09\/2019 07:58:19.144 +1000","TenantID":"101100"}


Event:

TargetHostClass(rv81),ObserverHostDepartment(obsdep),TargetHostFunction(rv82),SourceHostID(rv77),RetentionPolicyName(rv192),TargetHostDepartment(rv98),SourceHostGeoData(srcgeo),Severity(sev),TargetHostName(dhn),ObserverIP(obsip),SearchTargetID(rv172),TargetHostCriticality(rv84),RetentionPolicyID(rv171),TargetIP(dip),TargetServiceName(dp),TenantID(tid),EventTime(dt),ObserverTZMonth(estzmonth),CollectorNodeName(port),SourceHostName(shn),VendorOutcomeCode(voc),ObserverHostClass(obsclass),SourceIP(sip),ReporterIP(repip),ReporterHostID(repassetid),EventID(id),TargetHostLongitude(dlong),ObserverHostGeoData(obsgeo),Vulnerability(vul),SentinelProcessTime(spt),TargetHostGeoData(destgeo),ObserverTZDayInYear(estzdiy),EventName(evt),SentinelServiceID(src),ObserverTZDayInWeek(estzdiw),ObserverTZDayInMonth(estzdim),SourceHostLongitude(srclong),ProductName(pn),SentinelProcessingComponent(rt2),ObserverHostFunction(obsfunc),ObserverHostName(sn),ObserverType(st),TenantHierarchyID(rv1),CollectorPluginName(agent),IdTApprovedAccountAdmins(cv81),IDManagedSystems(cv82),TargetHostLatitude(dlat),ObserverHostLongitude(obslong),NetworkZone(cv97),SentinelID(rv121),CollectorPluginID(rv122),ObserverHostCriticality(obscrit),Message(msg),ObserverTZHour(estzhour),SourceHostLatitude(srclat),TargetHostCountry(rv30),ObserverCategory(rv32),MinRetentionDate(rv164),ObserverHostLatitude(obslat),ObserverTZ(estz),TenantName(rv39),ConnectorID(rv23),ObserverTZMinute(estzmin),CollectorID(rv22),RawDataRecordId(rv25),EventSourceID(rv24),CollectorManagerID(rv21),SourceHostCountry(rv29),Tags(rv145),ObserverHostCountry(obscountry)
Physical,ISAG,Main Server,0,System Events,ISAG,"-37.8330862,144.9455179",4,xxxxxx,x.x.x.x,B4F289F0-7F5F-1036-96D5-000C294C00E8,Critical,6E1CCA35-4BD4-102D-91CD-000C2907C76D,192.168.245.3,httpd,101100,Tue Apr 09 07:58:19 AEST 2019,3,Apache HTTPD,www.isag.melbourne,GET /media/images/favicon_16x16.ico HTTP/1.1,Physical,x.x.x.x,x:x:x:x:x:x:x:x,0,BD49D9E9-3C75-1037-B677-000C294C00E8,144.9455179,"-37.8330862,144.9455179",0,Tue Apr 09 07:58:19 AEST 2019,"-37.8330862,144.9455179",99,+1000],B4F289F0-7F5F-1036-8B40-000C294C00E8,3,9,144.9455179,Apache HTTP Server,Apache HTTP Server,Main Server,xxxxxx,A,0,Apache HTTP Server,0,0,-37.8330862,144.9455179,LAN,B4F289F0-7F5F-1036-9632-000C294C00E8,A5E13B30-5A4A-102C-9069-005056C00008,Critical,+1000] by www.isag.melbourne,7,-37.8330862,AU,WEB,Mon Jul 08 10:00:00 AEST 2019,-37.8330862,Australia/Melbourne,ISAG,B4F289F0-7F5F-1036-8B48-000C294C00E8,58,B4F289F0-7F5F-1036-8B40-000C294C00E8,6349D9E9-3C75-1037-B6BD-000C294C00E8,B4F289F0-7F5F-1036-8B4A-000C294C00E8,C76D2820-C395-1029-BB86-001321B5C0B3,AU,Sentinel,AU


I know its hard to spot, but it appears that the date parsing is causing the problems....it doesn't seem to cope with the " +1000" GMT identifier, so all the fields get offset and its just plain rubbish.

I grabbed the SDK and tried looking, but am completely lost. I think its in the release.js (???) line 282:


var evtDate = DateTime.parseExact(this.fields[3].substr(1), "dd/MMM/yyyy:HH:mm:ss", this.fields[4].substr(0, 5));


But unsure, and only reference to parseExact is Microsoft's C# one, but this is JS....so...????

Visit my Website for links to Cool Solution articles.
0 Likes
6 Replies
ScorpionSting Absent Member.
Absent Member.

Re: Apache HTTP

I take that back, I don't think it's date...


LogFormat "%v %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""


That fails....but I have something on the network that seems to hit this format:


LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""


So, I think its issues in parsing the Virtual Host in the log file....but I really need that field to identify which host as I have several hosts sending through.

Visit my Website for links to Cool Solution articles.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Apache HTTP

On 2019-04-09 00:34, ScorpionSting wrote:
> So, I think its issues in parsing the Virtual Host in the log
> file....but I really need that field to identify which host as I have
> several hosts sending through.


The collector was designed to parse the vhost from the log file name.
Did you setup your directory layout according to
https://www.netiq.com/support/sentinel/plugins/pre/collectors/Apache_HTTP-Server_2011.1r5-201903060149-preview.html#DeviceConfiguration_section
?

NOTE: The actual file names of the log files is not critical (adjust the
file pattern appropriately), but that the virtual domain must be in the
path two levels up from the log files to be captured so that the
Collector can properly determine the virtual domain name

--
Norbert
0 Likes
ScorpionSting Absent Member.
Absent Member.

Re: Apache HTTP

klasen;2498032 wrote:
On 2019-04-09 00:34, ScorpionSting wrote:
> So, I think its issues in parsing the Virtual Host in the log
> file....but I really need that field to identify which host as I have
> several hosts sending through.


The collector was designed to parse the vhost from the log file name.
Did you setup your directory layout according to
https://www.netiq.com/support/sentinel/plugins/pre/collectors/Apache_HTTP-Server_2011.1r5-201903060149-preview.html#DeviceConfiguration_section
?

NOTE: The actual file names of the log files is not critical (adjust the
file pattern appropriately), but that the virtual domain must be in the
path two levels up from the log files to be captured so that the
Collector can properly determine the virtual domain name

--
Norbert


Using rsyslog...custom log file for each vhost, so access/error isn't actually used by main logging (just all the other c**p that tries to http)...

Visit my Website for links to Cool Solution articles.
0 Likes
ScorpionSting Absent Member.
Absent Member.

Re: Apache HTTP

klasen;2498032 wrote:
On 2019-04-09 00:34, ScorpionSting wrote:
> So, I think its issues in parsing the Virtual Host in the log
> file....but I really need that field to identify which host as I have
> several hosts sending through.


The collector was designed to parse the vhost from the log file name.
Did you setup your directory layout according to
https://www.netiq.com/support/sentinel/plugins/pre/collectors/Apache_HTTP-Server_2011.1r5-201903060149-preview.html#DeviceConfiguration_section
?

NOTE: The actual file names of the log files is not critical (adjust the
file pattern appropriately), but that the virtual domain must be in the
path two levels up from the log files to be captured so that the
Collector can properly determine the virtual domain name

--
Norbert


Not sure why this worked previously and only started to misbehave after I started playing with plugin updates....I haven't changed apache vhost logging config for some time...

Its just a complete pain having to configure directories just for Sentinel then having to retrofit that change into awstats, logrotate, etc...

Would be nice if there was some way to configure the collector to map the file path to the log syntax using the apache % parameters... (i.e. /path/to/apache.log = "%v %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"")...then have the release.js use that for mapping.

Visit my Website for links to Cool Solution articles.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Apache HTTP

Hi,

in comparison to the combined LogFormat


LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-agent}i\"" combined


the collector was written to handle, your has an extra field (%v) at the
beginning:


LogFormat "%v %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""


You could try moving it to the end and then assign this.fields[10] to
this.domain in customParse()

Otherwise you need to customize Record.prototype.parseAccess()

--
Norbert
0 Likes
ScorpionSting Absent Member.
Absent Member.

Re: Apache HTTP

klasen;2498123 wrote:
Hi,

in comparison to the combined LogFormat


LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-agent}i\"" combined


the collector was written to handle, your has an extra field (%v) at the
beginning:


LogFormat "%v %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""


You could try moving it to the end and then assign this.fields[10] to
this.domain in customParse()

Otherwise you need to customize Record.prototype.parseAccess()

--
Norbert


Thanks Norbert

Visit my Website for links to Cool Solution articles.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.