Highlighted
Absent Member.
Absent Member.
317 views

Sample Collector build


Hi,

I used the SDK to create a new collector but will do with some help....

I want to build a collector with multiple map files so lets say below is
an example log:

01/03/13 14:33:22 NOTICE 10.10.4.3:5555 65.46.44.56:80 access_out
user123

Regex for the log would appear something like this:

(\S+)\s(\S+)\s([A-Z]{4,7})\s(\S+):(\d+)\s(\S+):(\d+)\s(\S+)\s(\S+)
Finally, below are the RegEx groups and what they extract:

RegExp.$1 = date
RegExp.$2 = time
RegExp.$3 = severity (need a map to convert to numbers)
RegExp.$4 = source IP
RegExp.$5 = source port
RegExp.$6 = target IP
RegExp.$7 = target port
RegExp.$8 = vendor message (need a map to covnert to a descriptive
message)
RegExp.$9 = Initiating username

Here is my problem.....

How do I create a new map file (containing "name" and "value" fields)
and link the respective regex values extracted from the log? In our
example I need one for severity where say "NOTICE" will be given a
severity value of 1; "WARNING" given a severity value of 4 and so on.

I also need a second map file for vendor message where "access_out" will
have a description value of "Successful connection established" or
something like this.

Now I need the collector to use the values from the map file and not the
one that are present in the log. Obviously I want to build such a
collector so I can add/remove/edit descriptions from a map file rather
than messing with a code.


Can someone guide me on how do I do this within the SDK please? If such
a collector is already build (i.e. mapping), could someone please send
me the development build?


Any help is much appreciated.


Regards,
Pras


--
pimpalp
------------------------------------------------------------------------
pimpalp's Profile: https://forums.netiq.com/member.php?userid=5587
View this thread: https://forums.netiq.com/showthread.php?t=50191

0 Likes
4 Replies
Highlighted
Absent Member.
Absent Member.

Hi Pras,
almost all collectors use KeyMaps
(http://www.novell.com/developer/plugin-sdk/ref/api/2011.1/symbols/KeyMap.ht
ml) to this kind of mapping, e.g. to set severity.

Basically you need to do the following:

1) create a CSV file, e.g. serverity.map

2) create a KeyMap object in init()
this.MAPS.severity = new KeyMap(this.CONFIG.collDir + "severity.map");

3) map the value in normalize()
this.severity = instance.MAPS.severity.lookup(RegExp.$3, 0);

Norbert

P.S.: If you have complex regexes, I suggest you use the following pattern,
as RegExp.$N is limited to 9 capturing groups:

var tokens =
/^(\S+)\s(\S+)\s([A-Z]{4,7})\s(\S+):(\d+)\s(\S+):(\d+)\s(\S+)\s(\S+)/.exec(m
essage)
if (tokens != null) {
this.severity = instance.MAPS.severity.lookup(tokens[3], 0);
}

>>> On 06.03.2014 at 14:04, pimpalp<pimpalp@no-mx.forums.netiq.com> wrote:


> Hi,
>
> I used the SDK to create a new collector but will do with some help....
>
> I want to build a collector with multiple map files so lets say below is
> an example log:
>
> 01/03/13 14:33:22 NOTICE 10.10.4.3:5555 65.46.44.56:80 access_out
> user123
>
> Regex for the log would appear something like this:
>
> (\S+)\s(\S+)\s([A‑Z]{4,7})\s(\S+):(\d+)\s(\S+):(\d+)\s(\S+)\s(\S+)
> Finally, below are the RegEx groups and what they extract:
>
> RegExp.$1 = date
> RegExp.$2 = time
> RegExp.$3 = severity (need a map to convert to numbers)
> RegExp.$4 = source IP
> RegExp.$5 = source port
> RegExp.$6 = target IP
> RegExp.$7 = target port
> RegExp.$8 = vendor message (need a map to covnert to a descriptive
> message)
> RegExp.$9 = Initiating username
>
> Here is my problem.....
>
> How do I create a new map file (containing "name" and "value" fields)
> and link the respective regex values extracted from the log? In our
> example I need one for severity where say "NOTICE" will be given a
> severity value of 1; "WARNING" given a severity value of 4 and so on.
>
> I also need a second map file for vendor message where "access_out" will
> have a description value of "Successful connection established" or
> something like this.
>
> Now I need the collector to use the values from the map file and not the
> one that are present in the log. Obviously I want to build such a
> collector so I can add/remove/edit descriptions from a map file rather
> than messing with a code.
>
>
> Can someone guide me on how do I do this within the SDK please? If such
> a collector is already build (i.e. mapping), could someone please send
> me the development build?
>
>
> Any help is much appreciated.
>
>
> Regards,
> Pras
>





0 Likes
Highlighted
Absent Member.
Absent Member.


Just one other caution: the regex results that are returned into
RegExp.$N are "temporary" - any subsequent application of a regex will
overwrite them. For that reason, it's likely a good idea to go ahead and
store each parsed-out value into the Record object and then manipulate
that. Something like:
/\S+)\s(\S+)\s([A-Z]{4,7})\s(\S+):(\d+)\s(\S+):(\d+)\s(\S+)\s(\S+)/.test(this.s_RXBufferString);
this.time=RegExp.$2;
this.severity=RegExp.$3;
this.sip=RegExp.$4;
....
etc...

Then you can further convert those values in your normalize() method as
appropriate, as Norbert mentioned, using KeyMaps.

Lastly, in your Rec2Evt.map, you need to list each Record member
property (after parsing and normalization) and map it to the formal
output event schema.


--
DCorlette
------------------------------------------------------------------------
DCorlette's Profile: https://forums.netiq.com/member.php?userid=323
View this thread: https://forums.netiq.com/showthread.php?t=50191

0 Likes
Highlighted
Absent Member.
Absent Member.


did solved the problem? can you post the entire release.js? thank you


--
diogobsb
------------------------------------------------------------------------
diogobsb's Profile: https://forums.netiq.com/member.php?userid=7491
View this thread: https://forums.netiq.com/showthread.php?t=50191

0 Likes
Highlighted
Absent Member.
Absent Member.


Hi,

My Maps are not working......... Here are details - please help.

Example data String:
12/06/2014 10:18:49 : duration 1m 52s : admin : Backup complete. : The
backup file is located at:
/sentinel_backup/BACKUP/Sentinel_App/sentinel_bkp-06_12_2014.tar.gz


Regex:
(.*?)\s:\s(duration.*?)\s:\s(.*?)\s:\s((.*?)\.\s:.*)

Event Name (evt) on RegExp.$5 - our example will give "Backup complete"
Replacing " " with "_" using "this.evt.replace(/\s/, "_")" command
so our output is "Backup_complete"

Severity Map file as follows:
Backup_failed,4
Backup_complete,1


Declared a map file in init() as follows:
this.MAPS.severityMap = new KeyMap(this.CONFIG.collDir +
"severity.map");


using the map in .parse() as follows:
this.sevr = instance.MAPS.severityMap.lookup(this.evt);


Using Rec2Evt.map file as follows:
Severity,sevr


Unfortunately the map isn't working and I'm not sure where I'm going
wrong!!


--
pimpalp
------------------------------------------------------------------------
pimpalp's Profile: https://forums.netiq.com/member.php?userid=5587
View this thread: https://forums.netiq.com/showthread.php?t=50191

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.