How to re-parse raw data records in Sentinel

1 Likes

ISSUE:

Collectors sometimes fail to parse an incoming raw data record correctly. To check if modifications to a collector fix the problem, Sentinel needs to process the same data again.

Another use case is to take data collected on one Sentinel system and feed it to another one to test correlation rules.

HOW TO STEPS:

Identify the EventSourceID (here: 17C387E0-EDB3-102F-BDFB-000C2979A9E3).

 

01 search for original event.png

 

Find raw data files in the online or archive storage (note: data in online storage is not flushed to disk immediately, you possibly have to wait until the file is closed at the top of the hour)

novell@sentinel7:~> ll /var/opt/novell/sentinel/data/rawdata/online/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/
total 4
-rw------- 1 novell novell    0 Jan 23 00:00 23-0000.gz
-rw------- 1 novell novell 9454 Jan 23 09:00 23-0800.gz

novell@sentinel7:~> ll /var/opt/novell/archive/*/rawdata_archive/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/
total 16
-rw------- 1 novell novell 3434 Jan 22 23:15 22-2200.gz
-rw------- 1 novell novell 9852 Jan 23 00:15 22-2300.gz

 

Extract raw data records of interest into a file (see also How to get the raw data record for a Sentinel event). Raw data files are compressed with gz. Therefore use zcat or zgrep to extract the data.

zgrep -h -m 1 nonexisting /var/opt/novell/sentinel/data/rawdata/online/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/*.gz >> /home/novell/raw-data-sample.json

 

You can inject a numeric id into the data to make the resulting events more easy to find. This requires the jq JSON processor. (Note: The TestDataID (rv102) field is not shown in the UI  but you can use its tag in searches, e.g. rv102:4711)

zgrep -h -m 1 nonexisting /var/opt/novell/sentinel/data/rawdata/online/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/*.gz | jq -c '.i_TestDataID=4711' >> /home/novell/raw-data-sample.json

 

Add a File connector to the collector that should parse the collected raw data.

02 add connector.png

 

Give it a proper name.

03 configure connector.png

 

Add an event source to the newly created connector and specify the file name.

04 add event source file path.png

 

Set it to always read data from the beginning on restarts. That will cause Sentinel to automatically re-parse the file if you deploy and updated collector plugin.

05 start behavior.png

 

Set connection mode to Connector Dump.

06 connection mode.png

 

Give it a name, disable the watchdog and run it.

08 event source general.png

 

Search for the event again. There will be two now. The one with the ObserverType of "T" is the replayed one.

09 st-t.png

Labels:

How To-Best Practice
Comment List
Related
Recommended