How to re-parse raw data records in Sentinel



Collectors sometimes fail to parse an incoming raw data record correctly. To check if modifications to a collector fix the problem, Sentinel needs to process the same data again.

Another use case is to take data collected on one Sentinel system and feed it to another one to test correlation rules.


Identify the EventSourceID (here: 17C387E0-EDB3-102F-BDFB-000C2979A9E3).


01 search for original event.png


Find raw data files in the online or archive storage (note: data in online storage is not flushed to disk immediately, you possibly have to wait until the file is closed at the top of the hour)

novell@sentinel7:~> ll /var/opt/novell/sentinel/data/rawdata/online/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/
total 4
-rw------- 1 novell novell    0 Jan 23 00:00 23-0000.gz
-rw------- 1 novell novell 9454 Jan 23 09:00 23-0800.gz

novell@sentinel7:~> ll /var/opt/novell/archive/*/rawdata_archive/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/
total 16
-rw------- 1 novell novell 3434 Jan 22 23:15 22-2200.gz
-rw------- 1 novell novell 9852 Jan 23 00:15 22-2300.gz


Extract raw data records of interest into a file (see also How to get the raw data record for a Sentinel event). Raw data files are compressed with gz. Therefore use zcat or zgrep to extract the data.

zgrep -h -m 1 nonexisting /var/opt/novell/sentinel/data/rawdata/online/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/*.gz >> /home/novell/raw-data-sample.json


You can inject a numeric id into the data to make the resulting events more easy to find. This requires the jq JSON processor. (Note: The TestDataID (rv102) field is not shown in the UI  but you can use its tag in searches, e.g. rv102:4711)

zgrep -h -m 1 nonexisting /var/opt/novell/sentinel/data/rawdata/online/17C387E0-EDB3-102F-BDFB-000C2979A9E3/2020*/*.gz | jq -c '.i_TestDataID=4711' >> /home/novell/raw-data-sample.json


Add a File connector to the collector that should parse the collected raw data.

02 add connector.png


Give it a proper name.

03 configure connector.png


Add an event source to the newly created connector and specify the file name.

04 add event source file path.png


Set it to always read data from the beginning on restarts. That will cause Sentinel to automatically re-parse the file if you deploy and updated collector plugin.

05 start behavior.png


Set connection mode to Connector Dump.

06 connection mode.png


Give it a name, disable the watchdog and run it.

08 event source general.png


Search for the event again. There will be two now. The one with the ObserverType of "T" is the replayed one.

09 st-t.png


How To-Best Practice
Comment List