I have been sending ESM events to hadoop using a syslog CEF forwarder for some weeks and now I would like to send previous events from ESM to Hadoop. As the forwarder only works in real time I'm testing this data transfer tool.
I'm exporting the data (in cef, csv and keyvalue) to file destination in order to check the fields exported, etc., and I found some differences in the fields:
- The severity field of the CEF header is exported as text by the forwarder but as number by the transfer tool (for example Low vs 1)
- The categoryDeviceType field is exported as catdt by the forwarder but as categoryDeviceType by the transfer tool
- The Custom Device * Label fields are exported with a different names, for example lblString1Label instead of cs1Label
- The content of some fields are wrong:
- The Geo Latitude and Longitude fields don't have the decimal point. For example: slong=-3773956298 instead of slong=-0.3773956298
- The IPv6 fields don't have the ':'. For example: c6a4=fe80000000000000025xxxxxx instead of c6a4=fe80:0:0:0:25xxxxxx
I hope this information could help you in improving the tool.
I've tested the tool with Express 4.0.