How Monitor Event are Dropping?
How do we track or troubleshoot all events which are coming to connector parsing or not? the reason why is I'm asking this normally, with high event volumes, file queue can build up faster leading to significant delays, up to the point when file queue becomes full, connector starts dropping events. The person who monitor ArcSight never notice it, am I right at this point? Since analyzer for admin never knows it and no proper way to track it. So How is there any way to t-shoot this problem?
Not entirely sure I know what you are saying here.
With the later releases of the SmartConnector framework, we provide the ability to take unparsed logs and to place them in a separate file for later processing (which is fine, because we process the timestamp correctly and this will be correlated correctly). But this is a mechanism that doesnt use a queue as such and hence wont drop data, so not sure that this is the problem here.
What a connector will do is fill the local cache as needed, but by default this is quite large (if I remember correctly thats 10 files of 1GB each!). So you shouldnt be getting dropped data there either.
From the sounds of it, you are suffering higher loads due to the time being processed on the incorrect parsed messages (incorrect parsing is probably the single biggest killer of processing rate on a connector) and as a result this is causing other issues with the connector. In this case, I would absolutely look to fix this as a matter of urgency. Get this sorted and pretty much everything else will be better.
If you cant fix this, I would look at mechanisms to optimize the connector too. For example, increase the memory (broken parsing will cause the processing to fail (generate an error) and start again, so memory will help but wont necessarily fix anything. But more memory is good. Also, look at options to reduce the load on the connector as a whole, such as make sure you arent doing excessive aggregation or processing.
But to follow on though, I would recommend you take a look at the following link:
I mention the Activate content that is available for connector monitoring. While I would recommend that you are careful with this and dont look to just implement it without understanding the implications (such as how many connectors you have), please note that it does specifically have some content around identifying when a connector isnt parsing data correctly AND it will give you a dashboard of the events as they happen! Maybe you can take a look at the content and take what you need? That would give you a simple but very effective dashboard of unparsed messages that you can easily identify.
Thanks for reply. Recently I integrated Cyber Ark with ArcSight but I reused bluecoat syslog connector to collect the event instead of having another syslog connector. What I noticed so far is some events from Cyber Ark has passed and most of them not parsed. I couldn't find them in anywhere.