Aggregation Fields - WUC
We recently installed the Unified connector for pulling Win 2008 domain controller logs.
We configured the connector to use field based aggregation. I added the target user name and source user name fields in addition to the default fields.
Should I include any other fields? Please help me, if you have done a similar setup.
We are using the following fields for aggregation:
Also, take a look on the "filter-out" option - the windows logs collect a lot of white noise, you must filter out everything that is not relevant to you.
Thanks Marian for responding. One problem with aggregating on only these fields is that often critical information is logged in deviceCustomStrings. Example: When a user is added to a group, the custom fields will log very valuable information. Have you considered aggregating on these fields? Or does this not apply for your environment?
Hello, Uday. We will look into your suggestion. On the other hand, I think a little unusually to have more than a single "add user to group" event for the lifespan of the aggregation window, so this event will fall out of the aggregation mechanism.
Thank you for your suggestion.
This is what we currently use on our WUC.
and Filter out Condition:
( categoryOutcome EQ "/Success" And deviceEventClassId In ( "Microsoft-Windows-Security-Auditing:4624", "Microsoft-Windows-Security-Auditing:4634", "Security:538", "Security:540", "Security:576" ) And destinationUserName Contains "$" ) Or ( categoryOutcome EQ "/Success" And deviceEventClassId In ( "Security:672", "Security:673", "Security:674", "Security:617", "Security:677", "Security:676" ) ) Or name In ( "A Kerberos service ticket was requested.", "A Kerberos authentication ticket (TGT) was requested.", "Privileged Service Called", "Privileged object operation", "A new process has been created", "A process has exited" ) Or deviceEventClassId In ( "Microsoft-Windows-Security-Auditing:4662", "Microsoft-Windows-Security-Auditing:4672", "Microsoft-Windows-Security-Auditing:4932", "Microsoft-Windows-Security-Auditing:4933", "Security:836", "Security:837", "Security:552", "Security:562", "Security:565", "Security:566", "Security:576", "Security:578" )
Have you seen any performance differences with "in" vs "=". This is at the connector level so I'm not really sure it would make a difference since this isn't indexed yet, but just crossed my mind.
Good question - always a good idea to steer clear from CONTAINS and IN where possible, since they are relatively slow operators, especially on indexed data. I don't think its so critical at the connector level, but its always worth double checking.
Try using Logfu to make sure you have the relevant data on how the connector is working though. It should give you some indications if you have it operating slowly and not being able to keep up. Usually with excessive memory and caching being a good clue here, but you should be able to spot something.
The best way I believe to approach this is first to define the events that you want to send to ESM.
Once you have the events, then review the field mapping for each event to the ArcSight Schema.
What you should then have is a list of fields that you need.
When using aggregation you need to ensure that you will get the fields that you need. However it doesn't mean that you need to aggregate on all those fields. What I normally end up doing is having a list of key fields and I also enable "Preserve Common Fields".
For example if you want to maintain all the agent information, then I normally select agentID only. This is because all other agent related fields would be the same from the same connector and thus be shown, because you enabled "Preserve Common Fields".
Likewise if you select all the fields, then of course you won't gain much in aggregation. Proxy data is a typical example here. If you need the URL information, then you need to aggregate on it, but of course URL's are unique, so the aggregation benefit you gain is somewhat low (we normally get about 15-20%).
From a performance perspective we found that the time frame and event count has a much bigger impact on performance than the number of aggregated fields does.