Highlighted
malik.kececiogl Respected Contributor.
Respected Contributor.
509 views

Best practice for filterout.

Hi,

Do you have any best practice methods for filterout garbage logs, for Microsoft type of logs there are lots of garbage logs i want to filter out , not only microsoft but also there are lots of other vendors that creates lots of garbage logs , indeed i know how to filter out but i dont know exactly which logs should be filtered out .  Of course in the name of custom use cases needed, some logs can be important to customer while other name it as garbage but there must be some kind of best practice on filteringout. Any suggestion about that would be great.

Regards.

Labels (3)
0 Likes
4 Replies
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: Best practice for filterout.

While its not necessarily best practices as such, I would look at the following and build out from there:

  1. Any log data that is not categorized tends to be irrelevant - there are no hard and fast rules here and clearly some new log data that isn't being parsed correctly could be missed if you take this approach, but usually uncategorized data isn't useful as its not security relevant or has context to support a use case. So if the categorization fields are blank - filter out.
  2. Check through your Windows rules specifically - this was always the issue with standard content and anything that you might add on top - without the ability to understand what events trigger which rules, it was always going to be a best guess. Activate fixes this by defining what events hit which filters which in turn hit which rules. So with Activate you can know this, but otherwise you are guessing a little. However, again, understanding which rules require which events does help you screen out everything else.
  3. Focus on the use cases / mission - if you are mostly concerned about authentication and users who try to bypass, escalate or incorrect use their credentials, you can screen out everything that is not in reference to this. So you can screen out all of the process logs and so on. In this case, I cannot recommend enough ultimatewindowssecurity.com - go to the quick reference guide section (Free Quick Reference Chart for the Windows Security Log ) and fill in your email. You will then get a simple PDF that categorizes the event log numbers into groups. Either use this to build your filter or use it to focus on specific types. It starts to get easy here. Also, take a look at the other articles that they have published there - all VERY useful.
  4. Broadly speaking you can expand 1 a little further - again, if a log doesnt have relevant information about the sourceUserName or destinationUserName (or attackerUserName or targetUserName) then you can usually broadly ignore these too. You might get IP address information, but without specific data about who did what, then its broadly irrelevant at the initial stage.

My recommendation (and this doesnt go down with the "I need everything" way of thinking), but focus on what you can do with the logs that support it. So filter out as much as you can and lets start building value for what rules you have in place. Screen out with the points above and then build out from there. As you develop and increase capabilities, you can then adjust the filters and allow more logs to enter ESM and be correlated accordingly. Of course, I would absolutely recommend that you send EVERYTHING to Logger, so you have a record, but there is absolutely no reason to send the data to ESM / Express for correlation.

There is a common misconception with some in the industry that you need to send all of the data to ESM for correlation purposes. While there may not be specific rules in place, other unconnected ones will be able to add some 'intelligence' to what is going on. While this might be true for a certain amount of data, its not for everything else. Data lacking IP addresses, username information and even categorization simply won't get correlated correctly (in ESM, Express or anything else for that matter) and its just a cheap and simple ploy by the vendor to get you to license more.... if you dont need it, dont do it.

Think of it this way - would you go to the Oil company (Shell, BP, Mobil, Total or any of the others) when you are asking for an energy saving program? Guess what recommendations you will get?

0 Likes
rhope Acclaimed Contributor.
Acclaimed Contributor.

Re: Best practice for filterout.

Paul Brettle has been very thorough as always in his answer above and I don't want to confuse things but the other thing you may wish to consider is a 'filter in' vs 'filter out' approach. This is one of the architecture discussions we occasionally get into as the Connector -> Logger -> ESM architecture lends itself to filter in, whereas a Connector to ESM and Logger simultaneously requires you to contort the filters to achieve this. However, Connector -> Logger -> ESM comes with other challenges e.g. scalability. I like filter in for situations where you have a tightly defined set of use cases with known inputs. Filter out is a situation you generally get into when you are trying to tune unknown content.

0 Likes
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: Best practice for filterout.

Richard makes a very valid comment here.

From an architecture point of view, I much prefer to use the dual destination approach. The ability to control what gets sent to which system and not have the limitations of sending through Logger does seem to be better overall. Logger is great and logically it makes sense to use forward data into it and then out of it for correlation purposed. But with limitations around the forwarding speeds, the use of particular filters on the forwarding rules AND the potential single point of failure, it makes more sense to just use dual destinations.

When we look to the future too, the use of multiple destinations will most likely be the norm. With the advent of the ADP (ArcSight Data Platform) for 1.x and 2.x as well as more and more customers using things like Hadoop for analytic and long term retention purposes, I see one set of connectors sending to a mixture of systems.

And the principles of filter in and out make more sense in this context too. When sending to a platform for analytics purposes, sending all data makes sense, but you will "pay" for this in performance, storage and processing speeds. But you do want to do this for the rare, unusual patterns and outlier events. But since a real-time correlation engine isn't necessarily best a looking at these, you can safely filter these out and just focus on the specific that you are looking for.

Hope this makes sense.

0 Likes
rhope Acclaimed Contributor.
Acclaimed Contributor.

Re: Best practice for filterout.

I agree, and I generally use dual destinations for all the reasons you mention, the only reason I mentioned the Logger forwarding is that it somewhat enforces filter in whereas the Smart Connector filters appear to be designed with filter out in mind. That's not to say you can't do filter in with a Smart Connector filter, you just filter out everything that's NOT what you want but the logic to achieve this can get a little confusing...

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.