Re: ESM 6.91c automatically disables rules
Unfortunately, we have reworked this rule many times and have made it as specific as possile to reduce the number of partial matches but as there are just so many events that relate to these rules. There ends up being a large number of partial matches no matter what we do.
Thank you anyway for your suggestions 🙂
Re: ESM 6.91c automatically disables rules
This is quite an interesting discussion i must say, and while i do understand the frustration i do think it is a fair reaction at this point from both sides, though there is certain things that should be pointed out.
The automatic disabling of rules is doing what it was designed to do, by disabling rules that are creating much more load than 1 rule should do, and instead of giving it the possibility of just DoS'ing the whole system, it rather disables one rule than melting down.
Seeing that the only possible solution posted was to disable this functionality, i wanted to give new potential readers more options to choose from, though in options it's more of a explanation/description, so maybe other actions can be taken instead.
First of all, Joins in general should always be a a last effort thing, where all other possible implementations has been considered first and ruled out as not working, not feasible to manage, or too complex, even then it should also be considered then if the rule/usercase is worth the heavy load it will generate.
If you do utilize joined/matched events there is certain technical things to keep in mind:
Each event you create in this rule creates both "partial match" and "full match" in waiting to see if it will also match the other events in that rule. For every single partial and full match it will be stored in memory and not removed until the time you have set for this match to happen has gone, this waiting time should never go above let's say 1 minute, especially on high traffic environments. When utilizing these join rules, the conditions specified in your event should follow a specific order, depending on what you want to match, all superindexed fields should be applied at the top of the conditions, starting with the condition that will give you the least amount of hits, though this is best practice also outside of joins.
When more than 1-2 conditions are met in an event, it creates a partial match, even in environments seeing 10-15k EPS it should never have more than 100 000 partial matches per rule, or the rule would be considered "wrong" or "broken" by the system.
Them same principle also applies to aggregation, even on single event rules, though it is at least a bit nicer, as the partial match count is lower.
Here is a old example i like to user:
A rule with 2 events are created, the first event looks for logins happening on physical consoles in your datacenter, while the second event (marked as negated to look if something hasn't fired) looks if someone has swiped their badge before entering, if this does not exist an event is triggered.
The issue here is the waiting time for these events, an engineer might take a long time before logging in, maybe he makes some coffee first.
This means you will have to leave that window open for a long time, to ensure that the rule correlation happens, while going against the practice of leaving this time window very small.
So to implement this in a more efficient solution, you can do something like this:
Create 2 lightweight rules (if you are only going to do something with an activelist or sessionlist, then lightweight rules should always be used), one for when the badge is swiped to enter the datacenter, and one for when he swipes out.
On swiping to enter, you add the name or username of this user to an activelist for tracking, using the username as a key field.
On swiping to exit, you create an action to remove the existing entry of that user in the same activelist. In certain cases to prevent it from increasing indefinitely we also put a time to live of let's say 1 day, if 2 people enters with one card etc.
Now that we have this activelist, we just need to create another rule, triggering on logon failed+successful on this datacenter console, and check if the user is in that list, if he is not he must be a thief 🙂
This principle can be applied on almost any type of join cases, and activelists are much more efficient ways to handle longer term correlation. If it is for short term then having aggregation on one rule can solve this instead, still utilizing the tracking list only after aggregation has been filled.
In certain implementations removing the usage of "for every event" action, or "for every aggregation" can be replaced with some of the less impactful options.
This was not saying that other implementations are WRONG, though it's important to clarify that just disabling features that are quite useful and important is not always the best approach for fixing the real issue in the background.
I have yet to see many usercases that can only be implemented with joins.
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.