aguida79 Trusted Contributor.
Trusted Contributor.
896 views

ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Hello,

One of our customers is having a problem with some Lightweight rules, that ESM is deactivating automatically because of high CPU usage.

We have a script that generate CEF events containing all the data of our Customer's AD infraestructure (computers, users, members), and other script that generate CEF events containing all the data of our Customer's IDM Solution. At first, we used a Model Import connector type to import that data in Active Lists on ESM. But a few years ago, the Model Import connector became unavailable as one of the connector types offered by HP. At the same time Lighweight rules appeared on ESM, so, we started to use Lightweight rules to capture that CEF events and put that information in Active Lists. Obviously the amount of events is huge, but we never have problems with this before.

After we upgrade ESM from 6.8 version to 6.9.1 version, we started to have the problem mentioned in the title. We already know this behavior of ESM but with Standard rules, not with Lightweight ones.

Does anybody here knows a workaround to modify/disable this new behavior of deactivating Lightweight rules of ESM 6.9.1? It is really a must have for us. We have this solution of generate CEF events and import that events in Active Lists by Lightweight rules in a lot of Customers...

Thanks in advance.

Regards,

Alejandro Guida

Labels (2)
Tags (1)
0 Likes
7 Replies
Acclaimed Contributor.. Shaun Acclaimed Contributor..
Acclaimed Contributor..

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Have you checked MySQL to see whether its tuned optimally for the frequent active-list updates?

0 Likes
Highlighted
aguida79 Trusted Contributor.
Trusted Contributor.

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Hello,

We found a new default parameter in server.defaults.properties file from ESM 6.9.1, that it was not present on ESM 6.8 file.

The parameter:

# Max cpu time a rule is allowed, as percentage of all rules

rules.max.fractional.cpu=50

Maybe it is posible to set up this parameter for example to a value of 95, to avoid the auto disable of the rule. I'm thinking in that idea because of this:

Or maybe there a a way to disable this parameter at all.

Anybody knows something about this new parameter?

Thanks in advance.

Regards,

Alejandro Guida

0 Likes
Knowledge Partner
Knowledge Partner

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Hi,

I'm facing the same issue with ESM 6.11. Rule is lightweight. rule condition: deviceVendor=X AND deviceProduct=Y. there are 60.000 events in 24h matching this condition. So, match count is very low for this rule.
How can a rule impacts such kind of cpu use and cause the manager deactivate the rule?

Thanks.

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Hi,

The behaviour being described here is part of the performance stats introduced to try and protect overall ESM stability from certain rogue rules.

The property 'rules.max.fractional.cpu' states that any rule who's fractional CPU time (compared with that of all rules) is greater than the value stated by the property, will be deactivated.

The default value is 50, meaning that if any rule is seen to be using 50% or more time than all rules put together,  it will be disabled.

The statistics can be found as the the "Sortable Rule Stats" data monitor on the "Rules Status" dashboard.

It's possible that even  though this rule is processing 60,000 events per day. that those events all tend to arrive at the same time.  Some connectors do tend to send their events periodically rather than continuously.  If your 60,000 events all arrive within 2 mins, then it is plausible that any rule processing those events could be subject to this performance counter and disabled.  The rules performance statistics do not currently account for heavily batched events from connectors like this.

I cannot see a way for this property to be disabled altogether.

The effects could be mitigated by increasing the fractional value higher (for example to 80 or 90), though this value will then be attributed to all rules.  There is no way to set a value on a per-rule basis.

Best regards,
Darren

ArcSight Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Does any rule depend on another rule? You might have something creating loops in your environment.

For example, if you are creating rules based on events directly from logsources and not other ArcSight rules, please also define Type in your condition. Best practice would be "Type != Correlated", which means you get both base and aggregated events, but not correlated, hindering ArcSight into triggering by its own events, which is one of the most normal causes for this behaviour.

I also gave quite a extensive answer on a similar post a few days ago, linked here: https://community.softwaregrp.com/t5/ArcSight-User-Discussions/ESM-6-91c-automatically-disables-rules/m-p/1657780/highlight/true#M43572

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Knowledge Partner
Knowledge Partner

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

Thanks for the information. Events aren't received at once, they are syslog events actually. There is no Type condition in the rule. I'll add this and monitor the status.

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: ESM 6.9.1 is deactivating Lightweight rules because of CPU usage

I would also like to mention that it should be narrowed down as much as possible, and have the conditions that cleans most of these events away at the top, except leaving superindexed fields at the very top is always good, so a condition should be narrowed than much more than what you are describing.

Example would be:

Type != Correlated

Device Vendor = VENDOR

Device Product = Product

These should be at the top of all your rules as a start, then you can start with narrowing it down, here is an example:

FILTER = Inbound Traffic only Filter

Device External ID = 1234

And so on, and use this to populate your Activelist, it should also only populate if it doesn't already exist, because if you want counts instead, then query and trends is the correct way to go 🙂

Please let us know how it ends up!

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.