Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Highlighted
jjavier Absent Member.
Absent Member.
517 views

AlleviatingTrend Performance in 6.0c with Rules & Active Lists

We had hoped our upgrade to 6.0 CORR we would alleviate our Trend performance problems but apparently we missed Ray Doty’s post on things to know before going to ESM 6. We’re now looking for alternative design methods to improve performance.

Background:

We need to run ad-hoc reports against IIS events for one of our websites. The dataset is too large to query the DB directly; when we do the reports fail. We therefore created a Trend to report against. After the upgrade the Trend started failing and causing system problems. From what I’m told Reports and Trends still use SQL syntax which is then inefficiently translated into queries to CORR.

https://protect724.arcsight.com/message/11865#11865

https://protect724.arcsight.com/message/12148#12148

Possible Solutions

Replace with Active List

Brook suggested using a Rule to populate an Active List then trending against the Active List instead of the DB. I setup a simple rule and the Trend is working well except for one problem. Active Lists are limited to 500,000 entries, and depending on the hour we can pull in upwards of 1m entries in an hour.

Question 1: How do we work around the 500k AL limit?

Question 2: Are there any performance concerns or tuning suggestions using an AL is this way? I’m guessing Partially cached is an option I’ll want to use as well as Optimize Data.

Split the Trend

200k of the entries are the same short stem /b, that we need to track. It won’t completely alleviate the 500k AL limit but I was looking at moving these stems into a 2nd Trend. Since these stems are all the same I’d like to further roll-up the aggregation from the current 1 minute timeframe on the Connector to a full hour by IP in the Trend. I was running into trouble aggregating in the AL because the entries persist after going into the Trend. As I type this I realize I should be aggregating in the Trend and not the AL. I’ll give that a shot but I still need to get around the 500k limit for the other stems. For reporting reasons it would be difficult to split up the trend further.

Basic Numbers from a 1 Hour Period:

Unique Stems Total: 303,659

Total Non Aggregated requests (i.e. individual lines in the trend): 770,228

Total Aggregated Requests: 8,593,527

Non Aggregated Requests for /b: 198,742

Total Aggregated requests to /b: 7,427,918

Unique IPs for /b: 101,015

Labels (2)
0 Likes
Reply
6 Replies
bondarets Absent Member.
Absent Member.

Re: AlleviatingTrend Performance in 6.0c with Rules & Active Lists

AL limit could be tuned via manager properties.

activelist.max_capacity=500000

in server.defaults.properties

0 Likes
Reply
brian.freedman@1 Absent Member.
Absent Member.

Re: AlleviatingTrend Performance in 6.0c with Rules & Active Lists

Have you thought about maybe tackling this problem from the bottom up?

Do you have Field Based Aggregation turned on? Since you are only doing reporting off your data, just identify only the valuable fields for your report and aggregate on those. I have seen some great improvements in data flow by using aggregation.

0 Likes
Reply
jjavier Absent Member.
Absent Member.

Re: AlleviatingTrend Performance in 6.0c with Rules & Active Lists

We expanded it to 1.5m today and are testing. Thank you.

0 Likes
Reply
jjavier Absent Member.
Absent Member.

Re: AlleviatingTrend Performance in 6.0c with Rules & Active Lists

For the /b stems we do use field based aggregation. The rest of the events are highly unique and offer little aggregation gains.

0 Likes
Reply
brian.freedman@1 Absent Member.
Absent Member.

Re: AlleviatingTrend Performance in 6.0c with Rules & Active Lists

I think this might be a case for FILTER based aggregation. I don't know much about Filter Based Aggregation, but it might be something worth looking at.

just my 2c

0 Likes
Reply
jjavier Absent Member.
Absent Member.

Re: AlleviatingTrend Performance in 6.0c with Rules & Active Lists

A problem I'm having now is I need to check if an IP is in an Active List. In the old Trend I used a Filter for an InActiveList check. The Trend would then store a 1 or 0 based on if the IP was in the list or not. For most of my Binary checks I've been able to use the new CustomConditionalEvaluation Function in the Local Variable but it doesn't allow for an InActiveList check. The old (broken) Trend shows the "ConditionalEvaluation" function but only the "CustomConditionalEvaluation" function is available in the new Trend which doesn't allow the use of filters.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.