Absent Member.
Absent Member.
29 views

Tuning ESM 6

When I first installed ESM 6, I noticed query performance seemed a bit unpredictable.  After some research I noticed the system would run itself out of memory and start using swap, regardless of how much memory the system had available (512GB for example).

After a few weeks of testing, I came up with the following configuration to eliminate swapping and make performance more predictable.  As you read, keep in mind that this is a single socket server with 24GB of physical memory (8GB heap for the manager).

Disable THP

Transparent Huge Pages is a newer feature that came with RHEL 6.  Based on the number of bugs recently discovered on bugzilla.redhat.com I decided to disable it.

/etc/rc.local

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled


Enable huge pages and tune swap parameters

Even after stress testing (5k-9k EPS w/ simultaneous scheduled reports + active channels) the system never swapped or ran out of memory using these parameters.

vi /etc/sysctl.conf

kernel.shmmax = 16642998272

kernel.shmall = 4160749568

vm.swappiness=0

vm.hugetlb_shm_group=500

vm.nr_hugepages=7168

vm.min_free_kbytes=131072

vm.dirty_ratio=8

vm.dirty_background_ratio=4

Use memlock

I always used memlock for huge pages with Oracle, so I kept the same practice here.

vi /etc/security/limits.d/90-nproc.conf

@arcsight soft memlock 16252928

@arcsight hard memlock 16252928

Configure java processes to use huge pages


vi /opt/arcsight/logger/current/arcsight/logger/bin/scripts/servers.sh

ARCSIGHT_JVM_OPTIONS="-verbose:gc -Xms256m -Xmx1536m -XX:+UseLargePages -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=utf-8 "

vi /opt/arcsight/manager/config/server.wrapper.conf

wrapper.java.additional.12=-XX:+UseLargePages

Configure mysql to use huge pages and tweak limits/buffers

Enabled huge pages by adding the "large-pages" parameter.  I've also read that the mysql buffers are more efficient than letting the OS buffer, so I tweaked them slightly.  Lastly I increased the sort_temp_limit to ensure my reports that do sorting won't fail (this actually affects disk space, not memory).

vi /opt/arcsight/logger/data/mysql/my.cnf

# The MySQL server

[mysqld]

large-pages

key_buffer_size=1G

sort_temp_limit = 50G

# You can set .._buffer_pool_size up to 50 - 80 %

# of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 1G


Results


The system has been running for days without swapping or crashing.  Memory utilization never reaches 100% and the system properly releases used memory after queries complete.

mem-vs-swp.png


11 Replies
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Update: Up for 112 hours, still no swapping.  🙂

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Update for ESM 6.5 (& RHEL 6.4): The parameters above still appear to work.  I've had a v6.5 system up for 7 days without swapping.

The only difference I've seen so far is that you'll need to use wrapper.java.additional.13 for server.wrapper.conf.

-Joe

0 Likes
Reply
Highlighted
Respected Contributor.. Respected Contributor..
Respected Contributor..

Re: Tuning ESM 6

Hi Joe,

You did really great job in this article!

Would you mind provide exact configuration for 6.5 on 24GB RAM?

Thank you,

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Just to state it explicitly for v6.5:

vi /opt/arcsight/manager/config/server.wrapper.conf

wrapper.java.additional.13=-XX:+UseLargePages

I picked this up after I figured out the adjustment on my own. I tend to be a little slow to the party.

Thanks Joe for all your hard work. You continue to impress me.

0 Likes
Reply
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Re: Tuning ESM 6

FYI - Confirmed that these performance tuning parameters still work with AS ESM version 6.5.1

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Thanks for verifying!

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Hey Guys!
Would you please confirm that these options are still working in ESM 6.8?

0 Likes
Reply
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Re: Tuning ESM 6

I am no longer utilizing Huge Pages for 6.8c Patch 1. Though system specs are 2TB memory, 60 CPU cores, FusionIO PCIe cards for CORR. Moving to Patch 2 soon, and upgrade to ESM 6.9c on the horizon for testing.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Just to clarify guys,

In ESM 6.8 P2 in order to use large pages you have to use additional option number 16:

wrapper.java.additional.16=-XX:+UseLargePages

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Tuning ESM 6

Thanks Nikolay!

Ali3n0ne, please let us know if you're able to measure any performance benefit to using over 256GB of memory.

0 Likes
Reply
Highlighted
Senior Member..
Senior Member..

Re: Tuning ESM 6

Which additional option is 16 pointing to in the default wrapper file. I am on 6.8 Patch 1 and want to make sure the parameter is set correctly.

Thanks.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.