Please excuse my ignorance if any below points are invalid or, badly worded, I am a *nix newb, just trying to understand something that has been bothering me.
Initially what looked like a memory leak in ESM 6 (is anyone seeing this in 5.x or prior as well?), it seems that this is unintended consequences of an intended feature of RHEL/Linux than a specific problem with ArcSight.
The question: Why does the system continually cache (what I understand being files read by the OS) to the point where there is zero 'free' memory (but plenty of cache), and then start swapping? From what I understand the OS is supposed to (under normal circumstances) release the cached memory to applications. Specific scenarios (and I guess there is an algorithm) decides when this does not occur.
|Link: Gentoo Forums :: View topic - Linux Memory Management or 'Why is there no free RAM?'|
|Link: Experiments and fun with the Linux disk cache|
|Link: Linux cached memory: Over 85% of cached memory and using swap - Server Fault|
As for why a server might swap data instead of releasing cache, it may be the case that your cached data was being read much more than your data stored in memory. Programs sometimes have pages that they rarely, if ever, visit. That space is better utilized by caching.
This seems to ruin performance for ESM, once the cycle starts, and it never seems to clear up until you drop_cache on the box and restart services...
Here are some references I've found helpful regarding caches and swapping configurations:
It seems the key flag to relieve this behavior may be the swappiness setting in /etc/sysctl.conf. We have tested with various numbers (60, 10, 1, and 0) and the only which keeps this behavior from recurring is 0.
I was hoping that maybe someone can help me understand why the Linux filesystem is deeming that cached disk-based files are more important than memory pulled out by what I assume are ESM/MySQL and decides to swap instead of providing some of the cached memory?