Of course I read this comment AFTER i clicked the restart rules engine button which of course didnt start the rules engine back up. It would be handy to know how to start the rules engine when the restart doesnt work...if the restart ever works that is ;-D
You need to add a line in the server.defaults.wrapper.conf file, in the additional parameters section.
My line: wrapper.java.additional.12=-XX:ReservedCodeCacheSize=256m
The index (12 in my case) needs to be adjusted depending on your file and changes you may have done to it.
NOTE: Do not modify this parameter without first consulting support. This is a pretty dramatic change and there is definitely a wrong way to do it!
We finally have tested this in both our production and development environments. We haven't had a reoccurance in dev in about 3 months and in prod in about 1 month. This seems like a success. We have had moments where the ESM will start to cache across the connectors, but it seems to recover without a restart. Thanks for helping everyone, I am sure there may be other symptoms that are similar, but this seems to be resolved on our system.
Ray, I'm glad your problem was also resolved with this magical line. Our prod box has been running strong for 3 months now. I did also saw occurences where cache would build up across all the connectors, but it quickly goes back to normal, so it's fine with me. Global cache rarely gets over 1 million (with 12-14k EPS), so no worries there.
I recently had a health check performed and the ESM crashed and burned while he was here. The fix mentioned above was added and I am running at 44K EPS with a load of ~10 for two weeks straight without any issues.
After the EPS load is doubled - from 2000 EPS to 4000 EPS - we have the performance issue too - here are some details:
So far the point is to check in server BIOS if Hyper-threading is enabled or disabled.
Like I said in your other thread, I'm pretty sure HT won't fix anything. The only fix seems to be the additionnal java line.
I agree to a point with Answer. I think the HT change is not a bad idea based on some research we did internally, but most of our issues were software related, not hardware unfortunately. If you're going the route of checking hardware and BIOS, some HP servers have power profile issues as well which can cause slowness.