Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
matslofva
New Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

1. Yes, seems like it.

2. Logs are rolled over. In normal operation we have a Full GC about every 2 minutes.. I seem to believe it was about the same.

3. Really inconsistent but I would say around 40-50% (this is unacceptable for what we're doing.. Seems like only persisting events is generating way to much cpu utilization).

4. No significant changes during that time, no.

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Every two minutes is pretty fast for Full GCs, I've heard that hourly Full GC is what youre shooting for most optimally, alongside GC times being quick (I don't know the number on that might be 200mb/sec)...  Do you have RAM which you can allocate to your ESM JVM Heap?  (server.wrapper.conf file - config parameters wrapper.java.initmemory and wrapper.java.maxmemory)

Thanks for sharing!

0 Likes
superman Respected Contributor.
Respected Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

This may be an moot and an addressed by support point,    has anyone been able to run esm in debug mode ?

0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

superman wrote:

This may be an moot and an addressed by support point,    has anyone been able to run esm in debug mode ?

I haven't tried debug mode in 6.x yet, but running debug mode in 5.x slows the system down to under 1000 EPS.  (Which I imagine would be too slow to recreate your issue)

-Joe

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Just had the caching problem this morning.  I did take some time to check on the Full GCs and here's the result:

INFO   | jvm 1    | 2013/12/12 04:26:03 | [Full GC

INFO   | jvm 1    | 2013/12/12 04:26:07 |  5027791K->1940325K(16700032K), 3.4398970 secs]

--

INFO   | jvm 1    | 2013/12/12 05:26:07 | [Full GC

INFO   | jvm 1    | 2013/12/12 05:26:10 |  5101306K->1980106K(16696000K), 3.4956920 secs]

--

INFO   | jvm 1    | 2013/12/12 06:26:10 | [Full GC

INFO   | jvm 1    | 2013/12/12 06:26:14 |  5970906K->1982132K(16688896K), 3.4655490 secs]

--

INFO   | jvm 1    | 2013/12/12 07:26:14 | [Full GC

INFO   | jvm 1    | 2013/12/12 07:26:18 |  6086238K->1987097K(16682816K), 3.7416110 secs]

--

INFO   | jvm 1    | 2013/12/12 08:26:18 | [Full GC

INFO   | jvm 1    | 2013/12/12 08:26:22 |  7181117K->2086567K(16661056K), 4.3971330 secs]

--

INFO   | jvm 1    | 2013/12/12 09:26:22 | [Full GC

INFO   | jvm 1    | 2013/12/12 09:26:26 |  8715083K->2064446K(16682048K), 4.4700140 secs]

--

Caching started at about 09:40

--

INFO   | jvm 1    | 2013/12/12 09:52:20 | [Full GC

INFO   | jvm 1    | 2013/12/12 09:52:22 |  11269172K->2242501K(16532096K), 2.6180600 secs]

--

INFO   | jvm 1    | 2013/12/12 10:10:49 | [Full GC

INFO   | jvm 1    | 2013/12/12 10:10:51 |  11163293K->2158772K(16332608K), 2.5266630 secs]

--

Manager restarted here

--

INFO   | jvm 1    | 2013/12/12 10:18:43 | [Full GC

INFO   | jvm 1    | 2013/12/12 10:18:43 |  5744K->5238K(16078208K), 0.1411190 secs]

--

INFO   | jvm 1    | 2013/12/12 10:19:55 | [Full GC

INFO   | jvm 1    | 2013/12/12 10:19:56 |  159839K->141899K(16078208K), 0.7561660 secs]

--

INFO   | jvm 1    | 2013/12/12 10:20:26 | [Full GC

INFO   | jvm 1    | 2013/12/12 10:20:26 |  152324K->137378K(16078208K), 0.6157070 secs]

--

INFO   | jvm 1    | 2013/12/12 10:20:26 | [Full GC

INFO   | jvm 1    | 2013/12/12 10:20:27 |  137506K->136679K(16078208K), 0.5874090 secs]

So it does seem to do full GCs more often when the problem is there.

0 Likes
superman Respected Contributor.
Respected Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Would the increased frequency of GC mean there is more G to collect ?

Does memory free up after the GC ?

0 Likes
heiko.hansen@hp Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

As we are facing the same symptoms with ESM 5.2 Patch 2, I wouldn't bet it's a CORR-E issue. Anybody else seeing that on 5.x?

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Now that you're talking about it, we might have been facing the same issue with 5.2.... We always blamed Oracle for the problems we had, but it might not be the actual case...  We had way lower EPS that we have right now (2k vs 10k)...

0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I think it's a valid point given the amount of code shared between 5.x and 6.x on the manager side.  Do you think the issue first appeared in 5.2 patch 2?

-Joe

0 Likes
heiko.hansen@hp Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

We started with a fresh 5.2 Patch 2 install, so there is no experience with previous 5.x versions.

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

It is interesting you mention this.  We didn't have similar experience in 5.2, we had other issues, but not this specific one...  Our investigation does seem to be leaning towards manager code and memory management, but were still troubleshooting...

0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I'd like to try to replicate the issue in a lab with ESM 6.5.  Are these the correct test parameters to trigger the problem?

  • Multiple connectors feeding a single ESM instance (connector version >= 6.x)
  • Total event throughput > 10,000 EPS
  • Minimum test duration = 2 weeks

Are any of those affected using peer search or archives?

Thank you,

Joe

0 Likes
deathbywedgie1 Frequent Contributor.
Frequent Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

We average 4-5k EPS during the day on the manager that we encountered this on recently, and it only took about 10 days of uptime before we encountered it. I had not yet enabled peer search, but we do use archives. I look forward to your lab test results. There's no telling what other user or content behaviors contribute to the issue, but if you're able to reproduce it with basic (but high volume) forwarding in a lab without a lot of user activity, that will be very meaningful.

--dbw

0 Likes
heiko.hansen@hp Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I wish I also had a chance to do labs like that.

Over here, it's also between 5 and 7 kEPS, multiple connectors (20-ish to 40-ish) 6.x and also some 5.x. Test duration could be longer, I would reckon 2 to 4 weeks.

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I've been bleeping my lab at 15k eps for about a week and a half now. No problems to this day. I do have far less content in the lab than in production, so my guess is that content does play a role. The problem arises in production about every 10ish days.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.