Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

There's always time for labs if you don't sleep Heiko.  😉

Questions for users that see the problem in production:

1) Are you running any add-on modules like identity view?

2) Do events expire by retention or are you letting them overwrite themselves due to 0% free space?

3) Are you using trends?

4) Are you using reactivated events from offline archives?

-J

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

1) We have IT Gov, PCI and Cisco packages. But most of the rules are disabled / not yet implemented.

2) They expire by retention (33 days), we have 90% free space.

3) Yes, quite a lot of trends (more than I'd like to have). Also quite a lot of Data Monitors / scheduled reports.

4) We are not using archives.

0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Merci!

My ESM 6.5 instance is now running at 6500 EPS (multiple connectors and products, mostly TestAlert replays with some production logs coming off my firewall).

While that's cooking for a few weeks, I recommend we compare the configuration of our systems.

  1. Anyone having the issue that is NOT using trends?
  2. Anyone else using the IT Gov, PCI or Cisco packages?
  3. Anyone using using payload sampling and payload retrieval?
  4. Anyone using using cases or annotation/stages?

-Joe

0 Likes
superman Respected Contributor.
Respected Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

ESM 6.0 Patch 1 ( Patch 2 not installed ).

22-30K eps for 30 days+  ;  no reboot or manager restart  ;  no issues

  1. Anyone having the issue that is NOT using trends?   -    One Trend size 200'000 entries with 5 fields size 12-14 characters excluding timeStamps
  2. Anyone else using the IT Gov, PCI or Cisco packages? - Only One FISMA Package
  3. Anyone using using payload sampling and payload retrieval? - Not using yet.
  4. Anyone using using cases or annotation/stages? - Yes, Annotations on less than 0.01% of events

    Free space 77%

        Disabled heaviest DataMonitors provided with stock content.


0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

More questions to try to narrow this down. (including @ superman since his system is working)

  1. How much memory are you assigning to the manager?
  2. Which connector versions are you using?
  3. Does any of your data contain IPv6 fields?
  4. Are you using SSD or HDD for storage?

Thank you,

-J

0 Likes
matslofva
New Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

Add to that (in case of superman) all customized parameters in my.cnf, postgresql.conf and server.properties would be great.

Thanks,

David

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

1. 16G

2. 6.0.2 up to 6.0.7, but they all go thru Loggers before getting to the ESM. We have 14 Loggers.

3. I think some WUC events do contain IPv6 fields... At least, I do see some errors on the Loggers about IPv6 fields not being the right format.

4. Hitachi SAN Storage

0 Likes
superman Respected Contributor.
Respected Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution
  1. How much memory are you assigning to the manager?

16GB

  1. Which connector versions are you using?

6.0.7 - ALL

  1. Does any of your data contain IPv6 fields?

Nope

  1. Are you using SSD or HDD for storage?

Fusion IO.

No custom tuning in the my.cnf or server.properties other than the change allowing for more space for sorting and grouping queries - this was provided by support.


0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I reviewed the posts again and I've compiled the following list to describe a system that should exhibit the ingest issue.

  • ESM 6.0 - 6.5
  • Manager heap size = 16384
  • SSD or HDD storage
  • > 10,000 EPS total via multiple v6.x connectors

Is anyone having the issue that is NOT receiving events from Logger?

Is anyone having the issue that is NOT using trends?

-Joe

0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I ordered another SSD as I'm running out of space during testing.  The only foundation package I installed was "ArcSight Content Management", so there's not much content running.  However I noticed that arc_system_data (referred to as "system storage" in the HP docs) keeps growing even after arc_event_data levels off.

The ratio I see is that for every 2.5GB in arc_event_data growth, I see 1GB in arc_system_data growth. (take this as an estimate because ESM 6.5 seems to write with a minimum chunk size and I can't see the boundary)

Is anyone else seeing this?  How much space are you assigning to arc_system_data on your production systems? (the install guide lists 500GB max for some reason)

Thanks,

J

0 Likes
jbur Absent Member.
Absent Member.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I know you guys are busy, but I'm hoping this might be the question that leads to a solution.

Are all the ESM 6.x systems that are having the problem receiving events that are NOT coming from a connector (i.e. Logger forwarder, ESM forwarder, etc)?

Thanks,

-Joe

0 Likes
Answer Honored Contributor.
Honored Contributor.

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

My ESM is receiving events from 12 loggers plus 2 Smart Connectors (DNS and MVM).

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

We have our arc_system_data set to 2TB.  Likely system/default trends and/or lists are filling this space up.  Could probably find via a MySQL command to show size on each table.

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

I am receiving only from connectors, no loggers are in the mix.

0 Likes
Established Member.. raymond.doty
Established Member..

Re: ESM6 ingest first indication of something else bad..?

Jump to solution

jbur wrote:

More questions to try to narrow this down. (including @ superman since his system is working)

  1. How much memory are you assigning to the manager?
  2. Which connector versions are you using?
  3. Does any of your data contain IPv6 fields?
  4. Are you using SSD or HDD for storage?

Thank you,

-J

1) 32GB

2) Almost all 5.2.7, maybe 10-20% 6.0+

3) Extremely heavy IPv6 in all of our event feeds

4) SSD

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.