ESM6 ingest first indication of something else bad..?

[jbur]

There's always time for labs if you don't sleep Heiko.  😉

Questions for users that see the problem in production:

1) Are you running any add-on modules like identity view?

2) Do events expire by retention or are you letting them overwrite themselves due to 0% free space?

3) Are you using trends?

4) Are you using reactivated events from offline archives?

-J

[Answer]

1) We have IT Gov, PCI and Cisco packages. But most of the rules are disabled / not yet implemented.

2) They expire by retention (33 days), we have 90% free space.

3) Yes, quite a lot of trends (more than I'd like to have). Also quite a lot of Data Monitors / scheduled reports.

4) We are not using archives.

[jbur]

Merci!

My ESM 6.5 instance is now running at 6500 EPS (multiple connectors and products, mostly TestAlert replays with some production logs coming off my firewall).

While that's cooking for a few weeks, I recommend we compare the configuration of our systems.

1. Anyone having the issue that is NOT using trends?
2. Anyone else using the IT Gov, PCI or Cisco packages?
3. Anyone using using payload sampling and payload retrieval?
4. Anyone using using cases or annotation/stages?

-Joe

[superman]

ESM 6.0 Patch 1 ( Patch 2 not installed ).

22-30K eps for 30 days+  ;  no reboot or manager restart  ;  no issues

1. Anyone having the issue that is NOT using trends?   -    One Trend size 200'000 entries with 5 fields size 12-14 characters excluding timeStamps
2. Anyone else using the IT Gov, PCI or Cisco packages? - Only One FISMA Package
3. Anyone using using payload sampling and payload retrieval? - Not using yet.
4. Anyone using using cases or annotation/stages? - Yes, Annotations on less than 0.01% of events
   
   Free space 77%

        Disabled heaviest DataMonitors provided with stock content.

[jbur]

More questions to try to narrow this down. (including @ superman since his system is working)

1. How much memory are you assigning to the manager?
2. Which connector versions are you using?
3. Does any of your data contain IPv6 fields?
4. Are you using SSD or HDD for storage?

Thank you,

-J

[matslofva]

Add to that (in case of superman) all customized parameters in my.cnf, postgresql.conf and server.properties would be great.

Thanks,

David

[Answer]

1. 16G

2. 6.0.2 up to 6.0.7, but they all go thru Loggers before getting to the ESM. We have 14 Loggers.

3. I think some WUC events do contain IPv6 fields... At least, I do see some errors on the Loggers about IPv6 fields not being the right format.

4. Hitachi SAN Storage

[superman]

1. How much memory are you assigning to the manager?

16GB

1. Which connector versions are you using?

6.0.7 - ALL

1. Does any of your data contain IPv6 fields?

Nope

1. Are you using SSD or HDD for storage?

Fusion IO.

No custom tuning in the my.cnf or server.properties other than the change allowing for more space for sorting and grouping queries - this was provided by support.

[jbur]

I reviewed the posts again and I've compiled the following list to describe a system that should exhibit the ingest issue.

• ESM 6.0 - 6.5
• Manager heap size = 16384
• SSD or HDD storage
• > 10,000 EPS total via multiple v6.x connectors

Is anyone having the issue that is NOT receiving events from Logger?

Is anyone having the issue that is NOT using trends?

-Joe

[jbur]

I ordered another SSD as I'm running out of space during testing.  The only foundation package I installed was "ArcSight Content Management", so there's not much content running.  However I noticed that arc_system_data (referred to as "system storage" in the HP docs) keeps growing even after arc_event_data levels off.

The ratio I see is that for every 2.5GB in arc_event_data growth, I see 1GB in arc_system_data growth. (take this as an estimate because ESM 6.5 seems to write with a minimum chunk size and I can't see the boundary)

Is anyone else seeing this?  How much space are you assigning to arc_system_data on your production systems? (the install guide lists 500GB max for some reason)

Thanks,

J

[jbur]

I know you guys are busy, but I'm hoping this might be the question that leads to a solution.

Are all the ESM 6.x systems that are having the problem receiving events that are NOT coming from a connector (i.e. Logger forwarder, ESM forwarder, etc)?

Thanks,

-Joe

[Answer]

My ESM is receiving events from 12 loggers plus 2 Smart Connectors (DNS and MVM).

[raymond.doty]

We have our arc_system_data set to 2TB.  Likely system/default trends and/or lists are filling this space up.  Could probably find via a MySQL command to show size on each table.

[raymond.doty]

I am receiving only from connectors, no loggers are in the mix.

[raymond.doty]

jbur wrote:

More questions to try to narrow this down. (including @ superman since his system is working)


How much memory are you assigning to the manager?
Which connector versions are you using?
Does any of your data contain IPv6 fields?
Are you using SSD or HDD for storage?


Thank you,
-J

1) 32GB

2) Almost all 5.2.7, maybe 10-20% 6.0+

3) Extremely heavy IPv6 in all of our event feeds

4) SSD