josh_tonak Super Contributor.
Super Contributor.
457 views

Upgraded Connectors and Data Ingestion Reduced by 98%

The strangest thing happend after upgrading most of our 150 connectors to version 7.12. Both the ArcMC and Logger show data ingestion dropped from around 90GB/day to 2GB/day. There is no change in EPS rate so we aren't losing any data. Maybe there was a significant change in data compression? Most of the connectors were on version 7.8 before the upgrade. We have another Logger that receives about half of the data so it was usually at 45GB/day. Now it's at 11GB/day.

This definitely saves a lot of money for license costs but is it accurate? I'd like to be able to justify why our numbers dropped so drastically. Right now it's just a guess that the connector upgrade is the cause since the data drop and upgrade happened on the same day.

0 Likes
9 Replies
Frequent Contributor.. stelinson Frequent Contributor..
Frequent Contributor..

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

Hi Josh,

I'm upgrading a number of connectors to 7.12 this week. Will let you know if I see anything similar

 

0 Likes
josh_tonak Super Contributor.
Super Contributor.

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

Thank you
Micro Focus Expert
Micro Focus Expert

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

Did you happen to be upgrading from 7.8? I would like to replicate this to find the root cause of this if you certain that no data has been lost.

Is it possible to disclose any information about what type of logs?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
josh_tonak Super Contributor.
Super Contributor.

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

It was a mix of 7.7 and 7.8 that were all upgraded straight to 7.12. 

Mostly Unix syslog. Around 20 web application/flex connectors and about a dozen WUCs. The syslog connectors handle Unix servers, firewalls, NIDS, HIPS. A few syslog-ng for ESXi logs. 

Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

That is indeed interesting. The 7.8 had issues in certain environments and situations in which persistent connections between the Logger and the connectors never expired, causing a high increase in load and other weird issues.

I think you might have seen a result of that, since this only happened on the 7.8 version of the connector it is almost too big of a coincidence that this was exactly what you was running while seeing wrong numbers and was instantly resolved right after the patching.

Ref this announcement after the release of 7.8: https://community.microfocus.com/t5/ArcSight-User-Discussions/Reduced-EPS-performance-of-ArcSight-Smart-Connector-7-8-0-8070-0/m-p/1646238

This was also added to the release notes of 7.8 located here: https://community.microfocus.com/t5/ArcSight-Connectors/SmartConnector-Release-Notes-7-8-0-8070-0/ta-p/1648613?attachment-id=67443

Ref: "Reduced EPS to Logger Destination"

 

Another thing that comes into mind is sudden increased in amount of parsed vs unparsed events.

The logger compression works it's best when there is mostly or only the same fields are used by all events. If you have 100k events, and 99.9k is windows and 0.1k is something else that populates totally different fields then it impacts the compression.

The more difference the less compression you get. Maybe the upgrade added parsing support for earlier events that was unparsed or wrongly parsed.

If you take linux audit events for example, these create many events only for 1 single actual audit event. If this suddenly starts to parse correctly then the event merging functionality that this parser uses will also start working, highly decreases the amount of EPS actually ending up as a result. It might also kick in aggregation that should have been there in the first place 🙂

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
Frequent Contributor.. stelinson Frequent Contributor..
Frequent Contributor..

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

Hi Josh,

I've upgraded 6 syslog connectors, an O365 connector, a WINC and a sylog flex connector.I've not seen anything like what you 've experienced. The upgrade was trouble free and the event processing is as expected.

The upgraded was from 7.11.1.8143.0.  I upgraded to 7.11.1.8143.0 from 7.9.0.8084.0 a month ago.

 

 

 

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

ingestion is normally not counted on the transfered data, rather on the bytes in the events... interesing observation, keen to read what is the "root-cause".

stvhull_forces Trusted Contributor.
Trusted Contributor.

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

info from Release notes SmartConnectorReleaseNotes-7.8.0.8070.0.pdf

==================================================================

Reduced EPS to Logger Destination
Important: If you have not upgraded to 7.8.0, this step is not necessary.
A degradation in performance over time has been observed while using SmartConnector 7.8. with Logger. Please refer to the following Oracle links for more details:
Java Release notes - http://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html
JDK Bug - https://bugs.openjdk.java.net/browse/JDK-8199463

To avoid this issue:

Perform these steps preferably before upgrading the Connector to 7.8.
1. Update the agent.wrapper.conf file.
For ArcMC Managed Connectors Use the Diagnostics Wizard to update the agent.wrapper.conf. See “Running Diagnostics on a Container” (page 118) on ArcSight Management Center Administrator’s Guide).
For Unmanaged Connectors, use the agent.wrapper.conf file located in CONNECTOR_HOME/user/agent.
a. Add -Djdk.tls.useExtendedMasterSecret=false in agent.wrapper.conf
b. Add the following line and specify the correct (incremental) number after the wrapper.java.additional property.
c. Restart the Connector.
2. Restart all Logger Apache servers, including single Logger destinations and Logger pool. This step may be executed once all the connectors pointing to logger/logger pool are updated.
Note: Ensure the parameter is applied to all the 7.8.0 connectors that send events to Logger/Logger Pool.

dkuehner Super Contributor.
Super Contributor.

Re: Upgraded Connectors and Data Ingestion Reduced by 98%

I doubt that there is actually a reduction in log ingestion if the EPS is the same.

- The 7.8 issue would (if anything) cause an increase in EPS/volume after the upgrade to 7.12.

- The data volume is calculated before any compression, aggregation and whatsoever, so the pure logs you are reading with the connector should be counted. There should be no difference between any connector versions unless the connector is collecting more/less logs (e.g. API connectors looking for more events then before an upgrade)

 

My guess is that it´s basically a bug related to the data volume calculation. Just make sure you actually receive all the events you received before the upgrade and you are fine...

 

PS: I did not experience this behavior with any of my customers (some coming from 7.7, some from 7.10/11.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.