Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Super Contributor.. Carl_E Super Contributor..
Super Contributor..
1294 views

Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Up until recently, my architecture for sending events into Arcsight Logger and ESM consisted of sending all events to Logger and then forwarding events of interest into ESM.  According to HP Professional Services, there are a number of drawbacks to this; one of the largest being that the Logger forwarder can only handle a certain EPS rate for forwarding.

So with this in mind, I started redefining my SmartConnectors to dual feed both Logger and ESM right from the SmartConnector itself.

However, after putting a few of them in place, I started to wonder what the impact will be for my hardest working SmartConnectors.  For example, I have some Syslog SmartConnectors that deal with over 1000 EPS from firewalls and are aggregating and performing DNS lookups for source/destination. 

If I dual feed these SmartConnectors, won’t the SmartConnector be performing DNS lookups and aggregation twice (using requiring two times the Java Heap and CPU cycles) since these setting are defined at the destination level?

Is my thinking on this correct?  If so, how do other customers deal with this?

Thanks,

Carl

Labels (3)
0 Likes
1 Solution

Accepted Solutions
Ruslan Mikhalov Honored Contributor.
Honored Contributor.

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Carl,

Regarding caching - all DNS data connector keeps in memory and on stop/restart all succeeded lookups is written to hosts.txt (located ../connector_home/current/user/agent/hosts.txt). So regardless on destinations number connector uses its internal memory with dns lookups and hosts.txt.

But you should be very carefully with dns errors on connector cause they may impact connector performance greatly.

From the other hand huge amount of failed dns request may influence dns performance cause connector would try to resolve them over and over again (by default, once in a minute, i guess).

Regarding aggregation - you're right as it's configured per destination, so it needs more memory. You should check memory consumption on this connector after memory changes are applied. Check intervals between GCs and GC duration. Problems with memory might be the reason of performance issues on connector as well and may lead to unpredictable behavior.

Feel free to contact me.

Rgrds, Ruslan.

0 Likes
12 Replies
Ruslan Mikhalov Honored Contributor.
Honored Contributor.

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Carl,

Regarding caching - all DNS data connector keeps in memory and on stop/restart all succeeded lookups is written to hosts.txt (located ../connector_home/current/user/agent/hosts.txt). So regardless on destinations number connector uses its internal memory with dns lookups and hosts.txt.

But you should be very carefully with dns errors on connector cause they may impact connector performance greatly.

From the other hand huge amount of failed dns request may influence dns performance cause connector would try to resolve them over and over again (by default, once in a minute, i guess).

Regarding aggregation - you're right as it's configured per destination, so it needs more memory. You should check memory consumption on this connector after memory changes are applied. Check intervals between GCs and GC duration. Problems with memory might be the reason of performance issues on connector as well and may lead to unpredictable behavior.

Feel free to contact me.

Rgrds, Ruslan.

0 Likes
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

From the other hand huge amount of failed dns request may influence dns performance cause connector would try to resolve them over and over again (by default, once in a minute, i guess).




Hi,

I have a question regarding what you wrote here: "From the other hand huge amount of failed dns request may influence dns performance cause connector would try to resolve them over and over again (by default, once in a minute, i guess)."

Is it possible to disable the feature where the connector try to resolve them over and over again? I would like to have 1 try and then just leave it and do not try again.

0 Likes
Respected Contributor.. kencheung1 Respected Contributor..
Respected Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Agreed to Ruslan.

The other thing is to monitor the traffic of DNS lookup if you have limit bandwidth.

0 Likes
Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

For the SmartConnectors that are performing DNS lookups in my environment, I also have them pointing to a replicated copy of my organization's primary DNS to avoid impacting DNS as a whole.

Something to consider if you are able to get a copy of DNS specifically for Arcsight.

Final idea on DNS that I have been kicking around but haven't implemented yet is for Linux servers that are purpose built for running many SmartConnectors..  Once you start distributing the work of a very high EPS source across multiple SmartConnectors, you end up increasing the number of DNS query by close to a factor of the number of SmartConnectors that the parsing the events.  A solution to this would be to simply install a local caching DNS service (BIND/named for example) and then point all the SmartConnectors to the local caching service.  By going this route, I'm thinking that you can even decrease the TTL to limit the size of each SmartConnectors local DNS cache since queries to a localhost caching service should not have a large performance impact.

What I'm describing above is only theoretical at this point unless someone else has already attempted this.

Carl

0 Likes
Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Ruslan,

Thanks for confirming and providing additional details on DNS and aggregation.

I personally find the design decision to only perform DNS and aggregation at the destination level to be a gap in these features.  If HPe recommends that customers dual feed Logger and ESM, then aggregation should also be possible prior to the events hitting the destination part of the internal SmartConnector event flow.

Looks like I’ll be writing up an enhancement for this if there is not one open already.

Cheers

Carl

0 Likes
60015535 Absent Member.
Absent Member.

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Carl,

I see your point but you should as well see from other customer perspective.

If aggregation is done pre-destination flow (lets say in main flow) then all destination will have same aggregation which might not be what customer want.

I have seen many different cases where customer wanted to do not have aggregation for Logger but only in ESM and same apply to filtering.

Saying that it is much better to have flexible destination configuration than having a static configuration for all of the destinations.

I would also suggest you to check this presentation from Protect 2014 about how to tune Syslog SmartConnectors to avoid hitting issues with memory or processing of events:

Best Regards,

Stefano Berian

0 Likes
Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Stefano,

Understanding that flexibility is the name of the game without adding too much complexity, I would argue that because customers have different needs, aggregation should be possible at both pre-destination flow and with the destination flow itself.

Unless I'm mistaken there are other config options that can either be set globally or per destination.

Best,

Carl

0 Likes
Honored Contributor.. DanyK7 Honored Contributor..
Honored Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Carl

Its not theoretical, We've been running like this for about 2 years now.

Since we are running about 15-20 smart per server, once a smart has asked for a resolution, all other smarts on the same host dont have to look much farther.

We went one more level. All our servers that host smartconnectors have a bind caching and forwarding server to a central bind and caching server.

Only the central (and its backup) dns caching is allowed to poke around in the organization's real DNS.

With this setup and the disabling of IPV6 (not here yet internally) we reduced by over 96% our dns queries.

I wouldnt dream of a setup without this anymore.

Dany

0 Likes
Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hey Dany,

I was hoping someone would confirm they were using this setup out in the field...  and if it is coming from you, my idea of implementing this likely came from our discussions of your rsyslog setup.

I have it running now as well and it gives a level of reliability and flexibility that I wouldn't want to do without!

Other than pointing to your caching server, do you also tweak some of the other SmartConnector DNS settings?  TTL for example?

0 Likes
60015535 Absent Member.
Absent Member.

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Dany,

This is also what i am suggesting to many customers that are having latency in response from their DNS servers and because of this impacting SmartConnectors A LOT.

Unfortunately it looks like that installing something to have a DNS cache locally at OS level is not something they would like to do and rely only on DNS cache in SmartConnector (JAVA) which is even putting more load on SmartConnector himself.

Good to hear that there are customers that indeed are using SmartConnectors only for normalizing the events and not for other features that even if implemented should not be delegated to the SmartConnector in toto.

Best Regards,

Stefano Berian

0 Likes
60015535 Absent Member.
Absent Member.

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Hi Carl,

Understand your concern but why then not having a SmartConnector that is cascade to another one?

The first will do normalization and aggregation for all events and send to the second using CEF syslog destination and this will only be responsible for the transport part to both Logger and ESM destinations.

Not much of a hassle if you are having enough resources for 2 SmartConnectors (this will also decrease the resources needed on your first SmartConnector).

Best Regards,

Stefano Berian

0 Likes
Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: Aggregation and DNS lookup on SmartConnectors with multiple destinations

Jump to solution

Stefano,

That is the workaround I'm considering but I was hoping for another answer on this... the more hops the event flow needs to go through, the more points I need to troubleshoot, maintain and design redundancy into.

And at that point, I might as well continue to forward events from Logger to ESM (at least for SCs that are aggregation).  As long as the load isn't too high to max out the Logger forwarding connector, I should be fine.

The SCs that are aggregating are sending to a Logger pool destination so I am already spreading out the Logger forwarding work to a very small "army of loggers".

BR,

Carl

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.