Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Highlighted
gvaltas Respected Contributor.
Respected Contributor.
372 views

ArcSight Load Balancer - UDP Multithreading

Hello,

Has anyone ever used multi threading on ArcSight Load Balancer specially for UPD sources/destinations?

I am running an LB in HA mode which has 4 sources (3 of them are TCP and 1 of them is UDP). The UDP source receiving logs from High Volume devices (e.g Fortigate).

Based on the loadbalancer.log file the following log have been observed:

2019-10-24 14:49:23,382 [ERROR][UDPPacketConsumer-<source_name>:1][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5002] events from the queue.
2019-10-24 14:49:23,417 [ERROR][UDPPacketConsumer-<source_name>:0][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5003] events from the queue.
2019-10-24 14:49:23,458 [ERROR][UDPPacketConsumer-<source_name>:0][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5004] events from the queue.
2019-10-24 14:49:23,488 [ERROR][UDPPacketConsumer-<source_name>:0][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5011] events from the queue.

After searching on the LB Installation/Configuration guide, there is an option called: udp.consumer.threads and is a Global Parameter.

Has anyone ever used this option? Any suggested value of threads?

 

Regards,

Grigoris

0 Likes
7 Replies
gvaltas Respected Contributor.
Respected Contributor.

Re: ArcSight Load Balancer - UDP Multithreading

Anyone who have observed ever the following errors:

2019-10-24 14:49:23,458 [ERROR][UDPPacketConsumer-<source_name>:0][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5004] events from the queue.
2019-10-24 14:49:23,488 [ERROR][UDPPacketConsumer-<source_name>:0][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5011] events from the queue.

------

Regards,

Grigoris

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: ArcSight Load Balancer - UDP Multithreading

Hi Grigoris,

The error message you are seeing it's by design.
For UDP Syslog, each UDP routing rule has a single thread only which is directly receiving packets of the network and dumping them into the processing system.
The Global parameter which you have mentioned is controlling how many threads process incoming datagrams and send them to the destination.
This could slightly improve performance but it's only required if a WAN message looks something like "Too many unsent events... Draining the events in the queue".

So the best way to increase throughput on UDP is to have multiple routing rules on different ports.

I hope this helps.

Regards,
Kresimir

gvaltas Respected Contributor.
Respected Contributor.

Re: ArcSight Load Balancer - UDP Multithreading

Thank you very much Kresimir,

So, when I observe the following errors:

2019-11-26 14:48:06,782 [ERROR][UDPPacketConsumer-IT_Fortigate:0][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5005] events from the queue.
2019-11-26 14:48:06,813 [ERROR][UDPPacketConsumer-IT_Fortigate:1][com.arcsight.lb.listener.b$a][run] - Too many unsent events. Discarding [5002] events from the queue.

that means that Load Balancer identifies queue to the destination UDP Connectors and discards 5000 events approximately and logs are lost? It is right or not?

Furthermore, due to I am observing "UDPPacketConsumer-IT_Fortigate:0" and "UDPPacketConsumer-IT_Fortigate:1" may the default thread number is 2?

 

Regards,

Grigoris

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: ArcSight Load Balancer - UDP Multithreading

Hi Grigoris,

You are welcome, I am glad I could clarify it.

You said yourself that the event count is quite high: "The UDP source receiving logs from High Volume devices (e.g Fortigate)."
For UDP, it will buffer events in memory depends on what you have configured in the udp.events.queue.capacity setting.
Once it reaches this limit, it will start dropping events. For TCP, the events in the queue are still persisted on disk as they will be stored on a disk.

I believe that by default there are 2 threads and you can change it in "UDP_CONSUMER_THREADS".

Don't forget to accept the solution if applicable as it can help others while facing the same situation you are having.

Regards,
Kresimir

gvaltas Respected Contributor.
Respected Contributor.

Re: ArcSight Load Balancer - UDP Multithreading

Thank you very much Kresimir,

It is quite clear now regarding the limits. However:

1. I can't reach the link you provided, it requires MF Stack A login credentials

2. I can't where DEFAULT_UDP_CONSUMER_THREADS is exactly located

3. I need some recommendations in order to configure properly udp.events.queue.capacity value. I suppose it depends on the received EPS on each source?

Regards,

Grigoris

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: ArcSight Load Balancer - UDP Multithreading

Hi Grigoris,

Sorry, I pasted the wrong link. You can access it here, it's just another community post:
https://community.microfocus.com/t5/ArcSight-User-Discussions/ArcSight-Loadbalancer-limits/td-p/1593896

This is where you will also find information about the limitation and EPS information.
So yes, it depends on how many events are in total sent to this LB by the source device.

It's basically the same setting which you mentioned, which is a global parameter (sorry for the confusion):
https://community.microfocus.com/t5/ArcSight-Connectors/Micro-Focus-Security-ArcSight-SmartConnector-Load-Balancer-1-4-0/ta-p/1642013

udp.consumer.threads: Specifies the number of threads used to read UDP packets from the network.

Other than that, I don't have other recommendations about specific values.
I hope I could help.

Regards,
Kresimir

gvaltas Respected Contributor.
Respected Contributor.

Re: ArcSight Load Balancer - UDP Multithreading

Perfect.

So I have to troubleshoot my environment because with the current setup the flow between ArcSight LB stops towards the ArcSight Connectors 3,4 times per day and I'am afraid that ArcSight LB detects large amount of Queue to destination pools and stops the traffic at random intervals to those destination pools.

Regards,

Grigoris

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.