Raw Syslog TCP Connector

[vigneshwar]

I followed this useful thread and set the agents[0].tcppeerclosedchecktimeout=30000

https://community.microfocus.com/t5/ArcSight-User-Discussions/quot-Too-many-open-files-quot-issue-causing-TCP-Syslog-connector/td-p/1572246

but I still have the connector failing with the Too Many Open files error from time to time. It doesnt fail immediately. It would run for 8 to 10 hours before it fails. It actually brings the connector down and fills the logs with the 

FATAL EXCEPTION:
INFO
jvm 1
2019/05/06 21:28:16
java.io.IOException: Too many open files
INFO
jvm 1
2019/05/06 21:28:16
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
INFO
jvm 1

Here is the link to the solution I came across

https://easyengine.io/tutorials/linux/increase-open-files-limit

I have not had any good suggestions yet from ArcSight Support.

Should I said open file limit at the OS Level?

Here is the link to the solution I came across

https://easyengine.io/tutorials/linux/increase-open-files-limit

Not sure what to increase it to.

Any thoughts or suggestions would help.

Regards

Vignesh

[Marius2]

This can also be caused by parsing issues if its causing too much unexpected activity in log files etc, but let's check the OS first.

Could you go into the arcsight user in CLI and give the output of "ulimit -a".

In the list it should say "open files", and it should be at least commonly 65535. 

[vigneshwar]

Marius

Thanks for your immediate response. I think I see 1024. Is that a problem?

-bash-4.2$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31157
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[Marius2]

That is indeed your issue.

Please change the value to something higher as mentioned in the earlier post and restart your server for the changes to take effect.

https://www.tecmint.com/increase-set-open-file-limits-in-linux/

[Lar]

This is part of the install process - check that install guide - search for "ulimit"

Cheers

[vitz1]

Hey @Lar, 

Thanks for you hint, however i was unable to find lulimits in the 

ArcSight-SmartConnector-User-Guide.

From the appliances I copied:  

• open files (-n) 65535

The   ESM-Installation-Guide says

• open files (-n)  65536
• max user processes (-u) 10240
  
  

For "some security reasons" RedHat below into a default install.

cat /etc/security/limits.d/20-nproc.conf 
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     4096
root       soft    nproc     unlimited

so I usually create /etc/security/limits.d/90-nproc.conf and put

cat /etc/security/limits.d/90-nproc.conf

*       soft    nproc     10240
*       soft    nofile    65536
root    soft    nproc     unlimited

Cheers

A