Raw Syslog TCP Connector

I followed this useful thread and set the agents[0].tcppeerclosedchecktimeout=30000

/cyberres/arcsight/f/arcsight-discussions/208678/too-many-open-files-issue-causing-tcp-syslog-connector-to-stop-working

but I still have the connector failing with the Too Many Open files error from time to time. It doesnt fail immediately. It would run for 8 to 10 hours before it fails. It actually brings the connector down and fills the logs with the 

FATAL EXCEPTION:
INFO
jvm 1
2019/05/06 21:28:16
java.io.IOException: Too many open files
INFO
jvm 1
2019/05/06 21:28:16
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
INFO
jvm 1

Here is the link to the solution I came across

https://easyengine.io/tutorials/linux/increase-open-files-limit

I have not had any good suggestions yet from ArcSight Support.

Should I said open file limit at the OS Level?

Here is the link to the solution I came across

https://easyengine.io/tutorials/linux/increase-open-files-limit

Not sure what to increase it to.

Any thoughts or suggestions would help.

Regards

Vignesh

 

  • This can also be caused by parsing issues if its causing too much unexpected activity in log files etc, but let's check the OS first.

    Could you go into the arcsight user in CLI and give the output of "ulimit -a".

    In the list it should say "open files", and it should be at least commonly 65535. 

  • Marius

    Thanks for your immediate response. I think I see 1024. Is that a problem?

    -bash-4.2$ ulimit -a
    core file size (blocks, -c) 0
    data seg size (kbytes, -d) unlimited
    scheduling priority (-e) 0
    file size (blocks, -f) unlimited
    pending signals (-i) 31157
    max locked memory (kbytes, -l) 64
    max memory size (kbytes, -m) unlimited
    open files (-n) 1024
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) 819200
    real-time priority (-r) 0
    stack size (kbytes, -s) 8192
    cpu time (seconds, -t) unlimited
    max user processes (-u) 4096
    virtual memory (kbytes, -v) unlimited
    file locks (-x) unlimited

  • That is indeed your issue.

    Please change the value to something higher as mentioned in the earlier post and restart your server for the changes to take effect.

    https://www.tecmint.com/increase-set-open-file-limits-in-linux/

  • This is part of the install process - check that install guide - search for "ulimit"

    Cheers

     

  • Hey , 

    Thanks for you hint, however i was unable to find lulimits in the 

    ArcSight-SmartConnector-User-Guide.

    From the appliances I copied:  

    • open files (-n) 65535

    The   ESM-Installation-Guide says

    • open files (-n)  65536
    • max user processes (-u) 10240

    For "some security reasons" RedHat below into a default install.

    cat /etc/security/limits.d/20-nproc.conf 
    # Default limit for number of user's processes to prevent
    # accidental fork bombs.
    # See rhbz #432903 for reasoning.
    
    *          soft    nproc     4096
    root       soft    nproc     unlimited

    so I usually create /etc/security/limits.d/90-nproc.conf and put

    cat /etc/security/limits.d/90-nproc.conf
    
    *       soft    nproc     10240
    *       soft    nofile    65536
    root soft nproc unlimited

    Cheers

    A