smartconnector syslog performance tuning

I have multiple software syslog smart connectors using syslog-ng daemon running on redhat linux, often there is queuing on the sending hosts. When I poke around to troubleshoot I see the receive queue in netstat on the smart connector box is filling up. From what I can tell the smart connector is slow to process incoming messages. Running a top doesn't reflect that the machine is overwhelmed and the smart connector java process is running between 25-40% CPU 7-0% memory so there is some headroom on the server. What are my options in terms of performance tuning the smart connector?

  • I think you're in luck!  There are a number of perf tuning steps that can be taken specifically on the syslog connectors that can help it get better performance.

    Below are the settings we use for our syslog connectors.  We have been able to realize about 2-3K eps in/out (outbound to 3 destinations) with these settings.

    a) The memory settings are simply that, the amount of memory allocated to the java process.  From what I have been told you essentially want the connector to be garbage collecting at regular intervals (not too long, not too short - I don't know what an exact timeframe is, but I believe 30min - 1hour is healthy - someone can help on that - 1-2 minutes or seconds is too short).  The settings are for init=starting memory and max=maximum memory to allocate.  I believe you can go up to 1536 max as it is a 32bit java process, but its not necessarily recommended as java garbage collection takes longer each time it is run based on the size of memory used.

    b) agent component thread: I believe this is the maximum amount of threads for the agent.  See agent.defaults.properties - it should have a description

    c) syslog parser multithreading and threadsperprocessor: These two go hand in hand.  This allows the syslog parser more threads to work with (primarily helps with inbound EPS bottlenecks), more threads loosely translates to higher inbound EPS, but this also has a impact on memory/cpu.

    d) Transport logger threading: These two settings enable threading for logger destinations.  I would suggest enabling these only if you know that you are using logger and you are caching to the logger destinations (and its not the logger which is causing the clog).  This is an outbound EPS tweak.

    e) Transport http threading: As with the logger threading, this helps with outbound EPS to https destinations (ESM is an http destination I believe).

    The settings below are what we use for our 1-2k EPS syslog connectors (we've actually tweaked up to about 5k EPS, but found it to be extremely unreliable, dropping data and not notifying at all, above 3000 EPS).  The threads and memory settings should all be tested and tweaked to see how they impact your environment.  I believe there are best practices around them but I don't have that information unfortunately (if someone else does it would be greatly appreciated).  This has been all gained/learned by our interactions with other users and A/S PS, so hopefully its all correct.

    You will also want to run the agent logfu and manually look through your agent logs for signs of issues (enableDropMode, YELLOW/REDZONE memory issues, etc) while you're tweaking so you can understand how things are going.  Check out the memory allocated vs memory used graphs in logfu, very useful...

    Again false legal mumbo jumbo: Test these and tweak them to your environment, something about each system and event feed being different... blah blah...

    1. Memory: agent.wrapper.conf
      • wrapper.java.initmemory=512
      • wrapper.java.maxmemory=1024
    2. Threading:
      agent.properties
      • agent.component.thread.enable.threshold=8
      • syslog.parser.multithreading.enabled=true
      • syslog.parser.threadsperprocessor=2
      • transport.loggersecure.multithreaded=true
      • transport.loggersecure.thread=4
      • http.transport.threadcount=4
  • I'm using the linux connector version 6.0.1.6574.0.

    I dont have an agent.wrapper.properties, I have an agent.wrapper.conf, this has the directives you listed but i want to make sure agent.wrapper.conf is the right file.

    Also another difference is that my agent.properties files has all the agent. settings as agent[0]. so I entered agent.component.thread.enable.threshold as agents[0].component.thread.enable.threshold=8

    Not sure if this right but no complaints in the logs.

  • About the agent.wrapper.properties, sorry.  That was my fault, you are correct on agent.wrapper.conf.

    Regarding the agent threading, no, for the ones that do not have a agents[0] in front of them, don't add it.  You can check the agent.defaults.properties to see the exact verbiage, but I believe its correct as written.