This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Collector development - Performance issues

Hello, everyone,

i have performance problems with a self developed collector.

The logs are separated with spaces and sometimes contain quotation marks. Below is an example for the log format.

valueA valueB valueC "valueD" ... valueZ

Currently I am separating the log using the safesplit() function. Then I assign the values directly to the variables.

Below is an example from the code

var input = this.s_raw_message2
myArray= input.safesplit(" ");

if (myArray[8] != zero) {
if (myArray[8].length >= 2) {
var vendorOutcomeCode = myArray[8];
e.VendorOutcomeCode = vendorOutcomeCode[8];
}
}

if (myArray[9] != zero) {
if (myArray[9].length >= 2) {
e.SourceIP = myArray[9];
}
}

The log contains about 30 own values, which I assign to the variables via the above way.
At the moment this collector only manages about 70 EPS. But it is necessary that this collector creates about 500 EPS.

My performance troubleshooting has shown that the performance issue is not directly caused by the safesplit function. It seems that assigning the array to individual fields drastically reduces the EPS.

Am I using the safesplit function or assigning the individual variables incorrectly?
Does anyone have any tips for tuning?

Parents
  • Can you paste your actual code? This does not look live valid Javascript:

    if (myArray[8] != zero) { }

     

    Norbert

  • Hi,

    you're right - I made a copy&paste mistake there:

    var input = this.s_raw_message2
    myArray= input.safesplit(" ");

    if (myArray[8] !=null) {
    if (myArray[8].length >= 2) {
    var vendorOutcomeCode = myArray[8];
    e.VendorOutcomeCode = vendorOutcomeCode[8];
    }
    }

    if (myArray[9] != null) {
    if (myArray[9].length >= 2) {
    e.SourceIP = myArray[9];
    }
    }

    I will remove the "!= null" checks and the length parts from my code. Maybe this will already give you the desired performance.
    Are there any other problems you might see alternatively?

    Many thanks in advance

     

  • Hi all,

    well, I tried to assign all objects in the array directly without checking the objects first.
    Afterwards I get only 53 EPS.

    Any other ideas how to tune the performance?

    Thanks

Reply Children
  • Hi woodsthegoods,



    How powerful is the test environment?



    I have servers with the minimum recommendations and I receive around 1500 eps without problems.

    And I use collectors out of the box and custom.



    Have you possibility to test collector in an environment reaching the minimal production hardware required from NetIQ?



    Regards,


  • For comparision, how fast is your environment, if you comment out the myArray stuff?

    In the SDK there is a _testPerformance Ant build target. This will give you an average over a longer period of time. See /cyberres/sentinel/w/sentinel_tips/13561/sentinel-collector-sdk-2014-updates-debugger

    Norbert

  • Thanks for your feedback.

    I don't really get it.
    I have now changed the collector - no more safesplit() but a regex directly to the raw messages.
    By assigning the same number of variables I get 500 EPS without problems.

    As soon as I comment out the assignment from the array, I also get 500 EPS.

    In my eyes the problems seem to come from assigning the array to single variables.

    Are there any general problems that could explain slow read access and subsequent write access to arrays?