• State Not Answered
  • Replies replies
    7
  • Subscribers subscribers
    0
  • Views views
    1353
  • Users 0 members are here
  • Locked Locked
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Collector development - Performance issues

woodsthegoods

Hello, everyone,

i have performance problems with a self developed collector.

The logs are separated with spaces and sometimes contain quotation marks. Below is an example for the log format.

valueA valueB valueC "valueD" ... valueZ

Currently I am separating the log using the safesplit() function. Then I assign the values directly to the variables.

Below is an example from the code

var input = this.s_raw_message2
myArray= input.safesplit(" ");

if (myArray[8] != zero) {
if (myArray[8].length >= 2) {
var vendorOutcomeCode = myArray[8];
e.VendorOutcomeCode = vendorOutcomeCode[8];
}
}

if (myArray[9] != zero) {
if (myArray[9].length >= 2) {
e.SourceIP = myArray[9];
}
}

The log contains about 30 own values, which I assign to the variables via the above way.
At the moment this collector only manages about 70 EPS. But it is necessary that this collector creates about 500 EPS.

My performance troubleshooting has shown that the performance issue is not directly caused by the safesplit function. It seems that assigning the array to individual fields drastically reduces the EPS.

Am I using the safesplit function or assigning the individual variables incorrectly?
Does anyone have any tips for tuning?

  • 0

    Hi woodsthegoods,

     

    As far as I can figure rom your explanation, you have always the values for each field.

    So, you don't need to see if the length of input field is different of 0, because it always will be.

    If the log have the complete values all time, you can assign the values directly:

    var vendorOutcomeCode = myArray[8]; e.VendorOutcomeCode = vendorOutcomeCode; e.SourceIP = myArray[9];

     

    But if you need to know if the record had no data for any input field, the you can write:

    if (typeof myArray[8] != "undefined") { var vendorOutcomeCode = muArray[8]; e.VendorOutcomeCode = vendorOutcomeCode; }

     

    Perhaps you need to research about the incoming log, to know which are the controls for the input data and avoid the use of repetitive onditions 

  • 0  

    Can you paste your actual code? This does not look live valid Javascript:

    if (myArray[8] != zero) { }

     

  • 0 in reply to   

    Hi,

    you're right - I made a copy&paste mistake there:

    var input = this.s_raw_message2
    myArray= input.safesplit(" ");

    if (myArray[8] !=null) {
    if (myArray[8].length >= 2) {
    var vendorOutcomeCode = myArray[8];
    e.VendorOutcomeCode = vendorOutcomeCode[8];
    }
    }

    if (myArray[9] != null) {
    if (myArray[9].length >= 2) {
    e.SourceIP = myArray[9];
    }
    }

    I will remove the "!= null" checks and the length parts from my code. Maybe this will already give you the desired performance.
    Are there any other problems you might see alternatively?

    Many thanks in advance

     

  • 0 in reply to 

    Hi all,

    well, I tried to assign all objects in the array directly without checking the objects first.
    Afterwards I get only 53 EPS.

    Any other ideas how to tune the performance?

    Thanks

  • 0 in reply to 
    Hi woodsthegoods,



    How powerful is the test environment?



    I have servers with the minimum recommendations and I receive around 1500 eps without problems.

    And I use collectors out of the box and custom.



    Have you possibility to test collector in an environment reaching the minimal production hardware required from NetIQ?



    Regards,


  • 0   in reply to 

    For comparision, how fast is your environment, if you comment out the myArray stuff?

    In the SDK there is a _testPerformance Ant build target. This will give you an average over a longer period of time. See /cyberres/sentinel/w/sentinel_tips/13561/sentinel-collector-sdk-2014-updates-debugger

  • 0 in reply to   

    Thanks for your feedback.

    I don't really get it.
    I have now changed the collector - no more safesplit() but a regex directly to the raw messages.
    By assigning the same number of variables I get 500 EPS without problems.

    As soon as I comment out the assignment from the array, I also get 500 EPS.

    In my eyes the problems seem to come from assigning the array to single variables.

    Are there any general problems that could explain slow read access and subsequent write access to arrays?

Resources

Support
Documentation
Learning Services
Partner Programs
Privacy
Compliance
Help
Company
Privacy Policy
Terms of Use
Cookies Preferences
Accessibility
Anti-Slavery Statement
Support
Contact Us
Careers
Code of Business Conduct and Ethics
The opinions expressed above are the personal opinions of the authors, not of OpenText. By using this site, you accept the Terms of Use. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.), Hewlett Packard Enterprise Company, or Micro Focus. As of January 31, 2023, the Material is now offered by OpenText, a separately owned and operated company. Any reference to the HP, Hewlett Packard Enterprise/HPE, and Micro Focus marks is historical in nature and the HP, Hewlett Packard Enterprise/HPE, and Micro Focus marks are the property of their respective owners.