Highlighted
Trusted Contributor.
Trusted Contributor.
411 views

Forward connector delay in writing events to csv

Hi,

We have adopted the forward connector for writing all the correlated events fired by the rule to the csv stored in the ESM linux server.Currently,we are facing an issue that there is a delay of maximum 2-3 minutes in the action of writing these events which is a critical problem for us since we have further actions which is automatically performed on these events.

Did any one face this issue before?

Can you please suggest any adjustments which i could do in the forward connector configuration to avoid the delay?

Regards,

Hima

Labels (2)
0 Likes
5 Replies
Acclaimed Contributor.
Acclaimed Contributor.

Re: Forward connector delay in writing events to csv

Batching will occur on any connector, such as SmartConnector, FlexConnector or Forwarding connector - we want to bunch up the data so that we optimize the write of the data when ever we can. I would take a look at the agent.properties for the forwarding connector to see what the settings are - not sure which settings are available, but you should see some in the file.

However, is there a reason why you are not using the Execute Command action from the rule? This will be triggered when the rule is triggered and will then run a script on the local ESM Manager server - you can specify the details of this and have it run the relevant action that you want. It will still generate a correlated event, so you have a record of this, but it will trigger the running of the script instantly and you can control things from there.

Be aware that this isnt great if you want to do hundreds of actions per second - but it will certainly cope with tens per second without a problem.

0 Likes
Highlighted
Trusted Contributor.
Trusted Contributor.

Re: Forward connector delay in writing events to csv

Hi,

Initially we were using the execute command but as we increased the number of rules,we figured out that the script was not executed for all of them and the EPS was also too high and we found some missing cases.Hence,we started using the  forward connector.But what surprised me was that the when the EPS is low especially during the night time,we found the maximum delay and in other cases,it was less.

Does this param agents[0].frequency has any effect on the frequency of events written to the csv?

Here i am attaching the agent.properties of our forward connector.

#ArcSight Properties File

#Tue Dec 06 10:36:34 CET 2016

agents.maxAgents=1

agents[0].AgentSequenceNumber=0

agents[0].arcsighthostname=usakspl001.internal.unicreditgroup.eu

agents[0].arcsightpassword=XXXXXXXX

agents[0].arcsightport=8443

agents[0].arcsightprotocol=https

agents[0].arcsightuser=XXXXXXXXX

agents[0].basewaittimeonagenttime=false

agents[0].contcachesize=-1

agents[0].continuous=false

agents[0].destination.count=2

agents[0].destination[0].agentid=3kW0001ABABCAAiCiGfL6GA\=\=

agents[0].destination[0].failover.count=0

agents[0].destination[0].params=<?xml version\="1.0" encoding\="UTF-8"?>\n<ParameterValues>\n    <Parameter Name\="csvfolder" Value\="/opt/arcsight/LogFwdConn"/>\n    <Parameter Name\="csvfields" Value\="event.name,event.startTime,event.agentReceiptTime,event.managerReceiptTime,event.agentHostName,event.agentAddress,event.eventId,event.sourceUserName,event.sourceAddress,event.bytesIn,event.deviceCustomNumber1,event.destinationUserId,event.destinationUserName,event.deviceEventClassId,event.requestCookies,event.deviceCustomNumber1Label,event.sourceUserId,event.requestContext,event.sourceUserPrivileges,event.categoryOutcome,event.categorySignificance,event.oldFileSize,event.categoryObject,event.requestMethod,event.oldFileId,event.rawEvent"/>\n    <Parameter Name\="interval" Value\="90000"/>\n    <Parameter Name\="writeheader" Value\="false"/>\n    <Parameter Name\="commentsprefix" Value\="\#"/>\n    <Parameter Name\="fipsciphers" Value\="fipsDefault"/>\n</ParameterValues>\n

agents[0].destination[0].type=file

agents[0].destination[1].agentid=3eioUOVEBABCH++7d0angDA\=\=

agents[0].destination[1].failover.count=0

agents[0].destination[1].params=<?xml version\="1.0" encoding\="UTF-8"?>\n<ParameterValues>\n    <Parameter Name\="host" Value\="127.0.0.1"/>\n    <Parameter Name\="port" Value\="1338"/>\n    <Parameter Name\="protocol" Value\="UDP"/>\n    <Parameter Name\="fipsciphers" Value\="fipsDefault"/>\n</ParameterValues>\n

agents[0].destination[1].type=cefsyslog

agents[0].deviceconnectionalertinterval=60000

agents[0].enabled=true

agents[0].entityid=enE001ABABCAAyCiGfL6GA\=\=

agents[0].eventrateunit=Second

agents[0].eventstreamtimeout=30000

agents[0].fcp.version=0

agents[0].filename=/

agents[0].fipsciphers=fipsDefault

agents[0].folder=/

agents[0].frequency=6

agents[0].id=3kW0001ABABCAAiCiGfL6GA\=\=

agents[0].markasreplayed=true

agents[0].maxrate=2147483647

agents[0].maxratesupported=5000

agents[0].maxsleeptime=30

agents[0].overrideagentinfo=true

agents[0].overridearcsightcategory=false

agents[0].overridezoneinfo=false

agents[0].persistenceinterval=0

agents[0].preserveagenttime=true

agents[0].preservedetecttime=true

agents[0].randomizeratetime=0

agents[0].setagenttimeasnow=false

agents[0].setdetecttimeasnow=false

agents[0].startpaused=false

agents[0].tempfileext=.replay.tmp

agents[0].timefactor=0

agents[0].type=superagent_ng

agents[0].zonedef=

loggersecure.transport.ssl.certs.checked=Tue Dec 06 10\:35\:46 CET 2016

remote.management.second.listener.port=10050

remote.management.ssl.organizational.unit=qVMZ0lABABCAAdqo5jNboA

0 Likes
Highlighted
Trusted Contributor.
Trusted Contributor.

Re: Forward connector delay in writing events to csv

Hi,

I was searching for alternate solutions  to deal with this delay.And i ended up installing an action connector.I tested it on one of our rule whcih gets fired once in 1 minute.Here it seems to work fine.I continued testing by adding one more condition to the rule in which it fired 3-4 times a second.Here i notied that the action connector is skipping the events(diddnt understand on what basis) and the action is executed only for few of them.It will be great if anyone could share some inputs if you have faced the issue before.I have already went through the action connector pdf 2016 but failed to get any hint.

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Forward connector delay in writing events to csv

For your question, I don't believe that frequency does what you are asking. You can try adjusting it, but at the moment I cannot find any reference to it and what it actually does. Sorry.

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: Forward connector delay in writing events to csv

Try here -

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.