Highlighted
ashis.sahoo@wns Absent Member.
Absent Member.
1056 views

Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

I have a static log file (XYZ.log) which updates every 30 mins.  It has 19 lines of data.  I created the flex connector and have SUCCESSFULLY managed to send the events to the console however the connector does not fetch data each time the log file is updated.  The agent.properties file is below for your reference.  Would appreciate if anyone can help me out with this.

#ArcSight Properties File

#Tue Aug 27 18:42:47 IST 2013

agents.maxAgents=1

agents[0].AgentSequenceNumber=0

agents[0].configfile=Web_Mon

agents[0].destination.count=1

agents[0].destination[0].agentid=3mzqIv0ABABCqeAFTAdtTlw\=\=

agents[0].destination[0].failover.count=0

agents[0].destination[0].params=<?xml version\="1.0" encoding\="UTF-8"?>\n<ParameterValues>\n    <Parameter Name\="port" Value\="8443"/>\n    <Parameter Name\="filterevents" Value\="false"/>\n    <Parameter Name\="host" Value\="arcsightmgmtbcp_sec"/>\n    <Parameter Name\="aupmaster" Value\="false"/>\n    <Parameter Name\="fipsciphers" Value\="fipsDefault"/>\n</ParameterValues>\n

agents[0].destination[0].type=http

agents[0].deviceconnectionalertinterval=60000

agents[0].enabled=true

agents[0].entityid=3mzqIv0ABABCqeAFTAdtTlw\=\=

agents[0].extractfieldnames=

agents[0].extractregex=

agents[0].extractsource=File Name

agents[0].fcp.version=0

agents[0].fixedlinelength=-1

agents[0].followexternalrotation=true

agents[0].id=3mzqIv0ABABCqeAFTAdtTlw\=\=

agents[0].internalevent.filecount.duration=-1

agents[0].internalevent.filecount.enable=false

agents[0].internalevent.filecount.minfilecount=-1

agents[0].internalevent.filecount.timer.delay=60

agents[0].internalevent.fileend.enable=true

agents[0].internalevent.filestart.enable=true

agents[0].logfilename=E\:\\Testing_Flex_Defacement\\Logs\\Logs.log

agents[0].maxfilesize=-1

agents[0].onrotation=None

agents[0].onrotationoptions=processed

agents[0].persistenceinterval=1

agents[0].preservedstatecount=10

agents[0].preservedstateinterval=30000

agents[0].preservestate=false

agents[0].roationonlywheneventexists=false

agents[0].rotationdelay=30

agents[0].rotationscheme=None

agents[0].rotationsleeptime=10

agents[0].startatend=false

agents[0].type=sdkfilereader

agents[0].usealternaterotationdetection=false

agents[0].usefieldextractor=false

agents[0].usenonlockingwindowsfilereader=false

remote.management.second.listener.port=10050

remote.management.ssl.organizational.unit=RMqFv0ABABCAAdNzb1KiyA

server.base.url=https\://arcsightmgmtbcp_sec\:8443

server.registration.host=arcsightmgmtbcp_sec

Labels (3)
0 Likes
1 Solution

Accepted Solutions
ashis.sahoo@wns Absent Member.
Absent Member.

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

Hi Joachim,

Thanks for the help, I kinda figured it out.  The logs in my log file did not have TIMSTAMPS and almost all of them would repeat.  The logger remembers what logs it sends to the destination and keeps it in its cache (assumption).  Thats the reason why the logs were not being send all the time(I guess it does it to avoid duplication of logs send).  I created a new cron which would fill in a dummy line (i.e 1 more than the current 19 lines logs) in the log file.  The connector send the dummy line and refreshed the cache/memory with the dummy line send as this was a new log.  Since the cache currently only contained the dummy line, it send the 19 log lines in the next cron cycle.  I created a cron to add a dummyline every alternate cycle.  So now its sends logs each time after it reads the dummy line .

Thanks for all the help.

Best Regards,

Ashis Sahoo

0 Likes
6 Replies
Established Member.. Ahedge
Established Member..

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

If the log file has events added to it then you want the change the agents[0].preservestate=false to

agents[0].preservestate=true

This tells the SmartConnector to remember the last event that was processed and keep checking for new events.  It will also preserve the state if the connector is stopped and restarted.

If the log file is recreated each time every 30 minutes then you need to look into file rotation options within the SmartConnector.

0 Likes
ashis.sahoo@wns Absent Member.
Absent Member.

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

Hi Arthur,

Thank you for the help, I have kept agents[0].preservestate=true however the problem I am facing is that the connector is not sending logs when the file is being updated. 

Example : at      10:15 connector sends logs to the Manager

                         10:45 No logs send

                         11:15 No logs send

                         11:45 No logs send

                         12:15 Send logs to Manager.

The log files do not show any errors.  If you have experience anything like this would like your inputs.

0 Likes
ashis.sahoo@wns Absent Member.
Absent Member.

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

The log files has a fixed set of 19 logs which are updated(the whole fileis update) ever 30 mins

eg:

google.com: OK

yahoo.com: OK

test.com: Failed

note: the log file is not appended it is updated.

0 Likes
ashis.sahoo@wns Absent Member.
Absent Member.

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

I am sorry I was very unrefined in giving you all the information

We have a script which runs every 30 mins.  The script checks for information and lets us know if the criteria is satisfied.

It checks this information for 19 rows and provides a response of OK or Failed

eg:

google.com : OK

yahoo.com: OK

gmail.com: OK

test.com: Failed

Now what we have observed with the current connector config with agents[0].preservestate=true.  The connector when it start gives the first logs.  When the script runs it updates the log file with new set of data.  Now it does fetch the new set of data at times.  I have ensured that the cron is running and have tested it to check and the cron runs every 30 mins.  So there is no error on the CRON Job. So ideally the connector should fetch data every time the script updates the log file but as per the observation the connector does not send it each time.  It sends the logs in random intervals and sometimes with a gap of more than 8 hours.

I am not sure why it does that.  I have checked the agent.log and agent.out.wrapper.log and there are no errors or warnings.

Not sure why this would not send logs every 30 mins. 

Thank you for your time much appreciated.

0 Likes
jring1 Trusted Contributor.
Trusted Contributor.

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

Usually a connector will try to read each line only once (at least for the runtime - or even longer when preservestate is true). The only time when it reads a file from start again is when it has detected a file rotation - which doesn't happen here for whatever reasons - probably because the data in the file is very uniform and has no timestamps.

Did you try to have the connector rename the file after reading it with:

agents[0].onrotation=RenameFileInTheSameDirectory

This way it should see when a new file is created by your cronjob.

Joachim

0 Likes
ashis.sahoo@wns Absent Member.
Absent Member.

Re: Flex Connector(Log Line) not fetching events once file is updated

Jump to solution

Hi Joachim,

Thanks for the help, I kinda figured it out.  The logs in my log file did not have TIMSTAMPS and almost all of them would repeat.  The logger remembers what logs it sends to the destination and keeps it in its cache (assumption).  Thats the reason why the logs were not being send all the time(I guess it does it to avoid duplication of logs send).  I created a new cron which would fill in a dummy line (i.e 1 more than the current 19 lines logs) in the log file.  The connector send the dummy line and refreshed the cache/memory with the dummy line send as this was a new log.  Since the cache currently only contained the dummy line, it send the 19 log lines in the next cron cycle.  I created a cron to add a dummyline every alternate cycle.  So now its sends logs each time after it reads the dummy line .

Thanks for all the help.

Best Regards,

Ashis Sahoo

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.