ALERT! The community will be read-only starting on April 19, 8am Pacific as the migration begins. Read more for important details.
ALERT! The community will be read-only starting on April 19, 8am Pacific as the migration begins.Read more for important details.
Ensign
Ensign
1558 views

persisted iis log files behavior

hello,

I'm having some issues while configuring a iis_multiserver connector, I'm reading log files from IIS 7.5 by mounting the log directories read-only from the IIS servers to the connector appliance.

Everything seems to be working, but if I restart the connector, arcsight re-reads all the files and re-send all the logs, it seems it doesn't care I set the "preservestate=true" property and "startatend=true", any hint? I have thousands of duplicates every time I restart the connector.

Here's part of my agent.properties, I anonymized some information:


agents[3].AgentSequenceNumber=3

agents[3].destination.count=1

agents[3].destination[0].agentid=3ZMChWUEBABCdsP0tV697Lg\=\=

agents[3].destination[0].failover.count=0

agents[3].destination[0].params=<?xml version\="1.0" encoding\="UTF-8"?>\n<ParameterValues>\n    <Parameter Name\="port" Value\="443"/>\n    <Parameter Name\="host" Value\="10.60.8.1"/>\n    <Parameter Name\="rcvrname" Value\="IIS"/>\n    <Parameter Name\="compression" Value\="Disabled"/>\n    <Parameter Name\="fipsciphers" Value\="fipsDefault"/>\n</ParameterValues>\n

agents[3].destination[0].type=loggersecure

agents[3].deviceconnectionalertinterval=60000

agents[3].enabled=true

agents[3].entityid=gtGhWUEBABCdsf0tV697Lg\=\=

agents[3].fcp.version=0

agents[3].foldertable.count=12

agents[3].id=3ZMChWUEBABCdsP0tV697Lg\=\=

agents[3].internalevent.filecount.duration=-1

agents[3].internalevent.filecount.enable=false

agents[3].internalevent.filecount.minfilecount=-1

agents[3].internalevent.filecount.timer.delay=60

agents[3].internalevent.fileend.enable=true

agents[3].internalevent.filestart.enable=true

agents[3].lastcharacterrechecktime=500

agents[3].persistenceinterval=0

agents[3].preservedstatecount=10

agents[3].preservedstateinterval=30000

agents[3].type=iis_multiserver


agents[3].foldertable[10].badsubfolder=bad

agents[3].foldertable[10].configfile=iis/iis_file

agents[3].foldertable[10].configfolder=config/agent/oldsdk/

agents[3].foldertable[10].configtype=sdkfilereader

agents[3].foldertable[10].delay=10000

agents[3].foldertable[10].encoding=UTF8

agents[3].foldertable[10].extractfieldnames=

agents[3].foldertable[10].extractregex=

agents[3].foldertable[10].extractsource=File Name

agents[3].foldertable[10].filesizecheck=false

agents[3].foldertable[10].fixedlinelength=-1

agents[3].foldertable[10].fixedlinelengthcontains=Fixed Number Of Characters

agents[3].foldertable[10].folder=/opt/mnt/[machine_name]/[SiteName]

agents[3].foldertable[10].followexternalrotation=false

agents[3].foldertable[10].ignoredwebsites=

agents[3].foldertable[10].latestlogonly=true

agents[3].foldertable[10].maxretries=-1

agents[3].foldertable[10].minfilelenght=-1

agents[3].foldertable[10].mode=PersistFile

agents[3].foldertable[10].modeoptions=processed

agents[3].foldertable[10].monitoringinterval=60000

agents[3].foldertable[10].preservestate=true

agents[3].foldertable[10].processfoldersrecursively=false

agents[3].foldertable[10].processinglimit=256

agents[3].foldertable[10].processingmode=realtime

agents[3].foldertable[10].processingthreshold=-1

agents[3].foldertable[10].processingtimeout=-1

agents[3].foldertable[10].retryinterval=1000

agents[3].foldertable[10].sleeptime=30000

agents[3].foldertable[10].startatend=true

agents[3].foldertable[10].triggerextension=.done

agents[3].foldertable[10].usealternaterotationdetection=false

agents[3].foldertable[10].usefieldextractor=false

agents[3].foldertable[10].usenonlockingwindowsfilereader=true

agents[3].foldertable[10].usetriggerfile=false

agents[3].foldertable[10].version=7.5

agents[3].foldertable[10].wildcard=u_ex*.log


thank you


Tags (3)
0 Likes
7 Replies
Commodore
Commodore

I've got the same exact issue.  The only work around I could come up with is to limit/remove the IIS logs in the folder you're parsing from after they've been processed.  So you could have a script that runs on the server that moves log files older than 2 days to another directory.  Or you could just remove them completely from the box after they've been read.

Pretty frustrating really because you know it SHOULD be working correctly... 

------
0 Likes
Absent Member.
Absent Member.

1. Make sure there is a file called "persisted.properties" in your "/opt/arcsight/connector_#/current/user/agent" directory. This file should have a list of the logs that it has already processed so it doesn't have to go through them again.

2. Here is what I have configured in the agent.properties file that is working for us.

agents[0].foldertable[0]..mode=PersistFile

agents[0].foldertable[0].modeoptions=processed

agents[0].foldertable[0].preservestate=true

agents[0].foldertable[0].startatend=true


0 Likes
Commodore
Commodore

What permissions does the account used to read the log files have?  I have the same setup as yours but our user had to be read only so Im assuming that might be part of the case.

------
0 Likes
Absent Member.
Absent Member.

Admin.

0 Likes
Ensign
Ensign

We too have the same setup and there is the file is a file called "persisted.properties" in "/opt/arcsight/connector_#/current/user/agent" directory. Our user has read only privileges: is this the problem?

I have many other connectors (not IIS) with same setup ( "preservestate=true" property, "startatend=true" and user with read only privileges), but in these cases all is working well.

0 Likes
Commodore
Commodore

Could be the issue.  Unfortunately Im unable to have that user setup with admin priviledges - something about having admin rights on a production server with a service account....I don't know why anyone would care about that sorta thing.  ha. 

Either way, its worth a shot if your thinking about troubleshooting.

------
0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Something may have changed since 4 years ago when I found this problem, but if it hasn't, here's the issue.  Since the IIS logs can have multiple columns and that's configurable by the IIS admin, the connector must read the header to find out the columns to parse.  If it starts at the end of the log, it'll never grab this header, and as such won't work.  I had a ticket open for a month before I figured this out. 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.