Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..
1521 views

Microsoft DNS FlexConnector doesn't work. File disappears after first processing.

Jump to solution

Hi,

I'm having Problems processing Microsoft DNS log files with FlexConnector. The logfile disappears after first successful processing an no new file will be created by Windows. I guess that the Connector sets a lock on the file and causes Problems at Windows site.

Does the usenonlockingwindowsfilereader Parameter works through a mounted Network share? It seems to be not...

###Background:###

• Microsoft DNS logs are saved in one log file. Path, max size and Content can be specified. When max size is reached, the log file rotates (Microosft DNS deletes file/Content and proceeds writing log entries to it)

•I Created a FlexConnector for it as the SmartConnector cannot parse the german timestamps and it's mode is set to "RenameFileinTheSameDirectory". This would Trigger Problems as another tool also processes this file.

•FlexConnector is using mode PersistFile to make sure the other tool can process the file too.

•Logfile is saved on the Microsoft DNS Server locally and the Directory is provided as a Share. This share is mounted on a SLES Server, where the connector is running on.

•Agent.Properties:

agents.maxAgents=1

agents[0].destination.count=1

agents[0].destination[0].agentid=xxx

agents[0].destination[0].failover.count=0

agents[0].destination[0].params=

\n\n    \n    \n    \n    \n\n

agents[0].destination[0].type=loggersecure

agents[0].deviceconnectionalertinterval=60000

agents[0].enabled=true

agents[0].entityid=xxx

agents[0].fcp.version=0

agents[0].foldertable.count=1

agents[0].foldertable[0].badsubfolder=bad

agents[0].foldertable[0].configfile=dns_tracelog_file

agents[0].foldertable[0].configfolder=/opt/arcsight/DNS/current/user/agent//flexagent/

agents[0].foldertable[0].configtype=sdkrfilereader

agents[0].foldertable[0].delay=10000

agents[0].foldertable[0].encoding=

agents[0].foldertable[0].extractfieldnames=

agents[0].foldertable[0].extractregex=

agents[0].foldertable[0].extractsource=File Name

agents[0].foldertable[0].fixedlinelength=-1

agents[0].foldertable[0].fixedlinelengthcontains=Fixed Number Of Characters

agents[0].foldertable[0].folder=/opt/arcsight/dns/

agents[0].foldertable[0].followexternalrotation=true

agents[0].foldertable[0].maxretries=-1

agents[0].foldertable[0].minfilelenght=-1

agents[0].foldertable[0].mode=PersistFile #--> Need to make sure the other tool can process the file too.

agents[0].foldertable[0].modeoptions=processed

agents[0].foldertable[0].monitoringinterval=30000

agents[0].foldertable[0].onrotation=None

agents[0].foldertable[0].onrotationoptions=processed

agents[0].foldertable[0].preservestate=true

agents[0].foldertable[0].processfoldersrecursively=false

agents[0].foldertable[0].processinglimit=256

agents[0].foldertable[0].processingmode=realtime

agents[0].foldertable[0].processingthreshold=3600000

agents[0].foldertable[0].processingtimeout=120000

agents[0].foldertable[0].retryinterval=1000

agents[0].foldertable[0].sleeptime=5000

agents[0].foldertable[0].startatend=false

agents[0].foldertable[0].triggerextension=.done

agents[0].foldertable[0].usealternaterotationdetection=true

agents[0].foldertable[0].usefieldextractor=false

agents[0].foldertable[0].usenonlockingwindowsfilereader=true --> I think this Parameter does not work for my deployment.

agents[0].foldertable[0].usetriggerfile=false

agents[0].foldertable[0].wildcard=Dns.log

agents[0].id=xxx agents[0].internalevent.filecount.duration=-1

agents[0].internalevent.filecount.enable=false

agents[0].internalevent.filecount.minfilecount=-1

agents[0].internalevent.filecount.timer.delay=60

agents[0].internalevent.fileend.enable=true

agents[0].internalevent.filestart.enable=true

agents[0].persistenceinterval=0

agents[0].preservedstatecount=10

agents[0].preservedstateinterval=30000

agents[0].type=sdkmultifolderreader

agents[0].unparsedevents.log.enabled=true

remote.management.second.listener.port=10053

remote.management.ssl.organizational.unit=xxx

Does anyone has experience with such Problems? Thank you in advance.

Cheers,  Demian

Labels (3)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Hi,

I finally found a solution.

A Little bit Background:

Microsoft DNS log Rotation works as follows:

1. Write log file until the specified Maximum size is reached

2. backup file to C:\Windows\System32\dns\backup\

3. deleting original file

4. create a new one and write to it.

Problem:

Windows can't create the file if ArcSight still has a read handle on it and the deletion can't be finished. Thus, the file just disappears.

Solution:

The Scenario above just happens, when the log file is written to a Partition other than C:\ (System Partition). If the file is written to C:\, Windows just renames the file (instead of copying it). As a consequence, Microsoft DNS can create a new file with the same Name.

You maybe Need to Change some Parameters in the Agent.properties. At least my FlexConnector is working fine...

View solution in original post

0 Likes
14 Replies
Highlighted
Established Member..
Established Member..

Hi Demian!

You just need to change flex connector type. Try to use "follower" instead of "reader".

0 Likes
Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Hi,

thanks for your fast reply. I used the "Multi Folder Follower" Flex connector. or what do you mean by "follower instad of Reader"?

Cheers, Demian

0 Likes
Highlighted
Absent Member.
Absent Member.

I'm using the ArcSight built-in MS DNS SmartConnector.

It's not working. I.e. I'm experiencing similar problems as OP.

After spending a couple of months with HP Support, I was told "It's a Microsoft problem".

I'm also mounting the files/windows shares on a Linux based Connector server. I have around 15 DNS servers I need to 'follow'.

Has anyone succesfully implemented this for MS DNS servers?

Regards,

0 Likes
Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

What SmartConnector Version are you using? There were some Problems which have been solved since Version 1.7 (as far as I remember).

Do you have write permissions on the share (Windows site)? Is the share mounted as rw?

0 Likes
Highlighted
Absent Member.
Absent Member.

Hello Demian,

Agent Version: 7.1.7.7600.0

The Windows shares are mounted read-only.

Regards,

Morten

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

You wouldn't believe how many times I have dealt with this problem. I have no idea how resolve it, I think it's a Windows problem.

In my experience this problem is inconsistent and happens when the file is truncated.

There are lots of people reporting this on the Internet.

Regards,

Vini

0 Likes
Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Try mounting it rw and make sure your Windows user accessing the share has write permissions too.

The SC mode is "RenameFileInTheSameDirectory"...

0 Likes
Highlighted
Absent Member.
Absent Member.

I am/was seeing a lot of memory warnings in the agent.log. Even after allocating 2GB RAM (maximum) I still got those. To me it almost looked like a memory leak of some sort.

The DNS servers created an 'Event ID’s 3152' at their end. Not particularly useful either.

Regards,

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

To be clear, this problem doesn't have ANYTHING to do with the connector. It can't even lock the file and the problem still happens.

In an environment I used to work it would happen with or without the connector reading the file.

Sorry to be the bearer of bad news.

0 Likes
Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Just to be clear. What exactly was your Problem? Disappearing of the DNS Log File? You mounted a Windows share on a Linux system?

By saying lots of People reporting such issues on the Internet, you mean even beside arcsight?

0 Likes
Highlighted
Absent Member.
Absent Member.

My problem was the disappearance of the DNS log file. The DNS service had to be restarted on the server to log again. A day or so later it would happen again.

I will try mounting the shares rw like you suggested.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.