WUC events: Detected Clearing of the log

[nwcook]

WUC connector running on Win2K 2012 64 bit (7.0.5.7132.0).

Configured to collect Security, System and Application logs from a number of hosts.

ArcSight seems to be generating events with the name "Event log rotated [Detected Clearing of the log]" for the application and system logs - but only for some servers and for some servers more frequently than others (e.g. a different ArcSight windows based connector server).

System, application, and security logs are configured to "overwrite as needed", but the size of the allocated log size varies across servers.

Oddly I don't seem to see this event occur for the Security log on any server.  And no, the event logs themselves are not being cleared.

Any ideas what is causing this and how to resolve?

[michael.loy@hpe]

Hi Norm,

In the agent.log and agent.out.wrapper.log files, look for any message saying "rotated".  This help in troubleshooting the issue. If you get a chance post your logs if you want additional feedback.

[nwcook]

Will post logs if more info required.

In agent.out.wrapper.log I see several events such as these:

INFO   | jvm 6    | 2015/01/08 13:23:02 | [Thu Jan 08 13:23:02 MST 2015] [INFO ] Policy for Log type [Security] for host [SERVER001] is adjusted. Oldest record [922837], last processed[949672]

INFO   | jvm 6    | 2015/01/08 13:37:06 | [Thu Jan 08 13:37:06 MST 2015] [INFO ] A log cleared for host-log type [SERVER001][Application]

Nothing really of note in agent.log.

I'm going to try to isolate my troubleshooting by creating a WUC to look at one server instead of the 140 I have currently.  That will reduce a pile of noise in the log files.

[nwcook]

Logs attached.    Interesting thing of note as well is that ESM created a new device object for the server 'connectorserver001' but it had the IP of connectorserver002.

[michael.loy@hpe]

Hi Norm,

Your logs are being cleared on the operating systems generating the events or at least it looks that way.  in this case it is "connectorserver001"  for system and application logs.

For a run down on some of the events:

10:04:36,506][INFO ][default.com.arcsight.agent.ub.hb][isAbleToConnectToHost] Connect to host [connectorserver001]

10:04:36,506][INFO ][default.com.arcsight.agent.ub.hb][reconnectToHost] Successfully created the rpc handle for  Host [connectorserver001] log [System

The connector has connected to the server

***

2015-01-09 10:04:36,616][INFO ][default.com.arcsight.agent.ub.hb][reconnectToHost] Connected to host [connectorserver001], EventLog [Application], read [0] events.

[2015-01-09 10:04:36,616][INFO ][default.com.arcsight.agent.ub.hb][reconnectToHost] Connected to host [connectorserver001], EventLog [System], read [0] events.

[2015-01-09 10:04:36,616][INFO ][default.com.arcsight.agent.ub.hb$0][run] setDeviceState,  Host [connectorserver001] log [Application], state changed to [true], time elapsed [0]

[2015-01-09 10:04:36,616][INFO ][default.com.arcsight.agent.ub.hb$0][run] setDeviceState,  Host [connectorserver001] log [System], state changed to [true], time elapsed [0]

[2015-01-09 10:04:36,616][INFO ][default.com.arcsight.agent.ub.hb$1][run] A log cleared for host-logType [connectorserver001][Application], last processed index [0]

[2015-01-09 10:04:36,616][INFO ][default.com.arcsight.agent.ub.hb$1][run] A log cleared for host-logType [connectorserver001][System], last processed index [0]

Application and System logs are empty or do not have any new updates since last time the WUC pulled.

Next steps.

Disable Application and system pulling from the WUC.

Wait until there are application or system logs on the server after the disable time on the WUC.

restart the WUC and see if the problem persists.

If you see those logs and it still generating these events then something will need a support case.  I did not see any bugs on this so it is a bit odd if you see the logs are not cleared.

If you are seeing the WUC hand shake and pull logs off of other servers successfully it can be isolated to a a particular server where system and application logs pulling is not working so it might be corrupt logs on that server.

[nwcook]

Thanks for the attempt, Michael, but it looks like I'm going to have to call tech support.

- Disabled App and system logs in the WUC.

- Ensured events appeared in windows log for both system and event.

- Re-enabled app and system logs in WUC.

- Also ensured I had correct Operating system (2012R2) in the WUC.

Seeing 10-12 "Event log rotated.." entries for system log every second.

I am not actually getting any real events from the system or application logs into ESM.

I have events dating back a month so VERY confident nothing is clearing the logs.  Especially 12 times/second as the WUC is detecting.

Just for kicks, I cleared the system and application logs manually.  No change in behaviour.

[nwcook · Accepted solution]

I ended up finding the solution myself.   

The problem is that Application and System logs require DIFFERENT permissions to access than “Security” logs. 

With Security logs you add the ArcSight credential to “Manage Auditing and Security Log” in the Local Security Policy via GPO.

With other logs you add the ArcSight Credential to the local “Event Log Readers” group via GPO (not recommended by me unless you are testing very carefully.  Our exchange crashed because we had exchange admin in that group on the local server and the GPO overwrote the setting and putting the exchange admin account in the GPO did not fix it).

I missed that setting in the doc because we run a windows 2003 domain.  This is not in the instructions for 2003 domain controllers within the document “SmartConnector Configuration Guide for Microsoft Windows Event Log – Unified”. 

So I messed up, but I blame TFM.

I had opened a case with support so I recommended they maybe add some better error messaging in the WUC logs to report an error reading the log as opposed to the log being rotated.  That totally sent me on the wrong path.