Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
PeSeKa Honored Contributor.
Honored Contributor.
552 views

Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

Hello, 

I have a issue with the smart connector for microsoft exchange message tracking. 

It is a "Multiple Server File" connector and reads logs from text files. As soon as I start the connector, it reads all the files and logs inside, but after a while it goes to block (and it does not read the logs anymore).

In the agent.log file I have the following warnings:

[2019-02-25 15:29:56,506][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 220
[2019-02-25 15:29:59,454][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551104983199}
[2019-02-25 15:30:22,496][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551104922483, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551104922483}
[2019-02-25 15:30:29,469][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105023229}
[2019-02-25 15:30:56,613][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 165
[2019-02-25 15:30:59,483][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105043244}
[2019-02-25 15:31:29,498][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105083274}
[2019-02-25 15:31:56,751][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 225
[2019-02-25 15:31:59,513][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105103289}
[2019-02-25 15:32:02,508][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551104922483, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551104922483}
[2019-02-25 15:32:29,527][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105143318}
[2019-02-25 15:32:56,890][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 220
[2019-02-25 15:32:59,542][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105163333}
[2019-02-25 15:33:29,556][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105203363}
[2019-02-25 15:33:42,520][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551104922483, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551104922483}
[2019-02-25 15:33:57,028][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 220
[2019-02-25 15:33:59,571][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105223378}
[2019-02-25 15:34:29,586][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105263408}
[2019-02-25 15:34:57,151][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 165
[2019-02-25 15:34:59,600][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105283423}
[2019-02-25 15:35:22,532][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105222520, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105222520}
[2019-02-25 15:35:29,615][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105323453}
[2019-02-25 15:35:57,274][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 220
[2019-02-25 15:35:59,629][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105343468}
[2019-02-25 15:36:29,644][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105383498}
[2019-02-25 15:36:57,381][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 210
[2019-02-25 15:36:59,659][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105403512}
[2019-02-25 15:37:02,545][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105222520, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105222520}
[2019-02-25 15:37:29,674][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105443543}
[2019-02-25 15:37:57,523][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 225
[2019-02-25 15:37:59,691][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105463559}
[2019-02-25 15:38:29,708][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105503592}
[2019-02-25 15:38:42,563][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105222520, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105222520}
[2019-02-25 15:38:57,618][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 110
[2019-02-25 15:38:59,725][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105523608}
[2019-02-25 15:39:29,741][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105563641}
[2019-02-25 15:39:57,745][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 165
[2019-02-25 15:39:59,758][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105583658}
[2019-02-25 15:40:22,583][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105522563, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105522563}
[2019-02-25 15:40:29,775][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105623690}
[2019-02-25 15:40:57,888][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 220
[2019-02-25 15:40:59,791][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105643707}
[2019-02-25 15:41:29,807][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105683738}
[2019-02-25 15:41:58,044][WARN ][default.com.arcsight.event.SecurityEvent][setModelConfidence] Number of bad threat level values received and corrected = 220
[2019-02-25 15:41:59,823][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105703754}
[2019-02-25 15:42:02,600][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[AETB[3y1JVj2UBABCABTfV87Xd9A==]]: {E0:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105522563, E1:3y1JVj2UBABCABTfV87Xd9A==.lastCheckpointTime=1551105522563}
[2019-02-25 15:42:29,838][INFO ][default.com.arcsight.util.AgentUtil][logfuLog] WatchDog[WDP[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443]]: {IT[loggersecure-3y1JVj2UBABCABTfV87Xd9A==][xx.xx.xx.xx][443].lastCheckpointTime=1551105743785}

--------------------------------------------------------------------------------

 

In the logs of the previous days instead, there are the following errors:

[2019-02-20 05:45:32,064][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:45:41,040][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:45:41,367][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:45:50,353][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:45:50,714][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:00,289][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:00,648][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:09,665][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:10,008][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:20,595][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:20,907][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:29,248][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:29,591][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:40,198][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:40,521][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:49,572][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:49,900][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:46:59,293][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:46:59,620][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:47:08,667][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:47:09,007][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:47:17,969][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:47:18,312][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:47:29,591][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:47:29,591][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:47:38,892][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:47:39,188][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.
[2019-02-20 05:47:49,729][FATAL][default.com.arcsight.agent.util.ao][run]
java.lang.OutOfMemoryError: GC overhead limit exceeded
[2019-02-20 05:47:50,056][ERROR][default.com.arcsight.util.memmon.MemoryMonitor][reportOutOfMemoryError] Caught out of memory error.

 

---------------------------------------------------------

Help me, please.

Best Regards

0 Likes
1 Solution

Accepted Solutions
Highlighted
Knowledge Partner
Knowledge Partner

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

Connector periodically performs garbage collection and full garbage collection. You can see this activity in your agent.out.wrapper.log as GC and Full GC. A healthy connector performs Full GC every hour.  If it starts to do it every 10 minutes, it means that the connector needs memory. I attached a health check document. Hope it gives a better idea.

 

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
6 Replies
Knowledge Partner
Knowledge Partner

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

First, I suggest putting the error log in code format here. (nex to photos button).

What is you memory settings in agent.wrapper.conf (under <connector home>/current/user/agent/)? If it's small, you can increase it. Do not give bigger than 4GB.

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
PeSeKa Honored Contributor.
Honored Contributor.

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

 

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=256

I set maxmemory = 512. Ok?

0 Likes
Knowledge Partner
Knowledge Partner

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

Hello,

 

i recommend to increased the Java heap size at least to 2048.

 

Best Regards,

 

Daniel

0 Likes
PeSeKa Honored Contributor.
Honored Contributor.

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

All right. But the connector block depends only on the memory according to you?

 

Regards

0 Likes
Knowledge Partner
Knowledge Partner

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

Hello,

 

for sure this will help.

Also, make sure that the destination for your parsed logs it is reachable as well and the SM it's not caching.

All the best,

 

Daniel

0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: Microsoft Exchange Message Tracking Smart Connector Error

Jump to solution

Connector periodically performs garbage collection and full garbage collection. You can see this activity in your agent.out.wrapper.log as GC and Full GC. A healthy connector performs Full GC every hour.  If it starts to do it every 10 minutes, it means that the connector needs memory. I attached a health check document. Hope it gives a better idea.

 

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.