firru Absent Member.
Absent Member.
1264 views

Windows Unified connector not sending logs

Hi All,

Am facing difficulties in rectifying the problem for the WUC.

We have installed a WUC on the VM machine, this WUC is collecting logs from 7 Domain controllers. till 2 days back everything was working fine but now the things changed and we are not receving logs from any of the DC. when I checked the agent logs found many warning message.

  • [2013-07-30 12:50:12,178][WARN ][default.com.arcsight.agent.l.v][getKeyValuePairs] No keys have been defined for an event with Event ID = [538], Event Log Type = [Security], Event Source = [Security], for Microsoft Windows keymap family = [Windows 2008 R2]. Please create or update the file [windowsfg\windows_2008\security.keymap.csv] with the appropriate keys for the event.
  • [2013-07-30 12:50:12,272][WARN ][default.com.arcsight.agent.jf.c][lookupAllByName] Cannot find information for

The connector is up and running but we are not seeing logs and also this connector is caching at very high rate. when checked the agent.wrapper file found an error as mention below.

  • INFO   | jvm 2    | 2013/07/29 14:44:54 | java.lang.OutOfMemoryError: unable to create new native thread
  • ERROR  | wrapper  | 2013/07/29 14:44:54 | JVM exited unexpectedly.

  • INFO   | jvm 4    | 2013/07/29 14:55:02 | [Mon Jul 29 14:55:02 AST 2013] [ERROR] Device connection to [ENTDCXXPTHZW1] down.(Host Unreachable)

Can I have any helpful input.

Thanks in advance

Labels (1)
0 Likes
Reply
9 Replies
jring1 Trusted Contributor.
Trusted Contributor.

Re: Windows Unified connector not sending logs

The lines

  • INFO   | jvm 2    | 2013/07/29 14:44:54 | java.lang.OutOfMemoryError: unable to create new native thread
  • ERROR  | wrapper  | 2013/07/29 14:44:54 | JVM exited unexpectedly.

are the most likely culprits. Look at Memory and GC/Full GC Messages in agent.out.wrapper.log and check wether mem usage there was ok or red. Also look at agent.wrapper.conf for the heap settings.

Is the VM really out-of memory or are you reaching the 32 bit boundary?

Joachim


0 Likes
Reply
firru Absent Member.
Absent Member.

Re: Windows Unified connector not sending logs

Hi jring,

Thanks for the reply.

I checked agent.out.wrapper.log and found the below info

  • INFO   | jvm 4    | 2013/07/29 16:12:27 | [GC 1107006K->1022928K(1193024K), 0.0090842 secs


  • INFO   | jvm 4    | 2013/07/29 16:12:37 | [Full GC 1087026K->208912K(1215296K), 0.4450278 secs]

And the heap memory is as below

wrapper.java.initmemory=512

wrapper.java.maxmemory=1536

The cpu utilization on the vm machine is 98%.

And also let me tell u that this connector caching many events.

0 Likes
Reply
jring1 Trusted Contributor.
Trusted Contributor.

Re: Windows Unified connector not sending logs

Have you checked why it is caching? Are all destinations ok and reachable?

wrapper.java.maxmemory=1536 is probably a bit high either for the physical memory of the VM or the 32 bit jvm...

Joachim

0 Likes
Reply
firru Absent Member.
Absent Member.

Re: Windows Unified connector not sending logs

We have 2 destinations ESM and Logger.

While all the events have been filtered out towards esm and am able to ping both the destination from the agent server.

Shall I reduce the jvm maxmemory..?

0 Likes
Reply
jring1 Trusted Contributor.
Trusted Contributor.

Re: Windows Unified connector not sending logs

Well, reducing wrapper.java.maxmemory will probably fix the crash with OutOfMemoryError but not your caching problem. Usual rule of thumb for wrapper.java.maxmemory is two times the value after a Full GC. If it is too large, Full GCs can take some time and the whole jvm is stopped during that time. Although .44s is not too bad...

Also it is not a bad idea to have wrapper.java.initmemory=wrapper.java.maxmemory at least for a test as this will show if wrapper.java.maxmemory is too high as the connector will not start at all then.

As for the caching problem, can you grep for Eps= and T= in agent.out.wrapper log and show us a few lines?

Joachim

0 Likes
Reply
firru Absent Member.
Absent Member.

Re: Windows Unified connector not sending logs

Below are the lines from agent.out.wrapper.log file

  • INFO   | jvm 175  | 2013/07/30 12:22:20 | [Tue Jul 30 12:22:20 AST 2013] [INFO ] {Eps=1124.5593220338983, Evts=460097}
  • INFO   | jvm 175  | 2013/07/30 12:22:20 | [Tue Jul 30 12:22:20 AST 2013] [INFO ] {C=0, ET=Down, HT=Up, N=Windows-PCI-2, S=0, T=0.0}
  • INFO   | jvm 175  | 2013/07/30 12:22:20 | [Tue Jul 30 12:22:20 AST 2013] [INFO ] {C=1155103, ET=Up, HT=Up, N=Windows-PCI-2, S=25375, T=0.0}

0 Likes
Reply
gbenga.ogunsaki1 Absent Member.
Absent Member.

Re: Windows Unified connector not sending logs

Hi,

Based on some of the log extracts posted on this thread, below are some of the assumptions we can make:

-Connector JVM constantly restarting possibly due to high rate of Full GCs.

During Full Garbage Collection, the jvm suspends every other operations and only resume when completed. Therefore, it will mean when there is high rate of Full GC, the connector is not performing its primary function - collecting events.

-Connector is caching events due to event transport down status.

When event transport to the configured destination is down, it will may be due to a number of factors, but usually when the destination is unable to handle the event volume received from the connector.

Therefore, some questions you must answer will include the following:

-Is the connector unable to forward events to all configured destinations?

-When was the first time this issue was observed?

-Has there been any configurations changes made to either the connector or configured destinations?

-Is the connector host on the same network/subnet with all of the DCs?

-What is the average EPS from each of the DCs

Also, my suggestion will be to upload a sanitized copy of the agent.properties file so we can understand your configuration.

Please, understand it may be best to have Technical Support help you analyze the issue.....my 2 cents

Cheers.

Gbenga.

0 Likes
Reply
Outstanding Contributor.. Pushpendra_Rathi Outstanding Contributor..
Outstanding Contributor..

Re: Windows Unified connector not sending logs

Hi Mohammed

The solution for this problem is to remove the logger destination....

and thn restart the agent, it will work perfect........... you can add the logger dest again.....

if you do not want to remove the logger thn you can also reboot the connector machine... it will work perfect but you might start getting error again....

.

Also make the memory same for both place...

wrapper.java.initmemory=512

wrapper.java.maxmemory=512

Cheers & Happy Ramadan...

0 Likes
Reply
Highlighted
Outstanding Contributor.. Pushpendra_Rathi Outstanding Contributor..
Outstanding Contributor..

Re: Windows Unified connector not sending logs

Hi Mohammed

The solution for this problem is to remove the logger destination....

and thn restart the agent, it will work perfect........... you can add the logger dest again.....

if you do not want to remove the logger thn you can also reboot the connector machine... it will work perfect but you might start getting error again....

.

Also make the memory same for both place...

wrapper.java.initmemory=512

wrapper.java.maxmemory=512

Cheers & Happy Ramadan...

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.