Highlighted
New Member.
1583 views

Major bug in Windows Unified Connector 5.2.1 release

As probably many of you on this board I was very excited when 5.2.1 came out because it incorporated a number of fixes for Windows Unified Connector. I upgraded a few of my agents and ran into a bug which in my opinion is more severe than the bugs in 5.1.7 and wanted to make the group aware of it in case anyone has also upgraded because there is a serious potential for missed events, which I know a lot of environments can't tolerate.

To check if you are impacted issue a Get Status command to a WUC agent. Among the data returned will be a list of all hosts that the connector is pulling events from in a format similar to the following:

server_name.domain.com[Security].......-1295739632
server_name.domain.com[Security].timestamp..1333997543000

The first line is the index of the last event received from that device. If that index is a negative number (and about a quarter of my hosts are) and you are running 5.2.1, chances are you are not getting all the events from that host. The initial connection is fine and it's able to maintain the connection, so there are no device up/down events, but I only get a very small number of events, and even those come in sporadically, probably when the event with the negative index gets rotated out, but that's just my guess.

I opened a case with support and dev identified a bug, CON-11448, so consider this an FYI and a warning.

Labels (2)
0 Likes
Reply
29 Replies
Highlighted
Respected Contributor.
Respected Contributor.

Re: Major bug in Windows Unified Connector 5.2.1 release

Gary,

Thanks for the Heads up!  I was about to upgrade.

Steve

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

Gary,

  Great thanks for the heads-up.  Any idea if it's fixed in 5.2.2?

Anton

0 Likes
Reply
Highlighted
New Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

Unfortunately it wasn't

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

I can confirm this issue with a handful of my systems as well. The only fix I am aware of at this time is to save a copy of the affected log and then clear the log. Doing so corrects the negative number and new events begin to come in again. The option I went with was a downgrade back to 5.1.7. One of the factors of commonaility between the systems reporting a large negative number and the ones that weren't was the event log size. On those systems, the size of the log on the disk is much greater than the maximum number set in group policy. Changing group policy to be larger or the same size as event log does not seem to correct anything. Can anyone confirm this commonaility?

0 Likes
Reply
Highlighted
New Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

Unfortunately clearing the event logs is not a possibility for me and I wonder if it's only a temporary solution. When the logs fills back up to some certain size, won't the index go back into negatives again?  I am not clear on the statement you make about the size of the log being larger than the maximum allowed by group policy. Theoretically that shouldn't be possible, since that's kind of the point of the group policy.

From my experience it looks like the busier hosts have this problem, so my theory is that Windows maintains an internal index of the last event log entry, probably from the point the log was last cleared, regardless of the size. I believe that value keeps constantly increasing on the host, even if the log is wrapping around. Clearing the log probably resets it back to zero. The maximum value of that index in Windows is probably larger than the variable in the WUC can hold. I am guessing WUC is using int as a data type to hold this value which has a maximum value of 2147483647 and they should have gone with double as the type or something along those lines

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

The size of the log file being larger than the group policy maximum was just as confusing to see as it is to explain. You are right that theoretically shouldn't be possible so I was trying to find evidence that the group policy setting permitted a larger size at one time, perhaps during server imaging but nothing conclusive.

From my experience the busiest of hosts don't seem to be affected by this. Hosts that generate about 15 eps are working fine under 5.2.1 but a host that generates about 3 eps was having issues. Your theory seems very plausible but there maybe some time of event log corruption here that could be adding to the problem.

0 Likes
Reply
Highlighted
Honored Contributor.
Honored Contributor.

Re: Major bug in Windows Unified Connector 5.2.1 release

Gary,

Good catch... out of curiosity what kind of setup are you running?

Smart connectors installed on a server if so what OS? or are you running the connector appliances? if so what model?

We're running 5.2.1 here and after seeing this I ran your check against ours and we appear to not be affected by this.

I'm running C5400 appliances in all but 1 LOB over there we're still on C5100 & C5200 series systems.

0 Likes
Reply
Highlighted
New Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

I am seeing this on C5100 and C5200's, but I don't think it's OS/platform related.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

Hello Everyone,

First of all, thanks to Gary for raising this as FYI for all of you and many thanks to everyone for sharing their thoughts and observations.

I want to assure you that all the right teams within HP ArcSight are engaged to troubleshoot this issue and isolate the root cause as soon as possible. ArcSight Support  Back line team and Engineering teams have been actively working on this effort.

If any of you have experienced this problem and have not yet opened a support ticket, I request you to do so. We will update this thread with more information from our investigation and next actions as it becomes available.

Thank you,

Medha Rangnekar

Manager, Technical Support (Tier 3)

ArcSight, an HP Company

HP Enterprise Security Products

Support Hotline (866)535-3285

0 Likes
Reply
Highlighted
Established Member..
Established Member..

Re: Major bug in Windows Unified Connector 5.2.1 release

Spent about 4 hours troubleshooting to find the exact same issue as is mentioned here, thanks for posting it...  We were previously notified to upgrade to >5.2.1 due to the connector device status monitoring bugs being resolved.

I will be doing further testing because this did not occur to all my connectors, it only seemed to happen to *some* (about 5 and growing by the minute) of my Win 2k3 servers (didnt seem to happen on the 2k8 servers), and not necessarily devices with high event volumes, it seemed somewhat sporadic, and also interesting that one connector could have a functioning server and non functioning on the same connector...

2k3_AD_Server[Security]....-2064389342
2k3_AD_Server[Security].timestamp..1334809498000
2k8_AD_Server[Security]....247322442
2k8_AD_Server[Security].timestamp..1334820125000
windowshoststable[0].Domain Name.............
windowshoststable[0].application.............false
windowshoststable[0].hostname................2k3_AD_Server
windowshoststable[0].locale..................en_US
windowshoststable[0].password................**********
windowshoststable[0].security................true
windowshoststable[0].system..................false
windowshoststable[0].username................
windowshoststable[0].windowsversion..........Windows Server 2003
windowshoststable[1].Domain Name.............
windowshoststable[1].application.............false
windowshoststable[1].hostname................2k8_AD_Server
windowshoststable[1].locale..................en_US
windowshoststable[1].password................**********
windowshoststable[1].security................true
windowshoststable[1].system..................false
windowshoststable[1].username................
windowshoststable[1].windowsversion..........Windows Server 2008 R2

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Major bug in Windows Unified Connector 5.2.1 release

Hello Everyone,

We would like to provide an update on the progress we are making on this issue. It appears that the Windows Unified Connector (WUC) starting from 5.2.1.6186 SmartConnector release build is not able to retrieve events from a Windows host/server that has a large Event Record ID value (in the range of billions). HP ArcSight Customer Support and Engineering teams are actively working on a resolution. We want to assure you that we are treating this issue as a top priority and will keep you posted on further updates.

Thank you for your patience.

Susan Li

Manager, Product Management

ArcSight, an HP Company

HP Enterprise Security Products

Support Hotline (866)535-3285

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.