olbap_zaid Absent Member.
Absent Member.
885 views

Sentinel stopped correlating events from collector manager

Hi everyone,
I have a working Sentinel server working with correlation rules from a long time ago. Yesterday correlations stopped firing with the patters we look for using our rules. Doing some research, I see that events coming from Collector managers are not firing correlations, meanwhile local events (like trying to log into Sentinel with worng password) fires correlations as expected.
Trying to see if there is any problem with my correlation rules, I´ve "test rule" against received errors, and they fires correlations as expected against recent logs, but deployed rules are not being able to see the same events that the test rule sees.
I can´t see errors or unusual things in addition to this situation. I´ve restarted sentinel server, correlation engine, undeployed the rules... but I can´t get correlation working.
I´m using Sentinel Version: 8.1.0.1_4000. My collector managers are remote, and the Correlation Engine is the one which comes with the Sentinel Server.


Has somebody had a situation like the one I´m describing?
Thanks in advance
0 Likes
6 Replies
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Sentinel stopped correlating events from collector manager

Have you verified that time is synchronized well across all of your
systems? If it gets too far out, something like thirty (30) seconds, then
correlation will not apply to the events that appear to be from the wrong
time.

It may also help if we had the exact text of your correlation rules that
are not working so we can understand the logic.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
ScorpionSting Absent Member.
Absent Member.

Re: Sentinel stopped correlating events from collector manag

ab;2485057 wrote:
Have you verified that time is synchronized well across all of your
systems? If it gets too far out, something like thirty (30) seconds, then
correlation will not apply to the events that appear to be from the wrong
time.

It may also help if we had the exact text of your correlation rules that
are not working so we can understand the logic.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.


You can confirm the timing issue by checking the server0.0.log - it will have something about excluding events because they are outside the 30 second window (can't remember it exactly off the top of my head - been a while)

Visit my Website for links to Cool Solution articles.
0 Likes
brandon-langley Absent Member.
Absent Member.

Re: Sentinel stopped correlating events from collector manag

olbap_zaid;2485056 wrote:
Hi everyone,
I have a working Sentinel server working with correlation rules from a long time ago. Yesterday correlations stopped firing with the patters we look for using our rules. Doing some research, I see that events coming from Collector managers are not firing correlations, meanwhile local events (like trying to log into Sentinel with worng password) fires correlations as expected.
Trying to see if there is any problem with my correlation rules, I´ve "test rule" against received errors, and they fires correlations as expected against recent logs, but deployed rules are not being able to see the same events that the test rule sees.
I can´t see errors or unusual things in addition to this situation. I´ve restarted sentinel server, correlation engine, undeployed the rules... but I can´t get correlation working.
I´m using Sentinel Version: 8.1.0.1_4000. My collector managers are remote, and the Correlation Engine is the one which comes with the Sentinel Server.


Has somebody had a situation like the one I´m describing?
Thanks in advance


We ran into this issue with a customer that had both ntp and VMware time synchronization running. Unfortunately the VM Host of the remote collector manager was 10 minutes off, so the system was constantly bouncing between being current and being 10 minutes behind.

For some odd reason because of this conflicting condition, the server wasn't able to detect the time disparity, and as a result no correlations happened.


The best way to ensure this isn't a problem:

1) Make sure time synchronization on all of our Sentinel server component (Server, Collector Manager, Correlation Engine) VMs do NOT have their time synchronizing with the VMWare host (in some cases this can also create performance problems due to how VMWare synchronizes time!)

2) Make sure all of your Sentinel server components (Server, Collector Manager, Correlation Engine) are using the same NTP configuration, and that they are sync'd to that configuration

I would make sure the above two things are correct along with the other suggestions this way, and hopefully this should ensure that at least the Sentinel backend is not the culprit (there may be other configuration issues such as 'trust event source time' and event sources that have inaccurate times, but those become easier to track down at that point.
0 Likes
olbap_zaid Absent Member.
Absent Member.

Re: Sentinel stopped correlating events from collector manag

Thanks for everyone. The first thing i thought was timensync. But i didn't know about vm time sync. I will ask to our vm team to analyze that. I will return soon with results.
0 Likes
Highlighted
olbap_zaid Absent Member.
Absent Member.

Re: Sentinel stopped correlating events from collector manag

yes, there was a problem with the time sync.
thanks to everyone
0 Likes
brandon-langley Absent Member.
Absent Member.

Re: Sentinel stopped correlating events from collector manag

olbap_zaid;2485083 wrote:
yes, there was a problem with the time sync.
thanks to everyone


Glad it was something straightforward! 🙂
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.