Thank you for all of your work on this. Appreciated!
The issue that was previously caught was a separate issue it turns out. The bug was referenced a little generically, "Logger Returning Inconsistent Results in Distributed Search". It was thought at that time that your issue fell under this bug. The fix for that issue was rolled into v6.1 (sometime in September/October).
When your case was escalated to my I found out via the developer that the changes in v6.1 were also in v6.0 P2. That was why I was anxious to get his opinion on why you would not be seeing the benefit of those code changes. Well, it turns out it was a completely new bug. I do not know why the changes in patch 2 were not release noted.
The issue we have with your system is not reproducible on our test systems. Whether that is due to an the uniqueness of the data or the uniqueness of the cluster is not yet known. I am going to attempt this repro with the data and query you have provided. It may take me some time to play back these events but I will begin tomorrow.
Thu Jul 30 22:25:10 GMT 2015
I think you make a typo mistake because the two searches in you first messages are completely different.
With the information you provide, there is no search include in the other one.
Regarding the post above, I am completely right with you.
It is a CRITICAL issue but if you check at the bottom of the article, there is already a bug LOG-14814 opened by ArcSight
meaning that using operators as AND, OR and NOT give wrong results.
I think that HP ArcSight has hidden this issue.
I would like to have a status about this because v5 and v6 of Loggers are impacted.
I will try to make some tests in our production environment based on Loggers v5.4 to confirm the issue.
But it is difficult to know the correct number of EPS in a large environment as the one I monitor.
I hope to receive soon a clear explanation from HP ArcSight Support when this will fixed and/or a workaround exists
to do not provide wrong information for management or CSIRC Teams (forensics)
We have a workshop with HP ArcSight end of august, I will ask them a status about this HIGHLY CRITICAL issue.
Thanks Michael, fixed the typo
This is actually a separate issue and has its own bug ID (though I've just now realized I don't have the number of the most recent ID filed but the previous 2 related IDs are posted above).
The problem here is that if you pick a day like 7/15/2015 and run a search and get 5 hits, then pick a time range like 7/10-20/2015 you will get a completed query that says 0 hits. Obviously you should have a minimum of the 5 hits from the 15th.
I have replicated this on different physical loggers, different data, and different queries in every version of logger 6 (.0, .0.1, .0.2).
In a "normal" workflow you might never know this issued existed, just as with the other bugs posted.
For example, if you get an alert in ESM for an event that happened today, you might want to know if anything else from the same actor has been going on over the past week. So you go to logger and immediately begin a 7 day search using several key parameters from the original event. The logger returns a completed query showing no events. So you think great, no other potential events of concern in my environment and since you had no reason to think you would need to run this query against each day individually you have no indication that anything is wrong with the results.
It is only by complete accident that we discovered this ourselves and it calls into question every investigation we have done for the past year.
I posted this because like the other bugs the affect is the same: data that cannot be relied upon.
I was torn between posting this to Protect and throwing HP under the bus or keeping it quiet and giving them time to work on a fix but since the bugtraq info has been posted I guess I no longer have to worry about that