Super Contributor.. Super Contributor..
Super Contributor..
373 views

Apparent BUG with Logger Pipeline queries

EDIT: Ok, I'm going to remove the "MAJOR" adjective from my title..... you can read my full update which is the 4th post below... but bottom line is that there is a REAL bug, however my syntax use of "WHERE" is not supported the way I used it.  More in 4th post.

I open a lot of bug reports.   I don't share all of them here, but I do on ocassion.  Due to its nature (read on) I consider this a major potiental significant issue, so I am sharing this early.  In all fairness Arcsight has not yet had a chance to respond.   It is recorded as  Incident: 111010-000059

In essence, I am reporting that there appears to be a bug with using "WHERE" clauses in pipleine queries.   I don't know the exact scope, of the bug, and it's nature is luikely different or broader than as described here. That is why  I'm looking for others to attempt to replicate this issue in your own way.  I am using the Appliance Logger, version 5.1 with all available patches as of today.

It will be easier to visualize this bug by looking at screenshots.

_deviceGroup IN ["dv_SDE_DC"]  | cef externalId deviceCustomString2 name | chart count by externalId deviceCustomString2 name | sort - _count

Cap1.Jpg

Those results seem reasonable, but becuase I was getting results where the contents of externalID was NULL (especuially further down the chart), I wanted to eliminate those.   SO, I modified my query to this:

_deviceGroup IN ["dv_SDE_DC"] where externalId IS NOT NULL | cef externalId deviceCustomString2 name | chart count by externalId deviceCustomString2 name | sort - _count

But yikes --- look at the results -- everything except the first result is gone

Cap2.Jpg

So, for the heck of it, I try with a different where clause:   Where eventID > 0.    Same ugly results:

_deviceGroup IN ["dv_SDE_DC"] where externalId > 0 | cef externalId deviceCustomString2 name | chart count by externalId deviceCustomString2 name | sort - _count

Same bad results:

Cap3.Jpg

SO, for reference, I decided to create a report query:

Cap3a.jpg

cap4.JPG

cap5.JPG

which give the same reults as the original query.

Again, note that all these queries are using a fixed time period that is old enough not to be affected by current incoming events.   They also did not "age out" during this period.

I'd urge you to try a similar style query, and see if you can observe similar behaviors.

Labels (1)
Tags (4)
0 Likes
Reply
4 Replies
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Apparent MAJOR BUG with Logger Pipeline queries

What happens if you re-order your search clauses like this:

_deviceGroup IN ["dv_SDE_DC"] | cef externalId deviceCustomString2 name | where externalId IS NOT NULL | chart count by externalId deviceCustomString2 name | sort - _count

0 Likes
Reply
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Apparent MAJOR BUG with Logger Pipeline queries

Great question..... it does not limit the result to a single entry,, HOWEVER, after 5 minutes, the query does not complete  and has only completed a few hundred rows.   In fact, it is well known to arcight that a WHERE clause  after a pipeline has extereme performance problems.

(To find this, search for "SEARCH OPERATORS", and scroll to the end oth the list of operators) (or check page 64 of the 5.1 Administrators Guide)

Usage: | where <expression>
<expression> can be any valid field-based query  expression, as described in Indexed  and Non-Indexed Fields.
A cef or  rex operator (to extract fields from matching events)  must precede this operator, as shown in the examples below.
<expression> can only be a valid field-based query  expression. Arithmetic expressions or functions are not  supported.
When the where operator is included in a query, the query performance  can be significantly impacted. This is a known issue and will be addressed in a  future release of Logger.
2. _ storageGroup IN ["Default Storage  Group"] | cef eventId deviceVersion | where  eventId=10006093313 OR deviceVersion CONTAINS "4.0.6.4924.1"
3. _storageGroup IN ["Default Storage  Group"] | cef deviceEventCategory eventId | rex  "deviceEventCategory=(?<categories>[^ ]*)" | where  eventId >=10005985569 OR  categories="/Agent/Started"
0 Likes
Reply
Highlighted
Super Contributor.. Super Contributor..
Super Contributor..

Re: Apparent BUG with Logger Pipeline queries

OK, so I realized I was using the WHERE syntax incorrectly.   While I will take some blame, Arcsight's bizzare mish-mash of semi-quasi-SQL and other pipeline selectors make it very difficult to compose anything beyond the most basic query.     And the fact that they do not filter or flag  the invalid syntax, but instead give a response does not help.

In any case, my bad syntax in its entirety was:

_deviceGroup  IN ["dv_SDE_DC"] WHERE externalId is NOT NULL| cef externalId   deviceCustomString2 name | chart count by externalId deviceCustomString2   name | sort - _count

(logger accepts, but gives invalid results that are difficult to detect if you don;t know your likely answer in advance)

What I should have used was:

_deviceGroup  IN ["dv_SDE_DC"]  AND externalId is NOT NULL| cef externalId    deviceCustomString2 name | chart count by externalId  deviceCustomString2   name | sort - _count

Basically, the WHERE clause should not be used in the inital statement before the pipeline.

What makes this so confusing is that this rule does not appear to be universal, it works sometimes, fails other times.   For example: these statements appear to work:

_deviceGroup IN ["dv_SDE_DC"] WHERE deviceVersion is NULL

_deviceGroup IN ["dv_SDE_DC"] WHERE deviceVersion is NOT NULL

In any case, I don't undertand all the ins and outs, but basically -- don't try to use WHERE clauses in Logger 5.1.  (Eveven after the pipeline it does not work well --- even the exisitng docs acknowledge this (see my preceeding post).

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Re: Apparent BUG with Logger Pipeline queries

First of all, the following query:

_deviceGroup  IN ["dv_SDE_DC"] WHERE externalId is NOT NULL

does not use the WHERE search operator. Since there's no pipe ('|') before WHERE, this ends up being a keyword search which looks for all the tokens in the query, namely, "where", "externalId", "is", "not", "null".

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.