Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Contributor.. security_suppor1 Contributor..
Contributor..
587 views

How to query log over 1000000 event?

I want to query and export log from Arcsight logger version 6.0.0.7307. But log over 1,0000,000 event.

Logger limit row of event is 1,000,000 right?

If yes, How to export this log?

Thank you,

Chonlapat.

Labels (2)
0 Likes
5 Replies
Micro Focus Expert
Micro Focus Expert

Re: How to query log over 1000000 event?

I think you can do this with Logger API and some scripting.

May I ask what is the use case?

What will you do with more than 1,000,000 events?

0 Likes

Re: How to query log over 1000000 event?

I have run into the 1,000,000 events limitation as well and would like to see an answer. Needed for external log analysis by a third party,  archival outside of the defined retention period, archival in a separate location due to an investigation, and eDiscovery request results are just a few use cases where I would want to do this.

Really the use case should not matter overly much and I am curious why you would ask as opposed to answering the question.

0 Likes
Outstanding Contributor.. douglas.baker@h1 Outstanding Contributor..
Outstanding Contributor..

Re: How to query log over 1000000 event?

If you are "...needed for external log analysis by a third party,  archival outside of the defined retention period, archival in a separate location due to an investigation" then all of them read to me as "trying to use interactive Anaylsis/Search" for something that isn't for a focused query and is thus the less than optimal (wrong) solution.

Look into a Forwarder for the above examples.

Keep on Searching interactively for more ah-doc and focused queries that 'should' return (much) less than 1M rows, i.e. that are interactively concise.

Really, the UseCase does matter in order to select the appropriate solution.

In addition to Interactive Search, noting of course that the Session has an administratively configured timeout that could be reached before the Search completed, and a Forwarder you could also selectively use the SOAP/REST interface an develop a variety of extraction options via external program control (see other blogs for sample code).

Match the solution to the UseCase.

The Logger gives you a variety of solution options, quite flexible really.

I'm w/ Aaron, as I explain above.

Doug

0 Likes
chris.allen3@hp1 Super Contributor.
Super Contributor.

Re: How to query log over 1000000 event?

I agree with Doug for being optimized.

I just wanted to add in as this is common request from the field.

If you truly need to export 1M+ events, use a Logger report...

the Linux utility "Zcat" has also been preferred by some running against the Logger archives.

If you want an event radar but the there is a large volume, use the chart with time span sub search operators.

example...

For a search against the last 7 days:

deviceProduct = ASA | chart sum(baseEventCount) span = 2h

Open the chart settings and increase the display limit to 84 (100 is max).

You can also save this search and set multiple saved search groups as a dashboard.

The "|top" and "|rare" sub searches can also provide useful indirect views into bulky data sets.

Cheers!

-Chris

0 Likes
Gayan Acclaimed Contributor.
Acclaimed Contributor.

Re: How to query log over 1000000 event?

Use Zcat utility. Its easy way to solve this issue.

Cheers

Gayan

Mr
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.