Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Regular Contributor.. digger5712 Regular Contributor..
Regular Contributor..
650 views

Logger no longer indexing incoming events

Hi,

Over the past couple of months on our 6.0 P1 Loggers, it's being noticed that every 1-2 weeks or so, the indexing of events will completely stop.  This is evidenced by the Summary page not updating with new data, yet on the Dashboard page you can confirm that events are still coming in unabated.  This has happened on two separate Loggers at different times, and the only thing that fixes it is a full reboot of this hardware appliance (they are L7400-SAN Loggers we bought a few years ago).  Invariably though, the problem will come back at seemingly random times for no obvious reason.

Obviously, this causes havoc with our scheduled and ad hoc searches that we perform on data newer than when the events were last indexed per the Summary page, especially if queries are run against Receivers...on queries run against Receivers, I'm finding that when using the non-indexed data the queries will return zero results, even when I know for certain events should be there.  In contrast, if I run essentially the exact same query but modified so that I query against a Device Group instead of the Receiver with the same name, then the query, though still running much slower than it should if it had been indexed, will give back results that I expect.

Looking through 7/24 I've found a couple others who have had a similar thing happen, and as far as I could see no resolution was ever found.  I have opened a support case w/ ArcSight Support and given them a couple of log dumps, but a week later I've still not had further direction from them even after repeated requests for an update.  All I've received so far from them is a generic "we are still looking at your logs".

In the meantime while I hope to find a root cause, my question is this:  In lieu of doing reboots every time this happens, is there a particular process or processes as listed in the "System Admin > Process Status" menu of Logger that I can  initiate a restart on that will kick-start the indexing in the same manner that a full reboot does?  I've already tried restarting the following services, with no luck in fixing the indexing issue:

apache

aps

connector

insp

web

The remaining processes I haven't tried, that I'm a bit nervous about restarting as I don't want to break anything worse than it already is, at least until I can confirm that restarting them can be done safely:

mysqld

postgresql

processors

receivers

reportengine

servers

snmp

Any assistance I can receive in this matter would be much appreciated, believe me.

Regards,

Doug Gillespie

Labels (2)
0 Likes
Reply
16 Replies
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

I am having this same problem, only fixed by a reboot. This is no joke when you have hundreds of scheduled reports and event forwarding configured.

Support also seems stuck with this.

0 Likes
Reply
Regular Contributor.. digger5712 Regular Contributor..
Regular Contributor..

Re: Logger no longer indexing incoming events

OK, minor update.

While I'm still no closer to finding out the root cause (and STILL haven't heard back from ArcSight support on this), I at least have found the process I can restart that appears to get the indexing working again, at least until it inevitably breaks again.

By restarting the 'servers' process in Logger, it also forces restarts of the following processes as well:

mysqld

processors

receivers

reportengine

web

After about 4 minutes these processes are back up again, and when I look back on the Summary page I can see that the indexing is now unstuck and is going through the process of trying to catch up with the backlog of non-indexed events that have piled up.  Currently it's indexing about 3 minutes of data for every minute of event collection, so it's going to take a couple of days for it to fully catch up.  It's not much better than a reboot, but at least it gets the indexing going without spending 10 minutes doing a full reboot, along with the paperwork and emergency change submissions that I would normally have to submit, and it's less impactful to users this way as well.

I'm not sure if it's just one of those child processes that needs restarting for this to work, it's possible restarting 'servers' is more heavyhanded than need be, but for now I'll consider this the safer way to go until I find out differently.

In the meantime, we'll see what ArcSight Support has to say.

0 Likes
Reply
Regular Contributor.. digger5712 Regular Contributor..
Regular Contributor..

Re: Logger no longer indexing incoming events

OK, it's been a few weeks so I figured I may as well give an update for anyone that is curious.

After sending multiple log dumps to ArcSight Support from my Logger and a lot of back and forth with them, they couldn't really find anything in the logs that gave a definitive answer as to why the indexing would silently stop working.  The only evidence they could find was that when the failure would happen, the Logger was in the midst of running some heavy scheduled reports.  Their suggestion was to go over their Best Practices to confirm that I was not running too many Receivers (we aren't, at least according to their recommended specs), that some of our queries being run with some of our reports were not as well optimized as they could be, and that there were some parts of our day where the Logger would be very busy running multiple large queries that could *maybe* be causing issues.  To that end, we went through an exercise where we killed some superfluous reports, and spaced some of the heavier reports out so they weren't as many running simultaneously.

The end result is that although there were some initial hiccups where the indexing service would stop, as of today on one Logger it's gone 3 full weeks without an indexing failure, and the other Logger it's gone 11 days and counting so far, whereas before it was failing every 1-3 days.  SO, it appears to be working better...though I'm not 100% certain if it was simply load balancing that improved matters, or the killing of a particularly bothersome report that helped.  At this point I'm going to give it one more week of monitoring to be on the safe side, and if all good then I'll close my ticket w/ ArcSight support.

0 Likes
Reply
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

Thanks for the follow up Doug. Its still happening to me. Im still working with support figuring it out.

They said forwarding connector is struggling with too many DNS resolutions, I will be working on that. But I will also check what you just said about reports.

0 Likes
Reply
psandeep08 Absent Member.
Absent Member.

Re: Logger no longer indexing incoming events

Hi Rodrigo, Any updates? We are facing same issue, we did all the method. Logger version appliance 7400,

1. restarted the logger

2.Database defragmented

3. deleted and reconnected logger destination

0 Likes
Reply
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

Hey Sandeep, which Logger version are you running ? As in 6.2 ?

There was a huge bug in 6.1 and earlier that was causing these kind of problems.

0 Likes
Reply
psandeep08 Absent Member.
Absent Member.

Re: Logger no longer indexing incoming events

Version 6.1

OS version:- 5.5 Tikana

0 Likes
Reply
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

You cant upgrade to 6.2 ?

The problem I had was that a partition filled up with old published reports that weren't deleted automatically because of the bug. And that caused all kind of havoc on Logger, mainly non indexing so no forwarding.

0 Likes
Reply
psandeep08 Absent Member.
Absent Member.

Re: Logger no longer indexing incoming events

Hi Rodrigo, Thank you for the hint, Logger 6.2 is NOT available in L7400 series as per release notes, only available in L7600,7500 & 3500.

Any hints on old report location?

0 Likes
Reply
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

Quick way to check this is just do a "df -h" on the linux interface.

Is any of your partitions full ?

0 Likes
Reply
Highlighted
psandeep08 Absent Member.
Absent Member.

Re: Logger no longer indexing incoming events

only 4 partitions .all are just below 60%

0 Likes
Reply
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

Hmm doesnt seem to be related then.

/opt/arcsight/userdata was the partition that was filled up for me, I had to manually delete some files for the indexing and forwarding to work.

The other times it was happening to me was solved by restarting Logger, defrag database or restarting the server process.

Have you tried support ?

0 Likes
Reply
psandeep08 Absent Member.
Absent Member.

Re: Logger no longer indexing incoming events

Support is asking me to delete all the unified logger forwarders and create regex forwarders. (I mean from logger to ESM use regex forwarder)

I couldn't understand how come forwarders having impact in indexing? After indexing the forwaders based on the condition will send to ESM?

Please correct if my understanding is wrong?

0 Likes
Reply
Super Contributor.. rcastellanos1 Super Contributor..
Super Contributor..

Re: Logger no longer indexing incoming events

Doesn't make sense to me either. I understand it works the way you just specified.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.