Highlighted
Super Contributor.. tony.zhang Super Contributor..
Super Contributor..
1350 views

Question about e-mail Notification stops sending after a while

Jump to solution

Basically, our email notifications would work for a few hours after a restart:

>>arcsight_services stop

>>arcsight_services start

when the restart is done, during the first few hours, whenever the rule is triggered, we would see these 3 events, and we do get notification emails as expected:

(index - Device Event Class Id - Audit Event Description - Name)

1 -  rule:101 - Rule fired OnEveryEvent. - Failed Login

2 -  rule:303 - Send to Notifier action - SendToNotifier: Success

3 -  notification:111 - Notification sent requires acknowledgement. - Notification sent (Acknowledgment required)

However, after a few hours, when a rule is triggered, we would only see 1 & 2; and we'd stop getting notification emails as well until we manually restart it again.

We've tried looking into patterns such as how long it would work after restart, or if it reached the max number of notifications limit, but couldn't find anything interesting; modified these settings in server.properties, still could not solve the problem. (sometimes it just sent 5-6 emails and then stopped working, plus we never got any notification:101-disabled due to queue too large)

All the daily scheduled reports are working properly, and the Test Destination emails also gets received.

so how can we look into this "Notifier" for why it decides to stop working? is there anything else we can try? Any help would be appreciated!

Tags (1)
0 Likes
1 Solution

Accepted Solutions
Super Contributor.. tony.zhang Super Contributor..
Super Contributor..

Re: Question about e-mail Notification stops sending after a while

Jump to solution

Just for others who may be encountering notification issues:

the notifications seem to be working fine after upgrading esm to 6.8 and uninstalling ASM package.

(These 2 were done together so not sure which one fixed the problem; both are possible)

No configuration/rule changes or any other changes made.

Havnt performed stress test to see if it permanently breaks again after hitting the daily limit; hope it doesn't.

View solution in original post

0 Likes
5 Replies
Super Contributor.. tony.zhang Super Contributor..
Super Contributor..

Re: Why is the sending notification action interrupted between "SentToNotifier" and "Notification sent"?

Jump to solution

According to https://protect724.hp.com/message/34562#34562,

"...You should try clearing your rule cache.  I know that it might be too little, too late, but rule cache hangs up our notifications sometimes. ..."

This is the one thing that we think we might want to try but haven't yet; according to another post https://protect724.hp.com/message/18661#18661, we are to delete everything in one of the folders (which contains a lot of java class files) we are new to arcsight thus very concerned about doing so.
Has anyone actually tried this? Does it solve the notification problem, and is it safe to remove this folder? Where can I find some introduction on rule cache management?
0 Likes
Honored Contributor.. wsladek1 Honored Contributor..
Honored Contributor..

Re: Question about e-mail Notification stops sending after a while

Jump to solution

We do it anytime there is a major issue... except rather than deleting anything we just rename the rules folder and recreate the subdirs.  The ESM will take care of the rest:

stop the manager

cd $MANAGER_HOME/

mv rules rules.`date +%m%d%y`

mkdir -p rules/{checkpoint,classes,Temp}

start the manager

0 Likes
Super Contributor.. tony.zhang Super Contributor..
Super Contributor..

Re: Why is the sending notification action interrupted between "SentToNotifier" and "Notification sent"?

Jump to solution

Hi wsladek,

Thank you for confirming;

we tried to do this last night but ran into problem;

so what we did was:

1. >> /etc/init.d/./arcsight_services stop

2. backup and remove everything under

<arcsight_home>\rules\classes

<arcsight_home>\rules\checkpoints

(we didn't touch the content within Temp folder because its not mentioned in the rule clearing procedures)

3. >> /etc/init.d/./arcsight_services start

and then after 3, the command immediately finished (usually it takes a while) and we were not able to connect from the console.

we looked through the logs and were not able to find anything causing this ( the arcsight_services status shows that manager is "initializing" and web service have "mixed state", all the rest are fine)

the arcsight_services start/stop/restart doesn't work, so we had to kill all the processes under account arcsight, moved all the backup rule folder contents back and restarted the services....

when we grepped for "rule engine" it did return something like "trying to rebuild rule folders" etc...

So the question would be which steps went wrong? are we expected to wait for a longer time for it to rebuild? what should we expect in the log files (like when the server finishes building rules)? do we lose all of the custom rules we made if we delete those 2 folders? is there a documentation I can read that's related to the functionality of each of these folders and procedures for clearing cache and stuff?

Any help would be really appreciated!@wsladek

0 Likes
Honored Contributor.. wsladek1 Honored Contributor..
Honored Contributor..

Re: Question about e-mail Notification stops sending after a while

Jump to solution

I am not sure - I have never done it that way before.  As I mentioned previously, we just rename the whole rules folder then recreate the subdirs.

0 Likes
Super Contributor.. tony.zhang Super Contributor..
Super Contributor..

Re: Question about e-mail Notification stops sending after a while

Jump to solution

Just for others who may be encountering notification issues:

the notifications seem to be working fine after upgrading esm to 6.8 and uninstalling ASM package.

(These 2 were done together so not sure which one fixed the problem; both are possible)

No configuration/rule changes or any other changes made.

Havnt performed stress test to see if it permanently breaks again after hitting the daily limit; hope it doesn't.

View solution in original post

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.