Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Established Member.. sbotharaj
Established Member..
781 views

Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hello there,

So I am now asked to introduce a logger appliance in a proposal where I have Express as the manager.  I have 2 agent servers reporting from tier 2 data centers to the main data center which has one agent server (to collect logs inside the DC itself) & one express manager to receive logs from the 3 agents, correlate & alert.

Question is,

Option 1:

should I introduce the logger in-line in the architecture so that all agents send logs to the logger which will in turn pass the logs to the express?

(My observation: Con: Logger becomes single point of failure  Pro: Only one copy of logs sent from agents via WAN (so BW efficient).

OR

Option 2:

should I introduce the logger in parallel with the express so that all agents send two copies of logs, one to the logger and the other to the express?

(My observation: Pro: Redundancy to some extent even if one of the components among Express and Logger fails. Con: Two copies of logs to be sent via WAN)

any other factors to be considered? Also if I go for Option 2, is it possible to apply two different filters in the same connector instance so that Logger receives all logs (for compliance requirement) while express receives only logs that are needed for correlation from InfoSec point of view?

Thank you.

Labels (2)
0 Likes
1 Solution

Accepted Solutions
Highlighted
balahasan.v1 Acclaimed Contributor.
Acclaimed Contributor.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hi Suresh,

We have deployed option one. Which is Bandwidth efficient. And we have automated the Failover for the Logger Forwarding Connector as well.

Keep in mind. Connector -> Logger -> Express works well for medium scale enterprises which are having low eps. In our case it's 1750. From 2 connectors. Since Logger has only one forwarding connector which has EPS limitations as well. Which in turn creates a lot of Cache and performance issues.

But we have automated the Failover of the Connectors from Express. When Primary Logger destination goes down. The Express rule will trigger the Event start command from Express.And hence there will no data loss.

Capture.JPG

Connector 01: Primary Destination -> Logger and Secondary Fail over Destination -> Logger 2/ESM. Register in ESM.

Capture.JPG

Rule 01:

For any Logger Forwarding Connector doesn't report/Shutdown event. (Optional - Add the entry to an Active List(Set AL entry expiry in 5 mins)

Only Sample Snap

Capture.JPG

Rule 02:

For any Logger Forwarding Connector report/Start up event. (Optional - Remove the entry from the Active List)

Only Sample Snap

Capture.JPG

Rule 03:

If the Logger Forwarding Connector doesn't report within 5 - 10 minutes(Upto you) the ActiveList entry will expire.So create an alert condition for that. Or Immediately as well if required.


Capture.JPG

Command Action: Event Flow Stop/Start

Capture.JPG Capture.JPG

0 Likes
17 Replies
stefan.oancea Outstanding Contributor.
Outstanding Contributor.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hello Suresh,

Have you taken into consideration Option number 3 - having the agents send events to the Express (as you have it now) and using a Forwarding Connector in order to forward everything you consider relevant from Express to the Logger? This is actually the situation which I come across most of the times and I find it suites best most of the environments. It actually resembles your Option 1, but with the Express and the Logger switched over.

Primary benefits of this approach:

1. No duplicate events within your network (so you conserve bandwidth)

2. You have the choice of sending everything to the Logger (including internal Express events, resulting correlation events as well as the basic primary events) or you can filter out exactly what you don't want to forward to the Logger

3. The changes that you have to make to your current infrastructure are minimal - just install the Forwarding Connector on the Express Appliance and you are good to go

You can find documentation for the Forwarding Connector in the Express 4.0 with CORR-Engine section here:

However, I would go for a newer release of the Forwarding Connector since I am almost sure it will work on Express as well.

You can download the Forwarding Connector installer from HP ESP Partner Portal; however you will not find it under the SmartConnectors section but under ESM/Express sections - it is a different installer than the general one for SmartConnectors.

Sorry for not answering your two Options, but if there is not anything preventing you from using scenario number 3 I can't see a reason why not. Hopefully I was not off-topic.

All the best,

Stefan

0 Likes
Established Member.. sbotharaj
Established Member..

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Thank you for the quick detailed response, Stefan.  Personally, I would consider option 3 (I did propose it) but it was not given much attention by our engineering panel.  I can try pushing this option with the helpful benefits you have quoted to the new deployments if not for the current deployment where I am restricted to go from among Option 1 & 2.

Quickly on Option 3, we would again end up losing logs if Express goes down yeah? or do we have fail open option?

Of course, your discussion is helpful and is aligned with the topic.  Keep responding & help people like us to improve.

Thank you.

0 Likes
pratikp Absent Member.
Absent Member.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hi Suresh,

Personally, I feel its always better to go for option 2 as you maintain redundancy which is always audit requirement. As all infrastructure logs are getting stored and maintained in SIEM solution, you can not keep risk open for single point of failure.

Also, you can forward different types of logs to 2 separate destinations ( logger and express) by applying necessary filters.

You can forward all logs including raw events to logger and whereas to express you can forward only aggregated and filtered events without RAW events.

I hope this helps.

Regards,

Pratik

0 Likes
stefan.oancea Outstanding Contributor.
Outstanding Contributor.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hello Suresh,

I am glad I can help.

Regarding failover and continuity mechanisms, I would bring into discussion the following:

1. SmartConnectors have the possibility to locally cache events for up to 50 GB when they see that the destination (Express or Logger in your case) is down: , page 58. Therefore, for a decent amount of time you can be sure no information will be lost until you get a destination back up.

2. SmartConnectors offer the possibility to have multiple simultaneous destinations configured as failover destinations: , page 73. This way you will not have duplicate events within your network, but you would have a failover mechanism in place.

Personally I have not implemented or tested the failover destinations mechanism, but I know it is available. From my point of view implementing the two options above is enough to ensure you don't lose information.

All the best,

Stefan

0 Likes
Established Member.. sbotharaj
Established Member..

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Thank you for the inputs.

Also, you can forward different types of logs to 2 separate destinations ( logger and express) by applying necessary filters.

I have not tried earlier.  So I install a single smart connector instance with two destinations Express and Logger, which creates two connector resources, one in Express and the other in Logger by which I can apply two different filters (one each in Express and Logger) for the same connector to send different set of logs?

0 Likes
pratikp Absent Member.
Absent Member.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hi Suresh,

You can apply filters on Express in below 2 ways

1. Through GUI console

Filter Out on console.png

2. By using runagentsetup on smartconnector configuration wizard.

For logger, you can apply filter configuration using runagentsetup on smartconnector configuration wizard.

Regards,

Pratik

0 Likes
Highlighted
balahasan.v1 Acclaimed Contributor.
Acclaimed Contributor.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hi Suresh,

We have deployed option one. Which is Bandwidth efficient. And we have automated the Failover for the Logger Forwarding Connector as well.

Keep in mind. Connector -> Logger -> Express works well for medium scale enterprises which are having low eps. In our case it's 1750. From 2 connectors. Since Logger has only one forwarding connector which has EPS limitations as well. Which in turn creates a lot of Cache and performance issues.

But we have automated the Failover of the Connectors from Express. When Primary Logger destination goes down. The Express rule will trigger the Event start command from Express.And hence there will no data loss.

Capture.JPG

Connector 01: Primary Destination -> Logger and Secondary Fail over Destination -> Logger 2/ESM. Register in ESM.

Capture.JPG

Rule 01:

For any Logger Forwarding Connector doesn't report/Shutdown event. (Optional - Add the entry to an Active List(Set AL entry expiry in 5 mins)

Only Sample Snap

Capture.JPG

Rule 02:

For any Logger Forwarding Connector report/Start up event. (Optional - Remove the entry from the Active List)

Only Sample Snap

Capture.JPG

Rule 03:

If the Logger Forwarding Connector doesn't report within 5 - 10 minutes(Upto you) the ActiveList entry will expire.So create an alert condition for that. Or Immediately as well if required.


Capture.JPG

Command Action: Event Flow Stop/Start

Capture.JPG Capture.JPG

0 Likes
Established Member.. sbotharaj
Established Member..

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Caching is part of our standard operating procedure.  We are doing it already.

Failover Destination - is a pragmatic approach.  I have forwarded my recommendation to the panel and waiting for their decision.

Thank you much, Stefan!

0 Likes
Established Member.. sbotharaj
Established Member..

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Not asking about the way to apply filter.  My question is can you apply two different filters from two managers on a single connector? It should not be possible I guess based on my understanding of ArcSight.

May be the following experts can clarify:

Cheers!

0 Likes
rkent1 Acclaimed Contributor.
Acclaimed Contributor.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hi Suresh,

If I go back up to the message immediately before Pratik's:

Suresh Prabhu B wrote:


So I install a single smart connector instance with two destinations Express and Logger, which creates two connector resources, one in Express and the other in Logger by which I can apply two different filters (one each in Express and Logger) for the same connector to send different set of logs?

Now, in this circumstance, you have one connector with two destinations: one to Express and one to Logger.

Connector:

     [Destination][Filter] -> ESM

     [Destination][Filter] -> Logger

What Pratik tried to outline to you is correct - you can set the filters separately, and you can do it via two ways:

Pratik Patil wrote:

1. Through GUI console

Filter Out on console.png

2. By using runagentsetup on smartconnector configuration wizard.

To add just a bit to Pratik's response: You could use option #1 above to push the filter for the destination from Connector -> ESM, or you could use option #2 on the connector itself to set the filter for *either/both* of the destinations.

One thing that I think might be the root of the confusion is when you ask "can you apply two different filters from two managers on a single connector?". Are you considering the Logger to be a manager of the connector?

0 Likes
pratikp Absent Member.
Absent Member.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Hi Suresh,

I hope your doubts solved now after Richard's and Bala's updates.

Thanks for adding your thoughts for better understanding

Regards,

Pratik

0 Likes
Established Member.. sbotharaj
Established Member..

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

Apologies for the slip.  I have never used "Modify Destination Settings" option in a connector agent set up .  Thanks a lot that now I understand that same instance can send different set of logs to different destinations using different filters (I like twisting )

Yes, I used manager as a loose term to indicate Logger .

0 Likes
Established Member.. sbotharaj
Established Member..

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

You have given a detailed and thorough solution to my situation.  Hats off to you! After reading your response I feel I have miles to go before sleep!

I would definitely use your suggestion for subsequent assignments.

Is there a possibility of tagging multiple "Correct Answers"?  In this case, I am tagging response as its the best answer but all the others have given correct answers and directions for me. I am grateful to all of them.

0 Likes
grace.chang Absent Member.
Absent Member.

Re: Express/Logger Deployment Scenario - Your Experience?

Jump to solution

That's great news, ! So glad to hear the community is giving you quality answers. Currently, there's only the option to mark one answer as "correct", but you can mark the rest as helpful. Great work, , , , !

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.