vsankar Absent Member.
Absent Member.
1986 views

Sync content between two ESM

Jump to solution

Hello,

I am looking for a solution to keep the content (not data) of two ArcSight systems(ESM) in sync (at least once a day). We have two systems - DR and Prod (both running 5.0Patch1) and the contents of the Prod needs to be in sync with DR, so that users can switch to the DRsystem in the event of failure on Prod. In the above architecture, we also have DR Logger feeding DR ESM and Prod Logger feeding Prod ESM, with the bunch of CA feeding events to both the Loggers
I was initially thinking of a daily export of system tables from Prod and import into the DR system. But this might not work as the system table dump also copies over the connector and I would loose the DR Logger-ESM connector in this approach.
The only other option I am left with is using packages or the archive command to maintain the sync, but my experience has been that both archive and packages are not as reliable as a system table import has been.
Any other ideas?
Thanks,
V
Labels (3)
0 Likes
Reply
1 Solution

Accepted Solutions
Highlighted
jbur Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

CSN27: Automated ArcSight ESM Content Replication

https://protect724.arcsight.com/docs/DOC-1476

-Joe

0 Likes
Reply
16 Replies
jbur Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

I think there was a user conference presentation on this topic.  If I find it, I'll let you know.

-Joe

0 Likes
Reply
vsankar Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

Hi Joe,

Thanks for the reply.

I was only able find some HA availability docs, which talks about using EMC, etc. Though that would be the ideally way to go, as a start, we are looking only to replicating content - users, active channels, assets, rules, reports, queries, etc. Since we have two dedicated Loggers in each environment,  we are assuming that the data is already replicated and will be available on the DR ESM.

Please do send me anything you find.

-Vinod

0 Likes
Reply
Absent Member.. Dean Farrington Absent Member..
Absent Member..

Re: Sync content between two ESM

Jump to solution

Aaron Wilson from SAIC did a presentation on a mechanism they developed at the Users Conference. Not sure if those presentations can be reposted since ArcSight only made them available to those who attended so far.

0 Likes
Reply
vsankar Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

Thanks! Do you know when he made the presentation? If so, the presentation name?  I have access to Protect 10 docs. 

0 Likes
Reply
Highlighted
jbur Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

CSN27: Automated ArcSight ESM Content Replication

https://protect724.arcsight.com/docs/DOC-1476

-Joe

0 Likes
Reply
vsankar Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

Thanks a lot Joe! This is very helpful. I will go through the presentation to understand how it is being done.

-V

0 Likes
Reply
dstrevinas Trusted Contributor.
Trusted Contributor.

Re: Sync content between two ESM

Jump to solution

Hello,

Is anyone kind enough to provide the presentation as the link above does not wrong while I do not seems to find it anywhere!!

Many regards!

Jim

0 Likes
Reply
Absent Member.. Dean Farrington Absent Member..
Absent Member..

Re: Sync content between two ESM

Jump to solution

The link works for me, but I think they only released the materials to those who attended Protect10 at the moment. I'm not sure the timing of the general release.

0 Likes
Reply
tliu Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

This is correct, the Protect '10 Session Materials are currently accessible by conference attendees only. Access is opened to the rest of the community about six months after the conference.

Best regards,

Trisha

0 Likes
Reply
dstrevinas Trusted Contributor.
Trusted Contributor.

Re: Sync content between two ESM

Jump to solution

Dear all

We are investigating different methods on performing the replication:

a) SAN Replication - Everything is in a SAN even the executables (very costy solution) - Events are replicated too

b) Replication using software like DRBD - Not tested, who is going to do ESM-Logger production with this one? - Events are replicated too

c) Content replication using packages (Protect '10 slides) - Propagate events from Main to DR using Connector.

d) System table export-import - Propagate events from Main to DR using Connector.

Regarding the System Table Export solution Vsankar pointed out that you lose the connectors. Will this idea be of any use?:

   a) Using a script, export a connector package from the DR site

   b) Export sys tables from the Main site

   c) Import sys tables to the DR site

   d) Import the previously exported Connector Package

If this helps, can we continue to my question? Does the system table export provide the following?:

a) Archived reports (low importancy - can be exported manually)

b) Active list data  (high importancy)

c) Session list data (high importancy)

d) Trend Data (high importancy)

e) Events contained in cases (medium importancy)

I am looking forward to hearing additional thoughs.

Regards,

Jim

0 Likes
Reply
Vini Acclaimed Contributor.
Acclaimed Contributor.

Re: Sync content between two ESM

Jump to solution

If you virtualise your environment you could use VMware Falt Tolerance to have an online replica of your Manager, you could also do this with the DB but it might impact performance.

One seemless option would be to do the replication of the DB using Oracle itself but that won't be cheap.

0 Likes
Reply
vsankar Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

Hi Jim,

Here are my observations when I tested the replication using system table export/import between the two ESM servers.

a) Archived reports - not done by the system table export, but you can just copy over the archived reports folder from the source server to the destination server.

b) Active list data  - yes, system table export/import gets you the active list data from the source.

c) Session list data - yes, system table export/import gets you the active list data from the source.

d) Trend Data - no, trend data is not imported.

e) Events contained in cases - not sure

In our approach, we have the Connector Appliance feeding events to both servers thereby ensuring event replication. Content replication is done by system table export/import.

Thanks,
Vinod

0 Likes
Reply
dstrevinas Trusted Contributor.
Trusted Contributor.

Re: Sync content between two ESM

Jump to solution

Hi Vinod,

Nice, we are also considering the dual feed as the best option so far. SAN replication may propagate a malicious deletion of a file to the remote SAN too, on the other hand the dual feed acts as a backup too.

My observation here is that in dual feeds one should not use content import/export. This is because, for example, an active list will be populated, on the fly, due to the correlation done on the dual-fed events. Thus, theoritically, trends/list/sessions lists should not be replicated in a scheduled fashion in dual fed ESMs.

The case events in ESM 5 have their own event table. Thus they are not subject to retention period. I still have to test whether this table can be exported and imported to the DR site without problems.

Finally, as soon as I complete my content replication scenario I will try to share some thoughts/results/code here. If anyone has already performed a working content replication, please share some information here or at least a do/don't listing!

Kind regards!

Jim

0 Likes
Reply
eoyola1 Absent Member.
Absent Member.

Re: Sync content between two ESM

Jump to solution

Yeah, we have a PS guy that suggested using the package utility method and despite of my my best efforts, there are too many workarounds and gotchas and manual intervention required for my level of comfort.

And it only partly works and can't really be automated.  Basically you have to break up all your content into small packages under 30MB in size and this is sometimes hard to do and needs rework whenever your content grows beyond the limit in a single package.

The package utility is really buggy, prone to get stuck indefinitly, emits random DB errors that noone can explain, and sometimes causes cache/db discrepancy in the ESM, so we opted to run it with the ESM offline during the sync process.  And it's slow.  If you let it time itself out (instead of killing it before its timeout), bad things happen that sometimes require manual database repair because it doesn't roll back a transaction properly.

Also the package utility prompts you for a bunch of stuff so you need to anticipate all the possible prompts and script reasonable choices using an expect-like tool or the whole thing will freeze forever.

You have to make sure to use an ArcSight account with a non-expiring password because there's a bug where if the password is expired, the package utility goes into a tight loop emitting "expired password" messages eventually filling a disk partition if you're logging the package utility output.

There are many more gotchas and workarounds and still there's some manual fixing we have to do virtually every time it runs.


-- Eli

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.