Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..
611 views

How to performe Arcsigth New deployement sizing

Dear Community

We want to do  an Arcsight Assessment /Sizing in order to identify all the requirements for the deployment :

Connectors capacity, ESM License, system requirements, EPS, Loggers, Event Brokers...

Let you know that we've 4 distributed big branches (connected via WAN Links) and the 5th one is the headquarter where the ESM will be installed, 

The collection will be from all the remote branches  containing many Assets (windows, Linux, network devices, security devices netflows...) and forwarded to the headquarter central site where the ESM is located

We are waiting for experts Guidance, this is a sizing from scratch to do a new deployment ( and correct and old deployment has been done with many mistakes )

Thanks in advanced

BR

0 Likes
18 Replies
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: How to performe Arcsigth New deployement sizing

Hello,

 

please contact your sales and presales representative to help you to scale your solution.

Unfortunate the documentation it's not enough.

 

Best Regards,

 

Daniel

Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Re: How to performe Arcsigth New deployement sizing

Hello,

 

Let you know that we've already do that, the results weren't good, also there is no enough guidance from the support, 

We prefer do you it ourselves exchanging with community experts, otherwise, the solution will be changed to another SIEM 

So we are doing our best to review correct and optimize the implementation, that's why we are asking to get some good feedbacks from Community members experience

 

BR

 

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: How to performe Arcsigth New deployement sizing

Hello, 

 

from your answer i understand the following "there is no enough guidance from the support". Sorry to inform you but the support is not responsible to help you how to perform ArcSigth New deployment sizing. The only thing that can do is to recommend you to follow the sizing from the official documentation. 

To be helped please contact your sales and pre-sales and discuss this with them not with the support and for sure they are glad to help. 

 

Best Regards, 

 

Daniel

 

Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: How to performe Arcsigth New deployement sizing

Hi Hemza,

 

I have done 2 Sizing for ESM, Loggers and SmartConnectors for a very large company.
If you want some help I can provide some info.

What is important to know is what is approximately

the number of logs you will collect and the retention.
the number of devices and the types, to estimate the EPS because EPS is extremely important.
the number of Use Cases/Dashboard and Analysts that will use ESM

With this we can already provide a good estimation.

Now if you want to do an infra with ESM v 7.X with multiple correlation engine, you have to contact the sales because it is very new and there is very few return on experience.
I don't know what is the difference between ESM v6;11 alone and ESM v7.x in distributed mode.

I hope this will help you.

For your info, we have build a very powerful ESM and we are very satisfied to have taken this choice because our ESM is working perfectly now for 60000 EPS collected at the connector level, we have of course filtered logs for ESM (max 15 000 EPS).

Thanks
Kind Regards

Michael

Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Re: How to performe Arcsigth New deployement sizing

Hi @mschleich Michael

Thank you very much for your reply  that contain many important keys to perform this task, 

At the moment we've Smart Connectors ArcMcs, loggers, ESM are installed and receiving logs

We have an estimation of the EPS received but we are facing big issues with connectors capacity and performance (Just an example)

the connector appliance can't handle and process all the Syslog events (huge traffic),  

that's why we need to review our sizing from scratch to put an optimized design, for example, to install a separate Syslog connector with EB Cluster

To do this task I've posted this thread hoping to receive some guidance in dealing with this kind of situations

Thanks

Hemza

 

 

 

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: How to performe Arcsigth New deployement sizing

Hi Hemza,

Believe me the connectors should not be your concerns points.

Why? Because now , since more than 2 years, ArcSight has provided solution to manage this problem.
Secondly, if you can as much as you can receive the logs in CEF, the EPS can be relatively high.
Normally, an ArcSight SmartConnector can relatively easily reach 1000 EPS and 2000 EPS for Syslog.

It is only when you use complex parsing regex or mapFiles that this number decrease and it is at that time that you use the great solution of ArcSight Connector Loadbalancing which is currently only available for Syslog or FileReader. I have only used for Syslog and it works perfectly, we have configured 5 Syslog SmartConnector to retrieve Unix OS logs of 10000 Servers which means more or less 2000 EPS per connector = 10000 EPS in Total.

We have used 9 CEF Syslog SmartConnector for Proxy logs, why 9 because of the mapFiles which is extremely complex as it extract all domain and subdomain based on the TLD thus you cannot reach the 1000 EPS, it is decrease to 400-500 EPS.

The problem for sure won't be your connectors, if you can use a VMware Infra, it is the best as you can increase the CPU, RAM as needed. Connectors will work, I told you we have configured 108 SmartConnectors that collect for 60000-70000 EPS max.
It is important to build your infra based on the max EPS. Why? To be sure that no connector will cache as this as a huge impact on the ESM for the correlation rules.

You have to size ESM properly, it depends of your objectives and the infra you want to build but a big ESM Servers will be always good.

I have more than 200 Rules, 250 AL, 150 Dashboard and few scheduled reports, our ESM is working at 10-20% 0 cache at all. No crash, work perfectly.

We have made a very good fine-tuning but very important, we have sized the connectors perfectly and we have properly configured them.

Loggers is the critical point for me. It is not as good as expected. If you can choose Splunk go for it. But it will be nice to keep ESM. I don't know when Micro Focus has decided to review their Loggers but if it is too far, choose another solution. If you have not a huge amount of logs, it could be acceptable. We have installed 23 Loggers in 1 pool and 3 Search Head but it is still slow and you cannot make any correlation wit Loggers thus this is extremely impacting.

I hope this info will help you to start the sizing.
If you need more precise info or if you have any question, do not hesitate to contact me.

Thanks
Regards

Michael

Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Re: How to performe Arcsigth New deployement sizing

Hi @mschleich 

Thank you very much for your shared info

Actually I'm trying to review and collect some details about our Arcsight deployment 

I'll try to share with you some cases in order to guide us for improvement points

Let you know that we 've Connector Appliance with ArcMCs (5 appliances distributed on 5 Branches and 1 ArcMC Appliance on every branch big )

(Assets-->Connectors ArcMC -->Logger) on branches --> ESM (One Central ESM)

The connectors are caching; we've tried to increase the memory... from config files but it doesn't work, some other issues has appeared.

can you tell me please how to size the connector properly?

The smart connectors you've installed are based on Linux centos and software?

Our infrastructure contains many important assets: Linux, windows Endpoints, server, Proxies, email filtring,  Vulnerability management products...etc

Thanks for your guidance Michael, I'll structure all identified  issues and back to you

Best regards

 

 

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: How to performe Arcsigth New deployement sizing

Hi Hemza,

I really don't understand why your contact by Micro focus ArcSight didn't tell you.
It is not a good design to send logs through the Loggers as there is a known bottleneck with the forwarder.

We are sending logs to 2 destinations ESM and Loggers Pool (for ESM we are filtering more than for Loggers) which is normal.

It works perfectly 0 cache for both destinations.
If you are facing a caching issue, it means that you connector is not properly sized or fine-tuned.
I need to know the logs types, the throughput max, number of source devices, logs format and connector type.

Then I will be in measure to help you but if you could change the design, it would be extremely better and also if your connector were installed not on the ArcMC (except of course if it is an appliance but in this case, you have no choice to decrease the load as there is a physical limitation)

We are collecting 60 000 EPS with more or less 110 Connectors sending to 2 destinations (ESM and a pool of 23 Loggers) 0 cache.

We have a SIEM Infra working perfectly. The issues we are facing are more due to missing critical features on Loggers like correlation by example.

Thanks
Kind regards

Michael

Highlighted
Trusted Contributor.. Trusted Contributor..
Trusted Contributor..

Re: How to performe Arcsigth New deployement sizing

Hi @mschleich 

Thanks for your reply, 

Please find below a summary about the actual architecture :

 

Arcsight Architecture_link issues.png

 The raison for witch the logs are send from connector to logger is the link connexion issues, if we forward logs from connector to ESM there is a risk of loosing logs/events  due to link unavailability.

To clarify :

Main Site : Connector->logger- > ESM

Branche sites: Connectors-> loggers

Only one ESM, DR configuration is in progress

@mschleich  you said that you are are collecting 60 000 EPS with more or less 110 Connectors :

can you provide me some clarification about the deployement architecture and how are the connectors  are configured ( every connector  for a type of products installed on VM  means  connector for MS Windows Active director, another for exchange, ... can you clarify please)

 

Thanks @mschleich 

BR

0 Likes
Highlighted
Acclaimed Contributor.
Acclaimed Contributor.

Re: How to performe Arcsigth New deployement sizing

Hi Hemza,

 

This is wrong, I had the same discussion several years ago with my manager.
ArcSight SmartConnectors works very well and a lot better than Loggers.
We have also some connectors far from ESM and we have no issue just maybe a small delay.

Did you try before to take this decision?

We have 2 ESM in HR but one is Active thus we have the same than you but we I have forced to do not send the logs to the Loggers, we send to both destinations.

No loss of events, use TCP to be sure that the events are sent and you will see.
What is the bandwidth of your bad link?

I will show you what we have. Do you have a private email address to do not share this on this site?
There is a status (real-time) this is why you don't see the 60 K as it is most of the time in the morning around 9:00 AM. We collect logs from +/- 60000 devices (network devices, AD Servers, Unix Servers and WKS).

ConnectorStatus_200518.JPG

 

We have 60 000 EPS at the the connector level but we send to ESM 13 000 EPS and more or less 50 000 EPS to Loggers as we filter some logs more due to space issue and retention as we want to keep data for 1 year. Thus we have filtered useless logs (after verification that it won't be useful for Forensics too)

There are a lots of events that don't need to be sent to ESM but we have kept the procedure to filter only events that we have identified to be useless not the opposite.

I have told you that if you want to increase the efficiency for logs collections, you can use ArcSight Load-Balancer SmartConnector, we use this a lot, for Proxy Logs and Unix OS/Auditd Logs.
The probability that all Connectors behind the LB are not available at the same time, is very low.
I don't think it is necessary but it could be an alternative.
I repeat, you don't lose any events with SmartConnectors as there is a local cache. You can fine-tune the connectors very well with JVM RAM, multi-threading, aggregation if necessary (personally we use it very rarely)

We are collecting more or less 3.4 billion logs per day on Loggers and 650-700 millions on ESM.
As you can see below:

LoggersEventsCollected.JPG

 

If you can try to send your the logs to both destination, we have configured a Pool of Loggers. You will see that it is working. I don't know the quality of your network lines but it should work.

Indeed, we have choose one or more connector per logs types, it is the only solution if you want to filter easily and later give some metrics to management. For AD by example, we are using WiNC Connector without WEC as it is not necessary as we have 36 AD Servers but for WKS logs we have 6 WINC for 20 WEC for 50000 WKS but no all are active at the same time thus consider 40000.

For AD WiNC we have very big VM 8 GB RAM, 8 CPU, it is normal as we send to ESM 1200 EPS per WiNC.
The key is the sizing and the fine-tuning of your connector but (in rare case, you could impact the global stability with some Use Cases like rules thus when you do this sizing and fine-tuning, you have to stop the more resources consuming rules, we had faced this issue I have reached the limit of ESM due to only 1 rule, ESM CPU were running at 90% in place of 10%, exactly and it is something that the Support didn't know, more than 1 year to find and to solve this issue. Now the SIEM is worked very well as never. So keep this in mind.

I hope this info will hep you but if you have any more specific question, do not hesitate.

Thanks
Kind Regards

Michael

Highlighted
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: How to performe Arcsigth New deployement sizing

@mschleich What an amzing write up, @Hemza  it think thats nothing you find in a best practice guide by ArcSight!
I can just really confirm what @mschleich  has said. Never forward from logger - this possibility was designed only for 1500 EPS not more (if it could handle1500 EPS at all).

if you want to hold the number of connections from your branches log. think about a second layer of smart connectors.

Layer A get feed/collects from the devices, and forward (in cef syslog)  to one  connector (for the local branch office) layer B that's connected to the ESM. however I would not do any filtering/aggregation nothing on that connector, add way more then 1 GB for caching, and ensure there is no Queue dropping. This one SC ( or maybe you loadbalance 2 SC) should be then be able to send 10k EPS easily, as the data are already in CEF.

you loose visibility, flexibility and managability (i.e. filtering needs to be done on arcMC level rather ESM) not only when there are issues - so i would not recommend that solution - but it is a possibility if you want to hold number of connections low.

 

KR

A

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.