What is the maximum EPS that ESM 6.5c can support?
Can you please give some examples of scenarios? For example:
- An ESM installed on HP Z620 with Intel® Xeon® Processor E5-1660 (15M Cache, 3.30 GHz, 0.0 GT/s Intel® QPI), 66GB RAM, 500GB hard disk, and RHEL 6.4 can support 5,000 EPS before caching occurs.
> The disk I/O is going to be a major factor.
Interesting. Whenever I talk to HP guys about disk performance, they tell me that I/O rate it is not so important in comparison with ESM 5.*
Correct, in comparison with 5.*, disk I/O going to be slightly less important, mostly because the number of persisted events is much higher at insertion time. However, query performance is going to be directly proportional to disk I/O performance. There is a certain threshold (which is different for every environment) below which the system becomes unusable from interactive standpoint.
The last I spoke with my HP Reps (not too long ago) , the rule of thumb was 7 IOPs per EPS. I talked to some consultants that told me that 10 IOPs per EPS was the sweet spot. Not sure what anyone else has heard at this point.
I am about to roll over to 6.5 and my calculations were on the 7 IOPs per insertion (times what I expected Maximum EPS to be) then threw on some extra overhead for the read IOPs.
Its actually the other way around - 10 events per 1 disk operation on average. So if you peak at 10,000 EPS, your storage heeds to support 1000+ IOPS. These are the numbers we've used for ESM 5.x and it takes into consideration indexing and queries. I would expect these numbers to be higher for 6.x - probably 12-15 events per disk operation, although we haven't reliably tested this yet.
Always keep in mind what's going to be writing and reading to/from your storage, and provide as much overhead as you can afford to ensure better query performance.
Here's a useful post on measuring disk performance.
The question for EPS is really a trick question on an ESM. If one is talking about just receiving and writing data into the ESM without doing anything else, sure, thousands can be the EPS. However, add some rule matches, some correlation, or even reports and that will affect your outcome. How to get the best read and write through would be based on the RAID setup (e.g. RAID 10) and then you would want to consider if you can perform more writes than reads, this can be setup on many RAIDS as well. Processor and memory are good however, in this day and age, one is not limited usually to those pieces of hardware. In closing, there are many moving parts and so everyone's experience will be different for EPS on an ESM; the question is begged, what do you want your ESM to do?
In closing, there are many moving parts and so everyone's experience will be different for EPS on an ESM; the question is begged, what do you want your ESM to do?
That's not really a helpful answer, is it? If I tell you "I want ESM to proactively find advanced targeted threats", how does that translate into hardware? The answer is, it doesn't. The volumes of data you need to collect will.
RAID10 configuration is not a panacea, and I'm a bit tired of people assuming RAID5 is a problem and moving to RAID10 will fix it. I've seen fantastically capable RAID5 arrays with large cache that beat any local disk configurations. It's also possible to set up a RAID10 with slow disks and have insufficient performance based on the data throughput.
The volumes of data you're moving through your system is the primary parameter for environment sizing. CPU and memory are important to a degree, but nothing bottlenecks the SIEM like poor performance of storage, regardless of the use cases. That's why understanding and measuring the disk throughput that your setup is capable of is absolutely key.
RAID 10 is quicker than RAID 5 using the same disks and hardware for reading and writing. Of course , RAID 10 can be a real 'boat anchor' if inferior hardware is used, but that is a red herring in this discussion. If one knows that there is a propensity for high volume of disk activity and no need to keep the logs for data retention needs, why not use RAID10? With all this set aside about RAID, consider rules & correlation. These can affect how many events can be taken in. If rules are written in such a way that they are correlating in memory, memory just may be your bottle neck depending on how many rules you have behaving this way. If in turn you decide to correlate to an active list, well, be prepared to write to disk and so the disk issue rears its ugly head.
I do not find RAID10 unreasonable to suggest over RAID5 using the same disks, RAID controllers, with the same caching abilities. Understanding with RAID10, with the same disks, one will have less storage space opposed to RAID5.
If one wants the ESM to rely heavily on disk, then build to something that can take high volume of IOPS.
If one wants the ESM to run a lot on memory, look to there.
I have seen SIEM's bottleneck because of storage and memory. I believe I provided scenarios that was requested in the original question.