How to poll logs from over 500 clients?
We are building a new image for about 500 systems and I currently use a Connector appliance to poll the Windows Security Events. I have an account log on to each machine in the container and pull the events. This can cause lag and failures in event polling because of the amount of systems. I was thinking of putting a smart connector on the image and then deploying the image to all the systems. This would allow for me to get the Application, System, and Security events on the system also allowing for encryption and compression. This systems are not on site so management can sometimes be difficult. What to does everyone else do when you want to poll logs from this many systems? What is the best practice? What are the pro's and cons?
Simple answer: dont use the WUC.
I worked at several company that tried the WUC on that many servers - mainly because of concerns about installing agents directly on every server that needed collection. For the few moment it seemed to work, it was simple hell to maintain.
Have you ever followed the ArcSight conferences about setting up the WUC by separating servers according to their normal EPS rate ? Its preposerous to even propose such a solution. With 500 servers or more, who's to say that the "profile" for a few servers wont change in an instant ?
I see 2 solutions, there's could be a lot more out there. But at least one I have experience with, positive experience with.
Go with Snare for Windows Agent from Intersect Alliance (Balabit also make a compatible agent I was told).
Its C code, not memory and CPU hungry as a Java process, and it takes the security event log and sends it thru tcp syslog to your smart connector.
ArcSight does support a Smart for Snare for Windows - not bad but not updated as frequently as the WUC, still very stable.
It can also do the system and application event log, and even custom event logs, but I have no experience with that - We only need the security event log so far.
Dont forget to install 1 Smart for Snare for every sustained 800-900 EPS you'll expect and a few more for surges. Then load balance your feeds towards your SmartConnector installs.
The commercial release of Snare supports TCP and buffering. The free only supports UDP.
Commercial cversion even support TLS if you are up for it - which is also supported by the SmartConnector.
They are even planning direct CEF support.
We had the free version at first but ran into problem with multiple timezones and loosing logs.
(the "u" in UDP is quite descriptive)
I am in no way affiliated with Snare. I just use the product and would never go back to WUC.
We had great support so far, even going to asking several features to better support our audit collection, that actuallly made it into production few weeks later (the use UTCTIME=1 feature was us!!!).
We are collecting more than 1k servers with almost no issues ever over 8 timezones.
For busy server, tell your IT guy that Snare will take one CPU and plan for it accordingly.
If you go with one WUC on every server, You'll need much more RAM and CPU and no logger or ESM will accept this many clients - you would need to concentrate into a syslog serveur.
Lately, Microsoft support centralization of event logs. ArcSight did mention at last Protect they would support that. It was too late for us to explore that solution - and beside why change a recipe that works ? Might worth your time checking into that.
Hoped this helped,
we have different experiences: we use WUC for few thousand Windows servers and the environment is pretty stable.
A rule of thumb - no more than 100 servers/connector.
I recently had an ArcSight engineer come and talk with me and a few from my office and he told us that ArcSight recommends no more than 200 systems on any one WUC. Since my office has about 1700 workstations he suggested we use the Windows event collector and send all the events to a connector on 1 server and have that connector forward them to our ArcSight. It will take some engineering by us but we will not spend any extra money, our ArcSight should be more stable (We installed a connector on every workstation which pretty much crashed our ArcSight since they say 250 connections max), and help with bandwidth issues.
For WUC you can always refer to our Protect presentation from 2012 called "HP ArcSight Windows Unified Connector a look under the hood - Moehadi Liang Girish Mantry.pdf" that can be found here in Protect724 at:
Regarding instead Windows Event Collector it is Microsoft function on how to collect logs from multiple hosts into a single server-host:
Then to pull data from this WEC server you can use normal WUC (or the new WINC that was released in the meanwhile).