Highlighted
Regular Contributor.. Hemza Regular Contributor..
Regular Contributor..
334 views

Windows Log Collection

Dear Community, 

We are looking for the best solution to collect logs and forward it to the connector /Logger

we've  5 sites and every location contain many windows servers (Active directory, SCCm, SCOM, Exchange servers, windows workstations...)

is it better to use SCCM or windows Unified events collector (WUC)  or WEC/WFC ?

 

Thanks

0 Likes
10 Replies
stvhull_forces Respected Contributor.
Respected Contributor.

Re: Windows Log Collection

WEF all the way.

especially now with Win 10 and Server 2019 which include Azure hybrid functionality. 

having multiple dedicated Windows Server Collector on which you would (preferably) have an additional partition in which you will install your Smart Connector (preventing the need of admin rights to the primary partition) is also a good idea. 

unfortunately, WEF is pretty time consuming (planning and development wise) depending on your subscription need and network (zoning) segregation.  not to forget the necessary Group Policy needed.

and...

be very careful... WEF could easily get your ADP EPS to the roof if not planned correctly. 

over here... we needed to perform a lot of testing in order to have enough "fine tuning" done.

others may have different opinion but... personally.. WEF... 

if you deal with UEBA, EDR and/or Threat Intel... WEF could be of great interest for them as well. 

Regular Contributor.. Hemza Regular Contributor..
Regular Contributor..

Re: Windows Log Collection

@stvhull_forces  Thanks for your reply

do you have a guide  to be used in  WEF deployement for Arcsight  that you can share 

If WEF machine is unavailable or link between  the connectors are down, does WEF cache the events of not, 

Thanks

0 Likes
stvhull_forces Respected Contributor.
Respected Contributor.

Re: Windows Log Collection

Hello again.

Big picture ... WEF is mainly divided in +/- 2 (depends how you see it). ME... i see it as WEF and WINC

1- the Windows portion, WEF is a functionality that Windows infrastructure administrators would need to enable. so... make sure you have a very good "collaborative" relation with them... 

they would also be the one "profiling" the subscription... meaning, they would evaluate what they would appreciate to be included in a ESM Used Cases. (remember, they are the Used Case customer as well as the provider of Security events at the same time).

they might need very specific security events (X,Y,Z plus maybe, USB or phones, Powershell or Sysmon). Now those days WEF can be very granular but would required (from the Windows Admins) to be setup to generate the security events for it, so they might want to evaluate the additional load on their servers (and workstation... and Network) before and after the GPO change. 

WEF is about instructing endpoint (workstation, Servers, Application on Servers...  etc) to forward their security events to a (or multiple ) Windows Collector Server... it all depends on which "WEF subscription" they fall under... 

2- then you have WINC (nice name from ArcSight to name the Smart Connector)...  WINC SC will read the file in the folder in which the WEF mechanism drop its security events in order to forward them to the destination (Transformation Hub, Logger, ESM,  etc). 

i strongly suggest that you install the WINC SC on each server it self (if you have multiple Windows Collector Server) SC will read, normalize the info locally and encrypt it before sending to destination. 

depending on how you build your ESM Smart connector folder structure ( in the ESM ressource tree)... and if you have a zoned/classification segregated infrastructure... you could be required to have 1 SC per subscription (ex: Win 10 Workstation) per zone per classification. again... this is up to you and your team.

as for the communication failure between the WEF client and server... i need to say that it all depends on the network structure you will setup. WEF subscription does provide fail-over.

From some info i was able to find, it appears that the "source initiated subscription" does provide caching... but this could be something that the Windows Admin could validate on your end.

as for the communication failure between the WINC SC and its destination.. WINC SC will cache as much as you configure them to do... if the communication is down for a very long time.. the SC will delete, in order, the security events with a lower significance first... and so on.. critical ones will be kept and deleted last. when the communication will come backup... it will create a bust of event on the destination.. 

i have attached some documentation which are kind of old but still can help. 

if you have the opportunity, look at the Activate Framework for pretty good package related to this... they will provide you additional guidance.

but again.. it all depends on the amount of time you have to create this "solution".

cheers. 

Super Contributor.. ejsimon Super Contributor..
Super Contributor..

Re: Windows Log Collection

I'm already using WEF/WEC and WINC but would like to do more filtering (fine tuning) of events we collect from the end devices (which would mainly be member servers and workstations).  I know this varies a lot depending on the company and needs, but do any of you have good listing of the Windows events you're filtering on?  

Thanks,

Eric

  

0 Likes
Regular Contributor.. Hemza Regular Contributor..
Regular Contributor..

Re: Windows Log Collection

Try to collect and forward events that are feeding usecases, and some additional security events,

Thanks
0 Likes
mschleich Acclaimed Contributor.
Acclaimed Contributor.

Re: Windows Log Collection

Hi Hemza,

I can answer to your main question that the best solution is to use WEF/WEC and WiNC for that purpose.
based on my experience with more than 20 WECs installed to collect more than 50000 workstations (until 800 million of events per day), for sure it is the best solution.

Now, if you have some servers like AD Servers, personally, I prefer to collect logs directly from the WiNC but it depends on your policy if you accept to have one inbound connection for the RPC bind.

Regarding the WEC, the point that you have to take into account is if you are authorized to place the WEC in the same Windows Domain as your sources because if it is not authorized, you have no other choice to use certificate to access the WEC and this increase a lot the complexity and the management.

If you choose source initiated, you have just to configure a GPO that will tell to each source to which WEC they have to forward logs which means nothing to do from the WEC side except to build the subscription.

It is important to define which sources will go to which WEC because this information won't be shown into the logs thus it is a bit annoying to troubleshoot, this is why it is good by example to choose a department by WEC or something similar that you could statically map, normally it should not change except if the GPO is modified. Based on practical experience, it is important.

For the time, it is really complex and long.

  1. For the WEC, you ask for big servers correctly sized and configured (CPU, RAM, HDD space, Network, no profile loaded) I insist to be able to place the WEC on the same domain.
  2. You ask to your admin to create a GPO for WEC forwarding.
  3. You build your subscription to define what you want to collect (event ID)
  4. You install the WiNC, you fine-tune them because they will be used a lot thus the default config won't be enough.
  5. That's it You create ArcSight Content to monitor and track your sources to be sure there is no mistake or missing sources. believe-me it works perfectly when it is properly sized. No delay, no cache, no lost of logs.

Indeed, regarding the performance and the complexity of the config, it works very well if you have sized properly your infra based on the EPS and source hosts.

Normally, we can consider max 5000 hosts per WEC but if you could reduce it to 2500, it will be better than for each WiNC, you have to be sure that the EPS won't be above 1000 (Sent to ESM) for Loggers no problem if you have 2000-2500 EPS per WiNC.

For collecting WKS logs with Sysmon enabled, we have installed 4 WiNCs (we filter events for ESM) for 20 WECs.

But on the other side, we have used several WiNC to collect logs form AD Servers.

WiNC is for me one of the best ArcSight SmartConnector because it used the windows API thus it is very efficient but also it permits to collect custom logs, it is a PUSH connector (WUC was a PULL connector), there is a filter capability to do not collect some events based on XPath Query and finally it is really stable.

If you have specific question or if you need more info, do not hesitate to contact me.

Thanks
Kind Regards

Michael

Regular Contributor.. Hemza Regular Contributor..
Regular Contributor..

Re: Windows Log Collection

@stvhull_forces @mschleich  Thanks for your usefull guidance , 

0 Likes
Regular Contributor.. Hemza Regular Contributor..
Regular Contributor..

Re: Windows Log Collection

Hi all, I want just to check another details : can SCCM be used for event collection and event forwarding ?

Thanks

0 Likes
mschleich Acclaimed Contributor.
Acclaimed Contributor.

Re: Windows Log Collection

Hi Hemza,

According to me no. It has not been designed for that purpose. 

But there is also  SCOM which could analyze Windows Events and you can build alerts on it but it has not built for a security purpose thus you will be limited at one point.

I don't understand why it is an issue to setup a proper WEC and the necessary WiNC you needs based on the throughput.

Because if you check my reply on this discussion or on another one, it is working very well.
it is not complex if you could place the WEC on the same domain as you sources.

I can help if it is necessary.
For me it is the best solution to collect Windows logs from many sources.

Even for AD, you don't need to use a WEC, you can directly use the WiNC and it works very well too.
I have already give the specifications. WiNC is currently one of the best ArcSight SmartConnectors in terms of efficiency, capability and stability.

Thanks
Regards

Michael

Regular Contributor.. Hemza Regular Contributor..
Regular Contributor..

Re: Windows Log Collection

@mschleich Thanks your reply, 

I want just to have a big picture about all the possible ways to collect event's ,

We are planning to plan Windows Event collection using  WiNC, 'ill contact you for any other question and assisstance,

Thanks ALL,

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.