Integrating OS logs from cloud environment VMs
I am looking for some guidance/recommendations for onboarding OS logs(Windows & *NIX) from AWS and Azure environments into on-prem ArcSight:
I have a heterogeneous cloud environment(AWS& Azure) with many subscriptions/accounts that are not joined to the domain. These environment contain both Windows and *nix hosts VMs and we are trying to find a good approach to integrate the logs from these VMs with our on-prem ArcSight.
These are our challenges:
- More than 50 Azure subscriptions and AWS accts
- Accts/subscriptions belong to the different units
- The environments are not domain joined
- They are also not connected to each other(all standalone environments)
- A VPN from the cloud provider to the on-premises site is not an option at this point.
- Current approach is to configure each linux server to forward logs to a Unix connector in each account/subscription.Connector then sends logs to on-prem Arcsight
- However, that means a syslog connector(aggregation point) will be required for each account/subscription)
- Not ideal from a management perspective
- Since the environments are not domain joined, using a WUC with a service account performing WMI calls to pull logs will not be an option
- Wondering whether I can leverage Windows event forwarding
- We may be able to leverage scripting for potential configuration/auditing changes needed
- Do I need to set up a connector for each subscription or is there a way to group them so I don't have over 50+ connectors(for both Windows and Linux) to manually set up and maintain?
These are the main questions I have as of now. I am looking to see if anyone might have had any experience in a situation like this before (basically getting connectors setup to non-domain hosts being sent over the Internet).
Thanks for the help.
Ok, lots of questions and points there, but there are a number of things to address.
Some initial comments though:
1) I am a big fan of Windows Event Forwarding (WEF). It is a simple, scalable and straightforward way to obtain log data. There are a number of advantages here in that you can use the NATIVE communications between Windows machines and bring the data to a central point - then collect the data from that server (either as WEF data natively or via an SQL conenction). Its easy, simple to enforce and since its native Windows communications, its easy to setup and manage. I would encourage anyone looking at Windows platforms to take a closer look at WEF.
2) You mention about getting the events from the platform also - this is an option that isnt widely published I am afraid, mainly because its pretty new. Azure as a platform does have the ability to deliver the data via this method and I would encourage anyone to take a closer look here:
You will see that we can collect the data via JSON - though you will need to write a FlexConnector to process this. Additionally, there is a new capability that Microsoft provides called AzureSIEM or AzSIEM. Its new and partly hidden, but expect it to be a lot more integrated in the coming months. The information about it has leaked out and you can find it here:
Take a look at the guide and it will cover off the way to integrate ArcSight SmartConnectors to pull and process the data. This means that you can receive the data through the Azure platform, rather than via the native Windows mechanisms. You will need the Azure SIEM parser for this also:
3) For Amazon, its a little easier since there is a mechanism that is available to do this, but at the moment, its pretty focused around S3 platforms rather than others. Expect to see this develop pretty quickly going forward, but take a look here:
Some generic information here:
And Richard Kent very kindly put together some information here:
Not sure I have answered your question there, but hopefully this is a start!
Is this still the sollution? using the AZSIEM (or AZLOG as it is called these days).
Using a JSONFlexConnector, we need parser files, Does anyone want to share the files they have?
I've been looking in using the WiNC Connector Azure parser, showin in the Marketplace, but i don't know if this is an old solution or a newer one.
Anyone care to share their thoughts?