Our vBulletin migration is complete.
Welcome vBulletin users! All content and user information from the Micro Focus Forums (vBulletin) site has been migrated to this site. READ MORE.
SIEM-TECH Honored Contributor.
Honored Contributor.
1895 views

"ArcSight Investigate" (RSA Conference announcement)


Anybody have Intel on this product outside of:

-10x faster search using HPE Vertica as an embedded high-performance DB

- Powerful and intuitive search experience using customizable dashboards

- Direct integration with Hadoop as a long-term data lake

- Fully integrated with ADP 2.0 & ESM

- Expected Early Q2 2017

I'm just curious what you guys/gals picked up at the conference etc.  I assume no documentation has been released yet.

What tool does it replace if any?  Or is it another tool in the massive stack?

Thanks all!

Can't wait to play....

-Mike

Labels (4)
10 Replies
Highlighted
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

So some comments and feedback for you:

Q) 10x faster search using HPE Vertica as an embedded high-performance DB

A) Yep, and thats pretty conservative too! Vertica is a high performance analytics database that scales extremely easily and comparisons against other solutions around search show massive performance gains. Simple things like being a true cluster solution and having support for multi-threading operation means its a lot quicker than other ways to do this. The numbers get even higher when you start doing analytics based stuff too - getting to 100, 200 or even higher performance increases.

Q) Powerful and intuitive search experience using customizable dashboards

A) The design goals are to not have a requirement to understand the schema or the language you would use to search with. With support for 'natural language processing' and a lookahead capability to suggest searches, fields and even data. The search language is extremely flexible and you also have access to SQL based language too - for the expert users obviously. As for dashboards, well the ability to create them, update and even customize them on the fly. You can even update the dashboard objects with drag and drop and it will update as you are doing it! No need to re-run searches or anything like that.

Also, its worth noting that support is provided to do advanced analytics and data processing with dashboards too. A good example of this is doing things like sum of data (interval) across an extended period of time and then grouping it by username. Sounds very specific, but the example is a lot of customers want to show data volume by user to illustrate abnormalities.

Q) Direct integration with Hadoop as a long-term data lake

A) Yes, the idea is to provide options and capability to integrate other data sources, such a Hadoop. The first release will provide this and you can look to define retention period and push data to Hadoop, seamlessly integrate with Hadoop for long term data and look to ingest data from Hadoop too. Lots more detail to come, but forcing customers to use one system over another doesn't make sense - so its better to integrate! This is just part of an overall strategy to provide an open platform for customers - not locked and closed.

Q) Fully integrated with ADP 2.0 & ESM

ArcSight Investigate will directly integrate with ADP so that you can feed data in and also bridge into ESM for investigations. More of this to come though - so keep a look out for this.

Q) Expected Early Q2 2017

A) Yep, customers are looking at a beta version now and you can sign up for an early look at Arcsight Investigate | HPE

As for some other points - does this replace anything? Not really. Its a new capability. Its well known that for some, Logger is fine. But for others its not fast enough or flexible enough for their needs. There is a demand for massive data storage and high speed search is high and this is what Investigate does. It doesn't replace Logger though, as it has its place. Logger has a secure database, data validation and compliance reporting packages - so it has its place. But for Investigate, its a new platform that provides a high speed platform for this massive amount of data and search performance.

As for the comment about massive stack - fair point, but be aware that Investigate isnt a single server solution. The point here is that it brings significant improvements and performance and that comes with a set of requirements. But, what it does do is reduce the dependence on ESM and Logger specifically. How? Actually its pretty simple - Logger can be used to focus on compliance use cases and hence reduce the footprint there. Additionally, because you don't need to store as much in ESM, you can reduce its footprint too (and not forgetting that you have a much reduced overhead because you arent using it for search either). And ArcMC (often overlooked here) brings further connector management and device status monitoring too - further reducing the overheads on ESM for this. The good news is that you can look to reduce the dependence on ESM and Logger, improve management with ArcMC and add a whole set of capabilities with Investigate that fills the gap around high speed search and investigate.

More to follow, but this is foundational and critical to adding a lot to customer deployments. And I haven't even scratched the surface with Kubernetes, Docker, microservices and scalability......

SIEM-TECH Honored Contributor.
Honored Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

Paul,

First off, you are a machine.  Your activity on here is commendable.  Thank you for the response, I really appreciate it.  I'll wrap my brain around this a bit and likely hit you up with a couple more questions.  But thank you, the response is greatly appreciated.

-Mike

SIEM-TECH Honored Contributor.
Honored Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

Paul,

In regards to your response, I understand HPE is moving towards removing services provided by ESM so it can focus on correlation, Logger will continue as it is and ArcMC has increased functionality etc.  What I am digging for is the intent behind this new tool.  Is it to compete with Splunk?  Could it not have been built in combination with logger?  I feel like Logger will be this expensive compliance tool now that there is limited use for (could be replaced with something open source), assuming Logger and Investigate ingest the same data.  I'm trying to better understand the underlying purpose outside of speed and flexibility what differs.

For what it's worth I'm a huge advocate for ArcSight, I'm just struggling with walking in front of a customer with:

UBA

DMA

ESM

Logger

ArcMC

Event Broker

Hercules

Investigate

The above requires a small Army and a dump truck full of gold to implement and sustain.  And I fully understand not every customer will have a need for all of the above and every situation is scenario-based.  I just wish it was slightly more condensed.

Thanks Paul,

Mike

pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

Good comments and its fair. But let me cover off a few things first:

UBA - dedicated solution to address user and entity behavior analytics for customers - DOESNT NEED TO BE an existing ArcSight customer at all and you can integrate this with IBM or Splunk or anything else. While it would be nice to add some capability on top of what you already have (say the SIEM), what is clear is that technically this is far from easy. Take a look at the terrible solution that is the IBM UBA solution! Splunk has a better approach, but its far from integrated (except the UI) and its driven by two different back-ends. It makes a lot of sense to have this separate.

DMA - a cloud based solution for monitoring DNS traffic for the 'calls home' from malware. While again, it would make a lot of sense to add this as capability to your SIEM solution, its the volume that is the killer here. It makes sense to leverage cheap compute and storage in the cloud and not have to pay licensing costs into your SIEM. Again, Splunk has a good solution here and even has an App for this - but you will pay for their collection - a lot!!! Simply not worth it and hence the need for DMA.

ESM - you know this bit.

Logger - Actually its now part of ADP or ArcSight Data Platform. You can use it, you don't have to use it. Some customers will, some customers wont. Bottom line is that its an integrated component that is available in ADP - use it if you want.

ArcMC - the central management aspect for ADP and a key part of it. Again, ADP is the key aspect here. ADP provides the collection (SmartConnectors), storage (Logger) and management (ArcMC) for your monitoring needs. Works with ArcSight as well as third party solutions such as Splunk, Hadoop and anything else too!

Event Broker - an instrumental and critical component, but one that is also optional. This is relevant going forward as a key infrastructure piece for log data distribution. And this is the critical part - for any solution, not just ArcSight. While it might be nice to start to say to customers 'you have to go with an ArcSight only architecture', lets be honest, that wont fly. In fact, it wont work for IBM, Splunk or anyone else either. With the advent of flexibility and multiple viable solutions that are driven using log data (such as UBA / UEBA for example), vendors locking in customers to a particular architecture and platform is not acceptable. Hence the use of Event Broker and Kafka. Now you can integrate Splunk, Elastic, Hadoop, UBA and others with ease! One collection technology, one distribution technology, multiple consumers to do different things! Ok, so there are a few things that need to happen on the way, but its this flexibility that is key. Oh, and EB is part of ADP - entirely optional too. For some customers they will never use it. For others its the single biggest reason to use ArcSight. But we arent forcing this on anyone - just making it available because of the openness it provides.

Hercules - internal project name for the longer term view of a fully integrated platform - forget it for the moment.

Investigate - a new platform designed to address specific requirements and to introduce our new approach to high speed data access and analytics. It brings Vertica as a back-end and uses Event Broker as the feed in (so in this case, you do need Event Broker to feed it). But it also brings a massive amount of flexibility and capability that we are only just starting to scratch the surface on.

To take your point on needing a dump truck, yes, there is a concern. But lets be 100% honest with ourselves here - ArcSight was never a small business solution and over the intervening years, this has been further accentuated by competitors in this market. ArcSight is really a larger enterprise solution and as such needs to have a component based architecture to fit customer requirements. While other competitors like Splunk do a good job of making things seem simple, they are just as complex and in many cases require more components anyway. Splunk does a good job of using the same components, just re-purposed to different tasks, making it seem simpler and 'lighter'. We call the components what they are, and for a lot of customers, this is what they want.

And to dig into the Logger point again - Logger is cheap, very cheap in comparison with other systems and likely to remain so. We absolutely DONT expect to see customers spin up hundreds of Loggers to get the retention and ingestion of data they require. This works for specific use cases and customers, but this is not what I would expect the general customer base to do. What I see is a situation where customers will use what they want and where they want it. For example, say a customer is impacted by PCI DSS. Typically this means that they have to keep around 1 year worth of data and it usually only impacts around 20% of their systems. Logger is ideal and simple for this. Its very straightforward, provides you a secure mechanism and we even provide the reporting packs now. However, you will most likely want to have access to the full 100% of the data in a high speed system, such as ArcSight Investigate. With speeds typically at least 10 times faster than other solutions, you can do more and faster.

But do you want to store a year online? Maybe, but with the bridge to Hadoop, Investigate can manage this for you and then push the data into Hadoop for longer term storage and management - and you can now run your own jobs against this data using your tool of choice! We arent going to charge you for how much data you want to store and we are certainly not going to charge for any other reporting on that data either. You have flexibility around this and you can do what you want - where you want! And we absolutely see the vast majority of our customers look at or currently implementing a strategy around Hadoop - hence the importance of getting this in Investigate.

For customers who have Logger today, they have a choice - they can move to Investigate or they can stay where they are (or a hybrid approach). But what is clear is when we ask customers what they want, its search speed. I do find it frustrating that the whole market around security monitoring, SOC's and SIEM's has boiled down to one thing only - search. Again, credit to Splunk on this, but they have changed the face of the market and focused it around the one thing they do well - search. Why did we build ArcSight Investigate? For a lot of reasons, but in this first release, because customers wanted high speed search, at a cost effective price and with capabilities to do more. This is what we have. Ok, so a lot of what is being provided is search, but I have to stress this is more than just search though. Its guided user interfaces, its providing a true set of capabilities behind the covers and its about addressing security use cases at scale - ok, its marketing headlines, but this 10x speed over the competition is no joke. Additionally, its 3x LESS hardware too! Oh, and did I mention that we take up less space too? Yes, its 3x less storage requirements too. Its not just another 'me too' capability here, its a key set of usable requirements today and a platform for the future.

I could go on, but let me finish off with a consolidated view though:

1) Data acquisition, storage, distribution and centralized management - ArcSight Data Platform (ADP)

2) High speed correlation, alerting and platform for use cases - ArcSight ESM

3) High speed search, guided investigation focused around L1/L2 users - ArcSight Investigate

4) User and entity behavior analytics to monitor the person and identity - ArcSight UBA

5) Dedicated solution for monitoring of malware and their activity - ArcSight DMA

Thats about it - still a comprehensive list, but hopefully this positions things a little better. Do I expect a customer to have all of it? Nah, would be nice, but unlikely. What do I expect a sophisticated and focused customer to do? I would say ADP with ESM and then add Investigate now. Building to add UBA as needed. DMA is powerful and flexible, but a lot of customers don't like the cloud aspect.

Hopefully this clarifies a few things.

pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

0 Likes
SIEM-TECH Honored Contributor.
Honored Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

Paul,

Good stuff, I appreciate it.  I think you addressed everything you could.  All fair...we are generally in a position to guide customers and with all the licensing transitions, new products etc it's a bit to deal with (trying to stay ahead of what's coming).  I'm looking forward to all the smoke clearing and all the announcements of late to come to fruition. 

Thanks again,

Mike

0 Likes
Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: "ArcSight Investigate" (RSA Conference announcement)

Paul,

A few follow up questions on your informative posts above.

  • "10x speed over the competition" - Does that include Logger?  Is Investigate 10 times faster in search over Logger?  Using the same hardware?  I don't have the competition's products so comparing to them doesn't say anything to me.
  • "3x LESS hardware too" - Same question, less hardware than what?  I don't have real world data on your competitors products.  What are you basing this statement on?
  • Does Investigate have improved RBAC over Logger?  I don't have cycles to participate in the early access program but I'm curious if there any improvements there.
    • Controlling who can access what events in Logger is painful if you want to get more nuanced in what sources group X can see, when compared to group Y or group Z. Is this easier to do with Investigate?
    • Also, centrally managing users who have access to multiple Loggers through ArcMC is an improvement over defining users manually on each Logger but the fact that ArcMC/Logger still can't leverage AD groups and that each individual user needs to be defined by manually is a pain point that I'm hoping isn't there with Investigate.  Any improvement here?

Thanks in advance

pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

Some feedback here:

Q) "10x speed over the competition" - Does that include Logger?  Is Investigate 10 times faster in search over Logger?  Using the same hardware?  I don't have the competition's products so comparing to them doesn't say anything to me.

A) When we did benchmarks to compare with 'competitors', we didnt use Logger. We actually used some benchmarking tools, content and data from a competitor and used their methodology. We saw numbers of around 10 - 100 times faster. So we went with the more conservative number of 10 times for the moment.

Logger is powerful and flexible and simple - but its got an upper limit (per node) performance. Some customers have done a great job of pushing it beyond what we thought, but it does have some upper limits on what we can do. Using ArcSight Investigate with Vertica as the back end means we can jump significantly beyond this without the limits. Its starts to get really powerful and unbounded.

Everyone hits on the search thing - somewhat misguided in some cases, but hey, we have to do what customers ask - but the performance gains get really impressive when we start looking at data set manipulation, real-time analytics and doing things like a 'JOIN' at scale. When we look at those (in comparison to competitor solutions) we are 1000x to 2500x times faster! This is where the future is and being able to process 5bn events for a statistical calculation of frequency and identify an anomaly (say frequency of login events for the identification of a 'low and slow' password guessing attack) and do it in less than 5 seconds WITH NO IMPACT, now we are talking. This is the difference and its dramatic.

Following on from my rather thinly concealed comment about searching though - search is required. Its essential. But it isnt the key. Its a part of the wider set of capabilities we need. Many organizations focus very specifically around this though. Don't get me wrong, there is a lot we can and should do with search. But we also need to have an eye on where we need to go and search is just a stepping block. When we apply real-time analytics and scale to the equation, then we can start to answer some questions we haven't been able to before. And thats when it starts getting cool.....

Q) "3x LESS hardware too" - Same question, less hardware than what?  I don't have real world data on your competitors products.  What are you basing this statement on?

A) Again, this is related to the answer above - we used a competitors benchmarking tool with their data, queries and methodology and on the same hardware, we found that we were 10x speed, 3x LESS storage and hardware to get to their same levels of performance. I cant say what we did, because it would open up a lot of questions about data that we are currently doing now - but it doesn't take much to figure who we  are comparing against and what benchmarking process we used. Suffice to say, we were really impressed with the gains we have and the reduced footprint we can implement.

Bottom line is that the underlying capability with Vertica is that we have a true clustered solution that cooperatively works together and that uses a true multi-threaded query process. While this isnt earth shattering functionality, it works and its reliable. So its great news and really impressive performance gains.

So to summarize this - to get the same performance as a competitor, you could use 3 times LESS hardware, 3 times LESS disk space and still see a performance improvement! But, as I have mentioned before, this isnt a single box solution.

Q) Does Investigate have improved RBAC over Logger?  I don't have cycles to participate in the early access program but I'm curious if there any improvements there.

    • Controlling who can access what events in Logger is painful if you want to get more nuanced in what sources group X can see, when compared to group Y or group Z. Is this easier to do with Investigate?

A) The version 1.0 will have a level of RBAC and the plan (of course, plans can change) is to have a mechanism that allows us to define users and roles that can define access to the data. We can then control what data they can view and even down to the fields. The exact details of this will be communicated later, but its going to be a step above that of Logger - however, in the first release, it will be somewhat limited to what we really envisage in the solution going forward.

    • Also, centrally managing users who have access to multiple Loggers through ArcMC is an improvement over defining users manually on each Logger but the fact that ArcMC/Logger still can't leverage AD groups and that each individual user needs to be defined by manually is a pain point that I'm hoping isn't there with Investigate.  Any improvement here?

A) Yes, it be centrally managed. Please note that in the 1.0 release, we have to have a separate user management component. Its a shame, but it was one of those 'go / no-go' questions that R&D teams constantly battle with. In this case, getting the centralized user management into ArcMC was a step too much in this release and hence got pushed out into a separate component. However, it will be brought into ArcMC in a later release, where you can then have consistent and centralized user management across all ArcSight components at this point. The plan is to have it easily defined and integrated with external tools, such as AD, so it will be simpler and easier going forward.

Please note that ArcSight Investigate will be one solution which will have users defined in it - Vertica uses a true clustering solution (and its cooperative too) so that you don't need to manage separate nodes in it as you would with Logger and peering. Simply put, if you say had a 3 node Vertica cluster and wanted to add another 2 nodes, you would simply go to the management interface, define the nodes (its all container based remember) and add them to the cluster. A few clicks and you are done. Vertica will then automatically build the profile, add to the cluster and then start building out the replication model etc. No need to define, adjust or manage indexing or even any form of configuration.

Vertica has an extremely clever mechanism to manage replication of data too, so it will handle this part too - a new node? Great, I will now break the data down and spread it across another node and add even more resiliency. But of course, and this is the critical point and difference to peering - we will have the data duplicated across the nodes too - lose one node, thats not a problem, no lost data! This is critical going forward and seriously builds on the resiliency and availability that micro services brings here.

And of course, I talk about scaling up, but you can scale down too - and if you want to make it automated? Now thats starting to get clever!!!

Super Contributor.. Carl_E Super Contributor..
Super Contributor..

Re: "ArcSight Investigate" (RSA Conference announcement)

Cool stuff...

One more question... with Vertica as the data store, is search time range (however misguided only talking about searching is) still based on time of consumption or can users now search for the time indicated in the event itself?

pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: "ArcSight Investigate" (RSA Conference announcement)

You have identified one of the issues with Logger and why it was / is limited in its capabilities around searching. Yes, Logger does index by receipt time and stores the data in the data blocks based on the time that it was received into Logger. If you have batch data coming in, you run the risk of having it across several data blocks and hence takes excessive time to search through. There was a feature added to Logger that did allow searching by receipt time or event time - but it quickly got pulled because it caused all sorts of performance issues (due to the way that CORR works at the back end).

Vertica does allow you to search by any of the time stamp fields - receipt, agent receipt or end time (or start time for that matter) and display data accordingly. So that restriction is gone. There also isnt a performance impact either, so thats sorted too.

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.