nicholas.hsiao@1 Absent Member.
Absent Member.
2391 views

Threat Intelligence Package - Protect724

=====v1.3 Update 2015/03/07 =====

In my mindset, there are couple Threat Intelligences sources

  • From External Source (Free one)
  • From Internal Source (Base on other technology deployed in company)
  • From Commercial Source (Such as HP ArcSight RepSM)


In v1.3, I add HP TippingPoint (from internal source) into it.  (It might not be that perfect yet).

          I created a ActiveList with possible C&C and Botnet detection filters of TippingPoint.

          So, if anyevent trigger tippingpoint with the following filters, then that could be a C&C/Botnet connection.

       1.png

After that, we have a rule to compare this active list and also real TippingPoint Event, once we trigger event, we could put the source into original IP reputation databse.   In this step, you need to enhance this rule to detect right traffic/source. Currently, I am putting 'Destination Address' into the database.  You might enhance this as well.


         2.png

        3.png


After that, original Threat Intelligence Dashboard could show the new source . You could see 'TippingPoint IRC' as the source (from the following snapshot.).

    4.png

Anyone could help me out other IDS/IPS solution?




=====v1.2 Update 2015/03/06 =====

  1. Collect "Integration Commands"

         

  • Cyber Attack Map - You need to launch an 'external web browser' to see that.
  • Logger Search - ACC - You need to define 'ESM Host', and 'ESM Port', and then input username and password. then you will get a Web Console and embedded in java console

         5.png

  • Open Source Threat Intelligence - Launch Threat Intelligence source URLs directly
  • Reputation Checker - Leverage 3rd vendors tools to check the IP reputation.

          'Cause this is from OpenSource, if you want to confirm with other IP reputation checker tool, you could right click integration commands -> "run command" to launch a web browser inside console.  And input ip there... as below snapshot.  (PS. you need to input IP address yourself)

   2.png3.png

    2. Add one more Dashboard into package  and some enhancement.

       4.png

===== v1.0 =================================================

Hi folks,

   I knew there are a lot of Open Source Threat Intelligence data feeds via perl or some other language.  However, Java is still my favor now.  So... here it is... I will also share the source code here, just in case anyone wanna verify that and modify something by yourselves.

****** Please provide me  more feedback.. including those Free/OpenSource Threat Intelligence URLs, suggestions.. or use cases.

How to use this

  • Deploy the arb file into your ESM 6.8c.
  • Uncompress the zip file into any specific folder.  You will see a .java file (source code) and couple .class file (bytecode)
  • Modify protectti.properties file.   It should be easy for you to understand.
  • open runProtect???.bat or run.Protect???.sh

        Modify the java path and also all paths in that command line.

Save that once you finish editing.

My suggestion is to put this in your FlexConnector (with latest jdk).  In my case, I installed these program in ESM server directly, and then leverage jre of arcsight ESM 6.8c. It's actually jre1.7.

  • Run the program via the bat or sh file you have.

log.syslog.enable - If you enable syslog in properties file, then the program will send syslog to your syslog smartconnector.  That would be much easy in most environment.  Remember to tune idle time, 'cause the program will load everything and then process that fast.  Put idle time to 200 or 300 min-seconds will help to reduce package lost.

log.file.enable - If you enable this, program will write into a log file with CEF format.  You need a FlexConnector Multiple Folder with batch/cef configuration there.   Program will write the file into a .temp file until it finishs all writing process.  Then it will rename it to .cef.log file.  So, remember to configure flexconnector widecard to read *.log file only.


log.console.enable - Normally, this is for debug purpose.  


  • Define schedule job (windows) or cron table (in linux) to load it by schedule.

My suggestion is to run it once per day.  Which should be enough.

    PS. In the zip file, I also attached cef log file.. it's about 5MB.  You could try to leverage this file to test your package also.

   1.png

    3.png4.png2.png

5.png

    edit protectti.properties file.

6.png

   Run crontab -e in unix, and define something like the following.

7.png

Nicholas

Message was edited by: Nicholas Hsiao Update original arb file to v1.2.

17 Replies
tkachouba Trusted Contributor.
Trusted Contributor.

Re: Threat Intelligence Package - Protect724

Great package Nicholas! You've been on a roll!

0 Likes
MarcNZ1
New Member.

Re: Threat Intelligence Package - Protect724

Hi Nicholas,

This looks pretty cool.

Any word on whether this works on Express?

0 Likes
nicholas.hsiao@1 Absent Member.
Absent Member.

Re: Threat Intelligence Package - Protect724

Marc,

I don't have time to create Express version yet.

But you could still use the zip file, it's the data feed program.

I will check if I have time to create or maintain an Express version. That might means I need to maintain two versions.

Are you using Express 3 or Express 4?


Nicholas

0 Likes
Markl Trusted Contributor.
Trusted Contributor.

Re: Threat Intelligence Package - Protect724

Great package Nicholas!!!


I´m using AExpress 5. I suppose that that it not works on Express 5, no?


Kind Regards,

Be Water My Friend
0 Likes
nicholas.hsiao@1 Absent Member.
Absent Member.

Re: Threat Intelligence Package - Protect724

Wait... Express 5.0 ?

I knew Express has 4.0, and it's my first time to know 5.0.  Could you show your console version to me ?

Nicholas

0 Likes
tkachouba Trusted Contributor.
Trusted Contributor.

Re: Threat Intelligence Package - Protect724

I haven't been able to install the package but do you have a full list of all the Threat Intelligence sources your script is pulling from?  I saw a couple in the screenshots but was interested in a comprehensive list.

0 Likes
nicholas.hsiao@1 Absent Member.
Absent Member.

Re: Threat Intelligence Package - Protect724

Hi,

    If you download the program, open the properties file. It's all in the properties file.

Best regards,

Nicholas

0 Likes
Jurgen
Visitor.

Re: Threat Intelligence Package - Protect724

Hi Nicholas,

Great post thank you for sharing,

The problem i see with these types of threat intel solutions, is that they import the information through a rule into an activelist. Why didn't you make a Network Model import connector that picks up the ip lists and converts them into a xml/.arb formatted file that imports automaticly as an activelist inside the ESM? (RepSM works like this) I don't want to load unnecessary triggered rules on my environment.

Kind regards,

Jurgen

0 Likes
nicholas.hsiao@1 Absent Member.
Absent Member.

Re: Threat Intelligence Package - Protect724

Jurgen,

    Thanks.  That's a good idea.  If customer already perform 'network modeling', then it should be easy for you to exclude those list from this package.  How do you think ?

    If we created another activelist in ESM, then I need to maintain that activelist... and meanwhile, user still have network modeling and asset modeling... so, put asset modeling or network modeling into the rule as exclude list might be better... let me know more inputs plz.

Best regards,

Nicholas

0 Likes
Jurgen
Visitor.

Re: Threat Intelligence Package - Protect724

Hi Nicholas,

There are several options for adding csv files into arcsight:

Add csv files into an activelist:

1. Create a custom archive file and use “arcsight archive” (not preferred)

2. ESM Console and right click on the AL and import (not preferred)

3. Send events to ESM via a flex connector and write a rule that populates the values to AL (not preferred)

4. Import Active Lists (AL) automaticly using a flexconnector/velocity templates.(preferred)

I was referring to building a flex connector which does for example: file reading. Which will pick up csv files and maps them with a velocity template to a activelist formatted xml file. Only problem is that in the past this only worked under some old 5.1.x smartconnector.

But now i see the new RepSM Smartconnector using the same technique but under a 7.0+ smartconnector. Is this possible again?

See the following presentation for more technical details:

There a How to written for creating the activelist importer using the ArcSight Smartconnector 5.1.7

https://protect724.hp.com/servlet/JiveServlet/download/39548-9612/ActiveListImportConnector_517.rtf

If you look into the following connector guide:

It uses .vm files to map the custom string fields just like the technique in 5.1.7 possibly use this connector for importing the lists used from the ip sources you used?

Kind regards,

Jurgen

Vini Acclaimed Contributor.
Acclaimed Contributor.

Re: Threat Intelligence Package - Protect724

Hi ,

I have to agree with , using a model import connector is by far the best approach.

Sending the intelligence via CEF to a connector works but it can become noisy and it is a little bit messy in my opinion.

Is it possible to add more thereat intel lists to your tool? Or have you hardcoded the lists in it?

Regards,

Vini

nicholas.hsiao@1 Absent Member.
Absent Member.

Re: Threat Intelligence Package - Protect724

Hi and ,

    Cool . ! And I think that would be interesting also.

    Let me see what I could do to modify my program for this.  Need more time for this.   Once I done, I will release new version again and let me know if you could try that for me also.

Nicholas

Highlighted
hpgbrice1 Respected Contributor.
Respected Contributor.

Re: Threat Intelligence Package - Protect724

Quick question! is the model import connector part of the regular connector package?

thanks,

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Threat Intelligence Package - Protect724

No, Model Import Connectors are not a part of the standard SmartConnector install, they are separate binaries.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.