This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

L1-Threat Intelligence - Indicators and Warnings

This is the official forum for discussing the basic ArcSight Activate L1-Threat Intelligence - Indicators and Warnings package, as described in the Activate Wiki.

Version 1.1.0.0 TI: (L1-Threat_Intelligence_-_Indicators_and_Warnings_1.1.0.0.arb)

Modified Resources:

/All Rules/ArcSight Activate/Solutions/Threat Intelligence/Indicators and Warnings/Populate Suspicious Address List

--
Prentice S. Hayes
Principal Product Manager | Security Operations
OpenText 

LinkedIn: 

https://www.linkedin.com/in/prenticeshayes/ 

Website: 

https://www.opentext.com/

Parents
  • Hi Community,

    I've install a STIX/TAXII server and tried polling for 1 specific collection. It's currently still running and it has been more than 1 hour now.

    1. Has anyone tried using a Stix/Taxii polling? How long does it usually take to finish one collection?
    2. Is there a minimum specs requirement for the server? Does this affect the performance of the polling?

    Thanks! :) 

     

  • Actually it depends on the data. which collection are you trying to download? It was working normal when I was using it.

     

  • If you don't specify a begin date, it pulls all the data. If the data is big, it takes long time to complete. I used the following command and it was completed in seconds.

    arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.Abuse_ch --today --output /tmp/ --debug

    Instead of --today option you can use "--days 7". it pulls data from last 7 days.

  • Hi Mr. Eugene,

    Thank you for that information, we will schedule it to run daily.

    But we are having trouble with the flex now. I used the flex config file that was with the "arcsight_stix_taxii.zip" file there seems to be no problem with the flex script. the csv file says it was processed but it is not on our ESM.


    I took a look at the agent.log and saw this:

    111111111111111.PNG


    Here are the files - it says that it was processed:

    22222.PNG


    here is the flexconn script that I used:

    delimiter=,

    text.qualifier="

    comments.start.with=\#

    trim.tokens=true

    contains.empty.tokens=true

     

    token.count=11

     

    token[0].name=otype

    token[0].type=String

    token[1].name=observable

    token[1].type=String

    token[2].name=indicatorType

    token[2].type=String

    token[3].name=firstdetecttime

    token[3].type=String

    token[4].name=lastdetecttime

    token[4].type=String

    token[5].name=score

    token[5].type=String

    token[6].name=confidence

    token[6].type=String

    token[7].name=producer

    token[7].type=String

    token[8].name=rdata

    token[8].type=String

    token[9].name=description

    token[9].type=String

    token[10].name=altid

    token[10].type=String

     

    event.name=__stringConstant("Collective Intelligence Feed")

    event.deviceFacility=__toLowerCase(otype)

    event.deviceSeverity=__toLowerCase(confidence)

    event.message=observable

    event.deviceCustomDate1Label=__stringConstant("First Detected Time")

    event.deviceCustomDate1=__createSafeLocalTimeStamp(firstdetecttime,"yyyy-MM-dd HH:mm:ss Z")

    event.deviceCustomDate2Label=__stringConstant("Last Detected Time")

    event.deviceCustomDate2=__createSafeLocalTimeStamp(lastdetecttime,"yyyy-MM-dd HH:mm:ss Z")

    event.requestUrl=__ifTrueThenElse(__contains(otype,"url"),observable,)

    event.sourceAddress=__oneOfAddress(__ifTrueThenElse(__contains(otype,"ipv4"),observable,))

    event.deviceCustomIPv6Address1=__stringToIPv6Address(__ifTrueThenElse(__contains(otype,"ipv6"),observable,))

    event.sourceDnsDomain=__ifTrueThenElse(__contains(otype,"fqdn"),__toLowerCase(observable),)

    event.sourceUserName=__ifTrueThenElse(__contains(otype,"email"),observable,)

    event.fileHash=__ifTrueThenElse(__contains(otype,"md5"),observable,)

    event.fileHash=__ifTrueThenElse(__contains(otype,"sha1"),observable,)

    event.fileHash=__ifTrueThenElse(__contains(otype,"sha256"),observable,)

    event.fileHash=__ifTrueThenElse(__contains(otype,"sha512"),observable,)

    event.fileHash=__ifTrueThenElse(__contains(otype,"uuid"),observable,)

    event.deviceCustomNumber1Label=__stringConstant("Score")

    event.deviceCustomNumber1=__safeToRoundedLong(score)

    event.deviceCustomNumber2Label=__stringConstant("asn")

    event.deviceCustomNumber2=__oneOfLong(asn)

    event.deviceCustomString1Label=__stringConstant("Sources")

    event.deviceCustomString1=__toLowerCase(producer)

    event.deviceCustomString2Label=__stringConstant("Reference")

    event.deviceCustomString2=altid

    event.deviceCustomString3Label=__stringConstant("Indicator Types")

    event.deviceCustomString3=indicatorType

    event.deviceCustomString4Label=__stringConstant("Related Data")

    event.deviceCustomString4=rdata

    event.deviceCustomString5Label=__stringConstant("Description")

    event.deviceCustomString5=description


    event.deviceProduct=__stringConstant("CIF")

    event.deviceVendor=__getVendor("Threat Intel")

    event.deviceVersion=__getVendor("2.1")


    Thanks for the help.

  • First of all, check agent.out.wrapper.log and search for "First event" lines.  If you see something like "First event from [Threat Intel|CIF|..... ] received, it means the parsing is OK. Then check your destination settings on the connector and search relevant logs on the ESM.

    If you don't see "Firs event" message in the logs, you need to check if there is a parsing issue.

  • Hi Mr. Eugene,

    We used a regex flex script and it works. i dont know what the problem of the delimted flex script.

    Thanks.

     

  • Hi Mr. Eugene and Community,

    Have anyone experienced this error?

    We are trying to poll on Hailataxii.com and the collection name is "EmergingThreats_rules".

    After a few minutes -> The polling stops -> then a csv file was written on the output folder but has no entry -> then an error prompts. (see screenshot below)

    error.png


    This is what the log file says:

    2018-12-12 10:39:12,586 : arcsight_stix_taxii : DEBUG : Error occurred while running client: list index out of range
    Traceback (most recent call last):
    File "c:\python27\lib\site-packages\arcsight_stix_taxii\client.py", line 837, in main
    related_objects=args.related_objects)
    File "c:\python27\lib\site-packages\arcsight_stix_taxii\client.py", line 206, in run_poll
    csvout.write(stix_object, **csv_row_options)
    File "c:\python27\lib\site-packages\arcsight_stix_taxii\storage\local.py", line 489, in write
    for otype, row in self.rows(stixobject, **kwargs):
    File "c:\python27\lib\site-packages\arcsight_stix_taxii\storage\local.py", line 359, in rows
    type_ = str(malware_instances.types[0])
    File "c:\python27\lib\site-packages\mixbox\typedlist.py", line 79, in __getitem__
    return self._inner.__getitem__(key)
    IndexError: list index out of range


    Thank you for the support! :)

  • Hi Josh,

    What version of the client do you use?

    arcsight-taxii-client -v

     

    Thanks,

    Bart

  • First, it's mr_ergene, not eugene :D

    I can poll the feed using following command(limited the data for 7 days and enabled the debug option on the command):

    arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.EmergineThreats_rules --days 7 --output /tmp/ --debug

    My client version is 1.1.3

  • Hello mr_ergene,

    Just to let you know, you are polling the collection guest.EmergineThreats_rules and Josh is polling guest.EmergingThreats_rules

    Both are valid on hailataxii, but I don't know if both collections are the same.

    When I poll both collections and using --days 356, I don't reveice any data can you confirm if this collection is empty, since you are using --days 7

    thanks

     

  • Lol, both collections don't have any data for the last year. Probably those collections are not being maintained and have dirty data.

  • Ok thanks.

    Acording to hailataxii.com the last update was on Fri May 25 15:18:06 2018 UTC. I checked guest.Abuse_CH and this collection is still up to date.

     The latest version of the ArcSight STIX/TAXII client (v 2.0.0) will have a fix for the error you have. 

    The ArcSight STIX/TAXII version 2 will be soon available, and will also support STIX/TAXII 2.x

     

Reply
  • Ok thanks.

    Acording to hailataxii.com the last update was on Fri May 25 15:18:06 2018 UTC. I checked guest.Abuse_CH and this collection is still up to date.

     The latest version of the ArcSight STIX/TAXII client (v 2.0.0) will have a fix for the error you have. 

    The ArcSight STIX/TAXII version 2 will be soon available, and will also support STIX/TAXII 2.x

     

Children