chrispols Absent Member.
Absent Member.

Re: Importing .txt files into active list automatically

Hi JP,

I see you updated the GIT again, I downloaded and updated. Results:

Processing Autoshun list from http://autoshun.org/files/shunlist.csv

Comparing 1067 downloaded to 0 cached lines

1067 new, 0 deleted, 0 unchanged lines

Sent 39354 New, 0 deleted, and 0 unchanged entries to 10.1.1.XX:514

one thing with this new version, if you don't have the Maxmind database cached, it fails with the following, I copied the DB across and then it worked fine. With the old version, it would pull the DB if not found.

Processing alienvault from https://reputation.alienvault.com/reputation.data

Comparing 12536 downloaded to 0 cached lines

12536 new, 0 deleted, 0 unchanged lines

Traceback (most recent call last):

  File "threataggregator.py", line 692, in <module>

    start(feeds, _db_add, _db_del, _db_equal)

  File "threataggregator.py", line 638, in start

    build_db(i['type'], i['url'], i['description'], db_add, db_del, db_equal)

  File "threataggregator.py", line 544, in build_db

    db_add.append(alienvault(url, compare_add))

  File "threataggregator.py", line 487, in alienvault

    repdb.add(ip, url, desc, priority=prio, reputation=rel)

  File "threataggregator.py", line 123, in add

    reader = get_geo_db()

  File "threataggregator.py", line 320, in get_geo_db

    print("Maxmind database not cached. Attempting to pull from {}".format(url))

ValueError: zero length field name in format

But I had a look in ESM, and looks good so far, will just need to verify, but awesome work, Thanks!

0 Likes
Ruslan Mikhalov Honored Contributor.
Honored Contributor.

Re: Importing .txt files into active list automatically

Hi everyone,

Actually there was a way to import entries directly to ActiveList using standard ArcSight Connector without EPS/rules/3rd-party tools.

I've tested it. And it worked.

If you still interested, I can check my notes and find this.

Regards.

0 Likes
seniorj@bennett Absent Member.
Absent Member.

Re: Importing .txt files into active list automatically

Resolved the unicode error.

0 Likes
seniorj@bennett Absent Member.
Absent Member.

Re: Importing .txt files into active list automatically

Hmm, small minor typo there that was  introduced when i was migrating to python3 support, Thanks, that was fixed too!

0 Likes
chrispols Absent Member.
Absent Member.

Re: Importing .txt files into active list automatically

Hi Ruslan,

I'm interested in having a look, if you could share, that would be great.

Thanks

Chris

0 Likes
jefferyhamstra Super Contributor.
Super Contributor.

Re: Importing .txt files into active list automatically

Has anyone been able to schedule the threataggregator.py to run via cronjob? I can run the pyhton script fine, but when I run it via cronjob, it starts the job, gets the alientvault update downloaded and throws a maxmiund cache error.

I'm assuming this cronjob doesn't have the ability to red  mysql or modify the cache, but am not sure?

0 Likes
kgraham Super Contributor.
Super Contributor.

Re: Importing .txt files into active list automatically

I'm interested.  Watching

0 Likes
Ruslan Mikhalov Honored Contributor.
Honored Contributor.

Re: Importing .txt files into active list automatically

Hi Kim, Hi Chris,

I've got this doc (attached), starting page 25 - describes the way to do it. I tried it myself and it worked well.

But you need ArcSight Connector 5.1.7 for that. I didn't manage to make it work with higher versions of connectors.

Regards.

0 Likes
Outstanding Contributor.. Pushpendra_Rathi Outstanding Contributor..
Outstanding Contributor..

Re: Importing .txt files into active list automatically

It works with latest connector as well...

0 Likes
Highlighted
Outstanding Contributor.. Pushpendra_Rathi Outstanding Contributor..
Outstanding Contributor..

Re: Importing .txt files into active list automatically

Such a long thread, making it so complicated by script and all 😞

Why don't you just use the Model Import Connector guys..

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Importing .txt files into active list automatically

Please tell us more, this sounds interesting.

0 Likes
chrispols Absent Member.
Absent Member.

Re: Importing .txt files into active list automatically

Hi Jeff,

I found it to be an issue with the "cache" folder reference. I made an explicit folder link for /opt/arcsight/scripts/threat-aggregator/cache/ instead of just cache/ and it resolved it. My cron works perfectly fine now. Give that a go.

Chris

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.