Highlighted
Honored Contributor.
Honored Contributor.
1317 views

Automatic updating of active list with csv

I think I've seen an article about this but I can't seem to find it.  Is there a way to update an active list from a csv using a smart connector? or other means?

What I want to do is use a script to dump some information to a csv, then have a smart connecter watch that directory and whenever a new csv shows up (say once a week) read it into an active list; either overwriting what was there or adding to the information already there.

Thanks.

Labels (1)
Tags (1)
0 Likes
Reply
12 Replies
Highlighted
Respected Contributor.
Respected Contributor.

It can be done (we are doing this at present).

1.  Create a flex conenctor that will load your csv file. Distinctively mark your events e.g. bt a custom deviceProduct string..

2. Create a lightweight rule that will wait for your speciffic events and add them to the activelist. Actually this is the only possible action for a lightweight rule.

Hope this helps

M.

0 Likes
Reply
Highlighted
Honored Contributor.
Honored Contributor.

Thanks.  I kinda used your solution along with a powershell script that runs via task scheduler to produce the csv files and a python script that sends each entry in the csv file to a CEF connector.  The rule then adds each entry to an Active List.

Sorry it took so long to acknowledge.

0 Likes
Reply
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Hi,

Kindly, can you show me in details how can I install the flex connector and read from csv file to feed arcsight channel.

I will appreciate to guide me for any document that will help me.

with all my thanks

0 Likes
Reply
Highlighted
Honored Contributor.
Honored Contributor.

I don't use a flex connector... here's my setup:

First I have the following powershell script:

import-module ActiveDirectory

get-aduser -filter * -SearchBase "OU=XXX,DC=YYY" -Properties Description | select SamAccountName, name, Description | export-csv c:\powershell\output\users.csv

(Modify the above to apply to your environment and what you want to capture in the ActiveList.  It is important to make sure you at least catch the username and the full name. For example you could capture only the domain admins, or only remote users or only users from a specific OU.  Depends on what you want to add to the AL.  Also, make sure you set up the AL to receive what you capture.)

I actually have multiple powershell scripts that pull different lists.  Disabled Users, Admin users, Remote Users are some examples that I pull from AD.  Each powershell script dumps into a different csv file in the same output folder.

Then I have have a python script that runs the the powershell scripts, parses the resulting files (i.e. users.csv) and sends each user/info as an event to the ESM.  The ESM has a simple rule that adds each user sent to an AL appropriate for the output file it came from.  Disabled Users go into a Disabled User AL, admin users go into an Admin User AL, etc.  Each AL is configured with a TTL of one day.

First, I must give a lot of credit to the following post and Greg Martin for providing a means of sending events to ArcSight via Python:      

Now, here's the python script:

import csv,os,sys,string,socket,time,subprocess,re

from datetime import datetime,timedelta,data

def syslog(message, level=5, facility=3, host='localhost', port=514):

     sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

     data = '<%d>%s' % (level + facility*8, message)

     sock.sendto(data, (host,port))

     sock.close()

subprocess.call('powershell -ExecutionPolicy Unrestricted C:\powershell\<name of above powershell script.ps>')  # repeat this step for as many powershell scripts as you have written

destination = ip_of_connector_listener

Header = 'CEF:0'

deviceVendor = 'HomeGrown ID'

deviceVersion = '1.0'

for file in os.listdir('c:/powershell/output/'):

     if file.endswith('.csv')

          if file.endswith('disabled_users.csv'):

               deviceProduct = 'Update Disabled Users'

               name = 'Update Disabled User Active List'

          elif file.endswith('admin_users.csv'):

               deviceProduct = 'Update Admin Users'

               name = 'Update Admin User Active List'

          elif file.endswith...... (do this for as many csv files you have created above)

          else: continue

          f = open(file)

          for line in f:

               if re.match(r'\A#Type',line):continue # skip the first line, which is column type. 

               if re.match(r'\A"SamAccountName"',line) # skip the first line, which is column names.

               names = line.split(',') # I could have used the csv module, but I decided to just split the line manually

               extra = " cs1Label=Description cs1=" + names[2].strip().strip('"')

               addon = 'msg=Update Active List duser=' + names[1].strip().strip('"') + ' duid=' + names[0].strip().strip('"')

               CEF = Header +"|"+ deviceVendor +"|"+ deviceProduct +"|"+ deviceVersion +"|0|Update Active List|0| " + addon + extra

               syslog(CEF, host=destination)

I have the above python script configured as a scheduled task that is set to run once a day (I'm in a Windows environment... I guess since it's powershell that goes without saying) .  It runs under a domain account that has admin privs so it can pull info from AD. 

I then have a rule in the ESM that looks for the deviceVendor and deviceProduct keywords and pulls the Destination User ID, Destination User Name and Custom String 1 (where I put the Description) from the event and adds that info to an specific AL (Disabled Users go into the Disabled Users AL, Admin Users into the Admin Users AL, and so on). 

Since the entries in the AL are set with a TTL of 1 day, I always have a fresh and accurate list of who is in each group.

Hopefully I transcribed the above correctly, let me know of any typos and if you have questions and/or need clarification.

Feel free to mod/improve/suggest alternates on the above.

0 Likes
Reply
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Thank so much for that.

0 Likes
Reply
Highlighted
Frequent Contributor.. Frequent Contributor..
Frequent Contributor..

Hi,

my python program is on my laptop and creating csv file. I want to take these event information to an Active List. is there any way to call that csv in arcsight and import these events to acitvelist.

Reagrds
SAAD

0 Likes
Reply
Highlighted
Honored Contributor.
Honored Contributor.

I've not found a way to do this from within arcsight.  Only send each line of the csv to arcsight via a connector.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

To Do this did you have to use Python or was there another way within Arc Sight?   Do you have a couple screen shots of what you had to do?

Regards,

Al McBride Jr.

0 Likes
Reply
Highlighted
Honored Contributor.
Honored Contributor.

I used the python script above. I've seen powershell scripts and perl scripts that do the same thing, but I don't know of a way within ArcSight that will notice a new or an updated CSV and read it DIRECTLY to an Active List.

You may be able to create a flexconnector to watch a directory and read through new created files.  But this is easier IMHO.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

Craig,

Thank you.  Yes, I don't know of a way either to do this within Arc Sight.  Thanks for sharing and great job!  We'll try this out soon.

0 Likes
Reply
Highlighted
Absent Member.
Absent Member.

I will show you one way using CEF. I pull in TOR exit nodes into a list by parsing input and printing CEF to a folder that is monitored by a multi-folder flex connector. Here is the script:

#!/usr/bin/perl

open (OUT, ">/home/arcsight/TOR/TORList.cef");

@torList = `curl -s  --proxy http://IP:PORT --proxy-user USER:PASSWORD http://torstatus.blutmagie.de/ip_list_exit.php/Tor_ip_list_EXIT.csv`;

foreach $node ( @torList ) {

chomp $node;

print OUT "CEF:0|GIS|TOR Exit Nodes|.1|GIS0201410101305|TOR Exit Nodes|0|src=$node\n";

}

I then have a lightweight rule that is looking for  Name="TOR Exit Nodes" The action on this rule writes the fields to an active list. The active list is simply the attacker address.

I noticed that I sort of like CEF as I can't find any examples any where that use a flex anymore to parse a CSV. Here another example of a script to read CSV and print CEF. Same deal as above rule looks for Name="GIS Bit9 Hash Exceptions" and writes to a list.

#!/usr/bin/perl

use Text::CSV;

open (OUT, ">/home/arcsight/Bit9Exception/hashList.cef");

my $csv = Text::CSV->new();

open (INF, "</home/arcsight/fileHash/list.csv") or die "Cat got my tongue";

while (<INF>) {

if ($csv->parse($_)) {

my @columns = $csv->fields();

$columns[0] =~ s/\\\\\?//g;

print  OUT "CEF:0|GIS|GIS Bit9 Hash Exceptions|.1|GIS220420130957|GIS Bit9 Hash Exceptions|0|filePath=$columns[0] fileHash=$columns[1]\n";

}

}

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.