Highlighted
konrad.kaczkows1
New Member.
794 views

arc_import_al.zip

Active List import script (PYTHON) - Version 0.7

Limitations:

csv import file must have the same column order as Active List

National language characters limited only to english prntable characters + some special characters [ list in table bellow ]

Example:

Active List:

IP,hostName

csv:

#IP,hostName

10.11.12.13,test.company.com

Fixed special character encoding in active list import over XML

Symbol

Description

ArcSight Active List MAP in XML

"Double quotes (or speech marks)"
&Ampersand\A
+Comma\C
<Less than (or open angled bracket)\L
>Greater than (or close angled bracket)\G
\Backslash\\
|Vertical bar\|

Fixed temporary files removing from /tmp directory - if AL was huge can use all /tmp space

Fixed verification of access to archive.log

[ tree = ElementTree.parse(TEMP_FILE) ...  IOError: [Errno 2] No such file or directory: '/tmp/AL_IN_ESM_INVALID' ]

Fixed TEMP_FILE access verification

- if no write rights generate new variable for TEMP_FILE

- Display line in CSV file with wrong (to high/low) number of columns

Things to add:

  • check capacity of Active List and compare to import file

THIS SCRIPT IS AFTER BETA TESTS on RedHat 6.5 with Python 2.6

Script have to executed on HP ArcSight Manager.

Test scenario at the end of post How does it work:

  • check if import csv file exist
  • check connectivity with ESM (validate if available, if password is correct and account is not blocked)
  • check if Active List exist on ESM  [ use /opt/arcsight/manager bin/arcsight archive -action export command ]
  • check if number of columns from Active List is the same as number of columns from csv file
  • prepare xml file/files to import
  • import xml file   [ use /opt/arcsight/manager/bin/arcsight archive -action import command ]
  • if syslog server is specified send CEF events to syslog server
  • if option -c was set - delete successfully imported files - otherwise change name to *.xml.done

Execution:

./arc_import_al.py -r 20 -l "/All Active Lists/BCC/al_IP" -f /opt/asset_import/al_IP.csv -m ManagerName -u UserName -p UserPass -s10.0.1.33 -P 514 -d -c

where parameters are:

REQUIRED

  -r 10                      [ numers of rows per single import ]

  -l Actve List           [ avtive list full URI in format "/All Avtive Lists/customer/malware" ]

  -f filename             [ if file contains space - use filename in " QUITAS " ]

  -m ESM manager   [ HP ArcSight ESM manager FQDN ]

  -u ESM user          [ HP ArcSight ESM import user ]

OPTIONAL

  -p ESM user pass  [ HP ArcSoght ESM user password ]

  -s Syslog Server    [ Syslog server ]

  -P Syslog Port       [ Syslog server port ]

  -c                          [ clean (delete) imported files ]

  -d                          [ debugging - display detailed information from processing ]

        

ADDITIONAL PARAMETERS

-h  [ help ]

-v  [ version ]

# Possible reconfiguration options:

#

# Place where are stored xml files for import: line 66

# export_dlobal_dir = "/opt/asset_import/active list"

#

# Device interface name: line 89

# CEF_dvc = get_ip('eth0')

Test scenarios

Test scenario 1:

- Active List 1 [ size: 400000, columns: 4, Type: Event-based ]

    Import rows: 331776

    Batch size ( -r ) : 100000

    Time of import :

       - processing time: 20 s

       - importing: 4 x 12 s

Test scenario 2:

- Active List 2 [ size: 1200000, columns: 1, Type: Field-based ]

    Import rows: 1100000

    Batch size ( -r ) : 200000

    Time of import :

       - processing time: 95 s

       - importing: 6 x 45 s

When Batch Size [ -r ] was set to 300k import failed.

Bellow ESM Active Channel

arc_import_al_ESM_channel.png

Bellow command for import attached csv file with imported arb.

!!!!! /opt/asset_import/ directory have to be created before !!!!!


/opt/scripts/arc_import_al.py -r 50000 -l "/All Active Lists/HPLAB/arc_import_al" -f /opt/asset_import/arc_import_al.csv -m ESM_NAME -u USER  -s IP_OF_SYSLOG_SERVER -P PORT_OF_SYSLOG_SERVER

K.

K.
Labels (3)
Tags (3)
2 Replies
StevenvandeBraak Outstanding Contributor.
Outstanding Contributor.

Re: arc_import_al.zip

Hi Konrad, great work!

Little thing to fix:  if you want to feed in integers, the numbers don't match in the activelist

Apparently ESM expects hex values when inserting integers into the AL

With strings it works fine but with numbers (integers in the AL), the script should first convert the numeric values into HEX.

I haven't tested with doubles and longs and dates yet

1) Test with a number as string field in AL (file: http-error-codes.csv)

Screen Shot 2015-07-08 at 22.32.44.png

2) Test with a number as integer in AL (same file)

Screen Shot 2015-07-08 at 22.32.03.png

3) Test with number as HEX in the CSV and AL field defined as integer (file: http-error-codes-hex.csv)Screen Shot 2015-07-08 at 23.02.14.png

http-codes.csv

#ErrorCode,Description

101,Switching Protocols

200,OK

201,Created

202,Accepted

203,Non-Authoritative Information

204,No Content

205,Reset Content

206,Partial Content

400,Bad Request

401,Unauthorized

402,Payment Required

403,Forbidden

404,Not Found

405,Method Not Allowed

406,Not Acceptable

407,Proxy Authentication Required

408,Request Timeout

409,Conflict

410,Gone

411,Length Required

412,Precondition Failed

413,Request Entity Too Large

414,Request-URI Too Large

415,Unsupported Media Type

416,Requested Range Not Satisfiable

417,Expectation Failed

500,Internal Server Error

501,Not Implemented

502,Not Implemented

503,Service Unavailable

504,Gateway Timeout

http-codes-hex.csv


#ErrorCode,Description

65,Switching Protocols

c8,OK

c9,Created

ca,Accepted

cb,Non-Authoritative Information

cc,No Content

cd,Reset Content

ce,Partial Content

190,Bad Request

191,Unauthorized

192,Payment Required

193,Forbidden

194,Not Found

195,Method Not Allowed

196,Not Acceptable

197,Proxy Authentication Required

198,Request Timeout

199,Conflict

19a,Gone

19b,Length Required

19c,Precondition Failed

19d,Request Entity Too Large

19e,Request-URI Too Large

19f,Unsupported Media Type

1a0,Requested Range Not Satisfiable

1a1,Expectation Failed

1f4,Internal Server Error

1f5,Not Implemented

1f6,Not Implemented

1f7,Service Unavailable

1f8,Gateway Timeout

konrad.kaczkows1
New Member.

Re: arc_import_al.zip

Steven,

try to export al by arcsight archive -m manager -u user -uri "/All Active Lists/AL with HEX" -f /tmp/al_with_hex.xml

and see what You will have in xml.

For example:

   <ActiveList id="Hqo-PzEkBABCXXoAG74V4lA==" name="Ascii_chars" action="insert" >

      <activeListEntries>

         <list>

            <map>

               <count>1</count>

               <creationTime>1416480734000</creationTime>

               <lastModifiedTime>1416480734000</lastModifiedTime>

            </map>

            <map>

               <count>1</count>

               <creationTime>1416480766000</creationTime>

               <lastModifiedTime>1416480766000</lastModifiedTime>

               <values>

                  <list>

                     <string>!</string>

                     <string>Exclamation mark</string>

                  </list>

               </values>

            </map>

Then You will see how ESM encode that.

This is walkaround, whe I'll have some time I'll do that.

K.

K.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.