Highlighted
bossemi Regular Contributor.
Regular Contributor.
1614 views

Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Dear All,

I need to find a way to export the rule definitions of all our rules in the ArcSight ESM6.91 content in a structured format into a HTML or XML file.
Currently I can achieve this manually via the rule context menue. When I select a rule I can rightclick the "print rule definition..." option and the Print Preview Window opens. Then I can select "export to HTML" and get a dialog window which wants me to assign a filename. This works quite okay for a single rule but it is very labor intensive work, because we always have to assign a unique filename.

To cut a long story short. I'm looking for a more convenient way to export this information. Preferably in a fully automated manner. Is there any onboard maintenance script available that might be useful, or are there any batch scrpts available I can use?

Note: The "export to html.." function which is directly available from the context menue doesn't fit, because the output is more consice and lacking some essential information that we need.

Kind regards,

Michael

1 Solution

Accepted Solutions
Micro Focus Expert
Micro Focus Expert

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hi Michael,

Apologies for my typo there -  indeed the table is arc_rules as you discovered.   I have now corrected my original post.

For the aggregation information, i do not believe we store this as neat columns in any table, but you can infer the aggregation going on from the rules definition.

The threshold information is always printed in the definition even if it not actually being used by the rule.  For example, a LightWeight rule does not use aggregation, but the following will still be printed with its definition:

<WhereClause TimeWindowSize="2" TimeUnit="Minute" Threshold="1">

Similarly, a rule that that is only using non-threshold actions (such as "On Every Event") is also not using the aggregation per-se,  but the definition for aggregation remains.  Such a standard event may still be using aggregation information from the aggregation panel to forward the event columns through to the correlation event etc.

Apart from that, you can see the aggregation information in the rules like this:

For the aggregation that requires aggregation on "unique" fields, you will see a <SelectClause> containing one or more lines with the "Distinct" tag.

For the aggregation that requires aggregation on identical fields, you will see those fields as GROUP BY enclosed by the <GroupByClause> tags.

So, for an example.  Where the export As HTML for an rule shows this:

Aggregation Summary:
Aggregate if at least 2 matching conditions are found within 1 Minutes AND these event fields are unique (event1.Bytes In, event1.Bytes Out) AND these event fields are the same (event1.Source Address, event1.Message, event1.Attacker Host Name, event1.Target Zone Resource, event1.Source Zone Resource, event1.Device Vendor, event1.Target Host Name)


The rule definition will show this for the first part matching on unique fields:

<SelectClause>
      <SelectColumn>
        <Expression>
          <Function Name="COUNT" Distinct="Yes" />
          <Variable TableAlias="event1" Column="bytesIn" />
        </Expression>
      </SelectColumn>
      <SelectColumn>
        <Expression>
          <Function Name="COUNT" Distinct="Yes" />
          <Variable TableAlias="event1" Column="bytesOut" />
        </Expression>
      </SelectColumn>
    </SelectClause>

...and this for the second part matching on identical fields:

   <GroupByClause>
      <Variable TableAlias="event1" Column="sourceAddress" />
      <Variable TableAlias="event1" Column="message" />
      <Variable TableAlias="event1" Column="attackerHostName" />
      <Variable TableAlias="event1" Column="targetZoneResource" />
      <Variable TableAlias="event1" Column="sourceZoneResource" />
      <Variable TableAlias="event1" Column="deviceVendor" />
      <Variable TableAlias="event1" Column="targetHostName" />
    </GroupByClause>

The aggregation window and threshold information is at the beginning of the WhereClause
:
<WhereClause TimeWindowSize="1" TimeUnit="Minute" Threshold="2">

So, it's not much fun having to parse all those items out of the rule definition, but I do not believe we are storing them anywhere else. Sorry about that!

I hope this is somehow useful Michael,

Best regards,
Darren

ArcSight Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
7 Replies
Micro Focus Expert
Micro Focus Expert

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hello Michael,

I have taken a look around (hoping to find some hidden tools) that might help here, but there appears to be nothing that takes the level of information detail provided by the "export to HTML" and batches it up nicely for a group of rules or indeed for all rules.

There is a possibility to export all rule definitions to an xml file through the mySQL command line tool as well.  The result is one log xml file containing all rules.  It would be possible to select just those rules that are active/enabled or simply dump all of them.  Exporting this way would not provide you with the nice extra format info that you get when exporting from the console.

What level of detail do you need?  How do you intend to use that information afterwards?  If XML is fine, and you're happy to post-format yourseff then dumping directly from the database may be enough for you.

WARNING

Please DO NOT update or insert lines into the mySQL database. Doing so could severely corrupt the database and may place it into an irrecoverable state.

We strongly recommend that a good, known backup is available before accessing the database manually. This is to protect against accidental erroneous updates.



Here is a method to get the rule definitions for all active rules to be output to a file /tmp/rule.output

1) /opt/arcsight/logger/current/arcsight/bin/mysql -uarcsight -p arcsight

Enter the database password when requested

2) mysql> tee /tmp/rule.output
3) mysql> select def from arc_rules where active=1\G
4) mysql > notee
5) mysql> \q

If the output works for you, then you could place this SQL into a file (for example /home/arcsight/dumpRules.sql) and then call it from the command line, or a cron job etc as required, outputting to a second file.  The only issue here is that you will need to provide the password in plaintext as part of the command line if you want to automate this (-p<password>). Basic example:

/opt/arcsight/logger/current/arcsight/bin/mysql -u arcsight -pmypassword arcsight < /home/arcsight/dumpRules.sql > /home/arcsight/dumpRules.out


Does that kind of output work for you?
Edited to correct arc_rules table typo

ArcSight Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
0 Likes
bossemi Regular Contributor.
Regular Contributor.

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hi Darren,

thank you very much for your quick reply. We had to slightly adjust the given sql command, then it worked.

select def from arc_rules where active=1\G

It's probably table arc_rules instead of arc_rule.

The output format itself works fine with me. We could parse out the information we need.
Unfortunately we are looking in particular for the aggregation summary information which is not part of the output of the previous sql command. Following an example from the manual html export  

Aggregation Summary</th>
</tr>
<tr>
<td width="100%">
Aggregate if at least <b>1</b> matching conditions are found within <b>2 Seconds
</b> AND these event fields are the same (<b>event1.Destination Address, event1.
Device Asset Resource, event1.Device Product, event1.Device Address, event1.Devi
ce Vendor, event1.Device Zone, event1.Device Host Name, event1.Destination Zone
Resource, event1.Device Zone Resource, event1.Destination Service Name, event1.C
ustomer, event1.dstAssetReference, event1.dstHostName, event1.Customer Resource<
/b>)</td>
</tr>
</table>

Is there a way to make this information also available?

Kind regards,

Michael

 

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hi Michael,

Apologies for my typo there -  indeed the table is arc_rules as you discovered.   I have now corrected my original post.

For the aggregation information, i do not believe we store this as neat columns in any table, but you can infer the aggregation going on from the rules definition.

The threshold information is always printed in the definition even if it not actually being used by the rule.  For example, a LightWeight rule does not use aggregation, but the following will still be printed with its definition:

<WhereClause TimeWindowSize="2" TimeUnit="Minute" Threshold="1">

Similarly, a rule that that is only using non-threshold actions (such as "On Every Event") is also not using the aggregation per-se,  but the definition for aggregation remains.  Such a standard event may still be using aggregation information from the aggregation panel to forward the event columns through to the correlation event etc.

Apart from that, you can see the aggregation information in the rules like this:

For the aggregation that requires aggregation on "unique" fields, you will see a <SelectClause> containing one or more lines with the "Distinct" tag.

For the aggregation that requires aggregation on identical fields, you will see those fields as GROUP BY enclosed by the <GroupByClause> tags.

So, for an example.  Where the export As HTML for an rule shows this:

Aggregation Summary:
Aggregate if at least 2 matching conditions are found within 1 Minutes AND these event fields are unique (event1.Bytes In, event1.Bytes Out) AND these event fields are the same (event1.Source Address, event1.Message, event1.Attacker Host Name, event1.Target Zone Resource, event1.Source Zone Resource, event1.Device Vendor, event1.Target Host Name)


The rule definition will show this for the first part matching on unique fields:

<SelectClause>
      <SelectColumn>
        <Expression>
          <Function Name="COUNT" Distinct="Yes" />
          <Variable TableAlias="event1" Column="bytesIn" />
        </Expression>
      </SelectColumn>
      <SelectColumn>
        <Expression>
          <Function Name="COUNT" Distinct="Yes" />
          <Variable TableAlias="event1" Column="bytesOut" />
        </Expression>
      </SelectColumn>
    </SelectClause>

...and this for the second part matching on identical fields:

   <GroupByClause>
      <Variable TableAlias="event1" Column="sourceAddress" />
      <Variable TableAlias="event1" Column="message" />
      <Variable TableAlias="event1" Column="attackerHostName" />
      <Variable TableAlias="event1" Column="targetZoneResource" />
      <Variable TableAlias="event1" Column="sourceZoneResource" />
      <Variable TableAlias="event1" Column="deviceVendor" />
      <Variable TableAlias="event1" Column="targetHostName" />
    </GroupByClause>

The aggregation window and threshold information is at the beginning of the WhereClause
:
<WhereClause TimeWindowSize="1" TimeUnit="Minute" Threshold="2">

So, it's not much fun having to parse all those items out of the rule definition, but I do not believe we are storing them anywhere else. Sorry about that!

I hope this is somehow useful Michael,

Best regards,
Darren

ArcSight Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
bossemi Regular Contributor.
Regular Contributor.

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hi Darren,

I've  build the parser. It works. Thanks for your support.

Kind regards,

Michael

 

ibaibhav
Established Member.

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution
Hi Michael,

Could you please help me with parser part? I have exported the rules in XML but not sure how to export in excel sheet or presentable format .
Thanks & Regards,
Baibhav Anand
0 Likes
bossemi Regular Contributor.
Regular Contributor.

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hi ibaibhav,

I've created a very simple parser for a very specific usecase, which is way beyond the scope of this forum. I've replied privately to you with a short description of the parser and the usecase.

KR

Michael

0 Likes
Respected Contributor.. aniket.govilkar Respected Contributor..
Respected Contributor..

Re: Question: How to export rule definition automatically in HTML or XML formatted file

Jump to solution

Hello Michael,

 

With reference to the above  post , can you  please share converter you created from XML to text on with me too . It would be great help and we need for one of huge assesment which we have to perform. 

 

If possible, can you  share on email address given . It would be great help . 

Thanks &  Regards,

Dhruv Shah

dhruv.shah@niiconsulting.com

 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.