Highlighted
dragoslungu Absent Member.
Absent Member.
968 views

XML Flex Connector generates identical events

Hello,

I'm trying to create a XML Flex Connector for NIST Vulnerability data feed (especially for the CVSS data in it) and I'm stuck in an odd Connector behaviour: the file is parsed and events are generated but all events are identical. The token data is extracted from the first XML trigger node and it's like the parser does not reset the token values for each trigger node and it does not read the token values from each node.

Sample XML (2 nodes, simpified ) from the file available at : http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-recent.xml

<?xml version='1.0' encoding='UTF-8'?>
<nvd xmlns:cpe-lang="http://cpe.mitre.org/language/2.0" xmlns:cvss="http://scap.nist.gov/schema/cvss-v2/0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:patch="http://scap.nist.gov/schema/patch/0.1" xmlns="http://scap.nist.gov/schema/feed/vulnerability/2.0" xmlns:vuln="http://scap.nist.gov/schema/vulnerability/0.4" xmlns:scap-core="http://scap.nist.gov/schema/scap-core/0.1" pub_date="2011-11-01T01:00:00" nvd_xml_version="2.0" xsi:schemaLocation="http://scap.nist.gov/schema/patch/0.1 http://nvd.nist.gov/schema/patch_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 http://nvd.nist.gov/schema/scap-core_0.1.xsd http://scap.nist.gov/schema/feed/vulnerability/2.0 http://nvd.nist.gov/schema/nvd-cve-feed_2.0.xsd">
  <entry id="CVE-2009-0900">
    <vuln:cve-id>CVE-2009-0900</vuln:cve-id>
    <vuln:cvss>
      <cvss:base_metrics>
        <cvss:score>4.1</cvss:score>
        <cvss:access-vector>LOCAL</cvss:access-vector>
        <cvss:access-complexity>MEDIUM</cvss:access-complexity>
        <cvss:authentication>SINGLE_INSTANCE</cvss:authentication>
        <cvss:confidentiality-impact>PARTIAL</cvss:confidentiality-impact>
        <cvss:integrity-impact>PARTIAL</cvss:integrity-impact>
        <cvss:availability-impact>PARTIAL</cvss:availability-impact>
        <cvss:source>http://nvd.nist.gov</cvss:source>
        <cvss:generated-on-datetime>2011-10-31T10:31:00.000-04:00</cvss:generated-on-datetime>
      </cvss:base_metrics>
    </vuln:cvss>
  </entry>
  <entry id="CVE-2009-0905">
    <vuln:cve-id>CVE-2009-0905</vuln:cve-id>
    <vuln:cvss>
      <cvss:base_metrics>
        <cvss:score>1.7</cvss:score>
        <cvss:access-vector>LOCAL</cvss:access-vector>
        <cvss:access-complexity>LOW</cvss:access-complexity>
        <cvss:authentication>SINGLE_INSTANCE</cvss:authentication>
        <cvss:confidentiality-impact>NONE</cvss:confidentiality-impact>
        <cvss:integrity-impact>PARTIAL</cvss:integrity-impact>
        <cvss:availability-impact>NONE</cvss:availability-impact>
        <cvss:source>http://nvd.nist.gov</cvss:source>
        <cvss:generated-on-datetime>2011-10-31T10:52:00.000-04:00</cvss:generated-on-datetime>
      </cvss:base_metrics>
    </vuln:cvss>
  </entry>

</nvd>

For testing, I just want to extract the CVSS Score and CVE ID :

namespace.count=8

namespace[0].prefix=default
namespace[0].uri=http://scap.nist.gov/schema/feed/vulnerability/2.0

namespace[1].prefix=xsi
namespace[1].uri=http://www.w3.org/2001/XMLSchema-instance

namespace[2].prefix=schemaLocation
namespace[2].uri=http://scap.nist.gov/schema/patch/0.1 http://nvd.nist.gov/schema/patch_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 http://nvd.nist.gov/schema/scap-core_0.1.xsd http://scap.nist.gov/schema/feed/vulnerability/2.0 http://nvd.nist.gov/schema/nvd-cve-feed_2.0.xsd

namespace[3].prefix=cpe-lang
namespace[3].uri=http://cpe.mitre.org/language/2.0

namespace[4].prefix=cvss
namespace[4].uri=http://scap.nist.gov/schema/cvss-v2/0.2

namespace[5].prefix=patch
namespace[5].uri=http://scap.nist.gov/schema/patch/0.1

namespace[6].prefix=vuln
namespace[6].uri=http://scap.nist.gov/schema/vulnerability/0.4

namespace[7].prefix=scap-core
namespace[7].uri=http://scap.nist.gov/schema/scap-core/0.1


trigger.node.expression=/nvd/entry

token.count=2

token[0].name=CVE_ID
token[0].expression=/nvd/entry/vuln:cve-id

token[1].name=cvss_score
token[1].expression=/nvd/entry/vuln:cvss/cvss:base_metrics/cvss:score


event.name=CVE_ID
event.deviceSeverity=__split(cvss_score,".","1")

event.deviceCustomNumber1=__safeToRoundedLong(cvss_score)
event.deviceCustomNumber1Label=__stringConstant("CVSS-Score")


event.deviceProduct=__stringConstant("CVSS-CPE")
event.deviceVendor=__stringConstant("NVD")
   
severity.map.high.if.deviceSeverity=7,8,9,10
severity.map.medium.if.deviceSeverity=4,5,6
severity.map.low.if.deviceSeverity=0,1,2,3

After running that, I end up with 2 events having identical values for the fields which were mapped to tokens, like event.name=CVE-2009-0900 for both events.

Anybody encounterd this behaviour?

Thank you,

Dragos.

0 Likes
11 Replies
dragoslungu Absent Member.
Absent Member.

Re: XML Flex Connector generates identical events

I don't know if it's Ok to reply to my own question, but I figured out how to fix this XML parser and I think it's better to post the solution than leave it unanswered.

So, for parsing this NVD CVE vulnerability XML feed:

http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-recent.xml

This parser works :

namespace.count=8

namespace[0].prefix=default
namespace[0].uri=http://scap.nist.gov/schema/feed/vulnerability/2.0

namespace[1].prefix=xsi
namespace[1].uri=http://www.w3.org/2001/XMLSchema-instance

namespace[2].prefix=schemaLocation
namespace[2].uri=http://scap.nist.gov/schema/patch/0.1 http://nvd.nist.gov/schema/patch_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 http://nvd.nist.gov/schema/scap-core_0.1.xsd http://scap.nist.gov/schema/feed/vulnerability/2.0 http://nvd.nist.gov/schema/nvd-cve-feed_2.0.xsd

namespace[3].prefix=cpe-lang
namespace[3].uri=http://cpe.mitre.org/language/2.0

namespace[4].prefix=cvss
namespace[4].uri=http://scap.nist.gov/schema/cvss-v2/0.2

namespace[5].prefix=patch
namespace[5].uri=http://scap.nist.gov/schema/patch/0.1

namespace[6].prefix=vuln
namespace[6].uri=http://scap.nist.gov/schema/vulnerability/0.4

namespace[7].prefix=scap-core
namespace[7].uri=http://scap.nist.gov/schema/scap-core/0.1

hop.node.count=1
hop.node[0].name=entry
hop.node[0].expression=/nvd/entry

trigger.node.expression=$entry

token.count=9

token[0].name=CVE_ID
token[0].expression=$entry/vuln:cve-id
token[0].node=.

token[1].name=cvss_score
token[1].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:score
token[1].node=.

token[2].name=cvss_access_vector
token[2].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:access-vector
token[2].node=.

token[3].name=cvss_complexity
token[3].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:access-complexity
token[3].node=.

token[4].name=cvss_authentication
token[4].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:authentication
token[4].node=.

token[5].name=cvss_confidentiality_impact
token[5].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:confidentiality-impact
token[5].node=.

token[6].name=cvss_integrity_impact
token[6].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:integrity-impact
token[6].node=.

token[7].name=cvss_availability_impact
token[7].expression=$entry/vuln:cvss/cvss:base_metrics/cvss:availability-impact
token[7].node=.

token[8].name=cvss_vuln_summary
token[8].expression=$entry/vuln:summary
token[8].node=.

event.name=CVE_ID
event.deviceEventClassId=CVE_ID
event.deviceSeverity=__split(cvss_score,".","1")

event.deviceCustomNumber1=__safeToRoundedLong(cvss_score)
event.deviceCustomNumber1Label=__stringConstant("CVSS-Score")

event.deviceCustomString1=cvss_access_vector
event.deviceCustomString1Label=__stringConstant("Access_Vector")

event.deviceCustomString2=cvss_complexity
event.deviceCustomString2Label=__stringConstant("Complexity")

event.deviceCustomString3=cvss_authentication
event.deviceCustomString3Label=__stringConstant("Authentication")

event.deviceCustomString4=cvss_confidentiality_impact
event.deviceCustomString4Label=__stringConstant("Confidentiality_Impact")

event.deviceCustomString5=cvss_integrity_impact
event.deviceCustomString5Label=__stringConstant("Integrity_Impact")

event.deviceCustomString6=cvss_availability_impact
event.deviceCustomString6Label=__stringConstant("Availability_Impact")

event.flexString1=cvss_vuln_summary
event.flexString1Label=__stringConstant("Vulnerability_Summary")


event.deviceProduct=__stringConstant("CVE-CVSS")
event.deviceVendor=__stringConstant("NVD")
   
severity.map.high.if.deviceSeverity=7,8,9,10
severity.map.medium.if.deviceSeverity=4,5,6
severity.map.low.if.deviceSeverity=0,1,2,3

Thanks to Till and Eugenie for posting the Maxpatrol FlexConnector, a true piece of work !

https://protect724.arcsight.com/docs/DOC-1796

Dragos.

0 Likes
pratibha Trusted Contributor.
Trusted Contributor.

Re: XML Flex Connector generates identical events

Hi ,

I have one query reagarding namespace. What if namespace named schemalocation having two different values for two different nodes?

0 Likes
pratibha Trusted Contributor.
Trusted Contributor.

Re: XML Flex Connector generates identical events

Hi,

What if i want to ignore namespace because in my xml file, values of namespaces are not fixed. It is varying. 

0 Likes
nils.guenther@t Honored Contributor.
Honored Contributor.

Re: XML Flex Connector generates identical events

Define the Namespace that is mostly used in your XML and do ordinary querying for all elements within that Namespace.

For the rest: XPath 2.0 introduced an easy way to implement namespace-agnostic expressions, Prepend each node name in the path with "*:", e.g. /*:rootElement/*:firstChild

0 Likes
pratibha Trusted Contributor.
Trusted Contributor.

Re: XML Flex Connector generates identical events

Hi Nils,

Thank you for your response. I will try this out. It will be more helpful if u give some sample logs and the corresponding parser u have created. That will help me more to understand the above mentioned concept. Anyways i will try by myself also.

Thank you.

0 Likes
pratibha Trusted Contributor.
Trusted Contributor.

Re: XML Flex Connector generates identical events

Hi Nils,

Thanks a tons!!!!

This idea worked.

Thank you so much you have solved my this pending issue.

0 Likes
pratibha Trusted Contributor.
Trusted Contributor.

Re: XML Flex Connector generates identical events

Again Hi Nils,

Now i am abletofetch the data but how to fetch the data from below scenario:

<MitigatingStrategies>

      <MitigatingStrategy>

            <StrategyID>1</StrategyID>

            <StrategyName>Deny by Default</StrategyName>

      </MitigatingStrategy>

      <MitigatingStrategy>

            <StrategyID>2</StrategyID>

            <StrategyName>something</StrategyName>

      </MitigatingStrategy>

</MitigatingStrategies>



here in this i have two same nodes MitigatingStrategy under MitigatingStrategies ,if i write expression /*:MitigatingStrategies/*:MitigatingStrategy/*:StrategyName ,every time it will give back the first one i.e Deny by Default,,,how to fetch data from both the nodes i.e Deny by Default and something.



0 Likes
nils.guenther@t Honored Contributor.
Honored Contributor.

Re: XML Flex Connector generates identical events

Basically you can't "collect" multiple nodes. The only way to deal with this is treating MitigatingStrategy as trigger nodes. This will result in multiple Events (one per MitigatingStrategy). This is not as bad as it might look on first sight. Use a smart rule to correlate those multiple events back to one event containing all data or write data to an AL.

0 Likes
pratibha Trusted Contributor.
Trusted Contributor.

Re: XML Flex Connector generates identical events

Hi Nils,

Thank you for your response. I figured out the solution and that worked.

0 Likes
dparker@siempli Absent Member.
Absent Member.

Re: XML Flex Connector generates identical events

Thanks!

0 Likes
nmbabkin1 Absent Member.
Absent Member.

Re: XML Flex Connector generates identical events

Hello Dragos!
Have you written you connector? Would you please share it with the community?


I'm also looking for connector that parses NIST Vulnerability Feed Data.

Thanks in advance.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.