Commander
Commander
484 views

Accuracy in ArcOSI

Jump to solution

Hi,

I have few queries to check with you regarding ArcOSI method in collecting the information from different sources.

I have a configuration file where I mentioned only one url http://feeds.dshield.org/block.txt . Executed the ArcOSI application and at the end, I got an output as attached and this is the same info which I got at my syslog server (Kiwi) side. Data is not matching when I checked this information with the data available at the dshield site.

My idea is, I will get the output which is same as in the link. Expecting response soon.

ArcOSI output.jpgDshield.jpg

Configuration file is

[PROXY]

enabled = no

host = proxy.localhost

port = 3128

user = none

pass = none

[IPSOURCES]

url1 = http://feeds.dshield.org/block.txt

-sudesh

Labels (1)
0 Likes
1 Solution

Accepted Solutions
Admiral
Admiral

I already noticed some issues with ArcOSI ( never tested with Bad Harvest ) because it uses a generic regex to extract IP and domain names which can lead to some unwanted entries being listed ( like the domain name of the web site which appears on the web page or its IP ) but here your results are completely different and I can't believe the problem is coming from the script.  Are you sure your proxy is not caching the page ?

Gaetan

View solution in original post

0 Likes
6 Replies
Absent Member.
Absent Member.

Hi Sudesh,

i personally never compared the Lists to be honest - i just tend to believe that arcosi (or Bad Harvest now) works. Interesting point yout got there though - could it be that there was an update of the dshield-list between the time when you executed the py script and the time you checked the website?

On Top of the dshield txt link you can always find the last updated time.

Regards,

Christoph

0 Likes
Commander
Commander

Yes I could. And I am sure, I executed ArcOSI after the update and there was no update performed by DShield during the time of execution.

-sudesh

0 Likes
Admiral
Admiral

I already noticed some issues with ArcOSI ( never tested with Bad Harvest ) because it uses a generic regex to extract IP and domain names which can lead to some unwanted entries being listed ( like the domain name of the web site which appears on the web page or its IP ) but here your results are completely different and I can't believe the problem is coming from the script.  Are you sure your proxy is not caching the page ?

Gaetan

View solution in original post

0 Likes
Commander
Commander

Thanks Gaetan for giving a thought. I believe caching was the problem.Cleared the caches and started as a fresh attempt and the data matches exactly.

Also, the dshield list used to update regularly and corresponding change in list causes the accuracy as well. I have tested it at different times and dshield updated the link at least 3 to 5 times during the execution of ArcOSI.

Right now, temporarily I solved the issue by removing the cache. However, this could happen in future when I automate the same. Is there any way to fix it permanently, so that it never reads from cache even though automated.

Once again, thanks.

-sudesh

0 Likes
Absent Member.
Absent Member.

Please check out our new update on ArcOSI/Bad Harvest!

0 Likes

Do you populate the lists by rules, XML archive tool or with velocity/connector?

I currently use scripts that write the XML AL's and import them using archive but it has issues when making it very large on imports. The writing directly to an activelist only works with an old connector and a lot of hassle.

Rules I dont like because I dont want to fire a rule for every entry  I put on these kind of lists.

I which there was an easier method of automated AL population/creation/updating.

What if the intel gets REALLY BIG? Tapping into an external DB with rules firing of scripts into that doesnt seems to be a very good idea either with ESM . Any thoughts on this?

/Steven

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.