Accuracy in ArcOSI

[sudesh]

Hi,

I have few queries to check with you regarding ArcOSI method in collecting the information from different sources.

I have a configuration file where I mentioned only one url http://feeds.dshield.org/block.txt . Executed the ArcOSI application and at the end, I got an output as attached and this is the same info which I got at my syslog server (Kiwi) side. Data is not matching when I checked this information with the data available at the dshield site.

My idea is, I will get the output which is same as in the link. Expecting response soon.

Configuration file is

[PROXY]

enabled = no

host = proxy.localhost

port = 3128

user = none

pass = none

[IPSOURCES]

url1 = http://feeds.dshield.org/block.txt

-sudesh

[chenselein1]

Hi Sudesh,

i personally never compared the Lists to be honest - i just tend to believe that arcosi (or Bad Harvest now) works. Interesting point yout got there though - could it be that there was an update of the dshield-list between the time when you executed the py script and the time you checked the website?

On Top of the dshield txt link you can always find the last updated time.

Regards,

Christoph

[sudesh]

Yes I could. And I am sure, I executed ArcOSI after the update and there was no update performed by DShield during the time of execution.

-sudesh

[GCA · Accepted solution]

I already noticed some issues with ArcOSI ( never tested with Bad Harvest ) because it uses a generic regex to extract IP and domain names which can lead to some unwanted entries being listed ( like the domain name of the web site which appears on the web page or its IP ) but here your results are completely different and I can't believe the problem is coming from the script.  Are you sure your proxy is not caching the page ?

Gaetan

[sudesh]

Thanks Gaetan for giving a thought. I believe caching was the problem.Cleared the caches and started as a fresh attempt and the data matches exactly.

Also, the dshield list used to update regularly and corresponding change in list causes the accuracy as well. I have tested it at different times and dshield updated the link at least 3 to 5 times during the execution of ArcOSI.

Right now, temporarily I solved the issue by removing the cache. However, this could happen in future when I automate the same. Is there any way to fix it permanently, so that it never reads from cache even though automated.

Once again, thanks.

-sudesh

[gregcmartin]

Please check out our new update on ArcOSI/Bad Harvest!

[StevenvandeBraak]

Do you populate the lists by rules, XML archive tool or with velocity/connector?

I currently use scripts that write the XML AL's and import them using archive but it has issues when making it very large on imports. The writing directly to an activelist only works with an old connector and a lot of hassle.

Rules I dont like because I dont want to fire a rule for every entry  I put on these kind of lists.

I which there was an easier method of automated AL population/creation/updating.

What if the intel gets REALLY BIG? Tapping into an external DB with rules firing of scripts into that doesnt seems to be a very good idea either with ESM . Any thoughts on this?

/Steven