Broken link monitoring
we have a requirement to monitor a particualr URL which has millions of Related Links and we need to monitor Broken links for the URL in question,
Previosuly we were using Link check monitor , However the maximum Links to this URL has increased to more than a million(may be 5 Million Links) , there was a dedicated Sitescope which only has one monitor which is link check and run the link check tool , This Sitescope server is already cofigured with 30 GB RAM and still there is memory leak exceptions
Increasing memory for this server might not be a solution , I am now running out of ideas what is the best approach to perform this
Any suggestions or Ideas please ?
Re: Broken link monitoring
Any ideas or solutions welcome ? Please let me know if we can achieve this ? I have a case opened for this with support but not getting proper support
Re: Broken link monitoring
Hi Bilal Aman,
I reviewed the issue and finally found the case you have opened for it.
The technical analysis of the issue revealed that the monitor runs out of Java Heap Space if the number of links it has to check exceeds a certain abount (roughly 1 million or so), with the data collected by the monitor occupies 9.5 GB of heap space.
You asked what Micro Focus plans to do about this, as the number of links doesn't decrease but it is planned to further increase them.
After quickly reviewing the monitor code I think there is no quick and easy solution for it. Even if you further increase the maximum JVM heap space, the monitor will sooner or later fail again. It's my understanding that the monitor would require a major rework to work differently so that it doesn't consume that much memory.
For this you can ask the engineer working on the issue to open a case with R&D to have them review the issue, perhaps there is an easy way to resolve the issue I didn't find.
One potential workaround might be to split the one monitor into multiple monitors, if the structure of the web page allows it.
If you have this structure (plain example)
with links to
you could setup one monitor for www.mypage01.com,
another one for www.mypage02com,
and so on.
With this you would split the load and also the resource consumption.
If the monitor fails with 1.000.000 links with exceeding 9.5 GB JVM heap space, dividing this one into ten individual monitors could (!!) easily increase the capacity and the number of links you can monitor.
You should make sure that the monitors run at different days, and not all once a week at the same time.
You also might consider updating SiteScope to a more recent release (we just released SiteScope 11.60).
While no changes to the monitor have been document in all the update (11.3x, 11.4x, 11.5x) it might be that the capacity of the monitor has increased due to internal changes.
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.