Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Outstanding Contributor.. bilal,aman Outstanding Contributor..
Outstanding Contributor..
325 views

Broken link monitoring

Hello Experts 

we have a requirement to monitor a particualr URL which has millions of Related Links and we need to monitor Broken links for the URL in question, 

Previosuly we were using Link check monitor , However the maximum Links to this URL has increased to more than a million(may be 5 Million Links)  , there was a dedicated Sitescope which only has one monitor which is link check and run the link check tool , This Sitescope server is already cofigured with 30 GB RAM and still there is memory leak exceptions 

Increasing memory for this server might not be a solution , I am now running out of ideas what is the best approach to perform this 

Any suggestions or Ideas please ?

 

Regards

Bilal Aman

0 Likes
4 Replies
Outstanding Contributor.. bilal,aman Outstanding Contributor..
Outstanding Contributor..

Re: Broken link monitoring

Hello Experts 

Any advise ?

Regards

Bilal Aman

0 Likes
Outstanding Contributor.. bilal,aman Outstanding Contributor..
Outstanding Contributor..

Re: Broken link monitoring

Any suggestions Experts ?

0 Likes
Outstanding Contributor.. bilal,aman Outstanding Contributor..
Outstanding Contributor..

Re: Broken link monitoring

Any ideas or solutions welcome ? Please let me know if we can achieve this ? I have a case opened for this with support but not getting proper support 

Regards

Bilal Aman

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Broken link monitoring

Hi Bilal Aman,

I reviewed the issue and finally found the case you have opened for it.
The technical analysis of the issue revealed that the monitor runs out of Java Heap Space if the number of links it has to check exceeds a certain abount (roughly 1 million or so), with the data collected by the monitor occupies 9.5 GB of heap space.

You asked what Micro Focus plans to do about this, as the number of links doesn't decrease but it is planned to further increase them.
After quickly reviewing the monitor code I think there is no quick and easy solution for it. Even if you further increase the maximum JVM heap space, the monitor will sooner or later fail again. It's my understanding that the monitor would require a major rework to work differently so that it doesn't consume that much memory.
For this you can ask the engineer working on the issue to open a case with R&D to have them review the issue, perhaps there is an easy way to resolve the issue I didn't find.
One potential workaround might be to split the one monitor into multiple monitors, if the structure of the web page allows it.
If you have this structure (plain example)
www.mypage.com
  with links to
    www.mypage01.com
    www.mypage02com
    ...
    www.mypage11.com
you could setup one monitor for www.mypage01.com,
another one for www.mypage02com,
and so on.
With this you would split the load and also the resource consumption.
If the monitor fails with 1.000.000 links with exceeding 9.5 GB JVM heap space, dividing this one into ten individual monitors could (!!) easily increase the capacity and the number of links you can monitor.
You should make sure that the monitors run at different days, and not all once a week at the same time.

You also might consider updating SiteScope to a more recent release (we just released SiteScope 11.60).
While no changes to the monitor have been document in all the update (11.3x, 11.4x, 11.5x) it might be that the capacity of the monitor has increased due to internal changes.

Greetings
Siggi

Customer Support
Micro Focus

If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.