WebInspect RegexExecution Errors

[stanhphogan]

I am attempting to run WebInspect 10.4 against a web application and the scan will complete the crawl, but when it gets started with the audit it will just sit there running with no progress for hours.  In looking at the scan log it shows the following error, and i am not sure how to fix these isseues.  I will say these are extremely large websites (17000 automated crawls), because i dont experience this one smaller

Thanks,

Scan Log Errors:

Warn RegexExecution_ProgressThresholdExceededAbort: RegexExecution: Pattern:<SCRIPT[^>]*(?:(?:/>)|(?:.*?</SCRIPT>)), Thread.Name:110df2ca-3dcf-49d2-90dc-3606ac08780b:PoolRequestManagerThread.ThreadProc[120], Thread.ManagedThreadID:120, Duration:00:00:00, StartTime:13:39:40.1755, EndTime:23:59:59.9999, Function:Replace(string input, string replacement), input:

8/1/2015 8:41:40 AM Warn RegexExecution_ThresholdExceeded: RegexExecution: Pattern:<SCRIPT[^>]*(?:(?:/>)|(?:.*?</SCRIPT>)), Thread.Name:110df2ca-3dcf-49d2-90dc-3606ac08780b:PoolRequestManagerThread.ThreadProc[120], Thread.ManagedThreadID:120, Duration:00:02:00.3109001, StartTime:13:39:40.1755, EndTime:13:41:40.4864, Function:Replace(string input, string replacement), input:

[HansEnders]

Stan;

I did not see that type of error among the details of the article "Scan Log Messages" found in the WebInspect Help guide (F1), nor did the key word RegexExecution show up.

From the Script details shown, it appears one of the injection engines probes is locked up (10 hours?).  I would take this to Fortify Support (support.fortify.com) for direct review.

They will most likely want a *.SCAN export of the offending scan plus the logs for the scan and for WebInspect itself.  The Scan Log tab will show the scan's ScanID, and you would then grab (Zip up) a sub-folder using that as its name from the Logs folder.  (Find your Logs folder path under the Application Settings > Directories).  The WebInspect application log will be a peer sub-folder found there that is named with all zeroes ("0000").  Dropping that combination of data to start your Support Case should provide plenty of details without delay.   😉    They can provide you a temporary FTP/sFTP/web file share if the SCAN file is too large to upload into the case directly.

Please report back here if it was a straight-forward error diagnosis or resolution.

PS - 17,000 is a huge Crawl, since I would expect the Audit Count to then be 3x or 4x the Crawl Count.  You might want to inquire with them on ways to make that Crawl more efficient without missing any areas, in order to save on time and still be as thorough.

[liuyao]

hello,

Has this problem been solved? I has a similar problems.

[ebell]

@liuyao what version of WebInspect are you seeing? You also mention similar problems...can you be more specific on what the error message is you are getting?

The RegexExecution_ThresholdExceeded error/warn message as see above could be a resource memort/cpu issue.

[liuyao]

Thank you for your reply!

The version is 19.1.The error message (regex)  is exactly the same as the picture provided by you.

From scan log, the printed  url is javascript and the response length is large, the regex is executed more than 2 minutes. Also, The error message of the same url can be 2500 times. This results slow scanning and large logs.

You think this caused by machine performance ? which machine ?

[ebell]

Without a scan file with traffic and logs everything else is pretty much speculation. With that being said, 

1. Is SPA enabled for this scan?
2. Any AV/AM running on the machine
3. Is this a physical or VM? If VM, make sure resources are dedicated to the VM.
4. Is it a dedicated machine or are other programs running? What are the specs on the machine (cpu, ram, etc.)?
5. Take a look at the Windows Event Logs (Application and System) to see if there is anything of interested in there

If you would like use to take a "deeper" look please open a ticket with support and provide an export scan where traffic monitor was enabled and make sure the check mark to include logs is checked in the export window.

[liuyao]

thank you!

Different WI to scan the same site has the same problem.I do not think it is a resourse problem.

One method is recommended by vendor supporter is increase RegexExecutionCompleteThresholdMS( default 60000, in C:\ProgramData\HP\HP WebInspect\SharedSettings.config).

I try to set 300000, but the problem is not solved.

The scan file is very large, it limits to send to others. So I'm looking for a similar solution  to try.

[ebell]

At this point I recommend opening a ticket with support to take a look at an exported scan where traffic and logs are included. Along with that file, send an export of your System and Application event logs. You can find information on how to collect this information here https://softwaresupport.softwaregrp.com/doc/KM03601146