Commodore
Commodore
10520 views

Web Inspect - slow scan progress - traffic monitor 'database or disk full' errors

Jump to solution

We have a scan that is progressing very slowly, the application being scanned has a lot of dynamic content.  We have done an automatic crawl first, and then de-selected some items (e.g. language localisation) from the site tree to minimize duplication of coverage.  We have had some errors (see below) with the traffic monitor database filling up, and subsequently we disabled trafic monitor for the remainder of the scan.  These errors just repeat every few seconds in the scan log, and i cannot find anyway of stopping them appear either...


HP.App.Sec.TrafficMonitor.Database System.Data.SQLite.SQLiteException (0x80004005) : database or disk is full
HP.App.Sec.TrafficMonitor.Database System.Data.SQLite.SQLiteException (0x80004005) : database or disk image is malformed
HP.App.Sec.TrafficMonitor.Database System.NullReferenceException: Object reference not set to an instance of an object


1.  Can anyone indicate how we clear up some space in the traffic monitor database?  We are using the SQL(lite) edition that is included within WI,  we do not have a separate full installation.  I've seen some references online to opening MS SQL Studio to delete the scan data (of other scans), but I don't believe this is a method we could use..  We cannot enable traffic monitor due to the disk being full.


2.  Shared requestor v Separate requestors.
We have currently configured 36 as our shared requestor setting.  However the user guide, mentions that using separate requestors results in significantly faster scans.  We would be interested if anyone could explain the performance and functional difference of shared requestor versus separate requestor?  If shared requestor is faster, why do we have an option for shared requestor at all?  Could anyone recommend a setting to use if we switched to separate requestor, given that we have 36 configured for shared?

3.  We have come across an option 'Perform Redundant Page Detection' which is currently 'unchecked': Highly dynamic sites could create an infinite number of resources (pages) that are virtually identical. If
allowed to pursue each resource, WebInspect would never be able to finish the scan. This option, however,allows WebInspect to identify and exclude processing of redundant resources.  Has anyone used this setting, has it been useful and would you recommend using this setting, or are there any drawbacks associated with using it?

This is a scan that we will need to repeat at some point, so we are really hoping to understand some of our settings better, and can hopefully progress it more quickly next time........

Labels (3)
0 Likes
1 Solution

Accepted Solutions
Micro Focus Expert
Micro Focus Expert

Hey Claire, I'm going to share a bit of what we covered privately so it does not look like this forum is dead.   😉

 

1.  Can anyone indicate how we clear up some space in the traffic monitor database?  We are using the SQL(lite) edition that is included within WI,  we do not have a separate full installation.  I've seen some references online to opening MS SQL Studio to delete the scan data (of other scans), but I don't believe this is a method we could use..  We cannot enable traffic monitor due to the disk being full.

A1:  This error is referring to the storage for the Traffic Monitor feature, not the SQL Express database used to house the scan data.  The Traffic Monitor feature is normally disabled by default due to the possibly large size of its logs, but your organization requires this TM feature always be used for trouble-shooting.  The TM log database (SQL Lite) is located under the Logs sub-folder for the scan in question.  Per the Application Settings > Directories panel, the Logs typically default to the user's AppData folder, e.g.  ..\AppData\Local\HP\HP WebInspect\Logs\.  Each scan is represented in that folder with a subfolder named using the scan's ScanID (see the Scan Log data for your scan).  If the TM feature were enabled, there will be a TrafficMonitor.db file within that scan's logs folder.  It sounds like your scan has gotten hung due to this DB file hitting some limit, and deleting or moving that file and disabling the TM feature in your Current Scan Settings should permit you to continue your scan.

 

2.  Shared requestor v Separate requestors.
We have currently configured 36 as our shared requestor setting.  However the user guide, mentions that using separate requestors results in significantly faster scans.  We would be interested if anyone could explain the performance and functional difference of shared requestor versus separate requestor?  If shared requestor is faster, why do we have an option for shared requestor at all?  Could anyone recommend a setting to use if we switched to separate requestor, given that we have 36 configured for shared?

A2:  The original scan had been configured as a Crawl-Only with 36 Single Shared Requestor threads, which was acceptable for the hardware resources and target server.  I do not know the precise reasons, but the Simultaneous threads always behaves faster.  It might be that every single thread maintains its own session state rather than all of them using one shared state that needs to be verified.  We switched the scan to Simultaneous threads with 12 threads of Crawl and 24 of Audit, essentially matching the 1:2 ration seen with the defaults of 5 and 10 respectively.

 

3.  We have come across an option 'Perform Redundant Page Detection' which is currently 'unchecked': Highly dynamic sites could create an infinite number of resources (pages) that are virtually identical.

A3:  I believe in this case, the Redundant Page Detection feature was helping some, but it could not account for all the dynamic URIs.  "RPD" will help when crawling large amounts of pages with identical structure and static content.

 

 

Additionally:

The user would manually review the Site Tree, deselect areas that had been crawled, and then proceed manually to the Audit phase.  In your case, most of these were alternat URI that fetched the same content as other URI within the same aplication.  To save on the Crawl time, we discussed entering some Session Exclusions to automatically strike-out those unwanted areas from ever being parsed or requested.


-- Habeas Data
Micro Focus Fortify Customers-Only Forums – https://community.softwaregrp.com/t5/Fortify/ct-p/fortify

View solution in original post

1 Reply
Micro Focus Expert
Micro Focus Expert

Hey Claire, I'm going to share a bit of what we covered privately so it does not look like this forum is dead.   😉

 

1.  Can anyone indicate how we clear up some space in the traffic monitor database?  We are using the SQL(lite) edition that is included within WI,  we do not have a separate full installation.  I've seen some references online to opening MS SQL Studio to delete the scan data (of other scans), but I don't believe this is a method we could use..  We cannot enable traffic monitor due to the disk being full.

A1:  This error is referring to the storage for the Traffic Monitor feature, not the SQL Express database used to house the scan data.  The Traffic Monitor feature is normally disabled by default due to the possibly large size of its logs, but your organization requires this TM feature always be used for trouble-shooting.  The TM log database (SQL Lite) is located under the Logs sub-folder for the scan in question.  Per the Application Settings > Directories panel, the Logs typically default to the user's AppData folder, e.g.  ..\AppData\Local\HP\HP WebInspect\Logs\.  Each scan is represented in that folder with a subfolder named using the scan's ScanID (see the Scan Log data for your scan).  If the TM feature were enabled, there will be a TrafficMonitor.db file within that scan's logs folder.  It sounds like your scan has gotten hung due to this DB file hitting some limit, and deleting or moving that file and disabling the TM feature in your Current Scan Settings should permit you to continue your scan.

 

2.  Shared requestor v Separate requestors.
We have currently configured 36 as our shared requestor setting.  However the user guide, mentions that using separate requestors results in significantly faster scans.  We would be interested if anyone could explain the performance and functional difference of shared requestor versus separate requestor?  If shared requestor is faster, why do we have an option for shared requestor at all?  Could anyone recommend a setting to use if we switched to separate requestor, given that we have 36 configured for shared?

A2:  The original scan had been configured as a Crawl-Only with 36 Single Shared Requestor threads, which was acceptable for the hardware resources and target server.  I do not know the precise reasons, but the Simultaneous threads always behaves faster.  It might be that every single thread maintains its own session state rather than all of them using one shared state that needs to be verified.  We switched the scan to Simultaneous threads with 12 threads of Crawl and 24 of Audit, essentially matching the 1:2 ration seen with the defaults of 5 and 10 respectively.

 

3.  We have come across an option 'Perform Redundant Page Detection' which is currently 'unchecked': Highly dynamic sites could create an infinite number of resources (pages) that are virtually identical.

A3:  I believe in this case, the Redundant Page Detection feature was helping some, but it could not account for all the dynamic URIs.  "RPD" will help when crawling large amounts of pages with identical structure and static content.

 

 

Additionally:

The user would manually review the Site Tree, deselect areas that had been crawled, and then proceed manually to the Audit phase.  In your case, most of these were alternat URI that fetched the same content as other URI within the same aplication.  To save on the Crawl time, we discussed entering some Session Exclusions to automatically strike-out those unwanted areas from ever being parsed or requested.


-- Habeas Data
Micro Focus Fortify Customers-Only Forums – https://community.softwaregrp.com/t5/Fortify/ct-p/fortify

View solution in original post

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.