Check Your Timeouts
At a customer site the other day I saw that they were having a problem with timeouts. Timeouts are something that many people will overlook, but they are very important because they can usually lead to inaccurate scan results.
Typical reasons for timeouts to occur:
- You may be experiencing serious network problems
- There may be an IPS or web app firewall between you and your target machine
- The web server or web application may be down
- Your scan settings may be too rough on the application
WebInspect will retry the attack as many times as your settings specify but it will sometimes falsely flag on things like buffer overflows, not find pages, and worse yet, not find critical vulnerabilities.
When an attack times out it is usually the result of the web app, the server, or the network closing the connection. To better determine which is the result, go to the scan log tab and look at the requests that are failing (the ones in red). To better examine the requests, double-click on the red text and scroll to the right. As an error message you should see something similar to "Request timed out", "Connection forcibly closed", or "Connection closed by remote host".
If sequential requests are failing...
...you may have lost your connection to the server, or the server (or application) may have gone down. Verify using your web browser that the app is indeed working and the server is up as well. On one customer site, WebInspect’s crawler found the administration section of the application’s framework and took the site down. If something like this is the case, make sure to exclude that section by un-selecting that area in your scan tree, or adding some values to "Excluded URLs" in your default scan settings. If the application is indeed up, change your timeout settings to the ones at the bottom of the page.
...you may have hit a problem page in the application. If you are currently on one of the parameter manipulation engines (i.e. Query Injection, Postdata Injection, etc) and all-of-a-sudden every request is failing, check the text in red. If it is the same filename every time, just different parameter attacks, you definitely hit a problem page. Some applications have pages that do not respond well to garbage stuck in their parameters. Rather than give an error message back to the user, they simply dangle the connection and WebInspect slowly times out. These problems should be reported to the developers and the problem page noted for further evaluation. There could actually be a vulnerability here, but there is definitely a problem with quality of service.
If random requests are failing...
...you may have an IPS (intrusion prevention system) or a webapp firewall between you and your target machine. If attacks fail that are signature attacks like "boot.ini" and "etc/passwd" but other requests seem to be fine, this is most definitely the case. Some security professionals do not know whether they have one of these devicess in between them and the application, and only by seeing WebInspect timeouts and then asking their networking teams do they find out. While scanning with one of these devices turned on is a good test of the device, you will want to turn it off (or ask the networking guys for a bypass) for a comprehensive vulnerability test of the web application. Just remember, it is better for you to find it through security scanning then a hacker down the road.
...you may have hit the limit of the application. See the section directly below.
If you are getting an extraordinary amount of timeouts...
...you may have an application that cannot handle the load of a web scan, or the amount of concurrent requests that we are making at a time. Sometimes applications closely watch a user’s session state, and when we make several requests at a time using the same session state, the application hangs. To prevent this, make the settings changes at the bottom of the page. If this does not help, you should consider speaking with the developers about the problem. With these slower settings WebInspect should not put a larger load then about 3 concurrent users of the application. You might also ask about putting the application on a better server.
Recommended WebInspect Settings
These settings are designed to slow a scan down as a result of timeouts. To change these in WebInspect, go into Tools -> Default Settings.
As is default, make sure the “Scan Properties” section on the left-hand side of the “General” tab is selected
Request Timeout: 60 (secs)
Reason: if you increase the time until a request times out, it will allow more time for a slower application (or server) to respond.
Retry Count: 5
Reason: if you increase the retry count, when a request fails WebInspect will retry it again and again, making sure that the attack takes place. Note: the total amount of requests is calculated like so: 1 (original) + 5 (retry count) = 6 total requests
Thread Count: 1
Reason: if you lower the amount of threads the scan will take longer, but WebInspect will be much gentler on the application (and server) and you should fix most of your timeout problems.
Click on “Advanced” on the left-hand side of the “General” tab
500 Total Timeouts
Reason: if you are having problems with the scan pausing after you reach a bunch of timeouts, increase this value. If after 500 total timeouts and the scan still pauses, you are definitely doing an inaccurate scan and you should consult your development and networking teams.
50 Consecutive Timeouts
Reason: if you see 50 timeouts in-a-row you have probably lost your connection to the server or the application has gone down, and 20 requests (the default) can be the result of an IPS or web app firewall.