scan job stuck at 9%

[spiritfong1]

hi

I am trying to scan a project by using a batch file that generated by scan wizard

However, It stucks at 9% and the log shows

"There is not enough memory available to complete analysis.  For details on making more memory available, please consult the user manual."

the project is just around 200k LOC and I take 32G ram to scan, whereas I could use the same configuration to scan 1800k LOC project successfully.

Is there any thing I can tune to complete the scan?

Thanks!

[graywt1]

Bigger source scan needs bigger java heap to interpret to the .nst code. There are two heaps in consideration (1) java heap, 32-bit java is up to 1.3 GB, default is 512MB, so we many need 64-bit to break out this boundary. (2) class heap, if too small, you will wait for long time for disk swap. if too big, you will wait for long time for the garbage collection. Select a proper class heap based on your jar overheads and how it is managed can speed up your scan.

• For JAVA HEAP: insert “-64 –Xmx4000M ” in the auditworkbench.cmd or sourceanalyzer
  
• For CLASS HEAP (optional, google these keyword prior to using them) : -XX:+UseParallelGC -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled

-----------------------if scan is done under AWB: after change, auditworkbench.cmd contains---------------

hard coded the launch pad location “C:\Program Files\HP_Fortify\HP_Fortify_SCA_and_Apps_4.42\bin”. Change 4.42 to whatever the AWB version you are using.  (You may not have write privilege to this location, so after modification, you need to save and run this .cmd from other places (i.e. desktop).

set FORTIFY_CORE_LIB="C:\Program Files\HP_Fortify\HP_Fortify_SCA_and_Apps_4.42\Core\lib"

set path=%path%;%FORTIFY_CORE_LIB%

call "C:\Program Files\HP_Fortify\HP_Fortify_SCA_and_Apps_4.42\Core\private-bin\awb\productlaunch.cmd" -64 -Xmx4000M -application com.fortify.awb.app.Main -product com.fortify.awb.app.MainProduct "-Dosgi.splashPath=platform:/base/plugins/com.fortify.awb.app" %*

--------------------if scan is done by command line---------------------------------

nohup /local/ssap/fortify-sca-3.80/bin/sourceanalyzer -b ###  -64 -Xmx4000M -XX:+UseParallelGC -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled  -Dcom.fortify.sca.ProjectRoot=/local/###/ -Dcom.fortify.WorkingDirectory=/local/###  -logfile /local/###/sca.log -scan -f '/local/###/###.fpr' &

[spiritfong1]

Thanks!

However, it still fails to analysis the application

1. I have tuned to "-64 -Xmx32G"  (total 40G in VM)

2. Fail to use command MaxPermSize and it shows

"Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 "

[graywt1]

The parameter MaxPErmSize is not supported under JDK1.8. It is being ignored and did not interrupt your scan.

At this point, I guess your problem is caused by recursive loop somewhere (usually cause by dynamically generating .js file). You can use a technique called "mobile build" - interpret  source code by file types (exculde .js files from all directories), and scan them together. (see SCA user guide for detail)

something like:sourceanalyzer -b bid_1 ../source/**/*.js                            //scan all .js files

sourceanalyzer -b bid_2 ...-exclude /source/**/*.js    ...     //scan all files exclude .js file

sourceanalyzer -b bid_3 ...

sourceanalyzer -b bid_1 -b bid_2 -b bid_3 -scan -f build_all.fpr

[spiritfong1]

Thanks wendy gray

I have tried to exclude one .js directory and it works now

May I know if there is any special with js files?

thanks!!!

[sri.pulla@veriz]

Hi Wendy Gray,

I am new to Fortify and I'm running the bat file generated out of the scan wizard. Also running into the exact same issue.

Could you please elaborate on the commands and do I run these commands before running the bat file? Any additional information will be greatly helpful.

Thanks!

[graywt1]

Fortify seems to have a hard time to scan if the js files are dynamically generated by the extjs or /gwt/node_modules  etc,

Notes:

(1) ~/working indicates a user preferred any dir, does not mean user's home dir.

(2) must define -Dcom.fortify.sca.ProjectRoot and -Dcom.fortify.WorkingDirectory to a ~/working for easy DELETION. After each translate/scan, there are huge .nst files under the ~/working directory. The hard disk filled up fast if you don't delete these artifacts. If not defined, it will be under user home directory and hard to find the location. If you only have 1 application to scan  you can remove theem from the command and define ~/working in the file <FIORTIFY_HOME>/Core/config/fortify-sca.property & fortify.property

(3a) -project-template : you can specify what be hidden after scan (all issue are in DB, just does not display by default, you can display them by "show hidden"

(3b) -filter : filter.txt  everything category/RULEGUID listed in this file will not be reporteed (no record in the DB either)

(3c) -rules: in addition to the fortify rulepack, you can create new rules (see custom rule manual for details)

(4) if you dont have code change, just want to scan with different rules, only run scan.bat will be enough

(5) if you have source code change, I would delete entire ~/working then run translate.bat

translate.bat

CLEAN bid1	~/sourceanalyzer -b bid1 -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working -logfile  ~/working/sca.log  -clean
CLEAN bid2	~/sourceanalyzer -b bid2 -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working -logfile  ~/working/sca.log  -clean
TRANSLATE bid1 (js files)	~/sourceanalyzer -b bid1 -verbose -debug -64 -Xmx11000M -Xss24M -XX:+CMSClassUnloadingEnabled -XX:+UseParallelGC -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working/ -logfile  ~/working/sca.log  -jdk 1.8 -classpath '~/source/WEB-INF/lib/*.jar:~/jars/**/*.jar:~/classes/**/*.class'  '~/source/**/*.js'
TRANSLATE bid2 (non js files)	~/sourceanalyzer -b bid2 -verbose -debug -64 -Xmx11000M -Xss24M -XX:+CMSClassUnloadingEnabled -XX:+UseParallelGC -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working/ -logfile  ~/working/sca.log  -jdk 1.8 -classpath '~/source/WEB-INF/lib/*.jar:~/jars/**/*.jar:~/classes/**/*.class'  -exclude ~/source/**/*.js  '~/source/**/*'

scan.bat

~/sourceanalyzer -b bid1 -b bid2 -64 -Xmx11000M -XX:+UseParallelGC -XX:+CMSClassUnloadingEnabled  -Dcom.fortify.sca.ProjectRoot=~/working -Dcom.fortify.WorkingDirectory=~/working -logfile ~/working/sca.log -scan -project-template ~/whatever_template.xml  -filter ~/whateverfilter.txt -rules ~/whatevere_custom_rule.xml -f  ~your_fpr.fpr'