Absent Member.
Absent Member.
5603 views

scan job stuck at 9%

hi

I am trying to scan a project by using a batch file that generated by scan wizard

However, It stucks at 9% and the log shows

"There is not enough memory available to complete analysis.  For details on making more memory available, please consult the user manual."

the project is just around 200k LOC and I take 32G ram to scan, whereas I could use the same configuration to scan 1800k LOC project successfully.

Is there any thing I can tune to complete the scan?

Thanks!

0 Likes
6 Replies
Absent Member.
Absent Member.

Bigger source scan needs bigger java heap to interpret to the .nst code. There are two heaps in consideration (1) java heap, 32-bit java is up to 1.3 GB, default is 512MB, so we many need 64-bit to break out this boundary. (2) class heap, if too small, you will wait for long time for disk swap. if too big, you will wait for long time for the garbage collection. Select a proper class heap based on your jar overheads and how it is managed can speed up your scan.

  • For JAVA HEAP: insert “-64 –Xmx4000M ” in the auditworkbench.cmd or sourceanalyzer
  • For CLASS HEAP (optional, google these keyword prior to using them) : -XX:+UseParallelGC -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled

-----------------------if scan is done under AWB: after change, auditworkbench.cmd contains---------------

hard coded the launch pad location “C:\Program Files\HP_Fortify\HP_Fortify_SCA_and_Apps_4.42\bin”. Change 4.42 to whatever the AWB version you are using.  (You may not have write privilege to this location, so after modification, you need to save and run this .cmd from other places (i.e. desktop).

set FORTIFY_CORE_LIB="C:\Program Files\HP_Fortify\HP_Fortify_SCA_and_Apps_4.42\Core\lib"

set path=%path%;%FORTIFY_CORE_LIB%

call "C:\Program Files\HP_Fortify\HP_Fortify_SCA_and_Apps_4.42\Core\private-bin\awb\productlaunch.cmd" -64 -Xmx4000M -application com.fortify.awb.app.Main -product com.fortify.awb.app.MainProduct "-Dosgi.splashPath=platform:/base/plugins/com.fortify.awb.app" %*

--------------------if scan is done by command line---------------------------------

nohup /local/ssap/fortify-sca-3.80/bin/sourceanalyzer -b ###  -64 -Xmx4000M -XX:+UseParallelGC -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled  -Dcom.fortify.sca.ProjectRoot=/local/###/ -Dcom.fortify.WorkingDirectory=/local/###  -logfile /local/###/sca.log -scan -f '/local/###/###.fpr' &

0 Likes
Absent Member.
Absent Member.

Thanks!

However, it still fails to analysis the application

1. I have tuned to "-64 -Xmx32G"  (total 40G in VM)

2. Fail to use command MaxPermSize and it shows

"Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 "

0 Likes
Absent Member.
Absent Member.

The parameter MaxPErmSize is not supported under JDK1.8. It is being ignored and did not interrupt your scan.

At this point, I guess your problem is caused by recursive loop somewhere (usually cause by dynamically generating .js file). You can use a technique called "mobile build" - interpret  source code by file types (exculde .js files from all directories), and scan them together. (see SCA user guide for detail)

something like:sourceanalyzer -b bid_1 ../source/**/*.js                            //scan all .js files

sourceanalyzer -b bid_2 ...-exclude /source/**/*.js    ...     //scan all files exclude .js file

sourceanalyzer -b bid_3 ...

sourceanalyzer -b bid_1 -b bid_2 -b bid_3 -scan -f build_all.fpr

0 Likes
Absent Member.
Absent Member.

Thanks wendy gray

I have tried to exclude one .js directory and it works now

May I know if there is any special with js files?

thanks!!!

0 Likes
Absent Member.
Absent Member.

Hi Wendy Gray,

I am new to Fortify and I'm running the bat file generated out of the scan wizard. Also running into the exact same issue.

Could you please elaborate on the commands and do I run these commands before running the bat file? Any additional information will be greatly helpful.

Thanks!

0 Likes
Absent Member.
Absent Member.

Fortify seems to have a hard time to scan if the js files are dynamically generated by the extjs or /gwt/node_modules  etc,

Notes:

(1) ~/working indicates a user preferred any dir, does not mean user's home dir.

(2) must define -Dcom.fortify.sca.ProjectRoot and -Dcom.fortify.WorkingDirectory to a ~/working for easy DELETION. After each translate/scan, there are huge .nst files under the ~/working directory. The hard disk filled up fast if you don't delete these artifacts. If not defined, it will be under user home directory and hard to find the location. If you only have 1 application to scan  you can remove theem from the command and define ~/working in the file <FIORTIFY_HOME>/Core/config/fortify-sca.property & fortify.property

(3a) -project-template : you can specify what be hidden after scan (all issue are in DB, just does not display by default, you can display them by "show hidden"

(3b) -filter : filter.txt  everything category/RULEGUID listed in this file will not be reporteed (no record in the DB either)

(3c) -rules: in addition to the fortify rulepack, you can create new rules (see custom rule manual for details)

(4) if you dont have code change, just want to scan with different rules, only run scan.bat will be enough

(5) if you have source code change, I would delete entire ~/working then run translate.bat

translate.bat

CLEAN bid1~/sourceanalyzer -b bid1 -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working -logfile  ~/working/sca.log  -clean 
CLEAN bid2~/sourceanalyzer -b bid2 -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working -logfile  ~/working/sca.log  -clean 
TRANSLATE bid1 (js files)~/sourceanalyzer -b bid1 -verbose -debug -64 -Xmx11000M -Xss24M -XX:+CMSClassUnloadingEnabled -XX:+UseParallelGC -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working/ -logfile  ~/working/sca.log  -jdk 1.8 -classpath '~/source/WEB-INF/lib/*.jar:~/jars/**/*.jar:~/classes/**/*.class'  '~/source/**/*.js'
TRANSLATE bid2 (non js files)~/sourceanalyzer -b bid2 -verbose -debug -64 -Xmx11000M -Xss24M -XX:+CMSClassUnloadingEnabled -XX:+UseParallelGC -Dcom.fortify.sca.ProjectRoot=~/working/ -Dcom.fortify.WorkingDirectory=~/working/ -logfile  ~/working/sca.log  -jdk 1.8 -classpath '~/source/WEB-INF/lib/*.jar:~/jars/**/*.jar:~/classes/**/*.class'  -exclude ~/source/**/*.js  '~/source/**/*'

scan.bat

~/sourceanalyzer -b bid1 -b bid2 -64 -Xmx11000M -XX:+UseParallelGC -XX:+CMSClassUnloadingEnabled  -Dcom.fortify.sca.ProjectRoot=~/working -Dcom.fortify.WorkingDirectory=~/working -logfile ~/working/sca.log -scan -project-template ~/whatever_template.xml  -filter ~/whateverfilter.txt -rules ~/whatevere_custom_rule.xml -f  ~your_fpr.fpr'

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.