This website uses cookies. By continuing to browse or login to this website, you consent to the use of cookies. Learn more.
OK
thawley
Cadet 3rd Class
Cadet 3rd Class
‎2018-10-05 17:02
5816 views

sourceanalyzer -incremental-base scans failing

I'm attempting to set up incremental scans for our project, but am running into issues. The issue claims to be a memory management complaint, but the error produced leads me to believe it's something else.

This is a simplified example due to sensitive information, but the inital phase works fine too:

sourceanalyzer -b fooabr "**/*" -source 1.8 -cp "WebContent/WEB-INF/lib/*.jar"

When I run either of the following, things run succesfully:

sourceanalyzer foobar -scan
sourceanalyzer foobar -scan -f foobar.fpr

 However when I run it with the incremental-base directive, I recieve the message "An unexpected error occurred during internal memory management. The scan will continue, but memory may be quickly exhausted and scan results may be incomplete. Please submit your scan log file to Fortify Support." in the console and a "java.io.NotSerializableException: java.util.regex.Matcher" in the log file.

sourceanalyzer foobar -scan -incremental-base -f foobar.fpr

I've tried different -Xsx allocations, -autoheap, and -64 to no avail. In a basic single Java file test I ran things worked, but that's not the case for our much larger project's codebase.

[2018-10-05 11:35:29.041 Thread-14 INFO]
Fortify Static Code Analyzer 18.10.0192 (using JRE 1.8.0_163)
[2018-10-05 11:35:29.043 Thread-14 INFO]
Args:
["-b", "foobar", "-scan", "-incremental-base", "-f", "foobar.fpr", "-logfile", "fortify.log"]
[2018-10-05 11:35:29.044 Thread-14 INFO]
VM Args:
"-XX:SoftRefLRUPolicyMSPerMB=3000 -Xmx15032385536 -Xss16M"
[2018-10-05 11:35:29.313 Thread-14 INFO]
Front End complete
[2018-10-05 11:35:29.995 Thread-14 INFO 1451]
Analyzing 819 source file(s)
[2018-10-05 11:35:30.117 Thread-14 INFO]
License Metadata: ...
[2018-10-05 11:35:30.922 Thread-14 INFO]
Loading nametable with 639 compilation units
[2018-10-05 11:35:40.095 Thread-20 INFO]
Loaded nametable with 160 compilation units
[2018-10-05 11:35:45.320 Thread-20 INFO]
Loaded nametable with 319 compilation units
[2018-10-05 11:35:53.082 Thread-20 INFO]
Loaded nametable with 478 compilation units
[2018-10-05 11:36:08.607 Thread-20 INFO]
Loaded nametable with 637 compilation units
[2018-10-05 11:36:27.433 Thread-57 WARNING 20511]
Unable to locate metadata for function trim at ...
[2018-10-05 11:36:36.706 Thread-14 INFO]
Completing call graph
[2018-10-05 11:36:47.295 Thread-14 INFO]
Constant Propagation: starting initialization of 639 compilation units
[2018-10-05 11:36:47.621 Thread-101 INFO]
Constant Propagation: initialized 160 compilation units
[2018-10-05 11:36:47.628 Thread-102 INFO]
Constant Propagation: initialized 319 compilation units
[2018-10-05 11:36:47.632 Thread-104 INFO]
Constant Propagation: initialized 478 compilation units
[2018-10-05 11:36:47.638 Thread-107 INFO]
Constant Propagation: initialized 637 compilation units
[2018-10-05 11:36:47.645 Thread-14 INFO]
Constant Propagation: initial constants for 5485 functions
[2018-10-05 11:36:49.569 Thread-14 INFO]
Constant Propagation: propagating constant parameters for 5485 functions
[2018-10-05 11:36:50.109 Thread-14 INFO]
Constant Propagation: propagating unresolved constants for 5485 functions
[2018-10-05 11:36:50.505 Thread-14 INFO]
Constant Propagation: completed
[2018-10-05 11:37:00.358 Thread-14 SEVERE 1142]
An unexpected error occurred during internal memory management. The scan will continue, but memory may be quickly exhausted and scan results may be incomplete. Please submit your scan log file to Fortify Support.
java.io.NotSerializableException: java.util.regex.Matcher
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at com.ergy.fset.FHashSet.writeObject(FHashSet.java:1638)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at com.ergy.fset.FTreeList.writeObject(FTreeList.java:1074)
	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at java.util.concurrent.ConcurrentHashMap.writeObject(ConcurrentHashMap.java:1413)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at java.util.EnumMap.writeObject(EnumMap.java:782)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at com.fortify.sca.backend.BackEnd.saveProgramModel(BackEnd.java:1132)
	at com.fortify.sca.backend.BackEnd.saveIncrementalData(BackEnd.java:1113)
	at com.fortify.sca.backend.BackEnd.analyze(BackEnd.java:635)
	at com.fortify.sca.Main$Sourceanalyzer.run(Main.java:708)
[2018-10-05 11:37:00.360 Thread-14 INFO 1459]
Analysis completed in 00:00
[2018-10-05 11:37:00.361 Thread-1 WARNING 20213]
exit(1)
0 Likes
Reply
Report Inappropriate Content
2 Replies
Caliban
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor
‎2018-10-22 15:25

Good day.

If you are using 18.10.  -mt (multithreading is the default) -64 is no longer supported or needed. SCA is 64Bit.

You do not need to set memory for 18.10 as SCA will correctly manage this.

We are not sure if you are aware of the following information.

 INCREMENTAL Analysis is 'not' fully implemented for all analyzers
 ONLY Configuration and the Semantic analyzers are working (see ** in list)
 The analyzers list:
1 buffer,
2 content,
3 **configuration,
4 controlflow,
5 dataflow,
6 findbugs,
7 nullptr,
8 **semantic,
9 and structural.

Please note the languages supported.
Java, C/C++, C#, and Visual Basic.
(Ref: pg 21 17.20 SCA Guide)

Also if your goal it improve SCAN time(s) please consider the SCA Performance guide PDF

Caution:

Incremental scans that meet the basic guideline can actually take longer for Scans to complete than for Scans that are optimized and run without -incremental-base features.

0 Likes
Reply
Report Inappropriate Content
Matthias Mangin
Cadet 3rd Class
Cadet 3rd Class
‎2019-08-12 14:33

Hello, 

I am facing the same kind of issue, and I cannot find how to fix it.

I am using the 18.20 version
For testing purposes, I am trying to scan incrementally the java code located under /infrabox/context/JavaVulnerableLab/src/main/java with the following instructions :

this java code can be found here : https://github.com/CSPF-Founder/JavaVulnerableLab

sourceanalyzer -b infrabox-fortify-integration /infrabox/context/JavaVulnerableLab/src/main/java
sourceanalyzer -b infrabox-fortify-integration -scan -incremental-base -f /infrabox/cache/fortify-results-incremental-$BUILD_ID.fpr
[2019-08-12 08:09:28.232 INFO  1451]
Analyzing 15 source file(s)
[2019-08-12 08:09:44.980 WARN  20511]
Unable to locate metadata for function init^ at /infrabox/context/JavaVulnerableLab/src/main/java/org/cysecurity/cspf/jvl/controller/EmailCheck.java:43:32
[2019-08-12 08:09:44.983 WARN  20511]
Unable to locate metadata for function put at /infrabox/context/JavaVulnerableLab/src/main/java/org/cysecurity/cspf/jvl/controller/EmailCheck.java:51:27
[2019-08-12 08:09:48.103 ERROR 1142]
An unexpected error occurred during internal memory management. The scan will continue, but memory may be quickly exhausted and scan results may be incomplete. Please submit your scan log file to Fortify Support.
java.io.NotSerializableException: com.fortify.messaging.Logger ...

without the -incremental-base, the same command doesn't produce this error. 

Do you know where this issue comes from ?

Best regards, Matthias

0 Likes
Reply
Report Inappropriate Content
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.