Exponentiation with fractional power causes invalid results

0 Likes

This article describes a possible computational error when using fractional exponents and describes a potential work around.

Problem:

Upgrading to version 7.x uncovered a computational error that occurs when executing iterative mathematical statements that involve exponentiation with fractional powers, such as square root. For example, executing the following statement in a loop 50 times could cause the resultant value of MY-CALC-INT-A to equal 1 instead of the correct value of 3.

COMPUTE MY-CALC-INT-A ROUNDED = 9 ** 0.5

Resolution:

Upgrade to version 8.x or higher.

A potential work around is available, but requires code modifications to the existing COBOL program source. To utilize this work around, an ON SIZE ERROR phrase may be added to the COMPUTE statement. For example:

COMPUTE MY-CALC-INT-A ROUNDED = 9 ** 0.5
ON SIZE ERROR …

Incident Number: 2264802

Old KB# 14459
Comment List
Anonymous
Related Discussions
Recommended