IBM PTFs required for correct maths calculations (precision) when using RDMLX on iSeries

Date:10 August 2007
Product/Version:Visual LANSA (RDMLX) on iSeries
Abstract:Incorrect precision calculations when using RDMLX on iSeries fixed via PTF
Submitted By:LANSA Technical Support
External(s):IBM PTFs
Last Review:November 2010

In Visual LANSA 11.x, it is possible to create RDMLX functions to execute on iSeries. During checkin and compile, C code is generated using the IBM C compiler.

In some cases the ILE C compiler (CRTCMOD and CRTBNDC commands) incorrectly calculates intermediate results for packed decimal division. There is available from IBM a PTF to correct this problem. If you experience any calculation anomalies, you should apply the appropriate PTF, depending on your OS version.

  • PTF number for V5R3 is SI27397
  • PTF number for V5R4 is SI27458

Sample code to illustrate the problem:

define field(#fielda) type(*dec) length(15) decimals(4) edit_code(J)
define field(#fieldb) type(*dec) length(15) decimals(0) edit_code(J) default(6000)
define field(#fieldc) type(*dec) length(15) decimals(4) edit_code(J) default(6000)

#fielda := #fieldb / 10000 or change field(#fielda) to('#fieldb / 10000')
results in #fielda = 0.0000 incorrect

#fielda := #fieldc / 10000 or change field(#fielda) to('#fieldc / 10000')
results in #fielda = 0.6000 correct

With the PTF applied, both answers are 0.6000. Refer to the PTF document for a full explanation.