Understanding Floating Point Precision in AutoCAD, Excel, etc.
AutoCAD has had 64-bit accuracy in the 32-bit platform for many years but still with 64-bit accuracy things can go wrong with with "unexpected" result.
Why is 1.320 - 1.3190 not equal to 0.001 when it really is 0.001?
Be careful when you make comparisons like this.
Both AutoCAD, Excel and many other applications are designed in accordance to the IEEE Standard for Binary Floating-Point Arithmetic (IEEE 754). The standard defines how floating-point numbers are stored and calculated.
A floating-point number is stored in binary in three parts within a 65-bit range: the sign, the exponent, and the mantissa.
1 Sign Bit
11 Bit Exponent
1 Implied Bit
52 Bit Mantissa
A detailed article that describes this thoroughly is Understanding Floating Point Precision, aka “Why does Excel Give Me Seemingly Wrong Answers?”



0 comments (add your comment):
Post a Comment |
Subscribe to the comments feed