Thursday, April 17, 2008

Understanding Floating Point Precision in AutoCAD, Excel, etc.

AutoCAD has had 64-bit accuracy in the 32-bit platform for many years but still with 64-bit accuracy things can go wrong with with "unexpected" result.

Why is 1.320 - 1.3190 not equal to 0.001 when it really is 0.001?

Excel caluculation

Be careful when you make comparisons like this.

Both AutoCAD, Excel and many other applications are designed in accordance to the IEEE Standard for Binary Floating-Point Arithmetic (IEEE 754). The standard defines how floating-point numbers are stored and calculated.

A floating-point number is stored in binary in three parts within a 65-bit range: the sign, the exponent, and the mantissa.

1 Sign Bit
11 Bit Exponent
1 Implied Bit
52 Bit Mantissa

A detailed article that describes this thoroughly is Understanding Floating Point Precision, aka “Why does Excel Give Me Seemingly Wrong Answers?”


Some of the latest blog posts

Subscribe to RSS headline updates from:
Powered by FeedBurner