Thursday, April 17, 2008

Understanding Floating Point Precision in AutoCAD, Excel, etc.

AutoCAD has had 64-bit accuracy in the 32-bit platform for many years but still with 64-bit accuracy things can go wrong with with "unexpected" result.

Why is 1.320 - 1.3190 not equal to 0.001 when it really is 0.001?

Excel caluculation

Be careful when you make comparisons like this.

Both AutoCAD, Excel and many other applications are designed in accordance to the IEEE Standard for Binary Floating-Point Arithmetic (IEEE 754). The standard defines how floating-point numbers are stored and calculated.

A floating-point number is stored in binary in three parts within a 65-bit range: the sign, the exponent, and the mantissa.

1 Sign Bit
11 Bit Exponent
1 Implied Bit
52 Bit Mantissa

A detailed article that describes this thoroughly is Understanding Floating Point Precision, aka “Why does Excel Give Me Seemingly Wrong Answers?”

1 comment:

  1. Restored comments

    Kevin said...

    Wikipedia's article http://en.wikipedia.org/wiki/Double-precision on double-precision floating point values will show you that they use 64 bits, not 65. One bit is used for the sign of the number. Eleven are used for the exponent (with an assumed offset to include negative exponents). the remaining 52 bits are used for the mantissa, so there are 53 bits of precision since the first bit (before the decimal) is always 1 for a non-zero number.

    Using 53 bits of precision leaves you theoretically with 15.95 reliable decimal digits. Compare this to single precision at 7.22 digits.
    July 25, 2008

    JTB World said...

    One bit, called the hidden bit or the implied bit, can be omitted if all numbers are required to be normalized according to the IEEE 754 standard.
    Here you also see the implied bit mentioned.
    http://support.microsoft.com/kb/78113

    ReplyDelete

Some of the latest blog posts

Subscribe to RSS headline updates from:
Powered by FeedBurner

Contact Us | About JTB World | Website General Terms of Use | Privacy Policy
^ Top of page

© 2004- JTB World. All rights reserved.