Contact US

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Precision problem

Precision problem

Precision problem

Here's a fun one. I have a date routine that among other things it grabs the last two digits of the year from the GETDAT subroutine.  Funny thing is when 2005 rolled around it was saying the year is 04!

I fixed the problem, but I was wondering if any of you experts know what going on in the bowels of the computer that would cause this.  Here's the code snippet:

* Example of return values:
*    DATE = '03/13/01'
*    TIME = ' 9:18 am'
TMP=IYR / 100.0
IYR = (TMP - INT(TMP)) * 100

To fix it I added this to the declarations:


and I changed

TMP=IYR / 100.0   to  TMP = dfloti(IYR) / 100.0

The clue that tipped me off was that I assigned
TMP - INT(TMP) to a variable and looked at it in the debugger. The value was 0.049999 something, something.  In our perfect base 10 world it should have been 20.05 - 20 = 0.05.

Yes, I know it's a precision problem, but I want to understand it.  Any takers?

RE: Precision problem

You took a perfectly reasonable and PRECISE integer and turned it into a binary floating point number.  

Had you left it integer and done something like MODULO(year, 100), you would have gotten the result without any error.  


RE: Precision problem

I agree with IRstuff's approach as the more elegant and correct way to tackle your original problem, but getting back to your question on precision, a floating point real variable is most unlikely to store 0.05 as exactly 0.05 but will store it as say 0.050001 in which case your original routine would have returned the correct year, but as you found out it had the value of 4.99999 (when multiplied by 100) and your routine then truncated it to 4, in circumstances when the MOD function is not practical it is standard practice to add 0.5 to your floating point variable before taking the integer, i.e. INT(4.99999 + 0.5) = 5 and also INT(5.00001 + 0.5 ) = 5 so any value > 4.5 and < 5.5 would return the integer value as 5

RE: Precision problem

Thanks guys.  I used the MOD function and it works like a dream.  I feel kinda goofy for not thinking of it.  I've used it in other applications.

I still don't understand all of the inner workings that caused this in the first place.  Yeah, yeah, I understand the principle, I'm just curious about exactly what's happening.

Sorry it took so long to get back. Thanks again.

RE: Precision problem

There's nothing mysterious.  An integer year is exactly represented in a binary format.  

Division by 100, which is not an exact binary number MUST result in an approximation in binary floating point.  It would not matter what precision you used, it would still be inexact.


Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members! Already a Member? Login


Low-Volume Rapid Injection Molding With 3D Printed Molds
Learn methods and guidelines for using stereolithography (SLA) 3D printed molds in the injection molding process to lower costs and lead time. Discover how this hybrid manufacturing process enables on-demand mold fabrication to quickly produce small batches of thermoplastic parts. Download Now
Design for Additive Manufacturing (DfAM)
Examine how the principles of DfAM upend many of the long-standing rules around manufacturability - allowing engineers and designers to place a part’s function at the center of their design considerations. Download Now
Taking Control of Engineering Documents
This ebook covers tips for creating and managing workflows, security best practices and protection of intellectual property, Cloud vs. on-premise software solutions, CAD file management, compliance, and more. Download Now

Close Box

Join Eng-Tips® Today!

Join your peers on the Internet's largest technical engineering professional community.
It's easy to join and it's free.

Here's Why Members Love Eng-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close