## floor w/ and w/o units gives different results

## floor w/ and w/o units gives different results

(OP)

Hi all, I'm pretty new to Mathcad (I'm still young ;)), but I have run across a pretty weird problem. I did try google/help/eng-tips without any luck.

I think my problem is best explained in the attached file but here it is:

My 'goal' is to floor a division incluing units m/mm. In my example, I divide 12.75m by 1275mm which should result in 10. But Mathcad actually return 9.99....998. 'flooring' this number would result in 9, where 10 is expected.

When I try to floor(12750/1275) I get 10, as expected.

I would like to know if there is any solution to this problem or if I'm just stuck with this error.

I think my problem is best explained in the attached file but here it is:

My 'goal' is to floor a division incluing units m/mm. In my example, I divide 12.75m by 1275mm which should result in 10. But Mathcad actually return 9.99....998. 'flooring' this number would result in 9, where 10 is expected.

When I try to floor(12750/1275) I get 10, as expected.

I would like to know if there is any solution to this problem or if I'm just stuck with this error.

## RE: floor w/ and w/o units gives different results

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

## RE: floor w/ and w/o units gives different results

Looks to me like 9 is the expected result.

## RE: floor w/ and w/o units gives different results

## RE: floor w/ and w/o units gives different results

BTW, my help for floor(z) states "greatest integer <=z"

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

(12.75 m) / (1275 mm)

is actually calculated internally as something like:

(12.75 * 1.0) / (1275.0 * (1.0 / 1000.0))

We human beings are "clever" enough to see that this is exactly equal to 10, but Mathcad evaluates this expression as being "equal" to 9.999999999999998 - which is the answer it gives for the expression with units. (Try it!)

http://julianh72.blogspot.com

## RE: floor w/ and w/o units gives different results

That's the result I get in Mathcad. The same computer gives a different result for a similar calculation in Excel. If I format the cell for the maximum 30 decimal places, I get 10 followed by 30 zeros. Any idea how Microsoft gets around the binary floating point limitation?

## RE: floor w/ and w/o units gives different results

Sorry! I can't help you with explaining why Excel and Mathcad get slightly different results in the 15th decimal place ...

... Other than to say they are programmed by different teams, using different compilers and different algorithms. (In fact, it would be quite remarkable if two different programs gave identical results to 15 decimal places for all mathematical floating-point expressions, given the accuracy constraints of representing real numbers in binary format!)

http://julianh72.blogspot.com

## RE: floor w/ and w/o units gives different results

## RE: floor w/ and w/o units gives different results

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

Nonetheless, the residual from the original question is 2^-49, which is 3 bits off from the 52-bit fractional part of a double precision float.

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

What makes you think that Mathcad and Excel would carry out EXACTLY the same sequence of calculations & function calls with EXACTLY the same binary representations for ALL terms in PRECISELY the same order?

Excel has no explicit means of representing units; in my example, I suggested that Mathcad MIGHT represent "1 mm" as "(1.0 / 1000.0) times 1 metre"; but it MIGHT store it as "0.001 metres"; or it MIGHT (probably!) use some other internal algorithm to handle the units.

I have no idea whether Mathcad does some sort of "symbolic shuffling" to determine the units of the solution, or if it multiplies and divides its internal representations of units (nor the sequence of such "cancelling" operations), or what. Bearing in mind the limits of precision of binary floating point maths, "0.001" and "(1.0 / 1000.0)" will not necessarily yield EXACTLY the same result carried across to the last available binary digit. Thus, multiplying by 0.001 or dividing by 1000.0 will not necessarily yield exactly the same result, because you are actually calling different processor functions (possibly in a different sequence of operations), using different stored binary numbers.

http://julianh72.blogspot.com

## RE: floor w/ and w/o units gives different results

I was also under the impression that math was more about the processessor than the higher level coding, explaining the pentium bug that affected all programs in the same way as I recall.

Good discussion here. Never really questioned floating point math since abandoning my slide rule. Thanks for the education.

## RE: floor w/ and w/o units gives different results

Do a "=pi()^2" and format the cell for 30 decimal places, and you'll see 16 zeros at the end, even though there are ostensibly a potential for 28 decimal places of precision.

However, Mathcad does have the capability of doing substantially high precision using its symbolic math processor, but that ptocessor is not unit-friendly.

If you're willing to mess up the units system a bit by redefining mm=0.001m, the symbolic processor will give you the correct numerical answer to 12.75m/1275mm, to whatever precision you want, and assign the answer to a variable. you could then use the numerical processor for the floor function on that variable.

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

I understand that software has to be programmed in certain ways (I have written some 'software' myself), but don't 'limitations' like the one discussed, just defeat the purpose of Math-software in some way?

If I wouldn't have used the floor function I would haven't have had a single problem with 10 actually being 9,99...98, but in this specific case it could actually be make-or-break. So is the lesson learned from this that you should be really carefully using 'non-math' functions, like floor? Or should you perhaps 'never' use units, which is one of the reasons I like Mathcad so much.

In my case using round will solve my problem. Are there any more tips/tricks people should know of before using software like this? It scared the h*ll out of me when I found this out though .

## RE: floor w/ and w/o units gives different results

This should done with care; otherwise, a global tolerance setting that high will give you silly results. Typically, advanced users will set the zero tolerance to 307, i.e., 10^-307 at the get-go.

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

Interesting story about 24 bit floating point leading to the death of 28 and injuring 100 more: htt

## RE: floor w/ and w/o units gives different results

More details in the links below, for anyone interested.

http

h

Doug Jenkins

Interactive Design Services

http://newtonexcelbach.wordpress.com/

## RE: floor w/ and w/o units gives different results

A typical scenario is to start range counter at T0 for the first pulse. Detect SCUD at T1. Set range gates for second pulse at T2+(T1-T0)-deltaT*Vscud/c-gateT to T2+(T1-T0)-deltaT*Vscud/c+gateT, where deltaT=T2-T0. Regardless of whether T0 and T1 and T2 are only relative time or absolute time, since the calculations only involve their time difference, the offset error should be irrelevant.

I can't argue with the notion that the system time might have been off, but it should be moot anyway.

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

The issue with using ANY software is to understand its constraints and limitations, and be practical in how you use it; in particular, be aware that digital computers have a finite precision, and that there is a difference between "exactly equal to" and "close enough for all practical purposes".

For example, IRStuff suggested using a tolerance of 10^-10 m (one Angstrom); if you are building a bridge, or planning a trip across the country, that should be "near enough". But what if your calculation relates to the "target" of the Large Hadron Collider? Now you have to hit a target which is the size of a nucleus - MUCH smaller than one Angstrom - so you would have to have a MUCH finer tolerance setting! Check out the definition of the "barn" as a unit of area in nuclear physics: http://en.wikipedia.org/wiki/Barn_(unit) - and yes, Mathcad does know how big (small!) 1 barn is!)

http://julianh72.blogspot.com

## RE: floor w/ and w/o units gives different results

I'm now indeed more aware of the constraints/limitations of software like this. I just found it out the hard way

Funny thing is, I was thinking about the LHC as well. Imagine 9,99...98 vs 10 being the difference of 'finding a Higgs-Boson particle' vs 'finding some random particle' ... I guess (hope) those scientist are well aware of these 'errors'

## RE: floor w/ and w/o units gives different results

With Mathcad's symbolic processor, you could crank barns, outhouses, and sheds to the 10th power and beyond all day long.

TTFN

FAQ731-376: Eng-Tips.com Forum Policies

## RE: floor w/ and w/o units gives different results

Mathcad's TOL = 0.001 works as epsilon, and to simplify consider the alternative function

jfloor(X) = floor(abs(X) + TOL)* sign(X)

We keep these sort of things in a standardized include file.