Unexpected number of decimal places when casting to xs:decimal

I have some simple code to find the centre of a box, given it's x value and width i.e. x + (width div 2). The x and width values are rounded to 2 decimal places (but are untyped) and when I rounded the result and cast it to xs:decimal I got lots of decimal places. Casting it to a xs:decimal first and then rounding it gave the expected number of decimal places.

My question is why do I get so many decimal places after rounding the value and then casting it to xs:decimal?

Although not in the examples below, casting to xs:double gives the expected number of decimal places.

If this is a problem I can raise an issue.

Some examples are below with the value they output. I cast the numbers to xs:double as the default type. :

    {xs:double(50.88) + (xs:double(100) div 2) => xs:double()}  gives 100.88
    {(xs:double(50.88) + (xs:double(100) div 2)) => round(1) }  gives 100.9
    {(xs:double(50.88) + (xs:double(100) div 2)) => xs:decimal() }  gives 100.8799999999999954525264911353588104248046875
    {(xs:double(50.88) + (xs:double(100) div 2)) => round(1) => xs:decimal() }  gives 100.900000000000005684341886080801486968994140625
    {xs:decimal(round((xs:double(50.88) + (xs:double(100) div 2)), 1)) }  gives 100.900000000000005684341886080801486968994140625
    {(xs:double(50.88) + (xs:double(100) div 2)) => xs:decimal() => round(1) } gives 100.9

Adrian

Replies (2)

RE: Unexpected number of decimal places when casting to xs:decimal - Added by Michael Kay 3 months ago

The short answer is, because that's what the spec says. Functions and Operators 3.1, section 19.1.2.3:

If ST [source type] is xs:float or xs:double, then TV [target value] is the xs:decimal value, within the set of xs:decimal values that the implementation is capable of representing, that is numerically closest to SV [source value]. If two values are equally close, then the one that is closest to zero is chosen.

Saxon uses infinite-precision representations for decimals, so the result is the decimal that is exactly equal to the supplied double, without approximation.

It's possible that whoever wrote that sentence in the spec was imagining that xs:decimal would generally have a smaller range or precision than xs:double. That would also account for why decimal is promoted to double rather than vice-versa.

You might find that you get a more pleasing result by casting the double to string and then casting the string to decimal. The rules for casting double to string don't say that you must deliver the string that is numerically closest to the double, they say (in effect) that you can deliver any string that round-trips back to the same double when cast in the other direction.

The Saxon rules for double-to-decimal do have some benefits when you are handling a heterogenous collection of decimals and doubles = it means that equality testing and ordering are fully transitive.

RE: Unexpected number of decimal places when casting to xs:decimal - Added by Adrian Bird 3 months ago

Ok.

Thanks.

(1-2/2)

Please register to reply

Project

Profile

Help

Saxon