Rounding bug in format-number() ?

Added by Anonymous about 14 years ago

Legacy ID: #8611175 Legacy Poster: JMescheder (jmescheder)

Hello, I have got problems with rounding errors in format-number(). This seems to affect all versions > 6.5.5 Example: <xsl:value-of select="format-number($testval,'#.##0,')"/> 1.234.567.890 <xsl:value-of select="format-number($testval,'#.##0,#')"/> 1.234.567.890,1 <xsl:value-of select="format-number($testval,'#.##0,0')"/> 1.234.567.890,1 <xsl:value-of select="format-number($testval,'#.##0,##')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,0#')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,###')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,00#')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,####')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,000#')"/> 1.234.567.890,120 <xsl:value-of select="format-number($testval,'#.##0,#####')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,0000#')"/> 1.234.567.890,1200 <xsl:value-of select="format-number($testval,'#.##0,######')"/> 1.234.567.890,12 <xsl:value-of select="format-number($testval,'#.##0,00000#')"/> 1.234.567.890,12000 <xsl:value-of select="format-number($testval,'#.##0,#######')"/> [b]1.234.567.890,1199999[/b] <xsl:value-of select="format-number($testval,'#.##0,000000#')"/> [b]1.234.567.890,1199999[/b] <xsl:value-of select="format-number($testval,'#.##0,########')"/> [b]1.234.567.890,11999989[/b] <xsl:value-of select="format-number($testval,'#.##0,0000000#')"/> [b]1.234.567.890,11999989[/b] <xsl:value-of select="format-number($testval,'#.##0,#########')"/> [b]1.234.567.890,119999886[/b] <xsl:value-of select="format-number($testval,'#.##0,00000000#')"/> [b]1.234.567.890,119999886[/b] <xsl:value-of select="format-number($testval,'#.##0,##########')"/> [b]1.234.567.890,1199998856[/b] <xsl:value-of select="format-number($testval,'#.##0,00########')"/> [b]1.234.567.890,1199998856[/b] <xsl:value-of select="format-number($testval,'#.##0,000000000#')"/> [b]1.234.567.890,1199998856[/b] Regards, J.Mescheder

Replies (5)

Please register to reply

RE: Rounding bug in format-number() ? - Added by Anonymous about 14 years ago

Legacy ID: #8611222 Legacy Poster: Michael Kay (mhkay)

You haven't said how $testval was initialised, but I imagine you did something like this: <xsl:variable name="testval" select="1234567890.12e0"/> in which case $testval will have a value that is the nearest xs:double value to 1234567890.12. Because no xs:double is exactly equal to this number, the nearest xs:double is probably something like 123456890.1199998856. The format-number() function will then format this value to as many decimal places as you request. So the rounding error is not coming from format-number(), it is coming from the string-to-double conversion that happened when the variable was initialized. If you need to avoid these problems, use xs:decimal rather than xs:double.

RE: Rounding bug in format-number() ? - Added by Anonymous about 14 years ago

Legacy ID: #8611258 Legacy Poster: JMescheder (jmescheder)

Thanks for the quick reply! Sorry, forgot to post the initialisation line: <xsl:variable name="testval">1234567890.12</xsl:variable> How should I alter the initialisation to avoid the problem?

RE: Rounding bug in format-number() ? - Added by Anonymous about 14 years ago

Legacy ID: #8611329 Legacy Poster: JMescheder (jmescheder)

Got it. <xsl:variable name="testval" as="xs:decimal">1234567890.12</xsl:variable> works as expected. Thanks for the hint!

RE: Rounding bug in format-number() ? - Added by Anonymous about 14 years ago

Legacy ID: #8611330 Legacy Poster: Michael Kay (mhkay)

I would write it as <xsl:variable name="testval" as="xs:decimal" select="1234567890.12"/> It's generally better to use the select attribute on xsl:variable rather than using child nodes, unless you actually want the value to be a tree structure. The as="xs:decimal" ensures that the value is a decimal rather than a double. In your code the value is actually a document node containing a text node; when you supply that as input to format-number it is atomized, giving an untypedAtomic value, and this is then converted to an xs:double.

RE: Rounding bug in format-number() ? - Added by Anonymous about 14 years ago

Legacy ID: #8611333 Legacy Poster: JMescheder (jmescheder)

Many thanks for your advice and explaination!

(1-5/5)

Please register to reply

Project

Profile

Help

Saxon