Project

Profile

Help

Bug #6210

closed

Format of numbers generated by fn:json-to-xml

Added by fouad MOUTASSIM 7 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
JSON
Sprint/Milestone:
-
Start date:
2023-09-28
Due date:
% Done:

0%

Estimated time:
Legacy ID:
Applies to branch:
10, 11, 12, trunk
Fix Committed on Branch:
12, trunk
Fixed in Maintenance Release:
Platforms:
.NET, Java

Description

Hello, We are using Saxonica 10.4 and we have a problem when using the function ** fn:json-to-xml.** ** We have numbers in the data. The sisue is we have scientific notation in the output xml (1.219496024E6 for example) Is there a way to keep the original number format without the scientific notation while using the fn:json-to-xml?

I saw a similar case with the function xml-to-json : https://saxonica.plan.io/issues/4860

Thank you in advance.


Files

payload.xml (6.04 KB) payload.xml fouad MOUTASSIM, 2023-09-29 15:48
m-tbw-ws00x-filter-business-keys.xslt (49.6 KB) m-tbw-ws00x-filter-business-keys.xslt fouad MOUTASSIM, 2023-09-29 15:48
Saxon_case.png (69.3 KB) Saxon_case.png fouad MOUTASSIM, 2023-09-29 15:58
Actions #1

Updated by Michael Kay 7 months ago

  • Status changed from New to In Progress

The specification of json-to-xml() says that when the JSON contains a number, the lexical representation of the number is retained in the XML. This appears to be working correctly in Saxon 10, 11, and 12 - the XML will only contain a number in exponential notation if it was already in exponential notation in the JSON.

Could you clarify exactly what you are doing and what output it produces?

Actions #2

Updated by Michael Kay 7 months ago

If I do this from the command line:

java net.sf.saxon.Query -qs:"json-to-xml('1234567890')" -t

the output is

<?xml version="1.0" encoding="UTF-8"?>
<number xmlns="http://www.w3.org/2005/xpath-functions">1234567890</number>
Actions #3

Updated by fouad MOUTASSIM 7 months ago

Hi Sorry for the confusion.

Today after further investigation we got the root cause issue.

we are using "map{'number-formatter': function($x){$x}}" as parameter in the xml-to-json() function to avoid having the scientific notation in the output as you advised us in 2020 and it works well.

but last month we opened a case (Support #6190) for slash escaping and you adiviced us to add "=> parse-json() => serialize(map{ 'method' : 'json', 'use-character-maps' : map { '/' : '/' } })" afer the xml-to-json() function to ignore the slash escape. it works well but it seems that it canceled the first fix regarding the scientific notation.

please find below the used functions and the results:

fn:xml-to-json(fwkmapping:document/document:content/document:xml/*, map{'number-formatter': function($x){$x}})

"ShipViaId":"BESSON/LTL" "EstimatedVolume":1219496.024

fn:xml-to-json(fwkmapping:document/document:content/document:xml/*, map{'number-formatter': function($x){$x}}) => parse-json() => serialize(map{ 'method' : 'json', 'use-character-maps' : map { '/' : '/' } })

"ShipViaId":"BESSON/LTL" "EstimatedVolume":1.219496024E6

Please find attached the payload and the xslt map.

Thank you in advance.

Best Regards. Fouad

Actions #4

Updated by fouad MOUTASSIM 7 months ago

please find a screenshot with actual results and desired result.

Actions #5

Updated by fouad MOUTASSIM 7 months ago

Hello,

Could you please provide us an update regarding this issue.

Thank you in advance.

Best regards.

Actions #6

Updated by Norm Tovey-Walsh 7 months ago

Indeed, running the output of xml-to-json through parse-json does "undo" the number formatting fix.

The JSON requirement to escape forward slashes is a frequent source of awkwardness.

I'm not immediately seeing a workaround that will accomplish both fixes, but I'll make sure we discuss it and see what we can come up with.

Actions #7

Updated by fouad MOUTASSIM 7 months ago

Thank you for your response.

we will wait for updates from your side.

Actions #8

Updated by fouad MOUTASSIM 6 months ago

Hello,

Any news regarding the issue ?

Best regards.

Actions #9

Updated by Vladimir Nesterovsky 6 months ago

I would not rely for json serializer to escape or unescape values. Formally, it is the same data. E.g. we often deal with Hebrew and some json serializers decide to escape Hebrew text, some - not. It is pointless to appeal here.

Basically same story is with xmls, which in addition may reorder attributes.

Actions #10

Updated by fouad MOUTASSIM 6 months ago

Thanks for the explanation. What do you suggest to solve the problem and get the desired result.

Actions #11

Updated by Vladimir Nesterovsky 6 months ago

I would suggest undoing the part

  • serialize(map{ 'method' : 'json', 'use-character-maps' : map { '/' : '/' } }) that you intended to use to prevent escaping of '/';
  • treat result string as json, which would mean that "\/" and "/" are the same values.
Actions #12

Updated by Michael Kay 6 months ago

I'm afraid I can't offer any workaround with the current product that allows you both to control the number formatting, and to avoid escaping of "/".

In the 4.0 specs we have added an option escape-solidus="no" to the JSON serialization method (so you no longer need the workaround with character maps), and at https://github.com/qt4cg/qtspecs/issues/530 I have proposed we should also add this option to the xml-to-json() function.

I'm also going to propose that we add an option for number formatting to JSON serialization.

Actions #13

Updated by Michael Kay 6 months ago

  • Tracker changed from Bug to Feature

Reclasifying this as a feature request. There is no bug -- Saxon is behaving correctly according to the specification.

Actions #14

Updated by Michael Kay 6 months ago

After attempting to specify a new number-format serialization property, I've come to the conclusion that it gets terribly complicated, and probably won't meet the requirements, because you actually want different formats for different numbers in the document.

The JSON serialization method already says that the processor can emit any format that's legal in JSON. So I think the simplest thing to do is to change the Saxon implementation to use exponential notation only in extreme circumstances, for example if the value to be output is an xs:double less than 1e-20 or greater than 1e20.

While looking into this, it also occurred to me that there is another workaround you could use: before serializing the output, transform it so that all the xs:double values are converted to xs:decimal. Saxon will not use exponential notation for xs:decimal values.

Actions #15

Updated by Michael Kay 6 months ago

  • Tracker changed from Feature to Bug

In investigating further I have found a case that IS a bug: the JSON serialization method does not correctly handle integer values outside the range of a 64-bit long. Added test case serialize-json-136

Actions #16

Updated by Michael Kay 6 months ago

  • Category set to JSON
  • Status changed from In Progress to Resolved
  • Priority changed from High to Normal
  • Applies to branch 11, 12, trunk added
  • Fix Committed on Branch 12, trunk added
  • Fix Committed on Branch deleted (10)
  • Platforms .NET added

Patch for integers greater than a long applied to 11.x, 12.x, and main branches.

On 12.x and main branches, changed the formatting of floats and doubles so exponential notation is only used outside the range 1e-18 to 1e18.

Actions #17

Updated by Michael Kay 6 months ago

  • Status changed from Resolved to In Progress

Reopening.

The new rules are generating unreasonable output, such as

"price":8.049999999999979394260662957094609737396240234375
Actions #18

Updated by Michael Kay 6 months ago

  • Status changed from In Progress to Resolved

Previous patch amended.

Actions #19

Updated by O'Neil Delpratt 5 months ago

  • Fixed in Maintenance Release 12.4 added

Bug fix applied in the Saxon 12.4 Maintenance release. Leaving it marked as 'Resolved' until fix applied on Saxon 11.

Please register to edit this issue

Also available in: Atom PDF