Saxon » SaxonJS

Reproduced.

The failure is in NodeJSPlatform.js line 467, where the code is looking at the returned JSON resource to see whether it starts with a byte-order mark. But in fact the variable in question doesn't hold the unparsed JSON as a string, it holds the object that results from parsing the JSON. (A classic problem of Javascript's weak typing).

I suspect there is confusion about whether the resource pool holds the unparsed or parsed resource.

This appears to be caused by https://github.com/axios/axios/issues/907 - axios is parsing the data as JSON even though we requested responseType="text".

There is a pull request with a fix to this problem at https://github.com/axios/axios/pull/2424, but as far as I can see the fix is not yet released.

Changing the code to JSON.stringify() the response if it is not text hits another problem: Saxon-JS now tries to parse the response as XML. That matches the documentation, which says that we only use JSON if the URL ends with .json; but that's clearly unusable, because the user doesn't get the chance to change the URL. So some re-design is needed here. (This was all put together rather hastily before release when we tested against some real-life use cases, and we tried to do the minimum necessary rather than doing a ground-up design).

As a workaround, the code works if you change the URL to https://swapi.dev/api/people/?extension=.json

But that's clearly not an acceptable long-term solution.

For me the use of https://swapi.dev/api/people/?extension=.json doesn't fix the problem; I have also used another URL which does end with .json, like <xsl:param name="url" as="xs:string">https://raw.githubusercontent.com/martin-honnen/martin-honnen.github.io/master/xslt/2020/test2020061801.json</xsl:param>, however, all attempts lead to the error "TypeError: c.codePointAt is not a function".

Sorry, when I said it fixed the problem, I meant that it fixed the second problem assuming the first one had already been patched.

I've been looking at whether we can add an attribute media-type="application/json" to ixsl:schedule-action. It's not easy because of the way we compile ixsl:schedule-action into a pseudo-function call with a fixed arity. We could do something messy like allowing

document="##application/json doc1.json"

so the media type is disguised as the first URI in the list.

I've now got this working with this design change. Currently this is on the Node.js path only. Specifically:

The document attribute is a whitespace-separated sequence of tokens. Each token is either a media type (prefixed "##"), or a URI. The media type applies to following URIs up to the next media type. The recognised media types are ##application/xml, ##application/json, and ##text/plain. If there is no media type, or if the media type is unrecognized, then ##application/xml is assumed. The "file extension" of the URI is now ignored.

If the request is for JSON, then we actually request text, but because of the axios bug, we actually get parsed JSON back. We stringify this and put it in the unparsed textResourcePool; it will be reparsed when json-doc() is called, but this time into XDM maps and arrays rather than into Javascript maps and arrays.

Need to consider whether we should do something similar on the browser path. The browser path is different because we use XmlHttpRequest.

Committing the change for now, and adjusting/rerunning test cases.

Debbie has pointed out that there's code on the browser path that deals with this problem and we can reuse the same logic. What happens on the browser is that the schedule-action retrieval always retrieves the resource as plain text, placing it in a responsePool, and the specific request (e.g. doc() or json-doc()) gets the unparsed resource from this pool and parses it accordingly. This means that there's no need to know the media type within the ixsl:schedule-action logic. I've implemented this simplification for this test case and it's working.

This means the only code changes for this bug are now (a) in Expr, handling schedule-action, drop the logic that looks at the file extension, and always retrieve as text; (b) in NodeJSPlatform, if we request text and the resource comes back from axios as parsed JSON, stringify the JSON.

See Bug #3461 for further details on the existing mechanism for browser fetches for ixsl:schedule-action.