Bug #4608
closedReading JSON resource over HTTP: TypeError: c.codePointAt is not a function
100%
Description
Following stylesheet fails with TypeError: c.codePointAt is not a function
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="3.0"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:ixsl="http://saxonica.com/ns/interactiveXSLT"
exclude-result-prefixes="#all">
<xsl:output method="adaptive"/>
<xsl:param name="url" as="xs:string">https://swapi.dev/api/people/</xsl:param>
<xsl:template name="xsl:initial-template">
<ixsl:schedule-action document="{$url}">
<xsl:call-template name="output-json">
<xsl:with-param name="url" select="$url"/>
</xsl:call-template>
</ixsl:schedule-action>
</xsl:template>
<xsl:template name="output-json">
<xsl:param name="url" select="$url"/>
<xsl:variable name="json-doc1" select="json-doc($url)"/>
<xsl:result-document href="json-doc-test-with-schedule-action-5.json" method="json" indent="yes">
<xsl:sequence select="$json-doc1"/>
</xsl:result-document>
</xsl:template>
</xsl:stylesheet>
Updated by Michael Kay over 4 years ago
Reproduced.
The failure is in NodeJSPlatform.js line 467, where the code is looking at the returned JSON resource to see whether it starts with a byte-order mark. But in fact the variable in question doesn't hold the unparsed JSON as a string, it holds the object that results from parsing the JSON. (A classic problem of Javascript's weak typing).
I suspect there is confusion about whether the resource pool holds the unparsed or parsed resource.
Updated by Michael Kay over 4 years ago
This appears to be caused by https://github.com/axios/axios/issues/907 - axios is parsing the data as JSON even though we requested responseType="text".
There is a pull request with a fix to this problem at https://github.com/axios/axios/pull/2424, but as far as I can see the fix is not yet released.
Updated by Michael Kay over 4 years ago
Changing the code to JSON.stringify() the response if it is not text hits another problem: Saxon-JS now tries to parse the response as XML. That matches the documentation, which says that we only use JSON if the URL ends with .json; but that's clearly unusable, because the user doesn't get the chance to change the URL. So some re-design is needed here. (This was all put together rather hastily before release when we tested against some real-life use cases, and we tried to do the minimum necessary rather than doing a ground-up design).
Updated by Michael Kay over 4 years ago
As a workaround, the code works if you change the URL to https://swapi.dev/api/people/?extension=.json
But that's clearly not an acceptable long-term solution.
Updated by Martin Honnen over 4 years ago
For me the use of https://swapi.dev/api/people/?extension=.json
doesn't fix the problem; I have also used another URL which does end with .json
, like <xsl:param name="url" as="xs:string">https://raw.githubusercontent.com/martin-honnen/martin-honnen.github.io/master/xslt/2020/test2020061801.json</xsl:param>
, however, all attempts lead to the error "TypeError: c.codePointAt is not a function".
Updated by Michael Kay over 4 years ago
Sorry, when I said it fixed the problem, I meant that it fixed the second problem assuming the first one had already been patched.
Updated by Michael Kay over 4 years ago
I've been looking at whether we can add an attribute media-type="application/json" to ixsl:schedule-action. It's not easy because of the way we compile ixsl:schedule-action into a pseudo-function call with a fixed arity. We could do something messy like allowing
document="##application/json doc1.json"
so the media type is disguised as the first URI in the list.
Updated by Michael Kay over 4 years ago
I've now got this working with this design change. Currently this is on the Node.js path only. Specifically:
The document attribute is a whitespace-separated sequence of tokens. Each token is either a media type (prefixed "##"), or a URI. The media type applies to following URIs up to the next media type. The recognised media types are ##application/xml
, ##application/json
, and ##text/plain
. If there is no media type, or if the media type is unrecognized, then ##application/xml
is assumed. The "file extension" of the URI is now ignored.
If the request is for JSON, then we actually request text, but because of the axios bug, we actually get parsed JSON back. We stringify this and put it in the unparsed textResourcePool; it will be reparsed when json-doc() is called, but this time into XDM maps and arrays rather than into Javascript maps and arrays.
Need to consider whether we should do something similar on the browser path. The browser path is different because we use XmlHttpRequest.
Committing the change for now, and adjusting/rerunning test cases.
Updated by Michael Kay over 4 years ago
Debbie has pointed out that there's code on the browser path that deals with this problem and we can reuse the same logic. What happens on the browser is that the schedule-action retrieval always retrieves the resource as plain text, placing it in a responsePool, and the specific request (e.g. doc()
or json-doc()
) gets the unparsed resource from this pool and parses it accordingly. This means that there's no need to know the media type within the ixsl:schedule-action
logic. I've implemented this simplification for this test case and it's working.
This means the only code changes for this bug are now (a) in Expr, handling schedule-action, drop the logic that looks at the file extension, and always retrieve as text; (b) in NodeJSPlatform, if we request text and the resource comes back from axios as parsed JSON, stringify the JSON.
Updated by Debbie Lockett over 4 years ago
See Bug #3461 for further details on the existing mechanism for browser fetches for ixsl:schedule-action
.
Updated by Michael Kay over 4 years ago
- Category set to IXSL extensions
- Status changed from In Progress to Resolved
- Priority changed from Low to Normal
- Applies to JS Branch 2.0 added
- Fix Committed on JS Branch 2.0 added
Also committed a documentation change. Marking as resolved.
Updated by Community Admin over 3 years ago
- Applies to JS Branch 2 added
- Applies to JS Branch deleted (
2.0)
Updated by Community Admin over 3 years ago
- Fix Committed on JS Branch 2 added
- Fix Committed on JS Branch deleted (
2.0)
Updated by Debbie Lockett over 3 years ago
- Status changed from Resolved to Closed
- % Done changed from 0 to 100
- Fixed in JS Release set to Saxon-JS 2.1
Bug fix applied in the Saxon-JS 2.1 maintenance release.
Please register to edit this issue
Also available in: Atom PDF Tracking page