Project

Profile

Help

Bug #4608

Reading JSON resource over HTTP: TypeError: c.codePointAt is not a function

Added by Michael Kay about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
IXSL extensions
Sprint/Milestone:
-
Start date:
2020-06-20
Due date:
% Done:

0%

Estimated time:
Applies to JS Branch:
2.0
Fix Committed on JS Branch:
2.0
Fixed in JS Release:
SEF Generated with:
Company:
-
Contact person:
-
Additional contact persons:
-

Description

Following stylesheet fails with TypeError: c.codePointAt is not a function

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  version="3.0"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:ixsl="http://saxonica.com/ns/interactiveXSLT"
  exclude-result-prefixes="#all">
  
  <xsl:output method="adaptive"/>
  
  <xsl:param name="url" as="xs:string">https://swapi.dev/api/people/</xsl:param>
  
  <xsl:template name="xsl:initial-template">
    <ixsl:schedule-action document="{$url}">
      <xsl:call-template name="output-json">
        <xsl:with-param name="url" select="$url"/>
      </xsl:call-template>
    </ixsl:schedule-action>
  </xsl:template>
  
  <xsl:template name="output-json">
    <xsl:param name="url" select="$url"/>
    <xsl:variable name="json-doc1" select="json-doc($url)"/>
    <xsl:result-document href="json-doc-test-with-schedule-action-5.json" method="json" indent="yes">
      <xsl:sequence select="$json-doc1"/>
    </xsl:result-document>
  </xsl:template>
  
</xsl:stylesheet>

History

#1 Updated by Michael Kay about 2 months ago

Reproduced.

The failure is in NodeJSPlatform.js line 467, where the code is looking at the returned JSON resource to see whether it starts with a byte-order mark. But in fact the variable in question doesn't hold the unparsed JSON as a string, it holds the object that results from parsing the JSON. (A classic problem of Javascript's weak typing).

I suspect there is confusion about whether the resource pool holds the unparsed or parsed resource.

#2 Updated by Michael Kay about 2 months ago

This appears to be caused by https://github.com/axios/axios/issues/907 - axios is parsing the data as JSON even though we requested responseType="text".

There is a pull request with a fix to this problem at https://github.com/axios/axios/pull/2424, but as far as I can see the fix is not yet released.

#3 Updated by Michael Kay about 2 months ago

Changing the code to JSON.stringify() the response if it is not text hits another problem: Saxon-JS now tries to parse the response as XML. That matches the documentation, which says that we only use JSON if the URL ends with .json; but that's clearly unusable, because the user doesn't get the chance to change the URL. So some re-design is needed here. (This was all put together rather hastily before release when we tested against some real-life use cases, and we tried to do the minimum necessary rather than doing a ground-up design).

#4 Updated by Michael Kay about 2 months ago

As a workaround, the code works if you change the URL to https://swapi.dev/api/people/?extension=.json

But that's clearly not an acceptable long-term solution.

#5 Updated by Martin Honnen about 2 months ago

For me the use of https://swapi.dev/api/people/?extension=.json doesn't fix the problem; I have also used another URL which does end with .json, like <xsl:param name="url" as="xs:string">https://raw.githubusercontent.com/martin-honnen/martin-honnen.github.io/master/xslt/2020/test2020061801.json</xsl:param>, however, all attempts lead to the error "TypeError: c.codePointAt is not a function".

#6 Updated by Michael Kay about 2 months ago

Sorry, when I said it fixed the problem, I meant that it fixed the second problem assuming the first one had already been patched.

#7 Updated by Michael Kay about 2 months ago

I've been looking at whether we can add an attribute media-type="application/json" to ixsl:schedule-action. It's not easy because of the way we compile ixsl:schedule-action into a pseudo-function call with a fixed arity. We could do something messy like allowing

document="##application/json doc1.json"

so the media type is disguised as the first URI in the list.

#8 Updated by Michael Kay about 2 months ago

I've now got this working with this design change. Currently this is on the Node.js path only. Specifically:

The document attribute is a whitespace-separated sequence of tokens. Each token is either a media type (prefixed "##"), or a URI. The media type applies to following URIs up to the next media type. The recognised media types are ##application/xml, ##application/json, and ##text/plain. If there is no media type, or if the media type is unrecognized, then ##application/xml is assumed. The "file extension" of the URI is now ignored.

If the request is for JSON, then we actually request text, but because of the axios bug, we actually get parsed JSON back. We stringify this and put it in the unparsed textResourcePool; it will be reparsed when json-doc() is called, but this time into XDM maps and arrays rather than into Javascript maps and arrays.

Need to consider whether we should do something similar on the browser path. The browser path is different because we use XmlHttpRequest.

Committing the change for now, and adjusting/rerunning test cases.

#9 Updated by Michael Kay about 2 months ago

Debbie has pointed out that there's code on the browser path that deals with this problem and we can reuse the same logic. What happens on the browser is that the schedule-action retrieval always retrieves the resource as plain text, placing it in a responsePool, and the specific request (e.g. doc() or json-doc()) gets the unparsed resource from this pool and parses it accordingly. This means that there's no need to know the media type within the ixsl:schedule-action logic. I've implemented this simplification for this test case and it's working.

This means the only code changes for this bug are now (a) in Expr, handling schedule-action, drop the logic that looks at the file extension, and always retrieve as text; (b) in NodeJSPlatform, if we request text and the resource comes back from axios as parsed JSON, stringify the JSON.

#10 Updated by Debbie Lockett about 2 months ago

See Bug #3461 for further details on the existing mechanism for browser fetches for ixsl:schedule-action.

#11 Updated by Michael Kay about 2 months ago

  • Category set to IXSL extensions
  • Status changed from In Progress to Resolved
  • Priority changed from Low to Normal
  • Applies to JS Branch 2.0 added
  • Fix Committed on JS Branch 2.0 added

Also committed a documentation change. Marking as resolved.

Please register to edit this issue

Also available in: Atom PDF Tracking page