Bug #3461


Asynchronous fetch for other external resources

Added by Debbie Lockett over 6 years ago. Updated almost 4 years ago.

Start date:
Due date:
% Done:


Estimated time:
Applies to JS Branch:
1.0, Trunk
Fix Committed on JS Branch:
Fixed in JS Release:
SEF Generated with:
Contact person:
Additional contact persons:


Currently we have defined ixsl:schedule-action/@document to allow XML documents to be asynchronously fetched; before calls on doc() and doc-available().

We should provide similar mechanisms for use with json-doc() and unparsed-text(), etc. - i.e. allow other external resources to be asynchronously fetched. Currently calls on these functions will just initiate synchronous fetches.

Note that unparsed-text() is complicated by the fact that the function allows a 2nd argument which provides the resource encoding.

Actions #1

Updated by Debbie Lockett about 4 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • Priority changed from Low to Normal

Changes implemented for asynchronous loading in the browser with Saxon-JS 2.0:

  • ixsl:schedule-action/@document can now also be used to asynchronously load other documents (e.g. text, and JSON), not just XML documents
  • internally we now first store the HTTP response in a responsePool, and then process the content only at the point that a resource retrieval function is actually called (e.g. fn:doc, fn:document-available, fn:json-doc, fn:unparsed-text, etc.); at which point the processed result is stored in one of two pools (documentPool for XML documents, or textResourcePool for other resources)

Some further internal implementation details:

When an asynchronous resource load is called (using ixsl:schedule-action/@document) internally, in the callback for the async "GET" XMLHttpRequest, rather than attempting to process the HTTP response directly (to put the result in the documentPool, as in Saxon-JS 1), we now store the response in the responsePool. The key for an entry is the absolute URI of the resource, and the value will either be an XError (if the fetch failed), or a new JSON object, with properties responseText (whose value is the responseText string from the HTTP response) and contentType (from the "Content-Type" header of the HTTP response).

Subsequently, when any of the resource retrieval functions is called (fn:doc, fn:document-available, fn:json-doc, fn:unparsed-text, etc.), we obtain the relevant response from the response pool, and now process the content as required. For instance, checking for invalid XML characters for the document functions and unparsed-text, but not for json-doc; and parsing as XML for the document functions. The result for the document functions is stored in the documentPool, and the result for the unparsed-text functions (including json-doc) is stored in the textResourcePool. (The intention is that the contentType response information may be used for handling encoding, e.g. when supplied in the 2nd arg of unparsed-text, but this is not yet fully working.)

These pools, and the response processing methods, are of course also used for synchronous loads (i.e. those from the internal platform.readFile method). Note that synchronous loads are discouraged in the browser. Also note that we currently only support synchronous loads in Saxon-JS 2 on the Node.js platform, for loading resources from file system, using the fs module.

As well as tidying up the three main fetch methods for XML and JSON documents, and unparsed text (now called fetchDoc, fetchJsonDoc, and fetchUnparsedText for clarity, and found in the new Fetch.js module); the methods for loading of other resources has also been tidied up (to try to eliminate unnecessary duplication, improve clarity, etc.); e.g. for fetching the source XML and SEF before initiating a transform.

Changes committed on the trunk 2.0 development branch: main changes are in BrowserPlatform.js, NodeJSPlatform.js, CoreFn.js, transform.js, and new module Fetch.js (which gathers relevant methods together, some previously in CoreFn.js e.g. fetchDoc, which didn't really properly belong there).

Actions #2

Updated by Debbie Lockett about 4 years ago

  • Status changed from In Progress to Resolved
  • Fix Committed on JS Branch Trunk added

The new method SaxonJS.getResource() has also been added to the API to be able to pre load resources asynchronously, the method returns a Promise. The results can then be supplied for a transformation in the new SaxonJS.transform() options documentPool and textResourcePool.

Actions #3

Updated by Michael Kay about 4 years ago

  • Fixed in JS Release set to Saxon-JS 2.0
Actions #4

Updated by Debbie Lockett about 4 years ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100
Actions #5

Updated by Debbie Lockett almost 4 years ago

  • Category set to API

Please register to edit this issue

Also available in: Atom PDF Tracking page