Saxon » SaxonJS

Can anyone provide some guidance on pinpointing the bottleneck in a transform?

This is a node.js implementation of Saxon-JS. I'm trying to increase the speed of transforming some XML documents so that I can provide a Synchronous API that responds in under 60sec ideally (230sec is the hard limit of the Application Gateway). I need to be able to handle up to 50MB size XML files as well.

I've run node's built profiler (https://nodejs.org/en/docs/guides/simple-profiling/). But it's tough to make sense of the results given that the source code of the free version of Saxon-JS is not really human-readable.

My Code

const path = require('path');
const SaxonJS = require('saxon-js');
const { loadCodelistsInMem } = require('../standards_cache/codelists');
const { writeFile } = require('../config/fileSystem');
const config = require('../config/config');
const { getStartTime, getElapsedTime } = require('../config/appInsights');

// Used for easy debugging the xslt stylesheet
// Runs iati.xslt transform on the supplied XML
const runTransform = async (sourceFile) => {
    try {
        const fileName = path.basename(sourceFile);

        const codelists = await loadCodelistsInMem();

        // this pulls the right array of SaxonJS resources from the resources object
        const collectionFinder = (url) => {
            if (url.includes('codelist')) {
                // get the right filepath (remove file:// and after the ?
                const versionPath = url.split('schemata/')[1].split('?')[0];
                if (codelists[versionPath]) return codelists[versionPath];
            }
            return [];
        };

        const start = getStartTime();
        const result = await SaxonJS.transform(
            {
                sourceFileName: sourceFile,
                stylesheetFileName: `${config.TMP_BASE_DIR}/data-quality/rules/iati.sef.json`,
                destination: 'serialized',
                collectionFinder,
                logLevel: 10,
            },
            'async'
        );
        console.log(`${getElapsedTime(start)} (s)`);

        await writeFile(`performance_tests/output/${fileName}`, result.principalResult);
    } catch (e) {
        console.log(e);
    }
};

runTransform('performance_tests/test_files/test8meg.xml');

Example console output:

❯ node --prof utils/runTransform.js
SEF generated by Saxon-JS 2.0 at 2021-01-27T17:10:38.029Z with -target:JS -relocate:true
79.938 (s)
❯ node --prof-process isolate-0x102d7b000-19859-v8.log > v8_log.txt

The stylesheet: https://github.com/IATI/IATI-Rulesets/blob/master/rules/iati.xslt

Example XML is attached

Snippet of the V8 log of the largest performance offender:

 [Bottom up (heavy) profile]:
  Note: percentage shows a share of a particular caller in the total
  amount of its parent calls.
  Callers occupying less than 1.0% are not shown.

   ticks parent  name
  33729   52.5%  T __ZN2v88internal20Builtin_ConsoleClearEiPmPNS0_7IsolateE
   6901   20.5%    T __ZN2v88internal20Builtin_ConsoleClearEiPmPNS0_7IsolateE
   3500   50.7%      T __ZN2v88internal20Builtin_ConsoleClearEiPmPNS0_7IsolateE
   3197   91.3%        LazyCompile: *k /Users/nosvalds/Projects/validator-api/node_modules/saxon-js/SaxonJS2N.js:287:264
   3182   99.5%          LazyCompile: *<anonymous> /Users/nosvalds/Projects/validator-api/node_modules/saxon-js/SaxonJS2N.js:682:218
   2880   90.5%            LazyCompile: *d /Users/nosvalds/Projects/validator-api/node_modules/saxon-js/SaxonJS2N.js:734:184

Thanks a lot. There aren't a ton of resources on this anymore to walk myself through. I've also already tried:

• Using the stylesheetInternal parameter with pre-parsed JSON (didn't make a large difference)
• Splitting the document into separate documents that only contain one activities <iati-activity> child element inside the root <iati-activities> root element, transforming each separately, and putting it back together this ended up taking 2x as long.

Best,

Nik