Project

Profile

Help

Bug #5338

closed

XX Compiler reports "Unknown accumulator NNNNN" when an accumulator-before or -after call occurs in a global variable

Added by Alexander Stein 7 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
XX compiler
Sprint/Milestone:
-
Start date:
2022-02-19
Due date:
% Done:

100%

Estimated time:
Applies to JS Branch:
2
Fix Committed on JS Branch:
2
Fixed in JS Release:
SEF Generated with:
Platforms:
Company:
Fed
Contact person:
Alexander J. Stein
Additional contact persons:
Wendell A. Piez

Description

Hello, we have a pipelines and software using Saxon HE (Java) variant and the SaxonJS variant with NodeJS to build CLI tooling. We have encountered a problem in which running Saxon HE will process a collection of XSL transforms in shell scripts (running Saxon HE 9.x and/or 10.x) and successfully execute. Developers wrapping compiling those same transforms into SEF artifacts receive compilation failures.

https://github.com/usnistgov/metaschema/issues/180

We receives exceptions like this only with SaxonJS.

Failed to compile stylesheet: Static error in XPath on line 17 in fixtures/accumulator_simple.xsl {accumulator-after('total-items')}: Unknown accumulator Q{}total-items
Error Q{http://www.w3.org/2005/xqt-errors}XTDE3340 at xpath.xsl#963
    Failed to compile stylesheet:
Error Q{http://www.w3.org/2005/xqt-errors}XTDE3340 at xpath.xsl#963
    Static error in XPath on line 17 in fixtures/accumulator_simple.xsl {accumulator-after('total-items')}: Unknown accumulator Q{}total-items
Failed to compile stylesheet: Static error in XPath on line 27 in fixtures/accumulator_basic.xsl {accumulator-after('total-items')}: Unknown accumulator Q{}total-items
Error Q{http://www.w3.org/2005/xqt-errors}XTDE3340 at xpath.xsl#963
    Failed to compile stylesheet:
Error Q{http://www.w3.org/2005/xqt-errors}XTDE3340 at xpath.xsl#963
    Static error in XPath on line 27 in fixtures/accumulator_basic.xsl {accumulator-after('total-items')}: Unknown accumulator Q{}total-items
Failed to compile stylesheet: Static error in XPath on line 28 in fixtures/accumulator_namespaces.xsl {accumulator-after('example:total-items')}: Unknown accumulator Q{https://example.com/ns/custom/0.1}total-items
Error Q{http://www.w3.org/2005/xqt-errors}XTDE3340 at xpath.xsl#963
    Failed to compile stylesheet:
Error Q{http://www.w3.org/2005/xqt-errors}XTDE3340 at xpath.xsl#963
    Static error in XPath on line 28 in fixtures/accumulator_namespaces.xsl {accumulator-after('example:total-items')}: Unknown accumulator Q{https://example.com/ns/custom/0.1}total-items

We have build a reproduction library here. With or without namespaces (I re-read the XSLT 3.0 standard), it is not clear to me if an accumulator requires a unique namespace or not. I built simple reproductions and none of these seem to compile with current SaxonJS from NPMJS. Instructions to reproduce in the repo below.

https://github.com/aj-stein-nist/issue180example.git

Let me know if this is my misunderstanding of syntax for accumulators, whether SaxonJS is working as intended, or whether Saxon HE is working as intended. Similar code compiles with Saxon as intended. Thanks.

Actions #1

Updated by Alexander Stein 7 months ago , visible to Alexander J. Stein (alexander.stein@nist.gov) from Fed

  • Company set to Fed
  • Contact person set to Alexander J. Stein

Hello, any updates on this issue?

Actions #2

Updated by Michael Kay 7 months ago

I have reproduced the problem and am re-familiarising myself with the code in this area.

The error originates from line 367 of parseFast.js. It is looking for an accumulator with name Q{}total-items, but the index of accumulator names in the static context is empty.

I am suspicious about line 176 in xpath.xsl which initializes the $accumulators tunnel variable with a reference to ex:accumulator (on the child axis). All other references to top-level components in this template rule use either .//ex:function or ./*:co/ex:function.

No, I've checked: the package element does have ex:accumulator as a direct child, the accumulator element exists and has the right @name value.

And at line 179 the $accumulators variable has the correct value {"Q{}total-items":"FM k[1AS ] v[1ADI ] "}.

Actions #3

Updated by Michael Kay 7 months ago

I note that the call on accumulator-after() succeeds if we convert <xsl:variable name="total-items-count"> to be a local variable rather than a global variable.

So I think that the problem might be that at xpath.xsl#196 when processing global variables, we aren't passing the tunnel parameter $accumulators through.

This seems to fix it. Solution: in the binding of variable $typed-globals at xpath.xsl#196, add:

<xsl:with-param name="accumulators" tunnel="true" select="$accumulators"
                            use-when="not($COMPILE_GLOBAL)"/>

Note in passing: the code here is over-complex because it includes multiple options for configuring the compiler, as a results of experiments to improve performance. We ought to decide which of these options we are actually using, and get rid of the code for the paths that are dormant.

Actions #4

Updated by Michael Kay 7 months ago

  • Subject changed from Discrepancy between behavior and standard conformance with accumulator-after in SaxonJS and Saxon HE (Java) to XX Compiler reports "Unknown accumulator NNNNN" when an accumulator-before or -after call occurs in a global variable
  • Category set to XX compiler
  • Status changed from New to In Progress
  • Assignee set to Michael Kay
Actions #5

Updated by Michael Kay 7 months ago

A couple of observations:

(a) you may be able to work around the problem by moving the accumulator-after call so it is not in a global variable, or by changing the argument to accumulator-after so it is not a string literal (e.g use a global parameter)

(b) It's probably rather inefficient to use an accumulator like this, if you only want the value at the end of the document. That's because the current accumulator value will potentially be stored on every node in the tree. Computing it conventionally using template rules with a special mode, or using xsl:iterate or fn:fold-left(), would probably perform better.

Actions #6

Updated by Alexander Stein 7 months ago

  • Additional contact person Wendell A. Piez added

OK, thanks Michael. I will evaluate this approach in the context of our code base once I sync up with the team, and more quickly in the same reproduction library of course.

I will follow up in the next few days.


From: Saxonica Developer Community
Sent: Friday, February 25, 2022 8:06 AM
Subject: [SaxonJS - Bug #5338] Discrepancy between behavior and standard conformance with accumulator-after in SaxonJS and Saxon HE (Java)

Actions #7

Updated by Alexander Stein 7 months ago

Also, thanks for this feedback on the transforms. I will definitely want to sync up with the team to address these as well, I need their feedback to understand the original code (not repro) and the implications of b better. We will follow up soon enough.


From: Saxonica Developer Community
Sent: Friday, February 25, 2022 8:14 AM
Subject: [SaxonJS - Bug #5338] XX Compiler reports "Unknown accumulator NNNNN" when an accumulator-before or -after call occurs in a global variable

Actions #8

Updated by Michael Kay 7 months ago

Added test case accumulator-090.

Actions #9

Updated by Wendell Piez 7 months ago

Hello Mike and Saxonians,

Deeper background/tldr: we installed the offending code into our pipeline as a replacement for earlier code that addressed functional requirements (wrt pipeline internals) that we were also splitting out and refactoring as we worked (under a deadline IIRC). So despite its being a significant performance improvement over what it replaced, I am not surprised to hear there are alternatives we can try as well to make it even better. Yet we do feel bound to report bugs we (think we) find -- and AJ being new to our team, we decided to assign this one to him.

So adding to AJ's thanks, we will definitely be looking at the proposed alternatives. Maybe xsl:iterate is the first obvious thing to try.

Actions #10

Updated by Michael Kay 7 months ago

Please keep reporting them! My commentary wasn't based on any thought no-one should be doing this, but more on thinking "why is it that no-one has found this bug before? What are they doing that other people aren't doing?"

Actions #11

Updated by John Lumley 7 months ago

When writing the code for the XX compiler, there are so many corner cases (many thousands of them?) that might need to be satisfied that it’s pretty much impossible to cover them all. In this case during the XPath compiling of global variables, rather than other declarations, such as templates, I neglected to add the tunnelled variable defining in-scope accumulators, hence the error.

Unfortunately none of the then XSLT test suite cases covered reference to an accumulator in a global variable…. hence the failure. These cases happen, but as Michael says, some of them are very rarely used in practice… That’s why reporting these to the Saxonica forum is so helpful!

Sent from my iPad

On 25 Feb 2022, at 22:03, Saxonica Developer Community wrote:

Please keep reporting them! My commentary wasn't based on any thought no-one should be doing this, but more on thinking "why is it that no-one has found this bug before? What are they doing that other people aren't doing?"

Actions #12

Updated by Michael Kay 7 months ago

  • Status changed from In Progress to Resolved
  • Fix Committed on JS Branch Trunk added

Fix committed.

Actions #13

Updated by Wendell Piez 6 months ago

Now we have solid alternatives and in the meantime we have also found a bug in applying this approach to the problem it is supposed to solve -- so good work all around, if exposing subtle bugs counts as good work. We will post again with more edge cases if we find any. With thanks for responsiveness and looking forward to the repair (since we are not done with accumulators even if we won't be doing quite this) - Cheers, Wendell.

Actions #14

Updated by Debbie Lockett 6 months ago

  • Fix Committed on JS Branch 2 added
  • Fix Committed on JS Branch deleted (Trunk)
Actions #15

Updated by Debbie Lockett 5 months ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100
  • Fixed in JS Release set to SaxonJS 2.4

Bug fix applied in the SaxonJS 2.4 maintenance release.

Please register to edit this issue

Also available in: Atom PDF Tracking page