Project

Profile

Help

Bug #4616

closed

default-collation

Added by Michael Kay over 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
XSLT Conformance
Sprint/Milestone:
-
Start date:
2020-06-26
Due date:
% Done:

100%

Estimated time:
Applies to JS Branch:
2
Fix Committed on JS Branch:
2
Fixed in JS Release:
SEF Generated with:
Platforms:
Company:
-
Contact person:
-
Additional contact persons:
-

Description

If an [xsl:]default-collation attribute appears in the stylesheet, the XX compiler appears to ignore it.

Given the input

      <xsl:copy-of select="saxon:highest($data, my:val#1)"
        default-collation="http://www.w3.org/2005/xpath-functions/collation/html-ascii-case-insensitive" />
      <xsl:copy-of select="max($data)"
        default-collation="http://www.w3.org/2005/xpath-functions/collation/html-ascii-case-insensitive" />

The XJ compiler generates a defaultCollation attribute on the sequence node representing the sequence constructor (I think it detects that the static context for both instructions is the same). It also generates a collation attribute on the fn:max function call - this is done by SystemFunction.export calling out.emitRetainedStaticContext().

The output of the XX compiler does not include the default collation anywhere, which means that neither the built-in function max() nor the extension function saxon:highest() can take it into account.

Actions #1

Updated by Michael Kay over 4 years ago

In principle this should be tested by XSLT3 test collations-0118.

However, that test, with most of the others in the collations test-set, is on the exceptions list because it depends on DTD parsing - the collation in use is represented using an entity reference.

I have modified most of the XSLT3 tests in the collations test-set so they no longer depend on use of entity references, which means it is now possible to run them in Saxon-JS. Most of them are failing.

Actions #2

Updated by Michael Kay over 4 years ago

A test with xsl:perform-sort

   <out xsl:default-collation="http://www.w3.org/2005/xpath-functions/collation/html-ascii-case-insensitive">
      <xsl:perform-sort select="$data">
        <xsl:sort select="."/>
      </xsl:perform-sort>
    </out>

shows that for this case, the default collation is being respected, but it is also output as a (non-namespaced) attribute in the result:

<out defaultCollation="http://www.w3.org/2005/xpath-functions/collation/html-ascii-case-insensitive">....</out>

If I move the default-collation attribute to the containing xsl:template element it works correctly.

The conversion of @xsl:default-collation to a no-namespace @defaultCollation attribute is done by the compiler in normalize.xsl line 288; it's not clear what the rationale is here.

Actions #3

Updated by Michael Kay over 4 years ago

  • Status changed from New to In Progress
  • Assignee changed from Michael Kay to John Lumley
  • Priority changed from Low to Normal
  • Applies to JS Branch 2.0 added
Actions #4

Updated by John Lumley about 4 years ago

The code in pastFast.bind(), which binds function calls, handles the case of arguments which are to be treated as collations, but not cases where absent arguments default to (any) default collation. As this processing operates (only) deep in XPath parsing, this needs this code section to get smarter about adding default collations.

Actions #5

Updated by John Lumley about 4 years ago

Adding:

 if(collationArg > args.length - 1) { // Need a collation - none supplied
      checkedArgs.push(new StringLiteral(env.defaultCollation));
 }

after the loop static checking arguments in parseFast.bind() seems to pass simple tests.

Actions #6

Updated by Debbie Lockett about 4 years ago

Patch committed. Further test failures in xslt30 collations test set to be reviewed.

Actions #7

Updated by Michael Kay about 4 years ago

Let's work through the tests in the collations test set in turn. Currently getting

RESULTS: 19 passes, 23 failures, 0 wrong errors, 0 not run

First failure is collations-0106. This declares a key; the xsl:stylesheet element has a default-collation attribute, which doesn't find its way onto the key declaration in the SEF.

Applying the same fix as for bug #4660 solves this one, Now getting:

RESULTS: 20 passes, 22 failures, 0 wrong errors, 0 not run
Actions #8

Updated by Michael Kay about 4 years ago

  • Assignee changed from John Lumley to Michael Kay

Next failure: collations-0109. This is a call on the fn:compare() function with a default-collation defined at the xsl:template level.

In the SEF file, the templateRule object has a correct defaultCollation property.

The implementation of fn:compare calls Compare.getCollationUri() to get the collation URI. This makes no attempt to access the default collation; instead it appears to fall back to the codepoint collation URI. Changing it to look for a default collation fixes this test and gets us to

RESULTS: 21 passes, 21 failures, 0 wrong errors, 0 not run
Actions #9

Updated by Michael Kay about 4 years ago

Next failure: collations-0111

This uses starts-with() with a default-collation on the containing xsl:value-of instruction.

The defaultCollation property is not present in the SEF file.

Fixing this by adding the defaultCollation attribute to the list of attributes copied at creating-new-nodes#289, gets us to

RESULTS: 31 passes, 11 failures, 0 wrong errors, 0 not run

However, this is rather unsatisfactory, because this fix was specific to a default-collation attribute appearing on the xsl:value-of instruction. If the code was wrong there, it's probably wrong elsewhere. But we'll push on for now.

Actions #10

Updated by Michael Kay about 4 years ago

Next failure is collations-0121.

Here an xsl:default-collation attribute appearing on a literal result document has surfaced in the final output as a defaultCollation attribute on the output element. This is the failure reported in comment #2.

The defaultCollation appears as an attribute of the element that's supplied to the template rule at creating-new-nodes#470. At this point it's impossible to distinguish it from a user-defined attribute of this name.

Changing attributes.xsl#65 to output leave the attribute named xsl:default-collation, rather than renaming it to defaultCollation; and changing creating-new-nodes#495 to pick up this attribute.

No; we also need to change normalize.xsl#291

I've now finally got rid of the defaultCollation attribute in the output, and the SEF file now looks correct, but it's still not using the specified collation. This is because we've generated a compareToString instruction that explicitly uses the codepoint collation: "comp": "GAC|http://www.w3.org/2005/xpath-functions/collation/codepoint".

This appears to derive from the XPath parser, Comparison.js#129 which calls env.getDefaultCollation().

The <xpath> element supplied to xpath.xsl#958, which drives the XPath parsing, doesn't have an @defaultCollation attribute.

Presumably this is because the template with mode=create.xpath in XSLT-SEF-compiler#1827 was called without the tunnel parameter $default-collation being set.

This is proving a battle. I'm now getting an error XTDE0420 with very little diagnostics.

Fixed that and its finally passing. Now getting:

RESULTS: 36 passes, 6 failures, 0 wrong errors, 0 not run

Can't help feeling there's a lot of quackery involved in this. We seem to have at least four mechanisms for passing the static context: individual tunnel parameters, an $attr tunnel parameter holding multiple attributes, a $staticContext tunnel parameter, plus individual attributes sometimes in the ex namespace and sometimes not. This could really do with rationalisation.

Actions #11

Updated by Michael Kay about 4 years ago

Next failure: collations-0127. This is giving

s=collations t=collations-0127
Actual error: XTSE0090
Expected success, got XTSE0090:

This is a new test; collations-0127 simply imports collations-0126.xsl, which has a default-collation on the xsl:stylesheet element.

The test driver gives us no useful diagnostics, but running the stylesheet from the command line gives:

Error XTSE0090 at xsl:import/@default-collation on line 6 in collations/collations-0127.xsl:
 xsl:import must not have attribute @default-collation

Adding a stack trace to f:wrongAttribute() in normalize.xsl gives:

WrongAttribute!!
    at function Q{MyFunctions}wrongAttribute in normalize.xsl#822

    Processing element <xsl:stylesheet>
    at template rule in static.xsl#131
    Processing element <xsl:stylesheet>
    at template rule in XSLT-SEF-compiler.xsl#388
    Processing document #1
    at template rule in XSLT-SEF-compiler.xsl#280

We're copying the default-collation attribute from the xsl:stylesheet element to all its children, which is fine: a default-collation attribute is allowed on any element, including xsl:import. The error is spurious. Fixed at static.xsl#544 by allowing all standard attributes on xsl:include/import.

Now getting

RESULTS: 37 passes, 5 failures, 0 wrong errors, 0 not run
Actions #12

Updated by Michael Kay about 4 years ago

Next failure: collations-0301

This one doesn't use a specific collation URI, rather it uses attributes such as case-order and lang to control sorting; and the problem with the result isn't the sort order, it's missing spaces (between first name and last name) in the output.

The spaces should be produced by a whitespace-only xsl:text element: <t:text> </t:text>. Perhaps my fix for bug #4681 affected this (the fix has been found to cause other test failures).

In the SEF file, the relevant xsl:text element becomes valueOf(str(val="")) which is incorrect (the space has been lost).

I think it's being lost in template name="staticText" (static.xsl#863) which after the fix for #4681 is now processing xsl:text elements, and needs special logic if they are whitespace-only.

Fixed this: modified the patch for #4681 so the xsl:next-match instruction is executed with xml-space-preserve=true.

Results now:

RESULTS: 41 passes, 1 failures, 0 wrong errors, 0 not run
Actions #13

Updated by Michael Kay about 4 years ago

The remaining test failure is collations-1006. This uses UCA collations with fallback=no. We are getting error FOCH0002 (unsupported collation). I think this is an acceptable outcome: the spec says:

If the fallback parameter is present with the value no, then the implementation must either use a collation that conforms with the rules in the Unicode specifications for the requested tailoring, or fail with a static or dynamic error indicating that it does not provide the collation

I will change the test to allow this outcome.

Results now:

RESULTS: 42 passes, 0 failures, 0 wrong errors, 0 not run
Actions #14

Updated by Michael Kay about 4 years ago

  • Category set to XSLT Conformance
  • Status changed from In Progress to Resolved
  • Fix Committed on JS Branch Trunk added

I'm now committing these changes. There are still some XSLT3 test failures, but I'm hoping they are unrelated. Current results:

Duration: 7m 40.15s
RESULTS: 6304 passes, 34 failures, 46 wrong errors, 5349 not run
Failures by test-set: 
  accumulator: 4
  attribute: 1
  bug: 1
  collection: 2
  document: 1
  error: 3
  function-lookup: 1
  global-context-item: 1
  higher-order-functions: 2
  match: 2
  merge: 2
  message: 4
  namespace: 2
  number: 1
  output: 1
  shadow: 1
  snapshot: 1
  system-property: 3
  type: 1

Actions #15

Updated by Michael Kay about 4 years ago

  • Status changed from Resolved to In Progress

Test collations-1006 is still failing. Although the test now allows FOCH0002, we aren't returning that; we are returning a success result that appears to ignore the requested collation.

The XX compiler is generating a SEF file in which the default-collation attribute is correctly present at the xsl:template level, but calls on functions such as starts-with() have an extra parameter injected, set to the codepoint collation URI.

Actions #16

Updated by Michael Kay almost 4 years ago

Confirmed that the status of collations-1006 is still as described: in the SEF file, the templateRule element has a correct defaultCollation attribute, but nested expressions such as compareToString and fn name="starts-with" have an explicit reference to the codepoint collation, which overrides the default.

Actions #17

Updated by Michael Kay almost 4 years ago

The XX compiler code at xpath.xsl#984 is expecting the attribute @defaultCollation to be present on the xpath element, but in this test case it is not present. The attribute is present when compiling an expression appearing on xsl:if/@test, but it is not present in an expression appearing in an AVT or TVT.

The code at value-templates#65 is not generating the @defaultCollation attribute on the generated xpath element because the value of the tunnel parameter uniform-static-context is true. The code that generates xpath elements for @test or @select attributes at XSLT-SEF-compiler#1826 generates a defaultCollation attribute regardless of the value of uniform-static-context.

Changed value-templates to do the same - i.e. to generate @defaultCollation and @xpathDefaultNamespace attributes regardless of the value of $uniform-static-context. This has the desired effect of generating the correct collation URI in the SEF file.

Actions #18

Updated by Michael Kay almost 4 years ago

Believed fixed; have committed the change (in value-templates.xsl line 84); not fully tested because of problems with the build environment.

Actions #19

Updated by Community Admin almost 4 years ago

  • Applies to JS Branch 2 added
  • Applies to JS Branch deleted (2.0)
Actions #20

Updated by Debbie Lockett almost 4 years ago

  • Status changed from In Progress to Resolved
  • Fix Committed on JS Branch 2 added
  • Fix Committed on JS Branch deleted (Trunk)

Confirm the fix works and has been tested (collations-1006 now passes).

Actions #21

Updated by Debbie Lockett almost 4 years ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100
  • Fixed in JS Release set to Saxon-JS 2.1

Bug fix applied in the Saxon-JS 2.1 maintenance release.

Please register to edit this issue

Also available in: Atom PDF Tracking page