Saxon

If I turn off any optimization with e.g. -opt:0 (and add declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method 'adaptive';) to code gives the wanted result.

Added as a test case to EXPath-binary-extra/EXPath-binary-powerset.

Problem reproduced.

In FilterExpression.java, evaluating $input[....], it has wrongly decided that the value of the predicate is independent of the focus.

When first calculated in FilterExpression.typeCheck(), the value of filterIsIndependent' is correctly set to false(andfilterIsPositionalis correctly set totrue`).

But in the optimize() phase, the re-calculation of filterIsPositional at line 389 wrongly sets it to false. Though on exit from optimize(), filterIsIndependent' is still correctly set to false`.

But then we attempt to rewrite the FLWOR expression as a "for" or "let" expression, which re-does the typeCheck() on the FilterExpression, and this time filterIsIndependent becomes true.

Note that the test runs correctly on the 11 branch.

I suspect (with no evidence) that the difference might be that the calls on bin:xx() functions are now implemented as integrated extension functions rather than reflexive extension functions, which may mean that they are handling context-dependent subexpressions better.

Correction: it doesn't work on the 11 branch. It no longer crashes, but it produces wrong output. The result on SaxonJ is

AA BB AA BB CC AA CC BB CC AA BB CC DD AA DD BB DD AA BB DD CC DD AA CC DD BB CC DD AA BB CC DD

and on SaxonCS we get

A B A B C A C B C A B C D A D B D A B D C D A C D B C D A B C D

(which is the same sequence without the doubling)

If I run under SaxonCS with !method=adaptive, the output is

[(),"A","B",("A","B"),"C",("A","C"),("B","C"),("A","B","C"),"D",("A","D"),("B","D"),("A","B","D"),("C","D"),("A","C","D"),("B","C","D"),("A","B","C","D")]

which is the same as BaseX. Phew!!!

The difference between the J and CS behaviour on the 11 branch is purely by chance. There is a bug in UTF8Writer.write(UnicodeString) whereby if the UnicodeString is a UnicodeChar, then the character is written twice. This bug doesn't affect the query as I was running it in SaxonCS because the output is being sent to a writer, not a stream, so the UTF8Writer is not used. But the UTF8Writer is present in SaxonCS and the bug could manifest itself under different circumstances.

Picking this one up after a gap: current status is that it works correctly on 11.x, but fails on 10.x.

The -explain output for the local:combination() function is

<declareFunction name="local:combination" arity="2" tailRecursive="false">
    <let baseUri="file:/Users/mike/bugs/2021/5110-Honnen/test.xq"
         ns="array=~ bin=http://expath.org/ns/binary err=~ fn=~ local=http://www.w3.org/2005/xquery-local-functions math=~ saxon=~ xs=~ xsi=~ xml=~"
         line="6"
         var="Q{}counter-bin"
         slot="2"
         eval="8">
      <javaCall name="Q{http://expath.org/ns/binary}from-octets" arg0type="1ADI">
        <varRef name="Q{}counter" slot="0"/>
      </javaCall>
      <filter line="7" flags="ib">
        <varRef name="Q{}input" slot="1"/>
        <gc line="10" op="&gt;" card="N:1" comp="CAVC">
          <javaCall name="Q{http://expath.org/ns/binary}to-octets" arg0type="1A2">
            <check card="1" diag="0|0||bin:to-octets">
              <javaCall line="9"
                        name="Q{http://expath.org/ns/binary}and"
                        arg0type="?A2"
                        arg1type="?A2">
                <varRef name="Q{}counter-bin" slot="2"/>
                <javaCall name="Q{http://expath.org/ns/binary}shift"
                          arg0type="?A2"
                          arg1type="1ADI">
                  <javaCall name="Q{http://expath.org/ns/binary}hex" arg0type="1AS">
                    <str val="1"/>
                  </javaCall>
                  <arith line="8" op="-" calc="i-i">
                    <fn name="position"/>
                    <int val="1"/>
                  </arith>
                </javaCall>
              </javaCall>
            </check>
          </javaCall>
          <int val="0"/>
        </gc>
      </filter>
    </let>
  </declareFunction>

which looks perfectly OK. And yet the query works if I set -opt:0. The only significant optimisations seem to be (a) the conversion of a FLWOR expression to a let expression, and (b) the inlining of variables $bin and $j - which looks harmless.

Actually, the thing that's critically wrong in the -explain output is the flags="i" on the filter expression. This means filterIsIndependent - the optimizer has concluded that the value of the predicate does not depend on the focus and can therefore be evaluated in the context of the outer expression - which fails because the outer context has no context item or position. So it's the calculation of dependencies that's wrong -- as explained in comment #3.

Getting closer, but it's tough going.

In the FLWORExpression

     let $j:= position() - 1,
          $bin := bin:and($counter-bin, bin:shift(bin:hex('1'), $j)) 
     return bin:to-octets($bin) > 0]

both the variables get inlined in turn, producing

bin:to-octets(bin:and($counter-bin, bin:shift(bin:hex('1'), position() - 1)) ) > 0]

and for some reason the cached properties of the GeneralComparison bin:to-octets($bin) > 0] (with no focus dependencies) are retained through the rewrite, so the final expression is considered to have no focus dependency either.

Note that because there are two variables, we are taking a slightly different route from normal - with one variable, we would probably turn the FLWORExpression into a LetExpression before eliminating the variable. This is no doubt why the problem has never been encountered before.

Fixed with two changes:

(a) in ExpressionTool.replaceSelectedSubexpressions(), change the recursive call from

replaced = replaceSelectedSubexpressions(....)

to

replaced |= replaceSelectedSubexpressions(...)

so that the variable replaced is true if any operand has been replaced, rather than just the last one.

(b) In ExpressionTool.replaceVariableReferences(), if the call on replaceSelectedSubexpressions() returns true, reset cached properties of the expression. For safety we do this throughout the subtree.

Note that although the test case didn't fail in 11.x, the code here is exactly the same and the same changes should be applied.

There's a secondary issue which is that these optimization rewrites aren't being traced when -explain is used.

Leaving bug as resolved until fix applied to the Saxon 10 maintenance release.