Saxon

I have set the " tracecodeinjector.isApplicable" to return true for FixedElement. And inside that FixedElement there is a "for" expression which is not yet wrapped.

Looking more into this in the method "net.sf.saxon.expr.parser.ExpressionTool.injectCode(Expression, CodeInjector)"

the expression is a TraceExpression over a FixedElement, but the code:

 else if (!(exp instanceof TraceExpression)){
            for (Operand o : exp.operands()) {
                o.setChildExpression(injectCode(o.getChildExpression(), injector));
            }

goes deep only if the expression is not yet a trace expression, in my case there is already a trace expression over the FixedElement but I still want it to go deep, so I will probably patch it something like:

    public static Expression injectCode(Expression exp, CodeInjector injector) {
        if (exp instanceof FLWORExpression) {
            ((FLWORExpression)exp).injectCode(injector);
        } else {
          for (Operand o : exp.operands()) {
            if (!(o.getChildExpression() instanceof TraceExpression)){
              o.setChildExpression(injectCode(o.getChildExpression(), injector));
            }
          }
        }
        return injector.inject(exp);
    }

There seem to be a number of separate but interacting problems here.

I think at some stage there was a change in strategy from injecting trace code at the time the expression tree is being built (i.e. during expression parsing), to injecting it during a recursive traversal of the tree after parsing is complete.

So Configuration.makeXQueryExpression() calls mainModule.getCodeInjector().process() which calls ExpressionTool.injectCode() which embarks on a recursive traversal; but the recursion stops if it hits an existing TraceExpression. This is going to happen, because the XPathParser and XQueryParser in many places call XPathParser.makeTracer which invokes the non-recursive TraceCodeInjector.inject().

ExpressionTool.injectCode() appears to be new in 10.x, and it looks as if it's primarily designed for XSLT, where it will successfully recurse down the XSLT-level instruction tree, until it hits an XPath expression, where it will find that the injection has already been done by the XPath parser. But in XQuery we don't get this two-language model, so it's less clear how this is intended to operate -- and it's clear that it hasn't been properly tested.

I think this is going to be a bit trickier to resolve than I first thought.

Right, I agree with everything, here are some things I did on my side to make this kind of work:

On the method "net.sf.saxon.Configuration.setCompileWithTracing(boolean)" I overwrote the injector for XQuery:

if (defaultStaticQueryContext != null) {
            if (trace) {
                defaultStaticQueryContext.setCodeInjector(new TraceCodeInjector() {
                	//////// OXYGEN PATCH START //////////////////
                	//EXM-49354 Create trace expressions over multiple XQuery constructs
                      @Override
                      protected boolean isApplicable(Expression exp) {
                        boolean appl = exp.isInstruction() || exp instanceof LetExpression || exp instanceof FixedAttribute
                            || exp instanceof FixedElement || exp instanceof FunctionCall || exp instanceof Choose;
                        return appl;
                      }
                      /**
                       * @see net.sf.saxon.trace.TraceCodeInjector#process(net.sf.saxon.trace.TraceableComponent)
                       */
                      @Override
                      public void process(TraceableComponent component) {
                    	  Expression newBody = ExpressionToolUtil.injectCodeXQuery(component.getBody(), this);
                    	  component.setBody(newBody);
                    	  ComponentTracer trace = new ComponentTracer(component);
                    	  component.setBody(trace);
                      }
                	//////// OXYGEN PATCH END //////////////////
                });

I created a "ExpressionToolUtil.injectCodeXQuery" similar to "net.sf.saxon.expr.parser.ExpressionTool.injectCode" but which recurses all the way:

public static Expression injectCodeXQuery(Expression exp, CodeInjector injector) {
      if (exp instanceof FLWORExpression) {
        ((FLWORExpression)exp).injectCode(injector);
      } else {
        if(exp instanceof TraceExpression) {
          Expression child = ((TraceExpression) exp).getChild();
          Iterable<Operand> childOps = child.operands();
          for (Operand operand : childOps) {
            operand.setChildExpression(injectCodeXQuery(operand.getChildExpression(), injector));
          }
        } else {
          for (Operand o : exp.operands()) {
            o.setChildExpression(injectCodeXQuery(o.getChildExpression(), injector));
          }
        }
      }
      return injector.inject(exp);
    }

I called "net.sf.saxon.expr.parser.ExpressionToolUtil.injectCodeXQuery(Expression, CodeInjector)" also from the "net.sf.saxon.query.XQueryParser.parseFLWORExpression()" on the return expression because I would not get callbacks inside returns.

I still have problems, I do not get callbacks for the "return" token itself and looking at how the TraceExpression changed I will probably never will. The "XQueryParser.parseFLWORExpression()"" method used to wrap the return expression inside a "TraceExpression" with a specific RETURN type but now trace expressions can no longer with specific types.

I also do not get callbacks for "let" expressions inside function declarations but I will look more over this one today.

Oh and I also do not get cals for the "then" and "else" tokens, I get calls for what's inside them but not for the tokens themselves.

About let expressions inside function declarations not being wrapped in trace expressions, in the method "XQueryParser.parseFunctionDeclaration(AnnotationList)" after the body is parsed I call my "ExpressionToolUtil.injectCodeXQuery" method:

Expression body = parseExpression();
                ExpressionTool.setDeepRetainedStaticContext(body, env.makeRetainedStaticContext());
                //OXYGEN PATCH START FOR EXM-49354, inject trace information in body.
                if(codeInjector != null) {
                  body = ExpressionToolUtil.injectCodeXQuery(body, codeInjector);
                }
                ///OXYGEN PATCH END FOR EXM-49354
                func.setBody(body);

Regarding "then" and "else", the tracing generally only traces the expressions that are executed, it doesn't attempt to identify their role within the parent expression. That's because we don't have this information readily available in a convenient place.

With these changes, trace level "normal" is giving output:

<query line="-1" module="test.xq">
  <element name="BigBoss_subordinates" line="1" column="1" module="test.xq">
   <element name="person" line="7" column="11" module="test.xq">
    <att name="id" line="7" column="11" module="test.xq">
    </att>
    <element name="name" line="9" column="14" module="test.xq">
    </element>
   </element>
   <element name="person" line="7" column="11" module="test.xq">
    <att name="id" line="7" column="11" module="test.xq">
    </att>
    <element name="name" line="9" column="14" module="test.xq">
    </element>
   </element>
   <element name="person" line="7" column="11" module="test.xq">
    <att name="id" line="7" column="11" module="test.xq">
    </att>
    <element name="name" line="9" column="14" module="test.xq">
    </element>
   </element>
   <element name="person" line="7" column="11" module="test.xq">
    <att name="id" line="7" column="11" module="test.xq">
    </att>
    <element name="name" line="9" column="14" module="test.xq">
    </element>
   </element>
   <element name="person" line="7" column="11" module="test.xq">
    <att name="id" line="7" column="11" module="test.xq">
    </att>
    <element name="name" line="9" column="14" module="test.xq">
    </element>
   </element>
  </element>
 </query>
</trace>

while trace level "high" gives:

<trace saxon-version="10.6" >
 <query line="-1" module="test.xq">
  <element name="BigBoss_subordinates" line="1" column="1" module="test.xq">
   <FOR var="$person" line="3" column="8" module="test.xq">
    <LET var="$link" line="4" column="8" module="test.xq">
    </LET>
   </FOR>
   <FOR var="$person" line="3" column="8" module="test.xq">
    <LET var="$link" line="4" column="8" module="test.xq">
     <WHERE condition="fn:exists(...)" line="5" column="11" module="test.xq">
      <WHERE condition="fn:compare(...) eq 0" line="5" column="11" module="test.xq">
       <element name="person" line="7" column="11" module="test.xq">
        <att name="id" line="7" column="11" module="test.xq">
        </att>
        <element name="name" line="9" column="14" module="test.xq">
        </element>
       </element>
      </WHERE>
     </WHERE>
    </LET>
   </FOR>
   <FOR var="$person" line="3" column="8" module="test.xq">
    <LET var="$link" line="4" column="8" module="test.xq">
     <WHERE condition="fn:exists(...)" line="5" column="11" module="test.xq">
      <WHERE condition="fn:compare(...) eq 0" line="5" column="11" module="test.xq">
       <element name="person" line="7" column="11" module="test.xq">
        <att name="id" line="7" column="11" module="test.xq">
        </att>
        <element name="name" line="9" column="14" module="test.xq">
        </element>
       </element>
      </WHERE>
     </WHERE>
    </LET>
   </FOR>
   <FOR var="$person" line="3" column="8" module="test.xq">
    <LET var="$link" line="4" column="8" module="test.xq">
     <WHERE condition="fn:exists(...)" line="5" column="11" module="test.xq">
      <WHERE condition="fn:compare(...) eq 0" line="5" column="11" module="test.xq">
       <element name="person" line="7" column="11" module="test.xq">
        <att name="id" line="7" column="11" module="test.xq">
        </att>
        <element name="name" line="9" column="14" module="test.xq">
        </element>
       </element>
      </WHERE>
     </WHERE>
    </LET>
   </FOR>
   <FOR var="$person" line="3" column="8" module="test.xq">
    <LET var="$link" line="4" column="8" module="test.xq">
     <WHERE condition="fn:exists(...)" line="5" column="11" module="test.xq">
      <WHERE condition="fn:compare(...) eq 0" line="5" column="11" module="test.xq">
       <element name="person" line="7" column="11" module="test.xq">
        <att name="id" line="7" column="11" module="test.xq">
        </att>
        <element name="name" line="9" column="14" module="test.xq">
        </element>
       </element>
      </WHERE>
     </WHERE>
    </LET>
   </FOR>
   <FOR var="$person" line="3" column="8" module="test.xq">
    <LET var="$link" line="4" column="8" module="test.xq">
     <WHERE condition="fn:exists(...)" line="5" column="11" module="test.xq">
      <WHERE condition="fn:compare(...) eq 0" line="5" column="11" module="test.xq">
       <element name="person" line="7" column="11" module="test.xq">
        <att name="id" line="7" column="11" module="test.xq">
        </att>
        <element name="name" line="9" column="14" module="test.xq">
        </element>
       </element>
      </WHERE>
     </WHERE>
    </LET>
   </FOR>
  </element>
 </query>
</trace>

These changes are now applied and committed on both the 10.x and 11.x branches.

Still to do: (a) on the 11.x branch, rationalise the way code injection is done, (b) add unit tests.

I've also looked at tracing within functions: to enable this, I've changed the code at XQueryFunction line 400 to read

                if (staticContext.getUserQueryContext().isCompileWithTracing()) {
                    namespaceResolver = staticContext.getNamespaceResolver();
                    staticContext.getCodeInjector().process(compiledFunction);
                    body = compiledFunction.getBody();
                }

It turns out the unit tests for XQuery tracing in 10.x are reporting success for completely spurious reasons - the method containing the relevant assertions is never called. But it had been corrected for 11.x.

Leaving bug as resolved until fix applied to the Saxon 10 maintenance release.