Saxon

My proposal would be that

(a) TraceListener.open() is currently provided with a reference to the Controller object. We should add to the Controller a method getAllTracePoints() which returns a map whose keys are all the Traceable objects in the stylesheet, and whose corresponding values are initialized to zero.

(b) We provide a new implementation of the TraceListener interface in which

(i) the open() method gets this initial map from the Controller,

(ii) the enter() method increments the count for the corresponding Traceable

(iii) the close() method serialises the histogram as XML to the output destination

Processing this XML then becomes the responsibility of XSpec, as it is at the moment.

A further suggestion was made that we should provide better customisation of exactly where trace points are injected into the code. Currently this decision is made by the specific CodeInjector; the one normally used for XSLT is the XSLTTraceCodeInjector, which makes decisions based on the TraceLevel (none, low, normal, high). A custom CodeInjector can be written, and registered with the XSLT compiler at the level of the CompilerInfo object. Writing a custom CodeInjector gives very low-level detailed control, while setting the TraceLevel is pretty coarse. Might there be a middle ground that is easier to configure?

Some questions from Amanda, with my answers:

1. We talked about categories A, B, and C as follows. Does every node in a stylesheet fall into one of those categories?

A. The node is a trace point and it was encountered N times, for some positive integer N. B. The node is a trace point and it was encountered zero times. C. The node is not a trace point.

I think the answer is yes. With perhaps a little bit of a caveat as to what exactly a "trace point" is. A trace point is actually a node injected into the compiled expression tree, not into the source stylesheet. The correspondence between nodes in the compiled expression tree and nodes in the source stylesheet isn't exact.

2. Will there be a documented list of elements or other patterns that are not trace points? Norm mentioned the idea of elements whose use-when attribute evaluates to false, which is why I wrote "other patterns".

Documented may be a bit ambitious! Yes, Norm is right, the first stage of stylesheet compilation (filtering out use-when sections) causes chunks of the stylesheet to disappear, these chunks will contain no trace points. There may be other things that we drop before we do trace code injection, such as unreferenced global variables. And then, in a part of the stylesheet that IS visited by the trace code injector, it makes decisions about where to put the trace points. In principle that's configurable by tweaking the trace code injector.

3. Are the following elements special as far as being trace points or being counted goes?

In some cases, it's not that Saxon is treating the instruction differently, it's that it's general rules affect the treatment. For example, there's a general rule that an "instruction" isn't traced if it is the child of a "non-instruction". That's very likely to affect, for example, the handling of sort keys, because the instructions to compute sort keys are compiled into children of a sort expression which isn't an actual instruction in its own right.

Also, the decisions here are primarily being made by the XSLTTraceCodeInjector which is pluggable. You could use a different CodeInjector with different algorithms if you chose.

• xsl:accumulator and descendants

The main difference here is when the code gets executed, and that shouldn't make any difference if we're only counting executions.

• xsl:assert and descendants

xsl:assert is dropped from the tree very early on if assertions are disabled, which is the default.

• xsl:attribute as a child of xsl:attribute-set, and deeper descendants

In principle that shouldn't make a difference. It's possible that the current code isn't injecting trace points for xsl:attribute set components, but that would be an oversight. We don't inline attribute set definitions, for reasons that I don't currently recall (I think there are edge cases that make it difficult).

• xsl:context-item

That's purely a static declaration, it's not executable, so it isnt' traced.

• xsl:evaluate

I think that dynamically executed code using xsl:evaluate isn't going to be traced. The instruction itself should be.

• xsl:fallback

This is a bit like xsl:use-when - we make an early static decision whether to include or exclude the code in the executable.

• xsl:map, xsl:map-entry

These compile down into XPath expressions, so they may be considered too fine-grained to be traced at the default tracing level for XSLT.

• xsl:matching-substring, xsl:non-matching-substring

Again, it's possible that these are considered too fine-grained.

• xsl:merge-action, xsl:merge-key, xsl:merge-source

The merge-action should be traced normally. The source and key are essentially declarations rather than executable instructions.

• xsl:on-completion

That's going to be treated as an operand of xsl:iterate; may be too fine-grained

• xsl:on-empty, xsl:on-non-empty

Like xsl:sequence, the XSLT instruction at source level is simply a syntactic wrapper for the XPath expression (assuming @select is used), so the instruction as such might disappear from the expression tree. There are some complications here discussed in bug #6428.

• xsl:param as a child of xsl:iterate

Could come under the rule that the generated code isn't under an instruction node.

• xsl:param as a child of xsl:template -- and does counting xsl:param mean the default value was used during execution?

Similar.

• xsl:perform-sort with select attribute or with xsl:sort as the only child element

xsl:perform-sort gets compiled into a sort expression.

• xsl:sort

This is primarily declarative. There may be instructions involved in computing a sort key, but they're likely to be buried inside the sort expression (which is a non-instruction).

• xsl:try, xsl:catch

Can't immediately think of any special considerations here. But we might be getting it wrong, because we're generating the same code as for a try/catch expression in XQuery.

• xsl:when, xsl:otherwise

All forms of conditional (xsl:if, xsl:choose, XPath if-then-else, XQuery typeswitch) generate a choose expression that's modelled on xsl:choose: it's basically a sequence of (condition, action) pairs.

• xsl:where-populated

Like try/catch, I think we might have implemented that as an expression rather than an instruction.

• xsl:with-param

Not sure.

• XSLT elements having a select attribute and no sequence constructor (e.g., xsl:sequence, xsl:break, xsl:copy)

Basically the default tracing rules for XSLT are to trace at the "instruction" level and that means some of the fine-grained events will be lost.

4. xsl:variable is tricky, as we discussed a little when Adrian brought up optimizations. What does the eventual design say about tracing variables, and is it different for global vs. nonglobal variables?

Variables are particularly prone to being inlined (that is, the variable reference is replaced by the expression bound to the variable). That's equally true for local and global variables. We talked about optimization: I think that in general the right approach is that the optimization strategy shouldn't be deliberately changed just because you're tracing, though sometimes the existence of trace code will stop the optimizer recognising code patterns that it would otherwise optimize. But the user has the option of disabling optimizations selectively, for example opt:-v disables variable inlining.

5. Is anything different for stylesheet nodes that are encountered via call to transform()?

Yes. What happens here is that the dynamic stylesheet is compiled and executed with the same trace code injector and trace listener as the master stylesheet. The trace listener, on fn:transform(), sees a nested call to open(). The timing trace listener handles this by creating one output table for each distinct transformation. I think the coverage listener should probably produce a single amalgamated report.

6. Beyond XSLT elements: Is there anything we need to know about treatment of text nodes? How about literal result elements?

It all works at the level of the compiled expression tree, where many of these distinctions no longer exist. For example an xsl:choose at the XSLT level compiles into exactly the same expression tree as an if - then - else at the XPath level, and an AVT compiles into exactly the same code as an xsl:attribute instruction. A literal result element generates the same code as an xsl:element instruction with a statically-known name and namespace.

7. Would you expect the tracing behavior to vary among Saxon editions (HE, PE, EE)? In a way that's straightforward to document for end users, or not really?

Apart from the fact that EE optimizes more aggressively, I wouldn't expect any differences.

8. The issue you created on saxonica.plan.io uses the word "encountered". Is that the same thing as reached, or hit? I am just wondering if there is some subtle difference that I might be missing.

No, I think they're synonymous.

You can also use the command line options

-s:data/books.xml -xsl:styles/books.xsl  -T:net.sf.saxon.trace.CoverageTraceListener -opt:-j -Tout:/Users/mike/Desktop/temp/coverage.xml

and in this case -Tout works as expected.

It occurs to me that we could take advantage of the fact that template rules are JIT-compiled by default: at the end of the run, we could look at all the template rules and report those that haven't been compiled, which means that they have never been matched by any node in the source document. That might be a more useful indicator than listing all the trace points within that template that weren't encountered at execution time.

To avoid the need to suppress JIT compilation, instead of scanning the stylesheet at the start of processing to find all the trace points, we could scan it at the end.

It may be worth pointing out here that a source xsl:template is compiled into multiple compiled templates - one for the named template if it has a name attribute, and one for each mode that it applies to, if it has a match attribute. This will result in multiple trace point reports for the same source code location. It would be helpful if the <T> output for a template rule indicated the relevant mode; this is harder to do for instructions within the template rule.

For the public release, it will be possible to run this using the command line option

-TC:coverage.xml

and JIT compilation will be suppressed automatically.

Thanks, Michael, for your replies and the prototype.

I am trying to run your prototype and having a problem. I placed your .java file in a directory structure net\sf\saxon\trace\CoverageTraceListener.java, navigated to the parent directory in a Windows cmd window, and executed javac -cp C:\saxon12.5\saxon-he-12.5.jar net\sf\saxon\trace\CoverageTraceListener.java. The command generated a .class file alongside the .java file. However, when I try running the Saxon commands you mentioned, it can't find the listener.

java -cp .;C:\saxon12.5\saxon-he-12.5.jar net.sf.saxon.Transform -s:sample.xml -xsl:sample.xsl --traceListenerClass:net.sf.saxon.trace.CoverageTraceListener -opt:-j -Tout:C:\temp\coverage.xml
Transformation failed: Incorrect value for --traceListenerClass: net.sf.saxon.trace.CoverageTraceListener: Failed to load net.sf.saxon.trace.CoverageTraceListener

java -cp .;C:\saxon12.5\saxon-he-12.5.jar net.sf.saxon.Transform -s:sample.xml -xsl:sample.xsl -T:net.sf.saxon.trace.CoverageTraceListener -opt:-j -Tout:C:\temp\coverage.xml
Transformation failed: Failed to load net.sf.saxon.trace.CoverageTraceListener

I also tried putting Saxon 12.5 in my current directory, alongside that net directory (and then adjusting the -cp part of my commands), but that didn't help.

Is there an issue with my net\sf\saxon directory structure being separate from the one in saxon-he-12.5.jar? Or are you able to spot something I've done wrong? When I run a Saxon command without the trace listener, the XSLT transformation runs fine. In case it's relevant, my Java version information is

openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment Microsoft-38106 (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM Microsoft-38106 (build 11.0.16+8-LTS, mixed mode)

I had the same problem as Amanda and assumed it was related to the .jar file.

So I changed the package name in the file, put the java file in the correct folder structure for my package name and added a couple of imports that were needed from the net.sf.saxon.trace package:

import net.sf.saxon.trace.Traceable;
import net.sf.saxon.trace.XSLTTraceListener;

That worked ok for me.

Thanks, Adrian. I changed the innermost package name only, to make the fully qualified name net.sf.saxon.coverage.CoverageTraceListener. Then I did the other things you mentioned, and now I am able to generate a trace! (Changing the entire package hierarchy didn't work.)

Next steps for me: generated traces for more stylesheets, study the results, and write back with questions/comments.

I modified the XSpec code coverage reporting locally so that it calls your prototype listener and transforms the resulting data into the format that XSpec is looking for, so that the rest of the code coverage reporting works. (I changed the XSpec data format slightly, by adding elements with count=”0” where the prototype listener reports count=”0” and making other modifications to account for meaning the absence of a hit, not a hit.) I generated XSpec coverage reports for most of the XSpec repo's coverage reporting tests using (a) this modified code base I just described; (b) a further-modified one where I undid some of the logic in XSpec that augments what the raw Saxon data is saying; and (c) the original XSpec code base with the existing listener.

I am in the process of comparing the three sets of reports, and I will write back here.

Overall initial impression: I am pleased that it didn’t require too much work to connect the code and obtain coverage reports with your prototype. If the new kind of listener is a better foundation for Saxon’s side of things, it seems like that will work out fine for XSpec. There are some already-reported trace issues that we’ll still like to see fixed; for example, I think https://saxonica.plan.io/issues/6457 is still an issue but I will double-check.

Great.

for example, I think https://saxonica.plan.io/issues/6457 is still an issue but I will double-check.

Yes, any issue to do with things not being reported will still be present. But hopefully we can now focus on those issues, rather than well-formedness problems,

One thing I forgot to mention in my comment 13 above: unlike the prototype histogram listener, the built-in trace listener produces the line/column coordinates I expect. For instance, when I execute

java -cp .;saxon-he-12.5.jar net.sf.saxon.Transform -s:sample-copy-of.xml -xsl:sample-copy-of-column.xsl -T -opt:0 -Tlevel:high

the trace elements for the two <xsl:copy-of> elements refer to line="9" column="40" and line="16" column="11" instead of line="9" column="0" and line="15" column="4", respectively.

I noticed that the new (in Saxon 12.5) GetLocation() method in the XSLTTraceListener isn't being used. Changing the line in the close() method from

Location loc = traceable.getLocation();

to

Location loc = XSLTTraceListener.getLocation(traceable);

gives the better column numbers in the 2 cases: Line 9 column 40 and Line 16 column 11.

However, I am confused about the line numbers 15 and 18 in the trace above, because I get 16 and 19 using the original version.