Saxon

I'm running the transformation successfully in the IDE (intelliJ). Currently I'm getting the same output (with incorrectly positioned commas) under both 9.6 and 9.7. I'm running under EE for convenience, I'll try changing some parameters or try running from the command line to see if that changes anything.

I had added debugging messages to the transform to report the values for position() and last() and that's where I was seeing differences between 9.6 and 9.7, but I was definitely getting different results from both versions.

It's possible I didn't provide the exact right version of the transform but the behavior definitely demonstrates the problem. I noticed this because the XSLT that had worked before 9.7 was now not working.

If I've got a configuration that demonstrates the bug, that's all I need for the moment. Working out why a different configuration doesn't show the bug is secondary.

Based on my original analysis, both 9.6 and 9.7 gave incorrect values for position() and last(), but they gave different incorrect values. The 9.6 incorrect values happened to work with my business logic.

I think the problem is that XSLT-defined space-stripping rules are not being applied to documents read using the collection() function. The stylesheet generally uses xsl:apply-templates with no select attribute, which selects all children including whitespace text nodes, and this means that when processing element nodes, position()=last() is always false.

Solving this is not easy. The machinery for the collection() function is carefully constructed to allow user customisation, and it's not clear where strip-space should fit in. Do we pass the strip-space information to the CollectionFinder to take note of as it chooses? Or do we apply strip-space to whatever it returns? The difficulty with doing the latter is that it is much more efficient to apply space stripping as a filter in the XML parsing pipeline than to retrofit it to an already-built document tree.

An added complication is the XSLT 3.0 rule that the space-stripping rules can vary from one package to another, so we need to know which package contains the collection() call. (But we don't yet do this for the doc() and document() functions...)

And then there's a problem with testing, because the XSLT3 test driver has its own machinery for defining collections which is not necessarily the same as the machinery used in other applications.

And we have to try and preserve API compatibility if we can, though the CollectionFinder apparatus is still fairly new so we can make changes if there's no alternative.

Oh dear: getting this working in the test driver is particularly difficult because it does all the work of building a collection up front, long before the collection() function is called. This might need a complete redesign of the way collections work in the test driver. Which is largely a distraction from getting the bug fixed in the actual product.

At least you identified the root cause--it seemed unlikely that there was actually a bug in calculating position().

I see the strip-space problem. If I was implementing my own collection handler I would probably be surprised to learn I had to do my own space stripping but on the other hand I would probably also be upset if there was a performance hit because of post-processing to remove spaces.

I would think a strong warning in the customization documentation and a readily-available SAX space filter would be appropriate.

I've added code to CollectionFn which in effect post-processes the results of the user-written (or system default) CollectionFinder to do the space stripping from returned documents. (It does this by wrapping the returned tree in a wrapper that performs the space stripping while the tree is being navigated).

This meets the conformance rules but it would be nice to add an optimization whereby the CollectionFinder can elect to take responsibility for space stripping itself. (It then has the option to be non-conformant if it chooses; so long as it declares that it's done all that needs to be done, Saxon will ask no questions.)

Seems reasonable.

I've added a method to AbstractResourceCollection (perhaps it can be promoted to the ResourceCollection interface in 9.8) whereby (a) Saxon tells the ResourceCollection what the whitespace stripping rules are, and (b) the ResourceCollection responds saying whether it agrees to take responsibility for implementing them (default is NO). Next step is that standard collection implementations like the DirectoryCollection will use this information to do whitespace stripping during parsing, and tell the CollectionFn code that they have taken care of it.

I have extended the patch so that it now implements the XSLT 3.0 rule that allows different whitespace stripping rules to apply to the same collection in different packages