Saxon

Problem confirmed.

I thought it might be the same issue as bug #4944, This appears not to be the case, though I think they are in the same area: new code was introduced in 10.x to allow the output of functions such as string-join() and unparsed-text() to be sent incrementally to the serializer without being buffered in memory first, and I think we're seeing problems with this code.

The bad output is produced by the string-join() at line 81; it looks as if the variable $infrastructure-text-line contains (in the part generated by line 127) individual data strings and " - " separators as individual items, which are then separated by newlines, as if the string-join at line 127 output all the arguments and separators as individual items without concatenating them into single strings. But I can't see how this can be happening!

The variable $infrastructure-text-line is lazily evaluated; the apply-templates calls don't actually happen until the variable value is needed, which happens when the xsl:message on line 80 is evaluated. I'm stepping through the evaluation of this xsl:message instruction in the debugger.

The debugging problem is actually an old one: there's a comment in the code about it. The FastStringBuffer class has a toString() method with side-effects (designed to reclaim unused space). Because the debugger calls toString(), this means that behavour within the debugger is different from normal behaviour. Moreover, if toString() is called at critical points (after adding data to the array and before updating the length) this can corrupt the content.

I've commented out the call on condense() in FastStringBuffer.toString() . This doesn't make the bug go away, but it might make it more easily debuggable.

OK, so I've traced the execution that produces the first <infrastructure> element, and it all looks fine. Which is unsurprising, because it is fine; the problems occur in the second execution. So let's trace that...

I missed the point where it happened, but the lazy evaluation of the variable by the count() expression within xsl:message on line 80 does indeed generate split items, even though they seemed OK when being added to the SequenceCollector.

Got it! It starts going wrong at the point where we've executed the expression sufficiently often to generate bytecode. Sure enough, with bytecode disabled (-opt:-c on the command line) the problem goes away.

That doesn't solve the problem, but it tells us where we need to look, and it explains the apparent arbitrariness of the symptoms. (Or some of them: it's still strange how it depends on the actual choice of separator.)

The construct for which bytecode is being generated is indeed the body of the template rule on line 126, comprising the offending call on string-join().

The data being fed by the compiled StringJoin.process() method to the Outputter is clearly wrong.

The push-mode compiler for StringJoin doesn't seem to have changed significantly between 9.9 and 10.x But perhaps it's more likely to be used in 10.x. It looks to me as if it only works if the string-join() is generating text nodes to be appended to an XML document, which isn't the case here (we're generating strings to contribute to a variable of type xs:string*). It looks to me as if the bytecode generator for string-join() needs a rewrite.

There's actually no point in generating fine-grained bytecode for fn:string-join(); we've learnt over the years that bytecode only improves performance where you can make decisions at compile-time that would otherwise be made at run-time, and that's not the case here. We should simply generate a call on the interpreted method. Having said that, we can't simply drop the compileToPush(), because that would lead to pull mode evaluation which uses more memory. We need code for PushableFunction implementations that generates an appropriate call.

Thanks for solving this problem. Either way, we remain firmly convinced that xslt is a genius language in general and Saxon a genius product in particular!