Bug #4938

Strange behavior of string-join()?

Added by Johan Gheys 3 months ago. Updated 2 months ago.

XPath conformance
Start date:
Due date:
% Done:


Estimated time:
Legacy ID:
Applies to branch:
10, trunk
Fix Committed on Branch:
10, trunk
Fixed in Maintenance Release:


I use Saxon-EE 10.3 and have a strange behavior when I use the string-join () function. I have uploaded a test case where the output folder contains both the expected result and the generated result. For some unknown reason, infrastructure text lines are split into multiple items when I use the string-join() function (line 127 in create-sts-report-xml.xslt). If I replace it with the concat() function, everything works fine. Or if I just process the last element (by uncommenting lines 18 and 65), the result - even with string-join() - is ok too. Can anyone explain what's going on here? (16.1 KB) Johan Gheys, 2021-03-15 16:53 (16.1 KB) Johan Gheys, 2021-03-18 09:17


#1 Updated by Johan Gheys 3 months ago

I think it's a bug since version 10.0. I uploaded a simplified version with 2 bat files giving the expected result with version and the wrong result with version 10.3.

#2 Updated by Michael Kay 3 months ago

  • Tracker changed from Support to Bug
  • Category set to XPath conformance
  • Assignee set to Michael Kay
  • Priority changed from Low to Normal

#3 Updated by Michael Kay 3 months ago

Problem confirmed.

I thought it might be the same issue as bug #4944, This appears not to be the case, though I think they are in the same area: new code was introduced in 10.x to allow the output of functions such as string-join() and unparsed-text() to be sent incrementally to the serializer without being buffered in memory first, and I think we're seeing problems with this code.

#4 Updated by Michael Kay 3 months ago

  • Status changed from New to In Progress

What's really weird is that if I change the string-join separator on line 127 from " - " to " ~ ", the output is now correct except that it contains a tilde instead of a hyphen. Which suggests something very strange indeed is going on.

Further oddities:

  • the problem goes away if I change the string-join delimited used at line 132, even though this doesn't seem to be involved at all in the incorrect output.
  • when I run the string-join() code in the debugger, I get an ArrayIndexOutOfBounds exception doing string.getChars(), which doesn't occur when I run without the debugger enabled.

This all suggests some fairly low-level data corruption is occurring.

#5 Updated by Michael Kay 3 months ago

The bad output is produced by the string-join() at line 81; it looks as if the variable $infrastructure-text-line contains (in the part generated by line 127) individual data strings and " - " separators as individual items, which are then separated by newlines, as if the string-join at line 127 output all the arguments and separators as individual items without concatenating them into single strings. But I can't see how this can be happening!

The variable $infrastructure-text-line is lazily evaluated; the apply-templates calls don't actually happen until the variable value is needed, which happens when the xsl:message on line 80 is evaluated. I'm stepping through the evaluation of this xsl:message instruction in the debugger.

The debugging problem is actually an old one: there's a comment in the code about it. The FastStringBuffer class has a toString() method with side-effects (designed to reclaim unused space). Because the debugger calls toString(), this means that behavour within the debugger is different from normal behaviour. Moreover, if toString() is called at critical points (after adding data to the array and before updating the length) this can corrupt the content.

I've commented out the call on condense() in FastStringBuffer.toString() . This doesn't make the bug go away, but it might make it more easily debuggable.

#6 Updated by Michael Kay 3 months ago

OK, so I've traced the execution that produces the first <infrastructure> element, and it all looks fine. Which is unsurprising, because it is fine; the problems occur in the second execution. So let's trace that...

I missed the point where it happened, but the lazy evaluation of the variable by the count() expression within xsl:message on line 80 does indeed generate split items, even though they seemed OK when being added to the SequenceCollector.

Got it! It starts going wrong at the point where we've executed the expression sufficiently often to generate bytecode. Sure enough, with bytecode disabled (-opt:-c on the command line) the problem goes away.

That doesn't solve the problem, but it tells us where we need to look, and it explains the apparent arbitrariness of the symptoms. (Or some of them: it's still strange how it depends on the actual choice of separator.)

#7 Updated by Michael Kay 3 months ago

The construct for which bytecode is being generated is indeed the body of the template rule on line 126, comprising the offending call on string-join().

The data being fed by the compiled StringJoin.process() method to the Outputter is clearly wrong.

The push-mode compiler for StringJoin doesn't seem to have changed significantly between 9.9 and 10.x But perhaps it's more likely to be used in 10.x. It looks to me as if it only works if the string-join() is generating text nodes to be appended to an XML document, which isn't the case here (we're generating strings to contribute to a variable of type xs:string*). It looks to me as if the bytecode generator for string-join() needs a rewrite.

#8 Updated by Michael Kay 3 months ago

There's actually no point in generating fine-grained bytecode for fn:string-join(); we've learnt over the years that bytecode only improves performance where you can make decisions at compile-time that would otherwise be made at run-time, and that's not the case here. We should simply generate a call on the interpreted method. Having said that, we can't simply drop the compileToPush(), because that would lead to pull mode evaluation which uses more memory. We need code for PushableFunction implementations that generates an appropriate call.

#9 Updated by Michael Kay 3 months ago

  • Status changed from In Progress to Resolved
  • Applies to branch 10, trunk added
  • Fix Committed on Branch 10, trunk added

Rewrote StringJoinCompiler.compileToPush() to generate a call on StringJoin.process().

I guess that my many misadventures while navigating the Brussels railway network were good training for this one.

#10 Updated by Johan Gheys 3 months ago

Thanks for solving this problem. Either way, we remain firmly convinced that xslt is a genius language in general and Saxon a genius product in particular!

#11 Updated by O'Neil Delpratt 2 months ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100
  • Fixed in Maintenance Release 10.5 added

Bug fix applied to Saxon 10.5 maintenance release.

Please register to edit this issue

Also available in: Atom PDF