Saxon

No, looking at the output more carefully, position() is 1 and last() is zero.

The approach to evaluating last() in an asyncronously-evaluated result-document instruction within a loop seems completely flawed. We're evaluating result-document using a context whose current iterator is a ManualIterator, and the getLast() method on the new context simply calls getLength() on the original iterator. This is a plain FocusTrackingIterator (not the multithreaded variety) so its getLast() method is not synchronised, and as far as I can see two result-document instructions can be consuming the iterator at the same time.

(Did this ever work, or is a new problem in 12.x?)

I've just run test threads-026 under Saxon 11 and it crashes.

The test was written in response to bug #3927. That bug entry contains a clear description of the problem. It says that the solution is to allocate a MultithreadedFocusTrackingIterator in cases where "we are in Saxon-EE, running a stylesheet that uses xsl:result-document, with allow-multithreading". This doesn't appear to be happening.

With Saxon11 the problem only occurs with bytecode enabled - a FocusTrackingIterator (non-multithreaded) is being created from within generated bytecode.

With Saxon12, the problem seems to be the method SequenceTool.focusTracker,() which always generates a FocusTrackingIterator (non-multithreaded) . Many of the expression elaborators are calling this method rather than using the factory in the configuration.

With a change to the FilterExpression elaboration code, the test is now consistently using a MultithreadedFocusTrackingIterator.

It now succeeds most of the time, but I still get an occasional failure -- especially when the test is run as part of the whole test set. The failure generally shows last="0", but I have also seen last="-1". I suspect a timing issue. Unfortunately adding tracing output, or using the debugger, affects the timing, which makes it hard to investigate.

With trace output, the test generally succeeds, and the trace output shows that it is all working as designed: the first time last() is called on the FocusTrackingIterator, it reads the base iterator to completion, saving the results in a list, and subsequent calls on next() read items from the list.

By trapping the case where last(0 returns -1 and using println diagnostics (the failure doesn't occur under the debugger), I have established that in the failure cases, the current iterator is a ManualIterator rather than a FocusTrackingIterator.

It seems likely that this is the ManualIterator created at EnterpriseConfiguration.processResultDocument(). Note that in this test case the xsl:result-document calls are nested, so the ManualIterator created for the outer xsl:result-document instruction will be the target of the last() call in the inner xsl:result-document.

The trace also indicates that we're creating a FocusTrackingIterator over a ManualIterator. This is unnecessary, because ManualIterator already implements the FocusIterator interface. I've fixed that, at least for the case where it's generated from FilterExpression. We still get the failure, but last() is now zero rather than -1.

The target of last() is also a ManualIterator in the success case. Suspicion therefore falls on how last() is calculated for a ManualIterator, which is by a callback to the getLast() method of the originating context.

There's definitely a bug here but it's proving very elusive.

I have a theory. Tracing FocusTrackingIterator.getLength(), it is returning -1 when pos is -1, which means the entire sequence has already been read. Changing FocusTrackingIterator to set last = pos when we find there are no more items in the base sequence could eliminate the problem. Needs confirmation.

I'm reasonably confident this was the key.

Explanation: we're iterating over a sequence S, and for each item we kick off an asynchronous call on xsl:result-document, which calls last() on the iterator. It is possible that the first call on last() will come when S has already been read to completion. It obviously makes sense that in that situation the iterator knows how long S was, and can return the result directly. Furthermore, if we don't do that, the alternative approach is going to give the wrong answer: it determines how many remaining items there are in the sequence (zero), and adds this to the current position, which has been set to -1 .

Note that an asynchronous call to position() doesn't suffer this problem, because the current position (and other stuff like the current regex group) was saved in the ManualIterator before firing off the asynchronous thread. The only reason this isn't done for last() is that evaluating last() just in case it's needed is excessively expensive, especially in memory.