Saxon

This is a tough one to solve.

We don't want to lose the MemoClosure mechanism whereby an expression $m[1] can be evaluated without fully materialising the value of $m. Yet at the same time there are iterators (not just the multithreaded iterator used for collections, but also for example an UnparsedTextLines iterator) that do need to be closed to release resources if they are not read to completion.

The only solution I can see at the moment is to distinguish iterators that need to be closed because they hold resources, and to avoid using these in a MemoClosure. This isn't easy of course because it's a transitive property: an iterator that's derived from one that holds resources itself needs to be closed so it can close the base iterator.

It's a little unfortunate that on the 11 branch we've discarded the general mechanism for SequenceIterator properties. Perhaps we should reinstate it.

I've added a method SequenceIterator.mustClose() with a default implementation that returns false. On MultithreadedItemMappingIterator the method returns true; on ItemMappingIterator it returns base.mustClose().

In MemoClosure.makeSequence(), if inputIterator.mustClose() is true, then the inputIterator is fully evaluated and sequence is set to a GroundedValue.

This solves the problem for this test case (running in the debugger demonstrates that all threads are closed down properly).

It remains to ensure that the new mustClose() method is properly implemented for all other implementations of SequenceIterator.

I've been doing this on the 11 branch; it also needs retro-fitting to 10.

This solution doesn't work in cases where we create a MemoClosure for the value of a variable, and at the time we create it, we don't know whether it will hold resources or not. For example:

<xsl:variable name="temp" select="my:f(), my:g(), my:h()"/>

where my:h() makes a call on fn:collection().

The variable $temp is represented by a MemoClosure whose inputIterator is a BlockIterator - effectively a concatenation of lazily allocated iterators - and at the time we create it, we have no idea whether it will contain an iterator that must be closed to release resources. We could assume the worst, that is, have mustClose() return true; but this would prevent lazy evaluation in many cases where it is perfectly safe.

Another idea: instead of the iterator telling the caller that it must be closed after use, have the caller tell the iterator that it can't guarantee to call close(). An iterator that needs to release resources on close will then have to ensure that all the resources needed are acquired, consumed, and released eagerly.

So we add a method SequenceIterator.getSafeIterator() which returns an iterator over the same items, such that it's OK if the new iterator is abandoned without a call on close(). For most iterators, getSafeIterator() will return this, but (for example):

• for a MultithreadedItemMappingIterator it will return SequenceTool.makeGroundedValue(this).iterate() (forcing eager evaluation)

• for an ItemMappingIterator it will set baseIterator = baseIterator.getSafeIterator();

• for a BlockIterator, it will call SequenceIterator.getSafeIterator() on each of the component iterators as it is constructed.

Another option is to move threading logic out from iterator to the code that iterates, e.g. if item signals that it supports (prefers) asynchronous evaluation.

This way thread cycles might be lifted on top.

I have implemented (currently on the 11 branch only) a solution based on comment #4. The SequenceIterator interface has a new method discharge() which is called (by MemoClosure only, at present) to indicate that the SequenceIterator should relinquish all resources while remaining prepared to deliver the remaining items in the sequence. The default implementation does nothing; delegating iterators call discharge() on their base iterators; and iterators that actually hold resources evaluate themselves eagerly and then relinquish the resources, substituting an iterator over the grounded results. The BlockIterator remembers that discharge() was called, and if so, it calls discharge() on each component iterator as soon as it is created.

The tests now seem to be working fine in SaxonJ-EE 11. But I've also applied them to SaxonCS 11, and I'm getting an occasional (non-repeatable) failure in the QT3 fn:collection() tests. I can't be sure what's happening here - I suspect it's unrelated, and is due to the collection() function in SaxonCS delivering collection results in unpredictable order (which is the subject of another open bug).