Saxon

Related: https://saxonica.plan.io/boards/3/topics/7336?pn=1&r=7347#message-7347

The issue raised by Vladimir in the help forum is rather more serious and difficult to solve than the original topic of this issue, but we may as well treat them together.

Basically, we are synchronising the construction of an index using the document node as the synchronisation object. Firstly, this is a long-running activity, so the lock may be held for some time, though it will only block other threads building an index on the same document. More significantly, when user code is executed under a lock, we have no way of knowing what other locks will be acquired. For example, we can (with some ingenuity!) construct a scenario where a document has two indexes, and one thread starts building index B while it is already building index A, while the other thread does the reverse: leading inevitably to deadlock.

I suspect that the answer to this is to allow two threads both to proceed with building the index, synchronising only at the point where the index is written to the data structure maintaining a collection of indexes. That is, don't hold any locks while executing user code.

I engineered a deadlock with this stylesheet:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   
   <xsl:variable name="data-1">
      <data1>
         <xsl:for-each select="1 to 100000">
            <entry id="{.}" value="london-{.+.}"/>
         </xsl:for-each>
      </data1>     
   </xsl:variable>
   
   <xsl:variable name="data-2">
      <data2>
         <xsl:for-each select="1 to 100000">
            <entry id="{.}" value="paris-{.+.}"/>
         </xsl:for-each>
      </data2>     
   </xsl:variable>
   
   <xsl:key name="one" match="entry" use="if (/data1) then key('two', @id, $data-2)/@id else @id"/>
   <xsl:key name="two" match="entry" use="if (/data2) then key('one', @id, $data-1)/@id else @id"/>
   
   <xsl:template name="main">
      <xsl:for-each select="1 to 20">
         <xsl:result-document href="out{.}.xml">
            <out>
               <xsl:value-of select="if (. mod 2 = 0) then key('one', '839', $data-1) else key('two', '839', $data-2)"/>
            </out>
         </xsl:result-document>
      </xsl:for-each>
   </xsl:template>
   
</xsl:stylesheet>

Note that these keys are considered "local" rather that "shared" because the key definition references a global variable. This means each transformation will have its own copy of the indexes. But that would happen anyway for indexes on a temporary tree, because each transformation has its own copy of the global variables.

It seems that more than one Bindery is being created. The method Controller.getBindery() creates the Bindery for a package on demand, and this is called when a global variable is evaluated. If the first attempts to evaluate global variables are within a concurrent thread, this method is not synchronized, so two threads can each create a separate Bindery.

Fixed this by sychronizing Controller.getBindery(). We now only have one copy of each global variable.

But the KeyManager is making multiple attempts to create the index for a document in the same thread, so we're not home and dry yet.

The test is now running, and reports the an error because the index is circular. Two things we now need to do:

(a) see if we can find a similar test that doesn't fail due to circularity

(b) test the path where the indexes are shared rather than local (which happens if they don't refer to global variables).

I tested the shared path by a temporary patch to force these indexes to be treated as reusable.