Further information about the problem.
The specs say that the current merge group is cleared on function calls and template invocation (§15.6: All invocation constructs set the current merge group and current merge key to absent.) We therefore make quite a few calls to
context.setCurrentMergeGroupIterator(null). The code is written to do nothing if the value is already null. But determining that it is null typically involves searching the property chain for a value, and in a stylesheet that doesn't use xsl:merge this will always search to the end. If the chain contains many other properties, which will often be the case if there is a deep level of recursion, this may be a lengthy search.
We could question whether the tiny saving achieved by changing the design in 12.0 is actually worth it. In 11.x, setting the current merge group iterator to null was a simple field assignment, and copying it when a new context is created was hardly going to be expensive. But overall, the reduction in the size (number of fields) in the XPathContextMajor object, and therefore the reduction in the cost of copying it, did show up in measurements as a benefit.
A change we could consider is to revert to holding the current template rule the old way, as a dedicated field. This is the only property that is likely to be set very frequently, and in cases I've looked at, it completely dominates the property chain. If we saved this in a dedicated field, it's very unlikely that the property chain would grow to thousands of entries, and searches for all the other properties would then be much shorter.
A more radical question is this: do we actually need to keep a full stack of Context objects through tail-recursive template calls? When a template call is tail-recursive, we reuse the Java stackframe of the caller to avoid a Java stack overflow, but we don't attempt to re-use (or discard) the context object for the caller. It seems that we do reuse the Context for function tail calls but not for call-template or apply-templates tail calls. There's a TODO in the CallTemplate code pointing out that I tried to fix this in June 2022 and hit difficulties. Perhaps I need to try again.