Saxon

The critical part of the stack trace is

at com.saxonica.ee.optim.OptimizerEE.reorderPredicates(OptimizerEE.java:560)

at net.sf.saxon.expr.FilterExpression.optimize(FilterExpression.java:449)

at com.saxonica.ee.optim.OptimizerEE.reorderPredicates(OptimizerEE.java:560)

at net.sf.saxon.expr.FilterExpression.optimize(FilterExpression.java:449)

at com.saxonica.ee.optim.OptimizerEE.reorderPredicates(OptimizerEE.java:560)

Suggesting that we are changing the order of predicates in a filter expression, re-optimizing the result, which possibly reverts back to the original order and therefore recurses indefinitely.

Hi Mike,

I have responded over the email with the latest XSLT and waiting for your reply for long time. Could you please help with the solution.

Thanks, Preethi Devaraj

It's becoming a bit of a tangled story....

The failure doesn't occur in my working copy of Saxon 9.9.1.5 (with patches applied preparing for the next maintenantce release.

Looking at the stack trace and the current code in comparison, the reason it doesn't fail is that the code at at com.saxonica.ee.optim.OptimizerEE.reorderPredicates(OptimizerEE.java:560) has been commented out. It looks as if it was commented out during the investigation of preicate reordering in issue

Mike,

What could me the interim solution? And when can we expect the next release?.

Also as I mentioned previously, we have migrated from the old net.sourceforge.saxon9PE to the latest Saxon9EE. As all the xslt worked fine with the old jar, do you recommend any suitable versions (compatible with Java8) other than the latest net.sf.saxon Saxon9EE. Such that the xslt would not require any changes as we consume it from a different system.

Interim solution is to switch off optimization using the -opt flag. Maintenance releases for a mature product like 9.9 are generally about every two months, but it depends on the urgency. - Michael Kay**

It turns out that the reason the reordering of predicates doesn't converge is integer overflow in the cost calculations. We estimate the cost of evaluating both predicates, say C1 and C2, and if C1 > 2C2 then we reorder them. But if 2C2 overflows and delivers a negative integer, then (C1 > 2C2) and (C2 > 2C1) will both return true. We could do the calculation using double or long arithmetic, or we could cap the cost estimate at some maximum (it's a very crude calculation anyway).

However, we should consider implementing some of the improvements made to the development branch following the investigation described in bug #4340 (see comment 10 et seq). These improvements consider all the candidate predicates collectively, rather than comparing them pairwise, and they also take into account the indexability of each predicate, which is a more important factor than the raw evaluation cost.

Hi Mike,

Should we expect any patches? or could you suggest how to use -opt flag in a Java Web application, where we refer the saxon jar as a library.

Thanks, Preethi Devaraj

We mark a bug as resolved when we have committed a patch (that's not particularly useful for EE users who don't have access to source code, however), and then we mark it closed when we ship a maintenance release containing that patch, which is usually within 6 weeks or so.

You can set optimizer options programmatically using Configuration.setConfigurationProperty() or Processor.setConfigurationProperty() with Feature.OPTIMIZATION_LEVEL

I have fixed the problem by changing the expression cost calculations to use double rather than int arithmetic.

Another approach to fix such overflow would be to switch to logarithmic scale, so instead of cost C it's possible to use c = log2(C). This way:

C2 = C1 * 2 would be expressed with c2 = c1 + 1 C3 = C1 * C2 would be expressed with c3 = c1 + c2 C3 = C1 + C2 would be approximated with c3 = max(c1, c2) + 1 C(n+1) = C1 + C2 + … + Cn would be approximated with c(n+1) = max(c1, c2, …, cn) + log2(N), where log2(N) is approximated with Java.lang.Integer.highestOneBit().

Isn't a logarithmic scale exactly what double floating point gives us? But in fact there's no need. The calculations are very crude (we really don't know what a//b will cost without any volumetrics on the instance data), we're just trying to distinguish simple expressions from complex ones.

I agree, it is the same. the only the difference is that you can stay with integer arithmetic and use + instead *.