Saxon

Confirmed that this also affects XQuery - added test case StringConstructor-027 which does

<a>Today is `{adjust-date-to-timezone(current-date(), ())}`</a>

This means we can't solve it simply by disabling this tokenization logic when doing XPath.

We also need to test perverse cases like

<a>This: {``[is `{``[text]``}`]``}`</a>

In both string constructors, where we have embedded expression of the form

`{expr}`

and value templates (XSLT/XQuery) where we have embedded expressions of the form

{expr}

we are relying on the tokenizer to deliver a terminal token ( "}" or "}`......" . Now, the problem is that the "}" at the end of a value template can validly be followed by a backtick. So for the tokenizer to do this correctly, it needs to know which delimiter is expected; but that's not a simple state setting because either of these expressions can be nested in the other, recursively.

So it looks as if we are asking too much of the tokenizer.

I think we probably need a complete change of strategy for parsing StringConstructors, handling them more like we handle direct element constructors, where the parser switches into reading individual characters from the input, with tokenization disabled. Unfortunately this is a pretty complex solution.

The bug also exists in 9.8.

Not sure what the current policy in terms of that branch is but it would be nice if it also gets the fix as without it using text value templates in XSLT 3 to create Markdown output where the backtick is an important character is not possible and one has to go back to use xsl:text/xsl:value-of.

@Martin, I think it's unlikely there will be any further maintenance releases for 9.8. We would do one only if such a patch is the only way to keep a customer on the road, which would imply (a) there's something stopping the customer moving to 9.9, and (b) there's no practical workaround. In general, we are no longer committing 9.8 versions of patches as a matter of course. If we were to make another 9.8 build for general release, we would review the set of bugs that are candidates for retro-fitting.

As a minor data point, I ran into this problem as well (also while creating Markdown-like output), and I'm glad to have found the explanation here so I can appropriately tailor the workaround. I'm also using Saxon-HE 9.8 due to problems with Saxon 9.9 that I haven't isolated yet (an uncaught java.lang.UnsupportedOperationException: "Cannot copy a variable reference whose binding is unknown") and based on the Saxon home page's indication that 9.8 is the most stable version. I'll try to create a report about the other issue.

@Evan, can I ask where you got the information that 9.8 is most stable? You should find that http://www.saxonica.com/download/download_page.xml says that 9.9 is most stable (this was updated some time ago, I think with the 9.9.1.2 release). Thanks.

@Debbie, I don't know where Evan looked but for the HE edition people do consult http://saxon.sourceforge.net/ and that continues to say "The most stable version is version 9.8"

@Debbie, the fifth paragraph on http://saxon.sourceforge.net/ reads:

"The most stable version is version 9.8, first released on 8 June 2018 (the same day as the XSLT 3.0 Recommendation), and regularly updated with maintenance releases."

Ah ha, thank you both - that information clearly needs updating...