Bug #4557

saxon:parse-html() called twice

Added by Michael Kay over 1 year ago. Updated about 1 year ago.

Start date:
Due date:
% Done:


Estimated time:
Legacy ID:
Applies to branch:
10, trunk
Fix Committed on Branch:
10, trunk
Fixed in Maintenance Release:


When called with -t, the tracing reveals that in executing the following query, the parse-html() function is called twice:

declare namespace saxon = "";

declare default element namespace "";

saxon:parse-html(unparsed-text('file:///xxxx/yyyy/profile.html'))//table[@class=zzzz']//tr[@class = 'mergedtoprow'][th = 'Country']/td//a//text()

Preliminary investigation shows the expression is translated into a call on the key() function. One call on parse-html() occurs while the index is being built, the other occurs during the key lookup.

As a completely separate question, there's no point in building an index if it is only going to be used once, and I'm surprised this isn't recognized as being the case.


#1 Updated by Michael Kay over 1 year ago

The expression translates into a call on fn:key() in which the call on parse-html() appears in the third argument. The first call on saxon:parse-html() happens while evaluating the arguments of this fn:key() call.

Evaluation of fn:key() (for the first time) then leads to a call on KeyIndex.constructIndex(). This calls NodeSetPattern.selectNodes() to select the nodes that need to be indexed; and this involves re-evaluation of the saxon:parse-html() expression.

It's complicated by the fact that there are two predicates in this query that are both indexable, which results in creation of two separate indexes. If I simplify the query to

saxon:parse-html(unparsed-text('file://xxx/profile.html'))//table//tr [@class = 'mergedtoprow'] /td//text()

then the problem still arises.

#2 Updated by Michael Kay over 1 year ago

Out of frustration with the poor messages being output, I have implemented a second optional argument for parse-html() and parse() to supply the base URI. (Perhaps for extensibility it should be a full-blown options map, with base-uri as one of the options).

When I change the query to use a local variable for the URI:

for $u in ('file:///xxx/profile.html', 'file:///xxx/books.html')
return saxon:parse-html(unparsed-text($u), $u)/table/tr [@class = 'mergedtoprow']/td/text()

I'm only getting one call on parse-html() for each document. But it's no longer using a key index, so we've simply bypassed the problem.

Now trying with parse-xml() in place of saxon:parse-html(). It's still using an index. But I'm puzzled that we now don't get any "Building tree... " messages. It seems that saxon:parse() and saxon:parse-html() construct the Builder using Controller.makeBuilder(), whereas fn:parse-xml() uses TreeModel.TINY_TREE.makeBuilder. I can't see any obvious reason for this difference. Changed parse-xml() to use controller.makeBuilder(), and I now get the messages: indeed, I get two messages, as with the parse-html() case.

If I change parse-xml(unparsed-text(XXX)) to doc(XXX), I still get "optimization" to use a key, but this time the document is only built once. I thought that might be because the result of doc() is cached, but this isn't the explanation: internally, there is only one call on doc().

#3 Updated by Michael Kay over 1 year ago

The difference between the doc() case and parse-xml(unparsed-text()) appears to be in SlashExpression#378, where expressions starting with a call on the doc() function are handled specially. This affects streaming so I don't think it's wise to change it.

#4 Updated by Michael Kay about 1 year ago

  • Category set to Performance
  • Status changed from New to In Progress

Coming back to this, I'm using the query


The tree is being built twice. The -explain output shows the rewritten query as

  <key name="Q{}kk101" line="0" flags="u">
    <p.nodeSet test="NE nQ{}b">
      <slash baseUri="file:/Users/mike/team/xmark/"
             ns="err=~ fn=~ local= saxon=~ xs=~ xsi=~ xml=~"
        <slash simple="1">
          <fn name="parse-xml">
            <str val="&lt;a&gt;&lt;b&gt;2&lt;/b&gt;&lt;b&gt;3&lt;/b&gt;&lt;/a&gt;"/>
          <axis name="child" nodeTest="NE nQ{}a"/>
        <axis name="child" nodeTest="NE nQ{}b"/>
    <cast baseUri="file:/Users/mike/team/xmark/"
          ns="err=~ fn=~ local= saxon=~ xs=~ xsi=~ xml=~"
      <data diag="1|0||=">
        <dot type="1NE nQ{}b"/>
    <docOrder baseUri="file:/Users/mike/team/xmark/"
              ns="err=~ fn=~ local= saxon=~ xs=~ xsi=~ xml=~"
      <for var="Q{}dd1161082381"
        <fn role="in" name="parse-xml">
          <str val="&lt;a&gt;&lt;b&gt;2&lt;/b&gt;&lt;b&gt;3&lt;/b&gt;&lt;/a&gt;"/>
        <fn role="return" name="key">
          <str val="Q{}kk101"/>
          <str val="2"/>
          <varRef name="Q{}dd1161082381" slot="0"/>

Stepping through the code, OptimizerEE.convertFilterExpressionToKey() has logic that avoids rewriting the filter expression to use a key if the filter expression creates a new document. This logic isn't being activated because the function metadata for parse-xml() doesn't have the NEW property. If we add this property, the rewrite doesn't take place, and the document is only built once.

Adding this property to parse-xml(), parse-xml-fragment(), saxon:parse(), saxon:parse-html(), saxon:new-NNNN().

#5 Updated by Michael Kay about 1 year ago

  • Status changed from In Progress to Resolved
  • Priority changed from Low to Normal
  • Applies to branch 10, trunk added
  • Fix Committed on Branch 10, trunk added

#6 Updated by O'Neil Delpratt about 1 year ago

Bug fix applied in the Saxon 10.3 maintenance release

#7 Updated by O'Neil Delpratt about 1 year ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100
  • Fixed in Maintenance Release 10.3 added

Please register to edit this issue

Also available in: Atom PDF