Saxon

I've got a colleague to test using an older version of saxon (9.5.1.7); she gets the expected result (empty regex-group(1)).

I also get it with the replace() function:

replace('1949', '([0-9]{1,2})?\s?([A-Z]{1}[a-z]{1,8}.?)?\s?([0-9]{4})', '$1')

Glad I'm not going mad ;)

This is one of those areas where the specifications for regular expressions are frustratingly unhelpful. Most of them say much the same as the F+O spec (which is the normative one here); "When parentheses are used in a part of the regular expression that is matched more than once (because it is within a construct that allows repetition), then only the last substring that it matched will be captured.". What the spec doesn't say, but what it probably should say, is that the only matches that count are those that are ultimately successful, that is, if you end up ignoring a match of part of the regex, as a result of backtracking, then the contents of any subgroups captured during that attempt should also be discarded. This seems to be what other regex engines do, and getting regular expression evaluation right often seems to be guided more by the actual behaviour of other engines than by any specification...

Saxon is doing what the F+O spec says, and not what it should say.

The backtracking logic was extensively rewritten for Saxon 9.6 to prevent stack overflows, and it is no doubt this rewrite that led to the regression.

There is no logic in the regex engine that attempts to clear captured subgroups during backtracking: a subgroup gets overwritten each time the capturing group matches something, but it doesn't get discarded if the backtracking unwinds beyond that part of the expression. So fixing this isn't going to be particularly easy. But I'm thinking about it. (I always had an uneasy feeling there might be a problem here, but didn't worry too much about it as all tests were passing.)

Turns out to be not as bad as I thought. The regex engine DOES have logic to clear "provisional" captured groups during backtracking; it just isn't invoking that logic on this particular path.

My first attempt at a patch, however, causes regression in one XQuery test case doing

analyze-string("how now brown cow", "(.*?ow\s+)+", "")