Saxon

I think that the assertion failed not because the result was false, but because an error occurred during its evaluation; the error was reported as a warning condition, stating

Internal error: value doesn't match its type annotation. Value "mxt" contravenes the
  enumeration facet "ZSY, ZSH, ZSO, ZSX" of the type of element D_3035

Reading the XSD 1.1 and XDM specifications rather carefully, it seems that in the tree that the XPath expression is processing, the element D_3035, whose content is invalid, should be annotated as xs:anyType. This gives us a peculiar situation, not encountered elsewhere, where the XPath expression is processing a mixture of typed and untyped data in the same document. It also means that some of the assumptions we make during optimization of the expression (which is treated as being schema-aware) are questionable.

We build the tree using the standard builder/validator, in which type annotations are allocated optimistically at startElement time. That is, the type annotation is what the type should be, if the element is valid. We don't have any mechanism in the tree builder to revert this if the element turns out to be invalid; in normal circumstances this would be a fatal error, and the tree would never be accessed, so its type annotations would be immaterial.

One option might be to avoid doing assertion processing on any tree that contains data reported as being invalid. In many cases, however, this would lead to some validation errors going unreported, making it difficult to correct all errors after a single validation run.

I note that the class AnnotationTreeBuilder contains the comment:

In the case of an element
node, the type annotation is added during the endElement call, after testing assertions against that element,
to satisfy the rule that when an assertion is evaluated, the target element has type AnyType but its attributes
and descendants are typed according to the appropriate schema declarations.

This might have been wishful thinking.

All tests are passing after this change.

I'm still a bit concerned that there could still be issues around optimization: however, we are not providing the assert expression with a static context item type, so there is little scope for the optimizer to make use of schema information, which suggests we should be OK.