Saxon » SaxonC

If you can add more information about your Linux setup, that would be really helpful:

• Distro
• Architecture
• Version

If you have a sample XML file that triggers the problem, please upload it here if you can, that would be very useful too...

Thanks, Matt.

Just to contrast the result running from Windows with the same Python sample and XML samples:

PS Microsoft.PowerShell.Core\FileSystem::\\wsl.localhost\Ubuntu-22.04\home\mh> python .\saxonc12xmlparse-test1.py
SaxonC-HE 12.4.2 from Saxonica
<root>test</root>
<root xml:lang="de">Dies ist ein Test: machen Umlaute wie ä, ö, ü etwa Ärger?</root>

Thank you very much for having a look into it to both of you. I work on Oracle Linux Server release 9.3, 64bit Intel. Let me know if you need more information, e.g. locale, installed locales maybe ?

Using just the bits of the Java code that the C++ parse code from the example uses, I can parse an XML file with a charset declaration of windows-1252 fine. That suggests that there's a problem with the version of the parser we're bundling with Saxon C. I am investigating further.

Perhaps (kind of a wild guess after a broad search by Google to find stuff related to GraalVM and unsupported encodings) it helps to use native-image -H:+AddAllCharsets when building SaxonC to get better/broader encoding/charset support.

Martin Honnen wrote in #note-7:

Perhaps (kind of a wild guess after a broad search by Google to find stuff related to GraalVM and unsupported encodings) it helps to use native-image -H:+AddAllCharsets when building SaxonC to get better/broader encoding/charset support.

No, that's bang on the money. I'm still digging – there's precious little direct documentation about which character sets are included that I've been able to find so far – but at a guess on Windows they include 1252 by default.

There's other stuff to consider as well – technically, windows-1252 is not a supported charset in XML, but it's worked (thanks to the Xerces team) on Java for years.

I'm also not sure what the size penalty for including all the charsets is, and whether there's even a mechanism for selectively including charsets in a native image library.

There's definitely more to explore with the bundled version of the Xerces parser too – whether the most recent versions might trigger inclusion of the charsets, for one.

Whatever the outcome is, this will require us to make a release to fix, and any workarounds in the meantime will involve parsing the XML files from strings that have been converted to UTF-8, which is trivial on Python but maybe less so under C++.

technically, windows-1252 is not a supported charset in XML

I try to avoid that word "supported" if I possibly can - it has so many different meanings.

Syntactically, XML requires the encoding to have the form [A-Za-z] ([A-Za-z0-9._] | '-')*

Semantically, the spec offers recommendations on what encodings should be accepted and how they should be named (which certainly don't include windows-1252), but the general rule is that a parser can support provide whatever encodings it chooses.

I suspect that Xerces allows you to use any encoding that's recognised by the Java VM, and that this can vary from one JVM (or JVM configuration) to another.