Saxon » SaxonC

After some more attempts I have now made two changes in the setup.py

if system() == 'Darwin':
    extra_link_args.append('-L../libs/darwin')
elif system() == 'Windows':
    extra_link_args.append('..\\libs\\win\\libsaxon-hec-12.4.1.lib')
else:
    extra_link_args.append('-L../libs/nix')

# extented modules
ext_modules = [Extension(
                         "saxonche", 
                        sources=[
                            "python_saxon/saxonc.pyx",
                            "../Saxon.C.API/SaxonProcessor.cpp",
                            "../Saxon.C.API/SaxonCGlue.c",
                            "../Saxon.C.API/SaxonCXPath.c",
                            "../Saxon.C.API/XdmValue.cpp",
                            "../Saxon.C.API/XdmItem.cpp",
                            "../Saxon.C.API/XdmNode.cpp",
                            "../Saxon.C.API/XdmAtomicValue.cpp",
                            "../Saxon.C.API/XdmFunctionItem.cpp",
                            "../Saxon.C.API/XdmMap.cpp",
                            "../Saxon.C.API/XdmArray.cpp",
                            "../Saxon.C.API/DocumentBuilder.cpp",
                            "../Saxon.C.API/Xslt30Processor.cpp",
                            "../Saxon.C.API/XsltExecutable.cpp",
                            "../Saxon.C.API/XQueryProcessor.cpp",
                            "../Saxon.C.API/XPathProcessor.cpp",
                            "../Saxon.C.API/SchemaValidator.cpp",
                            "../Saxon.C.API/CythonExceptionHandler.cpp",
                            "../Saxon.C.API/SaxonApiException.cpp",],
                        language="c++",
                        include_dirs = ['../Saxon.C.API/graalvm',],
                         extra_link_args = extra_link_args,
                       libraries=['..\\libs\\win\\libsaxon-hec-12.4.1']
                        ),
                ]

that seem to get me a step further but then I get another error:

   Creating library build\temp.win-amd64-cpython-312\Release\../Saxon.C.API\saxonche.cp312-win_amd64.lib and object build\temp.win-amd64-cpython-312\Release\../Saxon.C.API\saxonche.cp312-win_amd64.exp
CythonExceptionHandler.obj : error LNK2001: unresolved external symbol pysaxonapierror
  Hint on symbols that are defined and could potentially match:
    "struct _object * pysaxonapierror" (?pysaxonapierror@@3PEAU_object@@EA)
build\lib.win-amd64-cpython-312\saxonche.cp312-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.38.33130\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120

So I still need help on how to build the module under Windows.

No help possible here? Is that problem on CythonExceptionHandler.obj : error LNK2001: unresolved external symbol pysaxonapierror perhaps the one you had in an earlier release where Windows therefore didn't support PySaxonApiError on Windows? But that got fixed in later releases, are the released build files for Windows somehow not up to date with that relevant fix? Or have I just messed up my intent to follow the build instructions in the documentation?

Yes I think it is related. I will look into it in the week

It's hard to imagine how the release build could be out-of-date. Unless something very strange has happened, the Windows wheels were built with the same code that appears in the release.

I've fired off the Windows builds again in the hope that that's informative. I can try doing them locally on Windows tomorrow (or this week, anyway).

I am trying to follow the documentation at https://www.saxonica.com/saxon-c/documentation12/index.html#!starting/installingpython to build the module (not really the wheel) under Windows, to me the documentation there for Windows does not work but perhaps it is completely outdated, it mentions Cython while the whole stuff you have linked to doesn't even seem to use Cython.

I am not sure currently whether your Github action uses some already Cython compiled stuff and just downloads it with e.g. "Get the Windows x86 binaries" or whether Cython is not needed.

Anyway, I am even more lost than before, perhaps I just need to wait for SaxonC 12.5..

Or someone can fix https://www.saxonica.com/saxon-c/documentation12/index.html#!starting/installingpython for Windows during the week.

Cython is required. But I wonder if the problem is related to the C++ compiler you are using on your Windows machine. Problem have been reported when using msvc. See the bug issue comment here: https://saxonica.plan.io/issues/5936#note-2

I have checked that the file CythonExceptionHandler.cpp has the line extern "C" PyObject *pysaxonapierror;; and yes, I build under Windows and that uses MSVC 14.38.33130 cl.exe and link.exe.

Still failing with (my translation of partly German error messages)

   library "build\temp.win-amd64-cpython-312\Release\../Saxon.C.API\saxonche.cp312-win_amd64.lib" and object "build\temp.win-amd64-cpython-312\Release\../Saxon.C.API\saxonche.cp312-win_amd64.exp" are being created.
CythonExceptionHandler.obj : error LNK2001: unresolved external symbol "pysaxonapierror".
  notice for symbols that have been resolved and could possibly match:
    "struct _object * pysaxonapierror" (?pysaxonapierror@@3PEAU_object@@EA)
build\lib.win-amd64-cpython-312\saxonche.cp312-win_amd64.pyd : fatal error LNK1120: 1 nicht aufgelöste Externe
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.38.33130\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120

Original output with a a mix of English and German

   Bibliothek "build\temp.win-amd64-cpython-312\Release\../Saxon.C.API\saxonche.cp312-win_amd64.lib" und Objekt "build\temp.win-amd64-cpython-312\Release\../Saxon.C.API\saxonche.cp312-win_amd64.exp" werden erstellt.
CythonExceptionHandler.obj : error LNK2001: Nicht aufgelöstes externes Symbol "pysaxonapierror".
  Hinweis zu Symbolen, die definiert wurden und möglicherweise übereinstimmen:
    "struct _object * pysaxonapierror" (?pysaxonapierror@@3PEAU_object@@EA)
build\lib.win-amd64-cpython-312\saxonche.cp312-win_amd64.pyd : fatal error LNK1120: 1 nicht aufgelöste Externe
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.38.33130\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120

(I'm sure I typed a comment in here earlier today, but it's no where to be seen)

The actual command that the CI system uses to make the wheels is

python -m pip wheel \path\to\hec_pypi --wheel-dir=\path\to\temp --no-deps

Curiously that succeeds but all of my attempts to use build_ext or the like have failed with the same linking error reported above.

Digging a little deeper, the only difference (that I can see) between the way that wheel invokes the compiler and the way build_ext invokes the compiler is that build_ext adds -Ipython_saxon where wheel does not.

I haven't worked out where that's coming from, unfortunately.

Thanks for the update, Norm.

At least it sounds as if me trying to follow the build instructions from the documentation for Windows and getting a linking error is something you can reproduce.

Kind of a relief for me even if that doesn't help avoiding the linking error.

Yes. We're working on it. The build instructions work for Mac and Linux, and the wheels build on Windows, so we failed to notice the discrepancy. We're always working to improve testing, but Windows such a difficult platform to script that it's lagging a little behind.

I think we found the problem. Huge shout out to O'Neil for perserverance!

Make sure that you have Cython version 0.29.x installed and not something more recent like 3.0.5. It turns out there's some sort of bug in more recent versions of Cython that cause this linking problem (but only on Windows). We should probably try to track that down and report it.

The reason that -m pip wheel succeeds (and build_ext or bdist_wheel do not) is that the wheel command uses the versions of dependencies listed in pyproject.toml where we explicitly call out particular versions of setuptools and cython. If you just run the python commands by hand you get the version installed locally.

We made the problem worse by shipping a requirements.txt file with the wrong version of Cython in it. #facepalm.

Martin, if you have success with Cython 0.29.x, would you please let us know? Thanks!

Hi,

great, indeed after

pip install Cython==0.29.36
Collecting Cython==0.29.36
  Downloading Cython-0.29.36-py2.py3-none-any.whl.metadata (3.1 kB)
Downloading Cython-0.29.36-py2.py3-none-any.whl (988 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 988.3/988.3 kB 3.1 MB/s eta 0:00:00
Installing collected packages: Cython
  Attempting uninstall: Cython
    Found existing installation: Cython 3.0.5
    Uninstalling Cython-3.0.5:
      Successfully uninstalled Cython-3.0.5
Successfully installed Cython-0.29.36

the command python setup.py build_ext -if now runs through without problems (about linking) and creates a module saxonche.cp312-win_amd64.pyd.

So in terms of building the module the problem is solved.

I will need to check whether the module works fine but I suppose it will as the ones in your wheels do.

It should work, yes. And python setup.py bdist_wheel will make an actual wheel file that you can install. Bear in mind, however, that one of the cibuildwheel steps that's part of our distribution process is tinkering with the linking in the DLL. I think that an extension or wheel that you build yourself will have a dependency on the location of the underlying DLL. So just copying it to another machine might not work.

I think the Windows version of the tool that patches up the DLL linking is called delvewheel if you want to run it on your custom builds.

I struggling to use the built module, trying to set (in Powershell) $env:PYTHONPATH to the directory where the build module is but somehow python on trying to import something from saxonche tells me (sorry about the mix of German and English but it is literally what I get) "ImportError: DLL load failed while importing saxonche: Das angegebene Modul wurde nicht gefunden." which is (my translation) "ImportError: DLL load failed while importing saxonche: The named module could not be found.".

Not sure what else needs to be set or referenced.

This is my result I was typing while Norm's latest two messages arrived.

Will need to test whether the suggested python setup.py bdist_wheel does give some wheel I can install and use.

PS C:\Users\marti> $env:PYTHONPATH="C:\Program Files\Saxonica\libsaxon-HEC-windows-amd64-v12.4.1\pypi"
PS C:\Users\marti> python
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from saxonche import PySaxonProcessor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing saxonche: Das angegebene Modul wurde nicht gefunden.
>>> quit()
PS C:\Users\marti> ls $env:PYTHONPATH

    Directory: C:\Program Files\Saxonica\libsaxon-HEC-windows-amd64-v12.4.1\pypi

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          01/12/2023    12:35                build
d----          13/12/2023    16:23                python_saxon
d----          01/12/2023    20:27                samples
d----          01/12/2023    20:27                testsuite_driver
-a---          01/12/2023    20:27            182 conftest.py
-a---          01/12/2023    20:27            163 pyproject.toml
-a---          01/12/2023    20:27           4747 README.md
-a---          01/12/2023    20:27             50 requirements.txt
-a---          13/12/2023    16:25         657408 saxonche.cp312-win_amd64.pyd
-a---          13/12/2023    16:25           3132 setup.py
-a---          01/12/2023    20:27           4095 test_saxon_schema.py
-a---          01/12/2023    20:27          62774 test_saxonc.py
-a---          01/12/2023    20:27            533 tox.ini

The python setup.py bdist_wheel does build a wheel and e.g. PyCharm installs that but then doesn't find any PySaxonProcessor or other Py classes but offers autocompletion on from saxonche import to stuff like test_saxon_schema.

Hmm, I am afraid today that is all giving me some real headache; will try tomorrow, perhaps with a clean install, whether it works out without all those oddities.

I don't understand enough about how the Windows APIs work to suggest what's wrong. It seems like the DLL should be available on the machine where you built the library at the location where you built the library, but Windows does seem to agree.

AFAICT, this is the sort of problem that delvewheel was designed to fix. And if I ask delvewheel to show me the libraries required by the wheel, it identifies the SaxonC library. If I ask delvewheel to repair the wheel, it repackages the wheel (in a directory called wheelhouse I think, if you don't tell it to put it somewhere else) successfully. If I install that wheel, it works fine.

I think this is as good as its going to get. If someone with more Windows experience can explain how the pieces fit together (or should fit together) when delvewheel hasn't been run, I'd be happy to see that documented.

Not sure if bug must reopened but @Martin's comment #16 indicates that stubs are missing from the wheels.

As a result, IDEs - I use PyCharm - can't infer types, classes, methods, etc.

There's a separate PyPi package called saxonche-stubs that addresses this, but shouldn't the stubs be included in the wheels with every release/version?