Thanks for sending us your a detailed analysis of your issue. I will probably have to setup a python environment with Saxon/C to try and reproduce the seg error.

Hi,

Is it possible that you send a repo of your python code please.

Sure: https://github.com/ajelenak/pysaxon.

It does not have all the code I use for debugging. I can push a branch with that if you want. Also, I have created a Docker image for this work since I am on a Mac. Its Dockerfile is in the repo but I can also push the image I use to Docker Hub.

I tried a new approach in my Python test where I avoided calling @Item.getStringValue()@ (that calls C++ @XdmItem::getStringValue()@) and used instead @SaxonProcessor.getStringValue(Item)@ (calls C++ @SaxonProcessor::getStringValue(XdmItem*)@) and the test worked. The result is correct: @text1@.

The debug info:

In SaxonProcessor.__cinit__()
what = None
init = False
global _init = 1
C++: In SaxonProcessor::SaxonProcessor(bool l)
C++: l = 0
C++: refCount = 2
C++: before environ = 0xfa7810
C++: SaxonProcessor* = 0xe82730
SaxonProcessor* = 0xe82730
In SaxonProcessor.newXPathProcessor()
SaxonProcessor* = 0xe82730
before XPathProcessor* = (nil)
C++: In SaxonProcessor::newXPathProcessor()
C++: SaxonProcessor* = 0xe82730
C++: environ = 0xfa7810
after XPathProcessor* = 0x16395c0
In SaxonProcessor.parseXmlFromString()
SaxonProcessor* = 0xe82730
before XdmNode* = (nil)
C++: In SaxonProcessor::parseXmlFromString()
C++: SaxonProcessor* = 0xe82730
C++: environ = 0xfa7810
after XdmNode* = 0x1818b70
In XPathProcessor.setContextItem()
XPathProcessor* = 0x16395c0
XdmItem* = 0x1818b70
In XPathProcessor.evaluate_single()
XPathProcessor* = 0x16395c0
before XdmItem* = (nil)
after XdmItem* = 0x1818d60
In Item.size
XdmItem* = 0x1818d60
C++: In XdmItem::size()
C++: XdmItem* = 0x1818d60
In SaxonProcessor.getStringValue()
C++: In SaxonProcessor::getStringValue()
C++: SaxonProcessor* = 0xe82730
C++: XdmItem* = 0x1818d60
C++: environ = 0xfa7810

Result is: b'text1'

.In SaxonProcessor.__dealloc__()
SaxonProcessor* = 0xe82730
global _init = 1
Destroying C++ SaxonProcessor*
C++: In SaxonProcessor::~SaxonProcessor()
C++: SaxonProcessor* = 0xe82730
C++: refCount = 2
C++: environ = 0xfa7810
C++: Reducing refCount by one
C++: refCount = 1

The @environ@ variable this time contains the correct memory address throughout the test. Could it be that this static variable is for some reason not seen in the correct state by other Saxon/C classes?

Thanks for this. the docker feature interests me. Don't know much about but I am trying to use it. How do I get your docker image? Sorry I am asking beginner question as I have tried the command: 'docker build .'

I pushed a branch named @debug@ to the PySaxon repository with all the debugging code. I also updated the "README":https://github.com/ajelenak/pysaxon/blob/debug/README.md file with instruction on how to set up the development environment.

Few more notes:

• The Docker image does not have a debugger, or text editor. If you need those, or any other software, let me know and I can create another image.
• The installation location of the Saxon/C library is in the @$SAXONC_HOME@ env. variable.
• The @debug@ branch also contains two directories from the Saxon/C distribution with the source code of the C++ tests and the Saxon/C API. I included those because they contain the debug print statements. Also, because of the lack of text editor, the only way of making changes to those files is to have them in a Docker container's mounted volume. After making any changes, copy them into the install locations with a command like: @cp -puv Saxon.C.API/* $SAXONC_HOME/Saxon.C.API/@, and then rebuild the Python package with @python setup.py develop@.

Hi,

Thanks for posting instructions to set things up on docker. I have managed to get everything work and now I am trying to reproduce the seg error using your debug version. I cannot get the print-line statements as described earlier. It seems to be missing something or I have not copied the debug files successfully. I wonder if the *.cpp files are missing from the directory https://github.com/ajelenak/pysaxon/tree/debug/Saxon.C.API ?

My bad. Git was instructed to ignore @.cpp@ files. Fixed now. Please update the branch.

Also... this is the command that will only run the test that crashes and show all the debug output:

python setup.py test -a "-s pysaxon/tests/test_xpath.py"

Ok great thanks for the update.

I have managed to reproduce the failure.

However I am not getting the C++ print line statements. Here is what I get:

pysaxon/tests/test_xpath.py In SaxonProcessor.__cinit__()
what = None
init = False
global _init = 1
SaxonProcessor* = 0x1a6bba0
In SaxonProcessor.newXPathProcessor()
SaxonProcessor* = 0x1a6bba0
before XPathProcessor* = (nil)
after XPathProcessor* = 0x1950e30
In SaxonProcessor.parseXmlFromString()
SaxonProcessor* = 0x1a6bba0
before XdmNode* = (nil)
after XdmNode* = 0x1b37b30
In XPathProcessor.setContextItem()
XPathProcessor* = 0x1950e30
XdmItem* = 0x1b37b30
In XPathProcessor.evaluate_single()
XPathProcessor* = 0x1950e30
before XdmItem* = (nil)
after XdmItem* = 0x1b37d70
In Item.size
XdmItem* = 0x1b37d70
In Item.getStringValue()
XdmItem* = 0x1b37d70
SaxonProcessor* = 0x1a6bba0

JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: system exception at 0x00007fdbba44afb4

Hi,

I noticed in your github repository that you are using the Saxon/C 1.0.0 code base. Is it possible that you can move forward to the latest Saxon/C 1.0.1 code base? There have been many bug fixes in that maintenance release which may or may not resolve your problem but at least it just means we are working with the latest code.

I have managed to copy in the Saxon/C 1.0.1 code base, added the C++ debug statements from your github project and reproduce the error. I will be investigating it closely and report back shortly.

Output:

pysaxon/tests/test_xpath.py In SaxonProcessor.__cinit__()
what = None
init = False
global _init = 1
C++: In SaxonProcessor::SaxonProcessor(bool l)
C++: l = 0
C++: refCount = 2
C++: before environ = 0x165f130
SaxonProc constructor: jvm exists!
C++:jvm exists, environ-2 = 0x165f130
C++: SaxonProcessor* = 0x14c6690
SaxonProcessor* = 0x14c6690
In SaxonProcessor.newXPathProcessor()
SaxonProcessor* = 0x14c6690
before XPathProcessor* = (nil)
C++: In SaxonProcessor::newXPathProcessor()
C++: SaxonProcessor* = 0x14c6690
C++: environ = 0x165f130
after XPathProcessor* = 0x1cf4c00
In SaxonProcessor.parseXmlFromString()
SaxonProcessor* = 0x14c6690
before XdmNode* = (nil)
C++: In SaxonProcessor::parseXmlFromString()
C++: SaxonProcessor* = 0x14c6690
C++: environ = 0x165f130
xmlParseString, Processor: 566252480
XdmNode: 565769384
xmlParserString, ItemType: document-node(element(Q{}out))
after XdmNode* = 0x1ed90e0
In XPathProcessor.setContextItem()
XPathProcessor* = 0x1cf4c00
XdmItem* = 0x1ed90e0
In XPathProcessor.evaluate_single()
XPathProcessor* = 0x1cf4c00
before XdmItem* = (nil)
Properties size: 1
Parameter size: 1
size:2

XdmItem-getUnderlyingValue:
Type of itr:P8XdmValue

XdmItem-getUnderlyingValue:
Type of value:P8_jobject

XdmItem-getUnderlyingValue:
xpathString: //person[1]
after XdmItem* = 0x1ed9330
In Item.size
XdmItem* = 0x1ed9330
C++: In XdmItem::size()
C++: XdmItem* = 0x1ed9330
In Item.getStringValue()
XdmItem* = 0x1ed9330
SaxonProcessor* = 0x14c6690
C++: In XdmItem::getStringValue()
C++: SaxonProcessor* = 0x14c6690
C++: SaxonProcessor* = 0x14c6690
C++: In SaxonProcessor::getStringValue()
C++: SaxonProcessor* = 0x14c6690
C++: XdmItem* = 0x1ed9330
C++: environ* = 0

XdmItem-getUnderlyingValue:

JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: system exception at 0x00007f43648b48ee 

I noticed in your github repository that you are using the Saxon/C 1.0.0 code base. Is it possible that you can move forward to the latest Saxon/C 1.0.1 code base?

I apologize for that. I know the Docker image has the latest Saxon/C version. The @debug@ branch now has the latest code base, too.

Hi,

I have still not got to the bottom as to why the environ static variable changes to the value zero. The Python language and interface world is rather new to me. I feel I need to better understand how python interacts with the C++ interface. I am clutching at straw somewhat but I wonder if there is some sort of garbage collection going on. I will be adding a watcher method to keep track of the environ variable to track when and where it changes state throughout the execution of the program.

Hi,

Finally got to the bottom of this issue. Please see details of the same problem described on stackoverflow: http://stackoverflow.com/questions/4448296/static-variable-resetting-its-value-to-0-on-its-own-apparently

The declaration of environ should have the keyword extern before it. i.e.:

extern static sxnc_environment * environ;

I have created a bug issue for this, see: https://saxonica.plan.io/issues/2772

The patch fix will be available in the next maintenance release. You can just make the change locally or in your github project.

Thanks for reporting the issue.

Hi!

Thank you very much for looking into this issue. I do not use C++ myself and this is my first Python interface to a C++ library.

How can I be notified when the next maintenance release is out? When is it planned for?

Thanks!

You can copy the file SaxonProcessor.h which contains the fix from https://dev.saxonica.com/repos/archive/opensource/latest9.6/hec/Saxon.C.API/

Hello!

Building PySaxon with the new version of @SaxonProcessor.h@ fails with these two compiler errors:

/opt/saxon-HEC-v1.0.1/Saxon.C.API/SaxonProcessor.h:518:38: error: conflicting specifiers in declaration of ‘environ’
     extern static sxnc_environment * environ;
                                      ^
/opt/saxon-HEC-v1.0.1/Saxon.C.API/SaxonProcessor.h:521:23: error: conflicting specifiers in declaration of ‘jvmCreatedCPP’
     extern static int jvmCreatedCPP;

Were you able to build PySaxon with the new code base using my Docker image? Here is what I did:

Create a fresh new Docker container having the v1.0.1 code base.¶

@cd $SAXONC_HOME/Saxon.C.API/@¶

@curl -O https://dev.saxonica.com/repos/archive/opensource/latest9.6/hec/Saxon.C.API/SaxonProcessor.h@

@cd /opt/pysaxon@¶

@python setup.py develop@¶

Compile fails with the above errors¶

Yes you are right there are compile errors with that fix. Looking into it now

Hi,

I have made another attempt at fixing this build problem. Please can you checkout all C/C++ files from the svn repository and try building pysaxon again.

Hello!

Yes, I was able to build PySaxon with those files. Could not run it because the code references @libsaxoneec.so@.

Ok. I have made the change in the SaxonCGlue.h file to point to the HE/C library: libsaxonhec.so

Yes, the test is working now. Got the expected result: @b'text1'@.

Thank you very much for all the help!

Great. It looks like the underlying issue was to do with C++ inheritance within the python wrapper. The main difference was to make the getStringValue method in XdmItem as virtual.

Hello!

Interesting... I can confirm the C++ @virtual@ keyword does seem to fix the issue of the @SaxonProcessor::sxn_environ@'s value when accessed from other Saxon C++ classes. I wrote a PySaxon test like the testXPathValues in the @testXPath.cpp@ file and I had to declare the @XdmNode::getChildCount@ and @XdmNode::getChildren@ methods virtual in order to work.

However, I don't understand (yet) why making those methods virtual is the solution. The PySaxon code is not subclassing any of the Saxon C++ classes.