Maintenance: Planio will be observing a scheduled maintenance window this Tuesday, November 5, 2024 from 03:00 UTC until 06:30 UTC to perform urgent network maintenance in our primary data center. Your Planio account will be unavailable during this maintenance window.
Forums » Saxon/C Help and Discussions »
environ variable zero in one of SaxonProcessor methods crashes Python test
Added by Aleksandar Jelenak over 8 years ago
Hello!
A Python test resembling the @testXPathSingle@ in the @testXPath.cpp@ file causes a segmentation fault. A @jet_err@ file is generated with "@JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: system exception at 0x...@" message. The information in the @jet_err@ file points to the @SaxonProcessor::getStringValue()@ method as the cause of the crash. This is the very first real Python test so don't know if others would have similar problems.
Since the Python interface is basically just wrapping the various C++ pointers returned by the Saxon/C library I thought that printing those pointer addresses would be a good way of tracking down what is happing in both the Python and C++ code. Here's the information I gathered:
The @testXPathSingle@ works when I compile the @testXPath.cpp@ file so the SIGSEGV occurs only in the Python test. The difference is that the @environ@ variable in the @SaxonProcessor::getStringValue()@ method is zero for the Python test and not zero for the @testXPath@ executable. This variable is not exposed to Python as I consider it holding internal state so don't know why it appears zero for one SaxonProcessor method.
Below is the debug information, first for the @testXPath@ executable:
C++: In SaxonProcessor::SaxonProcessor(bool l) C++: l = 0 C++: refCount = 1 C++: before environ = 0 C++: after environ = 0xf1f7d0 C++: environ->myDllHandle = 0x7fbc2c206b40 C++: SaxonProcessor* = 0xf20b90 C++: In SaxonProcessor::newXPathProcessor() C++: SaxonProcessor* = 0xf20b90 C++: environ = 0xf1f7d0 Test: XPathProcessor with Saxon version=Saxon-HE 9.6.0.9J from Saxonica Test testXPathSingle: C++: In SaxonProcessor::parseXmlFromString() C++: SaxonProcessor* = 0xf20b90 C++: environ = 0xf1f7d0 C++: In XdmItem::size() C++: XdmItem* = 0x124ef90 Number of items=1 C++: In XdmItem::getStringValue() C++: SaxonProcessor* = 0xf20b90 C++: SaxonProcessor* = 0xf20b90 C++: In SaxonProcessor::getStringValue() C++: SaxonProcessor* = 0xf20b90 C++: XdmItem* = 0x124ef90 C++: environ = 0xf1f7d0
Note the last line above, where the @environ@ variable holds a valid memory address. For the Python test, debug info below, that value is zero and, hence, the segmentation fault. The debug information without the "@C++:@" prefix comes from the Python interface.
In SaxonProcessor.__cinit__() what = None init = False global _init = 1 C++: In SaxonProcessor::SaxonProcessor(bool l) C++: l = 0 C++: refCount = 2 C++: before environ = 0x221c820 C++: SaxonProcessor* = 0x2d23cb0 SaxonProcessor* = 0x2d23cb0 In SaxonProcessor.newXPathProcessor() SaxonProcessor* = 0x2d23cb0 before XPathProcessor* = (nil) C++: In SaxonProcessor::newXPathProcessor() C++: SaxonProcessor* = 0x2d23cb0 C++: environ = 0x221c820 after XPathProcessor* = 0x2c43810 In SaxonProcessor.parseXmlFromString() SaxonProcessor* = 0x2d23cb0 before XdmNode* = (nil) C++: In SaxonProcessor::parseXmlFromString() C++: SaxonProcessor* = 0x2d23cb0 C++: environ = 0x221c820 after XdmNode* = 0x2e1fad0 In XPathProcessor.setContextItem() XPathProcessor* = 0x2c43810 XdmItem* = 0x2e1fad0 In XPathProcessor.evaluate_single() XPathProcessor* = 0x2c43810 before XdmItem* = (nil) after XdmItem* = 0x2e1fca0 In Item.size XdmItem* = 0x2e1fca0 C++: In XdmItem::size() C++: XdmItem* = 0x2e1fca0 In Item.getStringValue() XdmItem* = 0x2e1fca0 SaxonProcessor* = 0x2d23cb0 C++: In XdmItem::getStringValue() C++: SaxonProcessor* = 0x2d23cb0 C++: SaxonProcessor* = 0x2d23cb0 C++: In SaxonProcessor::getStringValue() C++: SaxonProcessor* = 0x2d23cb0 C++: XdmItem* = 0x2e1fca0 C++: environ = 0
The SIGSEGV crash occurs after this. What is strange is that if I comment out the offending line of code, the correct @environ@'s value appears again when destroying that same SaxonProcessor object at the end of the Python test:
In SaxonProcessor.__dealloc__() SaxonProcessor* = 0x2d23cb0 global _init = 1 Destroying C++ SaxonProcessor* C++: In SaxonProcessor::~SaxonProcessor() C++: SaxonProcessor* = 0x2d23cb0 C++: refCount = 2 C++: environ = 0x221c820 C++: Reducing refCount by one C++: refCount = 1
Included is the @jet_err@ file.
jet_err_4886.txt (46.5 KB) jet_err_4886.txt |
Replies (26)
Please register to reply
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Thanks for sending us your a detailed analysis of your issue. I will probably have to setup a python environment with Saxon/C to try and reproduce the seg error.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Hi,
Is it possible that you send a repo of your python code please.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Sure: https://github.com/ajelenak/pysaxon.
It does not have all the code I use for debugging. I can push a branch with that if you want. Also, I have created a Docker image for this work since I am on a Mac. Its Dockerfile is in the repo but I can also push the image I use to Docker Hub.
I tried a new approach in my Python test where I avoided calling @Item.getStringValue()@ (that calls C++ @XdmItem::getStringValue()@) and used instead @SaxonProcessor.getStringValue(Item)@ (calls C++ @SaxonProcessor::getStringValue(XdmItem*)@) and the test worked. The result is correct: @text1@.
The debug info:
In SaxonProcessor.__cinit__() what = None init = False global _init = 1 C++: In SaxonProcessor::SaxonProcessor(bool l) C++: l = 0 C++: refCount = 2 C++: before environ = 0xfa7810 C++: SaxonProcessor* = 0xe82730 SaxonProcessor* = 0xe82730 In SaxonProcessor.newXPathProcessor() SaxonProcessor* = 0xe82730 before XPathProcessor* = (nil) C++: In SaxonProcessor::newXPathProcessor() C++: SaxonProcessor* = 0xe82730 C++: environ = 0xfa7810 after XPathProcessor* = 0x16395c0 In SaxonProcessor.parseXmlFromString() SaxonProcessor* = 0xe82730 before XdmNode* = (nil) C++: In SaxonProcessor::parseXmlFromString() C++: SaxonProcessor* = 0xe82730 C++: environ = 0xfa7810 after XdmNode* = 0x1818b70 In XPathProcessor.setContextItem() XPathProcessor* = 0x16395c0 XdmItem* = 0x1818b70 In XPathProcessor.evaluate_single() XPathProcessor* = 0x16395c0 before XdmItem* = (nil) after XdmItem* = 0x1818d60 In Item.size XdmItem* = 0x1818d60 C++: In XdmItem::size() C++: XdmItem* = 0x1818d60 In SaxonProcessor.getStringValue() C++: In SaxonProcessor::getStringValue() C++: SaxonProcessor* = 0xe82730 C++: XdmItem* = 0x1818d60 C++: environ = 0xfa7810 Result is: b'text1' .In SaxonProcessor.__dealloc__() SaxonProcessor* = 0xe82730 global _init = 1 Destroying C++ SaxonProcessor* C++: In SaxonProcessor::~SaxonProcessor() C++: SaxonProcessor* = 0xe82730 C++: refCount = 2 C++: environ = 0xfa7810 C++: Reducing refCount by one C++: refCount = 1
The @environ@ variable this time contains the correct memory address throughout the test. Could it be that this static variable is for some reason not seen in the correct state by other Saxon/C classes?
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Thanks for this. the docker feature interests me. Don't know much about but I am trying to use it. How do I get your docker image? Sorry I am asking beginner question as I have tried the command: 'docker build .'
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
I pushed a branch named @debug@ to the PySaxon repository with all the debugging code. I also updated the "README":https://github.com/ajelenak/pysaxon/blob/debug/README.md file with instruction on how to set up the development environment.
Few more notes:
- The Docker image does not have a debugger, or text editor. If you need those, or any other software, let me know and I can create another image.
- The installation location of the Saxon/C library is in the @$SAXONC_HOME@ env. variable.
- The @debug@ branch also contains two directories from the Saxon/C distribution with the source code of the C++ tests and the Saxon/C API. I included those because they contain the debug print statements. Also, because of the lack of text editor, the only way of making changes to those files is to have them in a Docker container's mounted volume. After making any changes, copy them into the install locations with a command like: @cp -puv Saxon.C.API/* $SAXONC_HOME/Saxon.C.API/@, and then rebuild the Python package with @python setup.py develop@.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Hi,
Thanks for posting instructions to set things up on docker. I have managed to get everything work and now I am trying to reproduce the seg error using your debug version. I cannot get the print-line statements as described earlier. It seems to be missing something or I have not copied the debug files successfully. I wonder if the *.cpp files are missing from the directory https://github.com/ajelenak/pysaxon/tree/debug/Saxon.C.API ?
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
My bad. Git was instructed to ignore @.cpp@ files. Fixed now. Please update the branch.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Also... this is the command that will only run the test that crashes and show all the debug output:
python setup.py test -a "-s pysaxon/tests/test_xpath.py"
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Ok great thanks for the update.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
I have managed to reproduce the failure.
However I am not getting the C++ print line statements. Here is what I get:
pysaxon/tests/test_xpath.py In SaxonProcessor.__cinit__() what = None init = False global _init = 1 SaxonProcessor* = 0x1a6bba0 In SaxonProcessor.newXPathProcessor() SaxonProcessor* = 0x1a6bba0 before XPathProcessor* = (nil) after XPathProcessor* = 0x1950e30 In SaxonProcessor.parseXmlFromString() SaxonProcessor* = 0x1a6bba0 before XdmNode* = (nil) after XdmNode* = 0x1b37b30 In XPathProcessor.setContextItem() XPathProcessor* = 0x1950e30 XdmItem* = 0x1b37b30 In XPathProcessor.evaluate_single() XPathProcessor* = 0x1950e30 before XdmItem* = (nil) after XdmItem* = 0x1b37d70 In Item.size XdmItem* = 0x1b37d70 In Item.getStringValue() XdmItem* = 0x1b37d70 SaxonProcessor* = 0x1a6bba0 JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: system exception at 0x00007fdbba44afb4
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Hi,
I noticed in your github repository that you are using the Saxon/C 1.0.0 code base. Is it possible that you can move forward to the latest Saxon/C 1.0.1 code base? There have been many bug fixes in that maintenance release which may or may not resolve your problem but at least it just means we are working with the latest code.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
I have managed to copy in the Saxon/C 1.0.1 code base, added the C++ debug statements from your github project and reproduce the error. I will be investigating it closely and report back shortly.
Output:
pysaxon/tests/test_xpath.py In SaxonProcessor.__cinit__() what = None init = False global _init = 1 C++: In SaxonProcessor::SaxonProcessor(bool l) C++: l = 0 C++: refCount = 2 C++: before environ = 0x165f130 SaxonProc constructor: jvm exists! C++:jvm exists, environ-2 = 0x165f130 C++: SaxonProcessor* = 0x14c6690 SaxonProcessor* = 0x14c6690 In SaxonProcessor.newXPathProcessor() SaxonProcessor* = 0x14c6690 before XPathProcessor* = (nil) C++: In SaxonProcessor::newXPathProcessor() C++: SaxonProcessor* = 0x14c6690 C++: environ = 0x165f130 after XPathProcessor* = 0x1cf4c00 In SaxonProcessor.parseXmlFromString() SaxonProcessor* = 0x14c6690 before XdmNode* = (nil) C++: In SaxonProcessor::parseXmlFromString() C++: SaxonProcessor* = 0x14c6690 C++: environ = 0x165f130 xmlParseString, Processor: 566252480 XdmNode: 565769384 xmlParserString, ItemType: document-node(element(Q{}out)) after XdmNode* = 0x1ed90e0 In XPathProcessor.setContextItem() XPathProcessor* = 0x1cf4c00 XdmItem* = 0x1ed90e0 In XPathProcessor.evaluate_single() XPathProcessor* = 0x1cf4c00 before XdmItem* = (nil) Properties size: 1 Parameter size: 1 size:2 XdmItem-getUnderlyingValue: Type of itr:P8XdmValue XdmItem-getUnderlyingValue: Type of value:P8_jobject XdmItem-getUnderlyingValue: xpathString: //person[1] after XdmItem* = 0x1ed9330 In Item.size XdmItem* = 0x1ed9330 C++: In XdmItem::size() C++: XdmItem* = 0x1ed9330 In Item.getStringValue() XdmItem* = 0x1ed9330 SaxonProcessor* = 0x14c6690 C++: In XdmItem::getStringValue() C++: SaxonProcessor* = 0x14c6690 C++: SaxonProcessor* = 0x14c6690 C++: In SaxonProcessor::getStringValue() C++: SaxonProcessor* = 0x14c6690 C++: XdmItem* = 0x1ed9330 C++: environ* = 0 XdmItem-getUnderlyingValue: JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: system exception at 0x00007f43648b48ee
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
I noticed in your github repository that you are using the Saxon/C 1.0.0 code base. Is it possible that you can move forward to the latest Saxon/C 1.0.1 code base?
I apologize for that. I know the Docker image has the latest Saxon/C version. The @debug@ branch now has the latest code base, too.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Hi,
I have still not got to the bottom as to why the environ static variable changes to the value zero. The Python language and interface world is rather new to me. I feel I need to better understand how python interacts with the C++ interface. I am clutching at straw somewhat but I wonder if there is some sort of garbage collection going on. I will be adding a watcher method to keep track of the environ variable to track when and where it changes state throughout the execution of the program.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Hi,
Finally got to the bottom of this issue. Please see details of the same problem described on stackoverflow: http://stackoverflow.com/questions/4448296/static-variable-resetting-its-value-to-0-on-its-own-apparently
The declaration of environ should have the keyword extern before it. i.e.:
extern static sxnc_environment * environ;
I have created a bug issue for this, see: https://saxonica.plan.io/issues/2772
The patch fix will be available in the next maintenance release. You can just make the change locally or in your github project.
Thanks for reporting the issue.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Hi!
Thank you very much for looking into this issue. I do not use C++ myself and this is my first Python interface to a C++ library.
How can I be notified when the next maintenance release is out? When is it planned for?
Thanks!
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
You can copy the file SaxonProcessor.h which contains the fix from https://dev.saxonica.com/repos/archive/opensource/latest9.6/hec/Saxon.C.API/
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Hello!
Building PySaxon with the new version of @SaxonProcessor.h@ fails with these two compiler errors:
/opt/saxon-HEC-v1.0.1/Saxon.C.API/SaxonProcessor.h:518:38: error: conflicting specifiers in declaration of ‘environ’ extern static sxnc_environment * environ; ^ /opt/saxon-HEC-v1.0.1/Saxon.C.API/SaxonProcessor.h:521:23: error: conflicting specifiers in declaration of ‘jvmCreatedCPP’ extern static int jvmCreatedCPP;
Were you able to build PySaxon with the new code base using my Docker image? Here is what I did:
Create a fresh new Docker container having the v1.0.1 code base.¶
@cd $SAXONC_HOME/Saxon.C.API/@¶
@curl -O https://dev.saxonica.com/repos/archive/opensource/latest9.6/hec/Saxon.C.API/SaxonProcessor.h@
@cd /opt/pysaxon@¶
@python setup.py develop@¶
Compile fails with the above errors¶
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Yes you are right there are compile errors with that fix. Looking into it now
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Hi,
I have made another attempt at fixing this build problem. Please can you checkout all C/C++ files from the svn repository and try building pysaxon again.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Hello!
Yes, I was able to build PySaxon with those files. Could not run it because the code references @libsaxoneec.so@.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Ok. I have made the change in the SaxonCGlue.h file to point to the HE/C library: libsaxonhec.so
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Yes, the test is working now. Got the expected result: @b'text1'@.
Thank you very much for all the help!
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by O'Neil Delpratt over 8 years ago
Great. It looks like the underlying issue was to do with C++ inheritance within the python wrapper. The main difference was to make the getStringValue method in XdmItem as virtual.
RE: environ variable zero in one of SaxonProcessor methods crashes Python test - Added by Aleksandar Jelenak over 8 years ago
Hello!
Interesting... I can confirm the C++ @virtual@ keyword does seem to fix the issue of the @SaxonProcessor::sxn_environ@'s value when accessed from other Saxon C++ classes. I wrote a PySaxon test like the testXPathValues in the @testXPath.cpp@ file and I had to declare the @XdmNode::getChildCount@ and @XdmNode::getChildren@ methods virtual in order to work.
However, I don't understand (yet) why making those methods virtual is the solution. The PySaxon code is not subclassing any of the Saxon C++ classes.
Please register to reply