Project

Profile

Help

XQuery compile failing in static type check

Added by Anonymous about 16 years ago

Legacy ID: #4981635 Legacy Poster: David Lee (daldei)

I have a very weird state dependant reproducable saxon crash. Saxon B 9.0.0.5 - from sourceforge Given this XQuery -------- declare variable $A as xs:string external ; declare variable $PATH as xs:string external ; declare variable $_ external; <foo attr='{$A}'/> -------- If I attempt to compile this query after having used saxon for a while in the same JVM then I get a crash with this callstack. null java.lang.NullPointerException at net.sf.saxon.expr.TypeChecker.staticTypeCheck(TypeChecker.java:112) at net.sf.saxon.functions.SystemFunction.checkArgument(SystemFunction.java:91) at net.sf.saxon.functions.SystemFunction.checkArguments(SystemFunction.java:75) at net.sf.saxon.expr.FunctionCall.typeCheck(FunctionCall.java:127) at net.sf.saxon.expr.ExpressionVisitor.typeCheck(ExpressionVisitor.java:181) at net.sf.saxon.instruct.SimpleNodeConstructor.typeCheck(SimpleNodeConstructor.java:116) at net.sf.saxon.expr.ExpressionVisitor.typeCheck(ExpressionVisitor.java:181) at net.sf.saxon.instruct.Block.typeCheck(Block.java:290) at net.sf.saxon.expr.ExpressionVisitor.typeCheck(ExpressionVisitor.java:181) at net.sf.saxon.instruct.ParentNodeConstructor.typeCheck(ParentNodeConstructor.java:166) at net.sf.saxon.expr.ExpressionVisitor.typeCheck(ExpressionVisitor.java:181) at net.sf.saxon.query.XQueryExpression.<init>(XQueryExpression.java:67) at net.sf.saxon.query.QueryParser.makeXQueryExpression(QueryParser.java:121) at net.sf.saxon.query.StaticQueryContext.compileQuery(StaticQueryContext.java:338) at net.sf.saxon.s9api.XQueryCompiler.compile(XQueryCompiler.java:177) at org.xmlsh.sh.shell.Expander.parseXExpr(Expander.java:377) ===== my code from here up The weird thing is that if I compile this query early on, then I dont get the crash later. I can only reproduce this after running about a dozen test scripts (some of which do XQuery code, others just build documents), then when hitting this test case I get the crash. If I run the test first, then the rest of the test (including this one) then I dont get a crash. The only static variable I have that I think could be effecting this is re-using the Processor instance, but I tried with and without reusing it (new on every compile vs reuse) and the same thing happens. Here's the code snippet that calls XQuery, but by itself it doesnt cause the crash, only after running other code first ... private XdmValue parseXExpr(String arg) { Processor processor = mShell.getProcessor(); XQueryCompiler compiler = processor.newXQueryCompiler(); XQueryExecutable expr = null; StringBuffer sb = new StringBuffer(); for( String name : mShell.getEnv().getVarNames() ){ XValue value = mShell.getEnv().getVarValue(name); sb.append("declare variable $").append(name) .append(" as ") .append( value.isString() ? "xs:string" : "item()*") .append(" external ;\n"); } sb.append("declare variable $_ external;\n"); sb.append(arg); try { expr = compiler.compile( sb.toString() ); // CRASH HERE XQueryEvaluator eval = expr.load(); for( String name : mShell.getEnv().getVarNames() ){ XValue value = mShell.getEnv().getVarValue(name); eval.setExternalVariable( new QName(name), value.toXdmValue()); } eval.setExternalVariable( new QName("_") , new XValue(mShell.getArgs()).toXdmValue() ); return eval.evaluate(); } catch (SaxonApiException e) { mLogger.warn("Error expanding xml expression: " + arg , e ); mShell.printErr("Error expanding xml expression"); } return null; } ----------------- I know this might be very hard to debug, any suggestions welcome ! I dont think I can trim down the path much, but I can try. I can also send the entire system if you were interested in debugging it yourself. Since its reliably crashing within saxon and not elsewhere I suspect a case of some static or otherwise kept state somewhere thats corrupting things within saxon. Side note, I am also running saxon calls in multiple threads, but I concurrently use any of the same objects (except Processor, and I've tried with and without reusing/sharing it). Any suggestions really appreciated ! -David Lee www.xmlsh.org


Replies (6)

Please register to reply

RE: XQuery compile failing in static type che - Added by Anonymous about 16 years ago

Legacy ID: #4982123 Legacy Poster: Michael Kay (mhkay)

This is certainly a weird one! The fact that it's crashing on this code: if (reqItemType == null || suppliedItemType == null) { throw new NullPointerException(); } suggests strongly that it's happened before: it's the kind of code that would only be inserted so that I could put a debugger breakpoint on it. However, that doesn't help with diagnosis! I think that you're almost certainly right that some data structure is being corrupted by earlier activity: it might well be the TypeHierarchy object, which is owned by the Processor/Configuration and is probably used in computing the values of reqItemType and suppliedItemType. I suspect it's related to some very specific feature of your workload, which I'm unlikely to be able to reproduce. So if you can put together something that generates the whole workload, I'd be very grateful. It doesn't matter how complex it is, I should be able to work backwards from the crash so long as I can reproduce it. Do you do anything unusual with ClassLoaders, by any chance? Michael Kay

RE: XQuery compile failing in static type check - Added by Anonymous about 16 years ago

Legacy ID: #4987882 Legacy Poster: David Lee (daldei)

I have posted both the source and built jar's (which includes saxon) in ftp://lee.calldei.com/pub/saxon_bug there are 2 zip's and 2 gz's but you only need one set or the other. Minimally you only will need the source zip or gz but its easier to get going if you install the binaries. You should be able to build from source if needed, there is a top level 'build.xml' ant file. I have reproduced this on both windows and linux, but have only done builds on windows to date. (should build on linux though). Reproduce by extracting the non-source zip/gz+tar file and running the main test cases. To run you need to set 1 environment variable XMLSH pointing to the top level install directory, and java 1.6 in your PATH, then run xmlsh by running one the scripts in the unix/ or win32 directories There are more details if you need it on starting xmlsh at http://www.xmlsh.org/Run ... but you shoudnt need them, just run the script or see what it does, it just runs java with the classpath set right. Then run the test cases cd test ./run_tests.xsh The crash occurs on test case core/core_xexpr.xsh There is an echo line in this test right prior to the code that triggers it: # echo the following crashes saxon sometimes X=<[<foo a1='{$B}'>{$A}</foo>] You can AVOID this crash by manually running this test prior to running the test suite cd test/core ./core_xexpr.xsh cd .. ./run_tests.xsh now the system is "primed" magically and will no longer display the crash in this instance of the JVM. Note: you'll need java 1.6 or later or you'll get some error output, but it doesnt affect this test, crashes with java 1.5 as well. If you have any problems getting this up and going, or if you have any questions building or navigating the source or anything at all please let me know. I'm doing nothing special with classpaths or classloaders. -David Lee

RE: XQuery compile failing in static type che - Added by Anonymous about 16 years ago

Legacy ID: #4988019 Legacy Poster: Michael Kay (mhkay)

Thanks. I worked out that to get xmlsh to run I needed to create a directory c:/tmp. Then I worked out (by use of xsh) that my current directory was c:/users/mike, and worked out how to navigate to the right place by an appropriate sequence of cd ../../etc. Paths starting "c:/" or "c:&quot; were rejected. Then I ran run_tests.xsh as suggested and got no crash. I got this: $ cd test $ ./run_tests.xsh running tests in core Running test core_andor.xsh Running test core_args.xsh Running test core_brace.xsh Running test core_case.xsh Running test core_for.xsh Running test core_here.xsh Running test core_if.xsh Running test core_subshell.xsh Running test core_vars.xsh Running test core_while.xsh Running test core_xexpr.xsh Differs at byte: 92 core_xexpr.xsh out/core_xexpr.xsh.out different output after exit running tests in builtin Running test builtin_colon.xsh Running test builtin_echo.xsh Running test builtin_exit.xsh Running test builtin_read.xsh Running test builtin_set.xsh Running test builtin_shift.xsh Running test builtin_source.xsh Running test builtin_xread.xsh $ That was with JDK 1.6.0_05. So I switched to JDK 1.5.0_06 and this time got the crash in typeChecker.staticTypeCheck. The challenge now will be working out how to run this in the debugger.

RE: XQuery compile failing in static type che - Added by Anonymous about 16 years ago

Legacy ID: #4988067 Legacy Poster: Michael Kay (mhkay)

OK, we're making progress. I've got it to crash in the IntelliJ debugger, which reveals that the static constant StringValue.SINGLE_SPACE has been corrupted (its typeLabel field is null). Now we've found the body, it should be a small step to find the murder weapon.

RE: XQuery compile failing in static type che - Added by Anonymous about 16 years ago

Legacy ID: #4989102 Legacy Poster: Michael Kay (mhkay)

Turns out this is essentially the bug reported by Mike Rutter: http://markmail.org/message/sbbwnyzo4o52l4c2 which I have fixed in the 9.1 branch but not in 9.0. The problem is that your first contact with Saxon is to call "new XdmAtomicValue()" and this is accessing the static variables in class BuiltInAtomicType before they are initialized by calling the static init() method. The safest workaround is to create a Saxon Configuration object (or a s9api Processor object) before you do anything else; this will cause the static initialization to be performed. I resisted retrofitting the 9.1 solution (which is adapted from Mike Rutter's proposed solution) to 9.0 because I feared it might be destabilizing. However, there seems to be a much simpler patch, which is to add the static initializer static { init(); } to class BuiltInAtomicType (and also BuiltInListType). I'll issue this as a Subversion patch.

RE: XQuery compile failing in static type check - Added by Anonymous about 16 years ago

Legacy ID: #4989521 Legacy Poster: David Lee (daldei)

Thank you ! This worked, I put new Processor(false); in the main class's static initialization block and solved the problem ! Also thank you for reporting the problems you had getting the app up and running. -David

    (1-6/6)

    Please register to reply