Saxon

Andrew Welch adds: In Java 8 the permgen area is gone, and classes can be garbage collected.

I managed to write a simple Java program which uses the Saxon's s9api interface to compile a stylesheet of 2.8k lines of XSLT in a for-loop 2000 times, see below a snippet of the program:

    public MultipleStylesheets(String args[]) {
        DATA_DIR = args[0];
        max_num = Integer.parseInt(args[1]);
        processor.setConfigurationProperty(FeatureKeys.GENERATE_BYTE_CODE, "true");
        processor.setConfigurationProperty(FeatureKeys.DEBUG_BYTE_CODE, "false");
        xsltCompiler = processor.newXsltCompiler();
    }

    public void loadStylesheets() throws SaxonApiException {
        StreamSource source1 = null;
        XsltTransformer trans[] = new XsltTransformer[max_num];
        for(int i = 0; i< max_num; i++){
            source1 = new StreamSource(new File(DATA_DIR + "/xmlspec.xsl"));
            trans[i] = xsltCompiler.compile(source1).load();
            // trans[i] = null; // **** - remove comment to force garbage collection of the XsltTransformer object after use. ***
            System.out.println("Stylesheet "+(i+1)+" loaded");
        }



    }

We observe in the loop the compiled stylesheet is loaded and stored in an array item of type @XstlTransformer@.

Experiments:

==========================

Bytecode enabled:

Running this program with bytecode generation enabled it runs out of memory in the 94th iteration of the loop, with the following exception:

Stylesheet 92 loaded
Stylesheet 93 loaded
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
	at java.lang.Class.getMethod0(Class.java:2670)
	at java.lang.Class.getMethod(Class.java:1603)
	at com.saxonica.bytecode.util.Generator.invokeInstanceMethod(Generator.java:260)
	at com.saxonica.bytecode.ItemCheckerCompiler.compileToIterator(ItemCheckerCompiler.java:66)
	at com.saxonica.bytecode.util.CompilerService.compileToIterator(CompilerService.java:716)
	at com.saxonica.bytecode.SlashExpressionCompiler.compileToIterator(SlashExpressionCompiler.java:70)
...

As mentioned earlier using the JVM options to increase memory (i.e. -XX:MaxPermSize=256m -Xmx -Xms and etc) helps avoid or delay this out of memory problem. I tried this with the settings MaxPermSize=128m and @MaxPermSize=256m@: execution of the program I got up to 209 and 439 iterations of the loop, respectively, before getting an out of memory error.

We found that setting '@trans[i] = null;@' (by removing the comment in the code above) forced garbage collection and the program ran successfully to the end. A fix of this sort will probably work for users as a workaround, if loading stylesheets from Java.

==========================

Bytecode disabled (Interpreted mode):

Running this program with bytecode generation disabled showed we also ran into out of memory problems, but this time reported in the GC. The failure occurred after 1542 iterations of the loop, see the the exception below. The out of memory error this time was not with the PermGen and we also observe that the memory load which can be handled is much higher under the interpreted mode.

Stylesheet 1542 loaded
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.AbstractList.iterator(AbstractList.java:273)
	at net.sf.saxon.expr.instruct.NumberInstruction.iterateSubExpressions(NumberInstruction.java:327)
	at net.sf.saxon.expr.Expression.setContainer(Expression.java:735)
	at net.sf.saxon.expr.Expression.setContainer(Expression.java:741)
	at net.sf.saxon.expr.instruct.Procedure.setBody(Procedure.java:55)
	at net.sf.saxon.expr.instruct.Template.setBody(Template.java:95)
	at net.sf.saxon.style.XSLTemplate.compileDeclaration(XSLTemplate.java:331)
...

As mentioned in comment #0 I am looking at giving more control to enable/disable bytecode generation at the level of individual compilation of the stylesheets.

Thank you Michael(s), O'Neil and Andrew for providing such excellent information so quickly. Indeed our maven tests are forked into a jvm with no permgen memory specification, so by adding -XX:MaxPermSize=128M this did indeed fix the problem. The tests had been failing at exactly the same location, late in the overall set, so the fix is very apparent. I suspect that this may have been an issue in our web app having to be restarted, because I was dynamically compiling stylesheets, and re-compiling them based on requests. I have made the overall set of compiled stylesheets static, setup at app start up. Hopefully that will help with performance and stability.

Cheers,

Peter

For 9.6 I have implemented a switch to control bytecode generation for XSLT at the level of the s9api XsltCompiler and the CompilerInfo object, independent of the setting in the Configuration. It defaults to the value in the Configuration at the time the XsltCompiler is created.