Project

Profile

Help

Memory not cleared on saxonica transformer

Added by Sherman Lee almost 10 years ago

I am calling the Saxonica Transformer multiple times to transform huge xml data and xslt to other text files but I notice the memory usage is high after calling the transform API and it does not clear the memory even I called ‘getUnderlyingController().clearDocumentPool()’ and close the transformer after every transformation. Please kindly advise on how to clear the memory after every transformation based on the coding below:

ByteArrayOutputStream out = new ByteArrayOutputStream();
xsltData = "" + "<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform\">" + "<xsl:output method="" + layout + ""/>" + "<xsl:template match="/">" + xsltData + "</xsl:template>" + "</xsl:stylesheet>"; Source xmlSource = new StreamSource(new StringReader(xmlData)); Source xsltSource = new StreamSource(new StringReader(xsltData)); Processor processor = new Processor(false); XsltTransformer transformer = processor.newXsltCompiler().compile(xsltSource).load(); transformer.setSource(xmlSource); Serializer output = new Serializer(); output.setOutputStream(out); transformer.setDestination(output); output.close(); transformer.transform(); transformer.getUnderlyingController().clearDocumentPool(); transformer.close();

Thanks in advance!!!


Replies (5)

Please register to reply

RE: Memory not cleared on saxonica transformer - Added by Michael Kay almost 10 years ago

My recommendation is to create a new XsltTransformer (by calling XsltExecutable.load()) for each transformation, unless you actually want to retain resources (such as keys) from one transformation to the next. The only reason to reuse an XsltTransformer is to reuse its resources.

Creating an XsltTransformer is a low-cost operation, provided that you re-use the XsltExecutable.

I'm a little surprised that clearDocumentPool() doesn't work for you; to understand why, it would be necessary to analyze a heap dump. There are other resources that could be the culprit, for example stylesheet parameters.

RE: Memory not cleared on saxonica transformer - Added by Sherman Lee almost 10 years ago

Thanks Michael for your quick response.

I believe a new XsltTransformer has already been created for each transformation:

XsltTransformer transformer = processor.newXsltCompiler().compile(xsltSource).load();

Even if I break down the code into this:

XsltCompiler xsltCompiler = processor.newXsltCompiler(); XsltExecutable xsltExecutable = xsltCompiler.compile(xsltSource); XsltTransformer transformer = xsltExecutable.load();

The high memory consumption issue still exist whenever transformer.transform(); is called.

As for the stylesheet parameters, it is just a simple stylesheet as shown:

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:output method="text"/> <xsl:template match="/">NAME,ALIAS <xsl:for-each select="OUTPUT/DATA"> <xsl:value-of select="GIVEN_NAME"/>,<xsl:value-of select="GIVEN_ALIAS"/> </xsl:for-each> </xsl:template> </xsl:stylesheet>

RE: Memory not cleared on saxonica transformer - Added by Michael Kay almost 10 years ago

OK sorry, I read your prose which says you are calling the transformer multiple times, and didn't look at your code which shows you are only calling it once.

In fact I see that you are even creating a new Processor for each transformation, which is an expensive thing to do, but means that there is no possibility for Saxon to hold on to any resources.

There have been some bugs in the past relating to caches (e.g. of type information) held in static data but there are no known problems in current releases. You haven't said which Saxon version you are running on.

Once you've checked you are using the current release (9.5.1.5), the next stage should probably be to get a heap dump (e.g. using HAT) to show which objects are accumulating in memory, and where they are linked from.

RE: Memory not cleared on saxonica transformer - Added by Sherman Lee almost 10 years ago

No worries for that.

I am using SaxonHE9.5.1.4. Btw, I am quite sure that the transformer is holding the memory as the high memory consumption issue will only occur if transformer.transform(); is called.

FYI, my xml data contains 2 millions of nodes thus I suspect it is too huge for the transformer to clear...

RE: Memory not cleared on saxonica transformer - Added by Michael Kay almost 10 years ago

Obviously the memory use of the transformer will be high if the source document is large. (To avoid that, you need to think about using a streaming transformation, which needs Saxon-EE).

But once the transformation is complete, the memory should be cleared by the garbage collector.

I'm confused now as to what you are saying. Is your problem

(a) that a single transformation uses a large amount of memory, or

(b) that the memory acquired during a transformation is not being garbage collected when the transformation ends?

You say you are sure that the transformer is holding the memory, but that's inconsistent with (b): if the memory is still locked down when the Transformer is released this would imply that the memory is held by something other than the Transformer.

    (1-5/5)

    Please register to reply