Saxon

Legacy ID: #7977359 Legacy Poster: Vladimir Nesterovsky (vnesterovsky)

Sorry, the problem was in a different place (in a custom try-catch implementation, that's were XPathContextMajor instances were linked in a wrong way).

Legacy ID: #7977885 Legacy Poster: Michael Kay (mhkay)

Thanks for reporting it, I will certainly investigate. It would be more useful to send me code I can run rather than a heap snapshot, since I will need to test any fix to be sure that it solves the problem.

Legacy ID: #7980298 Legacy Poster: Vladimir Nesterovsky (vnesterovsky)

Unfortunately I was not able to localize the problem, and I do not expect you to fix my bugs, but just in case if it's related to saxon somehow I'll quote here a function that's triggering the problem. When I change a line there (marked with '!!!!'), it starts to work without memory growth: /** * Executes xsl:apply-templates for the try items. * If operation fails, then executes xsl:apply-templates for the chatch items, * xsl:apply-templates is called in current mode. * * Current exception info is available during execution of exception handler * through current-error-info() function returning a sequence: * error as xs:QName; * error-description as xs:string; * error-object as item(); * * and through current-location-info() function returning a sequence of pairs: * ( * location as xs:string, * context as item() * )*; * * @param context - a dynamic context. * @param tryItems - items to execute. * @param catchItems - items to execute in case of error. * @return an output sequence. */ public static SequenceIterator tryBlock( XPathContext context, SequenceIterator tryItems, SequenceIterator catchItems) throws Exception { Controller controller = context.getController(); ParameterSet tunnelParameters = context.getTunnelParameters(); Mode mode = context.getCurrentMode(); SequenceOutputter sequenceOutputter = new SequenceOutputter(); sequenceOutputter.setPipelineConfiguration( controller.makePipelineConfiguration()); try { XPathContextMajor majorContext = context.newContext(); // !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! // IF I COMMENT THE FOLLOWING LINE, IT WORKS. majorContext.setReceiver(sequenceOutputter); TailCall tailCall = ApplyTemplates.applyTemplates( tryItems, mode, ParameterSet.EMPTY_PARAMETER_SET, tunnelParameters, majorContext, 0); while(tailCall != null) { tailCall = tailCall.processLeavingTail(); } } catch(Exception e) { // Release dirty data. sequenceOutputter = new SequenceOutputter();; sequenceOutputter.setPipelineConfiguration( controller.makePipelineConfiguration()); Object data = controller.getUserData(currentErrorKey, currentErrorKey); try { controller.setUserData(currentErrorKey, currentErrorKey, e); XPathContextMajor majorContext = context.newContext(); majorContext.setReceiver(sequenceOutputter); TailCall tailCall = ApplyTemplates.applyTemplates( catchItems, mode, ParameterSet.EMPTY_PARAMETER_SET, tunnelParameters, majorContext, 0); while(tailCall != null) { tailCall = tailCall.processLeavingTail(); } } finally { controller.setUserData(currentErrorKey, currentErrorKey, data); } } return sequenceOutputter.iterate(); }

Legacy ID: #7980497 Legacy Poster: Vladimir Nesterovsky (vnesterovsky)

Now, I have more symptoms. Whenever intermediate SequenceOutputter is used, the final sequece contains many elements that belong to different TinyTree(s). This corresponds to the logic of generation: a cycle calling some template, which in turn forms a couple of elements on each iteration. Here is the most internal template: <xsl:template mode="t:call" match="p:exit-state-handler"> <xsl:param name="id" as="xs:string" tunnel="yes"/> <xsl:variable name="exstate" as="element()?" select=" t:get-data-document($id)/*"/> <xsl:if test="$exstate/@STRING"> <xsl:value-of select="$exstate/@NAME"/> xsl:text=</xsl:text> <xsl:value-of select="$exstate/@STRING"/> </xsl:if> </xsl:template> The problem is that each TinyTree refers to LargeStringBuffer (charBuffer) with in turn refers to char[128K] containing text of element, which is a short line.

Legacy ID: #7980529 Legacy Poster: Vladimir Nesterovsky (vnesterovsky)

Did a crude experiment. Replaced LargeStringBuffer with that of version 9.1. Memory consumption has dropped and became stable.

Legacy ID: #7980635 Legacy Poster: Vladimir Nesterovsky (vnesterovsky)

Pure and simple: <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:t="this" exclude-result-prefixes="xs t"> <xsl:template match="/"> <xsl:variable name="extraction" as="element()" select="t:get-extraction-document()/"/> <xsl:variable name="resources" as="element()"> <xsl:for-each select="$extraction/"> <xsl:call-template name="t:exit-state-handler"> <xsl:with-param name="element" as="element()" select="."/> </xsl:call-template> </xsl:for-each> </xsl:variable> xsl:message <xsl:sequence select="$resources[self::exit-state]"/> </xsl:message> xsl:message <xsl:for-each select="$resources[self::property]"> <xsl:sequence select="text()"/> xsl:text </xsl:text> </xsl:for-each> </xsl:message> </xsl:template> <xsl:template name="t:exit-state-handler"> <xsl:param name="element" as="element()"/> <xsl:variable name="id" as="xs:string" select="$element/@id"/> <xsl:variable name="exstate" as="element()?" select=" t:get-existstate-document($id)/"/> <xsl:variable name="result" as="element()*"> <xsl:if test="$exstate/@STRING"> <xsl:value-of select="$exstate/@NAME"/> xsl:text=</xsl:text> <xsl:value-of select="$exstate/@STRING"/> </xsl:if> </xsl:variable> <xsl:sequence select="$result"/> </xsl:template> <xsl:function name="t:get-extraction-document" as="document-node()"> xsl:document <xsl:for-each select="1 to 3800"> </xsl:for-each> </xsl:document> </xsl:function> <xsl:function name="t:get-existstate-document" as="document-node()"> <xsl:param name="id" as="xs:string"/> xsl:document <xsl:value-of select="$id"/> </xsl:document> </xsl:function> </xsl:stylesheet> Just open task manager and look. :-)

Legacy ID: #7980906 Legacy Poster: Michael Kay (mhkay)

Thanks. The implementation of LargeStringBuffer changed in 9.2 to use fixed-length segments rather than variable-length segments, which had a significant effect in speeding access to text nodes in large documents. But you are right, it seems to be causing a large overhead in memory usage for small documents. It shouldn't be too difficult to correct this. In principle Saxon attempts to reuse the TinyTree structure so that when you create many small elements or documents, they all go in the same TinyTree. That optimization only works in simple cases and it isn't kicking in here either for 9.1 or 9.2. I thought that might be the cause but it seems not.