Saxon

On further investigation one thing we have noticed is a marginal increase in the size of the schema model. Is this something you would expect given changes that have been made to support the additional features of 9.6?

We have looked further into the increase in memory used by the schema grammar model and it seems to be connected to the move away from the use of name pools for QNames; in particular the fact that QNames are handled as char arrays, as opposed to integer pointers. In our case, we load the schema grammars into a DOM and deduplicate the strings (using String.intern() or similar) hence concatenating the parts of the QName into a char array actually increases memory usage.

Although the impact on an individual schema grammar is likely to be relatively small, if there are many such models in memory at once then the cumulative effect may be significant. For example, in our case we have a web service that is pre-configured with about 20 such models; each one is a few 10s of megabytes bigger but the cumulative effect is to use an additional 0.5Gb.

Thanks for reporting it. There seem to be two separate issues here, one with the size of compiled stylesheets and one with the size of compiled schemas. My experience with such things is that the devil is usually in the detail so it would be useful to have specimen programs that illustrate the problem that we can investigate - though I appreciate this can be difficult to achieve for example with a stylesheet that has extension function calls into a complex library. Memory usage isn't something we measure routinely in regression tests so it's entirely possible that some slippage has crept in between releases. Reducing locking contention on the NamePool has been a significant theme of recent releases and it's not at all impossible that the effort to reduce the use of shared memory has caused some increase in use of unshared memory; but it's also very likely that we can fix the problem if we can pin it down.

Without seeing your code, my guess would be that the increase in the number of instances of LocalVariableReference might be due to function inlining. There's no obvious change between 9.3 and 9.6 that would cause function inlining to happen more often than in 9.3, but as I say, the devil is in the detail and only looking at how the optimizer tackles your specific code is likely to show it up. Function inlining of course involves a space/time trade-off and we tend to assume that the primary objective is to reduce time rather than space.

I think the replication of classes associated with extension functions could well relate to the way you are manipulating the function library in the XPath static context: again, we would need to monitor execution of your code to get an understanding of this.

Continuing the same exploration, I tried changing two HashSets in ElementDecl (substitutionGroupMembers and complexTypesUsingThisElement) to simple lists. A slightly better improvement here: down to 16.7Mb. We've now cut the number of HashMaps down from 5400 to 2070; but the number of nodes in these hashmaps is only down from 51K to 39K; we've basically been knocking out the small hashmaps but it's the nodes in the larger hashmaps that are taking the space.

A significant number of the remaining hashMaps are in the finiteStateMachine.alphabet property, and this is unchanged from 9.3. (Most of these maps are likely to be very small, though...)

Off at a tangent: I can't find any code that uses the ElementDecl.substitutionGroupMembers() as a set, all uses seem to simply iterate over it so it might as well be a list, except possibly for eliminating duplicates during construction. However, this hasn't changed from 9.3, so it doesn't contribute to any regression since 9.3.

Why has the number of java.lang.String instances increased from 10,223 to 22,801? (And with it, the number of char[] instances from 10,030 to 29,270?)

A large number of the original 10,223 String instances are the values of attributes held in String[] arrays within AttributeCollectionImpl objects. The number of AttributeCollectionImpl objects is unchanged. Many of the extra strings appear to be held in NoNamespaceName objects which are themselves held in the NodeName[] array of an AttributeCollectionImpl, representing the names of attributes in the schema document. However, both the NoNamespaceName objects and the underlying strings are shared by all like-named attributes within a single document, so it seems surprising that this should account for so many Strings. The number of NoNamespaceName objects is only 657 so this isn't enough to account for all the extra Strings.

Olson-related timezone names seem to feature quite a lot, e.g "Azerbaijan Summer Time".

The generatedId properties of ElementDecl objects feature strongly. The targetNamespace property also features surprisingly often; one would expect the same String object to be reused here.

Beyond that, we seem to have an awful lot of Strings referring to internal stuff, e.g. class names within the Saxon code or within the JDK.

We haven't accounted for the difference. Perhaps I need to learn how to do OQL queries in JVisualVM.

I'm exploring another angle. In principle, the LinkedTree instances representing the parsed schema documents should be eligible for garbage collection once the schema component model has been built. Ensuring that this is actually the case should give a significant boost in memory usage. I'm analyzing the paths from the heap root to the XSDSchema object representing the document element of a schema document using jhat, which seems better at this kind of analysis than JVisualVM. (Took the heap dump using jmap; couldn't work out how to save a heap dump taken using JVisualVM for analysis by jhat).

It seems that the ElementDecl schema component has a link back to the LinkedTree XSDElement node via its namespaceResolver property. The AttributeDecl and AttributeUse components also hold namespaceResolvers with the same problem

We can solve this easily by copying the namespace context when necessary: we have the mechanism to do this with the SavedNamespaceContext class, designed for the analogous situation with XSLT processing. The Facet class is already doing this.

After making these changes, the size of a heap dump comes down substantially, from 17.5Mb to 10.1Mb, which is less than the Saxon 9.3 figure. Although this doesn't address the changes that caused the regression, it effectively solves the problem by finding compensating improvements elsewhere.

As a beneficial side-effect, I found there was code in the path for reloading a schema component model from an SCM file that appears to handle the namespace context incorrectly, and the change should fix this.

Since the namespace context is very often the same for all elements in a schema document, it would be useful to share the copies of the namespace context in the same way as we do for compiled stylesheets. I will look into that.

Thank you for the updates.

Note that we don't use any XSLT in our processor. It looks like you have found a similar pattern of memory use in the schema models as we did so, as far as we are concerned, no further action is required. I suspect some of the observations in my original report are worth us looking at more closely but don't seem to have been the cause of the increased memory use.

We will need to upgrade in order to incorporate your changes and wish to release our processor quite soon. This raises a few issues for us:

1. 9.6 is currently highlighted as the most stable and reliable release. Do you have a timetable for when 9.7 will be regarded as stable? What are the likely implications of using a non-stable release?

2. When is there likely to be a maintenance release on the 9.7 branch incorporating these changes?

3. Would you consider making these changes on the 9.6 branch? This would allow us to obtain the memory optimizations without having to adapt our processor to the changes in the API that occurred between 9.6 and 9.7. Note also that bug 2736 (reported today), which appears to be a regression between 9.6 and 9.7, will block our ability to upgrade.