Saxon

As another user (not support), I tend to use the -t command line option to get output like Memory used: 58.956.216, done in https://dev.saxonica.com/repos/archive/opensource/latest9.9/hej/net/sf/saxon/Transform.java through

System.err.println("Memory used: " + String.format("%,d", Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()));

I think, so value is "bytes" used. That gives an indication of the memory used by Saxon for the particular transformation.

Hi Martin,

Thanks, never realized -t did that. Sounds like to do what I need. Will try this and see how much wisdom that would bring

Thanks a lot Geert

A few years ago we made a substantial effort to reduce memory requirements for the indexes used for XSD unique and key constraints. I don’t think we’ve done the same for XSLT keys and there’s probably significant potential by recognising special cases, eg string-valued keys. The thing that adds a lot of overhead is the complex rules for untypedAtomic.

An index for xsl:key is implemented as Map <AtomicMatchKey, List<NodeInfo>> index. But it's more complicated than that. If any of the values selected by the use expression are untypedAtomic, then they will also be added to a list of``xs:untypedAtomic` values present in the index; and when key() is called with (say) an xs:date as the sought value, new entries will be added to the index obtained by converting each untypedAtomic value to an xs:date.

(UPDATED) The list of untypedAtomic values is used only for internally-generated keys supporting predicates that use the "=" operator. Explicit keys use the semantics of "eq" which do not have this complication (untyped atomic values only match strings).

If you're indexing strings and using the default collation, the AtomicMatchKey will be an object of class CodepointMatchKey; this object has a single field which is a UnicodeString, and a UnicodeString has two fields: an int holding a cached hashCode, and an array of 8-bit, 16-bit, or 32-bit integers depending on the highest codepoint present in the string. So for a 12-character key using the Latin-1 alphabet, I reckon the array is 24 bytes, the UnicodeString is 24 bytes, and the CodepointMatchKey is 16 bytes; if this identifies a node uniquely then the map entry will also hold an ArrayList of length 1, but we initialize the list with an initial length of 4, so I reckon the map entry as a whole is probably around 128 bytes.

The best way of reducing this is probably to avoid the List<NodeInfo> in the case where there is only one node, by making the entry hold either a NodeInfo or a List<NodeInfo> - it's not nice to lose static type safety in such cases, but there are certainly times when it's worth it. Holding an array of NodeInfo rather than an ArrayList would also give a small saving, but the most common case to optimize for is probably when the entries are all singleton nodes.

Two further ways to reduce the space occupied by indexes:

(a) we could make UnicodeString implement AtomicMatchKey, so for the common case where the key is a string or untypedAtomic, we don't need to wrap the UnicodeString in a separate CodepointCollationKey object.

(b) in the case of a TinyTree, we could hold the integer node number rather than a NodeInfo object. For indexes representing xsl:key, the nodes in the index all belong to the same document, and knowing the Tree and the node number is enough to quickly reconstitute a NodeInfo. The NodeInfo representing a node in a TinyTree has three fields (tree, nodeNr, parent) and furthermore, the presence of the NodeInfo in the index will prevent the parent and ancestor NodeInfo objects being garbage-collected.

I have implemented some of these ideas for the next major release: primarily (1) a singleton entry holds a NodeInfo rather than a List<NodeInfo>, and (b) class CodepointMatchKey is abolished, we use UnicodeString in its place.

A specialized index implementation for the TinyTree is a possibility, but a bit more complicated.

Hi Michael,

Thank you very much for giving this issue so much attention. I thought you might be interested in some statistics

Tests are done on a windows machine, transformations run on the command line And given that the processes take long anyway, I don't care too much about the Java warmup cost

The transformation tested a number of times on a 50 MB source file resulting in a 34 MB result file...

original XSLT runs 70 minutes and shows a memory usage of 196 MB using -t as Martin suggested

if I introduce xsl:key where ever it makes sense...

xsl:key XSLT runs 60 seconds and shows a memory usage of 148 MB using -t

I did expect the performance gain since I know experience linear behavior on the big files I am surprised about the memory consistently becoming smaller using keys, because I did not expect that to happen options for a theory about that

• since the original process is running for 70 minutes, there might be a lot of memory grokking up from different processes going on
• maybe Saxon EE spotted all the //foo[if = $thisid] constructs and used the extra memory in optimising those
• or... Tend to ignore that number as unreliable, but wanted to mention it anyway

Intrigued about your explanation in this thread, I thought about saxon:index If I am not mistaken the index key you have there is a string, unless explicitly typed in the XSLT So I rewrote the XSLT to use saxon:index and maps to replace the xsl:key

saxon:index XSLT runs 63 seconds and shows a memory usage of 142 MB using -t

I believe this is the interesting bit I have done a lot of different measurements and the trend is very clear using saxon:index tends to be an ignorable bit slower (average of 2-3 seconds on a 60 seconds process) saxon:index is using substantially less memory than xsl:key to achieve the same (142MB vs 148MB) In my particular case that is substantial because 6MB out of 148MB consistently seems to be index table overhead

Reading what you wrote before (and honestly not understanding all the details of it) the difference no longer was a surprise to me, the fact that the difference was so big actually was a surprise

I hope the numbers are somewhat useful for you In any case thanks again for giving this so much attention

Best regards

Geert

I can't see any obvious explanation why using an xsl:key should reduce the memory consumption, but I think I would need to study the code to try and understand it: for example, it might be that the code without keys is doing sorting-into-document-order, which would obviously consume memory.

The difference you're seeing between saxon:index and xsl:key is rather marginal. The reason for the difference isn't at all obvious, because the data structures used are very different internally. It might be that a slightly different scenario might give completely different results.