Saxon

How large is the serialized output? Does it work if you serialize to a file or to standard output? Is there memory pressure? Have you looked at a heap dump at the point where it hangs?

A StringBuffer uses an underlying char[] array so the data requires contiguous memory, I can imagine memory pressure causing difficulty in expanding the array.

The size of the xml before the transformation is 915.310.643 utf-8 chars (we did manage to write this the our database using the serializeNodeToString). When I run xdmNode.getStringValue().length() on this xml I get 147.421.083. After the transformation I still get 147.421.083 from xdmNode.getStringValue().length(), this is a bit strange because the transformations adds attributes to certain xml-tags which should increase the size. Anyway, when performing the serializeNodeToString on this transformed xml (still 915.310.643 chars, or more?) the program hangs ... I also tried calling the garbage colletion before the serializeNodeToString, no success.

I'm considering other ways of storing the xml in our database

• one option would be to split the xml into 3 parts (the xml contains 3 kind of elements) and then store these seperately and later on merge them again when needed

• another option would be to compress (zip) the xml before storing it (but probably it first has to be transformed to a string before it can be zipped)

• may it be possible to store the xdmNode as a blob in the database ? Any suggestions are welcome.

• we are working into access to our heap-dump (preparing visual vm)

• and we are also considering serializing the xml to file; is there a saxon method/function to do this?

A single tree of this size is certainly going to put a big strain on the system. The TinyTree allocates large contiguous arrays whose size is proportional to tree size, so it will need contiguous memory from Java heap allocator, which is asking a lot. Using the LinkedTree would need more memory in total, but it will be allocated as many small fragments, which is less likely to cause problems.

Note that the string value of an element node in XDM represents the concatenation of the text node descendants, which ignores any space occupied by attributes. Simply constructing the string value in order to determine its size could require a lot of space allocation (the way we do this changes significantly in Saxon 11) - doing sum(//text()!string-length()) is probably much cheaper.

Really, you should be trying to find a way to tackle this using XSLT 3.0 streaming. I don't know what your workload is actually doing with the XML, so streaming might or might not be possible, but a streamed solution would be much more manageable at this kind of data size.

-short update- We changed the treemodel for this xml to TINY_TREE_CONDENSED and after the transformation the serializeNodeToString took only 7 seconds. The size of the resulting xml was 1.198.309.034 chars. We wil run some more regression tests tomorrow and will also try the LINKED_TREE. -so far, so good-