Saxon

Thanks for the quick answer. I don't think we have the old Java on the machine any more (it's is our sys gurus that just made the major annual upgrade), so I can't test it myself (but maybe I will find a machine here that does have it).

I put the source files on http://nl.ijs.si/et/tmp/saxon/ (just please let me know when you have them, so I can remove them).

As you note, it's probably not a big thing as it won't work on such a big file anyway, still..

I will try with streaming mode, as soon as I figure out how to do it :)

Best,

Tomaž

PS: just had another crash reported with saxon on a web service - it looks like the version bundled with java 1.8 has problems with java 1.8. But that is probably something to report to Java folks, not you!

Thanks for running the tests and making the patch. OK, so file is simply

too big; still, it is, I guess, nice that a misleading error is not

reported.

I tried processing with the linked tree model - after one hour my

machine also ran out of heap space..

So, the moral is to work with smaller files, at least until Java handles

larger structures.

All the best,

Tomaž

Dne 22.12.2014 ob 18:37 je Saxonica Developer Community zapisal(a):

I won't pretend to understand Saxon datastructures and functions that

need to be modified but, yes, for me it would certainly be nice if I

could feed large files to XSLT; I'm working with corpora, where there

are lots of conversions to be done, and working on the whole corpus /

file is of course very convenient (it's not such a bit thing to split

it, but it's one more layer of complication). And my server has 47G of

memory, so no problem there.

So, very glad that it seems doable, and fingers crossed that a dark cold

winter day comes along to provide the opportunity to do it :)

All the best,

Tomaž

Dne 24.12.2014 ob 0:06 je Saxonica Developer Community zapisal(a):

Is this really fixed, or is it actually "won't fix"? When I run XSLT against a document which previously resulted in this error under Saxon-HE 9.6.0.5, I now just get the error:

java.lang.IllegalStateException: Source document too large: more than 1G characters in text nodes
        at net.sf.saxon.tree.tiny.LargeStringBuffer.addSegment(LargeStringBuffer.java:58)
        at net.sf.saxon.tree.tiny.LargeStringBuffer.append(LargeStringBuffer.java:120)
        at net.sf.saxon.tree.tiny.TinyTree.appendChars(TinyTree.java:405)
        at net.sf.saxon.tree.tiny.TinyBuilder.makeTextNode(TinyBuilder.java:380)
        at net.sf.saxon.tree.tiny.TinyBuilder.characters(TinyBuilder.java:362)
        at net.sf.saxon.event.ReceivingContentHandler.flush(ReceivingContentHandler.java:544)
        at net.sf.saxon.event.ReceivingContentHandler.endElement(ReceivingContentHandler.java:435)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1781)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2957)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
        at net.sf.saxon.event.Sender.sendSAXSource(Sender.java:440)
        at net.sf.saxon.event.Sender.send(Sender.java:171)
        at net.sf.saxon.Controller.transform(Controller.java:1692)
        at net.sf.saxon.s9api.XsltTransformer.transform(XsltTransformer.java:547)
        at net.sf.saxon.Transform.processFile(Transform.java:1056)
        at net.sf.saxon.Transform.doTransform(Transform.java:659)
        at net.sf.saxon.Transform.main(Transform.java:80)
Fatal error during transformation: java.lang.IllegalStateException: Source document too large: more than 1G characters in text nodes

This is reproducible by running even an identity transform against a large XML file (e.g. enwiktionary-latest-pages-meta-current.xml.bz2 from https://dumps.wikimedia.org/enwiktionary/latest/).

This looks like a won't fix issue after reading comment #5. I will ask Mike to confirm.

It was 'resolved/fixed" in the sense that we now produce an error message saying that Saxon limits have been exceeded, rather than an AIOOBE exception. Anything more than this would be a significant enhancement, not a simple bug fix.

As reported, I did look at the possibility of supported larger documents beyond the 2G limit. With strings and arrays in Java limited to 32-bit int addressing, it's not at all easy. The limits for a LinkedTree would probably be rather higher than for the TinyTree, which tries to reduce the number of Java objects and therefore ends up with objects whose size appriximates to the document size. It might be worth seeing how far you get with the Linked Tree: the actual amount of memory used will be higher, but I don't think you'll hit the same limits.