Saxon

Regarding code lines from Ant build-file:

	<target name="validate-final">
		<schemavalidate failonerror="no" noNamespaceFile="${xsd-file}" classname="com.saxonica.ee.jaxp.ValidatingReader" classpath="${saxon-jar}">
			<fileset dir="${process-dir}Entries" includes="**/*.xml"/>
		</schemavalidate>
	</target>

I'm afraid I don't know what Ant does with a call like this. It's a little surprising that it runs out of memory during Categories.getCategory() because that's reading a Saxon datafile (Unicode character categories) that should only need to be read once. This might suggest that Ant is calling Saxon thousands of times, rather than just calling it once to validate thousands of files. I'll need to try and reproduce it under the debugger to see what's going on.

The other problem here is that if Ant is invoking Saxon inefficiently, there's not all that much we can do about it. We have introduced configuration switches in the past so that Saxon can ignore some of the things Ant asks it to do, but that gets very messy.

I've taken a look at how the Ant schemavalidate task operates by running it in the debugger and seeing what calls are made on Saxon.

For each source document to be validated, it makes a new call on "new ValidatingReader()", and supplies the source schema by calling setProperty("http://apache.org/xml/properties/schema/external-noNamespaceSchemaLocation", fileName)

This means that Saxon is being initialised from scratch for every file processed, and there is no re-use of the compiled schema (let alone resources such as the Unicode character categories).

This is pretty bad news. It's not easy to do anything about it, and I wonder whether it's worth it anyway since Ant is no longer flavour of the month. The only thing we could conceivably do would be to cache schema information across calls in some static cache.

But you're not actually complaining about processing time, your'e complaining about memory. And the fact that each call initialises Saxon from scratch means that it's very unlikely to be Saxon that's accumulating resource in memory, it must be Ant that's doing so.

Your next step should probably be to try and get a heap dump to see what objects are accumulating in memory.

Alternatively, avoid using the schema validate task, and invoke Saxon's validation API directly, either using the command-line interface or calling a Java application that wraps the s9api API. Either way, you'll find it's quite easy to load the schema once and use it to validate multiple files.

Hmmm. At the point where it's running out of memory, it's calling net.sf.saxon.regex.charclass.Categories.getCategory(), which is reading an XML file. The method getCategory() should only do anything the first time it's called in a JVM (because it keeps the resulting information in static data). I've checked this by amending my test schema to use a regex that requires character categories, and it's only reading this XML file once. It seems rather surprising that you should run out of memory while reading this file (which isn't especially large) if it's only read once during the whole process. I can't explain that.

What's more, while it's reading the categories file it's also creating a TinyTree and the actual failure occurs when calling new TinyTree(): it looks like this is causing classes to be loaded, which surely can only happen if they aren't already loaded, unless there's some complication involving multiple classLoaders. I've no idea, of course, what the custom Ant ClassLoader is doing.

Strange.

Another way you could explore what's going on is to create your own implementation of ValidatingReader that delegates to the Saxon ValidatingReader; this would enable you to trace what calls are being made, and you could also consider caching the Saxon Configuration in static data. Unfortunately the ValidatingReader doesn't expose the Configuration that it uses, but you could get it by reflection.