Saxon

Additionally, I want to mention that if the central reason that this cannot be (easily) optimized is because of xquery reflection, e.g. fn:function-lookup(), I still request that a feature be added to check the main xquery string and the imported modules to see if they make use of xquery reflection, and if there is no xquery reflection, then apply the optimization. One should not suffer performance penalties for xquery reflection if our main xquery string and xquery libraries do not use xquery reflection.

PS: I am not sure if this would even be possible given the existence of ExtensionFunctionDefinition objects in our processor which are referenced by our xquery libraries. In that case, perhaps we could have a feature that says "we promise that we're not using xquery reflection, and please optimize assuming we are not", aka a feature to disable xquery-reflection and then optimize memory usage by removing unused shallow copy function definitions and (shallow) copy variable definitions.

You are correct that currently a main query is linked with its imported modules in such a way that if many queries import the same modules, some information from those modules will be replicated in each query. It's not a vast amount of data -- essentially an index of global function names and variable names -- but I can see that it builds up.

I think it's hard to avoid this for variables, because the way we manage global variables is to create a single pool of global variables each with distinct slot numbers. (Hard but not impossible, because the XSLT package mechanism does create a separate bindery for each package, and we could translate that design to the XQuery situation.) For functions, I think it's probably rather easier: the "function library" in the main module (which is essentially an index of functions) could contain a reference to the function library in the library module, rather than containing copies of its entries. We would need to change the way that we check for conflicts, because currently the checking for conflicts is done as part of the same operation as the building of the combined index, but that feels do-able. It would mean that the search for a function name takes a fraction longer, but that would be noticeable only if someone imports a very large number of library modules.

The internal design is greatly complicated by the fact that the XQuery compiler can encounter a reference to a function before it encounters its declaration, and by the rules for (a) cyclic imports of modules, and (b) dynamic features such as function-lookup and load-xquery-module. So none of this is easy.

I'm doing something similar to your test 4, importing a library with 10000 function declarations, and it's consuming about 2Mb for each query compilation. If I increase it to 20000 functions, that goes up to around 3Mb. So the overhead is around 1Mb per 10,000 functions, or 100 bytes per function, which is consistent with the theory that it's just adding an entry into an index.

By way of an experiment, I now tried the following:

• compile one query that imports the 20000-function library module.
• get a handle on its FunctionLibrary by doing FunctionLibrary first = exec.getUnderlyingCompiledQuery().getMainModule().getFunctionLibrary();
• add this function library as an extension function library to the XQuery compiler by doing: ((StaticQueryContextEE)compiler.getUnderlyingStaticContext()).setExtensionFunctionLibrary(first);
• Compile further queries without doing an import module (they access the functions as extension functions)

This works, and there is much smaller memory growth (about 4K bytes per query compilation).

This might provide the basis for a workaround.

The main difficulty with productising this is that it bypasses the checks on duplicated function names. Duplicates won't be detected, and the query will bind to whichever function is found first. However, it should be possible to change the implementation of "import module" so that the checking for duplicates uses a transient index of functions, which can be discarded once checking is complete, and then add a reference to the library module's function library rather than adding each function to the index.

Another possibility here is to reduce the cost of the check for duplicates by building into each library module a Bloom filter identifying the function names present; it's then a very fast check to confirm that there are no duplicates, followed by a more lengthy check if the fast check fails.

I'm coming to the conclusion that NOT maintaining the global index of functions at the level of a query executable has too many potential adverse consequences, on the internal structure of the product, on performance, and on diagnostics.

I'm wondering about an alternative approach, that would treat as a special case the execution of a main query module that consists solely of an "import module" plus a query expression - no other declarations in the top level. In this case it ought to be possible for the query expression to be compiled and execute with a static context that is simply a reference to the library module, meaning that there should be very little overhead in running different queries that take all their context from the same (set of) library modules. I'm not sure at the moment if this would be done using some new API, or some new language feature in the query, or a straight optimization detecting queries that fall into this pattern; or perhaps some combination of the three.

What about my proposal?

Ignoring ExtensionFunctionDefinition, it seems like an obvious optimization opportunity: detect when the main query and the imported xquery libs don't use xquery reflection, and then do dependency analysis and remove unused functions and variables from the XQueryExecutable object.

For ExtensionFunctionDefinition objects, include an opt-in flag on the individual ExtensionFunctionDefinition object that says "I promise not to use xquery reflection".

I'll keep that idea in mind; but I don't want to do an extra optimization pass unconditionally that very few users will benefit from, and I'm always reluctant to add more configuration switches that very few users will discover.

I've got another idea.

Currently, as I understand it, you're compiling a large number of queries (let's say 1000) each of which uses the same imported libraries; and you're saving each of these compiled queries as an XQueryExeecutable, presumably because it's going to be executed repeatedly with different parameters.

Now suppose instead that we could compile just one query, let's call it the master query, whose effect is to evaluate one of 1000 different XQuery expressions passed as a parameter? Conceptually this would do something like the xsl:evaluate instruction in XSLT - execute a supplied expression with full access to the functions in the static context of the caller.

But of course, you don't want to compile each of the 1000 expressions each time it is used, you want to compile each one once and then use it repeatedly.

I think it might be possible to achieve this effect today by use of extension functions.

Suppose the master query does:

import module lib="urn:library".
declare variable $query external;
saxon:compile-function($query);

where $query takes the form of an inline function expression, and saxon:compile-function (unlike the current saxon:compile-query) is defined to supply the static context of the caller to the query being compiled (in particular, the in-scope functions).

Then for each of the 1000 queries, instead of compiling a new XQueryExecutable, we would call the master query once supplying the query as a parameter; the call would return an XdmFunctionItem, and wherever the application currently invokes an XQueryEvaluator constructed from the XQueryExecutable, it would instead call the XdmFunctionItem with whatever run-time parameters are required.

I believe (needs confirmation) that it should be possible to implement the proposed saxon:compile-function() today as an integrated extension function, without any product changes, though possibly requiring some fairly deep delving into internal APIs. It's certainly possible if the query is restricted to XPath rather than XQuery syntax; XQuery might be a bit more difficult. I'll do some experiments to explore the possibility.

I think it might be possible to implement an XQueryExecutable.condense() method that eliminates data that has been retained only for debugging or for dynamic evaluation. This won't actually drop unused functions in library modules (because library modules might be used again for compiling another query), it will only remove them from the index held in the top-level module. The main difficulty will be testing that error paths don't crash if they try to access this information when producing diagnostics.

There's a risk that such a method could encourage flawed expectations. As proposed, it will only make a tiny difference to the memory used by one query; the benefit only comes in your use case where you have hundreds of queries sharing the same library modules. I'm conscious that I want to provide something that solves your particular problem but is also of general utility to a wider user base.

Currently, as I understand it, you're compiling a large number of queries (let's say 1000) each of which uses the same imported libraries; and you're saving each of these compiled queries as an XQueryExeecutable, presumably because it's going to be executed repeatedly with different parameters.

Correct. Although I might put my dream as closer to 100,000 or 1,000,000 XQueryExecutable objects per JVM, assuming many relatively small / simple main xquery strings.

There's a risk that such a method could encourage flawed expectations. As proposed, it will only make a tiny difference to the memory used by one query; the benefit only comes in your use case where you have hundreds of queries sharing the same library modules. I'm conscious that I want to provide something that solves your particular problem but is also of general utility to a wider user base.

I understand. I marked this as "low" priority originally, and I understand it is a feature request, and I understand that it should be done a "correct" way instead of a hacky way.

XQueryExecutable.condense()

I like that design too.

many relatively small / simple main xquery strings

Would it be acceptable to restrict these to be XPath expressions rather than XQuery expressions?

The XPathCompiler offers

XPathCompiler.addXsltFunctionLibrary (XsltPackage libraryPackage)

and it would be fairly straightforward to supplement this with XPathCompiler.addXQueryFunctionLibrary (XQueryExecutable libraryPackage).

These xqueries come from our customer. They could contain anything. It would not be limited to just xpath.

For the next major release, I have implemented an experimental optimization:

If an XQuery main module contains no function declarations, and contains a single module import, and the imported module is a precompiled library module, then we don't build a new function index for the main module, rather we reuse the function library already present in the precompiled library module.

Extending this to multiple imports isn't quite so straightforward. And unfortunately we can't just import a library that's the union of several libraries because function imports aren't transitive. We do know that the functions in different libraries are in different namespaces and therefore cannot conflict with each other. Oddly, however, XQuery allows a function in a main module to be in a namespace that is the same as that of an imported module, and this is a possible source of name conflicts.

This suggests that the library of user-defined functions should perhaps be a two-level structure organised first by namespace and then by local-name/arity, rather than being a single-level structure organised by namespace/local-name/arity as at present. It should then be possible to arrange that when a namespace is imported, we simply add an entry to the top-level index for that namespace, and only reconstruct the index for the namespace in the event that functions for that namespace come from more than one place.

Perhaps we could implement this structure using immutable maps. With a module import, we would start by adding a reference to the function library of the imported module to the importing module. If an additional function in the same namespace is encountered, we would make a non-destructive addition to this map, so the map in the main module and the map in the library module have now diverged, but still with reuse of parts that are common between the two.