Support #6039


Questions regarding QueryLibrary feature

Added by Jai Mashalkar 7 months ago. Updated 6 months ago.

Start date:
Due date:
% Done:


Estimated time:
Legacy ID:
Applies to branch:
Fix Committed on Branch:
Fixed in Maintenance Release:


We have multiple xquery files used in our product, each of which have nested imports. We are trying to build a solution to optimize compilation of these xquery modules using the QueryLibrary feature provided in saxon ee . There are some errors we face, can you let us know your recommendation for solving the below issues:

Problem There are some xqueries XQ1, XQ2 which have a set of nested imports as shown below XQ1 /
A D /
B C / C / B

XQ2 - B - C - B

Preferable Solution: Separate QueryLibrary for each of A, B, C, D, XQ1 and XQ2. Steps to be followed:

  • Maintain a cache of QueryLibrary
  • When any import is encountered during compilation, check the cache - if present link it, if not compile into a QueryLibrary and then link, so that it can be used in subsequent imports.

Issues: - Repeated imports - C is imported from both A and D. When A and D are imported into XQ1, the functions of C get added twice and result in "Function already present" error. This happens as the functions from C don’t get recognized as duplicate in the QueryLibraryImpl - link function. - Question: Is it ok to override the link function to allow proper checking of this duplicate function, so that it works in our case? Would there be any risks to doing this? If so, any other workaround recommended?

- **Cyclic imports** - Trying to create QueryLibrary for modules that have cyclic dependencies leads to infinite import loop, or function not found error if empty modules are attached to get over the infinite loop. 

Question: Is this by design(the cyclic imports should be packaged together in a single QueryLIbrary), or is there a way to create separate QueryLibraries for each of the imports in this case?

Files (28.7 KB) Saxonica unit tests for XQueryCompiler Michael Kay, 2023-05-15 16:24
Actions #1

Updated by Michael Kay 7 months ago

Thanks for asking. This will need a bit of research - I haven't used separately-compiled query modules in anger myself, and I'm not sure that many people do.

Actions #2

Updated by Jai Mashalkar 7 months ago

ok, thanks for the reply.

Could you give me a high level idea on the design, how it would work better -

  1. Top Down - Start compiling the xquery and create QueryLIbraries for modules encountered along the way.
  2. Bottom Up - Figure out modules that can be put into QUeryLibraries, construct them and then compile the main Query.

Ideally we would like approach 1 to work, but from the Saxon code, I get the idea that you expect it to work by 2nd way - QueryLIbraries should be present already in the static context if need to be reused. Is that understanding correct?

Actions #3

Updated by Michael Kay 7 months ago

It's difficult to work out from your description what you are actually doing at the code level: which APIs you are using, and how.

I'm attaching a Java module containing some of our unit tests in this area, seeing how we do it will hopefully give you some ideas. I'll then try and add some tests that are closer to the scenarios you describe.

It sounds a bit as if you are trying to maintain your own cache of separately compiled modules and link them manually, whereas our examples all rely on the cache maintained by the s9api XQueryCompiler.

Seeing some of your code would make it a lot easier for us to see what you are attempting and whether it can be made to work.

Actions #4

Updated by Michael Kay 7 months ago

I don't think it's possible to handle cyclic imports this way. I believe that when you compile a module, the modules it depends on must already be available, and that implies an ordering. (The original XQuery 1.0 specification forbade cycles, and our design in this area probably dates from that time.)

It's definitely a bottom-up model: the imported library modules should be present in the XQueryCompiler before the importing module is compiled. If you use a ModuleURIResolver to handle "import module" declarations as they are encountered, you could in principle attempt a recursive call on the XQueryCompiler to compile that library module, but I don't think it would be picked up by the current "import module" declaration.

The theory is that if the same module URI is encountered more than once in the import tree, whether through a cycle or through a diamond structure, the module is only compiled once: so you only get one set of functions and variables, with no duplicates. This is complicated by the fact that the spec allows more than one module to share the same module URI. In this case the configuration option Feature.XQUERY_MULTIPLE_­MODULE_­IMPORTS comes into play: if it is set, then Saxon effectively uses the location URI of the module rather than the module URI to decided whether two modules are the same. Furthermore, if you write your own ModuleURIResolver, then Saxon's ability to detect that you are importing the same module twice may depend on the way you set the SystemId property in the returned StreamSource objects.

I've described this in terms of the s9api (XQueryCompiler) API. If you use the internal, now deprecated, StaticQueryContext API the rules are rather similar but there are variations in detail. If you use XQJ, I don't believe you can use separately compiled modules.

Actions #5

Updated by Jai Mashalkar 7 months ago

Thanks for the detailed analysis. Its a little difficult to share a sample code here, I will try to explain at a high level

  • Yes we are trying to create a custom cache, since the XqueryCompiler and StaticQueryContext cannot be kept common for all of our xquery executions for various reasons such as high number of queries.
  • We are using the s9api, XqueryCompiler with a customer StaticQueryContext.
  • We are trying to plugin the QueryLibrary cache by overriding the getCompiledLibrary(namespace) method of StaticContext, which is checked before importing a module - it would return a QueryLibrary from cache, or else build one.

From what you described, at a high level, it looks like it would work for normal case, with the exception of cyclic import?

If you need more details let me know, I could share the POC code privately via saxon support email.

Actions #6

Updated by Michael Kay 7 months ago

Overriding system classes with your own versions means that you're outside the scope of what we test and support, but that doesn't mean it won't work. It does mean you're more exposed than most people to anything that we change from one release to the next, and it means that you're going to need to acquire a pretty deep understanding of some of the Saxon internals in this area.

Actions #7

Updated by Jai Mashalkar 7 months ago

Got it, will be careful and try to minimize the overriding. Thank you for your help!

Actions #8

Updated by Michael Kay 6 months ago

  • Status changed from New to Closed

Please register to edit this issue

Also available in: Atom PDF