Bug #2091
closedBytecode generation: performance on .NET
0%
Description
We are using the Saxon .NET EE API (version 9.4.0.6) for XSLT processing and are experiencing very different performance characteristics when running in licensed mode vs unlicensed mode. In licensed mode, compile time, transformation time, and memory consumption are significantly worse.
I've created a test harness that repeats the following 100 times:
-
Compile an XSLT
-
Apply the XSLT to 24 different XML files
Unlicensed mode
Max XSLT Compile 1.18 seconds
Avg XSLT Compile 151 milliseconds
Max Transform (XML file #1) 380 milliseconds
Avg Transform (XML file #1) 84 milliseconds
Memory Usage (at iteration 25) 36.71 MB
Total Elapsed Time 00:03:39
Licensed mode
Max XSLT Compile 179 seconds
Avg XSLT Compile 87.848 seconds
Max Transform (XML file #1) 2069 milliseconds
Avg Transform (XML file #1) 670 milliseconds
Memory Usage (at iteration 25) 588.29 MB
Total Elapsed Time 02:08:55
Observations
-
Licensed mode is orders of magnitude slower and higher memory usage
-
In Licensed mode, compilation got progressively slower with each iteration
-
In Licensed mode, the test harness failed after iteration 85 with an out of memory exception
Please let me know if there is more information that I can provide to help resolve the issue.
Files
Updated by O'Neil Delpratt over 10 years ago
Hi,
Thanks for submitting your performance results, which is surprising.
We would like to narrow down what is causing the slowdown in the license mode. Please may you report the performance with the bytecode generation feature on and off.
License Mode:
I assume the results already submitted is with bytecode generation on. To switch off the bytecode generation feature if running from the command line insert the option --generateByteCode:off
From the C# code you can do something like the following:
processor.Implementation.setConfigurationProperty("http://saxon.sf.net/feature/generateByteCode", "false");
You can of course use the constant FeatureKeys.GENERATE_BYTE_CODE@, which has the value @"http://saxon.sf.net/feature/generateByteCode"
Updated by Jeff Monnette over 10 years ago
Setting http://saxon.sf.net/feature/generateByteCode to "false" appears to
fix the problem. Can you please explain what this does?
On Mon, Jun 16, 2014 at 10:49 AM, Saxonica Developer Community <
dropbox+saxonica+f38e@plan.io> wrote:
Updated by O'Neil Delpratt over 10 years ago
- Assignee set to O'Neil Delpratt
- Found in version set to 9.4.0.6
The byte code generation feature is where we selectively generate byte code representing sub-expressions in the AST of the compiled stylesheet. Please take a look at the documentation on the byte code generation feature:
http://www.saxonica.com/documentation/index.html#!changes/optimization/9.3-9.4
I would like to investigate further what is causing the slowdown when byte code generation is on. It is possible that some path has not generated the byte code efficiently. There is also a possibility that this may be a known bug which has been fixed since 9.4.0.6.
Please may you send us a repro that we can reproduce the performance difference. You can either attach it to the bug issue or send it privately via email if confidential.
Thanks,
O'Neil
Updated by Jeff Monnette over 10 years ago
- File image.png image.png added
- File image.png image.png added
- File image.png image.png added
- File SaxonTestHarnessSimple.zip SaxonTestHarnessSimple.zip added
Here is a simplified test harness that just runs the compile. It takes
three command line arguments as shown in the examples below.
The first argument is the number of iterations to run.
The second is a Boolean to indicate whether the processor should run in
licensed mode or not. True = licensed mode.
The third is a Boolean to disable the GenerateByteCode option. True =
disabled.
The first example runs in licensed mode with byte code generated NOT
disabled. As you can see, it takes progressively longer to compile the
same XSLT on each iteration.
[image: Inline image 1]
The second example runs in licensed mode with byte code generation
disabled. Compile performance is much more consistent.
[image: Inline image 2]
The third example runs in non-licensed mode with byte code enabled.
Compile performance is similar to example #2.
[image: Inline image 3]
On Mon, Jun 16, 2014 at 11:52 AM, Saxonica Developer Community <
dropbox+saxonica+f38e@plan.io> wrote:
Updated by Michael Kay over 10 years ago
Just adding a note to show that I'm taking an interest in it, although O'Neil is leading the investigation. There are a number of other issues we've been looking at recently which might or might not be relevant:
(a) on .NET we've seen cases where there is a performance problem because the bytecode (which turns into IL code) that we generate accumulates in memory and never gets garbage collected. We've been experimenting with building Saxon under new versions of IKVM, (and becoming dependent on .NET 4.0 or 4.5), in order to alleviate this.
(b) we're aware of one or two paths in the code where bytecode generation has a negative impact (whether on Java or .NET). We'll be looking at whether this plays a factor here. The paths we know about concern filter expressions with constant predicates; it could be such a case, or it could be another similar case that we haven't encountered before.
(c) there are generally some performance anomalies on .NET that we haven't got to the bottom of; we think that these are associated with particular paths in the OpenJDK library that we use, but we have yet to discover the exact circumstances.
A regards the first problem (a), it seems to be good practice on .NET, at the moment, only to use bytecode generation on stylesheets and queries that will be used often enough to make it worthwhile, and to avoid it on a stylesheet that is only used once if the service as a whole is long-lived.
Updated by Michael Kay over 10 years ago
- Subject changed from XSLT Performance in Licensed Mode to Bytecode generation: performance on .NET
- Status changed from New to In Progress
I reviewed this with O'Neil yesterday; we came to the conclusion that the problem is similar to that described in https://saxonica.plan.io/issues/1833 for the Java case. I know we've investigated something similar recently but I can't find a tracker entry for it. Essentially, on older versions of .NET, when classes/types are dynamically loaded, they remain locked in memory and are never garbage collected. So if you compile a stylesheet with bytecode generation enabled, the memory this uses is never freed until the .NET instance shuts down. This explains the gradual increase in memory usage as you compile more stylesheets, and I think it also explains the performance degradation.
Our plan is to solve this in Saxon 9.6 by taking advantage of the fact that in .NET 4.0 and later, dynamically loaded types can be garbage collected; to make this work we have to build Saxon using a newer version of IKVMC, which also has the effect that Saxon stops working with older versions of .NET. For this reason, we don't think it's appropriate to do this in a maintenance release. As a workaround with Saxon 9.5, we advise disabling bytecode generation at the Processor level for workloads where this problem arises. As mentioned under #1833, we have introduced a more fine-grained switch controlling bytecode generation at the individual stylesheet/query level for 9.6, but in 9.5 it can only be done globally.
Updated by Michael Kay over 10 years ago
This page contains some useful information on problems with memory leaks connected with dynamically-generated assemblies:
http://documentation.red-gate.com/display/AMP8/Finding+a+dynamically+generated+assembly+leak
Updated by Michael Kay over 10 years ago
Also relevant is this IKVM blog entry:
http://weblog.ikvm.net/CommentView.aspx?guid=0df57cfc-1028-42c0-a706-c5f1bac529d0
Updated by Jeff Monnette over 10 years ago
Thanks. This is very helpful.
Can you tell me when 9.6 will be released?
On Fri, Jun 20, 2014 at 3:40 AM, Saxonica Developer Community <
dropbox+saxonica+f38e@plan.io> wrote:
Updated by Michael Kay over 10 years ago
We never publish release dates because we don't have planned dates: we release as soon as we've passed all the tests. But we've moved into the testing phase for the release and this generally takes a couple of months.
Updated by Michael Kay over 10 years ago
- Status changed from In Progress to Closed
Please register to edit this issue