Project

Profile

Help

Bug #2091

closed

Bytecode generation: performance on .NET

Added by Jeff Monnette almost 10 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
High
Category:
Performance
Sprint/Milestone:
-
Start date:
2014-06-16
Due date:
% Done:

0%

Estimated time:
Legacy ID:
Applies to branch:
Fix Committed on Branch:
Fixed in Maintenance Release:
Platforms:

Description

We are using the Saxon .NET EE API (version 9.4.0.6) for XSLT processing and are experiencing very different performance characteristics when running in licensed mode vs unlicensed mode. In licensed mode, compile time, transformation time, and memory consumption are significantly worse.

I've created a test harness that repeats the following 100 times:

  1. Compile an XSLT

  2. Apply the XSLT to 24 different XML files

Unlicensed mode

Max XSLT Compile 1.18 seconds

Avg XSLT Compile 151 milliseconds

Max Transform (XML file #1) 380 milliseconds

Avg Transform (XML file #1) 84 milliseconds

Memory Usage (at iteration 25) 36.71 MB

Total Elapsed Time 00:03:39

Licensed mode

Max XSLT Compile 179 seconds

Avg XSLT Compile 87.848 seconds

Max Transform (XML file #1) 2069 milliseconds

Avg Transform (XML file #1) 670 milliseconds

Memory Usage (at iteration 25) 588.29 MB

Total Elapsed Time 02:08:55

Observations

  • Licensed mode is orders of magnitude slower and higher memory usage

  • In Licensed mode, compilation got progressively slower with each iteration

  • In Licensed mode, the test harness failed after iteration 85 with an out of memory exception

Please let me know if there is more information that I can provide to help resolve the issue.


Files

image.png (11.4 KB) image.png Jeff Monnette, 2014-06-17 04:01
image.png (11.2 KB) image.png Jeff Monnette, 2014-06-17 04:01
image.png (11.6 KB) image.png Jeff Monnette, 2014-06-17 04:01
SaxonTestHarnessSimple.zip (15.9 MB) SaxonTestHarnessSimple.zip Jeff Monnette, 2014-06-17 04:01
Actions #1

Updated by O'Neil Delpratt almost 10 years ago

Hi,

Thanks for submitting your performance results, which is surprising.

We would like to narrow down what is causing the slowdown in the license mode. Please may you report the performance with the bytecode generation feature on and off.

License Mode:

I assume the results already submitted is with bytecode generation on. To switch off the bytecode generation feature if running from the command line insert the option --generateByteCode:off

From the C# code you can do something like the following:

processor.Implementation.setConfigurationProperty("http://saxon.sf.net/feature/generateByteCode", "false");

You can of course use the constant FeatureKeys.GENERATE_BYTE_CODE@, which has the value @"http://saxon.sf.net/feature/generateByteCode"

Actions #2

Updated by Jeff Monnette almost 10 years ago

Setting http://saxon.sf.net/feature/generateByteCode to "false" appears to

fix the problem. Can you please explain what this does?

On Mon, Jun 16, 2014 at 10:49 AM, Saxonica Developer Community <

> wrote:

Actions #3

Updated by O'Neil Delpratt almost 10 years ago

  • Assignee set to O'Neil Delpratt
  • Found in version set to 9.4.0.6

The byte code generation feature is where we selectively generate byte code representing sub-expressions in the AST of the compiled stylesheet. Please take a look at the documentation on the byte code generation feature:

http://www.saxonica.com/documentation/index.html#!changes/optimization/9.3-9.4

I would like to investigate further what is causing the slowdown when byte code generation is on. It is possible that some path has not generated the byte code efficiently. There is also a possibility that this may be a known bug which has been fixed since 9.4.0.6.

Please may you send us a repro that we can reproduce the performance difference. You can either attach it to the bug issue or send it privately via email if confidential.

Thanks,

O'Neil

Actions #4

Updated by Jeff Monnette almost 10 years ago

Here is a simplified test harness that just runs the compile. It takes

three command line arguments as shown in the examples below.

The first argument is the number of iterations to run.

The second is a Boolean to indicate whether the processor should run in

licensed mode or not. True = licensed mode.

The third is a Boolean to disable the GenerateByteCode option. True =

disabled.

The first example runs in licensed mode with byte code generated NOT

disabled. As you can see, it takes progressively longer to compile the

same XSLT on each iteration.

[image: Inline image 1]

The second example runs in licensed mode with byte code generation

disabled. Compile performance is much more consistent.

[image: Inline image 2]

The third example runs in non-licensed mode with byte code enabled.

Compile performance is similar to example #2.

[image: Inline image 3]

On Mon, Jun 16, 2014 at 11:52 AM, Saxonica Developer Community <

> wrote:

Actions #5

Updated by Michael Kay almost 10 years ago

Just adding a note to show that I'm taking an interest in it, although O'Neil is leading the investigation. There are a number of other issues we've been looking at recently which might or might not be relevant:

(a) on .NET we've seen cases where there is a performance problem because the bytecode (which turns into IL code) that we generate accumulates in memory and never gets garbage collected. We've been experimenting with building Saxon under new versions of IKVM, (and becoming dependent on .NET 4.0 or 4.5), in order to alleviate this.

(b) we're aware of one or two paths in the code where bytecode generation has a negative impact (whether on Java or .NET). We'll be looking at whether this plays a factor here. The paths we know about concern filter expressions with constant predicates; it could be such a case, or it could be another similar case that we haven't encountered before.

(c) there are generally some performance anomalies on .NET that we haven't got to the bottom of; we think that these are associated with particular paths in the OpenJDK library that we use, but we have yet to discover the exact circumstances.

A regards the first problem (a), it seems to be good practice on .NET, at the moment, only to use bytecode generation on stylesheets and queries that will be used often enough to make it worthwhile, and to avoid it on a stylesheet that is only used once if the service as a whole is long-lived.

Actions #6

Updated by Michael Kay almost 10 years ago

  • Subject changed from XSLT Performance in Licensed Mode to Bytecode generation: performance on .NET
  • Status changed from New to In Progress

I reviewed this with O'Neil yesterday; we came to the conclusion that the problem is similar to that described in https://saxonica.plan.io/issues/1833 for the Java case. I know we've investigated something similar recently but I can't find a tracker entry for it. Essentially, on older versions of .NET, when classes/types are dynamically loaded, they remain locked in memory and are never garbage collected. So if you compile a stylesheet with bytecode generation enabled, the memory this uses is never freed until the .NET instance shuts down. This explains the gradual increase in memory usage as you compile more stylesheets, and I think it also explains the performance degradation.

Our plan is to solve this in Saxon 9.6 by taking advantage of the fact that in .NET 4.0 and later, dynamically loaded types can be garbage collected; to make this work we have to build Saxon using a newer version of IKVMC, which also has the effect that Saxon stops working with older versions of .NET. For this reason, we don't think it's appropriate to do this in a maintenance release. As a workaround with Saxon 9.5, we advise disabling bytecode generation at the Processor level for workloads where this problem arises. As mentioned under #1833, we have introduced a more fine-grained switch controlling bytecode generation at the individual stylesheet/query level for 9.6, but in 9.5 it can only be done globally.

Actions #7

Updated by Michael Kay almost 10 years ago

This page contains some useful information on problems with memory leaks connected with dynamically-generated assemblies:

http://documentation.red-gate.com/display/AMP8/Finding+a+dynamically+generated+assembly+leak

Actions #9

Updated by Jeff Monnette almost 10 years ago

Thanks. This is very helpful.

Can you tell me when 9.6 will be released?

On Fri, Jun 20, 2014 at 3:40 AM, Saxonica Developer Community <

> wrote:

Actions #10

Updated by Michael Kay almost 10 years ago

We never publish release dates because we don't have planned dates: we release as soon as we've passed all the tests. But we've moved into the testing phase for the release and this generally takes a couple of months.

Actions #11

Updated by Michael Kay almost 10 years ago

  • Status changed from In Progress to Closed

Please register to edit this issue

Also available in: Atom PDF