Saxon

Thanks for reporting it.

I've deleted the attached ZIP file because it contained a Saxon licence key.

What level of CPU and memory utilisation are you seeing? If the CPU is fully utilised with 10 threads, then (unless you have more than 10 CPUs) doubling the number of threads means that each thread will take twice as long to complete, which would account for your observations.

I guess this is a synthetic workload, but in real life it's common to run the same transformation on many source documents, and in that situation you can get a big performance benefit by only compiling the stylesheet once.

If your monitoring tools show any lock contention within Saxon, then it would be useful to know about that.

Saxon10 on .NET is still supported, because .NET Framework 4.8 is still supported. However, it is becoming increasingly unlikely that we will produce any further maintenance releases for Saxon10, and we wouldn't do so for a performance issue unless it makes the product completely unusable.

Saxon10.9 is the latest version that runs on .NET Framework, but for .NET Core (or simply .NET) you should be using SaxonCS 12.6. Both are built from the Java source code, but in very different ways, so performance characteristics might be different. I don't know if your license will cover the latest version.

I haven't had much chance to look at your code. The XSLT code is Altova-generated code so it's pretty verbose and not easy to follow the logic. My immediate reaction would be that 30 seconds seems very slow. I tried it with SaxonJ from the command line (I don't have a Windows machine handy) and it took 977ms. So there's clearly something going wrong somewhere.

What timing do you get running a single transformation from the command line?

I looked at your C# driver code and that looks fine.

just a single tranformation takes between 3-4 seconds

OK, that's consistent with my Java figure, because the .NET product is often around a factor of 3-4 slower than Java (for reasons we have never fully understood). If it's taking 3 seconds in a single thread and 30 seconds with 10 threads that suggests that there's no real multithreading going on at all. What does the Console log look like?

We run the requests in Parallel.

Parallel.For(0, 10, _ =>
{
    saxon.RunTransformerEntEd();
});

From the console output, it looks like it is not running in parallel. Looks like it is executing serially. I wonder if there is a locking in Saxon API prevents these calls to execute in parallel.

Console output: Compile time: 1193 Compile time: 1193 Compile time: 1193 Compile time: 1193 Compile time: 1185 Compile time: 1193 Compile time: 1193 Compile time: 1193 Compile time: 1193 Transform time: 35974 Transform time: 38518 Transform time: 38747 Transform time: 38758 Transform time: 38772 Transform time: 38798 Transform time: 38864 Transform time: 38865 Transform time: 38872 Transform time: 38878

Well, there's clearly some parallelism there, because it's running all the compiles, then all the transformations. But it does look rather like some lock is holding things back.

Is there anything we can do the increase the performance?

I haven't identified anything yet, but will work on it.

I've now run your test case against SaxonCS 13.0 (under development) on .NET 6.0 on Mac. I know that's not your target environment but it's easiest for me to investigate on my current platform rather than setting up something new.

I'm getting the timings

Compile time: 395
Compile time: 395
Compile time: 395
Compile time: 395
Compile time: 396
Compile time: 394
Compile time: 395
Compile time: 395
Compile time: 395
Compile time: 396
Transform time: 19375
Transform time: 19386
Transform time: 19400
Transform time: 19405
Transform time: 19414
Transform time: 19428
Transform time: 19429
Transform time: 19432
Transform time: 19435
Transform time: 19475
Completed...

If I change the number of threads to 4, I get

Compile time: 277
Compile time: 277
Compile time: 277
Compile time: 277
Transform time: 5643
Transform time: 5643
Transform time: 5643
Transform time: 5645

and with 1 thread I get

Compile time: 236
Transform time: 2403

I added a trace at the point where the transform starts, and I'm getting

Compile time for : 4 = 289
Compile time for : 1 = 289
Compile time for : 0 = 289
Compile time for : 3 = 289
Compile time for : 2 = 289
Start transform : 0
Start transform : 2
Start transform : 3
Start transform : 4
Start transform : 1
Transform time for : 2 = 7776
Transform time for : 3 = 7776
Transform time for : 4 = 7776
Transform time for : 1 = 7777
Transform time for : 0 = 7779

So it's pretty clear that they are running concurrently rather than sequentially.

I'll come back to this after the weekend.

Sorry, I wasn't changing the number of threads, I was changing the total number of transformations performed. What we're seeing essentially is that running them in parallel takes more or less the same total amount of time as running them sequentially.

if I run them serially, I see better results. However, this is not ideal.

Compile time: 67
Transform time: 3535
Compile time: 61
Transform time: 1964
Compile time: 51
Transform time: 1980
Compile time: 49
Transform time: 2436
Compile time: 33
Transform time: 2268
Compile time: 53
Transform time: 1890
Compile time: 46
Transform time: 2434
Compile time: 58
Transform time: 1909
Compile time: 51
Transform time: 1851
Compile time: 51
Transform time: 1842
Compile time: 47
Transform time: 1949
Compile time: 47
Transform time: 2310
Compile time: 49
Transform time: 2451
Compile time: 48
Transform time: 1974
Compile time: 45

What are you trying to achieve? Are you trying to optimize throughput or latency?

Both. We get 100s of requests per sec that we need to keep the xslt compile/transform as efficient as possible.

Well, it's going to be a trade-off, isn't it?

I've tried running 20 transformations with different number of threads, and the figures (threads / av. transform time / total time / throughput):

2 / 2998 /31943 /  0.62
3 / 4218 / 31025 / 0.65
4 / 5347 / 28704 / 0.69
5 / 7763 / 32240 / 0.62
6 / 9948 / 34707 / 0.57

10 / 20636 / 42734 / 0.46

I'm running this on a machine with 4 cores, so as you would expect for a CPU-dominated workload, the throughput is highest with a concurrency of 4, but the latency is smaller with a lower number of threads. There is no benefit in having more threads than you have processors available, This is all very much what one would expect.

I don't know how many threads Parallel.For allocates by default. I'm sure there are more sophisticated ways of tuning the scheduling of parallel workloads on .NET, but the basic pattern is likely to remain the same.

I thought it might be interesting to look at the actual XSLT code, so I profiled it (on Java) using -TP:profile.html All this reveals is that nearly all the time is spent in the xsl:template match="/" root template, but that's hardly surprising since that accounts for 99% of the stylesheet. With monolithic code like this, the profiling isn't much use.

It's 10,000 lines of very turgid generated code, so no real prospect of examining it by hand.

A Java profile reveals nothing untoward.

I now there have been problems with performance with very large templates/functions like this in the past, but IIRC they were compile-time problems rather than run-time, and the compile time for the stylesheet looks OK.

OK, let's stand back.

You say you need to handle hundreds of requests a second, and each request is taking two or three seconds of CPU to execute, when run in a single thread. That's only going to be viable if you distribute the workload across hundreds of CPUs.

I strongly suspect the XSLT code could be made to go much faster. On the face of it it looks like a fairly uncomplicated transformation, Because it is generated code, it's doing a lot of unnecessary work, for example there are far more local variables than are needed. Saxon gives up on trying to optimize a template/function as big as this, because the cost of optimization increases exponentially with the size of the function or template. I suspect that means it's not even going to do trivial optimizations like replacing the expression xs:double(xs:decimal('2')) (line 11742) by the constant 2.0e0. But rewriting it by hand is a fairly daunting prospect (especially as you suggest this is just one of hundreds of different stylesheets), However, I would think it's feasible for a hand-written stylesheet to do the job in under 100ms.

Perhaps, because it's generated code, it's regular enough that you could consider attempting some automated rewrites? I've no idea what kind of improvement you might expect but it might be worth experimenting.

The only alternative seems to be to buy lots of hardware (or cloud capacity) and put in some load-sharing infrastructure to distribute the workload.

Running on Java rather than .NET might give you a 3-4 times improvement, which would be a good start.

Running on Java rather than .NET might give you a 3-4 times improvement, which would be a good start.

In fact on Java it's running in about 117ms, which is a much more reasonable time for this kind of transformation.