Commandline -repeat:N not working properly
I'm seeing weird behaviour when trying to benchmark Saxon 22.214.171.124 ee using the following command line:
java -server -Xmx2g -jar saxon96ee.jar -repeat:30 -o:result.xml test.xml test.xsl
When I use -repeat:30 I get:
* Average execution time over last 15 runs: 801.00544msms
Then again when I use -repeat:25 I get:
* Average execution time over last 13 runs: 829.111915msms
Again with -repeat:15 I get:
* Average execution time over last 8 runs: 817.645476msms
Also notice that the output has ms twice. When I run this with latest 126.96.36.199 version with:
java -server -Xmx2g -jar saxon95ee.jar -repeat:30 -o:result.xml test.xml test.xsl
* Average execution time over 30 runs: 765ms
My Java version is:
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
Running on Windows 7 x64 SP1
#1 Updated by Michael Kay over 5 years ago
- Assignee set to Michael Kay
- Found in version changed from 188.8.131.52 to 9.6
The repeated "ms" in the message is a simple bug.
The fact that if you request 30 repetitions, we report the average over the last 15 is by design. The purpose of using repeated runs is to discount the effect of Java VM warmup time, and we found that averaging over all the runs didn't do this very effectively, while measuring over the last half of the runs gives a much more reliable figure.
#4 Updated by Aleksander Karro over 5 years ago
Thanks for clarifying that. It makes sense now, although it's a bit confusing that the previous version would show the average over 30 last runs. I couldn't find anything in the change log about it under the Command Line interface section. Maybe it's worthwhile adding it as a note to the docs?
Please register to edit this issue