Bug #2165: Commandline -repeat:N not working properly - Saxon - Saxonica Developer Community

Actions

Send by e-mail Copy link

Bug #2165

closed

Commandline -repeat:N not working properly

Added by Aleksander Karro over 9 years ago. Updated over 8 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Michael Kay

Category:

Command Line

Sprint/Milestone:

9.6.0.2

Start date:

2014-10-07

Due date:

% Done:

100%

Estimated time:

Legacy ID:

Applies to branch:

9.6

Fix Committed on Branch:

9.6

Fixed in Maintenance Release:

9.6.0.2

Platforms:

Description

I'm seeing weird behaviour when trying to benchmark Saxon 9.6.0.1 ee using the following command line:

java -server -Xmx2g -jar saxon96ee.jar -repeat:30 -o:result.xml test.xml test.xsl

When I use -repeat:30 I get:

* Average execution time over last 15 runs: 801.00544msms

Then again when I use -repeat:25 I get:

* Average execution time over last 13 runs: 829.111915msms

Again with -repeat:15 I get:

* Average execution time over last 8 runs: 817.645476msms

Also notice that the output has ms twice. When I run this with latest 9.5.1.6 version with:

java -server -Xmx2g -jar saxon95ee.jar -repeat:30 -o:result.xml test.xml test.xsl

I get:

* Average execution time over 30 runs: 765ms

My Java version is:

java version "1.8.0_20"

Java(TM) SE Runtime Environment (build 1.8.0_20-b26)

Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)

Running on Windows 7 x64 SP1

Actions

Copy link

Updated by Michael Kay over 9 years ago

Assignee set to Michael Kay
Found in version changed from 9.6.0.1 to 9.6

The repeated "ms" in the message is a simple bug.

The fact that if you request 30 repetitions, we report the average over the last 15 is by design. The purpose of using repeated runs is to discount the effect of Java VM warmup time, and we found that averaging over all the runs didn't do this very effectively, while measuring over the last half of the runs gives a much more reliable figure.

Actions

Copy link

Updated by Michael Kay over 9 years ago

Status changed from New to Resolved

Actions

Copy link

Updated by Michael Kay over 9 years ago

A patch for the badly formatted message has been committed and tested on the 9.6 and 9.7 branches.

Actions

Copy link

Updated by Aleksander Karro over 9 years ago

Thanks for clarifying that. It makes sense now, although it's a bit confusing that the previous version would show the average over 30 last runs. I couldn't find anything in the change log about it under the Command Line interface section. Maybe it's worthwhile adding it as a note to the docs?

Actions

Copy link