Saxon

Initial MHK response:

Thanks for reporting it and for providing a simple repro.

The code has indeed changed here since 9.4; it used to read

       for (int i=used; i>index; i--) {

           array[i+1] = array[i-1];

       }

and this was changed to

       System.arraycopy(array, index, array, index + 1 + 1, used - index);

by an automatic IntelliJ refactoring. I haven't yet worked out why this rewrite is incorrect. Debugging is complicated by the fact that it's a Heisenbug; it doesn't occur in the debugger because FastStringBuffer.toString() calls condense() which changes the value.

The code appears to be unnecessary in the case where (as here) the "insertion" is at the end of the string,and the simplest fix may be to avoid doing it in that situation.

Michael Kay

Switching the code back to the 9.4 version does not cure the problem: the rewritten code does appear to be equivalent.

Running under 9.4, the Unicode decomposition is not generating any astral characters. 9.4 generates 55296 where 9.5 generates 65909. So it seems that as well as a bug in FastStringBuffer.insertWideChar() (present in both releases) we might have a new bug in Unicode denormalization. This could be related to bug 1842.

No, Unicode normalization is generating the same characters. What has changed is that in 9.4 it used a StringBuffer, in 9.5 it is using a FastStringBuffer. So the incorrect code in FastStringBuffer was always there, it just wasn't being used during Unicode normalization. So this explains the regression.

Changed the fix - it addressed the symptoms but not the cause. The cause is that too many chars are being shifted right to make room for the new char, which is caused by incrementing "used" before doing the shift. So the problem could also occur when inserting into the middle of the array. Changed the fix to increment "used" after doing the arraycopy, as done by the insert() method.