Project

Profile

Help

Bug #5552

closed

OnClose callback method call before generate files

Added by Vishnu Guatam over 2 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Internals
Sprint/Milestone:
-
Start date:
2022-05-31
Due date:
% Done:

100%

Estimated time:
Legacy ID:
Applies to branch:
11
Fix Committed on Branch:
11, trunk
Fixed in Maintenance Release:
Platforms:
.NET, Java

Description

I'm new to the SaxonCS library and I've noticed a bug where my xsl file generates multiple files at a time (approximately 10 files) and I want to do more processing after they're all generated, but the library calls the OnClose callback method too soon whenever a file is not created, and the callback method throws the error "file is used by another process." I've attached sample xml and xsl in a sample project.

Actions #1

Updated by O'Neil Delpratt over 2 years ago

  • Project changed from SaxonC to Saxon
  • Assignee set to Michael Kay
  • Found in version deleted (SaxonCS 11.3)
  • Applies to branch 11 added
Actions #2

Updated by Michael Kay over 2 years ago

  • File deleted (XsltConversion_31May.zip)
Actions #3

Updated by Michael Kay over 2 years ago

I have deleted the attachment because it contained a Saxon license file. Please never post your license file in a public place.

Actions #4

Updated by Vishnu Guatam over 2 years ago

When can I expect this to be resolved?

Actions #5

Updated by Michael Kay over 2 years ago

  • Tracker changed from Bug to Support
  • Status changed from New to In Progress

The OnClose() action on the primary output of the transformation will be called as soon as the primary output is complete. Because xsl:result-document operates asynchronously in Saxon-EE, at that point the transformer will still typically be writing to one or more secondary result documents (which accounts for the failure). It looks as if you want the "afterUploadAction" to run when the transformation is completely finished, in which case you can call it directly on exit from xsltTransformer.Run, rather than in an OnClose() callback.

In fact the OnClose() callback is designed primarily for use with secondary result documents. The Xslt30Transformer can nominate a ResultDocumentHandler to be invoked when xsl:result-document is called to produce a secondary result document; this can create an IDestination to receive that result document, and it can specify an OnClose() action to be invoked when the secondary result document has been completely written.

If you want your application to process the contents of secondary result documents after the transformation is complete, it may well be best not to write them to filestore at all. You can use the ResultDocumentHandler to write them somewhere else, either as an XdmDestination (an in-memory result tree) or perhaps as serialized XML in a StringWriter.

Actions #6

Updated by Vishnu Guatam over 2 years ago

I also tried xsltTransformer.Run (without onClose) than call next action but getting same issue, I am not able to find afterUploadAction method in library. I can find Close, onClose but not afterUploadAction.

Actions #7

Updated by Michael Kay over 2 years ago

afterUploadAction is the name of a method in your code that you posted, it's the method that you invoke in your onClose() callback.

As an experiment, it might be worth setting Feature.ALLOW_MULTITHREADING to false (using Processor.SetProperty()), to ensure that the secondary result documents are written in the main thread. I would be interested if that affects things.

It looks like you're running on Windows, which I've always found to be very sensitive about making sure that files are properly closed before attempting to read them. It's certainly possible that there's a problem in this area, but we need to eliminate other more obvious things first.

Actions #9

Updated by Vishnu Guatam over 2 years ago

I have tried several things also above commented facing same issue but waiting for resolution so is there any possible solution or we have to hit and try?

Actions #10

Updated by Vishnu Guatam over 2 years ago

For your information in .NET version afterUploadAction is not available.

Actions #11

Updated by Martin Honnen over 2 years ago

I have put together an example VS 2022 project which indeed seems to reproduce the problem of the SO question when run on Windows:


Error on line 15 column 4 of sheet2.xsl:
  FODC0002  Exception thrown by URIResolver resolving `result1.xml` against
  `file:///C:/Users/marti/source/repos/SaxonCSResultNotClosedTest1/bin/Debug/net6.0/sheet2.xsl'
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode
  mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode,
  FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode,
  FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode
  mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String
  path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions
  options, Int64 preallocationSize)
   at System.IO.File.OpenRead(String path)
   at Org.XmlResolver.Utils.UriUtils._getFileStream(String uri)
   at Org.XmlResolver.Utils.UriUtils.GetStream(String uri, Assembly asm)
   at Org.XmlResolver.Utils.UriUtils.GetStream(Uri uri)
   at Saxon.Callbacks.DirectResourceResolver.resolve(JJ_ResourceRequest request)
   at Saxon.Hej.lib.JJ_ResourceRequest.resolve(JJ_ResourceResolver[] resolvers)
   at Saxon.Hej.functions.DocumentFn.resolveURI(String href, String baseURI, String
  documentKey, XPathContext context)
: The process cannot access the file
  'C:\Users\marti\source\repos\SaxonCSResultNotClosedTest1\bin\Debug\net6.0\result1.xml'
  because it is being used by another process.

Saxon.Api.SaxonApiException
  HResult=0x80131500
  Message=Exception thrown by URIResolver resolving `result1.xml` against `file:///C:/Users/marti/source/repos/SaxonCSResultNotClosedTest1/bin/Debug/net6.0/sheet2.xsl': The process cannot access the file 'C:\Users\marti\source\repos\SaxonCSResultNotClosedTest1\bin\Debug\net6.0\result1.xml' because it is being used by another process.
  Source=SaxonCS
  StackTrace:
   at Saxon.Hej.s9api.XsltTransformer.transform()
   at Saxon.Api.XsltTransformer.Run(IDestination destination)
   at Program.<Main>$(String[] args) in C:\Users\marti\source\repos\SaxonCSResultNotClosedTest1\Program.cs:line 42

However, when run under Linux (WSL, Windows Linux Subsystem), it runs fine.

Actions #13

Updated by Martin Honnen over 2 years ago

Branch https://github.com/martin-honnen/SaxonCSResultNotClosedTest1/blob/ResultDocumentHandler/Program.cs does

transformer.ResultDocumentHandler = (href, baseUri) => { 
    var serializer = processor.NewSerializer();
    var fs = File.Open(new Uri(baseUri, href).LocalPath, FileMode.Create, FileAccess.Write);
    serializer.OutputStream = fs;
    serializer.OnClose(() => { fs.Close(); });
    return serializer;
};

which seems to work around the problem

Actions #14

Updated by Michael Kay over 2 years ago

The method Emitter.close() calls writer.close() only if the variable mustClose is true. In Saxon 11, it is never set to true. This applies to the Java code as much as to the C# code. (But failure to close a file tends to cause problems only on Windows).

The general principle is that Saxon should close a stream or writer if (and only if) Saxon created the stream or writer, typically because it was supplied with a file or URI. It shouldn't close a stream or writer that was supplied by the user.

The design changed in 11 so the Emitter is always supplied with a UnicodeWriter; the logic for constructing this (and therefore the knowledge of whether it needs to be closed) are in the new class ExpandedStreamResult.

Actions #15

Updated by Michael Kay over 2 years ago

The next part of the problem is specific to SaxonCS: the C# emulation of class StreamResult differs from the original in that the constructor supplying a File creates a FileOutputStream immediately, losing the information that the stream was system-constructed rather than user-constructed, and therefore needs to be closed by the system after use.

Actions #16

Updated by Michael Kay over 2 years ago

  • Tracker changed from Support to Bug
  • Category set to Internals
  • Status changed from In Progress to Resolved
  • Fix Committed on Branch 11, trunk added
  • Platforms Java added

I believe this is now fixed, though this can't be confirmed until we have run the tests on all platforms.

Actions #17

Updated by Debbie Lockett over 2 years ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100
  • Fixed in Maintenance Release 11.4 added

Bug fix applied in the Saxon 11.4 maintenance release.

Actions #18

Updated by Michael Kay about 2 years ago

  • Status changed from Closed to In Progress
  • Priority changed from High to Normal

On the 12.x branch (but not 11.x), while working on this bug, I changed the JsonEmitter and AdaptiveEmitter to have a mustClose flag, and to use it in the same way as the Emitter -- specifically, the output writer is not closed at close() time unless the mustClose flag is set.

The is causing the failure of a JUnit test, and indeed the failure of a command-line transformation using the JSON output method: the output file is empty because it is not closed.

This is because the ExpandedStreamResult created by SerializerFactory.getReceiver() has mustCloseAfterUse==false.

If we compare the path with the XML Output method, the difference is that XMLEmitter.close() calls writer.flush() even when the mustClose flag is not set. And sure enough, we can get JSON output by ensuring that writer.flush() is called unconditionally.

The call on flush() appears sufficient. With either output method, we are calling close() if the output destination is a file, but not if it is a stream/writer - for that case, closing is the caller's responsibility.

Actions #19

Updated by Michael Kay about 2 years ago

  • Status changed from In Progress to Resolved
Actions #20

Updated by O'Neil Delpratt almost 2 years ago

  • Status changed from Resolved to Closed
  • Fixed in Maintenance Release 12.0 added
Actions #21

Updated by Michael Kay over 1 year ago

The patch described in comment 18 has now been applied to the 11.x branch, to go out with 11.6. See also bug #5886.

Please register to edit this issue

Also available in: Atom PDF