Project

Profile

Help

Bug #5373

closed

Python multithreading code crashes

Added by O'Neil Delpratt over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Category:
Saxon-C Internals
Start date:
2022-03-07
Due date:
% Done:

100%

Estimated time:
Applies to branch:
Fix Committed on Branch:
Fixed in Maintenance Release:
Found in version:
11.2
Fixed in version:
11.3
SaxonC Languages:
SaxonC Platforms:
SaxonC Architecture:

Description

Reported by user in the following forum post: https://saxonica.plan.io/boards/4/topics/8523?page=2

The following SaxonC python example crashes:

import threading
import time

from saxonc import *

exitFlag = 0

class myThread (threading.Thread):
   def __init__(self, threadID, name, counter, xquery_processor):
      threading.Thread.__init__(self)
      self.threadID = threadID
      self.name = name
      self.counter = counter
      self.xquery_processor = xquery_processor
   def run(self):
      print ("Starting " + self.name)
      example(self.name, self.counter, 5, self.xquery_processor)
      print ("Exiting " + self.name)

def example(threadName, delay, counter, xquery_processor):
   while counter:
      if exitFlag:
         threadName.exit()
      time.sleep(delay)
      print("%s : %s" % (threadName, xquery_processor.run_query_to_string(query_text = '"Hello from SaxonC: " || current-dateTime()')))
      counter -= 1


saxon_proc = PySaxonProcessor(license = False)

# Create new threads
thread1 = myThread(1, "Thread-1", 1, saxon_proc.new_xquery_processor())
thread2 = myThread(2, "Thread-2", 2, saxon_proc.new_xquery_processor())

# Start new Threads
thread1.start()
thread2.start()
thread1.join()
thread2.join()
print ("Exiting Main Thread")

saxon_proc.release()

Crashes in linux as follows:

Starting Thread-1
Starting Thread-2

JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: system exception at 0x00007f71dca62d49
Please, contact the vendor of the application.
Core dump will be piped to "/usr/share/apport/apport %p %s %c %d %P %E"
Extra information about error is saved in the "jet_err_167047.txt" file.

Aborted (core dumped)

Related issues

Related to SaxonC - Bug #5050: java exception in multithreaded applicationClosedO'Neil Delpratt2021-07-29

Actions
Has duplicate SaxonC - Bug #4428: Multi-threading support of Python bindings Saxon-CClosedO'Neil Delpratt2020-01-14

Actions
Has duplicate SaxonC - Support #4942: How to prevent Saxon/C Python bindings from trying to start a new java VM when another is active? (JNI_CreateJavaVM() failed with result: -5)ClosedO'Neil Delpratt2021-03-18

Actions
Actions #1

Updated by O'Neil Delpratt over 2 years ago

  • Status changed from New to In Progress

I think I have finally tracked down the cause of this crash.

The bug is in the C++ code for JNI. The problem is we are not attaching the JNIenv variable in the new threads we create. Hence the java instances cannot use the JNIenv.

See the following article to a background on JNI in multithreading apps: https://stackoverflow.com/questions/31134009/how-to-keep-calling-java-instance-jobject-for-new-threads-created-by-c-in-jni

I experimented with this by attaching the JNIenv using the AttachCurrentThread method and now I am getting some expected output from the python script:

Starting Thread-1
Starting Thread-2
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:16.241Z
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:17.247Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:17.248Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:18.25Z
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:19.251Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:19.258Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T00:05:20.26Z
Exiting Thread-1


JET RUNTIME HAS DETECTED UNRECOVERABLE ERROR: runtime error
Thread 30AC2 ["Thread-1"] is terminated without notifying the JVM. Probably, "DetachCurrentThread" function was not called
Please, contact the vendor of the application.
Core dump will be piped to "/usr/share/apport/apport %p %s %c %d %P %E"
Extra information about error is saved in the "jet_err_199341.txt" file.

I am now getting a DetachCurrentThread crash probably I am not cleaning things after the 1st thread has finished.

Actions #2

Updated by O'Neil Delpratt over 2 years ago

  • Status changed from In Progress to Resolved

Bug fix applied to the C++ code where we check if the JNIEnv is available. If not we try to attach the JVM to the current thread.

The following method have been added to the SaxonProcessor:

    static void attachCurrentThread();

    static void detachCurrentThread();

Similarly in the PHP and, Python API.

See modified sample below:

import threading
import time

from saxonc import *

exitFlag = 0

class myThread (threading.Thread):
   def __init__(self, threadID, name, counter, proc, xquery_processor):
      threading.Thread.__init__(self)
      self.threadID = threadID
      self.name = name
      self.counter = counter
      self.xquery_processor = xquery_processor
      self.proc = proc
   def run(self):
      print ("Starting " + self.name)
      example(self.name, self.counter, 5, self.proc, self.xquery_processor)
      print ("Exiting " + self.name)

def example(threadName, delay, counter, proc, xquery_processor):
         while counter>0:
            if exitFlag:
               threadName.exit()
            time.sleep(delay)

            result = xquery_processor.run_query_to_string(query_text = '"Hello from SaxonC: " || current-dateTime()')
            
            if result is None:
               threadName.exit()

            print("%s : %s" % (threadName, result))
            counter -= 1
            proc.detach_current_thread

def example2(threadName, delay, counter, xquery_processor):
   while counter>0:
      if exitFlag:
         return
      time.sleep(delay)
      result = xquery_processor.run_query_to_string(query_text = '"Hello from SaxonC: " || current-dateTime()')
      if result is None:
         return
      print("%s : %s" % (threadName, result))
      counter -= 1


saxon_proc = PySaxonProcessor(license = False)

# Create new threads
thread1 = myThread(1, "Thread-1", 1, saxon_proc, saxon_proc.new_xquery_processor())
thread2 = myThread(2, "Thread-2", 2, saxon_proc, saxon_proc.new_xquery_processor())

# Start new Threads
thread1.start()
thread2.start()
thread1.join()
thread2.join()
print ("Exiting Main Thread")

Output:

Starting Thread-1
Starting Thread-2
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:30.021Z
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:31.026Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:31.027Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:32.029Z
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:33.03Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:33.036Z
Thread-1 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:34.039Z
Exiting Thread-1
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:35.033Z
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:37.043Z
Thread-2 : <?xml version="1.0" encoding="UTF-8"?>Hello from SaxonC: 2022-03-10T18:35:39.046Z
Exiting Thread-2
Exiting Main Thread
Actions #3

Updated by O'Neil Delpratt over 2 years ago

Notice in the example() method when we use threads we need to call the 'proc.detach_current_thread' property

Actions #4

Updated by Martin Honnen over 2 years ago

Sounds good, the example has the Python code proc.detach_current_thread without the () for calling, is that intended and working?

Actions #5

Updated by O'Neil Delpratt over 2 years ago

  • % Done changed from 0 to 100

Yes it is intended and working in the example code. It is a class property.

Available in the next maintenance release.

Actions #6

Updated by O'Neil Delpratt over 2 years ago

  • Related to Bug #5050: java exception in multithreaded application added
Actions #7

Updated by O'Neil Delpratt over 2 years ago

  • Has duplicate Bug #4428: Multi-threading support of Python bindings Saxon-C added
Actions #8

Updated by O'Neil Delpratt over 2 years ago

  • Has duplicate Support #4942: How to prevent Saxon/C Python bindings from trying to start a new java VM when another is active? (JNI_CreateJavaVM() failed with result: -5) added
Actions #9

Updated by O'Neil Delpratt over 2 years ago

  • Status changed from Resolved to Closed
  • Fixed in version set to 11.3

Bug fix applied in the SaxonC 11.3 maintenance release.

Please register to edit this issue

Also available in: Atom PDF