Project

Profile

Help

Support #4942

closed

How to prevent Saxon/C Python bindings from trying to start a new java VM when another is active? (JNI_CreateJavaVM() failed with result: -5)

Added by Kor Foo over 1 year ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Category:
Python
Start date:
2021-03-18
Due date:
% Done:

0%

Estimated time:
Found in version:
1.2.1
Platforms:

Description

I am building a flask api that allows users to pass an xml and a transformation that returns the xml on which the transformation is performed using Saxon/C's python API (https://www.saxonica.com/saxon-c/doc/html/saxonc.html).

The incoming endpoint looks like this (removed logging and irrelevant info):

@app.route("/v1/transform/", methods=["POST"])
def transform():
    xml = request.data
    transformation = request.args.get("transformation")

    result = transform_xml(xml, transformation)

    return result

The transform function looks like this:

    def transform_xml(xml: bytes, transformation: str) -> str:
        with saxonc.PySaxonProcessor(license=False) as proc:
            base_dir = os.getcwd()
            xslt_path = os.path.join(base_dir, "resources", transformation, "main.xslt")

            xslt_proc = proc.new_xslt30_processor()

            node = proc.parse_xml(xml_text=xml.decode("utf-8"))

            result = xslt_proc.transform_to_string(stylesheet_file=xslt_path, xdm_node=node)
            
            return result

The xslt's are locally available and a user should choose one of the available ones by passing the corresponding transformation name.

Now the problem is, this works (fast) for the first incoming call, but the second one crashes:

JNI_CreateJavaVM() failed with result: -5
DAMN ! worker 1 (pid: 517095) died :( trying respawn ...

What does work is changing the transform_xml function like this:

        proc = saxonc.PySaxonProcessor(license=False)

        xslt_path = self.__get_path_to_xslt(transformation)

        xslt_proc = proc.new_xslt30_processor()

        node = proc.parse_xml(xml_text=xml.decode("utf-8"))

        result = xslt_proc.transform_to_string(stylesheet_file=xslt_path, xdm_node=node)
        
        return result

But this leads to the resources never getting released and over time (1k+ requests) this starts to fill up the memory. I have tried fiddling with the JET garbage collector settings (-Djet.gc.no.swap -Djet.gc.ratio -Djet.gc.heaplimit -Djet.gc.defrag.holes.threshold), but this doesn't help to reduce the memory footprint.

It seems like Saxon is trying to create a new VM while the old one is going down.

I found this thread from 2016: https://saxonica.plan.io/boards/4/topics/6399 but this didn't clear it up for me. I looked at the github for the pysaxon repo, but I have found no answer to this problem.


Files

try_saxon2.zip (1.52 KB) try_saxon2.zip Daniel Haley, 2021-03-30 05:24

Related issues

Is duplicate of SaxonC - Bug #5373: Python multithreading code crashesClosedO'Neil Delpratt2022-03-07

Actions

Please register to edit this issue

Also available in: Atom PDF