Project

Profile

Help

Bug #4371

Saxon/C crashing in browser when run via PHP

Added by O'Neil Delpratt almost 2 years ago. Updated 22 days ago.

Status:
In Progress
Priority:
High
Category:
PHP API
Start date:
2019-11-01
Due date:
% Done:

0%

Estimated time:
Found in version:
1.2.1
Fixed in version:

Description

Reported by user:

Saxon-PE/C is crashing when run through the browser. It runs fine when PHP is run from the command-line.

I have managed to reproduce the issue.

php7_saxon.cpp (181 KB) php7_saxon.cpp O'Neil Delpratt, 2020-05-12 15:49

History

#1 Updated by O'Neil Delpratt almost 2 years ago

Update:

Still investigating this issue, but what I am noticing is that we are getting intermitted hanging in the browser. Sometimes the script run and sometimes it does not.

#2 Updated by O'Neil Delpratt almost 2 years ago

  • Status changed from New to In Progress

Update:

I have reinstalled Saxon/C PHP 1.1.2 version which runs as expected in the browser.

I also tried rolling back the PHP extension code from 1.1.2 release for Saxon/C 1.2. After making some minor changes to get the code compiling, I found the PHP script also runs fine in the browser.

Therefore I have narrowed down that the bug has been introduced in the PHP extension code between 1.1.2 and 1.2.1. This rules out the Excelsior Jet version change in the equation.

#3 Updated by O'Neil Delpratt almost 2 years ago

The fix made in bug issue #2055 is the cause of the hanging issue in the browser.

#4 Updated by Rein Baarsma almost 2 years ago

This bug is quite critical -- the extension is completely useless from 1.2.0 onwards for us. We had hoped to fix the memory problems by upgrading, but needed to downgrade directly after.

I've confirmed this bug is happening in the browser and functional tests, but not in unit tests and command line. It hangs consistently on every try.

We hope this issue will be fixed soon.

If it were up to me, I would change the severity to the highest option and "Found in version" to 1.2.0+

#5 Updated by O'Neil Delpratt almost 2 years ago

Hi, after much investigation I have got to the bottom of the problem but not yet yet worked out a solution.

We have some incompatibility with the cross compile tool we use (i.e. Excelsior JET) and PHP. The JET JNI environment is created as a static variable at the start of the program and released at the end.

The threads are created by the JET Runtime, e.g. GC thread, weak reference handler thread, finalizers thread, etc. Initially I thought best to create the Jet runtime for each request, but this did not work. So I tried to keep the Jet runtime alive for the lifetime of the extension, which could be reused between requests.

In the PHP extension we create the JET runtime (which has several threads) in the PHP_MINIT_FUNCTION(saxonc)

To make sure the clean up of the JET threads happens we have added a release method in the MSHUTDOWN_FUNCTION of the PHP extension.

The problem is when we run the PHP extension in the browser it just hangs. If we remove the code to manage the JET threads in the PHP_MINIT_FUNCTION the PHP scripts run, but does not kill the JET threads. The only way to kill the threads is a restart to the server.

In effect we need to reuse the JET threads between the PHP requests or find some way of killing them off.

I will revisit this problem again. We have one user resort to the shell_exec in the PHP script to run the Saxon/C extension.

#6 Updated by Michael Kay almost 2 years ago

Rein, thank you, we do realise this bug is a show-stopper for many users. It's often the case, however, that bugs like this can be the hardest ones to fix, and if it takes time, that's because of the difficulty of the problem, not because of lack of urgency.

#7 Updated by O'Neil Delpratt over 1 year ago

  • File php7_saxon.cpp added

Added patch for the hanging issue to the file php7_saxon.cpp

#8 Updated by O'Neil Delpratt over 1 year ago

  • File deleted (php7_saxon.cpp)

#10 Updated by s├ębastien bocahu about 1 year ago

Unfortunately we are experiencing the same bug. The provided patch seems to fix the hanging issue but I guess that it also brings back the memory leak / high memory consumption that 1.1.2 suffers from.

By the way, w/ patched v1.2.1, our code is now throwing errors that seem to be related to https://saxonica.plan.io/issues/4395

#11 Updated by O'Neil Delpratt about 1 year ago

Hi,

Unfortunately we have still not managed to get to the bottom of this bug issue and is still our priority to fix i the next release. I know of users having better success with the memory issue using PHP-FPM, but I have not tried it myself.

#12 Updated by Frank Arensmeier about 1 year ago

Hi!

I thought I share a function with those of you suffering from this bug. We use the Saxon module on several servers and we have seen that once in a while, servers stop responding because of the large number of zombie processes. Our temporary solution for now is to kill those child processes when execution has ended. Note that this has been tested for PHP-FPM 7.2 on Ubuntu LTS 18.

Kill switch function:

<?php
/**
 * @param int $minutes - minutes to wait until process is killed
 * @return bool
 */
function killCurrentThread(int $minutes = 0) : bool
{
    if (!stristr(php_sapi_name(), 'fpm-')) {
        return false;
    }

    $signal = SIGTERM;
    $pid = posix_getpid();

    // get a list of child processes for the current running process
    $cmd = "pstree -p {$pid} | grep -o '([0-9]\+)' | grep -o '[0-9]\+'";
    exec($cmd, $processlist);

    $processlist = array_map('intval', $processlist);
    $processlist = array_filter($processlist, function ($cid) use ($pid) {
        return $cid !== $pid;
    });

    $processlist = array_filter($processlist, function ($cid) {
        return posix_kill($cid, 0);
    });

    if (! count($processlist)) {
        return false;
    }

    $firstchild = array_shift($processlist);

    $prefix = sprintf('rmproc.%s.%s.', $firstchild, uniqid(true));
    $tmpfile = tempnam(sys_get_temp_dir(), $prefix);
    chmod($tmpfile, 0755);

    $script = <<<BASH
#!/bin/sh

PID={$firstchild}
sleep 5 && kill -s {$signal} \$PID  > /dev/null 2>&1
rm \$0
BASH;

    file_put_contents($tmpfile, $script);
    $cmd = sprintf(
        '/usr/bin/at now +%d min -f %s 2>&1 | /usr/bin/logger -i &',
        $minutes,
        $tmpfile
    );

    exec($cmd);
    return true;
}

// invoke the kill function during shutdown
if (function_exists("killCurrentThread")) {
    register_shutdown_function('killCurrentThread');
}

// do your magic with Saxon...

How this function works. It is important to invoke the function during shutdown (otherwise, your script will probably terminated before execution ends). The function first finds all child processes for the currently running process. If one of those zombified child processes is killed, all other zombies will die too. Next, the function saves a very short bash script to /tmp. And finally, script execution is scheduled with at - http://manpages.ubuntu.com/manpages/trusty/man1/at.1posix.html

Maybe this helps some of you. Thank you for your hard work! Looking forward for a solution.

Regards, Frank

#13 Updated by O'Neil Delpratt about 1 year ago

Thank you Frank for posting this workaround.

#14 Updated by Frank Arensmeier about 1 month ago

I realise that this is a hard nut to crack, maybe also because of the death of Excelsior JET? Any news?

#15 Updated by O'Neil Delpratt 22 days ago

Indeed this has been a long standing issue. So no update as yet.

Please register to edit this issue

Also available in: Atom PDF