Project

Profile

Help

Bug #2055

Apache allocates a lot of memory without releasing it

Added by O'Neil Delpratt over 6 years ago. Updated 9 months ago.

Status:
In Progress
Priority:
Normal
Category:
PHP API
Start date:
2014-04-17
Due date:
% Done:

100%

Estimated time:
Found in version:
1.1.2
Fixed in version:
1.2.0

Description

Issue reported by Petr Žák:

Apache allocates a lot of memory, which is not released after php script execution, but I hope that this is caused by wrong close() method implementation.

xsltExamples.php (2.01 KB) xsltExamples.php O'Neil Delpratt, 2014-04-17 12:36
screendump.jpeg (151 KB) screendump.jpeg Frank Arensmeier, 2018-05-16 14:42

History

#1 Updated by O'Neil Delpratt over 6 years ago

  • Category changed from PHP API to Saxon-C Internals
  • Status changed from New to In Progress

Update on memory usage issues:

I have been doing some extensive work on the performance testing of Saxon-C. This stems from our comparison work to other XSLT implementations from different platforms. This is inline with the project https://github.com/Saxonica/XT-Speedo which is currently being undertaken by colleagues at Saxonica.

In essence I have found that doing repeated executions of a transformation which involves the creation of a stylesheet and an XdmValue object I was getting the out of memory exception after running through a large number of test cases. I have addressed the exception in a number of ways:

  1. Implemented the methods releaseStylesheet and releaseXdmValue in the XsltProcessor and XdmValue classes, respectively.

  2. Implemented the compile method to cache the compiled stylesheet for repeated execution

  3. Fixed up the parseXmlFile and parseXMLString methods so that it can be used repeatably by the currently compiled stylesheet or in the one-shot transformation methods. As mentioned in point 1. the XdmValue object created can now be safely deleted.

There has been a quite a number of small fixes that will improve the overall performance of Saxon-C, which will be in the next release along with the fixes described above.

#2 Updated by O'Neil Delpratt almost 3 years ago

  • Status changed from In Progress to Closed

This bug has been resolved in later releases.

#3 Updated by Frank Arensmeier over 2 years ago

Hi!

I know that this bug has been marked as fixed. However, I am experiencing a very similar behaviour with a fresh install. My setup: Ubuntu LTS 16, Apache 2.4, PHP 7.0.30 (tested with PHP 7.1.17) and latest Saxon-HE/C 64bit. Observed behaviour is that Apache spawns a couple of child processes as soon as Saxon is invoked. Problem is that those child processes are not killed when PHP is done.

Any help greatly appreciated.

/frank

#4 Updated by Frank Arensmeier over 2 years ago

Short sample code that triggers the described behaviour:

newXsltProcessor(); ?>

/frank

#5 Updated by O'Neil Delpratt over 2 years ago

  • Status changed from Closed to In Progress

Thanks for reporting the problem you have found. We will investigate the problem and get back to you.

#6 Updated by O'Neil Delpratt over 2 years ago

Hi Frank,

I am failing to reproduce the problem you reported. Please can you paste the commands you used?

thanks

#7 Updated by Frank Arensmeier about 2 years ago

Hi!

Sorry for keeping you waiting.

A very simple test would be this one:

"; echo "Process ID :$pid".PHP_EOL; echo "Number of threads: $child_threads".PHP_EOL; echo str_repeat("-", 20).PHP_EOL; $sax = new Saxon\SaxonProcessor(); // output number of threads again $child_threads = trim(`ls /proc/{$pid}/task | wc -l`).PHP_EOL; echo "Number of threads: $child_threads".PHP_EOL; ?>

Running that code snippet in my local Docker container, I see that additional 9 child threads are running, but they stay alive until I manually kill them.

I am attaching a screen dump from "htop" - not sure if it makes it through…

Regards,

Frank

#8 Updated by Frank Arensmeier about 2 years ago

<?php
// output current PID and number of threads
$pid = getmypid();
$child_threads = trim(`ls /proc/{$pid}/task | wc -l`);

echo "<pre>";
echo "Process ID :$pid".PHP_EOL;
echo "Number of threads: $child_threads".PHP_EOL;
echo str_repeat("-", 20).PHP_EOL;

$sax = new Saxon\SaxonProcessor();

// output number of threads again
$child_threads = trim(`ls /proc/{$pid}/task | wc -l`).PHP_EOL;
echo "Number of threads: $child_threads".PHP_EOL;

#9 Updated by O'Neil Delpratt about 2 years ago

Thanks. I have managed to reproduce the problem. Investigating it now

#10 Updated by O'Neil Delpratt about 2 years ago

These extra threads are created in the initialisation of Excelsior Jet JVM. It is still unclear why these are created or how to kill these. A simple call to jet release has no effect.

#11 Updated by O'Neil Delpratt about 2 years ago

The threads are created by the JET Runtime, e.g. GC thread, weak reference handler thread, finalizers thread, etc.

These threads should live for the duration of the execution of the php script and close at the end of the PHP request.

To make sure the clean up of the JET threads happens we need to add back in the SaxonProcessor::release() method call probably in the MSHUTDOWN_FUNCTION or RSHUTDOWN_FUNCTION.

#12 Updated by Frank Arensmeier 11 months ago

I wonder if there is any progress. It would be valuable for me to know if this bug will be fixed any time soon.

#13 Updated by O'Neil Delpratt 11 months ago

  • Description updated (diff)
  • Category changed from Saxon-C Internals to PHP API
  • Found in version changed from 0.2 to 1.1.2

No progress I am afraid to say. Thank you for bringing this bug issue back up. I have tried following comment #11 and also tried added a release() method to the PHP API but the 13 threads are still showing up.

Currently investigating this further.

#14 Updated by O'Neil Delpratt 11 months ago

Update:

Having looked at this issue carefully it seems to me that the threads created by JET are killed following the execution of the PHP script.

In the browser I ran the script from comment #8.

Run 1:

Process ID :20947
Number of threads: 1

--------------------

Number of threads: 13

Run 2:

Process ID :21130
Number of threads: 1

--------------------

Number of threads: 13

Run 3:

Process ID :21189
Number of threads: 1

--------------------

Number of threads: 13

Notice the number of threads before the SaxonProcessor call is 1 and 13 after. In the each run the threads are not going increasing there the JET threads are killed.

Running Saxon/C on a pre-release.

#15 Updated by O'Neil Delpratt 11 months ago

Update,

So I got this all wrong.

If I look at an old process ID I see the JET threads still alive:

Process ID :20191
Number of threads: 13
--------------------
Number of threads: 13

I am currently investigating how C++ PHP extensions work and if it would be possible to keep the JET resources live between processes if we are unable to kill the JEt threads.

#16 Updated by O'Neil Delpratt 11 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Hi,

I am pleased to say I have finally fixed this bug issue. The fix was to setup the JET VM in the PHP_MINIT_FUNCTION(saxon) function and then calling SaxonProcessor::release() in the PHP_MSHUTDOWN function. All other calls on the release method were removed. Patch available in the next major release.

Result in executed PHP script:

Process ID :5363
Number of threads: 1
--------------------
Number of threads: 1

As a workaround for users you can do the following (In the php7_saxon.cpp file):

  1. Add the method:
void php_saxonc_initialize(void){
   if(SaxonProcessor::jvmCreatedCPP == 0){
        SaxonProcessor::jvmCreatedCPP=1;
    //SaxonProcessor::sxn_environ= new sxnc_environment;
        SaxonProcessor::sxn_environ= (sxnc_environment *)malloc(sizeof(sxnc_environment));

    /*
     * First of all, load required component.
     * By the time of JET initialization, all components should be loaded.
     */

    SaxonProcessor::sxn_environ->myDllHandle = loadDefaultDll ();

    /*
     * Initialize JET run-time.
     * The handle of loaded component is used to retrieve Invocation API.
     */
    initDefaultJavaRT (SaxonProcessor::sxn_environ); 
    }
}
  1. In the PHP_MINIT_FUNCTION(saxon) function add a call to the new function given above:
    php_saxonc_initialize();

  1. The PHP_MSHUTDOWN_FUNCTION now looks like the following:
PHP_MSHUTDOWN_FUNCTION(saxon) {
    SaxonProcessor::release();
    UNREGISTER_INI_ENTRIES();
    return SUCCESS;
}


#17 Updated by O'Neil Delpratt 10 months ago

  • Status changed from Resolved to Closed
  • Fixed in version set to 1.2.0

Big fix applied in the Saxon/C 1.2.0 release.

#18 Updated by Frank Arensmeier 10 months ago

Nice work!

#19 Updated by O'Neil Delpratt 9 months ago

  • Status changed from Closed to In Progress

I am reopening this bug issue because this fix presented in comment note 16 causes hanging issues when the PHP script is executed in the browser. Related to bug #4371

Please register to edit this issue

Also available in: Atom PDF