https://saxonica.plan.io/https://saxonica.plan.io/favicon.ico2014-05-09T08:47:21ZSaxonica Developer CommunitySaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=31552014-05-09T08:47:21ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Category</strong> changed from <i>PHP API</i> to <i>Saxon-C Internals</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul><p>Update on memory usage issues:</p>
<p>I have been doing some extensive work on the performance testing of Saxon-C. This stems from our comparison work to other XSLT implementations from different platforms. This is inline with the project <a href="https://github.com/Saxonica/XT-Speedo" class="external">https://github.com/Saxonica/XT-Speedo</a> which is currently being undertaken by colleagues at Saxonica.</p>
<p>In essence I have found that doing repeated executions of a transformation which involves the creation of a stylesheet and an XdmValue object I was getting the out of memory exception after running through a large number of test cases. I have addressed the exception in a number of ways:</p>
<ol>
<li>
<p>Implemented the methods releaseStylesheet and releaseXdmValue in the XsltProcessor and XdmValue classes, respectively.</p>
</li>
<li>
<p>Implemented the compile method to cache the compiled stylesheet for repeated execution</p>
</li>
<li>
<p>Fixed up the parseXmlFile and parseXMLString methods so that it can be used repeatably by the currently compiled stylesheet or in the one-shot transformation methods. As mentioned in point 1. the XdmValue object created can now be safely deleted.</p>
</li>
</ol>
<p>There has been a quite a number of small fixes that will improve the overall performance of Saxon-C, which will be in the next release along with the fixes described above.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=101682017-09-28T09:39:39ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li></ul><p>This bug has been resolved in later releases.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=115722018-05-08T13:22:25ZFrank Arensmeier
<ul></ul><p>Hi!</p>
<p>I know that this bug has been marked as fixed. However, I am experiencing a very similar behaviour with a fresh install. My setup: Ubuntu LTS 16, Apache 2.4, PHP 7.0.30 (tested with PHP 7.1.17) and latest Saxon-HE/C 64bit. Observed behaviour is that Apache spawns a couple of child processes as soon as Saxon is invoked. Problem is that those child processes are not killed when PHP is done.</p>
<p>Any help greatly appreciated.</p>
<p>/frank</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=115732018-05-08T13:27:19ZFrank Arensmeier
<ul></ul><p>Short sample code that triggers the described behaviour:</p>
<?php $sax = new Saxon\SaxonProcessor();
$proc = $sax -> newXsltProcessor();
?>
<p>/frank</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=115742018-05-08T13:39:28ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>In Progress</i></li></ul><p>Thanks for reporting the problem you have found. We will investigate the problem and get back to you.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=116082018-05-09T16:01:08ZO'Neil Delprattoneil@saxonica.com
<ul></ul><p>Hi Frank,</p>
<p>I am failing to reproduce the problem you reported. Please can you paste the commands you used?</p>
<p>thanks</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=116722018-05-16T12:42:27ZFrank Arensmeier
<ul><li><strong>File</strong> <a href="/attachments/3122">screendump.jpeg</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3122/screendump.jpeg">screendump.jpeg</a> added</li></ul><p>Hi!</p>
<p>Sorry for keeping you waiting.</p>
<p>A very simple test would be this one:</p>
<?php // output current PID and number of threads
$pid = getmypid();
$child_threads = trim(`ls /proc/{$pid}/task | wc -l`);
echo "<pre>";
echo "Process ID :$pid".PHP_EOL;
echo "Number of threads: $child_threads".PHP_EOL;
echo str_repeat("-", 20).PHP_EOL;
$sax = new Saxon\SaxonProcessor();
// output number of threads again
$child_threads = trim(`ls /proc/{$pid}/task | wc -l`).PHP_EOL;
echo "Number of threads: $child_threads".PHP_EOL;
?>
<p>Running that code snippet in my local Docker container, I see that additional 9 child threads are running, but they stay alive until I manually kill them.</p>
<p>I am attaching a screen dump from "htop" - not sure if it makes it through…</p>
<p>Regards,</p>
<p>Frank</p></pre> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=116732018-05-16T12:49:33ZFrank Arensmeier
<ul></ul><pre><code><?php
// output current PID and number of threads
$pid = getmypid();
$child_threads = trim(`ls /proc/{$pid}/task | wc -l`);
echo "<pre>";
echo "Process ID :$pid".PHP_EOL;
echo "Number of threads: $child_threads".PHP_EOL;
echo str_repeat("-", 20).PHP_EOL;
$sax = new Saxon\SaxonProcessor();
// output number of threads again
$child_threads = trim(`ls /proc/{$pid}/task | wc -l`).PHP_EOL;
echo "Number of threads: $child_threads".PHP_EOL;
</code></pre> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=116742018-05-16T13:47:09ZO'Neil Delprattoneil@saxonica.com
<ul></ul><p>Thanks. I have managed to reproduce the problem. Investigating it now</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=116762018-05-17T14:43:45ZO'Neil Delprattoneil@saxonica.com
<ul></ul><p>These extra threads are created in the initialisation of Excelsior Jet JVM. It is still unclear why these are created or how to kill these. A simple call to jet release has no effect.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=116772018-05-18T09:26:22ZO'Neil Delprattoneil@saxonica.com
<ul></ul><p>The threads are created by the JET Runtime, e.g. GC thread, weak reference handler thread, finalizers thread, etc.</p>
<p>These threads should live for the duration of the execution of the php script and close at the end of the PHP request.</p>
<p>To make sure the clean up of the JET threads happens we need to add back in the SaxonProcessor::release() method call probably in the MSHUTDOWN_FUNCTION or RSHUTDOWN_FUNCTION.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=142272019-09-06T12:46:23ZFrank Arensmeier
<ul></ul><p>I wonder if there is any progress. It would be valuable for me to know if this bug will be fixed any time soon.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=142282019-09-06T13:25:57ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14228/diff?detail_id=20033">diff</a>)</li><li><strong>Category</strong> changed from <i>Saxon-C Internals</i> to <i>PHP API</i></li><li><strong>Found in version</strong> changed from <i>0.2</i> to <i>1.1.2</i></li></ul><p>No progress I am afraid to say. Thank you for bringing this bug issue back up. I have tried following comment #11 and also tried added a <code>release()</code> method to the PHP API but the 13 threads are still showing up.</p>
<p>Currently investigating this further.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=142292019-09-06T16:20:57ZO'Neil Delprattoneil@saxonica.com
<ul></ul><p>Update:</p>
<p>Having looked at this issue carefully it seems to me that the threads created by JET are killed following the execution of the PHP script.</p>
<p>In the browser I ran the script from comment #8.</p>
<p>Run 1:</p>
<pre><code>Process ID :20947
Number of threads: 1
--------------------
Number of threads: 13
</code></pre>
<p>Run 2:</p>
<pre><code>Process ID :21130
Number of threads: 1
--------------------
Number of threads: 13
</code></pre>
<p>Run 3:</p>
<pre><code>Process ID :21189
Number of threads: 1
--------------------
Number of threads: 13
</code></pre>
<p>Notice the number of threads before the SaxonProcessor call is 1 and 13 after. In the each run the threads are not going increasing there the JET threads are killed.</p>
<p>Running Saxon/C on a pre-release.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=142302019-09-09T10:24:07ZO'Neil Delprattoneil@saxonica.com
<ul></ul><p>Update,</p>
<p>So I got this all wrong.</p>
<p>If I look at an old process ID I see the JET threads still alive:</p>
<pre><code>Process ID :20191
Number of threads: 13
--------------------
Number of threads: 13
</code></pre>
<p>I am currently investigating how C++ PHP extensions work and if it would be possible to keep the JET resources live between processes if we are unable to kill the JEt threads.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=142312019-09-10T14:23:58ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Hi,</p>
<p>I am pleased to say I have finally fixed this bug issue. The fix was to setup the JET VM in the <code>PHP_MINIT_FUNCTION(saxon)</code> function and then calling <code>SaxonProcessor::release()</code> in the PHP_MSHUTDOWN function. All other calls on the release method were removed. Patch available in the next major release.</p>
<p>Result in executed PHP script:</p>
<pre><code>Process ID :5363
Number of threads: 1
--------------------
Number of threads: 1
</code></pre>
<p>As a workaround for users you can do the following (In the php7_saxon.cpp file):</p>
<ol>
<li>Add the method:</li>
</ol>
<pre><code class="c syntaxhl" data-language="c"><span class="kt">void</span> <span class="nf">php_saxonc_initialize</span><span class="p">(</span><span class="kt">void</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="n">SaxonProcessor</span><span class="o">::</span><span class="n">jvmCreatedCPP</span> <span class="o">==</span> <span class="mi">0</span><span class="p">){</span>
<span class="n">SaxonProcessor</span><span class="o">::</span><span class="n">jvmCreatedCPP</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span>
<span class="c1">//SaxonProcessor::sxn_environ= new sxnc_environment;</span>
<span class="n">SaxonProcessor</span><span class="o">::</span><span class="n">sxn_environ</span><span class="o">=</span> <span class="p">(</span><span class="n">sxnc_environment</span> <span class="o">*</span><span class="p">)</span><span class="n">malloc</span><span class="p">(</span><span class="k">sizeof</span><span class="p">(</span><span class="n">sxnc_environment</span><span class="p">));</span>
<span class="cm">/*
* First of all, load required component.
* By the time of JET initialization, all components should be loaded.
*/</span>
<span class="n">SaxonProcessor</span><span class="o">::</span><span class="n">sxn_environ</span><span class="o">-></span><span class="n">myDllHandle</span> <span class="o">=</span> <span class="n">loadDefaultDll</span> <span class="p">();</span>
<span class="cm">/*
* Initialize JET run-time.
* The handle of loaded component is used to retrieve Invocation API.
*/</span>
<span class="n">initDefaultJavaRT</span> <span class="p">(</span><span class="n">SaxonProcessor</span><span class="o">::</span><span class="n">sxn_environ</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<ol start="2">
<li>In the PHP_MINIT_FUNCTION(saxon) function add a call to the new function given above:</li>
</ol>
<pre><code class="c syntaxhl" data-language="c"> <span class="n">php_saxonc_initialize</span><span class="p">();</span>
</code></pre>
<ol start="3">
<li>The PHP_MSHUTDOWN_FUNCTION now looks like the following:</li>
</ol>
<pre><code class="c syntaxhl" data-language="c"><span class="n">PHP_MSHUTDOWN_FUNCTION</span><span class="p">(</span><span class="n">saxon</span><span class="p">)</span> <span class="p">{</span>
<span class="n">SaxonProcessor</span><span class="o">::</span><span class="n">release</span><span class="p">();</span>
<span class="n">UNREGISTER_INI_ENTRIES</span><span class="p">();</span>
<span class="k">return</span> <span class="n">SUCCESS</span><span class="p">;</span>
<span class="p">}</span>
</code></pre> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=143412019-10-15T22:49:16ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Closed</i></li><li><strong>Fixed in version</strong> set to <i>1.2.0</i></li></ul><p>Big fix applied in the Saxon/C 1.2.0 release.</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=143582019-10-16T09:28:09ZFrank Arensmeier
<ul></ul><p>Nice work!</p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=144772019-11-05T12:40:07ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>In Progress</i></li></ul><p>I am reopening this bug issue because this fix presented in comment note 16 causes hanging issues when the PHP script is executed in the browser. Related to bug <a class="issue tracker-1 status-3 priority-3 priority-highest closed" title="Bug: Saxon/C crashing in browser when run via PHP (Closed)" href="https://saxonica.plan.io/issues/4371">#4371</a></p> SaxonC - Bug #2055: Apache allocates a lot of memory without releasing ithttps://saxonica.plan.io/issues/2055?journal_id=225322023-01-16T13:22:38ZO'Neil Delprattoneil@saxonica.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li></ul><p>This fix has now been resolved in SaxonC 11</p>