Bug #6392: Memory leak in PySaxonProcessor (Python) - SaxonC - Saxonica Developer Community

Actions

Send by e-mail Copy link

Bug #6392

closed

Memory leak in PySaxonProcessor (Python)

Added by Andy Knight 3 months ago. Updated 14 days ago.

Status:

Closed

Priority:

Normal

Assignee:

Matt Patterson

Category:

Start date:

2024-04-14

Due date:

% Done:

100%

Estimated time:

(Total: 0:00 h)

Applies to branch:

Fix Committed on Branch:

Fixed in Maintenance Release:

Found in version:

Fixed in version:

12.5

SaxonC Languages:

SaxonC Platforms:

SaxonC Architecture:

Description

Consider the following code fragment:

from saxonche import PySaxonProcessor import psutil

count = 0 prev = (mi := psutil.Process().memory_info)().rss

for _ in range(100): with PySaxonProcessor(license=False): pass if (count := count + 1) % 10 == 0: m = mi().rss print(f"{m - prev:,}") prev = m

It can be observed that memory usage increases over time. The exit functionality of the PySaxonProcessor class is not "cleaning up" as robustly as it should.

Python 3.12.2 saxonche 12.4.2 macOS 14.4.1 M2

Sub-issues 4 (0 open — 4 closed)

Actions

Copy link

Updated by Andy Knight 3 months ago

Apologies for incorrect code formatting in initial submission:

from saxonche import PySaxonProcessor
import psutil

count = 0
prev = (mi := psutil.Process().memory_info)().rss

for _ in range(100):
    with PySaxonProcessor(license=False):
        pass
    if (count := count + 1) % 10 == 0:
        m = mi().rss
        print(f"{m - prev:,}")
        prev = m

Actions

Copy link

Updated by O'Neil Delpratt 3 months ago

Project changed from Saxon-CE to SaxonC

Moving this project to the SaxonC project

Actions

Copy link

Updated by Matt Patterson 3 months ago

Status changed from New to In Progress
Assignee set to Matt Patterson

Actions

Copy link

Updated by Matt Patterson about 1 month ago

Status changed from In Progress to Resolved
Fixed in version set to 12.5.0

This has been a bit tricky to track down, largely because it looks like we had fixed it as a side-effect of making more general changes after 12.4.2 released, and before this was reported.

After more extensive testing using memray and a debug-enabled build, I can't see any evidence of memory leaks in the current build, which I was able to see when build a debug-enabled version of 12.4.

It's worth noting that, depending on system memory constraints, and other environmental factors, running this example for a large number of iterations may not result in the RSS usage of the process stabilising. On my machine, I've been finding it needs thousands of iterations before the OS starts reclaiming some of the assigned-but-unused memory, but memray reveals that heap usage has settled, is constant and not increasing.

Because of the similarity of this to a number of memory-related issues that were opened, I'm going to mark the others as children of this issue...

Actions

Copy link