TarInfo objects dripping information

Question

TarInfo objects dripping information

I've encountered an issue with my Python tool that reads through a tar.xz file and handles each individual file within it. The input file is 15MB compressed, expanding to 740MB when uncompressed.

Unfortunately, on a particular server with limited memory resources, the program crashes due to running out of memory. To troubleshoot this problem, I utilized objgraph to analyze object creation. It appears that the TarInfo instances are not being properly released. A snippet of the main loop structure looks something like this:

with tarfile.open(...) as tar:
    while True:
        next = tar.next()
        stream = tar.extractfile(next)
        process_stream()
        iter+=1
        if not iter%1000:
            objgraph.show_growth(limit=10)

The diagnostic output consistently shows:

TarInfo     2040     +1000
TarInfo     3040     +1000
TarInfo     4040     +1000
TarInfo     5040     +1000
TarInfo     6040     +1000
TarInfo     7040     +1000
TarInfo     8040     +1000
TarInfo     9040     +1000
TarInfo    10040     +1000
TarInfo    11040     +1000
TarInfo    12040     +1000

This pattern persists until all 30,000 files are processed.

To further investigate, I removed the lines responsible for creating streams and processing them. Surprisingly, the memory usage did not change - indicating a leakage of TarInfo instances.

This issue occurs across multiple operating systems including Ubuntu, OS X, and Windows, while using Python version 3.4.1.

python python-3.x memory-leaks tar objgraph

Answer 1

Answer №1

It appears that this behavior is intentional. The TarFile object stores all the TarInfo objects it contains in a `members` attribute. Every time you call the next function, the extracted TarInfo object is added to this list:

def next(self):
    """Return the next member of the archive as a TarInfo object, when
       TarFile is opened for reading. Return None if there is no more
       available.
    """
    self._check("ra")
    if self.firstmember is not None:
        m = self.firstmember
        self.firstmember = None
        return m

    # Read the next block.
    self.fileobj.seek(self.offset)
    tarinfo = None
    ... <snip>

    if tarinfo is not None:
        self.members.append(tarinfo)  # <-- the TarInfo instance is added to members

The members list will continue to grow with each extraction operation. While this functionality may be useful for methods like getmembers and getmember, it can be cumbersome in specific scenarios. One workaround suggested is to clear the members attribute while iterating through the items (as recommended here):

with tarfile.open(...) as tar:
    while True:
        next = tar.next()
        stream = tar.extractfile(next)
        process_stream()
        iter+=1
        tar.members = []  # Clear members list
        if not iter%1000:
            objgraph.show_growth(limit=10)

Answer 2

It appears that this behavior is intentional. The TarFile object stores all the TarInfo objects it contains in a `members` attribute. Every time you call the next function, the extracted TarInfo object is added to this list:

def next(self):
    """Return the next member of the archive as a TarInfo object, when
       TarFile is opened for reading. Return None if there is no more
       available.
    """
    self._check("ra")
    if self.firstmember is not None:
        m = self.firstmember
        self.firstmember = None
        return m

    # Read the next block.
    self.fileobj.seek(self.offset)
    tarinfo = None
    ... <snip>

    if tarinfo is not None:
        self.members.append(tarinfo)  # <-- the TarInfo instance is added to members

The members list will continue to grow with each extraction operation. While this functionality may be useful for methods like getmembers and getmember, it can be cumbersome in specific scenarios. One workaround suggested is to clear the members attribute while iterating through the items (as recommended here):

with tarfile.open(...) as tar:
    while True:
        next = tar.next()
        stream = tar.extractfile(next)
        process_stream()
        iter+=1
        tar.members = []  # Clear members list
        if not iter%1000:
            objgraph.show_growth(limit=10)

TarInfo objects dripping information

Answer №1

Similar questions

Running a .py file in IDLE from a folder separate from the Python installation location: How to do it?

Reorganize a dataset using suffix in Python

Unsuccessful conversion of JSON data to HTML using json2html in Python

Python 3: The list encounters a cycle of indices after passing index 47 with over 100 elements. What is the reason behind this behavior and how can it

Help with selenium web scraping

How to select the entire text in Python Tkinter by simply left-clicking

What is the process for retrieving information from custom_export within Otree?

What is the purpose of using socket.accept() function in Python and how should it be implemented?

`The `open...` function does not function properly when used in a scheduled task on Windows

Updating MySQL with values from a list using Python

Use a for loop to discover the words in a sentence along with their corresponding index positions

Having difficulty navigating to the subsequent page link

How to click on a hyperlink element without any attributes in Selenium using Python

Issues with the proper display of Bootstrap 4 in Django are causing problems

Concealed file selection in Python Automation with Selenium

Error encountered: Unable to decode JSON object in Django unit test for class-based view

Concern regarding the Functionality of Django URL Routers

Linear y-axis and log x-axis: finding the best fit line on a logarithmic scale

Troubleshooting Chrome driver version conflict with Selenium and PyInstaller in Python

Removing the initial characters from the beginning of every first line in every JSON file