Solving a ResourceT-related space leak in production

Oops I wrote this before I saw the example above:

Here’s an example in python (the same as the one before, but concrete this time), the same situation exists with streams and I assume conduit and the rest. The scope of lines is dynamic, not amenable to bracket. But it also illustrates how it doesn’t turn out to be a problem in python, due to GC. The File object is just to make closing noisy:

#!/usr/bin/env python3

def main():
    open('a', 'w').write(''.join(f"a{i}\n" for i in range(6)))
    open('b', 'w').write(''.join(f"b{i}\n" for i in range(6)))
    timings = [
        (0, 'a'),
        (2, 'b'),
        (4, 'a'),
    ]
    # with resourcet:
    for out in mix(timings):
        print(out)

def mix(timings):
    iters = [pad(start, fname) for (start, fname) in timings]
    while iters:
        outs = [next(iter, None) for iter in iters]
        yield [o for o in outs]
        iters = [iter for (iter, o) in zip(iters, outs) if o is not None]

def pad(start, fname):
    for i in range(start):
        yield ''
    for _, line in zip(range(3), lines(fname)):
        yield line

def lines(fname):
    return iter(File(fname))

class File:
    def __init__(self, fname):
        self.file = open(fname)
        # self.key = resourcet.acquire(self.file)
    def __iter__(self):
        return self
    def __next__(self):
        line = self.file.readline()
        if not line:
            self.close()
            # resourcet.release(self.key)
            raise StopIteration
        else:
            return line
    def close(self):
        print('** close', self.file)
        self.file.close()
    def __del__(self):
        print('** del close', self.file)
        self.file.close()

if __name__ == '__main__':
    main()

When I run it, the files are closed promptly, after the take 3, but they are closed by __del__ rather than close. In the real situation, it isn’t take 3, but another stream that goes to zero, but I think it comes out the same.
If python had a registry analogous to resourcet, it would be in the places where the comments are, and if it didn’t use weak pointers, it would also prevent del close until the backstop at the end of the with.

I’m sure there’s a more minimal example, but this is how I came across it!

Maybe bluefin can do it! It looks pretty similar in theory. The jumpTo stuff in take doesn’t exist in traditional streams. I don’t yet understand how the jumpTo done in take then triggers the corresponding jumpTo onEOF in linesOfFile… if it does… and who is calling who here.

1 Like