1. 10

I bumped into a (C)Python behaviour at work that I found quite surprising. Spent too much time debugging it until I learnt it was not a bug (some people I have spoken to have other opinions, here), but a CPython implementation detail and the way UNIX works cause this :)

If you want to try to guess / debug before reading:

I was doing this, to pass it to some API, similar to subprocess’ so it would write a binary’s stdout to this file. fd leak aside, this should work but did not, with no error whatsoever:

stdout_fd = open("/tmp/output.txt").fileno()
  1. 3

    I wish that Python teaching materials would teach the with statement, which has been with us since Python 2.5 over a decade ago, for context management. Context managers are easily the best way to ensure that file handles are cleaned up. Everything works even on PyPy or Jython, although the PyPy documentation does not say it explicitly.

    I find the comparison to Rust interesting. The author says:

    Rust is the only popular language that models [file-descriptor management] right, reducing the chances of messing up.

    I wonder to what degree a higher-level API would have removed the problem entirely. In Monte, file descriptors cannot be accessed at all, which might have prevented the problem, but all I/O is asynchronous, which might have exacerbated the problem.

    1. 1

      Completely agree with regards to context managers, they are a very elegant!

      In Monte, file descriptors cannot be accessed at all, which might have prevented the problem

      I’m not familiar with Monte but not exposing the backing fd is a good solution to the problem. If we can’t call close(2) on arbitrary file descriptors and the file object implementation does not expose the backing fd we should be good. The main issue is that Python’s API can’t change at this point, and some low-level APIs require file descriptors to be passed around, and Python semantics can’t capture this in way that misuses are prevented.

      I brought up Rust because I really like that Rust solves it very elegantly through not having a close wrapper in the standard library, the way ownership works, and the fact that crates that wrap close(2) have this function as unsafe.