1. 20

  2. 3

    PyCharm has this great feature that allows it to attach to a running Python program and debug it - how does this work? No changes to the code (like described for manhole) are required.

    1. 3

      I assume it’s using something like pyrasite. How old is this feature in PyCharm?

      I’d be surprised if they did it before gdb got the feature via pyrasite. It would be kind of sad, in a way, because that means that work was recreated just because Pycharm is proprietary.

      As to your question on this works, what gdb does (and any debugger, really) at least on Unix, is a ptrace syscall. The OS has to allow this syscall, which can be denied for permissions reasons. Once one process can control the other, it’s a matter of inspecting memory to figure out where the program stack and local variables are and, going further, knowing how CPython works so that you can recreate the Python stack and variables from there.

      1. 4

        PyCharm’s debugger is open source and almost always has been. That includes how they are attaching.

        1. 2

          Right, so they’re just calling gdb:


          This is still a small consolation for me, since such a widespread IDE is not free.

      2. 2

        You can do this without PyCharm as well. You need some additional instrumentation for gdb so that instead of looking at the innards of CPython itself you’d get new commands like py-list, py-bt: https://wiki.python.org/moin/DebuggingWithGdb

        But this Manhole thing actually sounds more convenient.

        1. 2

          This isn’t the same. This doesn’t let you inject code into the running process, only to inspect it. Pyrasite is the thing made the same gdb maintainers who implemented py-bt which does let you inject code.

        2. 1

          gdb would be one way.

        3. 2

          You can also do this to JVM processes, more or less, using LiveREPL. It’s intended to attach to Clojure processes, but you can attach it to anything running on the JVM, and it doesn’t require any cooperation—all you need is the script and some jar files, and to be running as the same user as the JVM process is. From there, it’s all just Java agent injection and dynamic classloaders and other stuff I don’t totally understand.

          It sounds like Manhole instead requires cooperation from the server, but if you can run as the same user anyway, it’s always theoretically possible to attach a debugger or something.

          I’m hearing people say “that sounds really dangerous in production” but the funny thing is that that’s the only place I’ve ever used it. Why? Because this is a tool of last resort, and if it weren’t production, I’d do it a different way. I’m not happy to have had to use LiveREPL on a production server, but it was necessary (couldn’t reproduce behavior in other environments, and needed to validate a guess as to the right fix) and we were careful: Take the server out of the cluster, replicate the bad behavior, attach, dig around inside, apply the proposed patch to the running application (ain’t Clojure grand?), validate the fix, terminate and replace the server.

          I’ve only had to do it twice, and it was totally worth it.

          1. 2

            Sounds a lot like what Erlang developers do with attaching REPLs and doing hot code reloading.

            1. 1

              Same goes for the Lisp family of languages.

              It’s wonderful for debugging since you can just explore the running system where the problem is.

              1. 1

                Yep. LiveREPL can help you inspect a running java process, but if it’s a Clojure app you can actually swap stuff out too, since it’s a Lisp with rebindable vars. :-)

          2. 1

            If someone can run arbitrary programs inside your container they already have plenty of access, so adding Manhole doesn’t seem like much of a security risk.

            Actually, this makes it very easy to change a running program and inject a bunch of code without any traceability. So, in terms of doing a post-break in analysis, it might be harder if the intruder used Manhole.

            1. 1

              Considering the security sort of implies you’d let this run in production? That seems like a poor decision, I’d appreciate analysis of the slowdown overhead this incurs. Especially given the site.

              Anything goes in dev, so perhaps it’s more right for staging. But if you’re deep enough in infrastructure to have staging already.. hmm.

              Cool project, though!

              1. 2

                It runs another thread waiting for connection. It has option to run the thread only after a signal is sent to it, in which case performance overhead is zero.

              2. 0

                This is cool.