1. 40
  1. 16

    I think my biggest complaint is that there’s no disclaimer on the box. Homebrew folks know about these problems, but they don’t warn folks that their virtual environments will be obliterated. If Homebrew explained that these problems happen, and that users might want to use a different source for installing Python interpreters, my ire would be lessened considerably.

    1. 6

      Yeah. If the Python package is just there to support other packages, and has these problems if used otherwise,they shouldn’t expose it. Either add a separate Python package for people to use, or just print an error telling people to get Python from some other source.

    2. 9

      This exact problem had plagued me for so long and it is so annoying. It was pretty much the final straw that made me move back to MacPorts.

      1. 6

        Similar issues are what sent me to nix.

        1. 2

          100% Agreed regarding Nix!

          My current home setup is Nix for everything & MacPorts for things I can’t get on Nix yet.

          1. 2

            I’ve been experimenting with Nix too. It’s nice, although port is so simple to use, whereas nix-env feels clunky/complicated in a very rpm way and the Nix language is… not straightforward :-)

            1. 2

              Yeah, nix-env is super clunky. I never use it unless I have to. I’ve got everything built declaratively. Nix the language definitely takes a minute to onboard, agreed.

              1. 1

                Last time I try it, I got oom.

          2. 3

            Agreed! There a lot of good things about MacPorts.

            1. 1

              Why did you migrate to Homebrew in the first place?

              1. 9

                Good question, and one to which I don’t really remember the answer. I’ve been using MacOS since 10.3 and had tried both Fink and MacPorts in the early days. My vague memory of both in the early days was they were fairly clunky, but Homebrew just worked.

                Homebrew has since morphed into this huge, slow thing that has become pretty annoying to use, whereas MacPorts now feels fast and slick in comparison.

                1. 4

                  Yeah. It feels like with this and other changes made over the last few years, Homebrew is coming close to falling off the back of the package manager treadmill.

                  Homebrew in its early days was good because it was lightweight, fast (for (as it was then) a package manager that could only build from source, at least), didn’t try to do stuff that was too clever, and also acquired a fairly nice and large package library fairly quickly thanks, I think, to a relaxed contribution policy that didn’t require individuals to take complete responsibility for packages. (I don’t know if Max Howell explicitly realized that keeping a package manager up to date with new versions of packages would be something a whole user community could do together if it were made easy enough, or if it just happened to work out like that, but in any case it did work very well in practice.)

                  Now it insists on automatically refreshing the package index every. time. I. do. anything. which makes startup slow, and is also doing this too-clever-by-half thing of trying to work out which packages I actually wanted to install and which just came along for the ride as dependencies. (Which isn’t how humans work: if one day I type ffmpeg and it happens to be there, it’s not necessarily clear to me that I only have it because I installed get_iplayer and it brought ffmpeg along, rather than having installed it myself explicitly on some now-forgotten date in the past. Computers have a far better memory for why things are installed than humans do, and humans will always get confused when things change because of implicit rules based on the computer’s perfect recall.) Meanwhile it’s also got stupider in some other ways, such as no longer being able to do builds from source with custom ./configure options etc.

                  I have similar memories of MacPorts being a pain to use, but who knows? It’s been ten years or something now. Maybe it’s worth another look.

                2. 3

                  I’ve been a FreeBSD user for a while so MacPorts always fit better for me. If my memory is correct, Homebrew had an initial learning curve that was shallower than the curve for MacPorts back in the day.

              2. 6

                Not sure if the author is unaware or not, but the asdf Python plugin is just a wrapper around pyenv. That’s actually all asdf does is provide a way to wrap something around version managers and provide a unified interface.

                Otherwise, yeah. Installing Python through Homebrew should only be done because another brew/cask/tap depends on it. You shouldn’t use it for actual development.

                1. 10

                  Author here. To be totally pedantic, I believe asdf uses the python-build back-end provided by pyenv rather than pyenv itself, but that’s admittedly a rather inconsequential distinction. For me, the real value of asdf is indeed the unified CLI syntax for adding/updating plugins and installing/removing versions. I like being able to use the same toolchain and UI to manage different versions of Python, Node.js, etc.

                2. 6

                  Yes, good article. I use asdf. I don’t want a python version manager, I want a version manager that knows about python. My asdf list shows me all the languages. It’s a single thing to learn. The name is ridiculous, that’s fine. But now I’m doing a recommendation and my own likes/dislikes are my own. I tried virtualenv, pyenv, conda and many others. I settled on asdf. It’s the closest to other communities. Combined with poetry, the python tooling is getting closer. I can’t believe it’s taking this long to approach ~2010 Rails’ QoL. But now I’m whining. Sorry.

                  I just wanted to say that I burned-in asdf for a year before I said: “I like it”. The plugins are all different people btw, so YMMV quite a bit there. So far, it’s fine to me but I encounter Blub type resistance with the number of similar tools. I guess this is no different than node/ruby in the past.

                  Ok but to illustrate something quite powerful (to me). In the future, python 4.0 will come out. mysterious winds start blowing. This is how I’d start to evaluate the upgrade (excuse any typos plz):

                  # assume you have a very nice test suite
                  $ asdf install python 4.0.0
                  $ asdf local python 4.0.0
                  $ git checkout -b python_4_bump
                  # edit pyproject.toml to require python 4
                  $ poetry upgrade  # if you think packages have python 4 support or something ...
                  $ poetry run test  # runs a task with taskipy, this is like npm's scripts
                  # run other tests
                  # get horrible errors
                  # fix errors, iterate
                  

                  This is fantastic.

                  The point being, you can know or at least find out. You have a checklist. You have a list of errors. You are simulating, executing your unique thing in a new world state. Python 4 will come out, Django/Flask/etc will version bump. This kind of situation is so less stressful with this fundamental tooling in place. Any version manager + package manager does this kind of stuff. I just think these tools are taking nods and notes from other language communities which is good (to me).

                  1. 1

                    Love asdf. Such a relief to be able to drop rbenv, nvm + a host of other version managers for a single tool.

                  2. 3

                    In my experience the thing that works is … installing from the Python.org installers. Seriously, they work as expected.

                    You need to basically invoke your virtualenv creation with the right Python version and then you’re off to the races.

                    1. 2

                      Yeah, why would you ever expect a package manager to expose a package you were supposed to use?

                      1. 2

                        The problem here is that they’re using a package manager to install a language and then using a hacky tool (virtualenv) which does some symlinking of binaries which are dynamically linked to libraries provided by that package manager, and then install and link other libraries using an entirely different package manager of which the original package manager is completely oblivious (pip inside the virtualenv), and then upgrade the underlying package against which the extension libraries were linked.

                        Of course this is going to break! In the extreme, think of this like installing Python 2, installing some libraries with pip and then upgrading to Python 3, and expecting those pip packages to just work without first rebuilding them against the new Python.

                        The real solution is to either do everything through the package manager (which means you’re stuck with whatever versions of Python libraries are provided by homebrew) or nothing (which means installing Python with some other tool).

                        edit: After re-reading, it seems to me Homebrew supports having multiple Pythons installed at the same time. So I guess the real problem is that it doesn’t have a mechanism to remember which Pythons you manually installed and want to keep. Or maybe it does and people are just relying on Python being installed through whatever other package and are not explicitly installing a specific version they want to rely on.

                        1. 3

                          I think you’re missing an important part of my article. The workflow you described worked very well for years, right up until Homebrew added two changes that caused everything to break, most notably the automatic cleanup that causes previous package versions to be deleted automatically.

                          To me, the real problem is that Homebrew does not mention this on their Homebrew and Python page. They could say, “Don’t rely on Homebrew Python versions, which can disappear at any time. You might be better off using asdf, pyenv, or Pythonz tooling to install and manage Python interpreter versions.”

                          1. 2

                            To me, the real problem is that Homebrew does not mention this on their Homebrew and Python page.

                            This would be a very helpful addition to the page. Have they refused the patch, or just not applied it yet?

                            1. 1

                              Fair enough; they broke something which used to work. Maybe it worked “by accident” in their vision, but if so many people relied on this accidental behaviour, it would behoove them to clearly inform users about it.

                              Note that I was more responding to the parent comment than your post though.

                            2. 3

                              Think of this like installing Python 2, installing some libraries with pip then upgrading to Python 3.

                              The problem here is that Homebrew decided to upgrade, not the user. If I upgraded to Python 3, I wouldn’t expect anything to work, but I didn’t - Homebrew did it for me without asking.

                              But what I’m really complaining about is the way the article describes it as “a misunderstanding” like users should know better. It’s a package manager. Exposing packages under obvious names that the user is not supposed to install is a horrible design failure, and I don’t like the way the article blames the user.

                              1. 4

                                I don’t like the way the article blames the user.

                                Article author here. Thanks for the constructive feedback. My initial reaction to this was, “Huh?? Did we read the same article?” Then I re-read it with your perspective in mind. And now I see your point. Allow me to explain how we got here. Short version: I was trying to be kind.

                                I actually wrote this article last summer, soon after Homebrew’s changes started wreaking havoc. Around that time, a user of VirtualFish (a virtual environment manager for Fish shell that I maintain) encountered the same infuriating problem and posted an issue about it in the VirtualFish repository tracker. I responded by posting a long rant and venting my frustration with how poorly the transition to the new Homebrew behavior was managed. In that comment you will see the basis upon which my article was written, which I purposely softened so as to respect the hard work that unpaid, volunteer Homebrew maintainers put into the project, for free.

                                I fear that in the process of toning down my frustration, I inadvertently left out the part about Homebrew’s culpability in neglecting to inform users about its suitability for Python development, leaving the opportunity for folks to walk away with the impression that I think the fault lies with users and their inability to understand. That was absolutely not my intention — quite the opposite. Mea culpa.

                                I hope that after reading my above-linked screed, you will see that I do not hold users accountable here. Perhaps I should amend the article to make that clearer. Thanks again for communicating your perspective, which will help me express myself better in the future.

                                1. 1

                                  Well that makes sense. Thanks for the explanation!

                              2. 1

                                There is still one way that it can break things even if you used another tool like pyenv to install the Python interpreters you actually use: pyenv compiles the interpreter locally, linking against the set of shared libraries you have at the time. Which doesn’t matter much for Python, except in the case of OpenSSL. Sometimes, Homebrew will “helpfully” clean up an older OpenSSL that some pyenv-compiled interpreters linked against, and suddenly those interpreters fail.

                                1. 1

                                  Can’t you tell pyenv to link Python against the system-provided OpenSSL though? Or is that so hopelessly out of date that it’s unusable?

                                  1. 1

                                    I don’t know how it is now. I know at one point you didn’t really have a choice because the system OpenSSL was too old for things like pip to even be able to speak to the Python Package Index.

                            3. 1

                              I’m not on a Mac for over a year, but when I was, usually I compiled the Python version from source, or used a docker container. It may not be the ideal solution for everyone, but it was for me.

                              1. 1

                                Very glad to see this as I was frustrated after installing Python with homebrew and using pyenv- only to have my working envs wiped out without warning. I must admit it’s down to my lack of understanding of how Python environment management works and it’s pushed me to get a bit deeper on that.

                                1. 1

                                  Isn’t this what versioned packages are for?

                                  brew install python@3.7
                                  brew install python@3.8
                                  brew install python@3.9
                                  

                                  I haven’t tried it myself, but is there some reason this doesn’t work with virtualenv?

                                  1. 9

                                    That works fine until you inadvertently get a point release, while upgrading some other homebrew package, then boom, all your virtualenvs go poof.

                                  2. 1

                                    I used pyenv for a minute but it seemed to screw a bunch of other stuff up, specifically IPython complained about running in a virtual environment, but there were other, mysterious, problems as well. Maybe if I’d used it through asdf’s consistent frontend I wouldn’t have made whatever mistake it was that I made that caused my entire Python toolchain to explode and become unusable.

                                    1. 3

                                      IPython complained about running in a virtual environment

                                      The message IPython gives is not, well-worded.

                                      The thing it’s trying to warn about is a case where the instance of IPython that’s running currently does not belong to the active virtual environment. IPython will do some gymnastics to make the virtual environment’s contents visible for you to import, but won’t be using the virtual environment’s Python interpreter, and won’t properly isolate what it can see or the side effects of what it’s doing. So the safe thing, and what it’s trying to suggest, is to create the virtual environment, activate it, and then pip install ipython so that the IPython you get when the virtual environment is active is one tied to that virtual environment.

                                      1. 1

                                        Yeah, see, that’s what I was doing. I looked it up at one point and IIRC there was a bug or something where it interpreted the pyenv environment as a virtual environment so it thought you were trying to run a virtual environment inside another virtual environment. There didn’t seem to be a reasonable solution at the time, and pyenv also seemed to cause some other problems I don’t recall specifically, so I gave up with the intention of trying again in the future.

                                    2. 1

                                      I’ve had the same issue with Homebrew Python and the constant breakage. But never come around to fix it before this post.

                                      On macOS Big Sur, this seems to “fix” it. Just applied it, but I can’t guarantee that it works in the longrun.

                                      $ brew unlink python3
                                      $ echo "/Users/$USER/Library/Python/3.8/bin" | sudo tee /etc/paths.d/user_python
                                      # Reload shell
                                      

                                      Note: Now python points to Python 2.7. Also, python 3 -m pip install -U #package# installs to the user folder automatically as the global folder is write-protected.

                                      1. 1

                                        Virtualenv supposed to be disposable. Using tools like poetry you just recreate the env and rerun poetry install. As long as poetry itself not within virtualenv but using python from brew directly it should not be affected by the upgrade. I see there’s brew formulae for poetry.

                                        Btw this post caught my eye because I wrote with similar title long time ago :)

                                        1. 0

                                          Who would install from some dodgy third party “package manager” when they can use the Python that Apple ships with the OS or the Python Software Foundation ships from python.org?

                                          1. 2

                                            The version that Apple ships is unsupported (2.7)

                                            1. 1

                                              Current macOS versions also ship Python 3.

                                              1. 2

                                                TIL. They’re removing Python, Ruby, and Perl from macOS at some point tho

                                                1. 1

                                                  Hiding, not removing. IIRC these will still be there, just not in PATH, so the user will not accidentally use OS-provided one.

                                          2. [Comment from banned user removed]

                                            1. 9

                                              Using virtualenv at all instead of installing dependencies from your system package manager.

                                              If you install all your Python dependencies from the system package manager, what do you do when one isn’t packaged? What do you do when the right version isn’t packaged? What do you do when two projects require conflicting versions of the same package?

                                              1. 2

                                                what do you do when one isn’t packaged

                                                Package it.

                                                What do you do when the right version isn’t packaged

                                                Upgrade the package, or upgrade your software, whichever one is outdated.

                                                What do you do when two projects require conflicting versions of the same package

                                                Patch the outdated one, send fix upstream.

                                                1. 1

                                                  If one is using Nix, as in sibling comments, then it is relatively easy and straightforward to have two different versions of a single package. Indeed, the trickery for Nix users tends to be in finding ways to support many different versions with a single Nix expression.

                                                2. 9

                                                  Using virtualenv at all instead of installing dependencies from your system package manager.

                                                  The virtualenv approach is the standard method, in the Python world, to get builds/deploys of a codebase with its own isolated copy of dependencies, and that approach has been adopted as the default and only way to do things by more recently-designed languages which abandoned the traditional Unix “every library goes in a shared systemwide location” approach that still pulls down older more “Unix-y” languages including but not at all limited to Python.

                                                  1. 0

                                                    virtualenv is a third-party tool, not a Python “standard”. If you abandon Unix for your faux “standards”, don’t be surprised when it bites back.

                                                    1. 16

                                                      virtualenv is a third-party tool, not a Python “standard”

                                                      If you mean this in the extremely narrow sense that use of the exact name “virtualenv” only ever applied to the third-party package, and that the module which does the same basic thing, that has been in the Python standard library for over eight years and was added as a result of a standardization of the third-party package, is technically named “venv” instead, then you are correct.

                                                      Perhaps if you argue that the still-extant third-party “virtualenv” package is technically a superset of the functionality of the standard-library “venv” package, you can also earn partial credit, though generally when people new to Python learn how to do this today they learn it as python -m venv using the standard-library module and its functionality, and not the extended feature set of the third-party “virtualenv”, or they learn it from using tools which provide a convenient abstracted interface which do not generally expose directly to the user which implementation of virtual environments is being used.

                                                      1. 3

                                                        Why is python any more “UNIX” than virtualenv?

                                                    2. 11

                                                      This basically reads as an anti-Mac rant.

                                                      1. 3

                                                        Only one of these has anything to do with Mac.

                                                      2. 4

                                                        Using macOS instead of an OS which (a) has its own packaging team

                                                        you’re conflating the job of an OS from that of package management here.