Every time Ruff comes up, the fact that it’s super-hyper-mega-fast is promoted as making up for the fact that it’s basically not extensible by a Python programmer (while the tools it wants to replace all easily are extensible in Python).
But the speed benefit only shows up when checking a huge number of files in a single run. And even on a large codebase:
In my editor, I only care about the linter being fast enough to lint the file I’m directly working in.
In pre-commit hooks, I only care about the linter being fast enough to lint the set of changed files.
In CI, on a codebase large enough that, say, flake8 would actually be taking significant time due to the number of files in a full-codebase lint, it’s overwhelmingly likely that the flake8 run still would effectively be noise compared to other things like the time to run a full test suite of that size.
So I’m still not sure why I should give up easy extensibility for speed that seems to offer me no actual practical benefit.
For these kinds of tools performance (both speed and memory usage) matters a lot, because codebases are effectively unbounded in size, and because for interactive use, latency budgets are pretty tight. There’s also Sorbet’s observation that performance unlocks new features. “Why would you whatchexec this on the whole code base? Because I now can”.
Now, if we speak strictly about syntax-based formatting and linting, you can get quite a bit of performance from the embarrassingly parallel nature of the task. But of course you want to do cross-file analysis, type inference, duplicate detection and what not.
The amount of things you can do with a good static analysis base is effectively unbounded. At this point, maybe Java and C# are coming to the point of saturation, but everything else feels like a decade behind. The primary three limiting factors to deliver these kinds of tools are:
performant architecture (you need smarts to avoid re-doing global analysis on every local change)
raw speed (with the right arch, things like parsing or hashing become bottlenecks)
the work to actually implement fancy features on top of fast base
This is high-investment, high-value thing, which requires a great foundation. And I would actually call that, rather than today’s raw performance, the most important feature of Ruff. We can start from fast linting, and then move to import analysis, type inference, full LSP and what not.
From my point of view, Python’s attempt to self-host all dev tools is a strategic blunder. Python really doesn’t have performance characteristics to move beyond per file listing, so it’s not surprising that, eg, pyright does its own thing rather than re-use existing ecosystem.
All that being said, extensibility is important! And Python is a fine language for that. Long term, I see Ruff exposing a Python scripting interface for this. If slow Python scripting sits on top of fast native core that does 90% o the CPU work, that should be fine!
For these kinds of tools performance (both speed and memory usage) matters a lot, because codebases are effectively unbounded in size, and because for interactive use, latency budgets are pretty tight.
Yet as I keep pointing out, my actual practical use cases for linting do not involve constantly re-running the linter over a million files in a tight loop – they involve linting the file I’m editing, linting the files in a changeset, etc. and the current Python linting ecosystem is more than fast enough for that case.
There’s also Sorbet’s observation that performance unlocks new features. “Why would you whatchexec this on the whole code base? Because I now can”.
But what’s the gain from doing that? Remember: the real question is why I should give up forever on being able to extend/customize the linter in exchange for all this speed. Even if the speed unlocks entirely new categories of use cases, it still is useless to me if I can’t then go implement those use cases because the tool became orders of magnitude less extensible/customizable as the cost of the speed.
Long term, I see Ruff exposing a Python scripting interface for this. If slow Python scripting sits on top of fast native core that does 90% o the CPU work, that should be fine!
I think the instant that interface is allowed, you’re going to find that the OMGFAST argument disappears, because there is no way a ruleset written in Python is going to maintain the speed that is the sole and only selling point of Ruff. But by then all the other tools will have been bullied out of existence, so I guess Ruff will just win by default at that point.
they involve linting the file I’m editing, linting the files in a changeset, etc.
Importantly, they also involve only context free linting. Something like “is this function unused in the project?” wouldn’t work in this paradigm. My point is not that you, personally, could benefit from extra speed for your current workflow. It’s rather than there are people who would benefit, and that there are more powerful workflows (eg, typechecking on every keypress) which would become possible
But what’s the gain from doing that?
At minimum, simplicity. I’d much rather just run $ foo than futz with git & xargs to figure out how to run it only on the changed files. Shaving off 10 seconds from the first CI check is also pretty valuable.
I think the instant that interface is allowed, you’re going to find that the OMGFAST argument disappears,
If you do this in the stupidest possible way, then, sure, it’ll probably be even slower than pure Python due to switching back and forth between Python and native. But it seems to me that that custom linting is amenable to proper slicing into CPU-heavy part and scripting on top:
start the API with specifying the pattern to match AST against. Pattern is constructed in Python, but the search is done by the native code. So Python is only involved at all for cases where there’s a syntactic match.
for further semantic lookups (what this identifier resolves to), Python calls into native, and again the heavy lifting is done by native
similarly, semantic lookups probably use some fancy memoization behind the scene, and that’s fully native.
often you want to do varios name based queries (like, all classes named “FooSomething”), and the internal of such text index can also be hidden from Python pretty well.
Importantly, they also involve only context free linting. Something like “is this function unused in the project?” wouldn’t work in this paradigm.
There are already flake8 plugins that detect that sort of thing.
At minimum, simplicity. I’d much rather just run $ foo than futz with git & xargs to figure out how to run it only on the changed files. Shaving off 10 seconds from the first CI check is also pretty valuable.
All the existing tools have a “copy/paste this into your pre-commit config” snippet and then it Just Works. If you are indeed rolling your own solution to run only on the changed files, then I think you should probably pause and familiarize yourself with the current state of the art prior to telling everyone else to abandon it.
Sorry if my comments read as if I am pushing anyone to use Ruff, that definitely wasn’t my intention! Rather, I wanted to share my experience as implementer of similar tools, as that might be an interesting perspective for some.
That being said, I think I want to register a formal prediction that, in five years or so, something of Ruff’s shape (Python code analysis as a cli implemented in a faster language, not necessary specifically Ruff, and not counting already existing PyCharm) would meaningfully eat into Python’s dev tool “market”.
I think Ruff will gain significant “market share”, but for the wrong reasons – not because of any technical superiority or improved user experience, but simply because its hype cycle means people will be pushed into adopting it whether they gain from it or not. I’m already dreading the day someone will inevitably file a “bug” against one of my projects claiming that it’s broken because it hasn’t adopted Ruff yet.
The “not extensible by a $lang programmer” was a reason for not pursuing faster tooling in better suited languages for the web ecosystem, and everything was painfully slow.
In my experience, esbuild (Go) and swc (Rust) are a massive improvement and will trade extensibility for the speed boost every time.
I’ve been using Ruff’s flake8-annotations checks to get a very quick list of missing annotations as I port a codebase. In a watchexec loop it’s substantially faster than getting the same information from MyPy or Pyright.
Likewise, in another codebase ruff --fix has already replaced isort (and flake8 and friends).
I’ve never needed the extensibility, though. I’m curious, what do you do with it?
In a watchexec loop it’s substantially faster than getting the same information from MyPy or Pyright.
I’m not sure why you’d need to run it over the entire codebase in a loop, though. Isn’t that the kind of thing where you generate a report once, and then you only incrementally need to check a file or two at a time as you fix them up?
Likewise, in another codebase ruff –fix has already replaced isort
Again, I don’t get it: isort will fix up imports for you, and my editor is set to do it automatically on file save and if I somehow miss that I have a pre-commit hook running it too. So I’m never in a situation where I need to run it and apply fixes across thousands of files (or if I was, it’d be a one-time thing, not an every-edit thing). So why do I need to switch to another tool?
I’ve never needed the extensibility, though. I’m curious, what do you do with it?
There are lots of popular plugins. For example, pylint on a Django codebase is next to unusable without a plugin to “teach” pylint how some of Django’s metaprogramming works. As far as I can tell, Ruff does not have parity with that. Same for the extremely popular pytest testing framework; without a plugin, pylint gets very confused at some of the dependency-injection “magic” pytest does.
Even without bringing pylint into it, flake8 has a lot of popular plugins for both general purpose and specific library/framework cases, and Ruff has to implement all the rules from those plugins. Which is why it has to have a huge library of built-in rules and explicitly list which flake8 plugins it’s achieved parity with.
I’m not sure why you’d need to run it over the entire codebase in a loop, though. Isn’t that the kind of thing where you generate a report once, and then you only incrementally need to check a file or two at a time as you fix them up?
I like to work from a live list as autoformatting causes line numbers to shift around as annotations increase line length. Really I should set up ruff-lsp.
Again, I don’t get it: isort will fix up imports for you, and my editor is set to do it automatically on file save and if I somehow miss that I have a pre-commit hook running it too. So I’m never in a situation where I need to run it and apply fixes across thousands of files (or if I was, it’d be a one-time thing, not an every-edit thing). So why do I need to switch to another tool?
I don’t use pre-commit because it’s excruciatingly slow. These things are really noticeable to me — maybe you have a faster machine?
I don’t use pre-commit because it’s excruciatingly slow. These things are really noticeable to me — maybe you have a faster machine?
Can you quantify “excruciatingly slow”? Like, “n milliseconds to run when k files staged for commit” quantification?
Because I’ve personally never noticed it slowing me down. I work on codebases of various sizes, doing changesets of various sizes, on a few different laptops (all Macs, of varying vintages). Maybe it’s just that I zone out a bit while I’m mentally composing the commit message, but I’ve never found myself waiting for pre-commit to finish before being able to start typing the message (fwiw my workflow is in Emacs, using magit as the git interface and an Emacs buffer to draft and edit the commit message, so actually writing the message is always the last part of the process for me).
I gave it another try and it looks like it’s not so bad after the first time. The way it’s intermittently slow (anytime the checkers change) is frustrating, but probably tolerable given the benefits.
I think my impression of slowness came from Twisted where it is used to run the lint over all files. This is very slow.
The way it’s intermittently slow (anytime the checkers change) is frustrating
My experience is that the list of configured checks changes relatively rarely – I get the set of them that I want, and leave it except for the occasional version bump of a linter/formatter. But it’s also not really pre-commit’s fault that changing the set of checks is slow, because changing it involves, under the hood, doing a git clone and then pip install (from the cloned repo) of the new hook. How fast or slow that is depends on your network connection and the particular repo the hook lives in.
I’ve never needed the extensibility, though. I’m curious, what do you do with it?
Write bespoke lints for codebase specific usage issues.
Most of them should probably be semgrep rules, but semgrep is not on the CI, it’s no speed demon either, and last I checked it has pretty sharp edges where it’s pretty easy to create rules which don’t work it complex cases.
PyLint is a lot more work, but lints are pretty easy to test and while the API is I’ll documented it’s quite workable and works well once you’ve gotten it nailed down.
Ah, so you and ubernostrum are optimizing workflows on a (single?) (large?) codebase, and you’re after a Pylint, rather than a pyflakes/flake8.
I’m coming at this from an OSS-style many-small-repos perspective. I prefer a minimally-configurable tool so that the DX is aligned across repositories. I don’t install and configure many flake8 plugins because that increases per-repo maintenance burden (e.g., with flake8 alone W503/W504 debacle caused a bunch of churn as the style rules changed — thank goodness we now have Black!). Thus, I’m happy to drop additional tools like isort. So to me Ruff adds to the immediately-available capabilities without increasing overhead — seems like a great deal!
It seems like Ruff might slot into your workflow as a flake8 replacement, but you get a lot from Pylint, so I’d keep using the latter. You could disable all the style stuff and use Pylint in a slower loop like a type checker.
pylint is, in practice, very memory hungry and frankly slow.
Now i can’t go from there to recommending ruff for the simple fact that ruff is not checking nearly enough stuff to be considered a replacement IMO. Not yet at least. But I’ll be happy to see better stuff happening in this space (disclaimer: I’m writing a rust-based pylint drop-in replacement. Mostly for practice but also because I really suffered under pylint’s perf issues in a past life)
My admiration for ruff comes from the fact that I now have a single tool and a single configuration place. I don’t have to chase how to configure 10 different tools to do linting and ensuring that my python project has some guardrails. For example, my big annoyance with flake8 is that I can’t add it’s config in pyproject.toml, it has to be a separate file. I really, really, just want to flip the switch and have various checks done on the codebase, and not scour the internet on how to configure these tools, since each has it’s own (quite valid) interpretation of what’s the right way to do things. I just want to stay away from ever creating setup.py and all those other things I never understood why are needed to package some interpreted code (my dislike for python’s packaging is leaking here :)).
I’m curious, what do you need to change in the tools replaced by ruff? What additional checks do you need to implement?
I personally do not care about the config file thing, and I wish people would stop bullying the flake8 dude about it. Way too many people, back when pyproject.toml was introduced for a completely different purpose than this, still treated its existence as meaning “all prior config approaches are now illegal, harass everyone you can find until they give up or give in”. Which is what people have basically tried to do to flake8, and I respect the fact that the maintainer laid out clear criteria for a switch to pyproject.toml and then just aggressively locked and ignored every request that doesn’t meet those criteria.
I’m curious, what do you need to change in the tools replaced by ruff? What additional checks do you need to implement?
I already gave a reply to someone else talking about the whole ecosystem of plugins out there for flake8 and pylint, and Ruff is not at parity with them. So even if I wanted to switch to Ruff I would not be able to – it lacks the checks I rely on, and lacks the ability for me to go implement those checks.
I’ve been slowly but surely giving up on Python for some time, and I’ve often struggled to articulate the reasons why. But having just read some of the flake8 pyproject stuff, it’s hit me that most of it could be described as bullying at some level or other.
Python itself arguably bullies its users, with things like the async -> ensure_future change, sum’s special case for str because the devs don’t like it, blah blah. (I want to say something about the packaging situation here, and how much of a pain in the ass it is to maintain a Python project to popular opinion standards in 202x, but I recognise that not all of this is deliberate.) Black’s founding principle is that bludgeoning people into accepting a standard is better than wasting time letting them have preferences. Long ago, when I frequented #python, SOP whenever anyone wanted to use sockets was to bully them into using an external dependency instead. And longer ago, when I frequented python-ideas, ideas that the in-group didn’t take to were made, along with their champions, to run ridiculous gauntlets of nitpicking and whataboutism.
Of course none of the exponents of this behaviour identify it as bullying, but then who would? The results are indistinguishable whether they’re being a dick, evangelizing best practices or just trying to save everyone’s time.
In short I think that, if you don’t want to be bullied into adopting a solution that doesn’t really work for you, you are in the wrong ecosystem.
Some of us use pyflakes on its own, and are thus used to the zero-configuration experience. The configurability of pylint is a net negative for me; it leads to bikeshedding over linter configuration.
This is entirely reasonable. In my case, I started a new job and new project, and I’m not invested heavily in the existing python based toolchain, so ruff was the right choice for us. I don’t like the way these sorts of minor differences get amplified up into existential crises anyway. And no, I’m not new on the internet, just tired of it all.
A thousand times this. $CUSTOMER had wildly divergent coding styles in their repos, and the project to streamline it meant configuring these traditional small tools and their plugins to conform to how PyCharm did things because it’s quite opinionated. And popular among people who are tired of it all.
The tooling included darker, which is fine, though I personally do not like all of black’s choices.
Eventually the whole codebase was blackened all at once and ruff replaced everything else.
The pre-commit is fast and my preferences aside, good arguments can be made for those two tools.
It is what it is, a business decision, and the right way to deal with it is to not elevate it to an existential crisis.
Outside the business world, if I had popular projects, I’d dislike ending up with PRs in the black style if the project wasn’t like that. Or having to set up glue for reformatting.
This is probably how all monopolization happens; people become rightfully tired of being ever-vigilant, and inevitably something bad will come out of the monopoly.
Like not getting OSS contributions because of the project’s formatting guidelines.
100% agree. This reminds me of the old “You have a problem and decide to use regexes to solve it. Now you have two problems.” Yes, your linting is faster but now “it’s basically not extensible by a Python programmer”, which means it’s more difficult for people to maintain their own tools.
ISTM, that Python is moving from being a glue language for C to being a glue language for Rust. It makes sense, but also makes me wonder why not just skip the Python part and go straight to Rust?
Maybe the ecosystem has matured since I tried to port some numerical python code to rust, but I found that the ~factor of two speedups I got in the end working in rust wasn’t worth the substantial productivity loss from working in a more restrictive language with a less mature ecosystem.
Python still uniquely possess the primary artifacts of multiple, giant multiple decade research and engineering projects in data analysis, ML, and scientific computation. While others are constantly fast-following, it’s hard to imagine it being replaced by Rust in any reasonable timeframe.
And that’s assuming people actually like using Rust for this sort of work which is unclear at best.
It makes sense, but also makes me wonder why not just skip the Python part and go straight to Rust?
Two aspects to this are that:
a) Rust is a compiled language, for many use cases such as small scripts having things like a repl to interactively explore a library is just nice, and you most likely don’t care about things like ownership in such contexts.
b) the demographic that uses python libraries is much bigger than the demographic that maintains python libraries. People maintaining python & rust libraries are then only a subset of the latter.
Every time Ruff comes up, the fact that it’s super-hyper-mega-fast is promoted as making up for the fact that it’s basically not extensible by a Python programmer (while the tools it wants to replace all easily are extensible in Python).
But the speed benefit only shows up when checking a huge number of files in a single run. And even on a large codebase:
So I’m still not sure why I should give up easy extensibility for speed that seems to offer me no actual practical benefit.
A bunch of thoughts here!
For these kinds of tools performance (both speed and memory usage) matters a lot, because codebases are effectively unbounded in size, and because for interactive use, latency budgets are pretty tight. There’s also Sorbet’s observation that performance unlocks new features. “Why would you whatchexec this on the whole code base? Because I now can”.
Now, if we speak strictly about syntax-based formatting and linting, you can get quite a bit of performance from the embarrassingly parallel nature of the task. But of course you want to do cross-file analysis, type inference, duplicate detection and what not.
The amount of things you can do with a good static analysis base is effectively unbounded. At this point, maybe Java and C# are coming to the point of saturation, but everything else feels like a decade behind. The primary three limiting factors to deliver these kinds of tools are:
This is high-investment, high-value thing, which requires a great foundation. And I would actually call that, rather than today’s raw performance, the most important feature of Ruff. We can start from fast linting, and then move to import analysis, type inference, full LSP and what not.
From my point of view, Python’s attempt to self-host all dev tools is a strategic blunder. Python really doesn’t have performance characteristics to move beyond per file listing, so it’s not surprising that, eg, pyright does its own thing rather than re-use existing ecosystem.
All that being said, extensibility is important! And Python is a fine language for that. Long term, I see Ruff exposing a Python scripting interface for this. If slow Python scripting sits on top of fast native core that does 90% o the CPU work, that should be fine!
Yet as I keep pointing out, my actual practical use cases for linting do not involve constantly re-running the linter over a million files in a tight loop – they involve linting the file I’m editing, linting the files in a changeset, etc. and the current Python linting ecosystem is more than fast enough for that case.
But what’s the gain from doing that? Remember: the real question is why I should give up forever on being able to extend/customize the linter in exchange for all this speed. Even if the speed unlocks entirely new categories of use cases, it still is useless to me if I can’t then go implement those use cases because the tool became orders of magnitude less extensible/customizable as the cost of the speed.
I think the instant that interface is allowed, you’re going to find that the OMGFAST argument disappears, because there is no way a ruleset written in Python is going to maintain the speed that is the sole and only selling point of Ruff. But by then all the other tools will have been bullied out of existence, so I guess Ruff will just win by default at that point.
Importantly, they also involve only context free linting. Something like “is this function unused in the project?” wouldn’t work in this paradigm. My point is not that you, personally, could benefit from extra speed for your current workflow. It’s rather than there are people who would benefit, and that there are more powerful workflows (eg, typechecking on every keypress) which would become possible
At minimum, simplicity. I’d much rather just run
$ foothan futz with git & xargs to figure out how to run it only on the changed files. Shaving off 10 seconds from the first CI check is also pretty valuable.If you do this in the stupidest possible way, then, sure, it’ll probably be even slower than pure Python due to switching back and forth between Python and native. But it seems to me that that custom linting is amenable to proper slicing into CPU-heavy part and scripting on top:
There are already flake8 plugins that detect that sort of thing.
All the existing tools have a “copy/paste this into your pre-commit config” snippet and then it Just Works. If you are indeed rolling your own solution to run only on the changed files, then I think you should probably pause and familiarize yourself with the current state of the art prior to telling everyone else to abandon it.
Sorry if my comments read as if I am pushing anyone to use Ruff, that definitely wasn’t my intention! Rather, I wanted to share my experience as implementer of similar tools, as that might be an interesting perspective for some.
That being said, I think I want to register a formal prediction that, in five years or so, something of Ruff’s shape (Python code analysis as a cli implemented in a faster language, not necessary specifically Ruff, and not counting already existing PyCharm) would meaningfully eat into Python’s dev tool “market”.
I think Ruff will gain significant “market share”, but for the wrong reasons – not because of any technical superiority or improved user experience, but simply because its hype cycle means people will be pushed into adopting it whether they gain from it or not. I’m already dreading the day someone will inevitably file a “bug” against one of my projects claiming that it’s broken because it hasn’t adopted Ruff yet.
The “not extensible by a $lang programmer” was a reason for not pursuing faster tooling in better suited languages for the web ecosystem, and everything was painfully slow.
In my experience, esbuild (Go) and swc (Rust) are a massive improvement and will trade extensibility for the speed boost every time.
I’ve been using Ruff’s
flake8-annotationschecks to get a very quick list of missing annotations as I port a codebase. In a watchexec loop it’s substantially faster than getting the same information from MyPy or Pyright.Likewise, in another codebase
ruff --fixhas already replaced isort (andflake8and friends).I’ve never needed the extensibility, though. I’m curious, what do you do with it?
I’m not sure why you’d need to run it over the entire codebase in a loop, though. Isn’t that the kind of thing where you generate a report once, and then you only incrementally need to check a file or two at a time as you fix them up?
Again, I don’t get it:
isortwill fix up imports for you, and my editor is set to do it automatically on file save and if I somehow miss that I have a pre-commit hook running it too. So I’m never in a situation where I need to run it and apply fixes across thousands of files (or if I was, it’d be a one-time thing, not an every-edit thing). So why do I need to switch to another tool?There are lots of popular plugins. For example, pylint on a Django codebase is next to unusable without a plugin to “teach” pylint how some of Django’s metaprogramming works. As far as I can tell, Ruff does not have parity with that. Same for the extremely popular pytest testing framework; without a plugin, pylint gets very confused at some of the dependency-injection “magic” pytest does.
Even without bringing pylint into it, flake8 has a lot of popular plugins for both general purpose and specific library/framework cases, and Ruff has to implement all the rules from those plugins. Which is why it has to have a huge library of built-in rules and explicitly list which flake8 plugins it’s achieved parity with.
I like to work from a live list as autoformatting causes line numbers to shift around as annotations increase line length. Really I should set up
ruff-lsp.I don’t use pre-commit because it’s excruciatingly slow. These things are really noticeable to me — maybe you have a faster machine?
Can you quantify “excruciatingly slow”? Like, “
nmilliseconds to run whenkfiles staged for commit” quantification?Because I’ve personally never noticed it slowing me down. I work on codebases of various sizes, doing changesets of various sizes, on a few different laptops (all Macs, of varying vintages). Maybe it’s just that I zone out a bit while I’m mentally composing the commit message, but I’ve never found myself waiting for pre-commit to finish before being able to start typing the message (fwiw my workflow is in Emacs, using
magitas the git interface and an Emacs buffer to draft and edit the commit message, so actually writing the message is always the last part of the process for me).I gave it another try and it looks like it’s not so bad after the first time. The way it’s intermittently slow (anytime the checkers change) is frustrating, but probably tolerable given the benefits.
I think my impression of slowness came from Twisted where it is used to run the lint over all files. This is very slow.
Thanks for prompting me to give it another look!
My experience is that the list of configured checks changes relatively rarely – I get the set of them that I want, and leave it except for the occasional version bump of a linter/formatter. But it’s also not really pre-commit’s fault that changing the set of checks is slow, because changing it involves, under the hood, doing a
git cloneand thenpip install(from the cloned repo) of the new hook. How fast or slow that is depends on your network connection and the particular repo the hook lives in.Write bespoke lints for codebase specific usage issues.
Most of them should probably be semgrep rules, but semgrep is not on the CI, it’s no speed demon either, and last I checked it has pretty sharp edges where it’s pretty easy to create rules which don’t work it complex cases.
PyLint is a lot more work, but lints are pretty easy to test and while the API is I’ll documented it’s quite workable and works well once you’ve gotten it nailed down.
Ah, so you and ubernostrum are optimizing workflows on a (single?) (large?) codebase, and you’re after a Pylint, rather than a pyflakes/flake8.
I’m coming at this from an OSS-style many-small-repos perspective. I prefer a minimally-configurable tool so that the DX is aligned across repositories. I don’t install and configure many flake8 plugins because that increases per-repo maintenance burden (e.g., with flake8 alone W503/W504 debacle caused a bunch of churn as the style rules changed — thank goodness we now have Black!). Thus, I’m happy to drop additional tools like
isort. So to me Ruff adds to the immediately-available capabilities without increasing overhead — seems like a great deal!It seems like Ruff might slot into your workflow as a flake8 replacement, but you get a lot from Pylint, so I’d keep using the latter. You could disable all the style stuff and use Pylint in a slower loop like a type checker.
I have both large and small codebases. I do use pylint in addition to flake8 – my usual approach is flake8 in pre-commit because it’s a decent quick check, and pylint in CI because it’s comprehensive. I’ve written up my approach to “code quality” tooling and checks in detail, and you can also see an example repository using that approach.
pylint is, in practice, very memory hungry and frankly slow.
Now i can’t go from there to recommending ruff for the simple fact that ruff is not checking nearly enough stuff to be considered a replacement IMO. Not yet at least. But I’ll be happy to see better stuff happening in this space (disclaimer: I’m writing a rust-based pylint drop-in replacement. Mostly for practice but also because I really suffered under pylint’s perf issues in a past life)
My admiration for ruff comes from the fact that I now have a single tool and a single configuration place. I don’t have to chase how to configure 10 different tools to do linting and ensuring that my python project has some guardrails. For example, my big annoyance with flake8 is that I can’t add it’s config in
pyproject.toml, it has to be a separate file. I really, really, just want to flip the switch and have various checks done on the codebase, and not scour the internet on how to configure these tools, since each has it’s own (quite valid) interpretation of what’s the right way to do things. I just want to stay away from ever creatingsetup.pyand all those other things I never understood why are needed to package some interpreted code (my dislike for python’s packaging is leaking here :)).I’m curious, what do you need to change in the tools replaced by ruff? What additional checks do you need to implement?
I personally do not care about the config file thing, and I wish people would stop bullying the flake8 dude about it. Way too many people, back when
pyproject.tomlwas introduced for a completely different purpose than this, still treated its existence as meaning “all prior config approaches are now illegal, harass everyone you can find until they give up or give in”. Which is what people have basically tried to do to flake8, and I respect the fact that the maintainer laid out clear criteria for a switch topyproject.tomland then just aggressively locked and ignored every request that doesn’t meet those criteria.I already gave a reply to someone else talking about the whole ecosystem of plugins out there for flake8 and pylint, and Ruff is not at parity with them. So even if I wanted to switch to Ruff I would not be able to – it lacks the checks I rely on, and lacks the ability for me to go implement those checks.
I’ve been slowly but surely giving up on Python for some time, and I’ve often struggled to articulate the reasons why. But having just read some of the flake8 pyproject stuff, it’s hit me that most of it could be described as bullying at some level or other.
Python itself arguably bullies its users, with things like the
async->ensure_futurechange, sum’s special case for str because the devs don’t like it, blah blah. (I want to say something about the packaging situation here, and how much of a pain in the ass it is to maintain a Python project to popular opinion standards in 202x, but I recognise that not all of this is deliberate.) Black’s founding principle is that bludgeoning people into accepting a standard is better than wasting time letting them have preferences. Long ago, when I frequented #python, SOP whenever anyone wanted to use sockets was to bully them into using an external dependency instead. And longer ago, when I frequented python-ideas, ideas that the in-group didn’t take to were made, along with their champions, to run ridiculous gauntlets of nitpicking and whataboutism.Of course none of the exponents of this behaviour identify it as bullying, but then who would? The results are indistinguishable whether they’re being a dick, evangelizing best practices or just trying to save everyone’s time.
In short I think that, if you don’t want to be bullied into adopting a solution that doesn’t really work for you, you are in the wrong ecosystem.
Some of us use
pyflakeson its own, and are thus used to the zero-configuration experience. The configurability ofpylintis a net negative for me; it leads to bikeshedding over linter configuration.This is entirely reasonable. In my case, I started a new job and new project, and I’m not invested heavily in the existing python based toolchain, so
ruffwas the right choice for us. I don’t like the way these sorts of minor differences get amplified up into existential crises anyway. And no, I’m not new on the internet, just tired of it all.A thousand times this. $CUSTOMER had wildly divergent coding styles in their repos, and the project to streamline it meant configuring these traditional small tools and their plugins to conform to how PyCharm did things because it’s quite opinionated. And popular among people who are tired of it all.
The tooling included
darker, which is fine, though I personally do not like all ofblack’s choices.Eventually the whole codebase was blackened all at once and ruff replaced everything else.
The pre-commit is fast and my preferences aside, good arguments can be made for those two tools.
It is what it is, a business decision, and the right way to deal with it is to not elevate it to an existential crisis.
Outside the business world, if I had popular projects, I’d dislike ending up with PRs in the black style if the project wasn’t like that. Or having to set up glue for reformatting.
This is probably how all monopolization happens; people become rightfully tired of being ever-vigilant, and inevitably something bad will come out of the monopoly.
Like not getting OSS contributions because of the project’s formatting guidelines.
100% agree. This reminds me of the old “You have a problem and decide to use regexes to solve it. Now you have two problems.” Yes, your linting is faster but now “it’s basically not extensible by a Python programmer”, which means it’s more difficult for people to maintain their own tools.
ISTM, that Python is moving from being a glue language for C to being a glue language for Rust. It makes sense, but also makes me wonder why not just skip the Python part and go straight to Rust?
Maybe the ecosystem has matured since I tried to port some numerical python code to rust, but I found that the ~factor of two speedups I got in the end working in rust wasn’t worth the substantial productivity loss from working in a more restrictive language with a less mature ecosystem.
For a lot of code that’s where I am, personally. For data analysis I’m still much happier with Python.
Python still uniquely possess the primary artifacts of multiple, giant multiple decade research and engineering projects in data analysis, ML, and scientific computation. While others are constantly fast-following, it’s hard to imagine it being replaced by Rust in any reasonable timeframe.
And that’s assuming people actually like using Rust for this sort of work which is unclear at best.
Two aspects to this are that:
a) Rust is a compiled language, for many use cases such as small scripts having things like a repl to interactively explore a library is just nice, and you most likely don’t care about things like ownership in such contexts.
b) the demographic that uses python libraries is much bigger than the demographic that maintains python libraries. People maintaining python & rust libraries are then only a subset of the latter.
Ruff’s been a great addition to the workflow at my day job. If I do Python at another place, I’d be looking to add it there.