I recently code-switched from working primarily in typescript to python3, and this article is a great summary of things I had to learn the hard way. Excellent work.
This is a great article! It’s very even handed in its recommendations, which isn’t something you always see where type annotations are concerned.
I like them but also appreciate the perspective of a LOT of smart people who don’t and say that they clutter the syntax and aren’t necessary for a language like Python.
So I use them for my own projects but avoid them at work where we have a senior dev who H-A-T-E-S them.
I’m glad you liked it. Thanks for taking the time to leave comment saying so, that is much appreciated.
People do have strong opinions about types. I used to be a typing enthusiast (with Haskell) but I think my view is now that formal methods are of only moderate value in most contexts. For example, how many systems for demonstrating correctness are predicated on having an accurate spec?
Re: cluttering the language. I think to some extent that is a fair charge. There is a lot of noise generated by typing annotations, especially when you’re wrangling nested generics (Dict[Dict[str, List[MyClass]]]). This is one of the long list of cons but there is an equally long list of pros. I would be a bit sad if every tiny Python program started to have type annotations - they are ugly and complex baggage - but I think that lots of commercial code (especially library code) really benefits from the increased clarity and the static analysis.
One of the things that made me start to like them is actually kind of non sequitor from a correctness standpoint: They give my IDE the ability to auto-complete EVERYTHING!
ducks and waits for the barrage of rotten tomatoes
I often feel like often this clutter is a sign that something needs to be changed and refactored in a proper dataclass or at least a type alias anyway. E.g. something like Dict[str, Dict[str, List[MyClass]]] is not very pleasant to work with in runtime either. Unless its use is very localized (e.g. lambda or local def return type), but in that case you can get away without type annotations I guess (unless you’re using --strict).
Very good point, especially confusing with stuff like –namespace-packages, etc. Just googled and apparently mypy got coverage report. Wonder if you tried it?
The coverage reports are news to me…I just ran the html coverage report on a small codebase with a lowish strictness level and it’s interesting reading. Hadn’t occurred to me before that log messages aren’t typechecked but I suppose that makes sense given their unusual type signature (eg log.info("a value: %s", that_value).
I don’t have much use for the output formats as I normally run mypy in emacs’ compile mode - it turns the errors into clickable links so you can jump to the error
I’m working with a Python codebase I’m trying to get typechecked, and I’m finding the art is getting all the configuration exactly right on what/how to include dependencies, Any tolerance, the Tangle, and more. This is helpful.
I’ve also enjoyed playing around with pyre-check, a tool to do static analysis and typechecking. It’s parallel OCaml, so it’s pretty fast, though the linked Dropbox post was very instructive on how mypy is feasible for large projects a well.
In my experience most projects above a certain size do have a tangle. I think the way that import works in Python is probably one of the greater matters for regret about the language as a whole.
pyre-check is on my list of things to try. There is also pyright which is worth investigation too.
I agree that most large Python programs have dependency issues (though maybe not worse than large C++ programs). And Python imports are overly flexible / dynamic.
I wanted to diligently avoid this for https://www.oilshell.org and I’m pretty happy with the results.
It uses a pure dependency injection style. So rather than every module potentially importing every other, almost every module is imported from the driver, and the objects are instantiated and wired together in main().
When I added MyPy type annotations, I hit all the “declaration deps” under if TYPE_CHECKING. So I know what are hard dependencies and where I just use the name of types – similar to #include vs. forward declaration in C.
I also was diligent in handling circular dependencies (which are inherent in a language evaluator). I have never used this “circular deps” pattern but I’m pretty happy with it:
It turned out to have some unexpected benefits when translating Oil to C++. I can slice up the program very easily and translate just that part – there is no tangle.
For example I translated the parser without translating the rest of the program, by writing a different main() and using MyPy annotations:
This is maybe not idiomatic Python but I think it’s better. Most Python programs rely on too many globals. Decorators seem to be a thing that encourage this and I generally avoid them. If you’re using many frameworks they’re hard to avoid though.
The next version of our flagship product is written in 100% type-{hinted,checked} Python 3 for the most part (there are lower-level parts in C). All told it’s around 10,000 lines of Python. This simply would not have been feasible without mypy.
I bet it was possible though: I’ve worked on projects much bigger than 10k lines without types. Tools like mypy help but it isn’t impossible to work without them. Facebook was once just an enormous PHP app (in some ways, it still is, but better structured). There are a lot of similar examples.
It’s less general than typed dataclasses, though I expect it uses less memory. I personally dislike the normal NamedTuple quite a lot in Python (always feels like a poor version of a slotted class) so I am not keen on the typed version either. YMMV of course.
I recently code-switched from working primarily in typescript to python3, and this article is a great summary of things I had to learn the hard way. Excellent work.
This is a great article! It’s very even handed in its recommendations, which isn’t something you always see where type annotations are concerned.
I like them but also appreciate the perspective of a LOT of smart people who don’t and say that they clutter the syntax and aren’t necessary for a language like Python.
So I use them for my own projects but avoid them at work where we have a senior dev who H-A-T-E-S them.
I’m glad you liked it. Thanks for taking the time to leave comment saying so, that is much appreciated.
People do have strong opinions about types. I used to be a typing enthusiast (with Haskell) but I think my view is now that formal methods are of only moderate value in most contexts. For example, how many systems for demonstrating correctness are predicated on having an accurate spec?
Re: cluttering the language. I think to some extent that is a fair charge. There is a lot of noise generated by typing annotations, especially when you’re wrangling nested generics (
Dict[Dict[str, List[MyClass]]]). This is one of the long list of cons but there is an equally long list of pros. I would be a bit sad if every tiny Python program started to have type annotations - they are ugly and complex baggage - but I think that lots of commercial code (especially library code) really benefits from the increased clarity and the static analysis.One of the things that made me start to like them is actually kind of non sequitor from a correctness standpoint: They give my IDE the ability to auto-complete EVERYTHING!
ducks and waits for the barrage of rotten tomatoes
I often feel like often this clutter is a sign that something needs to be changed and refactored in a proper dataclass or at least a type alias anyway. E.g. something like
Dict[str, Dict[str, List[MyClass]]]is not very pleasant to work with in runtime either. Unless its use is very localized (e.g. lambda or local def return type), but in that case you can get away without type annotations I guess (unless you’re using--strict).Good article, thanks!
Very good point, especially confusing with stuff like –namespace-packages, etc. Just googled and apparently mypy got coverage report. Wonder if you tried it?
Also error codes for more precise control
Couple more very useful flags are
--pretty--show-error-contextThe coverage reports are news to me…I just ran the html coverage report on a small codebase with a lowish strictness level and it’s interesting reading. Hadn’t occurred to me before that log messages aren’t typechecked but I suppose that makes sense given their unusual type signature (eg
log.info("a value: %s", that_value).I don’t have much use for the output formats as I normally run mypy in emacs’ compile mode - it turns the errors into clickable links so you can jump to the error
This is fantastic, thank you 😄
I’m working with a Python codebase I’m trying to get typechecked, and I’m finding the art is getting all the configuration exactly right on what/how to include dependencies,
Anytolerance, the Tangle, and more. This is helpful.I’ve also enjoyed playing around with pyre-check, a tool to do static analysis and typechecking. It’s parallel OCaml, so it’s pretty fast, though the linked Dropbox post was very instructive on how
mypyis feasible for large projects a well.Thanks I’m glad it’s helpful.
In my experience most projects above a certain size do have a tangle. I think the way that import works in Python is probably one of the greater matters for regret about the language as a whole.
pyre-check is on my list of things to try. There is also pyright which is worth investigation too.
I agree that most large Python programs have dependency issues (though maybe not worse than large C++ programs). And Python imports are overly flexible / dynamic.
I wanted to diligently avoid this for https://www.oilshell.org and I’m pretty happy with the results.
It uses a pure dependency injection style. So rather than every module potentially importing every other, almost every module is imported from the driver, and the objects are instantiated and wired together in
main().https://github.com/oilshell/oil/blob/master/bin/oil.py
When I added MyPy type annotations, I hit all the “declaration deps” under
if TYPE_CHECKING. So I know what are hard dependencies and where I just use the name of types – similar to #include vs. forward declaration in C.I also was diligent in handling circular dependencies (which are inherent in a language evaluator). I have never used this “circular deps” pattern but I’m pretty happy with it:
https://github.com/oilshell/oil/blob/master/core/vm.py
It turned out to have some unexpected benefits when translating Oil to C++. I can slice up the program very easily and translate just that part – there is no tangle.
For example I translated the parser without translating the rest of the program, by writing a different
main()and using MyPy annotations:http://www.oilshell.org/blog/2019/12/09.html#oil-native-shows-how-well-optimize-oil
This is maybe not idiomatic Python but I think it’s better. Most Python programs rely on too many globals. Decorators seem to be a thing that encourage this and I generally avoid them. If you’re using many frameworks they’re hard to avoid though.
The next version of our flagship product is written in 100% type-{hinted,checked} Python 3 for the most part (there are lower-level parts in C). All told it’s around 10,000 lines of Python. This simply would not have been feasible without mypy.
I bet it was possible though: I’ve worked on projects much bigger than 10k lines without types. Tools like mypy help but it isn’t impossible to work without them. Facebook was once just an enormous PHP app (in some ways, it still is, but better structured). There are a lot of similar examples.
[Comment removed by author]
With discipline it is not hard to go over 10,000 LOC using untyped Python. Type hints definitely help though.
Surprised there’s no mention of
typing.NamedTuple, that’s like my #1 go-to for “typed” Python.It’s less general than typed dataclasses, though I expect it uses less memory. I personally dislike the normal NamedTuple quite a lot in Python (always feels like a poor version of a slotted class) so I am not keen on the typed version either. YMMV of course.