This is a pretty rough article, which basically boils down to “Python type hints are optional and the interpreter doesn’t enforce them”. Which is… something you discover very early on in studying them (it was even introduced as “gradual typing”). If the author’s hope was for Python to have become a mandatory statically-typed language, I guess it’s a disappointment, but I don’t know of anyone who approaches Python type hints in good faith and has such an expectation.
There’s also the problem with this:
In Python 3.5, “type hints” were introduced.
To recap:
Syntatic support for annotating function and method definitions was added in Python 3.0.
Syntactic support for annotating variables and class members was added in Python 3.6.
What happened in 3.5 was that the syntactic feature of annotations gained the official semantic meaning of type hinting, and the typing module was added to the standard library to support that use case (previously, annotations had been officially un-semantic, with an encouragement to experiment with possible use cases for them, but people overwhelmingly used them for type hints).
Meanwhile, if you want something to complain about, here’s one: type hints and the surrounding tooling were added to Python in a way that strongly encouraged nominal typing, but real-world Python is much more “interfacely” typed, and it took years for the tooling to even begin to shift toward properly supporting that. Which is one reason why I’ve held off adding type hints to most of my own personal open-source Python projects — when the type hints don’t match the way the language is used, they’re not helpful.
“gradual typing” does imply that the type hints are optional, but when they are present they will be enforced. the real issue here is that “typed python” is a gradually typed language built on top of python, in much the same way typescript is built on top of javascript, and that the python interpreter is purely a python interpreter, not a typed-python one. mypy/pytype/pyright/etc are the typed-python typecheckers.
also, this is an interesting paper demonstrating that typed-python is really several dialects, each one depending on the specific type checker enforcing it.
Most of the “gradual” (portions of the code base can be untyped) type systems that are being applied to languages with large communities (TypeScript, Python’s type hints, Dart, etc.) are also “optional” (no effect on run-time behaviour), but they can vary in how clearly they communicate the optional-ness.
Some gradual type systems allow the optional-ness to be configured (e.g. Sorbet for Ruby), so you can choose whether to have run-time checks.
The term “optional” for speaking about “run-time checks” seems largely confined to academic literature, and doesn’t really seem to appear in user-facing type system documentation. A 2018 ECOOP paper from Chung et. al includes a helpful table comparing various gradual type systems by attributes like optional as well as others.
I do wish it were easier to be more precise type system behaviour, but unfortunately most adjectives placed in front of type systems are quite fuzzy with multiple meanings… 😬
thanks for the pointer, i skimmed the paper but it seems well worth a deeper read. (it’s a bit out of date wrt python though). the distinction i was trying to get at is where it says
The optional approach, chosen by TypeScript, Dart, and Hack, amounts to static type checking followed by type erasure.
python, by contrast, does not even do static type checking, it says that the type annotations are treated as metadata which you may check using an external tool if you so choose, or introspect yourself within your code, but they are otherwise unused[*] at both compile time and runtime.
[*] except for the inconsistency that if you have x: A the interpreter tries to look up the class A and store that in the metadata, rather than the string "A", so if A is not a valid class the compiler will complain. this is being fixed in an upcoming version of python, where all annotations will be treated as strings.
I’m wondering if Typescript is actually a major source of confusion here, because TypeScript is effectively a separate language that compiles down to JavaScript, while Python with type hints is still Python, not some sort of separate statically-typed language that compiles to Python.
And really, the same complaint applies for TypeScript, because TypeScript does not change JavaScript — the output JS is still a dynamically-typed language and has no useful memory or enforcement of the types you so carefully annotated into your TypeScript source.
In the same way, adding type hints to Python source code via the annotation syntax does not change Python-the-language to behave differently; it simply means that now you can run a tool to statically analyze your annotations and produce warnings/errors/etc. you can act on.
My guess is that the mypy developers reasonably thought that introducing such a major feature needs to be done gradually. If you see 100 errors when you run mypy the first time, that could be fixable in an afternoon. If you see 10,000, you just give up and stop trying.
To get to a good place, start with the default mypy configuration, then work your way up to strict compliance (see mypy --help and apply each of the --strict flags separately), and finally just use mypy --strict, which fixes most of the issues mentioned in this post. Or change to a compiled language.
I can’t understand why the author (a) wants to use Python, a dynamically typed language (b) wants to use type checking but (c) will give up on type checking that isn’t 100%-guaranteed-enforced
Add mypy to your CI. Congrats, now it’s enforced. There goes whole sections of this post.
Any is an escape hatch because, shocker, Python is dynamically typed (ie, it’s of type PyObject* but shhhh). Its existence doesn’t mean that things, in your own codebase, that are hinted aren’t helpful! Or at least it’s very strange to argue that “anything could be Any, and Any is bad, so let’s not use hints at all, so now everything is Any!”
I have never seen a statically typed language that didn’t have optional dynamic typing. Java has Object, Haskell has Dynamic (and a few other options), C has void*. Any is hardly news to the space
We were very early adopters of mypy, and I’m happy for it overall. It’s like a linter on steroids, for lack of a better term.
A frustrating issue is that because of how Any works and with stuff like third party deps, you end up with weird situations where entire functions just don’t get really type checked, and changing some part of the code in one file suddenly makes a seemingly unrelated function be type checked.
It’s good that errors show up, but it’s frustrating that you end up having to clean up “other people’s changes”.
My “project to do if I had infinite time” would be to fork the typescript type checker to support Python. structural subtying, very good union type resolution…. typescript’s just way better at answering a lot of the questions I have about types for my “standard enterprise app”. That being said mypy is a serious project and I think people are going to be adding types over time, if only because they are at least halfway-good documentation.
Yeah, Python isn’t going to check the types for you. There’s still a lot of value. By labeling the types, you are at least expressing intent. So it’s not “well what is the right thing to pass in”, but “I am passing in the wrong thing”, or “the type annotation is in fact wrong”. Which sounds equivalent, but is a bit more restricted and easier to think about.
TypeScript is the same, a developer puts an any in some place “just for now” and it spreads out tendrils into otherwise correct code, turning off typechecking and turning solid typedefs into lies at runtime.
I’m still not sure what the answer is, although allowing unknown to be used in places where you want to say “I don’t know” rather than any meaning “anything goes, do as you please” seems to be a good start.
I know typescript has a variant of this issue, but there’s something incidental (I believe) happening with mypy that makes this problem worse. It’s very weird but frequent in the large codebase I work on.
There is also a practical thing; the typescript type system captures more patterns than the mypy type system, so less stuff falls back to any.
In any case I am glad gradual typing exists overall.
IIRC TypeScript defaults to being way more aggressive with warning about “any”, and pretty much forces you to coerce “any” to a concrete type as soon as you see it.
IMHO this is a good idea, but the downside is that you may end up writing incorrect or incomplete types (coming up with the correct signature might require reading through half the code base) instead of being forthright about not knowing the type.
Sounds like typed-Python is newer and less mature than TypeScript. It takes time for the tooling and libraries to adapt.
(I’ve only been using TypeScript for a few months, but I love it. I had a project that required doing some JS, and my last experience writing nontrivial amounts of JS was awful, so I went with TS and I’m glad I did.)
Python doesn’t have a compile/build step where normally a statically typed language would enforce types.
Enforcing types at runtime is only slightly better than not having any type hints since the result will be the same: an exception getting thrown (due to misusing an untyped variable VS type hints finding a mismatch)
This basically leaves what we have today, smart tools (VS code, PyCharm etc) highlighting issues at coding time.. I know many devs out in the wild abhor the idea of using an IDE to write python code though. Plus discovering issues this way requires being physically on the file where the problem is.
The alternative is to introduce an optional pseudo-compile step, just like statically typed languages, where all type related problems (and perhaps others) would be discovered. And that’s Mypy :)
OP highlights all these, doesn’t offer an alternative so it is what it is 🤷♂️
The CPython interpreter — which for many people is “Python” — does have a compile step, because it’s implemented as a stack-based VM, and must first compile Python source code to bytecode for that VM. And the process of parsing Python is well-specified enough that there are static analysis tools for Python which produce and work with the abstract syntax tree rather than the source code.
It’s just that none of these currently read the annotations that are used for type hinting, or perform any type of enforcement of them.
And to be perfectly honest, I don’t understand why “it only counts if you have a tool of this exact name running at this exact time in this exact way” seems to be the argument here — people who work in, say, Java don’t just ignore all the stuff their IDE flags and say “eh, the compiler will catch it, and fixing it before I run the compiler is incompatible with my theory of what static typing is”. They try to clear that stuff before they invoke the compiler.
The CPython interpreter — which for many people is “Python” — does have a compile step, because it’s implemented as a stack-based VM, and must first compile Python source code to bytecode for that VM
Very true but that build step doesn’t get exposed to the python dev in the same way it gets exposed to say a c++ dev. It also doesn’t quite have the same usefulness in uncovering errors before runtime.
If there is an error in my C++ code, I’ll know about it before I’m able to run my code. OTOH, I can “build” and run my python code with errors as long as I don’t hit the parts of the code with the errors.
If there is an error in my C++ code, I’ll know about it before I’m able to run my code. OTOH, I can “build” and run my python code with errors as long as I don’t hit the parts of the code with the errors.
If your code is not syntactically valid Python, you find out at “compile time”. You can force this with the compileall module, and I recommend doing it at, for example, container build time so that you don’t have to have things compiling to bytecode later.
On the other hand, if your C++ contains a fatal logic bug or something that will segfault or so on, the compiler does not tell you and you won’t find out unless/until you hit the critical code path.
It was a big letdown that Python did not use guards, as found in other Smalltalk relatives, particularly E. Using the classic E-on-Java implementation, the article’s example correctly throws a type error:
? def foo :int := "hello"
# problem: <ClassCastException: String doesn't coerce to an int>
This is a pretty rough article, which basically boils down to “Python type hints are optional and the interpreter doesn’t enforce them”. Which is… something you discover very early on in studying them (it was even introduced as “gradual typing”). If the author’s hope was for Python to have become a mandatory statically-typed language, I guess it’s a disappointment, but I don’t know of anyone who approaches Python type hints in good faith and has such an expectation.
There’s also the problem with this:
To recap:
What happened in 3.5 was that the syntactic feature of annotations gained the official semantic meaning of type hinting, and the
typingmodule was added to the standard library to support that use case (previously, annotations had been officially un-semantic, with an encouragement to experiment with possible use cases for them, but people overwhelmingly used them for type hints).Meanwhile, if you want something to complain about, here’s one: type hints and the surrounding tooling were added to Python in a way that strongly encouraged nominal typing, but real-world Python is much more “interfacely” typed, and it took years for the tooling to even begin to shift toward properly supporting that. Which is one reason why I’ve held off adding type hints to most of my own personal open-source Python projects — when the type hints don’t match the way the language is used, they’re not helpful.
“gradual typing” does imply that the type hints are optional, but when they are present they will be enforced. the real issue here is that “typed python” is a gradually typed language built on top of python, in much the same way typescript is built on top of javascript, and that the python interpreter is purely a python interpreter, not a typed-python one. mypy/pytype/pyright/etc are the typed-python typecheckers.
also, this is an interesting paper demonstrating that typed-python is really several dialects, each one depending on the specific type checker enforcing it.
Most of the “gradual” (portions of the code base can be untyped) type systems that are being applied to languages with large communities (TypeScript, Python’s type hints, Dart, etc.) are also “optional” (no effect on run-time behaviour), but they can vary in how clearly they communicate the optional-ness.
Some gradual type systems allow the optional-ness to be configured (e.g. Sorbet for Ruby), so you can choose whether to have run-time checks.
The term “optional” for speaking about “run-time checks” seems largely confined to academic literature, and doesn’t really seem to appear in user-facing type system documentation. A 2018 ECOOP paper from Chung et. al includes a helpful table comparing various gradual type systems by attributes like optional as well as others.
I do wish it were easier to be more precise type system behaviour, but unfortunately most adjectives placed in front of type systems are quite fuzzy with multiple meanings… 😬
thanks for the pointer, i skimmed the paper but it seems well worth a deeper read. (it’s a bit out of date wrt python though). the distinction i was trying to get at is where it says
python, by contrast, does not even do static type checking, it says that the type annotations are treated as metadata which you may check using an external tool if you so choose, or introspect yourself within your code, but they are otherwise unused[*] at both compile time and runtime.
[*] except for the inconsistency that if you have
x: Athe interpreter tries to look up the classAand store that in the metadata, rather than the string"A", so ifAis not a valid class the compiler will complain. this is being fixed in an upcoming version of python, where all annotations will be treated as strings.I’m wondering if Typescript is actually a major source of confusion here, because TypeScript is effectively a separate language that compiles down to JavaScript, while Python with type hints is still Python, not some sort of separate statically-typed language that compiles to Python.
And really, the same complaint applies for TypeScript, because TypeScript does not change JavaScript — the output JS is still a dynamically-typed language and has no useful memory or enforcement of the types you so carefully annotated into your TypeScript source.
In the same way, adding type hints to Python source code via the annotation syntax does not change Python-the-language to behave differently; it simply means that now you can run a tool to statically analyze your annotations and produce warnings/errors/etc. you can act on.
My guess is that the mypy developers reasonably thought that introducing such a major feature needs to be done gradually. If you see 100 errors when you run
mypythe first time, that could be fixable in an afternoon. If you see 10,000, you just give up and stop trying.To get to a good place, start with the default
mypyconfiguration, then work your way up to strict compliance (seemypy --helpand apply each of the--strictflags separately), and finally just usemypy --strict, which fixes most of the issues mentioned in this post. Or change to a compiled language.Yeah, this is the way.
I can’t understand why the author (a) wants to use Python, a dynamically typed language (b) wants to use type checking but (c) will give up on type checking that isn’t 100%-guaranteed-enforced
mypyto your CI. Congrats, now it’s enforced. There goes whole sections of this post.PyObject*but shhhh). Its existence doesn’t mean that things, in your own codebase, that are hinted aren’t helpful! Or at least it’s very strange to argue that “anything could be Any, and Any is bad, so let’s not use hints at all, so now everything is Any!”I have never seen a statically typed language that didn’t have optional dynamic typing. Java has Object, Haskell has Dynamic (and a few other options), C has void*. Any is hardly news to the space
This is the way.
We did this early on and it has done wonders for code quality.
We were very early adopters of mypy, and I’m happy for it overall. It’s like a linter on steroids, for lack of a better term.
A frustrating issue is that because of how
Anyworks and with stuff like third party deps, you end up with weird situations where entire functions just don’t get really type checked, and changing some part of the code in one file suddenly makes a seemingly unrelated function be type checked.It’s good that errors show up, but it’s frustrating that you end up having to clean up “other people’s changes”.
My “project to do if I had infinite time” would be to fork the typescript type checker to support Python. structural subtying, very good union type resolution…. typescript’s just way better at answering a lot of the questions I have about types for my “standard enterprise app”. That being said mypy is a serious project and I think people are going to be adding types over time, if only because they are at least halfway-good documentation.
Yeah, Python isn’t going to check the types for you. There’s still a lot of value. By labeling the types, you are at least expressing intent. So it’s not “well what is the right thing to pass in”, but “I am passing in the wrong thing”, or “the type annotation is in fact wrong”. Which sounds equivalent, but is a bit more restricted and easier to think about.
TypeScript is the same, a developer puts an
anyin some place “just for now” and it spreads out tendrils into otherwise correct code, turning off typechecking and turning solid typedefs into lies at runtime.I’m still not sure what the answer is, although allowing
unknownto be used in places where you want to say “I don’t know” rather thananymeaning “anything goes, do as you please” seems to be a good start.I know typescript has a variant of this issue, but there’s something incidental (I believe) happening with mypy that makes this problem worse. It’s very weird but frequent in the large codebase I work on.
There is also a practical thing; the typescript type system captures more patterns than the mypy type system, so less stuff falls back to any.
In any case I am glad gradual typing exists overall.
IIRC TypeScript defaults to being way more aggressive with warning about “any”, and pretty much forces you to coerce “any” to a concrete type as soon as you see it.
IMHO this is a good idea, but the downside is that you may end up writing incorrect or incomplete types (coming up with the correct signature might require reading through half the code base) instead of being forthright about not knowing the type.
Sounds like typed-Python is newer and less mature than TypeScript. It takes time for the tooling and libraries to adapt.
(I’ve only been using TypeScript for a few months, but I love it. I had a project that required doing some JS, and my last experience writing nontrivial amounts of JS was awful, so I went with TS and I’m glad I did.)
Python doesn’t have a compile/build step where normally a statically typed language would enforce types.
Enforcing types at runtime is only slightly better than not having any type hints since the result will be the same: an exception getting thrown (due to misusing an untyped variable VS type hints finding a mismatch)
This basically leaves what we have today, smart tools (VS code, PyCharm etc) highlighting issues at coding time.. I know many devs out in the wild abhor the idea of using an IDE to write python code though. Plus discovering issues this way requires being physically on the file where the problem is.
The alternative is to introduce an optional pseudo-compile step, just like statically typed languages, where all type related problems (and perhaps others) would be discovered. And that’s Mypy :)
OP highlights all these, doesn’t offer an alternative so it is what it is 🤷♂️
The CPython interpreter — which for many people is “Python” — does have a compile step, because it’s implemented as a stack-based VM, and must first compile Python source code to bytecode for that VM. And the process of parsing Python is well-specified enough that there are static analysis tools for Python which produce and work with the abstract syntax tree rather than the source code.
It’s just that none of these currently read the annotations that are used for type hinting, or perform any type of enforcement of them.
And to be perfectly honest, I don’t understand why “it only counts if you have a tool of this exact name running at this exact time in this exact way” seems to be the argument here — people who work in, say, Java don’t just ignore all the stuff their IDE flags and say “eh, the compiler will catch it, and fixing it before I run the compiler is incompatible with my theory of what static typing is”. They try to clear that stuff before they invoke the compiler.
Very true but that build step doesn’t get exposed to the python dev in the same way it gets exposed to say a c++ dev. It also doesn’t quite have the same usefulness in uncovering errors before runtime.
If there is an error in my C++ code, I’ll know about it before I’m able to run my code. OTOH, I can “build” and run my python code with errors as long as I don’t hit the parts of the code with the errors.
If your code is not syntactically valid Python, you find out at “compile time”. You can force this with the
compileallmodule, and I recommend doing it at, for example, container build time so that you don’t have to have things compiling to bytecode later.On the other hand, if your C++ contains a fatal logic bug or something that will segfault or so on, the compiler does not tell you and you won’t find out unless/until you hit the critical code path.
About “dataclasses ignore type hints as well”, Pydantic will solve this for you.
It was a big letdown that Python did not use guards, as found in other Smalltalk relatives, particularly E. Using the classic E-on-Java implementation, the article’s example correctly throws a type error: