1. 41

  2. 14

    Some of the other things you get!

    1. 4

      I recently found the statistics module and was grateful to not have to roll my own stddev.

      1. 1

        Yesterday I found the secrets module, did a little dance, then realised it’s not in py2.7. Dancing stopped, because the project I was working on is mid-port to py3, so needs to support both for now

      2. 2

        There’s definitely mocking in Python 2 - is anything different about the Python 3 one?

        1. 3

          I’m not sure about functionality differences, but it’s part of the standard library in Python 3.

      3. 7

        We didn’t have dictionary comprehensions in 2 ??!! How did I live without them? Messily and verbosely, I guess. (Honestly, I forgot we didn’t have them). The slides are a great list of cool features I mostly didn’t know about. I guess I could start using them.

        1. 10

          Honestly, I forgot we didn’t have them

          In fact, you do have them. They were backported to 2.7, along with set comprehensions.

          1. 3

            In Python 2.4, you’d write:

            d = dict((key, some_function(key)) for key in some_iterator)

            instead of

            d = {key: some_function(key) for key in some_iterator}

            The older one isn’t as pretty. It’s not truly hideous though, not like implicit silent coercion between bytes and Unicode provided everything is ASCII, only to pop UnicodeDecodeError exceptions with utterly useless tracebacks(¹) in production at 3am.

            (¹ The traceback was usually useless for UnicodeDecodeError caused by things like b'føø' + u'føø' because the traceback will be thrown by whatever code tried to join the different kinds of strings together, but what you need to hunt down instead is what code produced two different kinds of strings. Very often with templating systems like ZPT, the string concatenation would happen as the final step of template rendering, after all of the view code has been run and none of it is left on the stack any more.²)

            (² In retrospect maybe we should’ve patched ZPT to check the types of strings as it produced them instead of just throwing everything into a big StringIO and calling StringIO.getvalue() (which subsequently calls ''.join(…)) at the end.)

          2. 6

            I appreciate the advocacy for Python 3, but I don’t know. Some of this stuff isn’t really all that great, or is just a library that could easily exist in Python 2.

            • pathlib: Doesn’t seem to be any reason this can’t be in Python 2.
            • Chained exceptions: You can wrap the cause in the raised exception if you create a custom exception. I call this “meh”
            • Fine grained OSError… This isn’t particularly revolutionary, and I’d wager people are more likely to miss the handling of an OSError because of it. I handled PermissionError but forgot about StatError (no idea if that’s a thing).
            • Everything is an iterator: I didn’t follow the “out of disk space” complaint. xrange existed in like Python 2.5 at least… as did itertools. Inconsistent API? OK, none of this is magic, rewrite it as itertools2 or something else.
            • No more comparison to everything… Yes. Please blow my program up at runtime. You adopted a dynamic language, you’re going to have a bad time once in a while.
            • asyncio, yield from – good additions. As is more “advanced” unpacking. But, if it’s limited to 3 things, it’s not enough.
            • bytes v str? I’m not sold on this being a great change, but then again, I’m a native English speaker.

            The change that speaks the most to me is the Enum type, but since I’m sure you can’t do things like ensure exhaustive checks, I fail to see how it’s any more useful than constants or other hacks. And, runtime verification that you reused the same name isn’t super compelling…

            1. 12

              No offense to you personally, but after seeing enough posts like this over the last few years I’ve come to the conclusion that most people still against Python 3 made up their minds a long time ago that Python 3 is bad, and they’re just not going to change no matter what.

              When somebody’s claiming it’s a good idea that (‘abc’ > 123 == True) then I just don’t know what to say.

              1. 4

                On the otherhand, in Python 3 map, filter, reduce are always True because they return a generator. In Erlang, anything is comparable to anything but the language clearly specifies the relative sort order of each data datatype so it’s predictable.

                For myself, I’m against Python in general but I happen to be productive enough in Python 2. Python 3 isn’t enough win to overcome the friction of switching to it. Most of the things that one seems to benefit from in Python 3 are for problems I don’t really want to solve in Python 3 in the first place. But that’s me.

                1. 2

                  No offense taken. I haven’t used Python regularly for … oh, 5 years(?), so I skipped over all the contentious debate and opinioning. I recently wrote a script for Python 2.7, but only because that’s what was available to me. Would have been happy to use Python 3.whatever, and fumble a little longer while I worked out the quirks that aren’t soldered into my brain.

                  I’m not suggesting that fixing 'abc' > 123 == True is worthless, or a bad idea, though I can see how my careless use of words would imply that. I’ll state my point more directly.

                  If you choose to program in a dynamic programming language like Python, you can only gain reliability by being diligent, and defensive about things like this. Let’s say I get a random JSON file as input. I know up front that the producer of this file nonchalantly mixes "123" and 123, or even if I don’t know that, if I’m expecting that these things be comparable as integers, I’m not going to blindly assume that the JSON doesn’t quote them. That’d be pretty presumptuous, (and dangerous) of me. (edit to add: I want to know that the data is bad as soon as possible, as it makes any side effects that happen later on easier to reverse (e.g. you often avoid causing them in the first place). Waiting til the last possible second to blow up just sucks on many levels.)

                  So, yes. Taking steps to remove these sorts of silly things from the language is a great step. Using it is as an example of an “awesome feature” to entice me to Python 3? Not very convincing.

                  Now, if the “awesome feature” was “Python 3 has contracts”, then you’re talking.

                  1. 4

                    I want to know that the data is bad as soon as possible, as it makes any side effects that happen later on easier to reverse (e.g. you often avoid causing them in the first place). Waiting til the last possible second to blow up just sucks on many levels.

                    Isn’t this a positive step in that direction though? Having 'abc' > 123 blow up at that point is better than having it silently evaluate to True to cause trouble later on, and easier to catch in tests.

                    1. 2

                      Yes. It’s a positive direction. But, no sane Python (or any other language, really) programmer should allow their data to go unvalidated long enough to get to that point in the first place.

                      That’s my point. But maybe this is part of the problem that leads to such horrible sec problems? Do programmers really hate validation that much?

                      1. 1

                        Yes. Validation is tedious and repetitive, exactly the kind of thing one goes into programming to avoid.

                        (To my mind the obvious thing is to move to strongly typed protocols like thrift or protobuf that handle at least the most basic parts of validation for you. But I guess a type-based solution would appeal less to Python programmers)

                        1. 2

                          Not to be that guy, but Python is strongly typed (it is also dynamically typed).

                          1. 1

                            Not to be that guy, but define “strongly typed.”

                            The phrases “strongly” and “weakly” typed are so nebulous that they are practically useless.

                            1. [Comment removed by author]

                              1. 1

                                What does it mean for “type safety to be preserved”? I think you are probably going for the definition “in strong typing, concatenating a string and an integer is an error, as no implicit coercion is allowed.” Is that accurate? Where as, “weak typing,” such as maybe JavaScript, it’d be allowed.

                                The problem with these terms is that their definition is not a completely agreed upon. Some people use strong to mean static and weak to mean dynamic, some people use strong to mean good and weak to mean bad. Some people equate strength with fewer implicit coercions. I find it better to avoid them entirely.

                                Ben Pierce (author of the famed “Types and Programming Languages”) has this to say (in the linked book):

                                I spent a few weeks trying to sort out the terminology of “strongly typed,” “statically typed,” “safe,” etc., and found it amazingly difficult… The usage of these terms is so various as to render them almost useless.

                            2. 1

                              I never said it wasn’t[1]. I implied JSON was weakly typed, and I expect even Python fans would agree with that.

                              [1] Though for the record it isn’t. Python pretends operational validation is type checking for some reason, but this isn’t true.

                  2. 7

                    bytes v str? I’m not sold on this being a great change, but then again, I’m a native English speaker.

                    The str change is a really, really big deal. Even disregarding non-English languages, here are some sentences that will screw over Python 2 code:

                    • “Your total is £20.”
                    • “Is ‘gif’ pronounced /dʒɪf/ or /ɡɪf/?”
                    • “The circumference of a circle is 2πr, or ?r if you’re not a savage.”
                    • “Whoever made word processors default to “” instead of "" needs to be shot out of a cannon.”
                    • “A quarter note (♩) is half the length of a half-note.”
                    • “∀ x, y, z, n ∈ ℕ : n > 2 ⇒ x^n + y^n ≠ z^n”
                    • “WoTC retroactively changed Æther to aether for translation reasons.”
                    • “?”

                    That’s a ton of use cases, and we haven’t even left English yet! Off the top of my head, Python 2 doesn’t play nice with עִבְרִית, 中文, فارسی, ру́сский язы́к, 한국어, हिन्दी, ελληνικά, etc. Even Latin languages have problems in Py2: it can’t handle ç, ñ, ü, ß, etc.

                    Incidentally, here’s what the above post looks like as a python 2 string:

                    “> bytes v str? I\xe2\x80\x99m not sold on this being a great change, but then again, I\xe2\x80\x99m a native English speaker.\n\nThe str change is a really, really big deal. Even disregarding non-English languages, here are some sentences that will screw over Python 2 code:\n\n* “Your total is \xc2\xa320.”\n* “Is 'gif' pronounced /d\xca\x92\xc9\xaaf/ or /\xc9\xa1\xc9\xaaf/?”\n* “The circumference of a circle is 2\xcf\x80r, or \xf0\x9d\x9c\x8fr if you're not a savage.”\n* “Whoever made word processors default to \xe2\x80\x9c\xe2\x80\x9d instead of "" needs to be shot out of a cannon.”\n* “A quarter note (\xe2\x99\xa9) is half the length of a half-note.”\n* “\xe2\x88\x80 x, y, z, n \xe2\x88\x88 \xe2\x84\x95 : n > 2 \xe2\x87\x92 x^n + y^n \xe2\x89\xa0 z^n”\n* “WoTC retroactively changed \xc3\x86ther to aether for translation reasons.”\n* “\xf0\x9f\x94\xa5”\n\nThat's a ton of use cases, and we haven't even left English yet! Off the top of my head, Python 2 doesn't play nice with \xd7\xa2\xd6\xb4\xd7\x91\xd6\xb0\xd7\xa8\xd6\xb4\xd7\x99\xd7\xaa, \xe4\xb8\xad\xe6\x96\x87, \xd9\x81\xd8\xa7\xd8\xb1\xd8\xb3\xdb\x8c, \xd1\x80\xd1\x83\xcc\x81\xd1\x81\xd1\x81\xd0\xba\xd0\xb8\xd0\xb9 \xd1\x8f\xd0\xb7\xd1\x8b\xcc\x81\xd0\xba, \xed\x95\x9c\xea\xb5\xad\xec\x96\xb4, \xe0\xa4\xb9\xe0\xa4\xbf\xe0\xa4\xa8\xe0\xa5\x8d\xe0\xa4\xa6\xe0\xa5\x80, \xce\xb5\xce\xbb\xce\xbb\xce\xb7\xce\xbd\xce\xb9\xce\xba\xce\xac, etc. Even Latin languages have problems in Py2: it can't handle \xc3\xa7, \xc3\xb1, \xc3\xbc, \xc3\x9f, etc.”

                    Whereas it renders just fine in Python 3.

                  3. 3

                    This is pretty amazing:

                    from pathlib import Path
                    directory = Path("/etc")
                    filepath = directory / "test_file.txt"
                    if filepath.exists():
                    1. 5

                      What’s amazing about it? It’s a type with the / operator overloaded. I’m not sure that it’s all together better than calling a join function for file paths rather than having to understand what / means in that context.

                      1. 2

                        Also, easily ported to Python 2, as __div__ can be overloaded.

                      2. 1

                        Wow. Yeah. I’ve been primarily Python 3 for the past year, and this module is news to me.

                        1. 1

                          As a design pattern, I tend to favour passing structured data between programs as “here is a path to a directory containing files with a standard layout”¹. pathlib is awesome for that kind of thing: being able to take a path object and do (path / "metadata" / "manifest.txt").write_text("\n".join(items)) instead of five or six lines of os.path.join() and a context-handler to manage a file handle… it’s great.

                          It’s also super useful when writing tests for the above code. Previously, my test suites were littered with helpers to create a particular subdirectory inside a particular parent, or create a particular filename at a particular location. Now it’s just stuff like:

                          temppath = pathlib.Path(tempfile.mkdtemp())
                          inputpath = temppath / "metadata" / "manifest.txt"
                          inputpath.parent.mkdir(parents=True, exist_ok=True)

                          My one beef with pathlib is that it does cross-platform support in a weird way. pathlib.Path is the baseclass of both pathlib.PosixPath and pathlib.WindowsPath, but the intended interface is that you create a pathlib.Path and it automatically picks the correct subclass depending on which platform you’re on. This means that you can’t inherit from pathlib.Path to add your own utility methods: when you instantiate your subclass, it doesn’t have the required shenanigans for pathlib.Path to choose which subclass to instantiate.

                          I kind of wish that the current pathlib.Path had been named pathlib.BasePath or something, and at the bottom of the file they’d just put:

                          if os.platform == "nt":
                              Path = WindowsPath
                              Path = PosixPath

                          …then people using “Path” would have exactly the same behaviour, but inheriting from Path would do the right thing.

                          ¹: obviously this doesn’t work for RPC, but it’s nice for multiple programs on a single host, or multiple instances of the same program, especially if you can take advantage of POSIX-guaranteed atomic filesystem operations

                        2. 3

                          Better features aren’t enough if you make it hard for people to upgrade.

                          The Python dev team decided to be backwards incompatible because it was more fun and convenient, and destroyed millions of dollars of developer time spent upgrading. So yes, all those new features are great, and I’m happy to finally be in the position to write Python 3-only code.

                          But if the Python dev team had just done the extra work to use __future__ statements we all would have been using these features years ago. And people wouldn’t have to write these presentations shaming people for not having the resources to do a painful and sometimes impossible upgrade (if you don’t have unit tests you’re screwed.)

                          1. 4

                            The new handling of strings, bytes, and Unicode can not be made backwards compatible. It’s a necessarily breaking change.

                            1. 3

                              I’m skeptical. Python 2 has from __future__ import unicode_strings, being stricter about adding bytes and unicode together could have made a fine __future__ import, and so on.

                              In any case, even if that required a hard break, there were many stupid unnecessary changes that just made porting harder, with the benefit of really minor improvements in defaults.

                              1. 2

                                I’d recommend reading Nick Coghlan’s Python 3 Q&A. It goes into a lot of the motivations behind the backwards incompatibility break with strings and bytes, as well as why they made the other changes (and whether or not they were good ideas in retrospect).

                          2. 1

                            i can’t read the awesome features because of certificate error, anyone else?

                            1. 4


                              1. 3

                                Yeah it’s a github cert, only a problem for https anywhere users though as the link is boring http :(

                                1. 2

                                  thanks, i completely forgot having https anywhere enabled last night.. maybe was a little too late ;)

                              2. 1

                                I don’t code in Python but it seems to be that many of these are solutions to the fact that Python is dynamically typed.