I wonder why you consider unordered maps a failing rather than a reasonable design decision: You want an ordered set of hash keys? Order that list yourself or use a library implementation that does. I like the opinionated decision that Go made from the start, and eventually Perl arrived at, to intentionally produce unpredictable key ordering so naive programmers do not rely on an assumption that isn’t true (and probably incurs overhead).
Python has different hashing per process invocation for security purposes. They just keep a separate insertion ordered array for stable iteration orders.
Came here to post exactly that. Ordered maps are significantly more expensive than unordered ones, because hash tables. Making the more expensive collection the default is a bad idea, especially when the benefit is so small — I can only think of a few cases where it’s been needed.
This is a pet peeve of mine because I’ve run into several issues where some existing system or other decides to assign meaning to the order of keys in a JSON object — despite the standard saying they’re unordered — causing serious problems with processing such JSON in a language whose maps are unordered. (Several examples of this come from CouchDB, such as the way it used to use a JSON object to describe the following multipart MIME bodies.)
Though one would think “adding the ordering constraint makes it more expensive”, Python landed here because a better dict implementation gave insertion ordering for free.
Now, sure, maybe a billion years down the line we’ll find some other dic management strategy that is better, but Python is pretty mature and the dict change seemed to align well with a lot of stuff.
So Python was faced with either:
just exposing the key ordering on the standard object
Keep around this separate object (OrderedDict) despite dict now being able to fulfill its requirements, for what amount to philosophical reasons.
I think pragmatism won out here. And now you don’t have to tell beginners “remember, insertion order on dictionaries aren’t kept!”, You can just say that (or have beginners assume it, correctly if for the wrong reasons).
Ordered maps are significantly more expensive than unordered ones, because hash tables.
“Because hash tables” what? :-) A dict in Python was a hash table when it was unordered, and remained a hash table when it suddenly became ordered as a side-effect of them wanting to share the keys (which saves a lot of space, given that instances of the same class would all have the same keys). Here’s a concise explanation I wrote recently: https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/en/
As for “significantly more expensive”, I have no idea what you’re talking about!
As for “significantly more expensive”, I have no idea what you’re talking about!
It adds a separate data structure for the key/value list (or you could say it adds a separate entry-index list.)
It adds a memory indirection to every hash table probe.
When a key is removed, you have to mess with the entry list. Either you remove the entry, which requires sliding the other entries down and renumbering all the indices in the table; or you turn the entry into a tombstone that can never be reused (in which case you eventually need to GC the tombstones and renumber when there are too many.)
I’m sure this design is a win for Python, because Python (like JS and Ruby) uses dictionaries to store objects, so it happens to have a ton of dictionaries with identical sets of keys, from which keys are very rarely removed. So this actually saves a significant amount of memory. Without that special circumstance, I doubt the overhead comes close to paying for itself.
At least more expensive in memory, you need to keep an additional list structure. I don’t really get the point of ordered maps but they’re probably fine for python.
In the old unordered implementation, each bucket was an “entry” struct (hash, key, value).
In the new ordered implementation, they have a dense array of (hash, key, value) entries, one per value which is in the hash table. The hash buckets are integer indexes into the entries array, and they change the size of the ints depending on how many entries the dict has - 8 bit ints when there are very few entries, growing to 16, 32 or 64 bits as the number of entries goes up.
The hash table always has more buckets than it has entries because its load factor can never be 1. In CPython 2 and 3 the load factor is generally between 1/3 and 2/3 because they resize at a 2/3 load factor. So the number of buckets is going to be between 1.5 and 3x the number of entries. Having more int8_t or int16_ts in memory in order to have fewer PyObject*s in memory is a net win.
The above applies mainly to the combined dict layout. The combined dict layout is the one in which the dict contains both keys & values. The other layout is the “split” layout, where 1 copy of the keys is shared between N dicts, each of which just has a simple flat array of pointers to values. The split layout saves way more memory in the case that it’s designed for, which is storing the fields for instances of classes (taking advantage of the fact that constructors will typically assign exactly the same fields in exactly the same order for every object of a given class).
For a normal dictionary it won’t apply, right? If the array of (hash, key, value) behaves like a stack, it still consume more than the number of entries because you don’t want each insertion to re-allocate. The exception for object fields is a good one, but it’s only because python is a dictionary-based language; in general an unordered hashmap is smaller than an ordered one simply because there’ literally less information to store.
Both the entries & buckets arrays grow multiplicatively.
https://mail.python.org/pipermail/python-dev/2012-December/123028.html shows the arithmetic. From what I remember the new combined layout was designed primarily in order to make dicts use less memory. The fact that it also makes preserving ordering feasibly was a happy side benefit that Python devs chose to take advantage of.
in general an unordered hashmap is smaller than an ordered one simply because there’ literally less information to store.
That would only necessarily be the case if you were using a succinct data structure, which no hash table is. They’re a time/space tradeoff, using more space than necessary in order to save time.
More expensive in memory often turns out to be more expensive in CPU as well. Most to all fast hash maps I’m aware of use some form of open addressing to store the data in a flat array. The only way to maintain ordering in that case would be to bloat each element with pointers and iterate in a much less cache friendly manner. Your CPU is going to end up spinning waiting for data. For Python (and I assume Ruby), everything is already a PyObject*, so the overhead is much lower than it would be in a more value oriented language.
Iterating the entries of a python3 dict is not cache-unfriendly in the way iterating a linked list normally is. Think “arraylist” not “linked list”.
Keys are stored in a flat array of PyDictKeyEntry structs, each of which is a (hash, key, value) triple.
In a split layout dict, the value field is omitted.
In a split layout dict, the values are stored in a flat array of PyObject*s.
In a combined layout dict, the values are stored in the flat array of PyDictKeyEntry structs.
To be clear, I was referring to a hypothetical open addressing hash table design. Python squares that circle by but using open addressing. Since it can inline the PyObject* in the key table, you aren’t paying for an extra memory indirection. In a value oriented language (both those without GC and languages like Go), that extra memory indirection would be an unacceptable cost on every lookup.
I think the same design with the flat entries array & separate indexes into it could work without the PyObject* indirection. If keys & values are both fixed size, use exactly the same design with a fixed-size entry struct. If not, now the integers in the bucket list would be byte offsets to the starts of entries, rather than indices of entries (so you’d have to use the wider sizes sooner). And each entry would need to have one or two length fields too, depending on whether either the keys or values might happen to be fixed-size.
It would work in the sense that you can build a functional hash table like that, but that table would still be slower than an open addressing table due to memory indirection. You’re still paying an extra memory indirection on every lookup. In an open addressing table, you have a single memory indirection when looking up the bucket that contains the key. In the ordered table you outlined, you have a memory indirection to look up an integer and a second memory indirection to look up the actual key with that integer.
AFAIK the definition of “open addressing” is that the collision resolution mechanism is based on probing other buckets rather than following a linked list or something - not that there isn’t a memory lookup to get the entry.
I’m not aware of anyone in python complaining about a big regression to dict lookup time when 3.6 came out (offhand I see some references to microbenchmarks showing 3% perf differences). The interpreter is pretty slow so maybe it’s just getting buried, but apparently that 1 extra indirection isn’t blowing everything up.
Nowhere did I mention values. I’m specifically explaining the number of memory indirections before being able to compare the key. I absolutely believe that Python was able to maintain performance even with ordering. I’m simply trying to explain why that isn’t possible in general.
Is that much worse than indirection to the values? I get the impression it’s pretty rare to have to probe very many different buckets before finding the right one or running out
Separate thought: you could steal say 2 to 4 bits from the bucket integers and put top bits from the hash in them. Then when probing you can often reject a given bucket and move on to the next one without having to check the entries
First, there are a number of use cases where you don’t use the values, notably hash sets but also when checking if a key is in a map. If you store the values next to the keys in the ordered array, that has all the same memory trade-offs as having the keys and values in a single array (i.e. like Abseil flat_hash_map), except for the ordered version has more memory indirection and makes deletes substantially more expensive.
This is a pet peeve of mine because I’ve run into several issues where some existing system or other decides to assign meaning to the order of keys in a JSON object
You just gave me flashbacks… OAS and the idiotic way it uses the JSON field order to organise the endpoints in the UI. Makes it extremely hard to work on OAS specifications programmatically as you have to fight your libraries all the way.
To anyone who’s considering assigning meaning to JSON field order, please switch profession, you weren’t meant to be a programmer…
Perl hashes (maps) have never been ordered as far as I know. I think the feature is from AWK.
I don’t believe it’s a conscious decision to have unordered hashes to “keep newbies on their toes”. It’s simply more (machine) efficient not to have to order internally.
Edit I mostly reacted to the statement
I like the opinionated decision that Go made from the start, and eventually Perl arrived at
(my emphasis). Perl has had unordered hashes since the 1980s, while Go was released in 2009.
I had a coworker whose Perl code used the basic data structure of hashes of hashes of hashes … ad infinitum and by Ghod I’m approaching that level myself.
awk doesn’t preserve order, but you can choose from built-in features (might be gawk specific)
$ awk 'BEGIN{a["z"]=1; a["x"]=12; a["b"]=42; for(i in a) print i, a[i]}'
x 12
z 1
b 42
$ # index sorted in ascending order as strings
$ awk 'BEGIN{PROCINFO["sorted_in"] = "@ind_str_asc";
a["z"]=1; a["x"]=12; a["b"]=42; for(i in a) print i, a[i]}'
b 42
x 12
z 1
$ # value sorted in ascending order as numbers
$ awk 'BEGIN{PROCINFO["sorted_in"] = "@val_num_asc";
a["z"]=1; a["x"]=12; a["b"]=42; for(i in a) print i, a[i]}'
z 1
x 12
b 42
my %hash = ( b => 42, x => 12, z => 1 );
say " ==> dump the hash";
foreach my $key ( keys %hash ) {
say " $key $hash{$key}";
}
say " ==> order by key";
foreach my $key ( sort { $a cmp $b } keys %hash ) {
# we use 'cmp' here because the keys are strings
say " $key $hash{$key}";
}
say " ==> order by value";
foreach my $key ( sort { $hash{$a} <=> $hash{$b} } keys %hash ) {
# we use '<=>' because the values are numbers
say " $key $hash{$key}";
}
Output:
==> dump the hash
z 1
b 42
x 12
==> order by key
b 42
x 12
z 1
==> order by value
z 1
x 12
b 42
Because the result of the function keys %hash is a list, we can apply all sorts of fancy sorting to it, for example, sorting by value and then by key on a tie.
say " ==> add a new key with the same value as an existing one";
$hash{y}=12;
foreach my $key (sort { $hash{$a} <=> $hash{$b} || $a cmp $b } keys %hash) {
say " $key $hash{$key}";
}
z 1
x 12
y 12
b 42
Perl hashes (maps) have never been ordered as far as I know. I think the feature is from AWK.
Maybe I am mistaken. I stopped following Perl circa 2013. I recall that the ordering was an implementation detail and platform-specific but consistent and predictable. So of course, people relied on that and the Perl community said, “No, THAT’S WRONG!” (probably tchrist on something else… but why not reuse a good opener?) but it wasn’t actually fixed for a while.
Thanks for expanding. I think I remember something like that. In any case, it’s possible that for some small number of keys, the return order would be deterministic (like if the keys were simply one-character strings) and beginners, without internalizing the documentation, observed this and started to rely on a behavior that broke down in other cases.
Perl has never guaranteed any ordering of the hash keys, and the ordering has already changed several times during the lifetime of Perl 5. Also, the ordering of hash keys has always been, and continues to be, affected by the insertion order and the history of changes made to the hash over its lifetime.
It’s ergonomics, much like the namedtuple collection mentioned in the same breath. The change being referred to removed the extra step of importing OrderedDict from the collections library when you cast a namedtuple into a dict. If that dict shouldn’t be ordered, there’s probably also no reason for namedtuple to exist. Collections also has other silly-but-useful beauties like defaultdict.
The choices about many such things in Python seem absurd when you sit down as an engineer to architect an application.
When you’re doing something like data science, the percentage of code you write that will never be run again dramatically outweighs even code a software engineer would refer to as prototype code. There are 100x or more the circumstances in which you’d type several dozen keyboard characters, run something best characterized as a “code cell“, then delete it because you were wrong (or start a new cell that doesn’t necessarily follow the previous cell in execution order). It’s an activity with requirements halfway between an interactive shell and an executable file.
When 90% of your work is loading arbitrary or novel data inputs and poking at them to see if they have any life, nothing matters more about your tool than your ability to churn through this iteration cycle quickly.
Over the past 10 years, the percentage of Python used directly as a human tool (not a language to build human tools) has dramatically shifted the audience for language improvements. Maybe away from what is appropriate for software development, maybe not.
I write Python for a profession, and there is no application I would engineer in Python instead of Go. But I also think any professional Go developer who just spent the day e.g. unmarshalling json can appreciate there are other activities we all do besides engineering.
Despite its many other failings, there’s no other tool I’d reach for before Python when some new thing comes at me and I say, “now what’s THIS bullshit.” Quirks like stuffing values into a data structure and getting them back in an intuitive way is part of this charm.
To put it another way with less fanfare: if you have data in a map that’s less useful to you because that map is ordered, that’s probably already data that shouldn’t be in a Python map. This remains true if you’re already in the middle of writing Python code when this happens (we have non-Python data structures in Python).
… But I also think any professional Go developer who just spent the day e.g. unmarshalling json can appreciate there are other activities we all do besides engineering.
(emphasis mine) Exploratory programming in go involving json (“nominally curly braced or bracketed UTF8 blobs”) is awful (I’ve also felt this a while back in D-Lang’s std.json library and got into the habit of using other libraries, C++ json libraries are excellent in terms of the programmer interface). If anyone thinks unordered maps is not ergonomic they’ll faint when they deal with json.
From my experience, having maps preserve insertion order is so much more convenient that it “deserves” to be the default. Additional “evidence” to that is Ruby and Python switching to do exactly that.
I know preserving order is good for job security because I’ve written this genuine line of code for a real project:
FIELDS = list({v: k for k, v in list(FIELD_MAP.items())[::-1]}.values())[::-1]
But other than that, I can’t think of a time when explicitly using OrderedDict felt like an inconvenience, and there are two obvious benefits: it doesn’t constrain the implementation of dict, and it tells the reader you’re going to do something that cares about the order.
I feel like I’m perhaps missing something. But I meant OrderedDict—as in: in the unusual event that I need ordering, it doesn’t bother me to explicitly ask for it.
I can think of exactly two kinds of thing where I cared. One was implementing LRU caches in coding challenges
The other was a dirty hack. I was reinventing protobuf (but for json, with simpler syntax, and better APIs for the target languages), and the code gen was done by defining a python class with the appropriate members, and later looping over their contents. I used metaclass magic to replace the default method dict with an ordered one, then iterated the members of all classes in the namespaces to do code gen:
class Event(metaclass=msg):
member = int
other = str
...
for c in scrape_classes():
for m in c.members():
lang.gen()
For most other things, I don’t think I wanted insertion order – it’s either been key order or value order.
Where are you using them often enough that it matters?
This isn’t a substantive criticism.
Python is 29 years old; it’s a given that backward compatibility is a major constraint on evolving the language.
The same will be true of NGS in 2042.
A new language designed last week might make different choices but that’s true of any deployed system.
This post would be much more interesting if you describe the alternatives and provide evidence for why your preferred choices are superior.
Python’s PEPs are an excellent example of high-quality language design and criticism.
Here are the examples from your post with a more detailed comparison and metasyntactic variables that can be filled in.
Any or all of these could be wrong; I haven’t used either language in anger.
Walrus Operator
“Python 3.8 added the walrus operator.
NGS is an expression language so a separate operator is not required.
I avoid the common assignment-vs-equals bug by $rationale.”
Positional-only Parameters
“Python 3.8 allows specifying positional-only parameters in Python methods. This was already permitted in methods implemented in C but couldn’t easily be done in Python. NGS uses the same approach as Python but $differences”
LRU cache
“Python’s functools.lru_cache permits negative arguments for backward compatibility. NGS uses a fail-fast approach to error handling because $rationale.”
Dictionary Ordering
“Python 3.7 made insertion order in dictionaries part of the language specification. This change allows the developers to simplify APIs and the implementation. Denial of service attacks can be an issue for dictionaries but Python avoids this risk by using the secure SipHash hashing function. NGS uses deterministic insertion order in dictionaries because of $reason. The implementation guards against DoS attacks by $method.”
So why criticise it in the context of the 3.8 release? This article really doesn’t add anything new to the discussion, and honestly neither does most of this comment thread.
The several mentions of NGS make me suspect that this is written almost as an advertisement. Consider the article’s conclusion:
From my perspective, all languages suck, while NGS aims to suck less than the rest for the intended use cases (tl;dr – for DevOps scripting).
NGS itself seems to have commits from Ilya (the author), so it seems like he’s at least a contributor. Combined with other people’s comments about this article’s lack of substance, I’m a little wary.
I recently saw another article authored by try NGS people about NGS being better because it has a method that ensures that an array has exactly one element, which was then promptly edited to say “yeah NGS isn’t the only one with this”. Every language looks better than others to their creator. But I don’t think a new language is the solution, especially if it has no groundbreaking features (and I don’t know if NGS has groundbreaking features)
I recently saw another article authored by try NGS people about NGS being better because it has a method that ensures that an array has exactly one element, which was then promptly edited to say “yeah NGS isn’t the only one with this”. Every language looks better than others to their creator. But I don’t think a new language is the solution, especially if it has no groundbreaking features (and I don’t know if NGS has groundbreaking features)
Do you have any examples of languages that became successful because of a groundbreaking language feature?
Most successful languages in the last 30 years have a blend of ideas from earlier languages:
Python - ABC, SETL, Occam
Ruby - Perl, Smalltalk, Lisp
Java - Objective C, C++
JavaScript - Self, C
Go - C, Limbo, Emerald
PHP - Perl
R - S, Scheme
Swift - Objective-C, Rust, Haskell, Ruby, Python, C#, CLU, others
C# - C++, Delphi, Java
PowerShell - Korn shell
It seems much more important to have a compelling use case, some degree of luck, and a platform or library that’s easier to use than the incumbent.
Corporate backing may have become a requirement; most of these languages were either created in the 1990s or had corporate backing.
In NGS, I am throwing in syntax and features (after deliberation) which are geared towards intended use cases, nothing that I would consider “groundbreaking”, just convenient. Sometimes much more convenient than in other languages.
Not many of the features are unique but they fit the niche well. The “chance” of this language is targeting the niche. I have learned to not throw in anything that is “maybe a good idea” (or at least mark that “experimental”).
Examples of features specifically targeting the niche:
Syntax for running external commands, syntax for running + parsing the output of external commands
I didn’t see it before. I do think that creation of a new programming language must be justified. If it’s “it would be so cool to have a new language” - either keep your fun/learning project for yourself or at least state the aim explicitly in a prominent place.
Unfortunately I can not answer your question fully at the moment because of my current constraints. I’ll try to get to this later. Partial answer is at
NGS being better because it has a method that ensures that an array has exactly one element, which was then promptly edited to say “yeah NGS isn’t the only one with this”
It’s practically not feasible to go over every existing language to check the claim. “I’m not aware of any language that has this” is my approximation. The mistake was pointed out to me and I promptly updated the post as not to mislead the readers.
Everything else is addressed in today’s update of the post.
My concern is with the GIL in the reference implementation (cpython), and how Python is slowly but surely falling behind due to failure to address this.
A couple of points here. The GIL is actively being worked on. Also, after programming for several years professionally with python I’ve never really ran into a situation where the GIL was a hindrance for me. Is it ideal? No. Is it actually a problem for 90% of python programmers? Also, no.
I see complaints like this spread all over the net, but no one ever has a concrete use case they encountered first hand where the GIL was a problem.
Finally, for the record, ocaml has had a GIL since inception as well and is only just now getting rid of it. It’s not like python’s GIL is strange or somehow worse than any others.
Maybe, but you probably should account that it’s probably at least partially a self reinforcing thing. If I know I need to write a program that uses parallelism, then I don’t even briefly consider Python. I just automatically skip over it and use a more capable language. And I also won’t complain on the Internet every time this happens either. So you won’t even hear about my use case.
I don’t really get your comment to be honest. I thought it was broadly accepted that a GIL is a significant problem.
By the same token, if the GIL wasn’t as terrible as it was made out to be, then so many people wouldn’t have sunk untold amounts of labor into trying to fix it. Or, so many people wouldn’t be complaining about it. I don’t recall reading about any recently, but I do remember at least a few failed herculean attempts at fixing it.
I guess if “significant problem” is defined to be “a problem that prevents something from achieving the success of Python,” then yeah, sure, it’s trivially true that the GIL isn’t a significant problem. Which is kind of missing my point. I guess I’m just saying, “why are you complaining about people who complain about the GIL?” Shrug. I just thought everyone kind of understood it was a big pain in the ass.
By the same token, if the GIL wasn’t as terrible as it was made out to be, then so many people wouldn’t have sunk untold amounts of labor into trying to fix it. Or, so many people wouldn’t be complaining about it.
I’ve observed these patterns:
Developers really needing parallelism - As you note they immediately skip Python, but if they do try Python, they quickly discover the problem and can understand why it exists. There is relatively little complaining, and they quickly move on writing an extension, or move to another language. When talking about their problem, it’s clear they know they the constraints.
Language and runtime implementors - They often have similar skillset of experience to the first set, but are keen to try and fix Python. This results in Unladen Swallow and many other efforts. Some of these, like Dave Beazley’s work on helping people to understand the GIL, have more lasting value for folks coming along later.
People looking to speed up their Python programs - A lot of GIL conversations come up in this context. In my experience, the better conversations are constructive and people aren’t trying to trash Python and the GIL. Many people don’t know about it, but often jump to the assumption that it’s the problem. Inexperienced developers are often understanding the difference been concurrency, parallelism, whether things are CPU-bound, etc. Experienced developers are helping inexperienced people to understand where their problem is first. Scientific programmers are discovering that they need to follow the common pattern of writing C/C++ to speed up their code.
The “GIL bad” people - This set of people who don’t like Python creating noise which frustrates experienced Python programmers. They create, or respond to Python problem listicles, or suggest that the GIL is the issue for some problem when it’s clearly not.
Pattern number 4 seems to outnumber the others in terms of discussions that occur on HN, Twitter, etc. I’ve spoken to developers who have little Python experience, but they’ve heard of Python’s GIL problems without understanding much about it. In a way, GIL gets used like garbage collection to signify a failing, without any context.
OK, I guess that’s fair. I like your framing. I just got miffed at someone saying that it wasn’t a significant problem and that there were no concrete use cases for it. I get that your fourth group of people might be blowing it out of proportion in a broad strokes kind of way (and that sort of group definitely exists for a lot of things on the Internet), but I guess the reverse happened in this thread: minimizing the problem. It just seems to me that of course most Python users aren’t going to have an issue with the GIL. The GIL is such a PITA and so widely known that it’s almost certainly a classic case of survivorship bias.
We all know I’m “someone”; no harm in admitting it at this point.
.. that it wasn’t a significant problem and that there were no concrete use cases for it.
I fully admitted it’s a problem, though, I stand by my original statement that I don’t believe it to be a significant problem. I am also fully aware that there are valid use cases for getting rid of the GIL. My complaint is that people love to trash python because of the GIL but don’t (can’t?) list a single instance of how the GIL has somehow caused them issues.
The problem is there’s a lot of hyperbole around the topic, and many people – not necessarily you! – don’t catch on to that, or realize that “the GIL is a real thing with real performance implications that you need to consider for some types of code” and “the GIL is not an all-devouring monster” can both be true simultaneously.
Coupled to that, it’s one of those perpetual complaint topics. Every discussion-forum thread about Firefox, for example, there’ll be at least a couple people who show up to post their pet years-old bugzilla link and say that because Mozilla refuses to address it they can’t take Firefox seriously. With Python there’s a similar pattern where you can guarantee that even if the topic at hand had nothing to do with them, the fact that Python was mentioned will bring a few types of formulaic comments out, one of which is “have they removed the GIL yet”, and none of which really lead to new or constructive discussion.
They use processes: https://docs.python.org/3/library/multiprocessing.html and that’s fine for most parallel solutions. If somebody knows python very well, switching languages is probably not worth it unless they have a rather specific problem. Of course, you know Rust very well, so there’s no good reason why you shouldn’t use that when parallelism is needed.
Yes, I’ve use multiprocessing many times and it’s pretty much always been a dreadful experience unless my use case is the simplest kind of parallelism where you just need to chunk some data and do some truly independent compute tasks.
I see complaints like this spread all over the net, but no one ever has a concrete use case they encountered first hand where the GIL was a problem.
I used to maintain a python extension that could occasionally need to fetch something from the network while the GIL was held. It sucked when used from a web application.
We were able to work around it, and I don’t think I’d call it a major problem. Nor do I think it’s a common one. But I did run into it first hand and it was painful.
I had some trouble with GIL few years ago when I did a job for a client. Had to use forks. Can you elaborate about “falling behind”? I perceive GIL as kind of constant issue/problem which does not change over time.
Except that I’ve dropped Python-like functions on the floor for now, in the name of time. I need to get the “basic” Oil language working and fast, with shell-like “procs”.
I remember you gave some good feedback on the Oil language last October/November. I’d appreciate any more feedback/help (e.g. on Zulip). Here are some idioms I wrote up
If you want a better design for named and positional args, you could help me push on Oil, and then you could have a shell like this, without doing the rest of the work :)
excerpt:
For https://www.oilshell.org/ , which has Python/JS-like functions, I chose to use Julia’s function signature design, which is as expressive as Python’s, but significantly simpler in both syntax and implementation:
Basically they make positional vs. named and required vs. optional ORTHOGONAL dimensions. Python originally conflated the two things, and now they’re teasing them apart with keyword-only and positional-only params.
A semicolon in the signature separates positional and named arguments. So you can have:
func f(p1, p2=0, ...args ; n1, n2=0, ...kwargs)
So p2 is an optional positional argument, while n1 is a required named argument.
And then you don’t need * and star star – you can just use … for both kinds of “splats”. At the call site you only need ; if you’re using a named arg / kwargs splat.
And “having a small feature set” does not improve a language inherently (perhaps it makes implementors lives easier).
There are things that are useful. Things that are less useful. And you should judge things based on those costs instead of establishing strong red lines. Especially when lots of these features are responses to people writing Python code in the wild requesting these kinds of changes to improve real experiences.
And you should judge things based on those costs instead of establishing strong red lines.
99.9% of languages cross all red lines and give a damn all about doing some cost analysis before doing things¹, so I think tit would be nice to have at least one language that upholds some quality standards.
Especially when lots of these features are responses to people writing Python code in the wild requesting these kinds of changes to improve real experiences.
If you do language-design-by-popularity-contest, don’t be surprised if the language looks like it. :-)
¹ Just have a look at the replies (including yours) having an aneurysm simply for mentioning that adding features doesn’t improve a language. If a feature improves a language, then it’s the job of the people who want to add it to prove this², not for others to argue against.
² Oh and also: People who want to add a feature should also be responsible to remove an existing feature first.
99.9% of languages cross all red lines and give a damn all about doing some cost analysis before doing things¹, so I think tit would be nice to have at least one language that upholds some quality standards.
See the PEPs.
Just have a look at the replies (including yours) having an aneurysm simply for mentioning that adding features doesn’t improve a language. If a feature improves a language, then it’s the job of the people who want to add it to prove this², not for others to argue against.
Again, see the PEPs for the feature inclusion. Also, there’s no need to talk down on others because they have a different opinion than you.
Oh and also: People who want to add a feature should also be responsible to remove an existing feature first.
Why do you have to remove another feature? Do you not have enough space? If so, a larger storage device is probably a better solution than arbitrarily restricting feature addition because a “feature” has to be removed first. Also, what is a “feature”? How do you define what is considered a “feature”? A method? An object? A function? A property of an object? A modification to the behaviour of an object?
I have occasionally jokingly suggested that when someone breaks out the Saint-Exupery quote about perfection, we should further perfect that person by taking something away – perhaps their keyboard! Which is a fun way to point out that such assertions absolutely depend on context and nuance.
Your assertions not only lacked nuance, they verged on straw-man arguments: for example, as another reply pointed out, the Python team do require discussion and debate including pro/con analysis prior to adding new language features, as evidenced by the PEP process and by the lengthy email discussions around many PEPs. So when you asserted your distaste for people who don’t do that, the only possible conclusions to draw are that it’s a red herring because you know Python does do this, or that it’s a misrepresentation because Python does it but you’re not acknowledging it.
On a deeper level, there are at least two common views about features in languages/libraries/products/etc., and those views appear to be fundamentally incommensurable. But neither view is objectively and universally correct, yet many people argue as if their view is, which is not useful.
require discussion and debate including pro/con analysis prior to adding new language features
Even by your own wording, it’s not even remotely an equal fight.
One side is “pro”, the other side “against” – which is far removed from a process that says “we want to improve the language, let’s examine whether this can be done by removing something, keeping things the same, or adding something”.
Let’s not even pretend that the pro/con analysis is unbiased or factual, as long as your process treats people who don’t want to add things/remove things as “the bad guys”.
at least two common views
That explains the complete meltdown of one side as seen in this thread, I guess?
Here’s a comment where I argue against adding a new feature to Python.
I believe in thinking about costs and benefits when adding language features. I will admit I like shiny stuff, but I also want features that fit in with the rest of the language.
I also write Python on a daily basis, and I know of places where the walrus operator would have made things cleaner, without getting in the way later. I don’t think it is that important (like do: while or just a case statement would be much more helpful) and I would rather not have the walrus operator if it meant we could have kept Guido and not started a big acromonious fight about it.
I just fundamentally dislike the concept of “too many features are bad”. Good features are good, that’s why they’re good!
Now, lots of languages add features that interact badly with each other…. that’s where you’re really getting into actual messes that affect users.
I might accept “adding features is not the same as improving a language”.
But features which allow for better code reuse (e.g. function decorators), improve expressivity (e.g. list comprehensions), or provide useful control structures (e.g. iterators and generators) can absolutely improve a language. All of my example features were added to Python after it was released, and they all improve that language.
Lack of features leads to books of programming patterns — they’re called so because they require each reader of the code to actively pattern match blocks of code to identify them as what would just be features in another language.
What’s the background on this statement? Are you suggesting that if your language is composed of maximally orthogonal and flexible features then additional features are unnecessary?
Additionally, adding more features after-the-fact will always result in lower quality than if the feature was designed into the language from the start.
I think C#’ properties are a good example of this, Java Generics, …
I wonder why you consider unordered maps a failing rather than a reasonable design decision: You want an ordered set of hash keys? Order that list yourself or use a library implementation that does. I like the opinionated decision that Go made from the start, and eventually Perl arrived at, to intentionally produce unpredictable key ordering so naive programmers do not rely on an assumption that isn’t true (and probably incurs overhead).
perlsec says that Perl uses unpredictable key ordering for resistance to algorithmic complexity attacks.
JFYI Rust also took the same approach. Deliberately randomly different between program runs. The position was to prevent HashDoS attacks.
https://doc.rust-lang.org/std/collections/struct.HashMap.html
Python has different hashing per process invocation for security purposes. They just keep a separate insertion ordered array for stable iteration orders.
Came here to post exactly that. Ordered maps are significantly more expensive than unordered ones, because hash tables. Making the more expensive collection the default is a bad idea, especially when the benefit is so small — I can only think of a few cases where it’s been needed.
This is a pet peeve of mine because I’ve run into several issues where some existing system or other decides to assign meaning to the order of keys in a JSON object — despite the standard saying they’re unordered — causing serious problems with processing such JSON in a language whose maps are unordered. (Several examples of this come from CouchDB, such as the way it used to use a JSON object to describe the following multipart MIME bodies.)
Though one would think “adding the ordering constraint makes it more expensive”, Python landed here because a better dict implementation gave insertion ordering for free.
Now, sure, maybe a billion years down the line we’ll find some other dic management strategy that is better, but Python is pretty mature and the dict change seemed to align well with a lot of stuff.
So Python was faced with either:
OrderedDict) despitedictnow being able to fulfill its requirements, for what amount to philosophical reasons.I think pragmatism won out here. And now you don’t have to tell beginners “remember, insertion order on dictionaries aren’t kept!”, You can just say that (or have beginners assume it, correctly if for the wrong reasons).
“Because hash tables” what? :-) A dict in Python was a hash table when it was unordered, and remained a hash table when it suddenly became ordered as a side-effect of them wanting to share the keys (which saves a lot of space, given that instances of the same class would all have the same keys). Here’s a concise explanation I wrote recently: https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/en/
As for “significantly more expensive”, I have no idea what you’re talking about!
I’m sure this design is a win for Python, because Python (like JS and Ruby) uses dictionaries to store objects, so it happens to have a ton of dictionaries with identical sets of keys, from which keys are very rarely removed. So this actually saves a significant amount of memory. Without that special circumstance, I doubt the overhead comes close to paying for itself.
At least more expensive in memory, you need to keep an additional list structure. I don’t really get the point of ordered maps but they’re probably fine for python.
From what I remember, Python’s dicts got smaller in memory, not larger, when they added the ordering feature.
It’s worth looking at the notes at the top of https://github.com/python/cpython/blob/v3.8.5/Objects/dictobject.c - the authors went to a lot of trouble to explain their work.
In the old unordered implementation, each bucket was an “entry” struct (hash, key, value).
In the new ordered implementation, they have a dense array of (hash, key, value) entries, one per value which is in the hash table. The hash buckets are integer indexes into the entries array, and they change the size of the ints depending on how many entries the dict has - 8 bit ints when there are very few entries, growing to 16, 32 or 64 bits as the number of entries goes up.
The hash table always has more buckets than it has entries because its load factor can never be 1. In CPython 2 and 3 the load factor is generally between 1/3 and 2/3 because they resize at a 2/3 load factor. So the number of buckets is going to be between 1.5 and 3x the number of entries. Having more
int8_torint16_ts in memory in order to have fewerPyObject*s in memory is a net win.The above applies mainly to the combined dict layout. The combined dict layout is the one in which the dict contains both keys & values. The other layout is the “split” layout, where 1 copy of the keys is shared between N dicts, each of which just has a simple flat array of pointers to values. The split layout saves way more memory in the case that it’s designed for, which is storing the fields for instances of classes (taking advantage of the fact that constructors will typically assign exactly the same fields in exactly the same order for every object of a given class).
For a normal dictionary it won’t apply, right? If the array of
(hash, key, value)behaves like a stack, it still consume more than the number of entries because you don’t want each insertion to re-allocate. The exception for object fields is a good one, but it’s only because python is a dictionary-based language; in general an unordered hashmap is smaller than an ordered one simply because there’ literally less information to store.Both the entries & buckets arrays grow multiplicatively.
https://mail.python.org/pipermail/python-dev/2012-December/123028.html shows the arithmetic. From what I remember the new combined layout was designed primarily in order to make dicts use less memory. The fact that it also makes preserving ordering feasibly was a happy side benefit that Python devs chose to take advantage of.
That would only necessarily be the case if you were using a succinct data structure, which no hash table is. They’re a time/space tradeoff, using more space than necessary in order to save time.
More expensive in memory often turns out to be more expensive in CPU as well. Most to all fast hash maps I’m aware of use some form of open addressing to store the data in a flat array. The only way to maintain ordering in that case would be to bloat each element with pointers and iterate in a much less cache friendly manner. Your CPU is going to end up spinning waiting for data. For Python (and I assume Ruby), everything is already a PyObject*, so the overhead is much lower than it would be in a more value oriented language.
Iterating the entries of a python3 dict is not cache-unfriendly in the way iterating a linked list normally is. Think “arraylist” not “linked list”.
Keys are stored in a flat array of PyDictKeyEntry structs, each of which is a (hash, key, value) triple. In a split layout dict, the value field is omitted.
In a split layout dict, the values are stored in a flat array of
PyObject*s. In a combined layout dict, the values are stored in the flat array of PyDictKeyEntry structs.The implementation for iterating both the keys & values at the same time is at https://github.com/python/cpython/blob/v3.8.5/Objects/dictobject.c#L3715 - functions for iterating only-keys or only-values are just above it.
To be clear, I was referring to a hypothetical open addressing hash table design. Python squares that circle by but using open addressing. Since it can inline the
PyObject*in the key table, you aren’t paying for an extra memory indirection. In a value oriented language (both those without GC and languages like Go), that extra memory indirection would be an unacceptable cost on every lookup.I think the same design with the flat entries array & separate indexes into it could work without the
PyObject*indirection. If keys & values are both fixed size, use exactly the same design with a fixed-size entry struct. If not, now the integers in the bucket list would be byte offsets to the starts of entries, rather than indices of entries (so you’d have to use the wider sizes sooner). And each entry would need to have one or two length fields too, depending on whether either the keys or values might happen to be fixed-size.It would work in the sense that you can build a functional hash table like that, but that table would still be slower than an open addressing table due to memory indirection. You’re still paying an extra memory indirection on every lookup. In an open addressing table, you have a single memory indirection when looking up the bucket that contains the key. In the ordered table you outlined, you have a memory indirection to look up an integer and a second memory indirection to look up the actual key with that integer.
AFAIK the definition of “open addressing” is that the collision resolution mechanism is based on probing other buckets rather than following a linked list or something - not that there isn’t a memory lookup to get the entry.
I’m not aware of anyone in python complaining about a big regression to dict lookup time when 3.6 came out (offhand I see some references to microbenchmarks showing 3% perf differences). The interpreter is pretty slow so maybe it’s just getting buried, but apparently that 1 extra indirection isn’t blowing everything up.
Nowhere did I mention values. I’m specifically explaining the number of memory indirections before being able to compare the key. I absolutely believe that Python was able to maintain performance even with ordering. I’m simply trying to explain why that isn’t possible in general.
Is that much worse than indirection to the values? I get the impression it’s pretty rare to have to probe very many different buckets before finding the right one or running out
Separate thought: you could steal say 2 to 4 bits from the bucket integers and put top bits from the hash in them. Then when probing you can often reject a given bucket and move on to the next one without having to check the entries
First, there are a number of use cases where you don’t use the values, notably hash sets but also when checking if a key is in a map. If you store the values next to the keys in the ordered array, that has all the same memory trade-offs as having the keys and values in a single array (i.e. like Abseil flat_hash_map), except for the ordered version has more memory indirection and makes deletes substantially more expensive.
You just gave me flashbacks… OAS and the idiotic way it uses the JSON field order to organise the endpoints in the UI. Makes it extremely hard to work on OAS specifications programmatically as you have to fight your libraries all the way.
To anyone who’s considering assigning meaning to JSON field order, please switch profession, you weren’t meant to be a programmer…
Perl hashes (maps) have never been ordered as far as I know. I think the feature is from AWK.
I don’t believe it’s a conscious decision to have unordered hashes to “keep newbies on their toes”. It’s simply more (machine) efficient not to have to order internally.
Edit I mostly reacted to the statement
(my emphasis). Perl has had unordered hashes since the 1980s, while Go was released in 2009.
The “llama book” has this to say: “The order is jumbled because Perl keeps the key-value pairs in an order that’s convenient for Perl so that it can look up any items quickly. You use a hash either when you don’t care what order the items are in, or when you have an easy way to put them into the order you want.”
I think the quote should be amended to read
;)
I had a coworker whose Perl code used the basic data structure of hashes of hashes of hashes … ad infinitum and by Ghod I’m approaching that level myself.
awkdoesn’t preserve order, but you can choose from built-in features (might begawkspecific)Thanks for expanding on AWK!
This is the equivalent Perl code
Output:
Because the result of the function
keys %hashis a list, we can apply all sorts of fancy sorting to it, for example, sorting by value and then by key on a tie.Maybe I am mistaken. I stopped following Perl circa 2013. I recall that the ordering was an implementation detail and platform-specific but consistent and predictable. So of course, people relied on that and the Perl community said, “No, THAT’S WRONG!” (probably tchrist on something else… but why not reuse a good opener?) but it wasn’t actually fixed for a while.
tchrist on something else: “You are wicked and wrong to have broken inside and peeked at the implementation and then relied upon it.”
Thanks for expanding. I think I remember something like that. In any case, it’s possible that for some small number of keys, the return order would be deterministic (like if the keys were simply one-character strings) and beginners, without internalizing the documentation, observed this and started to rely on a behavior that broke down in other cases.
Quoting from the link that @dbremner posted:
(Emphasis in original).
It’s ergonomics, much like the namedtuple collection mentioned in the same breath. The change being referred to removed the extra step of importing OrderedDict from the collections library when you cast a namedtuple into a dict. If that dict shouldn’t be ordered, there’s probably also no reason for namedtuple to exist. Collections also has other silly-but-useful beauties like defaultdict.
The choices about many such things in Python seem absurd when you sit down as an engineer to architect an application.
When you’re doing something like data science, the percentage of code you write that will never be run again dramatically outweighs even code a software engineer would refer to as prototype code. There are 100x or more the circumstances in which you’d type several dozen keyboard characters, run something best characterized as a “code cell“, then delete it because you were wrong (or start a new cell that doesn’t necessarily follow the previous cell in execution order). It’s an activity with requirements halfway between an interactive shell and an executable file.
When 90% of your work is loading arbitrary or novel data inputs and poking at them to see if they have any life, nothing matters more about your tool than your ability to churn through this iteration cycle quickly.
Over the past 10 years, the percentage of Python used directly as a human tool (not a language to build human tools) has dramatically shifted the audience for language improvements. Maybe away from what is appropriate for software development, maybe not.
I write Python for a profession, and there is no application I would engineer in Python instead of Go. But I also think any professional Go developer who just spent the day e.g. unmarshalling json can appreciate there are other activities we all do besides engineering.
Despite its many other failings, there’s no other tool I’d reach for before Python when some new thing comes at me and I say, “now what’s THIS bullshit.” Quirks like stuffing values into a data structure and getting them back in an intuitive way is part of this charm.
To put it another way with less fanfare: if you have data in a map that’s less useful to you because that map is ordered, that’s probably already data that shouldn’t be in a Python map. This remains true if you’re already in the middle of writing Python code when this happens (we have non-Python data structures in Python).
(emphasis mine) Exploratory programming in go involving json (“nominally curly braced or bracketed UTF8 blobs”) is awful (I’ve also felt this a while back in D-Lang’s std.json library and got into the habit of using other libraries, C++ json libraries are excellent in terms of the programmer interface). If anyone thinks unordered maps is not ergonomic they’ll faint when they deal with json.
From my experience, having maps preserve insertion order is so much more convenient that it “deserves” to be the default. Additional “evidence” to that is Ruby and Python switching to do exactly that.
I know preserving order is good for job security because I’ve written this genuine line of code for a real project:
But other than that, I can’t think of a time when explicitly using OrderedDict felt like an inconvenience, and there are two obvious benefits: it doesn’t constrain the implementation of dict, and it tells the reader you’re going to do something that cares about the order.
…unordered?
I feel like I’m perhaps missing something. But I meant OrderedDict—as in: in the unusual event that I need ordering, it doesn’t bother me to explicitly ask for it.
I was confused by your comment.
“I can’t think of a time when I wanted to use an ordered dict and I felt inconvenienced that the default dict was not ordered.”
“Because ordering is requested explicitly (as opposed to if dict was ordered by default) the implementation of dict is not constrained.”
This part is fine but is confusing if one interpreted the previous clauses of your comment in a different way.
Use a list, seriously; arrays’ raison d’etre is to provide you with a collection of ordered items.
I can think of exactly two kinds of thing where I cared. One was implementing LRU caches in coding challenges
The other was a dirty hack. I was reinventing protobuf (but for json, with simpler syntax, and better APIs for the target languages), and the code gen was done by defining a python class with the appropriate members, and later looping over their contents. I used metaclass magic to replace the default method dict with an ordered one, then iterated the members of all classes in the namespaces to do code gen:
For most other things, I don’t think I wanted insertion order – it’s either been key order or value order.
Where are you using them often enough that it matters?
This isn’t a substantive criticism.
Python is 29 years old; it’s a given that backward compatibility is a major constraint on evolving the language.
The same will be true of NGS in 2042.
A new language designed last week might make different choices but that’s true of any deployed system.
This post would be much more interesting if you describe the alternatives and provide evidence for why your preferred choices are superior.
Python’s PEPs are an excellent example of high-quality language design and criticism.
Here are the examples from your post with a more detailed comparison and metasyntactic variables that can be filled in.
Any or all of these could be wrong; I haven’t used either language in anger.
Walrus Operator
“Python 3.8 added the walrus operator.
NGS is an expression language so a separate operator is not required.
I avoid the common assignment-vs-equals bug by $rationale.”
Positional-only Parameters
“Python 3.8 allows specifying positional-only parameters in Python methods. This was already permitted in methods implemented in C but couldn’t easily be done in Python. NGS uses the same approach as Python but $differences”
LRU cache
“Python’s functools.lru_cache permits negative arguments for backward compatibility. NGS uses a fail-fast approach to error handling because $rationale.”
Dictionary Ordering
“Python 3.7 made insertion order in dictionaries part of the language specification. This change allows the developers to simplify APIs and the implementation. Denial of service attacks can be an issue for dictionaries but Python avoids this risk by using the secure SipHash hashing function. NGS uses deterministic insertion order in dictionaries because of $reason. The implementation guards against DoS attacks by $method.”
There was opportunity in Python 3, which broke this backward compatibility; Old code will likely not run as-is.
Python3 was released 12 years ago. There are schoolchildren learning to write code who weren’t yet born at the release of Python 3.
12 year ago is recent relative to 29.
So why criticise it in the context of the 3.8 release? This article really doesn’t add anything new to the discussion, and honestly neither does most of this comment thread.
The several mentions of NGS make me suspect that this is written almost as an advertisement. Consider the article’s conclusion:
NGS itself seems to have commits from Ilya (the author), so it seems like he’s at least a contributor. Combined with other people’s comments about this article’s lack of substance, I’m a little wary.
I recently saw another article authored by try NGS people about NGS being better because it has a method that ensures that an array has exactly one element, which was then promptly edited to say “yeah NGS isn’t the only one with this”. Every language looks better than others to their creator. But I don’t think a new language is the solution, especially if it has no groundbreaking features (and I don’t know if NGS has groundbreaking features)
Ilya is NGS’ main author. It’s a rant on his personal blog so he probably supposes the reader already knows this.
Do you have any examples of languages that became successful because of a groundbreaking language feature?
Most successful languages in the last 30 years have a blend of ideas from earlier languages:
It seems much more important to have a compelling use case, some degree of luck, and a platform or library that’s easier to use than the incumbent.
Corporate backing may have become a requirement; most of these languages were either created in the 1990s or had corporate backing.
In NGS, I am throwing in syntax and features (after deliberation) which are geared towards intended use cases, nothing that I would consider “groundbreaking”, just convenient. Sometimes much more convenient than in other languages.
Not many of the features are unique but they fit the niche well. The “chance” of this language is targeting the niche. I have learned to not throw in anything that is “maybe a good idea” (or at least mark that “experimental”).
Examples of features specifically targeting the niche:
log(),debug(),retry()methodsHave you read Frank Atanassow’s advice on designing a language?
He takes a strongly negative attitude towards creating languages but does pose some good questions.
I didn’t see it before. I do think that creation of a new programming language must be justified. If it’s “it would be so cool to have a new language” - either keep your fun/learning project for yourself or at least state the aim explicitly in a prominent place.
Unfortunately I can not answer your question fully at the moment because of my current constraints. I’ll try to get to this later. Partial answer is at
Alternatives:
https://github.com/ngs-lang/ngs#have-you-heard-of-project-x-how-it-compares-to-ngs
Note that there is no alternative at which a looked and said: “OK, it’s aligned with my vision so much that doing NGS is not needed anymore”.
Yes, I’m the main author.
It’s practically not feasible to go over every existing language to check the claim. “I’m not aware of any language that has this” is my approximation. The mistake was pointed out to me and I promptly updated the post as not to mislead the readers.
Everything else is addressed in today’s update of the post.
Minor language complaints aside, I love Python.
My concern is with the GIL in the reference implementation (cpython), and how Python is slowly but surely falling behind due to failure to address this.
A couple of points here. The GIL is actively being worked on. Also, after programming for several years professionally with python I’ve never really ran into a situation where the GIL was a hindrance for me. Is it ideal? No. Is it actually a problem for 90% of python programmers? Also, no.
I see complaints like this spread all over the net, but no one ever has a concrete use case they encountered first hand where the GIL was a problem.
Finally, for the record, ocaml has had a GIL since inception as well and is only just now getting rid of it. It’s not like python’s GIL is strange or somehow worse than any others.
Maybe, but you probably should account that it’s probably at least partially a self reinforcing thing. If I know I need to write a program that uses parallelism, then I don’t even briefly consider Python. I just automatically skip over it and use a more capable language. And I also won’t complain on the Internet every time this happens either. So you won’t even hear about my use case.
I don’t really get your comment to be honest. I thought it was broadly accepted that a GIL is a significant problem.
To paraphrase myself, I’m not saying it isn’t a problem. I don’t think it’s the significant problem it’s made out to be.
I don’t think you can explain python or ocaml’s overwhelming success if the GIL is a terrible as it’s made out to be.
By the same token, if the GIL wasn’t as terrible as it was made out to be, then so many people wouldn’t have sunk untold amounts of labor into trying to fix it. Or, so many people wouldn’t be complaining about it. I don’t recall reading about any recently, but I do remember at least a few failed herculean attempts at fixing it.
I guess if “significant problem” is defined to be “a problem that prevents something from achieving the success of Python,” then yeah, sure, it’s trivially true that the GIL isn’t a significant problem. Which is kind of missing my point. I guess I’m just saying, “why are you complaining about people who complain about the GIL?” Shrug. I just thought everyone kind of understood it was a big pain in the ass.
I’ve observed these patterns:
Pattern number 4 seems to outnumber the others in terms of discussions that occur on HN, Twitter, etc. I’ve spoken to developers who have little Python experience, but they’ve heard of Python’s GIL problems without understanding much about it. In a way, GIL gets used like garbage collection to signify a failing, without any context.
OK, I guess that’s fair. I like your framing. I just got miffed at someone saying that it wasn’t a significant problem and that there were no concrete use cases for it. I get that your fourth group of people might be blowing it out of proportion in a broad strokes kind of way (and that sort of group definitely exists for a lot of things on the Internet), but I guess the reverse happened in this thread: minimizing the problem. It just seems to me that of course most Python users aren’t going to have an issue with the GIL. The GIL is such a PITA and so widely known that it’s almost certainly a classic case of survivorship bias.
We all know I’m “someone”; no harm in admitting it at this point.
I fully admitted it’s a problem, though, I stand by my original statement that I don’t believe it to be a significant problem. I am also fully aware that there are valid use cases for getting rid of the GIL. My complaint is that people love to trash python because of the GIL but don’t (can’t?) list a single instance of how the GIL has somehow caused them issues.
The problem is there’s a lot of hyperbole around the topic, and many people – not necessarily you! – don’t catch on to that, or realize that “the GIL is a real thing with real performance implications that you need to consider for some types of code” and “the GIL is not an all-devouring monster” can both be true simultaneously.
Coupled to that, it’s one of those perpetual complaint topics. Every discussion-forum thread about Firefox, for example, there’ll be at least a couple people who show up to post their pet years-old bugzilla link and say that because Mozilla refuses to address it they can’t take Firefox seriously. With Python there’s a similar pattern where you can guarantee that even if the topic at hand had nothing to do with them, the fact that Python was mentioned will bring a few types of formulaic comments out, one of which is “have they removed the GIL yet”, and none of which really lead to new or constructive discussion.
They use processes: https://docs.python.org/3/library/multiprocessing.html and that’s fine for most parallel solutions. If somebody knows python very well, switching languages is probably not worth it unless they have a rather specific problem. Of course, you know Rust very well, so there’s no good reason why you shouldn’t use that when parallelism is needed.
Yes, I’ve use
multiprocessingmany times and it’s pretty much always been a dreadful experience unless my use case is the simplest kind of parallelism where you just need to chunk some data and do some truly independent compute tasks.I used to maintain a python extension that could occasionally need to fetch something from the network while the GIL was held. It sucked when used from a web application.
We were able to work around it, and I don’t think I’d call it a major problem. Nor do I think it’s a common one. But I did run into it first hand and it was painful.
I had some trouble with GIL few years ago when I did a job for a client. Had to use forks. Can you elaborate about “falling behind”? I perceive GIL as kind of constant issue/problem which does not change over time.
Other languages’ reference/mostpopular implementations do not suffer from it, thus Python is left behind.
While cpython does have the GIL problem, other implementations of Python successfully got rid of GIL. It is not a hopeless situation. It can be done.
About positional arguments, here’s what I wrote a few months ago with regard to Oil borrowing from Julia:
https://news.ycombinator.com/item?id=21253729
Except that I’ve dropped Python-like functions on the floor for now, in the name of time. I need to get the “basic” Oil language working and fast, with shell-like “procs”.
I remember you gave some good feedback on the Oil language last October/November. I’d appreciate any more feedback/help (e.g. on Zulip). Here are some idioms I wrote up
http://www.oilshell.org/preview/doc/idioms.html (will post this to lobste.rs later, after some polishing)
If you want a better design for named and positional args, you could help me push on Oil, and then you could have a shell like this, without doing the rest of the work :)
excerpt:
For https://www.oilshell.org/ , which has Python/JS-like functions, I chose to use Julia’s function signature design, which is as expressive as Python’s, but significantly simpler in both syntax and implementation:
Manual:
https://docs.julialang.org/en/v1/manual/functions/index.html…
Comparison:
https://medium.com/@Jernfrost/function-arguments-in-julia-and-python-9865fb88c697
Basically they make positional vs. named and required vs. optional ORTHOGONAL dimensions. Python originally conflated the two things, and now they’re teasing them apart with keyword-only and positional-only params.
A semicolon in the signature separates positional and named arguments. So you can have:
So p2 is an optional positional argument, while n1 is a required named argument.
And then you don’t need * and star star – you can just use … for both kinds of “splats”. At the call site you only need ; if you’re using a named arg / kwargs splat.
Sounds like a very good direction
Adding features does not improve a language.
And “having a small feature set” does not improve a language inherently (perhaps it makes implementors lives easier).
There are things that are useful. Things that are less useful. And you should judge things based on those costs instead of establishing strong red lines. Especially when lots of these features are responses to people writing Python code in the wild requesting these kinds of changes to improve real experiences.
99.9% of languages cross all red lines and give a damn all about doing some cost analysis before doing things¹, so I think tit would be nice to have at least one language that upholds some quality standards.
If you do language-design-by-popularity-contest, don’t be surprised if the language looks like it. :-)
¹ Just have a look at the replies (including yours) having an aneurysm simply for mentioning that adding features doesn’t improve a language. If a feature improves a language, then it’s the job of the people who want to add it to prove this², not for others to argue against.
² Oh and also: People who want to add a feature should also be responsible to remove an existing feature first.
See the PEPs.
Again, see the PEPs for the feature inclusion. Also, there’s no need to talk down on others because they have a different opinion than you.
Why do you have to remove another feature? Do you not have enough space? If so, a larger storage device is probably a better solution than arbitrarily restricting feature addition because a “feature” has to be removed first. Also, what is a “feature”? How do you define what is considered a “feature”? A method? An object? A function? A property of an object? A modification to the behaviour of an object?
[Comment from banned user removed]
I have occasionally jokingly suggested that when someone breaks out the Saint-Exupery quote about perfection, we should further perfect that person by taking something away – perhaps their keyboard! Which is a fun way to point out that such assertions absolutely depend on context and nuance.
Your assertions not only lacked nuance, they verged on straw-man arguments: for example, as another reply pointed out, the Python team do require discussion and debate including pro/con analysis prior to adding new language features, as evidenced by the PEP process and by the lengthy email discussions around many PEPs. So when you asserted your distaste for people who don’t do that, the only possible conclusions to draw are that it’s a red herring because you know Python does do this, or that it’s a misrepresentation because Python does it but you’re not acknowledging it.
On a deeper level, there are at least two common views about features in languages/libraries/products/etc., and those views appear to be fundamentally incommensurable. But neither view is objectively and universally correct, yet many people argue as if their view is, which is not useful.
Even by your own wording, it’s not even remotely an equal fight.
One side is “pro”, the other side “against” – which is far removed from a process that says “we want to improve the language, let’s examine whether this can be done by removing something, keeping things the same, or adding something”.
Let’s not even pretend that the pro/con analysis is unbiased or factual, as long as your process treats people who don’t want to add things/remove things as “the bad guys”.
That explains the complete meltdown of one side as seen in this thread, I guess?
So you’re deliberately trolling.
Here’s a comment where I argue against adding a new feature to Python.
I believe in thinking about costs and benefits when adding language features. I will admit I like shiny stuff, but I also want features that fit in with the rest of the language.
I also write Python on a daily basis, and I know of places where the walrus operator would have made things cleaner, without getting in the way later. I don’t think it is that important (like
do: whileor just acasestatement would be much more helpful) and I would rather not have the walrus operator if it meant we could have kept Guido and not started a big acromonious fight about it.I just fundamentally dislike the concept of “too many features are bad”. Good features are good, that’s why they’re good!
Now, lots of languages add features that interact badly with each other…. that’s where you’re really getting into actual messes that affect users.
My first comment was a direct dismissal of the approach to language design you describe.
There is no misunderstanding or need for further explanation here.
I might accept “adding features is not the same as improving a language”.
But features which allow for better code reuse (e.g. function decorators), improve expressivity (e.g. list comprehensions), or provide useful control structures (e.g. iterators and generators) can absolutely improve a language. All of my example features were added to Python after it was released, and they all improve that language.
I think we’ll have to agree to disagree on that.
All of the things you have mentioned have costs associated with them, and only very rarely do the improvements rise above that.
Lack of features leads to books of programming patterns — they’re called so because they require each reader of the code to actively pattern match blocks of code to identify them as what would just be features in another language.
I think we can do way better, we shouldn’t let the misdeeds of 70ies’ languages cloud our judgement.
What’s the background on this statement? Are you suggesting that if your language is composed of maximally orthogonal and flexible features then additional features are unnecessary?
Yes.
Additionally, adding more features after-the-fact will always result in lower quality than if the feature was designed into the language from the start.
I think C#’ properties are a good example of this, Java Generics, …