Threads for chreke
-
-
Been in industry almost 20 years. The solo tech stack diagram terrifies me. I don’t see how that is productive at all for side projects.
-
It’s not a side project, it’s for his business.
-
-
-
I’ve always liked Ansible for small-to-medium sized projects; it seems to strike a nice balance between simplicity and power. It doesn’t solve the provisioning problem but IME most shops very rarely need to provision new hardware or services, so doing it manually thought AWS, DO or whatever usually works well enough. Also, since it’s basically just an SSH for-loop on steroids it’s pretty easy to debug—just repeat the command manually!
Also, since it doesn’t pretend to be declarative you can easily do things like run a database migration, something that seems to be treated in the Cloudformation / K8s world as something weird and esoteric that require a bunch of esoteric hacks just to get working.
Am I the only one? I only ever hear about Ansible once in a blue moon, but I don’t know of any alternatives that work better for my use case. Or are there better alternatives I don’t know about?
-
I’ve tried Ansible only a little bit, but my experience with it is definitely not positive. For example, you say it doesn’t pretend to be declarative, but it kinda does, and you have to work your way around the bits where the declarativeness doesn’t work. For example, if you add a cron entry or a user or something like it in your script (or maybe you’re looping over a list and create the things in the list), when you take out one entry, that entry doesn’t get deleted. So you have to adjust your script to delete it conditionally (because the host may or may not have executed the previous version of your Ansible recipe). Also, you have to take care to make sure your recipe is idempotent, which is not always obvious.
And of course you may end up with different systems which have slightly different state. For example, Ansible supports multiple operating systems, but the subtle differences will still show up when you’re writing recipes. And what if users have manually made some changes one a subset of hosts that your Ansible recipe will run on?
At work, we’ve had a colleague write some Ansible tasks, which were basically unmaintainable and we had to scrap the entire thing and rewrite a different solution from scratch - but that might just have been the rest of the team’s unfamiliarity with the tool at the time.
-
-
Those are mostly fine. That’s just a normal three tier app, plus they outsourced authentication to a service (Cognito) and they outsourced syslog to a service (CloudWatch).
One thing that does suck is the usability of Application Gateway (this is the thing for invoking lambdas from HTTP requests) but eh oh well.
(Edit: deleted an incorrect paragraph.)
-
-
I always found the “Let it crash” very convincing. One of the most convincing talks I know is not even about Erlang. But I’ve never actually coded anything substantial in either Erlang or Elixir myself. I went the other way instead with Rust and static typing.
But I’m curious, for those of you who do work on such systems, does it deliver on its promise? Is it simpler? More robust? And is modern Elixir done in the same vein of “Let it crash” and very few tests or verification?
-
Rust follows the “let it crash”-philosophy, its panic system is Erlang-inspired. I used to be even stronger baked into the language, when it still had language-level tasking with a runtime. The nomicon chapter on unwinding still calls it out
You can see that in the tasking/threading APIs where a panic crashes that component and another part of the system is responsible for handling.
-
I’ve had to deal with more than one Rust service that take this philosophy to heart and so will fully crash the entire program in the presence of, say, a network connection timeout to a non-business-critical API endpoint. Maybe this isn’t the intended effect of the panic approach to error management, but it does seem to be a common outcome in my experience.
The problem here is a mismatch of expectations. It’s nominally OK to crash an Erlang actor in response to many/most runtime faults, because Erlang actors always operate in a constellation of redundant peers, and failure is a first order concern of their supervisor. That crash impacts a single request.
But e.g. systemd is not the OTP, and OS processes don’t operate in a cluster. A service running as an OS process is expected to be resilient to basically all runtime errors, even if those errors mean the service can’t fulfill its user-facing requirements. If an OS process crashes, it doesn’t impact a single request, it impacts every request served by the process, every other process with active connections to that process for any other reason, assumptions made by systemd about the soundness of that binary, probably some assumptions about off-host e.g. load balancers shuttling traffic to that instance, everything downstream from them, etc. etc.
If “crash-only” means “terminate the request” and not “terminate the process” then all good! But then “crash” isn’t the right verb, I don’t think, as crashing is pretty widely understood to mean the OS level process of the program. Alas.
-
Yeah, I think this is an acute case of catchy, but wildly misleading terminology. What is really (as in Elang or Midori) understood as proper “let it crash” is two dual properties:
- abandoning the current “thing”
- containing abandonment to some well-defined boundary, such that:
- abandonment don’t propagate outside of this boundary
- tearing things down at this boundary doesn’t compromise the state
- restarting at the boundary is a well-defined operation which can fix transient errors
- the actual blast radius from abandonment is small
Everyone gets the first point, buts it’s the second one which matters, which is hard, and which leads to simplicity and reliability.
-
To expand on this, Rust does only marginally, if at all, better here than you average $LANG:
- the build-in boundary is OS thread, which is often too coarse-grained, there’s catch_unwind for do-it-yourself boundaries. There’s nothing to protect from thread monopolizing the CPU due to an infinite loop bug. Some errors (stack overflow, OOM) abort the process bypassing the recovery mechanism.
- UnwindSafe machinery in theory helps somewhat with the tainted state problem. In practice, it’s too cumbersome to use and people often silence it. I had one spectacular bug where UnwindSafe would’ve saved couple of days of debugging, if it wasn’t silenced due to it tripping a compiler bug.
- nothing to make restart workable, do-it-yourself again.
-
But e.g. systemd is not the OTP, and OS processes don’t operate in a cluster. A service running as an OS process is expected to be resilient to basically all runtime errors, even if those errors mean the service can’t fulfill its user-facing requirements.
I think this might be oversimplifying. Whether it’s reasonable to let a service continue to fulfill tasks despite encountering a serious fault is not clear cut. Example: A service that has a lot of shared state, say thread bound caches of various sensitive user data, a crash might lead to failed cleanups and subsequent data leaks.
-
Let me rephrase my claim to be more precise.
A service running as an OS process is generally expected to be resilient to runtime errors. If a runtime error puts the service in a state where it can no longer fulfill user requirements, and that state is transient and/or recoverable, it is usually preferable for the service to continue to respond to requests with errors, rather than crashing.
-
-
-
-
In my experience across several companies and codebases in Elixir, I’d say the following things.
“let it crash” can lead to clean code. It also originated out of a design space that I believe doesn’t map as directly onto modern webshit as folks want to believe. This is neither good nor bad, it’s just an occasional impedance mismatch between system design philosophies.
“let it crash” encourages some people, drunk on the power of OTP and the actor model, to grossly overcomplicate their code. They decide deep supervision trees and worker pools and things are needed when a simple function will do. This is the curse of the beginner Elixir or Erlang developer, and if properly mentored this goes away quickly. If not properly mentored it progresses to a case of conference talks and awkward libraries.
Testing and verification in the BEAM ecosystem is weird, and until recently was both best and worst in class depending on what languages you were up against. Dialyzer for example is a marvelous typechecker, but there is growing suspicion that it is severely stunted in the sorts of verification it is categorically capable of. On the other side, property-based testing is strictly old-hat over in at least the Erlang ecosystem and has been for quite some time iirc. Other folks are catching up.
(Testing is also–in my opinion–most often valuable to solve team coordination problems and guard against entropy caused by other humans. This is orthogonal to language concerns, but comes out when you have larger webshit-style teams using BEAM stuff when compared with the origin of Erlang.)
Robustness is quite possible. I’ve alluded elsewhere to how a running BEAM instance is more of a living thing (gasp, a pet!) than most contemporary app models (pour one out for Smalltalk)…this unlocks some very flexible things you can do in production that I haven’t really seen anywhere else and which make it possible to do things without downtime during an incident that most folks would just look at and go “wat.”. On the other hand, you have to design your systems to actually enable robustness–writing your standard webshit without structuring the application logic to have affordances for interactivity or process isolation or whatever means you’re basically using the BEAM like you would other more conventional systems.
(You can also, as I’ve done with Python on at least one occasion, build a BEAM-like actor model with fault tolerance. The idioms are baked into Erlang and Elixir, but you can with sufficient effort reproduce them elsewhere.)
(Also, “let it crash” doesn’t mean you won’t sometimes have to wrap your whole VM in a systemd unit to restart things when, say, an intern pushes to prod and blows all the way up the supervision tree.)
-
In a sense, yes—an example I’ve run into several times is that a service you depend on becomes intermittently unresponsive. In a “regular” software service, unless you clutter your code up with retry and “fail soft” logic (basically your own ad-hoc OTP) this usually means a hard error, eg a end-user received an error message or a service needed to be restarted by the OS process manager.
In Erlang the system can usually deal with these kinds of errors by automatically retrying; if the operation keeps failing, the error will propagate up to the next level of the “supervision tree”. Unless it makes it all the way up to the root the application will keep running; sometimes the only indication that something went wrong is some log output.
The nice thing about “Let it crash” is that you don’t have to consider every possible failure scenario (eg what happens if this service call returns malformed data? What if it times out?). Instead of trying to preempt every possible error, which is messy and basically intractable, you can focus on the happy path and tell OTP what to do in case of a crash.
That said, “Let it crash” is not a silver bullet that will solve all errors in your distributed system; you still have to be acutely aware about which parts of the system can be safely restarted and how. The nice thing is that the base assumption of OTP is that the system will fail at some point, and it gives you a very powerful set of tools to deal with it.
Another thing that makes Erlang more robust is the process scheduling model: Processes are lightweight and use preemptive multitasking with fair scheduling, which means you’re less susceptible to “brownouts” from runaway processes.
-
But I’m curious, for those of you who do work on such systems, does it deliver on its promise? Is it simpler? More robust? And is modern Elixir done in the same vein of “Let it crash” and very few tests or verification?
I can only speak to the first half, and it is completely dependent on the team culture. If the team is very OTP/Erlang native, it can work out incredibly well. These teams tend to be extremely pragmatic and focused on boring, obvious, naive ways to solve problems, using global state as needed even!
However, when OTP/Erlang collide with a team trying to treat it like anything else things can go badly…. quickly.
-
-
Hey @chreke, you said
The end result of STEPS was KSWorld, a complete operating system including both a document editor and spreadsheet editor, which ended up being about 17 000 lines of code.
Do you know if the source code is available? Can people actually download and run KSWorld?
-
I don’t think so unfortunately, which I think is kind of weird. However, you can find Gezira, JITBlt and Nile on Dan Amelang’s GitHub page: https://github.com/damelang
-
-
Isn’t it at least as good to have a language that allows little-language expressiveness without loss of big-language functionality?
A number of languages are good at letting you define rich DSLs that also happen to be valid programs in the larger language, at the cost of some constraints on syntax. LISP comes immediately to mind, of course, but rich DSLs are also common in other languages like Kotlin.
-
This is a valid point. My counter-argument would be that in some cases you want that loss of big-language functionality, e.g. to make static analysis easier. Some problems I have with macro-based DSLs is that:
- The runtime usually doesn’t know or care whether you started off from a macro or not, since it only sees the expanded code. This means that e.g. error messages may lack context, or be worded in a confusing manner, etc. Macros is a wafer-thin abstraction, and unpredictable things can happen when you break it.
- You don’t know what “rules” from the original language apply in the macro world. An example that comes to mind is the Cascalog DSL for Hadoop. Since Hadoop ships JARs to different machines for running its distributed MapReduce, this means any function references you use need to be named; supplying a lambda function will not work. This breaks the expectations from the Clojure “host” language in a rather brusque manner, where you’re used to treating them as interchangeable.
Don’t get me wrong—I really like macros and basically think all languages should default to using S-expressions and macros, but I’m not 100% convinced about building DSLs with them. Racket seems interesting for this very reason—it actually does allow you to take things away, by letting you “opt out” of the standard library.
-
That’s quite an interesting argument. Perhaps once you get to a certain point with macros it’s better to reimplement them as a proper language. You can still use s-expressions though - “just” pass unevaluated lists to a “compiler” function that converts the code to a state machine.
The SRE syntax does this, and there are plenty of examples of assemblers that take s-expression syntax and produce bytecode (e.g. sassy for x86, my own bpf assembler and glls for GL shaders).
-
-
I never really like “we need something as elegant as Maxwell’s equations” argument because as soon as you substitute anything into the equations you get a gigantic mess that’s so big everybody uses alternate approaches to solve stuff. PDEs are no joke.
More relevant to your actual point, IMO the biggest barrier for little languages is tooling adoption. LSP/Treesitter plugins are 1) a lot of work to make, and 2) enormous productivity boosters. That pushes people to stay with big languages with extant tooling.
EDIT: I really wouldn’t trust the STEPS program’s claims that they got feature-parity of a 44,000 LoC program with only 300 lines of Nile, not without carefully evaluating both programs. Alan Kay really likes to exaggerate.
-
Yeah, tooling-wise you’re really swimming upstream as users won’t even have syntax highlighting from day one.
I guess the upside is that certain tools become much easier to create for little languages (eg Bret Victor’s graphical Nile debugger—for example, writing a simplified regular expression engine could realistically be an undergraduate homework project—but it’s still work that needs to be done, and it’s the kind of work that won’t pay anyone’s bills.
-
Regarding Nile VS Cairo—yes, if you read the fine print the graphics rendering pipeline actually works out to be much more, as that doesn’t include some of the systems underlying Nile, like Gezira. Looking at the STEPS final report you can see that the total amount of graphics code ends up being somewhere closer to 2000-3000 lines. As I mentioned briefly in the article, part of that size reduction was probably due to replacing hand-rolled optimizations with JITBlt .
However, if you think about it, JITBlt isn’t really connected to the whole “little language” concept at all; you could’ve probably achieved the same thing by adding a JIT compiler as a library in Cairo, or by using some really gnarly C++ template tricks, and end up with a significant size reduction of the Cairo code base instead.
(There might be some other caveats as well that I’m not aware of—for example, there could be a lot of backwards compatibility code in Cairo that they could just skip implementing as Nile is a green-field project)
-
EDIT: I really wouldn’t trust the STEPS program’s claims that they got feature-parity of a 44,000 LoC program with only 300 lines of Nile, not without carefully evaluating both programs. Alan Kay really likes to exaggerate.
The claim about Cairo really stood out to me. We run Cairo, nobody runs Nile, after all. If there is this amazing library that does all the same things in just 300 lines of code, then it would have been a no-brainer to adopt it when it came out… unless there is some details that we would rather not talk about.
-
I see 3 such potential “details”:
- Saying that Nile’s performance “compete” with Cairo may mean it is comparable, within the same order of magnitude. Up to twice as slow. And even if it was only 5% slower people would probably refuse to switch.
- The 300 lines don’t count the rest of the STEPS compilation toolchain, which I believe takes about 2K lines of code. They’re just not counted here because those 2K lines are used for the whole system, not just Nile.
- As far as I know Nile doesn’t have a C API. This one is probably a deal breaker.
-
-
The link is paywalled for me, is there an alternate source?
-
-
Some of the points of SQL, like not writing an explicit algorithm, apply to Prolog and similar languages while being more generalistic than “little languages”
-
Good point—I initially had a Prolog example in the article but dropped it in favour of shell scripts, as I thought they better illustrated the idea of “little languages”. But Prolog might actually be more of a happy medium; I don’t have high hopes of it ever becoming mainstream but I’d be very happy to be proven wrong!
-
-
I would happily make a long-term wager that small, bespoke languages will not replace high-level languages in any meaningful way. I have three reasons for believing this:
- The relative time/effort savings of “using the right language for the job” are not as great as they seem.
- The cost of working with multiple languages is greater than it seems.
- Political forces of organizations, and the individual force of laziness, favors high-level languages.
-
-
As someone who has done a lot of research on program semantics, static analysis and verification, I have gone from thinking little DSLs are ugly to thinking they are the future (as you can see from my other comments in Lobsters).
The key idea is that verifying programs written in Turing-complete languages is hard. We should either try to automate as much as possible (and here static analysis might be the most practical approach, but it is somehow less popular than other techniques) or switch to DSLs, where verification is much easier.
I think what could make DSLs take off is some tooling to construct them (like Racket) paired with some tooling for building verifiers.
-
An interesting related example to this big languages vs DSLs debate is the design of Julia vs Python in the context of ML.
Julia has taken the big Turing-complete language route, whereas Python is just a framework to build little DSLs (or not so little, JAX has around 200 operators). Julia is undoubtedly more elegant, but it’s a lot trickier to implement differentiable programming on a huge language vs a tiny DSL.
So, the end result is that nobody uses Julia to build large ML models because autodiff is not robust enough. You can find a very long and illuminating discussion here: https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/2
-
-
What’s interesting about “little languages” in a Lisp system is that they are no different than libraries. If you use that same argument in non-Lispy languages, “libraries are little languages,” then “little languages” are already incredibly successful and are absolutely the future. The API is a “little language.”
Personally, I want these “library languages” to be able to provide more to the higher level languages they are designed for, but I seem to be in the losing side of this, too. I recently wrote a small wrapper on top of Go’s
database/sqlto make it easy to do transactions across our database abstraction, and, of course, the only way to make this work reasonably was to take afunc () erroras a sort of “block” and on error rollback, else commit. But of course, most Go programmers would cringe at this, and even I cringe at it in some ways. -
Lisp is a funny comparison to make.
One thing I like about Lisp (Common Lisp, specifically) is that the language lets you embed “little languages” and DSLs, so this trade-off doesn’t need to exist there - if I need a new “language” to describe something, I can create it and use it easily without messing around with new tooling, editor support, etc. It’s basically no different than using a library.
-
-
I think I mostly agree here… but a couple points:
- Your usage of “high-level” seems off here. Do you mean “general purpose” or just “big”, maybe, instead? I’d argue that many/most of the little languages in question are also high-level by any common definition.
- I think one thing this misses is another route to success for little languages, which is to bypass “traditional software development” altogether and allow end users and domain experts to put together things that are “good enough.” There’s a long history of things like Access and Excel, up to IFTTT and the glut of modern “low-code/no-code” projects. I’d argue some of these absolutely have (and will continue to have) that sort of success.
-
-
Oh is it time to hype dsls again? That makes sense as we’re starting to all get a little embarrassed about the levels of hype for functional programming.
I guess next we’ll be hyping up memory safe object oriented programming.
-
-
-
-
I’ve just started learning Eiffel and like what ive seen so far, just curious what do you consider its mistakes?
-
- CAT-calling
- Bertrand Meyer’s absolute refusal to use any standard terminology for anything in Eiffel. He calls nulls “voids”, lambdas “agents”, modules “clusters”, etc.
- Also his refusal to adopt any PL innovations past 1995, like all the contortions you have to do to get “void safety” (null safety) instead of just adding some dang sum types.
-
-
Racket!
-
-
-
-
-
-
-
This, basically. Single dynamic dispatch is one of the few things from Java-style OO worth keeping. Looking at other classic-OO concepts: inheritance is better off missing most of the time (some will disagree), classes as encapsulation are worse than structs and modules, methods don’t need to be attached to classes or defined all in one batch, everything is not an object inheriting from a root object… did I miss anything?
Subtyping separate from inheritance is a useful concept, but from what I’ve seen the world seldom breaks down into such neat categories to make subtyping simple enough to use – unsigned integers are the easiest example. Plus, as far as I can tell it makes most current type system math explode. So, needs more theoretical work before it wiggles back into the mainstream.
-
I’ve been thinking a lot about when inheritance is actually a good idea, and I think it comes down to two conditions:
- The codebase will instantiate both Parent and Child objects
- Anything that accepts a Parent will have indistinguishable behavior when passed a Child object (LSP).
IE a good use of Inheritance is to subclass
EventReaderwithProfiledEventReader.-
-
Inheritance can be very useful when it’s decoupled from method dispatch.
Emacs mode definitions are a great example. Nary a class nor a method in sight, but the fact that
markdown-modeinherits fromtext-modeetc is fantastically useful!On the other hand, I think it’s fair to say that this is so different from OOP’s definition of inheritance that using the same word for it is just asking for confusion. (I disagree but it’s a reasonable argument.)
-
-
-
My impression is that inheritance is extremely useful for a peculiar kind of composition, namely open recursion. For example, you write some sort of visitor-like pattern in a virtual class, then inherit it, implement the
visitmethod or what have you, and usethisto recurse between the abstract behavior of traversing some structure, and your use-case-specific code. Without recursion you have to basically reimplement a vtable by hand and it sucks.Well, that’s my only use of inheritance in OCaml. Most of the code is just functions, sum types, records, and modules.
-
-
-
IMO there is a fundamental difference between Erlang OO and Java OO to the point that bringing them up in the same conversation is rarely useful. Erlang actively discourages you from having pellets of mutable state scattered around your program: sure, threads are cheap, but that state clump is still a full-blown thread you need to care for. It needs rules on supervision, it needs an API of some kind to communicate, etc, etc. Erlang is at it’s best when you only use threads when you are at a concurrency boundary, and otherwise treat it as purely functional. Java, in contrast, encourages you to make all sorts of objects with mutable state all over the place in your program. I’d wager that MOST non-trivial methods in Java contain the “new” keyword. This results in a program with “marbled” state, which is difficult to reason about, debug, or apply any kind of static analysis to.
-
-
In all honesty, you sound quite apologetic to what could be arguably considered objectively bad design.
Attaching methods to types essentially boils down to scattering data (state) all over the code and writing non pure functions. Why honestly cannot understand how anyone would think this is a good idea. Other than being influenced by trends or cults or group thinking.
Almost the same could be said about inheritance. Why would fiting a data model in a unique universal tree be a good idea? Supposedly to implicitly import functionality from parent classes without repeating yourself. Quite a silly way to save a line of code. Specially considering the languages that do it are rather verbose.
-
Why honestly cannot understand how anyone would think this is a good idea. Other than being influenced by trends or cults or group thinking.
Here’s a pro tip that has served me well over many years. Whenever I see millions of otherwise reasonable people doing a thing that is obviously a terribly stupid idea, it is always a lack of understanding on my part about what’s going on. Either I am blind to all of the pros of what they are doing and only see the cons, or what they’re doing is bad at one level but good at a different level in a way that outbalances it, or they are operating under constraints that I don’t see or pretend can be ignored, or something else along those lines.
Billions of lines of successful shipped software have been written in object-oriented languages. Literally trillions of dollars of economic value have been generated by this software. Millions of software developers have spent decades of their careers doing this. The though that they are all under some sort of collective masochistic delusion simply does pass Hanlon’s Razor.
-
To be honest, the more I study OOP (or rather, the hodgepodge of features and mechanisms that are claimed by various groups to be OOP), the less room I see for a genuine advantage.
Except one: instantiation.
Say you have a piece of state, composed of a number of things (say a couple integers, a boolean and a string), that represent some coherent whole (say the state of a lexer). The one weird trick is that instead of letting those be global variables, you put them in a
struct. And now you can have several lexers running at the same time, isn’t that amazing?Don’t laugh, before OOP was popular very prominent people thought it was a good idea to have global state in Lex, Yacc, or error handling (
errno). So here’s my current guess: the success we attribute to OOP doesn’t really come from any of its overly hyped features. It comes from a couple very mundane, yet very good programming practices it adopted along the way. People attributed to the hyped stuff (such as inheritance) a success they have earned mostly by avoiding global variables.Abstract data types are amazing, and used everywhere for decades, including good old C. The rest of OOP though? Contextual at best.
-
It has been the opposite for me.
- typecast everything to and from object in early versions of java
- EJBs 2
- Bower package manager. Its creator wrote on stack overflow that he was confused when he created the project and that it was essentially useless.
- Ruby gems security incident
- Left pad fiasco
- Apache web server htaccess configs
I could go on with more esoteric examples to an ever growing list.
All these had criticism screaming long before they happened: why?
-
-
-
-
-
-
-
The first versions of Simula and Smalltalk didn’t have inheritance either. Self and other prototypal object-oriented languages don’t use traditional inheritance either. We still call all of them object-oriented.
Honestly, it’s well beyond time that we retire all programming language paradigm terms. Modern languages simply aren’t organized into paradigms they way older simpler languages were.
It’s like we’re looking at a Honda Accord and arguing over whether it’s a penny farthing or a carriage. The taxonomy no longer makes sense.
-
-
-
-
-
The main thing about OOP that needs to die is the idea that OOP is a coherent concept worth discussing on its own. Talk about the individual concepts as independent things! It’s much more productive.
-
-
I think “OOP” generally means “features of object-oriented languages that I don’t like” to a lot of people. The people using those languages don’t generally get into paradigm arguments.
(Personally, I consider inheritance to be common in OOP languages but not a particularly interesting or salient part of them. Many early OOP languages didn’t have inheritance and prototypal ones have an entirely different code reuse model.)
-
For some people “OOP” means “features of languages I do like”. For instance I’ve seen people include templates/generics/parametric polymorphism and unnamed functions as core parts of OOP… having learned CamlLight (OCaml without the “O”) in college, I confessed I was quite astonished.
-
-
You say that but it means different things to different people. I don’t disagree that your definition would be a good one if you could get people to agree on it, but I can’t assume that when other people say “OOP” that’s what they’re talking about.
-
-
-
-
-
I think it will come back, rediscovered as something new by a new generation disillusioned with whatever has been the cool solves-everything paradigm of the previous half decade. Perhaps this time as originally envisaged with a “Scandinavian school” modeling approach.
Of course it never left as the first choice for one genre of software… the creation of frameworks featuring default behavior that can be overridden, extended or changed.
Those languages you mention (Go, Zig, Rust) are primarily languages solving problems in the computer and data sciences, computing infrastructure and technical capability spaces. Something is going to be needed to replace or update all those complex aging ignored line-of-business systems.
-
-
There isn’t really any need to “hype” DSLs because they’re already widely used in all domains of programming:
- front end: HTML / CSS / JavaScript, and most JS web frameworks introduce a new DSL (multiple JSX-like languages, Svelte, etc.)
- back end: a bajillion SQL variants, a bazillion query languages like Redis
- builds: generating Ninja, generating Make (CMake, Meson, etc.)
- there at least 10 CI platforms with their own YAML DSLs, with vars, interpolation, control flow, etc.
- In games: little scripting languages for every popular game
- Graphics: scene description languages, shader languages
- Compilers: LLVM has its own TableGen language, languages for describing compiler optimizations and architecture (in the implementation of Go, a famously “not DSL” language), languages for describing VMs (Ruby)
- Machine Learning: PyTorch, TensorFlow, etc. (these are their own languages, on top of Python)
- Distributed computing: at least 10 MapReduce-derived frameworks/languages; there are internal DSLs in Scala for example, as well as external ones
- Mathematics and CS: Coq, Lem, etc.
All of these categories can be fractally expanded, e.g. I didn’t mention the dozens of languages here: https://github.com/oilshell/oil/wiki/Survey-of-Config-Languages – many of which are commonly used and featured on this site
If you think you don’t use DSLs, then you’re probably just working on a small part of a system, and ignoring the parts you’re not working on.
ALL real systems use tons of DSLs. I think the real issue is to mitigate the downsides
-
Oh yes but at the same time if you haven’t seen the hype for DSLs then you haven’t spent long enough in the industry to go through that part of the hype cycle. DSLs are what they are and it looks like we might be entering a hype cycle where people want to make them out to be much more.
-
I don’t agree, I’ve been in the industry for 20+ years, there are plenty of things more hyped than DSLs (cloud, machine learning, etc.)
DSLs are accepted standard practice, and widely used, but often poorly understood
I’m not getting much light from your comments on the subject – you’ve made 2 claims of hype with no examples
-
Here’s an example of recent hype https://www.codemag.com/Article/0607051/Introducing-Domain-Specific-Languages
Here’s some hype from the year 2000 https://www.researchgate.net/publication/276951339_Domain-Specific_Languages
Arguably the hype for 4GLs was the prior iteration of that specific hype.
I’m not arguing that DSLs are bad - I’m saying that they’re one of the things on the roster of perfectly good things that periodically get trumpeted as the next big thing that will revolutionize computing. These hype cycles are characterized by attempts to make lots of DSLs when there isn’t a strong need for it or any real payoff to making a language rather than a library.
-
-
-
I know it might sound a bit controversial, but the way I see it we need to reach a new level of abstraction in order for large-scale software development to be sustainable. Some people might say AI is the way forward, or some other new programming technique. Either way I don’t think we’ll get there by incrementally improving on the paradigms we have—in order to reach the next level we’ll have to drop some baggage on the way up.
-
I mean, humans aren’t getting better at groking abstraction, so I don’t know that “new levels of abstraction” are the way forward. Personally, I suspect it means more rigor about the software development process–if you’re building a tall tower, maybe the base shouldn’t be built with a “move fast and break things” mentality.
-
Groking abstractions isn’t the problem, at the end of the day abstractions are just making decisions for the users of an abstraction. Over-abstraction is the root of many maintainability woes IMO, the more a programmer knows what’s actually going on underneath the better, but only to the degree that it’s relevant.
-
-
-
-
-
I’ve much respect for Archive.org and Brewster but “Ever try reading an ebook you paid for 10 years ago?” and “Without active maintenance, we will be lucky if our digital books last a decade.” are stretching it by quite a lot.
I still have ebooks from Amazon from more than 10 years ago and I can read them with no problem. Maybe that won’t be the case in another 10 years, and there may be formats that don’t make it a decade – but there’s a lot of us walking around with 10+ year old collections via the Kindle or similar that aren’t (yet) having problems with our libraries.
While I agree with Kahle on a lot of the larger points, I worry that an average person reading this is going to note that they’ve had a Kindle for a decade and still manage to read “old” content just fine and consider the larger points suspect.
-
They’re concerned because they store books in arcane formats designed to be difficult to download, because you have to “borrow” ebooks from the Internet Archive digital library. No, don’t just convert it to EPUB 3 and add a download link, that’s too complicated - reprocess and “re-invigorate” it yearly for some reason and make it really hard to read. Borrowing ebooks with an hour lease… inane.
-
you have to “borrow” ebooks from the Internet Archive digital library. No, don’t just convert it to EPUB 3 and add a download link, that’s too complicated
You realize that if Internet Archive did what you suggest, they would be sued out of existence by book publishers? In fact, there is an ongoing lawsuit right now by 4 publishers asking $19,000,000 damages and the deletion of 1.4 million e-books. https://www.dailydot.com/debug/internet-archive-lawsuit/
-
They’re being sued out of existence by book publishers for offering to let you “borrow” ebooks so, I don’t imagine things would be any different if you could just scrape them. :D
-
-
I think it’s worth beating a drum that the Gutenberg Project is a failure. Every person you know, nerd or not, has heard of and used Wikipedia. Only a handful of nerds have heard of the Gutenberg Project. A major reason for this is that GP thinks that the only hard part of the problem is text entry. But distribution is just as important! Without making an easy to use and easy to read product, GP basically might as well not exist at all. The only people it benefits are grey market scammers who sell free books on Amazon for 99 cents.
-
The Gutenberg project offers 60,000 out of copyright books, which are mostly over 100 years old. It’s pretty niche.
This thread is about the Internet Archive, which is a much larger project offering a vast amount of current material, including books, movies and a historical archive of the world wide web. I use it frequently.
-
-
-
As someone who switched from Kindle to Kobo, my Amazon ebooks are useless unless I download them to my PC (Amazon already makes this a little annoying), download Calibre, download the De-DRM plugin, and strip the DRM from them. Maybe most people stick with Kindle for a longer period of time than me, but the fact that they might lose access to all of their Amazon books also makes it hard to switch away.
-
I stopped using my Kindle after Amazon pushed an OTA update that bricked my device. Now I’m on Kobo, but I’m still subject to the same problem if the Kobo people ever screw up my device.
-
-
Yeah, and there’s also a lot of stuff being swept under the rug here.
You can keep a physical book readable long enough for multiple future generations to enjoy it. But that requires both up-front and ongoing effort. Up-front to ensure the book was made using materials of sufficient quality and resilience to last that long. Ongoing to ensure it’s stored and cared for in a way that will minimize long-term damage to it from its environment, and potentially restoring it occasionally – new binding, new covers, careful cleaning, etc.
And most infrastructure for book storage is not particularly high quality these days. Home bookshelves/bookcases are also often made of cheap/lower-quality materials and designed to be thrown away and replaced every few years.
So when the article laments that digital books require “active maintenance”, my only response is “Well, so do physical books”. The implication that physical books just automatically last for centuries is not only wrong, it’s bordering on insulting to the people – librarians and curators and collectors and amateurs who love their books – who do put in the “active maintenance” work to preserve physical books, and whose work is simply erased by this framing.
-
Agreed—if you live in an area with relatively dry air then it’s not much of a problem, but humidity really does a number on books. Not to mention that if you don’t get expensive hardbacks the only thing holding a book together nowadays is a mixture of cheap glue and wishful thinking.
-
-
-
🔥
-
I’m sure they’ve factored this in, but something that would make me a bit nervous about hardware servers is the need to make sure you’re always over capacity. Not counting basic redundancy (eg extra application servers for temporary failover) you would also have to plan for handling load spikes and hardware failures.
As far as I understand the lead time for getting a new server could be in the order of days or weeks, so the consequence of underestimating your server needs could be running a degraded service for several days. With cloud you can tune your server capacity “just in time”, which would make me sleep easier at night at least.
(However, maybe the cost savings are so great you can run at something like 25% capacity as your baseline and still save money?)
-
When I ran the numbers for our service, the cost for AWS was ~7x higher than colo, because we’re bandwidth heavy. There’s definitely room if your usage profile is the right shape and high enough scale to make the ops worthwhile.
-
As far as I understand the lead time for getting a new server could be in the order of days or weeks
Couple of thoughts:
- At bigger providers you can usually get new servers within minutes, at other hours. Of course that’s different if you run your own datacenter, but again there’s a very big chance that you won’t be in a situation where you won’t be pretty good with a big additional server, because for a large amount of businesses adding even a single server with a couple dozen cores and 200GiB of RAM can get you quite far. And that’s something you get within minutes at big providers and probably can get quickly otherwise as well. And if you can anticipate it at all none of these will make issues.
- It’s usually A LOT cheaper to hold additional capacity compared to the big cloud providers and since you already keep that additional capacity it is certainly available faster than a cloud provider is able to provision it
- I’ve been in a situation where GCP (Google) was unable to add servers in a region for months (or more), which sucked because a contract that was made specified the country
- Yes the savings are that great. That’s why I would highly recommend at least giving it a bit of thought. I think you can typically go a lot higher than 25% capacity while still saving quite a bit of money. Depending on what you do you might even benefit from reduced overhead if you scale load vertically rather than horizontally to a degree. Of course you want to have your system to fail over to, but typically there is some overhead involved in scaling out, that you might not need to deal with when just scaling up, which is typically more expensive for cloud providers, because their calculation naturally is how many compute instance they can squeeze into a physical server. If you have know your load and have your planned free capacity you might be able to just get exactly what you need for hardware and for whoever runs the data center (you or some hosting company) it’s mostly just another one or two units in their rack.
Since the resources are less dynamic the prices ware too, which has the benefit of easier cost estimates, especially before starting a project. You need a basic grasp plus generous additional capacity and since things like traffic and so on are part of the package there are fewer things that might spike or that you need to optimize for on the costs side. You can say you need that server and an equivalent one to fail over for example and you get an exact price that’s also very stable. For some businesses that’s a benefit in itself.
-
Thanks for the great write up! I’m not really in a position right now where physical servers would make much sense, but if / when I get there I’ll definitely keep this in mind. I’m a big fan of keeping things simple, and as was pointed out in the OP, cloud is not always as simple as the vendors make it seem. Additionally, the way cloud pricing works you’re often incentivized to run a lot of small servers instead of a few big ones, which further increase operational complexity.
-
-
That depends though. It might not make sense on the latency side. In best case you use a hosting company with some basic semi-cloud offerings (ie hourly billed vServers). These are fairly common, and even places that mostly offer colo have these. Then you can just choose. I tend to also like starting out with essentially the cheapest dedicated server, if it’s affordable and simply not have to worry for the initial development and finding out eventual resource usage.
But that leads to another topic. In many situations that are some variation of your usual CRUD app the thing that really takes the load is the database, while all the applications do is waiting for it to respond and then serializing to JSON. The logic inside actual application for such projects is frequently really low. So you are anyways only bound by that and then the question is if you run your database yourself or use something managed. I tend to always go with running it myself for various reasons (it’s something I am comfortable with, I have hit shortcomings with managed instances on big providers, I see being able to use the latest features as a core advantage, etc.). Then when it actually grows I rent bigger servers and fail over to the bigger server. And databases are usually something where you can’t shrink again on managed instances - unless that changed. Also while there’s mechanisms to grow it’s usually something where you need to know how you grow for it to do so smoothly and not for example slowing down everything by a lot while it grows. And databases tend to be very expensive in managed/cloud solutions.
-
Yeah, that makes sense—if you spend the upfront integration cost you a hybrid solution I guess it lets you buy as much “earthquake insurance” as you think you need. (However, from reading the post it looks like it was this kind of solution they were migrating away from at Basecamp)
-
-
-
Python is an anti-social language. It focuses primarily on the needs of developers and not as much on the user of the software.
I think that’s close, but not quite it. It focuses on the needs of some developers to the exclusion of others. It does not enforce any sort of balance.
Say someone on your team has a Jupyter notebook they use to calculate something important to the company. Great! Okay, now try to run it on the developer who sits across from them at the office. Bzzt, nope, not going to work. You’ll need to start pip freezing and hope that there aren’t any important C build tools needed to get it working.
Have a web team with separate frontend and backend devs. Enjoy explaining to the frontend devs what a virtual env is and why they should care. :-) Python’s “it works on my machine” factor is so bad that it propelled Docker into prominence. But your coworker who is a good designer and HTML editor might not know what a PATH is, let alone Docker.
There’s the classic XKCD about his Python installs being a “superfund site”. Here’s the thing though: Randall Munroe is a developer.
This article is being poorly received here—just pip freeze or use Poetry or follow some other conflicting advice!—because for these developers there are patterns in place that work. But for me as someone who tries to bridge between different people at shops that aren’t all Python experts, it’s a nightmare.
-
-
Not to drag out the Go vs Python flamewar more than necessary, but I do think this is a good way to get a handle on the culture gap between the two languages. Lots of things in Go are inconvenient or ugly for the individual but good for the group or others.
So, eg it’s a pain to me that I can’t just version tag my projects however I want, but I have to use the ugly /v2 semantic version imports system. But for other users, it’s good that they have clear and simple rules about versions and importing that work across all packages. Lots of times, when you see someone having a problem with Go, it’s because they want to do something non-standard, and then the pro-Go people end up sounding like cultists because we say “oh no, it’s for your own good not to have unused imports” or whatever. But the core is the culture clash: am I working for Go or is Go working for me? If you work for Go, it benefits the ecosystem, but maybe you would rather just do your own thing.
-
To paraphrase a Jedi master: When as old as Python your favorite language is, look as good it will not.
Or to just directly quote Stroustrup: There are only two kinds of languages: the ones people complain about and the ones nobody uses.
Basically everything you complain about from Python is directly or indirectly a consequence of its age and widespread adoption. Check back in a few decades to see if Go gets to a similar spot.
Also:
This article is being poorly received here—just pip freeze or use Poetry or follow some other conflicting advice!—because for these developers there are patterns in place that work.
I see one pretty bombastic chain going after the author’s tone. You’re the only comment that’s mentioned
pip freezeso far.It may well be that people who aren’t you don’t have the same experiences you do with certain tools and come to different conclusions. You’re free to do what works for you, but you seem to spend an inordinate amount of time on this site, from what I recall, trying to put down other people’s preferences. Maybe do less of that?
-
Basically everything you complain about from Python is directly or indirectly a consequence of its age
Another phrasing: many of the things that older languages do poorly and newer languages do well are a direct consequence of people having learned from the older languages’ missteps.
-
Basically everything you complain about from Python is directly or indirectly a consequence of its age and widespread adoption. Check back in a few decades to see if Go gets to a similar spot.
I’m certain that languages like Go will accumulate warts over time, but I can’t help but think that newer languages will fair better structurally. There have been so many huge advances to basic programming practices in the past 30 years and older languages were not built with them in mind. Programming languages built 30 years ago didn’t have to think about package managers, formatter, fuzzing, testing, documentation sites, and probably a lot more.
I’m sure that there will be new things that today’s languages haven’t considered, but at least they have examples of how to accommodate those major features from the last 30 years to draw from.
-
I’m sure that there will be new things that today’s languages haven’t considered, but at least they have examples of how to accommodate those major features from the last 30 years to draw from.
Again, I think time is not going to be as kind as Go’s advocates want it to be. I’ll pick on one specific issue and one general issue to illustrate why:
A specific issue: Go, for all that people are promoting it as better than Python on “packaging”, initially shipped with… “just list some Git repos as strings at the top of your file” as the “package manager”. And a lot of details of Go’s “proper” added-much-later package manager are still influenced heavily by that early design choice. The infamous
/v2thing, for example, is a consequence of how bad Go’s initial ability to specify and target versions of dependencies was. And the whole interface of Go modules is kind of clunky and shows as having been retrofitted on after the language was already in pretty wide use, rather than something that was thoughtfully designed to go with the language. And with Go’s policies being what they are on compatibility (or at least in theory), I’m not sure it’s possible to improve significantly on this in the future.And a general issue: Go is, well, sort of infamous for actively refusing to learn from “the last 30 years” (and really more than just 30 years) of programming language design and evolution. It’s one of the most common criticisms of Go! The error handling, the weird quirks and limitations of the type system, the difficulty-bordering-on-outright-lack of interoperability, the massive hostility to generics… all of this and more is important stuff from the past few decades of learning about how to make programming languages that Go is proud of not making use of.
So I’m not convinced at all that Go is going to look significantly better at its 30th birthday than Python did.
-
Go, for all that people are promoting it as better than Python on “packaging”, initially shipped with… “just list some Git repos as strings at the top of your file” as the “package manager”.
TBH, I had forgotten about that. I can’t imagine that will age well. Just thinking about the Heroku and Gitlab free tier changes that have happened I can only imagine the sort of bit rot that might occur with direct references to GitHub repositories.
-
-
-
-
I think Python developers don’t really understand how bad it is because they know how to work around it. They have already picked their favorite python environment manager for their own projects, they know exactly what to do with other people’s code when it has a requirements.txt or a setup.py or a venv file or a pipenv file or a pyenv file or a virtualenv file or a poetry file or whatever else. And for system stuff, they make conscious and informed decisions about whether to install a package using their system package manager or using pip, and when using pip, they know whether they want the package locally or globally, and they know how to resolve problems where they have already installed a package globally with pip and their system package manager wants to overwrite the same files.
It’s not like it’s the only community with that problem. C++ people may be a bit too quick to dismiss concerns about how hard memory management is, because we know how to choose between unique_ptr and shared_ptr and raw pointers and references and stack allocation, and we may have found strategies to read the multi-megabyte template or overload resolution errors and find the nugget of useful information in the sea of useless logs. But in general, C++ people probably don’t expect non-C++ developers to deal with C++ code as much as Python developers expect non-Python developers to deal with Python code.
-
-
-
Markdown changed your “dash dash global” to “emdash global”, which made me misread your comment.
Virtual envs can be used like node_modules, but a) they aren’t the default b) they need to be activated (or install a tool to activate them) c) they also copy the Python libraries themselves, but not enough to keep it from breaking when I upgrade homebrew Python.
Python has all the pieces for dependency management, but it’s just pieces and you have to have someone on your team write a blog post about why at your company we’re all going to do X. No one writes a blog post about how they use node_modules. :-)
-
I didn’t say it’s perfect (or even good), but it’s pretty much the same situation as node just with bad defaults. If people understand node they should be able to understand the python situation. (And at this point I mostly assume that front end devs know node, which might be a poor assumption.)
-
No, you don’t have to “activate” a node_modules folder, and frankly, it’s very embarrassing and hacky that virtual envs require it. (Again, my otherwise competent coworker did not know what a PATH was. How can I explain sourcing a Bash script vs running it?) I first saw activation with Ruby Gems, and I remember thinking as an arrogant Python user, “oh those monkey patching Ruby people! They’re even monkey patching my shell variables, how gross!” Little did I know that Python would copy it soon after. It really should have been made a default many years ago now, and to TFA’s point, it’s very anti-social that it has not been made the default.
-
My dude, the question you posed was “how do I explain this to someone?” I proposed explanation by analogy. Analogy does not imply a 1 to 1 correspondence. It does not imply that the things are equal in value. The idea here is that this new thing serves a similar function to another thing that they might know. It serves as a great jumping off point to discuss the differences. (Nested dependencies vs one common dependency, how PATH works, etc.)
Do I love virtual envs? No, I don’t, but that wasn’t your question. I get the sense that you never really wanted an answer to the question. You wanted to complain. I don’t really want to listen to you complain so I’m going to ignore this thread.
-
It’s a rhetorical question. I haven’t worked with that colleague for three years.
Any analogy is going to have points of commonality and points outside the scope of the analogy, but “venv is like node_modules” does very little work as an analogy. The only things they have in common is having project dependencies somewhere. Everything else is different. I guess it’s better than having no point of reference, but it leaves a big gap.
-
-
-
-
-
My understanding is that packages installed with pip do not ‘own’ their dependencies - as in, if you install package A that requires version 1 of package B, package B will be in the global scope and could conflict with user-installed package C that need version 2 of package B. Is that correct?
-
-
-
-
-
Another favorite feature of mine is VLOOKUP, which is essentially a SQL JOIN for your spreadsheet. Pivot tables can be used for data aggregation, so if you combine them you’ve pretty much got yourself a relational database.
Something I heard just recently was that SPJ worked hard during his time at Microsoft to get functional programming into Excel; you can hear the story in this podcast: https://open.spotify.com/episode/2vJ0blId04xsVmQ8Mrt1bc?si=QcQFIWvCRb-baRP56p2OwA
-
Very interesting, thanks for sharing! Really cool that you took the time to write a technical paper about it as well. For anyone interested in the subject of Lisp for game development I can highly recommend this article about Naughty Dog’s use of a proprietary Lisp for their games: http://www.codersnotes.com/notes/disassembling-jak/
-
-
-
Here’s a link to Open GOAL, the ongoing attempt to reverse engineer this particular language.
-
That’s really cool. TBH it feels like GOAL and its implications, ie the viability to use a “highly dynamic” language like Lisp for game development, has really flown under the radar. Hopefully this project can bring more attention to the fact that you don’t need to build your game in C++ 😉
-
-
-
Hell yeah LFG!
-
It’s fun to see how different languages handle the same patterns. My fav Clojure relies on plain maps as it’s core data structure and explicitly rejects paragraphs like this:
Not only do dicts allow you to change their data, but they also allow you to change the very structure of objects. You can add or delete fields or change their types at will. Resorting to this is the worst felony you can commit to your data.
-
I haven’t coded that much Clojure (although I keep tabs on it), but as someone who jumps back and forth between Python and JavaScript I’ve noticed the same distinction.
I think the reason dicts are not used as much in Python as objects in JavaScript and maps in Clojure is that dicts feel like second-class citizens; first of all, dict access is awkward. Compare
foo[“bar”]with egfoo.baror(:bar foo)—both JS and Clojure have affordances for literal key access which are missing in Python.Secondly, tooling support (autocompletion etc) has historically been very poor for dicts. This has sort of changed with the introduction of types dictionaries, but then you have to wrestle with the awkward dict syntax again.
Ultimately I think this often is to Python’s detriment—for example, translating Python classes to a wire protocol often involves quite a lot of boilerplate. Luckily there are libraries like Pydantic that provide good solutions to this problem, but it’s still not as seamless as eg serializing Clojure maps.
-
It’s funny you mention JS because if I needed an arbitrary key-value map I would not just use literal keys, but rather a Map. Consider what happens if you try to use a key named ‘toString’, or ‘hasOwnProperty’.
I also don’t think you should just be shoving your in-memory representation over the wire unless it’s extremely simple, especially in situations where you might need to change it and your client and your server might get out of sync in terms of versions.
-
for example, translating Python classes to a wire protocol often involves quite a lot of boilerplate. Luckily there are libraries like Pydantic that provide good solutions to this problem, but it’s still not as seamless as eg serializing Clojure maps.
I guess I don’t really get this one.
Pydantic is the hot new kid on the block, sure, but if you’re building networked services this stuff is table stakes and has been for years and years. If you use Django there’s DRF serializers. If you don’t use Django there’s Marshmallow. In both cases the tooling can auto-derive serialization and deserialization and at least basic type-related validation from whatever single class is the source of truth about your data’s shape, whether it’s an ORM (Django or SQLAlchemy) model, or a dataclass or whatever.
So I literally cannot remember the last time I had to write “quite a bit of boilerplate” for this. Maybe if I were one of the people I see occasionally who insist they’ll never ever use a third-party framework or library? But that seems like a problem with the “never use third-party”, not with the language or the ecosystem.
-
-
-
At the same time, I understand Clojure has this inclination toward keys that aren’t just any old string, but are namespaced and meaningful.
And even with that, sometimes it can be very hard to get your bearings when you’re jumping in at a random piece of code. Figuring out what the map might look like that “should” go into a certain function can be very difficult.
I wonder if Clojure programs at a certain complexity would still translate from wire format maps to domain model maps?
I work on a complex codebase and we use a Malli derivative to keep the wire format stable and version the API. The internal Malli model is translated to JSON automatically through this spec, and it also ensures that incoming data is well-formed. It’s all rather messy and I’m not sure if I wouldn’t prefer manual code for this because Malli is quite heavy-handed and its metaprogramming facilities are hard to use and badly documented.
-
-
-
Dicts are not opaque, it’s wrapping them in bespoke custom classes that makes data opaque.
Dicts aren’t opaque in the sense of encapsulation, but they’re opaque in the sense of making it harder on the developer trying to figure out what’s going on.
If I’m working with a statically-typed codebase (via mypy), I can search for all instances of a given type. I can also look for all accesses of a given field on that type. It’s not possible to usefully do this with a dict, since you’re using dicts everywhere. You also can’t say “field X has type Y, field Z has type Q” unless you use TypedDict and then at that point you don’t gain anything from not using a real class.
Similarly, I can look at the definition for the class and see its fields, methods, and docstrings. You can’t do that with a dict.
I’ve been working with a codebase at $WORK that used dicts everywhere and it was a huge pain in the ass. I’ve been converting them to dataclasses as I go and it’s a lot more convenient.
-
-
I’ve used TypedDict as a transitional measure while doing the dicts-to-dataclasses thing; it was definitely super helpful there.
-
-
The primary idea being that if a dictionary is useful in a given circumstance, then a dictionary with type assertions is often even more useful. The motivation section of the PEP expands on that a little.
-
-
-
-
-
Please post the article here if you do write it, cuz I don’t know much about Python best practices.
-
-
-
-
My personal progression for systems is to start with something like heroku, and only when that starts being insufficient, move to something containers-as-a-service (ECS etc), and when that starts being insufficient, I move towards something like Nomad on EC2.
There’s a lot to be said for using Packer to make AMIs and rolling them in an ASG too.
-
Agree - I feel like prescribing ECS as an entry-level tier is really overkill if you’re just a small team trying to find product-market fit. Also, it’s not like Heroku is in any way a dead end; there’s very little lock-in (except for maybe an addiction to convenience) so if you ever want to migrate to Docker you just add a few more lines to your Procfile and rename it to Dockerfile.
Last time I used ECS it still required quite a bit of configuration to get up and running (configuring an ingress, availability zones, RDS, ECR, setting up CI, etc etc). Also, there were a few common use cases that were surprisingly difficult to set up, such as running a one-off command on deploy (eg for triggering database migrations) or getting logs shipped to Kibana, things which can be done with literally a single line of config or a few clicks of the mouse on Heroku.
TBH I’d rather run on regular compute nodes on something like Digital Ocean and deploy with Ansible than use ECS. Kubernetes and ECS feel like they’re solutions for the problems of managing a compute fleet—most people don’t have a compute fleet, but by using Kubernetes they get the same level of complexity anyway
-
-
Agreed - Heroku is really expensive once you go beyond the starter tiers. However, ECS and Kubernetes may be cheaper on paper, but what you’re really doing is trading hosting fees for man-hours. At a certain point that trade makes perfect sense (when you’ve got enough manpower) but I’ve seen several instances of people making the switch without realizing how costly it would be to do their own SRE.
-
-
I’ve always been surprised that the Packer -> AMI -> ASG approach isn’t more popular. (I mean, I get why it isn’t.) It can take you really really far. There was even a (short-lived) startup around the idea, Skyliner. It’s not very efficient, from a bin-packing/k8s-scheduling POV, but efficiency is not a high priority until you are at a scale where margins cause you real pain. So we’re in a place today where it is under-theorized, under-documented, and under-tooled, as an ops discipline, compared to more complex alternatives. Too bad, it’s like the best medium/middle-sized scale approach I know about.
-
-
Author here. I heard of Cue on lobsters actually but only recently looked into it.
( I swear someone had mentioned to me that it was much more familiar looking to DevOps types than DHALL, but I can’t find the comment now.)
This is my write-up on attempting to use it to validate some YAML.
-
-
Hi @chreke! Thanks for listening to the podcast!
-
-
-
I think the reason it’s primarily “grumpy old developers” (and I count myself amongst that crowd) complaining about software bloat is that we were there 20 years ago, so we have the benefit of perspective. We know was possible with the limited hardware available at the time, and it doesn’t put today’s software in a very flattering light.
The other day I was editing a document in Pages and it made my MacBook Pro slow down to a crawl. To be fair my machine isn’t exactly new, but as far as I can tell Pages isn’t doing anything that MS Word 2000 wasn’t doing 20 years ago without straining my 200Mhz Pentium. Sure, Pages renders documents in HD but does that really require 30 times the processing power?
-
This might be selective memory of the good old days. I was in high school when Office 97 came out, and I vaguely remember one of my classmates complaining about it being sluggish.
-
I think there’s A LOT of this going around. I used Office 97 in high school and it was dog shit slow (tick tick tick goes the hard disk)! Yes, the school could have sprung for $2,500 desktops instead of $1,500 desktops (or whatever things cost back then) but, adjusted for inflation, a high-end laptop today costs what a low-end laptop cost in 1995. So we’re also comparing prevailing hardware.
-
-
-
Word processing programs were among the pioneers of the “screenshot your state and paint it on re-opening” trick to hide how slow they actually were at reaching the point where the user could interact with the app. I can’t remember a time when they were treated as examples of good computing-resource citizens, and my memory stretches back a good way — I was using various office-y tools on school computers in the late 90s, for example.
Modern apps also really are generally doing more; it’s not like they stood still, feature-wise, for two decades. Lots of things have “AI” running in the background to offer suggestions and autocompletions and offer to convert to specific document templates based on what they detect you writing; they have cloud backup and live collaborative editing; they have all sorts of features that, yes, consume resources. And that some significant number of people rely on, so cutting them out and going back to something which only has the feature set of Word 97 isn’t really an option.
-
When a friend of mine showed me Youtube, before the Google acquisition, on the high-school library computers, I told him “Nobody will ever use this, it uses Macromedia Flash in the browser and Flash in browser is incredibly slow and nobody will be able to run it. Why don’t we just let users download the videos from an FTP server?” I ate those words hard. “grumy old developers” complain about software bloat because they’re always looking on the inside, never the out. When thinking about Youtube, I too was looking on the inside. But fundamentally people use software not for the sake of software but for the sake of deriving value from software.
In other words, “domain expert is horrified at the state of their own domain. News at 11.”
-
I think a large problem is that “timeless ideas” in computer science are often presented in the context of a specific technology—while the ideas may be timeless, the context isn’t, which makes it easy to dismiss both as outdated. For example, I remember getting a lot of value out of Damian Conway’s Perl Best Practices—it’s a great book, even if you’re not a Perl programmer, but you won’t realize that until you’ve read it. In a similar vein, MySQL Performance Tuning is a great introduction to understanding database query planning in general, but if you’re exclusively using Postgres you probably wouldn’t pick it up.
On the whole I’d like more books like The Pragmatic Programmer which takes a higher-level approach and isn’t explicitly tied to any specific language or technology.