Threads for elldritch
-
Pleasantly accessible! We need more articles like this in the ecosystem that explain scary-looking things like GHC extensions in terms of approachable and useful examples.
-
-
Oh, this is monadic do-syntax! It’s always awesome to see a more conventional language adopt generalized monads.
(For the uninitiated, the basic premise of monads is “a thing that is then-able”, and it looks like Gleam’s flavor is “a function that takes a callback as its final argument”.)
In JavaScript, the equivalent is async/await, which is a monadic do-syntax that’s hardcoded to the Promise monad. It’s awesome that Gleam has chosen to provide a general do-syntax here rather than taking the common route of only providing hard-coded support for specific common monads.
-
You may also enjoy Shape Up from the Basecamp folks, which discusses this idea using their notions of “scopes” and “slicing”.
-
This is a little behind the current ecosystem. For folks stumbling across this now, at work we use Rel8, which is a wrapper over Opaleye. Absolutely excellent - a lot of the Opaleye composability benefits with great ergonomics for common use cases.
-
Preventing all runtime panics seems like an impossible task. Your program can always run out of memory.
Given that programs therefore need to handle unrecoverable situations anyway, it doesn’t seem that bad to me to allow judicious use of panics by developers to signal “I’ve reached an unrecoverable state” or “the consuming developer has made a mistake here”, especially since panics can be caught.
Two other articles I’ve read before with interesting perspectives on this topic:
- Error Handling in Node.js, especially the section on operational vs. programmer errors. It’s interesting how focused on reliability and debuggability the original Joyent-centric Node community was! (Bryan Cantrill has a great talk on this.)
- The Error Model in Midori, also discussing how bugs aren’t recoverable errors.
-
I think that statement is right when it comes to programs that run on general purpose operating systems like Windows and Linux. I think it’s less accurate when it comes to say firmware where you might have no multitasking, no operating system, no dynamic allocation and have rules disallowing recursion (mutual or self recursion) to enforce strictly-bounded stack usage.
-
running out of memory is a good example of a tough error. But you can set up a program to reserve space for error handling (And yeah maybe that fails too, but if you have 0 memory and your program can’t even start up….)
I think that there’s a false dichotomy though. For example in Python I can write a web service, and have a wrapper around my views with a “catch-all” exception that will handle almost everything. It probaly won’t handle OOMs cleanly, but there are huge classes of “just give up” errors that get handled by this.
Of course having exceptions is a whole thing, and
Resultwins the ergonomic argument! But honestly having exceptions for “sorry, I give up here, please go up to some part of the stack and assume everything’s busted” is a very nice feature for not having your entire program fall over! -
-
Predictable performance is the real killer feature of NoSQL. NoSQL databases aren’t necessarily faster, but it’s easier to understand what exactly a query is mechanically doing (e.g. scanning this table) and therefore easier to understand query performance under load. Performance degrades slowly and predictably under load instead of suddenly and without warning when the query planner changes its mind. The sharding story is also mechanically simpler, which makes it easier to understand how sharding and adding nodes impacts performance.
The trade-off is that lots of these concerns (how should my query work mechanically?) get pushed onto the application developer, and the application has to build its own joins, FKs, and integrity constraints. Your schemas end up being more tailored for your specific access patterns, and it’s harder to write new queries ad-hoc. For a new query to be performant, you’ll sometimes need to build a new set of tables.
Postgres vs. Dynamo is a lot like functional vs. imperative: you’re trading expressiveness for more predictable and controllable performance. AWS is an example of a team where this trade often makes sense - they gave a great talk about how they require teams to use Dynamo so that performance is stable under high load.
-
I strongly agree with this perspective.
Lots of programming (especially as taught in school) is about the relationship between the programmer and the machine. How do I get the computer to do the thing I want? What are techniques to get desired behavior out of the computer?
I think a key difference between software engineering as a profession and programming as a craft (e.g. as it’s taught in schools) is that an enormous part of software engineering is about preserving the intent of your code. Without preserving intent (“the program does X with the intent of accomplishing goal Y”), coordinating changes to a program done by multiple people over a long period of time becomes very difficult. This is because it becomes hard to tell which parts of the program are meant to work the way they do vs. just incidentally working the way they do. Is this weird if-statement with a specific condition a temporary hack, or is it a deliberate code path that handles some known edge case? Without knowing this, it’s much harder to determine whether deleting this if-statement cleans up some now-irrelevant code or will cause a spooky breakage elsewhere in the system.
Preserving intent comes in lots of ways. Sometimes it’s using code idioms, sometimes it’s comments, sometimes it’s design docs. But it generally needs to be a deliberate goal of the writer. Many heuristics for writing code such as DRY or SOLID are at their core about preservation of programmer intent.
Peter Naur has a great article on this called Programming as Theory Building, where he calls this “preserving the theory” (or “preserving the program”).
-
I work on a team that primarily writes Haskell. We’re part of a small-ish (~20 person) engineering org where most people come in knowing Go or JavaScript or Ruby. We’ve had good (surprising!) success onboarding new engineers to Haskell. Zero to production-ready code review takes around 6 weeks, which isn’t that different from most languages.
Because of this, I’ve had quite a bit of experience teaching working programmers what monads are, and seen a ton of different approaches. Pedagogically, I would say that “monads as monoids of endofunctors” is probably one of the least effective approaches we’ve tried. There’s just too many concepts here that you need to introduce at the same time. Monoids make sense to most people (programmers write a lot of monoids day-to-day, so it’s easy to motivate a monoid with concrete examples like strings, lists, numbers, or Builders), but endofunctors and categories usually do not. What is a category? What is an example of a category in code I’ve seen before? These questions tend to trip people up because it’s hard to understand something until you’ve played with it for a bit (alas, You Can’t Tell People Anything).
The approach I’ve found to be most effective is to start from something a working programmer has seen before, and compare and contrast. Fortunately, most programmers usually do have a good starting point! Depending on their native language, this is usually a Promise or a Future or a continuation or a Stream - usually some kind of object that represents a computation, where these objects can be sequenced to construct larger computations. I’ve had great success working with JavaScript programmers and telling them “monads are a lot like then-ables (Promises), and async/await is a lot like do-syntax, and here’s where they’re similar, and here’s how they differ”.
I think this approach succeeds because it’s much easier to teach a concept by starting from something that someone has actually used before (and by using their existing vocabulary) than by starting from scratch. If I tell someone “monads are values that represent computations that can be sequenced”, I get a lot of blank stares. If I follow that up with “just like Promises!”, I see a lot of lightbulbs go off.
(This is why I think it’s actually particularly hard to teach applicative functors. Lots of Haskell pedagogy attempts to go in the order of the typeclass hierarchy, teaching functors then applicatives then monads. I think it’s actually more pedagogically effective to skip around, teaching functors then monads then applicatives, because it’s hard to motivate applicatives as “functors containing functions” and easy to motivate applicatives as “weaker monads when you don’t need the power and you want to constrain what the computation can do”.)
-
“Monoid in the category of endofunctors” is true statement, but is an old joke that escaped containment.
Can you elaborate on your async/await comparison? I haven’t been in the JS world for a long time.
Re:
Applicative: I used to help teach the System-F (nee Data61) FP Course, and we seemed to have reasonable success motivatingApplicativeby trying to map a binary functiona -> b -> cover a functorf aand getting “stuck” with anf (b -> c). This motivatesApplicativeas the typeclass that provideslift0andlift[2..\infty]functions, if you think ofFunctoras the typeclass that provideslift1(as an alternative perspective onfmap).-
map a binary function
a -> b -> cover a functorf aThe hard part about this example is that the immediate response is “why would I ever do this?”. I don’t think I have a compelling answer for that. I’ve found that helping someone answer “why would I ever use X?” for any concept really helps them understand the concept better.
Can you elaborate on your async/await comparison?
Sure! TL;DR: async/await is do-syntax for the
Promisemonad.JavaScript has a concept of “Promises”, which are asynchronous computations that will eventually return a value. Promises are more convenient than using callbacks since you avoid the Pyramid of Doom problem (although they trade this off with some annoyances of their own), and most of the JS ecosystem has settled on Promises as a primitive for asynchronous computation. (Fun fact: Node actually had Promise support really early on, then switched to callbacks for most standard library functions, then switched back to Promises! There’s a really interesting aside about this in Platform as a Reflection of Values, a talk by Bryan Cantrill who was there at Joyent at the time.)
Promises provide a couple primitives:
-
resolve(value: T): Promise<T>, which takes a regular value and “wraps” it into a Promise that immediately provides the value. -
then(this: Promise<A>, f: A -> Promise<B>): Promise<B>, that takes aPromise<A>and a functionA -> Promise<B>and provides you with aPromise<B>. Mechanically,thenwaits untilthishas a value ready, and then runsfand returns the result.
(There are some other primitives for constructing promises from callbacks, synchronizing multiple promises, doing error handling, etc., but they’re not important for this comparison.)
Why do you have to use
thenin order to call a function using the Promise’s value? Because the Promise’s value isn’t guaranteed to be there yet! What you’re really saying is “here’s a computation that I want to run once the value becomes available”. (If you know monads, this should start sounding familiar.)So programmers usually wind up with a bunch of code that looks like this:
somethingThatReturnsAPromise() .then(a -> { // Do a bunch of synchronous stuff, // then end up doing something // asynchronous, and return a Promise. }).then(b -> { // etc. }).then(c -> { // etc. }) // This eventually becomes a long chain.Later, JavaScript adopted what’s called “
await/async” syntax, which is a syntax sugar that allows you to write code like this instead:// Some types: // somethingThatReturnsAPromise: () -> Promise<A> // someOtherAsyncThing: A -> Promise<B> // yetAnotherAsyncThing: B -> Promise<C> a = await somethingThatReturnsAPromise() // Do stuff... b = await someOtherAsyncThing(a) // etc. result = await yetAnotherAsyncThing(b) // result: Promise<C>This is a bit clearer and a bit more convenient to write, and basically desugars down to the same thing.
Now, how is this all related to monads? Well Promises are actually just an instance of monad! (Not really because of implementation details, but they’re monadic in spirit.)
Let’s look at how they compare. Monad is a typeclass, which basically means it’s an interface (or really more like a Rust trait). (Again not really, but correct in spirit - most of what I’m saying here is going to be not-super-technically correct.) A type is-a monad when it implements that interface. The monad interface basically looks like:
interface Monad for type T<A> { resolve(value: A): T<A> then(this: T<A>, f: A -> T<B>): T<B> }(Haskell’s actual
Monadtypeclass names these functionsreturninstead ofresolve, andbindinstead ofthen.)Hold on! Doesn’t that look familiar? Yes, this is almost exactly what Promise provides! Now let’s look at do-syntax:
// Some types: // somethingThatReturnsT: () -> T<A> // someOtherThingThatReturnsT: A -> T<B> // yetAnotherThingThatReturnsT: B -> T<C> result: T<A> // Where T is a monad result = do a <- somethingThatReturnsT() // Do stuff in let-bindings... b <- someOtherThingThatReturnsT(a) // etc. yetAnotherThingThatReturnsT(b)Huh… isn’t that almost exactly async/await? Right again.
Why is this useful? Two reasons:
- It turns out that there are lots of monads. This idea of “give me some computations, and I’ll sequence them together in a certain way” turns out to be very general. In Promise, the “way I sequence computations” is “wait until the asynchronous value is available”. In Maybe, it’s “don’t run the next computations if previous computations didn’t succeed”. In State, it’s “let future computations
askfor a value that previous computations haveput”. All of these implementations of Monad differ in how they implementthenandresolve. - It turns out that you can write functions that are general across every single kind of monads! For example, you can write
whenorunlessorforeverorsequence(seeControl.Monadfor other examples). This makes it easier to reuse code and concepts across different kinds of monads. If you can also write your business logic to work across different kinds of monads, it also makes testing easier: you can use a monad that provides logging by writing to disk in production, and swap that out for a monad that provides logging by saving log messages in memory in testing, all without changing a single line of code in your business logic. Monads are like type-checked, effect-tracked dependency injection! (This idea is its own whole blog post - TL;DR: monads and DI are both about effects, so they wind up being used for similar purposes.)
Sorry in advance for typos or incoherence - most of this is off of the top of my head.
-
Thanks for the detailed post. Is your type signature for
then(this: Promise<A>, f : A -> B)incorrect? It should be aPromise<B>, right?The hard part about this example is that the immediate response is “why would I ever do this?”
We’d usually motivate this with the
Maybeapplicative: applyfto both arguments if both areJust; returnNothingotherwise. Then consider other applicatives and notice how an embarrassingly large amount of imperative programming is subsumed by things you can build out of applicative operations (traverseandsequencein particular). -
-
-
-
I understand your approach, and while I think it’s effective, it does have some well-known weaknesses. The most important thing to emphasize is the algebraic laws, but your teams are working in Haskell, which is oblivious to those laws; it is easy for programmers to learn the entirety of the Monad typeclass in practice but never realize the laws. (I was such a Haskeller!)
What is a category?
C’mon, really? I know that you’re just playing, but this question really needs to be given a standard answer. A category is a collection of structures plus a collection of transformations from one structure to another. We usually call the structures “objects” and the transformations “arrows” or “morphisms”, but this is merely a traditional note so that readers can read common category-theoretic documentation. The main point of a category is to consider composition of transformations.
What is an example of a category in code I’ve seen before?
The following systems are categories which I’ve used in serious code. Each system has a proper title, but this is again merely mathematical tradition. This should not be a surprising collection; computer scientists should recognize all of these components, even if they’ve never considered how they’re organized.
-
Set, whose objects are sets and arrows are functions -
Rel, whose objects are sets and arrows are relations -
P, whose objects are sets and arrows are permutations -
Circ, whose objects are the natural numbers and arrows are Boolean circuits -
Mat_R, whose objects are the natural numbers and arrows are matrices over some semiringR -
Eff, whose objects are sets with computable equality of elements and arrows are computable functions -
Pos, whose objects are partial orders and arrows are monotone maps
In addition, every programming language gives a category via the native type theory construction. This is actually misleading for Haskellers, since the category Hask is actually bogus, but there is a non-
Haskcategory which accurately describes how Haskell transforms data.-
The most important thing to emphasize is the algebraic laws
Yes, agreed. Something I’ve found is useful for teaching is to explicitly map the maximally precise terminology that the Haskell community likes to use back to similar, vaguer terminology that working programmers are familiar with.
For example, I’ve found more success teaching “algebraic laws” as “coding conventions but motivated because look at how useful these equalities are”. Most programmers are familiar with conventions (e.g. don’t do side effects in your
toStringimplementation), but I’ve found a lot of people get hung up on the terminology of what “algebraic” means and what “laws” are.A category is a collection of structures plus a collection of transformations from one structure to another.
The difficulty I’ve had with a definition like this is answering “why is looking at something as a category useful to me as a programmer?”. I’ve found that providing a satisfying answer to this question for any concept (“why is X useful?”) helps tremendously with making the concept stick.
There are definitely categories in the list of examples you’ve provided that I’ve worked with (e.g.
Set,Rel,P) in other languages, but where I’ve been struggling is providing a satisfying answer to “if I could look at this as a category in a programming language I’m familiar with, what would be easier?”.It’s been relatively easier for me to answer this question for ideas like effect tracking (“your logging library will never secretly call the network again”) or algebraic data types (“you’ll never have a struct whose fields are in an inconsistent state”), but I haven’t found a satisfying answer for categories yet. If you’ve heard of one, I’d love to use it!
-
A disconnect between (nominally) mathematical and pragmatic perspectives I’ve noticed that seems important to address: people who aren’t conditioned to think as mathematicians do tend to, understandably, relate to mathematical invariants as inscrutable laws passed down from the universe, when in fact they are just as much determined by a criterion of usefulness as any programming construct is. Take the case of linear transformations as a tensor product between vectors and covectors: considered on their own, the only matrices you get from just that operation are singular. Such a purity of definition results in a universe of discourse (singular matrices) that is, frankly, not very useful, so we go on to consider sums of such products so that we can talk about linear transformations with full rank.
It’s certainly the case that what counts as “useful” to a mathematician might very well diverge wildly from what counts as useful to a programmer, but it’s been my experience that conveying the understanding that mathematical constructs are chosen as much on the basis of what they can do as are programming constructs is an essential aspect of answering the question of “why is framing this practical concern in terms of a more abstract mathematical construct useful to me as a programmer?”
-
All that said, though, I still harbor a suspicion that most invocations of category theory in the context of programming are fetishistic and convey little in the way of enhanced understanding of the shape of the problem at hand. The mere fact of dealing with a problem where one is operating on objects within a category and morphisms between them doesn’t at all imply that conceiving of the problem in category-theoretic terms is useful, even in the case of a lone creator who is free to organize their work according to whichever models they see fit, and much less in the case of someone who must work with others (which is to say, someone whose work could ever matter) and develop a shared mental model of their collective project with unquestionably competent people who nonetheless are in general unlikely to share the category-theoretic lens. It’s pretty silly to me to cite the mere fact of any one category’s objects and morphisms being a frequent domain of consideration as evidence of the notion that thinking of that domain in terms of its categorical representation is inherently useful.
-
If literally nothing else, category theory is tightly related to programming language theory and design via type theory. I’ve explained before how this works for the MarshallB programming language, but it also works for languages like Joy. Categories also show up in computability theory, due to the fact that Turing categories are restriction categories.
-
I’m absolutely in agreement with you on that, in that category theory provides an extraordinarily useful set of tools for modeling constructive type theories. Again, though, among all the problems that might ever be solved by a Turing-complete symbol system, how many benefit notably from a category-theoretic framing? To be clear, I’m coming at this from the position that cases like Haskell’s Monad typeclass don’t qualify, since they seem to be constructs that are merely influenced by category theory rather than being faithful representations of some category-theoretical construct: consider that
returnas implemented isn’t actually a natural transformation η in the theoretical sense, because it doesn’t actually map between functors: the identity functor is merely implied or assumed, even though the input is merely an object of Hask that need not be a functor itself.
-
-
-
-
-
-
Thanks, the description of explaining things in terms of what the programmer already knows helped me a lot. I have been working in c# for years and I knew that I have been using and understanding the use of monads most of that time through linq. But I was having issues translating that to a more abstract definition and overall concept. After reading your comment I just googled ‘monads in c#’ and got this https://mikhail.io/2018/07/monads-explained-in-csharp-again/ which explained to me how what I already new related to the more general concept.
-
-
Admit it. If you browse around you will realize that the best documented projects you find never provide that docs generated directly from code.
Is this saying that you shouldn’t use Javadoc or pydoc or “cargo doc”, where the documentation is located in the source files? So, from the previous point, it’s essential that docs live in the same repo as the code, but not the same files as the code? Seems like a pretty extreme position relative to the justification.
-
As a concrete example, Python’s official documentation is built using the Sphinx tool, and Sphinx supports extracting documentation from Python source files, but Python’s standard library documentation does not use it - the standard library does include docstrings, but they’re not the documentation displayed in the Standard Library Reference. Partially that’s because Python had standard library documentation before such automatic-documentation tools existed, but it’s also because the best way to organise a codebase is not necessarily the best way to explain it to a human.
As another example in the other direction: Rust libraries sometimes include dummy modules containing no code, just to have a place to put documentation that’s not strictly bound to the organisation of the code, since
cargo doccan only generate documentation from code.There’s definitely a place for documentation extracted from code, in manpage-style terse reference material, but good documentation is not just the concatenation of small documentation chunks.
-
Ah, I was thinking of smaller libraries, where you can reasonably fit everything but the reference part of the documentation on one (possibly big) page. Agreed that docs-from-code tools aren’t appropriate for big projects, where you need many separate pages of non-reference docs.
-
-
There’s definitely a place for documentation extracted from code, in manpage-style terse reference material, but good documentation is not just the concatenation of small documentation chunks.
Can’t agree enough with this. Just to attempt to paint the picture a bit more for people reading this and disagreeing. Make sure you are thinking about the complete and exhaustive definition of ‘docs’. Surely you can get the basic API or stdlib with method arity and expected types and such, but for howtos and walkthroughs and the whole gamut it’s going to take some effort. And that effort is going to take good old fashioned work by technical folks who also write well.
It’s taken me a long time to properly understand Go given that ‘the docs’ were for a long time just this and lacked any sort of tutorials or other guides. There’s been so much amazing improvement here and bravo to everyone who has contributed.
On a personal note, the Stripe docs are also a great example of this. I cannot possibly explain the amount of effort or care that goes into them. Having written a handful of them myself, it’s very much “a lot of effort went into making this effortless” sort of work.
-
-
Relevant links:
- Elixir Documentation index
- Elixir standard library documentation looks like it has plenty of non-reference documentation and is generated from the code.
- However, Elixir’s Getting Started guide from the website’s Learning section is written as standalone documentation, in a repo separate from the code.
-
-
The entire D standard library documentation is generated from source code. Unittests are automatically included as examples. It’s searchable, cross-linked and generally nice to use. So yeah, I think this is just an instance of having seen too many bad examples of code-based docs and not enough good ones.
When documentation is extracted from code in a language where that is supported well, it doesn’t look like “documentation extracted from code”, it just looks like documentation.
-
Check out Four Kinds of Documentation. Generated documentation from code comments is great for reference docs, but usually isn’t a great way to put together tutorials or explain broader concepts.
It’s not that documentation generation is bad, just that it’s insufficient.
-
-
-
As someone whose written Lisp, Ruby, and Node on one side, and Go and Haskell on the other, I’ve always been curious about this apparent split between strongly typed languages and languages with really good interactive connect-to-production drop-into-a-useful-debugger-on-crash REPLs. Is this intrinsic to the language, or just a result of differing focuses on the tooling?
I’m currently leaning towards “this is just a side effect of tooling focuses”, although I have yet to see even a really compelling UX sketch for a debug-it-in-prod REPL for a typed language. (For example, what would it mean to rebind a name to a value of a different type while paused computations may still refer to that name after resuming?)
-
It depends what you mean by “debug”. If you want to inspect values, execute functions, modify value of same type - that’s supported by a lot of languages. GDB will let you do all three.
But the idea of using a value of a different type is… interesting. I don’t see why would you ever want to do that in production. You can of course replace an object with a vtable-compatible one. (Or however your language does dispatch) But a completely different type? Technically: You’d need to also replace all the following functions to match the change and all places which produce that value. Practically: Why not take a core dump and debug this outside of production instead?
(An incompatible type also doesn’t make sense really - the code won’t do the right thing with it in about the same way a dynamically typed language wouldn’t)
And finally as a devopsy person I’d say: “if you feel the need to drop into a debugger in production that means we should improve tracing / logging / automated coredump extraction instead.”
-
I’ve always been curious about this apparent split between strongly typed languages and languages with really good interactive connect-to-production drop-into-a-useful-debugger-on-crash REPLs
This split is indeed interesting and it seems deep and hard to overcome. Languages emphasizing static typing provide a certain level of confidence, if it typechecks it is likely correct. But this requires an enforcement of type constraints across the program otherwise that promise is broken.
Interactive languages are more pragmatic. In Smalltalk, modifying a method in the debugger at runtime will compile and install it in the class, but existing instances of that method on the callstack will refer to the old one until they are unwound. There basically is no distinction in runtime / compile time which in practice works very well but is not really compatible with the confidence guarantees of more statically typed approaches.
I always wondered what a more nuanced compile time / runtime model would look like, basically introducing the notion of first-class behavior updates like Erlang. In different generations of code different type constraints apply. Which in turn means that for soundness the generations wouldn’t really be able to interact.
-
-
This reminds me of one of my favorite blog posts of all time: You can’t tell people anything. This is a must-read for every new project leader on my team.
It’s just so hard to go from not knowing a thing to knowing that thing without having actually done the thing. Systemization of a process can help scale new processes a bit. But at the end of the day, organizations are made of people, and no amount of process or systems or tools will save you from people who don’t get it. Your people have to get it first, then the systems and processes come after as a way to scale the learnings of those people.
(For the equivalent idea on the technical design side, see Programming as Theory Building.)
-
What are some alternatives to SQL that avoid some of these pitfalls?
One really interesting SQL library I’ve seen is opaleye, which bypasses a lot of the incompressibility issues by constructing queries in a Haskell DSL using reusable and composable monadic primitives, and then compiling those down into SQL (and automatically generating the repetitive parts of the query as part of that compilation).
-
-
I might be mistaken, but I don’t understand how this tool or the practices in this article get you a truly repeatable build.
To me, having a repeatable build means you produce the same binary artifacts when you build your checked-in source code no matter when or on what machine you run the build on. But using a tool like Docker seems to already make this impossible. If a Dockerfile allows you to
RUN apt-get install foo, then running that command at time T1 will give you a different answer than running at time T2.It seems to me like you can’t have real repeatability unless you have repeatable dependency semantics all the way down to the language level for each dependency manager you’re using. Tools like Blaze get around this by forcing you to use their own dependency management system that basically requires you to vendor everything, which guarantees repeatability. But I don’t see an analogous system in Earthly.
-
We debated ourselves a lot about what the right term is for this. From community feedback, we learned that there is a term for 100% deterministic builds: Reproducible builds. Earthly is not that. Bazel is. Earthly (and the practices that this guide talks about) has some consistency, but as you point out, it doesn’t get you all the way. We called this “repeatable builds” as an in-between term. The reasoning is that for many use-cases, it’s better if you are compatible with most open-source tooling, but are not 100% deterministic, rather than go all the way deterministic, but usability is heavily impacted.
-
-
-
elldritch
2 years ago
|
link
| on:
Software designers, not engineers: An interview from alternative universe
I like this article a lot. Two particular insights that really resonated with me:
- “The most important thing in software design is problem framing” (and all of its presented corollaries e.g. assuming you’re working with an ill-defined problem, etc.).
- “Knowledge about design is hard to externalize” (and therefore most learning is done through apprenticeship).
These two stood out to me because they both align strongly with my experience, and seem like they’re not really taught in a standard CS curriculum. School definitely taught me a lot about how to program, but very little about how to solve problems with software.
It feels almost like my university curriculum was missing a finishing course in actually solving real problems (although I can imagine designing such a course would be really difficult since it would need to effectively be an apprenticeship - maybe this is the role that internships are intended to fill?).
-
Thank you for making this! It’s always surprising to me (coming from Go and Node) when I find basic tooling like this that’s missing in Haskell.
My work projects use Cabal, but I’ll take a look at adding Cabal support when I find the time.
-
Sweet! Actually I added support tonight. I’ve never used a cabal.project file, so I kind of guessed, but if you want to try it out for a spin and let me know any issues in a DM please do: https://github.com/dfithian/prune-juice
-
-
This is a particular reputational problem for unusual language choices because failures of a team or product tend to be blamed on the weirdest things that a team does. When a Go project fails due to design issues, people rarely say “this failed because of Go” because there are so many examples of projects written in Go that did not fail. (On the flip side, design failure is also more likely because there are so many more examples of successful Go projects to follow than there are examples of successful Haskell projects.)
-
On the flip side, design failure is also more likely because there are so many more examples of successful Go projects to follow than there are examples of successful Haskell projects.
I feel like this is a big part of it. Another issue is that often the developers choosing languages like Haskell are less likely to have used it at a large scale than people working with conventional languages. For many people this is because you don’t get the opportunity to work with a language at scale without using it at work.
-
-
For an average project… I seriously doubt the “less likely to have used it at a large scale” part is true. Many teams don’t have a person on board who’d have practical experience using something at an actually large scale, much less designing for a large scale.
Just because there are more people with that experience in the general population, doesn’t mean a particular team has one of those. Also, many projects never reach a truly large scale, and with modern hardware, large is really quite large.
Of course, “using $language isn’t a stand-in for good software design”. But first, it’s true for any language, and second, there are already enough people trying to discourage everyone from moving beyond the current mainstream.
-
I can’t tell you the number of Python shops that said the same thing, only to have their engineers perform expensive rewrites in more performant language runtimes. Twitter here is the poster child of doing a large-scale rewrite from Rails into Scala for performance reasons. It’s true that large scale does not happen to every company, but I also think optimizing solely for developer ergonomics will put you in a bind any time you do need to scale.
Of course, “using $language isn’t a stand-in for good software design”. But first, it’s true for any language, and second, there are already enough people trying to discourage everyone from moving beyond the current mainstream.
Who? So many Silicon Valley companies have seen success embracing new technological stacks (like Rails and Django when they were hot, to Scala when it became cool, Go when it came onto the scene, and now Rust), so I can’t see why a new company would not want to use a new language if their devs were comfortable in it and confident in its ability to scale to the market they wished to address.
the current mainstream
There’s wisdom in accepting tried-and-true solutions. I’ve dug into new language libraries only to find undocumented settings or badly chosen constants more times than I can count. Edge cases in LRU caches, bugs in returning connections back into connection pools, circuit breakers that will oscillate between on-and-off when degenerate conditions are hit, these are all examples of algorithms that were incorrectly implemented, and it’s not always clear if the value proposition of a new language overcomes the complexity of implementing tried-and-true algorithms. Choosing an “unproven” language is not a light decision, especially when several engineers will spend their time and money on making it work.
-
Most places aren’t Silicon Valley. Most companies aren’t startups. Most projects aren’t web services with open registration.
And it’s not about optimizing for developer ergonomics. Advanced type systems are about correctness first of all. It’s just sort of taking off with Scala, Swift, and Rust. Sort of. Those clearly still face more opposition than 70’s designs like Python, Ruby, or Go.
-
Most places aren’t Silicon Valley. Most companies aren’t startups. Most projects aren’t web services with open registration.
According to StackOverflow [1], Typescript is a top 10 language, and Rust, Swift, and Typescript are top 20 languages. So while there is a bias for tried-and-true languages like Python and Java, languages with “advanced” type systems are in the top 20.
Advanced type systems are about correctness first of all
Is it? I’m not sure if there’s any proof that advanced type systems actually lead to more correct programs. I certainly don’t remember that when I was in grad school for PLT either. I understand that practitioners feel that type systems allow them to make more correct programs with less cognitive load, but I would bin that into developer ergonomics and not correctness until there’s proof positive that ML-based type systems do actually result in more correct programs.
It’s just sort of taking off with Scala, Swift, and Rust. Sort of. Those clearly still face more opposition than 70’s designs like Python, Ruby, or Go
Or, perhaps, most programmers genuinely do not enjoy working with heavily restrictive type systems or with monad transformer stacks and free monads. Haskell is a year older than Python and is an order of magnitude less widely used than Python. It’s clear that recent languages like Rust, Swift, Typescript, and Scala do use a lot of ideas from ML-descended type systems but it’s not clear whether they are popular as a whole or not. Many languages have taken pieces of these type systems (such as the
Optional/Maybesum type) and bolted them onto their language (Kotlin), but I think that ML-style languages will always be enjoyed by a minority, though how large that minority is time will tell.-
I understand that practitioners feel that type systems allow them to make more correct programs with less cognitive load, but I would bin that into developer ergonomics and not correctness until there’s proof positive that ML-based type systems do actually result in more correct programs.
This is an accurate statement I think. No type system can guarantee you are more correct unless the “spec” you are implementing is more correct. What it does help with is giving the developer more information on where they are starting to deviate from the spec. But even then it doesn’t guarantee that they are implementing the spec correctly in the type system. ML Type systems give you early information that your code won’t work as implemented when it hits certain edge cases. Those edge cases will sometime but not always manifest as bugs in production. So depending on the ergonomics you favor in your programming language you will either love that early information, be ambivalent, or you will dislike it.
-
-
-
-
-
-
-
michiel
2 years ago
|
link
| on:
What Silicon Valley "Gets" about Software Engineers that Traditional Companies Do Not
Let’s reframe this: maybe it’s more challenging to be an engineer (or a software manager) in a small company with a small customer base. Where building the wrong feature is a mistake that will cost you a year’s revenue. Where alienating your main client will lead to insolvency. Where you can’t just launch a new product and sunset it two years later.
FAANG’s enormous scale and customer-lock in makes it very inconsequential for them make mistakes. I’m sure that’s very comfortable for the engineers working there. But I wouldn’t make the mistake of confusing the dividend of those companies’ enormous success with the source of it.
-
This is an important point. The motivations of an individual engineer (learning, problem solving, building a CV) may not automatically align with the motivations of a company (sustainable operations). This is not to say that large slathers of management are needed to align the motivations, it’s just that this alignment is needed.
-
I’m not sure size is the key factor here. I worked at Google and now I work at a series B startup (joined at 3, now at 50) and we do a lot more of “makers (engineers, designers, etc.) talking to customers” at the startup than at Google. In fact, one of my biggest complaints about Google was that the game of telephone between the person making something and the person using the thing that was made was so long that it was very difficult as a maker to get meaningful feedback on whether you were building the right thing (because feedback from customers got passed through various PMs, etc. and lost detail at each step).
-
I’ve worked for a company that prided itself on its customer support. And to be fair, their people were really good at talking things over with customers, making them feel appreciated, maybe offer a little discount for the next month. Anything rather than admit there’s a bug and escalate it. I think that strategy worked well for the company, but it made product development rather frustrating.
-
-
-
This sort of article is deeply frustrating to me. I manage a team that develops a Haskell codebase in production at work, and for every engineer that (rightly) looks at this sort of incoherent, imprecise rambling and dismisses it as FUD, many more add it to their mental list of “times when Haskell has made someone quit in anger and confusion”.
There are valid complaints about Haskell, and its beginner unfriendliness. I made some very specific complaints about its syntax and tooling’s learning curve over in the thread on Hacker News (link: my comment on HN), and I can actually think of two more complaints[1] off of the top of my head this morning. There are definitely valid criticisms!
But the mysticism of “ooh, types and purity and monads are weird and spooky and Not Actually Good(tm)!” is FUD that is so difficult to evaluate critically, and I think makes folks who would otherwise pick Haskell up in a couple of weeks[2] really afraid to try it out.
[1] First, type declaration syntax is symmetric to value declaration syntax. This is elegant and beautiful and minimal, but really confuses beginners on what names are values and what names are types. It doesn’t help that values and types have different namespaces, so
acan be a value in some contexts and a type in others. Second, Haskell has a very minimal no-parentheses syntax to type declaration. Again, this is elegant and minimal - once you grok it, you understand why they would choose this path to make mathematical symmetries clear - but it’s very confusing for beginners when types, type constructors, and data constructors are not syntactically distinguishable.[2] Historically, our internal estimate for onboarding to Haskell is about 6 weeks from zero to production-ready code review. This is not much longer than other languages. However, zero to side project for Haskell is definitely a week or two, and this is definitely genuinely longer than other languages (e.g. zero to side project for Go is like 30 minutes).
-
The premise of this article seems to be “filesystems vs. database is not the right way to frame technologies because the requirements they solve for do not conflict”. That is, providing support for database-like operations (e.g. transactions) does not intrinsically preclude providing support for filesystem-like operations (e.g. storing large objects).
I’d argue that this premise is incorrect, because it considers requirements only from the perspective of capabilities when in reality there are also requirements from the perspectives of performance and cost. Databases and filesystems have dramatically different characteristics in terms of how much it costs to store some amount of data (remember, databases need to build indexes) and how quickly I can query and search for data. The reason I don’t mind working with two sets of technologies (at least, for now) is because I have intrinsically different requirements for the kind of data I put in a database vs. the kind of data I put in a filesystem, and it would be cost-prohibitive for me to use a database/filesystem hybrid abstraction.
I don’t find “the correct framing is controlled vs. uncontrolled changes” to be a compelling argument for using a hybrid system - what I would find to be extraordinarily compelling for a hybrid system is “here is an explanation of how we managed to build database-like capabilities at filesystem-like cost”.
-
I’d argue that this premise is incorrect, because it considers requirements only from the perspective of capabilities when in reality there are also requirements from the perspectives of performance and cost
May be if the title (and therefore the implied premise) of the article was replaced into:
“Data lifecycle, the application way” or something like that It would reflect the intended capabilities better.
There is certainly a need to have technologies that can reflect ‘Life-cycle’ of an application object, and not just the structure of that object.
The document oriented databases, tried to (perhaps not always successfully) reflect a ‘structure’ of an application object, but not its life-cycle. Instead the application developers have to write code to accommodate the life-cycle.
In my reading, boomla recognizes that life-cycle gap, and seems to want to go beyond what we have today.
We look to day at ‘append only’ and ‘authenticated databases’ approaches are means to ‘simplify’ the life-cycle gap I noted above, but those are simplifications, are more of ‘atomic’ building blocks of something bigger, in my view.
When I architecture a system, that deals with ‘critical data’, I want to think of the data lifecycle as a whole from the time it created, it is read, it is transacted with, it is archived, it is backed up, it is restored, it is reviewed for compliance, it is referenced (this is a hard problem, that probably was not solved by MS’s CreateFileTransactedA that @malxau mentioned)
Today, I have to ‘custom design’ an ecosystem (assuming a large enterprise) around the above. But there is more that can be done in that space from the basic technologies prospective.
I agree with @liftM that may be trying to project the idea into known ‘light’ formalism (like databases or filesystem) may not carry the message, the intent in the best way.
-
That’s an interesting way of looking at it. I have no clue about that space. Could you rephrase the gap you see?
One aspect I understand is audit-ability. Boomla can store every single change and currently does so. But I hardly see this being unique to Boomla, every copy-on-write filesystem does that. Backups, archives, and restoration all work but again I don’t see the uniqueness here.
I don’t quite follow what you mean with “it is transacted with”, “it is reviewed for compliance”, “it is referenced”. Maybe by transacting with, you mean the file is accessed? I see how that could be used for audition purposes, but again, that would not be unique.
I’d love to understand this.
I actually see the biggest value of Boomla in the entire integrated platform. It simplifies writing applications and eliminates problem spaces. Looking at any one of the eliminated problems, one can say, nah, that’s not a big issue. Yet as they add up, one ends up having a platform that’s 10x simpler to work with. That’s what I’m really after.
Thanks for your comment!
-
-
If I understand correctly, the argument is that databases have in-memory indexes which require lots of memory while filesystems don’t do that and as a result need much less memory.
I don’t see why filesystems couldn’t index structured file attributes the same way databases index the data. Boomla doesn’t currently do that thus its memory footprint is super low. In the future, apps will define their own schema similar to database tables and define any indexes. At that point one will have to ability to create in-memory indexes.
Again, I do not see this as conflict. Did I miss something?
-
The conflict is that I don’t need indexing on structured file attributes and I don’t want to pay for that extra capability. Cost is just as much of a requirement for me as capability.
I think that filesystems and databases genuinely are intrinsically different abstractions (as opposed to the article’s premise that filesystems and databases are not intrinsically different, and that the real differentiation for data storage abstractions is along the “controlled vs. uncontrolled changes” axis) because they have different cost profiles, optimized for different workloads.
databases have in-memory indexes which require lots of memory while filesystems don’t do that and as a result need much less memory
[nit] Database indexes also live on disk - indexing on a field necessarily requires more space (and therefore costs more) than only storing the field.
-
I think we are both right in different contexts.
If your main concern is cost, sure, that’s an optimization. In that case you should do whatever you can to reduce storage requirements.
The context I was exploring is a general purpose storage system for web development. In this context, storage space is not the key thing to optimize for, it is the developer’s brain capacity and working efficiency.
-
-
-
If anyone is curious about the mechanics behind how type families etc. work, I would strongly recommend Alexis King’s intro to typeclass metaprogramming.