I have to say that I personally can very much see where he’s coming from. GitHub contributions are dealt with in a very frustrating way (IMO they’d do better not allowing issues and PRs at all). There’s a bit of a religious vibe to the community; the inner circle knows what’s good for you.

That said, they may very well be successful with their approach by a number of metrics. Does it hurt to loose a few technically minded independent thinkers if the language becomes more accessible to beginners?

Where I see the largest dissonance is in how Elm is marketed: If the language is sold as competitive to established frameworks, you’re asking people to invest in this technology. Then turning around and saying your native modules are gone and you shouldn’t complain because no one said the language was ready feels a bit wrong.

I’ve shipped multiple production applications in Elm and attempted to engage with the community and I can say that their characterization perfectly matches mine.

Native modules being removed in particular has caused me to no longer use Elm in the future. I was always ok with dealing with any breakage a native module might cause every release, and I’m even ok with not allowing them to be published for external consumption, but to disallow them completely is unreasonable. I’m sure a number of people feel the same way as I do, but it feels impossible to provide meaningful feedback.

I work for a company that began using Elm for all new projects about a year and a half ago. That stopped recently. There are several reasons that people stopped using Elm. Some simply don’t like the language. And others, like the author of this post, want to like the language but are put off by the culture. That includes me. This article closely resembles several conversations I’ve had at work in the past year.

I vaguely recall some news around 2003 (I think it was) about Pixar switching from Sun to Intel hardware, and porting renderman.

Section 7.2 of the paper actually uses grep/ripgrep as a basis of comparison. It seems the two have the same or better performance than Sparser, which still wins out by a small margin for the most selective queries.

Yes. Always use grep first, even if one awk would do. This special one purpose only tool really cut the time down. Especially when you want to work on less than 100 million lines out of a billion lines.

I agree, I used to use JAWStats a PHP web app that parsed and displayed the AWStats generated data files to provide visually appealing statistics a lot like Google analytics but entirely server side with data originating from apache/nginx log files.

It’s a shame that it was last worked on in 2009. There was a fork called MAWStats but that hasn’t been updated in four years either :(

For a while I self hosted my feed reader and web analytics via paid for apps, Mint and Fever by Shaun Inman but those where abandoned in 2006. It seems like all good software ends up dead sooner or later.

You want GoAccess. Maintained, and looks modern. Example. I’m using it and it has replaced AWStats for me completely.

They see some use in formal methods for model checking. The Alloy Analyzer converts Alloy specs to SAT problems, which is one of the reasons its such a fast solver.

There’s also this talk on analyzing floor plans.

A previous submission on SAT/SMT might help you answer that.

I’ve used Z3 to verify that a certain optimized bitvector operation is equivalent to the obvious implementation of the intended calculation.

Just typed up the two variants as functions in the SMT language with the bitvector primitives and asked Z3 for the satisfiability of f(x) != g(x) and rejoiced when it said “unsatisfiable.”

Keiran King and @raph compiled a SAT solver to WebAssembly to use for the “auto-complete” feature of Phil, a tool for making crossword puzzles (source).

I found this library (and its docs) interesting. They go over some practical examples.

https://developers.google.com/optimization/cp/

I have personally used them when writing code to plan purchases from suppliers for industrial processes and to deal with shipping finished products out using the “best” carrier.

Thanks for the shout out :) I listened to the end of the talk (thanks for the pointer by @msingle), and basically everything he says is accurate. I found his 2010 master’s thesis several years ago, and it is a great read. Section 3 on Related Work is very good.

https://scholar.google.com/scholar?cluster=9993440116444147516&hl=en&as_sdt=0,33

Oh and Oil have very similar motivations – treating shell as a real programming language while preserving the interactive shell. Oh seems to be more Lisp-like while Oil simply retains the compositionality of the shell; it doesn’t add the compositionality of Lisp. (In other words, functions and cons cells compose, but processes and files also compose).

I mention NGS here, which probably has even more similar motivations:

http://www.oilshell.org/blog/2018/01/28.html

The main difference between Oh and Oil is the same as the difference between NGS/Elvish/etc. and Oil: Oil is designed to be automatically converted from sh/bash.

My thesis is that if this conversion works well enough, Oil could replace bash. If I just designed a language from scratch, I don’t think anyone would use it. Many people seem to agree with this. After all, fish has existed for 15 years, and is a nicer language than bash for sure, but I’ve seen it used zero times for scripts (and I’ve looked at dozens if not a hundred open source projects with shell scripts.)

However as he correctly notes (and I point out in the FAQ), Oil doesn’t exist yet! Only OSH exists.

The experience of implementing OSH and prototyping the OSH-to-Oil gave me a lot of confidence that this scheme can work. However it’s taking a long time, longer than expected, like pretty much every software project ever.

I’m not really sure how to accelerate it. Maybe it will get there and maybe it won’t :-/ No promises!

I “front-loaded” the project so that if I only manage to create a high-quality implementation of OSH (a bash replacement), then I won’t feel bad. I’m pretty sure people will use that if it exists.

I maintain a “help wanted” tag for anybody interested in contributing:

https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22

This is mainly helping with OSH, as Oil is a concentrated design/implementation effort that is hard to parallelize. Feel free to join https://oilshell.zulipchat.com too!

Michael MacInnis mentions talking to @andyc ~44:25 in the talk, but I’d also like to hear his opinion on Michael’s work.

There’s also Nils M Holm’s books: http://t3x.org/

There are a lot of books on compilers, but not as much for interpreters (i.e. the bytecode compiler + dynamic runtime combo that many languages use).

Not a book, but you still might find this paper interesting: Engineering Definitional Interpreters:

Abstract: A definitional interpreter should be clear and easy to write, but it may run 4–10 times slower than a well-crafted bytecode interpreter. In a case study focused on implementation choices, we explore ways of making definitional interpreters faster without expending much programming effort. We implement, in OCaml, interpreters based on three semantics for a simple subset of Lua. We compile the OCaml to x86 native code, and we systematically investigate hundreds of combinations of algorithms and data structures. In this experimental context, our fastest interpreters are based on natural semantics; good algorithms and data structures make them 2–3 times faster than interpreters. Our best interpreter, created using only modest effort, runs only times slower than a mature bytecode interpreter implemented in C.

An even better formulation would be “Here is the source code for an app where I didn’t use a framework. It has users, and here are my observations on building and deploying it”.

In other words, “skin in the game” (see Taleb). I basically ignore everyone’s “advice” and instead look at what they do, not what they say. I didn’t see this author relate his or her own experience.

The problem with “when you should” is that the author is not in the same situation as his audience. There are so many different programming situations you can be in, with different constraints, and path dependence. Just tell people what you did and they can decide whether it applies to them. I think I basically follow that with http://www.oilshell.org/ – I am telling people what I did and not attempting to give advice.

(BTW I am sympathetic to no framework – I use my own little XHR wrapper and raw JS, and my own minimal wrapper over WSGI and Python. But yes it takes forever to get things done!)

I definitely agree with this change. It’d get more people thinking architecturally, something that’s sorely needed.

Yup. I should also write a blog post on “invoking the compiler via a shell script”.

The main thing to know is that the .c source files and -l flags are order dependent. With a makefile, most people use separate processes to compile and link so I think it doesn’t come up as much.

I don’t use Make as a build tool, but I find it quite handy to collect small scripts and snippets with PHONY targets that don’t attempt any dependency tracking. Make is almost universally available, the simple constructs I use are portable between gmake and BSD make, and almost every higher-level tool out there understands Makefile — so coworkers using various IDEs and the command lin can all discover and run the “build”, “this test”, “download dependencies”, “run import process”, “lint”, etc tasks. If I need a task that’s more than two lines, I put it in a shell script.

Although some languages now come with tooling that understands scripts, such as Cargo or NPM, I still find a Makefile useful for polyglot projects or when it’s necessary to modify the environment before calling down to that language specific tooling.

About two years ago I finally sat down and read the GNUMake manual. It’s very readable, and it’s more capable than just about any other make out there. For one project, the core of the Makefile is:

%.a :
    $(AR) $(ARFLAGS) $@ $?

libIMG/libIMG.a     : $(patsubst %.c,%.o,$(wildcard libIMG/*.c))
libXPA/src/libxpa.a : $(patsubst %.c,%.o,$(wildcard libXPA/src/*.c))
libStyle/libStyle.a : $(patsubst %.c,%.o,$(wildcard libStyle/*.c))
libWWW/libWWW.a     : $(patsubst %.c,%.o,$(wildcard libWWW/*.c))
viola/viola         : $(patsubst %.c,%.o,$(wildcard viola/*.c))     \
            libIMG/libIMG.a         \
            libXPA/src/libxpa.a     \
            libStyle/libStyle.a     \
            libWWW/libWWW.a

The rest of it is defining the compiler and linker flags (CC, CFLAGS, LDFLAGS, LDLIBS) and some other targets (clean, depend (one command line to generate the dependencies), install, etc). And this builds a program that is 150,000 lines of code. I can even do a make -j to do a parallel build. I’m not entirely sure where all this make hate comes from.

Yeah I agree that you often have to cobble something together from existing parts to achieve something big. That’s one of the lessons I learned from reading Stallman’s biography. As far as I remember, GCC, Emacs, and many other GNU projects all started from borrowed code.

They didn’t just start typing in a blank text file. That would have pushed it from a “huge undertaking” to “unfinishable”.

The key though is you have to deeply understand all those parts and not just blindly copy them… otherwise you’ll get an incoherent system. But if you choose carefully, this can really accelerate the project and make it possible to ship.

Thanks! It’s going well, but more slowly than I would like, which is pretty much every software project ever :)

I wish I had time to work on the Oil language, but I’m sort of putting that off until OSH is in better shape and has users, which I think requires OPy and at least one other thing.

I haven’t had as much time to participate here and on /r/programminglanguages, but I noticed this post [1] and the one about flags vs. args. It sounds like we’re thinking about things very much along the same lines.

Just like there are memory-safe languages, there should be “string-safe” languages. I’m not sure if Oil will get there, but it definitely needs to go on that direction. This related to my comment about security here [2]

So yeah hopefully I can get to Oil sooner rather than later, and I would love to have your input. Or maybe we can think of some way to graft safety onto OSH. The idea of shell doing all the command line parsing and then serializing to

cmd "${flags[@]}" -- "${args[@]}"

I think is somewhat useful / promising. Basically, the shell is completely ignorant of which things are flags and which things are args right now. But it doesn’t have to be. It might not even require Oil.

[1] https://rain-1.github.io/escaping.html

[2] https://lobste.rs/s/jn62zk/persuasive_language_for_language#c_avqn7r

As I understand it (and as noted by alva above), Fuschia competes with Linux, not Java. It’s a microkernel, not a language VM or a language. The article was technically confused – the Oracle line was just a throwaway and not terribly accurate.

Java is going to be in Android forever, simply because there are hundreds of thousands of apps written in Java. Kotlin was a logical move because it’s very compatible with Java.

This is cool! I’ve been thinking of doing something like this for my Python compiler:

http://www.oilshell.org/blog/2018/03/04.html

That is, instead of compiling to bytecode, compile to C.

Have you tackled garbage collection yet? I suppose it is no harder to do in C code than interpreted bytecode, but I haven’t really gotten into the details.

I’m very much on that team. I’m giving a full conference talk about it in Barcelona next week. Different framing work well for different people, so I picked this one for this post, as I wanted it to be short, and this framing is shorter.

Yeah I think we browsers have become too big, monolithic, and homogeneous. I would like to see more diversity in web clients. Those clients could use WebAssembly outside the context of the browser.

The browser has a very specific and brittle set of security policies, and WebAssembly doesn’t change that. It will inherit the same problems that JavaScript has.

Imagination Fuel

Love It. The captures a lot of meetings I have been trying to have with folks at work. They are thinking low level performance fixes for things, while necessary, they are having a huge problem jumping up a couple abstraction levels and thinking transformatively.

People don’t even use sanitizers and fuzzers, so I’m not sure why you would expect them to rewrite in Rust. It’s literally 1000x less effort.

As far as I can tell, CloudFlare’s CloudBleed bug would have been found if they compiled with ASAN and fed about 100 HTML pages into it. You don’t even have to install anything; it’s built right into your compiler! (both gcc and Clang)

I also don’t agree that “nearly every mass vulnerability has a shared root cause”. For example, you could have written ShellShock in Rust, Python, or any other language. It’s basically a “self shell-code injection” and has very little to do with memory safety (despite a number of people being confused by this.)

The core problem is the sheer complexity and number of lines of unaudited code, and the fact that core software like bash has exactly one maintainer. There are actually too many people trying to learn Rust and too few people maintaining software that everybody actually uses.

In some sense, Rust can make things worse, because it leads to more source code. We already have memory-safe languages: Python, Ruby, JavaScript, Java, C#, Erlang, Clojure, OCaml, etc.

Software engineers should definitely spend more time on security, and need to be educated more. But the jump to Rust is a non-sequitur. Rust is great for kernels where the above languages don’t work, and where C and C++ are too unsafe. But kernels are only a part of the software landscape, and they don’t contain the majority of security bugs.

I would guess that most data breaches these days have nothing to do with memory safety, and have more to do with bugs similar to the ones in the OWASP top 10 (e.g. XSS, etc.)

https://www.owasp.org/images/7/72/OWASP_Top_10-2017_%28en%29.pdf.pdf

Edit: as another example, Mirai has nothing to do with memory safety:

https://en.wikipedia.org/wiki/Mirai_(malware)

All it does it try default passwords, which gives you some idea of where the “bar” is. Rewriting software in Rust has nothing to do with that, and will actually hurt because it takes effort and mindshare away from solutions with a better cost/benefit ratio. And don’t get me wrong, I think Rust has its uses. I just see people overstating them quite frequently, with the “why don’t more people get Rust?” type of attitude.

not only software engineers, almost the entire IT industry has buried it’s head in the sand and is trying desperately hard to hide from the problem, because “security is too hard”. We are pulling teeth to get people to even do the minimal upgrades to things. I recently had a software vendor refusing to support anything other than TLS 1.0. After many exchanges back and forth, including an article from Microsoft(and basically every other sane person) saying they were dropping all support of older TLS protocols because of their insecurity, they finally said, OK we will look into it. I’m sure we all have stories like this.

If you can’t even bother to take the minimum of steps to upgrade your security stacks after more than a decade,(TLS1.0 released in 1999 and TLS 1.2 is almost exactly a decade old now) because it’s “too hard”, trying to get people to move off of memory unsafe languages like C/C++ is a non-starter.

But I agree with you, and the author.

Many Go programmers seem to feel very comfortable with global state. When I join new organizations or projects, I often find myself needing to educate and socialize the problems that come from that. This post is just a formalization of the things I’ve been saying informally for a long time.

I wish I knew why this was so relatively prevalent in the Go ecosystem. If I had to guess, I’d speculate that it’s because a lot of the example code in e.g. books and tutorials doesn’t really shy away from global state.

Yup, this was my comment when this appeared a year ago on HN:

In other words, use a pure dependency-injection style, and ZERO globals.

https://news.ycombinator.com/item?id=14521894

I don’t see the need to distinguish between paths and strings.

They’re different types: a path is a string which is or could be the pathname of a file or directory on disk. Since they’re different types, treating them differently yields all the standard benefits of strong typing.

This was my thought as well, anything more complex than basic types or char*, you’re essentially serializing/deserializing with all the performance problems that entails.

Here’s my take on it: https://news.ycombinator.com/item?id=15776629

I also don’t think it’s a great paper. It’s long on ideas but short on evidence, experience, and examples. I don’t think you’re missing anything.

In my top-down view I tend to focus on the life cycle of software. The fact that software gets more complex over time, in a very tangible way. If we could avoid monotonically adding complexity over time, life would be much better.

Thanks for the interesting commentary. Some parts definitely resonated, particularly about the half-features and difficulty of knowing how and where to make the right change.

This is only the germ of an idea, but it is perhaps novel and perhaps there is an approach by analogy with forest management. Periodic and sufficiently frequent fires keep the brush under control but don’t get so big that they threaten the major trees or cause other problems.

Could there be a way of developing software where we don’t look at what is there and try to simplify/remove/refactor, but instead periodically open up an empty new repo and move into it the stuff we want from our working system in order to build a replacement? The big ‘trees’ of well understood core functionality are most easily moved and survive the burn, but the old crufty coupling doesn’t make the cut.

Some gating on what would be moved would be needed. The general idea though is that only sufficiently-well-understood code would make it across to the new repo. And perhaps sufficiently reliable/congealed black boxes. It would interplay a lot with the particularly language’s module/package and testing systems.

The cost would be periodic re-development of some features (with associated time costs and instability). The benefit would be the loss of those code areas which accrete complexity.

This paper was written in 2006, two years before Applicatives were introduced. The Haskell community’s understanding of how to best structure programs has been refined a lot in that time and I think you underestimate the insights of this paper even if it is only a refinement of Brooke’s ideas from 40 years ago.

“This was helpful, because it helped me to focus on the ‘causes of complexity’ portion of OP.”

That’s the part I read in detail. I skimmed the second half saving it for later since it was big. The first half I liked because it presented all those concepts you listed in one location in an approachable way. It seems like I learned that stuff in pieces from different sub-fields, paradigms, levels of formality, etc. Seeing it all in one paper published ahead of some of these things going into mainstream languages was impressive. Might have utility to introduce new programmers to these fundamental concepts if nothing else I thought. Understanding it doesn’t require a functional programming or math back ground.

Ok, so that’s their requirements. Minimax mentions things like business factors. You mention changes with their motivations. I’ll add social factors which includes things that are arbitrary and random. I don’t think these necessarily refute the idea of where the technical complexity comes from. It might refute their solutions for use in the real world such as business. However, each piece of the first half is already getting adoption in better forms in the business world on a small scale. There’s also always people doing their own thing in greenfield projects trying unconventional methods. So, there’s at least two ways their thinking might be useful in some way. From there, I’d have to look at the specifics which I haven’t gotten to.

I do thank you for the Reddit search given those are about the only discussions I’m seeing on this outside here. dylund’s comments on Forth were more interesting than this work. Seemed like they were overselling the positives while downplaying negatives, too, though.