1. 18

    I like to think about these things, but don’t have much hope. Here are my points:

    • Networking shouldn’t be an afterthought. Distributed computing should not be as difficult as it is. Transparently interacting or using resources from other systems should be something you don’t have to think about. I don’t care about hardware. I don’t care about CPU architectures. I don’t care about GPUs. I don’t care about drivers. All computers form an transnational turing machine.
    • Object capabilities should be a primitive concept. Imagine sharing a screen: That shouldn’t be the hassle it is, you should just be able to give someone read access to a segment or the whole display. The same applies to Files (but we probably shouldn’t have files), Hardware access, etc.
    • Hypertext should be a everywhere. The web has shown how attractive the idea is, but browsers are cursed to contain it, which is getting harder and harder. Project Xandu had good ideas about this, and HTTP is a shallow copy. We need complex links that can point to miscelanious parts of the system, and ideally also have back-references. You probably want a lof of cryptography for something like thise, to avoid the centralisation of power.
    • Logic and UI should be separate. Unix programms regard the standard output and input as the default UI, everything else is a side effect. Instead we should have the ability for a program (or procedure, algorithm, …) to produce complex data, that doesn’t only mean something in a specific environment (Powershell), but is universally understood. A terminal-like environment could display the results line-by-line, but it should be transformed into a graphical representation using a table, or a graph (or whatever one might come up with later).
    • Programming should not be a specialist’s affair. We have two classes of people, those who are at the mercy of computers, and those who can use them. This shouldn’t be the case, because the former are in a much weeker position, getting lost, getting overwhelmed, and sometimes even abused by those who know better. A proper operating system cannot be based on the lie, that you don’t need to know anything to use a computer: To be a responsible user, you need to know some basics. A simple programming language (I would like something like Scheme, but that’s just be) should be integrated into the system, and the user shouldn’t fear it. It’s a direct link to the raw computational power that can be used.

    In some sense, I like to think of it like Plan 9, without the Unix legacy, but that seems to simplistic. The interesting thing about Unix, is that despite it’s limitations, it creates the fantasy of something better. Inbetween it’s ideal power and it’s practical shortcomings, one can imagine what could have been.

    1. 13

      Programming should not be a specialist’s affair. We have two classes of people, those who are at the mercy of computers, and those who can use them. This shouldn’t be the case, because the former are in a much weeker position, getting lost, getting overwhelmed, and sometimes even abused by those who know better. A proper operating system cannot be based on the lie, that you don’t need to know anything to use a computer: To be a responsible user, you need to know some basics. A simple programming language (I would like something like Scheme, but that’s just be) should be integrated into the system, and the user shouldn’t fear it. It’s a direct link to the raw computational power that can be used.

      I think the ultimate problem is that most people don’t want to program. They want to accomplish a task, and for the most part, someone else has programmed the tool to accomplish the task. They don’t want to build the tool. Us freaks who want to write tools are few and far. It’s trhe same reason cars have mechanics.

      1. 5

        I don’t think that programming has to be the same as “building the tool”, but more along the lines of what @spc476 mentions with Excel. Especially when you take “Logic and UI should be separate”, one can imagine that programming doesn’t even have to mean “writing text in a text editor”, but could be a GUI afair, where you work on connection tools in a graphical representation, trivially connecting components of your system, without depending on another tool.

        Yes, not everyone want’s to be a car mechanic, nor do I, but to drive a car you need to get a drivers license, and that is the reason we can assume people can take some basic responsibility. We don’t have that for computers, and that’s why the responsibility has to be delegated to Microsoft or Apple. If we want to think of a computer as a tool, not a toy, I argue that a basic understanding for computational thinking should be assumable, and would help everyone.

        1. 2

          I sincerely believe enso (née Luna) has a serious fighting chance to fill this gap. Though they’re taking their time :)

      2. 10
        • Networking: QNX was network transparent. It was wild running a command on computer 1, referencing a file from computer 2, piping the output to a program on computer 3 which sent the output to a device on computer 4. All from the command line. The IPC was fast [1] and network transparent, and used for just about everything.
        • Hypertext: The only operating system I know of that uses extensive form of hypertext is TempleOS (I don’t think it’s HTML but it is a form of hypertext) that extends pervasively throughout the system.
        • Logic and UI: There are bits and pieces of this in existence. AmigaOS has Rexx, which allows one to script GUI programs. Apple has (had?) something similar. Given that most GUI based programs are based around an event loop, it should be possible to pump events to get programs to do stuff.
        • Programming: True, but there is Excel, which is a programming language that doesn’t feel like one. Given an easy way to automate a GUI (similar to expect on the command line), and teaching people that computers excel (heh) at repeated actions could go a long way in giving non-programmers power.

        [1] In the early-to-mid 90s, I had friends that worked at a local software company that wrote and sold custom X Window servers. Their fastest X Window server ran on QNX.

        1. 3

          Programming: True, but there is Excel, which is a programming language that doesn’t feel like one. Given an easy way to automate a GUI (similar to expect on the command line), and teaching people that computers excel (heh) at repeated actions could go a long way in giving non-programmers power.

          One program idea I’ve had was a spreadsheet that users could “compile” into a simple gui. Analysts already use it as an adhoc RAD tool. Why not give them an actual custom GUI for their efforts?

          1. 3

            There was something like that in KDE, it was called krusader or something like that.

        2. 7

          Transparently interacting or using resources from other systems should be something you don’t have to think about.

          Then everyone will run headlong into the fallacies of distributed computing, unfortunately. This is why things like CORBA and DistriibutedObjects failed. Networking is not transparent, much as we would like it to be.

          At least not in a normal imperative programming paradigm, like RPC. You can get a lot of transparency at a higher level through things like async replication, e.g. Dropbox or [plug] Couchbase Mobile. But even then you have to be aware & tolerant of things like partitions and conflicts.

          1. 4

            Your first point is pretty much what Barrelfish is designed for, go check it out!

            1. 4

              Programming should not be a specialist’s affair. We have two classes of people, those who are at the mercy of computers, and those who can use them.

              Indeed, the power dynamics are way out of control in software today. On the orange site, @akkartik describes this well via an analogy to food production: nearly all software today is restaurant-style, while almost none of it is home-cooked.

              For anyone interested in this topic, I would suggest looking into the Malleable Systems Collective (the Matrix room is most active) and Future of Coding communities, as it comes up in those places regularly.

              1. 3

                Programming should not be a specialist’s affair.

                Jonathan Edwards has been working on this problem for a long time. It goes well beyond the OS.

                1. 2

                  The same applies to Files (but we probably shouldn’t have files)

                  Could you elaborate on this? Why no files?

                  Logic and UI should be separate. Unix programms regard the standard output and input as the default UI, everything else is a side effect. Instead we should have the ability for a program (or procedure, algorithm, …) to produce complex data, that doesn’t only mean something in a specific environment (Powershell), but is universally understood. A terminal-like environment could display the results line-by-line, but it should be transformed into a graphical representation using a table, or a graph (or whatever one might come up with later).

                  There was an interesting newsletter post about emacs being interface independent. I’m not too familiar with emacs, but it struck me as an intriguing and beautiful idea.

                  1. 5

                    Could you elaborate on this? Why no files?

                    Maybe it’s clearer, if I say file system. It might be too much to throw out the concept of a digital document, but I have come to think that file systems, as we know them on POSIX systems, are too low level. Pure text, without hyperlinks would be a wierd thing in an operating system where everything is interconnected, and why directories shoudln’t have to be a simple tree (because tools like find(1) couldn’t do proper DFS in the 70’s), but instead could be any graph structure of sets, or even computed.

                1. 29

                  This is a poor submission because it does not contain any context about what is happening and it is link to a diff. In the future, I would look for something more substantial to submit. As it stands, this is a submission suitable for Twitter, not Lobste.rs.

                  1. 9

                    What about – hear me out – a 128kb club? One may say that 640K ought to be enough for anyone.

                    I do agree that web needs a diet. I don’t think that making a list of websites helps anyone. Making a copy of a list of websites is even less smart.

                    1. 7

                      Don’t worry, the club to end all clubs is here: https://0kb.club

                      1. 1

                        I really started something here xD I love it!

                      2. 2

                        Why are you so angry? How does making this list hurt anyone?

                        1. 22

                          I don’t think it hurts anyone, but I don’t think it helps either. In short, I think the question is really, what does it do? From my perspective, it does nothing.

                          And saying that it is basically useless does not mean I endorse heavy web pages. I am very much in favour of reducing bloat in sites. What I do find tiresome is the incessant complaining about it accompanied by poorly thought out rebellions, like trying to shame sites with arbitrary metrics or proposing dead-end protocols. Minimalism, in and of itself, should not be the goal. There needs to be a real incentive to making smaller sites that actually matters to those making them. Currently, no such thing exists. None of these proposals solve that or even acknowledge that it is a problem. It’s all predicated on the baseless belief that “smaller is better” will somehow win over hearts and minds. Rest assured that it won’t.

                          1. 3

                            There needs to be a real incentive to making smaller sites that actually matters to those making them

                            I think you misunderstand the motive for making larger sites in the first place.

                            These proposals won’t catch on in the large, because big companies are solely interested in profit. That is the only purpose of a company. They are not interested in technological innovation unless it services that goal. They are not interested in energy efficiency or any of the arguments you can make in favour of reducing web page size as long as the majority of their market has enough internet bandwidth and enough computing horsepower to cope. Actually, it’s beneficial to keep the status quo or build even more onto the tower of babel, not just for developers (despite the fact that alternative software could be written at a lower level of abstraction that does the exact same thing), but for companies and OEMs particularly – to propagate wasteful, costly technologies. Doing so ensures that, for example, telecommunication companies can continue getting grants for otherwise unnecessary infrastructure changes, or that OEMs can continue to sell laptops with ridiculously high ‘horsepower’ that really do not do anything different task-wise to laptops from a decade ago, and means Google et al can continue their trend of locking everyone in to Their Software™. It ensures that the wheel keeps spinning, if you will.

                            But the point isn’t that they catch on in the large, and it isn’t to ‘win hearts and minds’[0] the point is to pull back to not just a more minimalist web, but also to carve out new spaces as alternatives to a system that has been thoroughly branded, commoditized, and marketed. To create spaces that feel more personal, more familar, and are hand-crafted rather than Delivered and Targeted For You™

                            Sure, these new proposals won’t catch on ‘in the large’. That’s not the point here, at all.

                            [0] Whatever that means – seriously, there is no chance, at this time, of a new protocol taking hold regardless of the features it boasts. At least in the wide sense that you seem to mean.

                            1. 2

                              Sure, these new proposals won’t catch on ‘in the large’. That’s not the point here, at all.

                              Then they are doomed to obscurity and the constant complaining about the bloated web will continue.

                              1. 1

                                And once again, that is not the point.

                                The fact that they are creating new communities, new groups of specific styles of ideas, is valuable. Those connections with other people are valuable.

                                And regardless of that, if you are only concerned with the narrative of “we need to build the future and nothing else has any value”, it’s easy to show that time and time again, historically, small communities with ‘weird’ ideas serve to be melting pots for the technology of the future. We see this with places like, Xerox Parc, CSAIL, some Usenet sub-communities, LambdaMOO, etc.

                                Places like 2f30, suckless, 9front, et al led to the creation of Wayland, to a certain degree, for a modern example. Javascript and Python were influenced by Lisp, despite the fact that Lisp as a whole did not grow into an industry standard. Hell, despite my particular dislike about the borderline-abuse that goes on within the Merveilles community, they push out pretty good and interesting technology on the regular – orca has itself altered how a fair chunk of people create music interactively.

                                It’s not about creating the future directly, it’s about being able to communicate with people more directly and explore ideas that you wouldn’t have otherwise, and enjoying building unique spaces and connections with other people that are more genuine than those an algorithm has selected for you. It’s about creating spaces in which we can think differently, and from within that space, those ideas will in some shape or form, influence the future.

                            2. 1

                              I’ve had similar thoughts but didn’t have the words to express them as well. Lists like these are cool and all but like, so?

                              I’m more interested in things like code golfing. Don’t just have a minimal site, do something fantastical that only requires a small amount of scripting. That is impressive.

                        1. 13

                          Lobsters doesn’t have an NTP tag, but if you run anything that depends on NTP, the end of this post should make you a little worried.

                          1. 4

                            An ntp tag would be too partisan. A time tag would work instead.

                            1. 3

                              This post is an excellent argument for abolishing leap seconds. Adding seconds regularly causes issues, subtracting them will only be worse. I’m in favor of ignoring them for the next 200 years or so, by then either civilization will have collapsed or AIs can take care of it.

                              1. 7

                                Or the industries that are negatively affected by leap seconds could use a time scale that’s not affected by them, like TAI or MJD. Then it’s just a matter of treating UTC like any other civil time (which it already is, de facto), by deriving it from TAI via a table of leap seconds.

                                1. 5

                                  I agree with things not mattering much in practice due to civilization being done in the mid-term anyway, but c’mon, can’t people try to make an effort to get at least the basic things right? Where is the craftman’s pride?

                                  There is no fundamental reason why software dealing with time has to be such a shitshow. And if you can’t deal with leap seconds, simply pick a time without them. Problem solved.

                                  1. 5

                                    I worked out a while ago that leap seconds will add up to something like half an hour over a millennium. Within a time zone, the difference between noon and the sun reaching its zenith varies by up to an hour. Ignoring leap seconds for a thousand years will mean that noon is off as a result of drift by less than it is off because of accidents of geography. There is no benefit from leap seconds for anyone other than astronomers and a huge amount of pain for everyone else.

                                    If, in two thousand years, we’ve accumulated enough drift that time zones are one over from where they’d be, it’s a lot easier to adjust everyone’s time zone by one hour than it is to do a leap hour (assuming anyone actually cares about the relationship between the time and the position of the sun in 2,000 years).

                                    1. 6

                                      Astrophysics grad student here. I’ve worked on enough of LIGO’s timing system and enough GPS/UTC/MJD conversions for multi-messenger searches to say that leap seconds are as much of a pain for astronomers as for anybody else!

                                2. 2

                                  You weren’t kidding.

                                1. 5

                                  Also, it depends on which AWK implementation. mawk is generally much faster than gawk.

                                  Setting up the rough equivalent to what is in the post for the first “benchmark”, I get 0.164s for cut, 0.225s for mawk, and 0.413s for gawk. Similar ratios with the other test.

                                  I find the conclusion of the post to be pretty flimsy.

                                  1. 2

                                    I agree the conclusion is flimsy. Much also depends upon the CPU. I just tested gawk-5.1.0 as between 1.20x and 1.55x faster (4 diff CPUs) than mawk-1.3.4_p20200106 for “extracting column 2”.

                                  1. 16

                                    These commandments are best described by Agans in his book Debugging, which is probably the best book on debugging out there. Agans’ rules are below and are available in a sample on the site for the book. The ones that also appear in this post are marked.

                                    • Understand the system
                                    • Make it fail (*)
                                    • Quit thinking and look
                                    • Divide and conquer (*)
                                    • Change one thing at a time (*)
                                    • Keep an audit trail (*)
                                    • Check the plug (*)
                                    • Get a fresh view (*)
                                    • If you didn’t fix it, it ain’t fixed
                                    1. 7

                                      These commandments are best described by Agans in his book Debugging, which is probably the best book on debugging out there.

                                      I like that review of it some random dude wrote.

                                      1. 6

                                        I came here to post this same list - this is a great book :) It’s important to combine “understand the system” and “if you didn’t fix it, it ain’t fixed” in such a way as to really understand that if you don’t know why the issue happened, it is going to come back again. This is a type of information that humans are so terrible at acquiring, and are probably wired in such a way that we benefit from blinding ourselves to it, because it’s an example of an information hazard that causes us harm by simply acquiring true information which inconveniences us further. At some point, debugging is an act of self-harm due to the impact of acquiring hazardous information that imposes additional responsibilities on us or our team or tarnishes reputations etc…

                                        1. 3

                                          Yes indeed, but they are common sense. Working in mechanical automation, we were doing the same… I even had to go through certification. and the first things was to check the plug…

                                          1. 2

                                            I now buy copies of this book just to lend to junior engineers.

                                          1. 2

                                            I found the MANIFEST.in example to be poor one because, frankly, the spec language itself is just bad. It struck me that it’s not a “power” problem so much as it’s a design problem. The language really has the feel that it was made for ease of implementation, hence the reliance on state that is an unreasonable burden on the user.

                                            1. 3

                                              This is a good presentation, but I find it funny how terminals want to shoehorn in what has been available in GUIs for a while.

                                              1. 9

                                                The terminal is still my most productive environment. Typing commands and little loops and globs and pipelines and whatever is inherently faster for me than any sort of GUI manipulation.

                                                This is especially true if you work across machines (again, for me). X and Plan 9 almost got remote graphical applications right, but it never quite worked seamlessly and it got to the point where it seems like people only ever use remote command lines and now most of the major OS’s have done away with seamless remote GUIs if they ever had them to begin with.

                                                Something that also would’ve improved my GUI experience, and which I hacked in to dwm years ago, is window swallowing. When I’m working in the terminal, writing my little loops and pipelines and availing myself of the power of the command line, when I launch a graphical text editor my attention is popped to another window.

                                                I realize this is what a lot of people want, but for me, the concept of the editor taking over the window I’m in makes so much more sense to me intuitively and makes context switching so much easier. Plan 9 did it and a few other GUIs did it but no mainstream ones do, and certainly not when dealing with remote graphical applications.

                                                1. 2

                                                  I think it’s interesting to debate if this is right or not. I’m not a fan of using technologies in a way that it wasn’t indented, because I feel like it might always come back and bite you. People do crazy things with terminals, and even though it’s usually mostly aestethic, it breaks stuff for specifc users. I like to use the Emacs terminal, and therefore stick to non-interactive tools that avoid taking over the whole terminal or jumping around. It’s a minority, but good design shouldn’t be at anyone’s disadvantage. The concept of graceful degradation, as sometimes seen applied to websites, might be relevant here too. Not everything should have to work everywhere (most modern terminal emulators can’t plot graphics either), but assuming everything is a vt100 shouldn’t be the default either.

                                                  1. 1

                                                    As a GNU Emacs user, I find it funny that I have way better control over my preferred OpenType font (JetBrains Mono) in my terminal than I do using the GUI version of the editor. I hate wide spacing between letters in monospace fonts, so I like to adjust the kerning to scrunch the letters together. Try this in Emacs, you’ll find you can’t, regardless if you set the :width to condensed, semi-condensed, or ultra-super-mega-condensed. Also try configuring ligatures, sylistic sets, or the fallback behavior – those kind of detailed font tweaks aren’t exposed by Emacs, yet they are in certain terminal emulators.

                                                    1. 3

                                                      There is something to be said for having a minimal protocol between the “GUI Server” (frontend) and the backend ;) But doesn’t seem like Display PostScript is coming back, so we’re stuck with the terminal.

                                                      1. 2

                                                        Not with that attitude. I’ve been seriously considering using a DPS derivative to solve the problem of the hideous and useless terminal (but don’t hold your breath).

                                                        I still don’t understand why there’s been so little (re)invention in this area, save for game engines. Once you gain some basic understanding of how GUI libraries are put together, it is fairly straight-forward to reimplement them given a text shaping/rendering engine and a rasterizer, such as Pango with Cairo (the latter itself a derivative of Postscript, just like HTML canvas’ 2D context).

                                                        Misusing a Unix terminal as a GUI server doesn’t make it easy to create user interfaces, despite expectations.

                                                        1. 1

                                                          Years ago I wrote an experiment where the terminal could be flipped into a special mode via escape sequence that would then accept a Lua script, and would flip the window to a Cairo-based canvas that was manipulable via Lua.

                                                          The experiment “worked” but it turns out I’m really bad at GUI programming, especially at such a low level. But it was a neat experiment.

                                                          I’m curious what your DPS derivative would look like? DPS is neat.

                                                          1. 1

                                                            It’s a shame that no such experiment has really “seen the light of day”, except for one web-based prototype.

                                                            There are quite a few choices in this area, so me saying “derivative” mostly hints at there being a possible need for divergence. I haven’t really had a close look at DPS, so I take it mostly as a concept, as I have intimate knowledge of Cairo and the two are nearly the same thing.

                                                            If you have some sort of canvas, you can start with a Widget:

                                                            • abstract get_size()
                                                            • abstract render(context)
                                                            • abstract on_key(key, state)
                                                            • abstract on_mouse*(x, y, state)

                                                            Define a container such as VerticalBox:

                                                            • children[]
                                                            • heights[]
                                                            • private recalc(): ask each children for how high they want to be etc.
                                                            • render(): iterate through children, translate origin, possibly clip, save state, call their render method, restore state
                                                            • on_key(): find which child has keyboard focus and forward the event, possibly bubble up
                                                            • on_mouse*(): see what child could contain the given point

                                                            Perhaps a Button:

                                                            • text
                                                            • private depressed
                                                            • abstract on_click()
                                                            • get_size(): measure the text, add padding
                                                            • render(): clear with background colour, draw ridge, measure the text, render it in the center, adjust if depressed
                                                            • on_key(): call on_click() if the key is Enter/Space
                                                            • on_mouse(): toy with depressed, maybe call on_click(), invalidate the widget’s region

                                                            It’s all rather simple like this. It will take some time but it doesn’t require deep thought to get it working.

                                                            The context passed to the render method just queues/forwards DPS commands. Fonts can either be rendered on the client and glyphs cached on the server (XRender method, undesirable here), or you need to have a backwards channel to query font properties–if you ban scaling or rotation transformations and stick to translations only, it won’t depend on DPS state, and if you disregard ligatures and other nice-to-haves, you can shape it yourself trivially.

                                                            What’s hard is how to make this work in a non-hierarchical way, which I don’t think I need–a vector graphics alternate screen is fine for me, and it is possible to make a graphical shell with this–you can have parts of the screen emulating a vt100 and parts of it either retaining some DPS output or running an application–a graphical tmux is quite possible. It should also be possible to do things like embed text editors–this is extremely desirable. In general, most issues are about conceptual design.

                                                            What’s nice is that if these GUIs run in a mode, you can throw terminfo and other terrible legacy burden over the wall.

                                                            1. 1

                                                              What do you think about having the terminal be a Wayland compositor, that will place windows below where you started them? Then you won’t have the problem of having to integrate with every graphics API.

                                                              1. 1

                                                                Wayland adds ridiculous levels of complexity and the subwindows become completely opaque.

                                                                With DPS you can impose arbitrary high-detail scaling and even have a universal text selection mechanism.

                                                                It could be added as an optional feature, though.

                                                  1. 5

                                                    What is notable about this other than the fact that we’re apparently just supposed to know about it?

                                                    1. 4

                                                      While this one looks like a “we’re still relevant, dammit” moment… to those who want OpenVMS to live, any news from VMSSoftware are good news. It’s the company doing the porting to x86-64, so if they succeed, VMS can live. If not, it will die with IA64.

                                                    1. 2

                                                      …artisanal greping…

                                                      What does this mean? It sounds like flowery language for something we do every day: searching.

                                                      1. 8

                                                        My favorite line read today so far was in this article, and shows that the author is not taking himself seriously when he writes such things:

                                                        Now for the fun part: a crapton of grepanalysis ✨.

                                                        1. 3

                                                          Sometimes the job is too boring to explain in a straightforward fashion.

                                                          1. 3

                                                            It is meant to be flowery language for exactly that.

                                                            Not every technical post needs to be written drily. I enjoy Jordan’s blog.

                                                          1. 1

                                                            I never felt comfortable in IDEs. It’s so easy to loose yourself, and because you stop thinking in terms of buffers, people stop programming that way, that in turn might even increase your dependence on IDEs. I’m not sure if that is what we want, at least as long as programming is actually still done with files, in byte-sequences, etc. It is imaginable, that there is a potential to make a step forwards, to leave behind some of the complexity that didn’t scale well. In my eyes, it reflects the general sad state of computing we have cought ourselves in.

                                                            Supporting evidence here is Go. According to survey results, text editors are stably declining in popularity in favor of IDEs.

                                                            My personal theory is that in the last few years, Go tooling has broken (such as go-code for autocompletion), making it more burdensome to maintain it seperatly. Of course, replacing it with gopls, a LSP implementation, that self-describes itself as alpha software isn’t that great either, so I get it if people who don’t want to bother with all these things choose IDEs, just to forget it.

                                                            What annoys me, is that for a long time, Go was one of the best languages to work without IDE support, probably also because of it’s Plan 9 heritage, and the interest to make it work in Acme.

                                                            1. 2

                                                              It’s so easy to loose yourself, and because you stop thinking in terms of buffers, people stop programming that way, that in turn might even increase your dependence on IDEs.

                                                              This seems to imply that thinking in buffers is somehow different (or better?) than thinking in whatever displays the code in an IDE. And is a dependence on an IDE really much different than dependence on Emacs or Vim?

                                                              1. 1

                                                                I think it is different, thinking in terms of buffers (or rather files) works with the IDE approach, but the other way around is harder, which is arguably a limitation of the more traditional file-based approach.

                                                            1. 11

                                                              When designing tools, it is important to consider both of these attributes, and it helps to keep the intended audience in mind. I think that many programmers today are overly concerned with usability, and insufficiently concerned with utility. Some programmers (although this sort prefers “developer”) go so far as to fetishize usability at the expense of utility.

                                                              If anything, I’d say that not enough programmers focus on usability. Most software systems are hard to use, inconsistent, and have regular arbitrary changes to the user interface. It doesn’t matter what your system is capable of if no one can use it. The history of computing is filled with extinct systems that were hard to use in comparison to a competitor.

                                                              I argue that most programmers are professionals who are going to invest several years into learning the craft. This is the audience for whom I design my tools. What trouble is it to spend an extra hour learning a somewhat less intuitive code review tool when the programming language whose code you’re reviewing required months to learn and years to master?

                                                              The pay-off for learning a new programming language is often quite clear, e.g. I need it for work, the market prospects are good, or there are compelling use cases. The pay-off for learning a code review tool is smaller and potentially negative - I may not end up using it.

                                                              I write tools to maximize the productivity of professional programmers. Ideally, we can achieve both usability and utility, and often we do just that. But, sometimes, these tools require a steeper learning curve. If they are more useful in spite of that, they will usually save heaps of time in the long run.

                                                              It’s often unclear as a potential user why the complexity is present.

                                                              • The model is genuinely different, e.g. APL, Forth, Haskell, Prolog.
                                                              • The domain is inherently complex.
                                                              • The complexity is required for some rare use cases that I’ll never run into.
                                                              • The complexity exists for historical reasons, e.g. termios, POSIX, and Sendmail.
                                                              • The user interface was an afterthought; the designer focused their attention on the implementation.
                                                              • The system evolved arbitrarily without an overall design.
                                                              • The design was done under a severe time crunch, e.g. JavaScript.
                                                              • The designer tried but was unsuccessful in creating an easy to use system.

                                                              The first two reasons are rare; if your design is complex it’s probably for one of the other reasons.

                                                              1. 2

                                                                The pay-off for learning a new programming language is often quite clear, e.g. I need it for work, the market prospects are good, or there are compelling use cases. The pay-off for learning a code review tool is smaller and potentially negative - I may not end up using it.

                                                                There are two tools I am willing to put up with a steep learning curve: an editor and a programming language. Those two tend to transcend working environments, less so the language (although language pedigree or paradigm does). To some degree, the system on which these things run (*nix, Windows, and so forth).

                                                                Code review systems, emulators, search tools, version control system, and especially email clients (among other tools) are not things that I feel I should be spending my time learning in great detail. These are very likely determined by the organization you work for. At none of my jobs have I had the choice of email client, for example. Do I care? No. These things are just incidental to the work at hand. I will eventually learn to use what the workflows of the group are built around. It’s typically not that hard.

                                                              1. 4

                                                                Nice introduction (based on both tutorials). One suggestion would be to use

                                                                -v FPAT='[^,]*|"[^"]+"'
                                                                

                                                                instead of

                                                                BEGIN { FPAT = "[^,]*|\"[^\"]+\"" }
                                                                

                                                                If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk.

                                                                I can’t work my head around this quote. That’s a ridiculous claim. Even for a experienced programmer, learning a new programming language in 2 weeks, let alone 2 hours would be nothing short of a miracle. I’ve been using awk for past 2-3 years or so and I wrote a book on GNU awk one-liners earlier this year (https://learnbyexample.github.io/learn_gnuawk/). I’m no where close to knowing all of awk

                                                                1. 6

                                                                  I’m no where close to knowing all of awk

                                                                  Awk or GNU Awk? Awk itself is very small and simple. If you have mawk on your system, then man mawk will probably outline the entire thing (about 1000 lines). I have a copy of The AWK Programming Language from 1988 and it’s a thin book that’s easy to digest.

                                                                  GNU Awk has so many extensions you can’t possibly learn it in any reasonable amount of time, and frankly, most of those extensions are not all that useful. Awk is stunningly great for simple text processing and incredibly useful, but as a general purpose programming text processing language, which what GNU Awk seems to be going for, it’s subpar. You are better off with something like Perl or Python because data structures and functions in Awk kind of suck. (I swear the extensions in GNU Awk were added to deal with the bizarreness of some of the GNU builds where a tool was used, the situation changed that really called for a different tool, but for whatever reason the existing tool was extended in unnatural ways to deal with it.)

                                                                  1. 1

                                                                    Yeah, I had GNU awk in mind when I meant all of awk. Just glanced through man mawk and it is short indeed. I’d say I’m reasonably familiar with most concepts. I know getline for basic usage, but tend to avoid because of caveats (see http://awk.freeshell.org/AllAboutGetline). Also, my awk usage is limited to one-liners for most part, so I haven’t bothered to learn about functions (which has significant spaces in syntax).

                                                                    I’d disagree with your take on GNU awk extensions. Many of them are useful for one-liners too. For example: FPAT, multicharacter and regexp based RS (plus RT), FIELDWIDTHS, in-place editing, BEGINFILE, ENDFILE, array sorting with PROCINFO, 4th argument to split, 3rd argument to match and so on.

                                                                    And I do use Python or Ruby these days if I need to write a program file instead of one-liners.

                                                                    1. 1

                                                                      sort

                                                                      Is possible to implement sort() in awk (this is a quicksort):

                                                                      function swap(array, a, b,
                                                                              tmp)
                                                                      {
                                                                              tmp = array[a]
                                                                              array[a] = array[b]
                                                                              array[b] = tmp
                                                                      }
                                                                      
                                                                      function sort(array, beg, end)
                                                                      {
                                                                              if (beg >= end) # end recursion
                                                                                      return
                                                                              a = beg + 1 # 1st is the pivot, so +1
                                                                              b = end
                                                                              while (a < b) {
                                                                                      while (a < b && array[a] <= array[beg]) # beg: skip lesser
                                                                                              a++
                                                                                      while (a < b && array[b] > array[beg]) # end: skip greater
                                                                                              b--
                                                                                      swap(array, a, b) # found 2 misplaced
                                                                              }
                                                                              if (array[beg] > array[a]) # put the pivot back
                                                                                      swap(array, beg, a)
                                                                              sort(array, beg, a - 1) # sort lower half
                                                                              sort(array, a, end) # sort higher half
                                                                      }
                                                                      

                                                                      This sorts the array values using integers keys: array[1], array[2], … It sorts from array[beg] to array[end] included, so you can choose your array indices starting at 0 or 1, or sort just a part of the array.

                                                                      Example usage: with the both function above:

                                                                      {
                                                                              LINES[NR] = $0
                                                                      }
                                                                      
                                                                      END {
                                                                              sort(LINES, 1, NR)
                                                                              for (i = 1; i <= NR; i++)
                                                                                      print(LINES[i])
                                                                      }
                                                                      

                                                                      Performance is far from terrible!

                                                                      $ od -An /dev/urandom | head -n 1000000 | time ./test.awk >/dev/null
                                                                      real    0m 19.23s
                                                                      user    0m 17.90s
                                                                      sys     0m 0.12s
                                                                      
                                                                      $ od -An /dev/urandom | head -n 1000000 | time sort >/dev/null
                                                                      real    0m 4.39s
                                                                      user    0m 3.00s
                                                                      sys     0m 0.10s
                                                                      
                                                                      1. 1

                                                                        I’d disagree with your take on GNU awk extensions.

                                                                        The stuff you cite can be useful (although I use Awk pretty much every day and I think I’ve used FIELDWIDTH once and that’s about it), but then there’s the other stuff.

                                                                        And that’s not even all of it.

                                                                    2. 2

                                                                      I do not know GAWK, even though I’ve read Robbins’s book at some point in time. However, what Ultrix called nawk I read in a day and was productive writing “almost C” in nawk and very few idioms right away. Then Perl4 came along and I switched.

                                                                    1. 2

                                                                      Speaking from experience, processing CSV files with Awk (even GNU Awk) is a fool’s errand. Use something that already handles all the weird corner cases.

                                                                      That said, Awk is great and you should learn it. You really can learn it in under a day.

                                                                      1. 1

                                                                        There is csvawk: https://github.com/DavyLandman/csvtools

                                                                        It uses a C converter from .csv to a custom binary delimiter unexpected from input, then calls awk setup with this delimiter.

                                                                        It sadly uses a custom BEGIN { IFS = "..." } instead of awk -F "...", but it should really not be too hard to convert it to use -F instead.

                                                                        1. 1

                                                                          Yeah, here’s what a robust solution for parsing csv with awk looks like: https://stackoverflow.com/questions/45420535/whats-the-most-robust-way-to-efficiently-parse-csv-using-awk

                                                                        1. 4

                                                                          I hope this submission isn’t too much like an advertisement. I was just excited to hear more about it, ever since it was announced.

                                                                          1. 2

                                                                            Indeed, it looks too much like an advertisement. It might have been better to just link the product brief.

                                                                            1. 3

                                                                              That would also be too much like an advertisement, in my opinion.

                                                                          1. 2

                                                                            This is a good analogy. And I enjoy your writings @hwayne, but:

                                                                            Flossing takes like three minutes a day.

                                                                            That’s way longer than even my dentist takes to floss my teeth at an appointment. Either I’m really good at it or I’m doing something wrong. Here’s hoping my meagre attempts and making my software correct isn’t on par with my apparently deficient flossing technique!

                                                                            1. 4

                                                                              Indicating the return status of the last command is really useful, but I’ve found that just colour is too subtle most of the time. If you do this, you may want to experiment with putting the error code in the prompt if it’s not zero. Something like

                                                                              %(?..(%?%))
                                                                              

                                                                              to print (1) if the exit status was 1, for example.

                                                                              1. 1

                                                                                Riiight, I tried doing that and didn’t like it. So far that color-coded prompt seems to be noticeable enough for me, but maybe that’s because of novelty, I’ll see. :)

                                                                                1. 1

                                                                                  Fwiw, I just use:

                                                                                  setopt print_exit_value
                                                                                  

                                                                                  That way my prompt never changes, and I get the actual return code printed out blatantly like so:

                                                                                  zsh: exit 129   ~/sync.sh
                                                                                  

                                                                                  I’d rather know the actual return code, and do nothing if everything is ok and not clutter the prompt at all.

                                                                                  1. 2

                                                                                    That’s exactly what I replaced with colorful >. :-) I’ve been living with print_exit_value for a long time but it’s nice when command output is exactly command output. :)

                                                                                    1. 1

                                                                                      Heh, I’d rather have the extra lines in this case so its obvious that something exited non zero. To each their own!

                                                                                2. 1

                                                                                  for those who care about a visible exit code, I found

                                                                                  setopt PRINT_EXIT_VALUE
                                                                                  

                                                                                  a nice personal solution. It does not go in your prompt

                                                                                  1. 1

                                                                                    I gotta refresh my tabs more often heh jinx but I agree entirely.

                                                                                  2. 1

                                                                                    IMO, I’m not a huge fan of this, as the fact that the exit status is not zero does not always mean an error has occurred. I suppose it really comes down to workflows and what tools you use all the time. But my prompt turning red because my compile failed is not any more informative than the three pages of errors I got prior to that. :)

                                                                                    1. 2

                                                                                      …exit status is not zero does not always mean an error has occurred.

                                                                                      Indeed. I just use it as a useful point of information for those commands whose error codes are siginificant or I can’t tell actually failed. It’s always a personal preference thing, though.

                                                                                  1. 2

                                                                                    Stuff like this is one of the things that annoy me about zsh. You can write all sorts of symbol noise for things like PROMPT, globs and expansions. Personally I think

                                                                                        local last_exit=$?
                                                                                    
                                                                                        case $last_exit in
                                                                                            0) last_exit="${fg[green]}${last_exit}$reset_color" ;;
                                                                                            *) last_exit="${fg[red]}${last_exit}$reset_color" ;;
                                                                                        esac
                                                                                    

                                                                                    is more readable.

                                                                                    EDIT: Despite the comment below, you can use ANY string in PROMPT (or PS1) in ZSH. At worst, you have to worry about escaping special characters associated with the PROMPT DSL, namely ‘%’ and ‘)’ which are escaped with ‘%%’ and ‘%)’ respectively.

                                                                                    This is doable trivially in ZSH:

                                                                                    zshprompt_escape_percent() {
                                                                                        REPLY=${*//\%/\%\%}
                                                                                        REPLY=${REPLY//\)/\%\)}
                                                                                    }
                                                                                    zshprompt_escape_percent $prompt_string
                                                                                    PROMPT=$REPLY
                                                                                    

                                                                                    thus using the above snippet I posted could be used like

                                                                                    precmd() {
                                                                                        local last_exit=$?
                                                                                    
                                                                                        case $last_exit in
                                                                                            0) last_exit="${fg[green]}>$reset_color" ;;
                                                                                            *) last_exit="${fg[red]}>$reset_color" ;;
                                                                                        esac
                                                                                        zshprompt_escape_percent $last_exit
                                                                                        PROMPT=$my_prompt$last_exit
                                                                                    }
                                                                                    
                                                                                    1. 1

                                                                                      It would be helpful to indicate how you would do this since you can’t put this sort of thing in your PROMPT. (And if you can’t, indicate that as well.)

                                                                                      1. 1

                                                                                        you can indeed use it in PROMPT (or PS1). you just need to worry about, and escape, % by doubling them up; there is also ) which is escaped with the sequence %). I generally don’t use PROMPT though since mine is multi-line.

                                                                                        I updated my comment though, thanks.

                                                                                    1. 10
                                                                                      #define macro(x) do { foo(x); bar(x); } while(0)
                                                                                      

                                                                                      The example macro, as it stands, evaluates x twice, which is a macro faux pas.

                                                                                      Also, it is ill-advised to use a macro as a statement within another statement aside from a compound statement, regardless of the do {...} while (0) convention. It very likely will mislead the reader into thinking it’s actually an expression. This is a matter of style for the most part, but generally a good one, in my experience.

                                                                                      1. 1

                                                                                        very likely will mislead the reader into thinking it’s actually an expression

                                                                                        Why? Do you never call void functions?

                                                                                        1. 3

                                                                                          Sure I do. But then, if you use this macro in an if condition, you’ll be a bit surprised.

                                                                                          1. 2

                                                                                            No more so than if you used a function or variable returning void, or a structure. I don’t see as it’s particularly of concern.

                                                                                            1. 5

                                                                                              The point is to signal to the reader immediately that they are dealing with something that may only be well defined from a specific context that is beyond what one would expect from the language. Since macros can expand to pretty much anything, it’s helpful to the reader to know that what they are looking at may be very different than what it seems. That’s why the convention of uppercase naming for macros is useful. And also why it’s useful to put them in a context that doesn’t lead you to believe it might be a function and evaluated as such. (A statement that is just foo(); looks more like a function than anything else. But if you write it { foo(); } then it’s more likely to signal to the reader that there may be more going on than what appears to be a function call.)

                                                                                              And of course that can be violated and of course you’ll find out one way or another. Like I said, it’s a matter of style. I think it’s poor style.

                                                                                              It also doesn’t help that the original macro is an utterly uncompelling example. If you are writing a macro to do a common sequence of operations, you are better off to use a function (possibly inline) to avoid the problems of text replacement from macros in the first place.

                                                                                              1. 1

                                                                                                Yes, I completely agree with you. A function would suit this case better. However, that is not the point of the post.

                                                                                                1. 2

                                                                                                  What you have is not wrong (although the multiple evaluation is questionable), but the motivation needs to be there for the post to have more impact. What your post describes is a well-known problem in C. The do while (0) approach has been around for a very long time. With the long history of C and the sheer number of lines of code out there, it would probably be better if you found an actual example from real code to make the point.

                                                                                                  Here is one to consider. It’s in GCC and it even uses the the style that I disagree with, although in this case it’s only used in a single function so maybe there’s a better case for it. The history of it shows that it didn’t use the do while (0) pattern at first, which led to a dangling else issue. That has a lot more to it than a foo bar cooked up example. It’s also instructive to show why it is not a function, even though it looks like it could be.

                                                                                                  I understand that you’re probably learning your way through C. That isn’t a problem. However, you may want to cite some actual code to show the interesting (or weird) parts of the language. Code search is actually a reasonable thing these days and there’s no shortage of C code to look at. Good luck!