1. 19
  1. 17

    There isn’t always a tidy solution for everything. I used to believe that if I just look at a problem the right way, it would disappear. There are cases where you can remove complexity by making a smart design decision, but that is very rare. Usually, you simply need to deal with the problem directly.

    Most of these smart design decisions have been made already, but we are seeing frameworks being developed to solve specific problems more efficiently, this is how I expect improvements to continue. I don’t expect that there is going to be a major breakthrough.

    1. 19

      I think we should just accept that programming is hard and that text is the most efficient way to represent a program and be done with this discussion.

      We spend endless amounts of time bikeshedding the right syntax, indentation level, tabs vs spaces, or where to put code in the structure of files,

      This is a straw man. I can’t remember the time I had a discussion about syntax. Mature developers have moved on years ago. I do remember some conversations (not discussions) about the structure of files, but this was always in the context of modeling the program. So it was actually a conversation about the program and therefore very useful.

      Or even better, I want to specify a goal like “I want this function to not take this parameter” and let the system figure out how to transform the program to achieve this goal. I can imagine a system that can combine small transforms into larger ones and use some AI magic to figure out how to compose them to achieve the specified goal.

      This is not going to happen. If we have an AI that understands what I mean when I just remove a parameter we will have created an AI that is as smart as a human and we won’t need any programmers anymore because the robots will take care of everything. Sounds nice, but I don’t expect it to happen in my lifetime.

      No matter how you put it, you will need a medium to communicate your program. To the computer and to other developers. Text is the worst medium for that, except for all the others. So instead of hoping for some magical solution that makes this all go away, I’d rather go into the other direction and just try to get better at manipulating text.

      There is so much untapped potential. No matter if you’re the type of developer that uses a fullblown IDE or one that prefers vi because vim is ‘too bloated’, I bet you can learn some neat tricks in a single afternoon that will boost your productivity for the rest of your career. This is something you can do right now, without any help from others. You definitely don’t need to wait for some mindblowing breakthrough from the smartest people in our field to get better and make your own life easier.

      1. 17

        This is why we can’t have nice things.

        I’m so very tired of hearing various forms of “everything we had in the 70s is the best possible solution” The idea that we somehow accidentally settled on perfection about ten minutes after programming was invented is farcical. We need people to try new interesting models of programming, and this obsession with plain text is an anchor around the neck of anybody trying to find a path forward.

        1. 4

          I kind of agree with you, but then practically every other day I see articles here about people re-discovering or re-implementing concepts that have been in Lisp for 50 years. A lot of times it’s not even as good as what Lisp had.

          1. 3

            I can empathise with this a ton. I think the newer and younger generations of software engineers and programmers get a lot of flak for wanting to try new approaches to age-old tech. I really do hate a lot of the ancient stuff, older folks say “you’ll get used to it” but I’d rather take a shot at rethinking some things.

            1. 2

              To give a late reply to this: I am not against new things. I was more reacting to the general vibe of the post. I felt it as: “This is hard, can someone else please make it easier for me?” My reaction would be different if it was more constructive.

              I don’t think the 70’s had the best possible solution for everything. Although for while I liked to stir up conversations with fellow tech enthusiasts stating that “Nothing fundamentally new has been invented in tech after the 70’s. Convince me otherwise” and see people really struggle to come up with something. That doesn’t indicate that there weren’t any new things, or that what we have cannot be improved on a non-fundamental level. But it does indicate that progress is really hard and that we’re going to be stuck with this mess for a while. So please do try to improve, but don’t hold your breath and get comfortable with how it works now for your own sanity.

            2. 2

              I’m a bit more optimistic. “If we have an AI that understands what I mean when I just remove a parameter we will have created an AI that is as smart as a human” - there’s a lot we can do on terms of code transformation that doesn’t require any general AI. As long a we keep to “equivalent behaviour” transformations, it’s a fairly well known territory even. That’s exactly what the optimisation stages do in a compiler. If we know how to inline and propagate constants, we can surely generate candidates for “remove this parameter”.

              No matter how you put it, you will need a medium to communicate your program.

              The way to communicate the program does not have to be the same way it’s processed/stored. The same way you draw vector graphics, you communicate with the graphical interface, even though the internal model has a lot of splines - you never see those.

            3. 10

              Programmers can’t even agree on basic things like when to test, how to test, if unit tests are any good or if TDD is the only true way to build software. This is a good hint that we haven’t found the right way, yet.

              Or maybe it’s a hint that programming is a complex social activity and there will always be more than one way to do it? I can’t think of a single profession that has arrived at One True Way to practice it. People are dynamic and will always come up with new and diverse ways to solve problems. That diversity is a strength, not a weakness.

              1. 4

                Refactoring doesn’t even need to exist as a separate concept or activity - programming itself is that. We regularly change the whole program every day.

                I believe we should already be doing that, even without more tooling than we have today.

                1. 6

                  One of my bitter observations from my corporate days is that the prominent “refactoring” bouts in team-level schedules is one of those awkward consequences of strict Agile orthodoxy. Management leads have to pay lip service to all the agile hoodoo because it’s just how things are now. But they also have to reconcile it with the obligation to call things done at the end of every sprint, otherwise they have no clear progress to report, and they can’t give their higher-ups that warm fuzzy feeling of being in control, which tends to be detrimental to job security. However, this framework also makes it difficult to throw prototypes away or just admit that you got a design wrong to some degree and you gotta change some things. If you do that, somebody might get the (utterly heretical!) idea that there’s only so much thinking you can do over 1-2 weeks and maybe bending over backwards to fit everything in fixed-lengths sprints is stupid sub-optimal.

                  So instead we put these “refactoring” bars in the Gantt chart, which is really just fancy jargon for “we gotta think about it a little longer and program it some more” but you can’t say that, otherwise John Frum will stop sending supply planes on fruit day and the world will end.

                  1. 11

                    If you have a Gantt chart, then nothing you do resembles “agile” at all, no matter how much you say the word Sprint.

                    1. 2

                      Shh, you can’t say that, either!

                    2. 4

                      there’s only so much thinking you can do over 1-2 weeks

                      That is a very bitter take. More realistically, I’d say that there’s only so much you can actually think through. Most of the real issues you’ll see only in practice. Most of the time you have to try something and see how it well or badly it does in the real world. And if it doesn’t work well you can almost always redo it. Often, the half-baked solution is enough to bridge the gap to the proper solution. So the focus on getting things done is not a bad one.

                      And getting something done now and redoing it properly later also means you get to invoice the customer twice, which may not be great for the customer, but at least allows you to keep your job and actually get to make the improved version.

                      And often a stupid thing that works for most cases today is better than a perfect thing that works for all cases a year from now.

                      However, I do agree that it’s important to be able to take the time you need to design things well and that doesn’t fit all that well with Strict Agile(TM) because even though hammock time can be explicitly scheduled, your subconscious won’t obey deadlines. But yeah, try selling that to customers…

                      1. 1

                        That is a very bitter take. More realistically, I’d say that there’s only so much you can actually think through. Most of the real issues you’ll see only in practice. Most of the time you have to try something and see how it well or badly it does in the real world.

                        Of course. Thing is, when you run into one of these things, it’s important to be able to recognize that they’re not quite done the first time around. But it’s very hard to fit “advanced enough that we can check how it holds in practice” in some communication frameworks, so we get to resort to the “done” + “erm, it’s done but we have to do some refactoring” artifice. You can’t just say “we gotta do some more programming”, because that brings up the words nobody wants to hear – “but I thought we had it done last week?”. So you have to come up with some sort of “programming light” activity – something you can say still needs to be done while maintaining the “the task is done” fiction.

                        The focus on getting things done (and, I might add, on early prototyping) is absolutely something I agree with. It’s really important to be honest about it though :-D.

                        1. 2

                          I guess it’s that honesty that’s a bit missing in some organizations. Places where I’ve worked it would be fine to say it needs another round of improvements (and if you want you can call it refactoring, because sometimes that’s all it is).

                          The way I’d typically do it is mark the original ticket as “done”. and make a new ticket with the improvements to be done, then link it to the original for reference. I don’t think there would be a lot of places that would disprove of that. Nobody wants a ticket that drags on for months, that’s bad for everyone’s morale.

                          Though, to be fair, I have one ticket that I’ve been chipping away at for… checks… ZOMG, it’s been 8 years now, that’s my current record. This is in a small open source project where the number of tickets is manageable and we have no customers breathing down our neck and we feel like we can take our time and do things right. So I certainly see your point of view!

                          1. 2

                            I guess it’s that honesty that’s a bit missing in some organizations. Places where I’ve worked it would be fine to say it needs another round of improvements (and if you want you can call it refactoring, because sometimes that’s all it is).

                            Yep. I mean, I’ve done this ticket-splitting thing smoothly, too. Thing is, the ability to do so depends on an understanding that:

                            1. There are things which can be mostly done on the first try, but also things which are definitely going to need some hammering around the edges once they’re integrated, and
                            2. That this is inherent to some types of tasks/code, and not representative of team leads’/project managers’ ability to manage estimates and task scope.

                            For example, API-level work inherently falls under #2 – there are best practices for API design, yes, but there are going to be all sorts of interface improvements, performance issues, and even major flaws that you won’t be able to identify until the API has seen some use outside the team that developed it. Managing this in terms of the typical “extra X% for refactoring and review buffer” is pretty much doomed to fail.

                      2. 1

                        strict Agile orthodoxy

                        It’s funny you say that, since most of what you describe is more a result of taking some guidelines and treating them like hard and fast rules (as some folks do). But sadly, “trust your staff, and give them good feedback mechanisms” is a lot harder to sell than, you know, anything you can seel courses and certificates for.

                    3. 4

                      One of my personal wishlist items is a machine-translatable language keyword set. koro is a Bengali (বাংলা) version of the Go compiler and toolchain that can ~switch between Bengali and English. Many years ago, I worked on but never published a set of aliases for coding in Scheme in Esperanto but never got to the point of single-command language switching.

                      I’ve thought about a syntax such as this:

                      @lang(iso="eo", method_name="faru_io" doccomment="ĉi tiu estas dokumentada komento")
                      @lang(iso="en_US")
                      def do_something(self):
                        """
                        This is the doc comment.
                        """
                        return self.something_useful
                      
                      # These would be equivalent returns
                      assert do_something() == faru_io()
                      # and, ideally, equivalent objects
                      assert do_something == faru_io
                      

                      This would at least enable multilingual method names and documentation that with some tooling could switch between languages. I realize that having multiple languages in a single codebase could be a nightmare so the system would likely have a “language used for versioning” and tooling would automatically translate for the language of the viewer.

                      Why? Accessibility. I’m a native anglophone, but I don’t think people should have to suffer through English to learn how to code and have a successful career in it, regardless of where they are in the world.

                      1. 3

                        Rewriting commit history (or rather beating it into shape – usually a necessary activity) is still done with sticks and stones:

                        • There are tools like gitk, that lets you see which commit does what, and git-revise that let you split, reorder and join commits, but nobody has made a GUI tool to visibly drag a hunk from one commit to another. The need to context switch is not helped by the lack of a stable commit id either.
                        • Diffs are not even text based; they are line based, which cause a lot of “conflicts” in the process that aren’t. And each conflict must be resolved twice when reordering two commits because it’s two commits. And if you use autoformatting, you also get pointless style changes into the mix, or worse, all the surprises at the end (which are very conflict inducing to backpropagate).
                        • When interactively selecting pieces for a new commit (either with git add -p or git revise -c) you can’t edit a hunk you’ve already split, well you can, but it won’t apply because it’s split.
                        1. 3

                          Compilers with stronger type systems is a step in the right direction. We talk less about what it takes to refactor at my work than I recall at previous shops (we use Rust, and before that Haskell). Having any compilation step helps heaps. Hopefully there will be more ingenious ideas that help make the art of programming more enjoyable while allowing us to tackle bigger and harder problems.

                          1. 2

                            I think this is a little too limited to the minutiae and plumbing of programming, and should instead consider some progress from our current (old) design methods of functional decomposition, structured analysis, and data modeling (OO modeling seems to no longer exist).

                            Outside of the domains of computing and the data sciences, live the systems that make the world work. The systems that manage money, commerce, taxes, insurances, transport, health, education, scheduling and environment. How to model these systems with some predictability into code is also a programming breakthrough needed?

                            1. 2

                              Once you commit to using a tool, the stuff you build with it lives on a kind of desert island. Whether you use FFIs or HTTP to interact with that stuff, it’s never going to be as good as being on the island using the tool natively. The fact that, for most programmers, the stuff is text files doesn’t change that. It’s the same with any kind of tool and any kind of artifact the tool produces or works with.

                              Thinking optimistically, though, there are some very interesting and perhaps more concrete visions for how programming can be improved, such as:

                              • Content-addressing to separate the names of things and how they’re formatted from their AST (i.e. what they actually do).
                              • ML’s type system, which has already been hugely influential in improving type systems in many other languages to establish correctness at compile time and reduce boilerplate with pattern matching.
                              • Distinguishing at the type level between pure vs. effectful code with algebraic effects or abilities.
                              • Capabilities or permissions baked into the framework if not deeper, like in WASI.

                              There are lots of folks here much more attuned to PLT than me. What are some others?

                              1. 1

                                Program is not a text

                                I like Lamdu as a solution to this, and I’d suggest people sign up for their OpenCollective. There is only one donator besides me.

                                1. 1

                                  “Writing glue code and boilerplate is a waste” - True but it’s most of the code we write now.

                                  1. 2

                                    So: huge opportunity!

                                    1. 2

                                      I think that’s also exactly the kind of code that’s least amenable to improvements. “get something from this URL, post something to that URL” is quite specific and not likely to be any shorter than it already is without losing precision. No matter how you look at it, we still need to tell the computer what to do when it hasn’t already been written. And glue code is typically not already written.

                                      1. 3

                                        s3:mybucket/hello.txt ← 'Hello World!'

                                        Native-GUI distributed system in a tweet.

                                        text → ref:s3:bucket1/msg.txt.

                                        If you have the right abstractions, you can replace a lot of glue code with “

                                        1. 4

                                          That’s like those examples where a UNIX pipeline is so much shorter than a “real” program just because it happens to fit exactly the case it was designed for, and has a lot of the same caveats:

                                          • What about error handling and recovery? Looks like this will just error hard.
                                          • What about authentication handling in S3? Looks like this can handle only one account.
                                          • What if I want to store and retrieve structured data (like JSON)?

                                          And on and on and on.

                                          There’s a lot of complexity and not all of it can simply be swept under the rug.

                                          1. 2

                                            That’s like those examples where a UNIX pipeline is so much shorter than a “real” program just because it happens to fit exactly the case it was designed for

                                            As I explain in the post: yes, it looks like code golf. But it isn’t.

                                            The key to getting compatibility is to constrain the interfaces, but constraining interfaces limits applicability. Unix pipes/filters are at one end of the spectrum: perfectly composable, but also very, very limited. ObjS has Polymorphic Write Streams, which also follow a pipe/filter model but are much more versatile.

                                            Another example that is quite composable is http/REST servers, which is why we have generic intermediaries. Again, ObjS has an in-process, polymorphic equivalent that has proven to be extremely applicable and very versatile.

                                            These two also interact very synergistically, and things like variables and references also play along. While I did build it, I was a bit blown away when I figured out how well everything plays together.

                                            And ObjS is a generalisation, so call/return is still available, for example you can hook up a closure to the output of a textfield.

                                            textfield → { :msg | stdout println:msg. }.

                                            Of course it is easier to just hook up stdout directly:

                                            textfield → stdout

                                            What if I want to store and retrieve structured data (like JSON)?

                                            You pop on a mapping store and get dictionaries. You can also pop on an additional mapping store to convert those dictionaries to custom objects, or configure the JSON converter store to parse directly to objects.

                                            #-countdockerimages
                                            scheme:docker := MPWJSONConverterStore -> ref:file:/var/run/docker.sock asUnixSocketStore.
                                            docker:/images/json count
                                            

                                            What about authentication handling in S3?

                                            Similarly, you set up your S3 scheme-handler with the required authentication headers and then use that.

                                            What about error handling and recovery?

                                            Always a tricky one. I’ve found the concept of “standard error” to be very useful in this context, and used it very successfully in Wunderlist.