1. 9
  1. 10

    Still, governments relying on a system too arcane for most working engineers can be perceived as a structural failure. Murphy’s plea for COBOL engineers is also a sign that local, state, and federal governments have overwhelmingly failed to update their technologies to meet the needs of citizens.

    So what’s the alternative? Serious question, because I don’t see a clear “let’s just do […]”-solution here.

    The problem with these COBOL systems isn’t so much that it’s written in COBOL, but that’s it’s a mess reflecting decades of changes in laws which are often hastily and hackish implemented (gotta finish it before the law takes effect). That it all originated from a time when global state was normal and structured programming wasn’t the norm isn’t helping either.

    A few years ago I worked on a system to manage rental contracts in NL, and it was a problem even there: laws would change even as we were building it. And then you run in to the “oops, didn’t consider that, that’s actually quite hard to add in the current design”-kind of issues, and either have to refactor a lot or hacketyhack it on.

    And re-doing it all from scratch is both costly and risky; the required domain knowledge is often complex, and starting with a MVP and iterating on that also doesn’t necessarily work very well in this space.

    Furthermore, what will the situation be in 30 years? Will we have a “new COBOL” then? What kind of tech would be suitable for the next 30, 40, or 50 years? Rewriting all systems every few decades from scratch because some tech fell out of favour doesn’t strike me as a good idea. Clearly we need some more vision here than “just rewrite it in $popular_current_language”.

    So I don’t think it’s a failure of government, or at least not exclusively. It strikes me as a failure of the tech industry in failing to train people, as well as in not providing a stable long-term platform to build these kind of systems on, as well as the somewhat fickle nature of our democratic system.

    On the other hand, the industry is still comparatively young, and we’re all trying to figure out these issues as we go along, but I think it’s a lot more complex and nuanced than “governments have overwhelmingly failed to update their technologies”. I think this is a classic case of someone looking in from the outside, going “zomg”, but not fully understanding why things are the way they are, and then drawing far too simplistic conclusions.

    1. 6

      There is no alternative. You either treat software as done and encounter this and bitrot or you treat software as a living thing that needs to change over time and incur the maintenance cost. You don’t really get to treat software as done and then revisit it decades later with identical productivity as you would if you constantly moved it.

      That’s just a cost-benefit trade-off that we weren’t aware of in the past, so no one consciously made it. But now with 60 years of history, we know this to be true, and so we can make an informed decision to pay the cost ongoing or to pay the cost at change time.

      I suspect the incentives are towards the latter automatically.

      1. 2

        If the software “just works” and does what it needs to do, what are you going to do with it? Keep refactoring it or rewriting it in a new language every day? Eventually you’d have to agree that yes, the software is “done” and stop pouring money into it like a bottomless pit.

        Perhaps it would be a solution to formalize the behaviour better and add many tests (or even a formal proof? We have the technology…) so that rewrites are easier. If the tests can be run outside of the program (i.e., written independently of the language used to implement it) this should be quite possible. Even a team completely unfamiliar with the code base could theoretically write a new identical program from scratch if it was needed.

        But then you run into the cost/benefit issue. If you have piles of code implementing complex logic, and a new law is passed, you’d be able to implement logic for the new law in a day, or you’d be able to rewrite the entire thing in a “modern” language in two months, what is the best way to spend tax payers’ money?

        Perhaps it truly is best to have specialized shops which know these old languages and keep training new people. Then it just becomes a marketing issue, because who wants to write this kind of code? Personally, I think it would be pretty awesome to learn COBOL (I’m always interested in learning different languages), but then after one or two such actual projects I’m sure I’d get burnt out/annoyed/tired of it. These kinds of codebases don’t sound like they’d have the cleanest nicest code to work with, and the original engineer probably isn’t around to ask questions of. I’ve done enough projects where I took over from someone else and the code base was a total mess. Making a mistake when “refactoring” this kind of code could be very dangerous/costly, and without tests you’ll always have this nagging feeling that you might’ve gotten it wrong.

        1. 1

          I don’t think anyone disagrees on that, but the question is, where do we go from the current situation?

          Perhaps they should be rebuilt from scratch, somehow, but I fear that it won’t be done with a clear long-term vision on how it will look like in 40 or 50 years, meaning we’ll be in the same boat in 20 years if we do.

        2. 6

          Will we have a “new COBOL” then?

          Honestly, Java already feels like this.

          1. 3

            PHP is a more likely candidate for a “new COBOL”, but I don’t see Java or PHP going away anytime soon.

            1. 3

              Critical government and banking business logic isn’t written in PHP though, whereas Java actually is used in that space.

              1. 2

                You’d be surprised.. But it depends on which kind of “critical”. I know of risk assessment software for credits, but not transactions for accounts.

              2. 1

                No way, PHP breaks too many things on each release. Lately its stabilized somewhat, but it’s nowhere near the status like they say COBOL has, where a program written in the sixties still works unchanged. I challenge you to find nontrivial PHP code written 10 years that even works on PHP 7. You might even have trouble finding code that’s 5 years old that works in the current version, given that PHP 7.0 came out in 2015 (not everyone switched to it immediately).

                Java probably comes closer, but even there I think newer versions break backwards compatibility (I’m not a Java programmer, so maybe I have this wrong?). Perhaps one of those other older languages still in use like Common Lisp or ADA?

                Seriously though, this really is a problem of this industry in general. On the other hand, you don’t want to be using dead languages that no longer evolve (which Common Lisp kinda is, and COBOL definitely is), and having layers upon layers of old cruft doesn’t sound particularly appealing, either.

                1. 2


                  I feel Ada was almost this for a while (almost since it was conceived, even - C programmers shit on it as “verbose defense contractor language” at the time), but now there seems to be a resurgence of it, thanks to Rust.

                2. 1

                  But Java 9 did start deprecating APIs…

                3. 1

                  I remember reading an article saying there were already problems with systems built on very old Java versions that can’t really be upgraded and that few people have practical skills in. I can no longer find the article though :-(

                4. 1

                  What kind of tech would be suitable for the next 30, 40, or 50 years?

                  Most likely (in descending order):

                  • C: the world’s most portable programming language. Tons of CPU design revolves around compiling/running C programs. Even bootstrapped languages like Golang still have C-based compilers (gccgo).
                  • JavaScript: even if WebAsssembly takes the web by storm, way too many pages still run JS. The ecosystem will need to be kept alive.
                  • Fortran. Fortran still beats C at numerical work concerning matrices. BLAS and LAPACK are used by tons of mathematical software, including SciPy. As long has HPC and complex numerical work are a thing, Fortran will have its place alongside C as the lower-level language of choice.
                  • Java: IT runs on Java 8.
                  • Python: its scientific stack is more than just NumPy, SciPy, Matplotlib, and Sympy; it also has libraries like AstroPy, SunPy, SpacePy, PlasmaPy, Biopython, and PsychoPy. Python’s mathematical/scientific stack is unparalleled. Tons of young people (like me) learned it as one of their first languages; they’re teaching it to middle-schoolers now. People are unlikely to forget their first language.

                  It’s not a sexy list, but I’m pretty sure at least four of these five will still be quite relevant in 50 years.

                  Okay, let’s try some more fun/risky predictions. I predict that at least two of these memelangs somewhat newer languages will be a thing:

                  • Rust
                  • Go
                  • Zig
                  • Crystal
                  • Nim
                  • Brainfuck (will be a thing as long as Turing Machines are taught in CS classes).

                  I’d like it if Nim, Rust, and Zig continue to be a thing. Rust and Go have the highest odds.

                  1. 6

                    The problem is much deeper than “which languages will still exist in some form in 30 years”, it also has to be a suitable language for this kind of space, be a stable language which doesn’t change every 5 years, a robust language where mistakes are hard, should probably be clearly defined by a standard, preferably not controlled by a singly entity, and something you will still be able to find programmers for relatively easy in 30 years.

                    I mean, C or Fortran are still going to be around, but are clearly not the best suited languages for this. I don’t know what the “best” language would be, but I think it’ll probably be closer in the direction of Ada – although perhaps not Ada specifically – than Python or Rust.

                    1. 1

                      Java and C# are the most likely candidates.

                      JavaScript and all the modern webtechs are going to be a nightmare to maintain, “This gov’t website was built with angular 7 and has been left to rot for 20 years, now we need to update it”. By their nature SPAs require a rest backend, so it’s probably going to be easier to replace the whole front end.

                      But the backend is where all gnarly business logic, nasty integrations and 1000 line sql queries happen.

                      1. 1

                        I’m not advocating these languages, it’s just reality.

                  2. 4

                    COBOL isn’t the best of all possible languages but the real problems are financial.
                    New Jersey has the worst finances of any state in the nation.
                    Connecticut ranks 48th for finances.
                    This has undoubtedly affected maintenance budgets, projects, and hiring.

                    1. 4

                      This hit the orange site yesterday, and one of the top comments for a while indicated that the real issue was less to do with the technology or language, and more to do the management and administration. Later discussion brought up fairly consistent descriptions of historical finance and management issues in New Jersey In particular.

                      I’m not in any position to agree or disagree, but in my experience technology and/or language aren’t always the most significant factor in a system that’s “falling apart”.

                      1. 4

                        Zed Shaw had very interesting Tweetstorm on this:


                        1. 4
                          1. Set a deadline of end of year 2020 to have the COBOL system totally replaced, and offer a $2 million dollar pooled bonus to any employees that can pull it off and that stay to pull it off. Those employees from #1 would still get their retirement, and sweet bonus.

                          Yeah, a hastily built system built in a matter of months to get a cash bonus will surely be well designed and stand the test of time 😒

                          1. 2

                            It’s not that much different from current religion.

                            Gotta get your stories done by the end of the sprint!

                            Hurry, hurry, hurry!

                          2. 2

                            Wow, Zed is still at it? Missed this guys crazy rants.

                            1. 1

                              I wonder if he still uses Python 2

                          3. 3

                            Any non-maintained system will fall apart anyway, so Cobol is not an issue here. Rewriting it in Java, .net or any new fancy environment will cause the same issue in the future - maybe even quicker, as these languages are evolving much faster than Cobol.

                            1. 1

                              Running COBOL requires legacy IBM hardware? But COBOL is a fairly simple language? Then someone should write a COBOL-to-XXX transpiler, where XXX is some mainstream language, and make a fortune selling it to all these governments and businesses. If they pay some attention to readability of the generated code, the customers could retire the COBOL source and do future maintenance in a language lots of people know.

                              1. 4

                                This was in fact done. Multiple times. The one I know is NACA, a COBOL-to-Java transpiler. It was used to migrate 4 million lines of COBOL in production.

                                1. 3

                                  Very impressive project, particularly the CICS support. Looking through the github I didn’t see anything related to JCL, was that supported also? Did you participate in the migration of the code base?

                                  I worked at a company that had similarly migrated a legacy system to java using a cobol to java transpiler. This company attempted to continue developing the app using the transpiled java source code. The java source code read like assembler. After a few years the converted version was dropped and they had reverted back to the mainframe cobol version, which luckily (for them) had not been decommissioned yet.

                                2. 3

                                  Any programmer can pick up COBOLin a few weeks. It is a much simpler language than Java, which is routinely used to replace COBOL systems, despite being a poor fit for the domain.

                                  1. 2

                                    For mainframe hosted apps, Cobol as the language is incidental; it’s the hosting environment that’ really locks them in; Jcl for batch processes and CICS for interactive green screen applications. Microfocus has had a mainframe rehosting solution that’s remarkably complete for many years now (10-15 years that I know of). There are options out there, but they require will and financing to implement.

                                    1. 1

                                      NACA includes CICS emulation. It’s an entirely solvable problem.

                                    1. -1

                                      What are my taxes for?

                                      1. 2

                                        This episode of Yes Minister should answer your question: The Empty Hospital

                                        1. 1

                                          Same thing budgets are always used for… building new things rather than maintaining exiting critical infrastructure. Doesn’t matter if it’s governments building roads or Google building yet another instant messenger platform, maintenance isn’t sexy.