1. 25

I know this is probably too broad of a topic, but there are some great tools out there that allow one to use solutions created by experts, to build something new without actually possessing expert knowledge in that topic. Some examples are:

  • LLVM: you can build a programming language that compiles to LLVM IR — and get all the optimizations for free.

  • SAT solvers — convert your NP-hard problem to SAT and run a SAT solver on it.

  • Boehm(-Demers-Weiser) garbage collector — collect garbage without actually explicitly writing any garbage collecting logic.


  2. 20

    Sqlite. A database you can bring anywhere, robust and better tested than almost anything else.

    1. 12

      This was a tool written by three of the smartest people in the field at the time (Aho, Weinberger, Kernighan) to solve a problem they had often.

      awk is still an excellent tool for exploring unstructured, semi-structured, and record-based data, and can be abused for lots of other use cases. Despite it’s age, it still includes many features that people expect from modern languages (hash tables, not having to pre-declare variables, easy use of regexes). If you know awk, you can throw out a lot of other shell tools, such as grep, cut, head, tail, and many more. Where I might have reached for a dozen or two line Python script, I often just write a dozen or two characters of awk.

      “The AWK Programming Language” is one of the clearest and most concise technical books I’ve ever read, and definitely the best ROI I’ve gotten from reading any book.

      It’s a tool I use every day. I imagine it was difficult to write in the languages available in it’s day, but more importantly the clarity of thought and foresight of it’s creators still shines through to this day.

      1. 9

        TCP/IP. UNIX and the accompanying philosophy. Vim, emacs.

        However if you’re limiting the question to code tools, I’d include a few things I know from the Python ecosystem:

        Dask Pandas

        Really the question is too broad. I could spend all day listing and not come even close :)

        1. 7

          NACL is a library wrapping crypto primitives in a manner that’s easy to use. It provides public key crypto, secret key crypto, and cryptographic signing. And because it’s built on simple concepts, there’s a version in almost every language.

          1. 11

            You’ll find that libsodium, which forked from NaCl (and is now almost unrecognizable as a fork save for a few areas), is the thing that actually caught on. NaCl barely missed the mark in terms of ease of use, libsodium filled the missing gaps.

          2. 7

            Spend a few minutes learning LaTeX.

            Markdown is fine for READMEs and the like, but whenever I’m writing a design document, I write it in LaTeX. Most folks use Google Docs, but if I’m writing something substantial, I find I’m better off writing the first draft in LaTeX, then export some HTML, and put it into Google Docs for everyone else to edit/comment/etc

            There’s a /lot/ to learn about typesetting that’s largely ignored, but the fact you can simply compile some beautifully rendered documents is a useful tool.

            1. 8

              LaTeX is great but the language itself has a steep learning curve. I’ve written papers in it and to this day maintain my résumé using it.

              I’ve spent the last few months designing a workflow around the pandoc ecosystem and really like what I’ve got out of it. My team has now authored a few internal papers using the workflow, which is comprised of pandoc and a few filters, a Markdown file for each chapter, some graphics in various formats, and a Makefile with several relevant tasks that glue it all together. We’re producing professional-looking mathematical documentation in no time with folks focusing on writing and not typesetting. I’ve got other teams lining up get educated on how to use it. It’s overengineering at its finest but the workflow has proven worth every minute spent.

              1. 2

                +1 for LaTeX resume/CV. It makes it SO much easier to customize for each application.

                1. 1

                  Is one of these papers public? Or do you have a post on your process and outcome?

                  1. 3

                    None are public. You may have inspired me to write a post about the process, though!

                2. 1

                  I prefer markdown and a static site generator (Jekyll) for design docs because I want others to contribute to, build on, and maintain. It’s hard enough getting contribs from analysts and pms and whatnot in markdown, I’m not brave enough to try latex source.

                3. 5

                  This is broad — technically any big library is this (GUI toolkits, game engines, etc. included) — but I feel where you’re going. It’s about like… specialized, interesting, deep problem domains.

                  ISPC allows you to make SIMD programs (SSE/AVX/NEON) by writing GLSL-esque almost-C code.

                  Things like Hazelcast and DeltaCrdt look pretty fun: you get distributed data structures that look like normal ones, no distributed systems knowledge required (until something goes wrong, probably :D)

                  LLVM IR

                  A similar concept has emerged in formal verification: you can use Boogie or Why3 as an intermediate verification language. Build a translator from a programming language to one of these and get verification condition generation (and launching of solvers, etc.) for free.

                  SAT solvers

                  More broadly applicable are the SMT solvers: instead of just booleans, they support numbers, bitvectors, arrays, strings and so on. And SMT solvers themselves stand on the shoulders of giants, these examples from CVC4’s build:

                  cvc4_option(USE_GLPK     "Use GLPK simplex solver")
                  option(USE_CRYPTOMINISAT "Use CryptoMiniSat SAT solver")
                  option(USE_LFSC          "Use LFSC proof checker")
                  option(USE_SYMFPU        "Use SymFPU for floating point support")
                  1. 4

                    Any tool that provides cryptographic primitives that can be used in other projects. I wouldn’t trust myself to roll my own crypto nor should I.

                    1. 5

                      You’re basically asking for large libraries accessible through Maven Central, aren’t you? There are many. Lucene, rxjava and libphonenumber are very different examples of libraries that provide stuff you can’t or won’t implement yourself. They’re very different, Lucene gives you a lot of code to do something (a classic library), rxjava gives you basically an additional way to organise your code, and libphonenumber the result of thousands of hours of mindnumbing drudgery collecting data about phone numbers and number formats around the world… if you need that.

                      Oktaplanner, AWS, the list goes on. IMO Maven Central (and the other, similar reposotories of open source, even npm) have changed the way software can be developed. There are so many libraries to do all kinds of things.

                      It has disadvantages, but on the whole I love it. At one customer I wrote a <300loc module that did both language analysis, link/site retrieval, HTML parsing and some analysis, talked to a database and also to a remote service. Less than 300 lines. This is not the world of 1989, 1999 or even 20009.

                      1. 3

                        Not really a “tool”, but if I had to pinpoint the single most important skill that I learnt with computers: TOUCH TYPING REALLY FAST. Everybody can learn it with a moderate effort (a couple of weeks), and once you have this skill you attain a higher level of enlightenment. It’s like sight reading for serious musicians: just as important, but definitely easier to learn!

                        1. 2

                          http + html/(any human-readable structured data format, e.g. json, xml, yaml), space age (now passed) technology, so ubiquitous that it’s considered boring.

                          Surviving artefacts from an explosion of networking technologies and protocols.

                          Send text over addressed, single-use, sockets is a winner, primitive though it is. It is the horse drawn carriage (wheels, axels, horseshoes, government cobbles) of our age.

                          Hundreds if not thousands of libraries to help you do it… I like language-hosted middleware for the server (rack/ring/undertow) and almost any old thing for the client… The fact that it seems simple, like getting a post-card from a far away land, is a testament to it’s greatness.

                          1. 1

                            ANTLR (ANother Tool for Language Recognition)

                            1. 1

                              I think close to your examples are OS as library frameworks. One of the earlier (but now unmaintained example) was OSKit. Which allowed you to write your own kernel in just a few lines of code. Some example kernels that do e.g. networking in a few lines of code:


                              I guess OSKit’s spiritual successors are unikernels, which allow you to convert a program into an OS, sometimes with minimal modification: