1. 8

    Finally going to start “Project Alexandria” which is a highfalutin code name for a plan to insert all my books into a Postgres database and supply a simple front end (with little or no JavaScript) to query it.

    1. 3

      Bonus points for giving your personal projects code names. Way cool. :)

      1. 1

        Decided to go with some shell scripting to pull the book info and ready it for insert. Learned about jq, IFS and process substitution today :)

        if [ -z "$1" ];
                echo -n "missing ISBN number!"
                exit 1
                # IFS == internal field separator
                # read attempts to read nbytes of data from the object referenced by
                # the descriptor fd into the buffer pointed to by buf
                # < <(process substitution)
                # feeds the output of a process (curl) into stdin
                # e.g., echo <(echo bar) creates a temp file descriptor
                IFS="|" read title authors published desc \
                < <(curl -s "https://www.googleapis.com/books/v1/volumes?q=isbn:$1&key=$key&maxResults=1" \
                | jq -r '.items | map([.volumeInfo.title,.volumeInfo.authors[0],.volumeInfo.publishedDate,.volumeInfo.description]| join("|")) | join("\n")')
                echo -n $title
                echo -n $authors
                echo -n $published
                echo -n $desc
        1. 1

          Metadata only, or content?

          1. 1

            Metadata only for now, but I would love to find a way to grab my underlines from the actual texts. That’s far further down the line.

        1. 9

          Seems to me there are at least three topics that get conflated into one discussion here.

          1. What is the world like? That is, can I faithfully represent the world inside a static type system? I think the answer is no, corbin seems to make a good case. (But I’m weak on CT)

          2. What should my code be like? That is, what types of code do I want to create and look at while I’m programming? This is what most of the discussion on larger forums like HN is about.

          3. What is the frickin’ actual problem I’m trying to solve? This is what jgt is getting to. I don’t care about anything else in the universe aside from “what is this thing here and what do I have to do to it?” The static-vs-dynamic type discussion seems to be around what sorts of superstructures you want around executing this solution. Many times we lose track of just actually solving the problem by getting caught up in #1 or #2

          We need incremental strong typing. That is, we need enough type control over our data in order to do one small thing and only that. Once we start modeling the universe or debating what makes good code we lose track of that.

          1. 5

            I’ve been both coding and consuming books and videos about how to be a good coder for many, many years.

            Lately, I’ve been teaching what I call “Good Enough Programming”. If the code does something useful, and you can walk away from it, it’s good enough. All of this 50 year code is like that.

            Oddly enough, nobody talks much about good enough programming. In fact, if you take a look around, it’s not the natural state of things. Projects go on forever and their software is always being tweaked, poked at, and updated. We’ve got a lot of coders wanting to be perfect, kick-ass programmers, but not a lot of coders writing good enough code. It’s very odd.

            1. 3

              I think the difference for software engineering is the trope of “changing the world”, which means hiring “animals” (Paul Graham’s essay: http://www.paulgraham.com/start.html) who pour in their heart and soul into somebody else’s property in a hope of attaining riches, which is held up by society as the goal to attain. Throw in the fact that the field is constantly 50% buzzed 20 year olds, and you get a culture of coders wanting to be perfect.

              Kudos to you for fighting the good fight! Code that puts food on the table today and tomorrow is good enough and should allow you to sleep well at night.

            1. 2
              • Escape Room with the adult children and their SOs. Yay! It’ll be our first one.
              • Finish reading “The Metaphysical Club” which, so far, is freaking awesome. Lots of foundational and history stuff here that supports my Info-Ops series. I love history, philosophy, and science, plus it won the Pulitzer
              • Hoping to play some pick-up card games with the family this evening if they decide to hang around
              • A little bit of light coding. I have some pure functions scatter around in my F# projects that I’d like to bring together and make a dotnet project template for console projects. But that might drift into early next week. We’ll see
              1. 1

                Dang it, I just shut down probably a dozen of my sites over the last year.

                When I start back up, however, it’s all going to be static and serverless. I’m done with honking around with servers and software when I should be focusing on content.

                The only thing I’m wondering is how to do a serverless wiki. I haven’t put much thought into it. Should be a fun puzzle to figure out later this year.

                1. 2

                  Why do you think a wiki is special?

                  I only wrote a wiki software once but if you leave out the interactive stuff, it’s still just a collection of markdown files linking to each other.

                  1. 1

                    Yup, it sure is.

                    Have you tried out the wikimedia software? That’s the free version of the code that runs wikipedia?

                    There’s a ton of other stuff aside from just files linking to one another. I thought it was kinda cool and liked all the bells and whistles.

                    If I ever get around to setting a new wiki up, I’ll need to figure out how much of that feeling was golly-gee and how much was this-is-something-I-actually-need.

                    To your point, markdown by itself will do a ton of heavy-lifting. There’s also a bunch of markdown add-ons that do even more.

                    1. 1

                      Ah well, ok - depends on the definition of wiki, of course :)

                1. 6

                  We learn through pressure, stress. People who learn a lot tend to exert self-pressure on themselves for various reasons, sometimes good, sometimes bad. But there’s always some form of stress associated with learning, whether it’s a baby trying to communicate that it’s hungry or a neurosurgeon trying to perfect a new technique so he will be respected by his peers.

                  So really in any career there’s only two questions. One, are you continuing to learn new things? Two, if so, what and why?

                  We have computer science figured out. That is, programming in a computer language is a skill you can learn and walk away from, using it the rest of your life. We do not have making technology for others figured out at all. What makes a good team, what makes a good working environment, what kinds of things work well under which conditions and what do not. That means you can be continually learning these things for the rest of your life.

                  Since it’s unsolved, it also means there are no right or wrong things to learn.

                  Having a life should come first, above working, above coding, above all else. But having a life should have some form of learning involved with it. People who are professionally curious tend to continue to learn in various fashions and places, using what they learn in trying new things at work. If you’ve got 14 side hustles and are putting in 60 hours at work a week simply because it’s what you think you should be doing, you’re probably more part of the problem than part of the solution. If, on the other hand, you’re in the same spot for five years doing the same things and haven’t grown, you’re also part of the problem. It’s more complicated than that.

                  1. 3

                    I set up my first pi-hole about a year ago. I loved it. Then, about four months ago, I accidentally hosed it up doing some other things not related to the pi-hole code.

                    I went several months without one. Wow. Even with max ad-blocking on my PCs, I noticed a difference. So this past weekend I wiped the drive and reinstalled. Yay!

                    My only remaining question: I also have a new home network situation where I’m using an ISP-provided modem to do all the DHCP work. I want to take over that in the pi, but it might be a hassle having to call the ISP. Also I am not overly impressed with the UI for the DHCP server in the pi-hole.

                    1. 4

                      Just had a great conversation with my friend Michael. He and I are both interested in this technical topic we haven’t seen covered in the literature around programming. I think we might start setting up an example project to walk through some scenarios, maybe make a few videos later on. Should be a hoot.

                      1. 5

                        I’m curious. Are you intentionally not mentioning what the technical topic is?

                        1. 2

                          Yes. I’m not trying to be coy, but I want to make sure we frame it up well. To me we were talking about something obvious that didn’t need much explanation. I’ve found that I suck when I try to describe things like this since there are so many things I take for granted that others might not. So I’m waiting on jumping in and getting feedback in order to have some sort of catchy/explanatory title.

                      1. 7

                        I’ve spent some time trying to find the magic words or slogan to help companies with this problem. I think they see it, they just are unable to deal with it.

                        Here’s the best I’ve got so far:

                        Good companies approach a problem, they thrash, they organize, they codify, and then they automate. Bad companies approach a problem, thrash a bit, then assign it to a person.

                        It’s a good way to end up with a lot of people, all personally responsible for “impossible” problems.

                        1. 3

                          Fear is a good form of persuasion.. I wonder if there’s a way to frame it so that everyone that can cause “social” problems (many levels to decision makers, lots of ex-engineers on a project, lots of engineers in general, …) somehow are afraid to do so.

                          1. 9

                            Much of unneeded organizational complexity is driven by fear/risk-mitigation motivations. Things are just too broadly-applied and there’s no trade-offs being discussed. At the policy level, you easily grok if we made a list, process, or committee for some risk, the risk goes away. What you don’t grok is how that impacts everybody else. And to your point, the tendency is for nobody to tell you. A 100 people all adding 5 minutes to a dev’s time each week might all have great reasons to do so and the risks are dire. But these things add up. Who wants to go fight a battle to save 5 minutes? Even if you win you still lose.

                        1. 16

                          I am creating a website with a map of all the public pianos in the world 🎹. Happily learning a lot of leaflet, postgres and jquery in the process.

                          1. 2

                            This might be a fun little app, “Play the Piano for Me”, where people visit public pianos and take pics playing them.

                            1. 3

                              Actually, some of the data comes from Instagram where people do exactly that! The data source is been the trickiest part, there are lots of places on the internet where people post about public pianos but there is no unified source so the information is extremely fragmented. I’m attempting to unify it but, based on the previous attempts, it seems unlikely I’ll be successful. Sometimes you have to try!

                            2. 2

                              What a great idea!

                            1. 19

                              “..Using computers is a public, socal act…”

                              I don’t want to dive into a critique of this essay. I’m happy the author is working through these issue, if only from a rather odd angle (software freedom).

                              But I do think there’s an important point here about humans, not computers. Humans are tool-using and social creatures. People say things like this all of the time. I don’t thnk we realize the depth of the point.

                              If I want to join two pieces of wood, I gather the wood, I gather the nails and hammer, and I hammer the wood together. It is quite clear, both to me and any outsider passing by, what I am doing. People can offer to lend a hand. They might speculate on my style, wonder what other work I was planning. If I were working in a construction team, my boss could see what I was doing, the style and technique I masted. Perhaps he might offer pointers or demonstrate a better way of doing tings.

                              As tech guys, we’ve focused so much on the supposed goal of the activity, joining wood together, that we’ve convinced ourselves that as long as that goal is reached, we’re done. Likewise nobody ever sits down and thinks about all of that social crap when they us a hammer. They just want wood joined.

                              I read the other day that the Navy is moving away from glass command centers for ships. Good for them. Can you imagine the deck of a battleship in the middle of a war where everybody was contrrolling things from their smart phones? What a nightmare! How are things going? How’s the ship and crew respondning. Beats me. All I can see is a bunch of folks poking at little plastic boxes. There’s no observation, training, or team interaction there. Just somebody holding a box that might do anything … or nothing. They could all be playing Donkey Kong. Who knows?

                              More to the point, when you confront many of us tech people with this, instead of admitting the problem, they just double-down. Well, you want to know how the ship is doing? Just give us the metrics! We’ll make dashboards! Woot! But I don’t know the metrics. There are no metrics. Human society is more than just bits flying around. I need to manage and interact with people, not stare at updating graphics. The real world doesn’t work lke a Sci-fi movie.

                              Then we get into the whole “buy a general-purpose computer for one reason, end up using it for something else” problem, but that’s for another day. Our denial of the problem is probably a much bigger problem than the problem itself.

                              1. 6

                                I don’t want to dive into a critique of this essay. I’m happy the author is working through these issue, if only from a rather odd angle (software freedom).

                                I wrote this mess (as indicated by the authored below the title). If you do get around to it, I’d be very curious to hear your criticism, setting aside my inability to write a normal text. As I point out a few times, I’m very sceptical about the idea. The thought experiment is just a vehicle to think about these concepts.

                              1. 1

                                Related: does anybody know if there’s a programming/mathematical model of the solar system? I’d like to do a rocket simulator, but it won’t really shine unless there are multiple planetary bodies in it.

                                1. 2

                                  This maybe?

                                  Consider that allowing for n-body physics will introduce some chores into the game (e.g., station keeping) that might not be fun to deal with after awhile and some unexpected behavior that might be hard for new players to get (e.g., weird trajectories).

                                  I would rather have a good simulation of the atmosphere than n-body gravity. That would give you more bang for your bucks (realistic drag, planes, helicopters, etc).

                                  1. 1

                                    That’s a good point, but even just the earth-moon system would be enough to do some interesting orbital stuff, like lunar insertion orbit.

                                    1. 2

                                      I’m no astrophysicist but I think that if you just want to model a system with just the earth and the moon you don’t need a very complex system. You could just model the gravity well for each body and “jump” between spheres of influence when appropriate. That’s how kerbal space program models gravity.

                                      Things get complicated when you want to calculate the effects of earth, the moon, and the sun on the spacecraft. That will require an n-body physics engine.

                                1. 9

                                  I find the “if you can’t explain it to a child” analogy more and more problematic the more I think about it. The question here is, explain for what purpose? The most dangerous thing is not lack of understanding, it’s illusion of understanding. You can easily explain the basic idea behind RSA to a novice (even a literal child). Explaining it in enough detail to implement it correctly is a different story.

                                  Until you really understand something, you can’t know what is irrelevant and can be ignored. You can miss things that turn out very relevant, and when it comes to applications exposed to the Internet, consequences can be quite serious.

                                  1. 2

                                    Often the more we internalize knowledge and behaviors the less we can verbalize them.

                                    1. 1

                                      This is a good point which I agree with and expand on in the article.

                                      I think the “explain to a child” is widely misunderstood. It’s not that everything is simple. It’s being self-aware enough and being able to understand a random kid enough to simplify into common simple terms along maybe a million different axes, depending on the situation and the kid. That requires a much deeper understanding than simply being able to do something by rote.

                                      1. 2

                                        Explain to a random kid or a bright kid? IMO ‘explain to a kid’ is just meant to mean you could explain it from first principles, the ‘kid’ representing someone with no existing background/domain knowledge.

                                        I don’t think it means you should be able to explain it to literally anyone and have them understand it.

                                        1. 2

                                          I guess a correct version should be “if you can’t teach it to a novice given enough time, you don’t really understand it”. For many things, a short explanation is going to be useless even if it’s not lying by omission. One can easily explain basics of calculus to an actual child. A derivative shows how fast a function is growing, like with speed and acceleration. Now what? It doesn’t give one nearly enough information to start using it. Maybe it gives some motivation to study it further, but no more than that.

                                          As of verbalizing, I have a feeling that verbalizing may not really be enough. At this moment I can tell the formal definition of a limit offhand, not even because I remember the words, but because it’s obvious how to say “as the item number grows, the item gets closer to a certain value” with quantifiers. However, I remember it well that when I first encountered that definition in high school times, it was absolutely confusing to me. What has changed? Did I get more comfortable with quantifiers? Or better at following definitions with multiple levels of recursion? No idea. My attempts to explain things to novices are routinely met with a blank stare no matter what analogies I employ, until they actively try to understand it themselves and explore counterexamples and implications.

                                      1. 2

                                        Finishing up an essay on what it means to be a full-stack developer. I’m not crazy about the phrase, but I think it’s important for developers to embrace the entire work, whether they want to specialize in one small aspect of it or not.

                                        1. 40
                                          1. The dependencies of a system are part of that system and cannot be treated as a black box. The maintenance of the entire system is your responsibility.

                                          …and then the first item on the list is “package manager”. As I understand it, the whole point of a package manager is to treat dependencies as a black box, so I can say “install GIMP” (or whatever) and all the required dependencies magically appear without my having to understand them.

                                          Simplicity is definitely a noble goal, but I’ve seen a bunch of simplicity manifestos over the years, and they usually boil down to “all the things I’m comfortable with and none of the things I don’t care about”, and that’s not really simple, that’s just convenient. Real simplicity hurts to use, like Forth, or the Lambda Calculus, which is why we invented all these complex insulating layers like package managers and interactive shells and alphanumeric keyboards to begin with.

                                          1. 4

                                            I’ve seen a bunch of simplicity manifestos over the years, and they usually boil down to “all the things I’m comfortable with and none of the things I don’t care about”

                                            That’s a good point, and it’s spot-on. I was interested in noting that the author starts off saying that simple software breaks in simple ways. Complex software breaks in compex ways.

                                            I think that might be getting closer to the truth: it’s not about the tools, it’s about your interaction with the technology over time. If you could create a solution gluing wires on a breadboard, as long as you never came back to it, would it matter what tools you had used?

                                            We may be looking at the wrong end of the stick here. We keep looking at languages, platforms, and tools in terms of features and attributes. Their real value (or not) lies in how they interact with humans over the full development lifecycle.

                                            1. 4

                                              Forth hurts to use

                                              And we love the pain.

                                              1. 3

                                                It doesn’t say the installation of your system is your responsibility, but the maintenance of your system. That means that, when the package breaks, you need to be prepared to find out why and fix it. The package manager can resolve and manage dependencies for you, so long as you’re aware of what the dependencies are and acknowledge that they can come along.

                                              1. 7

                                                I love this. Very cool idea and site.

                                                This is respectful of a lot of things: bandwidth for the user, consideration of unnecessary user distractions, disregard of playing content to the metrics, etc. It also shows a desire to compare and contrast what we need from the web as opposed to what we get. If you spend your time trying only to do the minimum necessary to tell me what you think is important without any other “engagement” activities, I’ll return the favor and spend the time trying to consume what you’re creating. It shows that you value our relationship.

                                                1. 18

                                                  Further to this point. Strive to design your data structures so that ideally there is only one way to represent each value. That means for example NOT storing your datetimes as strings. This will imply that your parsing step also has a normalization step. In fact storing anything important as a string is a code smell.

                                                  1. 9

                                                    A person’s name should be stored as a string. City names. Stock symbols. Lots of things are best stored as strings.

                                                    1. 3

                                                      Names of things are best stored as strings, yes.

                                                      1. 1

                                                        What I recommend though is to encode them in their own named string types, to prevent using the strings in ways that they are not meant to be used. We often use that to encode ID references to things which should be mostly “opaque BLOBs”.

                                                      2. 3

                                                        Should stock symbols contain emojis or newlines, or be 100 characters long? Probably not. I assume there are standards for what a stock symbol can be. If you have a StockSymbol type constructed by a parser that disallows these things, you can catch errors earlier. Of course then the question is what do you do when the validation fails, but it does force a decision about what to do when you get garbage, and once you have a StockSymbol you can render it in the UI with confidence that it will fit.

                                                      3. 3

                                                        Interesting. If you don’t mind, I’d like to poke at that a bit.

                                                        Why should you care what anything is stored as? In fact, why should you expect the rest of the universe, including the persistence mechanism, to maintain anything at all about your particular type system for your application?

                                                        1. 7

                                                          They can maintain it in their own type system. The issue is information loss. A string can contain nearly anything and thus I know nearly nothing about it and must bear a heavy burden learning (parsing or validating). A datetime object can contain many fewer things and thus I know quite a lot about it lowering the relearning burden.

                                                          You can also build and maintain “tight” connections between systems where information is not lost. This requires owning and controlling both ends. But generally these tight connections are hard to maintain because you need some system which validates the logic of the connection and lives “above” each system being connected.

                                                          Some people use a typed language with code generation, for instance.

                                                          1. 3

                                                            @zxtx told the reader trying to leverage type systems to design their own program a certain way that derives more benefit from type systems. The rest of the universe can still do their own thing.

                                                            I’m not sure the string recommendation is correct. There’s several languages that are heavily based on strings powering all kinds of things out there successfully. I’ve also seen formally-verified implementations of string functionality. zxtx’s advice does sound like a good default.

                                                            We probably should have, in addition to it, verified libraries for strings and common conversions. Then, contracts and/or types to ensure calling code uses them correctly. Then, developers can use either option safely.

                                                            1. 3

                                                              Sure some things inevitably have to be strings: personal names, addresses, song titles. But if you are doing part-of-speech tagging or word tokenization, an enumerative type is a way better choice than string. As a fairly active awk user I definitely sympathize with the power of string-y languages, but I think people new to typed languages overuse rather than underuse strings.

                                                              1. 2

                                                                Unfortunately, even folks who have used typed languages for years (or decades) still overuse strings. I’m guilty of this.

                                                            2. 3

                                                              I admit to going back and forth on this subject….

                                                              As soon as you store a person name as a PersonName object…… it’s no longer a POD and you’re constricted to a tiny tiny subset of operations on it…. (With the usual backdoor of providing a toString method)

                                                              On the other hand Bjarne Stoustrup’s assertion that if you have a class invariant to enforce… that’s the job of an object / type.

                                                              Rich Hickey the clojure guy has an interesting talk exactly on this subject with an interesting different take….

                                                              Instead of hiding the data in a type with an utter poverty of operators, leave everything as a pod of complex structure which can be validated and specified checked and asserted on using a clojure spec.

                                                              ie. If you want something with a specific shape, you have the spec to rely on, if you want to treat it as ye olde list or array of string….. go ahead.

                                                              1. 11

                                                                I stuck to simple examples of the technique in my blog post to be as accessible as possible and to communicate the ideas in the purest possible way, but there are many slightly more advanced techniques that allow you to do the kind of thing you’re describing, but with static (rather than dynamic) guarantees. For some examples, I’d highly recommend taking a look at the Ghosts of Departed Proofs paper cited in the conclusion, since it addresses many of your concerns.

                                                                1. 1

                                                                  Ok. That took me awhile to digest…. but was worth it. Thanks.

                                                                  For C++/D speakers it’s worth looking at this first to get the idea of phantom types…


                                                                2. 4

                                                                  As someone who worked professionally with both Clojure (before spec but with Prismatic Schema) and OCaml and I have to say I utterly prefer to encode invariants in a custom type with only a few operations instead of the Clojure way of having everything in a hashmap with some kind of structure (hopefully) and lots of operations which operate on them.

                                                                  My main issue writing Clojure was that I did apply some of these (really useful and versatile) functions on my data, but the data didn’t really match what I had expected so the results were somewhat surprising in edge cases and I had to spend a lot of brain time to figure out what was wrong and how and where that wrong data came to be.

                                                                  In OCaml I rarely have the problem and if I want to use common functions, I can base my data structures on existing data structures that provide the functions I want to over the types I need, so in practice not being able to use e.g. merge-with on any two pieces of data is not that painful. For some boilerplate, deriving provides an acceptable compromise between verbosity and safety.

                                                                  I can in theory do a similar thing in Clojure as well, but then I would need to add validation basically everywhere which makes everything rather verbose.

                                                                  1. 3

                                                                    I’ve used Clojure for 8 years or so, and have recently been very happy with Kotlin, which supports sealed types that you can case-match on, and with very little boilerplate—but also embraces immutability, like Clojure.

                                                                    With Clojure, I really miss static analysis, and it’s a tough tradeoff with the lovely parts (such as the extremely short development cycle time.)

                                                                  2. 3

                                                                    The ability to “taint” existing types is the answer we need for this. Not a decorator / facade sort of thing, just a taint/blessing that exists only within the type system, with a specific gatekeeper being where the validation is done and the taint removed/blessing applied.

                                                                    1. 2

                                                                      In Go, wrapping a string in a new type is zero-overhead, and you can cast it back easily. So it’s mostly just a speedbump to make sure that if you do something unsafe, you’re doing it on purpose and it will be seen in code review. If the type doesn’t have very many operators, you might have more type casts that need to be checked for safety, but it’s usually pretty easy to add a method.

                                                                      On the other hand, the Go designers decided not to validate the string type, instead accepting arbitrary binary data with it only being convention that it’s usually UTF8. This bothers some people. But where it’s important, you could still do Unicode validation and create a new type if you want, and at that point there’s probably other validation you should be doing too.

                                                                      1. 1

                                                                        The last one is the best.

                                                                        Instead of scaling out code, we should be scaling out tests. We’re doing it backwards.

                                                                        I’ve been meaning to put together a conference proposal on this but haven’t gotten around to it. It’s the kind of thing that blows people’s minds.

                                                                        1. 1

                                                                          Can you expand a little on this? Sounds interesting.

                                                                          1. 3

                                                                            People don’t understand what tests do. If you ask them, they might say they help your code be less buggy, or they show your business customers that your program does what they’re paying for.

                                                                            That’s all true, but horribly incomplete. Tests resolve language.

                                                                            That is, whether it’s science, programming, running a business, or any of hundreds of other areas where human language intersects science, tests are the only tools for determining what’s true or not in unambiguous terms. Come up with some super cool new way of making a superconductor? Great! Let’s have somebody go out and make it on their own, perform a test. If the test passes, you’re on to something. Yay! If the tests fails? Either you’re mistaken or the language and terms you’re using to describe your new process has holes the reproducer was unable to resolve. Either way, that’s important information. It’s also information you wouldn’t have gained otherwise without a test.

                                                                            In coding, as I mentioned above, we have two levels of tests. The unit level, which asks “Is this code working the way I expected it to?” and the acceptance level, which asks “Is the program overall performing as it should?” (I understand the testing pyramid, I am simplifying for purposes of making a terse point). But there are all sorts of other activities we do in which the tests are not visible. Once the app is deployed, does it make a profit? Is your team working the best way it can? Are you building this app the best way you should? Are you wasting time on non-critical activities? Will this work with other, unknown apps in the future? And so on.

                                                                            We’ve quantitized some of this with things like integration testing (which only works with existing apps). Frankly, we’ve made up other stuff out of whole cloth, just so we can have a test, something to measure. In most all cases, when we make stuff up we end up actually increasing friction and decreasing productivity, just the opposite of what we want.

                                                                            So how do we know if we’re doing the best job we can? Only through tests, whether hidden or visible. How are we doing at creating tests? I’d argue pretty sucky. How can we do tests better? More to the point, if we do tests correctly, doesn’t it make whatever language, platform, or technology we use a seconary-effect as opposed to a primary one? We spend so much time and effort talking about tools in this biz when nobody can agree on whether we’re doing the work right. I submit that this happens because we’re focusing far, far too much on our reactions to the problem than the problems themselves. If we can create and deploy tests in a comprehensive and tech-independent manner, we can then truly begin discussing how to take this work to the next level. Either that or we’re going to spend the next 50 years talking about various versions of hammers instead of how to build safe, affordable, and desirable houses, which is what we should be doing.

                                                                            There’s a lot missing in my reply, but once we accept that our test game sucks? Then a larger and better conversation can happen.

                                                                            1. 1

                                                                              It will take me some time to digest this properly… it’s a completely different angle to which I usually approach the matter. (I’m not saying you’re wrong, I’m just saying you coming at it from such a different angle I’m going to have to step back and contemplate.)

                                                                              To understand where I’m coming from let me add…

                                                                              I regard tests as a lazy pragmatic “good enough” alternative to program proving.

                                                                              If we were excellent mathematicians, we would prove our programs were correct exactly the way mathematicians prove theorems.

                                                                              Except we have a massive shortage of that grade of mathematicians, so what can we do?

                                                                              Design by Contract and testing.

                                                                              DbC takes the raw concepts of program proving (pre-conditions and post conditions and invariants) and then we use the tests to setup the preconditions.

                                                                              Writing complete accurate postconditions is hard, about as hard as writing the software, so we have a “useful subset” of postconditions for particular instance of the inputs.

                                                                              Crude, very crude, but fairly effective in practice.

                                                                              My other view of unit tests is closer to yours…

                                                                              They are our executable documentation (proven correct and current) of how to use our software and what it does. So a design principle for tests is they should be Good, Readable understandable documentation.

                                                                              Now I will shutup and contemplate for a day or two.

                                                                              1. 1

                                                                                We are saying the same thing. Heck we might even be agreeing. We’re just starting from completely opposite sides of the problem. Formally validating a program proves that it matches the specification. In this case the formal specification is the test.

                                                                                I think when I mention tests you may be thinking of testing as it was done in the IT industry, either manual or automated. But I mean the term in the generic sense. We all test, all the time.

                                                                                What I realized was that you can’t write a line of code without a test. The vast majority of times that test is in your head. Works for me. You say to yourself “How am I going to do X?” then you write some code in. You look at the code. It appears to do X. Life is good.

                                                                                So you never get away from tests. The only real questions are what kinds of tests, where do they live, who creates them, and so forth. I’m not providing any answers to these questions. My point is that once you realize you don’t create structure without some kind of tests somewhere, even if only in your head, you start wondering exactly which tests are being used to create which things.

                                                                                My thesis is that if we were as good at creating tests as we were at creating code, the coding wouldn’t matter. Once again, just like I don’t care whether you’re an OCAML person or a Javascript person, for purposes of this comment I don’t care if your tests are based on a conversation at a local bar or written in stone. That’s not the important part. The thing is that in various situations, all of these things we talk about doing with code, we should be doing with tests. If the tests are going to run anyway, and the tests have to pass for the project to be complete or problem solved, then it’s far more important to talk about the meaning of a successful completion to the project or a solution to the problem than it is to talk about how to get there.

                                                                                Let’s picture two programmers. Both of them have to create the world’s first accounting program. Programmer A sits down with his tool of choice and begins slinging out code. Surely enough, in a short time voila! People are happy. Programmer B spends the same amount of time creating tests that describe a successful solution to the problem. He has nothing to show for it.

                                                                                But now let’s move to the next day. Programmer A is just now beginning to learn about all of the things he missed when he was solving the problem. He’s learning that for a variety of reasons, many of which involve the fact that we don’t understand something until we attempt to codify it. He begins fixing stuff. Programmer B, on the other hand, does nothing. He can code or he can hire a thousand programmers. The tech details do not matter.

                                                                                Programmer B, of course, will learn too, but he will learn by changing his tests. Programmer A will learn inside his own head. From there he has a mental test. He writes code. It is fixed. Hopefully. Programmer A keeps adjusting his internal mental model, then making his code fit the model, until the tests pass, ie nobody complains. Programmer B keeps adjusting an external model, doing the same thing.

                                                                                Which of these scale when we hire more coders? Which are these are programs the programmer can walk away from? Formal verification shows that the model meets the spec. What I’m talking about is how the spec is created, the human process. That involves managing tests, in your head, on paper, in code, wherever. The point here is that if you do a better, quicker job of firming the language up into a spec, the tech stuff downstream from that becomes less of an issue. In fact, now we can start asking and answering questions about which coding technologies might or might not be good for various chores.

                                                                                I probably did a poor job of that. Sorry. There’s a reason various programming technologies are better or worse at various tasks. Without the clarification tests provide, discussions on their relative merits lack a common system of understanding.

                                                                            2. 1

                                                                              ADD: I’ll add that most all of the conversations we’re having around tech tools are actually conversations we should be having about tests: can they scale, can they run anywhere, can they be deployed in modules, can we easily create and consume stand-alone units, are they easy-to-use, does it do only what it’s supposed to do and nothing else, is it really needed, is it difficult to make mistakes, and so on. Testing strikes me as being in the same place today as coding was in the early-to-mid 80s when OO first started becoming popular. We’re just not beginning to think about the right questions, but nowhere near coming up with answers.

                                                                              1. 1

                                                                                Hmm… In some ways we hit “Peak Testing” a few years back when we had superb team of manual testers, well trained, excellent processes, excellent documentation.

                                                                                If you got a bug report it had all the details you needed to reproduce it, configs, what the behaviour that was expected, what was the behaviour found, everything. You just sat down and started fixing.

                                                                                Then test automation became The Big Thing and we hit something of a Nadir in test evolution which we are slowly climbing out of…

                                                                                This is how it was in the darkest of days…

                                                                                “There’s a bug in your software.”

                                                                                Ok, fine, I’ll fix how do I reproduce…..

                                                                                “It killed everything on the racks, you’ll have to visit each device and manually rollback.”

                                                                                (Shit) Ok, so what is the bug?

                                                                                “A test on Jenkins failed.”

                                                                                Ok, can I have a link please?

                                                                                “Follow from the dashboard”

                                                                                What is this test trying to test exactly?

                                                                                “Don’t know, somebody sometime ago thought it a good idea”.

                                                                                Umm, how do I reproduce this?

                                                                                “You need a rack room full of equipment, a couple of cloud servers and several gigabytes of python modules mostly unrelated to anything”.

                                                                                I see. Can I have a debug connector to the failing device.


                                                                                Oh dear. Anyway, I can’t seem to reproduce it… how often does it occur?

                                                                                “Oh we run a random button pusher all weekend and it fails once.”

                                                                                Umm, what was it doing when it failed?

                                                                                “Here is a several gigabyte log file.”

                                                                                Hmm. Wait a bit, if I my close reading of these logs are correct, the previous test case killed it, and the only the next test case noticed…. I’ve been looking at the wrong test case and logs for days.

                                                                        2. 1

                                                                          Because throughout your program you will need to do comparisons or equality checks and if you aren’t normalizing, that normalization needs to happen at every point you do some comparison or equality check. Inevitably, you will forget to do this normalization and hard to debug errors will get introduced into the codebase.

                                                                          1. 1

                                                                            Ok. Thank you. I figured out what my hang up was. You first say “Strive to design your data structures so that ideally there is only one way to represent each value.” which I was completely agreeing with. Then you said “In fact storing anything important as a string is a code smell” which made me do a WTF. The assumption here is that you have one and only one persistent data structure for any type of data. In a pure functional environment, what I do with a customer list in one situation might be completely different from what I would do with it in another, and I associate any constraints I would put on the type to be much more related to what I want to do with the data than to my internal model of how the data would be used everywhere. I really don’t have a model of how the universe operates with “customer”. Seen too many different customer classes in the same problem domain written in all kinds of ways. What I want is a parsed, strongly-typed customer class right now to do this one thing.

                                                                            See JohnCarter’s comment above. It’s a thorny problem and there are many ways of looking at it.

                                                                            1. 1

                                                                              I think ideally you still do want a single source of truth. If you have multiple data structures storing customer data you have to keep them synced up somehow. But these single sources of data are cumbersome to work with. I think in practice the way this manifests in my code is that I will have multiple data structures for the same data, but total functions between them.

                                                                              1. 2

                                                                                Worked with a guy once where we were going to make a domain model for an agency. “No problem!” he said, “They’ve made a master domain model for everything!”

                                                                                This was an unmitigated disaster. The reason was that it was confusing a people process (determining what was valid for various concepts in various contexts) with two technical processes (programming and data storage) All three of these evolved dramatically over time, and even if you could freeze the ideas, any three people probably wouldn’t agree on the answers.

                                                                                I’m not saying there shouldn’t be a single source of data. There should be. There should even be a single source of truth. My point is that this single point of truth is the code that evaluates the data to perform some certain action. This is because when you’re coding that action, you’ll have the right people there to answer the questions. Should some of that percolate up into relational models and database constraints? Sure, if you want them to. But then what do you do if you get bad incoming data? Suppose I only get a customer with first name, last name, and email? Most everybody in the org will tell you that it’s invalid. Except for the marketing people. To them all they need is email.

                                                                                Now you may say but that’s not really a customer, that’s a marketing lead, and you’d be correct. But once again, you’re making the assumption that you can somehow look over the entire problem space and know everything there is to know. Do the mail marketing guys think of that as a lead? No. How would you know that? It turns out that for anything but a suite of apps you entirely control and a business you own, you’re always wrong. There’s always an impedance mismatch.

                                                                                So it is fine however people want to model and code their stuff. Make a single place for data. But the only way to validate any bit of data is when you’re trying to use it for something, so the sole source of truth has to be in the type code that parses the data going into the function that you’re writing – and that type, that parsing, and that function are forever joined. (by the people and business that get value from that function)

                                                                                1. 2

                                                                                  I suspect we might be talking a bit past each other. To use your example, I might ask what it means to be a customer. It might require purchasing something or having a payment method associated with them.

                                                                                  I would in this case have a data type for Lead that is only email address, a unique uuid and optionally a name. Elsewhere there is code that turns a Lead into a Customer. The idea being to not keep running validation logic beyond when it is necessary. This might mean having data types Status = Active | Inactive | Suspended which needs to be pulled from external data regularly. I can imagine hundreds of different data types used for all the different ways you might interact with a customer, many instances of these data types created likely right before they are used.

                                                                        3. 1

                                                                          Mostly agree, but I’d like to add that the ability to pass along information from one part of the system to another should not necessarily require understanding that information from the middle-man perspective. Often this takes the form of implicit ambient information such as a threadlocal or “context” system that’s implemented in library code, but as a language feature it could be made first-class.

                                                                        1. 13

                                                                          First, let’s stipulate that lots of people don’t know what they’re doing or why in technology development. Unless you’re coding, it doesn’t really feel as if something is going on, i.e., it’s a waste of time. In most cases, it is. Your instincts are correct.

                                                                          Second, I did not finish reading the article. I did not finish because the author doesn’t know what they’re talking about, or rather he is responding to the way he’s been taught and observed stand-ups happening, not how they actually work.

                                                                          ”…The majority of meetings are a waste of time. And in my opinion, one flavor of meeting that tops the charts in uselessness is the “status update” meeting. You know this meeting— the meeting where everyone gets together to share what they’ve been doing…”

                                                                          Yes. Shoot for meeting-free work areas. The way you do that is dynamically get together and talk about stuff as needed. The way you do that? A stand-ups. Stand-ups are the (in my mind) only fixed time and place where everybody gets together and asks for help. You don’t give status, you don’t report to anybody, you don’t ask or answer questions unless you can’t make out what the person is saying. You just take a minute and verbally review where you are and ask for help if you need it. Then we get to work. Maybe somebody has a problem that is going to be huge, so we hang out for an hour or two trying to solve it. Maybe nothing’s going on. Great! Two minutes later we’re all working on our own stuff. Works either way. It’s dynamic.

                                                                          The problem here is 1) people are used to status reports so that’s what they gravitate towards doing, 2) there’s pressure to build-up your work and deny having any problems, and 3) when it’s working right there is no positive feedback. It’s like brushing your teeth. You do it, it’s quick, and if you’re doing it right you never think about it. In fact, the more you think about it, probably the worse you’re doing it.

                                                                          1. 4

                                                                            In my experience, unless time limits for people talking are heavily enforced, they are largely a waste of time. If no one is willing/able to enforce hard limits for the amount of time any one person has in the stand up to talk, you almost always end up with senior people on the team who enjoy listening to themselves speak taking up 95% of the time, with the remaining 5% of the time given to those who need real help.

                                                                            1. 9

                                                                              Yeah they can go wrong in a lot of ways. Turns out having people talk to one another isn’t as simple as writing a for-next loop (grin).

                                                                              One of the fun things I used to do when teaching standups is a game where a team has to do a standup, but one person is the “ringer” – they’re given a dysfunction to display and the rest of the group has to deal with it. I found it was much easier to teach dysfunctions when people were only playing as opposed to directly addressing them.

                                                                              There’s also the guy that has to question everything, the one who never needs help, the one talking about nothing to do with work, and so on.

                                                                              1. 1

                                                                                That’s a fascinating approach. Thanks for sharing it.

                                                                          1. 7

                                                                            I decided I’ll try a new language for this year’s Advent of Code, so I’ll move away from my usual picks: Nim and Python. I chose OCaml, since I really like F# but I couldn’t make it as usable as I would like on Linux.

                                                                            For a November warm-up, I decided to use OCaml for AoC 2017 tasks, so I’ll be updating my existing AoC 2017 repo with OCaml solutions. These are my first steps in OCaml, so if anybody has any advice on how to improve my solutions, please let me know.

                                                                            1. 2

                                                                              Best of luck! I used OCAML to get into F# when it first came out.

                                                                              I guess the problem you have with F# is development in linux? I eventually settled on developing in windows and then moving the files over for deployment, but if I started today I might give VSCode in nix a chance. Interested in hearing how your experience goes.

                                                                              1. 2

                                                                                I guess the problem you have with F# is development in linux?

                                                                                Yes, I have tried it about a year ago and I didn’t like I needed Mono to make FSI work. But I’ve heard that .net core 3 should make FSI possible without Mono. Maybe I’ll give it another try in 2020.

                                                                                As far as OCaml goes, I find the experience superb. utop is such a nice REPL, and I could be productive using NeoVim in no time (merlin and ocp-indent are great).
                                                                                (But I still like the looks of F# programs more, it feels nicer and more elegant)

                                                                                1. 2

                                                                                  I managed to get utop running on my Windows machine once when trying to dabble in OCaml, and I had my mind blown. I hold it as the best repl I seen for now in my life. If Nim had such an amazing REPL, wow… one can dream ;)

                                                                                  1. 2

                                                                                    Yes, I have tried it about a year ago and I didn’t like I needed Mono to make FSI work.

                                                                                    What’s wrong with Mono?

                                                                                    1. 1

                                                                                      I dodged JVM and .NET in the past since they were large runtimes vs a memory-safe language with little to no runtime footprint. Maybe they shrunk them or have tools to since then.

                                                                                      What’s Mono currently like in terms of dependency size for the app? And how much is unsafe code?

                                                                                      1. 2

                                                                                        Disclaimer: I don’t have a constant eye, this is my understanding. Mono and Core are moving towards moving more into pure managed code or commodity libraries with ideally amortized security costs. Core tries to use modern C++ when possible for safety, Mono is considering rewriting some components in Rust (though that makes my thing much harder!).

                                                                                    2. 1

                                                                                      I did mono at first too. Great job and hard work those guys have put in.

                                                                                      I’m all .net core now, though. It just works everywhere.

                                                                                      I found that I eventually started thinking of F# apps as all being unix-style commands. This eliminated several entire categories of PITA that I would normally have to put up with, and I was still able to do whatever I wanted. I love all of the tooling and fameworks that are out there, but it’s quite easy to get “upside down” in a project and spend more time on the tools than you do the problem. YMMV.

                                                                                      1. 2

                                                                                        While we’re discussing F#, let me ask you: are there F# books, tutorials, or some other learning material which doesn’t assume you have previous C# (and .net in general) experience? Most of the things I found were along the line “how to rewrite your C# stuff in F#”.

                                                                                        1. 1

                                                                                          I don’t know. If that’s true, it must be a marketing thing. Perhaps the most lucrative audience are the current C# guys looking to try new stuff?

                                                                                          That leads me to the next question, coming from where? BASH-like script experience? Java? PHP? Any coding at all? Can you work an IDE, set up a project, etc.

                                                                                          30 years ago, when I taught myself C++, I started with the “How to program” books with just C, even though I already knew how to program. After a few of those, then I moved to the C++ books. Same with F#. I started with OCAML, but heck, there weren’t any other books out there back then. Surely somebody has a getting started with F# book, right?

                                                                                          For what it’s worth, I hang out in various F# forums from time-to-time. While I love the community, and I think F# and .net core are the go-to for most large solutions right now, I found that a lot of folks publish material that looks more like they’re trying to impress the other F# coders than teach anything. There’s a lot of pointy-headed “let’s do something that looks really cool using obscure language features people will be amazed at!” content going around. (Having said that, most all of them are also great for helping out new folks, as opposed to some of the other language communities) Some of them are also trying to reinvent MDD, but that’s a longer story.

                                                                                          Best of luck! Ping me if I can help any.

                                                                                    3. 2

                                                                                      Note that Windows support in OCaml has improved a lot in the last few years. OPAM works on Windows (https://fdopen.github.io/opam-repository-mingw/installation/). It still needs Cygwin to work (bundled with that OPAM installer), but compiled executables are pure Windows PEs.

                                                                                      There’s also an option to cross-compile to PE executables on Linux (https://github.com/ocaml-cross/opam-cross-windows), though the cross-repo is not very complete yet, but the point remains—you can develop for Windows without Windows.

                                                                                  1. 3

                                                                                    Writing: creating a diagram about scaling out groups of people, not just software projects. It’s part of a book I’m working on about learning at scale.

                                                                                    Family: hoping to spend some time imitating a vegetable and streaming with TW

                                                                                    Coding: I don’t know. Probably nothing.

                                                                                    Other: Selling a pickup truck, filling the hot tub for the season. I’d also like to get in some fall photography if there’s time.