1. 5

    Transactions for sled! I’ve been going through the literature and arriving at a concurrency control scheme loosely based on cicada but a little less complexity around timestamp synchronization, but giving it a clear path toward implementing less centralized timestamp allocation in the future if 100+ core machines become targets. It has been super interesting as a distributed systems person working on a main-memory database to see techniques from the distributed world being applied recently to reduce coordination costs of transaction techniques. I’ve been holding off on releasing sled as alpha because of a few known rare dataloss bugs, but I think it might be better to just roll forward and be clear that it’s got bugs still, which will be the case whether I know about them or not, because it’s still so young.

    1. 8

      I’m working on two projects in my spare time right now:

      -going through Type-Driven Development With Idris, to learn Idris for the dependent-types goodness

      -trying to teach myself the nuts and bolts of type theory by implementing Hindley-Milner type checking in the toy programming language interpreter I’ve been working on for a while. I’ve found a few resources about exactly how to go about doing this (most notably, this Haskell conference talk and this section from Stephen Diehl’s incomplete tutorial on implementing a Haskell-like functional programming language. I’ve actually had a bit of trouble translating the code from those resources, which is in Haskell and assumes a particular design for the AST data structures, to my own interpreter, which is in Rust and has a different AST design. If anyone is knowledgeable about actually implementing Hindley-Milner in a real programming language, I’d love to get some advice.

      1. 4

        I’ve got that book sitting in front of me right now! It’s so cool how you can specify protocols and valid state transitions so nicely in idris! I’m fantasizing about using it as the basis for a smart contract language.

        1. 4

          I tried going through that book, too, but I was not convinced by the examples it used. I recall one example which added the runtime length of an array to the typesystem. Nice, but you still needed do (runtime) checks for the length…

          1. 2

            You don’t always need to do runtime checks for the length. For example, if the length of a vector is n you can use a data type Fin n to represent an index. If you ask a user for an input, you’ll have to check it there - but once you have a Fin n you can pass it around your program without checking it. It’s always a valid index.

          2. 1

            If anyone is knowledgeable about actually implementing Hindley-Milner in a real programming language…

            I have experience with doing it in Haskell, but it seems like you’re implying that Haskell is not a real programming language…

            1. 1

              Haha far from it, I’ve written ostensibly-useful code in Haskell myself. There are a number of impedence mismatches between Haskell-style code and Rust-style code, that have made it hard for me to take the example Haskell code I’ve seen for Hindely-Milner and apply it to my own project.

          1. 9

            I foresake C as someone who works with it all the time. The author of this post makes the point that it’s worth knowing. I think that’s totally true if you interact with low-level systems.

            I definitely don’t buy the point about distributed systems usually requiring C because of performance reasons. As a distributed systems engineer, most of my peers work in memory safe languages, with a few things along data paths being in C, and every once in a while people may peer into the kernel to reason about an issue in the networking stack, but I’d imagine that most people who bill themselves as distributed systems engineers today are terrible at C and it probably doesn’t hurt them very much.

            When I foresake C I don’t advocate for its ignorance. I advocate for learning all you can about memory corruption, and being honest as a biased human who is building things for other biased humans and with other biased humans. C is a great language to use for learning about bugs and exploitation techniques. There is too much macho bullshit from prominent C engineers, and it catches on with incompetent engineers who make the world a more dangerous place.

            1. 4

              Many timeseries projects today seem to be borrowing techniques from Gorilla, particularly around compression, so it’s pretty relevant.

              1. 11

                Thank you for the wonderful comments last week.

                I wrote an Earley parser. And a Pratt parser. The Pratt parser is what I’ve been looking for all this time: a modular recursive descent parser. What it lacks in formalism it makes up with in brevity and power-to-weight.

                Now, I need to choose a host language. I’d like to pick Rust, but I’m not sure it has a ready-made GC solution right now, and I don’t want to go down that rabbit hole. That leaves C++, JVM, or OTP. Any thoughts?

                1. 3

                  What kind of language are you looking to interpret/execute? The three platforms you mention all have really different tradeoffs.

                  1. 3

                    A Lisp-esque language under the hood with a non-Lisp syntax on top. Idea is the functional paradigm can subsume the other two big paradigms (imperative/logic). Can use the CEK machine for proper tail call handling, so that isn’ta requirement of the host. Big thing I’m looking for is a GC (whether lib or built-in) and a language I like that I can target it with.

                  2. 2

                    For rust, you can wrap everything in a Rc, or if you have multiple threads an Arc, or if you want tracing GC you can use this, or if you just need epoch-style reclamation there’s crossbeam-epoch or if you just need hazard pointers there’s conc. I’ve had a lot of success with crossbeam-epoch in lock-free systems I’ve built.

                    1. 1

                      Rc (and friends) would need cycle detection, no? Maybe the thing to do is just use Rc and do research on cycle-detection algorithms to see if they are hard or not.

                      I looked at Epoch and hazard pointers and wasn’t sure if they were ok as a general GC. I need to do more reading. Thanks!

                      1. 2

                        Yeah, you can create memory leaks with Rc cycles in rust. But this is rarely an issue in most use cases. Rust memory can feel a little confusing at first, but cycles tend not to come up once you learn some different idioms for structuring things in non-cyclical ways.

                        For example, if you want to build a DAG, you can quickly implement it with a HashMap from ID to Node, where ID is some monotonic counter that you maintain. Each Node can contain Vec’s of incoming and outgoing edges. You can implement your own RC-like thing that tracks the sum of indegree and outdegree, and when it reaches 0, you just remove the Node out of the containing hashmap. For the cases where performance or concurrency concerns rule out this approach (which are rare and should not be pursued until this is measured to be a bottleneck) you can always write Rust like C with unsafe pointers, Box::into_raw, dereferencing inside unsafe blocks, and free’ing by calling Box::from_raw (actually calling drop() on that if you want to be explicit about what’s happening, but it will be dropped implicitly when it goes out of scope). Use mutexes on shared state until… basically always, but if you REALLY want to go lock-free, that’s when you can benefit from things like crossbeam-epoch to handle freeing of memory that has been detached from mutable shared state but may still be in use by another thread.

                        Feel free to shoot me an email if you’re curious about how something can be done in Rust! I know it can be overwhelming when you’re starting to build things in it, and I’m happy to help newcomers get past the things I banged my head against the wall for days trying to learn :)

                    2. 2

                      FWIW, many languages written in C or C++ use arenas to hold the nodes that result from parsing . For example, CPython uses this strategy. I’m pretty sure v8 does too. So you don’t manage each node individually, which is a large load on the memory allocator/garbage collector – you put them all in a big arena and then free them at once.

                      1. 2

                        Save the earth , use C++ or OTP

                        1. 1

                          You also have Go and .NET Core as possible host runtimes.

                          1. 1

                            What about Nim? It seems to be a memory-safe language with low-latency GC, macros, and produces C. I mean, the Schemes are ideal if doing language building with LISP thing underneath since they start that way.

                          1. 8

                            Rushing to get my lock-free rust bw-tree-backed embedded database to an alpha state before FOSDEM next weekend, where I hope to encourage a few people to give it a shot for low-impact workloads. Thinking about ways of applying real-time scheduling to threads in a concurrent property testing library I’m writing to tease out bugs in the bw tree to get similar results to PULSE when used with quviq’s quickcheck for erlang. I will name my first child “Determinism” after the past few months of intense debugging…

                            1. 34

                              The recruiters who perform the first layer of filtering will usually have a set of buzzwords or technologies that they have been told are associated with candidates worth talking to. This might include things like prometheus, kubernetes, golang, aws, puppet, etc… If you are applying for a specific job, try to figure out what stack they use, and familiarize yourself with it by working through a tutorial, so that you can mention at least a basic level of familiarity with tech that the recruiters will be often filtering for. To the good companies that want SRE’s who are curious enough to dig into unfamiliar territory, being open about your previous unfamiliarity and willingness to dive in anyway can be a strong positive signal. But recruiters don’t always share the same cultural values around this as managers or future teammates, so use your discretion about how open you are about this with the recruiter. Some teams really value curiosity, but the recruiters often don’t get that message, and will hear that you’ve only done a quick tutorial on something and will mash the “reject” button in their candidate pipeline management interface.

                              When I hire for SRE teams, I care about one thing above all else: ability to drill into unfamiliar systems that are behaving strangely. General curiosity and demonstrations of learning new things are possible indicators of this mindset to me. I actually usually prefer to hire people who are more curious than people who are less curious and more experienced because the curious people will be getting more out of the job on a personal level, and I love teaching. A lot of teams don’t prioritize teaching though, due to mismanagement, incompetence, laziness, or fear-driven mindsets that are afraid a newcomer will outperform them. Avoid these places like the plague. They are the norm, unfortunately, but they will not help you grow at a rapid pace, and if you can afford to keep playing the interview lottery, you should really hold off until you get a place that gives you a strong signal about mentorship.

                              I test for drilling proficiency by asking questions about tools that are common, and can be assumed to have some basic familiarity around, like linux or the web, and when we get to a part that they are unfamiliar with, I let them know that they can use me as Google to ask questions about how they can drill into specific behaviors. I ask about how things can fail. What are different ways a file can fail to be created, etc… (they can still use me as Google to look into the specific implementation details of things, or we can strace stuff together, etc…). Basically, I try to show them windows they can use to peer into things I’m asking about the system if they are not already familiar, and then I try to get a sense of how creatively they can use these new tools I’m giving them. That’s how the job will look, so that’s how I try to make the interview.

                              Most people suck at interviewing. They will ask you about minutae about tools that you may not be experienced with yet. It’s important to keep a calm head, let them know about where your current level of experience is with these specific technologies, and then explain to them how, in the real world, you would dive into the problem to understand and solve it. You exist to solve problems, deflect bad interview trivia questions with confidence in your ability to solve problems by drilling into them. If the team sucks at interviewing, that’s on them, and the team is less likely to be made up of experienced people who are also good to work with. People skills are the most important engineering skills.

                              If you want a laundry list of tech to get familiar with for modern SRE-like jobs:

                              • linux fundamentals, what’s an inode, what’s a process, what’s a socket
                              • AWS / GCE
                              • kubernetes
                              • prometheus


                              Different teams will prioritize coding skills differently. The most common thing I see recruiters filtering for is making sure you have one scripting and one compiled language under your belt. Golang is a good bet, as it’s taken over the pop infrastructure world.

                              Have fun :)

                              1. 2

                                That was really helpful!

                              1. 3

                                Raw sockets are the strcpy of the current age of terribly-written distributed systems. People stand no chance of finding bugs in the systems they create on their own laptops. The fact that slow, incredibly low bug:cpu cycle black-box partition testing is so successful in finding bugs should be a screaming red alarm that we need to write our systems in ways that are more amenable to testing on our laptops in the first place.

                                That means having a pluggable transport layer that can be used with real sockets in production, a simulator that messes up traffic according to the asynchronous network model, or a hit-tracing fuzzer. If you’re using raw sockets in your core serving logic, your architecture is broken, and you will have far more bugs and higher costs to fix the few bugs you discover.

                                1. 7

                                  I recommend looking for a University that does it to learn or work on one of their projects. I suspect it’s very helpful to have experienced people to help you through your first year or two of verifying anything significant.

                                  In any case, here’s a write-up from one of best in field with advice and one book reference. The other books people mention are Certified Programming with Dependent Types by Chlipala and Software Foundations. If picking based on tooling, Coq and HOL (esp Isabelle) are used on best projects in software with ACL2 being used most for hardware.

                                  It also helps to see what lightweight verification is like if you need some motivation, a fallback, or just tell you something aint worth proving. Alloy (see site) or TLA+ (learntla.com) are best for that imho.

                                  1. 2

                                    Thanks Nick. I suspect it would be helpful to have experienced mentorship as well. It’s certainly a challenging - and large - area. Working with a University sounds like a great idea.

                                    Thanks for the link! I suppose it’s time to get back to writing algorithms and proving things ;)

                                    1. 4

                                      Well, it might also help to read them to see what tools you even want to use. As always, I suggest something you will enjoy in a category with practical applications immediately or down the line. There’s more practical stuff now than ever. Formal methods still ain’t at Github’s level, though. Still more to do. :)

                                      Here’s a few areas of research to consider for what to verify:

                                      1. Obviously, a formal semantics, compiler, or optimization for a specific language worth the time. Rust and Nim come to mind. People also keep building important stuff in C++ which was only partly done. Haskell had one, formal semantics done but no verified compiler. Maybe a useful Scheme like Racket or Chicken.

                                      2. Tooling for automated proof of safety, termination, and so on for programs in a language like above with minimal specs a la SPARK or Frama-C. Optionally working on SAT/SMT solvers to get such languages closer to 100% on more problems with less schemes like ghost code. There’s lots of potential there.

                                      3. Verifying models used in actual languages for safe/effective concurrency (eg Eiffel SCOOP) and/or parallelism (eg Cray Chapel or Taft’s ParaSail). Possibly a mockup in existing language with macros of such features with verified translator to regular code.

                                      4. Verifying client-server or peer-to-peer protocols that have a lot of uptake for various properties. Optionally, making something like Verdi that already does it easier to use for non-experts or increased automation.

                                      5. Verifying important data structures and sequential algorithms for correctness. Just make sure there’s a C implementation with x86 and ARM7-9 compatibility. If it performs well, people can wrap it in any language with C FFI.

                                      6. GUI’s and graphics stacks. That occasionally gets work but not much of it. Graphics drivers are notorious for crashing. Just automated verification of them for safety like Microsoft’s SLAM and no race conditions might be helpful. Fully verification of an OpenGL stack or something might also be interesting. For GUI’s, something like Nitpicker would be easy whereas for GUI programming a DSL compiling to major toolkit would be the route. Maybe an OpenCL model. Who knows.

                                      7. Increasing usability, automation, and performance of any tool people have used to do anything on that the lists above. There’s already lots of people increasing their assurance. Cost of verification is a more important problem right now, though. The lightweight methods need more power. The heavyweight methods need more automation. I did speculate about and find a HOL-to-FOL translator used to do the latter once. That, the use of model-checkers to filter before proving, and results of SAT/SMT tooling in general suggest there’s lots of potential here.

                                      So, there’s you some ideas that might have immediate or long-term impact on problems that matter. There should also be something in that list you find fun. Maybe also something you know a lot about already. That can help. Either should have potential for projects worth spending a lot of time on.

                                      1. 2

                                        Thanks for the list. Programming languages are interesting to me and partly what drives my interest here, especially tools like Disel.

                                        I’m guessing #7, automation, will be increasingly interesting the further along I get here.

                                      2. 4

                                        If you’d like to practice on a juicy target, I invite you to join an effort to eradicate bugs in a database I’m working on in my free time, sled! It has a simple interface, but lots of optimizations that really need to be reliable. Nobody wants to use a database that very quickly deletes their data :P

                                        I’ve been working extensively with property testing and a few tricks for shaking out interesting thread interleavings, and these approaches have yielded enough bugs to keep me busy for the last 6 months, but it’s time to really ramp up the rigor.

                                        These are the approaches I believe will lead to the discovery of interesting bugs:

                                        • formally specify the lock-free algorithms in use for the IO buffer and tree using TLA+, alloy, spin, iris etc… with the goal of identifying concerns that have not been considered in the implementation
                                        • reproduce the functionality of quviq’s erlang PULSE scheduler using either ptrace or SCHED_FIFO/SCHED_RR in a test harness for performing parallel property testing as a rust library. Bring some of the awesome features of quviq’s quickcheck into the rust world.
                                        • implement a concurrency testing library for rust that utilizes ptrace and z3 to get a nice user-friendly practical implementation of Maximal Causality Reduction to identify a minimal set of relevant interleaving schedules, and then use ptrace to schedule the threads according to the generated schedules to suss out crashes or violations of specified invariants.

                                        Future directions include:

                                        • building a versioned transactional store on top of sled
                                        • building a horizontally scalable linearizable kv store on top of sled
                                        • building a location-agnostic (phones, multi-dc, PoP’s) store powered by OT & CRDT’s on top of sled

                                        The future focus on distributed systems will involve lots of interesting simulation, as well as an attempt to unify concurrency testing and distributed systems simulation. This is sort of a holy grail for me, and I hope to create tooling that lets people build significantly more reliable distributed systems, even more than the databases themselves.

                                        Let me know if any of these directions sound like things you would be interested in collaborating on! You can find my email on github if so. Having this stuff on my github has resulted in a bunch of interesting people reaching out about jobs, and I haven’t been asked to do a single technical interview after referring companies to the project to see my output. This is a 100% non-commercial endeavor at this point, but I see it as earning interesting future job opportunities at the very least. I can’t tell you if commercial formal methods people will appreciate your work on these systems or not, but this is a real system that fills a real pain point (existing embedded db’s either have crappy read perf or crappy write perf, are generally confusing for people to configure, and often have poor consistency guarantees), and applying advanced testing techniques to this should actually save people from facing huge issues.

                                        1. 1

                                          I may have to dig with this. I find having practical examples to chew on while learning quite valuable. Your work looks great, congrats on your success!

                                      3. 2

                                        “I recommend looking for a University that does it to learn or work on one of their projects. I suspect it’s very helpful to have experienced people to help you through your first year or two of verifying anything significant.”

                                        True, you can found some of the courses and lectures in a list.

                                      1. 6

                                        Making lots of progress toward stabilizing my rust lock-free bw-tree! Hopefully I’ll have an alpha out soon :) The goal for the next week is to have stable disk utilization while under extreme contention and fragmentation. Now that crash testing and interleaving tests have teased out much of the low-hanging fruit, it will soon be time to turn my attention to dynamic instrumentation for teasing out exotic race conditions :] If anyone is curious about lock-free high performance stateful systems in rust, feel free to ping me, and I’d love to spend some time teaching potential collaborators. There are a ton of really juicy database features that I’d love to delegate to other people who are curious about learning how to build them!

                                        1. 30

                                          I’ve worked part-time for about six years of my career. I started it because I’d repeatedly burned out of full-time jobs. Working 3 days/week was great for me, far more rewarding than the added salary I passed on could have been. Aside from lower work anxiety, I had time to write two books, give three conference talks, get engaged, get married, take up several hobbies, and enjoy life thoroughly. My work has been overwhelmingly better: I stay out of rabbit holes, I recognize deep patterns, I prioritize ruthlessly, I deliver the things my users didn’t realize they need. It’s not magic, it’s just downtime for my unconscious to noodle around with problems without pressure.

                                          I think working part time is a hugely valuable experience for anyone who doesn’t have a pressing need for dollars in the door (eg to pay off US medical bills or student loans). There are plenty of blogs out there on frugal living + investing (I recommend MrMoneyMustache and Bogleheads wiki), so developers can easily live comfortably and still save significantly towards retirement.

                                          1. 5

                                            I’m trying to pull back my working to part-time as well. Unfortunately many companies seem to want full-time or nothing. I’ve switched over to consulting to give me more freedom, we’ll see how that goes. I’m taking around 1.5 months off from work right now which is great. For the first few weeks it felt awkward to have no reason to do anything at any particular time, but after awhile it’s become really pleasant.

                                            1. 12

                                              About a year and a half ago I stopped working full-time, and it’s been really wonderful. I found I can work 2 months on a SF salary and live for a year in Berlin pretty comfortably. Sometimes I pick up more consulting work when I want structure, and sometimes I think about moving back to NYC where I would have to work a little more regularly, but I wouldn’t change anything about how I’ve spent my time up until now. I’ve been able to dive really deeply into a bunch of things I would never have had the time or energy to pursue if I were still a wageslave. The things I’ve built in my free time have also turned into tons of job opportunities, and I’ve stopped doing technical interviews now that people can just look at work I put on github and stuff. So, it can lead to lots of nice career things, too. I don’t want to stop engineering, but I am quite happy to live life outside of a shitty startup office a bit more.

                                              Almost no jobs will appreciate it when you tell them you’d like to work less. But if you go into a new thing with clear expectations set, I’ve found it to be much easier.

                                              1. 6

                                                This is awesome! How do you go about getting consulting work - do you look for clients, or do they approach you? Did you have a ramp-up period before you felt comfortable that you’d have enough consulting work when you need it?

                                                1. 3

                                                  I think most opportunities come my way because I genuinely enjoy talking to people about systems and engineering, and when I don’t have a full-time job I can spend more time communicating about those things. It’s networking, but for something that doesn’t feel gross to me. I am lucky to have this alignment between my personal interests and what businesses currently value. My current gig came from a comment I made here on lobste.rs ;)

                                                  A key to being comfortable is having enough runway where I know I will be OK for a while if I don’t find any work. This means being careful about burn rate. I consider potential purchases and recurring obligations in terms of how much life without work I’m giving up to have them. When my friends from work were increasing their rent to keep up with 30% of their salaries (or more) I was building the buffer that would keep me calm without work. They are worth a lot more money than me now, but I’ve been able to grow in ways that I’m extremely grateful for. Also after quitting my last full-time job I went through a period of intentional “unlearning of engineer-in-a-fun-city spending habits” which gave me a lot of peace of mind, by tripling my runway.

                                                  When I decided to stop working full-time, I didn’t know if it was going to just be a long break or a whole new arrangement. After getting over burnout from the SF startup I was at, I cold-emailed a company doing interesting work to me, and they enthusiastically agreed to a 10 hour/wk contract. That showed me that I might be able to keep it going.

                                                  When you pay 1/7 the rent, even a small trickle of engineering work feels like a geyser.

                                                  1. 1

                                                    Thanks, this is an excellent approach.

                                              2. 3

                                                Unfortunately many companies seem to want full-time or nothing. I’ve switched over to consulting to give me more freedom, we’ll see how that goes.

                                                While this is true, as Mike points out in the interview it’s possible to convince some companies some of the time to hire you part-time anyway. It’s much more effort, and you need to be willing to push back much harder. But it can be done. Since it’s not the default, you really want to only mention part time bit after company has committed to hiring you.

                                            1. 2

                                              I think it’s important for any kind of infrastructure decision to evaluate how commonly other people do similar things with the tool. Assuming a uniform bug density across functionality space, how much of that functionality are you going to be the first one to ever exercise? The more you’re using something in a way you haven’t heard of others doing, the more you need to be willing and capable of fixing the system yourself, or be able to wait on the maintainers to fix things. Just because a big name is behind the tool, it doesn’t mean it gets used at all in the way you intend to use it.

                                              Do bugs get fixed at all anymore?

                                              Are there people from outside the origin organization that contribute to an ecosystem of supporting tooling? Will the entire ecosystem become legacy when the hot SF startup that built it dies / cancels the project?

                                              Was it built by someone who has ever been on-call before? Or do you have to write telemetry into the system before you can safely deploy it?

                                              Am I going to have to teach the organization everything about it, or is there a rich ecosystem of existing documentation?

                                              Do the creators support graceful transitions across major upgrades, or do you need to write your own export/import tooling when bugfixes you depend on lie on the other side of the great migration?

                                              All this should be considered after actually determining semantic fitness, of course!

                                              1. 49

                                                Thanks very much for this detailed response.

                                                Free speech and public moderation is a core community value on this site. After all, the site was founded in response to capricious, arbitrary, and unannounced moderation on HN.

                                                That said, I think that jcs might have been a bit too light in his hand in moderation and I approve of pushcx trying to improve the quality of discussion here.

                                                1. 26

                                                  I agree, especially with this sentiment:

                                                  Communities like Usenet, 4chan, and YouTube with little to no human moderation sink into useless garbage.

                                                  As much as technologists want to believe that the right software will enable the virtues of humanity to shine through, it’s ultimate a human problem that can’t be solved with a technological solution. I wish I could find it now, but there was a study on unmoderated vs moderated hate speech in online communities which found that moderating hate speech does, in fact, extinguish it, as opposed to it finding other avenues.

                                                  Moderation does matter.

                                                  1. 8

                                                    At Tumblr I think we solved it (mostly) mechanically. I left in 2015 so some of this may have changed, but we felt very strongly that “sunlight is the best disinfectant” and we did not delete content unless it was child porn, self-harm, or gore. You can still go see terrorist cat blogs posting under #catsofjihad. The community was great about shutting down those who were abusive, and the mechanics of response were important for that. Once you created a post, anyone could reference your original, even if you deleted it. Tumblr threads are not trees, but the particular chain of responses that are favored by whoever decided to respond, giving responders full control over the context, making responses far more clear. You lose the full-picture, but gain clarity of context. This was backed up by a support team that was usually great about taking action when abuse was happening. The clarity of context is what I believe gave the community the tools it needed to effectively self-police. This is something I believe is sorely missed in other platforms.

                                                    1. 3

                                                      we felt very strongly that “sunlight is the best disinfectant”

                                                      This is, unfortunately, naïve, even to the point of being malicious—always in consequence, even if not in intent. And exponentially moreso online, where vectors for abuse are numerous and practically zero-cost.


                                                      1. 6

                                                        I don’t think your link has anything to do with the mechanism in my comment. Twitter gives total contextual power to the content creator, not to the responders. A malicious user deletes the tweet and the responses lose their power. This mechanism totally strips the community’s ability to self-police. I’m in agreement with the author of the link you posted, and I think twitter has made terrible trade-offs. There’s a reason tumblr has tended to be the place where many members of marginalized communities find the first group of people that accepts them, and these communities flourished there. That doesn’t happen with G+.

                                                        1. 5

                                                          There’s a reason tumblr has tended to be the place where many members of marginalized communities find the first group of people that accepts them, and these communities flourished there.

                                                          To be fair, this only applied if the community was not sufficiently odious that Tumblr nuked them.

                                                          1. 1

                                                            What do you mean? We let terrorists post anything that was legal…

                                                            1. 5

                                                              If memory serves there were a bunch of tags and adult blogs (some of whom were legitimately scummy folks doing commerce) that got rounded up and purged. I think around the time of Gamergate there were also some blogs purged, notably [REDACTED–see edit].

                                                              (Do yourself a favor: don’t search for that last one. There is nothing there that will make you a happier person. It’s some of the most hateful and misogynistic stuff posted online. Again, do not read it. I found it by accident through webcomics circles, and saw it descend pretty immediately into awful shit.)

                                                              EDIT: On second thought, I’m going to actually censor that last one. They’re shitty human beings and I don’t want to drive traffic their way, even for the curious.

                                                              1. 3

                                                                I mind this kind of censorship, be it self-induced or by moderators: I believe it is everyones own judgement to see for themselves and learn about it. Why mention it in description but not allow further research? In my opinion, your post is no longer credible because I can not verify it independently.

                                                                (Edit: Reverted autocorrect. Perhaps add public record of post edits?)

                                                                1. 1

                                                                  I remember a few cases where certain tags were blocked on mobile because if one of the app store testers found porn using a test of the app it could lead to them rejecting an update. Once, this unfortunately included tags like bisexual, gay and lesbian. It was not a fun decision to make, because it hurt the communities we tried so hard to protect, but Apple was holding the business hostage with this, and we chose to block lgbt tags in the short term and figure out something else later. There was significant blowback.

                                                                  We often mass-blocked clusters of associated spammers, which is uncontroversial as far as I know. Other controversial mass-blocks may have happened, but I don’t remember them.

                                                    2. 7

                                                      I agree, on both points. Transparency has always been a core goal of the site, and should continue to be. And I also think it makes sense to do at least slightly more moderation than there has historically been.

                                                      1. 1

                                                        I personally think the quality is/was fine and don’t care if there are flame wars as long as they’re not just 4chan style insults.

                                                      1. 1

                                                        I like parts of the transaction model. It’s how I would/do do it - the commit collection, batching and asynchronous reply model.

                                                        That said, why are we still using a decades old domain specific language from 1979 (SQL) to interact with databases.

                                                        A language that has an extremely low impedance match with “data”, when we have a perfectly good language from two decades earlier… 1958 (LISP) that does a better job, and doesn’t require an ad-hoc query planner that trys (and fails) to outsmart the person planning the query.

                                                        Not only that but clearly someone didn’t get the memo that relational database models are so “1995”.

                                                        I applaud the efforts, and cynicism aside it really looks like they are doing their best here, and appreciate there is still some time to go before SQL falls away.

                                                        Unfortunately I can’t really like the fact that the authors are working this hard on what really is a dead paradigm.

                                                        Very well made paper however.

                                                        Obviously this comment comes off as authoritative, snarky and denigrating to the people who did the work.

                                                        That’s really not my intention however; it’s more like

                                                        “just sayin…”

                                                        1. 4

                                                          Probably because it’s the least dead paradigm in the data world. Tools are useful to the extent that they can be employed to solve problems, and the first part of that is minimizing the number of new concepts the user must learn before they can solve problems.

                                                          1. 1

                                                            Just to go further in agreement in relation to your comment.

                                                            Select * from Customers where Balance < 0

                                                            I just made that up so it probably isn’t valid SQL. It’s years since I wrote a lot of SQL.

                                                            Sure, that is easy and as you said, it helps people get going and solve problems.

                                                            But look what they just did. They learned a DSL that wasn’t needed. Transducers like Map/Filter/Reduce (or better) are much clearer for them and the machine.

                                                            Furthermore those translate to many other languages and compose much more easily.

                                                            I’m not convinced it is easier to learn SQL than just learning operators based on basic set theory.

                                                            Not only that, but extending SQL to graph databases, geospatial etc.

                                                            Sure, it can be done, and has been, but only at the cost of vendor specific lock-in or extremely obtuse syntax.

                                                            I used SQL a great deal back in the day. Before I “saw the light”. It’s just there are things that are equally expressive but compose much better and honestly not that much more difficult.

                                                            I think that the problem is a matter of how principles are introduced.

                                                            Oh you need to do data? You need to use SQL.

                                                            It doesn’t work like that.

                                                            1. 0

                                                              Yes I get that and you are right of course.

                                                              When SQL started it really seemed quite good and worked awesomely.

                                                              Then it got extended - a lot.

                                                              It got extended to such a great extent that when I look at SQL now, I feel like the thousands of people who have spent so much time in learning it properly have been painted into a corner, and I really feel sorry for them.

                                                              I have good friends who are top notch SQL DBA’s but they can’t transfer their skills easily. They are upset and have been going through the five stages of grief for some time.

                                                              Data is not flat relational tables anymore, (you could argue it never was), and I really feel bad that they really did do “Computer Science” to a high level on a very obscure technology that is destined to the same fate as COBOL.

                                                              Obviously they get paid a lot. So there is that.

                                                          1. 4

                                                            TiDB doesn’t get much attention in the english-speaking infrastructure world, but they deserve more. They have been putting a lot of effort into reliability for the last year and a half, and are using things like namazu and simulation to great effect. More people building distributed systems need to know about simulation, as it’s one of the best ways to get your laptop to behave more like an actual (possibly partitioned) cluster without having to slowly spin up a bunch of containers / vm’s. Simulation can get very high bug/cpu cycle ratios.

                                                            1. 3

                                                              Small correction: 128-bit CAS is supported on all(?) x64 processors. I used it for a lock-free hash table one time, since 16 character keys should be enough for anybody :).

                                                              1. 1

                                                                Thanks for the correction!

                                                              1. 3

                                                                Would you be willing to link a TLA+ model of a lock-free algorithm? I think it’d be really cool to see how an “obviously correct” system has a subtle bug.

                                                                1. 3

                                                                  I actually set out to write an article about correctness tooling, but then I realized I had 4 blog posts worth of stuff, so I split away this (probably already too long) part giving some background on lock-free algorithms! I linked to your TLA+ manual in the section about responsible construction, but I agree that it could be nice to demonstrate an obviously correct system that TLA+ shows to have a subtle bug. Do you have a particular subtly broken yet simple algorithm in mind? I worry that this is already too long, but it could be nice to demonstrate just how subtle some bugs can be.

                                                                  1. 3

                                                                    Do you have a particular subtly broken yet simple algorithm in mind?

                                                                    I’m not familiar with lock free algorithms and after reading your essay I never want to be D:

                                                                    If there’s a broken algorithm you know of I’d be happy to write the TLA+ spec, though!

                                                                  2. 3

                                                                    I did some quick Googling on that. Here’s one for Rust who is working through a large number of algorithms:


                                                                    EDIT: Wait, it’s the same guy. I probably got that bookmark from him on Lobsters lol. Well, stay tuned for periodic updates there anyway haha.

                                                                    Most of the research uses either custom tools, SPIN, or provers. The top teams recently shifted to augmenting the normal provers to handle this stuff to get more assurance. Obviously, not much useful for the masses. The few I found with model-checking were paywalled and of unknown quality. These non-paywalled papers were representative of a few angles I saw in my search:




                                                                    I bring them up as some of the properties or analyses driving them might be ported to TLA+. Note that I threw that last one in there to give you a taste of the SPIN work that I say is often similar to TLA+ in application. They have good, code examples in it on top of discussions about model-checking linearizability for systems with out-of-order execution.

                                                                  1. 4

                                                                    If you’re a security practitioner, teaching yourself how to hack is also part of the “Hacking is Cool” dumb idea. Think about it for a couple of minutes: teaching yourself a bunch of exploits and how to use them means you’re investing your time in learning a bunch of tools and techniques that are going to go stale as soon as everyone has patched that particular hole. It means you’ve made part of your professional skill-set dependent on “Penetrate and Patch” and you’re going to have to be part of the arms-race if you want that skill-set to remain relevant and up-to-date. Wouldn’t it be more sensible to learn how to design security systems that are hack-proof than to learn how to identify security systems that are dumb?

                                                                    I read this as “Just do it right, don’t waste time learning how it goes wrong.”

                                                                    1. 3

                                                                      Yeah I wouldn’t go all the way to the point of “Just don’t make mistakes”, but I do think there is something to be said for taking things slower and spending more time designing carefully up-front, rather than falling into the Penetrate and Patch cycle. However I’d also want to use all of the tools available at my disposal to verify the design, but they can’t replace having a secure design, nor can they (fully) make an insecure design secure.

                                                                    1. 7

                                                                      I really appreciate your writing about correctness! As someone who writes stuff that has devastating consequences for having bugs, I am always trying to come up with more tools to help me. Even this little tool has teased out SO MANY bugs in concurrent programs:

                                                                      # usage: ./shufnice.sh name_of_process_to_shuffle
                                                                      while true; do
                                                                        PID=`pgrep $1`
                                                                        TIDS=`ls /proc/$PID/task`
                                                                        TID=`echo $TIDS |  tr " " "\n" | shuf -n1`
                                                                        NICE=$((`shuf -i 0-39 -n 1` - 20))
                                                                        echo "renicing $TID to $NICE"
                                                                        renice -n $NICE -p $TID

                                                                      Keep speaking the good word :)

                                                                      1. 12

                                                                        I disagree with this post. I’m also a professional and my time is valuable too. However, two of the three suggestions they made are taking significant time away from me and my team for evaluating a candidate that potentially can’t even write code to solve a simple task. Part of an interview process is to filter out people before it gets to that point, so we’re not wasting employees time.

                                                                        1. 10

                                                                          I think coding challenges are optimised for candidates who are looking for a job. I’ve been in that boat once, and when you’re actually looking for a job your “valuable time” is of course bet spent trying to get said job (by doing coding challenge or whatever else).

                                                                          Most of the time, though, I’m being recruited. I’m not going to do a coding challenge for a recruiter.

                                                                          1. 1

                                                                            Taking an entire day to work with them (unpaid) still strikes me as really weird.

                                                                            1. 1

                                                                              Think of it as a great way to find out if these are people you would want to work with every day before you actually have to do that.

                                                                          2. 8

                                                                            I disagree with this post. I’m also a professional and my time is valuable too.

                                                                            I have the same problem with it as a hiring manager– how do I screen out socially capable but inept developers– and I share the author’s opinion when I’m the candidate– this tells me nothing about why I want to work for you. Each side wants the other to make an unequal commitment so it amounts to a single round game with that candidate. As a candidate with a choice of games, I don’t want to play this one and it signals disregard for/trivialization of the candidate’s investment and work history. For the hiring side, this plays out multiple times and there is investment in posting, screening, reviewing, etc. so regardless of this round my total investment is higher but not visible.

                                                                            So what have I personally done? When I’m the candidate, I refuse to do the coding challenge and say, like the author, check my repos and talk to my references (unless the problem catches my interest, then I might). I have that luxury. When I’m the employer? Up front I explain how it works and what timeline they can expect as part of a 15-minute phone screen for basic info with someone technical. Then I arrange a round of 45-60 minute calls: a technical call with someone on the team and a social/technical call where I try to get them to talk with me about their work in detail and many of the non-code aspects of their job, habits, tools, designs, etc. They’ll probably have a call with my manager or another peer. Then, if I’m satisfied but not sure, I bring them in or have a video chat and they do a quick coding test. This wastes less of their time, makes my commitment visible, and seems to work but it is not a scalable process.

                                                                            1. 7

                                                                              I have a portfolio and some github projects. This is where most of my hiring emails come from. So when a company doesn’t spend the time to check that out, and they want me to implement some trivial thing that doesn’t generate value for them, I don’t have time for them either.

                                                                              I’ve had companies pay me to be a consultant for a week before giving me an offer, which was a nice way to learn about who they are. On the other hand, sometimes companies give me job offers now before I know anything about them, and I have to pump the brakes and talk to more of them before I feel comfortable going into something long-term.

                                                                              1. [Comment removed by author]

                                                                                1. 6

                                                                                  They didn’t ask for the hiring managers time, they asked for developers time. Either way, these people have other job responsibilities too and their time is important to defend.

                                                                                2. 1

                                                                                  …evaluating a candidate that potentially can’t even write code to solve a simple task.

                                                                                  In the post, they talk about how they have a blog, numerous GitHub repositories, etc. At that point it should be obvious they can code. The interview then should be more about “fit” or whatever, IMHO.

                                                                                  1. 5

                                                                                    They aren’t the only candidate we would interview and in my opinion, it is better to have a consistent process. If every candidate had a similar level of public presence to evaluate then maybe that would be different.

                                                                                    1. 7

                                                                                      So, again IMHO, at that point you’re basically throwing out someone with passion and talent due to bureaucracy. If I come to you with decades of experience/conference talks/published papers/lots of open source software to review/whatever…and you ask me to spend 30 minutes doing trivial work, you’re basically implying that I’m lying to you and/or that your company cares more about process than people.

                                                                                      Again, this is IMHO.

                                                                                      1. 7

                                                                                        I’m saying that you’re not the only person applying for the job and I need to treat everyone the same, so we’re not giving preferential treatment.

                                                                                        1. 3

                                                                                          I know, but…maybe you should give preferential treatment to people who are obviously better candidates. :)

                                                                                          1. 9

                                                                                            Some of the best engineers I know have zero public presence. Some of them are extremely humble and don’t like public flashiness. Some of them have families and maintain a strong work-life balance with non-tech stuff. Never assume those with a strong public presence are giving the world the whole picture. You still want to drill into the parts of their personality that they don’t highlight.

                                                                                            1. 4

                                                                                              Why does having a public portfolio make someone an obviously better candidate? What makes a candidate obviously better? Arbitrary social metrics? Ability to speak quickly about technical topics? Ability to talk bullshit without it sounding like bullshit?

                                                                                              How do you know a candidate is obviously better without having them go through the same process and pipeline?

                                                                                              1. 3

                                                                                                How do you know a candidate is obviously better without having them go through the same process and pipeline?

                                                                                                If the code in their GitHub account is as good or better than what would be tested by my coding test, why subject them to that? Ask harder questions, ask questions about the things that the coding test wouldn’t cover (including “soft” things that would judge a good fit), etc.

                                                                                                Why does having a public portfolio make someone an obviously better candidate?

                                                                                                Which surgeon would you rather have? The one nobody’s ever heard of, or the one who has published articles on the relevant surgical procedures, who goes to conferences to learn more about surgery, who obviously is passionate enough about medicine that they would study it even if they weren’t getting paid?

                                                                                          2. 8

                                                                                            There are, unfortunately, a lot of liars out there. I won’t say that industry hiring practices are anywhere near ideal, but as an interviewer it was astonishing how many people with great optics were incapable of coding. Someone would speak with great passion about all their projects and yada yada, and id be fairly convinced they could do the job, then I’d ask the most basic question imaginable. Splat.

                                                                                            I guess it helps if you choose to believe the test isn’t meant for you, but for all the other pretenders.

                                                                                            1. 7

                                                                                              Even more surprising to me is that people who can’t actually code are somehow able to sustain careers as developers. It started making a lot of sense to me why good developers are in such high demand after I had the opportunity to do some interviewing and found that a frustratingly large amount of applicants can’t actually code, even if they look great on paper and are currently employed as developers.

                                                                                              I think it’s incredibly risky to hire a developer without seeing code that they have written, be it from open source contributions or a coding test if necessary.

                                                                                              1. 3

                                                                                                Onsite nerves can kick in. It sure as hell did for me. I hate white boarding and I lockup. Totally brain freeze. That said, if it’s a basic one like writing a loop…well, they somehow lied their way into the onsite. Thing is, a take home coding challenge can weed out those people pretty fast. If they do that and come in and fall flat on their face before the whiteboard I don’t totally discount them. Anyway, there’s no perfect solution. There is always the potential to hire someone that is great at coding interviews and then sucks at real world problems.

                                                                                                1. 2

                                                                                                  This is exactly my company’s experience. Half of the candidates would just bomb out on the C++ test even though they have studied it at high school/college/university and worked with it at their job for 5-10 years. How?!? Because they were either doing Java, not C++, or they were actually managing a team that did C++ and never had to touch it themselves (Well since leaving school at least).

                                                                                                  1. 1

                                                                                                    What I don’t understand is why this is so hyper-specific to developers. You never hear UI designers talking about processes like this.

                                                                                                    1. 6

                                                                                                      Really? I’ve heard UI designers talk about it a lot.