Threads for aphyr
There are sort of two critiques here–one is whether to arrange data in a columnar (e.g. all ant types stored contiguously, all ant sizes stored contiguously) or row (for each ant, all properties stored together) oriented format. Another is just that pointer-chasing is expensive, and if you aren’t going to reference data multiple times, you don’t necessarily need that pointer indirection.
On the latter problem (and perhaps to some extent, the first) readers might be interested in something like Zach Tellman’s Vertigo, which offers nestable, packed structs and arrays backed by flat JVM ByteBuffers. Rather than chasing multiple pointers, you can jump directly to any field at any offset immediately. Vertigo also shows some possibilities for friendly iteration syntax, lazy views over structs, etc.
I’m confused: WAS this a Byzantine fault? The post only describes a network partition: every node appears to have followed the Raft protocol correctly. That’s a normal behavior of asynchronous networks, not a Byzantine failure. It sounds like the partition interfered with Raft elections, but didn’t actually cause a safety violation.
In the RAFT protocol, cluster members are assumed to be either available or unavailable, and to provide accurate information or none at all.
This is sort of… half-true. Raft is designed to preserve safety under any asynchronous network conditions, including partitions like this one, and they haven’t described any kind of safety violation. Like all consensus systems, network partitions can interfere with availability, and during a partial network partition like this one you can wind up with less-than-ideal availability. Now… the Raft paper does say:
[Nodes] are fully functional (available) as long as any majority of the servers are operational and can communicate with each other and with clients.
And this claim is violated here! At the same time, we should understand that claim in the context of the paper’s repeated cautions around availability:
… availability (the ability of the system to respond to clients in a timely manner) must inevitably depend on timing. For example, if message exchanges take longer than the typical time between server crashes, candidates will not stay up long enough to win an election; without a steady leader, Raft cannot make progress.
Regardless, we do have what looks like a violation of the majority-liveness claim: it sounds like a partially isolated node can rapidly advance its own epoch through elections, forcing a well-connected majority component to go down. Curiously, there ARE mechanisms in the Raft paper which are specifically intended to address this:
To prevent this problem, servers disregard RequestVote RPCs when they believe a current leader exists. Specifically, if a server receives a RequestVote RPC within the minimum election timeout of hearing from a current leader, it does not update its term or grant its vote. This does not affect normal elections, where each server waits at least a minimum election timeout before starting an election. However, it helps avoid disruptions from removed servers: if a leader is able to get heartbeats to its cluster, then it will not be deposed by larger term numbers.
Does this mechanism fail to address this partial-partition scenario? Did etcd opt not to implement it? Or could there be a bug in etcd? Worth investigating!
So if I understand this right, the intended usage is that you model the expected behavior, you instrument your real code to produce the traces and then you deploy it or fuzz it to get a lot of traces you can check for linearizability violations?
In practice what’s the easiest way to capture the traces? Just log some JSON? Derive it from otel spans?
You do need to ensure that whatever system records the history of events is itself Linearizable (hand-waving a bit here; you can actually get away with slightly weaker properties). That could be a single-node program which records invocations and completions in a Linearizable in-memory structure. Or you could journal them (synchronously!) to a Linearizable database before invocation and after completion, in production.
What you can’t do is, say, stamp log lines or OTEL events with local node clocks–those aren’t necessarily synchronized, and the checker might tell you your program violated Linearizability when it didn’t. You also can’t send them to a database asynchronously, because the message recording the invocation might be delayed and arrive at the database after the operation was actually performed. You also can’t use a Serializable or Causally consistent database, because it might fail to record your events in real-time order. A Sequential log is OK, but the scope of the Sequential object has to be be the entire log; otherwise you could (you guessed it!) get reorderings.
Another thing to be aware of is that if you log to the same datastore that’s under test, you could influence the results. For example, if your data store has an issue with stale reads in read-only transactions, and you add an “insert log line” between every read transaction down the same connection, you could see the bug become invisible.
In practice is it enough to synchronize the clocks using something like ntp?
I mean… no? Imagine you detected a Linearizability violation in a history recorded using timestamps on, say, Linux machines. From this you could conclude that either a.) the system being measured violated Linearizability, or b.) the history-collecting system had, at some time, poor synchronization; or the VM paused; or the OS paused; or the application paused; and so on. It doesn’t allow you to make a provable claim of correctness or incorrectness.
Maybe?
If you’re talking about something like EC2 timesync, where clock errors are ~50us, and your storage system is remote and best-case a couple hundred microseconds, and you’re careful about which end of the clock bound to pick, you can still get good results.
NTP in general, probably not, unless you’re very careful about the way you set it up and have great time infrastructure.
would it work to use lamport clocks? or stamp messages with the ones they’ve seen and then topologically sort (functionally the same thing but different)?
Sort of. You can use messages between nodes and a strategy like Lamport clocks to establish a partial order which is consistent (because messages always flow forwards in time) with the real-time order. However, you’d fail to detect violations of Linearizability over timescales shorter than the message propagation interval.
hrm—forgive me if these are stupid questions, but
wouldn’t any really linearisable logging scheme induce synchronisation between nodes, potentially also causing you to fail to detect some bugs? are there bugs that are still systematically more likely to be caught or can only be caught that way? (it seems not very nice because it sacrifices scalability)
IIUC, the difference between sequential consistency and linearisability is effectively that linearisability is consistent with communication ‘outside the system’ where seqcst doesn’t have to be. and lamport clocks or causal ordering should not miss sequential consistency violations. but if there are any classes of ‘external communication’ that we’re interested in in some context, couldn’t we try to bring those into the testing/logging system too? (i think this is similar to using a linearisable logging system, except that you don’t record all the ordering relationships—so you don’t get to full linearisability—only some of them, but what you do record you get in a distributed way)
Yep! It’s actually impossible to measure precisely. Jepsen does this by putting all clients on a single node, allowing fast connections between clients and servers, and (optionally) adding latency between servers; this lets it see errors “faster than light”, so to speak.
In general, I wouldn’t worry about scalability. Linearizability checking is NP-hard and relies on total histories–both of those are already bad for scalability. Even if that weren’t the case, you’ll get the most bang for your buck testing in a controlled environment where you can induce failures, rather than in production, where failures are (I hope!?) comparatively infrequent.
I’m not sure what this means exactly, but I should note that Causal is, in general, weaker than Sequential.
Sure, you could do that, but then the property you’re measuring is something like Causal or Sequential, rather than Linearizable.
Yes you’ve got it! If the project is in Go you could just write the fuzzer and test harness in Go. Otherwise yes you’d want to write some intermediate format from your system and then write a Go program to convert it for Porcupine.
Footnote 1 is interesting: https://jepsen.io/analyses/jetcd-0.8.2#fnref1
To my knowledge, this is the first time a client does not want to give official permission to publish the findings?
In any case, will that lead to changes in the contract negotiations for future efforts?
Clients routinely engage Jepsen for private analysis work; that’s just fine and the work is often never published. This was an odd case. Jepsen was asked to take a few days to re-run an existing test suite against the then-current version of etcd. We didn’t have budget or time for a full public analysis, but I volunteered to do that work–polishing the test suite, investigating anomalies, collecting exemplars, writing a report, editing, etc.–on my own time, after the engagement. The client was enthusiastic, but once I finished the report, the client’s legal department wouldn’t authorize the acknowledgement.
I’m not exactly sure what to change–in 9 years of doing this work it’s only happened once! One thing I’m considering is that when Jepsen takes on a client that asks for analysis of third-party systems, we include explicit contract terms giving permission to publicly acknowledge the company in any possible written report as having funded research into that system. That also avoids any sense of a system having been “tainted” by having some privately-funded contribution to its test suite.
In broader terms, I’m not sure what constitutes a meaningful contribution requiring a funding acknowledgement. This is generally simple: a single company, usually the vendor, funds the development or extension of a test suite and the time to investigate and write the report. That company gets acknowledged as the funder. But in a broader sense, something like 50-100 clients have funded improvements to Jepsen’s core libraries. At some point those contributions become diffuse–the library is a general-purpose commons and the clients generally have no idea what databases I’ll go on to test. Database-specific test suites also become commons, over time: the etcd test suite includes years of independent work by Jepsen, work funded by Stripe, work funded by the Linux Foundation, and work funded by an undisclosed client. My sense is that the purpose of a funding disclosure is to acknowledge why an investigation happened, rather than every past contributor, and that suggests Jepsen ought to disclose funding more on the basis of who paid for the investigatory work, rather than all past contributors to the software used in that investigation.
Lots to think about.
Thanks for the long answer and for humoring my curiosity! Makes sense to treat it that way and make it visible.
That is interesting behavior, especially from a CNCF (https://cncf.io/) company, who you’d hope would be more open to security/reliability concerns.
This is, as always from Jepsen / @aphyr, great work. And it strongly makes its central point that
If I might be so bold as to suggest one possible further improvement for future posts, I did find the diagrams in this post less useful than in most Jepsen posts: this time, the problem isn’t in any of the arrows (quite a few bugs found by Jepsen appear to ultimately be fixed by adding a lock to enforce a dependency/“arrow”), but - spoiler! - in the assumption that a transaction submitted to the library will be executed at most once. That might be better visualized in a different fashion (just a list of actually-executed serial transactions?)
(Aside: my commiseration on the situation described in footnote 1. That must have been frustrating to all involved!)
I thought about this one a bit–you can imagine drawing the “true serial execution” including retried transactions, but if you do that, the anomaly actually disappears! These diagrams make it obvious that the user-observable history actually does have a circular flow of information. I thought about including alternate versions of the diagrams, but decided that the prose explanation was clear enough on its own.
I’m a simple man, I see a Jepsen post, I upvote (and read, and always learn a ton). Thanks @aphyr!
Thank you!
It look me a long time using Datomic in anger to really grok the peculiarities of its transaction model, particularly the way transaction request data composes (or doesn’t). I really wish I’d had this blog post when I started, it captures everything very well–not just what the model is, but its implications for writing correct software.
I’m curious what you think about the all-at-once transaction semantics. Do you find that lacking intermediate states helps reduce bugs? Ever hit an issue with transaction function composition like the one described in the paper?
I have definitely hit issues with transaction-function composition. As you write in your piece, most of the time this results in datom conflict errors and so it gets caught; I’m not sure we have run into the case where the resulting transaction is non-conflicting but semantically invalid (like your grants example).
There are a lot of times where we might want to use a transaction function or an entity predicate, but don’t because it’s too expensive to run on the transactor. (Even though our workload is usually read-constrained, our writes are bursty and if transactions are slow we will eventually get backed up.) A pattern we’ve used in a few places is using
d/withto speculatively evaluate a set of transaction-request data on a peer, then query the result. This gives you something akin to the visibility into intra-transaction state that you get in other databases. Of course, the caveat is that the database version against which you speculatively evaluate the data on the peer may be different from the version against which you end up transacting on the transactor, so you need to use a CAS to ensure that important invariants have not changed.(I believe you can use
d/within a transaction function, too. But I have been leery of doing this for performance reasons.)Side note, but what a solid recommendation of Clojure. It’s fun to see software written in Clojure get such praise. I’m glad that all of Datomic’s claims hold true. Now all I need is a reason to use it!
It’s a really neat database! It took a few weeks to smoosh my head into Datomic’s model, but… like Lisp, Haskell, Datalog, Forth, or Erlang, internalizing that model lets you see the whole concept of “databases” differently. Simple stuff is tricky! Tricky stuff is easy! I really wish other databases had Datomic’s “DB snapshot as a value” concept.
If you ever have time, it would be fascinating to hear snippets of what this looked like! I’ve fondly admired Dataomic from afar for a while. I’ve used Elixir professionally so feel like Iv’e had a similar kind of experience, similar to how you mentioned Erlang of course.
Either way, thank you for your work.
The team behind Datomic has not only a great language but some of the best programmers. Just as a thought experiment, I wonder what they would have achieved with JavaScript :-)
Jepsen is great. As I commented elsewhere https://jepsen.io/consistency is a go-to for me. I don’t use MySQL so I don’t have too much to say about the substance of this particular test, but I wanted to take a moment to appreciate how this is an excellent example of exposition for the web.
Many of these points below aren’t necessarily unique or novel, and are subjective, but it shows an attention to detail and service to the reader.
More on the substantial side, glad to see they also tested RDS as well, since that’s certainly going to be relevant to many.
Thanks for writing this. I work my butt off on Jepsen’s writing, of course, but folks don’t necessarily see the weeks of work I put in to typesetting, color choice, visualization design. Stuff like color-blind accessibility, italic families, footnote backlinks, hand-editing code snippets to fit well within column widths… As someone with a bit of background in book arts, it makes me really happy to see people appreciate those aspects of the work.
As you note, every report gets compiled (thanks to Pandoc and a pile of TeX templates, Lua Pandoc filters, and custom HTML rewriting in the web site’s Clojure backend) to both HTML and PDF, including embedding (sometimes interactive!) SVGs. Some of the awkward choices in layout and wrapping come about from trying to write and compile for both formats–everything has to work in two-column text, on large browsers, and on phones.
A lot of the typographical stuff you’re noting is from Pandoc & XeTeX. I write in Markdown with a lot of embedded LaTeX hints (-/–/—, math typesetting, etc) and it does a darn good job of translating that to both PDF and HTML. The web site code also uses a bunch of the Flexmark typographical extensions to Markdown. Body fonts are TeX Gyre Schola. Titles and some metadata are Museo Slab and Museo Sans.
Some of Jepsen’s design is just BAD. The SVG renderings of anomalies from GraphViz? That stuff is such a hack, and it sticks out like a sore thumb. It’s machine-generated, it gets written as an SVG, munged to EPS, there’s terrible bounding-box hacks, it’s just… it’s a nightmare. Kind of amazed it works at all.
On the writing side–one of the things I work really hard on, both in my writing and teaching, is having a scale of affordances. Jepsen reports should be straightforward and specific for database people and researchers. But they should also have top-line recommendations that a motivated VP or CTO can digest. Most importantly, I want newcomers to have handholds throughout the work. They may skip some of the jargon or formalism, but in every section there should be some kind of beginner-friendly exposition that helps them learn something new, and motivate further learning.
Oh I didn’t realize you were the poster so I’m glad you saw my comment!
The effort does show, and thanks for sharing your tooling and process. I think it was the moment I opened the PDF and saw how consistent it was with the HTML that I went back and started writing this comment, because like you said that can be a surprisingly large amount of work. Typesetting a fixed-size PDF is one thing; typesetting for effectively 3+ formats at once is another. I was recently doing something similar and seeing how far I could take web-typesetting so it was all pretty fresh for me. (Side note, I wish there was a better way to scale down the code fonts but keeping the weight consistent. Variable fonts maybe?) Also amusing to note that with P3 and Rec2020 we literally have more colors to worry about too.
I also didn’t realize until you mentioned it how awkward the SVG across both formats would be. Hopefully we’ll see improvements in diagram-setting tooling, it’s still so manual and finicky.
I wanted to write more on the prose but worried my comment was already too long. It is interesting to optimize for both skimming and depth. A while back I read that one book, How to Read a Book (usually gets a chuckle when I mention to someone I felt the need to read such a thing) but I remember its main thesis being: If you only had 30 seconds to extract as much information from a book, what would you do? How about 5 minutes, 30 minutes, an hour, a day? And developing methods for all those levels. What you said reminded me of applying that in reverse, providing something at each level that can further guide their attention where it’s most useful.
(Funny enough, that book had a little rant about how most books are way too long, but to be honest How to Read a Book was way longer than it needed to be. Nonetheless that book has stuck with me. I write better abstracts now, I don’t immediately skip tables of contents, and I felt vindicated in my habit of skimming and economizing my attention.)
I’ve written a half-dozen libraries which had to deal with paginated APIs, and it’s always a little awkward threading the state through lazy-seq properly–this will be a welcome addition!
A small shout-out to the push-sum family of gossip systems, which you can use to obtain asynchronous, exponential-fast convergence for things like rate-limiting counters, quantiles, etc. https://manishyadav.dev/blog/gossip-push-sum-protocols
Fast or elegant. Writing Java in Clojure for higher performance is meh. This is what we get when trying to use a too high level of abstraction. It is really good that we have a simple way to observe performance and fix the bottlenecks though.
The reality is that idiomatic Clojure without any optimizations is fast enough for vast majority of situations. Clojure is already significantly faster than a lot of popular languages like Ruby or Python out of the box. So, the amount of times you actually need to do these kind optimizations is pretty rare in most domains Clojure is used. However, having the ability to tweak performance to this level is really handy when you do end up in a scenario where that’s needed. I’m very much a fan of writing code with maintainability in mind as the primary concern, and optimizing things on case by case basis as the need arises. Clojure is hands down one of the best languages I’ve used in this regards.
Any sources, if you don’t mind my asking?
Totally anecdotal, but my rewrite of the Riemann monitoring system from Ruby to Clojure improved throughput by something like two orders of magnitude right off the bat, and that was the first thing I ever wrote in Clojure. With a little work, we went from ~700 events/sec (in ruby) to ~10 million events/sec per node. Not an apples-to-apples comparison there–that’s years of optimization, batching and network tricks, and different hardware, but like… coming from MRI, the JVM was a game changer.
Here’s a post from AppsFlyer discussing a significant performance boost when we moving from Python to Clojure.
I completely agree with you. My only problem is that is very hard to sell Clojure to people. Both clients and co-workers. Clojure and F# are the two default languages I use a lot for writing code for myself. Unfortunately team mates and clients force me to use Python, TS, C#, etc.
My team’s been using Clojure for close to a decade now, and what I observed over that time is that developers experienced in imperative style often have a hard time getting into FP. Yet, when my team hires students who have little to no programming experience, they’re able to pick up Clojure very quickly and tend to love the language.
I think the big problem is with mismatched expectations. People build up a set of transferable skills that allow them to quickly move from one language to another as long as those languages are within the same family. If you learn a language like Ruby then you can quickly pick up Python, Js, or Java. You’ll have to learn a bit of syntax and libraries, but how you approach code structure and solve problems remains largely the same.
However, using a language like Clojure or F# requires a whole new set of skills that you wouldn’t pick up working with imperative languages. People tend to confuse this with FP being inherently more difficult as opposed to just being different.
I don’t really understand this critique. It feels like there’s two separate things here: one is the question of an equivalence class for “equal but not identical” objects, and another is modeling and control of state and side effects. For the first, Clojure can compare memory addresses just like Java can with
==:(identical? [:a] [:a])returnsfalsebecause the two vectors have different memory addresses. For the second, Clojure has a rich set of reference types (what Clojure calls “identity”) for explicitly modeling changing state, including vars for the global environment, promises, atoms, and STM refs for changing data, anddefrecordanddeftype, in addition to all the JVM container types (e.g. AtomicReference) for Weirder Stuff. It’s not at all a “purely” functional approach–you can do that, but I think the default Clojure path leans a lot more towards eagerness and mutability rather than, say, an effects system or monadic transformers.I sometimes wonder if the benefits of immutable data structures aren’t overshadowed by the added complexity needed to use them efficiently in practice.
Unless I’m misunderstanding, a Zipper is basically an iterator that makes traversing and updating immutable trees more efficient? Does anybody have real life examples of how these are used? Or maybe an example comparing a language with mutable data structures and Clojure with Zippers?
I think it might be better to think of a zipper as something that gives you suspendable/resumable iteration. One place I find them useful is when building UIs where you need to step through a list of data. At work I use a zipper on a page where I have a list of items that are related to the main entity a user is viewing. The UI lets a user step back and forward through these one at a time with next/previous buttons. A zipper gives me all of these operations in O(1) time, and also lets me query the state for “is there a previous page? is there a next page?”.
This is doable if you maintain a list of items along with the length of the list and some
0 <= i < length xs, but now I think all the operations I want become harder to obviously see, and also harder to obviously verify I got things correct (e.g., did I remember to incrementi? Did I bounds check?)Thank you! It’s interesting to hear a real life use case, and I hadn’t considered the suspendable/resumable aspect. I can see that being a real benefit when iterating over a complicated structure.
Is the interface extendable for user defined types? One gripe I have with Common Lisp is that the built-in iteration with (loop) isn’t easily extended for user defined types. The popular package ‘iterate’ allows it, but it’s an extra dependency to pull in.
I suppose one day I should just learn Clojure.
Yes, the interface is extensible–zippers basically take a few functions which define the children of a node, a root, and how to create new nodes. This makes them really convenient when you want to do ad-hoc modification of data structures that weren’t necessarily intended for traversal–say, for example, an HTML document, an SVG, a b*tree, a web API, or ad-hoc tree representations:
(node child1 (child2 subchild1 subchild2))or{:node "foo" :children [{:node "bar"} ...]}–you can lift them into a zipper, traverse or transform them in some way, then spit back a new structure without anyone being the wiser. If your trees are code, this is a nice way to write compiler optimizations and complex macros–you could, for instance, statically apply DeMorgan’s laws to simplify deeply nested boolean expressions in a complex conditional.This is super helpful for complex transformations like the kind you might do with XSLT/XPATH on XML documents: I could use a zipper to find instances of (e.g.) an image followed by a paragraph (at any level of nesting) within a
<div>with a particular class, then rewrite that HTML to, I dunno, add some special tags and wrap both photo and following paragraph in a single div. Really handy for transforming, say, the results of a Markdown processor if the output it gives you isn’t quite right for the context you want to display it in.Of course you can do all this mutably too, but zippers give you two big advantages. One is that they allow you to extend tree traversal and manipulation over arbitrary structures that weren’t necessarily intended for it. Second, because the zippers themselves are immutable, you get some nice atomic properties: if I have a node with two children
aandb, I can compute a newa'andb'and replace them atomically based on bothaandbwithout having to create temporary variables or (potentially) deep copies. You get arbitrary rollback, it plays nicely withcompare-and-set, etc.Not always the right solution–of course their performance isn’t going to be as good as direct traversal–but they’re a surprisingly useful tool. :-)
I have to admit, I did raise an eyebrow at the idea that Tigerbeetle’s correctness relies on well-behaved wall clocks during VM migration–in a database which also explicitly claims strict serializability. Viewstamped replication is an established consensus protocol, so that’s not a red flag per se, but we know from Lamport 2002 that consensus requires (in general) at least two network delays. However, Tigerbeetle claims “We also can’t afford to shut down only because of a partial network outage”, which raises some questions! Do they intend to provide total availability? We know it’s impossible to provide both strict serializability and total availability in an asynchronous network, but in a semi-sync model, maybe they can get away with something close, ala CockroachDB, where inconsistency is limited to a narrow window (depending on clock behavior) before nodes shut themselves down. Or maybe they’re only aiming for majority-available (which is what I’d expect from VR–it’s been a decade or so since I’ve read the paper and I only dimly remember the details), in which case… it’s fine, and the clocks are… just an optimization to improve liveness? I dunno, I’ve got questions here!
Thanks for the awesome questions, @aphyr!
The clocks are… just an optimization for the financial domain of the state machine, not for the consensus protocol.
As per the post, under “Why does TigerBeetle need clock fault-tolerance” we explain that “the timestamps of our financial transactions must be accurate and comparable across different financial systems”. This has to do with financial regulation around auditing in some jurisdictions and not with total order in a distributed systems context.
As per the talk linked to in the post, we simply need a mechanism to know when PTP or NTP is broken so we can shut down as an extra safety mechanism, that’s all. Detecting when the clock sync service is broken (specifically, an unaligned partition so that the hierarchical clock sync doesn’t work, but the TigerBeetle cluster is still up and running) is in no way required by TigerBeetle for strict serializability, it’s pure defense-in-depth for the financial domain to avoid running into bad financial timestamps.
We also tried to make it very clear in the talk itself that leader leases are “dangerous, something we would never recommend”. And on the home page, right up front (because it’s important to us) we have: “No stale reads”. If you ever do a Jepsen report on TigerBeetle, this is something I guarantee you will never find! :)
You might also recall that I touched on this in our original email discussion on the subject back on the 2nd July, when I suggested that CLOCK_BOOTTIME might be better for Jepsen to recommend over CLOCK_MONOTONIC going forward, and where I wrote:
You can imagine TigerBeetle’s state machine (not the consensus protocol itself) as processing two-phase payments that look alot like a credit card transaction that has a two-phase auth/accept flow, so you want the leader to timestamp roughly close enough to true time, so that the financial transaction either goes through or ultimately gets rolled back after roughly N seconds.
Hope that clarifies everything (and gets you excited about TigerBeetle’s safety)!
P.S. We’re launching a $20K consensus challenge in early September. All the early invite details are over here and I hope to see you at the live event, where we’ll have back-to-back interviews with Brian Oki and James Cowling: https://www.tigerbeetle.com/20k-challenge
Aha, thank you, that makes it clear. You’re still looking at potentially multi-second clock errors, right? It’s just that they’re limited to the window when the VM is paused, instead of persisting indefinitely?
You know, I’m delighted you say this, because this is something I’ve discussed with other vendors and have encountered strong resistance to. Many of my clients have either felt the hazard didn’t exist or that it’s sufficiently improbable to worry about. I still don’t have any quantitative evidence to point to which suggests how frequently clock desync and/or skew occurs, or what kinds of user-facing impact resulted. I think CockroachDB choosing to up their default clock skew threshold to 250 (500?) ms was suggestive that something was going wrong in customer environments, and I’ve had private discussions to the effect of “yeah, we routinely see 100+ ms offsets in a single datacenter”, but that’s not something I can really cite. If y’all have done any sort of measurement to this effect, I’d love to read about it.
My terrible memory strikes again! Yes, I do recall this now, thank you. :-)
What a mess. I founded #riemann for discussion and support of a monitoring system about ten years ago. Tentatively moved to libera after the whole mess last week, but because so much extant infrastructure mentions the Freenode channel, I left a note in the topic that we’d moved. Freenode nuked the #riemann channel; it now forwards to ##riemann, and the topic is gone. This has got to be a huge hassle for projects with logging and bot integration.
Worth mentioning: this might work some of the time, but in an asynchronous (e.g. real) network, it could be unsafe. The described locking scheme does not actually ensure no two processes hold the lock at the same time. Even if it did, it would not ensure that side effects, like writing to block storage, would be safe. Martin Kleppmann has a terrific overview of why “distributed locks” generally don’t do what people think, and what to do instead: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
Thanks for this feedback. Actually my design already takes this problem into account.
Kleppmann warns that a process can freeze for an arbitrary amount of time due to garbage collection, network problems, CPU starvation, etc. I mitigate this in my design through two ways:
It’s described in section “Long-running operations”.
Kleppmann proposes the use of fencing tokens, which indeed guarantees that such problems don’t occur, but it requires sufficient support by all systems. My design doesn’t provide as strong of a guarantee, but we can make it arbitrarily approach 100% safety by configuring the various timing settings (TTL, refresh interval).
Since 2017 the interest for Clojure dropped significantly, almost to zero, to a 2008 level (the language was created in 2007): https://trends.google.com/trends/explore?date=all&q=%2Fm%2F03yb8hb
This sounds scary. No one would invest in such a curve.
More, the founders / the company behind Clojure were bought up last year by a bank. We all know what this means in other areas.
And so on.
I’ve started learning Clojure a month ago. And these are my back-thoughts on it.
The trends chart for Apache Spark shows interest in that technology near a 5-year low and trending downward. Interest in SQL, Java, JavaScript have been on downward trajectories for 15+ years, and are currently at an interest level metric very near to Clojure.
Would it be fair to call it scary to invest in those technologies?
If you think that’s scary, wait ’til you see the same graph for Java!
https://trends.google.com/trends/explore?date=all&q=%2Fm%2F07sbkfb
Or C#!
https://trends.google.com/trends/explore?date=all&q=%2Fm%2F07657k
Or JavaScript!
https://trends.google.com/trends/explore?date=all&q=%2Fm%2F02p97
Or C!
https://trends.google.com/trends/explore?date=all&q=%2Fm%2F01t6b
Or C++!
https://trends.google.com/trends/explore?date=all&q=%2Fm%2F0jgqg
I used to think Google Trends correlated with language popularity, but these are pretty strong counterexamples.
Right, G Trends shows even React is in a serious downward spiral.
Point taken, thanks for everybody clarifying this.
I don’t. What’s the concern about being owned by a bank?
This also happened to Elixir and it seems to be doing fine?
Google trends isn’t really a useful metric. What’s more interesting is that there are more and more companies using Clojure commercially. For example, we had Clojure/north conference in Toronto where lots of people presented from companies that are entirely built on Clojure stack. There are lots of new companies popping up doing innovative stuff with Clojure every year. Roam Research being a good example.
The communities on Slack, Reddit, and Clojureverse are very active, and constantly growing.
There are now projects like Clojurists Together for funding open source ecosystem and ensuring that it’s sustainable. Incidentally, one of the first things that happened from Cognitect being bought by Nubank was that they started funding open source developers on Github.
Clojure is a mature language, with a large ecosystem around it, and a very active community. It’s not a hype driven language, but it’s very much sustainable and has been for many years now.
Definitely, I choose Clojure/Script as an alternative to JavaScript web dev due to all the above.
However I still don’t feel safe, because of the language popularity. For example on the 2020 Stack Overflow Dev Survey (https://insights.stackoverflow.com/survey/2020#technology) Clojure didn’t hit the list. A presence there would be reassuring.
I see Clojure a one way path: take a deep breath, go down into the rabbit hole (yes, Clojure learning is not simple at all, Clojure is unlike others) and never look back.
This seems like a pretty limited perspective… Learn more languages and you’ll see that Clojure is easier to learn (and better to use) than most if not all.
If the syntax, style, or ideas seem foreign, than all the better! You can write (C, Lisp, Cobol) in any language, and learning the pros and cons of each style is never time wasted.
Clojure is the 9th language I’m learning.
So far I find it so strange like Assembly. And functional programming such a shift when I transitioned from procedural programming (C) to object-oriented programming (C++).
These makes one cautious.
For example, with React was no question to learn it, to invest in. It was the solution for the problem I was waiting for ages.
On Clojure I can’t see really that clear path. Functional programming, for example, is solved elsewhere more thoroughly and in a simpler way (https://github.com/MostlyAdequate/mostly-adequate-guide).
That’s why language popularity would be a good indicator whether to adopt it, or not.
However, on HN, the comments on this same article are more alarming: https://news.ycombinator.com/item?id=27054839
It seems to explain why the language popularity is dropping. Clojure starts as a nice promise, then problems rise, people flock away.
Been writing Clojure professionally for a little over nine years, both on teams of hundreds and as a solo engineer. I can’t speak to popularity, but Clojure has been (and remains!) an exceedingly nice language choice for long-running services and desktop applications. It combines a well-designed standard library, reasonable performance, an emphasis on immutability and concurrency-safety without being dogmatic about evaluation semantics, just the right amount of syntax, excellent JVM interop, and access to a huge swath of Clojure and other JVM libraries. It’s a generally mature, stable language which solves data-oriented problems well. The major drawbacks, in my view, are a lack of static typing (though I’ve used spec, schema, and core.typed to some avail here), unnecessarily unhelpful error messages, slow (~5 to ~20s) startup times, and that working with JVM primitives efficiently is more difficult than one might like. And, of course the usual JVM drawbacks: garbage collection, for instance.
None of this is really new per se. I wouldn’t worry too much about popularity metrics or library churn rate–one of the nice things about Clojure is that it’s fairly stable, and libraries from a decade ago tend to work basically unchanged. After Scala, that was a breath of fresh air for me. What I’d ask, were I in your position, is whether the language’s ergonomics, libraries, and speed are suitable for the kind of work you’re trying to do.
My team’s been using Clojure for around a decade now, and things have only been getting better all around in that time. I think the most important part is that there are a lot of companies using it nowadays as their core platform. There is a lot of commercial interest in keeping the language and its ecosystem alive and active. I don’t think Clojure will ever get big like Java or Python, but I really don’t see it going away either at this point.
It’s also worth noting that Clojure can be sustainable with a smaller community because it piggy backs on JVM and Js runtimes. We have access to entire ecosystems from these platforms, and can seamlessly leverage all the work from the broader community.
I’m not so sure about Google trends as real data point… but there seems to be less buzz, but people are still using it.. and I don’t think there ever was a real hype.
I had noticed that my personal interest had diminished a bit and when most of the people from the irc channel migrated to Slack I didn’t join them. Stuff still seems to get regular updates and as just a casual user no Clojure release really excited or disturbed me - that could be because I’d neber used it to its full potential (likely) or that they were just iterating in small steps and not being revolutionary (also likely). I don’t think I’ve had to do meaningful changes over the years to the codebases I started between 2011 and 2013 and they run on the lastest Clojure version…
I don’t think the author seems to be aware that the people fighting do not care about losing the rest of the world.
They do not care about driving out engineers that don’t share their politics, they do not care about driving out engineers that just want to ignore politics, and they certainly do not care that by picking their sides as they have they’ve grouped a lot of good people in with genuine bigots and assholes of the highest caliber.
They will burn down free software to save it from itself, willfully ignorant of the stripmining of the ecosystem by small and large companies alike as long as they pay the correct lip-service.
Again, the author is trying to apply logic to actions and a rabid progressivism that have, at best, a distant relationship with rationality. They don’t even care about obvious fact-checking and games-of-telephone, because it doesn’t matter once they’ve whipped themselves into a frenzy. They cannot be reasoned with and the attempt isn’t worth the effort or risk; the juice isn’t worth the squeeze.
That boat sailed for a number of reasons. I miss it too, buddy. :(
And yet, a big talking point is “hey, there were never any good old days”–because people have to rewrite the past to support the actions in the present to get to a desired future. Old as time.
Ding ding ding, we have a winner. A huge dimension of all of this is a power grab–as long as people like RMS are around, you can’t be the new RMS, so he must be destroyed. As long as there is an FSF, you can’t be the new FSF, so you must seek to destroy the FSF. As long as there are people in power, you can’t be in power, so you must destroy those people in power–and right now, politically, what are the best tools for that? What rhetoric is in vogue?
This is obvious to any student of history or revolution.
(a cynical/troll-ey point would be to make the same observation about systemd/linux, but that’s neither here nor there.)
~
I feel bad for this author, and I feel worse that I think he and I are probably sharing the same lot. To anybody else who feels similarly: just go away. Walk away from this culture war stuff, give wide berth to both heretic and inquisitors, and don’t try to understand the madness–just treat it as such. Go write code, go make things, and let the passions of the time deaden and pass. There’s nothing here for you but argument and suffering. Don’t risk your job, your livelihood, or your friendships.
Return after the storm and build anew.
Sock, remember when you found out that I was disappointed in your conduct on lobste.rs and you took that seriously?
I’ve been reading your comments for something like seven years now and I remain consistently disheartened. Engaging with you is the missing stair of this community, and I’m not going to follow up on this thread, but I want you to know that your oh-so-civil, “trying to do better”, “just raising questions”, “why won’t you be rational”, socially regressive trolling is a big part of why I don’t spend more time here, and why I caution people who ask me for invites to lobste.rs.
You’re still doing it, and I sincerely wish you’d stop.
Nice link. I don’t know if you meant to highlight them, but there are some gems, like:
@friendlysock “Missing stair” is a political shibboleth of the same political faction that finds RMS unacceptable. It’s a metaphorical way of calling you an abuser, just like how they call RMS an abuser. The same goes for mocking your civility when commenting, or labeling your words as trolling. These are political attacks, and I would encourage you to view them as such, rather than doubting yourself and worrying that you should change your behavior.
You are a dramatic individual. I’ve really tried to give you a chance, but I just can’t take any more of your whining. Seriously, you’re doing the same thing you accuse others of doing. The world isn’t ending, moral panics have existed for thousands of years.
Please follow your own advice. Seriously.
Well, yes; and they have also caused great amounts of harm and hurt to innocent individuals. Shrugging it off as “oh, these kind of moral panics have happened before” doesn’t strike me as a very strong rebuke. We could substitute “moral panics” in your quote with all sorts of things (theft, murder, war, rape) and have exactly the same argument.
(I don’t disagree that ’sock is perhaps a bit overly dramatic though).
That doesn’t make moral panics good, or unworthy of political opposition. I’m not sure if @friendlysock’s dramatic rhetoric is maximally politically-effective, but he’s not wrong to express the thought.
What exactly do you think “culture war” means in our society? What do you think that you’re defending?
To be the child who points out that the emperor is naked: Christianity and the entire family of Abrahamic religions rest upon mistruths, propaganda, ahistorical claims, and a large amount of what we’d now call human-rights violations and war crimes. What do you actually think you mean when you say that you’re “sharing the same lot” as the author?
Sorry, but to be blunt: Do you actually produce Free Software? It’s cool if you don’t, but you shouldn’t expect pigs to take chickens seriously.
I do, and I keep those activities quite divorced from Lobsters.
Look, I’m about as atheist as they come but going on about historial war crimes when a religious person wants to starts a broader conversation about inclusivity is extremely inappropriate. This isn’t /r/atheism.
The key part,
There is also Tribes which provides a custom-hosted version (I run mine here).
See https://jointhefedi.com if you want to quickly try out the Fediverse.
In all fairness, Mastodon has one of the least spec compliant ActivityPub implementations out there. It gets stumped with a lot of valid payloads that were generated by other services inasmuch as having to implement Mastodon’s quirks is mandatory if one wants to do development for the fediverse.
Maybe an unpopular opinion, but without Mastodon ActivityPub would be living the life it was living before, used by dozens of nerds.
Of course that’s not a proper discussion point to some, you may or may not like its ideas and technical features, but to me it was kinda useless when it was only identi.ca and statusnet and whatnot. I’m saying this as someone who was pretty involved in many FLOSS projects at the time. Utterly useless. It was Twitter if you wanted a thing like this and 90% happened on mailing lists and IRC anyway.
Oh, I fully agree that Mastodon is overall a force for good in the Fediverse, at least in the fact that it made it popular with the non technical crowds, but I still wish they would work harder at some things related to ActivityPub compliance. Probably my own service will not be super compatible with it, as it skirts webfinger - something that Mastodon can’t do user discovery without. :(
I didn’t look into it very deeply, so can’t comment if they made some shortcuts for time to market, or enable stuff that would’ve been hard to do, or just because they were careless or simply didn’t care…
From my perspective they’re prioritizing the features that makes them a better micro blog platform than the features that makes them a better ActivityPub one.
I would like to say that being the major player in this niche they should take their responsibilities in this regard more seriously, but in the end they work on what they enjoy more, and that’s absolutely fine.
Agreed. I am present on a few Mastodon instances but my personal instance is Honk which is a very opinionated and pure (I guess?) ActivityPub server/client/thing
The servers recommended on that page are some of the most notorious in the fediverse, notable for hosting bigoted shitheads and having nazi-friendly moderation policies.
If you sign up on them, you will find yourself blocked by basically all fediverse instances with active and competent moderators.
This was one of the reasons I stopped using the Fediverse. I don’t like the concept of full-on instance-bans to begin with (something like warnings for out-going actions and filtering for unrequested ingoing actions would be more appropriate). I’m not sure if federation necessarily has to lead to fragmentation, but some people seem to accept it as a necessary tool and don’t care if anyone has a different opinion. In my case I wanted to hear what people on the spinster server had to say, but it was blocked on the instance I was on (ironically this made me go out of my way to listen to the points of radical feminists, which I don’t think was the intention).
Part of the problem with Mastodon specifically is that it has inherited a lot of the worst Twitter-culture by presenting itself as “Twitter with better moderation”, while paradoxically decentralisation is usually understood as a means to avoid being shut down by a central authority. Then again, it all ties into more fundamental issues with the Fediverse and how it presents itself as “each server is it’s own community”, while at the same time I don’t care about what server another person is using. The only thing I am interested in is the moderation policy and how well they administer the server.
The part of the Fediverse I still remain hopeful for is Peertube.
Instance bans allow for coexistence without cohabitation. You always have the choice of choosing your own policy domain/deferring to someone else. Forcing all nodes to be wide open would remove a lot of point and cause unnecessary annoyance.
To a degree yes, thought I’d still rather that not be the case, because I rarely agree with someone on everything, meaning I have to administer an instance myself. But it is not only a personal issue, with instance bans threads are also fragmented, so depending on your perspective, you might unknowingly not see the entire conversation going on, leading to more confusion than necessary.
Instance bans are sledge hammers that are applied to eagerly (I do think they make sense for actual spam servers). Maybe the situation has improved since, but I remember there only being three states:
Where I think that there should be more going on between 2. and 3.
There are degrees between 1 and 2, at least on mastodon. Admins can “silence”, meaning posts from that instance won’t show up in the federated timeline by default. If I’m not mistaken, there’s also “mute”, meaning interactions from that instance won’t be shown to the muting instance unless there’s a preexisting relationship between the actors.
I should also note that instance bans are not really a thing– you can mute an instance at a user level, but your data is still sent there and you must trust that server’s administration.
this seems to be an issue even if the instance isn’t banned. I see this happen with my small instance, where viewing the thread on the hosting instance (or from an account on another instance) shows different posts, and I’m pretty sure the missing posts aren’t from blocked instances.
iirc mastodon will fetch replies upthread, but not downthread: that is, if the chain goes X -> Y -> Z, and your instance is made aware of post Y (someone follows the poster, it gets boosted, whatever) then it will fetch X but not Z. this is why some people have a norm to boost the last post in a thread, as opposed to the first. this isn’t a technical limitation, since pleroma (the other big fedi server) will fetch the entire thread.
of course, in either case, if one of the posts in the thread is private and you don’t follow the person you’ll just break the thread entirely, but there’s not much that can really be done there.
oh wow, that’s confusing. :|
Due to how ActivityPub works, you need to have near-ultimate trust of an instance if you wish to federate with them. If you believe the admins are bad actors, using acceptance of harmful ideologies as a proxy for that, then you can’t trust them with your user’s data, and must defederate.
This isn’t necessarily against what the blockers wanted! What is called “censorship” on the fedi is usually about protecting their own users. Trans folks don’t want to have to see the same tired take on trans exclusionism for the fifth time today, nor do they want their posts to be seen by those folks.
As you discovered, there was absolutely nothing stopping you from finding out more from the spinsters, and nothing stopping you from making an account there either, right?
If we think decentralization is the key to freedom, then we can’t stop short of free association.
What do you mean by “trust them with your user’s data”? Is there something a server can only access if they are federated, that a “blocked” instance couldn’t see via it’s public feed?
I get that an instance would decide to mute another instance by default, but if a user explicitly requests to receive data, why should they not be able to interact?
A user’s private posts are always federated to any instance that has a single actor subscribed to it. That means that instance is storing a user’s private posts. If the admin’s a bad actor, they could see the private posts even if they’re not authorized to normally.
so private posts are not actually private, much like Facebook, though for totally different reasons. great.
Yes. Unfortunately, if you view private data disclosure as a security issue, Masto/ActivityPub is less secure than a centralized platform.
There’s hopes that CapTP will solve many of these concerns.
It’s similar to plaintext email, no? As long as the plain text traverses a server somewhere it can be read by the server admins.
As far as I know, end-to-end encryption isn’t supported by AP.
yes, but email doesn’t use the term ‘private’ anywhere. I think many(most?) people understand that email is not useful for HIPAA or other things where privacy matters.
Many people sign up for things with firstname @gmail.com, and then claim the account owner “hacked” them. Many people think that companyname @somecustomdomain.com means you work for them. Many people think that anything @someother.tld means you actually meant @someother.tld.com.
I don’t think most people understand anything about email.
I think with all things, it’s complicated. I’m sure people in their 70’s and older who have very little exposure to email are likely not very versed.
For the average professional that is legally required to care about privacy, then I think they mostly have the understanding that email != private communication.
Developers SHOULD know better, but they still do stupid things with email, because it’s the only thing you can reasonably assume someone has. (like login with email, use email for password recovery, etc) There are sane things you can do to help mitigate these things, like single use tokens, etc, but.. I’m sure there are still tons of code out there that doesn’t do these things.
I agree email ADDRESSING, which is what you mostly are referring to, is full of assumptions and mostly none of them can be assumed. The only thing you can mostly assume from user@domain, is that the domain admin at some point thought that user should exist. :)
I agree with you it’s a bit of a branding problem.
I’m just so used to the store and forward model of email and NNTP that I just applied that model to the fediverse too. And I have not heard anything about E2EE in the “mainstream” Fediverse.
Citation needed.
One of the servers recommended on that page, gleasonator.com, actually was created by someone that experienced bigoted behavior from mastodon’s toxic and neoracist moderation policies: https://blog.alexgleason.me/gab-block/
As a queer person and regular fedi user, I concur that these servers are notorious. Multiple accounts from shitposter.club harassed a trans friend of mine just this week because they posted a selfie to their timeline. Freespeechextremist’s users have a habit of sea-lioning their way into my mentions; I think the last one was an extremely tedious “wow aren’t gay people bigoted” monologue mixed with Q-anon rants. Freespeechextremist.com, shitposter.club, spinster.xyz, and glindr.org (another Alex Gleason joint) all have the dubious distinction of being on the relatively short mastodon.social and mstdn.social blocklists for hate speech, harassment, and transphobia. With the exception of mstdn.social, this is not a general-purpose instance list: these instances all share moderation policies aligned with reactionary views on gender and sexuality.
That transphobic bigot wasn’t ejected by mastodon’s moderation policies. Mastodon is the service, moderation responsibilities lie with the server admins.
That transphobic bigot was ejected by todon’s moderation policies, because, as he so proudly proclaims, his bigotry is contrary to the server’s stated goals and aims.
Those goals, aims and indeed the moderation policy are clearly stated on the server:-
Gleason is a bigot. That bigotry was noted by other todon users (I number myself among them) and he was shown the door.
Gleason is a peddler in transphobic bigotry. Its an essential part of who he is. His “sex-essential” “gender-critical” nonsense is a paper-thin mask for hatespeech against a marginalised element in society.
You have now defended him, Freedom of Speech Zealotry, White Supremacists and transphobic bigotry up and down this story, which you appear to have posted just to link to the aforementioned listing site for hatespeech and bigots.
You can put all the ten-dollar words you want all over your post, I can say without hesitation that you’re both posturing and a troll.
All you are doing, in your anonymous account to boot, is to accuse other people (Gleason and now me–that are not anonymous, neither are afraid to hide behind a mask) without evidence and without engaging rationally (as in without refuting the central point) but merely with politically loaded language (as in resorting to thinly veiled ad hominem).
Lobsters would be better off without such toxic comments expressing actual bigotry, and I assume on good faith that you did not intend that, and is writing in a state of not being with a sound mind - so I suggest you take a break.
They out themselves as a transphone one sentence into the blog post. I’m sure many transphobes think being told they’re a transphobe is toxic.
They also had no problem joining Gab and admitting that it’s full of, their quote, “literal nazis” in the same article.
Friendly warning: anything anywhere that mentions transphobia or nazis becomes a bozo bit here on Lobsters. Don’t try to argue semantics, don’t appeal to actual text or logic or history, don’t waste yours or anybody else’s time–just steer clear of it and save those cycles for making things or engaging in communities with more mature discussion capabilities.
You’re too wise for this place
Wisdom is what you get when do something really stupid but take notes.
…I’ve taken a lot of notes.
“Women’s sex-based rights” is absolutely a dogwhistle for transphobia, and if you look at what he wrote in his own words he says that ‘transgenderism [was] first popularized on Tumblr’ (?????), links the “TERF is a slur” page, and says “transgender ideology is fiction”. He’s transphobic through and through.
I’m also extremely unconvinced that there’s no way to prevent people from being actual literal Nazis while not hurting marginalized people. Like, if someone was to come in to the comments section of a Lobsters post and say “by the way, I think we should kill all the Jews”, they’d get flagged and banned, right?
And you’re ignoring the fact that constantly seeing people say that they think people like me (hi, I’m trans) are abominable freaks that are better off dead, or even ‘just’ mentally ill people who need to stop pretending, is likely to push me away from a place. This is going to happen with any sort of ‘free speech’-focused Masto instance: the bigots will migrate to your instance because they get kicked off elsewhere, and the people who don’t want to have to deal with bigots are going to go elsewhere.
And, going back to the list, it’s not just gleasonator. As someone who’s used Fedi for several years, every single one of those instances aside from mstdn.social is one that I’ve had shitty experiences with. And it’s not a coincidence that mstdn.social is the only one that’s described as not allowing racism or sexism!
Yes. And it’s happened: a few years ago a comment on a story about net neutrality attempted to use that to explain why the U.S. should commit genocide in the middle east. I deleted it and banned the author.
… now that’s a leap.
branching off from the thread, that quote is infuriating. they make the following argument:
therefore:
i.e. propagating the speech of who people arguing for an ethnostate and committing real-life violence against minorities is somehow beneficial for those same minorities. fucking inane.
My comment below is terribly off-topic, I think.
A transphobe is someone who fears or has a negative perception of trans people. Supporting “women’s sex-based rights” is the same as saying that people born with female sex organs have different rights than trans people who are women. That is a negative perception of trans people who are women. Saying that women who were born with female sex organs have different rights than trans people who are women is, precisely, transphobia.
Your comment is a low-effort attempt to deny that basic fact; if you recognize that trans people exist, saying they should be denied affordances that cis people have is clearly a manifestation of transphobia.
That claim is so obviously false and inflammatory that I have flagged your comment as a troll. I’ve done you the courtesy of leaving this comment explaining why even though my own comment should rightly be flagged as offtopic. That’s because I’m assuming some good faith even though the obviously false and inflammatory nature of your comment makes me think that’s vanishingly unlikely.