Threads for ec
-
I just have an unencrypted backup at a physically secure location. There are existing legal procedures for e.g. allowing your next of kin to come out and have a locksmith open a safe and it doesn’t make sense to over-complicate from that baseline expectation. If you don’t have a safe, or the physical security of your property isn’t sufficient, then I’d go with something like a safety deposit box.
If you use GPG or shamir secret sharing nobody is going to figure it out! Your secrets will die with you. The physical-world legal system is good, and has already solved this! Use it!
-
-
-
There are enormous penalties for law firms that leak privileged client information. As a result, even small ones tend to have decent infrastructure. I’d recommend using that for recovery keys and instructions rather than for the data because it’s a bit painful to update these things.
-
A have a notary acquaintance and let me tell you, their security is… not that good?
To name just two glaring issues:
- Remote management from multiple IT companies over VPN.
- Landlord having full access to the premises.
As for safety deposit boxes, they are basically uninsured. If you miss the payment, contents will disappear who knows where and in the US, they are mostly unregulated.
-
Out of curiosity, are you referring to the notary’s or the law firm’s security? Or are they a notary who work for a law firm? In the USA my experience is that the regulation and licensing for notaries is pretty weaksauce, basically “you are confirmed to be a real person with up to date contact info, and you pinky-swear not to cheat”.
-
-
-
To an extent, it depends on how tech savvy those you are sharing your secrets with are. Like, I have a friend who used shamir secret sharing to lock themselves out of social media via some access rube goldberg machine with a way for a threshold of friends and family to either access it themselves if they thought it was necessary or let them back in, and it actually somehow worked for their purposes. But that person had a circle of friends/family who were mostly in IT already, so.
Edited to add: maybe try portions of this out with some real but low stakes thing first to see whether the bits do work for you - eg using SSS as a way to do quorum based access to some thing.
-
allowing your next of kin to come out and have a locksmith open a safe
…
If you use GPG or shamir secret sharing nobody is going to figure it out
Seems in either case, you’re assuming the next of kin will hire an expert to figure it out?
I worry that a physical safe will end up on one of those “Hey Reddit, I just bought a house and found a locked safe in it!” threads.
Feels like whatever we do, it’s most important to have great instructions that are clear with references about who can help resolve them.
-
Agree on instructions, but some instructions are easier than others. As long as your next of kin know that you have a safe, the rest is solved with a phone call by the executor of the estate.
Compare to: which telephone number does a non-technical person call to hire someone to help them operate a shamir secret sharing algorithm?
Also: if this plan did “backfire”, and thousands of redditors started going through my personal files and talking about me decades after I die, I think that would be kinda neat?
-
Hopefully any of numbers that are on the instructions I included? I know dozens of people who can figure out GPG for example (or more complicated technical instructions if included), I’m sure they already outnumber locksmiths (or certainly will by the time I’m dead).
But yes, if we’re in it for a Reddit treasure hunt, then we might need a whole other conversation path. :P
Also one more note: Avoid safety deposit boxes, they’re unsafe and totally unregulated. There’s dozens of stories of people losing their contents.
-
-
-
The lawyer solution could work, especially since he is bound by law (is he?) to follow through with the arrangement. However:
- how much would that cost? I bet the sum is non negligible, and would cost many more times than any other technical solution that doesn’t involve lawyers;
- what if you don’t have access to such lawyers? if living outside the US and other “rich” countries, such a service would be perhaps not even be accessible to the “average” person, and perhaps it’s not even covered by the law, thus you’ll have to really trust your lawyer;
- how easy (and costly) is it to update your instructions?
-
Most legal firms will hold your will for a fairly negligible amount and you can put passwords and so on in a sealed envelope with that for no additional cost.
-
-
Anyone doing family law. There are many in any town in the UK, though this probably varies a bit between countries.
-
-
-
-
-
-
I looked into a safety deposit box some years ago, and it was absolutely ridiculously expensive, on the order of a significant percentage of my income. Then there’s horror stories about banks losing stuff, throwing it away, or letting random people access it. Not sure how common those are, but considering the abysmal security basically everywhere, I don’t see how this is an option.
-
-
This sounds like the computer you want, but what makes you think it’s the computer “we” (the majority of people) want/need? It sounds like you want a typewriter with a little display on it somewhere. We had those 30+ years ago, they sold ok, but when laptops appeared people found they liked those better. (A small niche market of writers did form a cult following of these devices.) What evidence do you have that people would want to return?
Your concerns are all about text entry. Most people are not writers or coders. Many people create things with computers that are not textual — drawings, photographs, movies, animations, music — and require or benefit from large color screens. Or that are textual but based more on manipulation of lots of displayed text, like spreadsheets or page layouts.
The goal of “build a computer that will last for 50 years” has a certain appeal, but why would I want to use a 50-year-old computer? If I had a perfectly working PDP-11 in my house it would make a cool objet d’art (i love the front panel) but I wouldn’t use it for anything serious. Twenty years from now would I be satisfied using a 1992 PC/Mac instea of whatever is state of the art then? Jesus, no. I know some retro enthusiasts like playing with these old systems, but good luck selling them to a mass market.
-
I think a little more technical flexibility would take us a long way towards some of the goals outlined in the original “a computer to last 50 years” article without sacrificing so much (actually, practically everything) in technical terms.
For example, lots of laptop discarding happens around trivia: broken keyboards that are too expensive to replace in service and too difficult to replace at home, broken displays that are super expensive to replace because the parts are hard to source on an open market, dead batteries that are hard to replace because they’re soldered and/or the cells can’t be sourced anymore.
Lots of these things are technically trivial to resolve, with minor (sometimes 0, in the first two cases) compromises. The PR machine that made the bezel fashionable would have a field day selling a slightly thicker laptop that’s sustainable and reduces waste, so users (and corporate acquisitions departments) would absolutely want them, whether they realize it or not.
Lots of computer waste in general is in the form of poor-quality peripherals. Back when laptops were more expensive and desktops were the norm, you’d see discarded keyboards all over the place. Twenty years ago, sixty bucks or so got me a basic, but good quality Logitech membrane keyboard that I still use today – the only visible problem is that some of the keys are slightly deformed, and most of the keys no longer have visible labels after twenty years of constant use.
An embarrassing number of modern computers give out mechanically or electrically before they become obsolete. Back when I started doing freelance work full time I had no idea if it’d work, so I didn’t want to invest much, and my first work machine was a seven year-old i3 laptop that worked just fine. Using a seven year-old machine for development in 2003 would’ve been unthinkable.
So I think that, with Moore’s law having been dead and buried for like a decade now, just mandating basic repair rights and/or educating users about peripherals would take us a long way towards long-lasting computers. Not fifty-year computers, but “a computer that you get to keep for ten years” is totally doable, and keeping some of the bulkier innards for longer than that is doable, too.
And you can just run some fancy distraction-free app on it to make it feel like a typewriter if that’s your thing, I dunno, I just use emacs :-).
-
The part of this that actually does sound appealing to a large-ish swath of people is figuring out a way to do durable media. If I could buy a hard drive that I could fill with family photos and keep on a book shelf for 15-20 years, that would be huge! As it stands, I suspect unless I’m quite lucky I’ll either not be able to read any data at all, or will have some loss of data due to corruption.
This situation directly drives the phenomenon where most people keep all their data on some corporate server in the cloud for safety. If your data was “paper book” level safe sitting on your shelf, then I suspect a lot of people would skip their $10/month Dropbox bill.
The M-Disc kind of gets at this, but it’s write-once-read-only media (WORM), and thus more of a headache than most people are up for. Also, they’re no longer manufactured, computers don’t have drives for them, etc.
Ideal solution might be a cassette of optical media, with some firmware that can create a “write-several-times, read-many” interface using something like a log structured file system that knows how to defragment data to allow you to replace expended discs. If the thing could accept Apple Airdrop, it could be a pretty slick addition to create some durable memories in a mobile/ephemeral content-consumption world.
-
From what I’ve heard, durable mass media is a difficult and unsolved problem. I’m not aware of anything on the market that can be expected to last longer than a decade. Apparently the current solution is to copy archives onto new media periodically.
I know there are research groups working on things like 3D crystals with microscopic holes burned into them with lasers, but those are still in the lab…
-
Blu-ray is >10 years easily (100s of years theoretical as per ISO testing). If we hadn’t paused R&D due to streaming video taking off we’d probably be looking at another 5-10x bump in density/cost reduction — which would be sufficient to build an “infrequent write”, multi-disc consumer storage solution that could feasibly store the sorts of data volumes common today.
-
-
As I understand it the issue with CD / DVD longevity is that they use organic dyes, whereas the Bluray is an inorganic phase change that is more stable over time. And yes, an external USB 3.1 version is ~$100.
https://en.wikipedia.org/wiki/Blu-ray_Disc_recordable#HTL_(high_to_low) https://en.wikipedia.org/wiki/CD-R#dyes
-
-
-
-
-
Your concerns are all about text entry.
As a programmer, I get the most power by being able to enter text. I understand that some tools allow people to create effectively on a touch screen: I saw an artists work on drawings, I saw my youngest brother build an animation through what looked like a Bret Victor-esque scripting engine with an effectiveness that seemed to exceed my own skills with an IDE . But to make or modify those, we need text.
Text entry, I believe, is a requirement to general computation.
It is also the hardest thing to sell: while better screens are just better, better text entry methods require learning. That’s why we’re still stuck with QWERTY (I’m not, but this is a source of friction). If we’re in it for the long run, text input is probably the thing we should think about the hardest.
-
Text entry is a requirement. But every computer, just about, supports it. Yes, most keyboards are onscreen and/or crappy. But most people don’t need the absolute best keyboard, for the same reason most people don’t need the absolute best drill or wrench or fountain pen.
Programmers are a niche. A computer optimized for programmers might be cool, but don’t expect to interest the rest of the population in it.
-
-
We haven’t exhausted the skill ceiling on computing systems designed for experts and we all benefit from exploration of this space. The only benefit of building for mass appeal is it’s easier to build a billion dollar corporation around.
The chief reason to build a computer that’s useful for 50 years is to insulate one’s personal computing environment from outside pressure to upgrade. Upgrading should be an aesthetic and financial choice, not a necessity to keep your household functional.
-
-
Is there any value in UUID over a 64bit numeric format like Twitter Snowflake? If so, is there any value in UUID over extending Twitter Snowflake to a 128 bit numeric format?
I am confused why it’s not more common to roll your own own surrogate key identifier numbers to match an application’s particular use case, desired sort order, desired partition method, desired cryptographic randomness, etc.
-
I think rolling your own is plenty common. It doesn’t seem too troublesome to do. The big reason to pack your custom ID into a UUID is interoperability. They’re easily recognizable in plaintext formats and natively supported by a plethora of databases and serialization format with a semantic ID type. I think many people start with normal UUID these days, and then “upgrade” to a home grown semantic UUID once the need arises. New formats the codify standard practice are nice.
-
-
ec
11 months ago
|
link
| on:
Verifying properties of a financial contract in Elixir (question in a forum)
The first response seems right, but I would define this property in the database, not in the application.
I would store this state using the full contract details as primary key, with signatures joined against that particular state. A hash or a surrogate key could be used if the contract state is particularly large.
That way you represent exactly what contract the parties have signed against, and if the contract changes the join condition is false.
If you do that, the desired property is easily proven in the relational model.
-
I was expecting this to be about getting a consistent random distribution across the
[-1.8*10^308, 1.8*10^308]64 bit floating point range.It would be interesting to account for the increasing ULP as the mantissa consumes more and more of the precision
-
Reminder: UUID is just a (really big) number. You can use whatever numbers you want for keys.
For example, if you want sequential keys, but have multiple nodes and want an ability to randomly partition the keyspace, this would probably do the job in only 64 bits – 48 bits for sequence, 5 for partition = 32 partitions, and 11 for node_id.
create or replace function sequential_id(node_id integer) returns bigint language sql as $$ select nextval('seq1') -- sequence, LSB | ((floor(random() * (1<<5))::bigint)<<48) -- partition | ((node_id::bigint & 2047) << 53) -- node identifier, MSB $$; # select id, id&281474976710655 seq, (id&8725724278030336)>>48 part from k order by seq asc; id | seq | part ---------------------+-----+------ 1303510617147048054 | 118 | 23 1318991740866134135 | 119 | 14 1364590687093260408 | 120 | 16 1359524137512468601 | 121 | 30 1302947667193626746 | 122 | 21 1344605963746803835 | 123 | 9 1335598764492062844 | 124 | 9 1333909914631798909 | 125 | 3 1297318167659413630 | 126 | 1 1318710265889423487 | 127 | 13 1337287614352326784 | 128 | 15 1363183312209707137 | 129 | 11 1338694989235880066 | 130 | 20 1328561890074296451 | 131 | 16 (14 rows) > bitstring(8725724278030336) "0000000000011111000000000000000000000000000000000000000000000000" > bitstring(281474976710655) "0000000000000000111111111111111111111111111111111111111111111111"This is not necessarily better than UUID, but it might be for your use case! Put another way: the designers of a specific UUID implementation may have been optimizing for your specific use case, but more likely they weren’t.
-
There ought to be a standardized “CGI 2” that works similar to a mashup of CGI + Lambda – a single connection per process, but processes can be reused, with the application server starting up new processes to handle connection concurrency.
I’d use a simple framing protocol over stdin/stdout that falls back to CGI 1 if the right
CGI_VERSIONenvironment variable is not set.Conceptually doing something like this:
> {"request":"GET","path":"/index.html","headers":{"Content-Type":"text/plain"}}\n < {"code":200,"message":"OK","length":13}\n < Hello world.\n\n > {"request":"GET","path":"/page.html","headers":{"Content-Type":"text/plain"}}\n > {"code":403,"message":"OK","length":0}\n-
-
FastCGI is more typically implemented as just a transport, so you need to arrange for a daemon to be listening on a given socket. That breaks the magical “drop files in a directory” type of a workflow — and in most cases you may as well just deploy nested HTTP.
All of this is about developer experience, so I think it’s important how the typical application server implements it.
Examples: https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#the-fastcgi-transport https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#example
I guess maybe I just want actual FastCGI support in the servers that I use? Hmm 🤔
-
FastCGI on Apache works the same as dropping files in (at least in my config, i guess it could be different for other people), it manages the worker processes for you starting and stopping as needed.
-
-
-
-
-
-
This feature is glorious. Will save so much time and bugs.
-
-
Rather than attempting to estimate the entropy of a scheme, wouldn’t it be better to just define a scheme that represents randomness directly in the encoding? If you target N bits of randomness, then you will not fall into these sorts of traps if you directly encode those bits in a symmetric number scheme, such as base16, base32, or diceware (base
~12.92).The math is sort of interesting, but I’m not sure why anyone would in practice want to do anything other than directly encoding randomness.
-
I became interested in this because I was trying to develop highly memorable passwords. For example, you might want to choose a random grammatical English sentence as your passphrase. A natural way to do this might be to choose a random parse tree of a given size, but if you do that you’ll have some duplicated sentences (with ambiguous parses).
-
There are also the EFF word lists, which are designed to be memorable, real, and chosen by dice roll. I particularly like the list where each word can be uniquely identified by its first three letters: https://www.eff.org/deeplinks/2016/07/new-wordlists-random-passphrases
-
-
-
-
-
To me, the relevant thing that would merit this level of caution is approximately (feel free to substitute more relevant real world nouns): a bad actor, Dolores Umbridge, has somehow taken over control of Hogwarts, and most of us are very concerned about that. How can the Order of the Phoenix communicate, calculate, and scheme against these Death Eaters without blowing our cover?
The most useful “Dolores Umbridge” defense is a maximally simple protocol based on commonly available software and gear. Convenience of use is relatively unimportant (the order of the phoenix is extrinsically motivated), but safety and correctness of use (“opsec easy mode”) is very important.
How come so many people are interested in what I would categorize as “casual laptop use that can also defend against 3 letter agency attacks”, but few are working on a widely disseminated protocol for solving the Dolores Umbridge problem?
The point is, it seems like if you push very far beyond “properly patched and configured laptop”, you quickly get into contingency planning more than dealing with a real day to day threat landscape. Maybe that’s worth a contingency plan, but I’m not sure it’s worth considering as a day-to-day threat.
-
What git command line do I run to remove a maintainer?
-
-
-
-
It’s not authoritarian to think that people who aren’t engaging in good faith shouldn’t be running core tooling projects used by the entire software industry. Applying a fork doesn’t solve the issue that a toxic person is leading a massive community effort.
Furthermore, this isn’t about solving it for me – I know how to use git already. It’s about increasing accessibility for newcomers, who won’t know how to apply patches and recompile.
-
So long as you only think so.
Where can I see a single example of engaging in bad faith, or any toxicity for that matter?
It could be argued that core tooling shouldn’t change at all, and a change like this would confuse the documentation, or break things. Though this has happened already with the master → main switch, as well as with some changes to the porcelain. git is rather bad from both viewpoints.
-
-
This is under the confusing of documentation. The more ways to do it, the more confused it is. Changing terms in any place would also be a source of confusion. I admit to not having read it in detail, but no miracle is possible.
I mostly just scanned it, early on found out it literally lies (“everyone” means “people I agree with”), and figured out it’s just someone publicly moaning, so not worth the attention.
And then there was this comment where someone disrespects other people’s work, of course.
-
-
-
-
-
Well then you better put in the effort to get your fork into distribution repositories.
-
-
-
-
Yes it is; the way the term “shouldn’t” is used there presumes it is or should be in your authority.
-
Applying a fork doesn’t solve the issue that a toxic person is leading a massive community effort.
Sure it does: if you do better, people switch projects, and the origin of the fork stops being a massive community effort. How many Hudson developers are there today? How many Jenkins developers are there today?
How about Gogs vs Gittea?
-
-
-
-
julia> sum(0:99) julia> reduce(+, 0:99) julia> maximum(ls) julia> max(ls...)
-
I don’t think I understand what richer colors would look like. Is this like real life colors that I would see in day to day life, or is this some sort of new purple that I’ve never witnessed?
-
Real-life colors. There are various ways to turn a triple of values from 0 to 255 (or floats from 0 to 1) into an actual color that humans perceive. sRGB, which is the traditional color space, would require negative values to represent some of those colors, and since the sRGB components traditionally correspond to how much of red/green/blue you mix in, this is clearly meaningless (you can’t mix in a negative amount of a color!).
Display-P3 can represent more colors without going negative, as you can see in this image. The extra colors are colors you’ve seen already, it’s just that sRGB has no way to name them. If your setup is such that you can actually see the WebKit logo in this image, then that’s a color that you cannot represent in CSS.
It’s a bit hard to talk about for the same reason that you can’t show what a 4K image is like on a sub-4K display.
-
-
What about partitioning?
- Partition your table into date buckets
- Drop the partition(s) when they are no longer needed
- Never have to explicitly issue or replicate a
DELETE
-
-
I had this problem and solved it with partitioning as well. I think time-series data is prone to this problem, which is why there are time series databases and plugins for ordinary relational databases. Outside this circumstance, I just don’t seem to run into bulk data removal that often.
-
CGI has pretty terrible performance since it launches a process per HTTP request. That’s a common reason for people’s blogs to keel over when slashdotted. Do you have plans to adopt a faster API? (Theres FastCGI, but I think often people just build a little HTTP server into their blog engine and then proxy their main server to it.)
-
-
I’m undecided on that point. I may end up doing some static generation in the background / under the hood, where the static pages are essentially just a pre-rendered cache. For now, CGI is a pleasure for prototyping. Just move the new thing to /cgi-bin; there’s no need to restart a service etc. when something changes.
The output from this program is highly cachable; the only time it is going to change is on a database update.
-
It’s definitely not ideal. fork/exec takes about 1-2ms for an AOT compiled program with a small runtime. You should expect to be able to get something like 10-100 reqs/s per core with CGI: I see about 100-150 req/s for some trivial C cgi script on my dinky laptop with XAMPP.
Historically it was worse for Perl, Python, etc partly because requires/imports/includes in dynlangs were typically awfully slow. (Still are for most, dunno if state of the art moved on.)
-
-
Stream ciphers that use cryptographic hashes have a risk of running into cycles. This is when calling the transform function on the state will eventually go back to a previous one. In order to mitigate this, we can use a block cipher instead.
The “proper” way is just to use a bigger state. The usual rule of thumb for overcoming the birthday paradox is to double the number of bits, so I would expect 256 bits of capacity would be sufficient for a 128-bit security level. But then you can’t use raw MD5 as your sponge function, which sort of defeats the author’s purpose of “can we make MD5 do everything”.
The author mentions ChaCha20 as an alternative permutation. I’ve wondered before how dangerous it would be to use that as a sponge function, because it’s relatively simple and has a nice state size of 512 bits: split it in half, and you can absorb/squeeze 32 bytes at a time, with the other 256 bits being your capacity. Being no more than an armchair cryptographer, I’m vaguely aware that you can’t just turn any permutation into a sponge and call it good – but I’m not qualified to say what a permutation needs to have in order to make a good sponge.
I see some rough edges in the article (most glaring one is that the described block cipher isn’t really a block cipher). But I’m really happy to see people learning about new fields and “shaking their sillies out” with respect to cryptography.
-
-
Ah, I found someone to ask questions to. Let me bother you a little now that I’ve caught you.
I think when performance is not a top priority, you can just use 1 byte or even one bit to absorb and keep the rest as your capacity. This should at least reduce your chances of messing up severely. With ChaCha you can also change the number of rounds to play around with the mixing quality/performance trade-off.
I’d like to ask you why the thing I called block cipher isn’t really a block cipher. I’d love to improve my terminology and correct any mistakes in my mental model if you have time to elaborate.
-
What you have right now is a single function
get_block(key, counter)that returns a block of pseudorandom bits. For a block cipher, you need a pair of functions:-
encrypt(plaintext block, key)that returns the encrypted block -
decrypt(encrypted block, key)that returns the plaintext block
What you have mimics a block cipher used in CTR (counter) mode, which is a commonly used mode. But block ciphers can be used in other modes as well – CBC is a common one.
-
-
-
-
One issue not addressed here is that plain text protocols are typically hand implemented (a recursive descent parser), where binary protocols are often machine generated (protobuf, grpc, etc.). HTTP is in theory parse-able by a number of parser generators, but in practice it and other text based protocols are hand implemented, leading to bugs, security vulns, and other problems.
The ideal seems to be something like HTTP, where a client/server can fall back to 1.1 if 2.0 isn’t jointly supported, and is general enough to support most anything you’d want to do with a protocol. Similarly, most machine generated formats like protobuf have a text based format as fallback.
-
And this, in turn, leads to security problems. Parsing untrusted data is one of the biggest sources of security vulnerabilities. The easier a protocol is to parse, the easier it is to secure. Most binary protocols require trivial binary pattern matching to parse. About the only check that you ever need to do is whether an offset is contained within a packet and even in an unsafe language it’s pretty easy to abstract that away into a single place. Binary protocols can often just use the header information as-is and parse lazily. Flat Buffers, for example, just uses the wire-protocol message in a buffer and provides accessors that do any offset calculation or endian conversion necessary. The cost of parsing is so low that you can re-parse a field every time you access it.
-
I feel like if the issue is that text protocal are using handcrafted and unsecure parser, the ideal would then be to have client/server that only use 2.0. Allowing fallback is status-quo and doesn’t fix any security hole.
The way I see it, the ideal would be to have some non-default dev-mode where text protocol are enabled, and/or those text protocol only support the simplest subset of robust features.
-
Grpc is huge. It depends on a large amount of hand written parsing code. Using it is unlikely to reduce the amount of hand written parsing in your system.
I don’t mind binary protocols, especially if I can handle them with something like Python’s struct module, but grpc is just a bad example. It’s amazing how little functionality they packed into such a huge amount of code.
-
-
freddyb
2 years ago
|
link
| on:
Project idea feedback on "ActivityPub but for sandboxed general purpose computation"
If you want to run code, then wasm should be your thing. By default wasm doesn’t define any APIs and if there are some, they are handed in from whatever is instantiating the module (this is also known as capability-based security, because it can only access things that you explicitly put into its namespace).
There are WebAssembly runtimes that are not in the browser, so that might solve your use case? Specifically, if you define your own APIs and hand them into the module.
-
I think the whitelist approach is a must here, but in your view what is the benefit of webassembly over an approach like gVisor or mbox where a supervisor process proxies all system calls?
That supervisor process can choose to expose as wide or as narrow an API as is desired in either case.
-
-
munksgaard
2 years ago
|
link
| on:
Comparison of Rust async and Linux thread context switch time and memory use
From the summary, it sounds like async is always as good or better than true threads. So when should I use a thread instead of async?
-
I think this is a limited view. What I’m seeing here is that threads, an arguably simpler concurrency primitive, can scale quite competitively on modern operating systems.
Therefore, while you should continue to use async/epoll etc. when there is a true need (e.g. building web facing software like haproxy, nginx, envoy, caddy), many use cases will run competitively using a simpler blocking threads model and it might be worth evaluating whether you’d prefer working in that paradigm.
-
Async requires separating I/O-bound tasks from CPU-bound ones (you need to specifically spawn a task on a threadpool if it’s going to hog the CPU). If you can’t reliably isolate the two workloads, you’re going to have I/O latency problems. In that case it’s better to sacrifice threads for mixed workload.
Similarly if you’re dealing with FFI or non-async libraries, you have no guarantee what they’ll do, so it’s best to use a real thread just in case.
async/awaitin Rust creates a single state machine for every possible await point. If your code is complex with lots of on-stack state intermixed with awaits, then this state machine can get large. Recursion requires heap allocation (because it ceases to have compile-time-fixed call graph depth, so the state becomes unbound). If your code is complex, with lots of state and recursion, a real thread with a real stack may be better. -
If I understand rust’s async correctly or rather that it follows other languages’ async/concurrency patterns, you use async to improve single thread/core performance, it allows you to continue if the thread hits something like an io wait or similar non-cpu task. Multi-threading can do this but is better when processing for durations, like a browser.
cli tools that are intended to do one thing and exit quickly like
lsfor example may degrade performance by adding threads, but may improve performance by adding async. Now if as mentioned you have a browser and you have already maxed out what a single thread can do using async it’s time to split using threads to allow you to use more of the available cpu cores to do its job.golang explanation of concurrency vs parallelism: https://blog.golang.org/waza-talk
-
Performance is only one metric. It is indeed true, to the first approximation, that async is not worse than threads in terms of performance. The most convincing argument for the opposite is that async might have worse latency, as it’s easier to starve tasks, but I haven’t seen this conclusively demonstrated for Rust.
Another important metric is ergonomics or “is programming model sane?”. In this regard, async, at least in Rust, is not a sane programming model. Stack traces do not make sense, you get extra “failure modes” (any await might not return (when the corresponding task is dropped)), and there are expressively restrictions (recursion, dynamic dispatch, and abstraction are doable, but awkward).
-
To play the devil’s advocate, sync, blocking model is also far from perfect. It doesn’t handle cancellation or selection between several sources of events. That is, if you do a blocking read call, there isn’t a nice way to cancel it.
With async, because you need to redo the entirety of IO anyway, you can actually go and add proper support for cancellation throughout.
-
-
When a syscall/FFI blocks, or when you might have to crunch a lot of numbers are good cases.
-
Here I thought Postgres already had a queue functionality built in. I may be thinking of its pubsub stuff though.
Anyone know what the gotchas for this sort of approach may be? I can imagine lots of different jobs fighting over access to the same table, but that already happens in databases anyway…
If you are sharing a cluster, then saturating connections is definitely a concern.
Yeah you are thinking of notification channels. It can’t be used for queues because message posted to these channels are not stored (like redis pubsub). So if a listener misses a message due to for eg. downtime, the message is just lost.
It can be used to wake the workers and let them try to
select for update skip lockedand perhaps acquire a job to process without them needing to poll constantly, though.I haven’t actually tried this, but I posted it a year ago
What is SKIP LOCKED for in PostgreSQL? Most work queue implementations are wrong
And a comment says maybe that this is over-optimizing for FIFO semantics ?
Also related: https://news.ycombinator.com/item?id=29599132
Because Postgres uses MVCC, I think you need to worry about different transactions seeing completely different queue states. I think you are certainly capable of creating a race condition by having two separate transactions both trying to accept the same item at the same time. I believe for these reasons, building a queue is considered a “no-no” in Postgres-land; I remember reading a great article about it some years back.
However, in searching for this article (which I couldn’t find), I found a lot of articles describing how people had successfully built a queuing system in Postgres. So it may be that either the failure scenarios are really far-fetched or that there are techniques (like row locking as described in the article) which make it reliable.
The transaction isolation level, which is independent of MVCC or any other underlying implementation of it, keeps workers from reading the wrong state value out of the row. The problem with using Postgres as a queue is its default overly coarse row-level locking. In the past few years they’ve extended the locking model to support the queue use case, allowing concurrent workers to lock and update rows without blocking each other.
The crates.io job queue does it by using
SELECT FOR UPDATE ... SKIP LOCKED. The job runners lock the first open row in the table, work it, then either delete the row on success or update the state to failed.This is what I’ve done, although never deleting the rows but just setting the state (easier to just run deletes as a vacuum under times of lower load).
I would have thought that doing an UPDATE would have the same impact as a DELETE: both creates garbage rows that must be vacuumed. So you double the number of rows that need vacuuming.
There’s two edge cases I can think of where it can be worth it. One is if the table has big columns with TOASTed data. The other is if the table has a ton of indexes. In both cases the UPDATE might avoid dealing with those “right now”, so if your DB is stressed during peak but has low load times it can make sense to batch the deletes.
Postgres has HOT updates where if the update fits on the same page it can be done in place without creating vacuum load.
I wonder if that can be used to avoid queue bloat. https://www.postgresql.org/docs/current/storage-hot.html
Yes, we set our fillfactor to something like 10%. Or whatever the value is that means you have more space for HOT updates.
I had forgotten about HOT updates! Looks like they don’t apply if you have an index on the affected column, however. That might be workable for a queue table if it’s small enough that not indexing the column is acceptable—but that then begs the question of whether deferring the deletes is necessary.
https://www.postgresql.org/docs/current/storage-hot.html
I don’t think this optimization applies to deletes.
As long as your worker accepting a job from the queue involves a write to postgres, you’re fine. If you have a bunch of workers accepting work at the same time you can get a lot of transaction retries, so it can be useful to serialize workers access to the queue table. And of course if your job queue is high volume you can shard it and serialize access to different shards. But these are all performance optimizations and not needed for correctness.