Threads for lukaszkorecki
-
One important limitation: You can’t use it with connection pooling proxy (pgbouncer). So as soon as you need to manage hundreds of connections to the DB its not usable.
-
I’m a bit of a noob here - doesn’t that depend on the type on the pool mode (session, transaction, statement)?
-
Sorry, yes you are right! Session pooling allows all Postgresql features.
However session pooling is only useful for limited use cases such as maintaining a long-lived connection for short-lived clients. In most cases, people use transaction pooling to handle more connection than postgres would allow.
-
Got it! We’re currently using connection poolers at the application level (Clojure + hikari-cp connection pool) and it’s been working great so far - it give us session-mode connections, but without overloading our PG instances. I wonder if at some point we will need something like pgBouncer
-
Depending on your architecture, you can use a single dedicated connection per worker process to LISTEN and dispatch notifications to other threads within the process. Then you can use transaction pooling for all your other threads, reducing the total number of connections needed.
-
Yes, that’s more or less what I had in mind - the notification listener doesn’t have to process the jobs itself. Are you aware of any existing open source solutions that implement this kind of strategy? All I could find are Python and Ruby implementations where threads are not really a thing.
-
I don’t know of any, but it’s pretty straightforward. Especially since your dedicated listen thread only needs to LISTEN, you can still NOTIFY normally through a transactional connection pool.
Have each worker thread create a promise,
.putto a shared JavaBlockingQueue, then deref the promise (blocking). Have the dedicated LISTEN thread.takefrom the queue, send LISTEN to PostgreSQL, and then(deliver worker-promise notify-payload). Taking from the promise queue before sending LISTEN to PostgreSQL ensures you don’t LISTEN before you can use the payload.You could even use a dedicated connection pool in the LISTEN thread with
maximumPoolSize=1, reusing all of Hikari’s reconnect logic.-
That’s super helpful - thank you! I guess I could also play around with
core.asyncinstead ofBlockingQueue, although that’s more an implementation detail.-
You probably could, promises and async code go hand in hand.
Unpopular opinion: unless you’re writing something approximately similar to a load balancer, async is a waste of time. It’s 2020, mutexes are fast, context switching is cheap, and asynchronous code comes with plenty of poorly understood problems.
-
Not sure if it’s an unpopular opinion, I’ve been avoiding core.async as much as possible, as on the server there’s not many reasons to use it and it doesn’t solve any problems for me (or my team - we have ~12 Clojure applications in production right now).
I’m also not convinced if it’s useful in Clojurescript, but admittedly I didn’t build anything significant to have a better opinion.
-
-
-
-
the notification listener doesn’t have to process the jobs itself. Are you aware of any existing open source solutions that implement this kind of strategy
Take a look at rxJava (or rxCPP [1] or rx.NET – same semantic, but different languages). Eg review this article:
https://vlkan.com/blog/post/2016/07/20/rxjava-backpressure/
I am assuming that
- a) you are within single process
- b) you want one producer (eg your listener).. although multiple producers will work
- c) you want multiple non-blocking consumers within same process as the producers
- d) you want some control over backpressure (eg slow down the producer,if some consumers cannot keep up). you want to use queue with backpressure monitoring, and multiple clients
Of course, if your listeners are out of process (and/or running on different machines), then rxJava will not help.
I will also say, that rxJava has been really helpful in just writing mobile app clients (Android) because there are a number of things, when you write these clients, that require reacting to ‘events’ (Eg responses coming from backends, purchase/subscription acks from playstore, even user clicks.., etc). So this is not just for server-side code.
In my view, the rx set of libraries, their semantic, and use cases are exceptionally useful to learn and that knowledge will not become ‘irrelevant’ a few years, instead, you will be able to utilze it (and probably same APIs for next decade or more). Even in programming languages that do not have native threads …
-
-
-
-
-
-
Server hardware is getting better and better, with more and more cores. The standard 100 connection limit of PostgreSQL is making less and less sense as time goes on.
Fun fact: I just installed PostgreSQL on a server with a 3900X and it compiled in less than a minute with
make -j24. (Yes, it is a completely different workload. But it’s still really cool!)edit: missing “on”
-
-
Super handy! I was looking into it recently as I’m prototyping a job processing system on top PG (with
select... for update skip lockedand triggers on top of that).-
-
Ha, I know - my long term plan is to replace RabbitMQ with the new system so we can offer per-tenant queues, something that’s nearly impossible to do in Rabbit at scale (or most queueing systems for that matter)
-
-
Not really - Kafka topics are a resource on their own, so per customer topic is not a great strategy in the long term. Segment has a great write up about building something what I have in mind, but at much, much larger scale: https://segment.com/blog/introducing-centrifuge/
-
-
-
-
-
That’s what I use it for, maps very nicely to go’s channels and I love not bringing in some other system to run. Then again my stuff isn’t public and I’m the only user at the moment, so we’ll see if I end up regretting rolling my own at some point.
I based mine off ruby’s queue_classic.
-
-
It’s perfectly OK to use libraries that are not updated for the past 2 years in production. The community takes stands on its workflows and stick to it for year
I really like all the ideas behind Clojure, especially this one. I’d like software to be more like hardware. How often do I have to update my microwave’s software or refrigerator’s software? Software should just do its job without fuss. Unfortunately we have the opposite trend, where hardware is becoming more like software, being in constant need of updates (e.g. our phones).
So I like these ideals. I’ve watched almost all of Hickey’s talks, and I even skimmed a book on Clojure “for fun”.
But unfortunately I never really crossed paths with the JVM professionally or personally, and it’s still not something I want or feel a need to dip my toes into. Despite writing some JavaScript, I also mostly avoid the NPM ecosystem, which ClojureScript uses as far as I understand.
Has anyone tried any other production-quality Clojure implementations or Clojure-influenced languages that are interesting?
On a different note, I watched a talk about Clasp and it looks pretty cool, e.g. the way it uses LLVM to integrate seamlessly with C++ and code generation. I wonder if an analogous Clojure / C++ hybrid would also make sense? Maybe not because as far as I understand Common Lisp has a history of generating fast code, while Clojure doesn’t. Despite the superficial similarities they are pretty different, from what I can tell.
-
I’m of course biased, but take a look at CHICKEN Scheme. It’s Scheme, which is a Lisp dialect. Scheme is very clean, and there are several stable and mature libraries for it (“eggs”) that haven’t changed much but are still being maintained. CHICKEN is fast, compiles to C and makes interfacing with C very easy.
Yes, the recent major release of CHICKEN itself (from 4 to 5) was a massive change, but these kinds of huge changes don’t tend to happen all that often (this one was way overdue). Upgrading code is usually simple and can be done in a backwards-compatible way if you take some care.
-
Seconded! CHICKEN is awesome and we used it to write small monitoring daemons on couple of occasions. Can’t wait to try out v5 soon!
-
-
Please don’t be scared of the JVM and Java dependencies - Clojure’s interop is very neat and super straightforward to use. Admittedly, I was intimidated by it in the early days but once I crossed that bridge it feels very natural, and as others stated it’s not something you do very often. In the last 4 years, 20 Clojure services and ~25 internal libraries we had to write Java code maybe once or twice and deal with one weird behavior coming from Maven.
I think I need to write my own “~4 years of Clojure” post ;-)
As far as other Clojure-inspired languages - you’ll find the ecosystem lacking because it’s a niche of a niche - unless you like to write your own libraries for everything.
-
You really don’t need to know much about the JVM to use Clojure. It’s not different than having Ruby or Python runtime on your system. Regarding ClojureScript, you don’t actually have to use NPM with it. It has its own compiler, and you can opt into using NPM if you need modules from it.
-
-
- MailMate on the desktop
- Apple Mail on my phone
-
I can work around this issue, but I find it much more worrisome that their gateways are discontinued entirely. I use Slack through the IRC gateway, because I don’t want to have a CPU & memory gobbling browser tab for every organization I join.
This means that sometime in the not-to-distant future, using the gateways at all won’t be possible.
-
Yes. It leaves me with the impression that gateways might exist primarily to ease the on-boarding process of new customers by being able to tell them, that they can continue using their IRC or XMPP clients. Once a company becomes a customer, the discussion is effectively finished and individual users will be pressured to use the electron or browser client anyways because Slack is what the company uses now.
So, sadly, there’s little incentive for Slack to support those bridges as long they are good enough to say “yes, we’ve got them” in the beginning.
-
-
What if the gateways are still running by chance, and no one dares to touch them because the dev(s) that built it left the company.
That would imply that a not too small company with lots of resources would run totally unmaintained software in production, right? I hope they don’t, and if they do, they could least say so in their documentation and advertisements.
Anyway, as you wrote, it doesn’t make sense to spend money enhancing something that most users don’t care about.
That’s not what I wrote! I meant that there are little incentives for companies who care a lot about profit maximization and less about supporting users with more uncommon needs. But it would of course ‘make sense’ to spend money on such features if one would like to create a good communication platform for a more diverse set of users!
-
Yes, I’ve seen them mention their XMPP gateway support in advertising material. They certainly don’t mention that it’s broken and entirely unsupported.
-
-
-
-
-
If you’re using Weechat then give wee-slack a go - I believe its lead developer works at Slack. I switched from IRC gateway some months ago because my team insists on using threads and emoji reactions, and I insist on using tools which don’t consume tons of resources. So far it’s been great, it supports most of the features I need (threads, emoji reactions, some slash commands) and I don’t care about others (search, file uploads)
-
Seconded; wee-slack is a lifesaver.
Where I work they won’t even turn the gateway on, but wee-slack talks directly over the same websocket API that the official client uses. It also lets me SSH into my work computer from my personal machine and keep my login credentials only on my work machine.
-
-
This means that sometime in the not-to-distant future, using the gateways at all won’t be possible.
A bit like some time in the not-too-distant future, you won’t be able to unlock a new Android/iOS cellphone without something like “Face ID”.
For your convenience and security, of course. Only a terrorist would want a phone without Face ID!
The commonality here is that the masses don’t care about the alternatives, and so, the alternatives end up becoming impractical.
-
-
Nice. I made something like that a while ago - also in Bash, but with more features and only with GitHub support.
-
-
That was the fun part of building it. Everyone should write their own testing library in Bash ;-)
-
-
-
lukaszkorecki
6 years ago
|
link
| on:
What is the oldest (retro) machine that you still use in production?
I have a vintage early 2015 Macbook Pro, you know the ones that still had the Esc key. Tremendous machine.
-
Great read. Also:
As you know, I’m joking on this article, since as we both know, Erlang copied Scala.
found in the comments.
-
It reassures me that a lot of these guys have “boring” desktops. My desktop is mostly Emacs, Firefox and way to many terminals. Definitely not a contender for a ricer setup that people post on /r/unixporn
-
My favorite part is how they’re all apologizing for being boring.
-
You either get stuff done or have a nice looking desktop ;-)
-
-
One that got a little popularity at one time was my Rouge (wayback website). It got a few mentions in books and I was pretty chuffed about it.
That’s really cool, the syntax for blocks interopt (is that what it is?) looks interesting:
Any reason why this couldn’t be just a function passed in? (Since blocks can be procs/lambdas/anything that implements
call, sometimes)The block form with arguments is syntax sugar that mirrors sugar in Ruby. It’s worth noting that a block isn’t a regular argument in Ruby — it has special treatment in the VM. So in Ruby:
Note how the last example fails, because
eachdoesn’t take any positional arguments.The first two forms are largely identical, although there is a difference in how the bytecode turns out because blocks get some special treatment in codegen too. (If you’re interested, try
puts RubyVM::InstructionSequence.compile('[1,2,3].each {|e| puts e}').disasm, and then again for the&lambda {…}form.)Here’s the equivalent in Rouge using Ruby interop:
Of course the ~rougey~ way would probably involve an as-yet unwritten
doseqimplementation. This thing didn’t get too far off the ground in terms of Clojure-compatibility.Thanks for responding. It gave me a chance to play around with Rouge, which I haven’t done in a good seven years or so. :)
Ah true - I forgot about the special handling of blocks. There’s more to it, for example: implicit blocks with
yield- it’s been a while since I wrote any Ruby that uses that stuff extensively.I wish I had some spare time to play around with it - based on the “seven years” part, it’s not only me ;-)