1. 1

    I snipped the title because it was too big: The Largely Untold Story Of How One Guy In California Keeps The World’s Computers Running On The Right Time Zone. (Well, Sort Of)

    outline link - https://outline.com/g7SPsW

    1. 2

      umm, it seems it would require different signups and login for different services? It would be great if you can add SSO

      1. 1

        SSO is on our roadmap. I just haven’t had the time nor the knowledge to set it up. :)

      1. 1

        I hang out in a couple programming language development chat channels

        Can you share these channels if they are public and accept new people?

        1. 2

          Probably they mean #lang-dev on the Rust discord (https://discord.gg/rust-lang-community) and the Programming Languages discord, which I unfortunately no longer have a link to.

          1. 2

            Correct on both counts. The latter is https://discord.gg/yqWzmkV ; it’s actually an offshoot of Reddit r/programminglanguages , which I’d forgotten. The Rust discord is more fun to actually talk to people in, but the Programming Languages discord occasionally has outright deluges of interesting and/or useful information.

        1. 11

          This is a great write up, but I’m a bit confused by the title and headline:

          Inserting One Billion Rows in SQLite Under A Minute

          Current Best: 100M rows inserts in 33 seconds.

          Shouldn’t this blog post be called Inserting One Hundred Million Rows in SQLite Under A Minute?

          1. 3

            Hey, sorry for the misleading title. I started with ‘Fast SQLite Inserts’ and it had many iterations. In the title, I wanted to intend that I want to insert 1 billion rows under a minute on my machine. I thought the current title is fine, since I got LGTM for earlier drafts. The detail about on my machine is also important since mine is a two year old laptop and all the measurements are done on it.

            Also I got another feedback that title should indicate that it is a test database and emphasise that it is not durable.

            I am wondering the right way to convey all of this in the title yet also keep it short.

            1. 10

              I think the title is close enough but could be renamed to 100M and still be impressive. What you said makes sense, but generally we don’t see titles that didn’t actually happen unless they’re prefixed with “Attempting to” or something.

              1. 3

                hey yellow, thank you for reading my post and for the feedback. I have updated the title on my blog post, prefixed with “Towards” and also requested mods to edit title here as well.

                1. 1

                  Thanks! Nice post!! :)

                2. 3

                  Yeah, I agree. I think it’s a great blog post and the results are impressive. The title is just a bit misleading.

            1. 3

              The companion book also looks pretty nice - https://www.stateright.rs/

              1. 29

                Sun Microsystems tries to sell Brendan Gregg’s own software back to him, with the GPL and author credit stripped (circa 2005).

                Great article.

                1. 15

                  Hey, spoiler alert missing ;)

                  1. 11

                    I think we should have a spoiler tag

                    1. 5

                      Or maybe just add the TLDR acronym

                    2. 1

                      Of course by replying we help keep this comment at the top. (I’m doing it as well, argh.)

                  1. 3

                    Why it was getting rejected so many times, what were the troublesome permissions?

                    1. 4

                      Google, basically, took a hatchet to apps with SMS or Call Log permissions a couple of years ago, and these were the permissions the app required. Google has, rightly so, deemed these permissions to be sensitive, so access to these permissions are limited. As for how I got this sorted, the truth is the changes I made were many and it really is difficult for me to tell you which few, or dozen, were specifically related to the rejections. Unfortunately, it’s likely a whole bunch of changes helped move the ball forward.

                      1. 2

                        Google doesn’t like it when apps access text messages IIRC

                      1. 1

                        Interesting post! Were all the bugs caught by MIT test suite?

                        1. 1

                          I guess it would be hard to know for sure if all bugs were caught, but IME the test suite was pretty rigorous. They’re more in the vein of “systems tests” or integration tests since they’re timing-dependent and opaque to the implementation.

                          I wouldn’t consider my toy implementation to be production ready, but the provided test suite was rigorous enough to show me where I messed up

                        1. 14

                          But why? Ain’t broken, do not fix it. Better cd? Seriously? Better cat? Seriously??? I wish I had the time to rewrite such trivial things in Rust for no good reason. Literally zero value is added by this.

                          1. 27

                            Literally zero value is added by this.

                            This is literally false. I use bat for its syntax highlighting, and it is a significant value add for me over cat.

                            1. 6

                              But that is not what cat is for. cat is for concatenation, nothing else.

                              1. 6

                                I used to use cat to quickly view short files in the terminal. bat replaced cat for me:

                                https://github.com/matklad/config/blob/f5a5a9db95b6e2123d2588fe51091fa53208c6e6/home/.config/fish/set_universals.fish#L28

                                I am sure that this matches significant fraction of usages when people type cat. I’d even say that viewing files + “priming the pipe” (for which <in.txt prog is the “proper” tool) represent majority of cat usages in the wild.

                                So, I don’t find “cat is for concatenation, nothing else” statement useful, as it doesn’t describe the reality.

                                1. 3

                                  it does for me. I never use cat to view files, I either use less (sometimes more, if I am on a very restricted system) or vim. I think you could remove cat from my system an I would not even notice, that is how often I use it. Most cli heavy people I know tend to be similar.

                                  1. 1

                                    Then you were misused it as you should have been using less for viewing files. That is the purpose of pager.

                                    1. 9

                                      less is less convenient to me. I do not want to have to type q to close it, and I do want the file to be visible while I type further commands.

                                      On a meta level, I just don’t get these prescriptivist comments. I would understand “less is more convenient than cat” framing, but what is the value of “cat should be used only for concatenation” statement eludes me.

                                  2. 5

                                    cat(1) is also, depending on your platform, for line numbering, non-blank-line numbering, the removal of runs of multiple blank lines, the explicit display of end-of-lines, the explicit display of tabs and form feeds, converting all non-printables to a human-readable escaped form, and for line ending conversion.

                                    1983 called and wants its boring whinge back.

                                    1. 1

                                      Right, and now we have a program for bonbatenation. It replaces cat and less. I don’t see the problem here.

                                      1. 1

                                        To be clear, bat’s pager capabilities start and end at running an external pager if stdout is a tty. It supplements less, it doesn’t replace it.

                                        I wonder how much stuff would break if you taught cat to do the same thing. Nothing springs to mind.

                                  3. 14

                                    If humanity had followed the “if it ain’t broke, don’t fix it” maxim to the tee without exception, we would all still be living in the stone age.

                                    1. 8

                                      And we might actually be better off that way. At least, a lot less likely to drive half the planet to extinction, and probably not significantly less happy (hedonic treadmill, etc).

                                    2. 11
                                      • Bat is awesome. It’s not really a cat replacement, but a “terminal file viewer with syntax highlighting”. I would never use bat in, say, a script to concatenate files. Two tools for two different jobs.
                                      • I haven’t tried loc or tokei, but cloc is a giant ball of slightly buggy perl. Run cloc on a big enough project and you’re bound to encounter issues.
                                      • Cut can only split based on single-byte delimiters. Lots of tools try to do some form of alignment by using multiple space characters, or they use a syntax like key: value, so cut usually doesn’t cut it. Having to reach for awk just to get the third column of a thing is unfortunate, so a better cut seems like a good idea.
                                      • Sudo is security critical. Every now and then, there are new memory issues found in sudo which couldn’t have existed if sudo was written in a safer language.
                                      • People like writing system monitors. Htop, bashtop, bpytop, gotop, the list goes on. I’ve even written a few myself. It only makes sense that someone would eventually write system monitors in Rust.
                                      • Hyperfine isn’t really a replacement for time. Rather, hyperfine is a fairly complete benchmarking solution which does multiple timed runs, warmup runs, detects outliers, and can compare multiple commands against each other. It’s become my ad-hoc benchmarking tool of choice.
                                      • Zoxide is certainly not just a cd replacement. It’s a completely different way to navigate the filesystem, learning which paths you usually go to so that you can write part of a path and have zoxide take you to a frequently used directory which matches your pattern. If you’re gonna complain about zoxide, complain that it’s a reimplementaiton of something like autojump (Python) or z.lua (lua).

                                      I don’t have any experience with the rest of these projects, but I would bet that the same pattern holds for most of them. It seems like, actually, a lot of value is added by many of these projects.

                                      1. 5

                                        tokei replaced my use of sloccount, which is also a giant ball of Perl and assorted support programs.

                                        It’s slightly faster than either. On FreeBSD /usr/src:

                                        • cloc: 3 minutes 40 seconds
                                        • sloccount: 3 minutes 16 seconds
                                        • tokei: 2.1 seconds
                                      2. 9

                                        How sure are you that the existing pieces of software are in fact not broken (in the sense of having some memory-safety-related security vulnerability that might get discovered and exploited tomorrow and become the next Heartbleed)?

                                        1. 2

                                          I have been using broot instead of tree and it is quite a joy.

                                        1. 17

                                          I have no idea what Oxide is and why it is relevant for Lobste.rs, but this specific post looks like it’s just a big ad for some server hardware. Is there any interesting technical content that I’m missing?

                                          1. 18

                                            I think it’s because two of the C-level people are “lobste.rs-approved”. Bryan Cantrill was the Joyent CEO, and wrote DTrace. Jessie Frazelle is a former Docker, Inc. employee who played a significant part in the popularisation of containers beyond Docker itself. (mostly during her time at Microsoft)

                                            But your comment is relevant though.

                                            1. 2

                                              Bryan was CTO of Joyent

                                              1. 1

                                                You’re right. My mistake. I can’t edit my comment anymore :P .

                                            2. 5

                                              It’s hyperscaler hardware most people have no use for. I think people are interested on the engineering work going on, w/ work on low-level boot firmware and…. rewriting it in Rust.

                                              1. 10

                                                The annoying thing is that the submission has nothing to do with the engineering, at its core. It isn’t a deep-dive into any of the sales points, no real technology is discussed.

                                                And now these folks just got a rather expensive ad slot on a slow-traffic site with high-value eyeballs.

                                              2. 7

                                                Just fanboyism here I am afraid

                                              1. 10

                                                For reference, this is one chapter out of the (free and excellent) book Operating Systems: Three Easy Pieces, by Remzi Arpaci-Dusseau and Andrea Arpaci-Dusseau . The full thing is available here: https://pages.cs.wisc.edu/~remzi/OSTEP/

                                                1. 2

                                                  by Remzi Arpaci-Dusseau and Andrea Arpaci-Dusseau

                                                  Idd, I’m becoming quite a fan of the work done by those two. From research on SSD internals, to file-systems and databases and now I found this gem.

                                                  1. 3

                                                    From research on SSD internals, to file-systems and databases and now I found this gem.

                                                    Newbie here, I am trying to learn about file systems and databases. Can you please link to those resources?

                                                    1. 1

                                                      Apart from the references in a paper itself to find previous work, I’m a big fan of the citations feature of papers found via researchgate.net. If you have one good paper, i.e. The_Unwritten_Contract_of_Solid_State_Drives then just scroll to the “Citations” section on that website. This way you easily discover new research that builds upon that. Also I found the textbook Database System Concepts very complete, it also refers to research papers for further reading. I’ll keep posting links to stuff that I find interesting, currently in the area of file systems and databases.

                                                1. 5

                                                  Meanwhile, PyPy is around 4x faster than CPython.

                                                  1. 6

                                                    Annecdote ain’t data, but I’ve never been successful at getting PyPy to provide improved performance. My use cases have been things like running tooling (Pylint is extremely slow under PyPy, much moreso than CPython), just running web apps, and a lot of other things that aren’t benchmarks.

                                                    I don’t want to be too critical of PyPy, I imagine it gets a lot of what a lot of people want. But I don’t know what real workloads end up benefiting from it.

                                                    1. 4

                                                      PyPy upstream generally treats slowness as a bug and is willing to expend resources to fix it, if you’re willing to file issues with minimal test cases. (Here is a recent example bug about slowness.)

                                                      Anecdotes aren’t data, but about a decade ago, I ported a Minecraft server from Numpy and CPython to array.array and PyPy, and at the time, I recorded a 60x speedup on a microbenchmark, and around a 20x speedup for typical gameplay interactions, resulting in a backend that spent most of its time sleeping and waiting for I/O.

                                                      As long as we’re on the topic, it’s worth knowing that PyPy comes with a toolkit, RPython, which allows folks to generate their own JITs from Python. So, if one wanted more speed than was available with Python’s language design, then PyPy provides a route for forking the interpreter and standard library, and making arbitrarily distant departures from Python while still having high performance. For example, if we can agree that Dolphin implements “real workloads”, then PyGirl (code, paper) probably does as well.

                                                      1. 3

                                                        Yeah to me it helps to think of workloads in these categories (even if there are obviously way more than this, and way more dimensions)

                                                        1. String / hash / object workloads (similar to web apps. Similar to a linter, and similar to Oil’s parser)
                                                        2. Numeric workloads (what people write Cython extensions for; note that NumPy is written largely in Cython.)

                                                        JITs are a lot better at the second type of workload than the first. My experience matches yours – when I tried running Oil with PyPy, it was slower and used more memory, not faster.

                                                        Also, I think that workload 1 is the more important one for Python. If I want to write fast numeric code, it’s not painful to do in C++. On the other hand, doing string/hash/object graph workloads in C++ is very painful. It’s also less-than-great in Rust, particularly graphs.

                                                        So while I think PyPy is an astonishing project (and that impression grows after learning more about how it works), I also think it doesn’t speed up the most important workloads in Python. Not that I think any other effort will do so – the problems are pretty fundamental and there have been a couple decades of attempts.

                                                        (In contrast I got much better performance results adding static types manually, and semi-automatically translating Oil to C++. This is not a general solution as its labor intensive and restricts the language, although there are some other benefits to that.)

                                                        1. 1

                                                          I see the outline of your point, but I’m not sure on the specifics. In particular, a mechanism is missing: What makes strings, dictionaries, and user-customized classes inherently hard to JIT, particularly with a PyPy-style tracing metainterpreter?

                                                          Edit: Discussion in #pypy on Freenode yielded the insight that CPUs have trouble with anything which is not in their own list of primitive types, requiring composite operations for composite types. Since JITs compile to CPU instructions, they must struggle with instruction selection for composite types. A lesson for language designers is to look for opportunities to provide new primitive object implementations, using the CPU’s existing types in novel ways.

                                                          Our experience in the Monte world is that our RPython-generated JIT successfully speeds up workloads like parsing and compiling Monte modules to bytecode, a task which is string- and map-heavy. Our string and map objects are immutable, and this helps the JIT remove work.

                                                          1. 1

                                                            Yes the JITs do a lot better on integers and floats because they’re machine types.

                                                            The performance of strings and hash tables is sort of “one level up”, and the JITs don’t seem to help much at that level (and for some reason lots of people seem to misunderstand this.)

                                                            As an anecdote, when Go was released, there were some benchmarks where it was slower than Python, just because Python’s hash tables were more optimized. And obviously Go is compiled and Python is interpreted, but that was still true. So that is a similar issue.

                                                            So there are many dimensions to performance, and many workloads. Saying “4x faster” is doing violence to reality. In some cases it’s the difference between being able to use PyPy and not being able to use it.

                                                          2. 1

                                                            SciPy has some cython code along with a bunch of fortran code but NumPy is all C.

                                                            1. 1

                                                              Ah sorry you are right, I think I was remembering Pandas, which has a lot of Cython in its core:

                                                              https://github.com/pandas-dev/pandas/tree/master/pandas/_libs

                                                            2. 1

                                                              cython is also a translator to C. why didn’t you use cython for oil?

                                                              1. 1

                                                                It generates code that depends on the Python runtime, and Cython is a different language than statically-typed Python. I don’t want to be locked into the former, and translating the code is probably even more labor intensive than what I’m doing (I leveraged MyPy team work on automatic type annotation etc.). It also wouldn’t be fast enough as far as I can tell.

                                                            3. 3

                                                              pypy is 4x faster…. for long-running tasks that allow the jit to warm up. Lots of python workloads (e.g. pylint) run the interpreter as a one-off so pypy won’t help there. Interpreter startup speed is also critical for one-off workflows and pypy isn’t optimized for that either.

                                                              1. 3

                                                                I think it’s more like 10x-100x faster OR 10% slower for different workloads – “4x” doesn’t really capture it. See my sibling comment about string/hash/object vs. numeric workloads.

                                                              2. 2

                                                                I used PyPy recently, for the first time and I had a nice experience. I am experimenting with SQLite and trying to figure out the fast ways to insert 1B rows. My CPython version was able to insert 100M rows in 500 is seconds, same in PyPy took 150 seconds.

                                                                The best part was, I did not have to change anything in my original code. It was just drop in, as advertised. Ran it with PyPy and got the speed bumps.

                                                              3. 2

                                                                Specifically, we want to achieve these performance goals with CPython to benefit all users of Python including those unable to use PyPy or other alternative virtual machines.

                                                                1. 1

                                                                  Apparently the goal is a 2x speed up by 3.11 and a 5x speed up in 4 years.

                                                                  1. 4

                                                                    Yes. Assuming that those numbers are not exaggerated, I expect that PyPy will still be faster than CPython year after year. The reasoning is due to the underlying principle that most improvements to CPython can be ported to PyPy since they have similar internal structure.

                                                                    In GvR’s slides, they say that they “can’t change base layout, object layout”. This is the only part of PyPy’s interpreter which is structurally different from CPython. The same slide lists components which PyPy derived directly from CPython: the bytecode, the stack frames, the bytecode compiler, and bytecode interpreter.

                                                                    Specializing bytecode has been tried for Python before; I recall a paper which monomorphized integers and other common builtin types. These approaches tend to fail unless they can remove some interpretative overhead. I expect that a more useful product of this effort will be a better memory model and simpler bytecodes, rather than Shannon’s grand explosion of possible bytecode arrangements.

                                                                    1. 1

                                                                      I’m curious about mypyc personally. Seems to me like (c)python is just hard to optimize and depends too much on implementation details (the C API) to be changed; to get a significant leap in performance it seems like using a statically typed, less dynamic subset, would give significantly higher speedups. Of course the downside is that it doesn’t work for old code (unless it happens to be in this fragment).

                                                                      1. 1

                                                                        Monomorphizing code does not always speed it up. There are times when tags/types can be checked for free, thanks to the dominating effects of cache thrashing, and so the cost of dynamically-typed and statically-typed traversals ends up being similar.

                                                                        It’s not an accident that some half-dozen attempts to monomorphize CPython internals have failed, while PyPy’s tracing JIT is generally effective. Monomorphization can remove inner-interpreter work, but not interpretative overhead.

                                                                        1. 2

                                                                          Well by “less dynamic” I also mean not having a dictionary per class and this kind of stuff :-). I should have been clearer. tag checks is one thing, but performing dictionary lookups all the time to resolve identifiers or fields is also very heavy. The statically typed aspect, I have no idea if it’s truly necessary, but it’d make it easier to implement, right?

                                                                1. 11

                                                                  we settled on $11,000 per month as a baseline salary

                                                                  Am I missing something about working in the USA, or is that very high? Especially for a project running on donations trying to reach sustainability.

                                                                  1. 22

                                                                    That’s $132000/year, which would be a junior developer level of salary at many US tech companies.

                                                                    Also, in the US anyone who is a contractor or “self-employed” pays an automatic additional 15% tax on their wages/salary above the normal income tax (which still applies), and has to fund all their own benefits (health insurance, pension, etc.) from their salary as well (regular employees of companies often get that at least partially funded by the employer).

                                                                    1. 29

                                                                      From my experience, salaries like that at a Junior level are only found in the Bay area and at FAANG like companies. For a lot of the devs who work at small/medium sized companies in reasonably priced areas, junior dev salaries are closer to $60,000-$90,000.

                                                                      1. 8

                                                                        $132k/year for a jr dev is about 100k more than I’ve seen in areas of the US. Even in NYC, you can expect maybe $75-90k/year as a jr dev at most companies. That seems extraordinarily high.

                                                                        1. 5

                                                                          … I got paid a lot more than 32k/yr as a junior developer straight out of school, with bad grades, in the south atlantic, in the aughts.

                                                                          If you’re getting paid 32k/yr as a programmer, please quit and work at chipotle - you’ll get paid more, probably have better benefits and at least you’ll get free lunch, saving another ~$7/workday.

                                                                          1. 1

                                                                            Edit: disregard, I can’t read.

                                                                        2. 3

                                                                          That’s $132000/year, which would be a junior developer level of salary at many US tech companies.

                                                                          Is levels.fyi considered good source to check the average salaries? or any other sites?

                                                                          1. 5

                                                                            levels.fyi is good if you’re looking at joining a Big Tech (tm) company or similar companies near Silicon Valley, but it’s definitely not representative of salaries across the country.

                                                                        3. 8

                                                                          someone on the orange site mentioned the twitter post by one of the babel founders which claims that the funding went to the wrong hands for some time, though they seem to be biased.

                                                                          The discussion over there also has some links that seem to proof these claims.

                                                                          1. 2

                                                                            Yeah, he seems to have deleted that because he was mostly considering commits and such and ignoring all the work around the project that isn’t just code, and he rightfully got a lot of crap for it.

                                                                            The scope of the project and the associated busy work around it is very different from when he started it.

                                                                            1. 1

                                                                              Like what? What other work was done by Henry that doesn’t result in commits, comments and other visible artefacts?

                                                                              1. 1

                                                                                Some of the stuff they mention in the post possibly. Public speaking, other activities involved in fund raising and supporting the project that probably don’t all take place directly in the repository and more likely in other points of contact.

                                                                                This isn’t so different from work done in engineering organizations. As your responsibility or level of seniority increases, the size of your output might lower, but the impact might be higher.

                                                                                At the same, time, the quantification that was originally pointed out was a number of pull requests, issues, etc, which is highly disingenuous.

                                                                                In Open Source projects in particular, an issue investigation can stretch out for weeks or months. PRs can have different sizes, and some of them can perhaps have a lot of work gone into them (research, etc) that isn’t directly visible in code committed. There’s also the time spent in reviewing and understanding contributors work. And all of this becomes even more complex and time consuming as a project grows in size and complexity.

                                                                          2. 7

                                                                            One thing to consider is that this number is likely fully-loaded (I imagine that in practice everyone is being paid like a contractor), so there’s a big chunk of that that disappears into taxes, and the takehome is probably more similar to if someone was pulling in $7k/month at a “normal company”. Seems pretty reasonable as take home in places like the US (with high rent + health insurance).

                                                                            I don’t think that it’s unreasonable for full-time maintainers to expect a salary at that level.

                                                                            1. 4

                                                                              I don’t think that it’s unreasonable for full-time maintainers to expect a salary at that level.

                                                                              I agree. The thing everyone is forgetting about is that for someone of the skill level required to maintain a popular OSS project is likely able to command a salary easily in this range. There are hundreds of companies that would hire them remotely at a salary well above $132k p/year. It’s not just FAANG companies in this range anymore.

                                                                              I mean, sure there are lots of companies paying 60 to 90k. Maybe those devs are happy with the amount. Maybe they are unhappy, but not motivated to run through interviews. Maybe they haven’t put themselves out there enough with OSS or giving presentations. We don’t know their situations, but if a developer can command a higher salary its not really in our best interests as a community to criticize them. (The fact that some say its unwarranted in this particular situation is a different matter)

                                                                            2. 1

                                                                              Even if that’s a huge salary for most tech jobs outside of SF and NY, if we consider the amount of value Babel has created for all the companies around the world and if the project was compensated accordingly they should be drowning in money.

                                                                            1. 8

                                                                              This is interesting and I am prolly going to use it in our services. Could you talk more about the underlying protocol?

                                                                              Also, aren’t these sentences contradicting:

                                                                              So we rewrote gRPC and migrated our live network. DRPC is a drop-in replacement that handles everything we needed from gRPC

                                                                              and

                                                                              It’s worth pointing out that DRPC is not the same protocol as gRPC, and DRPC clients cannot speak to gRPC servers and vice versa.

                                                                              How it is a drop in replacement if both client and server needs to be changed

                                                                              1. 15

                                                                                This is interesting and I am prolly going to use it in our services. Could you talk more about the underlying protocol?

                                                                                The wire format used is defined here.

                                                                                Logically, a sequence of Packets are sent back and forth over the wire. Each Packet has a enumerated kind, a message id (to order messages within a stream), a stream id (to identify which stream), and a payload. To bound the payload size, Packets are split into Frames which are marshaled and sent.

                                                                                A marshaled Frames has a single header byte, the varint encoded stream and message ids, a varint encoded length, and that many bytes of payload. The header byte contains the kind, if it is the last frame for a Packet, and a control bit reserved for future use (the implementation currently ignores any frame with that bit set).

                                                                                Because there’s no multiplexing at this layer, the reader can assume the Frames come in contiguously with non-decreasing ids, limiting the amount of memory and buffer space required to a single Packet. The Frame writing code is as simple as appending some varints, and the reading code is about 20 lines, neither of which have any code paths that can panic.

                                                                                How it is a drop in replacement if both client and server needs to be changed

                                                                                There’s a drpcmigrate package that has some helpers to let you serve both DRPC and gRPC clients on the same port by having the DRPC clients send a header that does not collide with anything. You first migrate the servers to do both, and can then migrate the clients.

                                                                                The drop in replacement part refers to the generated code being source code API compatible in the sense that you can register any gRPC server implementation with the DRPC libraries with no changes required. The semantics of the streams and stuff are designed to match.

                                                                                Sorry if this was unclear. There’s multiple dimensions of compatibility and it’s hard to clearly talk about them.

                                                                                1. 8

                                                                                  (not the one who asked). Thank you for the comment. Do you plan to document the protocol somewhere (other than code)? It seems to me a RPC protocol needs a healthy ecosystem of implementations in many languages to be viable long term :-)

                                                                                  (edit: I mean document it cleanly, with request/reply behavior, and talk about the various transports that are supported.)

                                                                                  1. 6

                                                                                    You are absolutely right that having good documentation is essential, and I’ve created a github issue to start to keep track of the most important bits I can think of off the top of my head. To be honest, I’ve always struggled a bit with documentation, but I’m trying to work on it. In fact, the current CI system will fail if any exported symbols do not have a doc string. :) Thanks for bringing this up.

                                                                                  2. 2

                                                                                    Thanks for the explanation! Are there any plans for writing Python client/server?

                                                                                    1. 4

                                                                                      I do hope to get more languages supported. I created an issue to add Python support. I don’t currently deal with any Python code, so help will probably be needed to make sure it “fits” correctly with existing codebases.

                                                                                      1. 1

                                                                                        I have started looking into the drpcwire code, I will wait for the documentation. Meanwhile, is there any place, like IRC, Discord where drpc/storj developers hang out? The Python issue is tagged help-wanted, so, can I be of any help?

                                                                                1. 1

                                                                                  This looks really nice. Reminded of UI of Pinboard

                                                                                  1. 1

                                                                                    thanks!

                                                                                  1. 3

                                                                                    From the article:

                                                                                    According to benchmarks, writes to SSDB are slightly slower but reads are actually faster than Redis!😲

                                                                                    How a disk based solution can be faster than Redis? Here is the link to the benchmarks.

                                                                                    1. 2

                                                                                      It looks that this is not read latency, but read operation per second. I think since SSDB is multithreaded as opposed to Redis, allows it to be faster even though it’s using a disk.

                                                                                      NVMe SSD could also help benchmarks.

                                                                                      1. 1

                                                                                        That actually explains why! Thank you.

                                                                                    1. 2

                                                                                      I am learning about setting up replication and cluster in Postgres. I sorta “broke” MongoDB by inserting duplicate records on a collection which had unique constraints. Now I am wondering what would happen in case of Postgres.

                                                                                      I am also trying learn cgo and rust FFI, trying to use a rust library in go.

                                                                                      1. 1

                                                                                        Postgres doesn’t have a multiple-writable-node configuration, so the consensus mechanism is much simpler (and not subject to that class of bug).

                                                                                        Even if it was, the indexing would take out a write lock lock (unless you used the recent “concurrently” option, in which case it’d fail right at the end of the index build).

                                                                                        1. 1

                                                                                          Postgres doesn’t have a multiple-writable-node configuration

                                                                                          My mongo setup also didn’t have multiple writable node config, all the changes had to be made via primary only. Or what I am missing?

                                                                                          1. 1

                                                                                            I misunderstood. Now that I’ve gone back and reread the relevant bits in more detail:

                                                                                            Postgres has “Logical” and “Streaming” replication modes builtin.

                                                                                            In “Streaming” mode, it transmits the changes to the on-disk representation of the database (including indexes) in order. This is expensive (network bandwidth use = disk write bandwidth use), but easy to make correct.

                                                                                            In “Logical” mode, it transmits the statements, along with enough information to reproduce the order of operations & volatile data they need. There’s lots of ways this could go wrong under the hood!

                                                                                            In practice, I’m not aware of anyone finding bugs in “logical replication” in any stable release; the order-of-operations stuff lets postgres sequence all the same obtain lock/release lock code to get the same result.

                                                                                      1. 1

                                                                                        brew is a mediocre replacement.

                                                                                        As someone who uses brew, can you tell me what I am missing?

                                                                                        1. 7

                                                                                          An issue I’ve encountered with brew is with software versions. Homebrew assumes that you want the most recent version of all the software. There is no simple way to install a previous version of a piece of software without digging around the git repo and finding the commit for that version, downloading the script, and manually telling brew to use that script.

                                                                                          Homebrew is also pretty poor when it comes to version pinning. It only pins the specific version specified. There is no way to pin match on major versions. Which means that when a point release of a software package is released you must manually unpin the old version, install the new version, and then re-pin the new version.

                                                                                        1. 4

                                                                                          You might want to read the papers on OKws. This was the OK Cupid web server architecture. It was designed by a few people from MIT and makes clever use of Unix domain sockets.

                                                                                          1. 4

                                                                                            MIT 6.858 Computer Systems Security covers OKws as a case study. Well worth a watch IMO (or at least a skim of the lecture notes).

                                                                                            1. 2

                                                                                              is this the one that you had in mind ? may you please let me know ? thank you kindly !

                                                                                              1. 2

                                                                                                Not GP, but seems correct. You can check the Github repo too and they have linked the paper

                                                                                                1. 2

                                                                                                  Yes, that is one of the papers.

                                                                                              1. 1

                                                                                                A couple of clarifying questions:

                                                                                                1. You state that if you haven’t received an ack within X milliseconds, to mark the current message as sent and proceed. If you don’t care about retries, why not remove the requirement to listen to acks in the first place?
                                                                                                2. How important is event ordering to you? For most event architectures, it’s worth it to quash that requirement due to increased complexity.
                                                                                                3. What’s worse: a user not receiving a message, or a user receiving more than one copy of a message?
                                                                                                1. 2
                                                                                                  1. I get acks 85%-90% of the times. So, I would like to optimise it so that it is ordered for maximum number of users and let it go out of order for few. Also, by adding this X amount of delay, the message is usually sent to user as ordered. The messages are going out of order when I send them instantly.

                                                                                                  2. The current system is unordered and works really well (scale, maintainability). However, a lot of messages are sent out of order. So, ordering is very important. My naive solution is to add a delay of X ms after every message and it should solve for most cases. However, I would be slowing down simply and I don’t want to do that.

                                                                                                  3. User not receiving a message is worse. But I would try not send multiple times either.

                                                                                                  1. 4

                                                                                                    Have you considered enabling PubSub ordering, with the ordering key being the user/room? Some of the tradeoffs are that you will be limited in your throughput (1MB/s) per ordering key, and will be vulnerable to hot sharding issues.

                                                                                                    After enabling ordering, if the ordering issue still impacts a greater fraction of users than you would like, then the problem is most likely on the upstream side (Insta/WhatsApp). AFAIK there is no ordering guarantee for those services, even if you wait for acks.

                                                                                                    My advice: if the current solution is working great without ordering, I would strongly suggest sticking with it.

                                                                                                    1. 2

                                                                                                      Once I enable ordering on the queue, it becomes difficult to distribute the work within multiple workers, right?

                                                                                                      if the current solution is working great without ordering, I would strongly suggest sticking with it.

                                                                                                      I so wish to do this, but seems I can’t :(

                                                                                                      1. 3

                                                                                                        Has someone actually quantified how impactful out of order messages are to the business? This is the kind of bug that a less-technical manager or PM can prioritize highly without doing due diligence.

                                                                                                        Another suggestion is to make a design, and be sure to highlight whatever infrastructure costs are changing (increasing most likely), as well as calling out the risks of increasing the complexity of the system. You have the agency to advocate for what you think is right. If they decide to proceed with the design then go ahead and get the experience and you’ll find out if the warnings were warranted over time.

                                                                                                        Quantifying the impact is a good exercise for you to do anyway, since if you build the system you can then put an estimate of the value you created on your resume.

                                                                                                        1. 2

                                                                                                          Correct; you will only be able to have one worker per ordering key, or you lose your ordering guarantee.

                                                                                                      2. 2

                                                                                                        If you want to avoid duplicates and lost messages, the only solution is to use idempotent APIs to send messages. If you do not receive an ack within some time, resend the message idempotently; lost sends/acks are repeated and the API provider filters the duplicates. Only proceed to sending message N+1 once you eventually succeed sending message N.

                                                                                                        If your API provider does not provide an idempotent API, then you could try to query their API for your last sent message and compare it with the one you plan to send. But this is slow and, since it’s not atomic / transactional, is very racey.