Threads for amontalenti

  1. 24

    Thank you, Lobste.rs! 🦞 It remains my favorite single site on the internet, many years running. (I joined 8 years ago.)

    Looking back across all my submissions, there was some lovely “crusty” discussion on this article, “On the use of a life”, penned by cpercival.

    https://lobste.rs/s/d8wxhi/on_use_life

    Although the submission suffered from a few “barnacles”, too. (That is, a few off-topic downvotes.)

    1. 4

      This is a little bit of a nitpick to an otherwise very good article, but I was a little surprised the article didn’t mention that ** is also the “raise to the power” operator as well as the multiplication operator.

      1. 1

        Good point. Also, beyond numeric multiplication, * is used more generally as a shorthand for “repeat” on multiple data types, so that "-" * 30 means repeat the string "-" 30 times to create a 30-char horizontal rule string, and [0] * 30 means repeat the list [0] 30 times to make a 30-element list of zeroes.

        1. 2

          When I’m explaining it to people newer to Python I generally avoid names like “power operator” or “multiplication operator” or”addition operator” and instead just focus on the symbol, because operator overloading means there’s no single universal semantic meaning for any of these operators. So + is an addition operator on some types and a concatenation operator on others; * is multiplication on some types, repetition on others, and so on. Better to avoid trying to stick one and only one default meaning onto them.

      1. 7

        How will you ensure that you can still build zig from sources in the future?

        1. 27

          By forever maintaining two implementations of the compiler - one in C, one in Zig. This way you will always be able to bootstrap from source in three steps:

          1. Use system C compiler to build C implementation from source. We call this stage1. stage1 is only capable of outputting C code.
          2. Use stage1 to build the Zig implementation to .c code. Use system C compiler to build from this .c code. We call this stage2.
          3. Use stage2 to build the Zig implementation again. The output is our final zig binary to ship to the user. At this point, if you build the Zig implementation again, you get back the same binary.

          https://github.com/ziglang/zig-bootstrap

          1. 7

            I’m curious, is there some reason you don’t instead write a backend for the Zig implementation of the compiler to output C code? That seems like it would be easier than maintaining an entirely separate compiler. What am I missing?

            1. 2

              That is the current plan as far as I’m aware

              1. 1

                The above post says they wanted two separate compilers, one written in C and one in Zig. I’m wondering why they just have one compiler written in Zig that can also output C code as a target. Have it compile itself to C, zip up the C code, and now you have a bootstrap compiler that can build on any system with a C compiler.

                1. 2

                  In the above linked Zig Roadmap video, Andrew explains that their current plan is halfway between what you are saying and what was said above. They plan to have the Zig compiler output ‘ugly’ C, then they will manually clean up those C files and version control them, and as they add new features to the Zig source, they will port those features to the C codebase.

                  1. 2

                    I just watched this talk and learned a bit more. It does seem like the plan is to use the C backend to compile the Zig compiler to C. What interests me though is there will be a manual cleanup process and then two separate codebases will be maintained. I’m curious why an auto-generated C compiler wouldn’t be good enough for bootstrapping without manual cleanup.

                    1. 7

                      Generated source code usually isn’t considered to be acceptable from an auditing/chain of trust point of view. Don’t expect the C code generated by the Zig compiler’s C backend to be normal readable C, expect something closer to minified js in style but without the minification aspect. Downloading a tarball of such generated C source should be considered equivalent to downloading an opaque binary to start the bootstrapping process.

                      Being able to trust a compiler toolchain is extremely important from a security perspective, and the Zig project believes that this extra work is worth it.

                      1. 2

                        That makes a lot of sense! Thank you for the clear and detailed response :)

                      2. 2

                        It would work fine, but it wouldn’t be legitimate as a bootstrappable build because the build would rely on a big auto-generated artifact. An auto-generated artifact isn’t source code. The question is: what do you need to build Zig, other than source code?

                        It could be reasonable to write and maintain a relatively simple Zig interpreter that’s just good enough to run the Zig compiler, if the interpreter is written in a language that builds cleanly from C… like Lua, or JavaScript using Fabrice Bellard’s QuickJS.

                        1. 1

                          Except that you can’t bootstrap C, so you’re back where you started?

                          1. 2

                            The issue is not to be completely free of all bootstrap seeds. The issue is to avoid making new ones. C is the most widely accepted and practical bootstrap target. What do you think is a better alternative?

                            1. 1

                              C isn’t necessarily a bad choice today, but I think it needs to be explicitly acknowledged in this kind of discussion. C isn’t better at being bootstrapped than Zig, many just happen to have chosen it in their seed.

                              A C compiler written in Zig or Rust to allow bootstrapping old code without encouraging new C code to be written could be a great project, for example.

                              1. 5

                                This is in fact being worked on: https://github.com/Vexu/arocc

                  2. 1

                    Or do like Golang. For bootstrap you need to:

                    1. Build Go 1.4 (the last one made in C)
                    2. Build the latest Go using the compiler from step 1
                    3. Build the latest Go using the compiler from step 2
                  3. 3

                    Build the Zig compiler to Wasm, then run it to cross-compile the new compiler. Wasm is forever.

                    1. 11

                      I certainly hope that’s true, but in reality wasm has existed for 5 years and C has existed for 50.

                      1. 2

                        The issue is building from maintained source code with a widely accepted bootstrapping base, like a C compiler.

                        The Zig plan is to compile the compiler to C using its own C backend, once, and then refactor that output into something to maintain as source code. This compiler would only need to have the C backend.

                        1. 1

                          I mean, if it is, then it should have the time to grow some much needed features.

                          https://dl.acm.org/doi/10.1145/3426422.3426978

                        2. 1

                          It’s okay if you don’t know because it’s not your language, but is this how Go works? I know there’s some kind of C bootstrap involved.

                          1. 4

                            The Go compiler used to be written in C. Around 1.4 they switched to a Go compiler written in Go. If you were setting up an entirely new platform (and not use cross compiling), i believe the recommended steps are still get a C compiler working, build Go 1.4, then update from 1.4 to latest.

                        3. 2

                          How do we build C compilers from source?

                          1. 3

                            Bootstrapping a C compiler is usually much easier than bootstrapping a chain of some-other-language compilers.

                            1. 4

                              Only if you accept a c compiler in your bootstrap seed and don’t accept a some-other-language compiler in your seed.

                              1. 3

                                Theoretically. But from a practical point of view? Yes, there are systems like Redox (Rust), but in most cases the C compiler is an inevitable piece of the puzzle (the bootstrapping chain) when building an operating system. And in such cases, I would (when focused on simplicity) rather prefer a language that depends just on C (that I already have) instead of a sequence of previous versions of its own compilers. (and I say that as someone, who does most of his work in Java – which is terrible from the bootstrapping point-of-view)

                                However, I do not object much against the dependence on previous versions of your compiler. It is often the way to go, because you want to write your compiler in a higher language instead of some old-school C and because you create a language and you believe in its qualities, you use it also for writing the compiler. What I do not understand is why someone (not this particular case, I saw this pattern before many times) present the “self-hosted” as an advantage…

                                1. 2

                                  The self-hosted Zig compiler provides much faster compile times and is easier to hack, allowing language development to move forward. In theory the gains could be done in a different language, but some of the kind of optimizations used are exactly the kind of thing Zig is good at. See this talk for some examples: https://media.handmade-seattle.com/practical-data-oriented-design/.

                                2. 1

                                  But you could make a C compiler (or a C interpreter) from scratch relatively easily.

                          1. 5

                            Possibly relevant, my blog post on Lucene entitled “Lucene: The Good Parts”.

                            https://blog.parse.ly/lucene/

                            1. 52

                              I’m always relieved when I see a doctor look things up instead of just relying on what they remember from 20 years past.

                              1. 24

                                Doctors also don’t typically use Google/Wikipedia. They use a service called UpToDate*, which is commercial and has its articles written by verified MDs and researchers, but with Google-like search and Wikipedia-like organization. There’s an interesting profile/obituary of the company’s founder here:

                                https://www.statnews.com/2020/04/25/remembering-uptodate-creator-burton-bud-rose/

                                *: I know because my wife is an MD.

                                1. 9

                                  I assume this varies by region, but some surveys suggest that a lot of doctors (especially junior doctors) end up using Wikipedia or other free resources in practice. E.g. a 2013 study of European doctors found that:

                                  In line with previous research [6], general-purpose search engines (eg, Google), medical research databases (eg, PubMed), and Wikipedia were popular resources, while specialized search engines were unpopular. […] Even though point-of-care databases (eg, UpToDate) provide reliable, evidence-based clinical information, the reported use among physicians was shown to be limited. A possible explanation could be that most physicians are not willing to pay high subscription fees to access medical information.

                                  1. 3

                                    Yea, perhaps my word choice on “typically” is doing too much work there, or, assuming too much. I revise my statement to say, “Doctors have the option not to use Google/Wikipedia, but still be able to look up quality information with simple search queries.”

                                    I’m sure you are right that, as the study suggests, many doctors don’t use a service like UpToDate. It is definitely a cost (although, a trivial cost, compared to other healthcare costs, and ridiculously easy to justify on ROI basis for a practice or hospital). Many of my wife’s friends who did career changes out of medicine even keep their UpToDate subscription (on a personal basis), simply to be able to guide their own (or their family’s) care a little when they are seen by other doctors. IMO, UpToDate is a great service and every MD should have access.

                                    Also, I should mention that my wife is quite young, as far as doctors go, and there is a generational divide here. Many doctors who came of age before the information era were forced to “search their brain” for all the answers, so I imagine many of those doctors haven’t adapted to the internet age merely out of habit.

                                  2. 9

                                    If you have it. I do a lot of Google still. No institutional site license here and I’m not wild about the per cost.

                                    (source: also an MD)

                                    1. 3

                                      I use it to the tune of 100+ CME points a year so to me it’s worth the $52/mo. I know colleagues who split a subscription too.

                                    2. 4

                                      I’ve observed my GP doctor type stuff into Google and click on a few links. Usually whatever page they land on looks like an official source of some kind, rather than an SEO optimised opinion piece (so, they probably don’t click on the first result).

                                      I’m generally fine with that - I trust my Dr to have enough background understanding to gauge whether an article is factual or not.

                                  1. 70

                                    Nobody knows how to correctly install and package Python apps.

                                    That’s a relief. I thought I was the only one.

                                    1. 8

                                      Maybe poetry and pyoxidize will have a baby and we’ll all be saved.

                                      One can hope. One can dream.

                                      1. 4

                                        After switching to poetry, I’ve never really had any issues.

                                        pip3 install --user poetry
                                        git clone...
                                        cd project
                                        poetry install
                                        poetry run python -m project
                                        

                                        You can pull the whole install sequence in a Docker container, push it in your CI/CD to ECR/Gitlab or whatever repo you use, and just include both the manual and the docker command in your readme. Everyone on your team can use it. If you find an issue, you can add that gotcha do the docs.

                                        Python is fine for system programming so long as you write some useful unittests and force pycodestyle. You loose the type-safety of Go and Rust, yes, but I’ve found they’re way faster to write. Of course if you need something that’s super high performance, Go or Rust should be what you look towards (or JVM–Kotlin/Java/Scala if you don’t care about startup time or memory footprints). And of course, it depends on what talent pools you can hire from. Use the right tool for the right job.

                                        1. 2

                                          I’ve switched to poetry over the last several months. It’s the sanest installing python dependencies has felt in quite a few years. So far I prefer to export it to requirements.txt for deployment. But it feels like about 95% of the right answer.

                                          It does seem that without some diligence, I could be signing up for some npm-style “let’s just lock in all of our vulnerabilities several versions ago” and that gives me a little bit of heartburn. From that vantage point, it would be better, IMO, to use distro packages that would at least organically get patched. I feel like the answer is to “just” write something to update my poetry packages the same way I have a process to keep my distro packages patched, but it’s a little rotten to have one more thing to do.

                                          Of course, “poetry and pyoxidize having a baby” would not save any of this. That form of packaging and static linking might even make it harder to audit for the failure mode I’m worrying about here.

                                        2. 1

                                          What are your thoughts on pipenv?

                                        3. 5

                                          I’d make an exception to this point: “…unless you’re already a Python shop.” I did this at $job and it’s going okay because it’s just in the monorepo where everyone has a Python toolchain set up. No installation required (thank god).

                                          1. 4

                                            I think the same goes for running Python web apps. I had a conversation with somebody here… and we both agreed it took us YEARS to really figure out how to run a Python web app. Compared to PHP where there is a good division of labor between hosting and app authoring.

                                            The first app I wrote was CGI in Python on shared hosting, and that actually worked. So that’s why I like Unix – because it’s simple and works. But it is limited because I wasn’t using any libraries, etc. And SSL at that time was a problem.

                                            Then I moved from shared hosting to a VPS. I think I started using mod_python, which is the equivalent of mod_php – a shared library within Apache.

                                            Then I used a CherryPy server and WSGI. (mod_python was before WSGI existed) I think it was behind Apache.

                                            Then I moved to gunicorn behind nginx, and I still use that now.

                                            But at the beginning of this year, I made another small Python web app with Flask. I managed to configure it on shared hosting with FastCGI, so Python is just like PHP now!!! (Although I wouldn’t do this for big apps, just personal apps).

                                            So I went full circle … while all the time I think PHP stayed roughly the same :) I just wanted to run a simple app and not mess with this stuff.

                                            There were a lot of genuine improvements, like gunicorn is better than CherryPy, nginx is easier to config than Apache, and FastCGI is better than CGI and mod_python … but it was a lot of catching up with PHP IMO. Also FastCGI is still barely supported.

                                            1. 2

                                              nginx, uWSGI, supervisord. Pretty simple to setup for Flask or Django. A good shared hosting provider for Python is OpalStack, made by the people who created Webfaction (which, unfortunately, got gobbled up by GoDaddy).

                                              I cover the deployment options and reasoning in my popular blog post, “Build a web app fast: Python, JavaScript & HTML resources”. Post was originally written in 2012 but updated over the years, including just this month. See especially the recommended stack section at the end, starting at “Conclusion: pick a stack”, if you want to ctrl+f for that section. You can also take a peek at how OpalStack describes their Python + uWSGI + nginx shared hosting setup here. See also my notes on the under the hood configuration for nginx, uWSGI, and supervisord in this presentation, covered in the 5-6 sections starting from this link.

                                              You’re right that there are a lot of options for running a Python web app. But nginx, uWSGI, supervisord is a solid option that is easy to configure, high performance, open source, UNIXy, and rock solid. For dependency management in Python 3.x you can stick with pip and venv, remotely configured on your server via SSH.

                                              My companies have been using this stack in production at the scale of hundreds of thousands of requests per second and billions of requests per month – spanning SaaS web apps and HTTP API services – for years now. It just works.

                                              1. 2

                                                I’m curious, now that systemd is available in almost all Linux distributions by default, why are you still using supervisord? To me it feels like it is redundant. I’m very interested.

                                                1. 1

                                                  I think systemd can probably handle the supervisord use cases. The main benefit of supervisord is that it runs as whatever $USER you want without esoteric configuration, and it’s super clear it’s not for configuring system services (since that’s systemd’s job). So when you run supervisorctl and list on a given node, you know you are listing “my custom apps (like uwsgi or tornado services)”, not all the system-wide services as well as my custom app’s ones. Also this distinction used to matter more when systemd was less standard across distros.

                                                  1. 1

                                                    Understood! Thanks very much for taking the time to explain!

                                                2. 1

                                                  Hm thanks for the OpalStack recommendation, I will look into it. I like shared hosting / managed hosting but the Python support tends to be low.

                                                  I don’t doubt that combination is solid, but I think my point is more about having something in the core vs. outside.

                                                  PHP always had hosting support in the core. And also database support. I recall a talk from PHP creator Ramsus saying how in the early days he spent a ton of time inside Apache, and committed to Apache. He also made some kind of data limiting support to SQL databases to make them stable. So he really did create “LAMP”, whereas Python had a much different history (which is obviously good and amazing in its own way, and why it’s my preferred language).

                                                  Similar to package management being outside the core and evolving lots of 3rd party solutions, web hosting was always outside the core in Python. Experts knew how to do it, but the experience for hobbyists was rough. (Also I 100% agree about not developing on Windows. I was using Python on Windows to make web apps from ~2003-2010 and that was a mistake …)

                                                  It obviously can be made to work, I mean YouTube was developed in Python in 2006, etc. I just wanted to run a Python web app without learning about mod_python and such :) Similarly I wish I didn’t know so much about PYTHONPATH!

                                                  1. 1

                                                    I agree with all that. This is actually part of the reason I started playing with and working on the piku open source project earlier this year. It gives Python web apps (and any other Python-like web app programming environments) a simple git-push-based deploy workflow that is as easy as PHP/Apache used to be, but also a bit fancier, too. Built atop ssh and a Linux node bootstrapped with nginx, uWSGI, anacrond, and acme.sh. See my documentation on this here:

                                                    https://github.com/amontalenti/webappfast-piku#build-a-web-app-fast-with-piku

                                                    1. 1

                                                      Very cool, I hadn’t seen piku! I like that it’s even simpler than dokku. (I mentioned dokku on my blog as an example of something that started from a shell script!)

                                                      I agree containers are too complex and slow. Though I think that’s not fundamental, and is mostly Docker … In the past few days, I’ve been experimenting with bubblewrap to run containers without Docker, and different tools for buliding containers without Docker. (podman is better, but it seems like it’s only starting to get packaged on Debian/Ubuntu, and I ran into packaging bugs.)

                                                      I used containers many years ago pre-Docker, but avoided them since then. But now I’m seeing where things are at after the ecosystem has settled down a bit.

                                                      I’m a little scared of new Python packaging tools. I’ve never used pyenv or pipx; I use virtualenv when I need it, but often I just manually control PYTHONPATH with shell scripts :-/ Although my main language is Python, I also want something polyglot, so I can reuse components in other languages.

                                                      That said I think piku and Flask could be a very nice setup for many apps and I may give it a spin!

                                                      1. 1

                                                        It’s still a very new and small project, but that’s part of what I love about it. This talk on YouTube gives a really nice overview from one of its committers.

                                                  2. 1

                                                    In addition to @jstoja’s question about systemd vs supervisord, I’d be very curious to hear what’s behind your preference for nginx and uWSGI as opposed to caddy and, say, gunicorn. I kind of want caddy to be the right answer because, IME, it makes certificates much harder to screw up than nginx does.

                                                    Have you chosen nginx over caddy because of some gotcha I’m going to soon learn about very unhappily?

                                                    1. 2

                                                      Simple answer: age/stability. nginx and uWSGI have been running fine for a decade+ and keep getting incrementally better. We handle HTTPS with acme.sh or certbot, which integrate fine with nginx.

                                                      1. 1

                                                        That’s a super-good point. I’m going to need to finish the legwork to see whether I’m willing to bet on caddy/gunicorn being as reliable as nginx/uWSGI. I really love how terse the Caddy config is for the happy path. Here’s all it is for a service that manages its own certs using LetsEncrypt, serves up static files with compression, and reverse proxies two backend things. The “hard to get wrong” aspect of this is appealing. Unless, of course, that’s hiding something that’s going to wake me at 3AM :)

                                                3. 3

                                                  Why is Python’s packaging story so much worse than Ruby’s? Is it just that dependencies aren’t specified declaratively in Python, but in code (i.e. setup.py), so you need to run code to determine them?

                                                  1. 9

                                                    I dunno; if it were me I’d treat Ruby exactly the same as Python. (Source: worked at Heroku for several years and having the heroku CLI written in Ruby was a big headache once the company expanded to hosting more than just Rails apps.)

                                                    1. 3

                                                      I agree. I give perl the same handling, too. While python might be able to claim a couple of hellish inovations in this area, it’s far from alone here. It might simply be more attractive to people looking to bang out a nice command line interface quickly.

                                                    2. 6

                                                      I think a lot of it is mutable global variables like PYTHONPATH which is sys.path. The OS, the package managers, and the package authors often fight over that, which leads to unexpected consequences.

                                                      It’s basically a lack of coordination… it kinda has to be solved in the core, or everybody else is left patching up their local problems, without thinking about the big picture.

                                                      Some other reasons off the top of my head:

                                                      • Python’s import mechanism is very dynamic, and also inefficient. So the language design kind of works against the grain of easy distribution, although it’s a tradeoff.
                                                      • There’s a tendency to pile more code and “solutions” on top rather than redoing things from first principles. That is understandable because Python has a lot of users. But there is definitely a big mess with distutils + setuptools + pip + virtualenv, plus a few other things.
                                                      • Package managers are supposed to solve versioning issues, and then you have the tricky issue of the version of the package manager itself. So in some sense you have to get a few things right in the design from the beginning!
                                                      1. 5

                                                        Ruby’s packaging story is pretty bad, too.

                                                        1. 3

                                                          In what way?

                                                          1. 4

                                                            I don’t know, it’s been a long time since I’ve written any Ruby. All I know is that we’re migrating the Alloy website from Jekyll to Hugo because nobody could get Jekyll working locally, and a lot of those issues were dependency related.

                                                        2. 4

                                                          Gemfile and gemspec are both just ruby DSLs and can contain arbitrary code, so that’s not much different.

                                                          One thing is that pypi routinely distributes binary blobs that can be built in arbitrarily complex ways called “wheels” whereas rubygems always builds from source.

                                                          1. 5

                                                            Not true. Ruby has always been able to package and distribute precompiled native extensions, it’s just that it wasn’t the norm in a lot of popular gems, including nokogiri. Which by the way, ships precompiled binaries now, taking couple of seconds where it used to take 15m, and now there’s an actual tool chain for targeting multi arch packaging, and the community is catching up.

                                                            1. 2

                                                              Hmm, that’s very unfortunate. I haven’t run into any problems with gems yet, but if this grows in popularity the situation could easily get as bad as pypi.

                                                            2. 1

                                                              Thanks for the explanation, so what is the fundamental unfixable issue behind Python’s packaging woes?

                                                              1. 1

                                                                I could be wrong but AFAICT it doesn’t seem to be the case that the Ruby crowd has solved deployment and packaging once and for all.

                                                            1. 2

                                                              I just run pkg install some-python-package-here using my OS’s package manager. ;-P

                                                              It’s usually pretty straightforward to add Python projects to our ports/package repos.

                                                              1. 3

                                                                Speaking from experience, that works great up until it doesn’t. I have “fond” memories of an ex-coworker who developed purely on Mac (while the rest of the company at the time was a Linux shop), aggressively using docker and virtualenv to handle dependencies. It always worked great on his computer! Sigh. Lovely guy, but his code still wastes my time to this day.

                                                                1. 1

                                                                  I guess I’m too spoiled by BSD where everything’s interconnected and unified. The ports tree (and the package repo that is built off of it) is a beauty to work with.

                                                                  1. 4

                                                                    I’m as happy to be smug as the next BSD user but it isn’t justified in this case. Installing Python packages works for Python programs installed from packages but:

                                                                    • They don’t work well in combination with things not in packages, so if you need to use pip to install some things you may end up with conflicts.
                                                                    • The versions in the package repo may or may not be the ones that the thing you want to install that isn’t in packages need, and may conflict with the ones it needs.
                                                                    • The Python thing may depend on one of the packages that depends on Linux-specific behaviour. The most common of these is that signals sent to the process are delivered to the first thread in the process.

                                                                    In my experience, there’s a good chance that a Python program will run on the computer of the author. There’s a moderately large chance that it will run on the same OS and version as the author. Beyond that, who knows.

                                                                    1. 3

                                                                      I mean, we used Ubuntu, which is pretty interconnected and unified. (At the time; they’re working on destroying that with snap.) It just often didn’t have quiiiiiite what we, or at least some of us, wanted and so people reached for pip.

                                                                      1. 1

                                                                        Yeah. With the ports tree and the base OS, we have full control over every single aspect of the system. With most Linux distros, you’re at the whim of the distro. With BSD, I have full reign. :-)

                                                                        1. 3

                                                                          But it could still be the case that application X requires Python 3.1 when application Y requires Python 3.9, right? Or X requires version 1.3 of library Z which is not backwards compatible with Z 1.0, required by Y?

                                                                          1. 3

                                                                            The Debian/Ubuntu packaging system handles multiple versions without any hassle. That’s one thing I like about it.

                                                                            1. 1

                                                                              Does it? Would love to read more about this if you have any pointers!

                                                                              1. 2

                                                                                I guess the main usability thing to read about it the alternatives system.

                                                                            2. 2

                                                                              The ports tree handles multiple versions of Python fine. In fact, on my laptop, here’s the output of: pkg info | grep python:

                                                                              py37-asn1crypto-1.4.0          ASN.1 library with a focus on performance and a pythonic API
                                                                              py37-py-1.9.0                  Library with cross-python path, ini-parsing, io, code, log facilities
                                                                              py37-python-docs-theme-2018.2  Sphinx theme for the CPython docs and related projects
                                                                              py37-python-mimeparse-1.6.0    Basic functions for handling mime-types in Python
                                                                              py37-requests-toolbelt-0.9.1   Utility belt for advanced users of python-requests
                                                                              py38-dnspython-1.16.0          DNS toolkit for Python
                                                                              python27-2.7.18_1              Interpreted object-oriented programming language
                                                                              python35-3.5.10                Interpreted object-oriented programming language
                                                                              python36-3.6.15_1              Interpreted object-oriented programming language
                                                                              python37-3.7.12_1              Interpreted object-oriented programming language
                                                                              python38-3.8.12_1              Interpreted object-oriented programming language
                                                                              
                                                                  2. 1

                                                                    Fwiw, I’ve had good luck using Pyinstaller to create standalone binaries. Even been able to build them for Mac in Circleci.

                                                                    1. 1

                                                                      It can feel a bit like overkill at times, but I’ve had good luck with https://www.pantsbuild.org/ to manage python projects.

                                                                    1. 5

                                                                      This was a surprisingly interesting article that wasn’t actually that much about Scala usage. It was more about how the developer environment is setup at a large and very technical company that happens to use Scala for most of its foundation code. Almost entirely with open source and public cloud tooling. We are talking hundreds of developers and millions of lines of production code, spanning several languages, and spanning closed source and open source production projects, many with important tie-ins to third party open source or public cloud APIs.

                                                                      Here is a paragraph from the bottom of the article that summarizes what was covered:

                                                                      One point of interest is how generic many of our tools and techniques are. Our CI system, devboxes, remote cache, test shards, etc. are not Scala-specific. Neither is our strategy for dependency management or linting. Much of these apply regardless of language or platform and benefit our developers writing Python or Typescript or C++ as much as those writing Scala.

                                                                      1. 1

                                                                        Though this is from 2014, this just showed up on ClojureTV a few days ago.

                                                                        From the YouTube description:

                                                                        This talk was given at EuroClojure 2014. While this has been hosted elsewhere previously, this is a new edit with slides integrated (for the first time) and improved audio.

                                                                        Here is the write-up of what core.async channels are all about, from the Clojure website:

                                                                        The objectives of core.async are:

                                                                        • To provide facilities for independent threads of activity, communicating via queue-like channels
                                                                        • To support both real threads and shared use of thread pools (in any combination), as well as ClojureScript on JS engines
                                                                        • To build upon the work done on CSP [Communicating Sequential Processes] and its derivatives

                                                                        It is our hope that async channels will greatly simplify efficient server-side Clojure programs, and offer simpler and more robust techniques for front-end programming in ClojureScript.

                                                                        1. 6

                                                                          I know the OP personally and was in conversation with him when this blog post was written. We were both CTOs of startups with big time series data that so happened to have several engineering staff based in the Cville, VA tech community. Baron was the CTO of VividCortex, a database observability and performance metrics system. I was the founding CTO of Parse.ly, a high-traffic web content analytics system. We each built relatively huge time series systems – the Parse.ly one eventually summarized 1.5 petabytes of customer data using 15-20 TiB of time series index data. The backend behind the Parse.ly system that I helped design is called mage. It went into production in 2015, and is still in production with largely the same design, scaling to 100-1000x the customer data it originally stored.

                                                                          I think the requirements that Baron laid out here are pretty spot-on. A colleague of Baron’s, Preetam Jinka, maintains a blog post with a big list of open source time series databases, which started in 2016 and has been maintained up through late 2019.

                                                                          At the time, none of these were particularly good options: we ended up building our system atop these components:

                                                                          • Amazon S3: used to keep raw data at rest
                                                                          • Elasticsearch: aggregation resting place; time series indices, storage, and live massively-parallel user queries
                                                                          • Kafka: used to keep raw data in motion
                                                                          • Cassandra: used to time-order recent enriched events in a cluster setting
                                                                          • Storm: used to create micro-batch streaming 5-minute & 1-second rollups with very low latency (Kafka => Cassandra => Elasticsearch)
                                                                          • Spark: used to create large-batch 24-hour rollups from events, leveraging elastic compute nodes (S3 => Elasticsearch)

                                                                          We unified the Storm and Spark layers with shared Python code between pyspark (already popular in the Spark community) and some open source libraries we released ourselves, called pystorm and streamparse.

                                                                          We used Elasticsearch for the actual time series index storage and query flexibility, taking advantage of its recently-added support for aggregations, columnar storage, and time-based indices, as described here. We also wrote our own higher-level query DSL atop the system, so that we could make queries against Elasticsearch using a SQL-like interface with explicit support for time series constructs. This “whole system” is what is described as “mage” post above.

                                                                          Building a scalable system like this in 2015-2016 was pretty darn tough (but we pulled it off). These days, you see some cloud databases explicitly tackling time series use cases, like Google BigQuery and Amazon Athena / Redshift-Spectrum, but even those can’t support super low latency or interactive queries very easily. They are more like “SQL alternatives” to using Apache Spark with Amazon EMR. You also see some time series databases that are centered around what I consider to be the “graphite” or “rrdtool” model, which throws away too much data to support the kinds of query flexibility that Baron describes here. A modern example of that is using something like Prometheus with Grafana, but I consider that to almost be a completely different category of data storage. Interesting, but different.

                                                                          In recent times, a couple of open source databases that seem to be going “all-in” on time series use cases are TimescaleDB, ClickHouse, and Druid. These have more of the facilities that Baron describes here, but have some of their own quirks.

                                                                          1. 1

                                                                            Could you expand on Prometheus being different, please? I’m guessing influxdb falls into the same “different” category?

                                                                            1. 1

                                                                              Yea, InfluxDB is also sort of in this category.

                                                                              Maybe the simplest way to explain is to pick the most pathological style of queries:

                                                                              • imagine you have to do unique value counts (cardinality) using data sketches like HyperLogLog++

                                                                              • imagine you have high cardinality labels, e.g. each event can have any number of strings associated with it, and there are millions of such unique strings in the data corpus

                                                                              • and imagine you want to performantly group by each label, sort by each unique count (set cardinality count), over any arbitrary time window (last 5 minutes, last 24h, last 7d, last 365d, but also Aug 1 - Oct 1, or Jan 1 - Feb 1, etc.)

                                                                              This is a pretty “straightforward” thing to do with the mage system I described, leveraging the aggregations framework in Elasticsearch and its “term grouping”, “sum”, and “cardinality” facilities. But is quite impossible to do with graphite/rrdtool systems, including Prometheus, to my knowledge.

                                                                              Baron used to demonstrate this by saying, “I want my time series database to not only show me the slowest measurements on a server by running a time range span with a by-server grouping and a performance metric sum resampled by minute to render my graph, but I also want to be able to get a listing of what exact queries were slow that contributed to that graph”. Put another way, this is about data lossiness. Many “time series databases” are efficient because they throw away the raw events entirely (e.g. rrdtool, graphite). They perhaps “only” store counts/metrics, and sometimes even statistically sampled counts/metrics. A good time series database as Baron describes would likely keep 100% of the unsampled raw events somewhere cheap (we used Kafka & S3 for this), and then build increasingly cpu/io-efficient “rollups” of raw events over time in the database, without complicating the query language or scaling story.

                                                                              Usually deep historical queries are a bit more “lossy” than recent-time queries, so that storage gets cheaper as data “ages out”. In our system, for example, we threw away hourly/minutely time resolution for very old data, only keeping daily resolution, but still with full support for arbitrary dimension slice/dice, including for metrics powered by data sketches (like unique count cardinality).

                                                                              In all cases, though, there should be some way to rebuild the world from the raw data to compute new time series views or migrate schemas. This is also something that becomes very tricky with graphite/rrdtool systems.

                                                                              1. 1

                                                                                Ok, i see. I’m not sure about Prometheus, which is why I asked, but Influxdb should be able to do all of that just fine. If not through original influxql, then definitely through flux language. The raw events are available / queryable including the stream processing and you can do expiry at different schedules for it and for the rollups. In v2 there are nice tasks for it https://docs.influxdata.com/influxdb/v2.0/process-data/common-tasks/downsample-data/

                                                                                Now I’m curious if someone familiar with Prometheus can confirm. I thought it doesn’t immediately rollup and throw away raw events like rrd.

                                                                                1. 2

                                                                                  Prometheus can also store raw metrics data, and it use remote storage for out-aged metrics to save costs. Rollup can be achieved via recording rules.

                                                                                  1. 1

                                                                                    Yea, very likely. I think TimescaleDB can do this as well. I am not familiar with “modern” InfluxDB, only the state of the project back in 2014-2016 when Baron was writing about it and I was designing our system. At that time, InfluxDB was more like “graphite with a cluster story”.

                                                                            1. 2

                                                                              Also: it’s not very well known, but Jupyter has a pluggable kernel architecture. Even the name Jupyter is a portmanteau of “Julia, Python, and R” – three core languages supported by Jupyter via kernels. It is best known as “IPython Notebooks” due to the project’s history (and because parts of Jupyter are implemented in Python), but other languages work, too. The Clojupyter project provides such a kernel for Clojure. And it actually works pretty nicely. I am a Python + Clojure (see also: Clojonic) programmer so I use it relatively frequently.

                                                                              1. 3

                                                                                People might be missing some context that reviewing the June 2019 research article for this allocator is particularly timely because in October 2021, this allocator was used as a replacement for “pymalloc” in a proposed changeset written by Sam Gross against the CPython interpreter which removes the GIL (Global Interpreter Lock), potentially enabling multi-core parallelism for many workloads for Python code.

                                                                                This is described briefly on LWN here. I also x-posted a nice breakdown of this CPython proposal from CPython Developer-in-Residence Łukasz Langa, which can be found here on lobste.rs.

                                                                                1. 1

                                                                                  Both mimalloc and snmalloc are much simpler than tcmalloc or jemalloc, primarily (AFAICT) because they don’t use thread-local freelists, which seems to be the primary source of complexity in other high-performance allocators. I feel like “you need thread-local freelists for performance” is perhaps the biggest contemporary myth in allocator design.

                                                                                  1. 2

                                                                                    I feel like “you need thread-local freelists for performance” is perhaps the biggest contemporary myth in allocator design.

                                                                                    I don’t think that’s quite an accurate characterisation. Snmalloc does have thread-local freelists. There’s a philosophical difference between something like jemalloc and snmalloc. Jemalloc is a single-threaded global allocator that then does local caching to try to reduce contention. Snmalloc is a single-threaded local allocator that uses an actor-model design to allow multiple instantiation. Every thread owns an snmalloc allocator object. This object owns a load of chunks, which it uses to perform allocation. Each chunk is used to allocate slabs, which are fixed size allocations. On the slow path, we construct a freelist of a particular size after creating a slab and then that freelist is exclusively accessed by a single allocator. If you free from another thread, the allocation is batched and then sent as a message to the owning allocator (if you have more than 64 allocators, it may be sent indirectly, which adds some latency to memory recycling but improves throughput), which pops it into the local freelist.

                                                                                    The hypothesis for snmalloc was that designing an actor-model memory allocator with a fast shared-memory message-passing scheme would give better performance than trying to do lock-based concurrency. So far, this hypothesis has been supported by the evidence.

                                                                                    1. 1

                                                                                      Yeah, I wasn’t clear in my terminology, sorry. I was referring to the common design where the consumer can directly allocate from a cache of objects freed by that thread but allocated by other threads. That’s what I think is commonly referred to as a “tcache” in tcmalloc and (later) jemalloc. This of course leads to false sharing and requires complex GC mechanisms to eventually return cached objects to their allocating heap or to a global heap when the cache gets too large. The key difference with modern allocators like snmalloc and mimalloc is that they never allocate from memory originally allocated by another thread, right?

                                                                                      1. 2

                                                                                        The key difference with modern allocators like snmalloc and mimalloc is that they never allocate from memory originally allocated by another thread, right?

                                                                                        That’s more or less what I said, but I oversimplified very slightly. It’s true for all small and medium allocations but for large allocations we allocate entire pages with mmap or a platform equivalent. When you free one of these, it’s cached locally in the freeing allocator and then returned to a central pool (snmalloc never returns address space, only physical memory), where it can then be used to satisfy later large allocations or can be split up to provide new chunks.

                                                                                        Aside from that corner case, I agree that this is the correct characterisation.

                                                                                1. 9

                                                                                  I don’t like the examples simply because you’re using a lot of filler text in a work environment where everyone’s just trying to get things done.

                                                                                  You: What titles? Yeah, I can definitely do that. What products are you wanting on there?

                                                                                  To me makes much more sense as something like “We might be able to, which titles?”. It doesn’t suggest you have time for it, it doesn’t suggest you’re promising to do it (never promise, surely), and it saves you reiterating their question.

                                                                                  You: We haven’t touched prices today. Hmm, I haven’t had anyone mess with prices today. What products are involved? I’ll check it out

                                                                                  Again, why not just “The prices have changed?” and maybe “Which products, buddy?” if they reply with a dry “yeah”. What’s with all the unnecessary verbosity? The corporate nicety is something we can all see through, and we have to spend time extracting the core message from “friendly” messages just as if they were sarcastic or unkind messages. Let’s just skip that part entirely.

                                                                                  1. 16

                                                                                    nicety

                                                                                    A “nicety” is a small or precise detail, not the state of being nice; that’s “niceness.”

                                                                                    What’s with all the unnecessary verbosity?

                                                                                    Human communication is filled with redundancy; English especially. We do not communicate using minimal ASTs because the signal-to-noise ratio in all of our communication channels is very poor. Adding some additional information requires adding many additional words. When the post suggests “Hmm, I haven’t had anyone mess with prices today. What products are involved? I’ll check it out”, that’s communicating more than “The prices have changed?”. It communicates, “I am surprised to hear this and doubt it’s related to anything I have control over. Give me some more information so I can work on it - but it’s not a bother and I’m not trying to get rid of you, which would be a reasonable reading of that question.”

                                                                                    You may not want to provide that reassurance and context-setting, but it’s not reasonable to say that it’s “unnecessary verbosity.”

                                                                                    1. 11

                                                                                      Perfectly said!

                                                                                      And if you understand what @mtset said but still can’t imagine why you would want to convey all that, just remember that you might not benefit from all that information but the recipient might, and perhaps far more often than you realize. Some examples:

                                                                                      • I worked with a software quality tester who always felt like bringing problems to programmers made him a burden. The total lack of explicit, POSITIVE receptivity to their bug reports over a long period of time created that feeling, and that’s reasonable.
                                                                                      • I worked with a developer who was absolutely terrified of messaging senior developers on the team because he was afraid of how his questions would be perceived. Being explicit that those questions are welcome was almost as important for this person’s career as answering the questions.
                                                                                      • Program Managers often have no idea how a request will be received because they often don’t know the magnitude of what they are asking, in terms of investment required. Every interaction you have with a person where you have a lopsided amount of information means they will be happy to get all that extra context.

                                                                                      Etc…

                                                                                      1. 8

                                                                                        The key thing that is communicated with all the extra words is “Even if I’m not sure I agree, your input is valid and I am taking it seriously.”

                                                                                      2. 7

                                                                                        Again, why not just “The prices have changed?” and maybe “Which products, buddy?” if they reply with a dry “yeah”. What’s with all the unnecessary verbosity?

                                                                                        The author’s suggestion means something completely different than yours and the distinction is important. His response (nicely) indicates “That assumption sounds wrong but I’ll engage in figuring out the problem regardless” because ultimately the assumption being right or wrong is totally irrelevant.

                                                                                        And fwiw, “buddy” is never taken as friendly in most parts of the US (particularly the east coast) as it will be perceived as sarcastic friendliness.

                                                                                        1. 1

                                                                                          Okay, s/buddy/<your regional slang here>/, no big deal?

                                                                                        2. 6

                                                                                          we have to spend time extracting the core message from “friendly” messages just as if they were sarcastic or unkind messages.

                                                                                          The core message alone can easily be misinterpreted as negative/sarcastic/unkind — our brains are predisposed to discover threats. Look up “negativity bias.” In spoken communication there is usually enough metadata in the form of facial expressions or at least tone of voice, but textual communication is very easy to misconstrue.

                                                                                          1. 4

                                                                                            “Manners – simple things like saying ‘please’ and ‘thank you’ and knowing a person’s name or asking after her family – enable two people to work together whether they like each other or not. Bright people, especially bright young people, often do not understand this. If analysis shows that someone’s brilliant work fails again and again as soon as cooperation from others is required, it probably indicates a lack of courtesy – that is, a lack of manners.”

                                                                                            From “Managing Oneself” (1999) by Peter Drucker

                                                                                            1. 1

                                                                                              The examples lacked please/thank you also. I’m not arguing for the removal of politeness, but stripping unnecessary content.

                                                                                          1. 4

                                                                                            We do one day per week, affectionately called “bugday”. Bugs are tracked with GitHub issues, reported by staff, triaged by engineers and leads, then split up for bugday. When you are done with your split of bugs, you can return to project work, which we manage in a Notion board per Iteration (similar to Trello). If bugs are gnarly enough, they become iteration projects.

                                                                                            1. 1

                                                                                              That sounds like a healthy practice. Does it work well in reality?

                                                                                              I’ve commonly seen the triage step skipped, which is unfortunate because it makes working with bug reports so much more fun. Don’t know how many hours (days) I’ve spent fixing some badly reported bug.

                                                                                              1. 4

                                                                                                We’ve been doing it for 8+ years. It started when the engineering team was 3-4 people, and that team is now 25-30 people. It’s one of our most beloved processes (even if we are occasionally grumpy about a gnarly bug), so I’d say it works very well. Sometimes it requires some upkeep, e.g. declaring bankruptcy on certain bugs or classes of tickets, or tweaking the triage process. But, in general, we’ve kept up with “one day per week for bugs” and “keep the bug list small” for years, and I can’t imagine doing it any other way.

                                                                                                You’re right that it’s important to make time for the triage step. On our teams, what happened over time is that a team’s weekly checkpoint meetings would happen on Tuesdays and bugday would be Wednesday, so each team would spend 15 minutes in their weekly meeting triaging/splitting/assigning bugs so that could people could show up on Wednesday with clear assignments to whittle down the list.

                                                                                                The other nice thing about bugday is that it creates a “pressure relief valve” with the support team, but with a reasonable-ish SLA beyond “hotfix”. That is, either something is a “hotfix”, or it’s a “bugday ticket”. If it’s a “hotfix”, there better be a damn good reason (like, a security problem or serious breakage). But, if it’s not in one of those clearly awful areas, it’s just a bugday ticket, which means it gets worked on after the ticket has had some time (a few work days, maybe even a week) to sit, be triaged and lightly investigated, and prioritized alongside other tickets. This avoids the must-fix-now, interrupt-laden, reactive culture you see on a lot of teams with widely-used products.

                                                                                                Also, I’ll mention that as a product becomes very widely used, your ticket list starts to get dominated not by bugs, but more by “customer technical questions” (that can look like bugs on the surface). In this case, you really need to separate those two things, and have a technical support staff (perhaps even with light coding/debugging background) focus on the non-bugs, and also triage every ticket, and only escalate the “may-be-a-bug” or “definitely-a-bug” items for the core engineering team.

                                                                                                1. 3

                                                                                                  Sorry if this sounds stupid, but what does your workload look like on the other four days?

                                                                                                  It seems to me that devoting one day per week for bugs is to either:

                                                                                                  1. “Protect” bug fixing time by setting a minimum (no less than one day per week); or
                                                                                                  2. “Limit” bug fixing time by setting a maximum (no more than one day per week.)

                                                                                                  I realize you’re saying this process works for you, but I don’t understand how. If a process artificially limits bug fixing to less than what’s necessary, the number of bugs will grow over time. If it artificially allocates more time than is necessary, the number of bugs (or bug severity) should fall until the day seems scarcely worthwhile. In either scenario, the amount of time to spend seems like it should be subject to a feedback loop, right?

                                                                                                  1. 2

                                                                                                    No, not at all – this isn’t a stupid question whatsoever!

                                                                                                    So, I think it’s both a “protect” and a “limit”.

                                                                                                    For “protect”, it’s basically saying, “We’re going to think about paying down the bug list every week for up to a whole workday per engineer, if necessary.” So it sets a weekly cadence, weekly reminder, weekly focus.

                                                                                                    For “limit”, it’s saying, “We’re not going to work on bugs in drips and drabs all week, because we want to devote 80%+ of our ‘good hours’ to development of long-term projects. We’re also not going to let low or mid-priority bugs derail the iterative development process and the projects we’ve already committed to – we’re not going to devolve into a reactive culture. Only ‘hotfix’ bugs can ‘skip the queue’. Otherwise, we calmly work on our committed projects in 4-week timeboxed iterations.”

                                                                                                    It’s also a timeboxing technique: is there some way to fix this bug within a single day, rather than making it a 3-5 day investigation and fix?

                                                                                                    The timeboxing part is perhaps the most useful tool, and also answers your other question. For bugs that can’t get fixed in a single day, we end bugday with a question: “Can this bug fix wait till next week’s bugday, where we will only get another full workday to take a crack at it?” Or, “Is this bug big, gnarly, or important enough that we should ‘promote it’ to a full-blown team/iteration project?” Sometimes, low-priority bugs reveal underlying issues that truly deserve to get fixed, but need to get scheduled into an iteration to get fixed properly. The bugday process prevents us from doing this “preemptively”, instead, we do it “just-in-time”.

                                                                                            1. 5

                                                                                              Going to give learning Clojure a shot. I’ve been diving into Emacs some more so kind of excited to learn a more widely used lisp.

                                                                                              1. 6

                                                                                                Good luck! I’ve used Clojure as my main language at work for nearly 2 years now, and still liking it. I also use Emacs—I would recommend configuring flycheck-clj-kondo, and lsp-mode (clojure-lsp) for jumping around. FWIW my rather simple Clojure-related Emacs config is here.

                                                                                                1. 1

                                                                                                  This is fantastic! Thanks @stig

                                                                                                2. 1

                                                                                                  Coming from which language, out of curiosity?

                                                                                                  1. 2

                                                                                                    I write Java for the day-job, and on the side I’ve been mostly doing Rust and Go for a handful of years.

                                                                                                1. 3

                                                                                                  Lol the article says “Yes, it’s a lot like iMessage from Apple”. Which I guess is true from a consumer perspective. But the important thing is RCS is an open interoperable standard that many companies participate in. iMessage is Apple proprietary and one of the many tools they use to keep its customers locked into their systems.

                                                                                                  I switched from iOS to Android a year or two ago and iMessage was by far the most painful part of the change. At least Apple finally relented and implemented a way to remove a number from the iMessage tar pit. I still get complaints from iPhone users who try to message me… “why is it green? It doesn’t seem to be working?” iMessage was a great improvement over SMS in its day, shame that Apple is so monopolistic.

                                                                                                  1. 7

                                                                                                    In Apple’s defense, they simply built something better than the piece of junk that is SMS, giving their customers much better features and end-to-end security. That it provides a network effect that encourages lock-in is a side effect that justifies the investment.

                                                                                                    That it took the industry ten years to come up with something comparable is kind of sad, although I’ve been around a few standards committees so I can understand why. It’s not like any part of this is particularly difficult, except for the crypto.

                                                                                                    shame that Apple is so monopolistic.

                                                                                                    Any for-profit company will be, in a similar situation. As I noted in my previous comment, Google, Yahoo and AOL acted the same way with IM, Ma Bell acted that way with phones, and today’s social networks won’t adopt open standards like ActivityPub.

                                                                                                    Being disappointed companies act this way is like being disappointed wolves keep eating bunnies. Wolves play an important role in the ecosystem, but they’re not pets.

                                                                                                    1. 1

                                                                                                      Google’s approach with RCS is not monopolistic or lock-in. It’s an open system that relies on the cooperation of many parties. I’m under no illusions that choice is all altruism; it’s the business strategy that makes sense for Google. But it’s also good for users and should be encouraged.

                                                                                                      1. 3

                                                                                                        Google’s approach with RCS is not monopolistic or lock-in.

                                                                                                        After they extended, embraced, and extinguished an open standard (XMPP) with Google Talk. And had many other failed proprietary services (Hangouts, Allo, Google Chat). However, their lunch was eaten by iMessage, WhatsApp, Facebook Messenger, because they couldn’t stick to a single IM (Google Talk was quite popular in the days).

                                                                                                        The only reason Google pushes RCS is that the only way they still have a chance to get a finger in the pie of any messenger is by pushing an open standard.

                                                                                                        But it’s also good for users and should be encouraged.

                                                                                                        Why would you trust Google after they adopted XMPP (an open standard) and killed XMPP federation once Google Talk was big enough? Why would you trust the carriers, who do not want to be simple pipes, but want to charge you separately for everything?

                                                                                                        This is an attempt of the carriers and Google to take back control, under the guise of an open standard.

                                                                                                        1. 3

                                                                                                          Just like Apple’s direction is the business strategy that makes sense for them. But it’s also been good for users, who for the last ten years have had functionality unavailable with SMS/MMS, notably end-to-end encryption.

                                                                                                          1. 2

                                                                                                            Telegram, WhatsApp, and Signal are 3 apps that provide “functionality unavailable with SMS/MMS, notably end-to-end encryption.” But, all 3 projects have fully functional apps across Android, iOS, and web/desktop, with user bases measured in tens or hundreds of millions. iMessage is the only app of this kind that is a) bundled with a mobile/desktop operating system and b) not ported to any other mobile/desktop operating system other than the ones where it is bundled, despite there being many obvious user advantages to those ports. So I wouldn’t say iMessage has been “good for users”, at least not relative to an iMessage with even a basic Android port available, which would have cost Apple a rounding error of cost to produce.

                                                                                                            The behavior of Apple re: iMessage is not just a “business strategy”, it is a reprehensible lock-in strategy. At this point, it is behaving as social engineering to expand market share and reduce mobile platform diversity at a grand scale. It is also training a generation of young iPhone users to socially dislike certain friends/contacts on the basis of their mobile operating system, a form of large-scale fabricated user evil (whether intentional or not) that I haven’t seen in tech in a long time.

                                                                                                            1. 3

                                                                                                              I fail to see how it’s reprehensible of Apple to not port an app to a competing platform. Is everyone required to support competitors this way now? Is Telegram required to integrate interop with Signal and WhatsApp to avoid the horror of lock-in? (Because that’s what we’re talking about in this thread.)

                                                                                                              The blue vs green bubbles thing is simply a visual indicator of capabilities, since messages sent over SMS lack a lot of features. That some kids want to play status games with that is depressing, though probably inevitable given what teenagers are like. But … “fabricated user evil”? Really? Did you know that Apple also brazenly stamps its logo on the back of its phones, just so kids can ostracize peers who don’t have one? Shocking!

                                                                                                    1. 6

                                                                                                      Whether RCS becomes the Apple/Android messaging divide healer, and ends the blue vs green bubble saga, depends entirely upon whether Apple adopts the standard for iMessage.

                                                                                                      Whether Apple adopts the standard depends entirely upon whether the product managers at Apple view its lack of support as a feature or a bug. If it is about nefarious social engineering for Apple/iPhone marketshare, iMessage lacking support for RCS is a feature. If it is about reducing overall mobile user pain, iMessage lacking support of RCS is a bug. Our messaging future likely hangs in the balance of some PM’s view of Apple’s real obligation (whether that is to Apple shareholders, or to the world’s mobile users).

                                                                                                      1. 10

                                                                                                        We have proofs in writing (from Epic vs Apple trial) that Apple execs like iMessage having a network effect locking people in into the Apple ecosystem.

                                                                                                        Fiduciary duty means the choice of shareholders vs being nice to people doesn’t exist.

                                                                                                        Even if Apple implemented RCS (which I wouldn’t expect them to do), they’ll probably keep it having green bubbles out of spite.

                                                                                                        1. 18

                                                                                                          Fiduciary duty means […] more money for the shareholders.

                                                                                                          This meme is my pet peeve. From the horse’s mouth (US Supreme Court Hobby Lobby decision): “Modern corporate law does not require for-profit corporations to pursue profit at the expense of everything else, and many do not.”

                                                                                                          https://www.nytimes.com/roomfordebate/2015/04/16/what-are-corporations-obligations-to-shareholders/corporations-dont-have-to-maximize-profits

                                                                                                          1. 5

                                                                                                            The law might not require it, but there’s been plenty of activist shareholder lawsuits that a corporation would think twice before putting the good of consumers before profits.

                                                                                                            1. 1

                                                                                                              Long term good of the consumer means long term profits, as opposed to being a flash in the pan.

                                                                                                        2. 7

                                                                                                          Vendors with a closed network with high market share won’t willingly let it interoperate with other networks (open or not.) This happened with the previous generation instant-message networks, where well-meaning folks developed an open standard (Jabber) but the big players like Google, Yahoo and AOL wouldn’t adopt it in any meaningful way.

                                                                                                          Similarly, you don’t see Twitter or Facebook supporting ActivityPub.

                                                                                                          The only reason email has interoperability is because SMTP predates the commercial internet, and when all the “big” services like CompuServe joined the Internet none of them were big fish in this new sea, so they all gained by adding SMTP support.

                                                                                                          1. 5

                                                                                                            The listed examples seem like odd choices since AIM did embrace Jabber for awhile and Google embraced it so hard that their contributions led to the A/V infrastructure we still use to this day (and forms the basis of what became WebRTC) and for so long that people were still using Google accounts for Jabber years after they should have moved to a more featureful implementation.

                                                                                                            1. 1

                                                                                                              Once Google Talk gained critical mass, Google turned off federation. It was still using XMPP as a transport protocol but that didn’t matter to anyone because you couldn’t communicate with folks with non-Google Jabber accounts and so it was effectively a proprietary network.

                                                                                                              1. 4

                                                                                                                Honestly, federation turned off way later than most think and before it went away what mostly happened is the Jabber network started rejecting Google because Google refused to use TLS with a valid cert on their federated links. If you ran your own server or used one that had a special case config for Google it kept federating long after most people had been saying “Google abandoned” federation for years. One the federated server did die the Google Talk product and brand had itself been dead for years and client connections still worked (and were used by many I know) for years after that yet.

                                                                                                                Google kills whole products and protocols on a regular basis, sudden death of popular features is kind of their rep, yet the Jabber servers they ran had the slowest death of anything I’ve seen from them. Definitely not just yanked out of spite.

                                                                                                              2. 1

                                                                                                                They did support it as a client protocol, but never enabled federation. So you couldn’t follow or message an AIM user from Google Messenger or vice versa. The stated reason was to avoid incoming spam, but another reason was to retain their network-effect lock-in, i.e. “all my friends use AIM so I have to use it too.”

                                                                                                          1. 8

                                                                                                            There are two dangers in most project tracking tools. One is the one discussed here: the assumption of one project, one person. The second is the assumption that it is “OK” to have one person assigned to 2, 3, 4, 5, … N projects that are currently in-progress.

                                                                                                            My rule to fix this is that a product/engineering team’s entire iteration/sprint/whatever project plan should fit on a single 720p screen. You should be able to see todo & in-progress in that single screen. You should be able to see pairs and triplets of assignments on the same screen. Trello or Notion or GitHub Project Boards can accomplish this. Many tools cannot.

                                                                                                            Ideally, one person is only on one project, and, as suggested here, that person is joined by colleagues of diverse skillsets/experiences, forming a development “pod” of 2-4 builders per project.

                                                                                                            Occasionally, one person is spread onto two projects, but one is “primary focus”, and the other is “as an advisor/reviewer only”. That can work, with care. If one person is on two projects, they are likely “strained”, though. If they are on three or more projects, they are definitely spread thinly and the team should rebalance or de-prioritize projects. If your project board fits on a 720p display, your brain will spot this quickly. Focus is key, as this post suggests.

                                                                                                            I wrote about this more in “Software planning for skeptics”:

                                                                                                            https://amontalenti.com/2017/10/27/skeptics

                                                                                                            1. 4

                                                                                                              Occasionally, one person is spread onto two projects, but one is “primary focus”, and the other is “as an advisor/reviewer only”. That can work, with care.

                                                                                                              Yes, it seems to me that such an arrangement where one person can be both a “primary” on one task, and “secondary” on another one, is an acceptable compromise, especially if for cultural reasons people are reluctant to buy into an idea of “wasting time by assigning people redundantly to one task”. At very least there is some explicit redundancy and expectation of staying in the context of each project.

                                                                                                              I wrote about this more in “Software planning for skeptics”:

                                                                                                              https://amontalenti.com/2017/10/27/skeptics

                                                                                                              Good read! I agree a lot with most if not everything in there!

                                                                                                            1. 2
                                                                                                              • Helping my girlfriend finish up her 3D print of the Master Sword from The Legend of Zelda. We stuck supports within its hollow center and completed the gluing. Today we plan to prime and sand it.
                                                                                                              • Set my Habitica account back up and give it an overhaul. I’ve been relying on Google Tasks reminders for my daily todos but I wanted a reminder system that actually resets each night and needed to clear off all those reminders that clogged up my Google Calendar.
                                                                                                              • Review the Year Compass I filled out a couple months ago.
                                                                                                              1. 2

                                                                                                                If you are an Android user, I really enjoy Loop Habits for the Habitica use case. As described here:

                                                                                                                https://loophabits.org/

                                                                                                                And open source code here:

                                                                                                                https://github.com/iSoron/uhabits

                                                                                                                It solves the problem you describe, but doesn’t attempt the whole hokey “gaming dynamic” that Habitica pursues. Also, your historical habits data is stored as a simple SQLite DB.

                                                                                                                1. 1

                                                                                                                  Thanks! I’ll take a look.

                                                                                                                2. 1

                                                                                                                  I’d like to hear more about your Year Compass experience. I looked over the system and I’m really surprised that it requires you to spend time with your last year’s calendar. I know some people keep calendars but I never have and now I’m wondering if I’m really weird in this respect.

                                                                                                                  1. 1

                                                                                                                    That lack of a calendar does surprise me, but then I use Google Calendar instead of anything physical. My daily morning pages, journal entries, and Bullet Journal entries do provide additional context, though.

                                                                                                                1. 5

                                                                                                                  This isn’t so much a post about Firefox 90, as it is a post about some changes to the version of JavaScript that Firefox 90 executes. The changes are:

                                                                                                                  • Support for private class fields.
                                                                                                                  • Addition of .at() method for arrays, which supports negative indexing to “index from the end”, similar to Python (so -1 is the last item in the array).
                                                                                                                  • Addition of a “conic gradient” coloring option in Canvas 2D.
                                                                                                                  • New headers related to the HTTP fetch API.
                                                                                                                  1. 8

                                                                                                                    Private class fields, oh boy.

                                                                                                                    The enterprise programmers are not finished colonizing that poor wayward lisp yet. :(

                                                                                                                    1. 3

                                                                                                                      It’s kinda funny that they picked Perl/Ruby-esque sigil syntax for private fields inside basically-Java class syntax. Interesting mix.

                                                                                                                      1. 2

                                                                                                                        at()? Why not just use the normal indexing with brackets?

                                                                                                                        1. 3

                                                                                                                          Probably because negative indexes are already a thing. Semantically, arrays are just hash maps with a few utility methods. arr[10] = “hello” sets the string key ”hello”, and for (let i in arr) iterates over array keys as strings. arr[-10] already refers to the hash map element with key ”-10”.

                                                                                                                          Actually, originally, JavaScript didn’t even have an array type. Instead, objects with numeric string keys were used. That’s why the magical arguments variable is an “array-like object” instead of an array. Eventually, an actual array type was added, but as described above, it’s still just a hash map.

                                                                                                                          Yeah, JavaScript is a horrible language.

                                                                                                                          1. 3

                                                                                                                            I wish they would just fix the language instead of making these changes in a backwards-compatible way, and it doesn’t even need to be a breaking change if you add 'use strict 2021'; or something.

                                                                                                                            Now every JavaScript developer will need to remember two ways to access arrays, which are only subtly different, just in the off chance that someone used this weird bug/feature.

                                                                                                                            1. 1

                                                                                                                              Then they’ll have to admit the language is crazy and only exists because of coincidences.

                                                                                                                              After all the harsh words against less cool languages over the year that is too hard even after TypeScript have proven beyond all reasonable doubt that a typed JS-like language is not only technically possible but a massive productivity boost.

                                                                                                                              Now, don’t get me wrong: The JS community is cool and produces awesome software despite having to deal with a terrible no good language. In fact it makes it even more impressive.

                                                                                                                              But it is still like doing amazing artistry on a one-wheel-bike, propped up with gyro stabilizers like linters etc.

                                                                                                                              Amazing, but should have been totally unnecessary when ordinary bikes (e.g. Java) were already on the market.

                                                                                                                      1. 4

                                                                                                                        Has Amazon/OpenSearch documented yet how they plan to keep HTTP API compatibility with Elastic’s Elasticsearch server, so that the client libraries don’t diverge in the community?

                                                                                                                        1. 2

                                                                                                                          I recall some statements that they intend to keep API’s compatible for as long as it makes sense.

                                                                                                                          1. 1

                                                                                                                            Good question, maybe they’ll also fork the libraries