Threads for emidln

  1. 5

    I find it interesting that the author uses AST rewriting to convert |. I’ve seen approaches in the past which just use a filler object (like foo | into | bar, into being the filler) which has a special __or__ method that does the piping.

    I personally have experimented with a simple function just called pype() which also does some magic to avoid the need for lambda functions, see here.

    EDIT: I did a quick writeup of my solution.

    1. 7

      I always just reduced the value over the reverse of callables. It made higher order stuff, particularly data transforms, a lot easier to read. The code I used at my last job is open sourced here (the thread() function):

      We used pipeline() more than thread(), which takes only callables (no value) and returns a callables. It is functional composition with the order reversed such that the callables are applied from left to right.

      The nice advantage of using the ast is that you can get much closer to Clojure’s -> and friends. By working from a compiler hook (the ast transformer), you wouldn’t need to use partial/rpartial/lambda as much. The downside is that it is quite a bit more complicated than a one liner.

      1. 2

        That is very nice and clean, though I don’t really like lambdas in Python, they feel super verbose for what they are. I wish there was Haskell-style partial application.

      2. 6

        I’d considered that approach as well, but I wanted to avoid needing a wrapper for the leftmost expression and I also wanted to avoid needing to partially apply functions.

        Take the 3rd example from the post:

        Users.find_all() | group_by_category(max=5) | print()

        without transforming the ast that’d have to look something like:

        pipe(Users.find_all()) | partial(group_by_category, max=5) | print
      1. 6

        I’m not Celery’s biggest fan, however I can explain the modes to you. beat is for repeating tasks on a schedule, optionally using a crontab-like syntax for scheduling. A single beat (last I checked) must be in the cluster to kick off scheduled jobs and at least 1 worker must be present to execute said jobs. In some cases, where you don’t have periodic or scheduled tasks, you might not run any beat nodes and only worker nodes.

        Lately, I’ve been using AWS Lambda to run small jobs using their Cron Scheduling syntax. For larger jobs, I put a message on an queue. This has worked pretty well without involving a lot of infrastructure.

        If you have a box lying around with spare capacity, just using Cron is pretty fine.

        1. 2

          Of course you can always just watch the real movie in ASCII using vlc (or anything else that supports aalib (black and white) or libcaca (colorized) output) with something like vlc --vout caca $movie.

          1. 2

            I remember using aalib way back in University with mplayer. At some point it became broken and never fixed, but libcaca still seems to work fairly well with mpv. I just pulled up Star Trek DS9 in a terminal at work over SSH. Now I just need to forward the PulseAudio remotely :-P

          1. 4

            Most functional languages fit this category. Clojure is a good example of a language that provides great tools to avoid naming transient variables with its various threading macros.

            1. 3

              It’s not really a functional language, but fwiw this is also common in modern R code via magrittr pipelines.

              1. 1

                You still have to name functions.

                1. 3

                  If you want recursion you’ll have to name functions, unless you want to write out the combinator from first principles every time. (Naming the combinator fix would be cheating!)

                2. 1

                  That’s a good point…I hadn’t thought of that, and I use method chaining all the time! Do you find in Clojure that it is difficult to debug the intermediate states of a “thread” (not sure what term is appropriate here) such that intermediate variables would be more convenient? (I guess I’m thinking from a perspective of JavaScript’s method chaining, which seems similar…and it is somewhat frequent that I need to log intermediate values in a .map.reduce.filter chain.)

                  1. 4

                    In the standard library, you can debug -> using another (-> doto prn) and you can debug ->> using (#(-> % doto prn)), although I usually use taoensso.timbre/spy for this. A simplistic solution to work for both ->, and ->> is (defn debug [x] (prn x) x).

                1. 3

                  It seems to be a common theme these days…. People rediscovering time and time again why properly normalised data, ACID and a well thought through data model is important.

                  I walked into a discussion recently where they were bemoaning the fragility, brittleness and complexity of a large json data structure…

                  …my only comment was that I felt I had fallen into a time warp and I was back in the late 1980’s when people were bemoaning the problems of hierarchical databases and why a RDBMS was needed.


                  Sort of sad really.

                  I’m still waiting for the pro-sql types to wake up to what CJ Date has been saying for decades and to up their game beyond null’s and auto-increment keys….. but we can’t get there because we keep having to rehash the basic stuff of normalization and ACID.

                  1. 7

                    The problem is the lack of SQL databases that require less than days to set up replication in a robust manner. Schemas are not the problem. Arcane hard to administrate software is the problem. PostgreSQL replication requires a dedicated DBA. I’m keeping a close eye on CockroachDB.

                    1. 4

                      I use Amazon RDS at the day job. Unless you have enough data to justify a DBA for other reasons, RDS is inexpensive enough and solves PostgreSQL replication.

                  1. 1

                    Is there a compelling reason for a Vim user to switch?

                    I really don’t want to start an editor flame war here, but I’m genuinely curious about the advantages it might bring if I switch.

                    1. 3

                      I was a vim user for around 15 years. I still “think” in vim motions. I switched to Spacemacs using the vim-style keybindings afforded by evil-mode. I had a very extensively customized .vimrc and .vim (you know it’s serious when you use a directory) and used various “slime” modes over the years (vim-slime was my last one) to integrate a copy/paste into other tmux panes. I maintained a lot of custom code to enable an integrated repl experience for the languages I used the most (editing sql and testing with psql/sqlite3; editing python and interacting with ipython; editing clojure and interacting with the clojure repl). I had a fancy Unite-based workflow for file-finding and project management.

                      In short, everything I had customized was implemented out of the box on Spacemacs. I enabled modes for the languages I used (clojure, python, sql, html, javascript) and it all just kinda works. git/gist integration just works. The motions I used worked. It was very easy to figure out how to make an evil operator for executing code and then use that + motions to emulate the fine-grained selection I had become accustomed to with vim-sexp/vim-sexp-mappings-for-regular-people/surround.vim + tmux. The best thing was a ton of functionality was unlocked for clojure/python thanks to modes already being integrated and built. I could stop maintaining my vim installation and focus on other issues.

                      Some highlights for Spacemacs for someone whose only experience with emacs was knowing how to kill it (the wrong way mind you; C-z into pkill -9 emacs).

                      • Spacemacs is discoverable. The leader key is Space and once you hit that, a menu pops up with completions. In fact, these completions pop up whenever you start a key chord. They default to waiting 0.4 seconds, but I overrode that in the .spacemacs config file to be 0.1 second. This is awesome when you don’t know emacs. (This functionality is called helm and which-key).
                      • Spacemacs is curated. I didn’t know what the git interface in emacs is (magit if you’re curious). If I would have had to spend months copy/pasting/tweaking elisp I found online to get half of what I spent 15 years acquiring in vim, I would have never switched. In Spacemacs SPC shows that g is for git and then you see a bunch of actions explained. This is easy.
                      • Spacemacs is a very complete vim experience. Registers work. Macros work. The motions you expect work. You can use operators and define new ones. Your chords will basically work. The difference is that you can jump to the source code and use an elisp debugger inline in Spacemacs. This is pretty awesome for discovering/tweaking once you find something you want to change. elisp is in a lot of ways similarly high level like Python, Ruby, or Lua. You don’t have to “grok Lisp” to make some useful changes
                      • Spacemacs has decent documentation. In addition to SPC h which has a bunch of emacs builtin help/documentation, there is a full Read the Docs and each layer documents its shortcuts.
                      • Spacemacs has “Layers” which make it easy to add more curated sets of things so you can learn a piece at a time and have nice entry points for extension and learning the underlying system. Layers are sets of emacs packages and associated configuration/integration into the spacemacs ecosystem.
                      • Spacemacs is easy to configure. Most of what you need to do is encompassed in editing a heavily commented .spacemacs file that is well documented on the read the docs website. You can largely avoid copying/pasting random elisp snippets.
                      1. 2

                        I’ve been using vim very heavily for about 10 years now and I’m currently a week into trying out a switch to emacs.

                        The main thing I like is honestly that things look way nicer - my switch was fed mainly by frustration with weird terminal UI issues, but has been sustained by 1) evil-mode means I can keep my vim editing ways, 2) plugins like magit that are just so much more feature-rich & polished, and 3) Emacs lisp is much nicer to use than vimscript.