1. 9

    Great news! I am eager to try this!

    Turn on -XLinearTypes, and the first thing you will notice, probably, is that the error messages are typically unhelpful: you will get typing errors saying that you promised to use a variable linearly, but didn’t. How hasn’t it been used linearly? Well, it’s for you to puzzle out. And while what went wrong is sometimes egregiously obvious, it can often be tricky to figure the mistake out.

    So, basically, GHC just got its own “Syntax error” a la OCaml… just a bit more specialized :p.

    1. 11

      Maybe it’s just me, but to me OCaml’s errors are terse and unhelpful and GHC’s errors are… verbose and unhelpful. ;)

      There are interesting papers that show working ways to improve both, but I wonder why none of those improvements are in the mainline compilers.

      1. 2

        Good error reporting is easiest if it’s built into the compiler front end from the start. If a new algorithm comes along to improve the error information it’s almost never going to be a simple job to drop it into an existing compiler.

        You need type information & parse information from code that’s potentially incorrect in both spaces, so any error algorithm usually has to be tightly integrated into both parts of the compiler front end. That tight integration usually means that improving compiler errors is a significant amount of work.

        1. 3

          It varies. What puzzles me is that a lot of time ready to use, mergeable patches take much longer to merge than they should.

          Like this talk: https://ocaml.org/meetings/ocaml/2014/chargueraud-slides.pdf

          1. 1

            Do you also have a link for a patch for the improved error messages?

            A lot of work has been going on to move OCaml to a new parser and improve error messages. Even though there is a lot still needed to be done, latest releases have started improving a lot. Maybe we can still extract some useful bits from that effort and try again

            1. 2

              Turns out it was even made into a pull request that isn’t merged yet: https://github.com/ocaml/ocaml/pull/102

              1. 1

                Thanks. It is quite an informative PR actually, and explains why the change is not there yet and once can infer why it is easier to add informative messages in new languages and complier but it may be quite hard to retrofit them to seasoned ones

      2. 7

        Would you be kind enough to give me an ELI5 about what linear types are and what you can do with them?

        1. 29

          In logic, normal implication like A implies B means whenever you have A, you can derive B. You have tautologies like “A implies (A and A)” meaning you can always infinitely duplicate your premises.

          Linear implication is a different notion where deriving B from A “consumes” A. So “A linearly implies B” is a rule that exchanges A for B. It’s not a tautology that “A linearly implies (A and A).”

          The classic example is you can’t say that “$3 implies a cup of coffee” but “$3 linearly implies a cup of coffee” makes sense. So it’s a logical form that reasons about resources that can be consumed and exchanged.

          Same in functional programming. A linear function from type A to type B is one that consumes an A value and produces a B value. If you use it once with an A value then you can’t use it again with the same A value.

          This is nice for some performance guarantees, but also for encoding safety properties like “a stream can only be read once” etc.

          1. 6

            Keynote: Linear Haskell: Practical Linearity in a Higher-Order Polymorphic Language https://skillsmatter.com/skillscasts/11067-keynote-linear-haskell-practical-linearity-in-a-higher-order-polymorphic-language

            1. 5

              It can be used to model protocols with type signatures. The following is in theory what you should be able to do.

              data ConsoleInput
                  = Input String ConsoleOutput
                  | ExitInput
              
              data ConsoleOutput
                  = PrintLines ([String] ⊸ Console)
                  & PrintLastLines ([String] ⊸ ())
              
              greet :: ConsoleOutput ⊸ ()
              greet console
                  = let PrintLines f = console
                    in step2 (f ["name?"])
              
              step2 :: ConsoleInput ⊸ ()
              step2 ExitInput = ()
              step2 (Input input console)
                  = let PrintLastLines f = console
                    in f ["hello " ++ input]
              

              If you combine it with continuation passing style, you get classical linear logic and it’s a bit more convenient to use.

              If you model user interfaces with types, they should be quite useful.

              I’m also examining and studying them: http://boxbase.org/entries/2020/jun/15/linear-continuations/

              1. 1

                Wikipedia gives a reasonable overview. The closest analogy would be something like move semantics – for example ownership in Rust can be considered as manifestation of linear types.

                1. 6

                  Rust ownership is linear affine types. Linear types are similar but differ in the details. A shitty way of understanding it is affine types mimic ref counting and prevent you from having a ref count < 0. Linear types are more a way of acting like RAII in that you might create a resource but just “know” that someone later on in the chain does the cleanup.

                  Which I’m sure sounds similar but affine types allow for things like resource leaks but linear types should guarantee overall behavior to prevent it.

                  This all assumes my understanding and explanation is apt. I’m avoiding a ton of math and i’m sure the shitty analogy doesn’t hold up but behaviorally this is how I have it in my brain.

                  1. 2

                    Linearity Design Space: https://i.imgur.com/s0Mxhcr.png

                    1. 2

                      I’m personally of the stance that the 2020 linear ghc stuff is more <= 1 usage, and kinda misses out on a lot of really fun expressivity that can fall out of making full classical linear logic first class. But that’s a long discussion in its own right , and I’ve yet to make the time to figure out the right educational exposition on that front

                      1. 1

                        it definitely seems more limited in scope/ambition compared to the effort ongoing for dependent types, for better or worse. Can’t say I know much about what first class linear logic would look like, but perhaps now there will be more discussion about such things.

                        1. 2

                          The really amazing thing about full linear logic is it’s really sortah a rich way to just do mathematical modelling where everything has a really nice duality. The whole thing about linearity isn’t the crown jewel (though wonderfully useful for many applications ), it’s that you get a fully symmetric bag of dualities for every type / thing you can model.

                          The paper that really made it click for me was mike shulmans linear logic for constructive mathematics paper. It’s just a fun meaty read even at a conceptual level. There’s a lot of other work by him and other folks that taken together just point to it being a nice setting for formal modelling and perhaps foundations of category theory style tools too!

                      2. 1

                        Not sure I can agree that Uniqueness types are the same as Linear types. Care to explain they’re similar sure but not the same thing and your… screenshot of a powerpoint? isn’t very illustrative of whatever point you’re trying to make here.

                        And from my experience with Idris, I’m not sure I’d call what Rust has Uniqueness types.

                        1. 1

                          They are different rows in the matrix because they are different, of course.

                          it’s from this presentation about progress on linear ghc a little over a year ago https://lobste.rs/s/lc20e3/linear_types_are_merged_ghc#c_2xp2dx skip to 56:00

                          What is meant by Uniqueness types here is “i can guarantee that this function gets the unique ptr to a piece of memory” https://i.imgur.com/oJpN4eN.png

                2. 2

                  Am I the only one thinking this is not how you ship language features?

                  If the compiler can’t even report errors correctly, the feature shouldn’t ship.

                  1. 15

                    If the compiler can’t even report errors correctly, the feature shouldn’t ship.

                    Its more this is an opt-in feature with crappy error reporting for now using computer programming design features not in use in most programming languages. Its going to have rough edges. If we required everything to be perfect we’d never have anything improved. Linear types like this also might not have a great way to demonstrate errors, or the domain is new so why not ship the feature for use and figure out what kind of error reporting you want based on feedback.

                    1. 13

                      Many people do not realize that haskell is a research language and GHC is one of the main compilers for it. This is an experimental feature in a research language. If it works out well, then it will be standardized.

                    2. 5

                      Other people have sort-of said it, but not clearly enough I think. This is not a language feature being added. It is a feature-flagged experimental feature of a particular compiler. Most such compiler extensions never make it into real Haskell, and the ones that do take years after they are added to a compiler to make it to a language spec.

                      1. 4

                        for all practical purposes isn’t “real Haskell” defined by what ghc implements these days?

                        1. 2

                          Yes, all the other implementations are dead. They still work, but they won’t run most modern Haskell code, which usually uses a bunch of GHC extensions.

                          1. 1

                            You might say “isn’t it not popular to write standards-compliant Haskell these days?” and you’d be right. Of course it’s often trendy to write nonstandard C (using, say, GNU extensions) or nonstandard HTML/JavaScript. However, ignoring the standard being trendy doesn’t mean the standard doesn’t exist, or even that it isn’t useful. I always make sure my Haskell is Haskell2010, and I try to avoid dependencies that use egregious extensions.

                          2. 2

                            Honestly curious: are there any other Haskell compilers out there? Are they used in production?

                            Also, what is a definition of a true Haskell? I always thought it’s what’s in GHC.

                            1. 5

                              There’s a Haskell which runs on the JVM - Frege. But it makes no attempt to be compatible with the version of Haskell that GHC impements, for good reasons. Hugs is a Haskell interpreter (very out of date now, but still works fine for learning about Haskell.) There a bunch of other Haskell compilers, mostly research works that are now no longer in development - jhc, nhc98 etc etc.

                              But GHC is the dominant Haskell compiler by far. I don’t think there are any others in active development, apart from Frege, which isn’t interested in being compatible with GHC.

                              (“True Haskell” is the Haskell defined in the Haskell Report, but real world Haskell is the Haskell defined by what GHC + your choice of extensions accepts.)

                              1. 2

                                There are other compilers and interpreters. None of them is anywhere near as popular as GHC, and usually when one does something interesting GHC consumes the interesting parts.

                                There is definitely a standard, though: https://www.haskell.org/onlinereport/haskell2010/

                                The whole reason language extensions are called “extensions” and require a magic pragma to turn on is that they are not features of the core language (Haskell) but experimental features of the compiler in question.

                              2. 1

                                In short, GHC Haskell is a language designed by survival-of-the-fittest.

                                1. -2

                                  If you haven’t noticed, the language spec is dead.

                                2. 3

                                  Overly terse error messages are bad, but they are better than wrong error messages. Some things are much harder to give helpful error messages for than others.

                                  I wish people spend more time improving error reporting, at least in cases when the way to do it is well understood. There is no reason for say TOML or JSON parsers to just say “Syntax error”. But, YAML parsers are pretty much doomed to give unhelpful errors just because the language syntax is ambiguous by design.

                                  And then some errors are only helpful because we know what their mean. Consider a simple example:

                                  Prelude> 42 + "hello world"
                                  
                                  <interactive>:1:1: error:
                                      • No instance for (Num [Char]) arising from a use of ‘+’
                                      • In the expression: 42 + "hello world"
                                        In an equation for ‘it’: it = 42 + "hello world"
                                  

                                  How helpful is it to a person not yet familiar with type classes? Well, it just isn’t. It’s not helping the reader to learn anything about type classes either.

                                  1. 1

                                    I’ve seen some good suggestions on r/haskell for improving the wording of these errors.

                                  2. 2

                                    The error they’re talking about is a kind of type error they’ve not worked with. It’s produced if you forget to construct or use a structure. I I’m guessing it’s technically “proper” but the produced error message may be difficult to interpret.

                                    They’ve ensured it’s a feature you can entirely ignore if you want to. Everybody’s not convinced they need this.

                                    I otherwise dunno what they’re doing and I’m scratching my head at the message. Something like “Oh cool you’ve done this… … … So where are the types?”

                                    1. 2

                                      So you never got a C++ template error in the good olden days? Seriously though, it just got merged. It’s not released or “shipped” in any means.

                                      1. 0

                                        So you never got a C++ template error in the good olden days?

                                        No, because I looked at the language, figured out that the people involved completely lost their fucking mind, and moved on.

                                        Seriously though, it just got merged. It’s not released or “shipped” in any means.

                                        They took 4 years to arrive at the current state, which I’ll approximate at roughly 10% done (impl unfinished, spec has unresolved questions, documentation doesn’t really seem to exist, IDE support not even on the radar).

                                        So if you assume that there will be a Haskell version in the next 36 years, then this thing is going to end up in some Haskell release sooner or later.

                                        1. 2

                                          So if you assume that there will be a Haskell version in the next 36 years, then this thing is going to end up in some Haskell release sooner or later.

                                          Could you elaborate on this? If practical users of linear types will only use them if they have good error messages, and early testers want to work out the kinks now, what’s wrong with having a half-baked linear types feature with no error messages permanently enshrined in GHC 8.12?

                                  1. 1

                                    What happens if

                                    1. one file is modified on multiple offline devices
                                    2. and these devices get online
                                    1. 2

                                      Then you have a conflict. The conflict version is local to each machine and has to be fixed locally.

                                      So don’t do that. It’s not magic.

                                      1. 1

                                        Thanks. Does that mean you have to periodically check the website to ensure there are no (unintended) conflicts?

                                        1. 1

                                          By the website you mean Syncthing web interface (it runs locally)?

                                          But it’s actually not displaying the conflicts anywhere (unfortunately), so I’ve got an external script that checks for conflicted files every hour and shows a notification if there are any. Perhaps I should contribute and make it the default behaviour.

                                          1. 1

                                            Yes, by “website” I meant the Syncthing web interface.

                                            Not having conflict notifications is pretty much a bummer for me.

                                    1. 2

                                      I love Syncthing, and using it across multiple devices, including Android phones, to sync most of my files.

                                      For Android, I recommend (weirdly named) Syncthing-Fork, it’s some nice power saving features and sync conditions (e.g. so you can only sync select folders when you’re roaming). The only thing that annoys me is that Android OS doesn’t support symlinks, so I have to maintain separate Syncthing directories for Photos/Pictures/Downloads, etc.

                                      For sync problems: I have a script that’s doing ‘heartbeats’ every hour, merely by writing timestamp into a special file (e.g. /syncthing/.heartbeat/$DEVICE). This is running on every device (so each device has its own hearbeat file). In addition each device is checking for other devices’ heartbeats – if they fall behind (say) for a day, it shows error. So far that’s been mostly overly paranoid, and on all occasions the problems weren’t with Syncthing itself, but with misconfiguration (e.g. firewall issues, wrong sync conditions on the phone, etc). But I feel like this kind of functionality could be helpful in Syncthing by default, maybe I should contribute it.

                                      1. 2

                                        I feel like this kind of functionality could be helpful in Syncthing by default, maybe I should contribute it.

                                        Even if it doesn’t get merged, publishing your script somewhere publicly visible could prove useful to someone!

                                      1. 3

                                        That looks lovely! I’m a big hater of yaml-based configuration and reinventing the wheel, when things like that can be done in a proper programming language. I don’t have to manage many servers, but I like the idea of having reproducible state of the system. Like NixOs, but NixOs/Nix seems like a lot of work, whereas I’m 99% happy with simply using apt, pip and few ad-hoc commands.

                                        I was about to cleanup my current setup scripts (done with a bunch of scripts + Ansible), so I think I’ll give this tool a try. Do you think it makes sense for my usecase (personal desktop/laptop)? Are you using it for that purpose?

                                        1. 4

                                          I do think it makes sense - I do exactly the same :) So far I’ve used pyinfra for both ad-hoc/local box setup and also in production managing medium size (100’s of nodes) Elasticsearch clusters, amongst other things. An example is my (very WIP) MacBootstrap deploy: https://github.com/Fizzadar/MacBootstrap.

                                        1. 2

                                          heh, my idea for a better browser history a few years ago was extracting semantic markup (schema.org etc. — attempt to unify common vocabularies) and being able to have a history of not just pages, but e.g. organizations, people, events etc. mentioned on all the pages. Not sure if that would actually be useful though :D

                                          1. 1

                                            That would be really cool, like a proper Memex! But also I’d imagine very hard.

                                            1. 2

                                              this sort of entity extraction and classification is actually pretty straightforward with more or less modern nltk stacks - I did almost exactly this for work a few years ago, pulling people and companies from news articles.

                                              you’re going to want to search for:

                                              • Named Entity Recognition (NER)
                                              • Stanford NLTK

                                              check out this random tutorial I found (unaffiliated, just hate being given search targets without contextual examples)

                                          1. 12

                                            This is missing the “why”.

                                            • What are the benefits of this whole mess of new build infrastructure?
                                            • Who claims these are the new best practices, or are these just the author’s preferences?
                                            • Why are these tools curl | bash-ed onto my machine instead of packaged with my distribution like a piece of dev tooling I can actually trust to base my entire project on?
                                            1. 16

                                              Agree. I now have a sudden urge to write a guide how to start from very simple (i.e. system interpreter, no packaging, minimal verification) and how to progress further on. And how to decide what tools do you even need to consider and at which stage.

                                              In hindsight as a somewhat experienced Python programmer I can see how each of these tools can be helpful – but if I was a beginner I would be completely overwhelmed.

                                              1. 4

                                                Please do. I am by no means a beginner at Python itself, but most of these “guides” confuse me. I’ve gotten pretty far with a few venvs and some other minor tools. Why would I need all of these other things?

                                                It feels very much like people write these kinds of guides where they try to score points by cargo culting a lot without themselves understanding why they are doing things the way they are.

                                              2. 1

                                                I think “why” kinda answers itself. People who are familiar with setuptools don’t need an answer as they would gladly try something else. As for beginners they don’t need an answer either as you should really learn the newest and most popular standard.

                                                So who’s really asking “why”?

                                                1. 4

                                                  Everyone, beginners especially, has a right to ask for a rationale for the recommendations they receive.

                                              1. 34

                                                I wrote Miniflux 7 years ago for my own needs. Rewritten in Golang in 2017. The project still active and continue to receive contributions. There is always something to improve :)

                                                1. 5

                                                  I like this but managing the database myself isn’t ideal. Any reason why an embedded database isn’t used?

                                                  1. 3

                                                    Miniflux is excellent! No BS, clean and snappy interface.

                                                    I’m using it for my project that automatically searches for certain queires on github/hackernews/twitter/reddit/pinboard and puts the results in atom feeds. Kinda like Google Alerts, but actually useful and consumed through a functional RSS interface. I needed to slightly modify the Miniflux frontend for that (making it even more compact) – and it was pleasantly quick and clear (I had no prior Go experience either).

                                                    Thanks for your work!

                                                    1. 1

                                                      I was looking for something exactly like this!

                                                      1. 1

                                                        I have started using it yesterday. Moved away from Innoreader, mainly because I wanted to own the data and not worry about it in future, specifically manage starred articles etc… Miniflux is extremely easy to setup, especially if you’re using docker. I always run Postgres instances anyways, so it integrated well. For people who question db, it’s an advantage having a database like PostgreSQL. And I like that it’s ‘opinionated’ - tech stack and implementation perfectly considered. @0xfg - many many thanks for your work on this.

                                                        1. 1

                                                          Hey man, I’ve been using Miniflux for years. Thanks for the awesome work! I really got excited when you rewrote it in Go :)

                                                          Eventually I switched to FreshRSS because I can choose a SQLite database there. Together with docker/kubernetes this makes the setup slim and easy. I miss Miniflux’s straightforward user-interface, though.

                                                        1. 2

                                                          Wow, great to see this.

                                                          I think it’s a real shame that usually issues are separate from the source code – the search UIs are paginated/bloated, don’t work offline and most annoyingly, the cognitive burden of switching to github (or whatever hosting you’re using) and searching through is too high. When you can just use incremental search within your IDE without having to switch to the browser, it’s much saner.

                                                          I used org-sync for far, but git bug looks way more mature… Perhaps I’ll try to implement a text dump for git-bug too.

                                                          1. 6

                                                            Macropy is excellent when one wants to use syntactic macros. One problem I had was that, for my scripts, I cannot directly use macropy and other syntax modifiers since they rely on the import hooks, which means that you can not use them directly on the script file. You have to import them separately, which makes their use cumbersome.

                                                            This was my project to work around this issue. It works, but is very hacky, and requires adding a new codec to the Python installation.

                                                            1. 2

                                                              Wow, it’s kinda mad in a beautiful way, thanks for sharing! I’ll give it a deeper read later.

                                                            1. 3

                                                              I do it in Python occasionally.

                                                              1. 1

                                                                Have been bitten by this in python when a library I imported redefined a global object. It was not a fun debugging session.

                                                                1. 2

                                                                  Yeah, can imagine. I usually only use it under extreme circumstances, and with a context manager, to minimize the impact. Kind of similar to what patchy does (although patchy is even more extreme and less fun to debug)

                                                                2. 1

                                                                  Python only allow it to a certain extent. You can monkeypatch any module, even stdlib ones, but you can’t monkeypatch the builtin types or instances of builtin types. modules from C extensions are not monkeypatcheable as well, I think?

                                                                1. 3

                                                                  What to do then? Well, I write in Clojure now and let me tell you that most of the time using functions is actually okay! If you’re doing Python, try to use more NamedTuples, dataclasses and similar: this way you’re developing with data and not with objects.

                                                                  I feel like this post is missing the closure. How would you reorganize this specific code in Python, without the inheritance? How would the solution look in Clojure?

                                                                  1. 1

                                                                    I would just do a protocol and implement it. And in Python I would do the same - an abstract base class and then implement it. Not much savings in code anyway!

                                                                  1. 4

                                                                    I’ve been progressively writing about my personal knowledge setup here. TLDR: org-mode and various tools around it (some of which I’ve built myself).

                                                                    1. 1

                                                                      Why did you abandon gitit?

                                                                      1. 3

                                                                        IIRC, I figured it’s much faster to skip the browser step altogether and just use a text editor (sublime for me at the time) on top of a repository with markdown files

                                                                    1. 9

                                                                      0xB. I dislike the comment area where people are rude and also serves the ego.

                                                                      A thousand times this. This is why the only comment system I’ll ever put on my site will be echochamber.js

                                                                      1. 5

                                                                        Funny, but antisocial, as it’s effectively wasting the visitor’s time.

                                                                        1. 1

                                                                          For my blog, I have a comment area for people who want to leave a quick comment, even anonymously, but I also add backlinks to the discussions here, on Reddit, or on Hackernews. Often there is no need to comment because someone else discussed that it already before.

                                                                        1. 4

                                                                          I had a hard time figuring out ideal project structure, with Python however, it’s pretty simple

                                                                          1. ├── blueprint  # Our source code - name of the application/module
                                                                          

                                                                          Not trying to nitpick, I just found it kind of funny that the very first line is debatable and not so simple :) Many people find it safer to use some indirection and put the sources under src/ directory, because of the implicit current working directory in PYTHONPATH.

                                                                          I can see that there is also an issue in your repo, so you’re aware of it (it’s even referencing the same article).

                                                                          1. 3

                                                                            Also blueprint has special meaning in Flask, so if this is a python project and people might have used that, I find it a little confusing.

                                                                          1. 3

                                                                            oh wow, I’ve had a remove_prefix in so many utils.py files. nice to finally see it in the standard lib!

                                                                            1. 2

                                                                              Looks like it’s returning the original string if the string does not start/end with the prefix/suffix. A valid way to implement it, but I’ll have to keep my helpers which throw an exception in that case..

                                                                            1. 5
                                                                              • org-mode: I keep my thoughts, tasks, drafts, blog posts – everything here, it’s my organisational heaven

                                                                              • mypy has made me enjoy Python and benefit from the best of both (static/dynamic) worlds

                                                                              1. 2

                                                                                is org-mode that good?
                                                                                Is there a possibility to use it on ios/Ipad os?
                                                                                What is with online sync and merging?

                                                                                1. 4

                                                                                  is org-mode that good?

                                                                                  I’ve yet to encounter an ex-org-mode user, which is a pretty strong endorsement IMO.

                                                                                  You really need to be running emacs to use it, though, which has been too high a barrier for me so far (and ensures it won’t run on iOS).

                                                                                  1. 2

                                                                                    Now you have!

                                                                                    I never felt satisfied with org-mode because I found it too flexible - it was hard to know what the best way to do things was, and so I kept fiddling with my workflow rather than actually getting things done. I went through a few different tools, and settled on using Trello with a kanban-style workflow in 2018, which I can definitely say was a gamechanger for me.

                                                                                    1. 1

                                                                                      I’m using it on my Android with orgzly and it’s not so bad. Mostly only for adding new notes and searching (Emacs is superior in terms of organization), but it does most of what I expect from my org-mode files on the phone anyway.

                                                                                      Also there are peopel who run Emacs on the phone (not sure if you can do it on ios), so with an external keyboard it’s meant to be the same experience.

                                                                                    2. 1

                                                                                      AFAIK there are iOS apps, but extracting it out of Emacs kind of looses the point.

                                                                                      And it’s a text file, you can sync and merge is just like any other text file. Org is just the magic that interprets and manipulates the text.

                                                                                      1. 1

                                                                                        org is glorious. the IOS options aren’t too good, I jump between beorg and plain text edits via an text editor (buffer) when on mobile but neither is really there. Organice.200ok.ch/ comes very closer to what I’d think a good compromise would be but afaict is not a native app.

                                                                                    1. 3

                                                                                      Going to integrate Roam Research data into

                                                                                      • HPI (Human Programming Interface): that would allow for programmatic access, and make the data access a matter of import my.roamresearch
                                                                                      • promnesia: that would integrate URLs stored in Roam as a browser extension, so you’d instaltly know if you have posts from a blog or twitter account annotated in Roam

                                                                                      I’ve already figured out the JSON export script, so the rest will hopefully be straightforward!

                                                                                      1. 2

                                                                                        The third statement: cd "$(dirname "$0")". So many scripts assume the user runs them from the script directory.

                                                                                        (not sure if it works on Mac)

                                                                                        1. 3

                                                                                          I love the big idea.

                                                                                          For points 1, 2, and 3 I would call out are unix shell pipelines and decade old applescript, and somewhat SQL queries. But they’re not so easy to change, and they don’t mix and match with each other. I’d say a huge obstacle is that it’s easy to work with data inside a particular language/framework, but it’s not easy to move data between languages.

                                                                                          I have a very personal use case I call the “Cary Grant” problem. I want to watch all the Cary Grant movies I haven’t seen, ordered by top IMDB score, for the least money possible.

                                                                                          That means I’d need to do an inner join on 1. the movie collection of my nearby libraries, 2. what netflix has right now, while filtering out my personal list of movies I’ve already seen in a text file, and order by the IMDB score. I tried this for a few weeks, nobody wants to share their data.

                                                                                          I have plenty more ideas along these lines. I recently downloaded several years of my fitbit data and it’s a huge mess of json that lacks documentation and has differing formats for older years.

                                                                                          For point 4, I want a fitbit watch where the data never leaves my own servers, same for my location data and which cell phone towers I use.

                                                                                          Do you have interesting use cases that would require joining on disparate data sources?

                                                                                          1. 2

                                                                                            Yes! I think about this a lot. I’m kind of obsessed about collecting my data and using it and it’s insane how hard is it to get hands on it in order to do something with it.

                                                                                            differing formats for older years

                                                                                            Yep, it’s one of the big issues I have. I’m consolidating my efforts in the HPI package, which is responsible for normalising and potentially arbitrating data.

                                                                                            Do you have interesting use cases that would require joining on disparate data sources?

                                                                                            Yep! For me, one common usecase for joining data is various well-being stats. For example: I’ve got dashboard, for sleep, exercise, food, etc. I’ve posted some screenshots here.

                                                                                            Another tool along these lines l I’m working on is promnesia, it’s a browser extension which unifies annotations and bookmarks from different sources.

                                                                                          1. 1

                                                                                            I wonder, how feasible would it be to use Ansible as a backend (i.e. to benefit from the plugins/state transitions/etc), and use Lisp merely to make “writing the configs” part pleasant?

                                                                                            Like, I 100% appreciate the effort, but do we really need to rewrite, say, apt module from scratch? Can’t we benefit from the existing, working and tested code?

                                                                                            1. 2

                                                                                              Well, what do I expect from the new tool is not only easier configs writing but also, better error messages.

                                                                                              With Ansible sometimes it is very hard to understand what went wrong. Especially, while you are developing your own roles.

                                                                                              I hope this new tool will make this process less painful.

                                                                                              1. 2

                                                                                                I was thinking about this recently, using a combination of Ansible and Starlark! I find that jinja2 is really limiting and not nice to work with, but using Starlark for example would help tremendously!!