1. 6

    In the general case, I have developed a deep and long-lasting skepticism of DSLs. I was a very heavy proponent of them during my grad studies, and investigated pretty thoroughly using a rules engine for Space Invaders Enterprise Edition and a runtime monitor for Super Mario World.

    I went a little further down this path before I abandoned it for reasons unrelated to the DSL skepticism. That happened later. I just wanted to give context that I was actually naturally predisposed to liking them.

    What has happened in my time on this earth as a software engineer is the feeling that it is axiomatic that all DSLs eventually tend towards something Turing complete. New requirements appear, features are added, the DSL heads further towards Turing completeness. Except the DSL does not have the fundamental mechanics to express Turing completeness, it is by fundamental design supposed to not do that. What you end up with is something very complex, where users are performing all sorts of crazy contortions to get behavior they want, and you can never roll that back. I feel like DSLs are essentially doomed from the outset.

    I am much, much more optimistic about opinionated libraries as the means to solve the problems DSLs do (Ruby on Rails being the most obvious one). That way any of the contortions can be performed in a familiar language that the developer is happy to use and won’t create crazy syntax, and the library then called to do whatever limited subset of things it wants to support. For basic users, they’ll interact with the library only and won’t see the programming language. As things progress, the base language can be brought in to handle more complex cases as pre/post-processing by the caller, without infringing on the design of the library.

    At Google, we have a number of DSLs to perform many different tasks which I won’t go into here. Each one requires a certain learning curve and a certain topping-out where you can’t express what you want. I was much happier with an opinionated library approach in Python, where I could do a great deal of what I wanted without peering behind the curtain of what was going to be performed.

    1. 6

      sklogic on Hacker News had a different view: you start with a powerful, Turing-complete language that supports DSL’s with them taking the place of libraries. He said he’ll use DSL’s for stuff like XML querying, Prolog where logic approach makes more sense, Standard ML when he wants it type-safe in simple form, and, if all else fails or is too kludgy, drops back into LISP that hosts it all. He uses that approach to build really complicated tools like his mbase framework.

      I saw no problem with the approach. The 4GL’s and DSL’s got messy because they had to be extended toward powerful. Starting with something powerful that you constrain where possible eliminates those concerns. Racket Scheme and REBOL/Red are probably best examples. Ivory language is an example for low-level programming done with Haskell DSL’s. I have less knowledge of what Haskell’s can do, though.

      1. 3

        I think it’s a good approach, but it’s still hard to make sure that the main language hosting all the DSLs can accomodate all of their quirks. Lisp does seem to be an obvious host language, but if it were that simple then this approach would have taken off years ago.

        Why didn’t it? Probably because syntax matters and error messages matter. Towers of macros produce bad error messages. And programmers do want syntax.

        And I agree that syntax isn’t just a detail; it’s an essential quality of the language. I think there are fundamental “information theory” reasons why certain syntaxes are better than others.

        Anything involving s-expressions falls down – although I know that sklogic’s system does try to break free of s-expression by adding syntax.

        Another problem is that ironically by making it too easy to implement a DSL, you get bad DSLs! DSLs have to be stable over time to be made “real” in people’s heads. If you just have a pile of Lisp code, there’s no real incentive for stability or documentation.

        1. 4

          “but if it were that simple then this approach would have taken off years ago.”

          It did. The results were LISP machines, Common LISP, and Scheme. Their users do little DSL’s all the time to quickly solve their problems. LISP was largely killed off by AI Winter in a form of guilt by association. It was also really weird vs things like Python. At least two companies, Franz and LispWorks, are still in Common LISP business with plenty of success stories on complex problems. Clojure brought it to Java land. Racket is heavy on DSL’s backed by How to Design Programs and Beautiful Racket.

          There was also a niche community around REBOL, making a comeback via Red, transformation languages like Rascal, META II follow-ups like Ometa, and Kay et al’s work in STEPS reports using “IS” as foundational language. Now, we have Haskell, Rust, Nim, and Julia programmers doing DSL-like stuff. Even some people in formal verification are doing metaprogramming in Coq etc.

          I’d say the idea took off repeatedly with commercial success at one point.

          “Probably because syntax matters and error messages matter. Towers of macros produce bad error messages. And programmers do want syntax.”

          This is a good point. People also pointed out in other discussions with sklogic that each parsing method had its pro’s and con’s. He countered that they can just use more than one. I think a lot of people don’t realize that today’s computers are so fast and we have so many libraries that this is a decent option. Especially if we use or build tools that autogenerate parsers from grammars.

          So, IIRC, he would use one for raw efficiency first. If it failed on something, that something would get run through a parser designed for making error detection and messages. That’s now my default recommendation to people looking at parsers.

          “Anything involving s-expressions falls down – although I know that sklogic’s system does try to break free of s-expression by adding syntax.”

          Things like Dylan, Nim, and Julia improve on that. There’s also just treating it like a tree with a tree-oriented language to manipulate it. A DSL for easily describing DSL operations.

          “nother problem is that ironically by making it too easy to implement a DSL, you get bad DSLs!”

          The fact that people can screw it up probably shouldn’t be an argument against it since they can screw anything up. The real risk of gibberish, though, led (per online commenters) a lot of teams using Common LISP to mandate just using a specific coding style with libraries and no macros for most of the apps. Then, they use macros just handling what makes sense like portability, knocking out boilerplate, and so on. And the experienced people wrote and/or reviewed them. :)

          1. 2

            Probably because syntax matters and error messages matter. Towers of macros produce bad error messages. And programmers do want syntax.

            Another problem is that ironically by making it too easy to implement a DSL, you get bad DSLs! DSLs have to be stable over time to be made “real” in people’s heads. If you just have a pile of Lisp code, there’s no real incentive for stability or documentation.

            I’m so glad to see this put into words. Although for me, I find it frustrating that this seem to be universally true. I was pretty surprised the first time around when I felt my debugger was telling me almost nothing because my syntax was so uniform, I couldn’t really tell where I was in the source anymore!

            Some possibilities for this not to be true that I’m hoping for: maybe its like goto statements and if we restrict ourselves to make DSLs in a certain way, they won’t become bad (or at least won’t become bad too quickly). By restricting the kind of gotos we use (and presenting them differently), we managed to still keep the “alter control flow” aspect of goto.

            Maybe there’s also something to be done for errors. Ideally, there’d be a way to spend time proportional to the size of the language to create meaningful error messages. Maybe by adding some extra information somewhere that currently implicit in the language design.

            I don’t know what to do about stability though. I mean you could always “freeze” part of the language I guess.

            For this particular project, I’m more afraid that they’ll go the SQL route where you need to know so much about how the internals work that it mostly defeats the purpose of having a declarative language in the first place. I’d rather see declarative languages with well-defined succinct transformations to some version of the code that correspond to the actual execution.

            1. 1

              (late reply) Someone shared this 2011 essay with me, which has apparently been discussed to death, but I hadn’t read it until now. It says pretty much exactly what I was getting at!

              http://winestockwebdesign.com/Essays/Lisp_Curse.html

              In this essay, I argue that Lisp’s expressive power is actually a cause of its lack of momentum.

              I said:

              Another problem is that ironically by making it too easy to implement a DSL, you get bad DSLs!

              So that is the “curse of Lisp”. Although he clarifieds that they’re not just “bad” – there are too many of them.

              He mentions documentation several times too.

              Thus, they will have eighty percent of the features that most people need (a different eighty percent in each case). They will be poorly documented. They will not be portable across Lisp systems.

              Domain knowledge is VERY hard to acquire, and the way you share that is by developing a stable and documented DSL. Like Awk. I wouldn’t have developed Awk on my own! It’s a nice little abstraction someone shared with me, and now I get it.

              The “bipolar lisp programmer” essay that he quotes also says the same things… I had not really read that one either but now I get more what they’re saying.

              1. 1

                Thanks for sharing that link again! I don’t think I’ve seen it before, or at least have forgotten. (Some of the links from it seem to be broken unfortunately.)

                One remark I have is that I think you could transmit information instead of code and programs to work around this curse. Implicit throughout the article is that collaboration is only possible if everyone uses the same language or dialect of it; indeed, this is how version controlled open-source projects are typically structured: around the source.

                Instead, people could collaboratively share ideas and findings so everyone is able to (re)implemented it in their own DSL. I say a bit more on this in my comment here.

                In my case, on top of documentation (or even instead of it), I’d like to have enough instructions for rebuilding the whole thing from scratch.

                To answer your comment more directly

                Domain knowledge is VERY hard to acquire, and the way you share that is by developing a stable and documented DSL

                I totally agree that domain knowledge is hard to acquire but I’m saying that this only one way of sharing that knowledge once found. The other way is through written documents.

        2. 4

          Since I like giving things names, I think of this as the internal DSL vs external DSL argument [1]. This applies to your post and the reply by @nickpsecurity about sklogic’s system with Lisp at the foundation. If there is a better or more common name for it, I’d like to know.

          I agree that internal DSLs (ones embedded in a full programming language) are preferable because of the problems you mention.

          The external DSLs always evolve into crappy programming languages. It’s “failure by success” – they become popular (success) and the failure mode is that certain applications require more power, so they become a programming language.

          Here are my examples with shell, awk, and make, which all started out non Turing-complete (even Awk) and then turned into programming languages.

          http://www.oilshell.org/blog/2016/11/14.html

          Ilya Sher points out the same problems with newer cloud configuration languages.

          https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a-programming-language/

          I also worked at Google, and around the time I started, there were lots of Python-based internal DSLs (e.g. the build system that became Blaze/Bazel was literally a Python script, not a Java interpreter for a subset of Python).

          This worked OK, but these systems eventually got rewritten because Python isn’t a great language for internal DSLs. The import system seems to be a pretty significant barrier. Another thing that is missing is Ruby-style blocks, which are used in configs like Vagrantfile and I think Puppet. Ruby is better, but not ideal either. (Off the top of my head: it’s large, starts up slowly, and has version stability issues.)

          I’m trying to address some of this with Oil, although that’s still a bit far in the future :-/ Basically the goal is to design a language that’s a better host for internal DSLs than Python or Ruby.

          [1] https://martinfowler.com/bliki/InternalDslStyle.html

          1. 3

            If a programming language is flexible enough, the difference between DSL and library practically disappears.

            1. 1

              DSL’s work great when the domain is small and stays small and is backed by corporal punishment. Business Software is an astronomically large domain.

            1. 4

              return in a generator throws a StopIteration exception, which is a useful feature but not the same thing as yield from at all. That being said, yield from will also throw a StopIteration exception if the generator it’s yielding from throws one. StopIteration is nothing to be afraid of.

              1. 1

                StopIteration is nothing to be afraid of.

                StopIteration is definitely nothing to be afraid of, but a big change in logic, from a subtle change in code is. Since returning the value doesn’t do anything other than stopping the iteration, I would much rather not even being able to actually return a value (empty return statements are okay). This can be caught by linting, but I can see it easily passing code review if the diff doesn’t include both the return val and the yield.

                1. 2

                  You’re right, but I think it’s worth also remembering that return to yield from is really no smaller a change in code than say if to while.

                  1. 1

                    Yeah that’s quite true.

                  2. 1

                    In Python 2, a non-empty return inside a generator (any def that yield) would give you a syntax error

                    SyntaxError: 'return' with argument inside generator
                    

                    So it used to be the way you wanted it. I hadn’t noticed this change in Python 3.

                    I have an implementation of an interpreter for (a subset of) Python and I think it would be clearer if a different keyword than def (like gen) was used for generators. This would also mean the interpreter doesn’t need to look inside the body to know if its a function or generator.

                    1. 1

                      I think the gen can be a good idea, since it would make it also a bit clearer, for the reader. In general I’ve barely had any issues from confusing generators and functions so I guess the main benefit would still be for the interpreter. Also really interesting project you’ve got, the link you posted is broken though and redirects to lobste.rs/github.com/.....

                1. 3

                  Your experience unfortunately matches mine with generic solvers of all kinds (well, except one): its too slow for any input that could interest me. I’m amazed at how clear the write-up is despite things not going that well. I might try some of the same things but would be mum if asked to explain it.

                  What happens if, to break symmetry, you just said “there are at most # rooms (R) talks at any time”? And remove ROOMS entirely from the model.

                  Like

                  constraint forall(t in TIMES)(
                      sum(talk_times[talk] == t) <= R);
                  
                  1. 2

                    I just tried that, and it’s pretty nice! You can rip out sched entirely. I couldn’t find a good way to add symmetry on times, but that plus a crude constraint talk_time[talks[1]] = TIMES[1]; gives you another 5x speedup.

                    This gave me an idea: what if we have a variable set of the talks for each time and link them via int_set_channel? In theory, this would give you another huge bonus.

                    array[TIMES] of var set of talks: talks_per_time;
                    
                    constraint int_set_channel(talk_time, talks_per_time);
                    
                    constraint forall(t in TIMES)(
                       card(talks_per_time[t]) = R);
                    

                    In practice, Chuffed can’t handle var sets :(

                    1. 2

                      Oh yeah, good idea. So you’re basically saying the talks are partitioned into TIMES sets, each of size at most R [1]. Looks like MiniZinc even has a builtin partition_set but Chuffed won’t handle that either.

                      I couldn’t find a good way to add symmetry on times, but that plus a crude constraint talk_time[talks[1]] = TIMES[1]; gives you another 5x speedup.

                      If instead of using partition_set, you wanted to do this by hand, you could continue along the same lines as the crude constraint and say “the lowest indexed talk scheduled at time i or later is always scheduled at time i” (for all i).

                      [1] I think you still need “at most” instead of “equal” unless you happen to have (# rooms) * (# time slots) talks (or add dummy talks nobody likes to get this to be true).

                    2. 1

                      solvers of all kinds (well, except one)

                      And that exception would be…?

                      1. 1

                        Solvers for problems with continuous valued variables (float values) with linear constraints and objective.

                        This might be a somewhat disappointing answer since the problems you can specify are much less generic. Usually needs some work just translating the problem into the input format (if it can be done at all), which is opposite to MiniZinc’s interface improvement.

                    1. 7

                      I ended up away from my PC for a bit much sooner than I expected, so I didn’t get to take part in the discussion here, but I rather feel like most of the comments here missed the point.

                      While I’m not disputing that no programming language in practice can be turing complete, the mistake is jumping into thinking about implementations when the article is considering instead the abstract specification of the C language. In particular, the reason why C is not turing complete is because the specification requires pointer semantics that bake in enough implementation details that instead of breaking turing completeness, it’s required that no C implementation could ever be turing complete, even in the face of changing basic rules about computing.

                      In another thread of comments, there is the request as to a language that would satisfy the authors definition of turing completeness “in theory”: Brainfuck. If you consider brainfuck as “a bidirectional infinite tape with the following operators […]”, then we have a language specification that is turing complete, even if no implementation can be. One could argue of course that this means that the specification can never be fully and correctly implemented and that’s true, but you can also write specifications that equally state neither behaviours that force turing incompleteness (C), nor make explicit demands that are impossible to fulfill (brainfuck), and instead leave such details out entirely (imagine a language that deals only in objects and operations on them, without reference to their representation in any implementation whatsoever).

                      1. 1

                        Thanks for clarifying this a bit. I’m still confused.

                        the specification requires pointer semantics that bake in enough implementation details

                        If I understand, here you are saying there’s “too many details included in the spec”. But I don’t understand the next sentence fragment at all.

                        that instead of breaking turing completeness, it’s required that no C implementation could ever be turing complete, even in the face of changing basic rules about computing.

                        On its own, “it’s required that …” could make sense but the two sentence fragments don’t match and make no sense (to me) together.

                        In another thread of comments, there is the request as to a language that would satisfy the authors definition of turing completeness “in theory”: Brainfuck. If you consider brainfuck as “a bidirectional infinite tape with the following operators […]”, then we have a language specification that is turing complete, even if no implementation can be.

                        Could you say what the equivalent for C is here? Consider C as “a [???] with the following operations”?

                        1. 3

                          Could you say what the equivalent for C is here? Consider C as “a [???] with the following operations”?

                          The problem is that what the gp said isn’t quite true. Brainfuck isn’t a bidirectional infinite tape with the following ops. Brainfuck has a bidirectional infinite tape, and the following operators. C has a declarations, expressions, control flow, functions, memory, macros, etc. etc. It’s tempting to say that brainfuck is the bidirectional infinite tape, but it’s not, it’s a programming language that has only a couple of things.

                          1. 2

                            Could you say what the equivalent for C is here? Consider C as “a [???] with the following operations”?

                            The problem is that in C, there is required to be a maximum number of valid pointers and as a result there is required to be a maximum number of valid addressable bytes, of a maximum bit length. This means memory is bounded and the C specification describes a finite state machine.

                            You can implement a turing machine in Brainfuck. You cannot implement a turing machine in C.

                            1. 1

                              A programming language specification can leave the harsh realities of the real world to the implementation, and if you were to find some bizarre reality where genuine turing completeness is possible, you could in essence be able to implement some languages in a way that exploits this new theory of computation and create a genuinely turing complete language implementation.

                              In C, however, there are specific rules about how pointers must be implemented that mean even in this world, you’d have to choose between correctly implementing the spec, or being turing complete, as the rules themselves have implications that mean you could not exploit the new reality to achieve this. These restrictions aren’t apparent to us everyday because we don’t have a world where turing completeness is possible, but that doesn’t mean the specification has to reflect those limitations - by choosing to leave the details of representation to the implementation, implementors could choose to make it as strong as their current reality allows.

                              So, overall, what I mean in that sentence is that in C, implementations do not lose potential for turing completeness as set out by the spec, limited by reality, but instead start from a spec where it is already by definition not permitted.

                              As for what C “is” in comparison to brainfuck, really Elronnd has it. The language I used is a little bit of a red herring in that brainfuck isn’t the tape in the same way the closest thing that C would “be” is some abstract machine you could build from the spec. It’s easy to build a mental model where the only object is a tape/machine, but the language specs really instead have the machine and also have rules to about how they work and are manipulated. I can only apologise for being so relatively casual in a discussion where the language needs to be precise.

                          1. 1

                            The topic is interesting and perhaps the argument once I know what it is but the presentation is pretty bad. There are far too many implicit assumptions all over the place. I think that’s why we are confused. (For example, why are we looking at pointers? How is that relevant to Turing-completeness??)

                            Can someone say what exactly “C is not Turing-complete” means here?

                            Here’s what I have reconstructed from the comments at the moments. Please help me if you’d also like to get to the bottom of this.

                            What this author is trying to show is:

                            Considering an idealised C11 which guarantees only the minimum from its spec, it is not possible to implement a Turing machine with a tape of size N.

                            The author does not say what N is but the intended difficulty is supposed to relate to size. At first, this makes no sense: we could just make a program of size N. But this feels like cheating in other context too. We should be only allowed one program and on a real machine, it would just crash when it runs out of memory.

                            Ok, so let’s revise that to

                            Considering an idealised C11 which guarantees only the minimum from its spec, for any implementation of a Turing machine, there’s a TM program this implementation is unable to simulate for every amount M of memory.

                            But now if we have M memory, can’t we just allocate it all using malloc? Is that not in the spec? But the post talks about the heap! Some people are talking about bignum. Is the trouble that even if we can allocate the memory, we won’t be able to read/write to all of it because we are unable to store the address them?

                            [Stuff to fill in.]

                            And therefore, our simulator always has a tape of size N << M?

                            1. 2

                              Say you have an infinite heap, or better yet a heap that can grow with you as your usage grows. You can then yes use malloc to allocate as much as you want, but having an infinite tape is only one part of it, being able to process the tape is the more important part. How would you process that malloced memory? You’ll do that by using pointers, and if the size of pointers is statically known at compile time, what happens when your malloced memory has more bytes than your pointers can possibly represent? (remember your heap can grow infinitely but sizeof(T *) is known at compile time).

                              1. 1

                                (For example, why are we looking at pointers? How is that relevant to Turing-completeness??)

                                Because if C is not capable of unbounded memory then the question of whether it’s Turing complete is very easy to answer: no.

                              1. 2

                                I keep a bunch of markdown files. Its not really a wiki like the one shown here. Its mostly notes on how to redo things I’ve figured out before or (I’m hoping) short amount of text I can read to quickly get back up to speed. They are disjoint and don’t link much to each other.

                                I later made a web interface for it but then pretty much never used it.

                                1. 4

                                  I saw @rain1 post nand game here about a month ago and quite liked it. Then I made this. (Not too sure how to tag it though. The original was just ‘games’.)

                                  1. 2

                                    This all means that I think it’d be more useful to everybody if I posted a more comprehensive article on how I optimized my model rather than just give some highlights here. Then it might help other beginners out there. Said article is about 3000 words long and will be going up next week.

                                    My thoughts exactly. Thanks for noticing this and writing the second article!

                                    Also, I’m probably slow/too used to other things but just in case it helps others: in the MiniZinc syntax, the name of the variable (or parameter) is the rightmost token on a line (between : and ;).

                                    1. 2

                                      Can you say what some of the main conclusions are? Or is this primarily a survey? E.g.,

                                      this article identifies game facet orchestration as the central challenge for AI-based game generation

                                      In particular, we identify the different creative facets of games

                                      Are these the six facets: audio, visuals, levels, rules, narrative and gameplay? But the intro also suggests that only music (audio?) will be looked at. Or maybe its only meant as linguistic metaphor.

                                      we propose how orchestration can be facilitated in a top-down or bottom-up fashion,

                                      How?

                                      we conclude by discussing the open questions and challenges ahead.

                                      I’m guessing this is

                                      • Combinatorial Explosion
                                      • Learning the Mapping Between Facets
                                      • Orchestration with Human Designers
                                      • Evaluating Orchestration

                                      envisioned several high-level orchestration processes along a spectrum between purely hierarchical top-down generation and organic bottom-up generation.

                                      What are some examples of these processes?

                                      (I’m thinking this is just my unfamiliarity with the topic but the last two sentences of the abstract are saying almost the same thing.

                                      I wish abstracts in general gave more information about their conclusion like key pieces of information the authors would have liked to know before starting the project. Stuff that would have helped speed things up a lot.

                                      1. 4

                                        I’d describe it as a forward-looking survey. The starting point is that there’s a large existing body of work on procedural content generation, but they’re systems that generate one kind of thing at a time, like level generators, pixel-art makers, or procedural audio. Can you get to full-game generation by just plugging these kinds of generators together in some way? We discuss different architectures for orchestrating these kinds of generators: top-down, bottom-up, pipelined, etc., and survey existing systems that have already done some version of that.

                                        The six facets we propose are ones you mentioned, yeah. There are many other ways you could slice game design, so this isn’t necessarily the ultimate metaphysical truth about games, that they’re made of exactly six kinds of stuff. But it’s based on looking at what kinds of things existing procedural generators currently generate (level generators are probably the most common, but there’s work in the other five too).

                                        Yeah, the “orchestrate”, “jam”, etc. terminology is just a musical metaphor. We don’t only focus on game music here, but we use analogies like top down orchestra-style scoring/conducting, where every element of the overall system is given its centrally written part, vs. bottom-up jazz improvisation where they coordinate less hierarchically, etc. I can see how that can get confusing, sorry.

                                        The survey part of the paper is in the case study section, where we give an overview of nine existing systems that generate more than one kind of thing in tandem (e.g. rules, levels, and music). We translate what each of them does to our language of 6 facets and different orchestration strategies, to try to get an idea of how all the existing stuff relates to each other, and what it doesn’t do yet. The first author (A. Liapis) made some colorful triangle facet-orchestration diagrams for that section that I quite like. They summarize what each system is doing in a kind of pictogram language, showing which facets the system generates and how they interrelate (e.g. some systems have a pipeline, some scrape content off the web, some ask for user input at certain points, etc.).

                                        edit: I also wrote a short twitter thread earlier today with a more concise version of this explanation, for people who like twitter threads (I know, I know).

                                        1. 1

                                          Thanks! The figures in the survey are indeed really nice. (It wasn’t obvious before reading your comment that the arrows was the information about the orchestration process.)

                                      1. 1

                                        I’d also like to know What are you using for speech recognition underneath?

                                        Looks like its something generic since text replacement is used for postprocessing to improve accuracy.

                                                "please tab": "previous tab",
                                                "fort worth": "forward",
                                        

                                        or even

                                                "ghost app": "close tab",
                                                "collect a blonde": "select tab 1",
                                        
                                        1. 11

                                          Similar to Fibonacci, a piece of code that enlightened me was the simple PEG implementation, which is represented as a pair of mutually recursive procedures. I used to think of parsers as really difficult to understand, and hard to write. But the simplicity of PEG was an eye opener.

                                          def unify_key(key, text, at):
                                             if key not in grammar:
                                                 return (True, at + len(key)) if text[at:].startswith(key) else (False, at)
                                             rules = grammar[key]
                                             for rule in rules:
                                                 res, l = unify_rule(rule, text, at)
                                                 if res: return (res, l)
                                             return (False, 0)
                                          
                                          def unify_rule(rule, text, at):
                                              for token in rule:
                                                    result, at = unify_key(token, text, at)
                                                    if not result: return (False, at)
                                              return (True, at)
                                          

                                          Given a grammar as below, we try to unify the input string with the starting key – here expr. If To unify a key with the given string (unify_key), we check if the key is in the grammar. If it was not, then it is a plain token match, and we do simple string match. If not, we get the production rules defined in the grammar corresponding to the key, and try each rule one by one using unify_rule. We return as soon as any rule succeeds. The unify_rule is also simple. It gets the parts of the rule, and tries to match them in sequence using unify_key. If any of these fails, then return failure.

                                          grammar = {
                                              'expr': [
                                                  ['term', 'add_op', 'expr'],
                                                  ['term']],
                                              'term': [
                                                  ['fact', 'mul_op', 'term'],
                                                  ['fact']],
                                              'fact': [
                                                  ['digits'],
                                                  ['(','expr',')']],
                                              'digits': [
                                                  ['digit','digits'],
                                                  ['digit']],
                                              'digit': [[str(i)] for i in list(range(10))],
                                              'add_op': [['+'], ['-']],
                                              'mul_op': [['*'], ['/']]
                                          }
                                          

                                          A driver

                                          import sys
                                          res, till = unify_key('expr', sys.argv[1], 0)
                                          assert(till == len(sys.argv[1]))
                                          print(res)
                                          

                                          Using it:

                                          $ python3 peg.py '1+10/(3+2)'
                                          True
                                          $ python3 peg.py '1+10/x(3+2)'
                                          Assertion failure
                                          

                                          While this implementation can have really bad performance due to back tracking, all one needs to do to make it linear is to decorate the procedures with @functools.lru_cache(maxsize=None), which memoizes the parsing, and makes parsing linear. While this does not implement the complete PEG syntax, it implements enough to be complete in terms of the grammars that can be represented (see Ford 2004 for details).

                                          Edit: Thanks @asrp for finding that the original had issues.

                                          1. 2

                                            Tried it with something like

                                            print unify_rule(['expr'], '1+10/(3+2)', 0)
                                            

                                            There seems to be quite a few typos in here

                                            -   if key not in grammar: return (text[at:].starts_with(key), len(rule))
                                            +   if key not in grammar: return (text[at:].startswith(key), len(key))
                                            
                                            -      if not result: return False
                                            +      if not result: return (False, 0)
                                            
                                            -  return (True, len)
                                            +  return (True, l)
                                            

                                            There must be more errors since it still only matches the first character of the expression.

                                            I found confusing for both rule names and string token to be encoded as strings. The same goes for concatenation and ordered choice to both be encoded lists.

                                            Thanks for bringing up this example, though!

                                            1. 2

                                              Sorry about that, and thank you so much for taking time to run it. I took the code from an old presentation, and seems I made errors while typing it in.

                                              I have fixed it.

                                              1. 1

                                                Thanks for the new version! I understand what its trying to do better now.

                                            2. 1

                                              Thanks for the example! While PEGs are not familiar to me (I’ll need to look into them), there’s a lot of potential for minds to be blown in the parser world. Playing a few times with parser combinators did exactly that to me - if that’s something that’s not as familiar to you, then here’s a nice example in Scala.

                                              1. 1

                                                The parser combinators are a way to represent PEG directly in code. So you will find them quite familiar. I love combinator based parsing, they are quite powerful and enlightning – A while back, I did a few posts (1, 2, 3, 4, 5, 6, 7) on building a small concatenate language using Parsec (The original monadic combinator library in Haskell) for parsing. You may find them interesting.

                                            1. 2

                                              Looks like they didn’t run bibtex enough times. Here’s a version with citations fixed:

                                              https://profs.info.uaic.ro/~stefan.ciobaca/faoc2016.pdf

                                              Can anyone comment on the generality and efficiency of their method? I’ve only skimmed it very quickly but didn’t find discussion about that.

                                              1. 3

                                                I can’t tell you that as a non-specialist. What I can tell you is it’s based on matching logic that Rosu et al use in the K Framework. They used that to make a C compiler, lots of language specs, and a static analysis tool with low, false positives. They’re interesting to me since they operate in their own bubble of rewriting logic instead of Coq, Isabelle/HOL, etc. They built on Maude.

                                                1. 2

                                                  Interesting. I pretty much don’t know anything here. Thanks for the links.

                                              1. 3

                                                This is pretty nice. I just finished it.

                                                Up to the ALU, you could sort of see the components you are building will be useful. But afterwards, I didn’t always really follow the reasoning through. I wish it let you design the instructions, or at least let you think about them. That seems like the more interesting/difficult part.

                                                That and maybe a “hard mode” where you have access to all your previous components.

                                                Would something like this work at a decent speed on real hardware (if you don’t mind the size) or is there normally too much optimization going on?

                                                1. 2

                                                  it is kind of like the minimum most simple CPU. A lot of extra features to speed it up can be added. Like a pipeline.

                                                  1. 2

                                                    Right, but how many times slower do we expect if this is used unmodified?

                                                    The follow up question would be: what optimization to do if you could only pick 1 or 2?

                                                1. 3

                                                  https://blog.asrpo.com/

                                                  I write sporadically, about what I think needs to be written about. It mostly contains bootstrapping/recreating things from scratch things at the moment. There’s a sort-of overview here.

                                                  Its also intended as a reverse search engine of sorts where I try to find people interested in the same things.

                                                  1. 0

                                                    This is ill-advised.

                                                    You cannot define 1/0 and still have a field. There’s no value that works. Even when you do things like the extended real numbers where x/0 = infinity, you’re really just doing a kind of shorthand and you acknowledge that the result isn’t a field.

                                                    You can of course define any other algebraic structure you want and then say that operating on the expression 1/0 is all invalid because you didn’t define anything else and no other theorem applies, but this is not very helpful. You can make bad definitions that don’t generalise, sure, definitions that aren’t fields. But to paraphrase a famous mathematician, the difficulty lies not in the proofs but in knowing what to prove. The statement “1/0 = 0 and nothing else can be deduced from this” isn’t very interesting.

                                                    1. 1

                                                      Could you explain why, formally, defining 1/0=0 means you no longer have a field?

                                                      1. 7

                                                        I want to make an attempt to clarify the discussion here because I think there is some substance I found interesting. I don’t have a strong opinion about this.

                                                        The article actually defines an algebraic structure with three operators: (S, +, *, /) with some axioms. It happens that these axioms makes it so (S, +, *) is a field (just like how the definition of a field makes (S, +) a group).

                                                        The article is right in saying that these axioms do not lead to a contradiction. And there are many non-trivial such structures.

                                                        However, the (potential) issue is that we don’t know nearly as much about these structures than we do about fields because any theorem about fields only apply to (S, +, *) instead of (S, +, *, /). So all the work would need to be redone. It could be said that the purpose of choosing a field in the first place is to benefit from existing knowledge and familiar expectations (which are no longer guarantteed).

                                                        I guess formally adding an operator means you should call it something else? (Just like how we don’t call fields a group even though it could be seen as a group with an added * operator.)

                                                        This has no bearing on the 1/0 = 0 question however, which still works from what’s discussed in the article.

                                                        1. 1

                                                          As I understand it, you’ve only defined the expression 1/0 but you are saying that /0 isn’t shorthand for the multiplicative inverse of 0 as is normally the case for /x being x^-1, by definition. Instead, /0 is some other kind of magical non-invertible operation that maps 1 into 0 (and who knows what /0 maps everything else into). Kind of curious what it has to do with 0 at all.

                                                          So I guess you can do this, but then you haven’t defined division by zero at all, you’ve just added some notation that looks like division by zero but instead just defined some arbitrary function for some elements of your field.

                                                          If you do mean that /0 is division by zero, then 1/0 has to be, by definition, shorthand for 1*0^-1 and the arguments that you’ve already dismissed apply.

                                                          1. 4

                                                            The definition of a field makes no statements about the multiplicative inverse of the additive identity (https://en.wikipedia.org/wiki/Field_(math)#Classic_definition). Defining it in a sound way does not invalidate any of the axioms required by the field, and, in fact, does define division by zero (tautologically). You end up with a field and some other stuff, which is still a field, in the same way that adding a multiplication operator on a group with the appropriate properties leaves you with a group and some other stuff.

                                                            The definition of the notation “a / b => a * b^-1” assumes that b is not zero. Thus, you may define the case when b is 0 to mean whatever you want.

                                                            That people want to hold on to some algebraic “identities” like multiplying by the denominator cancels it doesn’t change this. For that to work, you need the assumption that the denominator is not zero to begin with.

                                                            1. 1

                                                              In what way, whatever it is you defined /0 to be, considered to be a “division”? What is division? Kindly define it.

                                                              1. 3

                                                                Division, a / b, is equal to a * b^-1 when b is not zero.

                                                                1. 2

                                                                  And when b is zero, what is division? That’s the whole point of this argument. What properties does an operation need to have in order to be worthy of being called a division?

                                                                  1. 3

                                                                    Indeed, it is the whole point. For a field, it doesn’t have to say anything about when you divide by zero. It is undefined. That doesn’t mean that you can’t work with and define a different, but still consistent, structure where it is defined. In fact, you can add the definition such that you still have the same field, and more.

                                                                    edit: Note that this doesn’t mean that you’re defining a multiplicative inverse of zero. That can’t exist and still be a field.

                                                                    1. 1

                                                                      In what way is it consistent? Consistent with what? As I understand it, you’re still saying that the expression 1/0 is an exception to every other theorem. What use is that? You still have to write a bunch of preconditions, even in Coq, saying how the denominator isn’t zero. What’s the point of such a definition?

                                                                      It seems to me that all of this nonsense is about not wanting to get an exception when you encounter division by zero, but you’re just delaying the problem by having to get an exception whenever you try to reason with the expression 1/0.

                                                                      1. 3

                                                                        I mean that the resulting structure is consistent with the field axioms. The conditions on dividing by zero never go away, correct. And yes, this is all about avoiding exceptions in the stack unwinding, programming language sense. The article is a response to the statements that defining division by zero in this way causes the structure to not be a field, or that it makes no mathematical sense. I am also just trying to respond to your statements that you can’t define it and maintain a field.

                                                                        1. 1

                                                                          It really doesn’t make mathematical sense. You’re just giving the /0 expression some arbitrary value so that your computer doesn’t raise an exception, but what you’re defining there isn’t division except notationally. It doesn’t behave like a division at all. Make your computer do whatever you want, but it’s not division.

                                                                          1. 5

                                                                            Mathematical sense depends on the set of axioms you choose. If a set of axioms is consistent, then it makes mathematical sense. You can disagree with the choices as much as you would like, but that has no bearing on the meaning. Do you have a proof that the resulting system is inconsistent, or even weaker, not a field?

                                                                            1. 1

                                                                              I don’t even know what the resulting system is. Is it, shall we say, the field axioms? In short, a set on which two abelian operations are defined, with two distinct identities for each abelian operation, such that one operation distributes over the other? And you define an additional operation on the distributing operation that to each element maps its inverse, except for the identity which instead is mapped to the identity of the distributed-over operation?

                                                                              1. 2

                                                                                It’s a field where the definition of division is augmented to include a definition when the divisor is zero. It adds no new elements, and all of the same theorems apply.

                                                                                1. 1

                                                                                  I’m bailing out, this isn’t a productive conversation for either of us. Sorry.

                                                                                  1. 1

                                                                                    You are correct. The field axioms are all still true, even if we extend / to be defined on 0.

                                                                                    The reason for this is that the axioms never “look at” any of the values x/0. They never speak of them. So they all hold regardless of what x/0 is.

                                                                                    That said, even though you can define x/0 without violating axioms it doesn’t mean you should. In fact it seems like a very bad idea to me.

                                                              2. 1

                                                                That doesn’t make it not a field; you don’t have to have a division operator at all to be a field, let alone a division operator that is defined to be multiplication by the multiplicative inverse.

                                                                1. 1

                                                                  What is division?

                                                                  1. 1

                                                                    zeebo gave the same answer I would give: a / b is a multiplied by the multiplicative inverse of b when b is not zero. This article is all about how a / 0 is not defined and so, from an engineering perspective, you can define it to be whatever you want without losing the property that your number representation forms a field. You claimed that defining a / 0 = 1 means that your numbers aren’t a field, and all I’m saying is that the definition of the division operator is 100% completely orthogonal to whether or not your numbers form a field, because the definition of a field has nothing to say about division.

                                                                    1. 1

                                                                      What is an engineering perspective?

                                                                      Also, this whole “a field definition doesn’t talk about division” is a bit of misunderstanding of mathematical idioms. The field definition does talk about division since “division” is just shorthand for “multiplicative inverse”. The reason the definition is written the way it is (excluding 0 from having a multiplicative inverse) is that giving zero a multiplicative inverse results in contradictions. When you say “ha! I won’t let that stop me! I’m going to define it anyway!” well, okay, but then either (1) you’re not definining a multiplicative inverse i.e. you’re not defining division or (2) you are defining a multiplicative inverse and you’re creating a contradiction.

                                                                      1. 1

                                                                        (I had a whole comment here, but zeebo is expressing themselves better than I am, and there’s no point in litigating this twice, especially when I feel like I’m just quoting TFA)

                                                                        1. 1

                                                                          Me too, I’m tapping out.

                                                          1. 1

                                                            Great article!

                                                            I’m a bit late to the party because I wanted to try this one myself by just eyeballing and refactoring.

                                                            Java concurrency

                                                            The first thing that confused me was the concurrency model. The methods are supposed to be atomic because of synchonized but wait obviously make it non-atomic (the operation is broken between before the wait call and after). Then what does notify do? Does it pause the thread that called notify? Or is the woken up thread paused but next to execute?

                                                            Looking at the first few Google results didn’t get me a clear answer to this.

                                                            I assumed the former and this could make a lot of take thread wake each other up, block on notify and then all continue pushing occupied into the negatives and return invalid values.

                                                            I think your spec assumes the latter.

                                                            Concurrency and testing
                                                            if random.randint(0, 10000) == 1234:
                                                                cause_error
                                                            

                                                            would also be hard to test, need many runs and still have a chance of missing the error. This feels like a missing feature of the testing tool. For any non-deterministic part, the test should be allowed to pick the “random” values. In this case, the test should be allowed to pick which thread notify wakes up. And which thread runs next if there is a choice.

                                                            This is not saying anything about the difficulty in implementing such a testing tool but it also seems necessary if there’s going to be non-deterministic parts.

                                                            1. 3

                                                              Then what does notify do? Does it pause the thread that called notify? Or is the woken up thread paused but next to execute?

                                                              I assumed the former and this could make a lot of take thread wake each other up, block on notify and then all continue pushing occupied into the negatives and return invalid values.

                                                              I think your spec assumes the latter.

                                                              To my understanding it’s neither: notify wakes up a thread but does not execute it, nor does it guarantee it’ll be executed next. It just adds the woken thread into the scheduled pool.

                                                            1. 3

                                                              You may already be aware of this but here’s an example about composability, especially with respect to editing parents. Its not primarily “for making UIs”.

                                                              The Alto & Squeak environments get close, and so does Tcl/Tk’s ‘wish’ interpreter, but we’re not there yet.

                                                              For each of these, which of your items are missing?

                                                              The beginning of some work along these lines is here: https://github.com/enkiv2/misc/tree/master/kamui

                                                              Can you post some screenshots and/or describe and discuss your current implementation, even if not final? It would help clarify some of your implicit assumptions about the problem and model. (For example, what are things you would consider a widget and not a widget.)

                                                              Every object has a visual representation.

                                                              Why only one? Or do you mean at least one?

                                                              Every visual representation corresponds to an object.

                                                              Also why only one? What about visually nested things? (Is a representation something already rendered or an abstract description?)

                                                              Edit: Also, can you post some links to some examples of UIs on TV you consider good examples of what you want to do?

                                                              1. 6

                                                                For each of these, which of your items are missing?

                                                                I consider Smalltalk to have too high an initial learning overhead for this purpose (since my goal is to have the composability of graphical elements be an ‘invitation’ to programming in general for casual users).

                                                                Tk has a lower initial learning overhead but there are a lot of inconsistencies in how different widgets work (along with problems like a single-thread model), and no system has been written using Tk that does composability in the natural way smalltalk environments do. (I attempted a TCL/TK implementation many years ago. It’s briefly described here. It quickly became unmaintainable, and the widgets were not flexible enough for my liking.)

                                                                Can you […] describe and discuss your current implementation

                                                                Sure!

                                                                Io is an extremely simple homoiconic language with a prototype-based object system. It looks a lot like rebol. I chose it because an important part of making a language look understandable to non-programmers is conceptual simplicity, and Io’s pointfree style and message-passing-like behavior is a good fit for a context where a lot of the coding is glue between objects.

                                                                Because of certain features I wanted that stock Io didn’t have and certain behaviors that might be bugs (and because Io is not maintained), I decided to write my own implementation in a language with primitives that are a better match for the parts of Io I consider important for this task.

                                                                Since Io has a prototype-object system and not a class-object system, exploration through cloning a working object and modifying its guts is possible at run time. I expect this to be a normal pattern for experimentation.

                                                                I have defined the ‘widget’ object as having a position, size, and bitmap. Any widget inherits these properties. So, any widget has a visual representation, in the sense that it has a bitmap & that bitmap gets blitted to the display at render time. (Of course, it can be transparent, or whatever.) If you look at my implementation, I’ve also got some logic for visually nested things: children are positions at offset with respect to their parents, always at a higher layer, and their position affects the computed size of their parent.

                                                                The importance of widgets having a visual representation is that the expected method of modifying a widget is by selecting it in the interface and inspecting its implementation. So, ideally, every live object would be a widget whose visual representation indicates its state. (I don’t plan to make non-widget objects impossible, but it’s bad form, since they’re going to be harder to find and change.)

                                                                Unfortunately, I haven’t gotten to the point of having anything render. Stock Io’s various graphics bindings won’t build for me, & my reimplementation has a ways to go. Screenshots might be misleading, though: my goal is to have a system that’s extremely mutable in its form. Squeak’s smalltalk-based desktop is a good representation of the kind of thing I’m aiming for.

                                                                can you post some links to some examples of UIs on TV you consider good examples of what you want to do?

                                                                I’ll dig up some. Alternately, TVTropes has some lists.

                                                                Think along the lines of the way viruses are depicted in the movie Hackers – as graphical objects with apparent properties. Obviously, viruses have other reasons for not exposing their nature graphically – the whole point of them is to be hidden from & hostile to the end user – but having a graphical representation that matches the end user’s mental model of a task (as opposed to skeuomorphism, wherein it matches a programmer’s model of a device that used to accomplish a similar task, or the current norm, where the UI maps onto the structure of the programmer’s code) is desirable, even though it can’t be attained unless the end user writes it themselves. (The goal here is to minimize the artificial friction users encounter in making their computing environment a natural reflection of their mind.)

                                                                1. 3

                                                                  Thanks, that’s a very interesting read and I understand better now.

                                                                  I have defined the ‘widget’ object as having a position, size, and bitmap.

                                                                  Does this mean everything will mainly be raster (as opposed to vector). What about other visual transformations? How (un)important are they? Or maybe my question actually is whether composition will mainly be on functionality or visual appearance as well?

                                                                  (I don’t plan to make non-widget objects impossible, but it’s bad form, since they’re going to be harder to find and change.)

                                                                  Hmm…what about overlays for viewing and editing widgets, like corners, text cursors and bounding boxes? I guess they could all be their own separate overlay widgets.

                                                                  Unfortunately, I haven’t gotten to the point of having anything render.

                                                                  That might not be an issue for prototyping, except possibly for testing speed. To try it out, you might have a mock “render” loop that just prints out everything that should be drawn (and where) for that frame.

                                                                  And some random questions about the implementation you linked too.

                                                                  I’m a bit concerned about calculateBbox and widgetLint’s potential resource (time) consumption.

                                                                  For setParent, shouldn’t the current parent, if there is one, also remove “self” as a child?

                                                                  What about user input? Do they just get passed up the tree from the child? What if you want to globally change keybindings afterwards? Does the prototype-object system help with this or makes it harder?

                                                                  A lot of what I’ve written here is probably too early to think about (although something like vector vs raster is pretty hard to change later on).

                                                                  I definitely agree with Squeak not being inviting enough. I haven’t used Tk all that much despite having made this but did find issues with its internal model a few times.

                                                                  I guess if Io is simple enough, it can even be part of the thing that’s to be rewritten/customized.

                                                                  I hope you go much further with these ideas!

                                                                  1. 4

                                                                    Does this mean everything will mainly be raster

                                                                    In the particular system I’m writing, I’m only supporting raster, because I am lazy. I’m adopting the model from Menuet, because rendering becomes trivial: clear buffer, blit existing bitmaps onto buffer at given positions in Z order, flip. (I have the necessary information to later optimize out totally occuluded stuff, or keep a dirty bit in order to determine what parts of the display should be updated, if performance becomes a problem, but since unlike the Applet system in Java the user code doesn’t re-draw on visibility, it actually will be less of an issue, with the downside being the occasional half-drawn widget being rendered for one frame.)

                                                                    whether composition will mainly be on functionality

                                                                    Composition is performed through message-passing. My goal is to provide the GUI equivalent of unix pipes.

                                                                    I guess they could all be their own separate overlay widgets.

                                                                    Yeah. It cleans the system up conceptually if there are no “special” classes of widgets that don’t count as widgets.

                                                                    To try it out, you might have a mock “render” loop that just prints out everything that should be drawn (and where) for that frame.

                                                                    The stage I’m at in development is concerned with rewriting Io so that it can actually run my program.

                                                                    I’m a bit concerned about calculateBbox and widgetLint’s potential resource (time) consumption.

                                                                    This is a proof of concept system & so I’m not concerned with performance at all. If major performance problems appear, I’ll look at some of the low-hanging fruit (like the Io intepreter re-parsing and re-tokenizing the entire expression over and over) first.

                                                                    What about user input? Do they just get passed up the tree from the child?

                                                                    Yes, from the focused child.

                                                                    What if you want to globally change keybindings afterwards?

                                                                    That’s not a facility I’m supporting, though generally speaking you’d only be creating key bindings for widgets that use them, so having layers of conflicting triggers would be bad form.

                                                                    Does the prototype-object system help with this or makes it harder?

                                                                    It depends on how you use it. The inheritance hierarchy is distinct from the widget-packing hierarchy, and both are used for message-handling (of which input handling is a part). It’s definitely possible to make a huge mess with this.

                                                                    A lot of what I’ve written here is probably too early to think about

                                                                    I actually made a lot of these decisions before I started writing any code, so you’re absolutely right to ask these questions. Because I expect to have between zero and one users, performance isn’t a priority, and I’m ignoring it in favor of making everything structurally & conceptually straightforward.

                                                                    if Io is simple enough, it can even be part of the thing that’s to be rewritten/customized.

                                                                    Io is pretty bare-bones and I’m looking to write as much as possible in it, as opposed to Tk, where complex things like the text entry widget are basically exposed from FFI. I probably won’t expose the actual parser to be rewritten, though.

                                                                    1. 1

                                                                      I’m curious – has there been any development on this since? Would you be interested in any volunteer work? I’ve got some time to kill, as I don’t start work for a while. PM me if you’re interested.

                                                                      1. 2

                                                                        I haven’t done any work on this for a while, no. I’d love somebody to pick up where I left off, or work on their own projects inspired by this.

                                                                        I’m likely to work on it very intermittently. I have a lot of side projects, and with the current load at work, I get a chance to commit to one maybe once every few weeks. This one is complicated, since I’m writing a whole programming language interpreter for it, so it’ll probably get less love than my simpler projects.

                                                                        1. 2

                                                                          I’ve started poking at it in https://github.com/tekknolagi/kamui (having filtered out the subfolder from your misc repository). I’m very new to Io and this idea, so we’ll see if I get anywhere at all.

                                                              1. 2

                                                                I’m going to give suggestions in a different direction and hope it helps anyways. I haven’t tried these Windows newer than 7 or in some case 8.

                                                                1. I don’t know if “Windows desktop workspace” is the same as a remote desktop connection. If yes, you can use a linux client like rdesktop to connect to it and run it in a window from your host (Mac or Linux). So you can work in your host and paste to/from Windows whose remote desktop’s window. At least for rdesktop, I remember clipboard across the two “just worked”.

                                                                2. If that’s not an option, at any time, Super+r (for run), wait a bit, type “magnify”, press enter. Click the + if needed.

                                                                You may need something with desktop effects (I forget the actual name, Aero?) enabled to work

                                                                This worked at least until Win 8. Same thing with “magnify” replaced by “narrator”.

                                                                1. You can copy the text from dialog boxes by selecting the box and pressing Ctrl-C.

                                                                2. From a regular Windows command line, you can “mark” and then copy/paste to them. “Mark” is one of the options if you click on the icon on the top left. It still annoying to use.

                                                                3. Even though it does less, I like Minimalist GNU for Windows better than Cygwin. (And possibly easier to setup with less permissions but its been a while.) I’d probably try to use ssh from the command line within MinGW.

                                                                1. 1

                                                                  As I miss low-level Ring0 debuggers like SoftIce, I clicked this title way too fast.

                                                                  Now I’m disappointed. Are we going to see any other software to interrupt your machine (and inspect/control however you like) in the future?

                                                                  1. 1

                                                                    As I miss low-level Ring0 debuggers like SoftIce,

                                                                    It was hard to tell from the video what exactly was going on. The wikipedia page two links away describes it as

                                                                    SoftICE is a kernel mode debugger for Microsoft Windows up to Windows XP. Crucially, it is designed to run underneath Windows such that the operating system is unaware of its presence. Unlike an application debugger, SoftICE is capable of suspending all operations in Windows when instructed. For driver debugging this is critical due to how hardware is accessed and the kernel of the operating system functions. Because of its low-level capabilities, SoftICE is also popular as a software cracking tool.

                                                                    So, one main point for my project is to have a live editor for the process/program being “debugged”. I don’t know how machine-level inspection and control would help here.

                                                                    Are we going to see any other software to interrupt your machine (and inspect/control however you like) in the future?

                                                                    What do you mean by any other software? The idea was to make a debugger from scratch from syscalls and up (well, almost since a wrapper is used).

                                                                    If instead you mean run multiple debuggers at once, currently ptrace prevent more than one from attaching to one process. I did think of attaching a debugger to the debugger itself though.

                                                                    From your experience, is it quick to make machine level interrupts and inspectors? And then to get the desired macroscopic effects? (For me, assembly was already borderline

                                                                  1. 3

                                                                    I wasn’t aware of this project. Thanks for posting it here! (Also, welcome back.)

                                                                    I will be writing weekly or semiweekly blog posts summarizing the progress since the last update.

                                                                    Aside from the progress summary blog posts, I will try to distill what I cover on stream into standalone articles.

                                                                    Did they manage to do this in the end? That would be the most interesting part to me.

                                                                    The closest thing I’ve found is this but that’s more of a list/index than articles.