1. 4

    In order to add a logout feature we had to add four more edges. This quickly gets untenable.

    I didn’t understand why this is really an issue though. It feels like increasing the size of a for-loop to me. I do like the nested representation and use nesting quite a bit.

    When you started drawing automata, I thought you were going to go the linear temporal logic or Computation tree logic route.

    This is from memory and Wikipedia and I could be getting it very wrong:

    is it possible to reach answers report without going through the students report?

    exists(always(not Student) and eventually(Answers))

    We can also ask it to give us an example where someone logs out and logs back in again:

    exists(eventually(Logout and eventually(Login)))

    (Not really an example.)

    finding deadlocks

    for all states s, always(next(Summary) implies eventually(s)) and always(next(s) implies eventually(Summary))

    “You can reach everything from Summary and you can reach Summary from everything.”

    1. 2

      I didn’t understand why this is really an issue though. It feels like increasing the size of a for-loop to me.

      Adding additional edges to an FSM makes it visually harder to parse. Since the point of an FSM is to convey information to humans, making it harder to parse makes it less useful as a specification.

      When you started drawing automata, I thought you were going to go the linear temporal logic or Computation tree logic route.

      My main spec language is TLA+, which can express the more sophisticated properties quite easily. The problem is that it doesn’t handle subtyping at all, when representing nested states as subtypes is a really elegant way to encode the spec.

      The reason Alloy can’t handle things like deadlocks or liveness is because it’s carefully designed to be as user-friendly as possible. Among other things, that means checking specs really really fast, which means using a SAT solver, which means no infinite traces. We’ve been thinking of changing it to use an SMT solver, but right now the costs are just too high to make switching worth it.

      1. 1

        Oh, right. I forgot Alloy isn’t TLA+. Thanks.

    1. 5

      Last week

      • Fiddled with different ways of attaching to processes and viewing their states.
      • Some other technical stuff that went well

      This was for the low level debugger I’m trying to make.

      So, from what I’ve read and seen, tools that attach and inspect other process tend to just use gdb under the hood. I was hoping for a more minimal debugger to read and copy.

      lldb almost does what I need because of its existing external Python interface but documentation for writing a stand-alone tool (started from outside the debugger rather than inside) is scattered. I haven’t managed to make it single step.

      Using raw ptrace and trying to read the right memory locations seems difficult because of things like address randomization. And getting more information involves working with even more memory mapping and other conventions.

      I wish all these conventions were written in some machine readable language agnostic way so I don’t have to human-read each one and try to implement it. Right now this is all implicit in the source code of something like gdb. This is a lot of extra complexity which has nothing to do with what I’m actually trying to accomplish.

      The raw ptrace approach would also likely only work for Linux. And possibly strong tied to C or assembly.

      The problem with the latter is that eventually I will want to do this to interpreters written in C or even interpreters written in interpreters written in C. Seems like even more incidental complexity in that way.

      An alternative is to log everything and have a much fancier log viewer after the fact. This way the debugged program only need to emit the right things to a file or stdout. But this loses the possibility for any interactivity.

      Plus, all of this would only be worth it if I can get some state visualization customizable to that specific program (because usually it will be an interpreter).

      Other questions: How to avoid duplicating the work when performing operations from “inside the program” and from “outside” through the eventual debugger?

      Other ideas: Try to do this with a simpler toy language/system to get an idea of how well using such a workflow would work in the first place.

      Some references

      This week

      • Well, now that I have a better idea of how deep this rabbit hole is, I need to decide what to do. Deciding is much harder than programming…
      • Or maybe I should do one of the other thousand things I want to and have this bit of indecision linger some more.
      1. 5

        I wrote a very simple PoC debugger in Rust if you are interested in the very basics: https://github.com/levex/debugger-talk

        It uses ptrace(2) under the hood, as you would expect.

        1. 1

          Thanks! I’ve a had a look at your slide and skimmed some of your code (don’t have Rust installed or running would be the first thing I’d do).

          I see that you’re setting breakpoints by address. How do you figure out the address at which you want to set a breakpoint though?

          How long did it take to make this? And can you comment on how hard it would be to continue from this point on? For example reading C variables and arrays? Or getting line numbers from the call stack?

          1. 2

            Hey, sorry for the late reply!

            In the talk I was setting breakpoints by address indeed. This is because the talked focused on the lower-level parts To translate line numbers into addresses and vice-versa you need access to the “debug information”. This is usually stored in the executable (as decribed by the DWARF file format). There are libraries that can help you with this (just as the disassembly is done by an excellent library instead of my own code).

            This project took about a week of preparation and work. I was familiar with the underlying concepts, however Rust and its ecosystem was a new frontier for me.

            Reading C variables is already done :-), reading arrays is just a matter of a new command and reading variables sequentially.

            1. 1

              Thanks for coming back to answer! Thanks to examples from yourself and others I did get some stuff working (at least on the examples I tried) like breakpoint setting/clearing, variable read/write and simple function calls.

              Some things from the standards/formats are still unclear, like why I only need to add the start of the memory region extracted from /proc/pid/maps if its not 0x400000.

              This project took about a week of preparation and work. I was familiar with the underlying concepts, however Rust and its ecosystem was a new frontier for me.

              A week doesn’t sound too bad. Unfortunately, I’m in the opposite situation using a familiar system to do something unfamiliar.

              1. 2

                I think that may have to do with whether the executable you are “tracing” is a PIE (Position-Independent Executable) or not.

                Good luck with your project, learning how debuggers work by writing a simple one teaches you a lot.

            2. 2

              For C/assembly (and I’ll assume a modern Unix system) you’ll need to read up on ELF (object and executable formats) and DWARF (debugging records in an ELF file) that contain all that information. You might also want to look into the GDB remote serial protocol (I know it exists, but I haven’t looked much into it).

              1. 1

                Well, I got some addresses out of nm ./name-of-executable but can’t peek at those directly. Probably need an offset of some sort?

                There’s also dwarfdump I haven’t tried yet. I’ll worry about how to get this info from inside my tracer a bit later.

                Edit: Nevermind, it might have just been the library I’m using. Seems like I don’t need an offset at all.

                1. 2

                  I might have missed some other post, but is there a bigger writeup on this project of yours? As to the specifics of digging up such information, take a look at ECFS - https://github.com/elfmaster/ecfs

                  1. 1

                    I might have missed some other post, but is there a bigger writeup on this project of yours?

                    I’m afraid not, at least for the debugger subproject. This is the context. The debugger would fit in two ways:

                    • Since I have a GUI maker, I can try to use it to make a graphical debugger. (Ideally, allowing custom visualizations created for each new debugging task.)
                    • A debugger/editor would be useful for making and editing [Flpc]((github.com/asrp/flpc) or similar. I want to be able to quickly customize the debugger to also be usable as an external Flpc debugger (instead of just a C debugger). In fact, it’d be nice if I could evolve the debugger and target (=interpreter) simultaneously.

                    Although I’m mostly thinking of using it for the earlier stages of development. Even though I should already be past that stage, if I can (re)make that quickly, I’ll be more inclined to try out major architectural changes. And also add more functionality in C more easily.

                    Ideally, the debugger would also be an editor (write a few instructions, set SIGTRAP, run, write a few more instructions, etc; write some other values to memory here and there). But maybe this is much more trouble than its worth.

                    Your senseye program might be relevant depending on how customizable (or live customizable) the UI is. The stack on which its built is completely unknown to me. Do you have videos/posts where you use it to debug and/or find some particular piece of information?

                    As to the specifics of digging up such information, take a look at ECFS - https://github.com/elfmaster/ecfs

                    I have to say, this looks really cool. Although in my case, I’m expecting cooperation from the target being debugged.

                    Hopefully I will remember this link if I need something like that later on.

                    1. 2

                      I have to say, this looks really cool. Although in my case, I’m expecting cooperation from the target being debugged.

                      My recommendation, coolness aside, for the ECFS part is that Ryan is pretty darn good with the ugly details of ELF and his code and texts are valuable sources of information on otherwise undocumented quirks.

                      Your senseye program might be relevant depending on how customizable (or live customizable) the UI is. The stack on which its built is completely unknown to me. Do you have videos/posts where you use it to debug and/or find some particular piece of information?

                      I think the only public trace of that is https://arcan-fe.com/2015/05/24/digging-for-pixels/ but it only uses a fraction of the features. The cases I use it for on about a weekly basis touch upon materials that are NDAd.

                      I have a blogpost coming up on how the full stack itself map into debugging and what the full stack is building towards, but the short short (yet long, sorry for that, the best I could do at the moment) version:

                      Ingredients:

                      Arcan is a display server - a poor word for output control, rendering and desktop IPC subsystem. The IPC subsystem is referred to as SHMIF. It also comes with a mid level client API: TUI which roughly correlates to ncurses, but with more desktop:y featureset and sidesteps terminal protocols for better window manager integration.

                      The SHMIF IPC part that is similar to a ‘Window’ in X is referred to as a segment. It is a typed container comprised of one big block (video frame), a number of small chunked blocks (audio frames), two ring buffers as input/output queue that carry events and file descriptors.

                      Durden act a window manager (Meta-UI).This mostly means input mapping, configuration tracking, interactive data routing and window layouting.

                      Senseye comes in three parts. The data providers, sensors, that have some means of sampling with basic statistics (memory, file, ..) which gets forwarded over SHMIF to Durden. The second part is analysis and visualization scripts built on the scripting API in Arcan. Lastly there are translators that are one-off parsers that take some incoming data from SHMIF, parses it and renders some human- useful human- level output, optionally annotated with parsing state metadata.

                      Recipe:

                      A client gets a segment on connection, and can request additional ones. But the more interesting scenario is that the WM (durden in this case) can push a segment as a means of saying ‘take this, I want you to do something with it’ and the type is a mapping to whatever UI policy that the WM cares about.

                      One such type is Debug. If a client maps this segment, it is expected to populate it with whatever debugging/troubleshooting information that the developer deemed relevant. This is the cooperative stage, it can be activated and deactivated at runtime without messing with STDERR and we can stop with the printf() crap.

                      The thing that ties it all together - if a client doesn’t map a segment that was pushed on it, because it doesn’t want to or already have one, the shmif-api library can sneakily map it and do something with it instead. Like provide a default debug interface preparing the process to attach a debugger, or activate one of those senseye sensors, or …

                      Hierarchical dynamic debugging, both cooperative and non-cooperative, bootstrapped by the display server connection - retaining chain of trust without a sudo ptrace side channel.

                      Here’s a quick PoC recording: https://youtu.be/yBWeQRMvsPc where a terminal emulator (written using TUI) exposes state machine and parsing errors when it receives a “pushed” debug window.

                      So what I’m looking into right now is writing the “fallback” debug interface, with some nice basics, like stderr redirect, file descriptor interception and buffer editing, and a TUI for lldb to go with it ;-)

                      The long term goal for all this is “every byte explained”, be able to take something large (web browser or so) and have the tools to sample, analyse, visualise and intercept everything - show that the executing runtime is much more interesting than trivial artefacts like source code.

                      1. 1

                        Thanks! After reading this reply, I’ve skimmed your lastest post submitted here and on HN. I’ve added it to my reading list to considered more carefully later.

                        I don’t fully understand everything yet but get the gist of it for a number of pieces.

                        I think the only public trace of that is https://arcan-fe.com/2015/05/24/digging-for-pixels/ but it only uses a fraction of the features.

                        Thanks, this gives me a better understanding. I wouldn’t minding seeing more examples like this, even if contrived.

                        In my case I’m not (usually) manipulating (literal) images or video/audio streams though. Do you think your project would be very helpful for program state and execution visualization? I’m thinking of something like Online Python Tutor. (Its sources is available but unfortunately everything is mixed together and its not easy to just extract the visualization portion. Plus, I need it to be more extensible.)

                        For example, could you make it so that you could manually view the result for a given user-input width, then display the edges found (either overlayed or separately) and finally after playing around with it a bit (and possibly other objective functions than edges), automatically find the best width as show in the video? (And would this be something that’s easy to do?) Basically, a more interactive workflow.

                        The thing that ties it all together - if a client doesn’t map a segment that was pushed on it, because it doesn’t want to or already have one, the shmif-api library can sneakily map it and do something with it instead.

                        Maybe this is what you already meant here and by your “fallback debug interface” but how about having a separate process for “sneaky mapping”? So SHMIF remains a “purer” IPC but you can an extra process in the pipeline to do this kind of mapping. (And some separate default/automation can be toggled to have it happen automatically.)

                        Hierarchical dynamic debugging, both cooperative and non-cooperative, bootstrapped by the display server connection - retaining chain of trust without a sudo ptrace side channel.

                        Here’s a quick PoC recording: https://youtu.be/yBWeQRMvsPc where a terminal emulator (written using TUI) exposes state machine and parsing errors when it receives a “pushed” debug window.

                        Very nice! Assuming I understood correctly, this takes care of the extraction (or in your architecture, push) portion of the debugging

                        1. 3

                          Just poke me if you need further clarification.

                          For example, could you make it so that you could manually view the result for a given user-input width, then display the edges found (either overlayed or separately) and finally after playing around with it a bit (and possibly other objective functions than edges), automatically find the best width as show in the video? (And would this be something that’s easy to do?) Basically, a more interactive workflow.

                          The real tool is highly interactive, it’s the basic mode of operation, it’s just the UI that sucks and that’s why it’s being replaced with Durden that’s been my desktop for a while now. This video shows a more interactive side: https://www.youtube.com/watch?v=WBsv9IJpkDw Including live sampling of memory pages (somewhere around 3 minutes in).

                          Maybe this is what you already meant here and by your “fallback debug interface” but how about having a separate process for “sneaky mapping”? So SHMIF remains a “purer” IPC but you can an extra process in the pipeline to do this kind of mapping. (And some separate default/automation can be toggled to have it happen automatically.)

                          It needs both, I have a big bag of tricks for the ‘in process’ part, and with YAMA and other restrictions on ptrace these days the process needs some massage to be ‘external debugger’ ready. Though some default of “immediately do this” will likely be possible.

                          I’ve so far just thought about it interactively with the sortof goal that it should be, at most, 2-3 keypresses from having a window selected to be digging around inside it’s related process no matter what you want to measure or observe. https://github.com/letoram/arcan/blob/master/src/shmif/arcan_shmif_debugif.c ) not finished by any stretch binds the debug window to the TUI API and will present a menu.

                          Assuming I understood correctly, this takes care of the extraction (or in your architecture, push) portion of the debugging

                          Exactly.

                          1. 2

                            Thanks. So I looked a bit more into this.

                            I think the most interesting part for me at the moment is the disassembly.

                            I tried to build it just to see. I eventually followed these instructions but can’t find any Senseye related commands in any menu in Durden (global or target).

                            I think I managed to build senseye/senses correctly.

                            Nothing obvious stands out in tools. I tried both symlinks

                            /path/to/durden/durden/tools/senseye/senseye
                            /path/to/durden/durden/tools/senseye/senseye.lua
                            

                            and

                            /path/to/durden/durden/tools/senseye
                            /path/to/durden/durden/tools/senseye.lua
                            

                            Here are some other notes on the build process

                            Libdrm

                            On my system, the include -I/usr/include/libdrm and linker flag -ldrm are needed. I don’t know cmake so don’t know where to add them. (I manually edited and ran the commands make VERBOSE=1 was running to get around this.)

                            I had to replace some CODEC_* with AV_CODEC_*

                            Durden

                            Initially Durden without -p /path/to/resources would not start saying some things are broken. I can’t reproduce it anymore.

                            Senseye
                            cmake -DARCAN_SOURCE_DIR=/path/to/src ../senses
                            

                            complains about ARCAN_TUI_INCLUDE_DIR and ARCAN_TUI_LIBRARY being not found:

                            Make Error: The following variables are used in this project, but they are set to NOTFOUND.
                            Please set them or make sure they are set and tested correctly in the CMake files:
                            ARCAN_TUI_INCLUDE_DIR
                            
                            Capstone

                            I eventually installed Arcan instead of just having it built and reached this error

                            No rule to make target 'capstone/lib/libcapstone.a', needed by 'xlt_capstone'.
                            

                            I symlinked capstone/lib64 to capstone/lib to get around this.

                            Odd crashes

                            Sometimes, Durden crashed (or at least exited without notice) like when I tried changing resolution from inside.

                            Here’s an example:

                            Improper API use from Lua script:
                            	target_disphint(798, -2147483648), display dimensions must be >= 0
                            stack traceback:
                            	[C]: in function 'target_displayhint'
                            	/path/to/durden/durden/menus/global/open.lua:80: in function </path/to/durden/durden/menus/global/open.lua:65>
                            
                            
                            Handing over to recovery script (or shutdown if none present).
                            Lua VM failed with no fallback defined, (see -b arg).
                            
                            Debug window

                            I did get target->video->advanced->debug window to run though.

                            1. 2

                              I’d give it about two weeks before running senseye as a Durden extension is in a usable shape (with most, but not all features from the original demos).

                              A CMake FYI - normally you can patch the CMakeCache.txt and just make. Weird that it doesn’t find the header though, src/platform/cmake/FindGBMKMS.cmake quite explicitly looks there, hmm…

                              The old videos represent the state where senseye could run standalone and did its own window management. For running senseye in the state it was before I started breaking/refactoring things the setup is a bit different and you won’t need durden at all. Just tested this on OSX:

                              1. Revert to an old arcan build ( 0.5.2 tag) and senseye to the tag in the readme.
                              2. Build arcan with -DVIDEO_PLATFORM=sdl (so you can run inside your normal desktop) and -DNO_FSRV=On so the recent ffmpeg breakage doesn’t hit (the AV_CODEC stuff).
                              3. Build the senseye senses like normal, then arcan /path/to/senseye/senseye

                              Think I’ve found the scripting error, testing when I’m back home - thanks.

                              The default behavior on scripting error is to shutdown forcibly even if it could recover - in order to preserve state in the log output, the -b arguments lets you set a new app (or the same one) to switch and migrate any living clients to, arcan -b /path/to/durden /path/to/durden would recover “to itself”, surprisingly enough, this can be so fast that you don’t notice it has happened :-)

                              1. 1

                                Thanks, with these instructions I got it compiled and running. I had read the warning in senseye’s readme but forgot about it after compiling the other parts.

                                I’m still stumbling around a bit, though that’s what I intended to do.

                                So it looks like the default for sense_mem is to not interrupt the process. I’m guessing the intended method is to use ECFS to snapshot the process and view later. But I’m actually trying to live view and edit a process.

                                Is there a way to view/send things through the IPC?

                                From the wiki:

                                The delta distance feature is primarily useful for polling sources, like the mem-sense with a refresh clock. The screenshot below shows the alpha window picking up on a changing byte sequence that would be hard to spot with other settings.

                                Didn’t quite understand this example. Mem diff seems interesting in general.

                                For example, I have a program that changes a C variable’s value every second. Assuming we don’t go read the ELF header, how can senseye be used to find where that’s happening?

                                From another part of the wiki

                                and the distinct pattern in the point cloud hints that we are dealing with some ASCII text.

                                This could use some more explanation. How can you tell its ASCII from just a point cloud??

                                Minor questions/remark

                                Not urgent in any way

                                • Is there a way to start the process as a child so ./sense_mem needs less permissions?
                                • Is there a way to view registers?
                                Compiling

                                Compiling senseye without installing Arcan with cmake -DARCAN_SOURCE_DIR= still gives errors.

                                I think the first error was about undefined symbols that were in platform/platform.h (arcan_aobj_id and arcan_vobj_id).

                                I can try to get the actual error message again if that’s useful.

                                1. 2

                                  Thanks, with these instructions I got it compiled and running. I had read the warning in senseye’s readme but forgot about it after compiling the other parts. I’m still stumbling around a bit, though that’s what I intended to do.

                                  From the state you’re seeing it, it is very much a research project hacked together while waiting at airports :-) I’ve accumulated enough of a idea to distill it into something more practically thought together - but not there quite yet.

                                  Is there a way to view/send things through the IPC?

                                  At the time it was written, I had just started to play with that (if you see the presentation slides, that’s the fuzzing bit, the actual sending works very much like a clipboard paste operation), the features are in the IPC system now, not mapped into the sensors though.

                                  So it looks like the default for sense_mem is to not interrupt the process. I’m guessing the intended method is to use ECFS to snapshot the process and view later. But I’m actually trying to live view and edit a process.

                                  yeah, sense_mem was just getting the whole “what does it take to sample / observe process memory without poking it with ptrace etc. Those controls and some other techniques are intended to be bootstrapped via the whole ipc-system in the way I talked about earlier. That should kill the privilege problem as well.

                                  Didn’t quite understand this example. Mem diff seems interesting in general.

                                  The context menu for a data window should have a refresh clock option. If that’s activated, it will re-sample the current page and mark which bytes changed. Then the UI/shader for alpha window should show which bytes those are.

                                  For example, I have a program that changes a C variable’s value every second. Assuming we don’t go read the ELF header, how can senseye be used to find where that’s happening?

                                  The intended workflow was something like “dig around in memory, look at projections or use the other searching tools to find data of interest” -> attach translators -> get symbolic /metadata overview.

                                  and the distinct pattern in the point cloud hints that we are dealing with some ASCII text. This could use some more explanation. How can you tell its ASCII from just a point cloud??

                                  See the linked videos on “voyage of the reverse” and the recon 2014 video of “cantor dust”, i.e. a feedback loop of projections + training + experimentation. The translators was the tool intended to make the latter stage easier.

                              2. 1

                                I’d give it about two weeks before running senseye as a Durden extension is in a usable shape (with most, but not all features from the original demos).

                                A CMake FYI - normally you can patch the CMakeCache.txt and just make. Weird that it doesn’t find the header though, src/platform/cmake/FindGBMKMS.cmake quite explicitly looks there, hmm…

                                The old videos represent the state where senseye could run standalone and did its own window management. For running senseye in the state it was before I started breaking/refactoring things the setup is a bit different and you won’t need durden at all. Just tested this on OSX:

                                1. Revert to an old arcan build ( 0.5.2 tag) and senseye to the tag in the readme.
                                2. Build arcan with -DVIDEO_PLATFORM=sdl (so you can run inside your normal desktop) and -DNO_FSRV=On so the recent ffmpeg breakage doesn’t hit (the AV_CODEC stuff).
                                3. Build the senseye senses like normal, then arcan /path/to/senseye/senseye

                                Think I’ve found the scripting error, testing when I’m back home - thanks.

                                The default behavior on scripting error is to shutdown forcibly even if it could recover - in order to preserve state in the log output, the -b arguments lets you set a new app (or the same one) to switch and migrate any living clients to, arcan -b /path/to/durden /path/to/durden would recover “to itself”, surprisingly enough, this can be so fast that you don’t notice it has happened :-)

            3. 3

              If you are looking for references on debuggers then the book How Debuggers Work may be helpful.

            1. 4

              Last week

              However, the more I try to do, the more conventions, file formats and Linux-specific knowledge is needed. I think I’ll reduce the scope and not try to make something too general.

              One example was trying to make a function call and break right after the call returns. That is, interrupt the current execution to make that call. In assembly, this would just be

              call func_name
              int3
              

              The simplest way is to just write these instructions below the current program counter, run and revert the write. But that wouldn’t work for recursive calls and if we’re near the end of the function, maybe we might write over something else that matters during the call to func_name?

              So now we could try to simulate the call by crafting a stack frame by hand but the program still wouldn’t break if it finishes. We can reserve some new memory with mmap (syscall #9) and write our instructions there, out of the way of anything else. But the parameter to most calls are relative and only take a 32 bit address difference as argument instead of 64. So I don’t know how to make a call that’s far and the mmaped chunk can be in a different segment as func_name.

              I ended up just writing func_name’s address into a register (rax) and ran call rax instead of call func_name.

              Maybe I should post about this. But its likely to be a lot of things you shouldn’t do…

              This week

              • Try to add enough to the debugger so the beginning of flpc is easier to make (if recreated from scratch).

              I probably have to look into dynamically adding functions to C at some point. Dynamically adding to assembly isn’t so bad since I mostly just have to find some empty space to write those bytes.

              I don’t know if I should go the dlopen way or if that’s more conventions and formats than its worth.

              1. 6

                Noodling around in Haskell: https://github.com/singpolyma/raw-materials

                1. 2

                  Thanks for trying it out! I think a couple of your snippets make good examples of the difference between code and specifications, and what makes specifications particularly nasty to write.

                  The system is required to prevent two trains from occupying the same track segment at the same time.

                  newtype Track = Track [TrackSegment]
                  data Train = Train
                  data TrackSegment = SegmentTrain Train | SegmentEmpty
                  

                  This type makes it impossible to have two trains in the track segment at the same time. I don’t think that’s our goal of this spec! When I see “the system is required to prevent X”, I infer two things:

                  1. X is a possible state, in that it can happen
                  2. X is an invalid state, in that it shouldn’t happen

                  In other words, it is possible for two trains to be on the same track segment, but our system should guarantee that, if everything follows the specification, our system will never cause two trains to overlap.

                  I don’t think Haskell’s type system is powerful enough to specify this, but you can probably do it in dependent types. Whether that’s the appropriate “raw material” for this specification is another question.

                  (Another possible issue is that the current type spec prevents anything else from being on the tracks except a train, like people. But whether that’s an issue dependents on what it is we’re specifying.)

                  The vending machine can prevent a coin-insertion event, but only the customer can cause it.

                  allowCoinInsert :: Signal (Maybe CoinInsert) -> Signal VendingMachineState -> Signal (Maybe CoinInsert)
                  allowCoinInsert coinInsertS stateS = runMaybeT $ do
                  	CoinsNotBlocked <- lift stateS
                  MaybeT coinInsertS
                  

                  We have here that the vending machine can prevent a coin-insert event, but we haven’t actually specified that only the customer can cause it. For all we know the vending machine spec is also allowed to trigger a CoinInsert.

                  We could add vending machine code and not have any CoinInsert functions, but that doesn’t quite work, either. Upon seeing that, all we know is that the code doesn’t have a means for the vending machine to insert a coin. But we don’t know if that’s intended functionality (perhaps for the next sprint), unplanned-but-okay if it does happen, or expressly forbidden. Similarly, we don’t know if any other parts of the system, whether the customer, a bypasser, or the snack-filler, are allowed to insert coins.

                  When a button is pressed, the machine should turn the lamp on within 250 milliseconds.

                  -- Hmm... I could use `timeout` to *check* if this happened,
                  -- but I don't think I could write anything that would
                  -- *guarentee* it always did.  It's unclear what the goal of
                  -- the expression is.
                  

                  It means that if you press the button, the light goes on within 250 milliseconds. :P

                  This is an example of hard real-time specification: you specify that the system must do something within a given time range. If your system takes 255 ms to flip the light, then your system is wrong. Hard time specifications are something I have zero experience in, nor have I ever used any specification languages that can express those properties.

                  The pipes and switches of the package routing machine form a binary tree.

                  data Pipe = EndPipe | Pipe Switch Switch
                  data Switch = EndSwitch | Switch Pipe Pipe
                  

                  This is a good example of how types can be used as a specification tool, which is something I tend to dismiss too easily.

                  1. 2

                    Hard time specifications are something I have zero experience in, nor have I ever used any specification languages that can express those properties.

                    Always happy to fix a problem like that. I did save a lot of design and verification papers on hard-real-time systems. I can try to dig up some examples for you if you want. If you did, I’m curious if you want just the general stuff or also some examples of custom stuff. They do a lot of purpose-built formalisms in that niche. I’m still not sure if that’s a good idea or not since benefits of people being on same page often outweigh benefits of new formalisms.

                    1. 2
                      data Pipe = EndPipe | Pipe Switch Switch
                      data Switch = EndSwitch | Switch Pipe Pipe
                      

                      My Haskell is rather poor but wouldn’t this also let you make cycles? What happens if you put the same value for both Pipes of a Switch?

                      The pipes and switches of the package routing machine form a binary tree.

                      From this description, I thought something like this would be valid? But how can it be expressed?

                          S
                         / \
                        S   \
                       /     \
                      S       S
                             / \
                            S   \
                                 S
                      

                      From this diagram, I’m also thinking that having two child switches for a pipe would be invalid.

                      Although really, I’d ask for the statement to be clarified first. There are a few other questions here.

                  1. 1

                    Interesting. The syntax is simple enough to be added to other shells.

                    I wish all the examples also provided sample outputs of the command being run.

                    The “Compression benchmark” example has inputs to cat shown in an arbitrary(?) order which was really confusing for me. The actual order is

                    printf file printf wc printf xz->wc printf bzip2->wc printf gzip->wc
                    
                    1. 3

                      Last week

                      • Prepared some presentation slides and demo
                      • Fiddled with lldb. Realized I might want a very different design.

                      This week

                      • Think about what’s actually needed for a debugger. Try different things. Still want the debugger to be general and work externally.
                      • Think of ways to sustain my activities.
                      1. 10

                        I have not read the entire paper. Just the first 5 pages and the conclusion.

                        I’d like to point out that there are significantly different ways for creating computer programs and this analysis seem to implicitly assume that the “static way” is the only way.

                        In the “static way”,

                        1. the programmers write large chunks of code in an editor (sometimes with some tests written beforehand),
                        2. the code is sent to a compiler which help point out errors,
                        3. if it compiles, the programs runs and the programmer observes the output
                        4. the programmers makes updates as needed. Eventually adding other large chunks of code.

                        (There are of course many variantions on this. Please correct me for a more accurate description.)

                        In the more “dynamic way” of programming, a programmer is not only created in small pieces but potential new lines are tried out in the interpreter before deciding to add them or not. It is typically characterized by a short feedback loop.

                        1. the programmers write a statement or line,
                        2. this is sent to the interpreter,
                        3. the programmer observes the result,
                        4. make modifications and go back to 1 and/or add/keep some that line for the final program.

                        Usually, there are lot of (language and IDE) features to make the 1-3 loop as fast as possible.

                        In the Samlltalk IDEs, its possible to even run programs with mostly missing methods and Classes and fill them in at runtime (without ever restarting). To make such workflows possible, “superfluous” dynamic features are absolutely needed but will never be seen in individual codebases in the final product.

                        The main reason I think this article assumes a strong static view is that they never discuss this aspect of dynamic languages (in the pages I’ve read).

                        We started our analysis by looking at all 1850 software projects stored in Squeaksource in a snapshot taken in early 2010.

                        They don’t look at dynamic feature use during the program creation phase at all, just the final product.

                        One issue is that its always possible to program in a more static way in a dynamic language and IDE. And the see the lack of help from the compiler as a hinderance. In one never uses the dynamic features then it is easy to conclude they are a hinderance. But these features aren’t there solely for the purpose of being included in the final program.

                        They found that the Python program in their corpus made a heavier usage of dynamic features during their startup phase, but that many of them also used some of these features during their entire lifetime.

                        Of course, to perform any kind of study, some kind of assumptions need to be made. But they don’t say that this aspect is ignored to simplify the study.

                        There are also things that can be done which can improve a language on one of the static/dynamic axis without regression on the other, especially at the beginning.

                        As a final note, I’d like to mention one last use of dynamic features: to learn about a new module. I’ll usually read the high level readme of a new module and then immediately start playing with its objects in the interpreter without reading the documentation for individual classes and functions.

                        Reflexive methods allow me to quickly find out what’s available and try things purely based on their names. I’ll send mock inputs to functions to try and figure out what their inputs and outputs are, observe the results and update my hypothesis about what they do.

                        I find this gives me a much better and certain understanding of foreign modules and its a lot of fun to just poke around instead of just reading!

                        Now I’m wondering if I should make a screencast of this.

                        1. 9

                          I completely agree with your point here, and I had the same reaction to the paper. I don’t program in Smalltalk anymore, but when I did, one of my favorite demos was what I called “debugger-driven development.” I’d start by just writing some code that would be what I wanted an API to look like–e.g., for a fortune program, maybe Fortune loadAll randomFortune–then execute DoIt, bringing up the debugger (since that class doesn’t even exist)–and then I’d add recursively add the missing methods until the original line completed. This leverages a whole swath of metaobject and reflective capabilities that the final code wouldn’t need, but without which, the entire Smalltalk development experience would be neutered beyond recognition.

                          My other common demo was pretty different, but also did a great job using metaprogramming and reflective capabilities during development not needed in prod: I’d find a web API that was pretty cleanly designed and relevant to the audience I was speaking to. Next, I’d show how to write a kind of generic catch-all implementation via #doesNotUnderstand:, like what you might do in Ruby or Python. But then I’d modify it to generate the missing methods on-the-fly, and then (using a deliberately-made mistake in my first implementation) show how things like method modification timestamps and reflection could find the improperly-generated methods and regenerate them. (Sometimes, I’d even demo regenerating them as-needed with #perform: if I felt like it.) Finally, when we were done, I’d delete the #doesNotUndestand: implementation, customize a route or two, and be done. The final product didn’t use reflection, metaclasses, #doesNotUnderstand, or any of that kind of stuff, but the entire development process did. And unsurprisingly, a lot of the code for development did use things like self basicNew, foo perform: (a, b) asSymbol, and so on, even though the final wouldn’t.

                        1. 2

                          How does this compare to other menus if its keyboard driven (both menu opening and item selection).

                          What about compared to some regexp search through the menu items?

                          1. 4

                            One good thing that my horrible ActiveX pie menus did handle nicely was keyboard navigation, shown in the following demo.

                            Four and eight item pie menus work very well with the numeric keypad and arrow keys, as well as joysticks and d-pads. You can also move to the next/previous item around the circle with tab / shift-tab, and type the name of an item to jump to it directly, hit escape to cancel the whole menu tree, backspace to peel back one submenu layer, etc, just like normal menu keyboard navigation.

                            It’s great to be able to navigate the same memorable menu trees with many kinds of input devices!

                            Terrible Example: ActiveX Pie Menus

                            This demo of an old OLE/ActiveX implementation of pie menus in Internet Explorer shows the weakness and complexity of the traditional desktop GUI MFC based approach to creating and editing pie menus with typical property sheets, scrolling lists and tree editors. It also has a weird way of paging between large numbers of items in pages of eight or fewer items each, which I was never very happy with. And programming complex user interfaces with XML/C++/ActiveX/OLE/MFC/Win32 is hell on earth.

                            https://www.youtube.com/watch?v=nnC8x9x3Xag

                            1. 1

                              Did you do any experiments specifically with joystick input? Because mouse drivers are intended to produce actual cursor position deltas, even trackpoint-style mice produce more information than is used in a pie menu (and while warping can make up for overshooting, it can be confusing for the user unless done carefully). On the other hand, while Mass Effect never nested their pie menus, they essentially made use of one of the two controller sticks as a pure direction-input device with good radial resolution (better than a d-pad, which has only four directions & thus can defeat the point of a pie menu if there are more than 4 options).

                              Using a dual-stick video game controller for a pair of pie menus that are statically placed means simultaneous navigation of two menus can be fast. Using fans instead of pies and orienting the fan such that the fan center is the center of the previous selection and the fan’s options are arrayed perpendicular to the swipe direction used to select it could make it possible to navigate and back out of many layers of menu pretty easily without extra selection gestures (the one complaint I had about the pie menus in The Sims 1)

                              1. 3

                                No, I haven’t done any joystick experiments myself.

                                Before he started spreading FUD about marking menus, Bill Buxton wrote an excellent paper on input devices in 1983, called “Lexical and Pragmatic Considerations of Input Structures”, which categorizes input devices and how their unique properties map to the problem domain, and discusses a lot of important concepts like pragmatics, chunking and closure, device independence, taxonomy of devices, and the nulling problem:

                                https://www.billbuxton.com/lexical.html

                                lexical: issues having to do with spelling of tokens (i.e., the ordering of lexemes and the nature of the alphabet used - symbolic or iconic, for example).

                                pragmatic: issues of gesture, space and devices.

                                Figure 1: Taxonomy of Input Devices. https://www.billbuxton.com/lexical1.gif

                                Continuous manual input devices are categorized. The first order categorization is property sensed (rows) and number of dimensions (columns). Subrows distinguish between devices that have a mechanical intermediary (such as a stylus) between the hand and the sensing mechanism (indicated by “M”), and those which are touch sensitive (indicated by “T”). Subcolumns distinguish devices that use comparable motor control for their operation.

                                Since you mentioned fans, I should point out that Simon Schneegans, the same guy who did the excellent Gnome-Pie, also did these amazing spoke-like and fan-out pies for his bachelor’s thesis:

                                The Coral-Menu: https://vimeo.com/51072812

                                The Trace-Menu: https://vimeo.com/51073078

                                Those are great on so many levels!

                                1. 2

                                  Thanks! A lot of food for thought here

                                  1. 1

                                    You’re welcome! I’m glad you’re hungry for it. ;)

                                    Enjoy some pizza: https://medium.com/@donhopkins/the-story-of-sun-microsystems-pizzatool-2a7992b4c797

                                    And some more pie too: https://www.youtube.com/watch?v=Xj1LFYSO2Kw

                              2. 1

                                Thanks!

                                Ignoring the thing its built on, that’s an interesting demo. I like that expert gesture-like mode. How well does it works in the end? Too bad this video doesn’t show keyboard or joystick use (or I missed it having skipped most of the editor portion).

                                1. 2

                                  It was very snappy because it was directly calling the Win32 API to manipulate windows and handle events (well, going through some layers of OLE/MFC, but not as squishy as a web browser). So mouse-ahead was quite reliable since it wasn’t polling for events and letting some of them slip through the cracks when the computer slows down (i.e. always), like some systems do.

                                  That was the great thing about NeWS: it had perfect, air tight, synchronous event handling, and never dropped one event on the floor, or sent a keystroke to the wrong window.

                                  Even my modern MacBook Pro always gets overheated, slows down, and when I open a new tab I have to stand back, be patient, and wait for the vehicle to stop moving completely before embarking with the mouse and keyboard.

                                  On Medium, I keep typing cmd-f to search before the page has settled down, and the first few characters of my search string that I type after cmd-f get inserted into the document somewhere at random, then the rest go into the search field! So I have to type cmd-f, sit back, take a breath, wait for a while, then type what I want to search for. Emacs on a 300 baud terminal was better than that!

                                  But I really hurt for an easy to use drawing API like cairo or canvas (i.e. I’m an old PostScript programmer, and it’s a huge pain in the ass carefully selecting resources in and out of the win32 GDI context absolutely perfectly every time without making the slightest mistake or else GDI memory would overflow and Windows would freeze up and the reset button would not work so you had to remove the laptop’s battery to reboot).

                                  But then again it was even worse programming on a Mac without any memory protection whatsoever! Those old Macs tend to freeze up instantly whenever you made the slightest programming mistake, instead of slowly grinding to a halt while glitching out the screen like Windows. To their credit, the PowerPC Macs could crash really really really quickly! You instantly knew when to reach for the “programmer’s switch”.

                            1. 8

                              The proposed solutions provided in the article all swarm around the idea of finding a third-party source randomness that everyone can agree on. Almost all the proposed solutions on the reddit thread do the same. (Props to this person for walking to the beat of a different drummer.)

                              I think they (or we) can do better! But, I don’t know how, yet.

                              I think the solution should be cryptographic in nature… So, I’ll try to get close to the answer by clicking anything in Wikipedia’s Cryptography Portal and connected nodes and sharing anything that looks remotely related.

                              These look really promising:

                              These look … interesting? Hopefully unnecessary.

                              1. 5

                                What about this?

                                1. Each party generates a secret string
                                2. Each party publishes the hash of their string to the entire group
                                3. Once all hashes have been published, each party publishes their original string to the entire group
                                4. The random number is the hash of the concatenated strings

                                There’s nothing in this protocol enforcing that the secret strings be random, but I believe that it’s game-theoretically in each party’s interest to do so, so as to avoid e.g. dictionary attacks. I can’t see how any party gains anything by generating a string that is anything except truly random, ensuring a random hash of the concatenated strings.

                                Am I thinking about this correctly?

                                EDIT: Ah, I see, this is basically the “commitment scheme” idea mentioned in the Wikipedia article you posted. Cool!

                                1. 1

                                  I came up with a variant of this, but instead of strings, each person picks a number, which is then hashed, then the resulting number is the sum of the chosen numbers, mod 20, plus 1.

                                  Another thing you could do is send a message, and the millisecond it was received at, mod 20, plus 1, is your number. You would have to trust that the internet makes the timing random enough, and that you can trust your chat system, but usually you can.

                                2. 4

                                  They don’t need to agree on the source of randomness, it just needs to be reasonably beyond manipulation by dev1. Like @asrp says, stuff like Bitcoin IDs will work. You could also hash the fifty first characters of the first article of a newspaper published the day of. Just as long as you announce the method before the event happens, and that the data source is sufficiently out of your control, you’re good.

                                  1. 2

                                    It depends on what they mean by their constraint 2. If there’s no input from the other devs or any third party then the only remaining source of information is the first developer and so I think it cannot be done.

                                  1. 1

                                    I found the question pretty interesting but the discussion of the (hidden) solution too long (everything after 3 seem to be a rehash of 3).

                                    State a timestamp in the future (distant enough for the email to reach everyone before the timestamp) and select a random public event at that time (or the first one to occur right after it).

                                    I think this was stated many times on reddit with bitcoin hash id as the public event.

                                    Example: If its 9:10 now, write an email with “take the bitcoin id that appears after 9:25 and apply <transformation to get it to a random number between 1 and 20>” and send it now. Wait. Look at the ID.

                                    Edit: actually on Reddit a lot of solutions of this form were suggested like solar activity or stock prices.

                                    1. 3

                                      Instead of Bitcoin, solar activity, or stock prices, there also exists the NIST Randomness Beacon.

                                    1. 1

                                      Thanks for posting this. I’ve been meaning to look into scraping with a headless browsers for a while.

                                      I noticed a lot of time.sleep around. Is there no way to get a callback (or even polling) instead of this? It doesn’t seem to be used for throttling.

                                      There’s also quite a few globals around. Is that because of how you setup threads?

                                      1. 1

                                        It is indeed used for throttling. Usually, you use selenium because of the elements that are loaded with JS. In most cases, if you don’t give them enough time to load selenium won’t be able to find them. If the page is fully static BeautifulSoup is just HTML parser that will do the job without any need to sleep. This is why you’ll see so often the combo selenium + BeautifulSoup. While you develop this kind of scripts is useful to see what is done by selenium first and then make the whole thing headless.

                                        There are more globals simply to prevent the need for having to call the same function redundantly and make it return something that doesn’t make much sense.

                                      1. 5

                                        Last week

                                        This week

                                        • Prepare some presentation a bit
                                        • Should really get back to that low level (C/asm) debugger/editor

                                        Its tempting to try and add some self-hosted servers because of this thread.

                                        My biggest issue is that the bigger servers don’t usually explain what they are doing, even when its simple. So each one is a lot of reading to try and configure to have it do what I actually want it to.

                                        1. 5

                                          Last week

                                          • Posted Collaborative Software Trades. Seems like interest is too low for this to go anywhere, among other things.
                                          • Tried to look up some recent animation libraries. Found popmotion and others by the same author.
                                          • Cleaned up updates to guitktk from the week before (mainly a new SDL backend) and pushed them.

                                          This week

                                          • Maybe think up of projects for Collaborative Software Trades anyways and add them there.
                                          • Maybe summarize my findings about animation.
                                          • I want to keep working on the low level (for C/asm) visual program editor but might have to shelve it for a bit.
                                          1. 2

                                            Not sure I can help with the Collaborative Software Trades, but the guitktk reminded me that there’s more I want to do with textual-based representations of non-textual things, such as diagrams. Just like PlantUML, Markdown, et al, use text to represent some (or mostly) non-textual things, but in a way that’s very different from, say, using XML or JSON to describe a diagram or richly formatted page, I’d like to extend that idea to GUI layouts (e.g., JavaFX-based interfaces), where a textual almost-WYSIWYG gets you 80% of the way there, and the other 20% can be done with some other mechanism.

                                            I’m also doing the same for online quizzes/tests that’s uses Markdown-like syntax, but allows for embedded questions and grading.

                                            Anyway, thanks for the ideas…even if the “trades” never happen, it’s kinda neat to see what other things people want in this realm.

                                            1. 1

                                              I’ve not heard of JavaFX before. Does it work well?

                                              I actually haven’t thought about the layout portion all that much. It’d be interesting if a PlantUML-like thing described UI, although I’ve never had UI large enough that having it would have made that much of a difference though. I almost went the other way with mostly manual placement of everything. (Its uglier, but faster.)

                                              1. 1

                                                JavaFX is the replacement for the Java Swing GUI toolkit and it’s as good as any cross-platform GUI toolkit can be.

                                                One of the nice things about a PlantUML-like layout is that it’s easy to version and see diffs. Also, a big problem that I see in complex layouts is the nesting structure, so if it could capture 60-80% of the use cases, it’d help a lot.

                                                1. 1

                                                  Oh, I thought JavaFX was some special UI describing language.

                                                  I don’t know if we’re talking about the same thing but PlantUML is pretty declarative and uielem might be interesting to you. The same idea can be wrapped around other UI toolkits. However, it isn’t WYSIWYG (but I prefer it to, say, trying to manually draw ASCII diagrams).

                                          1. 1

                                            Is it just me or is the quality really bad? Or is that to be expected?

                                            The speed parameter behaves very unpredictably

                                            $ time -p ./sam -speed 100 'hello world'
                                            real 1.40
                                            $ time -p ./sam -speed 200 'hello world'
                                            real 2.40
                                            $ time -p ./sam -speed 280 'hello world'
                                            real 0.42
                                            
                                            1. 7

                                              Yes, that’s the quality I remember, which was amazing at the time. :-D Just in case we might be getting different results, here is what it’s supposed to sound like: https://matracas.org/tmp/sam-hello-world.wav

                                              If you put a full-stop at the end, the intonation sounds a bit more natural but also unenthusiastic.

                                              As for the speed parameter, that’s 8-bit overflow: the slowest is at 256, and if you roll it over to 257 it’ll speak so fast it’ll just be a blip.

                                              https://matracas.org/tmp/sam-lobsters-reply.mp3

                                              1. 1

                                                Thanks! That explains it. It not bother by the intonation, much more by the audio artifacts when it “screeches”.

                                                At least that problem doesn’t get worse at higher speeds (around 20) and is still understandable. I know this is old (but never had a C64 to experience it with) but couldn’t tell if its a porting bug or not. Maybe I could have worded my last comment better.

                                              2. 3

                                                It ran on a system with 1MHz CPU and 64KB of RAM during the year they were cranking out state-of-the-art experiences like this one. Relative to the time, great sound quality to hardware ratio. Today, not so much but that comparison is cheating. ;)

                                                1. 3

                                                  Looks like before Moore’s law really kicked in, you just had to do with what you have. And they did an amazing job with that hardware from the looks of it!

                                                  Though I was comparing it to eSpeak, which is still pretty small but I don’t know its CPU and memory footprint. (That page says it was originally for Acorn/RISC_OS computers but then relaxed those constraints.)

                                                  1. 1

                                                    That looks cool. Didn’t know about it. Thanks for the link!

                                                  2. 3

                                                    Interestingly a 1930s analog system sounds a lot smoother. Partly I think because it doesn’t have to drive digital audio output; you effectively get a very high “clock rate” with a specialized analog circuit. E.g. Wikipedia says this 1939 device is running 10 bandpass filters, which a C64 probably can’t do in realtime. Kind of interesting that you could do it with 1930s tech though, just not the same kind of tech. (The Voder also cheats compared to a TTS system by having a trained human key in the phonemes.)

                                                    1. 1

                                                      Super cool! Too bad it needs month of training to operate

                                                      After months of practice, a trained operator could produce recognizable speech.

                                                      1. 1

                                                        That was really neat! It was human controlled, though. I think that disqualifies it for automated synthesis. The fact that it was analog certainly gives it an advantage since analog doesn’t pause (clockless realtime), uses a fraction of the power, and operates on mediums closer to vocals. I’ll do a submission close to tuesday afternoon you’ll like on them.

                                                        1. 1

                                                          That submission is here.

                                                      2. 2

                                                        Totally expected. Go read the Wikipedia Article - this code was released in 1982.

                                                        Making no assumptions about the reader, were you born yet? :) In any case, hopefully that will give you some context.

                                                      1. 4

                                                        Last week

                                                        • Posted some small(er) projects in collaborative software trades anyways.
                                                        • Wrote most of a post on animation APIs. Still need to add some stuff and proofread it.
                                                        • Wrote some simple example using my current APIs.

                                                        This week

                                                        • Finish that animation post.
                                                        • Add some more parameters like opacity (hard for some backend) for guitktk and use them with animations.
                                                        • Probably some time consuming administrative stuff.
                                                        1. 1

                                                          Prompted by this ask

                                                          1. 5

                                                            By analogy to the statistics example, maybe the best answer is

                                                            • graph theory: graphs, directed graphs

                                                            but that’s hardly informative. I think the hard part when applying graph theory is recognizing that there is a graph in your problem in the first place and clearly identifying what the vertices and edges are (and whether they are directed). I guess you might throw “trees” and “directed acyclic graph” in the list? Because recognition imply more immediate simplification.

                                                            I’d be interested in more examples from you, especially if you’ve been collecting these.

                                                            A number of these examples look like extra perspectives on an existing situation though. And I think this one isn’t meant to be used quite in the same way. When graph theory is applicable, it should naturally suggest itself to you as you model the problem (instead of the other way around). I suppose if you miss the fact there is a graph at all, it might. Its closer to “a distribution” and “game theory” in nature.

                                                            1. 5

                                                              Interesting idea. But you’re missing a couple of major drawbacks:

                                                              1. Software is usually a liability rather than an asset. It can increase your attack surface. It can be a time sink to maintain over time. It can not quite do what you want and bleed energy. It can be actively malicious, insidiously stealing your data or bitcoin. (I’m seeing a lot more awareness of these issues after the security vulnerabilities and Cambridge Analytica of the last few years. For example, this article can be distilled down to, “stop using software to trust people, software will never be trustworthy enough.” Software’s benefits are usually residents of Mediocristan, but its drawbacks often live in Extremistan.)

                                                              2. Dependencies matter. You list as an advantage that “Unless the language or library used is part of the requirements, programmers proficient in different languages can still trade.” But you’re reading that fact from exactly the wrong direction. Since anything I’m unable to get running is by definition useless, and since installing languages and libraries is non-trivial in the most general case, platform dependencies will grow to become an essential part of requirements. An obvious example: any programs I distribute are useless to someone running Windows. Another example: If I don’t want to install Java, you either don’t use Java, or you need to come up with a way to distribute your program with Java in some sandboxed fashion. (We all know how successful such attempts have been.)

                                                              With these issues in mind, I have two suggestions.

                                                              Suggestion #1: Reduce the unit of exchange. If people exchanged programs that could be built in an hour or two rather than a week or two, the programs would be more obviously right, more generally useful and easier to specify, detect duplicates, probably rely on fewer dependencies, have smaller risks for the author, etc., etc.

                                                              If the expected project size were to go down I can contribute a couple of small scripts I’ve built that I find very useful:

                                                              • search: looks in a unix-like directory tree for files containing all of a list of terms. Basically a search-engine-like interface atop grep. Useful for maildirs, but since I started using it I’ve also started structuring other data sources as directories with one file per unit.
                                                              • accumulate: an add-on for search above that allows me to try multiple searches when looking for something specific, without getting duplicate results on successive searches.

                                                              (Hmm, as I look at these I notice a couple more drawbacks to your idea: 3. Many programs are demand-creating rather than demand-satisfying. People may not know they want them until they see them in action. 4. Programs often work in systems. Unix pipes. Or operating on a directory of files with one file per unit, as I do above. Or pipes operating on Yaml. Individual programs in these sets aren’t useful without the right context and environment.)

                                                              Suggestion #2: Cater to a more specific kind of person. I can imagine at least two axes to segment programmers by: time rich vs time poor and money rich vs money poor. So for example someone money rich but time poor and someone time rich but money poor would end up transacting the conventional job for money. Your current approach has benefits along both time and money axes, but the cost is the drawbacks enumerated above. For example, I’m not particularly time rich, but I know that a program I write will do just what I want, nothing I don’t want, be fairly parsimonious of dependencies, and set at least some sort of lower bound for secure practices. So if I have an idea for a program it’s better to wait and build it myself rather than try to optimize too hard for time. But maybe there’s some way to mitigate the drawbacks if you focus on just one of the quadrants?

                                                              1. 2

                                                                Thanks for your thought on this.

                                                                I think you are mainly thinking of software that will be used on servers open to the internet and when I wrote this, I was mainly thinking of desktop apps used to make things (which may then be put on Internet-facing servers in production, but not the desktop app itself).

                                                                But aren’t these general problems with using open source libraries in your projects (and not something specific to trades)?

                                                                1. For the potentially malicious versions, shouldn’t having the source code at least help quite a bit in that direction.

                                                                Depending on how bad it is, I’d argue that you didn’t actually get the software you wanted if it ends up being so much trouble. (And consequently shouldn’t be used despite another user having made it, which is unfortunate but avoids the downsides.)

                                                                1. I’d say this would all just go into the requirements. I’m hope there’s similar enough requirements that they would center around major platforms (that makes matches easier), which is still less restrictive than a specific language or library.

                                                                Maybe I just happen to make stuff that’s more cross platform by default.

                                                                Since anything I’m unable to get running is by definition useless,

                                                                See the “usage” pattern I described. In my case, not being able to run it would still lose a lot of value, isn’t useless.

                                                                Suggestion #1: I’m happy to try this version if there’s enough people.

                                                                Though I don’t think I have anything I could safely say can be made in 1-2 hours from scratch. Any random hiccup can take up the entire time. It also makes thinking about and writing out the specs a bit less worthwhile.

                                                                But I don’t know. “A weekend” wasn’t that thought out of a time length either. So I’d give this one a go if more people prefer it.

                                                                If the expected project size were to go down I can contribute a couple of small scripts I’ve built that I find very useful:

                                                                I hadn’t thought of just releasing existing source. But that works too! Really any way works, even finding an existing project (if licenced right).

                                                                You still have to add programs that you want. And the way I described it, it could be that someone else is assigned to make search and accumulate (which you already have) while you’re assigned something else from your list.

                                                                People may not know they want them until they see them in action.

                                                                I guess people can browse the existing list, although this doensn’t help see it in action. I was thinking more of programs that do not yet exist so you couldn’t see them in action anyways.

                                                                Suggestion #2 This is an interesting suggestion but I didn’t entirely follow your discussion.

                                                                How would focusing on only one quadrant (potentially) help with the drawbacks listed? Do you mean all the time poor, money right would tend to want the same kind of things?

                                                                Or do you mean some of the quadrants wouldn’t find the drawbacks as much?

                                                                (I’m in no way in disagreement, just didn’t follow the reasoning entirely.)

                                                                1. 2

                                                                  Yes, you’re right that my drawbacks apply as well to existing open source. But you’re proposing something new, something where people’s reputations aren’t involved yet the way they’re involved in conventional projects. And something where I have to precommit to using something that doesn’t exist yet. The competition isn’t existing open source projects. The competition is me making something myself.

                                                                  (I’m trying to attack these drawbacks for open source as well.)

                                                                  See the “usage” pattern I described. In my case, not being able to run it would still lose a lot of value, isn’t useless.

                                                                  I don’t follow. I don’t see the word “usage” anywhere in OP.

                                                                  Maybe I just happen to make stuff that’s more cross platform by default.

                                                                  What sort of stack do you use? How do you test on multiple platforms?

                                                                  1. 1

                                                                    Thanks again for your thoughts. I have to admit, I’m even more confused, which could be good since it can be some important blind spot I’ve missed.

                                                                    Yes, you’re right that my drawbacks apply as well to existing open source. But you’re proposing something new, something where people’s reputations aren’t involved yet [in] the way they’re involved in conventional projects.

                                                                    I don’t know if you mean reputation as in skill level or reputation as in not being a bad actor.

                                                                    For the first, we probably need a few round of this to see how it goes. Since trades are per project, the more skilled can put it less time. (There’s still the question of quality of the output. But I think that would have the same discussion as in the “Failure” section.)

                                                                    For the second, I’m thinking the projects are small enough to make an audit not as hard.

                                                                    I mean, again, this reputation problem happens for new accounts that show up on github with new projects entirely made on their own. You still have to decide to use them or not somehow. I’m not saying open source doesn’t have its drawbacks, but collaborative trades isn’t intended to solve any of those problems. Its intended to improve coordination.

                                                                    And something where I have to precommit to using something that doesn’t exist yet. The competition isn’t existing open source projects. The competition is me making something myself.

                                                                    But if you hire someone or even thinking of writing the software yourself, you’re still precommiting to a non-existent program (with just a spec). Basically, I didn’t understant the extra burden your are trying to highlight when comparing this to writing everything yourself.

                                                                    Let me try to explain the difference anyways (but I might be going in a completely wrong direction).

                                                                    Scenario: You want programs A, B, C, D (they are independent and all don’t exist).

                                                                    • Writing it all yourself: Takes time of writing all four of A, B, C and D. Full control over everything.
                                                                    • Collaborative software trade: Takes time of writing one of A, B, C or D (you don’t get to pick which one, the matcher assigns it to you). Full control over your assigned program, no/little control over the others. When you complete your assigned program, it has to be released as open source.

                                                                    The result in both cases is that you now have all four programs. (In the collaborative trade, so does everyone else.)

                                                                    As described, you might have to modify the three programs that you receive to better suit your needs, but the people who made them took your wishes into consideration to make editing them easier. This does means you should take others wishes into account when make your assign program.

                                                                    In the case where I would write and release the source for all programs anyways, the trade is much better. Even if all other programs received this way are really bad, I can just (re)write them myself as I was going to do anyways without the trade. But most likely, I can at least salvage some useful things out of them. In the more optimistic case, what I get is much better.

                                                                    I don’t follow. I don’t see the word “usage” anywhere in OP.

                                                                    Sorry I meant the stuff I typed in the other reply in here:

                                                                    In my case, on top of documentation (or even instead of it), I’d like to have enough instructions for rebuilding the whole thing from scratch. This means favouring simpler internals and fewer large list of cases.

                                                                    For servers, we know to have script and configuration setup to restore state when the machine reboot. This would be the human version of that.

                                                                    As a bonus, on top of describing what works, add a few remarks about what’d be lost with some “obvious” simplifications to the system.

                                                                    I really only need the program or library to prod it to check and understand the properties it has.

                                                                    What sort of stack do you use? How do you test on multiple platforms?

                                                                    Most of what I’ve written is in Python, using Tkinter if graphics are needed. I did not expect the programs to be particularly cross-platform (and so no such test was done!) but I know people have tried at least on Mac. It could also be that it doesn’t work for some platforms and no-one said anything yet.

                                                                    1. 3

                                                                      I understand that scenario from the original article. But let me try to restate my objections:

                                                                      Takes time of writing all four of A, B, C and D.

                                                                      I agree. But the time taken to write them may not be important for everyone.

                                                                      this reputation problem happens for new accounts that show up on github with new projects entirely made on their own. You still have to decide to use them or not somehow.

                                                                      True. But I look at hundreds of open source projects over the years, but start using only a few. Less than 1% yield. With trades that sort of strike rate would yield nothing.

                                                                      if you hire someone or even thinking of writing the software yourself, you’re still precommiting to a non-existent program (with just a spec).

                                                                      But you usually have more steering control over the process. Hmm, perhaps there’s some way to add that back into your idea? Maybe four of us start out building programs, but we meet once a week or something to chat about how it’s going. This would make it easier for us to take each other’s wishes into consideration, as you put it.

                                                                      Even if all other programs received this way are really bad, I can just (re)write them myself as I was going to do anyways without the trade. But most likely, I can at least salvage some useful things out of them.

                                                                      I think this is the crucial disagreement. See my strike rate with GitHub projects above. I don’t think salvaging useful things out of bad projects is either common or economic. It can take way longer to understand a bad project that does something in a bad way with 1% overlap with what you care about, than it would to just build the 1% yourself surrounded by a different 99%.


                                                                      As I write this, I thought of a couple more objections that I hadn’t aired yet:

                                                                      1. I don’t usually have a list of programs I wish existed. I usually get ideas one at a time. And then I forget the ones that I wasn’t very into, and obsess one at a time about the ones that I really care about. So I don’t know how common it is for programmers to have a list of programs A, B, C and D they wish to exist.

                                                                      2. Oftentimes my goal when programming isn’t mainly about the utilitarian value of the final program. It’s the wish to create an “object of conversation”, a locus for collaboration with another programmer. If I found 3 other people who wanted the same programs to exist as myself, my preferred approach would be to collaborate with them on all four programs, rather than to go our separate ways and build one program in isolation. Working together just seems more fun.


                                                                      These are my objections, and I care about conveying them to you just so you understand them. But I don’t mean for them to discourage you. My objections are all abstract, and if you don’t find them important that’s fine. Just keep going, and either you’ll understand them better, or you’ll discover that my theoretical objections don’t actually matter in practice, that there’s a segment of programmers who can use this even if it doesn’t include me.

                                                                      1. 1

                                                                        Thanks again. Now I understand.

                                                                        These two probably go together.

                                                                        I agree. But the time taken to write them may not be important for everyone.

                                                                        1. I don’t usually have a list of programs I wish existed. I usually get ideas one at a time. And then I forget the ones that I wasn’t very into, and obsess one at a time about the ones that I really care about. So I don’t know how common it is for programmers to have a list of programs A, B, C and D they wish to exist.

                                                                        I think this is indeed one of the differences.

                                                                        If there is a list and the list keeps growing in size or its members in scope, the total estimated time can exceed human life expectancy. Because of how bad we are at estimate, even 10x less estimated time is cutting it too close.

                                                                        Most things I want are editors and before they exist, so there also needs some time to actually use them afterward. :)

                                                                        Having said that, indeed if time taken to write the programs is not important then trades don’t offer anything even in the most optimistic outcome (and so there’d be no reason to participate in that case).

                                                                        True. But I look at hundreds of open source projects over the years, but start using only a few. Less than 1% yield. With trades that sort of strike rate would yield nothing.

                                                                        Well, most projects I’ve looked at don’t have anything near my requirements taken into account. Do you mean that of the ones that matches your description, you only get 1% useful? Or of the ones matching keyword search had 1% yield?

                                                                        But you usually have more steering control over the process. Hmm, perhaps there’s some way to add that back into your idea? Maybe four of us start out building programs, but we meet once a week or something to chat about how it’s going. This would make it easier for us to take each other’s wishes into consideration, as you put it.

                                                                        I’ve thought about something along those lines. Basically, the same thing can be done with incorporating other’s requirement at any threshold since its all symmetric. So for the control gained on the other projects, you’d lose control over your own.

                                                                        With it set this way, I think you’d lose flexibility on skill difference and potential asynchonicity (if we all just agree to have the projects done by some point, people can start/stop any time between now and then). Depending on how the meetings are conducted, you might also lose

                                                                        I thought finding people who want the same things and have comparable abilities (for whichever measure) would be much harder. I haven’t run this yet so I don’t know.

                                                                        In my particular case, I’d be happy to just take the weekly discussions as the output. Although I’d just want the final summary, not the entire thread.

                                                                        It’d also be one meeting per projects since the three other people’s list aren’t the same. I don’t quite see how the logistics of that would work out yet.

                                                                        I think this is the crucial disagreement. See my strike rate with GitHub projects above. I don’t think salvaging useful things out of bad projects is either common or economic. It can take way longer to understand a bad project that does something in a bad way with 1% overlap with what you care about, than it would to just build the 1% yourself surrounded by a different 99%.

                                                                        You’re right. Although I think the difference in expected rate might more central to the disagreement. Yes, 1% would not make it worthwhile. (My requirements includes instructions for rebuilding it so I could tell if its worth it to dig through more easily.)

                                                                        I don’t have a particular reason to think the initial rate would be higher. However, I’d like to argue that this system can maintain a high rate if it already has one, while something like Github cannot. Namely, the programmers with the lowest failure rates on Github are likely to move to much more non-open source while everyone else stays.

                                                                        Here is failures can be made to net nothing (not sure how to implement that yet) and success is rewards at the system’s current rate then I think its stable (provided the time savings are a good enough reason to stay, which isn’t always the case as discussed much earlier).

                                                                        Oftentimes my goal when programming isn’t mainly about the utilitarian value of the final program. It’s the wish to create an “object of conversation”, a locus for collaboration with another programmer. If I found 3 other people who wanted the same programs to exist as myself,

                                                                        See the above, but its most likely that each one only wants one program in common with you. (Maybe this doesn’t affect what follows but its 3 times as much communication.)

                                                                        my preferred approach would be to collaborate with them on all four programs, rather than to go our separate ways and build one program in isolation. Working together just seems more fun.

                                                                        That’s partly in the “Lack of discussion” section. I don’t know how well that would work with people with different languages and libraries preference. High level discussion might still work.

                                                                        Oftentimes my goal when programming isn’t mainly about the utilitarian value of the final program. It’s the wish to create an “object of conversation”, a locus for collaboration with another programmer. If I found 3 other people who wanted the same programs to exist as myself, my preferred approach would be to collaborate with them on all four programs, rather than to go our separate ways and build one program in isolation. Working together just seems more fun.

                                                                        I think this is another major difference in objectives. I most definitely want the final product first. I’d much rather discuss the post-mortems with them than follow along.

                                                                        (Although I’m wondering if there’s more similarity than difference here. You want to discuss and work with people. I want to be in the same state as if I had those discussions and collaborations, but reached more quickly.)

                                                                        These are my objections, and I care about conveying them to you just so you understand them. But I don’t mean for them to discourage you. My objections are all abstract, and if you don’t find them important that’s fine. Just keep going, and either you’ll understand them better, or you’ll discover that my theoretical objections don’t actually matter in practice, that there’s a segment of programmers who can use this even if it doesn’t include me.

                                                                        Thanks again for chiming in. This discussion is much appreciated!

                                                              1. 4

                                                                hm, I think a few examples would be really useful. My first thought is that people’s requirements for the same project would be too specific for anyone to be satisfied with another’s implementation.

                                                                That and software tends to take forever to develop - like sometimes years if someone doesn’t have a specific deadline and is writing it solo in their spare time. I know the idea was to have smaller projects than that, but people are likely to underestimate by orders of magnitude

                                                                1. 2

                                                                  I cut the samples out because I have a pretty strong (and maybe unusual) bias in what I want. And this idea is meant to be more general.

                                                                  For all programs, here’s the general property I want.

                                                                  In my case, on top of documentation (or even instead of it), I’d like to have enough instructions for rebuilding the whole thing from scratch. This means favouring simpler internals and fewer large list of cases.

                                                                  For servers, we know to have script and configuration setup to restore state when the machine reboot. This would be the human version of that.

                                                                  As a bonus, on top of describing what works, add a few remarks about what’d be lost with some “obvious” simplifications to the system.

                                                                  I really only need the program or library to prod it to check and understand the properties it has.

                                                                  This means the language and library used doesn’t matter as much although being able to read the source without too much experience is definitely a plus.

                                                                  As I’ve said in that post, I still need to think of projects of that scope to add myself. So what follows need more thought.

                                                                  • A PEG grammar for Python 3. I want to use my own parsing library but don’t mind if it uses a different library. It has to be a parsing expression grammar (PEG) using ordered choice though. The parser itself need not be written in Python, although that’d be nice. Bonus points for concision.

                                                                  Definitely not for this time frame:

                                                                  • An extensible low level editor for interactively create low level programs (in C and assembly). I want something more visual and better than a debugger (workflow-wise, not necessarily feature-wise). Let me set/view regions of memory while the program runs or is paused.

                                                                  The editor itself needs to be “simple” with relying on its extensibility (which also means making it “easy” to extend).

                                                                  For a more doable project, even a much smaller (in terms of complexity, not resource use) gdb or lldb would be a good start.

                                                                  That and software tends to take forever to develop - like sometimes years if someone doesn’t have a specific deadline and is writing it solo in their spare time.

                                                                  Indeed and I’m unfortunately well acquainted with this problem. But this phenomenon is worst with two pieces of software, or three, or more…

                                                                  In some sense, these trades are a solution to this problem although it does not face it head on.

                                                                1. 2

                                                                  in what way did the spam/moderation problem manifest? what suggests zulip would not have those problems? i hope these questions don’t sound too pointed. i would just like to understand the decision, lest i must make a similar one in the future.

                                                                  1. 2

                                                                    There were almost no real messages on the mailing list – and I would get messages from mailman/dreamhost every day about whether to let spam in or not. There were a couple other reasons I forgot to mention:

                                                                    • I’m on the toybox mailing list, which also uses mailman on Dreamhost. Rob Landley has complained that the archives go missing peridiocally. There is a hole in his mailing list history.
                                                                    • Also, Gmail users get auto-unsubscribed because of some spam issue. It might be Google’s fault or it might be because Dreamhost is used to send a lot of spam. Dreamhost is good at web hosting, but the mailing list hosting doesn’t appear to be well-maintained.
                                                                    • Some people had asked for an IRC channel, so this format might encourage more messages. There’s not enough traffic for IRC + mailing list, but there might be for a combined medium.
                                                                    1. 1

                                                                      interesting. one hitch i’ve found is that i can’t figure out how to view threads without logging in to zulip. the mailing list had web archives which could be viewed without giving my email to dropbox. i don’t browse ml archives too much in practice, but it’s useful if you want to refer someone to a thread in a ml they’re not subscribed to. i’m sure there’s a way to enable this on the zulip side, and that way you’ll get the same utility that the ml provided and more.

                                                                      1. 1

                                                                        and I would get messages from mailman/dreamhost every day about whether to let spam in or not.

                                                                        But how does Zulip prevent this? Is it just not popular enough to have spambots for it or do they have something specific for curation?

                                                                        I suspect a lot of general purpose host get blacklisted for e-mail. I hadn’t thought of “don’t send e-mail” as a potential solution to that, but it seems to work. :)

                                                                        1. 2

                                                                          I’m not sure, but the fact that they have logins and are centralized means they can probably do a better job than a random mailing list.

                                                                          Dreamhost also probably has enough data to do something good about spam, but they apparently don’t. It costs engineers and money to fight e-mail spam.

                                                                          Here’s a good link about spam: Modern Anti-Spam and E2E Crypto (2014)

                                                                          https://news.ycombinator.com/item?id=12997025

                                                                          In my experience, Google’s spam filters have gotten significantly worse lately. I’m on the busybox mailing list and Gmail routinely marks those messages as spam. And it routinely rejects e-mails I sent from a cron job. So they’re having problems with both false positives and negatives.

                                                                          1. 1

                                                                            Thanks for the link and your thoughts on this! I actually didn’t know where to start looking.