1. 8

Feel free to tell what you plan on doing this weekend and even ask for help or feedback.

Please keep in mind it’s more than OK to do nothing at all too!

  1.  

  2. 4

    I am writing a new type of “data studio”. Somewhat similar to Jupyter notebooks or Googles Data Studio. But more direct, faster and easier to use.

    On top is a textarea, where you can paste in your data. Immediately it is turned into an interactive scatterplot. Then you have all the tools and settings to transform the data and change the chart settings. Everything instantly changes the chart. Like changing it to a box plot. Or grouping the data by day/week/month. Or extrapolating incomplete data. Etc etc etc.

    I build this because I found no suitable software to analyze my selftracking log. Everything out there feels too slow, too cumbersome or too inflexible

    1. 1

      I’m interested in this!

    2. 3

      I’m continuing to turn a shell (dash) into a library so that I can directly integrate it into an experimental terminal.

      As part of this I’m moving all the global state into a struct and passing it around, and as part of that I’m currently boggling at dash’s use of errno - it’s redefining it for glibc and I don’t understand why. If any C experts are interesting in taking a look see the following lines of code: main.h:45-50, main.c:67-69, main.c:97-99. My current best guess is that it’s somehow related to the fact that it longjumps out of its interrupt handler.

      1. 2

        Sounds interesting, what’s the benefit of linking it with the terminal? I can think of a few things, but I’m curious what you want to do with it.


        On a cursory glance, it doesn’t look to me like the dash_errno thing does much. Like it’s not really used anywhere?

        I would be inclined to take it out and see if the shell still works :) Not sure if dash has tests, but I have a whole bunch of shell tests if you like …

        1. 2

          I’m still working out how to best explain this, but basically I’m trying to create a more cohesive/interactive interface between the user and the shell. Things like the ability to filter output based on command/stdout/stderr after the fact. Ability to inspect the output of subshells after the fact. Live monitoring of background jobs/generally better support for running things in the background while still using the shell. An easy interface for taking commands you’ve run and turning them into a shell script. Support for running a shell script “interactively” (similar to in a debugger with step-in/step-over/step-to + live editing), etc.

          This is largely inspired by the good parts of blue ocean (jenkins) and xl-release.

          The cost of this is that I’m going to abandon using a tty - and therefore compatibility with a lot of programs that expect a tty. Possibly I’ll eventually implement a way to run an individual program in a tty, but it’s always going to be a second class citizen/compatibility mode.


          errno within the shell is defined to be *dash_errno - and is used two or three dozen times. After looking at glibc source code as well I’m pretty sure it is just a historical artifact, and yes it appears to still work when removed.

          A bunch of tests will definitely be useful in the near future. Looking at your work - is this the best/current documentation on where to find and how to use your tests?

          1. 2

            OK cool I’ve been thinking of things along the same lines. Although I think there could be a way not to abandon the TTY. What about a GUI that HAS a terminal for commands, so then:

            • the shell itself talks to the GUI ?
            • the child processes talk to the terminal

            For example, someone suggested that the shell’s input line is always at the top and separate from the terminal.

            So that input line could get send to the shell, and it will need ahook for autocomplete. However child processes of the shell like ps need to run in a terminal.

            With the current architecture, the shell and the child process share a terminal, but if the shell were factored as a library, then that wouldn’t need to be the case.

            Some links here: Shell as an engine for TUI or GUI

            (it even links to a thread about debuggers)

            I haven’t seen XL release but it seems related. If you have any screenshots or videos of what you’re thinking of, I’d be interested.


            Oil is eventually supposed to be embeddable C++ code with a C interface to support use cases like this. In fact I had a long discussion with the maintainer of the fish shell about embedding Oil in fish. For one, Oil uses almost no global variables, unlike bash.

            One problem I see with dash is that it doesn’t have any autocompletion support? I think that could limit things UI-wise.

            Fish has some of the cleaner code I’ve seen for a shell, although of course it’s not a POSIX shell.


            As far as tests, you can get a feel like this:

            oil$ test/spec.sh smoke  ~/src/languages/dash-0.5.8/src/dash
            smoke.test.sh
            case    line    dash    bash    mksh    osh     osh_ALT dash
              0       4     pass    pass    pass    pass    pass    pass    builtin
              1       8     pass    pass    pass    pass    pass    pass    command sub
            

            Notice the extra dash column which is running against a second version of dash.

            It’s not fully automated for running against say dash, but it could be. But just running those smoke tests and a few other suites could be useful, and I think is quite easy to setup ( as long as you have a Debian- / Ubuntu- ish machine).

            https://github.com/oilshell/oil/wiki/Contributing

            https://github.com/oilshell/oil/wiki/Spec-Tests

            Sample results here: https://www.oilshell.org/release/0.8.5/test/spec.wwz/survey/osh.html

            And feel free to ask me any questions… !

            1. 2

              Wow, yes we’re thinking along very similar lines I think.


              Your description of how to use tty’s sounds like my plan to eventually implement them, but I’m not keen on it being the default way to run commands. I’d like each command running in a tty to be in its own tty (so I can filter output by which command it came from), and I’d like to have running multiple commands in parallel be a very first class experience. The main problem with tty’s in my opinion is that as soon as you are doing anything that actually needs a tty - you basically need exclusive control over the output buffer or you get weird effects. I.e. interleaving output lines from different tty’s will often not make much or any sense (despite this being approximately what happens by default in terminals).


              I’m definitely not attached to dash as the shell. I would like something posix-like to start with because I think changing the terminal UI has already more or less exhausted my strangeness budget if I hope to get anyone to actually use it. The other thing is that (like you suggest in the github issue) I want to convert the shell I’m using to be async because of signals (and because it makes the api for using it cleaner). My current somewhat crazy plan for that is to:

              1. Rewrite everything global to be in a struct.
              2. Transpile the C source code to rust using c2rust.
              3. Apply the async keyword everywhere that blocks, which causes the compiler turn the code into a giant state machine (one of the under sold reasons for using rust - which has nothing to do whatsoever with safety).
              4. Replace blocking calls with async calls.
              5. Ideally also change all fork calls to just be making a copy of the global shell state and keep subshells running in the same process - but I’m not sure how simple that will actually be.

              Autocompletion support is an interesting point, my naive assumption was that I could handle that separately and have the interface to dash basically be execstring(shell, output_related_callbacks, "thing the user typed") - but I can see how that might be problematic. You know a lot more about shells than I do, so I’m curious if you think I need to start planning for how to handle this now, or alternatively that my original plan of “worry about this later” is going to be ok?


              My frontend work isn’t far enough along that screenshots are particularly instructive, nevertheless here’s a super early stage image. I don’t have access to xl-release anymore and can’t find screenshots/video showing what I want - but the short version of it is that it’s just about the only scripting system I’ve seen that had editing future steps while a script (pipeline) was running, restarting a script from an arbitrary point, and so on as first class features. The interface was pretty clunky, but it had a lot of good ideas.

              1. 2

                Yes I imagine that the GUI would have multiple TTYs for running multiple commands in parallel. This requires some cooperation from the shell as far as I can tell – it can’t really be done with bash or dash as is.

                The async part is interesting and tricky. wait() is a blocking call, and I wonder if you have to resort to the self-pipe trick to make it async. I feel like it changes the structure of the interpreter in nontrivial ways.

                BTW in Oil the interpreter <-> OS interface is mostly in a single file:

                https://github.com/oilshell/oil/blob/master/core/executor.py#L249

                This file has:

                • class Executor
                  • RunSimpleCommand
                  • RunBackgroundJob
                  • RunPipeline
                  • RunSubshell
                  • ….

                And actually we have a NullExecutor which I want to use for “pure” config file evaluation, and maybe we should have a WindowsExecutor, since Windows doesn’t have fork() or exec(), etc.

                But yeah making this async is an interesting problem. I will be interested in the results of that.


                Somehow I’m a little skeptical of the c2rust approach… it seems more direct just to prototype a mini shell in Rust, i.e. first the synchronous version, and then see how it integrates with the GUI.

                One place to start might be the xv6 shell. It’s only a few hundred lines but it has pipelines and redirects!

                https://github.com/mit-pdos/xv6-public/blob/master/sh.c

                Trying to make this async would be an interesting exercise (and it’s not clear to me how).


                To me the issue with autocompletion is that it should appear in the GUI and not in the terminal ? At least the way I was thinking of it. Ditto for command history.

                I guess what I was warning about is that if you do a ton of work with dash, and then it doesn’t have ANY autocomplete support, then you might have to switch to something else. But xv6 doesn’t either – although it’s much less work, and I think it exposes the core issues of shell more cleanly. In my mind what you’re doing is a fairly ambitious experiment, so it makes sense to try to surface the issues as quickly as possible!

                But it looks like you already have something working so maybe my concern about dash is overblown.

                What kind of app is it? Is it like a Linux desktop app?

                FWIW I created a #shell-gui Zulip channel if you want to chat about it:

                https://oilshell.zulipchat.com/#narrow/stream/266977-shell-gui

                I seeded it with this older wiki page, which nonetheless might inspire some ideas:

                https://github.com/oilshell/oil/wiki/Interactive-Shell

                1. 2

                  Yeah actually I went through the wiki and a lot of these have cool screenshots/animations:

                  https://github.com/greymd/tmux-xpanes

                  https://github.com/ericfreese/rat

                  https://github.com/hoeck/schirm

                  Though I don’t think anyone has done anything quite like what we’re discussing, probably because you have to hack up a shell to do it !! And that’s hard. But I hope Oil can become a library for such experiments. It seems like an “obvious” next step for shell.

                  I use shell because I want a fast, responsive, and automatable interface… but I want to do things in parallel, with UI that’s a little nicer than tmux, and I want some GUI support.

          2. 1

            I was going to suggest looking at the commit history, but it looks like that’s present in the very first commit … Maybe ask on the mailing list? https://git.kernel.org/pub/scm/utils/dash/dash.git/commit/src/main.h?id=05c1076ba2d1a68fe7f3a5ae618f786b8898d327

            The weird thing is that I wouldn’t expect this to actually do anything - isn’t main.h included after the standard library headers? That means that only dash would see the different copy of errno, but it never seems to use it!

            1. 1

              Dash does use its redefined errno sometimes (note the #define errno (*dash_errno)), but as far as I can tell after looking at the glibc source it is functionally identical to glibc’s errno. My best guess for now is that it’s a historical artifact.

              Asking on the mailing list would probably work - but I don’t really want to bother maintainers with “so I’m curious about this part of the code” questions when it’s unlikely to be particularly important.

          3. 2

            I am working on my little CLI password manager written in Rust https://github.com/stchris/passage

            While testing some dependency upgrades I found an embarrassing bug which was a good reason to finally add some integration tests. It turns out that working with the assert_cmd crate for this is an absolute pleasure.

            Overall the project has some very clear constraints after which I would like to cut a 1.0 release, call it done and use it for myself. I feel like I need the closure on this project before I move on to the next thing, especially that my free time is currently limited.

            1. 2

              Playing around with McMaster’s and OctoPrint’s REST APIs, and doing some CAD design work.

              1. 2

                Building a realtime markdown sharing app with granular permissions for a friend of mine. So far, it’s going great!

                1. 1
                  • Spending time with family

                  • I think I’m going to work off-and-on a bit to stand up a wiki on AWS. I had a wiki on a VPS for years, but it wasn’t worth the monthly charge to keep it. I_think_ if I go with the smallest-tiered EC2 instance, I can keep the wiki and only pay a few bucks a month for it. I’m also keeping an eye on AWS costs to make sure something doesn’t go wrong. I’ve had more than one client who ended up with five-figure or more AWS charges because of small details somebody missed. (I understand that this would never happen with my selection, and I’ve put a billing alert on my account to double-check, but I’m still interested in seeing how fast the bill grows. Good luck trying to sort through AWS pricing.)

                  • Going to publish an interview I did with a technical coach a couple of days ago. We didn’t get to actually code, but we talked a lot of about good tech practices and the difficulty of getting teams and orgs using them

                  • Re-plan my studio development room. I keep trying to dial-in the right amount of tech and furniture so that I can comfortably code and also shoot creative video and chat. No matter what I do, it seems like after a few weeks I’m ready to reconfigure it all. Sigh.

                  1. 4

                    Depending on what exactly you need, did you consider smaller cloud providers? Might help with pricing. Things like Vultr, Digital Ocean, Hetzner Cloud, etc.

                    1. 1

                      It’s a good point.

                      I felt it was worth a few bucks a month to have a reason to screw around with AWS. I spent some time getting up to speed on everything a couple of years ago and I don’t want to let those skills decay.

                      It’d be nice to configure the wiki on CloudFormation and then be able to dial it up or down or change the VPS whenever I feel like it. There are a few other things it would be fun to play around with if I ever have the time.

                      But yup, if you’re only shopping price, there are a ton of better places than AWS. Even if some of their stuff is competitive, they suck you in and then kill you on incidental and upcharges without you realizing it. And then, for all of that work, unless you’re prepared to spend even more time and money, you’re now locked-in.

                    2. 2

                      AWS is pretty expensive, check scaleway (their smallest instance costs 1.8 EUR per month) or hetzner cloud. Vultr is also pretty cheap and has more regions to choose from including US, Asia and even Australia

                      1. 1

                        I have really liked Vultr.com for small cloud instances. They have some for $2.50 / mo., plus a signup bonus of $100 - https://www.vultr.com/?ref=8737202-6G (ref link). Billing is much easier IMO too.

                        If you have a lot of AWS credit though, it may be a better option.

                      2. 1

                        Some housekeeping for https://github.com/dw/mitogen

                        1. 1

                          Refreshing my Golang and Cypress skills, and learning TypeScript + React, by writing a trivial iCalendar anonymiser + merger. Plus getting my Emacs config working well with both Golang and TypeScript. Something of a steep learning curve :)

                          1. 1

                            I’m reading lobste.rs and waiting for an ebook reader to arrive. Also scrolling through some old writings and examining whether to publish them or chuk them to trash.

                            1. 1

                              Technical? writing a bit more Selda queries (Haskell SQL). Maybe learn enough serverless to go from PureScript -> Slack bot.

                              Personally? Family bike ride, I hope. Coffee. More quarantining.

                              1. 1

                                Traveling home. Feeling dead. Holidays at some point turn into a matter of holding your breath until you arrive home.

                                1. 1

                                  Spending time with family, reading (I got a new Kindle in the Black Friday sale) and going through more boxes of stuff we’ve dragged around through many moves to finally get rid of. We’ve slowly shed stuff over many years, but it’s funny how every iteration we find more stuff we can get rid of.

                                  I’m also going through my photo collection. Because taking digital photos were “free”, I’ve got more than 20 thousand of mostly shitty or mediocre photos going back over a decade. I know there are some good ones, or at least photos that remind me of good times, but I can’t find them among the chaff. So I’ve decided I’m allowed to keep 1 photo per day (averaged over a year) and am aggressively pruning my photos. It’s a big job, so I’m doing only one year each weekend. To motivate myself, I note down how many I had, and how many are left after pruning.

                                  1. 1

                                    I’m porting a few of my old Qt4-based fixed pipeline OpenGL projects to use GLFW and modern OpenGL.

                                    1. 1

                                      Still trying to connect to freenode with WeeChat on my WSL, for some reasons it works for irc.tilde.chat but just not freenode :/ tried with ssl too but still connection timeout.

                                      1. 1

                                        Only in WeeChat? Did you try something like telnet to connect to it? Might reveal more about where things are going wrong.

                                        1. 1

                                          telnet irc.tilde.chat 6697 works, but for freenode:

                                          > telnet chat.freenode.net
                                          Trying 185.30.166.38...
                                          Trying 82.96.96.11...
                                          Trying 84.17.44.180...
                                          Trying 130.185.232.126...
                                          Trying 185.30.166.37...
                                          Trying 2a01:270:0:666f::1...
                                          Trying 2001:41d0:401:2000::1056...
                                          Trying 2607:5300:201:2000::8f6...
                                          telnet: Unable to connect to remote host: Network is unreachable
                                          
                                          1. 1

                                            telnet chat.freenode.net

                                            Even with port specified? telnet chat.freenode.net 6667

                                        2. 1

                                          We’re not having any connectivity problems today. Are there any errors or logs you can provide? Dropping by #weechat when connected from hexchat might be helpful.

                                          1. 1

                                            Installing weechat (apt) using the exact same method works perfectly on my other Linux machine (non-WSL, just Linux), it can auto-connect quickly and join channels, but using the same command on WSL (WSL 1 to be specific) just doesn’t connect, there isn’t any errors shown (not sure about log files though) and just says connection timed out, retrying in x seconds. Everything else in weechat seems to work fine (including being able to successfully connect and join tilde as mentioned above). I will see if I can join hexchat when I have time later today. This has been happening since a few months ago if I remember correctly, as far as I know it should be the problem with WSL.

                                            1. 1

                                              Maybe a default is different, can you copy your entire config from the linux box to the WSL machine?

                                        3. 1

                                          I’m primarily from a Linux background, so learning more about Win32 and DirectX 12. I’m going a little slow since I’m submitting pull requests to fix documentation typos and such as I go. I’m continuing the work I’ve been doing over the past few weeks, combining it with learning how Windows works and playing with Windows Performance Analyzer, VTune and NVidia NSight.