1. 22

This is the weekly thread to discuss what you have done recently and are working on this week.

Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.

  1. 12

    Continuing my review of debugging books. I’ve decided to go through my debugging books, review them all, and then write an overview/critique of them.

    This week I’m finishing (or have already finished) Agans’ “Debugging”, Grötker et al’s “The Developer’s Guide to Debugging”, and Polya’s “How to Solve It”. I hope to have the next essay in my series done and posted on the weekend.

    If you have a recommendation for a good book or paper, feel free to share or message me. I’m particularly interested in older (pre 1990) ones.

    1. 3

      I think literature reviews are incredibly valuable. Thanks for doing this!

    2. 8

      Reading more about Rust as I prepare to start a job that will largely have me writing in it.

      The new position puts me as the only engineer on a project initially and my usual languages are C and C++. I’m worried about safety of a system I’m writing by myself. Rust should provide sufficient safety and strictness to help mitigate a large portion of problems with writing low level code, especially since this will be completely green field development.

      I’m also excited to be using a modern toolchain for development (e.g. cargo) :).

      1. 4

        start a job that will largely have me writing in it.

        That’s a really exiting statement as far as the overall Rust ecosystem goes. Are you at a point where you’d be comfortable saying who this is for, or maybe at least a bit more about the type of software you’ll be building? I’m really curious to hear more about where Rust is heading in industry.

        1. 7

          In the interest of transparency, the job didn’t dictate the language - I did. I spent most of the Christmas holiday prototyping and experimenting with stuff to try and find a language I felt most suited the nature of the project (one engineer, strict performance requirements, strict/safe compiler). I’m an OS-level engineer by experience, having spent the entirety of my professional career working on lower level OS stuff.

          The project itself is under wraps but it’ll be low level OS-related code. Some OS services/daemons and some driver work. The code will need to be minimal overhead for CPU/memory and run on a variety of hardware setups. Where I can, I’m going to use Rust as much as possible.

          If I can find the time, I’d like to blog about and/or open source some if it as we go along :).

          1. 3

            I know I speak for others too when I say it will be great to hear about any of your experiences using Rust in production. Good luck and I hope you find it enjoyable!

      2. 6

        Working on implementing an Earley parser using this amazing tutorial. I’ve read that site about a dozen times, and last night it finally clicked. Money quote from the author:

        Most compiler texts are obsessed with slaying the compiler dragons of the 60s with 70s technology.

        If you have any sort of appreciation for algorithms or parsing, you should study it.

        1. 4

          The coverage in Dick Grune’s Parsing Techniques: A Practical Guide is great, and his page for the first edition includes PDFs. If you’re interested in parsers in general, it’s definitely worth picking up.

          I’ve also been working on implementing Earley parsers, and the Earley/intersection parsing approach seems particularly compelling. It’s described in chapter 13 in the second edition, doesn’t seem to appear in the first.

          1. 2

            So Earley will recognize all context-free grammars, including ambiguous ones? I have two problems with that:

            1. Doesn’t it have to return a “forest” of all possible derivations of ambiguous grammars? What do you do with that? That doesn’t seem to be solving the problem; it’s more like pushing it to a later stage.
            2. Context free grammars aren’t powerful enough to represent most languages we use (C, C++, shell, Ruby, Perl 5 and 6, etc.). So even if Earley magically solved problem #1, it still wouldn’t be an ideal algorithm.

            I’ve written about this length in various comments, but I think CFGs are the wrong abstraction for human programming languages. They’re simultaneously not powerful enough and too inefficient.

            My argument is analogous to the one here regarding CFGs and network protocols. This is admittedly is a fringe claim, where as “CFGs should be used to parse programming languages” is the claim you will read in nearly every textbook:


            One of the langsec people responded on HN: https://news.ycombinator.com/item?id=16126234

            I read that paper about “calc-regular languages” but it was immature – which the authors admitted – it still can’t handle some common constructs.

            Anyway, my overall point is that we should use formalism, but formalism should match practice, not the other way around. Practice shouldn’t be put in the straitjacket of a theory (especially when that theory doesn’t provide you with good efficiency or correctness guarantees, as CFGs don’t).

            (FWIW I have a very messy wiki page about the difficulties with parsing here: https://github.com/oilshell/oil/wiki/Parsing-is-Difficult

            I don’t believe any of them are addressed by Earley parsing. The main bugaboo I have now is that every language has at least TWO parsers. One for the compiler/interpreter, and a totally separate one for linting/formatting. The former ignores whitespace; the latter must recognize whitespace. Even a brand new language like Julia has two parsers!!!

            Any system based on CFGs don’t solve that problem. I do NOT want parse trees (which include the full derivation of the string from the grammar); I want ASTs or “lossless syntax trees”.

            1. 2

              Well, I’m not happy about how Earley detects ambiguity TBH. But I am looking for something that is simple enough to take the question off the table so I can continue moving forward for now.

              You ended up using Pratt parsing for Oil, correct? The choice of a parser for my project is still very much up in the air; the most important consideration is whether I can build the grammar immediately before parsing a module, which isn’t a great use case for a lot of existing tools. I don’t need the ability to mutate the parser while I’m parsing, as I don’t mind cheating by pre-processing to locate all the grammars in use.

              From my research, Pratt parsing looks really underutilized, but I wasn’t able to discern whether it was painful to build the rules programmatically after parsing patterns and such, and whether it’d be substantially different from PEGs. For example, if I’m building a for x in iterable statement, would matching the in keyword be painful, especially if the metagrammar specification was declarative in nature? I won’t be writing the led() and nud() methods by hand here, they’ll be generated from specifications provided by the user. Also, I was initially put off by all the precedences on everything, then later realized those are implicit in most grammar specifications anyway, so that is pretty unfounded.

              1. 2

                Also remember you can always use more than one parser. sklogic used to point that out on HN when people were choosing between one that succeeds/fails fast and one easy fod error messages. Something like that.

                His tool on “combinatorylogic” Github is a LISP that embeds a bunch of DSL’s he mainly uses for program analysis. So, his double-parser strategy might be well-informed. ;)

                1. 1

                  Oil is best described as 3 interleaved recursive descent parsers + one Pratt parser for expressions, with 15 or so lexer modes. [1]

                  Pratt parsing is really best suited for expression languages, not a whole programming language, as Crockford does in his widely read article (which I reviewed here [2]).

                  Although Pratt parsing isn’t a grammar-based approach, there is something called an “operator precedence grammar” which is a subset of CFGs, and what it handles roughly corresponds to that set of languages:


                  Bob Nystrom has a good way of putting it:


                  If recursive descent is peanut butter, Pratt parsing is jelly. When you mix the two together, you get a simple, terse, readable parser that can handle any grammar you throw at it.

                  I would add the caveat that this technique is really good for implementing existing languages. If you are designing a language from scratch, it might be helpful to use some kind of parser generator to guide the design.

                  In summary, I would say the parsing method you use depends very heavily on the language. And many languages are compositions of multiple languages. I don’t have experience with Earley but I’m sure the same thing is true: it’s good for some things and not good for other things.

                  [1] http://www.oilshell.org/blog/2017/12/17.html

                  [2] http://www.oilshell.org/blog/2016/11/02.html

                2. 2

                  Have you heard about parsing with derivatives?

                  I recently explored the possibility of doing this with Clojure’s spec. Here’s an example:

                  (s/conform ::s-expression "(abc  def)")
                  [[:sub {:left  \(
                          :nest  [[:atom [\a \b \c]]
                                  [:whitespace [\space \space]]
                                  [:atom [\d \e \f]]]
                          :right \)}]]

                  Notice that in the third line, the whitespace has been captured. A linter could examine whitespace and a compiler/interpreter could throw it out. Specs can be bidirectional too, so a formatting tool could take a data structure like the above, adjust white space, and unform it, producing formatted source code.

                  Regarding returning a parse forest, the author of parsing with derivatives goes into that here. On the next slide, the author alludes to the possibility of modifying the grammar during parsing (!) which may be of particular interest to you.

              2. 5

                Going to try to pick up Elm again, for the purpose of writing a game. Tried last year, let’s see if it goes better this time…

                1. 3

                  Just out of curiosity, any arguments to favor Elm over Reason?

                  1. 2

                    I like Reason, but it doesn’t enforce purity. IMO that’s the big reason to use Elm instead.

                    1. 1

                      Ecosystem mainly - though Reason certainly seems very interesting as well!

                    2. 2

                      Elm is great! I really liked the new error messages in the latest 0.18. They’re a bit overdue for an updated version though.

                    3. 5

                      As ever, I’m working on Dinit, a cross-platform init/service manager in C++. I’ve been making good progress but there’s still so much to do - sometimes it feels like I’ll never finish. But I would really like to be able to finish it, even if it never makes an impact on the rest of the OSS world, it would be the one major project that I took all the way from conception to finish myself.

                      I did get a pull request and an issue raised in the last few weeks, which were both pretty trivial but it’s good to know someone is keeping track.

                      1. 5

                        next week I’m out at a couple of conferences, so finding out who’s there (are you at dotSwift Paris or FOSDEM in Brussels?), highlighting the FOSDEM schedule and making sure my train tickets and hotel reservations are correct.

                        I’ve been slowly progressing my HURD side project, though my work on it so far this week has been migrating the VM from my home PC to a laptop that I’ll take with me to FOSDEM.

                        Work: some people management and team workflow things, doing job interviews, and then the odd bit of Javascript around the edges.

                        Music: one of my bands is in a conference on Saturday so plenty of practice!

                        1. 3

                          Going myself to FOSDEM! First time there, I still have a ton of homework checking the talks I am going to attend, which I’m just constantly delaying, probably because it is overwhelming as hell.

                          1. 1

                            it is. The mobile apps have favourites and alarms which make it easier to navigate than reading the whole thing, but you still have some work to do to star the interesting sessions :)

                          2. 2

                            I’m also heading to FOSDEM! My first time, and also the first big event I’ve been to, so I’m not too sure what to expect.

                            1. 1

                              I found it overwhelming when I first went, there’s so much on. The talks will be videoed so if you have a choice meet people at the booths or dev rooms.

                          3. 5

                            First EMT class tonight! It’ll be my first time back in school since I graduated college. It’ll probably be fine, but I’m still anxious.

                            It’s also cleanup week for me. I have some projects and essays that are 70% done before I lost interest and wandered away. I want to make at least some of them public, which means pushing through to the end.

                            1. 5

                              Home: finish up a small treebank viewer in Rust and gtk-rs. It is primarily for my own uses to quickly browse through CoNLL-X dependency treebanks and dump trees in the Graphviz dot and TikZ-dependency formats for teaching, papers, etc. Since this was my first project with gtk-rs, I was surprised at how complete the gtk-rs and related bindings are. What I mildly dislike is that I had to litter a lot of Rc<RefCell>s everywhere, which is not commonly the case in my daily Rust code. Also, communicating between a worker thread and a GTK main thread is kinda ugly [1]. Also, since Rust does not support inheritance, defining your own widgets is a bit cumbersome (typically, you wrap a Gtk+ widget and implement the Deref trait to provide the inner widget).

                              Anyway, given all the constraints, I think the gtk-rs people have really done a nice job!

                              Work: teaching (this week’s program: remainder of parse choices for dependency parsing and the implementation of CKY). Hopefully, I will have some time to work on a sequence to sequence learning project that I have started.

                              [1] https://github.com/gtk-rs/examples/blob/30549d056f21eafba755f8084f3cb8a85e4f6736/src/bin/multithreading_context.rs#L50

                              1. 4

                                Solved a couple lingering characterization problems in Farisa’s Crossing, which I hope to have in final draft by end-of-year and publish in 2019 (yes, it takes a long time).

                                Using a hybrid numerical/evolutionary approach (SGD for learning heuristics, genetic algorithms for inventing heuristics, since the “throw everything in the cauldron, and train a deep neural net” approach didn’t work) to see if I can get an AI to play Ambition at a competent level, preferably without ad hoc hand-holding.

                                Refreshed Scuba skills after 2 years out of the water; going diving in Mexico in February, and then Roatan in the summer.

                                1. 1

                                  @michaelochurch how to you keep track of characters and plot arcs? Do you use a spreadsheet, a notebook? Asking for a friend.

                                  1. 3

                                    I use Scrivener. And lots of revision. After enough time in a book I can keep track well enough to prevent continuity errors. Also I don’t have a complex plot as far as first books of epic fantasy series go.

                                2. 4

                                  I’ve been working on learning Godot the past week, which has been interesting. GDScript has been an interesting language to work in, as it’s the first time I’ve spent a lot of time in a language that is purely reference counted (python doesn’t count because it has a cycle detection in the garbarge collector).

                                  Other than that Godot has been fun to work with, if a little underdocumented in parts. (Though I’ve never found it to be fatally so).

                                  For work, I’ve found that using a greyscale screen (I’m using NegativeScreen on windows for this) is helpful for increasing mental focus. For me, it seems to clear the mental clutter that having lots of colors adds, which is helpful at the end of the day.

                                  1. 4

                                    My workplace has finally formally decided that managers such as myself are not expected to also be senior technical staff. This is in every way the right model, but now I have a bunch of technical obligations that I need to schedule handing off to others in our group, so that’s sort of a PITA, because I can’t just dump stuff context-free, but at the same time, there are other demands on my time. Eh. So it goes, in middle management. Some post mortems to run/attend, some reviews to write, some organizational chaos to overcome.

                                    Outside of work, I need to find some time to rebuild part of the big computer; my patience with fiddly PC non-sense is growing thin, and I am tempted to just part the thing out and get a dedicated storage box or something. But this too will pass, and I’ll miss having a big powerful desktop computer in six months. So I’ll try to wait it out.

                                    Otherwise, baby proofing now that #2 Daughter (9mo) is up on her feet and taking preliminary steps.

                                    1. 3

                                      my model for that hand-off is that I still do technical work, both during offloading and to keep my hand in (there are plenty of programmers who respect capable programmers more than capable non-programmers), but that I should not be on the critical path. When I delegate technical leadership, I make it clear that I’m available for reverse-delegating where I can help out so that the new technical lead feels less like they’ve been dropped in it.

                                    2. 3

                                      Just debugged a weird issue that was happening only on my laptop only in my compositor, so expecting to work on weston-rs more this week. 2018 is definitely going to be the year of Rust-powered Wayland on FreeBSD on the desktop :D

                                      1. 3
                                        1. An rPI-based device for streaming video and some metadata from the inside of a racecar to a remote RTMP server.
                                        2. An AsyncIO+protobuf based TCP server for bidirectional communication between a hardware device and a cloud-based service for controlling said hardware device.
                                        1. 3

                                          $work: i++; GOTO $work

                                          !$work: Organizing docs for tax season, researching workbench designs, trying to figure out local requirements around building permits. Some towns near me don’t require permits for structures smaller than some square footage, some do. I think I just have to call the City and ask, but I thought it’d be fun to read some local codes and see if I could find out for myself. My idea of fun is nonstandard.

                                          1. 3

                                            I’m currently preparing for another exam but since it’s less stressfull than the first, I have a bit more breathing room.

                                            Since my last comment, I’ve finished data recovery, lost a few days of work in total but nothing too serious. Thankfully I could recover some of the more sentimental data from an old harddrive.

                                            With my data back, I’m working a bit on my toy kernel, global state management could be improved so maybe I’ll finish that. Since my task manager was partially nuked during recovery (I should have committed it), I’ll have to rewrite it.

                                            I also started investigating web development with rust, the currently best options seem to be a combination of rocket and diesel as frameworks. I’ll see how that works out tho I’m optimistic.

                                            I’m also fixing some bugs for SECL, the next milestone is closing in, though I managed to work in some much needed improvements and fixes in just a few commits, there is now a proper way to do type conversion and a few methods expose this functionality into the config file (so you can enforce variables to be a certain type).

                                            1. 3

                                              Writing something that, I hope, will eventually become a text editor, multithreaded and extensible with MoonScript/Lua (or any other language via loadable libraries and external processes). The implementation language is Rust and I’m going to use tokio-rs for async IO and luajit for Lua. At the moment I have a basic rope implementation with Unicode support (including extended grapheme clusters thanks to the unicode-segmentation crate) that can pass some tests. The source code is here.

                                              1. 3

                                                Continuing work on improving my home setup, solved consistent image versions last week (using the last modified date of the folder containing the packer configuration for that particular image). This week I need to polish those base images to make sure they come up successfully & can pull/apply my masterless puppet git repo to themselves. Then begins the slow rebuild of the ~15 zones I have that I want to be managed by puppet. Some of them were hand built, others built from a puppetmaster so will be easier to port as I’ve got puppet configuration for them already.

                                                Also been given a Digital Oscilloscope by a friend, so I can investigate what signals my treadmill is sending from the headunit to the motor board, and attempt to replace it with a network-connected controller I can program. Intending on being able to collect stats from it (you walked for x duration at y average speed) for tracking purposes, as well as being able to start/stop/change speed on it from my terminal - I use it as a walking desk so I’ve already got a machine on it with terminal running.

                                                1. 3

                                                  I continue developing genact which is a useless activity generator that does something between pretending to do actual work and looking like a Hollywood computer terminal. It’s written in Rust and has Travis set up so it automatically builds for Linux, Mac, Windows and the Web in one push.

                                                  1. 3
                                                    • Tried to make a 0.4.alpha1 release of Oil yesterday, after the success with Linux distros [1].
                                                    • The OSH benchmarks run several configure scripts found in the wild, and they failed to finish, catching at least 3 bugs introduced since the last release! Testing is useful :)
                                                    • I’m working on those bugs now.
                                                    • Before release, I also need to write some blog posts explaining the project. Every release should have a “Why Write a New Shell?” link, because the Hacker News and Reddit /r/linux threads show that new readers are confused by that.

                                                    [1] https://www.reddit.com/r/oilshell/comments/7qn0p8/success_with_aboriginal_alpine_and_debian/

                                                    1. 2

                                                      nixwrt, which will some day - if it gets that far - be a Linux-based OS for consumer broadband routers (think WRT54 and its successors) based on the Nix packages collection. This week I aim to get it from the point of “something that boots and runs busybox sh on its serial console” to “something I can actually ssh into”

                                                      1. 2

                                                        I’m working on a new Wordpress theme Log Lolla. It is meant to help you create knowledge from your everyday news consumption. And yes it has a brutalist look.


                                                        If you want to get it please send me an email. Thanks :))

                                                        1. 1

                                                          For $CLIENT it’s a mixture of testing a heap of undeployed changes so we can finally get the CD setup in place for the prod environment, and supporting other developers who are having separate often-fatal issues running our dev environment under vagrant.

                                                          Hopefully I’ll get time to actually release the shell script library I’ve been working on publicly, possibly even some of the tools that are built upon it too.

                                                          And I’m having another go at keeping a regular cycling habit. I’m trying at night now, which means less crazy Thai drivers. Not none, but less at least.

                                                          Oh and of course, the continuing saga of fixing the utterly fucked electrical wiring in the house. This week it’s replacing an unearthed outside fluro tube with a new earthed LED “tube”, installing a couple of new LED spotlights, and maybe checking the lounge room down-lights for loose connections. Might even get to replace the remaining outside outlets that are all literally held in place by the wires and in one case has almost zero protection from the rain.

                                                          1. 1

                                                            This week, I’m doing some contract work to pay the bills.

                                                            But on Helmspoint (https://www.helmspoint.com) I talked to users and got some indirect feedback. I think I missed the mark a bit, and will need to go back to the drawing board. I use to think the data pipeline stage was different than the deployment stage when using ML in production. I believe there can be a single abstraction to use for both, and will be prototyping it out.

                                                            1. 1

                                                              Work is the usual, lots of churning along with Ruby. My main side project right now is trying to learn a decent amount of Go. I’m interested in seeing what I can do with the concurrency stuff people keep talking about, and it doesn’t seem too hard to learn. I’d like to write in the neighborhood of 100-1000 lines of useful code in it before I move on to something else.

                                                              1. 1

                                                                At work, I’m still trying to debug weird crashes in an open-source geometry kernel (vcglib), which is proving to be extremely tedious. Thank god for address sanitizer, at least.

                                                                Outside of work, continuing trying to get my sourdough down pat and consistent, as well as continuing work on my large-scale vertical pen plotter. This week, I’m trying to come up with the Jacobian that relates the lengths of the two chains it hangs from to its xy-position on the wall, so that I can start doing path planning in chain-length-space.

                                                                1. 1

                                                                  Finally crawling my way out of Jenkins Pipeline internals to build the front-end code to link from your pipeline view to any downstream builds that were triggered by it. I’ll be happy when it’s done, but the end value to users in no way reflects just how much (re)work that went into it. Such is life, and I’ve learned a boatload about how it all works internally, so no “raegrets”, just a bit bummed that the end result is so low-key :)

                                                                  1. 1

                                                                    Back from vacation and apart from work I hope to get my first proper PR to rust-clippy in a good state this week.

                                                                    1. 1

                                                                      At work I’m finishing up a backport of a small feature to a patch release.

                                                                      Outside of work I’m reading through Fluid Simulation for Computer Graphics and hoping to implement a simple simulation or two. I really like the way it explains the math behind fluid simulation, but I was hoping the “… for Computer Graphics” in the title meant it was a little more practical and focused on algorithms and data structures. I’m starting the 5th chapter, and so far it’s really heavy on math and differential equations, but really light on implementation details.

                                                                      1. 1

                                                                        Exploring the design space of the new file protocol for Jehanne.
                                                                        The only decision so far is it’s name: File Protocol. It has an important property: FP reminds 9P in hex. :-D

                                                                        Also, I’m coping with the unexpected reception of an (unrelated) article about AI.

                                                                        1. 1

                                                                          orking on Rustwell, a RESTful frontend to the Gnome Shotwell backend database.

                                                                          The plan is as follows:

                                                                          1. Put a Diesel & Rocket front end in front of the Shotwell SQLite database, with full standard CRUD capabilities.

                                                                          2. Add two tables to the Shotwell Database: One to define StorageResources, (NAS boxes, photo upload servers that you use, et cetera), and then one table to define Objects. Each photo or video in the existing Shotwell database can then have one or MORE rows in the Objects table, one for each copy of the photo/video. Once that is done, I can crawl my two NAS boxes and find exactly which photos are backed up and which are not (from years of disorganized backups), and (much later) then which photos are backed up elsewhere (Flickr, Google, Shutterfly, etc).

                                                                          3. Add RESTful clients to standard photo storage services.

                                                                          4 Compile the whole thing to my BeagleBone, and now have something that automatically syncs my two NAS boxes and uploads to Flickr (those things that should be uploaded t0 Flickr).

                                                                          5 Android app to talk to by Beaglebone, a la Sweet Home.

                                                                          6 Tor service?

                                                                          I am on step 2 now. ROcket works, and I am adding the routes.

                                                                          1. 1

                                                                            Still working on rewriting a Swing Java App from old school button handlers does it all into a MVVM approach.

                                                                            For fun, I saw an article here about building your own kernel for the Raspberry Pi - as I have a few lying around, I figure I’ll try that out.

                                                                            1. 1

                                                                              Working on porting some software to PASE - I’m doing pretty good for someone who’s never used AIX before and with no knowledge of POWER assembler; just applying my knowledge of hacking around old commercial Unix and codebase familiarity. We’re a lot further in that we thought we’d be! The big problem is toolchain, where the readymade ones kinda suck, so we’re tempted to build one ourselves. (The problem there is, GCC is a monster to build and may assume some things that don’t exist. LLVM doesn’t seem much better, if it’s even compatible at all.)

                                                                              1. 1

                                                                                This weekend I tested a few hash functions in SMHasher. I will spend this coming weekend reviewing a few open source projects for bugs and vulnerabilities.

                                                                                1. 1

                                                                                  Mapping out our current 6 year old architecture, identifying business goals, any existing issues and pain points, planning out something new.

                                                                                  Trying to implement some practices and ways of working around software development (presenting scrum to the team), also designing a new deployment workflow.

                                                                                  All whilst potentially interviewing at other companies.