1. 23

This is the weekly thread to discuss what you have done recently and are working on this week.

Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.

  1. 13

    Released the second release candidate for version 5.0.0 of CHICKEN Scheme yesterday.

    Will probably be doing some more testing on various platforms using VMs. If anyone would like to help test on exotic (but supported) OSes and/or architectures, that’d be much appreciated!

    1. 12

      Going to be looking at ways to exploit image storage on PGP Keyservers

      1. 9

        I hope you don’t make any progress. Or maybe I hope you do…

        1. 7

          I hope you do make progress. Rip them to shreds until they’re forced to fix them or bystanders use something better. ;)

        2. 10

          Multics email development continues at BAN.AI — working on MAIL-11 over DECnet and UUCP.

          1. 4

            i have used BAN.AI multics i recommend checking it out for sure

            1. 2

              Very cool! Must have missed your previous updates. Are you running atop modern hardware on something liek SIMH?

              Is there a multics media kit out there somewhere you bootstrapped from or were you the lucky owner of some actual period tape and the ability to read it?

              1. 5

                I’d recommend reading the past thread on the system, which is now a little dated (things move fast) but still a good introduction to the system.

                I started this project as something of professional agitator, contacting many Multics people and the past Multics-using sites trying to track down any leads on software (and hopefully backups) that may still exist, and finding what was run at each site. While successes are few and far between, we’ve recovered a surprising amount of software this way.

                In some cases software has been available from various disparate collections, but was never integrated, compiled, and run on post-MR12 systems until now. Sometimes this involves correcting OCR errors and retyping listings from scans and microfilm.

                In other cases there is known Multics software which is essentially ‘extinct’ but various descendants and forks still exist, so we are working on backporting/crossporting the existing software versions back to Multics - this includes programming languages like XPL and SNOBOL, software packages like REDUCE, MACSYMA, OMNITAB, TeX, etc.

                Finally, we are also working to develop new Multics software and ports. We will be distributing the software and updates through a “package management” system via an online MIKSD (Multics Internet Kermit Service Daemon), which we hope to have available and running next year.

                The most important utilities and tools from the efforts are being integrated back upstream and will appear in future releases of the base Multics distribution, where appropriate.

                The BAN.AI system hopes to become the Multics equivalent of the SDF system, and already has become the de facto “home base” for a lot of efforts — I’m humbled and grateful to be able to work with so heroes of computer culture and science.

                We recently completed the initial implementations of a few fun projects, including Internet email and DECnet.

                I’d really recommend logging in and reading the system news (pmotd -a) to get an idea of what we’ve been working on lately!

                1. 2

                  Thanks for the reply I’ll definitely do that! Has any thought been given to backporting ssh or telnet? Just wondering because I can’t connect with the web app due to cranky corporate firewalls :)

                  1. 2

                    You can mosh or ssh to dps8@m.trnsz.com and you can also telnet to m.trnsz.com on port 52815.

                    1. 2

                      Thank you very much! Exploring systems like this is so enlightening for folks like myself who really dig computer history and UNIX history in particular.

                      Like I couldn’t help but notice that the help system reminds me a lot of our long lost “friend” GNU info! :)

                      1. 1

                        The info pages are brief, usually similar to man pages.

                        The ‘paper’ documentation, most of which is in our document archive, not always up to date for the latest release however, is the main source, but is indexed online.

                        The where_doc (wdoc) command locates the appropriate manual. Try ‘wdoc fnp’ or ‘wdoc directory’ or ‘wdoc fortran’ - or any keyword.

                        There is also explain_doc (edoc) which summarizes a manual and shows you the table of contents. Example: ‘edoc AG92’ or ‘edoc AK50’

                        Between prints of the documentation, the errata is available online. You can use ‘lh errata’ to see all the available errata.

                        (Retyping all the documentation from scans back into “compose” source code is something nobody wants to do, a gargantuan task - but will probably have to be done eventually!)

                        Another thing, probably the inspiration for the bugs section of UNIX man pages, are error info documents. These document all the open trouble tickets that still shipped with a release. You can type ‘lh errors’ to see a list, or for example ‘help lisp.errors’ or ‘help probe.errors’

                        1. 1

                          Wow that’s a nice leg up. Thanks! WRT hand typing in the paper docs - could you use OCR and then hand correct the bits it gets wrong? That might at least reduce the task’s size somewhat?

            2. 9

              Familiarising myself with rustc, so that I can contribute to it.

              1. 8

                Focusing on theft again, after taking a break for several months. I’m well into a massive restructuring (to add multi-core support), which feels like it’s been going on forever. Now the main stuff is working, but I’m in the long tail of making sure error handling, config hooks, etc. work, now that some of them may be split between multiple address spaces.

                Other stuff that should make it in the next release:

                • Failure tagging: If the user code sets an identifier for a failure, it can avoid shrinking paths that would change into other known failures, so that common failures don’t shadow rarer ones. I have a proof of concept of this already, but it needs to be redone to work with multi-core, because that splits its state between processes.

                • Possibly replacing the RNG with something faster than Mersenne Twister 64, since outside of user code, theft spends essentially all of its time in the RNG.

                • A better interface for passing config to user hooks, and moving more common configuration things into official settings (for example: running until a time limit, rather than a number of test trials).

                • Better benchmarking of multi-core behavior (especially shrinking), and tuning with servers with 30+ cores in mind.

                • A function that provides a CLI interface, with argument parsing.

                • Improving the bloom filter implementation. (That will also become its own library.)

                • Lots of misc. improvements to shrinking.

                Other things that are on the horizon, but will probably be pushed until after the next release so it doesn’t take another year:

                • Coverage-directed test case generation. I have a proof of concept using clang’s SanitizerCoverage interface – I don’t want to make theft depend on a specific compiler, etc., but know what the interface would look like to be able to utilize that sort of info, whether it comes from clang or something more project-specific.

                • Better structural inference during shrinking (and coverage-directed generation). This should make shrinking faster.

                • Sort sort of persistence API for generator input.


                • Various performance improvements to our regex engine (with a focus on frontloading work at compile-time).

                • Implementing some succinct data structures, for efficiently querying a (currently) massive in-memory data set. I should be able to eventually open-source the parts that aren’t highly specific to our use case.

                1. 1

                  Re RNG. I always used xorshift if speed over quality was goal. Plus, it’s tiny. A quick look led me to someone saying xoshiro-256 is faster with good, statistical properties. So, there’s a few to try.

                  1. 2


                  2. 1

                    Coverage-directed test case generation. I have a proof of concept using clang’s SanitizerCoverage interface – I don’t want to make theft depend on a specific compiler, etc., but know what the interface would look like to be able to utilize that sort of info, whether it comes from clang or something more project-specific.

                    Oh wow, exciting! I actually started looking into the theft sources with the intention of doing this, great to hear you’re working on it yourself. Did you have any successes yet finding bugs that did not show up without feedback?

                    1. 2

                      I haven’t hooked it up into theft yet, I just confirmed that the interface will work like I expect – and there isn’t much point in starting until the multi-core stuff is done. It’s almost a complete rewrite (which is why it’s taken so long).

                      I want to add a more general coverage feedback hook, which could also be used with (say) line number info from Lua’s debug hook, hashes of runtime data, or something else that can be easily monitored and would correlate to meaningfully different behavior.

                  3. 8

                    Working through the Cryptopals challenges! Currently finishing up set #2 and enjoying it immensely.

                    1. 7
                      • Correcting homework projects (in Rust). Amazed by the quality of some of the submissions. These projects were relatively short time-wise, but that didn’t hold some students from implementing interesting machine-learning related projects, such as a small voice recognition system, spectral clustering, and neural net-based spell correctors.

                      • Preparing the first release of finalfrontier, a Rust package that implements training of word embeddings with the skip-gram model and subword embeddings (similar to fastText) and the structured skip-gram model (similar to wang2vec). One student is also working on dependency embeddings. (The project started out as small library to read fastText embeddings, so that I could easily use them in my Rust NLP projects, but I wanted some features that are currently not available in fastText, so I decided to implement training as well.)

                      • Contemplate more if I want to ‘host’ this project at GitHub, self-host with cgit (but losing some potential contributors, Travis CI, etc.), or some other place.

                      • Polishing a paper that I am co-author of.

                      • Packing up stuff at work, since we are moving to NL next week.

                      • Recently (private/home): picked build failures from the NixOS 18.09 branch, tried to fix them (new to Nix), and submitted some pull requests. It’s been fun and the Nix folks are very responsive! At least spaCy and some other ML projects should be working on 18.09 now ;).

                      1. 7

                        Finishing a Bare Bones wiki page for OSDev about the HiFive-1 RISC-V board!

                        1. 6

                          Oh…going to deploy Go + Postgres on DigitalOcean for an ongoing startup.

                          This will be awesome 🎉

                          1. 5

                            I worked on cleaning up the user interface of Synaptic (the trusty package manager on Debian systems) last weekend. I might have a look at it again in the evennigs.



                            First steps:



                            Current state:


                            1. 5

                              Diving into the guts of Linux and figuring out why my Alienware R5 17 doesn’t properly wake after suspend. Thankfully there’s an excellent doc on digging into this and I can’t wait to work through it: https://wiki.ubuntu.com/DebuggingKernelSuspend

                              (Resuming properly on wake from suspend is the last thing I need to get working on this new laptop. Otherwise it’s an amazing battle tank of a machine :)

                              1. 4

                                Making an idempotent Jenkins multibranch on OpenShift setup script.

                                1. 4

                                  I redesigned the landing page for https://newbusinessmonitor.co.uk/!

                                  I’m super happy with the direction it’s taking. I also hired a designer to make me a real logo. I’ll gradually add more marketing things like pricing page, and a blog. I connected the product to a printing and mailing partner, so now you can click on companies and send sales letters to them. We take care of all the printing, enveloping, and sending, so it’s super easy. It’s not available to users yet, but that’s the goal for this week.

                                  Tech: Haskell, Elm, PostgreSQL, Redis, NixOps/AWS.

                                  1. 4

                                    I’m programming (on the side) again after a 3-4 month hiatus!

                                    Rewriting my Discord bot in Haskell. It’ll be the basis of a few future blog posts where I show Haskell’s pit of success via functional core/imperative shell. Plan is to explain each layer separately: the pure code containing most of the logic, the constrained effect layer where we interact with the world, and the IO layer where we make it happen.

                                    The basics of the functional core are done and the shell compiles now, which is a big accomplishment for my rusty Haskell skills. I can connect to Discord successfully and respond to commands. This week I want to clean up the IO layer and write a command parser with parsec.

                                    1. 2

                                      very much looking forward to seeing these blog posts!

                                    2. 4

                                      I’m creating a collection of “Special offerings for open source projects”. If you guys know any devtool or service that should be on the list, please add it here: https://docs.google.com/spreadsheets/d/19rLT0G8L21aCETucxCWkbNUOmCfSWAl8MiA33wM1uxo/edit?usp=sharing Thanks!

                                      1. 4

                                        Fixing some bugs with the annotations feature in my voice control for chrome extension and adding some more fun plugins: https://www.lipsurf.com

                                        1. 4

                                          Preparing to release a big change to the Merit Lightwallet which has some major internal changes which improve transaction performance significantly.

                                          1. 4

                                            For work: Familiarizing myself with generative adversarial networks (GANs) for some possible projects. Although I’m an AI researcher and pretty familiar with machine learning in general (esp. statistical ML), I haven’t kept up that much with recent deep-learning libraries/tooling, which at this point seem to have become too successful in too many domains to continue to avoid. :-)

                                            For para-work: Rethinking some of my software use and trying to get into a set of habits I’m happy with. I recently switched to Miniflux as a feed reader, which I’m happy with so far (I pay for the $15/yr hosted version). I used newsboat for a bit but I find blog-reading actually works for me quite well while lying on a couch / in bed / etc., on my phone, so I wanted something with either an app or usable-on-mobile web interface, and Miniflux has that. Next up is calendaring. I haven’t yet decided if I really need syncing to my phone, which would impact what kind of setup works. If not, one option is running khal on my VPS, which would sit nicely alongside how I already do email, mu4e on the same VPS. I also might need to sync with a future Office365 work calendar.

                                            For future work: Sorted out my teaching for January 2019, and will be teaching two new-to-me classes, so starting to work on syllabi and assignments.

                                            • One is my own special-topics course, “AI and Games”, which will be probably about 1/3 each on AI game-playing algorithms, design of AI in commercial games, and generative stuff (procedural content generation, automated game design). I’ll probably use selected chapters from this textbook, this one, and miscellaneous course notes, papers, in-depth blog posts in a few cases, and maybe the odd GDC video.
                                            • The other is Programming Languages, which is a more standard course, but I have quite a bit of leeway on how I want to teach it. The main challenge is that it’s supposed to fill the role of what in my own BSc was split across two courses. The two I had were: 1) a survey-of-PLs style course, where you get introduced to the major programming paradigms and do little tutorial-level assignments in a half-dozen dissimilar ones (Scheme, ML, Prolog, asm, Haskell, etc.), and 2) design-and-implementation-of-PLs, covering parsing, ASTs, semantics, types, interpreters, compilers, etc. I need to fit some element of both of those into one course, in a way that isn’t an unreasonable workload. There are ways to do it, but requires a bit of thinking. So far I haven’t found a textbook that does precisely what I want, though I’m open to suggestions. I might go for a no-textbook approach with just my own course notes plus selected readings.
                                            1. 4

                                              An orthogonal WebSocket library for Go.


                                              1. 4

                                                Working on documentation and planning for the upcoming release of Wallaroo. Also going to the NYC Flink meetup tonight.

                                                1. 4

                                                  On Sunday we got a spontaneous drive failure in one of our RAID arrays. I was already planning on being at the data center to replace a drive in another RAID that had a bad sector (a much more routine and graceful failure condition).

                                                  The spontaneous drive failure turned out simpler. It’s replacement went in and sync’d in a textbook manner. The failed drive itself had somewhere north of 40k hours. An old drive, but comfortably inside it’s service lifetime. It came up “fine” once it was on the workbench. Fine here being defined as “capable of being erased before destruction.”

                                                  The replacement of a drive with a bad sector was not textbook. While inserting the replacement, an unrelated drive in the same power domain on the backplane kicked from RAID. We’ve seen this problem before, but the presentation here included symptoms I don’t believe we ever saw during workbench testing.

                                                  That drive is resyncing now and the original replacement of the drive with the bad sector is finished. Quite a way to start the week!

                                                  1. 4

                                                    I installed Magisk on my new Android (Gemini PDA) so I could install a self-signed trusted CA certificate and, using a VPN app called Postern, MITM all the HTTP(S) traffic with a Mac HTTP(S) proxy app called Charles, and finally, block hosts using an Android app called AdAway which provides a nice UI to manage /etc/hosts.

                                                    It’s really disturbing the amount of tracking that’s baked into an Android installation with a few basic apps installed, and the effort required to stop it. I was thinking of doing a write-up on it but maybe no one would care.

                                                    1. 2

                                                      I would care! show me how!

                                                    2. 4

                                                      Building a multi-tenant job scheduler for an ETL pipeline. We need to strike a balance between reasonable resource utilization and fairness. We can’t have one user monopolizing all our resources. I was pretty surprised I couldn’t find a reasonable open source or COTS solution that I could just plug into our existing pipeline. Everything I saw was for some specific pipeline system.

                                                      1. 1

                                                        I think that’s because most pipelines should have some sort of workload management built into them. Like airflow has priority weight (which works fairly well if you build it into feedback loops, etc). What system(s) are you using in your pipeline?

                                                        1. 2

                                                          Airflow was actually the first workflow system we evaluated. We needed to trigger one-time jobs based on messages in a queue - i.e. kafka or SQS, and that seemed not at all straightforward to do at the time we looked (about two years ago). It seemed like you would have to have a long-running job that polled for messages.

                                                          The other thing is that the priority weighting needs to be dynamic - to support something like round-robin per customer. That also seemed like it would not be a good fit. The workflow system we’re currently using is nextflow but with a custom system to kick jobs off.

                                                      2. 4

                                                        This week I’m working on 0xblockchain, it’s a lobsters sister site. Personally, I use this site to collect all post that I read or videos that I watch related to blockchain.

                                                        1. 3

                                                          Finishing up a small parser for map files to try and figure out if there’s a correlation between function addresses and apparent performance issues due to caching in an embedded device.

                                                          1. 3

                                                            Hopefully tonight I’ll be releasing a small update to my social network, Pnut. Today is its second anniversary, so I’m thinking about how to celebrate with the community.

                                                            [Edit: it says be descriptive!]

                                                            The largest feature is adding “badges” for users, based on accomplishments and to reward them for supporting the network and community. Users choose whether to display a badge on their profile.

                                                            I’ve also adapted more of the data to be accessible via RSS - post search and channel messages.

                                                            1. 3

                                                              A GUI for my music synthesizer, for Windows (first, though I want to port it later) in Rust.

                                                              1. 3

                                                                At work: Continuing development of my file-syncing tool with Rust 🙌

                                                                At home: Last week I had a sudden inspiration for a simple game, which would be a fun iPhone/Android app… and since then I’ve got a working prototype that my kids really enjoy playing and have been helping me to refine. Sooo… maybe launching my first phone app game soon? I’m 99.9999% sure if I do I’ll just make it $0.99 and have no ads whatsoever, maybe with a free version that just has a smaller number of levels. I haaaaaate advertising.

                                                                And my favorite part about the game is that as a puzzle game, it’s basically teaching the user reverse-engineering.

                                                                1. 3

                                                                  If you release a free version with a small number of levels, maybe think of way to skip those levels or track the progress of the user once he pay for the real version. It might be boring to re-do puzzles just to get back at where you were at.

                                                                  Edit: It’s really nice that have a nice prototype that your kids seems to enjoy :)! Was it hard to learn how to do the app? Where you familiar with making games before?

                                                                  1. 2

                                                                    That’s a good idea about skipping levels!

                                                                    I’ve never professionally made a game before, and never made a mobile app… so it’ll be interesting for sure. I can easily see falling down the rabbit hole of features like: achievements, preferences, tracking user metrics & engagement, etc.

                                                                    One of the features I think would be killer is auto-generated levels, which really raises the importance of testing & reliability so that for example users don’t get stuck.

                                                                2. 3

                                                                  For work: We’re trying to finishing up a release but we are struggling on the organization of our work, so we’re not outputting a lot of work.

                                                                  For personal project: Still working on PDB. The Groovy changes are finished but I did not have the expected results. So now I’m learning Vue.js to rewrite again the view layer. Hopefully, this will be a great choice and will help speed up the project. So far, learning Vue.js has been fun :) I’ll probably try it with Typescript, but this is not a pressing matter as much as just getting new features at some point.

                                                                  1. 3

                                                                    I’m spending the first part of the week implementing the circuit breaker pattern into our system so we’re hopefully a bit less susceptible to cascading failures between our services. Then I’ll move on to writing API documentation and fiddling with our Oauth2 / PKCE implementation, I think.

                                                                    1. 3

                                                                      I finished the CFP milestone for Moot. I’m moving on to the blind editing workflow next.

                                                                      Day job is a data-heavy medical client, interesting stuff that I can say almost nothing about unfortunately.

                                                                      Hopefully I will have more to talk about soon.

                                                                      1. 3

                                                                        I’m wrapping up a new model-driven test harness for an upcoming Wallaroo release and working with the rest of the team on ironing out the last few remaining bugs that we’ve found.

                                                                        1. 3

                                                                          Working on a small go library that uses different compression algorithms for compressing a stream of data. In the end it will be integrated into a sidecar (on Kubernetes) to proxy some traffic for internal applications.

                                                                          1. 3

                                                                            We have discovered a pathetic bug in Kafka’s handling of Avro encoded data, that is essentially unfixable without reworking vast swathes of unrelated codes. So, that’s fun! We get to put in a disgusting workaround and try to maintain, with a straight face, that our data backbone is reliable and trustworthy.

                                                                            Swutting software, Ford!

                                                                            1. 3

                                                                              Allergies, burnout, and mental health.

                                                                              1. 2

                                                                                Working through some PLT texts, looking for a msc theme in lightweight ormal methods

                                                                                1. 2

                                                                                  At work I’ve joined the (small) bug triage team, so most of my time has been dedicated to tracking down root cause on bugs, and then either fixing them or handing them off to a SME to fix. Some people don’t like it, but I enjoy the challenge, and I get to learn parts of the system I haven’t had to deal with before, so I don’t mind it at all.

                                                                                  Outside of work I’m still figuring out how to scan barcodes. I’m having fun trying to figure it out myself, but at some point I may just create a binding to ZBar and be done with it.

                                                                                  1. 2

                                                                                    I just finished releasing my first browser extension! I wrote typescript for the first time.

                                                                                    Check it out: Twitmarks - Twitter bookmarks for the web https://github.com/geekodour/twitmarks

                                                                                    1. 1

                                                                                      Working flat out to get through this week with a silly amount of deadlines, partially due to not knowing what was coming down the pipeline and partially due to things going sideways as customers come back to work after the summer and use the product.

                                                                                      Outside of that? Not sure. I have next week off as I’m out the country though. That’s going to be fun. 🇵🇱