1. 24

This is the weekly thread to discuss what you have done recently and are working on this week.

Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.

  1.  

  2. 12

    At home, I’ve been optimising my compression algorithm.

    The encoder hasn’t received much attention yet, but I found a very nice trick for optimising one small part and it doubled the speed of the whole thing, putting it on par with LZ4. With tweaking it should beat LZ4 quite handily.

    The decoder is where I’ve been focusing my efforts, and it destroys anything in the open source world. I’ve seen 60% faster with the same or a bit better compression than LZ4 on normal non-degenerate files, and I still have work to do. There’s definitely room for tweaking, and I want to try widening to AVX2. AVX is a bit derp though so I’m not sure if that will work out.

    This is all particularly exciting because LZ4 has the world’s fastest practical LZ encoder (ruling out things like density) and I believe I’m in the same ballpark as the world’s fastest decoder (but with much less compression :<).

    Next steps are to become #1 and figure out how to turn it into money.

    1. 10

      I believe there’s a ‘documentary’ called Silicon Valley that can be a good starting guide.

      1. 2

        I think they even had a real-world, case study toward the end involving life-like hands and cylindrical objects. Might be useful for bicycle games or something.

      2. 2

        Good luck with your project!

        I’m observing compression-related subjects from the sidelines occasionally. (Lately I started providing some moderately recent Windows binaries for some compression-related tools, that are not widely available.)

        Are you perhaps unaware of encode.ru forum? You can look there for some projects that were based on LZ4, like LZ4X or LZ5, and a lot of other interesting stuff, like promising RAZOR - strong LZ-based archiver for instance. You’ll find there knowledgeable people from the field and their insightful posts.

        1. 1

          Giving RAD a run for their money eh? :)

          1. 1

            Sort of. I’m trying to avoid taking them head on because I won’t win that. AFAIK they don’t put much effort into having super fast encoders, which leaves room for me to focus on end to end latency.

        2. 8

          Avoiding commercial work by rallying as many early pledges for a crowdfunding campaign I’m attempting tomorrow – to fund a piece of free software :) Likelihood of success is probably under 1%, but I’ve never seen a better project to try it out with, so going to give it a whirl.

          I hadn’t expected quite so much upfront planning to be required. It’s nice to say it’s “risk free”, but in reality, the danger of overcommitting when there is actual money involved, that is immensely stressful, and just the planning alone has been a significant sunk cost. Whether it works out or not, I’m not sure I’d try again.

          1. 8

            Preparing release 1.4.0 of greatest: (Edit: it’s posted.)

            • Add a flag to call abort() on test failure, so if tests are being run inside gdb, it will break on the failure with the call stack and other runtime info.

            • Add a function to set optional test name suffixes, so if a test is being run with parameter(s), it can print test results as the_test_ABC, the_test_DEF, etc. This is particularly useful when tests are being run in shuffled order. The test-name-based filtering includes this optional suffix in its matching.

            • Several other misc. improvements, especially work on the documentation.

            After that’s out, I’m going to work on a new release for heatshrink. It’s essentially done, and has been stable for years, but it wouldn’t hurt to flesh out the documentation more. Also, it’d give the project some recent activity – people seem to consider projects that haven’t had changes in 3 years stale, even when they’re really tightly scoped and feature complete.

            Strangest request: Someone connecting with me on LikedIn specifically to ask me to write a C# wrapper…Unlikely, though I will probably add blocking, bitstream-compatible implementations in C and Python to make porting easier. The majority of heatshrink’s complexity comes from being able to suspend and resume on IO, between any bytes.

            I will probably be back to working on theft again in a week or two – I have a massive restructuring underway to add multicore support. Since it interacts with nearly everything, there has been a ton of integration work, but it’s starting to come together.

            Work: Wrapping up loose ends so that we can start something really exciting soon. Also, working on a couple internal presentations about recent successes.

            1. 2

              The majority of heatshrink’s complexity comes from being able to suspend and resume on IO, between any bytes.

              Been a while since I heard someone mention that feature. Might be worth a write-up or detailed comment on how what makes it complex or how it’s done.

              1. 9

                It implements LZSS, one of the LZ77 family of data compression algorithms. Everything is based on a state machine, with an internal buffer for the data being processed. I chose LZSS because it does all of its processing using that buffer (the “sliding window”), so its CPU time and memory usage is very predictable.

                As a friend described the API: “you put in data, then turn the crank until nothing more comes out, repeat.” When you’re done, you call a function to notify it the end of the stream has been reached, and keep turning the crank until processing is finished. It isn’t that surprising, since the point of compressing / decompressing is to change the size of the data, but the assumption that the caller will keep calling to sink more data from a stream means that it can check its current state / where it left off, if there’s enough data buffered to move on to the next step, and how much room is left in the output buffer.

                I’m not saying it’s incredibly complicated, but a blocking, single-pass LZSS implementation is like a page of code; the suspend/resume logic is well over half of heatshrink, line-wise. There are also optimizations for doing decompression or compression on memory-constrained embedded systems – for example, the indexing in the encoder is simple, but either significantly faster or uses an order of magnitude less memory than other approaches I’ve seen. If you’re unpacking data out of ROM on a system with only a few hundred bytes of RAM to spare, approaches like gzip’s big in-memory hash table are a non-starter.

                1. 1

                  Appreciate the detailed description. That’s very interesting. :)

            2. 7

              This week I start my new (remote) job - super excited about it, and I hope it’s a good learning experience! I’m curious how the on-boarding process will go since I am waiting on my machine to arrive still.

              1. 4

                Congrats, I hope it works out for you. I’ve been primarily remote (with a few on-site trips) for 9 years now, and couldn’t imagine going back to working in a ‘regular’ office.

              2. 6

                Back to a hobby project I started long ago and restarted again recently: a time series datastore in Go using gRPC.

                So far have it up and running with a library I wrote previously for a fast ordered datatype. Watched a presentation about best practices for gRPC since the docs and examples seemed so thin and wound up finding out that yes, I was indeed doing it wrong. Rewrote my .proto and simplified my error handling late last week and switched over to the delightful logrus for logging.

                This week, when time allows, I will actually port over the TSDB in-memory format from Gorilla and add more structured input and aggregation queries for output. Next milestone after that is adding continuous queries & downsampling. I hope to have all that done this month if the weather stays crappy.

                1. 2

                  Ah Logrus! Slightly OT, but why did you opt for logrus over zap?

                  1. 2

                    Logrus (sorry for the bad link in my post btw) is older and popular, so it’s a great starting point for most projects that need leveled logging. It is pluggable as well, so it’s easy for me to add a configuration item later that will change the log output for my app to logstash, fluentd, JSON, etc without any extra work for me. It also defaults to a nice format when a TTY is detected, so debugging with colored levels and pretty offset times is another win for me.

                    Zap is great; it’s very fast and I would consider it for any application that wants to output structured data quickly (not just logs!). As always, though, there’s a tradeoff in flexibility for that speed. From the docs:

                    The zap package itself is a relatively thin wrapper around the interfaces in go.uber.org/zap/zapcore. Extending zap to support a new encoding (e.g., BSON), a new log sink (e.g., Kafka), or something more exotic (perhaps an exception aggregation service, like Sentry or Rollbar) typically requires implementing the zapcore.Encoder, zapcore.WriteSyncer, or zapcore.Core interfaces.

                    That tipped the balance for me; I wanted to support a few common format options and Logrus makes that super simple. My app doesn’t write structured data often enough to warrant that ease-of-use tradeoff.

                    1. 3

                      Cheers for your insights, nice to see why you opted for Logrus.

                      I mainly asked as I’ve been burnt by logrus before and managed to subsequently piss off half the Go community whilst doing so. Thanks!

                      EDIT: However it was not Logrus’ fault - but mostly Go’s lack of successful package management.

                      1. 2

                        I’ve been on the pissing off whole communities side before – hey at least it helps instill changes sometimes.

                2. 6

                  Hacking on Mono trampolines for PPC, by first studying how they work on z - the assembly there is actually pretty easy to read and understand by hand.

                  1. 6

                    We’ve finalized the speaker list for Heartifacts, a mental health and community building conference by Code & Supply† I’m organizing here in Pittsburgh. We’ll be announcing speakers this week once all have ACK’d. I’m excited about the line-up!

                    Recently, I made a few tweaks to homebrew-bundle to improve its usability and speed.

                    This week, I need to send my work Mac in for repairs, so I’m going to spend some time honing my home directory management stuff to more cleanly work on my personal Mac, which I’m subbing in for the duration that the work Mac is in the shop. It’s using an older, non-versioned version of my home directory stuff and is sorely out of date compared to the hotness on the work Mac.

                    group behind Abstractions and Uptime

                    1. 6

                      Trying to become a more organized and productive person. :)

                      Concretely, this involves:

                      • Blocking time-sucking sites like Twitter. [1]
                      • Each night choosing my tasks for the following day based on a few specific inputs
                        • A project list which I keep in my planner
                        • Any upcoming events. (Which again, should be in my planner)
                        • Anything that came up during the day which needs action soon
                      • Deciding which 3-4 tasks are the most important
                      • Assigning each important task a specific block of time
                      • If it appears that more time is available, blocking out time for a few more things
                      • Setting a visual timer at the beginning of each work block

                      This stuff may sound pretty simple, but it’s been pretty revolutionary for me. I’ve alway struggled with time awareness, but now I’m actually starting to get things done quickly and predictably. This curriculum has been amazingly helpful.

                      [1] Obviously I don’t consider Lobste.rs a time suck!

                      1. 5

                        Lots of hacking this week as I was forced to take some vacation. What better way is there than to spend it coding on my own projects? The big item will be posted to lobste.rs - but I also plan to start the revival of my experimental visual debugging / reverse engineering toolkit, Senseye (https://github.com/letoram/senseye/wiki) If you are interested in the domain - make sure to look at the voyage of the reverser presentation from BH2010.

                        1. 5

                          Around five years ago while I was still studying, and still very green as a developer I started, but never finished an iPhone client for Lobsters. For whatever reason I’ve been messing around with that idea again over the past few evenings, and I already have a nice little alpha version I’ve been using on my commute to and from work. I’m building it in React Native this time, which I’ve been using for a while now and mostly loving despite more than a few warts.

                          1. 2

                            I had thought about trying that for Android, but the web interface seems to do everything I would want to do anyways. Did you come up with something that an iPhone client can do that the web client can’t?

                            1. 1

                              Mostly I just had a few UI ideas I wanted to play around with, but there’s a few things I’d like out of a native app::

                              • Ease of sharing links to other apps so I can quickly send links I want to read to Instapaper / Pinboard, maybe behind a swipe gesture or something
                              • Tighter control over my reading experience with support for night mode and the ability to show links in Safari’s reader mode by default
                              • A slightly more touch optimised UI. Lobsters actually does a really good job of this for the most part, but there’s some limitations to what you can do on the web versus in a native app.
                          2. 5

                            I’m collecting ideas after my talk to the seminar Myths and reality of artificial intelligence – Theoretical issues and practical developments.

                            I had completely no idea of the bright people I was going to talk (and debate) with. A wonderful experience.

                            I look forward for their slides and the videos. My own slides are here.

                            Also, talking with audience after the seminar, I realized that the definition of intelligence that I proposed (slide 20 and 21) deserves a little more space, so I promised to write a more in-depth analysis about its advantages and its limits.

                            1. 5

                              This week is about Job hunting, and catching up on my Return To Teaching course.

                              Anyone needs I a remote systems administrator / technical support team leader - then I might be of some use 8~)

                              1. 4

                                Work:

                                • Improving the integrations to the PMSs we support to offer more data depth (rather than more PMSs)

                                Other:

                                • Reading more often now, reading The Checklist Manifesto right now, which is an easy read if you’re OK with not understanding 100% all the medical terms
                                • Starting to learn a bit more Java, gotta get used to the tooling difference between Java and Node
                                • New blog post! Just a short note on adblockers https://greduan.com/blog/2018-03-03-adblockers.html
                                • Small work on my RoarSS project, right now it’s got the barebones functionality. Just published it since some people are probably interested, I’ll probably keep spending time on it from time to time to make it more useful. Fair warning, I don’t use it day to day right now. :) https://github.com/greduan/roarss
                                • Taxes! Dutch taxes, it turns out, I am not required to report them for 2017, unless I expect to get money back, in which case I have up to 5 years. Swiss taxes are all about gathering the paperwork..
                                1. 4

                                  This week I’m tying up all lose ends ready to start my new role next week!

                                  Although this organisation is a shambles, I want to make sure no future developer is left handicapped due to my lack of diligence.

                                  1. 4

                                    I’ve been working on a bot for the CS/Math/Stats discord server at my university. I want this bot to be an introductory project for 1st and 2nd years to learn about contributing to an open source project. I spent most of the week refactoring it to be as easy as possible to contribute a module to. Then I wrote up a nice and detailed contributing doc as well as started creating, tagging and flushing out tickets that I’d been thinking about but hadn’t written down.

                                    This week I want to add a small tutorial for getting started as well as well as some more beginner friendly stories (and tagging them as such). After that I’ll be posting the link in the server and really encouraging the regulars to take a crack at contributing something to the bot.

                                    1. 4

                                      This week I’m doing tons of stuff on 44CON, in particular I’m seeing another training venue this week, I’m reorganising past content across a bunch of different channels and I’m hoping to do some work on a project for people to do at the event.

                                      Aside from that I’m doing a bunch of backend work on a cryptocurrency-related side project that’ll hopefully soft launch next week or the week after, and having spent some time prepping will be working on some reverse engineering tutorials for a training thing I’m setting up over the next few months.

                                      I’ve also decided I need to take a bit more time out to focus on building my strength back after some time out due to heart problems. I’m hoping for a physio referral from the cardiologist, but won’t hold my breath.

                                      1. 4

                                        Just put in my 2 weeks notice at my current job, moving out to move up :) I’ve got a week off between this job and the next one where I plan to do some reading on ElasticSearch and TensorFlow (along with renovating the in-laws house). This week will mostly be finishing up any outstanding tasks and saying goodbye to everyone.

                                        1. 3

                                          Working on “improving TCP performance” in Wallaroo. Its the first step in my project to improve the throughputs folks can achieve when scaling horizontally.

                                          1. 3

                                            Working on replacing zerorpc with plain ol’ TCP sockets in my Raft implementation in Python. zerorpc was super easy to get started with, but the message queuing and heartbeats really aren’t necessary in Raft.

                                            1. 2

                                              Last week I did a little bit of work on the wiki (I’m now storing edit history, though I don’t have a UI for using old versions yet), and did a little more learning about Godot. I also did a little bit of character art, but nothing major.

                                              This week is my first sprint at work. They rotate developers between sprinting and maintenance as the needs on either side suggest, due to the timing of my arrival, I’ve been on maintenance until now.

                                              1. 2

                                                Life got in the way the last two weeks, so I’ve made little progress on what I said I’d do two weeks ago https://lobste.rs/s/rptmkp/what_are_you_working_on_this_week#c_309kpj

                                                Going to resume learning by going through Mark Myers’ book on Javascript: https://www.amazon.com/dp/B00H1W9I6C

                                                Also going to pause pod/screencasting at least this week.

                                                1. 2

                                                  Inspired by a comment in last week’s post (or was it before that?) , I sat down and thought about how I would implement a backup system, so I wrote some toy code using the restic chunker and a bit of custom serialization.

                                                  I also setup a status page for my server using cachet on a shared hoster. That way I can monitor and respond to incidents more transparently. I want to add a ICMP ping module to my monitor addon since the HTTP plugin captures a 1000~ ms response from the host login page.

                                                  I might also do some progress on my SECL parser, the runtime context needs a CoW implementation so I can create locally scoped variables, functions and macros. I might also look into expanding it as a HTML templating language by ripping out most of the later parsing phases. Or alternatively I’ll just blow some memory and make full copies. Performance is good enough that I don’t care.

                                                  Lastly, I have backup sync on my radar, the backups on my server are still manually synced whenever I find time, I’m looking at using the OVH Archive Object Storage, which seems cheap enough for my purposes (and a bit cheaper than Backblaze B2, so I might do a full switch)

                                                  1. 2

                                                    Just started a project to prototype a small checklist app. Decided that for lulz I’d go with the unknown pain of learning JavaFX rather than the known pain of Electron, and constantly fighting CSS and JavaScript build systems. TBD wether or not that was a mistake :D

                                                    1. 1

                                                      A slack export viewer in Haskell and Reflex.