1. 33

This is the weekly thread to discuss what you have done recently and are working on this week.

Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.

  1. 20

    Job searching, and I could use a hand. I’ve been working a part time a few years and my boss gave me an ultimatum to move to FT. As long as I have to change over, I’m looking around, but the job sites are generally pretty terrible. If you know someone who’s good to work with, please copy and paste this to them:

    Peter Bhat Harkins is a senior web developer looking for more work in Rails or Python, or very interested to move towards either end of the spectrum of computing abstractions and work in Rust or Haskell. He’s been coding professionally for 15 years, hobby programming for 10 more. He’s spoken at RailsConf and other tech conferences, and has written a book on programming. He’s looking for smart, happy people to make useful things with on the web or in a network setting. He enjoys being tech lead and mentoring junior devs but is not on a management track. Chicago or remote. ph@push.cx - http://malaprop.org

    1. 16

      Getting settled in Silicon Valley! I start full-time at Cisco next Tuesday. I think I’ll go for a bike ride today and get some fresh air after the stress of buying a car yesterday.

      1. 1

        Scratch that - mom wanted me to visit. I’ll ride my bike tomorrow. :-)

      2. 11

        Most of my effort this week is going to be spent tying up pre-“launch” loose ends on PDFDATA.io, our PDF data extraction as-a-service. I wrote about it previously here.

        Just yesterday, I flipped the bit to make the initial pricing public, based on feedback from our initial customers. It’s only a first cut, and pricing is always fraught, but what’s there now will do for at least a little while.

        I’ve been idly investigating alternative distribution channels like offering PDFDATA.io via AWS Marketplace, Heroku add-ons, and similar. It’s hard to tell just how viable any of them are for very specialized services like this; nearly all of the obviously-successful offerings in those venues are general purpose stuff like databases, operations stuffs, CMSs, and so on. I’d love to hear from anyone that’s done well in those sorts of channels with something that is domain-specific to some extent.

        On the technical side, I’ll be smoothing over the rough edges on the shape of the actual API, i.e. what requests and responses look like, making sure things are set up to make the rest of the initial roadmap easy to integrate, etc. It’s Yet Another JSON API, so I blew a half a day properly reading up on JSON-LD before coming to my senses and rage-closing all of the corresponding browser tabs. As much as I’d like to have a “proper” schema for something like this, nothing I’ve seen around JSON has even close to a proportional payoff (either for us or users) given the immaturity and structural problems. If I were shipping around XML, then publishing a schema would be an obvious win, but XML just isn’t culturally acceptable at this point in most of the HTTP API market AFAICT.

        What I am doing is applying prismatic schema judiciously at API and certain backend boundaries to make sure that data going back and forth is “valid”. That’s not a big deal, but I do want to figure out how best to flush those schemas out to something suitable for (automated) inclusion in the slate docs. There’s a bunch of additional tooling that the community has built up around schema that I haven’t explored yet, and I’m hoping to find a gem I can build on there.

        1. 9

          I was trying to look for resources on “real time file parsing”, but I don’t know the true terminology (I suspect there’s one! “incremental parsing” seems to be it, although i can’t find more formal stuff).

          I wanted to know of the techniques that can be used to parse a file, hold the AST, and update the AST as the file is updated as well (content can be added, or removed, or inserted at the middle of the file). A poor man’s implementation could involve some filediff library (can libgit handle this?).

          1. 7

            Are you familiar with the concept of a zipper?

            1. 4

              any other explanation of what a zipper is. My understand of it is quite fuzzy

              1. 4

                Yeah, that explanation isn’t great. At its core, a zipper is a data structure for looking into and moving around another data structure. It essentially works by separating the “focus” (the part of the original data structure at which you are currently looking), and the “context” (the rest of the data structure).

                As a simple example, if you have a list like [1, 2, 3, 4, 5, 6], the zipper of this list would look like [[2, 1], 3, [4, 5, 6]]. The single element (‘3’) is the “focus,” and the two lists (the first one reversed, for efficiency) are the “context.” You can then define functions to move this zipper to the left ([[1], 2 [3, 4, 5, 6]]) or the right ([3, 2, 1], 4, [5, 6]).

                This notion of the zipper can be generalized to other data structure (a tree is a common example). In every case, the idea is how to efficiently walk through and view the innards of a particular data structure, particularly across repeated operations. If you want to operate on a segment of a tree, it’s easier to get a zipper of that tree with the focus on the desirable section, and then operate on that focus, than it is to repeatedly find the important part. In a functional context, the zipper facilitates this.

                Huet’s original paper on this is quite good, I think.

                1. 4

                  Superb explanation, thank you for making this so crystal clear.

                  1. 3

                    Thanks! I’m glad it helped! I had similar issues understanding zippers when I first learned about them. I really think introducing them with lists is easier than introducing them with trees, which is what most tutorials I’ve found do.

                  2. 1

                    That’s super and thanks a lot for the link to the orginal paper. When you say to move around another data structure, I suppose you mean without modifying the original one ? An I suppose it include create a copy modified of the data structure without modifying the original one

              2. 4

                “incremental parsing” generally refers to the idea of your parser being able to wait for input and/or be given more input.

                Say if you are writing a parser for c and your input so far has been

                struct foo {
                    int a;

                your parser can ‘halt’ and ‘wait for more input’ as it can clearly see this struct isn’t finished, this isn’t a complete program specification yet.

                However, to the best of my knowledge, “incremental parsing” doesn’t include the ability to modify previously parsed things - that is it is only additive and doesn’t allow for removal or modification.

                If you find a solution on this or a term to encapsulate it, please share, I would be very interested!

              3. 8
                • Maybe, just maybe, my work version of LLVM will be able to produce object files for mainframe systems by Friday.
                • Attempt to builld Clasp as the first step to actually doing something with it.
                • Thanks to prodding by BruceM, work through a Rust tutorial or two.
                • Work on a Gnus backend for browsing my papers database.
                1. 6

                  It’s a quiet week of revision again. My exams last week (biology and compilers) both went well, and now I’m revising for ‘chip multiprocessors’ (cache coherency, transactional memory, GPGPU computation etc), ‘Advanced Algorithms II’ (numerical stability/computation accuracy, nature inspired algorithms and complex network theory) and ‘AI & Games’ (game theory, mechanism design/macro economics). No matter how tedious revision gets, I can’t complain that the content is dull!

                  I’ve decided that I really want to write a compiler after my exams (inspired by the course I took). I think I’d start by working with simple arithmetic expressions, but aim build a complete toolchain to do it (lexer -> parser -> semantic analyser? -> IR -> optimiser -> machine code), and then maybe flesh it out with stuff like flow control etc later. This would also be good to get be programming again after a month or two of pretty much solid theory work!

                  1. 6

                    Finally got out to the mountain bike trails yesterday, had an awesome time.

                    This week doing OEC training Mon and Wed, going out to dinner Tues with my great-uncle who fought in the 1956 Hungarian revolution. Going shooting Thurs.

                    Busy week :)

                    1. 6

                      At client demand, continuing to learn node.js and javascript. It really reminds me a lot of tcl; you can palpably feel that design parameter (and ceiling) for the language was making scrolling tickers work, and that everything since then has been a loosely coordinated, patchwork triage under the effects of stockholm syndrome. It seems like it could be possible to be productive in the language and use it to solve problems on the server side, but the mountain is tall and looms.

                      On the personal front, I read Marie Kondo’s book (the lifechanging magic of tidying up), and had a strong epiphany. So I’ve filled five trashbags with clothes that no longer give me joy and am moving forward on a plan of radical possession reduction. As I head into my late 40s, it’s seeming more and more important not to have to fight through clutter and noise just to live; so many of my purchases in life have been to fill holes that can’t be filled, or soothe wounds that can’t be soothed, and now I’m surrounded by ineffective bandages. The next few decades must be different.

                      1. 5

                        Common Lisp: reading both ANSI Common Lisp and Practical Common Lisp, getting to understand the package management system. I like to jump in with a practical project, and my first project will be to write a reader for the 2bit DNA data format. This seems to be covered by the Parsing binary files chapter of PCL. This will be fun because it dives into Macros.

                        1. 5

                          If you’re looking to learn about packages in CL, I recommend The Complete Idiot’s Guide to Common Lisp Packages. Also, unlearn about packages from other languages. CL ones are… not the same. :)

                          Edit: also, if you can get your hands on a copy of Edi Weitz’s book, it won’t hurt, but it may be a bit advanced if you’re just starting out.

                          1. 2

                            The Guide to CL packages is good, but I think @kghose means ASDF systems and not CL packages

                            1. 1

                              Hi @geoffwozniak and @puercopop - thanks for the input! The links from @GeoffWozniak are very interesting, because of Lisp’s interesting concepts of namespace. It’s true that though as a “consumer” I’m benefiting from the automation of ASDF and Quicklisp, it is important for me to understand these core concepts too!

                          2. 5

                            If by package management system you mean the equivalent of npm, pip, etc. I’ll give you a quick rundown, in CL there are 3 different things:

                            1. ASDF: A system groups a collection of files. It handles which other systems have to be loaded before loading this system and in what order are the files in the system have to be loaded. It also lets you define other operations like test-system. 2 Quicklisp: It fetches ASDF systems from the internet. It uses leverages ASDF.
                            2. Packages: The equivalent of modules

                            Keep in mind the name of the asdf system doesn’t always correspond to the package, take for example Conspack, the name of the system is cl-conspack but of the package is conspack.

                            Although ASDF has an extensive manual, most people cargo cult it so I’ll give you a minimal ‘template’

                            ;; You'll see other people define an ad-hoc package for the system, this was from the time before ASDF provided the package asdf-user
                            (in-package #:asdf-user)
                            (defsystem #:my-system ; Can also be a string
                              :depends-on (#:cl-store) ; dependencies
                              :pathname "src/" ; This is optional, if you want to place the files in a different directory that the asdf definition
                              :components ((:file "package") ; Other
                                           (:file "utils")
                                           (:file "main")))

                            To use Quicklisp one has to run (ql:quickload <system-name>) in the REPL, if you want to make available a system (or system version/fork) not included in quicklisp, just include it in ~/quicklisp/local-projects/. Sometimes you’ll have to run (ql:register-local-projects) after copying the system to local-projects. You can search the systems in Quicklisp through (ql:system-apropos <search-term>)

                            For packages @GeoffWozniak’s link has you covered

                          3. 5

                            #ProjectBMW is back on the road, discovered not only was it leaking coolant from the thermostat housing but also from the expansion tank (which is part of the radiator on these cars). It’s still weeping a couple of drops from the expansion tank so it’ll need a new radiator, but not immediately which means I can drive it finally. So nice having a convertible again.

                            #ProjectBoat was unwrapped from winter storage this last weekend. It’s currently taking up the hardstanding I need to work on #ProjectBMW some more, so it needs fettling, hauling to the sailing club & launching for the season. This week is pretty much sorting out the small amount of woodwork it has, including fixing the mast step back to the hull from where it’s come adrift, and then working out how the hell the rigging goes together. (I never sailed it before it was bequeathed to me, so I don’t know exactly how it’s rigged. Going to be a fun problem, luckily there’s a couple of other flying fifteen’s at the club so I should be able to get advice on it there.)

                            Bit the bullet and reinstalled my physical server with SmartOS, still needs the routing for zones sorting out & IPv6 sorting on the external interface. Debating whether to order a /29 for it or continue with NAT behind just the single IP for all zones.

                            And finally at work it’s continuing to learn how to use ElasticSearch properly with parent/child documents for what we want to achieve. Been reading ElasticSearch Server (3rd Edition) and enjoying finding out more about features in ES and how to model the data slightly better than “here index this JSON object and let me query it later”.

                            1. 5

                              I am working on a side project called Nightreads - Github, MIT.

                              It’s basically a newsletter management kinda app. I use it to send emails.

                              Bit of back story: So I maintain a Slack community of Indian Programmers and we are close to 700 members now. We had a member called gnu who used to post interesting links every night. He suddenly disappeared from the community, no one knows the reason. But I wanted to continue the tradition and instead of sharing links in Slack, I am sending emails, so that it can reach wider audience. So people will receive one link every day. People also can choose their interests like python, law or security and receive emails on those topics only.

                              The app is almost done and will go live soon :)

                              If you want see the basic demo, here it is - demo. This is a demo app and you won’t be subscribed if you enter your email.

                              Since its basic, I am mostly leveraging features of Django Admin and learned some new things like, adding a custom view to admin panel etc.

                              1. 5

                                Starting a vacation this week, by first dragging my family along to LambdaConf, which I’m super excited about! We have friends in Boulder, so we won’t be taking part in any of the after hours festivities, but if you’re going to be around, look for the guy with the beard, and the Heroku shirt, with the really long last name that starts with a G.

                                And, I’m also giving a lightning talk, so I should probably prepare for that…

                                1. 4
                                  • Moving! Going from the muggy bayous of Louisiana to the mountains of New Hampshire! :-)
                                  • Preparing my coworkers to live without me for a week and a half.
                                  • Converting a short story into a nice mobi/epub edition.
                                  1. 4

                                    Home hosting: I will setup my new (5th generation) Intel NUC into a little home hypervisor to host fun stuff like a GitLab server and a Plex server. Very excited! It’s my first M.2 host. I can’t believe how physically tiny those SSDs are.

                                    Web apps: I’ll be starting to write a couple small web apps for the radio station I used to run. The current web technologies in use are Flask and Laravel, with a history of Code Igniter and Rails. I might stick to Flask because it’s fun and reliable, or I might finally learn Scala. Either way, I’ll probably go with Postgres this time (everything has been MySQL/MariaDB until this point).

                                    Books: And most important of all, I need to find a copy of The Restaurant At The End Of The Universe. I finished Hitchhiker’s Guide last weekend and I haven’t stopped thinking about it since.

                                    1. 2

                                      Hey, that sounds like fun!

                                      Those M.2 SSDs are so tiny. And so fast. I have 256GB one in my Chromebook and while this is completely anectdotal, it certain feels faster.

                                    2. 4

                                      My side project is an interface for comparing historical photographs of Washington, D.C. with Google Street View images. If you have ties to DC, you might find it interesting - Wymer’s DC. I would appreciate any ideas you might have to increase the usability of the interface (or any other suggestions).

                                      In my real job, I’m working on input validation. Sometimes people use the name of the organism (e.g. a bacterium) in various fields where it’s wrong (e.g. ‘strain’, which should be some alphanumeric identifier). Teaching the submission system to notice that.

                                      1. 1

                                        Oh - it works best in Chrome - Street View has a horrible bug in Firefox. :(

                                      2. 3

                                        Going to continue working on my Crystal OpenGL scengraph renderer. Made a bit of progress this week (as in something shows up on screen now), but there’s still a lot of work and optimizations to do. Once it’s all fairly usable, I’ll probably split out the C code that’s wrapping a lot of OpenGL functionality so it can be used as an independent library.

                                        1. 3

                                          We shipped the first release candidate for PureScript 0.9 yesterday, which means I’ll be spending time getting things like libraries, blog posts and tools updated, as well as fixing bugs. Hopefully we can have a stable release out by the end of the week. With the release looking more complete, I’m also looking forward to getting back to implementing interesting type system features. I started with programmable type errors yesterday, and I’m thinking about constraint kinds and a variant of functional dependencies as the next things to tackle.

                                          1. 3

                                            I’m building a healthcare analytics system aggregating disease/diagnosis, prescriptions/procedures, and patient demographics over a large population of (anonymized) health insurance claims. Its not going to un-fuck the american healthcare system, but hey - more data shouldn’t hurt either.

                                            1. 1

                                              Have you looked at the stuff Docgraph has?

                                              1. 1


                                                No I didn’t know about that! thanks for the link. I’ve actually used many of the same reference/source datasets, though.

                                                1. 1

                                                  Get in touch with them–they’re pretty friendly.

                                            2. 3

                                              Training a stacked denoising autoencoder to fix compression artifacts in video

                                              1. [Comment removed by author]

                                                1. 1

                                                  It’s not yet :). Ask me again at the end of the week.

                                                  I think the hard part is going to be stopping it from generating its own more annoying artifacts (think: puppyslugs). I have some ideas for ways to mitigate that but it’ll take some experimenting.

                                              2. 3

                                                Had a setback with broadband provision at my new house. EE, my chosen provider, contacted me after their engineer visited this morning, to say they could not actually provide the Fibre broadband they had sold me. They were very apologetic and initially offered unlimited regular broadband at no cost (just line rental) as compensation, but when checking they realised that they could not actually do that either. (The best they could do was £40/month with 20GB/month limit—utterly untenable in a household with kids addicted to streaming TV.) The bloke I spoke to then gave me a cash compensation for my wasted time & recommended I go with BT instead—according to him BT owns the exchange and is the only provider that will give me unlimited broadband through it. (This could have something to do with it being fairly rural where I live.)

                                                Continuing to sand down walls & windowsills in my living room in preparation for a paint job. Incredibly dirty job, but mostly done now. The paint has all arrived, so hoping to get a start on painting at least the ceiling this week. I chickened out from buying the belt-sander I had in mind and bought a random orbital one instead, as the reviews I read said these are a little easier to use. Not too happy with it as it clogs up quickly and vibrates tremendously—so not comfortable to use for a long time. I also bought a detail sander for the corners. This seems to do a fairly decent job at larger areas too, so I think I should have gone with the belt sander for the initial paint stripping, and used the detail one for the finer work.

                                                Counting myself lucky, once again, to live in a country with a decent national health service. 4yo son’s fever spiked to 40.3C and he started throwing up Sunday night at 7pm, so we called the NHS out-of-hours GP helpline and they booked us in for an appointment at the local hospital at 9:20pm. Got there in time, waited for about 15 minutes, and got to see a doctor. We hung around for an hour or so for observation and the doctor told us what to expect (diarrhoea—which he gave us nappies to cope with, since no shops were open at that time of night), and we were out at about 10:40pm. The UK is one of there countries I have lived in, the others being my native Norway and Hong Kong, all with free (at the point of use) health service. Now that I have a family to look after I don’t think I would even consider living anywhere without this basic safety net. (Yes—I recognise that I am incredibly privileged to be able to make this choice.)

                                                Re: work I’ve been off ill today as I managed to catch what my son had yesterday—albeit less dramatically. Nothing on fire at the moment so assuming I make it to work this week probably will just fast-tracking some low-hanging-fruit tickets to try to tackle our rather overflowing list of things to do. Also some prepping for our bi-monthly meetup in Germany next week. (Our entire tech team works remotely.)

                                                1. 2

                                                  More NuGet/python for work. At home, I’ve upgraded from Linux to FreeBSD on my VPS, and spent the evening turning on 2FA for all the services I use that support it. It’s unpleasant, but necessary, much the same as my quarterly password change exercise.

                                                  I’m also trying to install FreeBSD on bare metal, with the added … confusion? idiocy? brave act of naïvety? that I need to have root be a ZFS pool, and use UEFI so I can dual-boot to Windows without jumping through ASUS BIOS madness. I don’t know why I’m doing this.

                                                  1. 2

                                                    I am foolishly building a short messaging social network a la App.net or Twitter. Have the basics done. Need to document it, test, and let somebody in. I want to rebuild in Rust over the years, tackling pieces of the API incrementally. It’s a hobby project, with no intention of becoming a professional outfit.

                                                    Oh, and feeding the bees. They’re doing great. Love working with them. Italians are very calm.

                                                    1. 2

                                                      Outside of work:

                                                      I am also working on my toy programming language Icarus which is my attempt to explore ideas around static verification of mutability and type safety.

                                                      I am also working through Essentials of programming languages as a build up to eventually reading both Practical Foundations for Programming Languages and Types and Programming Languages

                                                      1. 2

                                                        Been working on evaluating a few libraries for work and stuck in a rut at home.

                                                        I haven’t been writing code at home at all, i don’t feel motivated to do so which is weird since that is all what i used to do earlier. Need some interesting problem to solve.

                                                        Basically i want to create or contribute to something, just don’t know what. this is a weird problem to have. :/

                                                        1. 2

                                                          I launched Larder, a bookmarking service for developers, a few weeks ago, and this week has been about adding more features to that. The web client and browser extensions are pretty solid now, but there’s plenty of auxiliary stuff that needs to be done.

                                                          I’ve been continuing to tweak the home page (I can’t help myself, even when it’s “good enough” and I should be working on other more useful features) and next I’ll be working on getting the Android app to beta stage. It’s hard to decide which features to prioritise, but I think having native mobile clients is probably more useful, especially in signalling that we’re invested in the project, than adding more of the developer-oriented features like importing starred Stack Overflow questions.

                                                          In terms of side projects, I’ve been making progress on my toy programming language, Braid. I have the lexing and parsing pretty sorted, and can generate enough basic LLVM IR to compile and run simple functions involving ints and strings.

                                                          1. 2

                                                            Meetings meetings meetings :)

                                                            Have to interview around a half dozen folks. The most interesting thing is we’re quantitatively measuring our TAMs (technical account managers) on how well they demo and, separately, how they handle some field support situations. While its nice to hire smart folks, it’s good to be able to do something like this to objectively qualify that their skills are staying sharp.

                                                            On the down side I have to do a fair amount of glue code / integration work with sales force this week.

                                                            Finally i get to start getting some ETL pulls together to start building our customer success measurement analytics (time permitting)

                                                            1. 2

                                                              Finally going to get a few dry days in a row, so it’s time to stain the deck. We did prep work a few weeks back, but it’s been raining nearly nonstop since then.

                                                              I need to put my leaf blower back together.

                                                              Work stuff, continuing on redoing some things for a different market. Nothing fun or interesting, though.

                                                              1. 2

                                                                Writing enterprise java for work. Writing ESP8266 C for fun. Trying to get slides and demos ready for Buzzconf Nights and looking forward to PyConAu and Compose in Melbourne. Trying to learn a bit about Hadoop and trying not to get distracted by all the things I don’t like about it. Trying not to get distracted by election season here.

                                                                1. 2

                                                                  Moving time! Graduated from UT after 5yrs this past Saturday, working on finding an apartment in the Bay Area before I start with Twitter on the 6th. When I get some downtime from apartment hunting, I want to mess with Clojure 1.9’s new spec stuff, but am trying to focus my efforts on learning and migrating to haskell.

                                                                  1. 1

                                                                    Congratulations on graduating :) Good look house hunting!

                                                                  2. 2

                                                                    A python irc bot: https://github.com/meskarune/autobot

                                                                    I still need to figure out a real plugin system, and configure bot admins.

                                                                    1. 1

                                                                      At work I’m going through my case backlog and closing things that have already been fixed and fixing a bunch of annoyances and smaller cases.

                                                                      Outside of work I’m creating a Common Lisp GUI application that plays MP3s and displays visualizations that are synchronized with the music. Not particularly useful, but it’s a fun project to get more familiar with CL, Qt/QTools/CommonQt, and the FFT.

                                                                      While creating that program, I found a bug in one of the QTools example programs, so I’m attempting to fix it and submit a pull request.

                                                                      I’m also hoping to go for a short backpacking trip next weekend, but need to get a plan together.

                                                                      1. 1

                                                                        still enjoying learning f#, will hopefully have v0.1 of my app released soon. i announced it on reddit as an incentive to keep on task.

                                                                        1. 1

                                                                          This week, I want to get better at something quite dull: maintaining a work log/journal. I’ve configured remember-mode in Emacs (a lovely small and simple package) and make a concious effort to log what I’ve done when I’m done with a task or milestone. Hopefully, given enough practice it will become a habit.

                                                                          1. 1

                                                                            Among other things, getting ready to go to PyCon next week! Say hi if you’re going to be there! I will often be found at the Recurse Center booth in the expo hall.

                                                                            1. 1

                                                                              I have just joined Zomato full-time.

                                                                              So, would be leading their research on data-driven adtech and marketing.

                                                                              Simultaneously, trying very hard to continue self-educating myself with Javascript and Scala.

                                                                              1. 1

                                                                                Trying to remember where the fuck guy before me put the EMC VNX2 Fast cache licence information.

                                                                                1. 1

                                                                                  Reading the “Programming Elixir 1.2” book by Dave Thomas, as (the backend of) the game I’m currently working on is mostly written in Elixir.

                                                                                  1. 1
                                                                                    • Code search for chrome extensions (2 fails with elasticsearch and grep/ag, gonna use postgres)
                                                                                    • Making it easy for people to start their Q&A website (mainly, automating question2answer hosting right now)
                                                                                    • Trying to contribute to Rocket.Chat to add a difference between @all and @user_name notifications