1. 5

    Finally a redesign! Looks great, too.

    1. 5

      Looks an awful lot like Slack with a different colour scheme.

      1. 1

        A different color scheme and a governing body that isn’t hell-bent on destroying anything standing in their way, I guess.

        1. 1

          Sure, I’m not exactly in love with Slack either. My point still stands: calling it a redesign is a bit strong.

    1. 2

      My first gut reaction: if you find yourself reaching for glom, refactor your data into something simpler so you don’t have to use it.

      I mean sure, there are some domains and use cases where it could be warranted, but the page tries too hard to sell it as a long-needed solution for a ubiquitous problem. Which it isn’t.

      1. 4

        It would be nice to refactor things to be simpler, but sometimes we have to work with other people’s data. For my job, I have to make a lot of API calls to deeply nested data. This tool seems like it could be useful.

        1. 1

          Yep, I’m looking forward to trying this tool with some JSON-serialized Java objects I have to deal with from a 3rd party API (yes it is as bad as it sounds).

        2. 2

          Maybe I’ve just been unlucky with the data I’ve worked with, but LinkedIn, Facebook, GitHub, Wikipedia, Twitter, and PayPal’s APIs (especially the midtier/internal ones) are all exceedingly nested. Could just be my experience, but nested data seems pretty ubiquitous to me!

        1. 28

          After reading the article and many HN comments, I found the headline to be highly misleading as if they’re targeting Signal for their activities in fighting censorship. It’s actually more incidental. They’re targeting a fraudulent practice Signal is doing that violates terms of service. Signal is doing it for good reasons but others might not. Google and Amazon are trying to stop it wholesale. A proper headline might be that “Several providers threaten to suspend anyone doing ‘domain fronting’ via hacks, including us.” Average person reading something like that would think it sounds totally to be expected. A technical person liking Signal or not should also notice the MO is an operational inconsistency that shouldn’t exist in the first place.

          So, they’re not doing a bad thing given the situation. They’re just an apathetic, greedy party in a business context fixing a technical problem that some good folks were using to help some other good folks deal with evil parties in specific countries. Sucks for those specific people that they did it but they’re not aiming at Signal to stop their good deeds. They’re just addressing an infrastructure problem that affects anyone hacking around with their service. Like they should.

          I wish Signal folks the best finding another trick, though.

          1. 16

            I think the correct headline would be “AWS is fixing a bug allowing domain fronting and calling it Enhanced Domain Protections”. An analogous situation would be console homebrew people exploiting buffer overflows in Nintendo games. Of course Nintendo should fix them, and like you, I root for console homebrew people to find another one.

            1. 3

              That’s another good one. It’s just a bug in their services. Them not fixing it would be more questionable to me.

            2. 9

              I found the headline to be highly misleading as if they’re targeting Signal for their activities in fighting censorship. It’s actually more incidental.

              And that’s why they immediately sent signal an email containing a threat to close the account immediately, instead of a regretful email telling them that this will stop working due to abuse prevention measures.

              1. 1

                It my experience that’s generally how they treat literally any issue.

              2. 5

                Signal is doing it for good reasons but others might not.

                I’m failing to think of a way to use domain fronting for a not good reason, especially one where the provider being fronted is still happy to host the underlying service.

                1. 4

                  There is nothing fraudulent about domain fronting. Show me one court anywhere in the world which has convicted someone of fraud for domain fronting. That’s a near-libelous claim.

                  Can you provide an example of a “bad reason” for domain fronting?

                  As the article points out, the timing of Amazon’s decision relative to the publicity about Signal’s use of domain fronting suggests that Signal is in fact the likely intended target of this change, not incidental fallout.

                  The headline is accurate. Your comment really mischaracterizes what is happening.

                  1. 3

                    I meant it in the popular definition of lying while using something. Apparently, a lot of people agree its use isn’t what was intended, the domains supplied are certainly not them, and service providers might negatively react to that. It would probably be a contract law thing as a terms of use violation if it went to court. I’m not arguing anything more than that on the legal side. I’m saying he was doing something deceptive that they didn’t want him to do with their services. Big companies rarely care about the good intentions behind that.

                    “the timing of Amazon’s decision relative to the publicity about Signal’s use of domain fronting suggests that Signal is in fact the likely intended target of this change”

                    The article actually says he was bragging online in a way that reached highly-visible places like Hacker News about how he was tricking Amazon’s services for his purposes. Amazon employees stay reading these outlets partly to collect feedback from customers. I see the cloud people on HN all the time saying they’ll forward complaints or ideas to people that can take action. With that, I totally expected Amazon employees to be reading articles about him faking domains through Amazon services. Equally unsurprising that got to a decision-maker, technical or more lay person, who was worried about negative consequences. Then, knowing a problem and seeing a confession online by Signal author, they took action against a party they knew was abusing the system.

                    We can’t just assume a conspiracy against Signal looking for everything they could use against it with domain fronting being a lucky break for their evil plans. One they used against Signal while ignoring everyone else they knew broke terms of service using hacker-like schemes. If you’re insisting targeted, you’d be ignoring claims in the article supporting my position:

                    “A month later, we received 30-day advance notice from Google that they would be making internal changes to stop domain fronting from working entirely.

                    “a few days ago Amazon also announced what they are calling Enhanced Domain Protections for Amazon CloudFront Requests. It is a set of changes designed to prevent domain fronting from working entirely, across all of CloudFront.

                    It’s a known problem they and Google were apparently wanting to deal with across the board per his own article. Especially Google. They also have employees reading forums where Signal was bragging about exploiting the flaw for its purposes. I mean, what did you expect to happen? Risk-reducing, brand-conscious companies that want to deal with domain fronting were going to leave it on in general or for Signal since that one party’s deceptions were for good reasons according to claims on their blog?

                    Although I think that addresses it, I’m still adding one thing people in cryptotech-media-bubble might not consider: the manager or low-level employee who made the decision might not even know what Signal is. Most IT people I’ve encouraged to try it have never heard of it. If you explain what it does, esp trying to get things past the governments, then that would just further worry the average risk manager. They’d want a brick wall between the company’s operations and whatever legal risks the 3rd party is taking to reduce their own liabilities.

                    So, there’s at least several ways employees would react this way ranging from a general reaction to an abuse confession online to one with a summary of Signal about dodging governments. And then, if none of that normal stuff that happens every day at big firms, you might also think about Amazon targeting Signal specifically due to their full knowledge of what they’re doing plus secret, evil plans to help governments stop them. I haven’t gotten past the normal possibilities, though, with Amazon employees reading stuff online and freaking out being most likely so far.

                    1. 3

                      This rings true to me (particularly the middle-management banality-of-evil take), bar one nitpick:

                      The article actually says he was bragging online in a way that reached highly-visible places like Hacker News about how he was tricking Amazon’s services for his purposes.

                      How did you get that impression? The article states:

                      We’re an open source project, so the commit switching from GAE to CloudFront was public. Someone saw the commit and submitted it to HN. That post became popular, and apparently people inside Amazon saw it too.

                      I haven’t read the mentioned HN thread, but that hardly constitutes “bragging online”.

                      1. 2

                        I can’t remember why I originally said it. He usually blogs about his activities. I might have wrongly assumed they got it out of one of his technical write-ups or comments instead of a commit. If it was just a commit, then I apologize. Thanks for the catch regardless.

                  2. 3

                    “Service provider warns misbehaving customer to knock it off after repeated RFC violations.”

                  1. 1

                    Dramatiq is licensed under the AGPL

                    Now I have three options:

                    • Make the codebase at work opensource (lol)
                    • Violate AGPL on purpose
                    • Use any of the other Redis-based task queues

                    I really don’t get what the author is trying to achieve with choosing a license like this.

                    1. 13

                      You could also buy a license.

                      1. 1

                        Ah, that makes sense. I didn’t see that, shame on me.

                      2. 4

                        I think commercial backing of some sort or another is the only way we can sustainably develop open source software long term and dual licensing seemed like the lowest friction way to get started. I’ll have to highlight that fact a little better in the docs! :D

                        1. 2

                          You are right of course. Other message frameworks like sidekiq seem to do alright: https://github.com/mperham/sidekiq

                          The challenge here is that Celery is in pretty great shape for a free solution. On the other hand Python’s support for high concurrency is changing rapidly so who knows maybe there’s room for a new player in this market.

                          1. 2

                            I’ve never met anyone IRL who’s worked with Celery and didn’t run into problems, so there’s definitely room for improvement in this area.

                            1. 2

                              It works like a charm with RabbitMQ as a backend. The rest is pretty experimental and breaks, especially at high volume. (I’ve been using Celery for >5 years)

                              1. 4

                                I’ve been using Celery professionally for about 3 years and dramatiq tries to solve many of the issues I’ve encountered using it. Some stuff that immediately springs to mind:

                                • Celery doesn’t support task prioritization. You have to deploy multiple sets of workers in order to prioritize queues.
                                • Celery has poor support for delayed tasks. Delayed tasks go on the same queue that normal tasks go on and they’re simply pulled into worker memory until they can be executed. This makes it hard to autoscale workers by queue size.
                                • Celery acks tasks as soon as they’re pulled by a worker by default. This is easy to change, but a bad default. Dramatiq doesn’t let you change this: tasks are only ever acked when they’re done processing.
                                • Celery tasks are not retried on error by default.
                                • Celery’s not well suited for integration testing. You’re expected to unit test tasks and to turn eager evaluation on for integration tests, but even then task exceptions will be swallowed by default. Dramatiq provides an in-memory stub broker specifically for this use case.
                                • The source code is spread across 3 different projects (celery, billiard and kombu) and it’s impenetrable. Its usage of runtime stack frame manipulation leads to heisenbugs.
                                • It’s easy for some of its more advanced “canvas” features to drop tasks.

                                All of the above are things that are first-class in dramatiq and there are definitely other things I’m not thinking of right now. That’s not to say that celery is bad, but I think we can do better and that’s why I made dramatiq. :D

                                1. 1

                                  Considering your experience, I was wondering what’s your take on rq? (others who used it, are obviously welcomed to chime in too)

                                  1. 1

                                    I don’t have much experience with RQ since it is Redis-only and I’ve generally preferred to use RabbitMQ as a message broker. However, a few things that seem like disadvantages to me with RQ are:

                                    • Messages are pickled so it’s strictly limited to Python and pickled messages are potentially exploitable. This also means you may sometimes send bigger messages than you intended over the network purely by accident.
                                    • Queue prioritisation is handled like it is in Celery: you have to spawn different sets of workers.
                                    • It forks for every job, so it’s slightly slower and forks that are killed b/c they’ve surpassed their time limits can leak DB connections if you’re not careful. I understand this may be swappable behaviour, however.
                                    • Similar to Celery, there isn’t a good integration testing story for RQ.

                                    Because I’ve criticised both Celery and RQ at this point, I feel it’s important that I mention a couple areas where they’re both currently better than dramatiq:

                                    • the obvious one: it’s newer than either of those and is less likely to be familiar to users. The extension ecosystem for dramatiq is nonexistent (though I will be releasing integration packages for Django and Flask soon!)
                                    • dramatiq doesn’t store task results and doesn’t offer a way to retrieve them. Adding that sort of functionality is trivial using middleware, but it’s not there ootb so if you absolutely need something like that and you don’t care about the things I have mentioned so far then you should look at Celery or RQ instead.
                                    1. 1

                                      Thank you for taking the time to post this!

                                      There are two other areas that bother me personally:

                                      • Python 3 only. While I would love to switch to Python 3, still need to maintain a large project in Python 2.
                                      • The AGPL license. The above project is open source too, but I want to keep it BSD licensed to stay “friendly” towards potential users. Ironically, for a commercial project I would worry less about your license of choice, as I wouldn’t mind buying the commercial license when needed.

                                      I share @jscn’s sentiment about Celery. I I was wondering if RQ, despite the above disadvantages might be more stable. At least their codebase should easier to grok (single repo)…

                                      1. 1

                                        Python 3 only. While I would love to switch to Python 3, still need to maintain a large project in Python 2.

                                        I’m considering adding Python 2 support, but it’s a hard thing to balance what with 2.x getting EOL’d in a little less than 2 and a half years.

                                        The AGPL license. The above project is open source too, but I want to keep it BSD licensed to stay “friendly” towards potential users. Ironically, for a commercial project I would worry less about your license of choice, as I wouldn’t mind buying the commercial license when needed.

                                        Understandable.

                                  2. 1

                                    Sure, that’s true. Did you ever look at https://github.com/RichardKnop/machinery that project is still really early. Probably much easier to compete with.

                            2. 1

                              beanstalkd, NSQ, resque, celery, huey, … — pretty much everything in this space is non-GPL. So “use any other queue thing” will definitely be a very popular option :)

                              1. 5

                                So “use any other queue thing” will definitely be a very popular option :)

                                That’s perfectly fine! I just want those people that get value out of my work to contribute back in some way. If someone makes a cost-benefit analysis and decides that they’d rather use celery over dramatiq because they prefer the cheaper option (although it’s worth mentioning that I give out free comm. licenses for one year per company) then that’s their prerogative. I’ll still be around a year later when they realise their mistake ;).

                            3. 2

                              Trying to achieve you not using this at work? That’s usually what I’m going for when I choose AGPL

                            1. 4

                              Finally getting around to reading The Parable of the Sower by Octavia Butler, after it’s been sitting on my desk for a few months. It’s very good so far.

                              1. 3

                                I read that a year or two ago and really enjoyed it. Thanks for the reminder that I’ve been meaning to read more of her.

                              1. 5

                                Specifically, a MongoDB exposed publicly with no password.

                                1. 6

                                  That was ransomed three times already.

                                  1. 6

                                    Once is a mistake, twice is carelessness, three times is they’re secretly fronting for the extortion ring.

                                  2. 1

                                    Yeah, the headline is little unfair – the fact that the datastore happens to be MongoDB is not really relevant. Any unsecured datastore would cause similar problems. You could argue that it’s a bit irresponsible for MongoDB’s default configuration to be insecure, but ultimately the responsibility lies with the developers of the application(s).

                                    1. 4

                                      Well, MongoDB does ship with unsafe and insecure defaults. I can see an argument that it’s fair to call out a product that is unsafe by default.

                                      1. 1

                                        It’s the job of the person using the tool to understand the tool and operate it safely.

                                        That’s like buying a knife, accidentally cutting somebody and then claiming that the knife is faulty for being sharp by default or for not coming with a sheath.

                                        1. 2

                                          But we’ve accepted that you can take simple precautions to prevent really terrible damage by selling them in sheaths or at least wrap it in some wad of news paper. There’s a certain level of basic protection you can provide so those who are new and ignorant won’t just accidentally kill themselves or others.

                                          It’s like saying well, Windows XP ships with all ports open to the network, but you should know that as a manager and you should understand you need to change the default settings to be safe. It’s standard industry practice. You’re probably right, they should know this, but this is useless garbage they shouldn’t need to know if they were all closed by default.

                                          1. 1

                                            It’s the job of the person using the tool to understand the tool and operate it safely.

                                            Of course.

                                            That’s like buying a knife, accidentally cutting somebody and then claiming that the knife is faulty for being sharp by default or for not coming with a sheath.

                                            No it’s not. A knife must be sharp to be useful. A database does not need to have an insecure configuration to be useful. It’s a purely unnecessary hazard to users.

                                            1. 1

                                              A knife doesn’t need a sheath to be useful.

                                              1. 1

                                                What I was trying to say is this: the reason a knife can cut you is the exact same reason that it is useful: it’s sharp. Take away the danger and you take away the utility. Nobody would ask for that.

                                                But a database is useful for storing data. Giving it insecure defaults does not make it more useful, it simply adds hazard.

                                                So to go back to your analogy, it’s more like if you buy a knife, and when you get it out of the package, the handle is wrapped in razor wire. That’s still not a perfect analogy because insecure defaults are presumably due to laziness or inattention, and the razor wire would be active malice. But it’s closer.

                                        2. 1

                                          The Register is known for tabloid-style headlines; rather surprised to see two articles from them on the homepage currently…

                                      1. [Comment removed by author]

                                        1. 3

                                          I voted up. But, it’s also interesting to consider Whether or not Instagram would have been as successful if they choose another language. Did Python grant them the ability to hire the exact right people at the time? Did it give them that extra productivity boost? But, now that they are successful, and can presumably grow the team out considerably, why haven’t they invested in other langs? Maybe they have? Maybe it’s truly not worth it? I mean, they are using Django and Postgres at their scale. That’s actually pretty damn impressive when you think about it.

                                          1. 2

                                            they are using Django and Postgres at their scale. That’s actually pretty damn impressive when you think about it.

                                            I feel like this is kind of like saying Facebook is using PHP and MySQL: technically true, but they’ve spent huge amounts of resources to heavily customise them to the point of probably being unrecognisable.

                                            1. 1

                                              Well, it looks like it’s not just Postgres anymore (shows how up to date I am!), but as of 2015 a slew of other things. I actually don’t think I’d be surprised if they used pretty stock Django (and maybe avoided or replaced some parts of it) for serving web requests. I’d expect there to be custom, hyper-optimized stuff for things like search, recommendations and other stuff like that, though. But, again, what do I know?

                                        1. 2

                                          Excited about this – Webmention along with the also in progress Micropub standard and the renamed Pubsubhubbub standard WebSub enables the creation of full featured self-hosted social media profiles in the style of Twitter and beyond.

                                          One can also create social media apps like Tweetbot for these social media profiles that can both read stuff in realtime (WebSub), post new posts and interactions (Micropub) and let other users know of ones interactions with them (Webmention).

                                          Lastly they can be integrated into the existing social media platforms and thus be usable from day one by what the IndieWeb movement calls POSSE (and PESOS) and by combining that with services like Brid.gy. That way dodging Metcalfe’s law and getting the full network effect of existing networks from day one while still being able to craft ones own space online with technologies that works fully standalone.

                                          Also excited about the already fairly large amount of independent implementations of all of these technologies – the specifications are truly proven to be possible to reimplement again and again and not to rely on any unspecified specific implementation detail of any one dominating library for interoperability.

                                          1. 1

                                            It’s nice to see all the informal R&D done by the IndieWeb folks coming together into specs. Having never messed with a Pub/Sub system before, I’m planning on implementing the WebSub spec as a Django app for some semi-dayjob-related learning.

                                          1. 20

                                            I saw this article earlier and decided not to submit. I was a bit disappointed with the content and here is why (note I didn’t check the linked ‘free’ book).

                                            1. The title speaks of a resurgence of C programming but the content has little to do with the language and recurring interest in C;
                                            2. They constantly intermix C and C++;
                                            3. It focuses solely on Arduinos & Raspberry Pi plus bits of seemingly random trivia.

                                            I expected a piece going deep on people re-discovering the importance of working on a lower level. Instead I found couple of passages that the language is still used to program hardware and that it influenced a lot of languages with it’s syntax.

                                            1. 5

                                              Thanks, this saved me some time trying to get psst their paywall.

                                            1. 8

                                              A company that values good work/life balance will have a much easier time hiring and retaining female and disabled employees. Women and the disabled often do not have the time to spend working 12 hours straight. They have other important obligations that need to be taken care of, such as their own health, children, elderly parents, etc. The upsides to a more diverse software engineering team are already well known, but companies are still struggling with retention and hiring.

                                              1. 16

                                                Some men also like to spend time with their families, have to take care of family members, need to exercise, have hobbies. It’s good to retain them too.

                                                1. 4

                                                  I’m 30 years old, got a wife and three kids (looking to add a fourth, maybe, someday). I come from a big family, and although I absolutely love what I do for a living, my family is the most important thing in my life, bar none. An employer who doesn’t understand that will find it hard to retain me.

                                                  tl;dr I agree with your statement.

                                                  1. 1

                                                    Tech companies are thus far getting by hiring a majority of men for their offices who can work long shifts. (I don’t think they should do this, but I’m talking about reality right now.)

                                                    A major reason why the gender gap exists in Tech is due to women not being able to work on call, over night or very long shifts. One of the big reasons for this is due to obligations at home that cannot be scheduled around or put off. I am not talking about a man’s “right” to have hobbies, I am talking about someone being cut out of an entire industry due to long shifts that many women and disabled folks will never be able to do.

                                                    Your comment has no baring on my point.

                                                    Of course men will also benefit from shorter working hours, and their families benefit having them around more. The author of the article goes into a great deal of detail into this already. I was merely pointing out an additional benefit of shorter hours – increased diversity in hiring.

                                                  2. 2

                                                    Men dont have to look after elderly parents, children or health?

                                                    1. 6

                                                      Completely agreed. Everyone benefits from work/life balance and I’d hope everyone supports it.

                                                      When asked to make a choice between career and family, however, … somebody in the household needs to choose career or there won’t be a household, and hopefully somebody chooses family, for the kids' sake. In the very common case of male-female monoamorous relationships, women are more likely to be the ones who choose family, because that’s the predominant social structure.

                                                      And, of course, disabled people don’t have a choice.

                                                      In the context of “how can we help retention for groups where it is generally very bad”, making that choice easier and less all-or-nothing is an enormous thing that companies can do. I agree - it will also help retention for everyone else.

                                                      Let me say in advance that I am about to offer an explanation of why your remark could draw hostility, but I don’t think you believe or are trying to suggest the view I’ll describe. I just want to head off acrimony.

                                                      I have been disappointed as I discover how many people really do think as follows:

                                                      1. We should help this marginalized group by doing a thing!
                                                      2. That thing would also help almost everybody.
                                                      3. Therefore, we shouldn’t do it.
                                                      1. 3

                                                        I don’t disagree. It just reads like the original comment mentally segregates men into one category, and women and disabled people into another. This sort of distinction is unnecessary in this context and could alienate men.

                                                        1. 1

                                                          Women, Men and the disabled have different lived realities where I am (the united states) and I cannot possibly comprehend how pointing out the differences between people’s lives == alienating non-disabled men.

                                                          Right now women do the majority of child care and elder care in the US. I made no comment on this fairness of this, I never said “the menz are evil patriarchs”, I am simply stating the reality of things right now without blame or judgments. People are reading into what I said and placing ideas there that never existed.

                                                          Of course I would like to see men be able to spend more time with their children and eldery parents. But right now the reality is that women are doing the majority of this sort of work in the US.

                                                        2. 2

                                                          I think it’s more than social convention when men “focus on career” and women on family. For me at least, I will be working while my partner stays home simply because I can command a higher paycheck than she can.

                                                          1. 1

                                                            True enough.

                                                          2. 1

                                                            I was merely pointing out an additional benefit of shorter hours – increased diversity in hiring. Obviously everyone benefits from not working 12 hours shifts. I have no idea why y'all are getting your panties in a twist. I just wanted to point out something that many people may not even consider – that more manageable work shifts == more women and disabled employees as a possible consequence.

                                                          3. 1

                                                            I am a man who does not look after his health. So that’s a solid 1/3 yes.

                                                        1. 3

                                                          I think newsblur qualifies. It’s MIT licensed, too.

                                                          1. -2

                                                            Is that a woman?

                                                            1. 3

                                                              Does it matter?

                                                              1. 2

                                                                It don’t matter when it’s Arcturian

                                                            1. 2

                                                              My new gig is with Sonatype, and Sonatype has a fully distributed structure. There are a few people at the head office, but almost everybody else works remotely. Some of us use co-working spaces, and some of us work from home, but everybody is on chat pretty much all the time during working hours. We use Hangouts and other systems extensively.

                                                              At my last two gigs (Samsung Research, and Apple Maps), I was the only remote worker, which makes for a monstrous disadvantage in communication and socialization. It was do-able, but my relationship with those jobs was very much “grinding through problems that are pushed down a tube to me”, rather than being fully integrated. With the best will in the world, it is extremely difficult to replicate the informal communication that face-to-face working fosters over a low-bandwidth channel like chat/email/video calling.

                                                              Sonatype is different; there’s no face-to-face channel to rebuild; everybody is subject to the same attenuated media. This means that everybody uses the same tools communicate; it’s analogous to everybody sitting in the same room. In addition, because the company is fully distributed, the historical culture isn’t strongly biased towards in-person work. I find it an enormous breath of fresh air.

                                                              1. 1

                                                                my relationship with those jobs was very much “grinding through problems that are pushed down a tube to me”

                                                                Wow, that’s the best description of being the only remote on a team that I’ve read yet. Really resonates with my experiences.

                                                                1. 1

                                                                  This is how my current (non remote) job often feels, so it’s definitely not limited to remote work. I think it’s more to do with how (un) collaborative the environment is, and working remotely tends to magnify that.

                                                              1. 4

                                                                Looking forward to the final release of this. I had a lot of fun helping out with technical reviews of a couple of chapters.

                                                                1. 3

                                                                  I’m confused, is he basically saying that working on open source stuff got him a job offer to work on closed source stuff and now he’s making lots of money and working from home?

                                                                  1. 1

                                                                    I think the lesson here is that if you have passion and use it a lot than great things might happen. In the case of the IT business it’s even easier, thanks to the great exposure that open source can give you. But anyway when you have passion you don’t do things strictly for the exposure or for the money…

                                                                  1. 4

                                                                    At work, been putting a lot of time into a query profiler for Elasticsearch. When profiling is enabled, you’ll now see a timing breakdown for each “node” in your query tree at each shard involved in the search. You’ll also see a tree of Collectors (Lucene objects which “collect” documents to score) which can provide an overlapping set of timings about how query vs post_filter vs aggs vs global aggs interact.

                                                                    The response is super verbose and a bit quirky due to Lucene mechanics (e.g. queries can “rewrite” differently depending on local data, so different shards may have slightly different representations of the query tree) … but I think it’ll be a huge boon to anyone trying to debug queries. In the past profiling queries relied on intimate knowledge of Lucene / ES and divining tea leaves, which is obviously not ideal :)

                                                                    For fun, I recently finished a MYOG ultralight backpack. Weights 7.5oz, ~40 liter capacity, water resistant. It’s survived a few 46er hikes so far, so I’m pretty pleased. Starting on the next version, which will be a little more rugged and a lot less voluminous. Also getting back into Rust + Mio side projects.

                                                                    1. 2

                                                                      I would love to hear more about both of these things! Any chance the profiler will be FOSS? Got a blog post about either one up your sleeve?

                                                                      1. 1

                                                                        Whoops sorry, I probably should have stated that I work at Elastic :) If all goes well, the profiler should make it into ES 2.1 core. It’s still pretty rough (I just pushed the shared branch, we’ll start iterating on it soon), but you can follow along here: https://github.com/elastic/elasticsearch/tree/query_profiler

                                                                        There is a bit of history and explanation in this PR too: https://github.com/elastic/elasticsearch/pull/12974

                                                                        I’ll definitely be writing an article about it, once it get’s merged. The output is a little tricky to read, so I we’ll need a primer on what the various components mean, and how to interpret the results. Keep an eye out for it sometime post 2.0 :)

                                                                        My personal blog burned to the ground a while back, so nothing written about the backpack yet. There are some photos here though: http://imgur.com/a/mkCfZ

                                                                        1. 1

                                                                          Looks like the profiler will be pretty handy. Your elasticsearch-inquisitor looks nice, too. I’ll have to have a play with it next time I’m debugging our search results.

                                                                          Is the pack based off a pattern? I’m a little hesitant about UL packs because I like structure you get with an internal frame.

                                                                          1. 1

                                                                            Fair warning, Inquisitor has been essentially unmaintained for a long time now…I’m not sure if it works with modern ES. It should theoretically work, since it is 1.0+ compatible, but you may find oddities or bugs :)

                                                                            No real pattern for the pack, just cobbled together from various articles on the internet and studying backpacks that I own. The main problem with my pack is that it’s too large: it can technically hold ~40 liters, but the lack of structure would make it awful (and probably not strong enough without ripping either). Realistically it only holds ~15lbs, which works fine for day trips….but I wouldnt want to use it for an overnight or longer.

                                                                            My next pack will be smaller and more rugged, to help maintain it’s structure more and less likely to overload. Some day, I’d like to tackle an arched external frame, where two carbon fiber rods are tensioned to provide structure and lift the bag off your back. Need a lot more practice first :)

                                                                    1. 1

                                                                      @work I’m fixing bugs in our Celery code and updating a Django project from 1.6 to 1.8. I’ve hit a few snags with the new migrations framework causing problems with foreign keys which is going to be a real pain to figure out. I’m also trying to kick off a regular lunch ‘n learn.

                                                                      @other I started getting up at 0530 again to get to BJJ class before work. It’s not easy getting out the door so early in the morning but I always feel less stressed at work when I hit the mats first thing in the day.

                                                                      1. 4

                                                                        I’m trying to catch up on the OO classics, finally started Refactoring Ruby Edition. Although I mainly work in Python, Ruby > Java.

                                                                        My biggest frustration with the book is the lack of hyperlinks. I’m planning to clone the content into a wiki so I can add links, with reworked examples in Python.

                                                                        1. [Comment removed by author]

                                                                          1. 2

                                                                            My only concern is that I don’t want to go posting the author’s copyrighted material, and it would largely be a “duplicate and add hyperlinks” kind of deal.

                                                                        1. 3

                                                                          I just finished the first book of the Southern Reach trilogy. It was completely enthralling from the first page. Reminded me at times of The Thing and also of Andrei Tarkovsky’s Stalker.

                                                                          As soon as it arrives, I’ll be reading the latest in the Little books, The Little Prover. https://mitpress.mit.edu/books/little-prover.

                                                                          1. 3

                                                                            Keep going with the Southern Reach trilogy, it gets even better.

                                                                            I’d also recommend checking out Roadside Picnic (the book that Stalker was based on) and Solaris by Stanisław Lem.

                                                                            1. 2

                                                                              I’ve hardly been able to put the Southern Reach books down. Just started the final book, Acceptance. And thanks for the recommendations, I never knew those films were based on novels!

                                                                          1. 11

                                                                            I don’t know. Part of me agrees, but part of me thinks if the argument holds for Python and C, then surely it holds for C and Assembly, too. And I don’t think anyone really needs to be able to program Assembly to be a great programmer. But then, I’m not that great a programmer so I could well be wrong.

                                                                            Also, and I say this as a non-Lisp-capable programmer, it seems to me that Lisp and C both teach you something about how computation works. C teaches you about how the machine works. Lisp seems to me to be more about how computation works in a more general sense. But then, I’m not a Lisp programmer so I could well be wrong.

                                                                            1. 3

                                                                              Presumably you could also make the jump from ASM down to hardware, too.

                                                                              1. 2

                                                                                If you haven’t coded up quicksort in transistors, you don’t really know computer science!

                                                                                1. 2

                                                                                  I built a computer from 5v TTL, implemented a compiler, and wrote a scheme in that compiler, and I have no idea about computer science

                                                                              2. 2

                                                                                I suppose using C puts you on par with manual transmission drivers as far as knowing how the car works, and using assembly puts you on par with mechanics who can take the car apart and rebuild the motor.

                                                                                Though I know plenty of mechanics who are terrible drivers, and I’m sure there are programmers who are great at assembly and extracting every nanosecond of performance out of an application but are terrible at big-picture design of complex applications. Analogies are hard.

                                                                                1. 2

                                                                                  Obviously there’s a point of diminishing returns, and there’s going to be disagreement about where that point is. I think it’s a question of how leaky the abstractions you’re building are, and given that criteria I think C is a reasonable sweet spot: translating between C and assembly is pretty straightforward.