1. 4

    So we let it all out. Our devices produced an average of 50 MB of (uncompressed) logs per day, each. For the baseline 100,000 devices that we discussed above, that’s about 5TB of logs per day. Ignoring compression, how much does it cost to store, say, 60 days of logs in S3 at 5TB per day? “Who cares,” that’s how much. You’re amortizing it over 100,000 devices. Heck, a lot of those devices were DVRs, each with 2TB of storage. With 100,000 DVRs, that’s 200,000 TB of storage. Another 300 is literally a rounding error (like, smaller than if I can’t remember if it’s really 2TB or 2TiB or what).

    I don’t follow this at all. How is the storage on the endpoints relevant? Were the centralized logs somehow reflected back out?

    1. 19

      As I understand it, the argument that if you’re already buying 200,000TB of storage for the client devices, you’re operating at a scale where you can easily afford to pay for another 300TB of storage for your logs, so you shouldn’t be worrying about the cost of the log storage.

    1. 6

      All of this, especially “simply waterfall . . .development in agile clothing” and everything on the “Key flags that a project is not really agile” list are painfully familiar from my time working on a federal government project. I’m definitely not bought into capital-A Agile, but the basics (get something working quickly, get lots of feedback from users, etc.) just seem like no-brainers for most projects. Instead what we got was slapping the “agile” label on because we worked in two week sprints and used story points on JIRA :(

      1. 3

        Congrats @hwayne!

        1. 7

          OK, I’m intrigued. Is there a beginners “Homebrew to Nix” guide anywhere that you can recommend? I regularly have breakages with home-brew, particularly with ruby and python tools. I regularly have to re-create my virtualenvs because the underlying python binary went away. This is vexing.

          1. 2

            I’m afraid there’s not a great answer. Things like the apt cheatsheet on the wiki and nix by example might be a good starting point. And then there’s the nix pills, which are great but pretty technical, but other than those and what’s documented in the manual/wiki it does require a bit of figuring things out on your own.

            The community is pretty active and there are a lot of helpful people on IRC/discourse that will answer questions and try to help if you run into problems. If you decide to give it a try.

            1. 1

              I gave it a try, migrating more and more to nix before fully removing last usages of Homebrew last night. I mostly did it by trial and error rather than those resources though… I didn’t find it particularly hard. The biggest issue was probably that I built Emacs 27 from git with home-brew, but ended up using the Emacs 26 Mac port, which meant I had to rewrite part of my config since the package.el initialisation has changed a bit.

              I know I’m barely scratching the surface, and haven’t done any customisations at all, but I’m pleased with the switch so far.

            2. 1

              That “underlying Python binary went away problem” with Homebrew gets me all the time too.

              I’ve similarly been curious about Nix for a long time, and am hoping to give it a try soon. I’ll probably test it on my server before my laptop though.

            1. 3

              I’d be curious to read more about the actual changes and improvements they made to WINE. I’m curious if Proton has some sort of fundamental architectural differences from WINE, or if it’s just that they’ve poured a lot of blood sweat and tears into fixing small issues and it adds up over time.

              1. 3

                Taken from https://steamcommunity.com/games/221410/announcements/detail/1696055855739350561

                Q: What is Proton exactly? How does it differ from normal Wine? Who worked on it?

                Proton is a tool distribution based on a modified version of Wine. The included improvements to Wine have been designed and funded by Valve, in a joint development effort with CodeWeavers. Here are some examples of what we’ve been working on together since 2016:

                • vkd3d[source.winehq.org], the Direct3D 12 implementation based on Vulkan
                • The OpenVR and Steamworks native API bridges
                • Many wined3d performance and functionality fixes for Direct3D 9 and Direct3D 11
                • Overhauled fullscreen and gamepad support
                • The “esync[github.com]” patchset, for multi-threaded performance improvements

                Modifications to Wine are submitted upstream if they’re compatible with the goals and requirements of the larger Wine project; as a result, Wine users have been benefiting from parts of this work for over a year now. The rest is available as part of our source code repository for Proton and its modules.

              1. 8

                I saw SAT solvers as academically interesting but didn’t think that they have many practical uses outside of other academic applications. … I have to say that modern SAT solvers are fast, neat and criminally underused by the industry.

                Echoing a good comment on reddit: The author didn’t list any practical applications!!! How can you then say they are criminally underused?

                The only one I know of is writing versioned dependency solver for a package manager (in the style of Debian’s apt). However, very few people need to write such code.

                What are some other practical applications? I think they are all quite specialized and don’t come up in day-to-day programming. Happy to be proven wrong.


                EDIT: I googled and I found some interesting use cases, but they’re indeed specialized. Not something I’ve done or seen my coworkers do:

                https://www.quora.com/What-are-the-industrial-applications-of-all-SAT-solvers

                http://www.carstensinz.de/talks/RISC-2005.pdf

                I can think of a certain scheduling algorithm that might have used a SAT solver, but without details I can’t be sure.

                1. 5

                  They see some use in formal methods for model checking. The Alloy Analyzer converts Alloy specs to SAT problems, which is one of the reasons its such a fast solver.

                  There’s also this talk on analyzing floor plans.

                  1. 4

                    A previous submission on SAT/SMT might help you answer that.

                    1. 4

                      I’ve used Z3 to verify that a certain optimized bitvector operation is equivalent to the obvious implementation of the intended calculation.

                      Just typed up the two variants as functions in the SMT language with the bitvector primitives and asked Z3 for the satisfiability of f(x) != g(x) and rejoiced when it said “unsatisfiable.”

                      1. 1

                        Hm this is interesting. Does this code happen to be open source?

                        1. 7

                          I’ll just post it here. :)

                          (define-sort Word () (_ BitVec 64))
                          (define-fun zero () Word  (_ bv0 64))
                          
                          ;; Signed addition can wrap if the signs of x and y are the same.
                          ;; If both are positive and x + y < x, then overflow happened.
                          ;; If both are negative and x + y > x, then underflow happened.
                          (define-fun
                              add-overflow-basic
                              ((x Word) (y Word)) Bool
                              (or (and (bvslt x zero)
                                       (bvslt y zero)
                                       (bvsgt (bvadd x y) x))
                                  (and (bvsgt x zero)
                                       (bvsgt y zero)
                                       (bvslt (bvadd x y) x))))
                          
                          ;; Here is a clever way to calculate the same truth value,
                          ;; from _Hacker's Delight_, section 2.13.
                          (define-fun
                              add-overflow-clever
                              ((x Word) (y Word)) Bool
                              (bvslt (bvand (bvxor (bvadd x y) x)
                                            (bvxor (bvadd x y) y))
                                     zero))
                          
                          (set-option :pp.bv-literals false)
                          
                          (declare-const x Word)
                          (declare-const y Word)
                          
                          (assert (not (= (add-overflow-basic x y)
                                          (add-overflow-clever x y))))
                          
                          (check-sat)
                          
                          1. 2

                            Here’s you an example of SMT solvers used for stuff like that. I added some more stuff in comments. You might also like some examples of Why3 code which is translated for use with multiple solvers. Why3 is main way people in verification community use solvers that I’m aware. WhyML is a nice, intermediate language.

                        2. 3

                          Keiran King and @raph compiled a SAT solver to WebAssembly to use for the “auto-complete” feature of Phil, a tool for making crossword puzzles (source).

                          1. 2

                            I found this library (and its docs) interesting. They go over some practical examples.

                            https://developers.google.com/optimization/cp/

                            1. 1

                              This looks like it’s about linear programming, not SAT solving. They’re related for sure but one reason it would be nice to see some specific examples is to understand where each one is applicable!

                            2. 2

                              I have personally used them when writing code to plan purchases from suppliers for industrial processes and to deal with shipping finished products out using the “best” carrier.

                            1. 1

                              I’d love to know more details about the user-space networking stuff. How is it similar / different to user-space networking on Linux? Do iOS / watchOS now have two independant implementations of TCP, TLS, etc., or are they reusing some of the code? What are the security implications of the memory mapping?

                              1. 1

                                Can anyone speak to the similarities and differences between Minio and Ceph (which I have some experience using)?

                                I googled around for a bit and my vague impression is that Minio seems simpler and sleeker to operate, but isn’t designed for quite the scale that Ceph is (and also doesn’t support the same menagerie of client interfaces), but I’d love to hear someone knowledgeable say more.

                                1. 5

                                  The first difference is that Minio is only an object storage while Ceph is object storage, block storage and even a filesystem when needed, Ceph also has several gateways like rados gateway to provide swift compatible API for example.

                                  Generally Minio is like a small UNIX tool designed to do one thing and do it simple and fast (like vi editor for example) while Ceph is like Emacs with tons of modules and functionalities …

                                  Hope that helped a little.

                                  Maybe little out of topic but only in the latest versions of Ceph (5.x) it does check checksums and have builtin compression, I can not recommend earlier versions of Ceph then 5.x for anything.

                                  1. 1

                                    That does make sense, thank you! And yes, my experiences with Ceph have been lackluster to say the least, although they are a few years out of date.

                                    1. 1

                                      Welcome.

                                      There is a chance that I will have chance to deep dive into Ceph 5.x in a while, if so I would share the thoughts for sure :)

                                1. 8

                                  We definitely need a similar place in western Europe…

                                  1. 4

                                    Oh yes, definitely. I bet a ton of people would be interested in signing up for something like that.

                                    1. 3

                                      Go to your local hacking space?

                                      I have a bit of trouble to understand the point of going through 3 application rounds to take time off, not get paid and meet other people taking time off and not getting paid.

                                      Sounds a bit like doing Erasmus…

                                      1. 6

                                        People have been doing write-ups about the benefits if you’re wondering about them. The mains ones I’m seeing across the write-ups are:

                                        1. A break from mentally-taxing work to only do the work they want. People tend to do one or more fun projects there instead of those they’re usually forced to by work. It’s also common to do both concentrated learning and building activities.

                                        2. Improved focus since they’ve left environments with a lot of distractions. Obviously, they might need a gameplan for email, phone notifications, etc. Based on prior data, I’ll also add their focus at RC might improve by the mental and financial commitment they made by going there. Many who might get distracted doing random stuff on the Internet at home will try harder to complete their project to avoid walking away with nothing. Double true if they lost pay like you indicated. That makes the trip more like an investment.

                                        3. A crowd of people to learn from or help. They usually like listening to interesting projects others are doing there. Face-to-face provides a different experience than just Googling summaries on their Githubs or something. Regardless, I speculate that these connections with others can be a mental break from or boost for the focused projects people are doing at RC. Kind of a pause for them to let stuff process with extroverts benefting directly having people to talk to on top of that.

                                        4. You might get a job. RC is run by recruiters. People with not-so-great employment doing 1-3 might have some job skills or portfolio additions to show off. That they keep funding it is either some serious charity or the fact that they’re getting enough people jobs to keep funding it. Probably a mix of both. So, there could be job-seekers visiting who would rather not be doing 20-30 sessions of whiteboard coding in front of non-coders assessing their skill. RC might be a better experience.

                                        Of course, this assessment is based on just a handful of posts I’ve read written by people who visited. Anyone who visited or works there feel free to correct anything that seems off.

                                        1. 7

                                          I would say these are all pretty spot on! I keep meaning to write a “Why RC?” page for our website that clearly and directly answers the question of why you might want to come, but for now you can get a pretty good idea by reading our about page and the things linked from it under “Further Reading”

                                          I think the only thing missing from is that RC is about more than just the time you spend at the retreat—you’re also joining a tight-knit community of peers dedicated to teaching and learning from each other. The value of this is tremedous, but hard to capture in words. If any other RC alums want to jump in here and try to explain it I think that would be great :)

                                          On point 4—all of RC’s operations, including our living expense grants for people underrepresented in programming, are funded by our recruiting revenue! You can read more about the career benefits of RC on our page about this and in our manual.

                                          1. 3

                                            I’m one of many alums. Here are a couple thoughts on my experience, which I found very positive and better than regular unemployment:

                                            1. I learned a lot about myself and improved a lot as a learner at RC - much more so, I think, than I would have if I had just been unemployed for a few months on my own.

                                            2. I am not as involved in the alum community as I’d like to be but it sure seems fun. It’s nice to feel like you share a connection with some pretty interesting and accomplished people. It’s a diverse crew and I like seeing what different people are thinking and learning about. I also like that, collectively, the community knows a lot and will help you with your problems if you need them.

                                          2. 0

                                            Sounds like a hacker space to me. My city has two, and if I hop onto a train I can get to another 5 in less than an hour.

                                            The whole concept of having application rounds asking someone for permission to spend your own, free, unpaid time on the things you want to do … that sounds very foreign to me, maybe it’s just some concept that’s more widely understood in the US.

                                            If grown-up people are unable to make an independent, autonomous decision on how to spend their time, this looks like a parenting/education failure to me.

                                            1. 2

                                              It is not a hacker space, any more than a blank wall beneath a bridge where you can spray paint is an artist’s colony.

                                              It’s a focused retreat for programmers to gather in one space for a period of time with other programmers, who are all there, as @jamesjporter says, to teach and learn from each other about how to be a better programmer, including theory, hardware, new models and techniques, etc.

                                              I don’t know what else you’re misunderstanding about it, but I suggest you withhold public judgement until you learn more.

                                          3. 4

                                            The social signal is different - “hacking on stuff” vs. “hacking on stuff, as a result of passing 3 application rounds”. Beyond that, the application filter presumably also has some sort of effect on who’s there, compared to regular “open” hacking spaces.

                                      1. 18

                                        Congrats! We’re excited to meet you :)

                                        1. 11

                                          Thanks! I’m excited too, and made a huge list of things I want to work on, probably enough to fill a year. But I also want to go with the flow and not plan too much :)

                                        1. 1

                                          What reasons do owners have to keep their etcd servers open to the internet? Is this mainly used for cross-datacentre communication?

                                          1. 5

                                            Unfortunately, I think it’s more common that this happens accidentally rather than deliberately, e.g. someone spinning up AWS servers and installing etcd in it’s default configuration on them without really thinking about the fact that they’re exposed to the internet.

                                          1. 1

                                            This is painfully similar to my own experiences with Ceph.

                                            1. 1

                                              Nthing the other recommendations to use an automated configuration management system like chef, ansible, puppet, or salt. They all support doing this pretty smoothly. For a real world example, here’s one where I conditionally template some ports (that vary by host) into an nftables config.

                                              1. 4

                                                Wow, thank you, this was really great. I see a lot of writing about how most developers don’t know how to do security well, but rarely do I see such a clearly illustrated and worked through example demonstrating what “doing security well” and doing it poorly means in a particular instance.

                                                I found this post pretty inspiring, both because it showed me a gap in my skillset as a programmer/sysadmin/whatever, and because the designs @alex_gaynor proposes are simple and well-explained and understandable to me, which makes me feel like I could fill this gap (i.e. security is a skill I could learn, not some arcane voodoo).

                                                1. 1

                                                  This is extremely cool! I’ve done a lot of work in scientific computing and devopsy stuff (both of which are domains in which this useful), so I’ve been excited about systems for executing code (particularly Python) on remote hosts for a long time, starting with a lot of exploration/work on IPython parallel, eventually moving more into systems like Salt and Ansible. Will definitely check this out. I sadly don’t have the time to hack on it much right now, but definitely keep posting updates here @dw!

                                                  1. 1

                                                    You probably know about this already, but Fabric is useful for this sort of thing, and it’s Python, too.

                                                    Edit: The article covers Fabric, of course. Should have read it fully before I posted :)

                                                  1. 5

                                                    Really cool. Does anyone have links to more write-ups of things designed this way, info on companies that have reputations for making them, etc.? I’ve recently developed something of an obsession with longevity and repairing rather than replacing my physical things. For example, I would love to be able to buy shoes in the style I tend to wear (sneakers/skate shoes basically) that don’t fall apart after a year of use.

                                                    1. 6

                                                      One place I know of that is focused on longevity is https://www.reddit.com/r/BuyItForLife/.

                                                      1. 1

                                                        Thank you! This is really great, exactly the kind of thing I was looking for. Disappointingly, the sort of shoes I’m looking for don’t seem to exist . . .

                                                        Maybe I need to adjust my personal style to be more amenable to durable clothing :)

                                                      2. 2

                                                        Another phrase to use looking for these things is “heirloom” designs or products. The word indicating they’re expected to be passed down. Longevity planning is a part of that. Also, here’s this nice article on how we got to non-durable appliances particularly:

                                                        https://lobste.rs/s/3rcyqa/they_used_last_50_years_appliance

                                                      1. 2

                                                        I skimmed through this, and it looked kind of like Go’s context? Is that the tl;dr? I’ve been out of the loop for Python 3, so I have a lot less context on the shift to async and coroutines, but as someone who writes Go, and used to love writing and using generators for random things in Python 2, I can see how this could be useful, and very interesting!

                                                        1. 2

                                                          You can read in the rationale:

                                                          Some languages that have coroutines or generators recommend to manually pass a context object to every function, see [1] describing the pattern for Go. This approach, however, has limited use for Python, where we have a huge ecosystem that was built to work with a TLS-like context. Moreover, passing the context explicitly does not work at all for libraries like decimal or numpy, which use operator overloading.

                                                          1. 2

                                                            Oh interesting. I’m going to upgrade to “reading” rather than skimming. Thanks for the inspiration!

                                                          2. 2

                                                            To me it sounded more like putting something like xlocal, which I’ve used and liked for this purpose in the past, in the standard library.

                                                          1. 27

                                                            I was accepted to the Recurse Center and am arranging the logistics of moving to NYC from mid-September to mid-December. If anyone in NYC would like to meet up, message me. I plan to study category theory, dependent types, and, if I can find a study buddy, tools like Coq and TLA+.

                                                            Codewise, I’m tinkering ahead on my current Haskell practice program, a solitaire solver. I’m also making demo Rails apps for my two factor authentication Rails gem.

                                                            1. 4

                                                              Check out @hwayne’s tutorial on learntla.com since it’s on a practical subset. Also, Alloy Analyzer has long been used to do something similar for structural analysis of programs with ease of use vs Coq etc.

                                                              http://learntla.com/

                                                              http://alloy.mit.edu/alloy/

                                                              Hopefully, you master it all. TLA+ and Alloy just knock out lots of low-hanging fruit with relatively few people quiting due to difficulty like what happens with big-time provers.

                                                              1. 3

                                                                Great news, looking forward to the blog posts! Is there something similar to Recurse Center in Europe?

                                                                1. 2

                                                                  There was “Hacker Retreat” in Berlin that was explicitly inspired by Recurse Center, but it disappeared in October or November 2015. I’m not aware of others, but I haven’t looked.

                                                                2. 3

                                                                  Glad to have more RC people here. Enjoy!

                                                                  1. 2

                                                                    This is wonderful news! Looking forward to hearing all about it :).

                                                                    1. 2

                                                                      Congrats again, we’re looking forward to having you :)

                                                                    1. 3

                                                                      China Mieville Kraken. Started fun, but has become a bit of a slog.

                                                                      Inventive character concepts, but boring characters.

                                                                      1. 4

                                                                        Not his best book, yeah. The City and The City is the best, if you haven’t read it.

                                                                        1. 2

                                                                          Kraken is pretty much Neverwhere, but by China Miéville. If you liked the concept but not the characters, you might prefer Neverwhere (there will be no surprises in it, though, Kraken basically spoils most of Neverwhere).

                                                                          Miéville’s less-derivative books are better. @itamarst recommended The City & the City, which is excellent; in true good sci-fi form, it takes a kind of weird concept and explores it to the limit (though it’s more exploring sociological than technological concepts). I’ll also recommend Embassytown (similarly takes a weird concept and runs with it) and Railsea (which is just fun and kind of silly).

                                                                          1. 4

                                                                            Kraken felt more to me like China Mieville’s American Gods. The Neverwhere analogue IMO is Un Lun Dun, which is both short and really good. Mieville gets to throw every idea he has into one book without having to worry about inconsistencies or torturing his characters or talking ‘bout socialism, and it’s absolutely delightful.

                                                                            When recommending Mieville I usually recommend TC&TC or ULD, because they’re both really approachable and are on opposite ends of his Weird Fiction spectrum: one is about taking a single idea and going as far as possible with it, the other just seeing what he’ll come up with next.

                                                                            1. 1

                                                                              I’ve never read Un Lun Dun, I think mostly because I tend to not like short books :/ Agreed The City & The City is probably a good book to read for folks who haven’t built up a tolerance to Mieville. My personal favorite of his though is The Scar.

                                                                        1. 4

                                                                          Cool stuff. I used to study genetics and systems biology and I really enjoyed “thinking computationally” about what it was cells and organisms were doing, especially during development. I do find, though, that practitioners of the more formal sciences (computer science, mathematics, physics) who find their way to biology have a tendency to parachute into some existing problem, come up with a relatively simple model that predicts some things correctly by conveniently ignoring a lot of experimental data, getting it published, and declaring the problem solved (this basically). Some of the papers linked in the OP come off this way, although I have only skimmed them so perhaps I am being ungenerous.

                                                                          1. 2

                                                                            I think the situation is that the concept of “computability” and “computational models” are simple and predictable, but can explain complex behavior like we see in biological systems. Whether or not the concept of computability can construct these complex systems, however, is another question entirely. I’m hopeful that the field of self-organization can shed some light on this issue of constructing complex systems that work. FWIW I like this short interview with De Landa as an intro to this way of thinking.