1. 1

      I appreciate the domain name.

    1. 1

      Gmail everywhere

      1. 3

        Kick ass @hwayne

        1. 7

          I still owe folks a blog post about total vs partial math in Pony and how that relates to division by 0. Sadly I’ve had zero time for that as I’ve been spending all my time on Wallaroo Labs work.

          Part of that post was going to be “and as a pre-1.0 language all this is going to change as we will be introducing partial integer math operators in the future”. Well, those operators are here. All the division by zero kerfuffle got someone inspired to implement the RFC that has been open and waiting to be.

          Still, I owe folks a post on partial vs total integer math and eventually that will come. Maybe when I’m on vacation in November, although honestly, that sounds like an awful vacation.

          1. 4

            Incidentally, have you watched Evan Czaplicki’s recent talk, ‘The Hard Parts of Open Source’? It sounds like you’re in the same situation (and I’ve probably contributed to that, sorry!).

            1. 2

              I just finished watching the talk. I enjoyed it. Thank you for the recommendation.

              1. 1

                I haven’t watched it. Evan was incredibly thoughtful and nice when I met him a few years ago at ICFP and hung out. From that interaction and the title, I imagine its something I would enjoy.

                Care to summarize it?

                1. 4

                  I think you would enjoy it. He examines patterns of behaviour in open source communities that seem hurtful, like ‘Why don’t you just do it like this?’, ‘What gives you the right to do this?’, and so on, and traces them back to the birth of hacker culture and other very interesting historical context that directly influence today’s online communities.

                  1. 18

                    That does sound interesting. It certainly expands past open source communities. Programmers in general are quite happy to critique the product of a series of tradeoffs without context.

                    We do this when we look at other people’s systems and pick out one thing to critique outside of the other feature. The Pony divide by zero kerfuffle was an example of that. Many people who knew nothing about Pony critiqued that single “feature” on the basis of the impact within system they know rather than as a “feature” within Pony as a whole (which in the end is what my blog post needs to be about). In that Pony case, a series of decisions that were made to make Pony safer moved it towards being part programming language and part theorem prover. It’s in an interesting place right now where we have downsides from both that lead to “interesting” things like divide by zero being zero because of a series of other choices. All in all, Pony is safer than many languages but, we can to find that there were a number of features needed to address issues like divide by zero. For example, dependent types, partial integer math as an option.

                    I think this happens in every system. You make a number of well intentioned decisions where they are the right decision but inevitabily, they are going to lead to “wat” and “ugh” moments as they come together. I’ve never seen a language that doesn’t have those and if you spend the time to understand the language and its choices, you can see how when favoring certain values, you would end up there. No tool will ever be perfect.

                    There’s a Bryan Cantrill talk on this that is really good: “Platform as a refection of value”.

                    Often times, we also see the results of constraints on the code. For example, perhaps there was an artificial but reasonable time limit. “This needs to be fixed but we only have a couple weeks to do it, what is the best we can make this in two weeks because other things are more pressing”.

                    I had real problems with this earlier in my career. I was incredibly judgemental. Sean in his 20s would have been all over Pony for the “stupid divide by zero”, Why? Well, I wouldn’t have taken time to understand the problem. Everything I knew had divide by zero as an error so that would “obviously be the right thing to do”. And in general, I lacked empathy. I had no ability to try and understand why someone would do something that I could see a reason to do. Worse, I didn’t care to understand. I just loved to go “wat” and laugh at things. I was awful towards PHP for example. Now, I recognize that PHP is an awesome tool for some tasks. I don’t really ever take on those tasks but that doesn’t make PHP any less valuable for those tasks.

                    I had to work incredibly hard on empathy. Its not something we do in my family. My mother, to this day, is still incredibly selfish and as one of her children, I picked that up. My stepfather hurt his back a couple years ago. My mother was somewhat concerned with his injury but mostly was annoyed with how it impacted on her life and the extra work she had to do because he wasn’t capable.

                    It wasn’t until I worked for an asshole CEO and was a team leader and had to try and hold my team together that I really started to get good at empathy. I realized that in order to deal with said asshole, I needed to try and understand why they did what they did. Without that understanding, there was no way I could get what my team needed from said CEO. There was no way, I could put together an argument that would speak to his needs, desires, and concerns. I developed this empathy skill for purely selfish purposes but, its turned out to be incredibly helpful in general. I have a much better appreciation and understanding of other people’s software. Where before I would judgementally dismissing things as crap, I have now often taken the time to understand why the software was the way it is, and, I’ve learned a ton in the process.

                    Anyway, I could write 5,000 more words on this topic and things tangential to it. Given the context, that seems like rambling to the extreme, so I’m going to stop now. Thanks for the talk recommendation. I’ll definitely check it out.

                    I’d really advise anyone who read this and found that it reasonated at all the check out that Cantrill talk. It’s really really good. And also, if you don’t think empathy and understanding can be valuable as an engineer, I’d pass along the advice to give it a serious try for a couple years. If you are like me, you will be amazed and delighted with the results.

                    1. 3

                      Thanks a lot for your very open and honest text, it was really moving and I’m glad you’ve made empathy a priority, and that it has worked out for you.

                      1. 2

                        Thanks for the context. I too struggle with being empathetic. We’re all such a deep well of emotions and desires, that sometimes I feel like if I try to open that door of trying to understand people on a deeper level, I’ll spend all my emotional budget on it. But even on a superficial level–what I try to do nowadays (not always successfully!) is realize that people probably do things that make sense to them and it’s OK for it not to make sense to me, because it doesn’t affect my life.

                        1. 2

                          Empathy and doing stuff with other humans is about being at the point where it does affect your life and still being able to deal with other people as fully formed, totally broken but in a different way to you, feeling, incomprehensible beings.

                          1. 2

                            That sounds about right. I’m still learning, I guess!

                        2. 2

                          “All in all, Pony is safer than many languages but, we can to find that there were a number of features needed to address issues like divide by zero. For example, dependent types, partial integer math as an option.”

                          I suggest a translation to WhyML in Why3 platform that feeds verification conditions to automated provers. Why3 is the middle-end that Frama-C, SPARK, and the Java one all use. They prove absence of things like you describe. The automation in SPARK is often over 90%. The backends keep improving.

                          So, my default recommendation for any type/verification of things like number ranges is either static analyzer that’s extensible or a langusge-specific front-end for Why3. A side benefit is SPARK-style annotations are easy for programmers to learn. And you can do property-based, test generation if proof is too hard.

                          1. 1

                            We do this when we look at other people’s systems and pick out one thing to critique outside of the other feature. The Pony divide by zero kerfuffle was an example of that. Many people who knew nothing about Pony critiqued that single “feature” on the basis of the impact within system they know rather than as a “feature” within Pony as a whole (which in the end is what my blog post needs to be about). In that Pony case, a series of decisions that were made to make Pony safer moved it towards being part programming language and part theorem prover. It’s in an interesting place right now where we have downsides from both that lead to “interesting” things like divide by zero being zero because of a series of other choices.

                            The main criticism I saw was criticism of it being presented as somehow more ‘mathematically pure’, at least around here, not criticism of Pony. The blog post claiming that it 1/0 = 0 was actually consistent with mathematics was nonsense.

                            1. 1

                              We never presented it as more mathematically pure. The langauge we used was that it was an unfortunate compromise. If you want to argue with @hwayne about his blog post, go for it.

                              1. 0

                                I never said you did

                  1. 7

                    Is it Github’s CI? It’s typical today’s marketing trick to make landing pages where everything is completely unclear. Documentation is probably only available after signing up for this feature.

                    1. 6

                      There is some documentation here: https://developer.github.com/actions/

                      1. 3

                        But its pretty sparse right now.

                        1. 2

                          Is it just webhooks?

                      2. 1

                        Seems more like IFTTT or Zapier than just CI.

                      1. 2

                        Has anyone used Pony for anything non-trivial? I really appreciate the actor model, and I’ve played around with OTP (Erlang and Elixir) some. There are some very, very good ideas there IMHO. However, I prefer statically typed languages, especially for large projects. Is Pony meant to solve problems in the same general space? What have people found it to be a good fit for?

                        1. 3

                          There’s the Wallaroo thing, which is most certainly non-trivial. Other than that I’m not aware of anything.

                          1. 3

                            The first version of the Pony runtime (which was accessed via C- the problems with that approach are what lead to the creation of the language) is/was at a large bank in the UK. I say, “is/was” because afaik its still in use but I have no recent information on that.

                            As @akavel points out, there’s Wallaroo. Which is what I work on with Pony.

                            I’m also aware of at least a couple other things that no one publicly discusses, but, in general, it’s still a small community of folks. That tends to happen with pre-1.0 languages. There needs to be a very compelling use case for folks to consider using.

                          1. 1

                            Going to give this a try on the Wallaroo Labs blog this week. Very timely, I was going to look into how to do this very soon. Thanks!

                            1. 9

                              I have been a mac user since the beige-toasters. There is no chance I would buy a computer with “DRM” crap preventing me from repairing it.

                              1. 2

                                Didn’t the beige toasters require an Apple-licensed tool to open or was that an urban legend spread in the besieged Amiga community?

                                1. 5

                                  It’s correct, they used Torx screws for the case: teardown.

                                  1. 3

                                    I just bought a torx screwdriver from Lowe’s last week. I’d hardly consider that to be locking users out of their hardware.

                                    1. 3

                                      They were hard to come by in 1984.

                                      1. 3

                                        Yet infinitely easier to bypass than software DRM.

                                        1. 2

                                          I started repairing Macs in 89 and had several torx screwdrivers so by then, they were easy to get.

                                          1. 1

                                            Good point.

                                        2. 3

                                          Tri-Point and Pentalobe are two common currently examples of this.

                                          In highschool/college I worked at an authorized repair shop for Apple. Torx are great honestly. They are easier to work with and less likely to strip than Philips. Tripoint & Pentalobe are a nightmare to work with, which is why they are often not used internally.

                                          1. 3

                                            Torx is just a better type of allen bolt

                                            1. 3

                                              Indeed, there’s valid reasons to use Torx.

                                        1. 6

                                          So, what’s the business model now?

                                          1. 3

                                            We are looking at a few options based on initial engagements with folks:

                                            • the standard “support” model
                                            • running Wallaroo as a hosted option in the cloud for folks. So for example, we install Wallaroo into your AWS account, hook it up to your data sources and run it for you.
                                            • enterprise version where we build around the core product and create value that isn’t of interest to the open source community but are to large enterprises. for example, i doubt many folks in the open source community are interested in using Wallaroo with Kdb but its come up on more than 1 occassion with larger enterprises.
                                            1. 4

                                              Third option works the best. I mentioned that in the licensing discussion in another thread. Active Directory integration was the example I gave. Things like that stay proprietary. Good luck regardless.

                                              1. 3

                                                Yeah, the thing we want to do is not have it be features that are integrated into the open source product. We don’t want to keep things from the open source option, rather we really want to build around it. It’s hard. It’s going to be interesting but, I think it’s the right thing to do.

                                                1. 6

                                                  Sounds risky to me. I respect your team’s commitment to principles, though.

                                                  Kind of adding to other comment, one thing I forgot to mention is you can just license it to businesses. That may sound counterintuitive since who would pay for open-source software they can get for free, right? A number of people on HN told me they license GPL software to businesses. One was at OpenSUSE but another just did custom apps for businesses that they allowed to go FOSS. They weren’t critical to competitive advantage, developer highlighted benefits of FOSS like extra features at lower cost, and they were cool with it. One pointed out that even Stallman encouraged people to charge for free software.

                                                  So, that’s possible at who knows what odds of success or level of revenue for Wallaroo. The main benefit they told me is businesses like having someone accountable for what they’re using. Someone to call, sue, or whatever. The other factor is that many believe you have to pay for good software. The “cheap” software is garbage. Free software must be even worse. Instead of arguing with them, those selling FOSS said they’d rather just let them pay a lot of money for the software and their own foolishness. :)

                                                  1. 3

                                                    O yes, that “license” with a “neck to wring” is part of what we cover in “support”. Sorry, we’ve been discussing it for so long that I forgot that many people rightly consider them to be different items.

                                                    It is risky but personally, I see the alternative as more risky. The infrastructure tooling space is dominated by open source products. Even if we create a better product, would we really be able to outpace a large open source project? Probably not. Could we lose lots of business to “its good enough” and people using another tool? Probably.

                                                    Those are also risks. So ya, risky but, we think worth the gamble.

                                                    1. 3

                                                      Good thoughts on infrastructure. I had one more idea but would’ve worked better if you GPL’d it. That is embedding it into commercial products that want to layer on top of your project or use one of its components. In the GPL model, they buy a copyleft exemption so they don’t have to GPL their product. Apache breaks that model given companies can just repackage it for free without legal risk from giving nothing back. Since you want it permissive, I tumbled idea around in head and it still might work.

                                                      There might be companies that want to integrate your stuff with theirs who would prefer to pay your company to do it faster (your experts) and with less integration risk. You could mention integration of Wallaroo into commercial offerings as a service Wallaroo Labs offers. I don’t think this would be common. It could be a nice supplement to main revenues if priced well to encourage the practice, you avoid any deals that look like a resource drain (or just charge extra for it), and Wallaroo kept modular enough to make that easy.

                                                      1. 2

                                                        I’m a huge fan of the GPL but man, it really scares large enterprises and makes selling to them much harder.

                                                        1. 4

                                                          Oh I hear you. I was mainly mentioning it for other readers that might be evaluating pro’s and con’s of various options in licensing their software. The paid service for embedding was for you. I could see why you’d want to avoid GPL if dealing with big business. I also mostly “thank” Microsoft for that.

                                                          1. 1

                                                            Why not offer Wallaroo Pro which is Wallaroo/GPL relicensed with a commercial license? That’s literally part of my business model with Sidekiq (LGPL) and Sidekiq Pro (commercial).

                                                            1. 2

                                                              It’s something we considered but again, fear of GPL will keep some developers from using Wallaroo for free in some enterprises. In the end, we wanted to leave that open as a possibility.

                                                              I’m comfortable with the GPL. Many people are not and their lack of comfort definitely ended up leading to a lack of comfort from other non-engineers at Wallaroo Labs. Apache 2 is a “safe license”. No one balks at it.

                                                              All of that said, I think the model you have is a completely valid one.

                                            1. 2

                                              I’ve used mutation testing at previous jobs. It was a handy tool albeit one where in our large codebase, the feedback cycle was very slow.

                                              1. 2

                                                I’m learning how to be a “developer evangelist”/developer outreach person. It’s a role we’ve been looking to fill at Wallaroo Labs and never did, so I’m transitioning over into the role. The fun part about being at a startup is doing things that are scary and new to you because, it needs to get done and you are the best person to do it.

                                                1. 1

                                                  Can someone here possibly explain to me what is the value add of Wallaroo? Maybe some real world use case when this could be used, or pain point it solves? I see articles from its authors surfacing occasionally on HN/Lobsters; it’s interesting to me esp. because of its use of the Pony language; but I repeatedly fail to grasp what is the product trying to do, where could it be useful. This feeling sometimes means to me that it’s some new, breakthrough technology (I had similar trouble understanding Docker for a long time). I understand I may just not have lived through some particular problem domain where this can be useful (in case of Docker, my lack of understanding was mainly because I had no Ops-related experience). Especially if you are a person having some tech pain/itch, and feeling that Wallaroo is a solution that could help you with it, please speak up and try to describe your context to me! TIA

                                                  1. 4

                                                    Imagine you have the following scenario: you have a chunk of data coming in every now and again. This could be user analytics, logs, whatever, but the point is that it never stops coming in.

                                                    You have to perform some kind of transformations on this data in order to make it usable. So let’s say it’s data in -> transform -> write transformed data to db.

                                                    This will work well if the data comes in rarely and you have enough resources to handle it all on one machine, one batch at a time.

                                                    Now, imagine the volume of the data increases so that you either cannot handle one batch fast enough before another batch comes in. Or the amount of data coming in with one batch is too much to handle on one machine.

                                                    You’ll want to speed up the execution of a batch. Maybe you’ll try making the batches smaller at the producer level (if you control it). This will buy you some time and may work for a while. If the volume/frequency of the smaller batches becomes too big to handle, you’ll make them smaller again.

                                                    Eventually, you’ll arrive at the limit: sending your data in entry-by-entry. At this point, your batch transform will probably cost more in startup time than the processing of one entry. You’ll also be running a bunch of these transform processes to make up for the increase in input frequency, on many machines perhaps.

                                                    Now what if you stepped back and decided up front: abandon the batch workflow and deal with every event as it comes in? This is what stream processors do. This way, you can stop thinking about files on disk, cron jobs, etc. You could define your data transformation functions, and have the stream processor apply them to data, while not having to worry about managing clustering and shipping bytes around.

                                                    There are a lot of stream processors out there now (Apache Flink, Kafka Streams, etc.), but to answer your original question: Wallaroo’s value-add is that it can run event-processing code that’s written in Python or Go, whereas the big players are mostly focused on Java as their first-class citizen.

                                                    I heartily recommend Martin Kleppman’s book “Designing Data-Intensive Applications” for an overview of this space in general, and the approaches to solving various data-intensive tasks.

                                                    1. 1

                                                      Thanks a lot!

                                                      I don’t think I was ever in such a situation yet, so that can explain why I don’t understand its pain points. As of now, I’m imagining if I found my imaginary data processor having to work entry-by-entry, I’d try to convert it to an always-on service, listening for the entries on some URL. Do I understand correctly, that the “stream processors” like Spark or Wallaroo are some helper frameworks, that provide some common components I’d have to add in such a service? I imagine I might need some authentication, so probably Spark/Wallaroo provide it out of the box? Also “clustering” seems to be thrown around a lot in this context, so I suppose the “stream processors” are aimed at data volumes so huge that one machine is not enough to process them? So if my data volumes are smaller, I probably don’t need such frameworks and a simple service will be easier to maintain?

                                                      1. 2

                                                        I’m going to speak to Wallaroo but Spark is probably the same in many areas.

                                                        What you get from Wallaroo is a common API for your tasks so that from one program to another, you have the same API to describe the data flow and transformations. That API gives you access to a number of features, mostly related to horizontal scaling (having more than 1 process do the work) and failure recovery.

                                                        Some frameworks are definitely very hard to get setup and you’d need to look at the tradeoffs of using them. A pro to use a framework from the beginning is that when its time to scale across more than 1 process, its a straightforward thing to do. There’s no rewriting. In Wallaroo, that’s running a command to add an additional process to your cluster.

                                                        Horizontal scaling (clustering) can give you a couple things:

                                                        • the ability to do more of “something”. that might be “hold more things in memory” or “process more data in parallel”
                                                        • ability to continue processing if you lose a process or a machine

                                                        Wallaroo, unlike some other options, is more about “mesh networking” than setting up a cluster ahead of time. I can be running a Wallaroo process for an application and then add more on the fly and that creates a cluster. Spark, Storm and some other tools you are setting up a cluster ahead of time.

                                                        This is a very long way of saying:

                                                        So if my data volumes are smaller, I probably don’t need such frameworks and a simple service will be easier to maintain?

                                                        It depends. There are tradeoffs. You need to decide what is important to you. If you ever have a specific use-case in mind where you would need to work through those tradeoffs, I’d be happy to chat about specifics.

                                                        1. 1

                                                          Thanks for the reply!

                                                          The problem is, I don’t know what tradeoffs! XD so I can’t subject them to discussion, as I still don’t see them coming :) Currently, if I need to process something, I go and write myself some service, deploy it with docker on a server and be happy! :)

                                                          But if by tradeoffs you mean the two bullet points you listed, then hmmm… what I seem to think of them, going in reverse:

                                                          • …if you lose a process or machine — for this I’d expect to be using a load balancer, haproxy or something; does Wallaroo bring something over it?
                                                          • …do more of “something”… — ok; so for me this again seems to confirm my vague image, that Wallaroo may be a tool I’d have to reach out for once the throughput of data I’d need to process would be too huge to fit processing powers of one physical machine… otherwise I can just split it over a pool of goroutines, right? Here’s where you didn’t seem to answer my question about “factory transmission belt”… would Wallaroo e.g. help me balance the number of goroutines in my app even on one machine? Or is this an incorrect intuition attempt on my side?

                                                          Also, I don’t really understand the first paragraph of your answer: what’s an “API to describe the data flow and transformations”? If I need to do some data transformations, I just write the code to do them… where’s an API here?

                                                          1. 1

                                                            The problem is, I don’t know what tradeoffs! XD so I can’t subject them to discussion, as I still don’t see them coming :)

                                                            Right, what I meant was, when you find yourself in that position, I’m happy to chat and lend you my perspective based on my prior experience.

                                                            would Wallaroo e.g. help me balance the number of goroutines in my app even on one machine? Or is this an incorrect intuition attempt on my side?

                                                            Wallaroo gives you an API that if you write to it, you can scale it across more than 1 process without having to rewrite it.

                                                            If I need to do some data transformations, I just write the code to do them… where’s an API here?

                                                            Its a standard API that you can pick up the code for any application and understand what it is doing versus “just writing some code” that doesn’t have any particular consistency from one application to the next.

                                                            In general frameworks give you two things:

                                                            • a standard API for accomplishing a task so you can move from one application to another without needing to relearn everything
                                                            • whatever the functionality of the framework is

                                                            For example, all Ruby on Rails applications share a common API basis that would make it easier to pick up a new application that is written in Rails rather than if each application in an organization where custom done on top of raw HTTP.

                                                            1. 1

                                                              Eh; OK, still can’t say I feel I’ve grasped the core idea, but thanks for the discussion! Maybe one day I’ll run into a situation when something you’ve said will come back to me as a perfect solution for a problem I’d have.

                                                    2. 2

                                                      Since @pzel already gave you a very good run-down about Wallaroo, here is one about Docker:

                                                      Imagine you need to setup a server to being able to run your software smoothly. You might make people pull your code, run some configuration script which does all the job (install scripts, move files) and runs the code. That’s great and all but what if you just want to play around with the application without messing your system? What if you run the software in parallel without changing a line of code? Here comes Docker!

                                                      The beauty of Docker, together with the sandboxing, is how easy it is to spawn tons of mini services which can then be load balanced. DevOps becomes a joy as you can treat each machine as an individual while having 1000 micro Dockers running in parallel doing tiny bits of work without even coding for it. You just make the image configuration and roll out.

                                                      Another use case is to just tell your new employee to pull image X from your private repo and they will have their dev environment setup in a matter of minutes.

                                                      1. 2

                                                        Thanks! Maybe I haven’t expressed this clearly in my original post, but I’ve actually already grasped Docker at some point :) Used it just as an example. Sorry for the confusion, and big thanks for your effort and goodwill! Really appreciated.

                                                      2. 1

                                                        Hi @akavel,

                                                        I’m the VP of Engineering at Wallaroo Labs. The folks who are building Wallaroo. I’d be happy to chat either here or perhaps over IRC/email that might work better. A question before I get started. What’s your familiarity with the “big data” space?

                                                        Are you familiar with:

                                                        • Apache Hadoop
                                                        • Apache Spark
                                                        • Apache Storm (or Flink)

                                                        Beyond that, anything about your background you want to share? From there I think I could give a explanation that you’d be able to grasp.

                                                        1. 1

                                                          I’m OK with discussing here, maybe it can be useful for other people too. Won’t make a lot of noise for the others, I hope. But if you feel you prefer email for some reasons, I can switch; I suppose the email on your github profile is correct?

                                                          To answer your questions: not familiar with “big data” space, nor with Hadoop/Spark/Storm/Flink :) Only the first name seems to ring some bell: some kind of Google-scale map-reduce engine IIRC? Or not? :/

                                                          Not sure what to share about my background that could be relevant. I know Go well; never worked on more than a few servers however. Or maybe I did… but the particular “webservice” I was contributing to was just copied over a few machines by Ops guys, and traffic was distributed between them; I never needed to think about it. And we communicated with some other “webservice”, which was hidden behind another load-balancer. Based on your questions, and @pzel’s answer, do I infer correctly that Wallaroo could start to be useful to me only once my processing stops to fit on one machine? When I have a cluster and want to load-balance a stream of data flowing through various services? Like managing supply & demand in a brick & mortar shop, or maybe avoiding congestion on a transmission belt in a factory? Is this what it does?

                                                          1. 1

                                                            do I infer correctly that Wallaroo could start to be useful to me only once my processing stops to fit on one machine? When I have a cluster and want to load-balance a stream of data flowing through various services?

                                                            Yes. Wallaroo would be useful in the scenarios you describe.

                                                            One big difference with something like Wallaroo vs the sort of horizontal scaling you’ve been doing is that Wallaroo is aimed at jobs where you need many processes working together to get the job done. The “webservice” approach is more “things that can work independently from one another”.

                                                            One of the big things that Wallaroo does it manage state for you. One common approach that folks do with data processing tools is have a computation engine and a data storage engine. So for example, I am using Storm to process data as it enters the system and I store the results into a database. This works as long as I don’t need to accumulate results over time.

                                                            Imagine for example, we want to do spam detection or DDos attack detection.

                                                            In those cases, you will want to accumulate state over time, probably in-memory, to be able to make your application decisions like “is this spam?”, “is an attack underway?”. If you accumulate state and store it in a database then the scalability of your application is going to be severely limited by regular round-trip to the database to fetch and store data. The more data you get, the worse the load on your database is going to get.

                                                            You are going to want that state in-memory to be able to make quick decisions and not limit your ability to handle more traffic.

                                                            When you move to using in-memory state, you will need to manage it in case you want to:

                                                            • increase your processing capacity or available memory by adding additional machines or process to your application
                                                            • restore your state if there is a crash

                                                            In the “increase processing capacity”, you’ll need to move state around to take advantage of the additional processes you have available. In the case of a crash, you’ll need a mechanism to bring your state back to the last known “good point”. Both of those things are very complicated to do correctly. With Wallaroo, we’ve built that in and you get it “for free” by using our API.

                                                            There’s more to state handling but that is a big part of it. This feels like place to stop and see if what I’ve said makes sense to you or if I should back up and work through some of the earlier problems that Wallaroo seeks to solve.

                                                      1. 1

                                                        So, is this about using OpenBSD as a development environment for working on MirageOS applications?

                                                        1. 7

                                                          Yes. More precisely, it is about running mirageos unikernels in vmm: See https://marc.info/?l=openbsd-tech&m=150743896827764&w=2

                                                        1. 2

                                                          So I’m pretty familiar with how cool befs is, but what are the other advantages of beos over unix?

                                                          1. 10

                                                            Here’s the 10,000ft view. Main advantage was performance through pervasive concurrency. My current box is Linux on an Intel Celeron. It’s responsiveness is worse than BeOS was on a Pentium due to inferior architecture. My favorite demonstration is this clip where they throw everything they can at the machine. Notice that (a) it doesn’t crash immediately vs OS’s of the time and (b) graceful degradation/recovery. I still can’t do that on crap hardware with Ubuntu without lots of lagging or something.

                                                            1. 2

                                                              When I used to demo BeOS for folks in NYC, I would throw even more at it than what is in the demo. It was amazing how good BeOS was. I’ve never had a better day to day OS in terms of stability and responsiveness. Even when it was in beta and I had to kill off servers from time to time, they would pop right back up and everything would keep going.

                                                              1. 3

                                                                I rarely hear something is better than the demo in practice. Wow. It’s architecture is worth copying today. I don’t know how close HaikuOS is to the architecture and stability under load. They might be doing their own thing in some places.

                                                                Anyone copying or reusing its principles today also has tools like Chapel, Rust, and Pony that might make the end result even better in both performance and stability. QNX and BeOS were the two I wanted cloned the most into a FOSS desktop. I hate freezes, crashes and data loss. Aside from hardware failure, no reason we should have to deal with them any more.

                                                                1. 2

                                                                  The relatively limited time that I got to use QNX was pretty nice. It was limited in some areas but from a stability and responsiveness standpoint, it was a joy to use.

                                                                  1. 3

                                                                    If you’re curious, John Nagle described how it balanced performance and stability here. He’s constantly encouraging a copy of its design on places like HN. I did find an architectural overview from the company itself in 1992.

                                                                    EDIT: Another person on HN described what the desktop experience was like. That person’s main memory was how big compiles would slow down their main workstation but not QNX desktop. It’s real-time design, maybe built-in priorities, made sure the UI parts ran immediately despite heavy load from other processes. The compiles got paused just enough for whatever he was doing. That sounded cool given one app can drag down my whole system or interrupt my text to this day.

                                                                    1. 3

                                                                      Back in the mid-90s I was hired to port a bunch of Unix programs to QNX. What blew me away about QNX was the network transparency in the command line. I could run a program on my machine A, loading a file from B, piping the output to a program that lives on C but run it on D and pipe that output to a local printer hooked to E, all from the command line. My boss would regularly use the modem attached to my machine from his machine (in the office next to mine).

                                                                      Now, this meant that all machines had to have the same accounts install on all the machines, and the inter-processing message passing was done over Ethernet, not IP, so it was limited to a single segment. Such a setup might not fly that well in these more security-conscience days.

                                                                      As far as speed goes, QNX was fast. I had friends that worked at a local company that sold commercial X servers and they said that the fastest X servers they had all ran on QNX.

                                                                      1. 2

                                                                        That all sounds awesome. I wonder if it’s still that fast on something like Intel Core CPU’s given how hardware changed (eg CPU vs memory bottlenecks). Some benchmarks would be interesting against both Linux and L4-based systems.

                                                                        If it held up, then someone should definitely clone and improve on it. Alternatively, port its tricks to other kernel types or hypervisors.

                                                          1. 7

                                                            I’m looking forward to figuring out how much of my daily work I can move over to Haiku. Probably not much at this point given how much of it is tied to specific applications (and Pony and Wallaroo for code related things). Hopefully over a few years, I can move more and more work to it. I really miss the days in the 90s when my primary work computer for 3 years was BeOS and my primary home machine(s) were BeOS for about 7 years.

                                                            1. 14

                                                              Why can’t Unikernels be SMP?

                                                              1. 9

                                                                IncludeOS has SMP support.

                                                                MirageOS doesn’t, but that’s more about OCaml’s threading model. HalVM does (or probably could, anyway) thanks to Haskell’s SMP support, but it’s got some other problems. Then there’s LING, which uses the Erlang VM on Xen. Not sure about their SMP story, but I’d bet it’s not intrinsically difficult, due to Erlang’s design. But then again, if you have a fine-grained message passing architecture and can host lots of little lightweight kernels, maybe the kernels themselves don’t need SMP at all.

                                                                1. 3

                                                                  No idea. Maybe they’re using some approach where the bare minimum of the kernel is linked in, but somehow no process management (implied by flat address space reference). I think you’re right though.

                                                                  1. 2

                                                                    No reason I can think of…..

                                                                    …except the stuff that does that in the linux kernel is the hardest and most mindbending to get right and is a product of man decades of work, and usually depends on the fine fine fine print of the actual CPU and memory architecture spec.

                                                                    Every programmer I know can write a multi threaded program that will work.

                                                                    Most of the time.

                                                                    Very very few programmers I know can write a significant multithread system that will be right 99.99999% of the time.

                                                                    I don’t think I have met one that get’s them right all the time….

                                                                    Multi-processor shared memory systems with multiple layers of caching are much harder to get right.

                                                                    1. 2

                                                                      I assume because the natural use case falls towards kilobytes instead of megabytes, and nobody needed to fit in kilobytes when they had multiple cores lying around.

                                                                      1. 1

                                                                        There’s no reason they can’t be. But the definition of unikernel they use in the post my preclude it.

                                                                      1. 2

                                                                        This was a very good talk. Definitely recommend once the video comes out.

                                                                        1. 11

                                                                          There was a comment in that HN post that stated:

                                                                          Lobste.rs is basically Hacker News without intransparent moderating based on whims, liberofascist social justice warriors, stalinists and feminazis. I like browsing it, but it doesn’t have as much activity as Hacker News.

                                                                          It was soon moderated and marked as “dead”.

                                                                          Inflammatory words, but the moderation part has some truth to it in my experience.

                                                                          1. 18

                                                                            i would mark any such post as troll too. There is no content beyond the insult - and the insults are unacceptably vitriolic.

                                                                            1. 11

                                                                              And they make us look bad. Many people will think that’s how comments on Lobsters look or how most of us think about HN. Neither is true. I tried hard in my summary there to avoid causing a quick dismissal by the crowd which that troll earned for themselves. I count it going dead as an example of HN moderation doing their job.

                                                                              1. 5

                                                                                Your summary was wonderful detailed and a great representation.

                                                                            2. 7

                                                                              liberofascist social justice warriors

                                                                              LOL what is that even supposed to mean

                                                                              1. 2

                                                                                “Bad bad people who are bad and I disagree with them”, I think.

                                                                              2. 3

                                                                                The presence of people that doesn’t ditch social issues was the exact reason I joined lobste.rs (complete opposite of what that person thinks what lobste.rs is). If I wanted a community of techbros, I would have joined any other community on the internet.

                                                                                1. 2

                                                                                  The presence of people that doesn’t ditch social issues was the exact reason I joined lobste.rs

                                                                                  Really? I don’t visit Lobste.rs as often nowadays, however I remember that when I joined the site it was because that it focused on tech (and stuff interesting to hackers) more than politics. Has that changed?

                                                                                  1. 4

                                                                                    It’s still focused on tech, but social issues also happen on tech. And from what I noticed visiting the website in the last months, people don’t just dismiss them, solely because they are social issues. Which I think it’s a good thing.

                                                                                    1. 4

                                                                                      Could you point me to some example threads here (or, better, the relevant tags) that you refer to as social issues?

                                                                                      1. 2

                                                                                        They are often tagged with culture or culture + person when it’s something very specific. I just don’t link the threads specifically because I don’t know how the moderation will feel about that. But, if you google “site:https://lobste.rs + term” with a term related with social issues, you might find them easily.

                                                                                    2. 3

                                                                                      It’s news to me…I’m still working on keeping that stuff out.

                                                                                  2. 3

                                                                                    The problem isn’t that HN is moderated in a politically biased way, it’s that it’s not really properly moderated at all. I’m sure this is what you actually meant, but it’s important to point out. I’m a leftist with liberal opinions and I also found HN’s moderation lacking.

                                                                                    1. 1

                                                                                      obviously whoever posted that comment has never seen my replies

                                                                                      1. 0

                                                                                        plz delete this

                                                                                      1. 5

                                                                                        GopherCon!

                                                                                        I’ll be speaking on Wednesday about various things I’ve learned about cgo performance while working on Go support for Wallaroohttps://github.com/wallaroolabs/wallaroo).

                                                                                        I had a very busy last few weeks at work so I’m still putting together my presentation. I had all the content ready but nothing in terms of slides. If there are any lobste.rs folks at GopherCon and are interested in chatting, drop me a note.