There was a recent discussion about whether file-based APIs were better or worse the specific ones (https://lobste.rs/s/ckqzbn/why_filesystems_have_loose_coupling_your).
I think a lot of the pros and cons of that API choice are illuminated by unveil.
Things can always be implemented either way (layered on top of the generic mechanism, or given their own specific mechanism). But if you then enhance the general mechanism with a new feature, you get to use that for all of the functionality you have layered on top.
(As a side thought - it’s kind of a nice thought experiment: If the only syscalls were open/close/read/write and friends - how would you (a) provide an interface for all other system calls and (b) put them together in a filepath hierarchy so that unveil() grouped things nicely.)
I know very little about either of these topics, but I wonder if it would be interesting to combine OpenBSD’s security syscalls like pledge and unveil (is there a generic name for these? Privilege-based security?) with Plan 9’s extreme everything is a filesystem approach.
The amount of syscalls in Plan9 is so small that pledge(2) wouldn’t make much sense.
And unveil(2) wouldn’t really be necessary as there are already mount namespaces, a process can unmount paths and then use rfork(2)s RFNOMNT flag to disallow mounting new resources.
In other words, Plan 9 obviated the need for these approaches. It really was the wave of the future, but it hasn’t reached the shore yet. Yet …
I’m even more of an idiot than the author, so I just used Anarchy Linux (previously Arch Anywhere). It worked like a charm on the first run, which I found surprising and impressive given my previous bad experiences with Linux installers.
My only complaint is that the website is down a LOT. On later installations, I would often encounter 404s because the anarchy-linux repository was unavailable. The workaround for this is to not opt in to the extra anarchy-linux repository (I’ve personally deleted it from my list because nothing in there is more useful than what’s available in the AUR).
Simple > correct resonates with me greatly. My day job is writing developer tools (specifically, build analysis) for big enterprises, and it’s surprising how often a simpler (but less correct) approach works much better than a correct approach. Our customers are often very okay with having approximately correct answers as long as (1) they understand how we’re approximating and (2) the tool’s algorithm is simple enough for them to work around it if they need special behaviour.
One example of this is Bower build analysis. The Bower configuration spec is extremely complex, and our first attempt at analysing the dependency resolution logic in Bower was bug-prone and often failed in ways that were difficult to debug. It turns out that the simpler approach (just read bower_components) worked well enough for customers about 95% of the time, and was significantly more reliable.
You could make an argument that if a simpler implementation is correct enough, it’s actually still correct (we’ve simply shifted the goalposts), but I think that “simple > correct” does a good job of capturing the spirit of the lesson.
Yes, the intuition around “correct” brings to mind a waterfall-style process where we start by defining (formally or informally) what it is we want to do, and this is carved in stone forever more. Anything which doesn’t fulfill that definition is “incorrect”.
Yet I’ve often found that something turns out to be a little hairier than I’d like, but if we redefine what it is we’re trying to do then a small, elegant, reliable implementation may fall out; even though this is “incorrect” according to the original definition.
As you say, this is moving goalposts. We could backsplain that our simple implementation is correct for the “actual” goal of ‘delivering customer value’ or somesuch, but that quickly becomes too hand-wavey to be useful. I think it’s better in these situations to think in an agile-like feedback loop, where we’re figuring out what we want and how to do it at the same time.
and our first attempt at analysing the dependency resolution logic in Bower was bug-prone and often failed in ways that were difficult to debug
As an aside, for things like this I try to keep in mind that predictability is a useful feature. It’s often preferable to have something that predictably fails in, say, 10% of cases, than it is to have a “smart” system which only fails 2% of the time but is difficult to tell when that will be. For the former we can draw a line around that 90% and say everything within it is solved; for the latter we’ll always have uncertainty.
Yep, this is how I figured out monads too, but when using Rust! There is more to them though - the laws are important, but it’s sometimes easier to learn them by examples first!
Can you show an example where a monad is useful in a Rust program?
(I’m not a functional programmer, and have never knowingly used a monad)
I learned about monads via Maybe in Haskell; the equivalent in Rust is called Option.
Option<T> is a type that can hold something or nothing:
enum Option<T> {
None,
Some(T),
}
Rust doesn’t have null; you use option instead.
Options are a particular instance of the more general Monad concept. Monads have two important operations; Haskell calls them “return” and “bind”. Rust isn’t able to express Monads as a general abstraction, and so doesn’t have a particular name. For Option<T>, return is the Some constructor, that is,
let x = Option::Some("hello");
return takes some type T, in this case, a string slice, and creates an Option<T>. So here, x has the type Option<&str>.
bind takes two arguments: something of the monad type, and a function. This function takes something of a type, and returns an instance of the monad type. That’s… not well worded. Let’s look at the code. For Option<T>, bind is called and_then. Here’s how you use it:
let x = Option::Some("Hello");
let y = x.and_then(|arg| Some(format!("{}!!!", arg)));
println!("{:?}", y);
this will print Some("Hello!!!"). The trick is this: the function it takes as an argument only gets called if the Option is Some; if it’s None, nothing happens. This lets you compose things together, and reduces boilerplate when doing so. Let’s look at how and_then is defined:
fn and_then<U, F>(self, f: F) -> Option<U>
where F: FnOnce(T) -> Option<U>
{
match self {
Some(x) => f(x),
None => None,
}
}
So, and_then takes an instance of Option and a function, f. It then matches on the instance, and if it’s Some, calls f passing in the information inside the option. If it’s None, then it’s just propagated.
How is this actually useful? Well, these little patterns form building blocks you can use to easily compose code. With just one and_then call, it’s not that much shorter than the match, but with multiple, it’s much more clear what’s going on. But beyond that, other types are also monads, and therefore have bind and return! Rust’s Result<T, E> type, similar to Haskell’s Either, also has and_then and Ok. So once you learn the and_then pattern, you can apply it across a wide array of types.
Make sense?
Make sense?
It absolutely does! I’ve used and_then extensively in my own Rust code, but never known that I was using a monad. Thanks for the explanation Steve.
But there’s one gap in my understanding now. Languages like Haskell need monads to express things with side-effects like IO (right?). What’s unique about a monad that allows the expression of side effects in these languages?
No problem!
This is also why Rust “can’t express monads”, we can have instances of individual monads, but can’t express the higher concept of monads themselves. For that, we’d need a way to talk about “the type of a type”, which is another phrasing for “higher minded types”.
So, originally, Haskell didn’t have monads, and IO was done another way. So it’s not required. But, I am about to board a flight, so my answer will have to wait a bit. Maybe someone else will chime in too.
A monad has the ability to express sequence, which is useful for imperative programming. It’s not unique, e.g. you can write many imperative programs using just monoid, functor, applicative or many other tools.
The useful function you get out of realising that IO forms a Monad is:
(>>=) :: IO a -> (a -> IO b) -> IO b
An example of using this function:
getLine >>= putStrLn
I should say Monad is unique in being able to express that line of code, but there’s many imperative programs which don’t need Monad. For example, just Semigroup can be used for things like this:
putStrLn "Hello" <> putStrLn "World"
Or we could read some stuff in with Applicative:
data Person = Person { firstName :: String, lastName :: String }
liftA2 Person getLine getLine
So Monad isn’t about side-effects or imperative programming, it’s just that imperative programming has a useful Monad, among other things.
You are way ahead of me here and I’m probably starting to look silly, but isn’t expressing sequence in imperative languages trivial?
For example (Python):
x = f.readline()
print(x)
x must be evaluated first because it is an argument of the second line. So sequence falls out of the hat.
Perhaps in a language like Haskell where you have laziness, you can never be sure if you have guarantees of sequence, and that’s why a monad is more useful in that context? Even then, surely data dependencies somewhat impose an ordering to evaluation?
For me, the utility of Steve’s and_then example wasn’t only about sequence, it was also about being able to (concisely) stop early if a None arose in the chain. That’s certainly useful.
but isn’t expressing sequence in imperative languages trivial?
Yes.
In Haskell it is too:
(>>=) :: IO a -> (a -> IO b) -> IO b
But we generalise that function signature to Monad:
(>>=) :: Monad m => m a -> (a -> m b) -> m b
We don’t have a built in idea of sequence. We just have functions like these. A generalisation which comes out is Monad. It just gives code reuse.
Maybe is an instance of a monad, and there are many different kinds of monads. If you think of Maybe as “a monad that uses and_then for sequencing”, then “vanilla” sequencing can be seen as “a monad that uses id for sequencing” (and Promises in JavaScript can be seen as “a monad that uses Promise#flatMap for sequencing”).
Yes, expressing sequence in eager imperative languages is trivial because you can write statements one after the other. Now imagine a language where you have no statements, and instead everything is expressions. In this expression-only language, you can still express sequence by using data dependencies (you hit this nail right on the head). What would that look like? Probably something like this (in pseudo-JavaScript):
function (next2) {
(function (next) {
next(f.readline())
})(function (readline_result) {
next2(print(readline_result))
})
}
with additional plumbing so that each following step has access to the variables bound in all steps before it (e.g. by passing a dictionary of in-scope variables). A monad captures the spirit of this, so instead of doing all the plumbing yourself, you choose a specific implementation of >>= that does your plumbing for you. The “vanilla” monad’s (this is not a real thing, I’m just making up this name to mean “plain old imperative sequences”) implementation of >>= just does argument plumbing for you, whereas the Maybe monad’s implementation of >>= also checks whether things are None, and the Promise monad’s implementation of >>= also calls Promise#then and flattens any nested promises for you.
What’s useful here is the idea that there is this set of data structures (i.e. monads) that capture different meanings of “sequencing”, and that they all have a similar interface (e.g. they have all an implementation of >>= and return with the same signature) so you can write functions that are generic over all of them.
Does that make sense?
There is a comment below saying it pretty succintly:
A monad is basically defined around the idea that we can’t always undo whatever we just did (…)
To make that concrete, readStuffFromDisk |> IO.andThen (\stuff -> printStuff stuff) - in the function after andThen, the “stuff” is made available to you - the function runs after the side effect happened. You can say it needed specific API and the concept of monads satisfies that API.
Modelling IO with monads allows you to run functions a -> IO b (take a pure value and do an effectful function on it). Compare that to functions like a -> b (Functor). These wouldn’t cut it - let’s say you’d read a String from the disk - you could then only convert it to another string but not do an additional effect.
EDIT: I might not have got the wording entirely right. I ommited a part of the type annotation that says the a comes an effect already. With Functor you could not reuse values that came from an effect; with Monad you can.
Oddly - this sounds like the author has just discovered dependency injection? I would have thought that concept would translate pretty well to Go. I’ve written a lot of Go, but I cut my teeth largely on C, C++, and C# so dependency injection has always been on my radar. When I wrote Go, I learned it and largely applied my own lessons from C, C++, and C#.
Due to compiler constraints and the language’s ethos, global state and func init feel weird in Rust (my current language). You can’t, without jumping through hoops, create objects that are partially constructed (e.g. using Option for uninitialized types). That said, even if you’ve got struct members that are of type Option, you are actually initializing it to something - sometimes it’s just None.
I don’t have enough context in Go land to know why this author’s argument might be a novel conclusion. Does anyone have some context? I’d love to learn more.
Many Go programmers seem to feel very comfortable with global state. When I join new organizations or projects, I often find myself needing to educate and socialize the problems that come from that. This post is just a formalization of the things I’ve been saying informally for a long time.
I wish I knew why this was so relatively prevalent in the Go ecosystem. If I had to guess, I’d speculate that it’s because a lot of the example code in e.g. books and tutorials doesn’t really shy away from global state.
It’s also related to the standard library itself having lots of globals. Which itself leads to bad things, like the cryptographic randomness source being trivially modifiable: https://github.com/golang/go/issues/24160
The Go community has a strong culture of writing application-specific code that is “good enough”, and tends to err strongly on the side of avoiding premature abstraction. For a significant number of use cases, globals (combined with a reasonable amount of documentation, testing, and good discipline) tend to be the “good enough” solution.
The thesis of Go’s culture is that premature abstraction often costs more than rewriting code. The argument is that you often know your domain better after writing a “good enough” first version, and that premature abstractions lock you in to specific designs more tightly that may be harder to change down the line.
It’s definitely not a conventional position, but it’s not indefensible – it’s hard to argue that Go developers are not pragmatic (or not productive).
Yup, this was my comment when this appeared a year ago on HN:
In other words, use a pure dependency-injection style, and ZERO globals.
This is scary. Because even at the time of install, after carefully reading all reviews, and inspecting the network tab everything looks okay. It’s only weeks/months later that the malicious code gets remotely loaded and executed.
Google needs to take a hard look at their Chrome Extension Store.
I just checked, there doesn’t seem to be an official way to stop Chrome extensions from auto-updating. This is madness.
Meanwhile, in Firefox, you can just click the checkbox to disable automatic updates in the add-ons manager (either globally or for a particular add-on).
They also seem to have a pretty decent review process for add-ons; I don’t know whether they’d catch malicious updates, but when I submitted my add-on, I got actual technical feedback about the code (like a suggestion to replace innerHTML with textContent on a particular line).
With how great Firefox Quantum is these days, maybe it’s time to consider switching?
A power user can always download the extension’s source code and install it separately in developer mode. Since that counts as a different “extension” than the one offered in the store, it won’t automatically update.
I’m probably in the minority here, but I think automatic updates as a default is a good idea. Ideally there would be a way to turn this off, but defaulting to automatic updates is better than defaulting to manual updates. The vast majority of updates fix real bugs and the vast majority of users won’t update manually.
I also think automatic updates as a default is a good idea. What I find odd is that you can’t disable auto-update on a per extension basis. That way I can only authorize official extensions to auto-update and then decide for every other extension. It would definitely reduce the risk of a popular extension being bought and re-uploaded with malicious behavior.
I’m a little perplexed that salary is an afterthought here–in a field where what we do is so directly tied to outsized profits, it’s weird we aren’t pushing harder to get a cut of the pie.
I unholstered my sarcasm gun, paused to think about it, and put it away again. I fully agree with you. I recently read this book which touches on the matter. I love the stability of being an employee and I hate dealing with clients for a reason or another. I also hate very many things about corporate life, and I wanna break out of that at some point, although it ain’t possible nor practical right now. Anyway I digress, it was a good read.
Because a higher salary won’t make you happier.
A good example of this is the amount of people unhappy at work, asking for a raison to justify their stay.
Anyway, what you seem to want is a better distribution of income in a company, no specifically higher salary, so the question would be, is income equality a factor of hapiness? Maybe.
No, I specifically want a better cut of the value that I produce.
If I grow sales by 5x by implementing a feature, there are some options, right?
Only two of those aren’t terrible. Only one of them is fair.
If you want to see a developer act do work that can bring in 10 million, cut them in for 5%. I can’t think of any dev who wouldn’t move heaven and earth to make 500K in a year.
Of course, the current system is more a deal of “You’re lucky to have a job here, here’s the salary, management/shareholders will skim off the profits that, by construction, they themselves could not have realized had you (or devs like you) agreed to it.”
And honestly, while I respect the problems of folks who aren’t in our industry making our wages, I also work very hard not to have those problems. I have no desire to be holding the bag when market trends correct themselves and we’re paid the same as non-devs while the people we let fleece us are fucking off in their Teslas to Moneyland.
This sounds like a great idea in principle, but how do you attribute whose work produced what value? This seems like a hard question to answer in general (i.e. not just for engineering roles), with maybe the exception of direct sales roles (where a commission based on deal size is often the norm). Even in sales roles, I think attribution is a hard question to answer: are your sales great because your salespeople are great or your application engineer is great or the intern you hired produced a ton of value by fixing a bunch of stuff that nobody had bothered to?
When I worked in adtech, we had similar difficulty trying to attribute clicks to specific ads. The honest truth seems to be that, in both ads and work, it’s hard to do attribution “fairly” when you have a high-touch process involving many people.
So, there’s a few different parts of that, but the one I’ll poke at is attribution.
A lot of sales folks work on commission: and yeah, that has pathologies, but it’s the case more often than not that a salesperson that puts in the work to seal a deal is pretty unquestionably the one that deserves a cut of the sale.
The idea that we can’t do basic accountability in engineering is something I disagree with. Some solutions:
At least in some fields, say e-commerce, it’s pretty obvious how to break things down. If an engineer builds the product page, builds the order logic, and builds the persistence, they’re pretty obviously the ones that deserve the credit. If one team builds, say, product search, it’s pretty easy to track what generated a sale and how the customer got there (they’re tracking the customers, right?) and give them a cut.
And one immediate objection to this is “but how do engineers that don’t do customer-facing stuff get rewarded?” And my answer to that is basically: if an engineer doesn’t directly do stuff that puts money in the hands of the business, they don’t actually generate value for the company and as such shouldn’t be rewarded a cut of the spoils. From a business standpoint, a bad engineer that ships continuously and drives sales is worth infinitely more than a great engineer that refactors in pursuit of perfection.
I’m still debating internally how hard I believe this line of reasoning, but it’s opening up some interesting tangents in my head so I don’t think it should be dismissed outright.
I am an SRE for an online retailer (I am not, but the sake of the argument I am).
If I screw up big time, I completely halt all the sales. On the other hand, if I do decently my job, my work goes unnoticed.
So, according to your model, should I earn the entire company profit or should I earn nothing?
Well, as I wrote, you don’t drive sales or make savings, you have rated new value…you’ve acted to preserve existing value.
You should still be well compensated for your work! Just not with a cut of the growth.
I disagree, a bit. A decent SRE is going to prevent an incredible amount of screwing ups that would potentially cost any given company a lot of money. In a way, their presence is a form of risk mitigation; there’s definitely value in that, but it’s less obvious. For example: At some point in my career I was asked to implement in emergency a feature that essentially mitigated a risk that, should it realize, would have cost them in the tens of millions. Once mitigated, the risk didn’t exist anymore, and I’d bet that there’s SOME value in that risk having been permanently mitigated. Probably less than tens of millions, but probably more than a pat on the back.
[edit:] I mean, it’s less obvious as opposed to “See, since I tweaked this endpoint last week sales have increased by 20k per day, where’s my cut?”
Individual contributors that actually lay hands on a feature and implement it get a cut of sales that touch that feature, or cost savings if it’s an efficiency improvement.
This might be combined with “competitive coder” schemes to get even more results if they themselves are getting the results they claim.
This is such a sad story.
This is also a good example of why making an android level user friendly desktop Linux distribution should be number one priority for FOSS developers. We need UX designers, Artists, and non-technicals in FOSS.
Development is one of the few “creative” areas where working for free is considered not-strange (it’s still not common). For various cultural reasons that essentially stem from the abuse of artists, most higher creatives strongly resist working for free.
If my phone were charged I’d take a picture of the document that discourages free work of just about any form to the creative students at my university.
A big part of this is probably the existence of copyleft licenses which provide a legal mechanism to guarantee that continued work will be contributed back to the community. I wonder if a similar mechanism exists for artists (perhaps creative commons?).
It’s also important that software has a useful notion of “contributing back to the original work”, and that contributing is both standardised (my copy of git and the language compiler probably works the same as yours) and idiomatic (diffs, patches, and PRs are all well-known tools). It seems possible in theory to have large, open source, collaborative design projects (where many designers contribute back to a single project under copyleft terms), but I’m not aware of a “standard” design format with both a critical mass of users and good support for decentralised contributions.
I think free work should be discouraged if someone else is profiting. However if it such a taboo because they have a history of being abused and low quality of life perhaps we should start an money pool to pay for contributors who don’t have a higher paying job like software development.
Personally, I don’t think FOSS is a good fit in that situation.
Just my opinion, but I feel the open source model works best when the contributors are working on the project because it’s something they want or need themselves, OR because the project offers some kind of unique and interesting challenge.
A novice friendly desktop doesn’t fall into either category.
Funny because the android OS gets a LOT of contributions. Just because something is novice friendly doesn’t mean it has to be crippled in any way. Also there’s a lot of unique and interesting challenges in making a user friendly operating system.
This project, like others I’ve found, doesn’t seem handle to Python dependencies properly. For example, Flask 0.12 is listed as having no dependencies, but actually it depends on Werkzeug, Jinja2, click, and itsdangerous. [1]
I think this is because these sites are grabbing package JSON data from Pypi, but many (most?) packages don’t declare their dependencies there. As far as I know the only way to accurately resolve dependencies for Python packages is
I’ve been working on a project that actually has the correct dependency graph for Python libraries. I’ve had to follow the approach above. It’s not quite ready to show the world, but I’m hoping it’ll be interesting for people.
Yeah python dependencies are not easily machine readable, and for the moment I’m trying to avoid executing setup.py files downloaded from the internet on the Libraries.io servers, any help contributing better python support would be great.
I’m trying to avoid executing setup.py files downloaded from the internet on the Libraries.io servers
That’s wise - I’ve seen all sorts of shenanigans in those files. Just importing some of them causes attempted sudo operations.
I’m not particularly familiar with the python world, but this sounds like the perfect use-case for containers, no?
Note I said containers not “docker”. I believe what you want is a quick “spin up $distro, install $package, analyse installed deps” flow, which imo would suit lxc/lxd perfectly.
I’ve hacked something similar together over here: https://github.com/librariesio/pydeps
Yup. The place I work at (shameless plug: https://fossa.io) does this using ephemeral Docker containers.
We scan projects to check if they’re compliant with the licenses of their open source libraries. To do this, we need to compute the dependency graph of a project. For most build systems (the exceptions are usually NPM and Golang tools), this means running a full build to execute any arbitrary build scripts.
If you only do static analysis of package manifests, you tend to overreport and underreport – you’ll miss packages brought in by build scripts, and you’ll have extra packages (or extra versions of packages) that are included in the manifest but might be unused/optimised out by the build system/brought in by version constraint solver weirdness.
FOSSA is hiring. We build tools to help enterprises use open source more effectively. The majority of code deployed to production today comes from a third party (often open source), but there are no good tools for doing risk management (e.g. license compliance, vulnerabilities, etc.) for enterprises looking to adopt open source at an industrial scale.
Our initial offering is license compliance, but our core technical competency is build and component analysis (we’ve released some of our analysis tools as open source here) with applications in security and more.
We went to market around a year ago and have seen massive adoption with zero marketing, including high-profile customers like Twitter, Docker, Oath (Verizon / Yahoo / Tumblr), Hashicorp, Tesla, the Linux Foundation, and the JS Foundation. If you use modern open source, chances are that you’ve used something that relies on FOSSA (e.g. Webpack, Kubernetes, Docker).
Personally, I enjoy working here because I’m working on an interesting technical problem, building a sustainable business (early-stage startups with real business models and real revenue are surprisingly hard to find in SF), and working with friendly teammates on a transparent and trusting (psychologically safe) team.
We’re hiring generalist software engineers, customer success managers, and people ops. SF is ideal, but remote is doable if there’s a good fit.
Happy to answer questions here, over DM, or addressed to leo@fossa.io.