This news caused the public release for XSA-267 / CVE-2018-3665 (Speculative register leakage from lazy FPU context switching) to be moved to today.
These embargoed and NDA’d vulnerabilities need to die. The system is broken.
edit: Looks like cperciva of FreeBSD wrote a working exploit and then emailed Intel and demanded they end embargo ASAP https://twitter.com/cperciva/status/1007010583244230656?s=21
Prgmr.com is on the pre-disclosure list for Xen. When a vulnerability is discovered, and the discoverer uses the responsible disclosure process, and the process works, we’re given time to patch our hosts before the vulnerability is disclosed to the public. On balance I believe participating in the responsible disclosure process is better for my customers.
Pre-disclosure gives us time to build new packages, run through our testing process, and let our users know we’ll be performing maintenance. Last year we found a showstopping bug during a pre-disclosure period: it takes time and effort to verify a patch can go to production. With full disclosure, we would have the do so reactively, with significantly more time pressure. That would lead to more mistakes and lower quality fixes.
This is a bad response to the issue. The bad guys probably already have knowledge of it and can use it. A few players deemed important should not get advanced notification.
Prgmr.com qualifies for being on the Xen pre-disclosure list by a) being a vendor of a Xen-based system b) willing and able to maintain confidentiality and c) asking. We’re one of 6 dozen organizations on that list–the criteria for membership is technical and needs-based.
If you discover a vulnerability you are not obligated to use responsible disclosure. If you run Xen you are not obligated to participate in the pre-disclosure list. The process consists of voluntary coordination to discover, report, and resolve security issues. It is for the people and organizations with a shared goal: removing security defects from computer systems.
By maintaining confidentiality we are given the ability, and usually the means to have security issues resolved before they are announced. Our customers benefit via reduced exposure to these bugs. The act of keeping information temporarily confidential provides that reduced exposure.
You have described a voluntary process with articulable benefits as “needing to die,” along with my response being “bad.” As far as I can tell from your comments you claim “the system is broken” because some people “should not get advanced notice.” I’ve described what I do with that knowledge, and why it benefits my users. I’m thankful the security community tells me when my users are vulnerable and works with me to make them safer.
Can you improve this process for us? Have I misunderstood you?
Some bad guys might already have knowledge of it. Once it’s been disclosed, many bad guys definitely have knowledge of it, and they can deploy exploits far, far faster than maintainers, administrators and users can deploy fixes.
You’re treating “the bad guys” like they’re all one thing. In actuality, there’s a string of bad guys from people who will use a free, attack tool to people who will pay a few grand for one to people who can customize a kit if it’s just a sploit to people who can build a sploit from a description to rare people who had it already. There’s also a range in intent of attackers from DOS to data integrity to leaking secrets. The folks who had it already often just leak secrets in stealthy way instead of do actual damage. The also use the secrets in a limited way compared to average, black hat. They’re always weighing use vs detection of their access.
The process probably shuts down quite a range of attackers even if it makes no difference for the best ones who act the sneakiest.
The process probably shuts down quite a range of attackers even if it makes no difference for the best ones who act the sneakiest.
I believe the process is so effective at shutting down “quite a range of attackers” that it works despite: a) accidental leaks [need for improvement of process] b) intentional leaks [abuse] c) black hats on the pre-disclosure list reverse engineering an exploit from a patch. [fraud] In aggregate, the benefit from following the process exceeds the gain a black hat would have from subverting it.
Well, it’s complicated. (Disclosure: we were under the embargo.)
When a microprocessor has a vulnerability of this nature, those who write operating systems (or worse, provide them to others!) need time to implement and test a fix. I think Intel was actually doing an admirable job, honestly – and we were fighting for them to broaden their disclosure to other operating systems that didn’t have clear corporate or foundation backing (e.g., OpenBSD, Dragonfly, NetBSD, etc). That discussion was ongoing when OpenBSD caught wind of this – presumably because someone who was embargoed felt that OpenBSD deserved to know – and then fixed it in the worst possible way. (Namely, by snarkily indicating that it was to address a CPU vulnerability.) This was then compounded by Theo’s caustic presentation at BSDCan, which was honestly irresponsible: he clearly didn’t pull eager FPU out of thin air (“post-Spectre rumors”), and should have considered himself part of the embargo in spirit if not in letter.
For myself, I will continue to advocate that Intel broaden their disclosure to include more operating systems – but if those endeavoring to write those systems refuse to honor the necessary secrecy that responsible disclosure demands (and yes, this means “embargoed and NDA’d vulnerabilities”), they will make such inclusion impossible.
We could also argue Theo’s talk was helpful in that the CVE was finally made public.
Colin Percival tweeted in his thread overview about the vulnerability that he learned enough from Theo’s talk to write an exploit in 5 hours.
If Theo and and the OpenBSD developers pieced enough together from rumors to make a presentation that Colin could turn into an exploit in hours, how long have others (i.e., bad guys) who also heard rumors had working exploits?
Theo alone knows whether he picked-up eager FPU from developers under NDA. Even if he did, there’s zero possibility outside of the law he lives under (or contracts he might’ve signed) that he’s part of the embargo. As to the “spirit” of the embargo, his decision to discuss what he knew might hurt him or OpenBSD in the future. That was his call to make. He made it.
Lastly, I was at Theo’s talk. Caustic is not how I would describe it, nor would I categorize it as irresponsible. Theo was frustrated that OpenBSD developers who had contributed meaningfully to Spectre and Meltdown mitigation had been excluded. He vented some of that frustration in the talk. I’ve heard more (and harsher) venting about Linux in a 30 minute podcast than all the venting in Theo’s talk.
On the whole Theo’s talk was interesting and informative, with a sideshow of drama. And it may have been what was needed to get the vulnerability disclosed and more systems patched.
Disclosure: I’m an OpenBSD user, occasional port submitter, BSDCan speaker and workshop tutor, FreeNAS user and recommender, and have enjoyed many podcasts, some of which may have included venting.
If Theo and and the OpenBSD developers pieced enough together from rumors to make a presentation that Colin could turn into an exploit in hours, how long have others (i.e., bad guys) who also heard rumors had working exploits?
It was clear to me the day Spectre / Meltdown were disclosed that there would be future additional vulnerabilities of the same class based on that discovery. I think there is circumstantial evidence suggesting the discovery was productive for the people who knew about it in the second half of 2017 before it was publicly disclosed. One can safely assume black hats have had the ability to find and use novel variations in this class of vulnerability for at least six months.
If Theo did pick up eager FPU from a developer under embargo that demonstrates just how costly it is to break embargo. Five hours, third hand.
If Theo did pick up eager FPU from a developer under embargo that demonstrates just how costly it is to break embargo. Five hours, third hand.
I have absolutely no idea what point you’re trying to make. Certainly, everyone under the embargo knew that this would be easy to exploit; in that regard, Theo showed people what they already knew. The only new information here is that Theo is every bit as irresponsible as his detractors have claimed – and those detractors would (of course) point out that that information is not new at all…
With respect, how is Theo irresponsible for reducing the time the users of his OS are vulnerable?
Like, the embargo thing sounds a lot to the ill-informed like some kind of super-secret clubhouse.
Theo definitely wasn’t part of the embargo, but it’s also unquestionable that Theo was relying on information that came (ultimately) from someone who was under the embargo. OpenBSD either obtained that information via espionage or via someone trying to help OpenBSD out; either way, what Theo did was emphatically irresponsible. Of course, it was ultimately his call – but he is not the only user of OpenBSD, and is unfortunate that he has effectively elected to isolate the community to serve his own narcissism.
As for the conjecture that Theo served any helpful role here: sorry, that’s false. (Again, I was under the embargo.) The CVE was absolutely going public; all Theo did was marginally accelerate the timeline, which in turn has resulted in systems not being as prepared as they otherwise could be. At the same time, his irresponsible behavior has made it much more difficult for those of us who were advocating for broader inclusion – and unfortunately it will be the OpenBSD community that suffers the ramifications of any future limited disclosure.
Espionage? You’re suggesting one of:
Someone stole the exploit information, leaked it to the OpenBSD team, a team known for proactively securing their code, on the off-chance Theo would then further leak it (likely with mitigation code), causing the embargoed details to be released sooner than expected,
OpenBSD developers stole the exploit information, then leaked it (while committing mitigation code), causing the embargoed details to be released sooner than expected.
The first doesn’t seem plausible. The second isn’t worthy of you or any of the developers on the OpenBSD team.
I’m sure you’ve read Colin’s thread. He contacted folks under embargo after he wrote his exploit code based on Theo’s presentation. The release timeline moved forward. OSs that had no knowledge of the vulnerability now have patches in place. Perhaps those users view “helpful” in a different light.
Edit: Still boggling over the espionage comment. Had to flesh that out more.
Theo has replied:
In some forums, Bryan Cantrill is crafting a fiction.
He is saying the FPU problem (and other problems) were received as a leak.
He is not being truthful, inventing a storyline, and has not asked me for the facts.
This was discovered by guessing Intel made a mistake.
We are doing the best for OpenBSD. Our commit is best effort for our user community when Intel didn’t reply to mails asking for us to be included. But we were not included, there was no reply. End of story. That leaves us to figure things out ourselves.
Bryan is just upset we guessed right. It is called science.
He’s also offered to discuss the details with Bryan by phone.
Intel still has 7 more mistakes in the Embargo Execution Pipeline™️ according to a report^Wspeculation by Heise on May 3rd.
Let the games begin! 🍿
What’s (far) more likely: that Theo coincidentally guessed now, or that he received a hint from someone else? Add Theo’s history, and his case is even weaker.
While everyone is talking about Theo, the smart guys figuring this stuff out are Philip Guenther and Mike Larkin. Meet them over beer and discuss topics like ACPI, VMM, and Meltdown with them and you won’t doubt anymore that they can figure this stuff out.
In another reply you claim your approach is applied Bayesian reasoning, so let’s go with that.
Which is more likely:
or
Show me the observed distribution you based your assessment on. Show me all the times Theo lied about how he came to know something.
Absent meaningful data, I’ll go with team of smart people knowing their business.
Absent meaningful data
Your “meaningful data” is 11 minutes and 5 seconds into Theo’s BSDCan talk: “We heard a rumor that this is broken.” That is not guessing and that is not science – that is (somehow) coming into undisclosed information, putting some reasonable inferences around it and then irresponsibly sharing those inferences. But at the root is the undisclosed information. And to be clear, I am not accusing Theo of lying; I am accusing him of acting irresponsibly with respect to the information that came into his possession.
Here is at least one developer’s comment on the matter. He points to the heise.de article about Spectre-NG as an example of the rumors that were floating around. That article is a long way from “lazy FPU is broken”.
Theo has offered to discuss your concerns, what you think you know, what he knew, when and how. He’s made a good-faith effort to get his cellphone number to you. If you don’t have it, ask.
If you do have his number, call him. Ask him what he meant by “We heard a rumor that this is broken.” Ask him what rumor they heard. Ask him whether he was referring to the Spectre-NG article.
Seriously, how hard does this have to be? You engaged productively with me when I called you out. You’ve called Theo out. Talk to him.
And yes, I get it. Your chief criticism at this point is responsible disclosure. But as witnessed by the broader discussion in the security community, there’s no single agreed-upon solution.
While you’ve got Theo on the phone you can discuss responsible disclosure. Frankly, I suggest beer for that part of the discussion.
Edit: Clarify that Florian wasn’t saying he knew heise.de were the source.
It is Bayesian reasoning, pure and simple.
That said, this is a tempest in a teacup, so call it whatever you want; I’m gonna go floss my cat.
Sorry – I’m not accusing anyone of espionage; apologies if I came across that way.
What I am saying is that however Theo obtained information – and indeed, even if that information didn’t originate with the leak but rather by “guessing” as he is now apparently claiming – how he handled it was not responsible. And I am also saying that Theo’s irresponsibility has made the job of including OpenBSD more difficult.
The spectre paper made it abundantly clear that addtional side channels will be found in the speculative execution design.
This FPU problem is just one additonal bug of this kind. What I’d like to learn from you is:
What was the original planned public disclosure date before it was moved ahead to today?
Do you really expect that a process with long embargo windows has a chance of working for future spectre-style bugs when a lot of research is now happening in parallel on this class of bugs?
The original date for CVE-2018-3665 was July 10th. After the OpenBSD commit, there was preparation for an earlier disclosure. After Theo’s talk and after Colin developed his POC, the date was moved in from July 10th to June 26th, with preparations being made to go much earlier as needed. After the media attention today, the determination was made that the embargo was having little effect and that there was no point in further delay.
Yes, I expect that long embargo windows can work with Spectre-style bugs. Researchers have been responsible and very accommodating of the acute challenges of multi-party disclosure when those parties include potentially hypervisors, operating systems and higher-level runtimes.
Thanks for disclosing the date. I must say I am happy that my systems are already patched now, rather than in one month from now.
I’ll add that some new patches with the goal of mitigating spectre-class bugs are being developed in public without any coordinated disclosure:
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/9474cbef7fcb61cd268019694d94db6a75af7dbe
Thanks for the clarification.
I don’t think early disclosure is always irresponsible (the details of what and when matter). Others think it’s never irresponsible; and some that it’s always irresponsible. Good arguments can be made for each position that reasonable people can disagree about and debate.
One thing I hope we can all agree on is that we need clear rules for how embargoes work (probably by industry). We need clear, public criteria covering who, what, when and how long. And how to get in the program, ideally with little or no cost.
It’s a given that large companies like Microsoft will be involved. Open-source representatives should have a seat at the table as well. But “open source” can’t just mean Red Hat and a few large foundations. OSs like OpenBSD have a presence in the ecosystem. We can’t just write the rules with a “You must be this high to ride” sign at the door.
And yeah, Theo’s talk might make this more difficult going forward. Hopefully both sides will use this event as an opportunity to open a dialog and discuss working together.
Right, I completely agree: I’m the person that’s been advocating for that. I was furious with Intel over Spectre/Meltdown (despite our significant exposure, we learned about it when everyone else did), and I was very grateful for the work that OpenBSD and illumos did together to implement KPTI. This time around, I was working from inside the embargo to get OpenBSD included. We hadn’t been able to get to where we needed to get, but I also felt that progress was being made – and I remained optimistic that we could get OpenBSD disclosure under embargo.
All of this is why I’m so frustrated: the way Theo has done this has made it much more difficult to advocate this position – it has strengthened the argument of those who believe that OpenBSD should not be included because they cannot be trusted. And that, in my opinion, is a shame.
Look at it from OpenBSD’s perspective though. They (apparently) tried emailing Intel to find out more, and were told “no”. What were they supposed to do? Just wait on the hope that someone, somewhere, was lobbying on their behalf to be included, with no knowledge of that lobbying?
I think Bryan’s straw-manning the Amazon leadership principles a bit. Integrity is woven throughout essentially all of them; earning trust, insisting on the highest standards, and customer obsession all mandate a high foundational level of integrity. That’s not to say every Amazon product, process or leader succeeds all the time, but it’s a disservice to tens of thousands of high-integrity Amazonians to suggest we don’t care deeply about it.
(disclaimer: I work for Amazon, but don’t speak for Amazon here)
I don’t agree in that I don’t think that integrity is in fact implied by any of them. For example, you can easily earn trust without integrity (e.g. Madoff Investment Securities), insist on high standards without integrity (e.g., Enron), and be customer obsessed without integrity (e.g., Uber). I’m not accusing Amazon of not having integrity (or at least, not necessarily), but I do accuse it of not having integrity as an espoused core principle (or even among the first 14!). And the bigger problem isn’t Amazon (frankly), it’s those that Amazon inspires, like (say) Uber. Uber’s core values (the idea for which supposedly came from an exec who came from Amazon) are entirely devoid of integrity – and indeed, some (like “fierceness” and “encourage toe-stepping”) practically assure that integrity be sacrificed to satisfy them.
I’ll agree it’s possible to read the Amazon LPs the way you do, but I think that’s unnecessarily incharitable in service of your (otherwise great and thought provoking) presentation.
At Amazon, ‘Earns Trust’ means what it literally says on its face; to earn the trust of your customers, your peers, your management, and your reports, without equivocation. And it’s drilled in, correctly, that trust is fragile; it is far easier to break it than to ever fix it once it’s broken. In order to keep and maintain trust, you have to work with integrity, compassion, an adult viewpoint, introspection, deliberation, openness, thoughtfulness, and a bunch of other non-mentioned (but critically important) motivations, and you have to keep those front and center all the time.
I think it’d be a pretty great idea if we exposed some of the internal training materials about the principles to the public. I think it’d help in discussions like this. I’ll bring that up.
My new minimum bar for a companies value statement is “Would Uber’s behaviour break these principles?”. Looking at Amazon’s principles, I don’t really see anything there that would be inconsistent with how Uber runs their business except for perhaps “Earn Trust”. I don’t think they have the same ethical values, but they don’t distinguish themselves from Uber in those principles.
This person going on about integrity is of the company that dropped lifetime plans they’d sold once they became too expensive, right?
Yes – though the lifetime plans long pre-date me (indeed, they date back to 2004), and the decision to turn them off in 2012 was one that I (unfortunately) had nothing to do with. (I was the VP of Engineering at the time.) No executive involved in any aspect of lifetime accounts (either the decision to offer them or the unfortunate decision to end them) has anything to do with the company – and indeed, the Joyent of that era is what many of us veterans refer to as the Bad Old Days. When I came to Joyent in 2010, I quickly realized that the founders of the company and I had some fundamental disagreements at some very basic levels – of which the lifetime plans (both that they were offered and the way in which they were terminated) is a particularly vivid example. The lesson from it all for me was that values absolutely must emanate from the top of a company, and that the most important (and most difficult) property to vet in any executive is their integrity. Fortunately, we at Joyent now have the luxury of working for a high-integrity CEO – but it took a while to get there (he was hired in 2014, over two years after the founding CEO was fired), and then longer still for him to build an organization around him.
If anything, this entire odyssey has made me much more overt and assertive about my own values. So yes, it’s the same company (or the same name, anyway) – and in a perverse and roundabout way, that’s not an accident.
It would be a violation of integrity if they did not honor the lifetime plans that they sold. There’s nothing wrong with changing your pricing structure. You can’t make them sell lifetime plans forever.
“I’m sorry, if you’ve got one principle in your organisation, its integrity? Right?”
Wrong. Effectiveness at achieving organization’s goals is the most important principle. Everything else comes second. If Im wrong, just get a market report on desktop OS’s where Microsoft shoukd either be bankrupt or less than 1% share. Similarly for mainframes where IBM probably is barely scraping by. Im probably typing this on an OpenMoko with cutting-edge hardware whose specs are open to aid trustworthy, competitive, driver development. As is EDA and fab’s cell libraries. Licensing for Oracle is a one-click process with exceedingly cheap results.
Good that high integrity prevails in the corporate world. I can wait to see what President(s) got elected and laws passed through the campaign contributions of good corporations. The future is probably bright for all economic classes.
Okay, so to play that out: if an organization has to choose between integrity and achieving its goals, it should choose to achieve its goals?! Perhaps we’re operating under different definitions of integrity (perhaps you have inferred it to mean “technical purity”?), but I can’t imagine how anyone would want to work in an organization where integrity is knowingly sacrificed to achieve larger goals – and I further don’t see how this could possibly be controversial.
To add another wrench to the mix: if your value system is that breaking the rules is ok to achieve goals, you can still have integrity (in the technical sense of the word) as long as you are clear that your values allow this. To use Uber as an example as well: the results of the Uber scandals, I find, ethically problematic. However they are completely in line with my understanding of the Uber culture and I’m not surprised or feel scandals demonstrate any dualism in Uber. Their communications have always been to do whatever it takes to succeed.
Note that this definition of ‘integrity’ is that someone/group is the same internally and externally. Unified. Which is, perhaps, a different definition that many people have or how it’s used day-to-day. This definition comes from Ray Dalio’s Principles book.
So, to play devil’s advocate, what competitive edge does integrity give in a modern business? What is the dollars-and-cents value of scruples?
Also, what do you you think caused this shift, if indeed it was a shift?
To me, it’s not a competitive edge – it’s a constraint. (And one that isn’t very constraining, honestly – especially if one takes a slightly longer view in terms of outcomes.)
As to what has caused a seeming shift away from valuing integrity, I can only speculate, but I suspect that it’s due to a confluence of factors – and I very much hope that it’s a localized and transient! That is, I hope that we will collectively look back on (say) Uber as symptomatic of the excesses of an era rather than a portent of worse to come…
And one that isn’t very constraining, honestly – especially if one takes a slightly longer view in terms of outcomes.
Yeah… Especially leadership changes or post-acquisition. I could never work for a company intending to get acquired or IPO while telling my customers I had their best interests in mind. Unless, I open-source almost everything and otherwise gave them a clean, exit strategy so a future tyrant couldn’t lock them in. Most tech people in proprietary or SaaS don’t do that. I know your company open-sourced a bunch of stuff. I don’t remember all the product details. Might be an exception. :)
Is this what we’re saying…
Do whatever makes money. If breaking the law doesn’t reduce profits, then break it. If being amoral (whose morality?!) doesn’t reduce profits, be amoral! etc.
Because that seems to be the contemporary mode.
I’m not saying that. I’m saying effectiveness. In some organizations, that means focus on money, In others, it might mean other things. Depends on the goal. Let’s illustrate with an more obvious and extreme example that happens in the real-world: non-profit that gets money to pay for cancer treatments, esp kids. The people on the sales end are naturally going to be talking to a lot of people who didn’t want anyone to sell them anything. They’re also going to talk to people that will only do it if they like the seller, works within their ideology, etc. As in sales in general, the person doing the selling should be bending their personality and truth to the ears of the listener to increase odds they’ll accepts while simultaneously constraining themselves enough to avoid long-term, negative impact. The latter is not even always necessary as the organization can “fire” the one person that got caught saying they were out of line and not representing the organization properly.
Meanwhile, this person acting on effectiveness more than integrity is saving a lot of lives getting a lot of donations. Acting with highest integrity for most people would reduce persuasive ability, reduce donations, and kill people as a side effect. Would you kill people regularly in such a job just to say you have higher integrity? ;)
Would you kill people regularly in such a job just to say you have higher integrity?
If you were receiving cancer treatment, would you accept it if you knew it came from swindling people?
I don’t know. That would be a tough decision. I stay realist. So, I’d decline and turn them in if I believed the money would get back to the victims. Otherwise, I’d be choosing between taking it or dying for the toys a LEO would buy for themselves after seizing it. I won’t die for that. So, I’d take the money and go after them if Im in remission. Also, a proxy to do same if I die.
I didn’t say I wanted to work at such a place. I countered the idea that integrity was most important by noting that most successful businesses sacrifice it in some way regularly to gain market share, profit, etc. All the market leaders do as well for all far as I can tell. So do almost all politicians and companies that write the laws that people with integrity are forced to operate under.
Given all that, I think businesses should be willing to sacrifice integrity when it helps them achieve goals, esp long-term. I prefer them to operate with integrity but their environment means most won’t. Many will even go under when competitors use low integrity marketing or cost-cutting strategies. Example: try competing in hardware market by having well-paid, first-world people assemble the components with 40 hour work week, breaks, vacations, and so on. You’ll go bankrupt if it ain’t defense or some high-margin industry.
I think both you and Bryan Cantrill have somehow conflated values & morals with capitalism. It’s apples and oranges. There is an argument to be made for (moral) principles/values, and there is an (economic) argument to be made for achieving goals when operating within the capitalist system. But the actions that these arguments would imply are not necessarily aligned and perhaps not even reconcilable.
Incidentally, I hold integrity in the highest regard in all endeavours (not just business). Amazing things can be built on the foundation of integrity.
the actions that these arguments would imply are not necessarily aligned and perhaps not even reconcilable.
So an action that’s carried out to achieve a capitalist end is simply external to any moral universe? The “values & morals” just don’t apply once an economic motivation comes into play?
many people here would love to say “no”, but it seems “yes” is the right answer here. Look at the “shareholder value” driven world today. More, more, more is the only thing that counts.
That’s not what I said. I said that acting morally may not be aligned with achieving success in a capitalist system, and may even be impossible in some cases (case in point: today’s oil companies). I’m not at all advocating amorality, but my point is that capitalism as a system has no built-in morality; morality is an independent set of constraints. People, however, have to operate within both sets of constraints, and choose their course of action accordingly.
Sure. I was just trying to poke a little at “conflated values & morals with capitalism”, because I don’t think the rant in this post does that. On my understanding, it’s saying rather that the problem with unconstrained capitalism is exactly that it doesn’t have values & morals applied to it often enough, if at all. I agree with that, and in my experience that’s usually the case because people metaphorically shrug their shoulders regarding values & morals, arguing “well of course not, my hands are tied, financial motivations have to take precedence, what would you have me do?”, as if this gives people and organisations free passes to behave badly in the name of profit. What I’d have people do would be to take a bit of personal responsibility for the world they live in and exercise a bit of values & morals in order to make that world a little bit less shitty, rather than them a little bit richer. In my mind, that’s what civilization is about; a generalized move away from selfish actions towards those aimed at benefiting a collective; and moves towards increased selfishness and individual people’s and organisation’s gains at the expense of others are steps back from civilization. But I recognise I probably live in cloud cuckoo land as far as most people are concerned, and the way politics and societies are moving right now seem to bear that out.
I may have misinterpreted the comment about Scott McNealy as I don’t know the details of that; it sounded like a moral judgement was cast as an act of a capitalist.
I agree with your view of civilisation, and I think it’s beneficial in that it’s aligned with our biological underpinnings - we evolved as social creatures, and hence complete selfishness and individualism are not natural for us.
Ah, got you. I thought the point of the McNealy reference was that even he, the most die-hard fan of free-market capitalism, still found it important to act morally and not lie or cheat or have to hide things from his children. Thus, that one could probably always make more profit by being less moral, but would then have to hide it from one’s children. Thus, that acting morally and making a profit aren’t incompatible, and being a capitalist doesn’t mean you have to act amorally (or immorally), but that you can reach some kind of compromise. And what Bryan Cantrill seemed to be saying was, hello, aren’t more and more companies refusing that compromise, targeting profit above all else, moral compass and/or integrity be damned, and isn’t that something we should be fighting against?
The thing is, coming from a Western European background, and having been raised by post-war Europeans, I find it hard to accept the kind of moral arguments for self-interest that US libertarians often make (e.g. that it’s immoral to provide health care, welfare, or any kind of help that stops people having to rely on themselves alone and fighting tooth and nail for survival) - and so I naturally assume that any morally positive act carried out in a capitalist system is morally positive because it’s intended to temper the capitalist/profit motive. Meaning that there are interpretations of ‘moral’ that are completely different from mine; meaning, although I disagree with them (natch), I might have misinterpreted the article as well, and almost certainly interpreted it according to my bias.
Regardless, you make a good point in your last paragraph too. I lapse all too easily into thinking “humans are all too selfish, there’s no way forward”, and what I see happening around right now tends to feed into that - but the reality is that humans are neither completely selfish and individualistic nor completely altruistic and collectivist, and the historical swing between the two is a reflection of that fact. Thanks for the reminder.
I like your interpretation of the post, I think it’s better than mine.
I also find it hard to accept arguments for self-interest. I come from an Eastern European background, where I saw the rise of kleptocratic capitalism and the tidal wave of self-interest that came along with it. I don’t think the results have been good.
That’s not what I said. I said that acting morally may not be aligned with achieving success in a capitalist system
That’s true. Those doing the most good need the most success in such a system if voters are apathetic. We see this with the big corporations straight-up writing laws to help them while hurting others. These pass quite a bit of the time. That makes capitalist outcomes and morality tied together in the current system in the U.S.. Probably many places. Good news is nothing stopped many of these billionaires from playing things beneficially once they got the market and got rich. People can still do good taking over as CEO’s, managers with decent budgets, and so on. Not always but plenty of the time.
P. T. Barnum thought integrity was important enough to make it one of his golden rules of money getting -
https://www.fourmilab.ch/etexts/www/barnum/moneygetting/moneygetting_chap21.html
Roger’s :proc appears twice in the index. The second link goes to the URL for DNS and the Art of Making Systems “Just Complex Enough” but the page loaded is, again, Roger’s :proc.
7074 says Hello World is missing entirely.
Anyone know to whom to forward these issues?
I’d try Bryan Cantrill. He might at least be able to direct you to the right person.
Sorry about that! We got the DNS link corrected. As for Marianne’s amazing talk, I’m afraid that we can’t make the recording available for legal reasons: while Marianne was authorized to give the talk (and her content was approved) she didn’t have the necessary White House approval to make the video available. Now, one can obviously be bummed out that the White House (and the government more generally) is getting in its own way here (there was nothing in her talk that shouldn’t be made public), but (for my own sanity) I choose to look at it the other way: we are (very!) lucky that technologists like Marianne not only put up with the government but actively try to improve it! So… sorry about that. You missed a great one, but if we put everything up, what reason would there be to make it to the next Systems We Love? ;)
According to All the talks from Systems We Love, 7074 says Hello World wasn’t recorded.
DNS and the Art… can be found 7h4m4s into the livestream.
Genius! I really enjoyed watching it remotely. And, Bryan, if you ever wanna do that in Europe, happy to help ;)
So, we’re trying to figure that out. Clearly, we’ve tapped into something here – but I imagine I don’t have to tell you that running a conference isn’t exactly a low-stress or low time-commitment endeavor. We also want to figure out what, exactly, constitutes the Systems We Love “brand” so we can franchise it out for others to run their own Systems We Love conferences. We also need to figure out sponsorships for future conferences; we at Joyent have no interest in making money off this, but losing less money would probably be nice. ;)
Definitely welcome ideas on any/all of this! The one thing we feel we know is that there is clearly demand for this content – which itself is inspiring!
Right. Yeah, as a co-chair of the European Data Forum 2013 and countless user groups I think I have a bit of an idea what it means. Suggest you have a look at the super successful Devoxx franchise here in Europe as well, as a reference. I’d immediately know two locations in Europe (incl. partner and logistics): CodeNode in London (with SkillsMatter) and Software Circus in Amsterdam (with Container Solutions) + I can certainly ask our friends at MSFT if they wanna support this as well.
This is such an awesome format and it would be a really great thing to make this accessible in person for Europe. Happy to do whatever it takes to make it happen!
This looks like it’ll be a lot of fun. I’m planning on taking a 1 day trip up from southern California to attend.
We’re glad you’re coming, and excited to see the enthusiasm for it – as simple as the notion is, it seems to have struck a nerve. And please consider submitting a talk idea as well!
The whole conference reminds me a lot of !!Con, which I spoke at in 2014. I plan to submit a talk for this, though haven’t quite stumbled upon a good idea yet.
Thanks for putting this together!
The similarity to !!Con is half (but only half) coincidence: when I approached Julia about being on the PC for Systems We Love, she mentioned her work on !!Con, a conference that I have never attended but have coveted from afar. Julia agreed to be on the Systems We Love PC (w00t!) and gave some very helpful feedback in terms of both drafting the call for proposals and structuring the mechanics of the PC. So while not directly inspired by !!Con, it’s also true that it taps into the same zeitgeist – may it be as successful!
Nice write-up on i432. Glad you took some time to read on it. I’ll add that Intel learned from their mistake quite a bit in the design of i960. It’s worth checking out. It’s basically a RISC + object descriptors + error handling + fault tolerance + parallel processing. It overdid it with more mistakes due to requirements of ridiculous BiiN project. They canceled that then started selling it as an embedded CPU. It performed well. Moreover, I see a subset of it that would’ve made a fine alternative to Intel 286, etc with way better security, reliability, and multicore later.
https://en.wikipedia.org/wiki/Intel_i960
What did you think of it?
This article is obviously terrible: inflammatory link-bait that is not merely technically naïve but technically ignorant and actively dangerous. Not that I would expect any less, but the lobste.rs crew clearly agrees: many (many!) more upvotes on the comments decrying the article that the article itself. So, serious question: why is this the top-rated lobste.rs article right now? Obviously, everyone here is disaffected by HN to some degree or another, but an HN algorithm that I actually like is that articles that receive more comments than upvotes are assumed to be generating more heat than light and drop off the front page. Worth considering something similar here?
Congratulations! Does this mean that Joyent will have the HR infrastructure to start hiring remote workers in Europe?? :)
if you need an office in brisbane australia, ive got one ready to go ;)
itd be nice to have alex back too.
That this (excellent, must-read) article was ignored on HN but rose to the top on lobste.rs really says it all for me; good on us all!
First, I have to say that I loved giving this presentation. Huge kudos to both the Papers We Love movement and in particular to the Papers We Love NYC team – especially Zeeshan Lakhani who set this whole thing up and was generally great to work with.
Secondly, I just wanted to offer myself up to be corrected on anything I mischaracterized about FreeBSD Jails. I tried to get the technical details correct, but the reality is that I know infinitely more about zones than jails, and may well have gotten some bit of history or other technical detail incorrect. (In particular, I still struggle with the fact that Sys V IPC hasn’t been virtualized on a per-jail basis; I don’t see how that could possibly be true, but it seems to be?!)
Finally, I apologize for the length: 1h45m is a bit of a truck pull!
This is such a fantastic talk, thank you! I giggled and learned a lot.
The biggest “a-ha!” moment for me was when you talked about the zone console - that you have access to the console and can log into it even before the zone is booted is one of those things I’ve often wanted in jails. Turns out it’s an enormous yak shave making it work! (So I guess let’s call it technically impossible)
Thanks for the kind words and I’m glad you enjoyed it – and if my only service is to explain that implementing the console was a giant pain in the butt, I think I’ve done right by Dan, if no one else. ;)
The documentation, does seem to imply that Sys V IPC is not virtualized – only allowed or disallowed in a jail. I also found a fairly recent mailing list thread, talking about the desire for the per jail namespacing of it.
I had found the docs, but not the mail thread – especially given its recency, that pretty much says it all! I’m still a bit flabbergasted that this still isn’t done 15+ years after the original work; jails are woefully incomplete with respect to databases as long as this remains unimplemented!
As a predominantly FreeBSD user, it is a bit depressing.
Lately I was playing around with SmartOS, and found the UI of vmadm to be quite nice, even with the use of json¹. It’s like someone actually considered the UI of it! In comparison, the usability of jails (and associated tooling) is rather awful – even the jail.conf file is some weird custom format thing. ;_;
¹: I guess I would prefer something like yaml. The inability to add comments to json is sometimes irritating.
This was a great talk – especially since I could watch it while donating platelets (an almost 2-hour process), so the timing was perfect for me!
btw, in this talk – moreso than others of yours I’ve seen – you remind me of Lewis Black (the comedian), in a good way.
And having lived through the dot-bomb, and seeing companies I worked for spend $$$ on E10Ks, it was great to hear a bit around that from the inside.
Interesting but mildly scary: patching out a sleep call from a running process.
a PostgreSQL function … was explicitly sleeping as part of the cost-based vacuum delay mechanism, which attempts to limit resources used by vacuum operations. In this case, with all database activity effectively blocked on this operation, this delay was only hurting us. Since these tunables are not dynamically adjustable, we manually patched the running process to avoid sleeping in this case. This improved performance considerably.
“Mildly scary” is certainly correct; here’s the log from our chat at the time.
I believe Postgres creates a directory per database, and a file per table. I think each index is a separate file too?
EDIT: reference
Ya, that’s what I thought. For somebody to notice that there’s a pause each time a new file is opened, that’s a lot of tables.
Postgres chunks its data across files – so it’s not that it was a lot of tables, just a lot of data. And there wasn’t a pause when it was opened, it was that the files themselves were on the platters and we could see Postgres coming off CPU and blocking on I/O. Using DTrace, we easily determined which files were inducing this – and could see that it was marching through them in numerical order. Based on this, we prefetched manually to stay ahead of it (and then ran into the second issue).
Nice! @bcantrill how did you find the AMA format and experience?
It was actually refreshingly good. To be totally honest, I went in assuming that this was the opportunity for every skeleton to come out of the closet (especially when the skeletons are so searchable!) and had prepared diplomatic answers to many mean-spirited questions. But surprisingly, the questions were (generally) earnest and thoughtful – and I especially enjoyed the questions from people who are contemplating computer science and/or software engineering and are wondering how to get started. Overall, a positive experience – and much, much more positive than I had anticipated.
FWIW thanks! I quite enjoyed reading though it this morning.
Thanks also for listing some recent talks in your opening. I admit to having been a bit baffled by the popularity of docker (seemed like a convoluted/glorified chroot), and am going to watch your talk on it and see if anything resonates.
edit clarification: popularity of docker vs something “better” like jails/zones, or even using lxc directly
Out of curiosity, since it doesn’t seem super likely that io.js and node are getting together anytime soon (and that’s beside the point anyways)…would Joyent be interested in moving off of v8 as a JS runtime if a better platform, like an open-sourced engine from Microsoft or duktape or something, became available?
One vision for future development of Node that we’ve spitballed around the office is definitely to refactor the platform so that it does not depend on a particular Javascript runtime. I’ve personally played around with Duktape and I think it’s eminently embeddable and quite neat. Obviously the performance will not be anywhere near that of a JIT-capable VM like V8 or Spidermonkey, but it’s interesting nonetheless.
Though there are chunks of the codebase that are written in C++ today, and clearly the entire platform is relatively wedded to running on top of V8 and libuv, at its heart Node is really a sort of “Javascript standard library”. There’s no reason not to write (or rewrite) most of that in Javascript, with a small, well-defined C (not C++) layer underneath that exposes the parts of a Javascript VM and the underlying OS that are commonly available.
It would then be reasonable to use Node as a Javascript library/platform on top of basically any VM – whether on top of V8 (with a stable C API/ABI on top), or Duktape, or JSC, etc. The node binary could even load different interpreters at runtime from shared libraries.
Though, obviously this is not just a decision for Joyent. As we move toward a Node Foundation it is even more a community-centric decision that ever before, both in terms of setting the direction and doing the work required.
I think that Node-The-Standard-Library and Node-The-Runtime-Implementation is an important distinction to make.
I think that building a minimal C scaffolding (libuv + a JS runtime) would a great place to go. Is there anyone currently working on that? Is there a test suite for Node or IO.js that verifies the Node-The-Standard-Library behavior (as seen from JS userland), and which we could use to test conformance of an alternative Node-The-Runtime-Implementation?
There are many challenges with providing an API compatibility test (or certification) suite. Some recent related work is StrictEE, a modified EventEmitter that can make assertions about event firing cardinality, ordering, etc. This is a critical step towards being able to prove what the API of Node even is, given how much of it is exposed through the EventEmitter pattern.
If you’re keen to work on things like that, there is definitely interest!
Thanks for doing that @bcantrill! I know there are all sorts of interesting folks doing open source stuff, but I found it refreshing to see your name out there. Cheers!
Does anyone know if this 64-bit work is useful at all for *BSD getting 64-bit support of Linux apps?
Also, I’m very ignorant of Docker and any relationship to *BSD and jails. Is there anyone trying to bring these things together? Is it worth it? Does a Dockerfile express enough to just make a jail from?
While it would be great if our work could help the BSDs out, my assumption would be that it basically won’t: most of the work that we did was specific to SmartOS (and much of it was kernel work). It’s all open source; check out the SmartOS source, in particular looking at usr/src/lib/brand/lx and usr/src/uts/common/brand/lx.
Don’t forget usr/src/common/brand/lx!
Afaik 64 bit Linux compat is available in FreeBSD and netbsd. And it’s just “work” away from working in OpenBSD.
FreeBSD has 32-bit support but not 64-bit support. Last I have heard is it’s a work in progress but I’m not aware of anything, even in 11-CURRENT, that supports 64-bit binaries.
The title definitely reflects some emotion, but it should be understood that this came from the author’s intense frustrations porting cgo to illumos. (As a disclaimer: I’m the one who put the author up to this – though it would be incorrect to blame his retirement[1] on this work.) Having dealt with the underbelly of Go in the past (on the initial Go port to illumos), I can understand the wellspring of this frustration: there is much in the Go implementation that leaves one asking “but… why?!” I also do think that there’s a bit of an attitude of smugness from the Go implementers that tends to exacerbate these frustrations – and I think that’s part of what Keith’s reacting to here. Which is not an attempt to apologize for the more vitriolic bits, but merely to explain where they’re coming from.
One thing that is clear to me: the author hasn’t actually written much (or perhaps any) Rust. This is clear to me because I think one of the traps that the merely Rust-curious fall into is a disproportional fear and loathing of the borrow checker. This is disproportional because it ignores many of the delightful aspects of Rust – for example, that algebraic types in a non-GC’d language represent a revolution in error handling. (I also happen to love the macro system, Cargo, the built-in testing framework, and a bunch of other smaller things.) Yes, the lack of things like non-lexical lifetimes can make for some wrestling with the borrow checker, but once one is far enough into Rust to encounter these things, they are also far enough in to appreciate the value it brings to systems programming.
To sum, the author shouldn’t weigh in on Rust (or any language, really) so definitively without having written any – or at least make clear that his perspective is informed by reading blog entries, not actual experience…
I believe it wasn’t your intent, but your commentary reads a bit like “Only true Rustaceans should be allowed to talk about Rust”.
Everyone should be allowed to talk about Rust. There is no authority that deserves to have the power to decide which people can or cannot talk about Rust.
That said, it’s also fine to say that the author’s opinion about Rust is untrustworthy because it bears the hallmarks of someone who has read about Rust but not actually used it themselves in any meaningful way. I myself agree that it’s possible to write lots of useful rust code without running into situations where the borrow checker trips you up, and that some of Rust’s best innovations are the “small” things like the algebraic types, macros, Cargo, etc. that are now available in a non-GC systems language.
I still use rustlang but share the same opinion as the author. Did I write enough of it to be trustworthy? :)
Nothing on that list was rustlang’s innovation.