Threads for dw
-
The article fails to account for reserved instance pricing, the sustained use discount, the free tier, and spot or pre-emptable instances.
Pricing on AWS/GCP is complex, but you can save a lot of money if you’re careful.
Though to be fair that complexity is one way they make money. You could save a lot of money, but it’s all too easy to overlook something.
-
Hi, OP here.
- I believe I am taking Google’s sustained use discount into account
- I haven’t included the free tier because it is marginal and I think most organisations will exhaust it fairly quickly
- I think spot and preemptable instances are not a general product but a specialist one: only some applications of virtual machines can tolerate getting evicted
I do discuss the issue of complexity later on. I don’t think it normally works to the advantage of the customer.
My intuition (and experience!) is that most real world AWS customers get bamboozled by the incredible complexity of pricing (especially when it’s presented in non-human readable units like “0.034773 per vCPU hour”) and wind up paying far, far over what the going rate of renting a computer should be.
-
-
There’s also a built-in Terraform provider https://www.terraform.io/docs/providers/hcloud/index.html
-
Agree, it seems to be 10 Euro/month for 8GB in the cloud plan.
I’m running a root server with 32GB of memory and 2TB hard disk at Hetzner for ~30 Euro / month (from the Serverbörse). I do not know about their support at all, but I am quite sure that from a US IT company I could only expect automated mails, anyway. So Hetzner cannot be any worse there.
Of course, root server and cloud hosting are two totally different beasts, but in my humble opinion it’s a choice the US-centric tech community too often does not even consider. The mantra is always the application has to be horizontally scalable.
-
It should just be noted that Serverbörse is usually based on dekstop machines and the like, often older CPUs and servers, so you might not want to rely on that if your application stack is considered mission-critical.
As for the cloud vs classic servers, it’s a different beast completely, yes. A lot of internet wouldn’t be alive if you had to pay a linux admin to configure your servers, deploy your apps and pay attention to traffic, script kiddies etc. But not having a lot of internet online could perhaps be considered a good thing, eh?
-
-
-
On preemptible/spot they both provide /liberal/ shutdown warnings, it is possible to run almost anything aside from long life connection hosts (e.g. websockets) or extremely stateful applications like databases. Use cases that don’t fit spot are approaching minority in 2020 with current infrastructure trends.
Re: DigitalOcean, I did a migration a few years back where AWS came out vastly cheaper than the equivalent configuration on DO mostly due to AWS instance reservations, which are a trivial factor to plan for when you’re already planning a large migration.
The one area I couldn’t possibly defend the cloud providers is bandwidth pricing. All other costs are a footnote compared to it for any account doing high traffic serving
-
Not an expert on this, but while it seems it is possible to run lots of things on hosts that may shut themselves down automatically, actually doing so will cost you more developer and devops time instead of just paying a little more for hosting. It seems likely that this is time you want to spend anyway, as part of making an application more resilient against failure, but it still makes the situation yet more complicated, and complexity usually serves Amazon more than the customer. (And I have a hard time believing that databases are approaching a minority use case with current infrastructure trends. ;-)
-
-
-
Personal example: I started to develop an application with AWS services (Lambda, SQS, EC2, S3). Later I changed it to an application for a “normal” server. I still wanted to store data to S3, but the cost to download it from there for analysis is just ridiculous. So the choice was to store to S3 and run on EC2 or not to store to S3. (I decided against S3).
-
What I mean is that data transfers between services in the same cloud x region are much cheaper than data transfers between clouds. So it’s more expensive to store logs in AWS and analyze them with GCP, compared to just analyzing them in AWS. You can’t take advantage of the best tools in each cloud, but are forced to live in one cloud, creating a lock-in effect.
If there was a rule that bandwidth prices must be based on distance, not whether the endpoints are within the same cloud, we’d see more competition in cloud tools. Any startup could create a really good logs-analysis tool and be viable, for example. This rule runs into some legitimate issues though. For example, if a cloud provider has custom network hardware and fiber between their own data centers, the cost of moving data between their zones might be much cheaper than sending it over the public internet to another cloud provider. Moreover, many different cloud services are co-located in the same data center. So it’s much cheaper to analyze logs using a service that already exists where the data is than to ship it off to another cloud.
The problem is big cloud vendors have little incentive to let users take their data out to another cloud. It’s going to be a market where only a few big players have significant market share, at this rate.
-
-
I agree completely on the bandwidth pricing. At this point, I think this should be considered a common public infrastructure, like roads etc. Yes, I understand that there are costs to providing it all, that some companies have invested in infrastructure privately, all I’m saying is that the traffic should be “free” for the consumers (and even for the business sector that the article OP is mentioning, companies hosting wordpress or timesheet or some similar small apps like that without major engineering teams).
-
-
-
-
Yep, it’s definitely true for existing apps. Converting a large stateful app to new world is a nightmare, but you get so many benefits, not least the problem of preemptibility and autoscaling are basically identical. The big Django app used to require 16 vCPUs to handle peak, so that’s how it was always deployed. Now it spends evenings and non-business days idling on a single t2.micro
In the case of a typical Django app though, if you’re already using RDS then the single most typical change is moving its media uploads to S3. It’s a 15 minute task to configure the plugin once you’ve got the hang of it, but yep, for a single dev making the transition for a single app, that probably just cost you a day
-
-
The one area I couldn’t possibly defend the cloud providers is bandwidth pricing. All other costs are a footnote compared to it for any account doing high traffic serving
Thanks, I came here to say that. The article didn’t even factor in bandwidth/network costs, which matter for both EC2 and S3 (not as familiar with the other cloud providers).
-
Anecdotally, from friends who work in the AWS machine: once you get to real (financial) scale with AWS - think “7 digits a month” or so - you’ll find Amazon is extremely happy to negotiate those costs.
Fiber ain’t free, but I wager that the profit margin is probably highest there.
-
-
-
-
-
I was going to ask how this relates to Mitogen, but then I noted you are Mitogen author. So this is the ‘premium’ version of Mitogen?
-
It integrates it, but other than that the efforts are distinct. I considered merging the library into Operon and giving it away from the outset, but this issue of the cost of support does not go away, and the existing project is much easier/cheaper to support. Simultaneously Mitogen has a lot of users, and basically breaking those users install scripts just to promote a new thing would be a dick move.
-
-
Not a single word on how it works
-
-
Can you describe it (vaguely, if need be)? Something about how RedHat would react if you reveal too much?
-
-
-
Interesting. GPL-licensed, but no way to download sources. Presumably if you buy a subscription, you can request sources under the GPL, and would then be free to distribute them.
This might be the first time I’ve actually seen a project use the GPL this way, which is surprising since AIUI this was the intent of the license in the first place. (I’m sure someone else has done this too. I would just expect it to be more common.)
-
Red Hat have been using the GPL this way for more than a decade, it’s their primary business model.
As for why it’s like this – primarily because a bug report can cost me $80+/hr in compute time alone to reproduce. I can’t accept users that I can’t support, so the only thing to gain from releasing the code for the time being is a lot of unhappy users.
-
-
Wouldn’t say it’s “common” but it’s not unheard of in fields that aren’t so closely related to computery things. I think most people where the engineering team has enough influence would just say “please make our lives easier and just let us publish the repo on github”.
Also I read the article that it will(might?) be provided in the future, just not now.
-
-
I (along with probably every other technical e-mail user) have had thoughts about a replacement mail protocol layered on modern standards, but having flicked through JMAP, the level of abstraction was.. not encouraging. It reminded me of SOAP.
Does anyone have experience with implementing it? Be-all protocols have a horrible habit of turning out to be garbage, and JMAP kinda reeks of it. It’s main saving grace is that it comes from a well-regarded industry participant, but that’s no guarantee either. Maybe it’s just an allergy to overly generic specifications
-
I can only tell you from user experience that it’s life changing and makes me not hate email. It makes GMail seem incredibly slow
-
How are you experiencing this? Through the Fastmail web app?
-
Yes open the web app on multiple devices and do things: move lots of messages, tag, send, receive, archive, etc. It’s instant across all web apps and very fast over IMAP.
Now do the same with GMail and watch how it lags before your changes synchronize stage across the Gmail web apps and even IMAP: it will take anywhere from 15 seconds to 5 minutes. I’ve made changes in GMail that took 5 minutes for it to show up in IMAP and vice versa. It’s crazy how bad GMail really is.
-
-
-
-
Have the OpenBSD project given much consideration into adopting a safer language? A 2019 rewrite in C of what is historically one of the buggiest pieces of C.. is eyebrow raising to put it mildly
-
Here’s an email from Theo de Raadt 1.5 years ago that answers that question
-
-
I don’t see the value of complaints against SPAs anymore. That ship has sailed about a decade ago! The reasons are about as complex as the reasons for the popularity of the web platform itself. It’s not the best solution, but nothing on the web ever is. Instead, it’s providing a particular set of tradeoffs such that many developers prefer it – even while taking user experience into consideration. For example, can anyone suggest how to make an old school web application that works offline?..
(Although I’ll admit, I’m prone to criticising the web platform myself on occasion. The whole thing, including SPAs, is really a terrible kludge. Nonetheless, I only make SPAs and not old school web applications.)
(Oh, and it’s also very useful to distinguish between web sites and web applications, but for some reasons these complaints rarely do.)
-
I don’t see the value of complaints against SPAs anymore.
I don’t see the value of this statement specifically. Even though SPAs are a step backward in a multitude of ways, it’s just how the world works now and we should all just accept that the ship has sailed?
can anyone suggest how to make an old school web application that works offline?
Look, I’m just trying to be able to still use the web on mobile when my connection is spotty. I think it’s a cool party trick that webapps can be cached entirely in service workers (or whatever) in order to make them work offline, but I’m just trying to read one article and then go about my day. But now, I’m sitting here on the metro waiting 20 seconds for a newspaper site to load.
Does a content site (like a newspaper or blog) need to work offline? If not, why do people build them with stacks that cause drawbacks in other areas (such as, from TFA, being slower overall).
(Oh, and it’s also very useful to distinguish between web sites and web applications, but for some reasons these complaints rarely do.)
It’s because everything is an SPA now. Newspapers, shopping sites, blogs. I think it’s great that rich applications can be built now – I use a fair share of web applications day to day! Mail client, Trello, couple of others. I’ve been in the “SPA haters’ club” for a few years now, consuming plenty of content to confirm my bias, and I’ve never heard anybody say “I sure wish Trello worked with JS turned off.” I’ve experienced a lot of “why do I have a running service worker for this shopping site?” or “why does this page show up blank with JS disabled and then it turns out to just be a blog post with no images once I turn JS on and reload?”
-
Well, so you see the value of SPAs, right? Your issue is that content sites use this architecture when they don’t need to, and do it at the expense of UX in some cases. OK, fine, but that’s not the same as saying “SPAs are useless, and we could do anything and everything by loading individual pages from the server”. Well, no, we can’t.
So my problem is that the complaints like the OP are usually a vague handwavy appeal to how wonderfully simple and fast it is to load pages from the server, usually combined with pointing fingers at a content site like MDN, and without taking the trouble to address any of the use cases for which that approach doesn’t work.
We shouldn’t just accept things as they are, but I think the specific ship that sailed is for complaints about SPAs vs plain old web pages in the context of web applications. There was a point when things could have gone a different way, but that was in 2004-2008, if memory serves. Now it would be far more productive to frame complaints in the modern context, where we have complex applications all over the web, which can’t be turned into simple web pages.
I hope this clarifies things.
-
Your comment appears to be replying to a point I didn’t make, and I’m frustrated by it, so I will reiterate.
We’re not talking about Trello and Google Docs. We’re talking about Medium, or any Squarespace or Wix site (which don’t work without JS).
There’s somebody like you in the comments of every article like this. “Don’t forget about web applications! You know, the ones that absolutely require client-side dynamism!”
Nobody’s forgotten about them. It’s impossible to. They’re everywhere. Your use-case is handled already. But now, everybody wants to build SPAs by default, even when doing so doesn’t provide any benefit. That’s the problem.
-
You’ve conveyed your frustration perfectly. I understand your point, I agree to a degree, and I think it’s fine to criticise this application of SPAs.
I could still suggest some benefits of SPAs even for content sites on the development side, but I don’t want to exasperate you any further. Thanks for the discussion.
-
For many many years I’ve blocked cookies by default, and only opened exceptions for sites I wanted to use in ways that required a cookie.
In more recent years I’ve also been blocking local storage by default and opening the occasional exception for a site I want to use that has a justifiable reason for needing it. But a staggering percentage of content-oriented things just flat-out break if you deny local storage, because A) they’re actually massive JS applications and B) nobody involved in their development ever considered this case (while people do seem to at least recognize when you disallow cookies).
For example, the browser I’m in right now does not have an exception for GitLab, so even trying to view a README of a repository in GitLab’s UI will fail with an eternally-spinning “loading” indicator and shows
TypeError: localStorage is nullin the browser console.-
I guess you’ve considered it, but for this reason self-destructing cookies (aka Cookie AutoDelete for Firefox) is much better at preventing breakage with most of the benefits of compete blocking
Rather than broken sites, as a European I only have to contend with endlessly confirming GDPR notices for sites I’ve already visited (although thankfully there is also a plugin for that!)
-
-
-
-
-
Does a content site (like a newspaper or blog) need to work offline?
Would e-readers be better if they required an always-on internet connection? I think there’s a lot of value in not needing an internet connection just to read… (Although many offline SPAs are poorly-written, or loaded with slow trackers, ads, fullscreen modals asking for your email address, etc.)
-
It’d be nice if I got to make the decision myself as to whether I want to read an article from the same source again before they cache their entire backlog for me. (slight hyperbole)
Personally I believe that ATOM is still the best designed way of accessing articles offline, and advancements in that system would be much more beneficial than pushing SPAs. Things like encouraging sites to actually put the full article in, rather than just a link to the website I’m trying to avoid.
-
-
-
Every old-school website I’ve ever used works just fine offline. “File” -> “Save As”.
-
Web application, not site.
-
See my sentence above about the distinction between websites and web applications.
-
For example, can anyone suggest how to make an old school web application that works offline
Maybe I’m showing my age here, but an ‘old school web application’ means flash, to me - and those overwhelmingly worked perfectly when you saved the page.
-
So do you think we should go back to the good old days of Flash and Java applets? You’ll probably recall that Flash had a bad track record for security and energy efficiency. This critique by Steve Jobs is still pretty good (even if hypocritical).
I don’t recall that saved .swf files were able to maintain any state either. Were they?
-
So do you think we should go back to the good old days of Flash and Java applets?
I know as well as you do how much of a dumpster fire flash security was. Java applets were… less disastrous, but I never saw one that wasn’t deeply unpleasant to use.
I don’t recall that saved .swf files were able to maintain any state either. Were they?
Very few applications maintain state after you close & reopen them. You could, though - flash applets could work with files on your computer (they had to use the flash builtin filepicker and only got access to files chosen through it).
-
So in comparison to how things were with Flash and Java applets back then, don’t you think SPAs are an improvement? Sure, they might be overused from the user’s point of view, but that’s not the same as saying they can easily be dispensed with by reimplementing with server-side page rendering.
Re state: that also doesn’t seem like a great user experience in comparison to SPAs.
-
You’ve put words in my mouth two comments in a row.
The way you have chosen to communicate does not reflect well upon you, and I’m not interested in engaging further.
-
I’m sorry if it came across that way, that’s not how I meant it. I was just asking questions to understand what you think. The comment re “they can easily be dispensed with” was more an interpretation of the OP, which is what I thought we were talking about.
-
-
-
-
-
-
Did… did you read the thread, or just reply to one comment in isolation?
-
-
No worries than :)
-
-
-
-
-
-
-
-
For small diagram-like SVGs there is also the final trick of inlining the resulting SVG directly into the page, removing the opportunity for any network delay to creep in at all, and allowing all those small files to share a compression dictionary.
I found the most effective approach was in the form of a url-encoded (not base64) data: URL, as it meant toggling inlining only involved changing
<img src=>attributes, which could be done centrally from a Django template tag, with no other changes in page structure. By using the less efficient URL encoding rather than base64, redundant substrings within each SVG are byte aligned so gzip over the whole page is much more effective, even though the decompressed page is usually much larger-
-
HTTP/2 push is great, but it requires the alignment of many things stretching across many layers, something like:
- a separate file or webapp headers to emit hints to the web server
- a web server to understand the hints
- all proxies and load balancers to accept and forward them
- a proxy to own the client and also persist a list of files that have already been pushed to it
- clients to receive and act on them correctly
for example with Firefox talking to Apache2, the browser acts completely differently on page refresh than it does for a new cold connection. No reason to avoid it, but some inline strings on a page are vastly easier to avoid messing up
-
-
-
Milliseconds on a frictionless sphere perhaps, but on typical 3G or transcontinental connections the removal of every DNS lookup, SSL connection setup or even a single roundtrip before the browser knows to fetch a file quickly adds up to multiple seconds. Offering a snappy experience isn’t a nice-to-have
-
-
-
A big downside of using SVG is this bug in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=935056. The minimum font size is also applied to text inside SVG (even if it is finally scaled up by a transformation).
I have found the
mergePathsfeature of svgo somewhat unreliable: it may chop off arrow heads. -
-
I switched off of Google products about 6 months ago.
What I did was I bought a Fastmail subscription, went through all my online accounts (I use a password manager so this was relatively easy) and either deleted the ones I didn’t need or switched them to the new e-mail address. Next, I made the @gmail address forward and then delete all mail to my new address. Finally, I deleted all my mail using a filter. I had been using mbsync for a while prior to this so all of my historical e-mail was already synced to my machine (and backed up).
Re. GitHub, for the same reasons you mentioned, I turned my @gmail address into a secondary e-mail address so that my commit history would be preserved.
I still get the occasional newsletter on the old address, but that’s about it. Other than having had to take a few hours to update all my online accounts back when I decided to make the switch, I haven’t been inconvenienced by the switch at all.
-
It’s really exciting to see people migrating away from Gmail, but the frequency with which these posts seem to co-ocur with Fastmail is somehow disappointing. Before Gmail we had Hotmail and Yahoo Mail, and after Gmail, perhaps it would be nicer to avoid more centralization.
One of the many problems with Gmail is their position of privilege with respect to everyone’s communication. There is a high chance that if you send anyone e-mail, Google will know about it. Swapping Google out for Fastmail doesn’t solve that.
Not offering any solution, just a comment :) It’s damned hard to self-host a reputable mail server in recent times, and although I host one myself, it’s not really a general solution
-
Swapping Google out for Fastmail solves having Google know everything about my email. I’m not nearly as concerned about Fastmail abusing their access to my email, because I’m their customer rather than their product. And with my own domain, I can move to one of their competitors seamlessly if ever that were to change. I have no interest in running my own email server; there are far more interesting frustrations for my spare time.
-
I can agree that a feasible way to avoid centralization would be nicer. However, when people talk about FastMail / ProtonMail, they still mean using their own domain name but paying a monthly fee (to a company supposedly more aligned with the customer’s interests) for being spared from having to set up their own infrastructure that: (A) keeps spam away and (B) makes sure your own communication doesn’t end up in people’s Junk folder.
To this end, I think it’s a big leap forward towards future-proofing your online presence, and not necessarily something comparable to moving from Yahoo! to Google.
-
for being spared from having to set up their own infrastructure that: (A) keeps spam away and (B) makes sure your own communication doesn’t end up in people’s Junk folder.
I’m by no means against Fastmail or Proton, and I don’t think everyone should setup their own server if they don’t want to, but it’s a bit more nuanced.
Spamassassin with defaults settings is very effective at detecting obvious spam. Beyond obvious spam it gets more interesting. Basically, if you never see any spam, it means that either you haven’t told anyone your address, or the filter has false positives.
This is where the “makes sure your own communication doesn’t end up in people’s Junk folder” part comes into play. Sure, you will run into issues if you setup your server incorrectly (e.g. open relay) or aren’t using best current practices that are meant to help other servers see if email that uses your domain for From: is legitimate and report suspicious activity to the domain owner (SPF, DKIM, DMARC). A correctly configured server SHOULD reject messages that are not legitimate according to the sender’s domain stated policy.
Otherwise, a correctly configured server SHOULD accept messages that a human would never consider spam. The problem is that certain servers are doing it all the time, and are not always sending DMARC reports back.
And GMail is the single biggest offender there. If I have a false positive problem with someone, it’s almost invariably GMail, with few if any exceptions.
Whether it’s a cockup or a conspiracy is debatable, but the point remains.
-
-
We’re not going to kill GMail. Let’s be realistic, here. Hotmail is still alive and healthy, after all.
Anyone who switches to Fastmail or ProtonMail helps establish one more player in addition to GMail, not instead of it. That, of course, can only be a good thing.
-
Just to bring in one alternative service (since you are right, most people here seem to advice Fastmail, Protonmail): I found mailbox.org one day. No experience with them though.
-
-
I still get the occasional newsletter on the old address, but that’s about it.
Once you moved most things over, consider adding a filter on your new account to move the forwarded mails to a separate folder. that way it becomes immediately clear what fell through the cracks.
-
Sorry, I wasn’t clear. E-mails sent to the old address are forwarded to the new one and then deleted from the GMail account. When that happens I just unsubscribe, delete the e-mail and move on. It really does tend to only be newsletters.
I suppose one caveat to my approach and the reason this worked so well for me is that I had already been using my non-gmail address for a few years prior to making the change so everyone I need to interact with already knows to contact me using the right address.
-
-
-
Trying to juggle multiple projects while squeezing out a minimum viable Mitogen release supporting Ansible 2.8. Azure Pipelines is being an asshole, so I’ve downed tools for the evening
-
-
I’m glad they mentioned that the filesystem operations will be faster. I wish they would explain further as to how they achieved that. This was my biggest concern using WSL last year while I was working on a Magento 2 site. Serving pages was slow, DI compilation was slow, JS build scripts were slow, everything was slow.
-
I’m guessing it’s probably the obvious route given the description they use – Linux gets its own block backing store like a regular VM and manages its own cache just like a regular disk, and all the ugly coherence issues are completely dodged. Nobody ever needed a 1:1 filesystem translation in the first place, and they’re too hard and/or impossible to build without suffering the woeful perf issues WSL1 had
Really sad they’re giving up on the WSL1 approach – an independent reimplementation of Linux had immense value for all kinds of reasons. Must be heartbreaking for a lot of people who worked so hard to get it to where it was
Anyone care to place wagers on how long it’ll take after release before it’s hacked to run FreeBSD? :)
-
Must be heartbreaking for a lot of people who worked so hard to get it to where it was
I thought the same thing! I’m also a bit bummed because the current implementation makes use of the “personalities” feature of the NT kernel, whereas the new one is a VM.
-
It is a bit of a shame indeed. Also a good example of Agile development. I suppose the initial developers thought it would be a nice way to go, so they tried it out and shipped it. I suppose they couldn’t be sure in advance that the file system would be that much slower, or that people would be that concerned about it. Now that they know that, they had to switch over to a VM to get the performance level that people demanded. Too bad, but hard to predict ahead of time.
-
-
-
-
Yeah this was the logical conclusion of the WSL team initially targeting ABI compatibility with Linux. Diversity of implementation would probably have had a better chance of surviving if the WSL team targeted POSIX source compatibility, like macOS. That would have given them more wiggle-room.
That’s not to say the WSL team made the wrong decision to target ABI compatibility, they likely have different goals for their corporate customers.
-
-
If they just wanted POSIX source compatibility, they could have continued development of the POSIX subsystem / SFU. Apparently there’s something to be gained from being able to run unmodified Linux binaries (otherwise we’d just have Windows Ports). My guess is that the goal is to grow Azure – it’s likely to be a bigger revenue source than Windows in the future.
-
-
Really sad they’re giving up on the WSL1 approach – an independent reimplementation of Linux had immense value for all kinds of reasons. Must be heartbreaking for a lot of people who worked so hard to get it to where it was
It seemed like it involved a lot of kludges onto NT though - to the point it seemed the easier approach to me was refurbishing the old POSIX subsystem and making it feel modern. There was a lot of (wasted) potential there.
-
-
-
-
I spent close to 8 weeks over winter researching an article exactly along this line, and so it was with some panic that I read this attempt at characterizing the issue. Gladly for that effort, this paper omits the majority of inseparably related considerations and relies too heavily on narrative for its conclusion.
Fork is a nuanced beast – it most certainly is a filthy hack and I set to write (as they have) how much I hated it, but the truth isn’t quite as absolute as presented here. Fork is deeply baked into UNIX, much in the same way processed beach dirt is baked into all our CPUs. Straight removal from any particular application isn’t possible for many reasons, not least since it regularly plays a dual role as a communication mechanism (including in brand new designs like systemd) that require deep rearchitecting to remove.
It has also deeply influenced surrounding UNIX APIs. For example, it is not possible to fully transition into a Linux container namespace without a double fork, or fully detach from a parent process on any UNIX (that I know of) without the same. Consider just that alone – how the classic daemonize() would be implemented without fork(), and what new and exotic interfaces / bending of process management rules would be needed to provide an equivalent interface. No suggestion of ‘removing’ fork is worth taking seriously without addressing problems like these.
The UNIX incantation of fork was a hack that borrowed its name from an only slightly more formalized concept (through a series of Chinese whispers), all of which predated the actualization of, and were hopelessly inadequate at addressing the problem they set out to solve – Conway’s fork predates the arrival of contemporary SMP architecture by more than 20 years.
-
-
That doesn’t really address many of the listed complaints. The problem, as stated, is that fork requires duplicating state which makes microkernels impossible, etc. How is posix spawn defined to work?
All process attributes, other than those influenced by the attributes set in the object referenced by attrp as specified above or by the file descriptor manipulations specified in file_actions, shall appear in the new process image as though fork() had been called to create a child process and then a member of the exec family of functions had been called by the child process to execute the new process image.
-
-
-
-
A lot of the criticism of “fork” is justified, but a lot also reminds me of what I saw in Linux kernel code years ago - many efforts to “clean up” code by people who did not understand the tradeoffs and had nothing valuable to contribute.
-
-
glibc should just add
__attribute__((deprecated))to the declaration offork()and we can revisit in 10 years. -
-
-
-
-
Back then, there was fork on UNIX vs spawn on VMS vs language-based or thread-like that Brinch Hansen was doing. My research showed spawn did things like set privileges, allow CPU/RAM metering, etc. Basically, the kind of stuff clouds added to Linux decades later. So, at least they’d argue spawn was better. Language-based methods are coming back with less overhead and more flexibility than UNIX or VMS process model. Worse isolation but hardware attacks are making me wonder how much it matters. fork is objectively worse than many ancient and new designs. It’s legacy cruft now.
Compared to systems then and shortly after, it does look like fork was a hacked together design driven primarily by hardware considerations of their underpowered machine. Maybe combined with their preference for minimalism. We know that quite a few things are in there solely due to them being on a PDP-11, though. So, that should be a strong consideration for any other design element that seems unnecessarily limited or prone to problems. My favorites being lack of prefix strings and incoming data flowing toward stack pointer rather than away from it (eg MULTICS).
-
were hopelessly inadequate at addressing the problem they set out to solve
And yet, 40 years later, it is the basis for 90% of the infrastructure software in the world. I’d like to create something that “hopelessly inadequate”.
-
It’s fair to say most infrastructure uses fork, if you mean to start up, but the majority of modern client/server infrastructure software thankfully does not rely on it for data plane processing – any software due to be exposed to Internet-sized traffic by necessity must use event-driven IO or some form of threading (or a mixture of both), all of which are competing multiprocessing paradigms, and required to achieve anything close to sensible performance on modern hardware.
The UNIX model was conceived at a time when the intended notion of a ‘client’ was literally an electronic typewriter attached to a fixed number of serial lines. I hope you can imagine the horror on the face of its designers were you to explain how this model was expected to cope with any subset of humans anywhere on the planet at any random moment demand one process to be dedicated to each of them.
-
I just wanted to point out that the “hopelessly inadequate” UNIX fork/exec design was a hugely successful design. Not only is it still the basis for a wide range of widely used software, but it was designed in a way that evolution to things like rfork and clone was practical.
BTW: “event driven” software seems to me to have been motivated by the shortage of programmers able to understand multi-process/multi-thread design more than anything else.
-
That’s what I call the COBOL and VB6 argument. A lot of people used it. It’s still around. So, it was a great design. Alternatively, they spread for business, social, and ecosystem reasons as much as for design. Then, design can’t take credit for popularity. Popularity can’t excuse the bad or outdated elements of design.
Same with fork. In this case, UNIX got wildly popular with a huge ecosystem. It trains people to use available features like fork. It doesn’t tell them about lots of old and new ways to do the job better. So, they use the thing that they were taught to which is there and works well enough. When running into its problems, they use some other non-UNIX-like approach such as monolithic, multi-threaded apps or event-driven design.
-
But those were very succesful designs. If they have become outdated, it’s good to try to understand what made them work so well for so long. Programmers have a really bad attitude about this. You are not going to find civil engineers who sneer at Roman Aquaducts or aviation engineers who think the DC3 was the product of stupid people who just didn’t get it. This is why programmers keep making the same errors.
-
Civil engineers would behave the same way if the difference between Roman Aquaducts and modern aquaducts was as invisible as the difference between pieces of software is. It’s not that programmers are a different kind of people: it’s that noticing the differences, and hence needing explanations for the differences, requires much more effort.
Seeing the ancient aquaduct immediately reminds you of the materials they had to work with, minor general historical knowledge tells you of the inferior technology/tools they had to work with and minor specific historical subject knowledge (of science, physics, civil engineering) tells you of the lack of knowledge they had to work with. That makes the primitive nature of the Roman Aquaduct comprehensible and even impressive.
Seeing a COBOL program doesn’t demonstrate any reason why it should seem so primitive. Knowledge of general history tells you the world was pretty much the same 40 years ago, which doesn’t explain anything. Even with knowledge of the history of computing it remains easy to underappreciate the vast lack of knowledge, experiences and experiments (such as COBOL itself) that they had to work with and that we’ve been able to learn from.
So I don’t think blaming ‘programmers’ as lacking historical sensitivity is helpful or fair. Which doesn’t mean that we shouldn’t try to find ways of increasing the historical sensitivity of our craftsfolk, because it would indeed be really helpful if more of them learned from mistakes of the past or perhaps even restored something accidentally lost.
-
-
-
Tell that to nginx, which does both.
-
Well, event programming is embedded in web programming just as fork/excec is embedded in unixes. If you want a widely used web server, you don’t have much choice. But I take back what I wrote which was unintentionally snotty about event driven programming. lo siento.
-
-
-
-
-
Are the problems with fork() not mostly implementation issues? I get what you’re saying, but the existence of e.g. spawn() suggests that the concept is sound.
-
-
inactive-user
edited
4 years ago
|
link
| on:
Ansible's creator on the state of contemporary devops and open source
The irony is that he’s now trying to build better tools that use embedded DSLs instead of YAML files but the market is so saturated with YAML that I don’t think the new tools he’s working on have a chance of gaining traction and that seems to be the major part of angst in that thread.
One of the analogies I like about the software ecosystem is yeast drowning in the byproducts of their own metabolic processes after converting sugar into alcohol. Computation is a magical substrate but we keep squandering the magic. The irony is that Michael initially sqauandered the magic and in the new and less magical regime his new tools don’t have a home. He contributed to the code-less mess he’s decrying because Ansible is one of the buggiest and slowest pieces of infrastructure management tools I’ve ever used.
I suspect like all hype cycles people will figure it out eventually because ant used to be a thing and now it is mostly accepted that XML for a build system is a bad idea. Maybe eventually people will figure out that infrastructure as YAML is not sustainable.
-
-
There are plenty of alternatives. Pulumi is my current favorite.
-
Thanks for bringing Pulumi to my radar, I hadn’t heard of it earlier. It seems quite close to what I’m currently trying to master, Terraform. So I ended up here: https://pulumi.io/reference/vs/terraform.html – where they say
Terraform, by default, requires that you manage concurrency and state manually, by way of its “state files.” Pulumi, in contrast, uses the free app.pulumi.com service to eliminate these concerns. This makes getting started with Pulumi, and operationalizing it in a team setting, much easier.
Which to me seemed rather dishonest. Terraform’s way seems much more flexible and doesn’t tie me to Hashicorp if I don’t want that. Pulumi seems like a modern SAAS money vacuum: https://www.pulumi.com/pricing/
The positive side, of course, is that doing many programmatic-like things in Terraform HCL is quite painful, like all non-turing programming tends to be when you stray from the path the language designers built for you … Pulumi handles that obviously much better.
-
-
You’re welcome to use a local state file the same way as in Terraform.
-
-
+100000
-
-
-
-
kristof
4 years ago
|
link
| on:
Ansible's creator on the state of contemporary devops and open source
Can someone elaborate on what he means by agile burning people out?
-
I interpreted it as in the context of free software contribution. It simply isn’t sensible on a modern software project to suggest spending considerable time working on random tooling, because project management style and language has evolved to the point where the value of doing so can’t even be captured. “I’d like to add [X] to software [Y]” .. “Great, but what user story does this relate to? We’re focusing on the billing feature this week - there are 17 tickets still open for that due end of sprint Friday”, etc.
-
-
I like the diagrams on these pages. Does anyone recognize the tool? I guess it’s possible they’re simply hand-drawn with great care.
-
It looks like they are “hand drawn” using Inkscape or something. I guess you could use Dia to achieve something similar.
-
The author used MS Powerpoint. The author is asking the community to suggest alternatives.
-
-
-
For reference, existing tools presently require full debug info installed, which on my Ubuntu VM is 583MB on disk for the main kernel image alone.
-
szbalint
4 years ago
|
link
| on:
Linux on Microsoft Azure? Microsoft has root-access by default (wa-linux-agent)
This is what running a VM in Microsoft Azure means. Microsoft controls the underlying datacenter hardware, host OS, hypervisors etc.
There is no defense against sustained physical access.
-
I think the difference raymii is making is between online and offline access - yes, they can unrack the servers and look at your disks with a magnifying glass, but online access where they can log in live to your running instance is a different threat model. If you rack hardware somewhere, sure, they have your hardware, but they most likely don’t have (an equivalent of) the root password. This story surprised me.
-
But we’re talking about virtual machines here, right? So you don’t need to unrack anything; your magnifying glass is just
/proc/$(pgrep qemu)/mem(or whatever the hyper-v equivalent is), to peruse at your leisure, online, from the host.(And even in the case of rented physical servers, there are still probably BMCs and such in scope that could achieve analogous things.)
-
But that is still more work than just executing commands via an agent that’s already running. You still have to do something to get root access to a specific machine, instead of being able to script against some agent and accessing all machines.
Leaving your door unlocked is one thing; setting it wide open with a sign “Enter here” is another.
-
-
-
There is absolutely nothing to be done from within a VM to prevent the host flipping a bit and backdooring it arbitrarily, or snapshotting it without shutting it down and doing the same. I’d be very surprised all the big names don’t have this functionality available internally – at least Google support live migration, which is the same tech.
There are open toolkits for doing arbitrarily nasty poking and introspection to a running VM, e.g. volatility framework
Hard to point fingers at Microsoft here
-
-
They can access the memory. That is equivalent to a root password. IMHO CPU supported memory encryption like Intel SGX is snake-oil at most, if you are targeted by the phisycal host of your VM.
Hosting in the cloud is a matter of trust and threat analysis.
-
-
I’m really suprised, it seems that everybody thinks it’s common knowledge and they seem to think it’s normal. I don’t like my hosting provider having this level of access to my data and machines. We are smart enough to find a solution to this, hosting infrastructure without giving up on all security…
-
With managed virtualized infrastructure, “this level of access” is completely unavoidable. They run the virtualized hardware your “server” is running on; they have complete memory and CPU state access, and they can change anything they want.
I guess it makes backdooring things marginally simpler to write a guest-side agent, but their actual capabilities are totally unchanged.
This is something that ought to be common knowledge, but unfortunately doesn’t seem to be.
-
The risk of your provider taking a snapshot of your disk and ram is always there with virtualization. But, you could encrypt the disk, which would make it harder for them (they have to scan ram for the key, then decrypt). But just an agent with root privileges… what bothers me the most I guess is that it is not made clear. A note in /etc/issue or motd with “we have full root acces in your vm, read http://kb.ms.com/kb77777 for more info” would make it clear right from the get-go.
-
-
-
No. Even the various “gov clouds” are mainly about isolation from other customers and data center location.
The cloud providers are executing the cpu instructions that the VM image provided by you (or picked from the store) contains. There isn’t any escaping that access level.
The only option is to actually run your own physical hardware that you trust in an environment you consider good enough.
-
In my comment about host and TLA resistance, I had a requirement for setups resistant to domestic TLA’s that might give orders for secrets to be turned over or use advanced attacks (which are getting cheaper/popular). It can be repurposed for an untrusted, host setup.
“If it has to be U.S. and it’s serious, use foreign operated anti-tamper setup. The idea is all sensitive computations are run on a computer stored in a tamper detecting container that can detect radiation, temperature changes, power surges, excessive microwaves, etc. Tamper detection = data wipe or thermite. The container will be an EMSEC safe and the sensors/PC’s will always be located in a different spot in it. The system is foreign built and operated with the user having no control of its operation except what software runs in deprivileged VM’s in it. Status is monitored remotely. It helps to modify code so that most sensitive stuff like keys are stored in certain spot in memory that will be erased almost instantly.”
The clouds aren’t built anything like this. They have total control like those in physical possession of hardware and software almost always have total control. They can do what they want. You won’t be able to see them do it most of the time without some clever detection mechanisms for security-relevant parts of the stack. That’s before we get to hardware risks.
Bottom line: external providers of computing services should always considered trusted with full access to your data and services. By default. Every time. It’s why I encourage self-hosting of secrets. I also encourage pen, paper, and people for most confidential stuff. Computers aren’t as trustworthy.
-
What is your threat model?
There is something based on selinux for Xen https://wiki.xen.org/wiki/Xen_Security_Modules_:_XSM-FLASK which can by design prevent the privileged “dom0” from reading the memory of nonprivileged guest domains. But that assumes you trust your provider to actually implement this when they say they do.
-
-
-
A note in /etc/issue or motd with “we have full root acces in your vm, read http://kb.ms.com/kb77777 for more info” would make it clear right from the get-go.
I think this is a combination of “common knowledge, so not worth mentioning specially” for users who already know this and “let sleeping dogs lie” for people who don’t already know. I mean, why press people with their noses on a fact that the competitor is equally mum about? Seems like a bad PR move; you’d get clueless people all alarmed and leaving your platform for reasons that are totally bogus, as any competitor has the same kind of access.
-
-
-
-
-
Dell XPS 13 9350 (over two years old now). Previous two were ThinkPad X series. None of them with 15” displays, though.
My main problem with cheap laptops, and even some expensive “consumer market” laptops is flimsy keyboards with poor key travel or (worse) flex in the top of the chassis when typing (I’m a relatively heavy typist.)
Plus I value a docking station or a USB type C cable where I can quickly plug in/out at my desk.
(Your priorities may vary, of course.)
If you’re on a budget, I recommend looking for something high specced and a couple of years old. My laptop before this one was bought used (two years old) and had belonged to the CTO of a high frequency trading company. Was optioned up completely when new, so build quality and specs were still way above anything available new at that price.
-
I have had the XPS 13 9343 for around ~three years I think. I think it’s great.
If you’re on a budget
I bought this particular one refurb from Amazon for ~$900. I feel like I gambled and got lucky.
After having used this one for so long, I think I’d prefer a laptop with more memory. Everything else has been excellent.
-
-
-
Thanks. I’m looking at the XPS15. the non-touch model is a strong contender.
-
-
-
After dragging my heels forever, I finally settled on an XPS last week as a replacement for the endless series of 2011 Macbook Pros I’ve been wearing out for the past 10 years (2007 Macbooks before that). I don’t like buying new hardware, so ended up with a 4K 9550 / i7 quad / 32 GB RAM from eBay.
The machine is almost everything I was hoping for, including the touchpad, with one exception: the panel response time is so bad you could measure it with a sand timer. Looking around, it seems this is a long-running complaint with XPS. I’m chatting to the seller to see if he repasted the machine because there was some trick to make the panel behave sanely, but otherwise, looks like this is not the Macbook replacement I’ve been dreaming of :(
Currently travelling with my trusty beaten up “hobo” Macbook Pro and its barely functional keyboard – it’s almost impossible to beat this machine, and it’s increasingly looking like its final replacement is going to be yet another 2011 Macbook Pro
Note that many of the XPS 13 models have soldered disk / RAM.
-
-
If you are willing to spend as much, the XPS 15” is great. For a cheaper option, consider Dell’s Inspirons. https://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-7000/spd/inspiron-15-7580-laptop/dncwwc120h. They used to be of awful quality but the new series is decent (15” 1080p IPS, metallic body, thin bezels, great linux support, reliable build quality, comes with dual-drives - SSD and HDD together). I’ve been using one myself since over a year now. But don’t expect more than 3 hours of battery life for serious work, webcam is garbage, and the aluminium edges will cut your wrists.
-
Tangentially related, it seems like macOS on M1 swaps very enthusiastically, to the point of possibly bringing SSD life well under a year: https://twitter.com/marcan42/status/1361151198921826308
(I didn’t want to post a twitter thread as a submission, but seems like it may be of interest.)
As marcan42 points out in the thread, this is clearly an OS software issue so it should be patchable in software as well.
Interesting. I’m up to 5.4TBW; kernel_task has written 69.5GB (!) in 3 days 1 hour of uptime. One to keep an eye on; thanks for sharing.
This is madness, I’ve only managed 15 TBW on a Samsung 970 EVO 1TB on an extremely heavily used laptop in 28 months, under 2% the drive’s warrantied TBW, and keenly aware I’ve been hammering the drive at various points. That also includes 2 h00j VMware VMs
Not a big problem, just swap out the SSD of your MacBook when it’s dead. Oh wait, it can’t be replaced :/ Another reason to support Right To Repait.
This might be controversial, but I think you’re just watching what happens when miniaturization and integration happens. The SSD on these is basically directly connected to the FSB, and that contributes to the performance of it. How do you make that replaceable effectively?
Your ALU used to be a discrete, replaceable component. Then it became integrated. Then L2 cache. Should it stop, especially if integration can make things more reliable (i.e RAM slot failure)?
I think “things that are consumables” such as batteries are those things that absolutely must be replaceable. SSD’s fit that category because they actually wear out over time.
But I think you raise good points about other discrete components, not being able to upgrade my RAM sucks, but if it’s more reliable, performant, cheaper and uses less power than alternatives, then it’s a compelling choice.
I agree that this is miniaturization and integration, but I’d argue it’s not strictly necessary for performance.
AFAIK the M1 RAM is LPDDR4X-4266 and you can buy DIMMs[*] in this specification as well. The SSD is NVMe and as far as I know there’s nothing special about the signalling compared to an off-the-shelf NVMe SSD.
I don’t have any numbers to back this up, but my gut feeling is that in the average upgradable laptop the number of lifetime failures that require replacing the RAM is going to be equal or higher than the number of lifetime failures that require replacing the RAM slot - so there’s a gain and a loss here.
I’d suggest it boils down to three things:
[*] EDIT: I originally thought you could buy SO-DIMMs in this spec, but maybe only DIMMs. I think it’d be technically possible to have a small size & replaceable standard for these, but maybe the industry is going with soldered RAM to the extent that it doesn’t exist.
I wonder how much putting RAM on as an MCM lets them run LPDDR at those speeds/latencies.
And yet, the only failing RAM I had in a machine in the last 10 years was a MacBook Pro with on-board RAM. If the machine actually had a DIMM slot, it could’ve been replaced without replacing the whole logic board. (Since the MacBook Pro was just two days old, they replaced the whole system, of course.)
This comment should be a separate post by itself. Thank you for the heads up!
Not a problem on mine somehow (918GB writes in 6 weeks).
I’m second thinking getting an M1 now, maybe I’ll wait for this to be fixed. Hopefully, in time for the new macbook pros. :p
Still my current Linux laptop is 4 years old, and has <10TB TBW on its nvme. I haven’t used it a lot in the past 6 months but it has been in used daily before that. So, 918GB in six weeks still seems like a lot.
shurg
Just checked my 3.5 year old MB12, it had 27.5TB writes over 182-ish weeks, which is roughly 0.9TB/6W. So yeah, it’s normal.
I’ve had a 2019 MBP from work for almost a year now, and I’m at 65.8 TB written. I don’t think this is an M1 problem so much as a macOS problem (if indeed it’s actually a problem).
Yes it’s certainly an OS issue.
Could be some combination of usage patterns with memory configuration. Like I don’t do npm or use any heavyweight IDEs, maybe they provoke the system to swapping out more.
FWIW smartctl claims 27 TB written on my mid-2012 MBA11. I’m no expert, but I think my wearout is all zeroed out. Can’t upgrade past 10.15, not sure if OS matters.
I have had this experience with macOS (VM) in general, testing my memory profiler’s OOM detection (https://pythonspeed.com/fil). it seems much more aggressive about swapping than Linux, to the point where I needed to come up with a different heuristic.