Please don’t copy the terrible overlay setup in Kubernetes YAML for infrastructure. I don’t see what this offers above the standard Terraform command/functionality.
Not necessarily an application performance but a team/infra performance increase. Some of the details are foggy, but I worked at a company that wanted to add search to a long list of items in a separate PostgreSQL database. The main database was approximately 1.5T in size. The design the team came up with was this new service that would slurp up all the data into an Elasticsearch cluster and then build specific searches around templates. Seems reasonable, and often what is reached for right? Sure, until our database guy chimed in one day and said, “You know… instead of spending $20k on this application’s infrastructure, you could just use the full-text search capabilities in PostgreSQL.”
Thankfully the team listened and not only did we save a bunch of cash, development time went from 3 months to build the service to a month or so to get the frontend assets right. It was awesome to see. PostgreSQL is great.
When people say “10 * time programmers don’t exist” I think of cases like these. They absolutely do exist: they are the people who take a step back, think about another solution, and avoid building the whole system altogether.
I’ve started writing a blog on this exact topic, because I’ve been complaining about it for 20 years: tech companies focus on hiring problem solvers, they don’t focus on hiring people who can identify the right problems to solve.
would be good to read!
I am of opinion that the hiring practices are even less relevant than the above.
I think the companies are hiring ‘Jeopardy contestants’ – not book/novel writers.
Both types deal with ‘words’ – but at a different level of composition, obviously.
Certainly there are different tasks needed a varying spectrum of the composition skills – but testing for a Jeopardy-like skills, does not reveal much of the composition skills.
However, these are the interviewing processes of today.
10x programmer isn’t someone who write 10x more code. It is someone who is willing to spend their time on finding solution that will suffice and will take 10x less time to implement.
I’ve watched the Hashicorp stuff from afar but never actually used it. The reference to NATS made me wonder if they considered using Serf, and if so, why they went with NATS instead. Serf does not seem to get much airtime for some reason.
Since you like questions…I assume you are migrating away from nomad because it no longer fits your abstractions. While I can guess, what are those reasons? From the sound it seems you are migrating to a home grown solution.
The way I expected this to work from the README and name is that it wrapped the PostgreSQL C code to run the lot in-memory, or something along those lines.
But looking at the source this is not how it works at all: NewDatabase() fetches a binary from https://repo1.maven.org, extracts it to a temporary path, and runs pg_ctl start with exec.Command(). Nothing is “embedded” in any way I would use the word, and it’s not even easy to implement that in the current version as it expects to read stuff from the filesystem (rather than an io.Reader or whatnot). It’s also just an unmodified PostgreSQL as far as I can tell.
Useful for some things? I guess? Maybe? But very limited. I’d just run PostgreSQL.
Also, this was so obviously written by a Java developer.
Good article though I kind of wish it showed some examples. I really like the idea behind OTP, and using this to orchestrate Go (or other services) makes a lot of sense. I will do some research to understand a bit more how Elixir handles this.
I actually toyed with doing something like this using FUSE, but I couldn’t get it to work very well. I was a bit out of my league writing something like that though, I just thought it would be a great way to use GitHub.
I think about this a lot. I agree with some others here that Plan 9 is the peak of operating system design, but I think some things I’d like to see (in random/brain storm fashion):
Expand on the idea behind Erlang and processes having a common messaging behavior. Almost everythin becomes “gen_server” like. I know Plan 9 has this idea of everything as a file server, but abstract that a little bit and create the same interface across the board.
Isolate hardware and software, meaning put everything into a fence that users can be given access to them. Jails and Solaris Zones are inspirations here, but go further. I know people hate it but the fact that MacOS requires a user to grant access to areas of their system is what I’m after, but go further and do it with hardware/kernel level structures as well. And I don’t want to be prompted, but it is something that I can configure (if that makes sense). To the point where as a power user I could drop this “fence” config onto my cluster and completely lock down everything except what I explicitly make available for processes to run. Almost like OpenBSD’s pf but for the security of my userspace.
ZFS but more workflows like git/fossil/mercurial. Make it easy to branch a path and “archive” it. Focus on the delta between states, allow me to snapshot/branch “a point in time” and just diverge into this reality without impacting my main files.
Functional programming methodology in tooling. What I mean by this pretty terrible phrase is something like the command line in pipes but as a language across the system. Complex data flows (streams, bytes, strings, objects, etc) enter tools and I can process and filter them and produce something else. Almost like IFTT and pf at a system level. This builds on the “everything is a gen_server” idea above.
Elevate the terminal and “user facing” commands more. Think Alfred, Quicksilver, but it is the terminal and user friendly so my mother-in-law used it. “open” is a tool that acts on data, that data might be an application, an image, a URL, etc. Again, think functional. :)
Finally make it so that things like dtrace/ebpf are just standard behavior, not these complex hooks into a system, but a part of the standard method for writing process, that they naturally produce that info that these types of tools can dig into.
Oh one more thing, packages/package manager should isolate all installable items (think Nix). And don’t choose some special “bucket”, use standards like tar/gzip. I don’t want to have to have special tools to peer into some package.
That is it for now. Just ideas, things I like today but that are not ubiquitous across a system.
If there’s a thing to take from Erlang, I think it’d be to use structured data (tuples, atoms, ints, strings/bitstrings) in messages, not just a stream of bytes like unix does. I think an OS where programs communicate via Erlang’s data structures would be much better and richer than anything built on unstructured bytes.
It should transcend machines. Ideally, the OS should have an immutable part and a mutable part (my data + all installed programs). That way, I’ll be able to push and pull around my workspace across machines.
It should have no processes. Processes are just hidden data. Let’s just have functions instead. All programs shall be functions (that is, return something).
Program composition capabilities. If programs are functions we can chain them. Similar to pipes but simpler to reason about.
File extensions are broken. File extensions are an afterthought to file’s having an associated program. I’d call this file types instead and store it in a separate field.
Much, much better filesystem. Atomic transactions. Plus structured data similar to a DB.
File level data deduplication built into the filesystem.
Undo/redo for the entire filesystem. Even for installing programs. If I’m unhappy, I should be able to undo.
Allow branching, merging, similar to Git.
All the above are ideas I had in mind for a long long time but all of them are reality now, it’s called the Boomla OS. I hope that’s not cheating just because I’ve made it real. :P
Some of the bits about branching/undo/mutability might be resolvable with ZFS. The UI for it could be much better though.
File extensions are broken. File extensions are an afterthought to file’s having an associated program. I’d call this file types instead and store it in a separate field.
It’s funny how everyone here seems to express a sigh of relief that the mentioned question in the title has been “debunked”. I don’t think it has been, here’s why: Instead of the app signature, Apple is sent the developer signature. However, how is that not a big problem? The author gives Mozilla as an example of how Apple couldn’t tell if you were using Firefox or Thunderbird, but most developers only have one successful app and it’s trivial for Apple (and everyone on your (public) Wifi) to infer which apps you’re running because of that.
This is a big deal and we should rightfully question this practice. I’m glad I left the Apple-ecosystem in 2012, because what we’re seeing here with all the code-signing madness and locking down is an iOS-ification of macOS, and there will be a point where you’ll be completely locked in and not able to run your hardware a few years beyond the last software update, because all the signatures will have expired. Have fun on this sinking ship if you still decide to stay!
Or maybe – and I know this is something you probably are not prepared to accept, possibly not even capable of accepting – just maybe… other people see this differently than you do, or want to take different sides of a trade-off than you do, or, well, just don’t have exactly the same opinions as you on every conceivable thing under the sun, but still are acting rationally on the same set of information as you, and you should learn how to disagree with them without painting them as inherently stupid and/or evil.
Wow, did my comment really hit that close to home? I could think of multiple (privacy-protecting and actually more efficient) other ways Apple could’ve shared revocations across their install-base. This is clearly about control and will only get worse. Anyone who believes in free general-purpose computing would be a madman staying in this ecosystem any longer.
Wow, did my comment really hit that close to home?
Seriously, just step back and take a look at yourself. You’ve demonstrated that it’s literally inconceivable to you that someone might disagree with you rationally and in good faith, by effectively saying you think it’s impossible for any person to disagree without being either mentally ill or outright evil. This is not a rational position and a rational person would not hold it!
Anyone who believes in free general-purpose computing
I will return to an analogy I’ve used before: Apple’s inclusion in their products of security features you personally dislike is as much a “war on general-purpose computing” as a generic corporate holiday greeting is a “war on Christmas”, and proponents of those two positions display, in my experience, roughly equal rationality and equal inability to be reasoned down from the precipices on which they’ve perched themselves.
Keep using your Mac, if you do and are in a moral conflict, and see what this process leads to. I’ve been watching this with Apple for a few years and have seen numerous examples in the industry. This is not an emotional but an objective standpoint, and I don’t care about your feelings. Bring some counterarguments (instead of vague philosophical assertations) please, because otherwise this discussion here leads nowhere.
there will be a point where you’ll be completely locked in and not able to run your hardware a few years beyond the last software update, because all the signatures will have expired. Have fun on this sinking ship if you still decide to stay!
This is not an emotional but an objective standpoint, and I don’t care about your feelings.
Uh huh, sure.
I think what you fail to see is that a majority of users require this level of security. Apple has already stated they will add a method to turn off this verification and for those “hackers” it is as easy as blocking the traffic. OHHHH, what’s that you say? An Apple user has a choice? By gosh, I think the computer still works, that is amazing!
I want to go back to your earlier point though, about hardware dying with a software updates. I’d like to point to the fact that I have a MacBook that is 7 years old and is still functioning just fine. Also, iPhones and iPads are notorious for their long term support compared to Android devices. Perhaps you are letting your emotions towards Apple get in the way here?
I think what you fail to see is that a majority of users require this level of security. Apple has already stated they will add a method to turn off this verification and for those “hackers” it is as easy as blocking the traffic. OHHHH, what’s that you say? An Apple user has a choice? By gosh, I think the computer still works, that is amazing!
As I said earlier, one can argue in favor of the benefits of this solution, but the implementation is strange and raises some serious concerns for privacy.
I want to go back to your earlier point though, about hardware dying with a software updates. I’d like to point to the fact that I have a MacBook that is 7 years old and is still functioning just fine. Also, iPhones and iPads are notorious for their long term support compared to Android devices. Perhaps you are letting your emotions towards Apple get in the way here?
I don’t have emotions against Apple and even recommend it to some of my customers who benefit from the ecosystem. Big Sur might be the last software update for your MacBook, and after a while (2-3 years) signatures will start expiring and you won’t be able to run software that’s otherwise perfectly fine.
Apple has been locking down macOS for quite some time now. Every release is just a few degrees hotter for the proverbial frog in the water-pot. While a few degrees at a time might not be noticeable or can even be disregarded, the water will end up boiling at some point.
You’re the one who made a series of claims, not least of which is that those who disagree with you are either evil (in that they knowingly and deliberately wage some type of war against “general-purpose computing”), or mentally ill (“madmen”, to use your term).
You’ve provided no evidence for this other than vague mutterings about how you’ve been “watching this” and “seen numerous examples”.
It is up to you, as the party making positive claims, to support said claims with evidence before you demand that someone else knock them down; this is, among rational people, the way debate and discussion is expected to proceed. And the fact that you don’t and likely won’t, but only retreat further into sneering (“I don’t care about your feelings”) and borderline insults, speaks eloquently about both you, and your claims.
Feel free to disregard what I’m saying, but the outcome will speak for itself. If you tolerate the obvious developments of the past few years, I can’t help you. Everyone is entitled to his own opinion and if you choose to spend your money there, it’s your freedom to do so. I’d still think of you as a madman, and you can’t shame me to think otherwise because I don’t measure myself to your standards.
Until you make a falsifiable claim and assert some evidence for it – other than your “feelings”, which is odd given what disdain you openly show for those of others – I will, because you are not participating in rational discourse.
I’d still think of you as a madman, and you can’t shame me to think otherwise because I don’t measure myself to your standards.
The fact that you still can’t handle someone disagreeing with you, without needing to insult or attack them, continues to speak volumes to anyone willing to listen.
Please don’t copy the terrible overlay setup in Kubernetes YAML for infrastructure. I don’t see what this offers above the standard Terraform command/functionality.
Hi, author here, I suggest you check out my talk at SREcon where I explain the problems it solves.
Cool, I will check it out.
Not necessarily an application performance but a team/infra performance increase. Some of the details are foggy, but I worked at a company that wanted to add search to a long list of items in a separate PostgreSQL database. The main database was approximately 1.5T in size. The design the team came up with was this new service that would slurp up all the data into an Elasticsearch cluster and then build specific searches around templates. Seems reasonable, and often what is reached for right? Sure, until our database guy chimed in one day and said, “You know… instead of spending $20k on this application’s infrastructure, you could just use the full-text search capabilities in PostgreSQL.”
Thankfully the team listened and not only did we save a bunch of cash, development time went from 3 months to build the service to a month or so to get the frontend assets right. It was awesome to see. PostgreSQL is great.
When people say “10 * time programmers don’t exist” I think of cases like these. They absolutely do exist: they are the people who take a step back, think about another solution, and avoid building the whole system altogether.
(And yeah, postgres is great)
I’ve started writing a blog on this exact topic, because I’ve been complaining about it for 20 years: tech companies focus on hiring problem solvers, they don’t focus on hiring people who can identify the right problems to solve.
would be good to read! I am of opinion that the hiring practices are even less relevant than the above.
I think the companies are hiring ‘Jeopardy contestants’ – not book/novel writers. Both types deal with ‘words’ – but at a different level of composition, obviously.
Certainly there are different tasks needed a varying spectrum of the composition skills – but testing for a Jeopardy-like skills, does not reveal much of the composition skills.
However, these are the interviewing processes of today.
10x programmer isn’t someone who write 10x more code. It is someone who is willing to spend their time on finding solution that will suffice and will take 10x less time to implement.
I’ve watched the Hashicorp stuff from afar but never actually used it. The reference to NATS made me wonder if they considered using Serf, and if so, why they went with NATS instead. Serf does not seem to get much airtime for some reason.
We used to use Serf to gossip load; I wrote a blog post about Serf, because I like it too:
https://fly.io/blog/building-clusters-with-serf/
Ultimately NATS was just easier to get working across Go and Rust for us, and it’s more flexible. But they’re pretty similar systems.
Thanks for sharing that with me!
Thanks for asking! :)
Since you like questions…I assume you are migrating away from nomad because it no longer fits your abstractions. While I can guess, what are those reasons? From the sound it seems you are migrating to a home grown solution.
@tptacek answered this on the orange site: https://news.ycombinator.com/item?id=30863610
Bump. :) I was interested to hear what the plan was there also.
https://lobste.rs/s/spvdwx/consul_at_fly_io#c_adpa5o
The way I expected this to work from the README and name is that it wrapped the PostgreSQL C code to run the lot in-memory, or something along those lines.
But looking at the source this is not how it works at all:
NewDatabase()fetches a binary fromhttps://repo1.maven.org, extracts it to a temporary path, and runspg_ctl startwithexec.Command(). Nothing is “embedded” in any way I would use the word, and it’s not even easy to implement that in the current version as it expects to read stuff from the filesystem (rather than anio.Readeror whatnot). It’s also just an unmodified PostgreSQL as far as I can tell.Useful for some things? I guess? Maybe? But very limited. I’d just run PostgreSQL.
Also, this was so obviously written by a Java developer.
That’s kind of nasty. Why not just use a shell script that makes and runs a docker container?
that would depend on docker, whereas the goal of this project is to depend only on the Go standard library.
It is possible to run a PostgreSQL database in memory which is what I expected to find. I agree this doesn’t really fit the “embedded” idea at all.
Any links to running a postgres database in memory for real?
Good article though I kind of wish it showed some examples. I really like the idea behind OTP, and using this to orchestrate Go (or other services) makes a lot of sense. I will do some research to understand a bit more how Elixir handles this.
I actually toyed with doing something like this using FUSE, but I couldn’t get it to work very well. I was a bit out of my league writing something like that though, I just thought it would be a great way to use GitHub.
I think about this a lot. I agree with some others here that Plan 9 is the peak of operating system design, but I think some things I’d like to see (in random/brain storm fashion):
That is it for now. Just ideas, things I like today but that are not ubiquitous across a system.
If there’s a thing to take from Erlang, I think it’d be to use structured data (tuples, atoms, ints, strings/bitstrings) in messages, not just a stream of bytes like unix does. I think an OS where programs communicate via Erlang’s data structures would be much better and richer than anything built on unstructured bytes.
All the above are ideas I had in mind for a long long time but all of them are reality now, it’s called the Boomla OS. I hope that’s not cheating just because I’ve made it real. :P
You and I are on the same page on a lot of these.
Ha indeed! The one on package isolation is also a big one I am also totally on the same page with you! Just like hardware / software isolation.
Some of the bits about branching/undo/mutability might be resolvable with ZFS. The UI for it could be much better though.
The original Macintosh had creator types.
It’s funny how everyone here seems to express a sigh of relief that the mentioned question in the title has been “debunked”. I don’t think it has been, here’s why: Instead of the app signature, Apple is sent the developer signature. However, how is that not a big problem? The author gives Mozilla as an example of how Apple couldn’t tell if you were using Firefox or Thunderbird, but most developers only have one successful app and it’s trivial for Apple (and everyone on your (public) Wifi) to infer which apps you’re running because of that.
This is a big deal and we should rightfully question this practice. I’m glad I left the Apple-ecosystem in 2012, because what we’re seeing here with all the code-signing madness and locking down is an iOS-ification of macOS, and there will be a point where you’ll be completely locked in and not able to run your hardware a few years beyond the last software update, because all the signatures will have expired. Have fun on this sinking ship if you still decide to stay!
Or maybe – and I know this is something you probably are not prepared to accept, possibly not even capable of accepting – just maybe… other people see this differently than you do, or want to take different sides of a trade-off than you do, or, well, just don’t have exactly the same opinions as you on every conceivable thing under the sun, but still are acting rationally on the same set of information as you, and you should learn how to disagree with them without painting them as inherently stupid and/or evil.
Wow, did my comment really hit that close to home? I could think of multiple (privacy-protecting and actually more efficient) other ways Apple could’ve shared revocations across their install-base. This is clearly about control and will only get worse. Anyone who believes in free general-purpose computing would be a madman staying in this ecosystem any longer.
Seriously, just step back and take a look at yourself. You’ve demonstrated that it’s literally inconceivable to you that someone might disagree with you rationally and in good faith, by effectively saying you think it’s impossible for any person to disagree without being either mentally ill or outright evil. This is not a rational position and a rational person would not hold it!
I will return to an analogy I’ve used before: Apple’s inclusion in their products of security features you personally dislike is as much a “war on general-purpose computing” as a generic corporate holiday greeting is a “war on Christmas”, and proponents of those two positions display, in my experience, roughly equal rationality and equal inability to be reasoned down from the precipices on which they’ve perched themselves.
Keep using your Mac, if you do and are in a moral conflict, and see what this process leads to. I’ve been watching this with Apple for a few years and have seen numerous examples in the industry. This is not an emotional but an objective standpoint, and I don’t care about your feelings. Bring some counterarguments (instead of vague philosophical assertations) please, because otherwise this discussion here leads nowhere.
Uh huh, sure.
I think what you fail to see is that a majority of users require this level of security. Apple has already stated they will add a method to turn off this verification and for those “hackers” it is as easy as blocking the traffic. OHHHH, what’s that you say? An Apple user has a choice? By gosh, I think the computer still works, that is amazing!
I want to go back to your earlier point though, about hardware dying with a software updates. I’d like to point to the fact that I have a MacBook that is 7 years old and is still functioning just fine. Also, iPhones and iPads are notorious for their long term support compared to Android devices. Perhaps you are letting your emotions towards Apple get in the way here?
As I said earlier, one can argue in favor of the benefits of this solution, but the implementation is strange and raises some serious concerns for privacy.
I don’t have emotions against Apple and even recommend it to some of my customers who benefit from the ecosystem. Big Sur might be the last software update for your MacBook, and after a while (2-3 years) signatures will start expiring and you won’t be able to run software that’s otherwise perfectly fine.
Apple has been locking down macOS for quite some time now. Every release is just a few degrees hotter for the proverbial frog in the water-pot. While a few degrees at a time might not be noticeable or can even be disregarded, the water will end up boiling at some point.
You’re the one who made a series of claims, not least of which is that those who disagree with you are either evil (in that they knowingly and deliberately wage some type of war against “general-purpose computing”), or mentally ill (“madmen”, to use your term).
You’ve provided no evidence for this other than vague mutterings about how you’ve been “watching this” and “seen numerous examples”.
It is up to you, as the party making positive claims, to support said claims with evidence before you demand that someone else knock them down; this is, among rational people, the way debate and discussion is expected to proceed. And the fact that you don’t and likely won’t, but only retreat further into sneering (“I don’t care about your feelings”) and borderline insults, speaks eloquently about both you, and your claims.
Feel free to disregard what I’m saying, but the outcome will speak for itself. If you tolerate the obvious developments of the past few years, I can’t help you. Everyone is entitled to his own opinion and if you choose to spend your money there, it’s your freedom to do so. I’d still think of you as a madman, and you can’t shame me to think otherwise because I don’t measure myself to your standards.
Until you make a falsifiable claim and assert some evidence for it – other than your “feelings”, which is odd given what disdain you openly show for those of others – I will, because you are not participating in rational discourse.
The fact that you still can’t handle someone disagreeing with you, without needing to insult or attack them, continues to speak volumes to anyone willing to listen.