Speaking of OneNote’s local notebook format, it seems to be documented (also this one) but it seems quite complex - I’ve never seen an implementation of it other than Microsoft’s own.
I’ve always thought that “advice” was insulting, condescending, and wrong. Obviously someone needs to write all those things, and that someone better had the experience which only comes after doing all those things for a while.
We need to encourage better understanding of tricky fields, and we need to encourage software diversity and fight against software monocultures.
Any monopoly or oligopoly is usually bad for the customer. Things stagnate at best or get destructive to customers at worst. Even when they’re good, attempts at reinventing the wheel slightly can be a lot better. For a recent example, VMS-style clusters with a database were pretty bullet-proof at high-availability within a few hundred miles at decent performance. I’m still glad people worked on alternatives necessary to build Spanner and FoundationDB, though. That’s why we get even higher performance on cheaper boxes at a global level with similar level of consistency. It didn’t happen overnight either: people stayed trying to reinvent databases and distributed services for a while before things lined up right.
4ad’s OpenSSL example is a good one. iOS vs Android for mobile apps is another.
Doesn’t your argument mostly apply to closed source software in a commercial setting? I think it’s possible Linux wouldn’t stagnate even if there were no alternative OSes.
Linux, as a product people consume, has a lot of diversity in it, from Ubuntu to Arch, to Slackware. As a kernel, no, Linus is pretty heavy handed on what the kernel cannot do these days, so we’re stuck with things like epoll until Linus changes his mind or moves on to another project.
But lets flip this on me and have me using an OS that I quite like and thinks makes a lot of right decisions: FreeBSD. Do I think stagnation would be a problem if FreeBSD was the only operating system? Definitely. FreeBSD has ended up steeling good ideas from other operating systems that were very unlikely to arise within FreeBSD. The security conscious programmers at OpenBSD might have a kinship to FreeBSD, but the values of FreeBSD to align enough with them to be able to innovate in the ways that interest them.
That you went right to Linux shows that it’s an outlier. The last time someone dug up data, Nadia Eghbal, most of the FOSS projects were in bad shape financially. Things like Linux are rare. Situation is opposite for proprietary software.
I’m aware of the funding issues with open source, but I think that’s a separate issue from what I’m trying to say. My point is: a monopoly or oligopoly of open source software wouldn’t be bad for the users of that software, because development would still continue due to the different incentive structure. Do you think that’s not true?
I think there’s at least two concerns: development drops off due to incentives not holding up (happens a lot); incentives, esp if corporate, can push the FOSS in directions people dont want. Red Hat and systemd is the first example coming to mind for Linux. Another was a scheduler I read about that improved things on desktop workloads that was rejected or pulled due to server focus of many stakeholders.
Good for users is relative. That there’s often competing interests means ability to diverge on some points is valuable. Whether you need it or not, who knows. Someone might.
OpenSSL has many problems but no matter how well funded OpenSSL could be it will have a security hole in it because it’s software. Having some competition at least means not all of the software on ones stack would have the security hole.
The common, concrete, argument given is usually security. A hole found in Linux probably does not affect BSD. VeriSign runs (at least?) three operating systems in their infrastructure for this reason. There is a push to support Tor better in BSD for this as well. It’s possible if x86 didn’t dominate our servers, Meltdown and Specter could have been less problematic (tangentially, I hope something comes of POWER9).
For more hand wavy arguments, the different project cultures has allows for different ideas to flourish with other systems picking up the things that win. Despite not being an OpenBSD user, I support it with my money because I want it to survive because I think the world is a better place with the ideas that bubble up to the top making it elsewhere. I think it would be challenging for someone with an OpenBSD ethos to participate in Linux.
My job involves pumping lots of bytes through kafka, doing some computations on said data, and then writing them to cassandra.
With more specialized tooling, this could be done far simpler, and cost a fraction of what the whole thing costs to run right now. Of course, at the expense of engineering resources to design, build, test, and operate it all. And, not to mention the time it’d take to do it.
I’m all for composable solutions from well tested components, but let’s be honest—composing infrastructure requires you to make a lot of compromises…like wasting CPU cores by adopting redis…, etc.
I think it might be better to say. “Always be clear about your level of experience and understanding when building something as important as a database. If building a database is a secondary goal and not a primary one you might not have time to do it right.”
In which the Microsoft OneNote team write their own database, and seemingly don’t regret the decision.
OneNote just corrupted my notes locally, so who knows, maybe they should have followed the general advice?..
Speaking of OneNote’s local notebook format, it seems to be documented (also this one) but it seems quite complex - I’ve never seen an implementation of it other than Microsoft’s own.
It might be interesting to poke around with…
If we followed all of these things:
I think we’d all be pretty damn bored.
I’ve always thought that “advice” was insulting, condescending, and wrong. Obviously someone needs to write all those things, and that someone better had the experience which only comes after doing all those things for a while.
We need to encourage better understanding of tricky fields, and we need to encourage software diversity and fight against software monocultures.
Not trolling…but why?
Any monopoly or oligopoly is usually bad for the customer. Things stagnate at best or get destructive to customers at worst. Even when they’re good, attempts at reinventing the wheel slightly can be a lot better. For a recent example, VMS-style clusters with a database were pretty bullet-proof at high-availability within a few hundred miles at decent performance. I’m still glad people worked on alternatives necessary to build Spanner and FoundationDB, though. That’s why we get even higher performance on cheaper boxes at a global level with similar level of consistency. It didn’t happen overnight either: people stayed trying to reinvent databases and distributed services for a while before things lined up right.
4ad’s OpenSSL example is a good one. iOS vs Android for mobile apps is another.
Doesn’t your argument mostly apply to closed source software in a commercial setting? I think it’s possible Linux wouldn’t stagnate even if there were no alternative OSes.
Linux, as a product people consume, has a lot of diversity in it, from Ubuntu to Arch, to Slackware. As a kernel, no, Linus is pretty heavy handed on what the kernel cannot do these days, so we’re stuck with things like epoll until Linus changes his mind or moves on to another project.
But lets flip this on me and have me using an OS that I quite like and thinks makes a lot of right decisions: FreeBSD. Do I think stagnation would be a problem if FreeBSD was the only operating system? Definitely. FreeBSD has ended up steeling good ideas from other operating systems that were very unlikely to arise within FreeBSD. The security conscious programmers at OpenBSD might have a kinship to FreeBSD, but the values of FreeBSD to align enough with them to be able to innovate in the ways that interest them.
That’s a good counterpoint, thanks.
That you went right to Linux shows that it’s an outlier. The last time someone dug up data, Nadia Eghbal, most of the FOSS projects were in bad shape financially. Things like Linux are rare. Situation is opposite for proprietary software.
I’m aware of the funding issues with open source, but I think that’s a separate issue from what I’m trying to say. My point is: a monopoly or oligopoly of open source software wouldn’t be bad for the users of that software, because development would still continue due to the different incentive structure. Do you think that’s not true?
I think there’s at least two concerns: development drops off due to incentives not holding up (happens a lot); incentives, esp if corporate, can push the FOSS in directions people dont want. Red Hat and systemd is the first example coming to mind for Linux. Another was a scheduler I read about that improved things on desktop workloads that was rejected or pulled due to server focus of many stakeholders.
Good for users is relative. That there’s often competing interests means ability to diverge on some points is valuable. Whether you need it or not, who knows. Someone might.
When the alternative is OpenSSL, I think the answer is self-evident.
Were the problems with OpenSSL really driven by the lack of competition, or by the lack of funding?
OpenSSL has many problems but no matter how well funded OpenSSL could be it will have a security hole in it because it’s software. Having some competition at least means not all of the software on ones stack would have the security hole.
The common, concrete, argument given is usually security. A hole found in Linux probably does not affect BSD. VeriSign runs (at least?) three operating systems in their infrastructure for this reason. There is a push to support Tor better in BSD for this as well. It’s possible if x86 didn’t dominate our servers, Meltdown and Specter could have been less problematic (tangentially, I hope something comes of POWER9).
For more hand wavy arguments, the different project cultures has allows for different ideas to flourish with other systems picking up the things that win. Despite not being an OpenBSD user, I support it with my money because I want it to survive because I think the world is a better place with the ideas that bubble up to the top making it elsewhere. I think it would be challenging for someone with an OpenBSD ethos to participate in Linux.
Might also get something done. :)
Or… adapting existing solutions to fit our problems might lead to vulnerabilities, additional cost, additional resource requirements, etc.
But, yeah, you might go faster. You might not. As always, it depends.
I don’t do any of this and pump terabytes of data through infrastructures and networks using stock software. It’s pretty pleasing.
My job involves pumping lots of bytes through kafka, doing some computations on said data, and then writing them to cassandra.
With more specialized tooling, this could be done far simpler, and cost a fraction of what the whole thing costs to run right now. Of course, at the expense of engineering resources to design, build, test, and operate it all. And, not to mention the time it’d take to do it.
I’m all for composable solutions from well tested components, but let’s be honest—composing infrastructure requires you to make a lot of compromises…like wasting CPU cores by adopting redis…, etc.
IMO, we still haven’t nailed some of those, esp web frameworks and OSes.
Don’t read the comments.
That’s actually the best advice. Adding it to the list. :)
I’ve had a FANTASTIC time writing my own database, and I intend to deploy something based on it to “production” thanks very much :D
I think it might be better to say. “Always be clear about your level of experience and understanding when building something as important as a database. If building a database is a secondary goal and not a primary one you might not have time to do it right.”
From Akin’s Laws of Spacecraft Design
Replace exploration with software, program with project, and launch vehicle with database.
Oops, too late :D