I wish projects like this would clearly say why they exist and why anyone might care to use/contribute/follow them.
My best guess from a quick search is that the authors either made it just for fun or they made it because they wanted ZFS features but don’t like the license conflict or the btrfs code.
The proposed value for Linux users seems to be “btrfs but better”.
If you’re interested in the origin, bcache itself is a stable, transparent way to split hot/cold data between multiple fast/slow drives through caching. The author at some point said effectively - you know, that’s halfway to a decent filesystem with good features, so let’s make it happen.
Note that the “Status” section is woefully out-of-date. Everything on that list is complete and stable, except for erasure coding (which is not yet stable). I don’t know about the status of DEX/PMEM.
Honestly, after fairly extensive experience with using Btrfs during 4 years of running openSUSE as my daily driver, the headline on the website tells you what you need to know:
“The COW filesystem for Linux that won’t eat your data”.
That’s it. That’s the story.
Btrfs is the most featureful FS in the Linux kernel but it is not reliable. I know lots of people are in deep denial about this, but it is not.
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck successfully repair all types of filesystem corruption. E.g. some other software or hardware bugs can fatally damage a volume.
WARNING: Using ‘–repair’ can further damage a filesystem instead of helping if it can’t fix your particular issue.
It is extremely important that you ensure a backup has been created before invoking ‘–repair’. If any doubt open a support request first, before attempting a repair. Use this flag at your own risk.
Btrfs is not trustworthy. If you fill the volume, it will corrupt. When it is corrupted, it cannot be fixed and you will lose the contents.
OpenSUSE uses a tool called “Snapper” to automatically take snapshots before software installation. This means progressively filling the root filesystem. Unless you manually manage this and manually prune them, it will fill up, and when it fills up, it is game over.
I have been using Linux since kernel 1.0 or so in 1995 (I suspect well over one hundred distros now, if you count major versions) and in the 20th century it was routine for Linux boxes to self-destruct. If the power went at a bad moment, boom, dead.
21st century Linux is different. It is solid now.
I used openSUSE for 4 years, and in that time, I had to format and reinstall my OS about half a dozen times or more, because Btrfs self-destructed. This was not a one-off experience.
Fedora doesn’t have any snapshotting functionality, and so it’s not using Btrfs for the thing that makes Btrfs special. It’s like a moped enjoying perfect engine reliability because you pedal everywhere and never start the engine.
(Why does it use it then? I think this is why: Fedora leans heavily on Flatpak and is increasing that. Flatpak is extremely inefficient and involves a lot of file duplication. Btrfs can compress that. That means it doesn’t look so bad.)
Btrfs self-corrupts on full volumes and it does not have a working repair tool.
That is not acceptable. Therefore Btrfs is not acceptable, and that’s without considering the issues around its RAID handling and so on.
Yeah. BTRFS is kind of shit. I was recently considering adding more storage to my system and found out that I could select between two allocation policies if my disks were mismatched (or if I added one without balancing). I can use the single profile and it will only write to the bigger disk until it is full (BTRFS’ allocation policy is most free space) or I can pick RAID0 and it will stripe across all disks until eventually it is only writing to a single disks. Both cases hotspotting.
I don’t get the complaint about --repair. This is the same situation as fsck in all filesystems where it exists. If you have a corrupted filesystem, the best a tool can do is make clever guesses about the way to recover. Hopefully the guesses will be right, but they may be wrong. They’re being explicit about it.
Like I said, I have fairly extensive, repeated, personal experience of this.
Fsck is simple and it usually works. I have used it on ext2, ext3, ext4, hfs+, and probably on reiserfs, xfs, and others.
Usually it fixes some damage, maybe small, and afterwards your filesystem works again. It’s as safe or safer than CHKDSK on the DOS/Windows family. It’s part of my routine maintenance.
I have never ever seen Btrfs repair successfully fix any form of damage. None at all, not even once.
What I have seen is a healthy working volume, which I only checked out of curiosity, destroyed by it.
Don’t be misled. What the warning means is “DO NOT RUN THIS”. It will destroy the volume.
There is no way to repair a damaged Btrfs volume, even if the damage is trivial.
I’ve installed openSUSE with Btrfs a few years ago. It worked kind of well, until I’ve tried Android development; the VM was unusable (stuttering, low fps, low performance) if the VM image was stored on a Btrfs partition. When I moved the VM to an ext4 partition, everything was smooth again. I’ve tried to search for the issue on various forums, but couldn’t find any fix for this, so I’ve just resigned from openSUSE and Btrfs altogether, and didn’t look back on it ever again. I haven’t experienced any data losses but from what you write, this was just a matter of time. So it’s good I’ve resigned.
One of the benefits over btrfs today is that bcachefs supports native encryption like ZFS does, meaning you don’t need to juggle LUKS + btrfs.
I think for the next while at least btrfs or ZFS will still be preferred but I suspect that bcachefs will be good enough that the (admittedly minor) effort of ensuring OpenZFS on Linux is working such dealing with lack of packages or kernel upgrades breaking things (hello ArchLinux dkms) will make it a success since it’s natively integrated into the kernel.
A big adopting point would be if container tooling such as containerd start to natively support bcachefs the way btrfs and ZFS are.
I think bcachefs is significantly better than ZFS or BTRFS due to flexibility. You can pick storage configuration per-directory, chance storage configuration at any time, stripe sizes change automatically as you add disks and mismatched disks are not a problem.
Overal bcachefs feels like you just add your disks, add your files and it more or less does the best possible thing given your placement constraints. ZFS placement is quite rigid and specified upfront and BTRFS placement is way too stupidly simple.
The only two platforms known to actually support zfs are FreeBSD and Linux. No other filesystem than FAT has a good chance at being writable from every major OS
I can know without further confirmation that the files at the other end are exactly what I was sending, and the transit was minimal. I also can then roll back on the remote server to before that send happened, in case I want to check something that was deleted during that sync.
I’m not sure there are “other tools”. Traditionally filesystems have dump/restore for backup/restore, there’s rsync for incremental sync and restic/borg/duplicity support encryption and authentication.
Zfs send/receive works zfs snapshots and native encryption to form a rich, yet simple, solution for sync and backup. It works well ssh for network transport - or eg USB media for local backup. It supports remote mounting (mount backup on backup host) if wanted.
Like most things zfs, it’s the combination of features that creates a powerful emergent whole.
ZFS […] If you have 2 disks mirrored you can’t add a 3rd disk and still have all data stored twice.
That isn’t currently the case anymore, you can add a third disk to a mirror setup and it’ll resilver just fine to have triple copies of everything.
Though the RAID still applies (2.2 adds DRAID, which does let you expand IIRC, at massive complexity cost and has issues still, RAID Expansion is still in the pipes).
Did you make a typo? The requirement is 3 disks and 2x redundancy. I’ll admit that I didn’t look into the configuration too closely but this seemed difficult to do with ZFS. Especially if you were starting from 2 disks and wanted to live migrate to 3.
My understanding is that at the end of the day you configure your performance/redundancy into a virtual block device. You then put a more-or-less single-disk filesystem on top of that. (It is smarter than your average single-disk filesystem because it is aware of the topology but IIUC the behaviour is basically the same). You can grow by adding more disks to the virtual block device but you don’t really have much option to change the existing devices in the pool. (As you said there are some limited options such as 2 disks replicated can become N disks replicated).
bcachefs (and BTRFS for this case) are fairly flexible about this. They see that you want 2 copies and they pick two disks and put the data there. Adding more disks gives them more options to pick from. With bcachefs you can get even more flexible and say “everything in this directory shall have 4 copies” and it will replicate that data more times (well it won’t proactively replicate existing data IIUC, it will replicate new writes or on a balance operation) even if that filesystem previously had no configuration with 4x redundancy.
ZFS’ internal structures are messy, but essentially there is two types of vdev’s; RAID and Mirror.
A non-mirror vdev (ie, just a plain disk, no redundancy), is simply a special case of a RAID1 with no partners (though internally it simply presents the device directly as pool member).
Mirror vdevs can be brought up and down in redundancy by adding or removing disks. If your pool is only mirror disks, you can even split it in half and have two pools with identical data. You can also remove and add mirror vdevs at your pleasure. Even convert single-disks into a mirror configuration.
RAID vdevs are much more strict. They cannot be expanded once created, they can’t be split and cannot be removed from a pool. This is due to the internal block pointers being in a special format to handle the RAID stripes.
The RAID Expansion work doesn’t even change that, it instead allows you to rewrite a RAID vdev in a way where it has more members than before. The old data will still only exist at the previous redundancy level (ie, same parity-data ratio) and only new data will benefit from the better parity to data ratio.
The objective stated was “a filesystem that has 3 disks, and all data has 2 replicas”. Number of data replicas < number of disks. This saves 50% of your storage space if your failure budget only requires 2 replicas.
You would have to ask them, but a bunch of people do complain about it.
Now that this comment the thread has helped me learn more about bcachefs, I don’t think the licensing was a huge deal. The bigger motivations are: bcachefs devs think they can do much better than ZFS or BTRFS; and also ZFS will never be first class on Linux (because Torvalds don’t want it in the kernel, etc)
Eh, Torvalds doesn’t ship binaries. It’s companies like RedHat/IBM, Amazon, Google, Samsung and Microsoft that ship kernel binaries and wouldn’t want legal exposure.
Canonical paid for & published a legal opinion and shipped binaries including ZFS for quite a while.
OpenZFS not being in the Linux kernel has the social effect that the kernel devs don’t maintain compatibility with it, which has resulted in occasional breakages, which puts a maintenance burden on distributors (which is maybe why Canonical are dropping/mothballing their ZFS interface code?).
Canonical, a small UK company, is not the same scale of legal target as big US companies like IBM, Google, Microsoft and Amazon. Some of them won’t even ship GPLv3 code let alone mixed GPLv2 & CDDL. I am not a lawyer, but I’ve been close enough to corporate lawyers IP discussions to expect that they prioritize covering their asses from huge law suits over enabling a few long tail filesystem features (from their perspective at least)…
Their value proposition is btrfs or ZFS but better, and especially faster. They think they can achieve it by using a simpler architecture inspired by databases.
The proposed value for Linux users seems to be “btrfs but better”.
That’s my rough understanding as well. Though I don’t think it’s supposed to be a direct clone of everything btrfs does — more that there’s been lots of promising filesystem ideas since ext3/4, and btrfs and bcachefs are drawing from that same pool.
I’m not 100% up to speed on why btrfs failed to take off. Is it just that it had issues with RAID not working/being unsafe?
As a relatively casual observer, my understanding is that:
Even in server and workstation applications, most usecases don’t need the advanced features of btrfs or it’s equivalent in ZFS, making the overheads not worthwhile for some users, and in some devices (embedded, phones, etc) it was not feasible as an EXT4 replacement.
COW filesystems tend to impose IO overheads that make them undesirable
Some of btrfs’s flagship features excepting COW can be replicated by existing features available to Linux users; XFS permits differential snapshots incremental backup of filesystems which can approximate snapshotting, and LVM2 allows existing filesystems to get features like block device spanning and online partition resizing.
Red Hat was the big driver of btrfs adoption, but they abruptly decided to abandon support for btrfs (even as an optional OS) in RHEL starting from RHEL 8, and Fedora consequently only uses btrfs on Workstation, not Server. My understanding is they were sponsoring a lot of the btrfs work, too.
For those who truly did need all of the options btrfs provided, ZFS provided more, didn’t have a reputation for corruption and unstability that btrfs had (probably unfairly after a few years), and had a lot more materials and documentation on management and deployment.
Red Hat in the end decided to create Stratis, a userland program which replicates some btrfs functionality as a management layer over XFS and LVM2, though I’m not sure how widely this is adopted relative to btrfs.
didn’t have a reputation for corruption and unstability that btrfs had (probably unfairly after a few years)
In my experience this reputation is not unfair at all when it comes to BTRFS’ advanced features. I’ve been running BTRFS in RAID5/6 from 2016 to beginning of 2023 and have experienced data corruption roughly every year in that period. Can’t complain though, BTRFS tools all warn you about RAID5/6 being completely broken every time you run them.
I don’t get it. Your setup was unsupported and documented as such with warnings. Why do you think it’s fair to say it’s unstable?
It’s like randomly moving some files in the system directories after clicking “yes, I know this will cause issues” and saying the system is possible to corrupt and unstable… You’re really on your own in that situation.
Which is a pretty good demonstration of why one might want to build a new filesystem vs btrfs… IMO RAID5 is a pretty fundamental feature for a modern filesystem, both ZFS and Linux md have had it working fine since forever, and and it’s been well-known to be entirely broken in btrfs for… what, seven years or more?
Coincidentally, the author apparently started work on bcachefs in 2015.
I don’t get it. Your setup was unsupported and documented as such with warnings. Why do you think it’s fair to say it’s unstable?
Because it’s unstable regardless of whether the unstableness is documented or not. Why would it be unfair to say something that is factually true? (This is a genuine question, I am wondering if I am lacking a subtlety in my understanding of the English language.)
If you’re doing something explicitly unsupported, then it shouldn’t affect how the system is described. It’s like spraying your computer with water or plugging it into 300V and saying it’s unstable because it crashes - the user guide did tell you not to do that. My phone has an option to enable some experimental GPU features which actually crash when starting games - I can’t say my phone is unstable, just because I explicitly enabled that myself. It’s a option we’re not preventing from taking, but doing those things only says something about us, not about the system.
I can’t say my phone is unstable, just because I explicitly enabled that myself.
You can’t say the phone is unstable, but you can say it’s unstable when you enable experimental GPU features, the same way I’m not saying BTRFS is unstable, I’m saying BTRFS is unstable when using is advanced features. I really don’t understand where our disconnect comes from.
Because it’s not unstable when using advanced features. Advanced features like compression, snapshots, many raid levels works just fine. It’s unstable when using experimental unstable features. And pointing that out is not meaningful, because it’s the same for an experimental unstable feature of any project by definition.
This article feels incredibly dissonant for me. On one hand taking cherry-picked and non-equivalent examples (such as comparing a complex message enum in Rust and days of the week in Go) makes it feel like it could be humorous, which seems to be what the author is going for given their statements about this being “for fun”, but on the other hand, the framing of these points as legitimate frustrations is odd.
I don’t think it’d make sense to correct the points and examples provided, but if these are attempts at being even slightly genuine then I think the author might benefit from spending some time understanding how to write Rust effectively instead of treating it as Go but with more things you have to do before the compiler will run your code.
While the other replies cover the standard approach (treating it as whitespace), that’s not always what you want, especially if you’re building a “blessed” parser for a language since a lot of use cases want to understand your comments (for example formatters which need to preserve comments and LSPs which might use documentation comments).
In my parser, every lexeme stores its position in the input text. (This can be done as a separate array.) Then a comment can just be a single COMMENT token; to get the content of the comment (if you need it for some reason, like doc generation), just grab the position and look up the original text.
(Though most parse rules treat it as equivalent to whitespace, and ignore it.)
Recognize them as a kind of whitespace, and let them get handled by the rule (which might be explicit or implicit) that ignores whitespace between significant tokens.
You treat it as whitespace; for example given functions like this:
static void eatspaces() { // consume whitespace and comments from input stream
for (;; nextchr()) {
int c = peekchr();
if (c == '#') { // comment
while ((c = peekchr()) != '\n' && c != EOF) nextchr() ;
continue;
} // you could match multiline comments, even nested ones, too
if (!aisspace(peekchr())) {
break;
}
}
}
static bool match_nospc(int c) {
eatspaces();
return match(c);
}
static void expect_nospc(int c) {
eatspaces();
expect(c);
}
and the parser:
static Expr primary() {
eatspaces(cm);
int c = nextchr();
if (aisdigit(c)) { // number literal
...
} else if (c == '(')) { // '(' EXPR ')'
Expr ex = expression();
expect_nospc(')');
return ex;
} else ...
}
static Expr unary() {
if (match_nospc('-')) { // '-' UNARY
return makeunaryexpr('-', unary());
} else if ... {
...
} else {
return primary();
}
}
I wouldn’t recommend doing things this way (unified lexer and parser), it can get pretty messy.
Sad to see this only available for enterprise orgs. While a lot of non enterprise orgs probably don’t need something like this, it’s still useful at smaller scales but not a whole $17/user/month more useful.
This is an interesting take on solving the egg and chicken problem of getting a python development environment installed. I have some concerns, like reinventing some substantial part of pip, and doing so on not-python. Is that a good idea? But Python was always thought-out so it could talk easy with faster languages, so, maybe it’s not a big deal.
As with a lot of stuff in this space, it will come down to what gets adoption. But from a cursory glance, I think I wouldn’t be made if this did.
An interesting avenue here is that the Rust ecosystem tends to end up with some high quality crates that can be easily shared across different tools that are adjacent (e.g. how ruff uses the RustPython parser, or shared libraries coming out of the git-branchless/jujutsu/sapling work). The posy README already mentions the pubgrub crate for dependency resolution for example.
In general though, I’m pretty excited about this. The current setup of package/tooling/etc. managers for Python works but it always feels like we’re taking a slight step back when coming from a Rust world. In particular, the idea of being able to do posy run ... and having it use the correct python version without needing an extra tool installed such as pyenv and ensuring all your dependencies are installed and there’s no interference with the global env (something which to my understanding PEP-582 doesn’t address) could be a significant developer experience boost.
Yeah, I love pyenv, best tool available for managing multiple python versions. But I don’t even have the words in English to describe how I feel about how it actually works, so I have to resort to my native language and call it a “gambiarra do caralho”.
The starlark library it uses has some nice extensions like type annotations, an LSP, linter, etc. The trend of Meta’s open source Rust projects spawning good libraries continues (see also sapling which has components used by git-branchless).
I haven’t managed to dig into their incremental computation engine DICE too much yet but my understanding is that it’s inspired by work on Shake, Salsa, Skip, and Adapton
From the start a large part of the rule footprint seems to be written in starlark as opposed to using built in rules. Bazel has been going in this direction and Please did this from the get go but it’s great for rule authors to be able to inspect the native rules for inspiration and learning.
It seems it will be taking a Gazelle based approach to language ecosystem integration (see reindeer for Cargo integration) as opposed to a more dependency inference based approach like Pants. Though there was no reason to suspect this approach would be considered, there are some definite ergonomic benefits.
It seems it will be taking a Gazelle based approach to language ecosystem integration (see reindeer for Cargo integration) as opposed to a more dependency inference based approach like Pants. Though there was no reason to suspect this approach would be considered, there are some definite ergonomic benefits.
I’m curious what you think the ergonomic benefits are for this approach? From my experience with Gazelle the upside is Go integration with other facets of the Bazel ecosystem, but the ergonomics are worse than directly using the Go toolchain.
I was mainly referring to Pants’ dependency inference when referencing ergonomics. This can be a lot nicer than having to maintain dependency references between files.
That being said, the biggest ergonomic benefits of something like Gazelle over the native toolchain is speed. The ability to run a single command to compile, lint, and test everything in a loop but only what changed is great. Go is pretty good in this space already, but some tools can be a bit bad with detecting incremental changes and only running new things.
I think it’s a nice format for certain setups and I quite like the simplicity of it (even if just from a theoretical compactness perspective).
In terms of the comparison to S-expressions on the comparison page I think a more apt compact version would be similar to say dune’s build files which makes the comparison look more like so:
(first-name "John")
(last-name "Smith")
Still not as compact as Jevko but maps quite well to how some OCaml libraries do S-expressions (where it’s used more as a config or interchange format in contrast to the lisp ecosystem).
Another syntax in this space that’s been posted recently is kdl. It’s more focused one ease of use than compactness but similar to the formats being compared.
What I like about syntaxes like this is that a nice one can let you express things declaratively that give the user a semblance of dynamic control without actually breaching the config language barrier.
For example, without needing a full language, you can add the feel of scripting to a build tool with the following KDL:
Yes, your comparison would be more apt, but much less in favor of S-expressions in terms of compactness. I tried to use the most compact representation possible, since that was the focus of the comparison. Even in this case Jevko is slightly more compact.
KDL looks nice, but it’s higher-level and more complex than Jevko.
You could build a simple language like KDL on top of Jevko.
And indeed, you could add features like you mention to it. Your example translated to that hypothetical language would look something like this:
for [file] in [**/*.rs] [
sh [rustfmt $file]
]
Although this would take a bit of work to implement properly.
One of the foundations I set up when starting a new user facing project is requiring all errors to include a message and a user facing message. This improves the user experience a lot when a random error from the stack bubbles up.
I like the article for an additional perspective on how to structure those messages.
The tricky part about errors is that they need to be different things to different consumers of them. In any given system, we have at least 3 consumer roles—the application, the end user, & the operator.
So for most systems you want to set up some sort of mechanism where errors or exceptions or whatever get handled by the system where possible, logged with sufficient details to allow debugging in other cases, and associated with nice user messages when they can’t be resolved automatically.
They denote that the function has side effects. I actually think it’s quite interesting as an approach to make it clear at a glance if a function is pure!
Symbol tags are terrific for typescript. I’ve used it extensively in the past especially combined with ts-pattern to getgood pattern matching (generally using a field called t on an interface/class).
Worth noting that newtype-ts is a semi-popular library for this in the fp-ts adjacent ecosystem.
I recently added some thoughts about type checking to the issue tracker which is a part of this project that I am fairly excited about. In general the team seems fairly smart and welcoming so I can see this being a great project to follow.
One item to note is that languages with managed runtimes will implement tricks like string interning to which can often make them a lot faster when dealing with lots of small strings like you have in lesers and parsers. A trick some Rust setups use is to use the smallstr library which you can use to store strings that are below a certain size inline and drastically reduce heap allocations. I’d be interested to see how the benchmarks go with this approach.
Honestly, I’m surprised that Rust’s String doesn’t have any form of small string optimisation. It has been standard in C++ for ages, and it seems like such an obvious optimisation.
One big difference is the lack of a null byte. In C++, an empty string needs to store the null byte somewhere. You either allocate for every empty string or add a bunch of branches to special case a static null byte. Rust empty strings are just empty since they aren’t compatible with C strings.
I find this article interesting mainly because it completely sidestepped the idea of making computing radically more efficient by reducing the amount of computing happening, using primarily lower level and more efficient systems and so on. It’s almost entirely about the act of engineering as opposed to the engineering itself. The closest it seems to get is touching on bandwidth costs but that’s entirely a user experience thing.
I think there’s a lot of room between this article and scavenge computing for discussing how to make things more efficient.
The problem you run into with Ansible (as an example of a stateless solution) is that removing resources from your config doesn’t ensure they’re removed from the cloud without some care. So say I create a VPC in the YAML, run Ansible and it gets built, then I remove it from my YAML and run Ansible again, the VPC will continue to exist indefinitely without adding some other steps.
By contrast, Terraform with state typically ensures that the resources will be cleaned up once removed from the HCL.
In theory you’ll end up with much more cruft over time the stateless way. Whether or not that is more painful than working with terraform state is a compelling question. I think it depends on the team size and apply process.
This is exactly correct. When Terraform was very early on, I think 0.3 or some such? Well before data resources – I initially did an on-prem to AWS migration solely using Terraform.
Unfortunately, it wasn’t quite up to snuff so after a few months I rewrote it all in Ansible, which ended up being far, far more verbose and had all the problems you listed. From an operator pov it ‘felt good,’ though.
Had I to do it again, I would likely use Ansible to ‘bootstrap’ all the Terraform stuff (s3 buckets and iam users/roles/keys) and do the rest with TF. Shooting for 100% Terraform (or really 100% only one tool) is usually the wrong path.
At a previous company we had a bootstrap terraform repo that would setup the basics for the rest of the automation, like backend buckets and some roles. The bootstrap terraform used local state, committed to the repository. It worked well enough.
My approach is generally to use a bootstrap Terraform stack which creates the buckets and so on that Terraform needs, then change the bootstrap stack to use the bucket it created as its state backend. Having some backups is useful (as with all statefiles) but it’s generally and easy and safe approach.
That’s thought provoking. I wonder if it would be reasonable to run Ansible (or some other stateless tool) in a mode where it went looking for things that didn’t exist and removed them. The flaw there would be that no Ansible config sets up an entire system from first principles, but assumes there are some tools already in place.
Maybe git or other source control could be used to store take on the state burden to detect removal.
The downside of that idea is that it is extremely common to have multiple Terraform workspaces managing resources at the same provider. If you did “destroy all resources that aren’t in your configuration” you’d end up removing, say, all the QA infrastructure every time someone did an update to production if the two are managed separately.
terraform et al are great in theory. it’s in practice where they often fall apart. it’s one of those domains where sanity must be enforced though disciplined conventions, which is hard.
If such culture existed, then ansible, terraform et al would probably never spring to existence.
This usage is very much motivated by a mindset of throwing a flashy tool at a problem, rather than understanding the fundamental problem and how to control it.
For example, using yet another AWS service out of their line up of hundreds, is a choice that is rarely questioned. Then that device has its own challenges… Ok, AWS offers yet another service to ‘solve’ them, and so on.
There is no time for discipline in this reality… Just keep adding stuff and hiring engineers to cope with the system untill the next company mass lay off.
This is also a thing in Puppet (another stateless system). The answer is that you have to put the absence of the resource in the config (in Puppet terms, ensure => absent). Then when the config has been applied everywhere it needs to be, you can delete that “tombstone” resource. For some resources there is also a purge attribute that means to delete all unmanaged resources found in a scope (like a directory).
I find the interpretation of trust in a language context to be a really interesting division. For example, looking at Hare’s first two design principles:
Trust the programmer.
Provide tools the programmer may use when they don’t trust themselves.
Are generics not implemented because we don’t trust the programmer not to make complex code? I could use the above principles to justify Rust and Haskell levels of compiler logic too.
I’m not saying Hare’s interpretation is wrong and I quite like what it’s trying to do, I just find that those two principles can have wildly varying interpretations.
It feels like going a bit against the grain here, but I agree with the article. It took until the very end of the thread to even get to some technical reasons for the package not being merged, everything in the lead up felt entirely like emotional reactions.
The thing about ends and means is that the means end up influencing the ends greatly. If you don’t want this MR to go ahead then the approach used might seem like a success, but there were many other approaches, a lot which would have been friendlier and led to a better outcome in terms of community and mutual respect between contributors and maintainers.
One reason is most likely that it’s unsound in many cases. The following snippet is the standard Typescript version of this unsoundness:
const dict: { x?: string } = { x: "here" };
const removeX = () => { delete dict.x; };
if (dict.x) {
removeX();
dict.x[0] // type checks even though x is now undefined
}
This is the reason why using the Map type in typescript doesn’t have flow typing.
This problem gets even worse when you factor in concurrency (especially in the face of async/await).
From the description alone, it’s not clear to me how early cutoff works in practice, when mtimes instead of content hashes are used: let’s say the hash for the compilation of some .o file changed, so we re-run the action, and get the same .o file! But now the .o file has a newer mtime, so it reruns also all the steps downstream and so on.
My understanding from the post and looking at the linked restat command in the manual, is that Ninja depends on the underlying compiler not modifying the output file when it won’t change, otherwise Ninja will only be able to see new mtimes. I might be incorrect though.
There isn’t really a flag option for “Author doesn’t want this posted” but given the most recent article on his blog I’d say it’d be a good faith measure to take this link down (and maybe ban the domain until it’s ready for release?).
Really, IMO that’s pointless anyway. Security by obscurity is silly. Just say “this is not even pre-release status and everything will break”, and don’t bother talking to anyone else about it unless you’re ready.
It’s not about security, but about attention and being able to control where one’s own efforts go. At least my read of the linked blog post suggests so.
That is just a way to generate engagement. People here are technical enough to understand what “ not finished” means. If you want to keep a secret, don’t tell anyone. Drew’s other domain is banned already.
Having written some complex dbt projects (and if PRQL officially supported macros it could be a good drop in replacement), I can definitely see the value in something like this. The first thing that jumped out to me that it gets right is to start with the table and work down. This is an enormous readability boost in large projects and leads to great intellisense. I saw recently this with import statements in Bagel recently and I can just say it’s a major improvement over how say Typescript does things.
One thing I’d love to see a system like this explore is a gradual type system ala Typescript as there’s no reason we can’t start to detect basic type interactions like only being able to sum a number. In addition to the .sql output, it could also provide a .d.sql file which contains a type specification of the output table. Source tables can be imported directly from the database or written by hand (also provides a convenient way to provide types for native functions/UDFs instead of having to natively support every dialect’s details in the parser). Even just having this do very basic “fall back to any if I see something unknown” inference would be a big step up over SQL.
I wish projects like this would clearly say why they exist and why anyone might care to use/contribute/follow them.
My best guess from a quick search is that the authors either made it just for fun or they made it because they wanted ZFS features but don’t like the license conflict or the btrfs code.
The proposed value for Linux users seems to be “btrfs but better”.
I think https://bcachefs.org/ is ok at summarising the main points.
If you’re interested in the origin, bcache itself is a stable, transparent way to split hot/cold data between multiple fast/slow drives through caching. The author at some point said effectively - you know, that’s halfway to a decent filesystem with good features, so let’s make it happen.
I did check their site. It told me what but not why.
The patreon has a lot more about the motivation / why part and an overview of the current options: https://www.patreon.com/bcachefs/about
Note that the “Status” section is woefully out-of-date. Everything on that list is complete and stable, except for erasure coding (which is not yet stable). I don’t know about the status of DEX/PMEM.
Thanks, that’s much more helpful.
Honestly, after fairly extensive experience with using Btrfs during 4 years of running openSUSE as my daily driver, the headline on the website tells you what you need to know:
“The COW filesystem for Linux that won’t eat your data”.
That’s it. That’s the story.
Btrfs is the most featureful FS in the Linux kernel but it is not reliable. I know lots of people are in deep denial about this, but it is not.
From the Btrfs docs:
From the SUSE docs:
You want a deep examination? Try the Ars one.
Btrfs is not trustworthy. If you fill the volume, it will corrupt. When it is corrupted, it cannot be fixed and you will lose the contents.
OpenSUSE uses a tool called “Snapper” to automatically take snapshots before software installation. This means progressively filling the root filesystem. Unless you manually manage this and manually prune them, it will fill up, and when it fills up, it is game over.
I have been using Linux since kernel 1.0 or so in 1995 (I suspect well over one hundred distros now, if you count major versions) and in the 20th century it was routine for Linux boxes to self-destruct. If the power went at a bad moment, boom, dead.
21st century Linux is different. It is solid now.
I used openSUSE for 4 years, and in that time, I had to format and reinstall my OS about half a dozen times or more, because Btrfs self-destructed. This was not a one-off experience.
Fedora doesn’t have any snapshotting functionality, and so it’s not using Btrfs for the thing that makes Btrfs special. It’s like a moped enjoying perfect engine reliability because you pedal everywhere and never start the engine.
(Why does it use it then? I think this is why: Fedora leans heavily on Flatpak and is increasing that. Flatpak is extremely inefficient and involves a lot of file duplication. Btrfs can compress that. That means it doesn’t look so bad.)
Btrfs self-corrupts on full volumes and it does not have a working repair tool.
That is not acceptable. Therefore Btrfs is not acceptable, and that’s without considering the issues around its RAID handling and so on.
Yeah. BTRFS is kind of shit. I was recently considering adding more storage to my system and found out that I could select between two allocation policies if my disks were mismatched (or if I added one without balancing). I can use the single profile and it will only write to the bigger disk until it is full (BTRFS’ allocation policy is most free space) or I can pick RAID0 and it will stripe across all disks until eventually it is only writing to a single disks. Both cases hotspotting.
bcachefs on the other hand has a simple policy that uses disks evenly and causes them to self-balance over time https://bcachefs-docs.readthedocs.io/en/latest/feat-multipledevices.html
I don’t get the complaint about
--repair. This is the same situation as fsck in all filesystems where it exists. If you have a corrupted filesystem, the best a tool can do is make clever guesses about the way to recover. Hopefully the guesses will be right, but they may be wrong. They’re being explicit about it.No, it’s not. Have you tried it?
Like I said, I have fairly extensive, repeated, personal experience of this.
Fsck is simple and it usually works. I have used it on ext2, ext3, ext4, hfs+, and probably on reiserfs, xfs, and others.
Usually it fixes some damage, maybe small, and afterwards your filesystem works again. It’s as safe or safer than
CHKDSKon the DOS/Windows family. It’s part of my routine maintenance.I have never ever seen Btrfs repair successfully fix any form of damage. None at all, not even once.
What I have seen is a healthy working volume, which I only checked out of curiosity, destroyed by it.
Don’t be misled. What the warning means is “DO NOT RUN THIS”. It will destroy the volume.
There is no way to repair a damaged Btrfs volume, even if the damage is trivial.
I’ve installed openSUSE with Btrfs a few years ago. It worked kind of well, until I’ve tried Android development; the VM was unusable (stuttering, low fps, low performance) if the VM image was stored on a Btrfs partition. When I moved the VM to an ext4 partition, everything was smooth again. I’ve tried to search for the issue on various forums, but couldn’t find any fix for this, so I’ve just resigned from openSUSE and Btrfs altogether, and didn’t look back on it ever again. I haven’t experienced any data losses but from what you write, this was just a matter of time. So it’s good I’ve resigned.
VMs are usable on btrfs, you just have to make sure the file or directory has copy-on-write disabled (
chattr +C).That’s interesting. I am not a developer but I run lots of VMs. I never had a problem with that.
It is 100% possible to run openSUSE on ext4, and it works perfectly. I have run both Leap and Tumbleweed like this.
Leap is very stable and a good config is
/on ext4 and/homeon XFS. That is pretty reliable and performant in my experience.With T’weed you are at the mercy of a bad update but it’s still doable.
One of the benefits over btrfs today is that bcachefs supports native encryption like ZFS does, meaning you don’t need to juggle LUKS + btrfs.
I think for the next while at least btrfs or ZFS will still be preferred but I suspect that bcachefs will be good enough that the (admittedly minor) effort of ensuring OpenZFS on Linux is working such dealing with lack of packages or kernel upgrades breaking things (hello ArchLinux dkms) will make it a success since it’s natively integrated into the kernel.
A big adopting point would be if container tooling such as containerd start to natively support bcachefs the way btrfs and ZFS are.
I think bcachefs is significantly better than ZFS or BTRFS due to flexibility. You can pick storage configuration per-directory, chance storage configuration at any time, stripe sizes change automatically as you add disks and mismatched disks are not a problem.
Overal bcachefs feels like you just add your disks, add your files and it more or less does the best possible thing given your placement constraints. ZFS placement is quite rigid and specified upfront and BTRFS placement is way too stupidly simple.
I wrote a bit more about this in a blag post a while ago: https://kevincox.ca/2023/06/10/bcachefs-attempt/#bcachefs and a bit more words about the features https://kevincox.ca/2023/06/10/bcachefs-attempt/#summary
Any cross platform support? Native encryption? Zfs send/receive type functionality? Encryption with discard support for SSDs?
Those are some of the major zfs features for me.
Ed: I see in a sibling comment that bcachefs support native encryption.
The only two platforms known to actually support zfs are FreeBSD and Linux. No other filesystem than FAT has a good chance at being writable from every major OS
ZFS is also the native filesystem for Illumos and other descendants of Solaris.
Well, there’s opensolaris of course. NetBSD?
Looks like MacOs is limping along, and windows label itself as “beta”.
https://openzfsonosx.org/forum/viewtopic.php?f=20&t=3743
https://github.com/openzfsonwindows/openzfs/releases
Can you explain the utility of ZFS send/receive? Specifically, what it offers that other tools do not.
I can know without further confirmation that the files at the other end are exactly what I was sending, and the transit was minimal. I also can then roll back on the remote server to before that send happened, in case I want to check something that was deleted during that sync.
I’m not sure there are “other tools”. Traditionally filesystems have dump/restore for backup/restore, there’s rsync for incremental sync and restic/borg/duplicity support encryption and authentication.
Zfs send/receive works zfs snapshots and native encryption to form a rich, yet simple, solution for sync and backup. It works well ssh for network transport - or eg USB media for local backup. It supports remote mounting (mount backup on backup host) if wanted.
Like most things zfs, it’s the combination of features that creates a powerful emergent whole.
From your link:
That isn’t currently the case anymore, you can add a third disk to a mirror setup and it’ll resilver just fine to have triple copies of everything.
Though the RAID still applies (2.2 adds DRAID, which does let you expand IIRC, at massive complexity cost and has issues still, RAID Expansion is still in the pipes).
Did you make a typo? The requirement is 3 disks and 2x redundancy. I’ll admit that I didn’t look into the configuration too closely but this seemed difficult to do with ZFS. Especially if you were starting from 2 disks and wanted to live migrate to 3.
My understanding is that at the end of the day you configure your performance/redundancy into a virtual block device. You then put a more-or-less single-disk filesystem on top of that. (It is smarter than your average single-disk filesystem because it is aware of the topology but IIUC the behaviour is basically the same). You can grow by adding more disks to the virtual block device but you don’t really have much option to change the existing devices in the pool. (As you said there are some limited options such as 2 disks replicated can become N disks replicated).
bcachefs (and BTRFS for this case) are fairly flexible about this. They see that you want 2 copies and they pick two disks and put the data there. Adding more disks gives them more options to pick from. With bcachefs you can get even more flexible and say “everything in this directory shall have 4 copies” and it will replicate that data more times (well it won’t proactively replicate existing data IIUC, it will replicate new writes or on a balance operation) even if that filesystem previously had no configuration with 4x redundancy.
No typo, all data can be saved thrice.
ZFS’ internal structures are messy, but essentially there is two types of vdev’s; RAID and Mirror.
A non-mirror vdev (ie, just a plain disk, no redundancy), is simply a special case of a RAID1 with no partners (though internally it simply presents the device directly as pool member).
Mirror vdevs can be brought up and down in redundancy by adding or removing disks. If your pool is only mirror disks, you can even split it in half and have two pools with identical data. You can also remove and add mirror vdevs at your pleasure. Even convert single-disks into a mirror configuration.
RAID vdevs are much more strict. They cannot be expanded once created, they can’t be split and cannot be removed from a pool. This is due to the internal block pointers being in a special format to handle the RAID stripes.
The RAID Expansion work doesn’t even change that, it instead allows you to rewrite a RAID vdev in a way where it has more members than before. The old data will still only exist at the previous redundancy level (ie, same parity-data ratio) and only new data will benefit from the better parity to data ratio.
A mistake in understanding then?
The objective stated was “a filesystem that has 3 disks, and all data has 2 replicas”. Number of data replicas < number of disks. This saves 50% of your storage space if your failure budget only requires 2 replicas.
Why would anyone be worried about the ZFS license? Oracle’s never sued anyone over IP they acquired from Sun have they?
You would have to ask them, but a bunch of people do complain about it.
Now that this comment the thread has helped me learn more about bcachefs, I don’t think the licensing was a huge deal. The bigger motivations are: bcachefs devs think they can do much better than ZFS or BTRFS; and also ZFS will never be first class on Linux (because Torvalds don’t want it in the kernel, etc)
I think that was a facetious comment; there’s certainly been at least one very high profile instance of Oracle doing exactly that.
Makes sense, thanks. I had forgotten about that case (generally not very interested in licensing disputes).
Eh, Torvalds doesn’t ship binaries. It’s companies like RedHat/IBM, Amazon, Google, Samsung and Microsoft that ship kernel binaries and wouldn’t want legal exposure.
Canonical paid for & published a legal opinion and shipped binaries including ZFS for quite a while.
OpenZFS not being in the Linux kernel has the social effect that the kernel devs don’t maintain compatibility with it, which has resulted in occasional breakages, which puts a maintenance burden on distributors (which is maybe why Canonical are dropping/mothballing their ZFS interface code?).
Canonical, a small UK company, is not the same scale of legal target as big US companies like IBM, Google, Microsoft and Amazon. Some of them won’t even ship GPLv3 code let alone mixed GPLv2 & CDDL. I am not a lawyer, but I’ve been close enough to corporate lawyers IP discussions to expect that they prioritize covering their asses from huge law suits over enabling a few long tail filesystem features (from their perspective at least)…
Some features bcachefs might have that ZFS lacks are defragmentation and “resizing flexibility” (links are to previous Lobsters comments).
Thanks, those links led me to the bcachefs manual, and that answered my questions.
https://bcachefs.org/bcachefs-principles-of-operation.pdf
Their value proposition is btrfs or ZFS but better, and especially faster. They think they can achieve it by using a simpler architecture inspired by databases.
That’s my rough understanding as well. Though I don’t think it’s supposed to be a direct clone of everything btrfs does — more that there’s been lots of promising filesystem ideas since ext3/4, and btrfs and bcachefs are drawing from that same pool.
I’m not 100% up to speed on why btrfs failed to take off. Is it just that it had issues with RAID not working/being unsafe?
As a relatively casual observer, my understanding is that:
Even in server and workstation applications, most usecases don’t need the advanced features of btrfs or it’s equivalent in ZFS, making the overheads not worthwhile for some users, and in some devices (embedded, phones, etc) it was not feasible as an EXT4 replacement.
COW filesystems tend to impose IO overheads that make them undesirable
Some of btrfs’s flagship features excepting COW can be replicated by existing features available to Linux users; XFS permits
differential snapshotsincremental backup of filesystems which can approximate snapshotting, and LVM2 allows existing filesystems to get features like block device spanning and online partition resizing.Red Hat was the big driver of btrfs adoption, but they abruptly decided to abandon support for btrfs (even as an optional OS) in RHEL starting from RHEL 8, and Fedora consequently only uses btrfs on Workstation, not Server. My understanding is they were sponsoring a lot of the btrfs work, too.
For those who truly did need all of the options btrfs provided, ZFS provided more, didn’t have a reputation for corruption and unstability that btrfs had (probably unfairly after a few years), and had a lot more materials and documentation on management and deployment.
Red Hat in the end decided to create Stratis, a userland program which replicates some btrfs functionality as a management layer over XFS and LVM2, though I’m not sure how widely this is adopted relative to btrfs.
In my experience this reputation is not unfair at all when it comes to BTRFS’ advanced features. I’ve been running BTRFS in RAID5/6 from 2016 to beginning of 2023 and have experienced data corruption roughly every year in that period. Can’t complain though, BTRFS tools all warn you about RAID5/6 being completely broken every time you run them.
I don’t get it. Your setup was unsupported and documented as such with warnings. Why do you think it’s fair to say it’s unstable?
It’s like randomly moving some files in the system directories after clicking “yes, I know this will cause issues” and saying the system is possible to corrupt and unstable… You’re really on your own in that situation.
Which is a pretty good demonstration of why one might want to build a new filesystem vs btrfs… IMO RAID5 is a pretty fundamental feature for a modern filesystem, both ZFS and Linux md have had it working fine since forever, and and it’s been well-known to be entirely broken in btrfs for… what, seven years or more?
Coincidentally, the author apparently started work on bcachefs in 2015.
Because it’s unstable regardless of whether the unstableness is documented or not. Why would it be unfair to say something that is factually true? (This is a genuine question, I am wondering if I am lacking a subtlety in my understanding of the English language.)
If you’re doing something explicitly unsupported, then it shouldn’t affect how the system is described. It’s like spraying your computer with water or plugging it into 300V and saying it’s unstable because it crashes - the user guide did tell you not to do that. My phone has an option to enable some experimental GPU features which actually crash when starting games - I can’t say my phone is unstable, just because I explicitly enabled that myself. It’s a option we’re not preventing from taking, but doing those things only says something about us, not about the system.
You can’t say the phone is unstable, but you can say it’s unstable when you enable experimental GPU features, the same way I’m not saying BTRFS is unstable, I’m saying BTRFS is unstable when using is advanced features. I really don’t understand where our disconnect comes from.
Because it’s not unstable when using advanced features. Advanced features like compression, snapshots, many raid levels works just fine. It’s unstable when using experimental unstable features. And pointing that out is not meaningful, because it’s the same for an experimental unstable feature of any project by definition.
I’ve had issues with snapshots and other raid levels too unfortunately (although less frequently than with RAID5/6).
Why don’t you run a less buggy RAID setup?
I was just having fun playing with BTRFS, I only used the raid for data I did not care about losing :)
Heh, fair enough. Hope you are doing well!
And a decent name.
bcachefssounds like something Facebook would run on their CDN nodes.Side note: sweet buddha, phoronix is full of ad cancer.
In fact, AIUI, FB is a big Btrfs user.
But then its data is highly volatile anyway.
This article feels incredibly dissonant for me. On one hand taking cherry-picked and non-equivalent examples (such as comparing a complex message enum in Rust and days of the week in Go) makes it feel like it could be humorous, which seems to be what the author is going for given their statements about this being “for fun”, but on the other hand, the framing of these points as legitimate frustrations is odd.
I don’t think it’d make sense to correct the points and examples provided, but if these are attempts at being even slightly genuine then I think the author might benefit from spending some time understanding how to write Rust effectively instead of treating it as Go but with more things you have to do before the compiler will run your code.
How do people deal with comments when combining lexing and parsing? An upfront pass to strip comments?
While the other replies cover the standard approach (treating it as whitespace), that’s not always what you want, especially if you’re building a “blessed” parser for a language since a lot of use cases want to understand your comments (for example formatters which need to preserve comments and LSPs which might use documentation comments).
You can check what rust-analyzer and black do for references.
In my parser, every lexeme stores its position in the input text. (This can be done as a separate array.) Then a comment can just be a single COMMENT token; to get the content of the comment (if you need it for some reason, like doc generation), just grab the position and look up the original text.
(Though most parse rules treat it as equivalent to whitespace, and ignore it.)
Recognize them as a kind of whitespace, and let them get handled by the rule (which might be explicit or implicit) that ignores whitespace between significant tokens.
You treat it as whitespace; for example given functions like this:
and the parser:
I wouldn’t recommend doing things this way (unified lexer and parser), it can get pretty messy.
Sad to see this only available for enterprise orgs. While a lot of non enterprise orgs probably don’t need something like this, it’s still useful at smaller scales but not a whole $17/user/month more useful.
This is an interesting take on solving the egg and chicken problem of getting a python development environment installed. I have some concerns, like reinventing some substantial part of pip, and doing so on not-python. Is that a good idea? But Python was always thought-out so it could talk easy with faster languages, so, maybe it’s not a big deal.
As with a lot of stuff in this space, it will come down to what gets adoption. But from a cursory glance, I think I wouldn’t be made if this did.
An interesting avenue here is that the Rust ecosystem tends to end up with some high quality crates that can be easily shared across different tools that are adjacent (e.g. how ruff uses the RustPython parser, or shared libraries coming out of the git-branchless/jujutsu/sapling work). The posy README already mentions the pubgrub crate for dependency resolution for example.
In general though, I’m pretty excited about this. The current setup of package/tooling/etc. managers for Python works but it always feels like we’re taking a slight step back when coming from a Rust world. In particular, the idea of being able to do
posy run ...and having it use the correct python version without needing an extra tool installed such as pyenv and ensuring all your dependencies are installed and there’s no interference with the global env (something which to my understanding PEP-582 doesn’t address) could be a significant developer experience boost.Yeah, I love pyenv, best tool available for managing multiple python versions. But I don’t even have the words in English to describe how I feel about how it actually works, so I have to resort to my native language and call it a “gambiarra do caralho”.
Some interesting notes on Buck2:
I’m curious what you think the ergonomic benefits are for this approach? From my experience with Gazelle the upside is Go integration with other facets of the Bazel ecosystem, but the ergonomics are worse than directly using the Go toolchain.
I was mainly referring to Pants’ dependency inference when referencing ergonomics. This can be a lot nicer than having to maintain dependency references between files.
That being said, the biggest ergonomic benefits of something like Gazelle over the native toolchain is speed. The ability to run a single command to compile, lint, and test everything in a loop but only what changed is great. Go is pretty good in this space already, but some tools can be a bit bad with detecting incremental changes and only running new things.
Ah right, it wasn’t clear to me what “this” was referring to, but I see you were referring to the opposite of what I interpreted!
Yeah I agree, I love the static analysis approach. I’d love to see a build tool double down on this. I’ve played around a bit with it myself.
I think it’s a nice format for certain setups and I quite like the simplicity of it (even if just from a theoretical compactness perspective).
In terms of the comparison to S-expressions on the comparison page I think a more apt compact version would be similar to say dune’s build files which makes the comparison look more like so:
Still not as compact as Jevko but maps quite well to how some OCaml libraries do S-expressions (where it’s used more as a config or interchange format in contrast to the lisp ecosystem).
Another syntax in this space that’s been posted recently is kdl. It’s more focused one ease of use than compactness but similar to the formats being compared.
What I like about syntaxes like this is that a nice one can let you express things declaratively that give the user a semblance of dynamic control without actually breaching the config language barrier.
For example, without needing a full language, you can add the feel of scripting to a build tool with the following KDL:
Yes, your comparison would be more apt, but much less in favor of S-expressions in terms of compactness. I tried to use the most compact representation possible, since that was the focus of the comparison. Even in this case Jevko is slightly more compact.
KDL looks nice, but it’s higher-level and more complex than Jevko.
You could build a simple language like KDL on top of Jevko.
And indeed, you could add features like you mention to it. Your example translated to that hypothetical language would look something like this:
Although this would take a bit of work to implement properly.
One of the foundations I set up when starting a new user facing project is requiring all errors to include a message and a user facing message. This improves the user experience a lot when a random error from the stack bubbles up.
I like the article for an additional perspective on how to structure those messages.
Yeah, I wrote about how to do that from the perspective of including error messages in Go, but it’s a universal concern. In prior blog post that I linked to, Ben Johnson wrote,
So for most systems you want to set up some sort of mechanism where errors or exceptions or whatever get handled by the system where possible, logged with sufficient details to allow debugging in other cases, and associated with nice user messages when they can’t be resolved automatically.
What’s up with the exclamation marks? For me they interrupt the flow of reading the code.
They denote that the function has side effects. I actually think it’s quite interesting as an approach to make it clear at a glance if a function is pure!
I assume they got the idea from Ruby, but in Ruby the “!” suffix on a mutating function is just a naming convention, not part of the language.
Symbol tags are terrific for typescript. I’ve used it extensively in the past especially combined with ts-pattern to getgood pattern matching (generally using a field called
ton an interface/class).Worth noting that newtype-ts is a semi-popular library for this in the fp-ts adjacent ecosystem.
I recently added some thoughts about type checking to the issue tracker which is a part of this project that I am fairly excited about. In general the team seems fairly smart and welcoming so I can see this being a great project to follow.
One item to note is that languages with managed runtimes will implement tricks like string interning to which can often make them a lot faster when dealing with lots of small strings like you have in lesers and parsers. A trick some Rust setups use is to use the smallstr library which you can use to store strings that are below a certain size inline and drastically reduce heap allocations. I’d be interested to see how the benchmarks go with this approach.
Honestly, I’m surprised that Rust’s String doesn’t have any form of small string optimisation. It has been standard in C++ for ages, and it seems like such an obvious optimisation.
One big difference is the lack of a null byte. In C++, an empty string needs to store the null byte somewhere. You either allocate for every empty string or add a bunch of branches to special case a static null byte. Rust empty strings are just empty since they aren’t compatible with C strings.
I find this article interesting mainly because it completely sidestepped the idea of making computing radically more efficient by reducing the amount of computing happening, using primarily lower level and more efficient systems and so on. It’s almost entirely about the act of engineering as opposed to the engineering itself. The closest it seems to get is touching on bandwidth costs but that’s entirely a user experience thing.
I think there’s a lot of room between this article and scavenge computing for discussing how to make things more efficient.
Interesting, thanks for writing.
The problem you run into with Ansible (as an example of a stateless solution) is that removing resources from your config doesn’t ensure they’re removed from the cloud without some care. So say I create a VPC in the YAML, run Ansible and it gets built, then I remove it from my YAML and run Ansible again, the VPC will continue to exist indefinitely without adding some other steps.
By contrast, Terraform with state typically ensures that the resources will be cleaned up once removed from the HCL.
In theory you’ll end up with much more cruft over time the stateless way. Whether or not that is more painful than working with terraform state is a compelling question. I think it depends on the team size and apply process.
This is exactly correct. When Terraform was very early on, I think 0.3 or some such? Well before data resources – I initially did an on-prem to AWS migration solely using Terraform.
Unfortunately, it wasn’t quite up to snuff so after a few months I rewrote it all in Ansible, which ended up being far, far more verbose and had all the problems you listed. From an operator pov it ‘felt good,’ though.
Had I to do it again, I would likely use Ansible to ‘bootstrap’ all the Terraform stuff (s3 buckets and iam users/roles/keys) and do the rest with TF. Shooting for 100% Terraform (or really 100% only one tool) is usually the wrong path.
At a previous company we had a bootstrap terraform repo that would setup the basics for the rest of the automation, like backend buckets and some roles. The bootstrap terraform used local state, committed to the repository. It worked well enough.
My approach is generally to use a bootstrap Terraform stack which creates the buckets and so on that Terraform needs, then change the bootstrap stack to use the bucket it created as its state backend. Having some backups is useful (as with all statefiles) but it’s generally and easy and safe approach.
That’s thought provoking. I wonder if it would be reasonable to run Ansible (or some other stateless tool) in a mode where it went looking for things that didn’t exist and removed them. The flaw there would be that no Ansible config sets up an entire system from first principles, but assumes there are some tools already in place.
Maybe git or other source control could be used to store take on the state burden to detect removal.
The downside of that idea is that it is extremely common to have multiple Terraform workspaces managing resources at the same provider. If you did “destroy all resources that aren’t in your configuration” you’d end up removing, say, all the QA infrastructure every time someone did an update to production if the two are managed separately.
terraform et al are great in theory. it’s in practice where they often fall apart. it’s one of those domains where sanity must be enforced though disciplined conventions, which is hard.
If such culture existed, then ansible, terraform et al would probably never spring to existence. This usage is very much motivated by a mindset of throwing a flashy tool at a problem, rather than understanding the fundamental problem and how to control it.
For example, using yet another AWS service out of their line up of hundreds, is a choice that is rarely questioned. Then that device has its own challenges… Ok, AWS offers yet another service to ‘solve’ them, and so on. There is no time for discipline in this reality… Just keep adding stuff and hiring engineers to cope with the system untill the next company mass lay off.
This is a classical problem with stateless tools, going as far back as
make.This is also a thing in Puppet (another stateless system). The answer is that you have to put the absence of the resource in the config (in Puppet terms,
ensure => absent). Then when the config has been applied everywhere it needs to be, you can delete that “tombstone” resource. For some resources there is also apurgeattribute that means to delete all unmanaged resources found in a scope (like a directory).I find the interpretation of trust in a language context to be a really interesting division. For example, looking at Hare’s first two design principles:
Are generics not implemented because we don’t trust the programmer not to make complex code? I could use the above principles to justify Rust and Haskell levels of compiler logic too.
I’m not saying Hare’s interpretation is wrong and I quite like what it’s trying to do, I just find that those two principles can have wildly varying interpretations.
I read it like:
It feels like going a bit against the grain here, but I agree with the article. It took until the very end of the thread to even get to some technical reasons for the package not being merged, everything in the lead up felt entirely like emotional reactions.
The thing about ends and means is that the means end up influencing the ends greatly. If you don’t want this MR to go ahead then the approach used might seem like a success, but there were many other approaches, a lot which would have been friendlier and led to a better outcome in terms of community and mutual respect between contributors and maintainers.
One reason is most likely that it’s unsound in many cases. The following snippet is the standard Typescript version of this unsoundness:
This is the reason why using the Map type in typescript doesn’t have flow typing.
This problem gets even worse when you factor in concurrency (especially in the face of async/await).
From the description alone, it’s not clear to me how early cutoff works in practice, when mtimes instead of content hashes are used: let’s say the hash for the compilation of some .o file changed, so we re-run the action, and get the same .o file! But now the .o file has a newer mtime, so it reruns also all the steps downstream and so on.
My understanding from the post and looking at the linked
restatcommand in the manual, is that Ninja depends on the underlying compiler not modifying the output file when it won’t change, otherwise Ninja will only be able to see new mtimes. I might be incorrect though.I guess even if the compiler doesn’t support it, a wrapper that hashes the file twice is cheap compared to running the compiler itself.
There isn’t really a flag option for “Author doesn’t want this posted” but given the most recent article on his blog I’d say it’d be a good faith measure to take this link down (and maybe ban the domain until it’s ready for release?).
I would hope we’re all colleagues here ;-)
Really, IMO that’s pointless anyway. Security by obscurity is silly. Just say “this is not even pre-release status and everything will break”, and don’t bother talking to anyone else about it unless you’re ready.
It’s not about security, but about attention and being able to control where one’s own efforts go. At least my read of the linked blog post suggests so.
They kind of gave that control away when they put everything on a public website, didn’t they?
Anyway now they’ve done it, the site is behind a password.
That is just a way to generate engagement. People here are technical enough to understand what “ not finished” means. If you want to keep a secret, don’t tell anyone. Drew’s other domain is banned already.
Having written some complex dbt projects (and if PRQL officially supported macros it could be a good drop in replacement), I can definitely see the value in something like this. The first thing that jumped out to me that it gets right is to start with the table and work down. This is an enormous readability boost in large projects and leads to great intellisense. I saw recently this with import statements in Bagel recently and I can just say it’s a major improvement over how say Typescript does things.
One thing I’d love to see a system like this explore is a gradual type system ala Typescript as there’s no reason we can’t start to detect basic type interactions like only being able to sum a number. In addition to the
.sqloutput, it could also provide a.d.sqlfile which contains a type specification of the output table. Source tables can be imported directly from the database or written by hand (also provides a convenient way to provide types for native functions/UDFs instead of having to natively support every dialect’s details in the parser). Even just having this do very basic “fall back to any if I see something unknown” inference would be a big step up over SQL.