Btw I really like the graphical style! The way the paragraphs are nestes according to which part of the thought are they about is amazing idea, it feels much easier to follow.
The actual content of the article seems to be just about one sentence long at the very end, the rest is just rehearsal of basics of shared pointer usage. Shame, I wanted to read a deeper analysis about shared ptr design issues.
Shame, I wanted to read a deeper analysis about shared ptr design issues.
The root problem with shared pointer is that it tries to be a generic thing over three quite different use cases:
I have a type that is always ref counted.
I have a type that is sometimes refcounted.
I have a refcounted type and I want to break cycles or provide caches with weak pointers.
The first is better solved with an intrusive reference count in a superclass because otherwise there’s no good way of going from this to a refcounted wrapper. The standard has std::enable_shared_from_this but this doesn’t require the objects to be constructed with make_shared so adds another word to contain a pointer from the object to its shared control block.
The additional state required for weak references adds even more overhead and 99% of the time it’s not used (I have never used std::weak_ptr, though I’ve used std::shared_ptr quite a lot). If you had a separate type for shared pointers that permit weak pointers then you could reduce the overhead in the common case.
There’s also the thing that just because the shared pointer’s refcount manipulation is atomic doesn’t mean that it’s thread safe (assigning to a shared pointer from two threads breaks in exciting ways), so you now have another atomic version.
I ended up writing my own intrusive ref-counted base class for $DAYJOB, for the above reasons. Unfortunately this is the kind of attractive nuisance that seems trivial at first, and it was trivial to get 99% of the way, but there were exciting edge cases that revealed themselves periodically, especially with multithreading, exceptions in constructors/destructors, and so on.
It would be really nice if std:: had taken care of this for me.
(I’m sure Boost has something like this, but my impression of Boost is it’s a gigantic tangle of yarn, and if I pick up one little trinket it’s going to drag a megabyte of other stuff along with it into my binary, plus double my build times.)
(I’m sure Boost has something like this, but my impression of Boost is it’s a gigantic tangle of yarn, and if I pick up one little trinket it’s going to drag a megabyte of other stuff along with it into my binary, plus double my build times.)
intrusive_ref_counter is the Boost solution for this. There’s the BCP command to extract only the required portions of the library, but it’s still going to include a lot of configuration code. I’d also write my own class for this.
One practical issue is that reference cycles can hide other bugs. After you fix the cycle with weak_ptr or some other technique, all those use-after-frees bubbling under suddenly surface. With weak pointers you can avoid a crash by checking if lock() returns a nullptr but good luck rewriting all the logic with that in mind.
Could call it an extreme case of accidental technical debt.
For insta-sharing of the modified videos / their “indexes”, maybe you could put them on IPFS? Sorry if you already mention this somewhere, I only glanced over the readme as I’m in a hurry.
I actually don’t modify the videos, I just download them and build a timetable (that I call an ‘index’) that remembers when was the video loud and when was it silent - the timetable file sizes range from 1K to 20K (I don’t even bother to compress them), most of them being in the ~4K range (it’s the javascript that speeds up the video while playing it). And this extension is not currently used much, so it’s not much of a concern as the sizes are manageable so far.
IPFS is interesting project and I need to finally look into it further. It would be really nice if I managed to run it entirely over IPFS and eliminate the need for a central server. I quite like the idea.
I played with the Go API recently, there was a lot of docs I had to wade through, but the final code was quite nice and short. Just remember to call “pin” and/or “provide”, one of them brought down sharing speed from behind NAT from some horrific 45 minutes to something much more bearable around 2 min. And experimental auto-relay; didn’t minimize to check which ones are the required ones. I know they also have JS lib on the same level of support I think.
This is awesome. And written entirely in bash as well. I really like the way it’s written. I never thought of writing http server in bash, but it makes perfect sense when you think about it. In the end, http protocol was designed for this. I might steal this idea later.
Btw, is it secure to let people create random files in your filesystem? I’m on phone so I can’t properly test it, but isn’t there danger of path traversal?
This certainly is pushing it! It accepts “.”, “..” and flags to ln, but there are no flags that you can use to make ln do nasty things, AFAICT. I tried -i, which indeed seems to send the server into an endless loop waiting for “input”. So a resource exhaustion attack might be possible, if you can hit the max open file descriptor or pid limit.
With -f as subdomain, I’d expect it to delete the target file, but unfortunately the check whether the target is the same as the source comes first…
ln "$yn" "$subdomain" with $subdomain equals -f, you get ln yes -f or something like it. Because of how most programs work, flags don’t necessarily need to come first. To fix this particular issue, it should read ln -- "$yn" "$subdomain".
I think I broke it, with -fbdrs as subdomain; this force removes the target file (which is yes or no) by creating a backup and then linking it to itself. Or something…
Huh this looks like a really convenient setup. My blog (if you can call it so, heh) is generated by ugly shell+awk combo I wrote. Everything is handwritten. It even uses a custom flavour of markdown. Basically I have directory with some 2020-06-06.md files, and a simple shell scripts load the markdown and build everything - the blog pages itself (which is just simple html file), index file (just a list of aforementioned blog pages), and very basic kind of javascript search (user can search via tags, without the need to have any backend).
I don’t need any setup, it’s just literally git clone away form working on any linux box, without the need to configure anything at all. And while the way the scripts are written, new features can be added easily (someone emailed me about lack of RSS, and I got it working from scratch, in awk, in under an hour), they get messy quick. And while I love the simplicity and that getting it work is possible under a minute (I use it for more things, for example for rendering my journal to html), I think like it’s slowly getting out of my control. I’ve thought of either rebuilding it in real programming language or switching to some static page generator. So thanks for this post!
Btw, consider using CSS prefers-color-scheme. It allows user to specify the preferred theme (light/dark), and lets websites adapt to the settings. It’s really convenient, not that hard to set up if you have sane CSS, and it boosts user experience a lot. All of my websites use this, and I am pushing pretty heavily when some site doesn’t use it. It is just great when you setup your OS to dark theme, and running applications as well as running websites instantly change their appearance to match the OS (and daytime).
Those two are unrelated decisions. The RSS code was bitrotting and that is why it was removed. Pocket is a bundled addon.
Yes, they could’ve shipped a bundled addon for RSS as well. But there are a gazillion RSS addons for Firefox that do a much better job than the built-in feature ever did. They are a single click away.
Export the details of your existing live bookmarks to an OPML file on your desktop, which other feed readers (including ones that are webextensions) support importing from.
Replace the live bookmarks with “normal” bookmarks pointing to the URL associated with the live bookmark.
Open a page on support.mozilla.org that explains what has happened and offers you options for how you could continue consuming those feeds.
So, the data was not destroyed, it was saved in multiple formats including standard formats made to be imported in other solutions. Documentation was provided to guide the transition.
“Bitrotting” is never a justification to remove a feature. It worked perfectly fine until they removed it. Now you have to install RSSPreview, so it’s an extra step to get this basic lightweight functionality.
Have you read the link I pasted earlier? The way the live bookmarks were built were not compatible with sync, used ancient firefox stuff that they are trying to get rid of and had no way to set read/unread status of a post which is a basic necessity of a rss reader. They also have telemetry showing few people using it and more people using more complete add-ons. Mozilla is making a huge effort trying to make Firefox more nimble and maintainable, and the feature was removed. I liked that feature too, but when you read what people maintaining that code and making decisions said about it you might understand why it was done.
The feature I miss is the RSS previews. The justification given in the blog post holds no weight:
the feed viewer has its own “special” XML parser, distinct from the main Firefox one, and has not had a significant update in styling or functionality in the last seven years.
It is illustrative that Mozilla (or whoever wrote this post) thinks seven years is too long to wait before a styling update. This attitude is responsible for the recent degradation of the about:config page, among other things…
You understand that in these seven years Firefox went through a major rework? Working towards removing XUL, XCB, and lots of old code. Adding the Rust bits and new components from Servo. Refactoring the whole UI into an HTML5 based UI. Seven years when we’re talking about what happened in the specific interval mentioned is a lot of changes to the browser code.
Also, as I’ve mentioned multiple times, that feature was:
Incomplete.
Not compatible with the current bookmark and sync.
Better served by multiple add-ons.
People complain about bloat all the time in Firefox. Keep saying that X, Y, Z, features should be add-ons. When they remove a feature that the power users were not using (because they were already using better add-ons) and the casual users were not using (because they didn’t know it existed) taking care to provide a migration path, save the data, and offer alternatives, people cry that they shouldn’t have done it.
It is impossible to please you folks. If they ship some new feature, it is bloat. If they remove something that is hard to maintain and broken, then it is a crime.
Also, the feature is one click away in the add-ons portal. Add-ons exist so that browser vendors don’t need to cover every single feature by themselves.
Also, as I’ve mentioned multiple times, that feature was:
Incomplete.
Not compatible with the current bookmark and sync.
Better served by multiple add-ons.
I think you may be referring to Live Bookmarks; I’m referring to RSS previews. I don’t believe these bullets apply to RSS previews.
I also don’t see how Firefox would know how many people used the RSS preview function. If they click on a link to an RSS feed, they would be using it without knowing that they were using it. Then suddenly those links become raw XML and they don’t know that they need the RSSPreview addon to render the page.
It is impossible to please you folks. If they ship some new feature, it is bloat. If they remove something that is hard to maintain and broken, then it is a crime.
Converting an RSS feed to HTML is an extremely simple task that provides functionality that people had come to expect with very little code. New features must meet a higher standard of usefulness and code quality because people aren’t already used to them.
If you stop to check Firefox source code and the changes they’ve been doing to “modernize” it away from XUL and many other older technologies inside the code base, you’ll notice that that source code was not up to date with how bookmarks work, it didn’t contain the RSS features people expected it to have (such as read/unread status), and was incompatible with sync and missing from the mobile browsers. That is why it was removed. Read the article, talk to people before spreading FUD.
The feature didn’t disappeared overnight without annoucing and planning a transition. There are a dozen RSS readers on the add-ons portal, all of which are better than that feature ever was. The feature never worked well, that is why it was axed.
Since when Mozilla focus feature-wise has been monetization? The RSS code was removed, as explained in the article that apparently no one read, because it was not compatible with the new features they added. The code was bitrotting because it was difficult to maintain and the target userbase moved to add-ons or services with more features. The feature was also incomplete from the point of view of those who make the browser and instead of devoting resources to it, when suitable add-ons are present in the ecosystem, they opted to remove it and focus on other tasks.
Not all features are eternal. Features get removed. This was announced, a path of migration was documented, the data was saved in multiple places. You could easily either move to a SaaS, or to an add-on and import all the same feeds from the OPML export.
Read the article, talk to people involved, go to mozilla community/slack/matrix and talk to people, and you’ll realize the feature was removed for many reasons none of which is monetization.
The pocket thing is related to monetization. What I’m talking that is not related to monetization is the Live Bookmarks removal that happened in Firefox 64, which was released in 2018. All I’ve been replying and chatting here is related to that feature, and not pocket. The Live Bookmarks removal happened more than 10 versions ago and was not motivated by monetization.
The quote above was part of the reasoning behind removing RSS support. In other words they wouldn’t have removed RSS support if there weren’t monetized alternatives like Pocket and sponsored content on the home page.
No, they told you the feature was going to be removed with months in advance. Provided docs on how to to take your data out, and links to alternative solutions for the feature. Your data should have been saved into two different backups: one OPML on your Desktop, and the bookmarks should still be there, they’d just be normal bookmarks instead of live bookmarks.
If this didn’t happen, then it is a bug and even though I am really sorry for you, it is the case that the only possible solution is to fill a bug report and try to figure out why the documented process failed so that it doesn’t happen to someone else.
Thanks for sharing! This is exceptionally readable, considering the topic. Author’s page also contains several interesting articles worth looking at. I really like their writing style, it’s easy to digest even for people not in the crypto field.
Hi, author here. I’ve written a debugger for sed in Rust. This was not only to learn rust, but to actually have a solid debugger for sed. I’ve started learning sed recently and decided to start writing various algorithms in it. And as sed doesn’t have numbers and can just filter/transform text, this is a challenge even for something like comparing two numbers. I’ve seen people do amazing things with it.
I’d be glad for any comments regarding code quality or the debugger itself.
For example, it doesn’t use strings-as-buffers as in C; it uses strings-as-values as in Python. (And it doesn’t use STL’s string class.) That’s a huge difference that you’ll see all over the code.
Despite that it’s actually faster than bash and zsh which are written in very low level C! I call bash and zsh “grovelling through backslashes and braces one by one”, and Oil doesn’t do that.
Oil is partly a shell and partly a software engineering experiment in “metaprogramming” – i.e. can you implement code with domain-specific, high level abstractions and still get good performance?
This post is probably 20-40% of the victory lap. I hope to have 80% by the end of the year – i.e. a working shell in pure native code, that’s measurably better than existing shells along several important dimensions.
If the project becomes self-sustaining (which is a big if), I will have time to optimize it in ways that are impossible if it were written by hand in C++, Rust, or Go. The abstractions are “right” – they don’t express irrelevant detail and give you more leeway in implementing them.
Honestly I think Rust (and manually written C++) are at the wrong abstraction level for writing a 10K line recursive descent parser. They would end up at more like 20 - 30K lines due to all the irrelevant details you need to take care of. (note that bash is a 140K line C program.)
However I encourage others to pursue that direction:
Oil isn’t written or Python or C++, or Rust or Go. It’s written in “OPy/mycpp”, ASDL, and an abstract regex dialect (actually “Eggex” if I self host.)
the 26K line osh-lex file is all DFAs generated from regexes.
the 16K line osh_eval file is the parsers and evaluators translated from statically typed Python. This will be more like 25K when the translation process is done.
the *_asdl files are language-independent type definitions expanded to C++ (and the types can be pretty printed)
Some more color: Oil is made of ideas; it’s not a big pile of code in a particular language
edit: also the reason to generate C++ and not Rust is basically to have fewer build deps and runtime deps (and that there’s no safety advantage to the latter). Note that libc is a required runtime dep of a shell; it’s closely tied to libc and the kernel.
Embedded systems are often Linux systems these days, which need a shell. And I want Oil to be usable on all CPU architectures, not just “first class” Rust or Go ones, etc.
This RFC is not on the roadmap for the language team for the moment.
We will consider it at earliest in 2020.
But I added them to ASDL with ease. I use them all over the AST and it’s a more natural expression of the language, and generates more compact C++ types:
One thing I’ve learned from this project is that algebraic data types have a lot of nontrivial design decisions. And you pretty often run into limitations of them in bigger programs.
In contrast, generating C++ means that the code is bog-standard and essentially 100% portable. What a shell expects from the operating system is extremely old and standardized by POSIX.
Although this doesn’t mean nobody should try writing Oil in Go, if you’re willing to deal with portability issues.
(Sorry for the deluge of replies, but I expect these questions to come up more in the future, so this is a good place to draft answers to them all :-) )
Thanks for letting us know. This is awesome feature and I’m looking forward to release. It’s really interesting that tar handles foo.txt and ./foo.txt in a different way, I didn’t know that. It’s great that you noticed it early, it might cause a lot of confusion.
Using cargo, Rust package manager, instead of directly calling compiler, produces more optimized binary even without any special switches or flags. Rust docs suggest users to use cargo instead of directly calling the compiler; this is the standard way to compile rust programs.
cargo build --release
This produces total of 95 syscalls, 20 of them or unique.
They’re blocking access from old devices for absolutely no technical reason; they’re blocking read-only access from folks that might not have any other devices at their disposal.
If you have an old iPod lying around, why on earth should you not be able to read Wikipedia on it? Absolutely no valid technical reason to deny access. Zilch. None. Nada.
There’s no reason it shouldn’t be possible to read Wikipedia over straight HTTP, for that matter.
I know next to nothing about security so correct me if I’m wrong, but doesn’t leaving old protocols enabled make users vulnerable to downgrade attacks?
You’re applying bank-level security to something that’s public information and should be accessible to everyone without a licence or access control in the first place. I don’t even know what sort of comparison to make here best, because in my view requiring HTTPS in the first place here was a misguided decision that’s based on politics, corporate interests and fear, not on rational facts. Postel’s law is also a well-known course of action in telecommunication, even Google still follows it — www.google.com still works just fine over straight HTTP, as does Bing, no TLS mandated from those who don’t want it.
I agree with you, I’d like to be able to access Wikipedia with HTTP, but this is in my opinion a different issue from disabling old encryption protocols.
Accessing Wikipedia with secure and up to date protocols might not be necessary to you but it might be for people who live under totalitarian regimes. One could argue that said regimes have better ways to snoop on their victims (DNS tracking, replacing all certificates with one they own…) but I still believe that if enforcing the use of recent TLS versions can save even a single life, this is a measure worth taking. It would be interesting to know if Wikipedia has data on how much it is used by people living in dictatorships and how much dropping old TLS versions would help these people.
It’s funny you mention it, because this actually would not be a problem under a totalitarian regime with a masquerading proxy and a block return policy for the https port and/or their own certificates and a certificate authority. See https://www.xkcd.com/538/.
Also, are you suggesting that Wikipedia is basically blocking my access for my own good, even though it’s highly disruptive to me, and goes against my own self-interests? Yet they tell me it is in my own interest that my access is blocked? Isn’t that exactly what a totalitarian regime would do? Do you not find any sort of an irony in this situation?
this actually would not be a problem under a totalitarian regime with a masquerading proxy and a block return policy for the https port and/or their own certificates and a certificate authority.
Yes, this is what I meant when I wrote “One could argue that said regimes have better ways to snoop on their victims”.
Also, are you suggesting that Wikipedia is basically blocking my access for my own good
No, here’s what I’m suggesting: there are Wikipedia users who live in countries where they could be thrown in jail/executed because of pages they read on Wikipedia. These users are not necessarily technical, do not know what a downgrade attack is and this could cost them their lives. Wikipedia admins feel they have a moral obligation to do everything they can to protect their lives, including preventing them from accessing Wikipedia if necessary. This is a price they are willing to pay even if it means making Wikipedia less convenient/impossible to use for other users.
If they left http, yeah, sure. But I don’t think any attack that downgrades ssl encryption method exists, both parties always connect using the best they have. If there exists one, please let me know.
There is no technical reason I’m aware of. Why does wikipedia do this? It’s not like I need strong encryption to begin with, I just want to read something on the internet.
I still have usable, working smartphone with android Gingerbread, it’s the first smartphone I ever used. It’s still working flawlessly and I’m using it sometimes when I want to quickly find something when my current phone has no battery and I don’t want to turn on my computer.
This move will for no reason kill my perfectly working smartphone.
Which exact page you’re looking at may be of interest, e.g. if you’re reading up on medical stuff.
Are you suggesting that we implement access control in public libraries, so that noone can browse or checkout any books without strict supervision, approvals and logging by some central authority? (Kinda like 1984?)
Actually, are you suggesting that people do medical research and trust information from Wikipedia, literally edited by anonymous people on the internet?! HowDareYou.gif. Arguably, this is the most misguided security initiative in existence if thought of in this way; per my records, my original accounts on Wikipedia were created before they even had support for any TLS at all; which is not to say it’s not needed at all, just that it shouldn’t be a mandatory requirement, especially for read-only access.
Are you suggesting that we implement access control in public libraries, so that noone can browse or checkout any books without strict supervision, approvals and logging by some central authority? (Kinda like 1984?)
I’m saying that you may not wish other people to infer what medical conditions you may have based on your Wikipedia usage. So TLS as the default is desirable here, but whether it should be mandatory is another question.
Are you suggesting that we implement access control in public libraries, so that noone can browse or checkout any books without strict supervision, approvals and logging by some central authority? (Kinda like 1984?)
PSST, public libraries in the western world already do this to some extent. Some countries are more central than others thanks to the US PATRIOT Act.
Not my experience at all; some private-university-run libraries do require ID for entry; but most city-, county- and state-run libraries still allow free entry without having to identify yourself in any way. This sometimes even extends to making study-room reservations (can often be made under any name) and anonymous computer use, too.
I still have usable, working smartphone with android Gingerbread, it’s the first smartphone I ever used. It’s still working flawlessly and I’m using it sometimes when I want to quickly find something when my current phone has no battery and I don’t want to turn on my computer.
This move will for no reason kill my perfectly working smartphone.
It’s not working flawlessly, the old crypto protocols and algorithms it uses have been recalled like a Takata airbag, and you’re holding on because it hasn’t blown up in your face yet.
our ire should be directed to the smartphone OS vendor for not providing needed updates to encryption protocols
I think it’s pretty clear that the user does not need encryption in this use-case, so, I don’t see any reason to complain to the OS vendor about encryption when you don’t want to be using any encryption in the first place. Like, seriously, what sort of arguments are these? Maybe it’s time to let go of the politics in tech, and provide technical solutions to technical problems?
I don’t think what passes for “postel’s law” is what Postel meant, anyway.
AFAICT, Postel wasn’t thinking about violations at all, he was thinking about border conditions etc. He was the RFC editor, he didn’t want anyone to ignore the RFCs, he wanted them to be simple and easy to read. So he wrote “where the maximum line length is 65” and meant 65. He omitted “plus CRLF” or “including CRLF” because too many dotted i’s makes the prose heavy, so you ought to be liberal in what you accept and conservative in what you generate. But when he wrote 65, he didn’t intend the readers to inter “accept lines as long as RAM will allow”.
As I understand it, they’re protecting one set of users from a class of attack by disabling support for some crypto methods. That seems very far from “absolutely no technical reason”.
As for HTTP, if that were available, countries like Turkey would be able to block Wikipedia on a per-particle basis, and/or surveil its citizens on a per-article basis. With HTTPS-only, such countries have to open/close Wikipedia in toto, and cannot surveil page-level details. Is that “no reason”?
As for HTTP, if that were available, countries like Turkey would be able to block Wikipedia on a per-particle basis, and/or surveil its citizens on a per-article basis. With HTTPS-only, such countries have to open/close Wikipedia in toto, and cannot surveil page-level details. Is that “no reason”?
I don’t understand why people think this is an acceptable argument for blocking HTTP. It reminds me of that jealous spouse scenario where someone promises to inflict harm, either to themselves or to their partner, should the partner decide to leave the relationship. “I’ll do harm if you censor me!”
So, Turkey wants to block Wikipedia on a per-article business? That’s their decision, and they’ll go about it one way or another, I’m sure the politicians they don’t particularly care about the tech involved anyways (and again, it’s trivial for any determined entity to block port 443, and do a masquerade proxy on port 80, and if this is done on all internet connections within the country, it’ll work rather flawlessly, and noone would know any better). So, it’s basically hardly a deterrent for Turkey anyways. Why are you waging your regime-change wars on my behalf?
Well, Wikipedia is a political project, in much the same way that Stack Overflow is. The people who write have opinions on whether their writings should be available to people who want to read.
You may not care particularly whether all of or just some of the information on either Wikipedia or SO are available to all Turks, but the people who wrote that care more, of course. They wouldn’t spend time writing if they didn’t care, right? To these people, wanting to suppress information about the Turkish genocide of 1915 is an affront.
So moving to HTTPS makes sense to them. That way, the Turkish government has to choose between
allowing Turks to read about the genocide
not allowing Turks any use of Wikipedia
The Wikipedians are betting that the second option is unpopular with the Turks.
It’s inconvenient for old ipad users, but if you ask the people who spend time writing, I’m sure they’ll say that being able to read about your country’s genocide at all is vastly more important than being able to read using old ipads.
So what’s to stop a totalitarian regime from doing the following?
Redirect all DNS queries to their own DNS servers? The root DNS servers use fixed IP addresses, so it would be easy enough to reroute those addresses to return any address they want.
Redirect all DoH to 1.1.1.1 (or other well known DoH addresses) to again, their own server? Is the CloudFlare public key installed on all browsers? How would you know you are hitting CloudFlare, and not TotallyCloudFlare served by TotallyLegitCA?
Given control over DNS, redirect users to TotallyWikipedia? Again, do you know what CA Wikipedia uses? They can then decode (doesn’t matter if it’s SSL/1.0 or TLS/1.3) the request and proxy it or send out security to question the loyalty of the citizen. Or you know, download the entirety of Wikipedia (which anyone can do), and serve up a cleaned up version to their citizens.
The difficulty is to setup/enrole TotallyLegitCA. How do you do that? If TotallyLegitCA is public, the transparency log will quickly reveal what they are doing. The only way to pull that seems to force people to have your CA installed, like Kazakhstan is doing.
That’s actually incorrect. There are various technical reasons. But also remember that they need to operate on a vast scale as a non-profit. This is hard.
Here are some technical reasons. I’m sure others will chime in as there are likely many more.
some attacks on TLSv1.0 can compromise key material which is used for the newer, secure versions of TLS
providing a read-only version without login over HTTP shouldn’t really add any new code except they’d be on a HTTP-2-only webserver if I’m not mistaken.
But I hear all the time that I must ensure my personal site uses HTTPS and that soon browsers will refuse to connect to “insecure” sites. Isn’t this a good thing Wikipedia is doing? /s
I have HTTPS on my completely static website mostly so that no one asks why I don’t have HTTPS, but on the other hand, the “completely static” part is only relevant as long as there are only Eves in the middle and no Mallories.
If serving everything over HTTPS will make the life of ISPs injecting ads and similar entities harder, it’s a good thing, until there’s a legal rather than technical solution to that.
I actually think that HTTPS is reasonable for Wikipedia, if for nothing else to hinder 3rd parties for capture your embarrassing edits to “MLP: FIM erotica” and tracing it to back to you. For a static, read-only site it just adds cost and/or a potential point of failure.
For some reason none loaded for me with newest Firefox, even when I unblocked everything relevant from uMatrix. Chromium with everything unblocked shown only the first one (???). Only chromium with all disabled extensions worked as expected.
Shame author didn’t add simple img, imagine how much more portable and reliable that would be.
Shell arithmetic expansion conflates data and code, but Oil statically parses arithmetic expressions instead.
Shellshock was also about the conflation of data and code, with the export -f misfeature that serialized a function to a string!
However I don’t really have answer to this flag vs. arg issue (flag being code, and arg being data), other than “use -- everywhere”.
Relatedly, in Oil, echo is no longer a special case because it supports -- (if you use bin/oil rather than bin/osh). So you can do echo -- $x, which is a safe version of echo $x.
In POSIX shell you have to use printf to echo an arbitrary string, although not many shell scripts follow that discipline!
You can’t use echo -- $x in POSIX shell because that would output 2 dashes also :)
If anyone has a better idea than “use -- everywhere”, let me know :)
I guess you write shell in two modes: the quick and dirty throwaway, and then Oil is supposed to allow you to upgrade that to a “real” program. So there could be a lint tool that warns about -- or just auto-refactors the script for you. Or maybe there is some option that breaks with shell more radically.
If you ever write a script that operates on untrusted files, always make sure the command will do exactly the thing you wanted it to do.
The problem is that when you ask yourself “does this do exactly what I want it to?”, you don’t have the imagination to come up with something like “filenames that look like flags will be interpreted as one.”
Someone who would make a safer, saner shell to write programs with would be a hero.
However, after thinking about it more, I realized that no solution will solve the dissonance inherent to any language like that. You will always be tripping up and compromising over where something is or is not quoted. All templating languages have this problem.
Instead, I’m currently of the opinion that an approach more like PowerShell, in which you call functions, not write the names of programs and arguments as text, is the right way forward. This removes the problem of quotation. The downside to this approach is that it requires work to produce the APIs. It’s fine if you have a large standard library, as PowerShell does, but being able to pull a binary off the shelf e.g. one written in e.g. Python or C should still be natural.
The missing part therefore, in my opinion, is that programs (in any system be it Linux, OS X, Windows, BSD), ought to be accompanied by a schema (could be in JSON, doesn’t matter), let’s say git and git.schema, which can be interpreted offline or “cold” – without running the program (very important) –in order to know (1) arguments/flags the program accepts (as commands or switches), (2) the types/formats of those inputs, (3) possibly list the side-effects of those commands.
This allows a shell or IDE to provide a very strong completion and type-checking checking story, and to provide it out of the box. A projectional editor would be satisfying here, too (even something as simple as CLIM’s listener).
When downloading a random binary online, you could additionally download the schema for it. The schema file itself can contain a SHA256 of the binary that it’s talking about, to avoid accidental misuse. Currently if you want completion for an exe, you have to generate some bash. So it’s clear that the need is already there, it’s just implemented in a poorly done way.
The upside to this approach is that it’s additive, no one has to change their existing software. Additionally, it’s easy to produce for old software; You can make a parser for --help or man pages to generate a “best guess” schema for a program. The reason you wouldn’t put this into a shell by default would be because some programs don’t accept --help and/or they run side effects and delete things. Existing opt parser libraries can generate such schemas, just like some of them can generate bash at the moment.
Another upside is that it can simply be a standard/RFC published freely, just like JSON.
I haven’t moved forward on this schema idea yet, because it’s hard to research existing solutions (hard to google for). It would be an FFI standard but instead of calling C functions you’re calling processes.
Yeah I agree with your diagnosis of the issue. You need a schema and a parser to go along with every command.
I say a parser because in general every command has its own syntax, in addition to semantics (the types of the arguments). Some commands recognize fully recursive languages in their args like find, test, and expr. (Although of course there are common conventions.)
It relates pretty strongly to this idea about shell-agnostic autocompletion we’ve been kicking around. Every command needs a parser and a schema there too.
And yes it’s nice to have the property that it’s a pure addition. It’s basically like TypeScript or MyPy (which I’m using in Oil). They handle a lot of messiness to provide you with a smooth upgrade path.
If you’d like to pursue these ideas I think Oil will help because it has an accurate and flexible shell parser :) I break shell completion into two parts: (1) completing the shell language, then (2) completing the language of each individual tool. That’s the only way to do it correctly AFAICT. bash conflates the two problems which leads to a lot of upstream hacks.
That’s really interesting, especially thanks for the dwheeler page.
I solve shell problems by using shellcheck all the time, it catches most mistakes one can make and it itegrates nicely into existing editors.
Oil looks really interesting and is certainly better than the shell+shellcheck combo, but I don’t think I want to write everything in new syntax that is not universal and might mean I’ll have to rewrite it in bash later anyway.
Well Oil is very bash compatible – it runs your existing shell/bash scripts. You can use it as another correctness tool on top of ShellCheck. It catches problems at runtime in addition at parse time.
There are a bunch of other examples I should write about too.
If your shell scripts are used for years, and keep getting enhanced, chances are that you will run into the limitations of bash. And you won’t want to rewrite the whole thing in another language. That’s what Oil is for :) It’s fine to stick with bash now, but as you use shell more and more, you will run into a lot of limitations.
Yes, this is also a good page about the issue, and related issues:
I disagree with all items marked as “#WRONG” in this site. Clean globbing is too powerful and beautiful to complexify with these mundane problems. Filenames are variable names, not data, and you get to chose them. What is actually wrong is the existence of files with crazy names. This should be solved at the filesystem level by disallowing e.g. the space character on a filename (no need to bother the user, it could be translated simply into a non-breaking space character, and nobody would notice except shell scripts).
An environment variable indicating the providence of each argument, and sending patches to getopt, argp, coreutils, bash, and your favorite unix programs.
If you want to write to for example stderr from laguage which doesn’t support stderr redirection (awk), you can instead write it to /dev/stderr and it will redirect to stderr automagically!
This also means you don’t have to remember the awful sh syntax for redirecting echo output stream to 2nd file descriptor, you can just do echo 'error' > /dev/stderr.
Btw I really like the graphical style! The way the paragraphs are nestes according to which part of the thought are they about is amazing idea, it feels much easier to follow.
Thank you! I was kind of afraid people wouldn’t find it too readable haha, glad to see someone enjoying it.
Nice! See L-systems: https://en.m.wikipedia.org/wiki/L-system
I’m blocked by cloudflare, even from different IPs (ray id: 7e9d6daf1c7ab38f). Anyone else has the problem? (Europe/Czechia)
I can access it just fine from Poland. Pretty close to the Czech border, if that counts :P
The actual content of the article seems to be just about one sentence long at the very end, the rest is just rehearsal of basics of shared pointer usage. Shame, I wanted to read a deeper analysis about shared ptr design issues.
The root problem with shared pointer is that it tries to be a generic thing over three quite different use cases:
The first is better solved with an intrusive reference count in a superclass because otherwise there’s no good way of going from
thisto a refcounted wrapper. The standard hasstd::enable_shared_from_thisbut this doesn’t require the objects to be constructed withmake_sharedso adds another word to contain a pointer from the object to its shared control block.The additional state required for weak references adds even more overhead and 99% of the time it’s not used (I have never used
std::weak_ptr, though I’ve usedstd::shared_ptrquite a lot). If you had a separate type for shared pointers that permit weak pointers then you could reduce the overhead in the common case.There’s also the thing that just because the shared pointer’s refcount manipulation is atomic doesn’t mean that it’s thread safe (assigning to a shared pointer from two threads breaks in exciting ways), so you now have another atomic version.
I ended up writing my own intrusive ref-counted base class for $DAYJOB, for the above reasons. Unfortunately this is the kind of attractive nuisance that seems trivial at first, and it was trivial to get 99% of the way, but there were exciting edge cases that revealed themselves periodically, especially with multithreading, exceptions in constructors/destructors, and so on.
It would be really nice if std:: had taken care of this for me.
(I’m sure Boost has something like this, but my impression of Boost is it’s a gigantic tangle of yarn, and if I pick up one little trinket it’s going to drag a megabyte of other stuff along with it into my binary, plus double my build times.)
intrusive_ref_counter is the Boost solution for this. There’s the BCP command to extract only the required portions of the library, but it’s still going to include a lot of configuration code. I’d also write my own class for this.
Knock knock!
Who’s there?
intrusive_ref_counter!intrusive_re——4! 5! 6! 7!
One practical issue is that reference cycles can hide other bugs. After you fix the cycle with weak_ptr or some other technique, all those use-after-frees bubbling under suddenly surface. With weak pointers you can avoid a crash by checking if
lock()returns a nullptr but good luck rewriting all the logic with that in mind.Could call it an extreme case of accidental technical debt.
For insta-sharing of the modified videos / their “indexes”, maybe you could put them on IPFS? Sorry if you already mention this somewhere, I only glanced over the readme as I’m in a hurry.
I actually don’t modify the videos, I just download them and build a timetable (that I call an ‘index’) that remembers when was the video loud and when was it silent - the timetable file sizes range from 1K to 20K (I don’t even bother to compress them), most of them being in the ~4K range (it’s the javascript that speeds up the video while playing it). And this extension is not currently used much, so it’s not much of a concern as the sizes are manageable so far.
IPFS is interesting project and I need to finally look into it further. It would be really nice if I managed to run it entirely over IPFS and eliminate the need for a central server. I quite like the idea.
I played with the Go API recently, there was a lot of docs I had to wade through, but the final code was quite nice and short. Just remember to call “pin” and/or “provide”, one of them brought down sharing speed from behind NAT from some horrific 45 minutes to something much more bearable around 2 min. And experimental auto-relay; didn’t minimize to check which ones are the required ones. I know they also have JS lib on the same level of support I think.
This is awesome. And written entirely in bash as well. I really like the way it’s written. I never thought of writing http server in bash, but it makes perfect sense when you think about it. In the end, http protocol was designed for this. I might steal this idea later.
Btw, is it secure to let people create random files in your filesystem? I’m on phone so I can’t properly test it, but isn’t there danger of path traversal?
He strips out slashes, so I don’t think so.
This certainly is pushing it! It accepts “.”, “..” and flags to
ln, but there are no flags that you can use to makelndo nasty things, AFAICT. I tried-i, which indeed seems to send the server into an endless loop waiting for “input”. So a resource exhaustion attack might be possible, if you can hit the max open file descriptor or pid limit.With
-fas subdomain, I’d expect it to delete the target file, but unfortunately the check whether the target is the same as the source comes first…Thanks for noticing the flags, I forgot that GNU tools are stupid and take flags after arguments.
EDIT: No ok there’s no excuse. I’m an idiot.
Does it really accept flags? Input should be split by spaces because of read if I’m not mistaken.
ln "$yn" "$subdomain"with$subdomainequals-f, you getln yes -for something like it. Because of how most programs work, flags don’t necessarily need to come first. To fix this particular issue, it should readln -- "$yn" "$subdomain".I think I broke it, with
-fbdrsas subdomain; this force removes the target file (which isyesorno) by creating a backup and then linking it to itself. Or something…Huh this looks like a really convenient setup. My blog (if you can call it so, heh) is generated by ugly shell+awk combo I wrote. Everything is handwritten. It even uses a custom flavour of markdown. Basically I have directory with some
2020-06-06.mdfiles, and a simple shell scripts load the markdown and build everything - the blog pages itself (which is just simple html file), index file (just a list of aforementioned blog pages), and very basic kind of javascript search (user can search via tags, without the need to have any backend).I don’t need any setup, it’s just literally
git cloneaway form working on any linux box, without the need to configure anything at all. And while the way the scripts are written, new features can be added easily (someone emailed me about lack of RSS, and I got it working from scratch, in awk, in under an hour), they get messy quick. And while I love the simplicity and that getting it work is possible under a minute (I use it for more things, for example for rendering my journal to html), I think like it’s slowly getting out of my control. I’ve thought of either rebuilding it in real programming language or switching to some static page generator. So thanks for this post!Btw, consider using CSS prefers-color-scheme. It allows user to specify the preferred theme (light/dark), and lets websites adapt to the settings. It’s really convenient, not that hard to set up if you have sane CSS, and it boosts user experience a lot. All of my websites use this, and I am pushing pretty heavily when some site doesn’t use it. It is just great when you setup your OS to dark theme, and running applications as well as running websites instantly change their appearance to match the OS (and daytime).
Dark mode is on my list, among other things, so I’m definitely adding that! Thanks for the tip 👍🏼
Pocket will now provide fascinating reads from trusted sources in the UK on new tab. I wonder why did they kill RSS support if they intended Firefox to become news aggregator and reader.
Those two are unrelated decisions. The RSS code was bitrotting and that is why it was removed. Pocket is a bundled addon.
Yes, they could’ve shipped a bundled addon for RSS as well. But there are a gazillion RSS addons for Firefox that do a much better job than the built-in feature ever did. They are a single click away.
They didn’t need to destroy RSS bookmarks.
They explained the reasons for removing it and there are a dozen add-ons that bring that feature and more back.
Removing support for something doesn’t mean you deliberately destroy user data.
From the linked article:
So, the data was not destroyed, it was saved in multiple formats including standard formats made to be imported in other solutions. Documentation was provided to guide the transition.
My data was.
I am sorry, it shouldn’t have been. Have you filled a bug report?
“Bitrotting” is never a justification to remove a feature. It worked perfectly fine until they removed it. Now you have to install RSSPreview, so it’s an extra step to get this basic lightweight functionality.
Have you read the link I pasted earlier? The way the live bookmarks were built were not compatible with sync, used ancient firefox stuff that they are trying to get rid of and had no way to set read/unread status of a post which is a basic necessity of a rss reader. They also have telemetry showing few people using it and more people using more complete add-ons. Mozilla is making a huge effort trying to make Firefox more nimble and maintainable, and the feature was removed. I liked that feature too, but when you read what people maintaining that code and making decisions said about it you might understand why it was done.
The feature I miss is the RSS previews. The justification given in the blog post holds no weight:
It is illustrative that Mozilla (or whoever wrote this post) thinks seven years is too long to wait before a styling update. This attitude is responsible for the recent degradation of the about:config page, among other things…
You understand that in these seven years Firefox went through a major rework? Working towards removing XUL, XCB, and lots of old code. Adding the Rust bits and new components from Servo. Refactoring the whole UI into an HTML5 based UI. Seven years when we’re talking about what happened in the specific interval mentioned is a lot of changes to the browser code.
Also, as I’ve mentioned multiple times, that feature was:
People complain about bloat all the time in Firefox. Keep saying that X, Y, Z, features should be add-ons. When they remove a feature that the power users were not using (because they were already using better add-ons) and the casual users were not using (because they didn’t know it existed) taking care to provide a migration path, save the data, and offer alternatives, people cry that they shouldn’t have done it.
It is impossible to please you folks. If they ship some new feature, it is bloat. If they remove something that is hard to maintain and broken, then it is a crime.
Also, the feature is one click away in the add-ons portal. Add-ons exist so that browser vendors don’t need to cover every single feature by themselves.
I think you may be referring to Live Bookmarks; I’m referring to RSS previews. I don’t believe these bullets apply to RSS previews.
I also don’t see how Firefox would know how many people used the RSS preview function. If they click on a link to an RSS feed, they would be using it without knowing that they were using it. Then suddenly those links become raw XML and they don’t know that they need the RSSPreview addon to render the page.
Converting an RSS feed to HTML is an extremely simple task that provides functionality that people had come to expect with very little code. New features must meet a higher standard of usefulness and code quality because people aren’t already used to them.
If you stop to check Firefox source code and the changes they’ve been doing to “modernize” it away from XUL and many other older technologies inside the code base, you’ll notice that that source code was not up to date with how bookmarks work, it didn’t contain the RSS features people expected it to have (such as read/unread status), and was incompatible with sync and missing from the mobile browsers. That is why it was removed. Read the article, talk to people before spreading FUD.
The feature didn’t disappeared overnight without annoucing and planning a transition. There are a dozen RSS readers on the add-ons portal, all of which are better than that feature ever was. The feature never worked well, that is why it was axed.
Since when Mozilla focus feature-wise has been monetization? The RSS code was removed, as explained in the article that apparently no one read, because it was not compatible with the new features they added. The code was bitrotting because it was difficult to maintain and the target userbase moved to add-ons or services with more features. The feature was also incomplete from the point of view of those who make the browser and instead of devoting resources to it, when suitable add-ons are present in the ecosystem, they opted to remove it and focus on other tasks.
Not all features are eternal. Features get removed. This was announced, a path of migration was documented, the data was saved in multiple places. You could easily either move to a SaaS, or to an add-on and import all the same feeds from the OPML export.
I guess that RSS is more difficult to monetize. They make money on Pocket using ads.
Read the article, talk to people involved, go to mozilla community/slack/matrix and talk to people, and you’ll realize the feature was removed for many reasons none of which is monetization.
From the article:
Seems to have plenty to do with monetization.
The pocket thing is related to monetization. What I’m talking that is not related to monetization is the Live Bookmarks removal that happened in Firefox 64, which was released in 2018. All I’ve been replying and chatting here is related to that feature, and not pocket. The Live Bookmarks removal happened more than 10 versions ago and was not motivated by monetization.
The quote above was part of the reasoning behind removing RSS support. In other words they wouldn’t have removed RSS support if there weren’t monetized alternatives like Pocket and sponsored content on the home page.
They removed live bookmarks in 2018.
The RSS reader was removed in 2018 alongside the live bookmarks feature. They are part of the same source. Stop spreading FUD.
Pocket sponsored content, which I don’t like either, is not related to the RSS feature.
What “other features to help users discover and read content” was the quote referring to?
So they destroyed my data in order to monetize me?
No, they told you the feature was going to be removed with months in advance. Provided docs on how to to take your data out, and links to alternative solutions for the feature. Your data should have been saved into two different backups: one OPML on your Desktop, and the bookmarks should still be there, they’d just be normal bookmarks instead of live bookmarks.
If this didn’t happen, then it is a bug and even though I am really sorry for you, it is the case that the only possible solution is to fill a bug report and try to figure out why the documented process failed so that it doesn’t happen to someone else.
Thanks for sharing! This is exceptionally readable, considering the topic. Author’s page also contains several interesting articles worth looking at. I really like their writing style, it’s easy to digest even for people not in the crypto field.
Hi, author here. I’ve written a debugger for sed in Rust. This was not only to learn rust, but to actually have a solid debugger for sed. I’ve started learning sed recently and decided to start writing various algorithms in it. And as sed doesn’t have numbers and can just filter/transform text, this is a challenge even for something like comparing two numbers. I’ve seen people do amazing things with it.
I’d be glad for any comments regarding code quality or the debugger itself.
Edit: I’ve written a bit about the way this was built.
What was the reason of choosing c++ over a fast memory safe language, be it traditional language like Go or something completely different like Rust?
Oil is memory safe by construction … that was one of the main points of writing it in a collection of high level DSLs!
Manually written C++ is “kinda” memory safe but not really, but that’s not what Oil is.
http://www.oilshell.org/blog/2019/06/17.html#why-is-it-written-in-python (I should change this to “DSLs”, because it’s written in Python in one important sense, but that description is misleading in another important sense)
For example, it doesn’t use strings-as-buffers as in C; it uses strings-as-values as in Python. (And it doesn’t use STL’s string class.) That’s a huge difference that you’ll see all over the code.
Despite that it’s actually faster than bash and zsh which are written in very low level C! I call bash and zsh “grovelling through backslashes and braces one by one”, and Oil doesn’t do that.
Oil is partly a shell and partly a software engineering experiment in “metaprogramming” – i.e. can you implement code with domain-specific, high level abstractions and still get good performance?
This post is probably 20-40% of the victory lap. I hope to have 80% by the end of the year – i.e. a working shell in pure native code, that’s measurably better than existing shells along several important dimensions.
If the project becomes self-sustaining (which is a big if), I will have time to optimize it in ways that are impossible if it were written by hand in C++, Rust, or Go. The abstractions are “right” – they don’t express irrelevant detail and give you more leeway in implementing them.
Honestly I think Rust (and manually written C++) are at the wrong abstraction level for writing a 10K line recursive descent parser. They would end up at more like 20 - 30K lines due to all the irrelevant details you need to take care of. (note that bash is a 140K line C program.)
However I encourage others to pursue that direction:
http://www.oilshell.org/blog/2020/02/recap.html#posts-i-still-want-to-write
If you look at this listing it may give you a sense of how Oil is architected now. The three biggest files are the 3 biggest DSLs.
https://www.oilshell.org/release/0.8.pre2/metrics.wwz/line-counts/oil-cpp.txt
Oil isn’t written or Python or C++, or Rust or Go. It’s written in “OPy/mycpp”, ASDL, and an abstract regex dialect (actually “Eggex” if I self host.)
osh-lexfile is all DFAs generated from regexes.osh_evalfile is the parsers and evaluators translated from statically typed Python. This will be more like 25K when the translation process is done.*_asdlfiles are language-independent type definitions expanded to C++ (and the types can be pretty printed)Some more color: Oil is made of ideas; it’s not a big pile of code in a particular language
https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/Oil.20is.20made.20of.20ideas.3B.20it's.20not.20a.20pile.20of.20code.20.2E.2E.2E (requires login)
edit: also the reason to generate C++ and not Rust is basically to have fewer build deps and runtime deps (and that there’s no safety advantage to the latter). Note that libc is a required runtime dep of a shell; it’s closely tied to libc and the kernel.
Embedded systems are often Linux systems these days, which need a shell. And I want Oil to be usable on all CPU architectures, not just “first class” Rust or Go ones, etc.
Here’s one example where using domain-specific abstractions is more expressive than Rust. Rust doesn’t have “first class variants” as I call them:
https://github.com/rust-lang/rfcs/pull/2593
But I added them to ASDL with ease. I use them all over the AST and it’s a more natural expression of the language, and generates more compact C++ types:
https://oilshell.zulipchat.com/#narrow/stream/121539-oil-dev/topic/Shared.20Variant.20Refactoring (requires login)
One thing I’ve learned from this project is that algebraic data types have a lot of nontrivial design decisions. And you pretty often run into limitations of them in bigger programs.
Another example from Haskell (and I think Scala has similar things, etc.): https://stackoverflow.com/questions/28244136/what-is-scrap-your-boilerplate
That’s a conflict between types and metaprogramming as I view it. Oil prefers to leverage metaprogramming.
http://www.oilshell.org/blog/tags.html?tag=metaprogramming#metaprogramming
Also on Go, you can’t write a POSIX shell in portable Go, basically due to it’s threaded runtime:
https://lobste.rs/s/hj3np3/mvdan_sh_posix_shell_go#c_qszuer
related: https://lobste.rs/s/6a6zne/some_reasons_for_go_not_make_system_calls
In contrast, generating C++ means that the code is bog-standard and essentially 100% portable. What a shell expects from the operating system is extremely old and standardized by POSIX.
Although this doesn’t mean nobody should try writing Oil in Go, if you’re willing to deal with portability issues.
(Sorry for the deluge of replies, but I expect these questions to come up more in the future, so this is a good place to draft answers to them all :-) )
Updated: I addressed this problem in Oil (and linked this thread)
http://www.oilshell.org/blog/2020/02/dashglob.html
Thanks for letting us know. This is awesome feature and I’m looking forward to release. It’s really interesting that tar handles
foo.txtand./foo.txtin a different way, I didn’t know that. It’s great that you noticed it early, it might cause a lot of confusion.Using cargo, Rust package manager, instead of directly calling compiler, produces more optimized binary even without any special switches or flags. Rust docs suggest users to use cargo instead of directly calling the compiler; this is the standard way to compile rust programs.
This produces total of 95 syscalls, 20 of them or unique.
While not significantly better, it’s something.
Used cargo 1.40.0.
Maybe some folk don’t understand what’s going on here, but this is in direction violation of Postel’s law:
They’re blocking access from old devices for absolutely no technical reason; they’re blocking read-only access from folks that might not have any other devices at their disposal.
If you have an old iPod lying around, why on earth should you not be able to read Wikipedia on it? Absolutely no valid technical reason to deny access. Zilch. None. Nada.
There’s no reason it shouldn’t be possible to read Wikipedia over straight HTTP, for that matter.
I know next to nothing about security so correct me if I’m wrong, but doesn’t leaving old protocols enabled make users vulnerable to downgrade attacks?
You’re applying bank-level security to something that’s public information and should be accessible to everyone without a licence or access control in the first place. I don’t even know what sort of comparison to make here best, because in my view requiring HTTPS in the first place here was a misguided decision that’s based on politics, corporate interests and fear, not on rational facts. Postel’s law is also a well-known course of action in telecommunication, even Google still follows it — www.google.com still works just fine over straight HTTP, as does Bing, no TLS mandated from those who don’t want it.
I agree with you, I’d like to be able to access Wikipedia with HTTP, but this is in my opinion a different issue from disabling old encryption protocols.
Accessing Wikipedia with secure and up to date protocols might not be necessary to you but it might be for people who live under totalitarian regimes. One could argue that said regimes have better ways to snoop on their victims (DNS tracking, replacing all certificates with one they own…) but I still believe that if enforcing the use of recent TLS versions can save even a single life, this is a measure worth taking. It would be interesting to know if Wikipedia has data on how much it is used by people living in dictatorships and how much dropping old TLS versions would help these people.
It’s funny you mention it, because this actually would not be a problem under a totalitarian regime with a masquerading proxy and a
block returnpolicy for the https port and/or their own certificates and a certificate authority. See https://www.xkcd.com/538/.Also, are you suggesting that Wikipedia is basically blocking my access for my own good, even though it’s highly disruptive to me, and goes against my own self-interests? Yet they tell me it is in my own interest that my access is blocked? Isn’t that exactly what a totalitarian regime would do? Do you not find any sort of an irony in this situation?
“Isn’t that exactly what a totalitarian regime would do?”
I think you may have overstated your case here.
Yes, this is what I meant when I wrote “One could argue that said regimes have better ways to snoop on their victims”.
No, here’s what I’m suggesting: there are Wikipedia users who live in countries where they could be thrown in jail/executed because of pages they read on Wikipedia. These users are not necessarily technical, do not know what a downgrade attack is and this could cost them their lives. Wikipedia admins feel they have a moral obligation to do everything they can to protect their lives, including preventing them from accessing Wikipedia if necessary. This is a price they are willing to pay even if it means making Wikipedia less convenient/impossible to use for other users.
If they left http, yeah, sure. But I don’t think any attack that downgrades ssl encryption method exists, both parties always connect using the best they have. If there exists one, please let me know.
There is no technical reason I’m aware of. Why does wikipedia do this? It’s not like I need strong encryption to begin with, I just want to read something on the internet.
I still have usable, working smartphone with android Gingerbread, it’s the first smartphone I ever used. It’s still working flawlessly and I’m using it sometimes when I want to quickly find something when my current phone has no battery and I don’t want to turn on my computer.
This move will for no reason kill my perfectly working smartphone.
Downgrade attacks are possible with older versions of SSL e.g. https://www.ssl.com/article/deprecating-early-tls/
Which exact page you’re looking at may be of interest, e.g. if you’re reading up on medical stuff.
Are you suggesting that we implement access control in public libraries, so that noone can browse or checkout any books without strict supervision, approvals and logging by some central authority? (Kinda like 1984?)
Actually, are you suggesting that people do medical research and trust information from Wikipedia, literally edited by anonymous people on the internet?! HowDareYou.gif. Arguably, this is the most misguided security initiative in existence if thought of in this way; per my records, my original accounts on Wikipedia were created before they even had support for any TLS at all; which is not to say it’s not needed at all, just that it shouldn’t be a mandatory requirement, especially for read-only access.
P.S. BTW, Jimmy_Wales just responded to my concerns — https://twitter.com/jimmy_wales/status/1211961181260394496.
I’m saying that you may not wish other people to infer what medical conditions you may have based on your Wikipedia usage. So TLS as the default is desirable here, but whether it should be mandatory is another question.
PSST, public libraries in the western world already do this to some extent. Some countries are more central than others thanks to the US PATRIOT Act.
Not my experience at all; some private-university-run libraries do require ID for entry; but most city-, county- and state-run libraries still allow free entry without having to identify yourself in any way. This sometimes even extends to making study-room reservations (can often be made under any name) and anonymous computer use, too.
It’s not working flawlessly, the old crypto protocols and algorithms it uses have been recalled like a Takata airbag, and you’re holding on because it hasn’t blown up in your face yet.
(my emphasis)
So you just use this phone to access Wikipedia, and use it for nothing else?
If so, that’s unfortunate, but your ire should be directed to the smartphone OS vendor for not providing needed updates to encryption protocols.
I think it’s pretty clear that the user does not need encryption in this use-case, so, I don’t see any reason to complain to the OS vendor about encryption when you don’t want to be using any encryption in the first place. Like, seriously, what sort of arguments are these? Maybe it’s time to let go of the politics in tech, and provide technical solutions to technical problems?
As per my comment, I do believe that the authentication provisions of TLS are applicable to Wikipedia.
Besides, the absolute outrage if WP had not offered HTTPS would be way bigger than now.
I find the connection to Postel’s law only weak here, but in any case: This is the worst argument you could make.
It’s pretty much consensus among security professionals these days that Postel’s law is a really bad idea: https://tools.ietf.org/html/draft-iab-protocol-maintenance-04
I don’t think what passes for “postel’s law” is what Postel meant, anyway.
AFAICT, Postel wasn’t thinking about violations at all, he was thinking about border conditions etc. He was the RFC editor, he didn’t want anyone to ignore the RFCs, he wanted them to be simple and easy to read. So he wrote “where the maximum line length is 65” and meant 65. He omitted “plus CRLF” or “including CRLF” because too many dotted i’s makes the prose heavy, so you ought to be liberal in what you accept and conservative in what you generate. But when he wrote 65, he didn’t intend the readers to inter “accept lines as long as RAM will allow”.
https://rant.gulbrandsen.priv.no/postel-principle is the same argument, perhaps better put.
IMO this is another case of someone wise saying something wise, being misunderstood, and the misunderstanding being a great deal less wise.
I can’t really understand advocating laws around protocols except for “the protocol is the law”. Maybe you had to be there at the time.
As I understand it, they’re protecting one set of users from a class of attack by disabling support for some crypto methods. That seems very far from “absolutely no technical reason”.
As for HTTP, if that were available, countries like Turkey would be able to block Wikipedia on a per-particle basis, and/or surveil its citizens on a per-article basis. With HTTPS-only, such countries have to open/close Wikipedia in toto, and cannot surveil page-level details. Is that “no reason”?
I don’t understand why people think this is an acceptable argument for blocking HTTP. It reminds me of that jealous spouse scenario where someone promises to inflict harm, either to themselves or to their partner, should the partner decide to leave the relationship. “I’ll do harm if you censor me!”
So, Turkey wants to block Wikipedia on a per-article business? That’s their decision, and they’ll go about it one way or another, I’m sure the politicians they don’t particularly care about the tech involved anyways (and again, it’s trivial for any determined entity to block port 443, and do a masquerade proxy on port 80, and if this is done on all internet connections within the country, it’ll work rather flawlessly, and noone would know any better). So, it’s basically hardly a deterrent for Turkey anyways. Why are you waging your regime-change wars on my behalf?
Well, Wikipedia is a political project, in much the same way that Stack Overflow is. The people who write have opinions on whether their writings should be available to people who want to read.
You may not care particularly whether all of or just some of the information on either Wikipedia or SO are available to all Turks, but the people who wrote that care more, of course. They wouldn’t spend time writing if they didn’t care, right? To these people, wanting to suppress information about the Turkish genocide of 1915 is an affront.
So moving to HTTPS makes sense to them. That way, the Turkish government has to choose between
The Wikipedians are betting that the second option is unpopular with the Turks.
It’s inconvenient for old ipad users, but if you ask the people who spend time writing, I’m sure they’ll say that being able to read about your country’s genocide at all is vastly more important than being able to read using old ipads.
I can think of several reasons:
So what’s to stop a totalitarian regime from doing the following?
The difficulty is to setup/enrole TotallyLegitCA. How do you do that? If TotallyLegitCA is public, the transparency log will quickly reveal what they are doing. The only way to pull that seems to force people to have your CA installed, like Kazakhstan is doing.
We’re talking about a totalitarian regime (or you know, your standard corporation who install their own CA in the browser).
That’s actually incorrect. There are various technical reasons. But also remember that they need to operate on a vast scale as a non-profit. This is hard.
Here are some technical reasons. I’m sure others will chime in as there are likely many more.
providing a read-only version without login over HTTP shouldn’t really add any new code except they’d be on a HTTP-2-only webserver if I’m not mistaken.
There are arguments for an inverse-postel’s law given in https://m.youtube.com/watch?v=_mE_JmwFi1Y
But I hear all the time that I must ensure my personal site uses HTTPS and that soon browsers will refuse to connect to “insecure” sites. Isn’t this a good thing Wikipedia is doing? /s
Edit also see this discussion: https://lobste.rs/s/xltmol/this_page_is_designed_last#c_keojc6
I have HTTPS on my completely static website mostly so that no one asks why I don’t have HTTPS, but on the other hand, the “completely static” part is only relevant as long as there are only Eves in the middle and no Mallories.
If serving everything over HTTPS will make the life of ISPs injecting ads and similar entities harder, it’s a good thing, until there’s a legal rather than technical solution to that.
I actually think that HTTPS is reasonable for Wikipedia, if for nothing else to hinder 3rd parties for capture your embarrassing edits to “MLP: FIM erotica” and tracing it to back to you. For a static, read-only site it just adds cost and/or a potential point of failure.
dmbaturin just said what the value add is. HTTPS prevents third parties from modifying the content of your static site.
Hmm, those sandbox embeds aren’t loading for me. (Firefox 68.)
For some reason none loaded for me with newest Firefox, even when I unblocked everything relevant from uMatrix. Chromium with everything unblocked shown only the first one (???). Only chromium with all disabled extensions worked as expected.
Shame author didn’t add simple img, imagine how much more portable and reliable that would be.
Yes, this is also a good page about the issue, and related issues:
https://dwheeler.com/essays/filenames-in-shell.html
As some might know, https://www.oilshell.org/ is very much about proper parsing and string safety.
Code and data are kept separate. For example:
export -fmisfeature that serialized a function to a string!However I don’t really have answer to this flag vs. arg issue (flag being code, and arg being data), other than “use
--everywhere”.Relatedly, in Oil,
echois no longer a special case because it supports--(if you usebin/oilrather thanbin/osh). So you can doecho -- $x, which is a safe version ofecho $x.printfto echo an arbitrary string, although not many shell scripts follow that discipline!echo -- $xin POSIX shell because that would output 2 dashes also :)If anyone has a better idea than “use
--everywhere”, let me know :)I guess you write shell in two modes: the quick and dirty throwaway, and then Oil is supposed to allow you to upgrade that to a “real” program. So there could be a lint tool that warns about
--or just auto-refactors the script for you. Or maybe there is some option that breaks with shell more radically.The problem is that when you ask yourself “does this do exactly what I want it to?”, you don’t have the imagination to come up with something like “filenames that look like flags will be interpreted as one.”
Someone who would make a safer, saner shell to write programs with would be a hero.
I started some work on a safer, more explicit shell, realizing that the fundamental offering of any shell is the ability to just type the name of a function and arguments unadorned. I called this “quotation”.
However, after thinking about it more, I realized that no solution will solve the dissonance inherent to any language like that. You will always be tripping up and compromising over where something is or is not quoted. All templating languages have this problem.
Instead, I’m currently of the opinion that an approach more like PowerShell, in which you call functions, not write the names of programs and arguments as text, is the right way forward. This removes the problem of quotation. The downside to this approach is that it requires work to produce the APIs. It’s fine if you have a large standard library, as PowerShell does, but being able to pull a binary off the shelf e.g. one written in e.g. Python or C should still be natural.
The missing part therefore, in my opinion, is that programs (in any system be it Linux, OS X, Windows, BSD), ought to be accompanied by a schema (could be in JSON, doesn’t matter), let’s say
gitandgit.schema, which can be interpreted offline or “cold” – without running the program (very important) –in order to know (1) arguments/flags the program accepts (as commands or switches), (2) the types/formats of those inputs, (3) possibly list the side-effects of those commands.This allows a shell or IDE to provide a very strong completion and type-checking checking story, and to provide it out of the box. A projectional editor would be satisfying here, too (even something as simple as CLIM’s listener).
When downloading a random binary online, you could additionally download the schema for it. The schema file itself can contain a SHA256 of the binary that it’s talking about, to avoid accidental misuse. Currently if you want completion for an exe, you have to generate some bash. So it’s clear that the need is already there, it’s just implemented in a poorly done way.
The upside to this approach is that it’s additive, no one has to change their existing software. Additionally, it’s easy to produce for old software; You can make a parser for
--helpor man pages to generate a “best guess” schema for a program. The reason you wouldn’t put this into a shell by default would be because some programs don’t accept--helpand/or they run side effects and delete things. Existing opt parser libraries can generate such schemas, just like some of them can generate bash at the moment.Another upside is that it can simply be a standard/RFC published freely, just like JSON.
I haven’t moved forward on this schema idea yet, because it’s hard to research existing solutions (hard to google for). It would be an FFI standard but instead of calling C functions you’re calling processes.
Yeah I agree with your diagnosis of the issue. You need a schema and a parser to go along with every command.
I say a parser because in general every command has its own syntax, in addition to semantics (the types of the arguments). Some commands recognize fully recursive languages in their args like
find,test, andexpr. (Although of course there are common conventions.)It relates pretty strongly to this idea about shell-agnostic autocompletion we’ve been kicking around. Every command needs a parser and a schema there too.
https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal
And yes it’s nice to have the property that it’s a pure addition. It’s basically like TypeScript or MyPy (which I’m using in Oil). They handle a lot of messiness to provide you with a smooth upgrade path.
If you’d like to pursue these ideas I think Oil will help because it has an accurate and flexible shell parser :) I break shell completion into two parts: (1) completing the shell language, then (2) completing the language of each individual tool. That’s the only way to do it correctly AFAICT. bash conflates the two problems which leads to a lot of upstream hacks.
That’s really interesting, especially thanks for the dwheeler page.
I solve shell problems by using shellcheck all the time, it catches most mistakes one can make and it itegrates nicely into existing editors.
Oil looks really interesting and is certainly better than the shell+shellcheck combo, but I don’t think I want to write everything in new syntax that is not universal and might mean I’ll have to rewrite it in bash later anyway.
Well Oil is very bash compatible – it runs your existing shell/bash scripts. You can use it as another correctness tool on top of ShellCheck. It catches problems at runtime in addition at parse time.
Example: Oil’s Stricter Semantics Solve Real Problems
There are a bunch of other examples I should write about too.
If your shell scripts are used for years, and keep getting enhanced, chances are that you will run into the limitations of bash. And you won’t want to rewrite the whole thing in another language. That’s what Oil is for :) It’s fine to stick with bash now, but as you use shell more and more, you will run into a lot of limitations.
I disagree with all items marked as “#WRONG” in this site. Clean globbing is too powerful and beautiful to complexify with these mundane problems. Filenames are variable names, not data, and you get to chose them. What is actually wrong is the existence of files with crazy names. This should be solved at the filesystem level by disallowing e.g. the space character on a filename (no need to bother the user, it could be translated simply into a non-breaking space character, and nobody would notice except shell scripts).
The problem is you don’t always; if I write a script and you run it, then the author of the script didn’t choose the filenames.
That is more or less the same conclusion the author makes: https://dwheeler.com/essays/fixing-unix-linux-filenames.html
An environment variable indicating the providence of each argument, and sending patches to
getopt,argp, coreutils, bash, and your favorite unix programs.If you want to write to for example stderr from laguage which doesn’t support stderr redirection (awk), you can instead write it to /dev/stderr and it will redirect to stderr automagically!
This also means you don’t have to remember the awful sh syntax for redirecting echo output stream to 2nd file descriptor, you can just do
echo 'error' > /dev/stderr.