1. 3

    In market share study results announced in May 2018, KaiOS beat Apple’s iOS for second place in India, while Android dominates with 71%,

    Impressive! I’d love to see them succeed where Firefox OS failed! (side note: This is a Firefox OS fork).

    1. 1

      It would be especially impressive if they achieve that success with Google’s money, while (it seems) competing against Google’s established mobile OS.

    1. 4

      Interesting. Could this have a damaging effect on open, free software available to us?

      1. 9

        Why would it have a damaging effect? Are you supposing that there are a lot of useful FOSS projects out there that are public only because their owners didn’t want to pay GitHub five dollars a month?

        1. 6

          They already push the agenda that you probably don’t want your side project or interview assessment to be public. I think that there could be projects locked away as they are not ‘ready’. Hackers won’t be able to learn from things they cannot see!

          Slippery slope argument in there somewhere… I’m more interested to see how they tie this product in to LinkedIn.

          1. 2

            From what I’ve seen of GitHub, that seems like a fairly reasonable supposition, especially if you leave out the FOSS ideology and just consider publicly accessible projects. More user control over privacy will definitely (as in “by definition”) result in less public access.

            1. 2

              Are you supposing that there are a lot of useful FOSS projects out there that are public only because their owners didn’t want to pay GitHub five dollars a month?

              Actually, yeah. A lot of cool repositories have a single commit, and were pushed to one time anywhere from 1-7 years ago. They’re super useful, but I can see people making that kind of repository private now that it’s free.

          1. 3

            His websites load instantly! That’s so cool. I love this guy. Maybe I should learn how to actually use html/css/js instead of base.css and stolen javascript snippets. Great article.

            1. 3

              Although I can’t find it, there was an article that visualized it as a pyramid. Most of it was HTML representing the content. A small part was CSS representing layout. The tiny tip on top was Javascript doing whatever wasn’t possible in HTML or CSS. If it was possible, use no Javascript at all. Then, you also use compression not just to boost transfer speed: to be considerate to the many, many folks on dial-up (slow) and mobile (data charges).

                1. 1

                  That’s it! Thanks! Also, I recommend his articles on the subject because they’re as entertaining as they are informative.

            1. 32

              This is a pretty spot-on example of why I stopped using Twitter; the first reply is someone telling Joe Armstrong that he’s never coded beyond hello world.

              1. 9

                What our Twitter reply is saying is of course Joe Armstrong is past Hello World, he’s saying that it’s impossible to be without dependencies and this is just “old guy” advice. Joe Armstrong is choosing to have his cake and eat it too chastising dependencies while the real world revolves around them.

                I think this example is anecdotal to Mr. Armstrong and that as a practice I’d rather people use their judgement. The argument stinks to me of handwaving and of the old class of developers who’d rather not learn new stuff.

                If you’re Google you can vert. up all you like. Don’t act like any old place but large ones can put up with an engineer trying to argue this in reality.

                1. 6

                  My guess is that most languages wrap system IO routines, which means that even a “hello world” isn’t truly dependency free.

                  Now, if Chuck Moore was making this statement, I’d fully believe it. He made a career out of building things to the exact spec necessary—the only dependencies were often encoded in the CPU (and other hardware)’s silicon.

                  1. 6

                    That’s not all! In his later years, Chuck Moore build an entire VLSI CAD system in Forth so he could design his own CPUs. Some of these were even fabbed. There might be a few left for sale.

                    Sounds extreme, but it’s true. That’s why we call him “Chuck Mooris”.

                    1. 5

                      He’s the kind of programmer that “full stack” should refer to. An elite level where you can do everything from apps to full-custom silicon.

                      1. 4

                        @technomancy and I have a t-shirt that’s been in the works for a while. It’s totally my fault that it isn’t currently available (as it’s been done for ages), but it celebrates Chuck in a “witty” way. This might just be the nudge that gets me to follow through with the rest. ;)

                  2. 14

                    I’m glad that person made that comment, actually, even if it was a bit rude.

                    When I first read Joe Armstrong’s statement “Of course - I try to write code with zero dependencies - code I wrote 25 years ago with zero dependencies still works today.”, my first thought was something along the lines of “hm, yeah, that’s sensible advice from Joe Armstrong, and he’s a well-known name he probably knows what he’s talking about”.

                    Then some random guy told him publicly that he was wrong in an un-nuanced way.

                    And I thought to myself, well, hm, that’s a rude way to put it - but yes, actually, “A software system simply cannot be built without components based on other components whether they are soft or hard dependencies.” has some merit to it as well. Maybe Joe Armstrong isn’t being completely literal when he says that’s he’s written telephony code for Ericsson base stations that’s run for 25 years with absolutely zero dependencies other than the compiler. Maybe some dependencies are getting “snuck in” via the compiler, or the hardware drivers for those base stations, or in changed hardware itself, even if the Erlang code running in the VM has been around for 25 years unchanged. Maybe the code he’s talking about that’s remained unchanged is itself a dependency for other code that changes more frequently - certainly there have been plenty of updates of the capabilities of the phone system over the past 25 years, and I doubt that Joe Armstrong’s quarter-century-old code now needs absolutely no additional augmentation now. Maybe telephony is a special domain, and eschewing dependencies is a good fit for solving telephony problems but a bad fit for, say, creating new pieces of software that do things undreamt-of 25 years ago.

                    And none of this is to say that Armstrong’s points about dependencies increasing fragility and causing code bloat are invalid. He’s 100% right to point out that it’s bad to add 200K of CSS in order to make a button look pretty. But maybe doing that is the least bad of several trade-offs (if you couldn’t use bloated CSS libraries, maybe web design would look terrible, or maybe a bunch of useful websites would just never have been built). My own opinion is that things like 200K CSS files for one button are a local but not global minimum - the entire web is highly path-dependent, and relies on a bunch of inefficient hacks to make it usable as a software distribution platform, but there’s also a huge breaking-backwards-compatibility cost in moving away from the web and towards a better software ecosystem that does the same things as the web but doesn’t encourage 200K CSS libraries. Maybe webassembly is a small step towards climbing out of that local minimum to something globally better.

                    1. 9

                      Should we really need a prompt to think that way, though? Whenever I hear anything I imagine the counter position as a test. It’s a way of getting at the nuance. I like to think that it’s something we should foster as an engineering mindset.

                      1. 11

                        I recently learned about the principle of charity: https://en.m.wikipedia.org/wiki/Principle_of_charity. Most online discussion and tools could use a lot more of it.

                      2. 9

                        The argument would be helped if the original responder didn’t descend to ‘ball-licking’ level: https://twitter.com/ethericlights/status/1075531837286555648

                        1. 7

                          “zero dependencies” but he does have the OTP system to support all this. Not like he’s rewriting the TCP socket logic in every program.

                          I think this is an argument for the seaparation of thought process between standard libraries and external dependencies. Standard libraries have very large backwards compat requirements in general, so relying on that is really not an issue. So having a standard library that is vast and covers a lot of non-controversial but tedious stuff (building HTTP requests, parsing JSON if that’s a thing, listening to sockets) can mean that you won’t feel the need to pull in external dependencies for the most part.

                          And what he said about vendoring in stuff and ripping out what you don’t need is… pretty good advice for lots of usages! Though libraries get added improvements, if you vendor in the libraries you can just change the stuff up to your specific use cases without much issue.

                          But yeah “I write a bunch of programs with no dependencies” kinda precludes a lot of stuff if you were doing something in C (for example). But zero-dep Python/Erlang(OTP) seems really doable in general

                          1. 3

                            Broadly agree, but zero-dependency C is often pretty doable too IME, for the kind of things where C shines. And even if you do need dependencies, well-known C libraries tend to be more stable than e.g. the nodejs ecosystem.

                      1. 7

                        I thought the conspiracy theory folks were wrong. It’s looking like they were right. Google is indeed doing some shady stuff but I still think the outrage is overblown. It’s a browser engine and Microsoft engineers have the skill set to fork it at any point down the line. In the short term the average user gets better compatibility which seems like a win overall even if the diversity proponents are a little upset.

                        1. 10

                          I thought the conspiracy theory folks were wrong. It’s looking like they were right. Google is indeed doing some shady stuff

                          If it’s an organization, you should always look at their incentives to know whether they have a high likelihood of going bad. Google was a for-profit companies aiming for IPO. Their model was collecting info on people (aka surveillance company). These are all incentives for them to do shady stuff. Even if they want Don’t Be Evil, the owners typically loose a lot of control over whether they do that after they IPO. That’s because boards and shareholders that want numbers to go up are in control. After IPO’s, decent companies start becoming more evil most of the time since evil is required to always make specific numbers go up or down. Bad incentives.

                          It’s why I push public-benefit companies, non-profits, foundations, and coops here as the best structures to use for morally-focused businesses. There’s bad things that can still happen in these models. They just naturally push organizations’ actions in less-evil directions than publicly-traded, for-profit companies or VC companies trying to become them. I strongly advise against paying for or contributing to products of the latter unless protections are built-in for the users with regards to lock-in and their data. An example would be core product open-sourced with a patent grant.

                          1. 9

                            Capitalism (or if you prefer, economics) isn’t a “conspiracy theory”. Neither is rudimentary business strategy. It’s amusing to me how many smart, competent, highly educated technical people fail so completely to understand these things, and come up with all kinds of fanciful stories to bridge the gap. Stories about the role and purpose of the W3C, for instance.

                            Having read all these hand-wringy threads about implementation diversity in the wake of this EdgeHTML move, I wonder how many would complain about, say, the lack of a competitor to the Linux kernel? There’s only one kernel, it’s financially supported by numerous mutually distrustful big businesses and used by nearly everybody, its arbitrary decisions about its API are de-facto hard standards… and yet I don’t hear much wailing and gnashing, even from the BSD folks. How is the linux kernel different than Chromium?

                            1. 16

                              While I actually am concerned about a lack of diversity in server-side infrastructure, the Linux kernel benefits, as it were, from fragmentation.

                              There’s only one kernel

                              This simply isn’t true. There’s only one development effort to contribute to the kernel. There is, on the other hand, many branches of the kernel tuned to different needs. As somebody who spent his entire day at work today mixing and matching different kernel variants and kernel modules to finally get something to work, I’m painfully aware of the fragmentation.

                              There’s another big difference, though, and that’s in leadership. Chromium is run by Google. It’s open source, sure, but if you want your commits into Chromium, it’s gonna go through Google. The documentation for how to contribute is littered with Google-specific terminology, down to including the special internal “go” links that only Google employees can use.

                              Linux is run by a non-profit. Sure, they take money from big companies. And yes, money can certainly be a corrupting influence. But because Linux is developed in public, a great deal of that corruption can be called out before it escalates. There have been more than a few developer holy wars over perceived corruption in the Linux kernel, down to allowing it to be “tainted” with closed source drivers. The GPL and the underlying philosophy of free software helps prevent and manage those kinds of attacks against the organization. Also, Linux takes money from multiple companies, many of which are in competition with each other. It is in Linux’s best interest to not provide competitive leverage to any singular entity, and instead focus on being the best OS it can be.

                              1. 3

                                Performance tuning is qualitatively different than ABI compatibility. Otherwise, I think you make some great points. Thanks!

                              2. 7

                                If there is an internal memo at Google along the lines of “try to break the other web browsers’ perf as much as possible” that is not “rudimentary business strategy”, it’s “ground for anti-trust action”.

                                It’s as good of a strategy as helping the Malaysian PM launder money and getting a 10% cut (which… hey might still pay off)

                                1. 5

                                  Main difference is that there are many interoperable implementations of *nix/SUS/POSIX libc/syscall parts and glibc+Linux is only one. A very popular one, but certainly not the only. Software that runs on all (or most) *nix variants is incredibly common, and when something is gratuitously incompatible (by being glibc+Linux or MacOS only) you do hear the others complain.

                                  1. 2

                                    Software that runs on all (or most) *nix variants is incredibly common

                                    If by “runs on” you mean “can be ported to and recompiled without major effort”, then I agree, and you’re absolutely right to point out the other parts of the POSIX and libc ecosystem that makes this possible. But I can’t think of any software that’s binary compatible between different POSIX-ish OSs. I doubt that’s even possible.

                                    On the other side of the analogy, in fairness, complex commerical web apps have long supported various incompatible quirks of multiple vendor’s browsers.

                                    1. 6

                                      Multiple OSs, including Windows, can run unmodified Linux binaries.

                                  2. 4

                                    How is the linux kernel different than Chromium?

                                    As you just said it,

                                    financially supported by numerous mutually distrustful big businesses

                                    There’s no one company making decisions about the kernel. That’s the difference.

                                    1. 4

                                      There’s no one company making decisions about the kernel. That’s the difference.

                                      Here comes fuchsia and Google’s money :/

                                    2. 1

                                      I am disgusted with the Linux monoculture (and the Linux kernel in general), even more so than with the Chrome monoculture. But that fight was fought a couple decades ago, it’s kinda late to be complaining about it. These complaints won’t be heard, and even if they are heard, nobody cares. The few who care are hardly enough to make a difference. Yes we have the BSDs (and I use one) and they’re in a minority position, kinda like Firefox…

                                      1. 2

                                        How much of a monoculture is Linux, really? Every distro tweaks the kernel at least to some extent, there are a lot of patch sets for it in the open, and if you install a distro you get to choose your tools from the window manager onwards.

                                        The corporatization of Linux is IMO problematic. Linus hasn’t sent that many angry emails percentually, but they make the headlines every time, so my conspiracy theory is that the corporations that paid big bucks for board seats on the Foundation bullied him to take his break.

                                        We know that some kernel decisions have been made in the interest of corporations that employ maintainers, so this could be the tip of an iceberg.

                                        Like the old Finnish saying “you sing his songs whose bread you eat”.

                                    3. 3

                                      It’s a browser engine and Microsoft engineers have the skill set to fork it at any point down the line.

                                      I think this is true. If Google screws us over with Chrome, we can switch to Firefox, Vivaldi, Opera, Brave etc and still have an acceptable computing experience.

                                      The real concerns for technological freedom today are Google’s web application dominance and hardware dominance from Intel. It would be very difficult to get a usable phone or personal server or navigation software etc without the blessing of Google and Intel. This is where we need more alternatives and more open systems.

                                      Right now if Google or Intel wants to, they can make your life really hard.

                                      1. 8

                                        Do note that all but Firefox are somewhat controlled by Google.

                                        Chrome would probably have been easier to subvert if it wasn’t open source; now it’s a kind of cancer in most “alternative” browsers.

                                        1. 5

                                          I don’t know. MIPS is open sourcing their hardware and there’s also RISC-V. I think the issue is that as programmers and engineers we don’t collectively have the willpower to make these big organizations behave because defecting is advantageous. Join the union and have moral superiority or be a mercenary and get showered with cash. Right now everyone chooses cash and as long as this is the case large corporations will continue to press their advantage.

                                          1. 9

                                            “Join the union and have moral superiority or be a mercenary and get showered with cash. Right now everyone chooses cash and as long as this is the case large corporations will continue to press their advantage.”

                                            Boom. You nailed it! I’ve been calling it out in threads on politics and business practices. Most of the time, people that say they’re about specific things will ignore them for money or try to rationalize how supporting it is good due to other benefits they can achieve within the corruption. Human nature. You’re also bringing in organizations representing developers to get better pay, benefits, and so on. Developers are ignoring doing that more than creatives in some other fields.

                                            1. 3

                                              Yup. I’m not saying becoming organized will solve all problems. At the end of the day all I want is ethics and professional codes of conduct that have some teeth. But I think the game is rigged against this happening.

                                            2. 2

                                              I don’t think RISC-V is ready for general purpose use. Some CPUs have been manufactured, but it would be difficult to buy a laptop or phone that carries one. I also think that manufacturing options are too limited. Acceptable CPUs can come from maybe Intel and TSMC and who knows what code/sub-sytems they insert into those.

                                              This area needs to be more like LibreOffice vs Microsoft Office vs Google Docs vs others on Linux vs Windows vs MacOS vs others

                                            3. 2

                                              They already are screwing us over with chrome, this occurrence is evidence of that.

                                          1. 38

                                            Wow. Microsoft engineer complains about “some seemingly anti-competitive practices done by Google’s Chrome team”. Now that is some piquant irony.

                                            Also, the page’s YouTube video appears to be blocked. Icing on the cake?

                                            1. 37

                                              …one of the reasons we decided to end EdgeHTML was because Google kept making changes to its sites that broke other browsers, and we couldn’t keep up…

                                              I can appreciate the shadenfreude of Microsoft’s new position, but this is a pretty legitimate concern. Especially if Google is/was doing that intentionally. What we need is a good incentive for Google to care about web standards and performance in non-Chrome browsers, but this move by Microsoft only drives things in the opposite direction.

                                              1. 12

                                                I don’t know if it’s intentional or not, but I am almost never able to complete reCaptchas in Firefox, it just keeps popping up ridiculous ones, like traffic lights that are on the border of three squares, and it keeps popping the same unsolvable ones for 2-3 minutes until I get tired/locked out of it and just use Chrome to log in, where somehow I always get sane ones and it lets me in first try. Anyone had the same?

                                                This video sums it up very well, although not Firefox specific: https://www.youtube.com/watch?v=zGW7TRtcDeQ

                                                (Btw I don’t use Tor, or public VPNs or any of the like.)

                                                1. 4

                                                  This is known to happen: Blocking via an unsolvable CAPTCHA.

                                                  1. 1

                                                    Ha! Thanks for this, I won’t keep trying anymore :)

                                                2. 17

                                                  Especially if Google is/was doing that intentionally.

                                                  I disagree that intention has anything to do with it. We have to judge these kinds of situations by the effect they have on the web, not on good feelings.

                                                  1. 7

                                                    Spot on. Intent only matters when it is ill. Even if not intended, the outcome is what matters materially.

                                                    1. 6

                                                      One reason intention matters: if the intention is to handicap edge, then it’s probably not serving some other purpose that’s good for all of us. If handicapping edge is a side-effect of some real benefit, that’s just a fact about the complexity of the web (it might still be a bad decision, but there are trade-offs involved).

                                                    2. 7

                                                      OK, let’s put aside the schadenfreude as best we can and examine the consequences. I think it’s fair to assume, for the sake of argument, that Alphabet Inc absolutely will do everything in its power, dirty tricks included, to derive business value from it’s pseudo-monopolist position. If Microsoft were to dig in their heels and ship a default browser for their desktop OS that didn’t play YouTube videos as well as Chrome does, would that harm Alphabet, or just Microsoft at this point?

                                                      I don’t really understand your talk of “a good incentive”. Think of it this way: what incentive did Google, an advertising company, ever have to build and support a web browser in the first place? How did this browser come to its current position of dominance?

                                                      1. 15

                                                        Google built a web browser because Microsoft won the browser wars and did nothing with IE for 10 years.

                                                        Their entire suite of products were web based and their ability to innovate with those products was severely hampered by an inadequate platform.

                                                        https://googleblog.blogspot.com/2008/09/fresh-take-on-browser.html

                                                        Chrome was revolutionary when it was released and many of the web technologies we take for granted today could never have happened without it.

                                                        I’m not entirely thrilled with everything it led too but whatever their motives now, Google had good reasons to build Chrome in the first place.

                                                        1. 23

                                                          I’m sure whichever visionaries were at Google at that point had great reasons to build Chrome. But Google isn’t the same company anymore, and Chrome doesn’t mean what it once meant.

                                                          “You could not step twice into the same river.” —Heraclitus

                                                          1. 11

                                                            That’s certainly ONE factor. The other is that Chrome by default makes “address bar” and “search bar” the same thing, and sends everything you type into the search bar to Google.

                                                            Same as Google Maps, or Android as a whole. I often navigate with Google Maps while driving. The implication is that Google knows where I live, where I work, where I go for vacation, where I eat, where I shop. This information has a monetary value.

                                                            If there is something Google does that is not designed to collect information on it’s users that can be turned into ad revenue, that something will eventually be shut down.

                                                            1. 9

                                                              “This information has a monetary value.”

                                                              Exactly. They are trying to build accurate profiles of every aspect of people and businesses’ existences. Their revenue per user can go up as they collect more information for their profiles. That gives them an incentive to build new products that collect more data, always by default. Facebook does same thing. Revenue per user climbed for them year after year, too. I’m not sure where the numbers are currently at for these companies, though.

                                                            2. 8

                                                              Google built a web browser because Microsoft won the browser wars and did nothing with IE for 10 years.

                                                              No, that was Mozilla. They together with Opera were fighting IE’s stagnation and by 2008 achieved ~30% share which arguably made Microsoft notice. Chrome was entering the world which already was multi-browser at that point.

                                                              Also, business-wise Google needed Chrome as a distribution engine, it has nothing to do with fighting browser wars purely for the sake of users.

                                                              1. 1

                                                                I’m not entirely sure what you mean by a distribution engine. For ads? Or for software?

                                                                I think business motives are extremely hard to discern from actions. I think you could make the argument that Google has been trying for years to diversify their business, mostly unsuccessfully, and around 2008 maybe they envisioned office software (spreadsheets, document processing, etc) as the next big thing. GMail was a surprise hit, and maybe they thought they could overthrow Microsoft’s dominance in the field. But they weren’t about to start building desktop software, so they needed a better browser to do it.

                                                                Or maybe they built it so that Google would be the default search engine for everyone so they could serve more ads?

                                                                Or maybe some engineers at Google really were interested in improving performance and security, built a demo of it, and managed to convince enough people to actually see it through?

                                                                I realize the last suggestion may sound helplessly naive, but having worked as an engineer in a company where I had a lot of say in what got worked on, my motives were often pretty far afield of any pure business motive. I got my paycheck regardless, and sometimes I fixed a bug or made something faster because it annoyed me. I imagine there are thousands of employees at Google doing the same thing every day.

                                                                Regardless, the fact remains that the technology they built for Chrome has significantly improved the user experience. The reason Chrome is now so dominant is because it was better. Much better when compared to something like IE6.

                                                                And even ChromeOS is better than the low-price computing it competes with. Do you remember eMachines? They were riddled with junk software and viruses rendering them almost completely useless. A 100$ Chromebook is such a breath of fresh air compared to that experience.

                                                                I realize there’s a cost to this, and I get why there’s a lot of bad press about Google, but I don’t think we need to rewrite history about it. I think we’re all better off with Google having created Chrome (even if I don’t agree with many of the things they’re doing now).

                                                                1. 5

                                                                  The reason Chrome is now so dominant is because it was better.

                                                                  There are two reasons why Chrome became so dominant:

                                                                  • Google makes deals with OEMs to ship Chrome by default on the new desktops and laptops. Microsoft cannot stop them because of historical antitrust regulations.

                                                                  • Google advertised Chrome on their search page (which happens to be the most popular web page in the world) whenever someone using another browser visited it. It looks like they’ve stopped, though, since I just tried searching with Google from Firefox and didn’t get a pop-up.

                                                            3. 3

                                                              The incentive to play fair would come from Google not wanting to lose the potential ad revenue from users of non-Chrome browsers due to them deliberately sabotaging their own products in those browsers. Not trying to imply that EdgeHTML was the solution to that problem or that it would somehow be in Microsoft’s best interest to stick with it, just that its loss is further cementing Google’s grip on the web and that’s a bad thing.

                                                              1. 3

                                                                All the user knows is “browser A doesn’t seem to play videos as good as browser B”. In general they can’t even distinguish server from client technologies. All they can do about it, individually, is switch browsers.

                                                                Now that Alphabet has cornered the market, their strategy should be obvious. It’s the same as Microsoft’s was during the Browser Wars. The difference is, Alphabet made it to the end-game.

                                                                1. 1

                                                                  The end game being anti-trust action? I’m not following your line of argument. Are you examining that particular consequence?

                                                                  1. 2

                                                                    The antitrust case against Microsoft ended up with not much happening, and that was 18 years ago. Do you have much confidence that there is likely to be an effective antitrust action against Google?

                                                                    1. 1

                                                                      I’m not the one making a case here.

                                                                      Your interpretation[1][2] of how a single historical case went doesn’t change the fact that antitrust action is bad for a company’s long-term prospects and short-term stock price. The latter should directly matter to current leadership. Companies spend a reasonable amount of time trying to not appear anti-competitive. @minimax is utterly ignoring that consequence of “dirty tricks”.

                                                                      [1] https://www.nytimes.com/2018/05/18/opinion/microsoft-antitrust-case.html illustrates the opposite perception. [2] https://arstechnica.com/tech-policy/2010/09/the-eternal-antitrust-case-microsoft-versus-the-world is more balanced, and points out the effect of the lawsuit on Microsoft PR, leadership focus and product quality.

                                                                  2. 1

                                                                    What makes Chrome’s position more of an end-game than what IE had in the early 2000s?

                                                                    1. 4

                                                                      You’re looking at it wrong. The question you really need to consider is:

                                                                      What makes Google’s position more of an end-game than what Microsoft had in the early 2000s?

                                                                      Microsoft was the dominant OS player, but the Internet itself was undergoing incredible growth. What’s more, no one existed solely within what’s Microsoft provided.

                                                                      Today, the Internet is essentially the OS for many (most?). People exist in a fully vertically integrated world built by Google - operating system, data stored in their cloud, documents written on their editor and emails sent through their plumbing… all of it run by the worlds most profitable advertising company, who just built themselves mountains of data to mine for better advertisements.

                                                                      Microsoft in the 00’s could only dream of having that.

                                                                      1. 4

                                                                        Your assessment of Google today strikes me as not completely unreasonable, although it does neglect the fact that only a small fraction of Internet users live so completely in Google’s stack; I suspect far more people just use Android and Chrome and YouTube on a daily basis but don’t really use Gmail or GSuite (Docs, etc.) very frequently, instead relying on WhatsApp and Instagram a lot more.

                                                                        And back in the 2000s there were definitely a large group of people who just used Windows, IE, Outlook, Hotmail, MSN & MS Office to do the vast majority of their computing. SO it’s not as different as you seem to believe. Except now there are viable competitors to Google in the form of Facebook & Apple in a way that nobody competed with MS back then.

                                                                        1. 2

                                                                          SO it’s not as different as you seem to believe.

                                                                          It’s incredibly different.

                                                                          When I used IE5 Microsoft’s tactic was to bundle it with Windows and use Microsoft-specific APIs to boost its performance, killing Netscape. If I used Chrome today, I’d find dark UI patterns are used to ensure my data is captured.

                                                                          Similarly, Office/Outlook/Windows in 2000 didn’t mine the files I was working on to enrich an advertising profile that would follow me across the internet. If memory serves, while Hotmail did serve advertisements, they were based on banner advertisements / newsletters generated by Microsoft, and not contextually targeted.

                                                                          The real risk here, I believe, is in both the scope and ease of understanding what’s happening today versus what Microsoft did. Microsoft’s approach was to make money by being the only software you run, and they’d use any trick they could to achieve that - patently anticompetitive behavior included.

                                                                          Google, on the other hand… at this point I wonder if they’d care if 90% of the world ran Firefox as long as the default search engine was Google. I think their actions are far more dangerous than those of Microsoft because they are much wider reaching and far more difficult for regulators to dig into.

                                                                          I suspect far more people just use Android and Chrome and YouTube on a daily basis but don’t really use Gmail or GSuite (Docs, etc.) very frequently, instead relying on WhatsApp and Instagram a lot more.

                                                                          Even if we take that as a given, this means most people are sending:

                                                                          • their location
                                                                          • the videos and pictures from their phone’s camera
                                                                          • their search history
                                                                          • a list of content they watched

                                                                          up to Google.

                                                                          1. 1

                                                                            Your assessment that Chrome is only a means to an end, the end being to have people continue using Google’s web search, seems dead on. But then you follow that up with a claim that doesn’t seem to logically follow at all.

                                                                            The reach of Google now relative to Microsoft 15 years ago is lower as a fraction of total users; it only seems higher because the absolute number of total users has grown so much.

                                                                            1. 3

                                                                              Doesn’t this depend on how you define a “user”, though? Google has a grip on search that would be the envy of IBM back in the day. Android is by far the most popular operating system for mobile phones, if not for computing devices writ large. They pay for Mozilla because they can harvest your data through Firefox almost as easily as via Chrome, and they prop up a competitor, in case the US DOJ ever gets their head out of their ass and starts to examine the state of the various markets they play in.

                                                                              1. 2

                                                                                Depends on how narrowly you define “search” too; Do you include all the searches people conduct directly on Amazon or Facebook or Siri?

                                                                              2. 1

                                                                                The reach of Google now relative to Microsoft 15 years ago is lower as a fraction of total users; it only seems higher because the absolute number of total users has grown so much.

                                                                                Android’s global smartphone penetration is at 86% in 2017[1]. And while the “relative reach” might be lower, the absolute impact of the data being Hoovered up is significant. In 2000, annual PC sales hit 130 million per the best figures I could find[2] … that’s less than a tenth of smartphone sales in 2017 alone.

                                                                                What does it matter that Google’s relative reach is lower when they control nearly 9/10 smartphones globally and proudly boast over two billion monthly active devices?

                                                                                1. 1

                                                                                  The level of control isn’t directly comparable. Microsoft sold Windows licenses for giant piles of money while Google licenses only the Play Store and other apps that run on Android. Android in China is a great example of the difference, although I guess Microsoft probably lost revenue (but not control over the UX) there via piracy.

                                                                  3. 1

                                                                    Think of it this way: what incentive did Google, an advertising company, ever have to build and support a web browser in the first place?

                                                                    Is this a real question asked in good faith? Maybe it’s just been a long day but I really can’t tell.

                                                                    1. 2

                                                                      I was going for Socratic. You’re quite welcome to assume anything you like about the goodness of my faith.

                                                                      1. 1

                                                                        Got it - always happy to play along with Mr. Socrates ;) I mostly wanted to be sure I wasn’t just biting on an obvious troll.

                                                                2. 11

                                                                  That’s just a picture of a blocked YouTube video to emphasis their point.

                                                                1. 5

                                                                  I’m not sure that the linux tag is quite warranted here, since the article is about customizing bash on Mac OS.

                                                                  Also, it’s only about bash tricks, and not about which terminal emulators support colorful, double-width unicode characters, and why or why not.

                                                                  1. 3

                                                                    Same for the ‘osdev’ tag, it seems like quite a stretch for that too.

                                                                    1. 1

                                                                      Definitely. I wasn’t sure as well but couldn’t find any other relevant tag. I was hoping for something like “bash” or “shell” or “terminal”

                                                                    2. 1

                                                                      The same config works also on Ubuntu (tested) and I assume in any other bash environment. I only happened to do the screenshots on a Mac. Anyway the intention of the article was to inspire some curiosity on bash customisations. It wasn’t intended to be exhaustive. With that in mind, I am more than happy to accept any suggestion on how to improve the article. Most likely it can use some constructive suggestion :)

                                                                      1. 4

                                                                        The article takes for granted the ability of your font and terminal emulator to display these characters properly. In my experience, that’s the “hard part”. The issues involved are at least worth a mention.

                                                                        1. 1

                                                                          Seems like it only works in terminals built using the VTE library too, which means others (xterm, rxvt, alacritty[?]) don’t support it.

                                                                          1. 1

                                                                            Most definitely. Updating the article right now. I’ll make sure to include this as well. Thank you so much!

                                                                      1. 22

                                                                        This immediately suggests another metric:

                                                                        DRYness = ULOC / SLOC

                                                                        Or if you prefer,

                                                                        WETness = 1 - DRYness

                                                                        (‘Violations of DRY are typically referred to as WET solutions, which is commonly taken to stand for either “write everything twice”, “we enjoy typing” or “waste everyone’s time”. ’)

                                                                        1. 2

                                                                          I was thinking the same. Immediately had a look at my main codebase at my day job. DRYness: 0.44. Subjectively I’d say it’s somewhat WET.

                                                                        1. 2

                                                                          Kay: Yeah. You want to get those from the objects. You want it to be a mini-operating system, and the people who did the browser mistook it as an application. They flunked Operating Systems 101.

                                                                          6 years later and we’re really, really regretting making the browser a mini-operating system.

                                                                          Kay: Well, I’m an old-fashioned guy. And I also happen to believe in history. The lack of interest, the disdain for history is what makes computing not-quite-a-field.

                                                                          […]

                                                                          Kay: No, I didn’t. I mean, I made up the term “objects.” Since we did objects first, there weren’t any objects to radicalize. We started off with that view of objects, which is exactly the same as the view we had of what the Internet had to be, except in software.

                                                                          The Smalltalk-72 book explicitly lists Simula as a “major influence”. Simula had objects.

                                                                          1. 1

                                                                            6 years later and we’re really, really regretting making the browser a mini-operating system.

                                                                            I’d argue that what we’re regretting is making the browser a full sized operating system. (That and trying to draw complicated animated interactive widgets with a system intended to render static rich tech documents.)

                                                                            1. 3

                                                                              Who exactly is the “we” here? The people who wrote, say, NSCA Mosaic? The early commercial browsers over which the “browser wars” were fought? The W3C, WHATWG, and other standards bodies? Present-day Chromium and Firefox developers and product managers? The vast number of (mostly non-technical) people who actually use web browsers? The people who write web applications? Infosec people? I suspect that if either of you were to try to make your gripes more precise, you’d find that they dissolve into a messy tangle of historical contingencies.

                                                                              I won’t claim any “regret” for decisions that I was never a part of. But I’d be quite satisfied with the browser-as-OS paradigm if the browser actually replaced the underlying OS rather than layering more leaky abstractions on top of it. ChromeOS is a step in the right direction. FirefoxOS was too, and I was quite disappointed in Mozilla for dropping it.

                                                                              1. 2

                                                                                “But I’d be quite satisfied with the browser-as-OS paradigm if the browser actually replaced the underlying OS rather than layering more leaky abstractions on top of it. “

                                                                                Best example.

                                                                                1. 1

                                                                                  I suspect that if either of you were to try to make your gripes more precise, you’d find that they dissolve into a messy tangle of historical contingencies.

                                                                                  Everything’s a messy tangle of historical contingencies. Historical contingencies are an explanation but not an excuse.

                                                                                  I don’t think anybody should confuse criticism of a regrettable situation with a search for an easily-identified figure to blame. Generally speaking, the best thing to do about problems is fix them (and figuring out why they’re problems is the first step in that process).

                                                                                  The great thing about the messy kind of historical tangle is that the exact same set of circumstances is not likely to occur again: an ill-timed attempt at asserting dubious trademark claims is unlikely to hobble a gopher revival, a fight between Mark Miller & Roger Gregory is unlikely to break the Autodesk timeline for a half-decade XOC project, Apple’s not going to poach half of the dev team on the next big Smalltalk environment and make them write assembly for a bitty box, & Commodore isn’t going to mis-manage the Amiga br– actually scratch that last one.

                                                                                  I’d be quite satisfied with the browser-as-OS paradigm if the browser actually replaced the underlying OS rather than layering more leaky abstractions on top of it.

                                                                                  I don’t have any particular problem with the OS-in-a-sandbox-in-an-application model. Smalltalk VMs make it work, after all. The leakiness of the abstractions aren’t my biggest complaint about it: as hardware performance growth slows, I expect more effort to be put into cutting unnecessary & counterproductive bloat in order to ensure that current releases perform acceptably on current hardware, and almost everything a browser does is unnecessary for an application sandbox.

                                                                            1. 4

                                                                              The configuration in this demonstration established that a computer can play Tetris better than a human, under normal real-time Tetris playing conditions.

                                                                              As phrased, that’s a pretty weak claim. Sure, I don’t doubt that a well-engineered computer system (especially at today’s clock rates) can play Tetris better than any human… but if you haven’t seen championship-level Tetris players, have I got a documentary for you!

                                                                              1. 14

                                                                                I’ve seen a few things on the internet (including here at Lobsters) saying, essentially, “Please, use Firefox out of concern for the ecosystem, even if it’s worse than the alternatives at <thing you care about>.” I do use Firefox, and have for the last year, but this rankles me a bit. I realize that Mozilla is (partially) a non-profit, and that even a for-profit corporation can’t do everything, but if you visit Firefox’s Bugzilla you can find tickets for obvious features that have been open for years. Here’s one that’s been open since April 2013 and which is still unassigned.

                                                                                Part of this is a PR/communication problem; Firefox is at a bit of a disadvantage in that we can all see a list of the things they are or aren’t working on right now. But every time Firefox gains a new feature that I don’t care about, I think about all of these tickets that have been open forever and have lots of comments and duplicates but which Mozilla has chosen not to work on.

                                                                                1. 10

                                                                                  Honestly, if we’re concerned about the web ecosystem, people should use Lynx more. The web would be better if most pages had to work in Lynx.

                                                                                  Edit: Which, after checking, Lobste.rs is very readable on Lynx, and you can login just fine. Unfortunately, the reply/edit links don’t work.

                                                                                  1. 4

                                                                                    I do actually know people using the web with lynx frequently (It’s a very nice browser for blind people used to the command line).

                                                                                    1. 1

                                                                                      Just so people don’t get the wrong idea, I should mention that the majority of blind people who use computers have been productively using GUIs for a couple of decades now. Yes, there are blind people who are more comfortable with the command line and even screen-oriented terminal-based applications like Lynx, but they’re a small and shrinking minority of a minority, and I would guess that even they have given in and started using a JavaScript-capable browser when needed. There is certainly not an economic barrier anymore. So don’t feel that you need to accommodate them.

                                                                                      1. 3

                                                                                        So a blind person who uses lynx should not be accommodated, because the barriers to switching to GUIs are not economic? If it’s harder for them to learn to use a GUI than to keep using lynx, why shouldn’t they be accommodated?

                                                                                    2. 3

                                                                                      Yup. I read lobste.rs in links and have to pop over to Firefox to reply. You can post a new comment, though.

                                                                                      1. 2

                                                                                        Lynx (and Links in text mode) are great options. If you still want graphics, but not the latest JS / CSS fads, maybe try Dillo or NetSurf. All these have independent rendering engines. The parts of the web that don’t work in simple browsers are largely the ones I can do without.

                                                                                      2. 10

                                                                                        I know its a pain when your pet issue doesn’t get fixed but really, bookmarklets.. I have never heard anyone irl ever mention one. 99.9% of web users probably don’t know what they are so I am betting when a mozilla dev has to choose what issue to work on there will be a lot more high priority tasks than fixing bookmarklets. I have had some real blocking issues with some new web features not working exactly to spec in firefox and I have seen a lot of them get fixed in reasonable timeframes because they affect more users.

                                                                                        1. 8

                                                                                          Yeah, but that’s not what they were doing. Mozilla invested all kinds of time into projects people weren’t demanding while not fixing problems their existing users were reporting. That’s not a good way to run a business if you have one product with serious competition. Gotta keep making that product the best it can be along every attribute.

                                                                                        2. 6

                                                                                          Heh, I should have guessed that was the bookmarklet bug. It is weird that a browser that supposedly empowers the user allows remote sites to dictate what code you’re allowed to run.

                                                                                        1. 14

                                                                                          Professional historians have a generic name for teleological historiography: Whig history. It’s a predictable and ubiquitous consequence of history being written (only) by the victors. As such, it’s still unfortunately predominant in the histories of science and technology.

                                                                                          1. 3

                                                                                            Check out A People’s History of Computing in the United States by Dr. Joy Rankin for a very different (and very exciting) history of personal computing that focuses on time-sharing communities, K-12 classrooms, BASIC programming courses, etc. = http://www.hup.harvard.edu/catalog.php?isbn=9780674970977

                                                                                            1. 2

                                                                                              So, sort of inevitable?

                                                                                              1. 3

                                                                                                Yes. Every culture has heritage: origin stories, ancestor myths, explanations of who “we” are and how we got to be that way. The idea of history as a dispassionate study of (relatively) objective truths is what’s unusual. It doesn’t come easy.

                                                                                            1. 3

                                                                                              Really nice write-up. To my sensibilities, this makes a better advertisement for Julia than any number of language feature blog posts!

                                                                                              1. 1

                                                                                                I don’t understand. What is so specific to Julia in this post that other language can not achieve?

                                                                                                1. 1

                                                                                                  It’s not that you couldn’t do Bezier curves in any other language! Just that this particular Julia implementation is quite uncluttered and elegant, which make the prospect of using Julia for similar exploratory work more appealing.

                                                                                                  1. 1

                                                                                                    I realized this is a good point, which begs the question. How much time has the author actually spent on producing this blog post?

                                                                                              1. 2

                                                                                                This article helps to lay out the legal mechanisms by which apple and the plaintiff will make their cases. It’s illuminating for me because I have no legal background.

                                                                                                The author takes a slight position against apple, but doesn’t try to hide that. I appreciate both the writing style and the honesty.

                                                                                                1. 1

                                                                                                  Ben Thompson is a high-quality commentator and regularly takes on some very big topics. This blog is recommended reading for anyone who wants a better understanding of how law, politics, economics, marketing, and business strategies in general influence consumer technology. Doubly recommended for anyone who believes that the influence mostly works in the other direction (the SV “disruptive tech is changing the world!” narrative).

                                                                                                1. 4

                                                                                                  My co-workers and I discuss various computer topics over lunch including research papers. And we’ve realized that most all of it was started in the 50’s, 60’s and 70’s. The ideas are decades old and we’re still doing research?

                                                                                                  And then we’re sad and lunch tastes like bitter tears.

                                                                                                  1. 5

                                                                                                    Just imagine working in a research environment! Bitter tears all day long. :-)

                                                                                                    Keep in mind, there’s a long way between a good idea and a practical, useful technology. Many (if not most) promising ideas in CS research have never been properly tried out “in practice”, and that represents a lot of good and interesting work to do. The whole field is very young and sorely lacking in historical perspective.

                                                                                                  1. 6

                                                                                                    Android did this and won.

                                                                                                    A text interface is a universal interface. Even ncurses-based (though I prefer REPL or even better, CLI).

                                                                                                    As soon as you add buttons, it gets messy and the need to contain it all arises.

                                                                                                    The web is more like a joke on that point :

                                                                                                    It went from:

                                                                                                    1. Ship the data. (your browser settings rule the rendering)

                                                                                                    To

                                                                                                    1. Ship the style of the website (with a new trend every 2 year contributing to split the scene into a cascade of design generation)
                                                                                                    2. Ship the data (extracting them has gotten harder with all these divs. Requires DOM manipulation).

                                                                                                    To

                                                                                                    1. Ship the heavy client. (Why would we call them light clients? They are not lightweight and behave as local programs. Most are even still trying to fetch data while offline).
                                                                                                    2. Ship the style of the webapp (you got to restructure a document rendering engine into an application engine).
                                                                                                    3. Ship the data (good luck actually getting the data with wget or curl !).
                                                                                                    1. 3

                                                                                                      A text interface is a universal interface.

                                                                                                      I think I agree with you overall, but I don’t think that terminal interfaces are all that universal. There are lots of idiosyncrasies both low-level (which escape codes are supported by which terminal emulators?) and high (copy-paste? undo-redo? CLI flags? completions?). Now, compared to desktop (let alone web) GUI frameworks, terminals are still more orderly, but I think that’s mostly just due to a low ceiling on their basic capabilities.

                                                                                                      1. 1

                                                                                                        That is why I prefer CLI and REPL-like interfaces where you just get text. Then you can even publish the text on the web, on a plain file, print it, pipe on it…

                                                                                                        Of course sometimes you need different kind of interfaces (any user of ed or ex here ?) and there, terminal are falling short…

                                                                                                        There is a set of common features accessible on every terminal I have ever seen, which ANSI.SYS escape codes, as used by linenoise.

                                                                                                    1. 5

                                                                                                      Parsing Models Cheatsheet

                                                                                                      I recently got the Earley parsing religion, which is O(n³) worst-case just like everything else, but can handle most sensible CFGs in linear time and space, even with infinite lookahead. No shift/reduce errors like yacc, no accidental infinite recursions like PEG parsers, just reliable operation.

                                                                                                      1. 1

                                                                                                        By the way, I think that GLL parsing can be similarly elegant and concise as that of Earley. I haven’t gotten around to translating that to Python though.

                                                                                                        1. 1

                                                                                                          The religion has its appeal… but doesn’t get the job done on its own. Can you recommend some implementations?

                                                                                                          1. 2

                                                                                                            If you are looking for a simple well-explained Python implementation that does the related optimizations such as Aycock’s fix and Leo’s right-recursion optimization, do look at ours. We are writing this as part of a larger book on fuzzing with grammars.

                                                                                                            1. 1

                                                                                                              Thanks! That book looks quite useful. I will share it around.

                                                                                                            2. 1

                                                                                                              The industrial-strength implementation is Marpa, which is a C library. There are bindings for various languages, but the original author’s Perl binding is the most complete.

                                                                                                              Searching for “earley” on nearly any language package manager will probably give you a bunch of implementations of varying quality.

                                                                                                              Naturally, I’m writing my own: as a learning exercise, and because I’ve got Strong Opinions about documentation quality I’d like to demonstrate. I haven’t yet got it to the point of public presentability, but it’s coming along nicely.

                                                                                                              1. 1

                                                                                                                Would love to see your progress. Are you going the SPPF route or are you building the trees after the Earley algorithm has completed?

                                                                                                                Here is our attempt at explaining Earley parsing in Python.

                                                                                                                1. 1

                                                                                                                  I currently have zero documentation, but if you’d like to take a look at the source, I put it on sr.ht.

                                                                                                                  I confess I don’t know exactly what SPPF actually is, except that Loup Vaillant couldn’t understand Scott’s paper about it, and Jeffrey Kegler independently invented something equivalent which he doesn’t really explain. Basically, I build a parse-tree node for each completed item by stitching together parse-tree nodes for non-terminal symbols that appear in the right order and cover the required span. I happen to do it while the parse is in progress, but I think the same approach would work if done afterward.

                                                                                                                  Thank you very much for that link! I just discovered the hard way that the Leo optimization breaks naïve parse-tree construction, and yours is the first document I’ve seen that seems to acknowledge that’s an issue. Unfortunately, it only mentions it in an exercise, so there’s no worked solution, but at least there’s a hint which is more than I’ve found anywhere else.

                                                                                                                  1. 1

                                                                                                                    Unfortunately, it only mentions it in an exercise,

                                                                                                                    The solution is there! Look at the LeoParser using Tagged class. (Most exercises do come with a worked solution.)

                                                                                                                    Thank you for your link.

                                                                                                                    1. 1

                                                                                                                      Ah, right! It was late at night when I first looked, now that I’ve had some rest I can see the solution.

                                                                                                                      Although I’m sure I’d fare better if I’d used your tutorial from the beginning, I’ve had some trouble following the code examples:

                                                                                                                      • Exercise 5 has a big block of text describing the concept of a deterministic reduction path, and then it has a block of code that adds a tag field to the State class. Nothing in the text motivates this addition, and it’s not actually used in the following LeoParser class.
                                                                                                                      • The tag field is eventually used in the solution to that first exercise, but very subtly: inside the new leo_complete() method, a single call is changed to have an extra parameter. That’s the only change to the previous leo_complete() function, and it doesn’t have a comment pointing out the change or even a keyword argument tag=state.name to highlight that something is being done with tags
                                                                                                                      • It turns out that tags are not even necessary to make a right-recursion-optimised recogniser, only a parser, so this is all distraction from the idea (“deterministic right reduction”) that the text is trying to convey and it would be very easy for a reader to ignore this scaffolding and assume they’ve understood the material.
                                                                                                                      • By the time we get to the hint that says “any time you see a tagged state, look at its end point”, there hasn’t been any discussion of how, when or why a state gets tagged, or what a state gets tagged with. It turns out that the code does provide answers to most of those things, but (as I said) it’s pretty subtle.
                                                                                                                        • EDIT: Now that I look more closely, the state is tagged with the LHS of the production that was originally produced, but the code never examines the tag content, so effectively the tag is a boolean “produced by Leo reduction” flag. Is that right?
                                                                                                                      • Another thing that confused me: inside the parse() method of the final LeoParser, it starts by calling parse_prefix(), checking the result, and then calling chart_parse(). If I understand correctly, that code will recognise the whole input string, then throw away the intermediate and final results, then start parsing again from scratch?
                                                                                                                      1. 1

                                                                                                                        Thank you for the detailed review. While the rest of the text has gone through multiple revisions, the exercises hasn’t received as much care as the main body other than checking that the idea is correct and the implementation works. I will go through the exercise again, and clarify the points you have noted; and update you.

                                                                                                                        1. 1

                                                                                                                          As I mentioned, I have updated my tutorial with your feedback in mind. The updated tutorial is here Exercise 5. Have I addressed all your concerns? Any place I can make it better?

                                                                                                                          Note that I took the easy way out in expanding the Leo items. I put back everything that was trimmed. Not just the ones required for derivation trees.

                                                                                                                          Thanks again for your feedback.

                                                                                                                          1. 2

                                                                                                                            Thank you for being patient! I’ve been studying your tutorial and working on my code, and I have something I think I’m happy with. Here are the notes I took as I was figuring things out:

                                                                                                                            • It’s a bit difficult for me to follow the text because I’m diving in at the end rather than reading from the beginning, and because I started with Loup Vaillant’s tutorial which uses different terms (“Earley set” instead of “column”, etc.) but that’s my problem, not a problem with the text.
                                                                                                                            • Thank you very much for the exhaustive tests; I found a problem with my existing recogniser due to RR_GRAMMAR4.
                                                                                                                            • It’s nice to have a diagram of the right-recursive-reduction chain, but I think the “topmost” reduction is depicted on the bottom, which is a little confusing.
                                                                                                                            • The “deterministic reduction” operation is described as resulting in a new completed item, so I was surprised to see the completion operation done in get_top() rather than in uniq_postdot()… why not have one method for the “one level” reduction and another method that calls it recursively for the topmost reduction?
                                                                                                                            • When it comes to actual parsing, the code adds a TState class and updates the Column class to use it… but TState seems identical to State in every way, so I’m not sure why this change is made. Is it just to distinguish between ordinary Earley items and Leo-reduced items? Wouldn’t adding an is_reduced boolean field be a clearer way to distinguish them?
                                                                                                                            • I assume the parameters to traverse_constraints() are named for how they’ll be called in the first iteration of the example structure, but I generally expect parameters to be named by what they mean in general, not in a specific instance. Is this a general convention for the textbook?
                                                                                                                            • If I understand the example code correctly, you’re adding a transitives field to each column, which stores a copy of each item in the reduction chain (get_top() iterates the chain recursively, and calls add_transitive() at each step). Isn’t that the O(n²) behaviour Leo is supposed to avoid?
                                                                                                                            • I’ve come up with my own approach:
                                                                                                                              • for each topmost item, we store the set of items in the reduction chain, and the numerically largest end offset for any item in the set
                                                                                                                              • inside get_top(), once we’ve found the topmost item for state_A, we add the one-level-reduced state st_B to the chain for that topmost item, and increase the end offset to state_A’s end offset, if it was smaller
                                                                                                                              • because our extra cache stores reduced items once per topmost, instead of once per column per topmost, I believe this preserves the O(n) behaviour of Earley parsing
                                                                                                                              • when we’re done recognising, we can updating the end-offset of each state in the chain to match the largest observed end-offset, and insert them into the regular state storage for the parsing phase to find.
                                                                                                                            • When implementing this, I had a bug where I was storing state_A in the chain instead of the one-level-reduced st_B. I discovered this bug because one of my testcases was the grammar <A> ::= 'a' | 'a' <A> and I wound up stretching all the non-recursive 'a' items to cover the right-recursion span instead of the recursive 'a' <A> items. You might want to add that test-case to your examples.

                                                                                                                            I’m not sure how helpful that all is to you, but I found it very helpful to have a working example to follow, especially for finer points of the reduction memoisation. Thanks again for your help!

                                                                                                                            1. 2

                                                                                                                              Thank you so much for the exhaustive review!

                                                                                                                              It’s nice to have a diagram of the right-recursive-reduction chain, but I think the “topmost” reduction is depicted on the bottom, which is a little confusing.

                                                                                                                              I agree! It is better represented with the topmost at the top.

                                                                                                                              The “deterministic reduction” operation is described as resulting in a new completed item, so I was surprised to see the completion operation done in get_top() rather than in uniq_postdot()… why not have one method for the “one level” reduction and another method that calls it recursively for the topmost reduction?

                                                                                                                              The reason I went with this organization is that the transitive has to be stored in the end column of st_B_inc. Unless I pass the end column of st_B_inc separately, I can not use uniq_postdot the way you suggest. (I think this also has to do with our different understanding of where transitives are stored. See the later point.)

                                                                                                                              When it comes to actual parsing, the code adds a TState class and updates the Column class to use it… but TState seems identical to State in every way, so I’m not sure why this change is made. Is it just to distinguish between ordinary Earley items and Leo-reduced items? Wouldn’t adding an is_reduced boolean field be a clearer way to distinguish them?

                                                                                                                              Indeed, the sole function of TState is to mark it as a transitive (the copy could have been done separately). You are right that in an independent parser that implements Leo’s reduction, this would have been better represented as a boolean field. However, we have an Earley parser that is separate in the main text, which I did not want to modify, and which I want to be importable from the main chapter. With this organization, a person who imports the State will not need to see the TState update. (I also like the fact that the transitive is a different kind of a state as it is a shorthand for a sequence of states, and hence worthy of getting its own class to indicate it. But that is subjective)

                                                                                                                              I assume the parameters to traverse_constraints() are named for how they’ll be called in the first iteration of the example structure, but I generally expect parameters to be named by what they mean in general, not in a specific instance. Is this a general convention for the textbook?

                                                                                                                              I actually switched back and forth multiple times between naming them in a general sense and naming them using the names in the example. Having found a rather large number of bugs, and having to re-understand what exactly was happening each time, I found it better to rely on the names in the example to clarify what exactly I was trying to do. (This is not the general convention followed in the book, and I will probably have to rename them at a later point. Until that time, this makes my life simpler :) )

                                                                                                                              If I understand the example code correctly, you’re adding a transitives field to each column, which stores a copy of each item in the reduction chain (get_top() iterates the chain recursively, and calls add_transitive() at each step). Isn’t that the O(n²) behaviour Leo is supposed to avoid?

                                                                                                                              Only the first time for any particular pattern. After the first time the transitive items are cached. See the quote from Loup Vaillant “Now, when we find the topmost item in the reduction path, we add the corresponding transitive item to the current Earley set. And all the Earley sets we just examined, while we’re at it.”. This is also my reading of Def 2.1 from Leo’s paper.

                                                                                                                              “To determine the topmost item on a deterministic reduction path we do not always have to construct the complete reduction path. Suppose $ [C -> \delta., m] $ is the topmost item on the deterministic reduction path above $[A-> \gamma., i]$, and $[B -> \beta., k]$ is some other item on this path. Then we add to set $I_k$ a so-called transitive item $[C -> \delta., B, m]$. Subsequently, if an item of the form $[B -> \beta’., k]$ is added to some set I_j, we can directly add $[C -> \delta., m]$ to $I_j$. “ (formatting is screwed up. But please see the link I gave above.)

                                                                                                                              That is, if $[B -> \beta., k]$ is some item in the path (at end column k), we add the transitive item to I_k.

                                                                                                                              On the other hand, my implementation of the parser is a naive one. The leo_expand essentially gives back all the gains made during the recognizer. It would be possible to have a clever implementation that only lazily expands when a reduced item is required. However, at this point, I have opted for a naive but simple and correct.

                                                                                                                              I’ve come up with my own approach: for each topmost item, we store the set of items in the reduction chain, and the numerically largest end offset for any item in the set inside get_top(), once we’ve found the topmost item for state_A, we add the one-level-reduced state st_B to the chain for that topmost item, and increase the end offset to state_A’s end offset, if it was smaller because our extra cache stores reduced items once per topmost, instead of once per column per topmost, I believe this preserves the O(n) behaviour of Earley parsing when we’re done recognising, we can updating the end-offset of each state in the chain to match the largest observed end-offset, and insert them into the regular state storage for the parsing phase to find.

                                                                                                                              This might work, however, I caution that Leo is rather hard to get right. (the deterministic reduction as I implement it does preserve O(n) behavior as I mentioned above.)

                                                                                                                              One of my testcases was the grammar ::= ‘a’ | ‘a’ and I wound up stretching all the non-recursive ‘a’ items to cover the right-recursion span instead of the recursive ‘a’ items. You might want to add that test-case to your examples.

                                                                                                                              Thank you so much! This actually exposed a bug I had. My implementation was not able to parse with the RR_GRAMMAR7 which is your test case. I am not entirely sure if my fix is the right one, but I don’t see a different explanation for it other than this.

                                                                                                                              I’m not sure how helpful that all is to you, but I found it very helpful to have a working example to follow, especially for finer points of the reduction memoisation. Thanks again for your help!

                                                                                                                              Your review has been very helpful, and thank you for this wonderful new year gift!

                                                                                                                              1. 1

                                                                                                                                On the other hand, my implementation of the parser is a naive one. The leo_expand essentially gives back all the gains made during the recognizer.

                                                                                                                                I found a way to avoid expanding all states (updated in the code) which is simple enough.

                                                                                                                            2. 1

                                                                                                                              Thank you very much! I haven’t had time to look at your changes yet, but I will try to get around to it very soon.

                                                                                                              1. 1

                                                                                                                This article has no type. At least it is not sound. I think serious writing about type theory starts with studying the aptly named untyped lambda calculus. Terms in the untyped lambda calculus have no type.

                                                                                                                1. 7

                                                                                                                  That’s one way to look at it. From another perspective, the terms of the “untyped” (some would say “uni-typed”) lambda calculus have only one type. That’s how TAPL presents it, for example. I consider that serious enough.

                                                                                                                  But would agree that studying the differences between Church’s 1936 “untyped” and 1940 “simply typed” lambda calculus is quite illustrative.

                                                                                                                  1. 2

                                                                                                                    It seems utterly pointless to talk about a type system where everything has one and the same type. The only point of a type system is to distinguish between different types. Otherwise you just . . . don’t have a type system.

                                                                                                                    1. 6

                                                                                                                      Sure, as long as you’re only talking about that one “untyped” language and its properties, there’s no point. But when you want to compare languages and their features, it’s all of a sudden very helpful.

                                                                                                                      This happens all the time in the history of math, by the way: as a theory develops and gets elaborated, something previously thought of as distinct gets recognized as a degenerate special case of something else. For example, natural numbers are included in the integers when we recognize the (not necessarily intuitive!) concept of negation, which arises out of subtraction, which in turn is desirable as a way of undoing addition.

                                                                                                                      1. 1

                                                                                                                        It can be argued that there’s no point to a type-system, it’s just a formal system, whether intended to be or not. It’s useful to know what these formal systems represent.

                                                                                                                        1. 1

                                                                                                                          It’s actually a very useful perspective for understanding the semantics of such languages: Dana Scott’s reflexive objects are essentially just a codification of the fact that untyped can be seen as unityped.

                                                                                                                      2. 1

                                                                                                                        Reminds me of that Guy Steele talk where he notices that type theorists primarily use an ad-hoc, unspecified, dynamically typed language to describe the denotational semantics of their language.

                                                                                                                        1. 1

                                                                                                                          Yes that is a good one! But the point is that the meta language is supposed to be smaller than the language and can be trusted… I think NG de Brujin discusses this issue in depth

                                                                                                                      1. -2

                                                                                                                        Untyped programs do exist. The author is just looking at the wrong direction. He’s looking down to hardware when he should look upwards toward abstract programming languages.

                                                                                                                        Here’s an example of an “untyped” program, it’s the ‘id’ function from Haskell: (λx.x) : (∀a. a -> a)

                                                                                                                        It works for every variable you pass in and the variable ranges over an infinitely large domain. The type in the program says that for every variable, you can get a variable and return the same type of a variable.

                                                                                                                        If you loosen from the idea that a program must represent a valid proof for some logical formula, then you can describe large variety of programs. The validity of the program is determined when the domain is selected. That’s how you get runtime errors and dynamic typing.

                                                                                                                        The original post is parroting of Robert Harper’s dumbest ideas. It’s equivalent to taking my “Dynamic typing is good” -post from the year 2014 and using it as the basis to refute the use of type systems.

                                                                                                                        It appears that the motive for writing the post came from a short discussion with somebody else. He hasn’t gotten to the agreement about the subject so he decided to write a very assertive blog post to drive his point.

                                                                                                                        1. 7

                                                                                                                          The haskell id function is typed (it is polymorphic, but it has a universal type).

                                                                                                                          1. 4

                                                                                                                            I’m afraid I don’t understand what definition of “typed” you are using here, could clarify a little?

                                                                                                                            I’m not sure what connection the proofs-as-programs idea has here, this merely states that some interesting type systems happen to mirror some interesting proof theories. You can trivially go from a type system to a logic, though the result is rarely worth studying. Going in the opposite direction is usually more interesting but seems irrelevant.

                                                                                                                            1. 2

                                                                                                                              Type systems not only mirror proof theories; they are isomorphic. And the isomorphism is very useful because theorems about programs can be stated in the same language as the programs themselves, as seen in languages such as Coq, Idris, Agda, Epigram… It gives verification powers to languages, and also gives implementation laboratories for logics.

                                                                                                                              1. 1

                                                                                                                                I avoided the use of the word “isomorphic” because it’s not clear in what category such an isomorphism should live. But yes, it is a very important symmetry!

                                                                                                                                1. 1

                                                                                                                                  Isomorphic as in a one to one mapping from and to one another so that you can always recover the original information/object after going from one representation to the other

                                                                                                                            2. 4
                                                                                                                              GHCi, version 8.0.1
                                                                                                                              λ :t id
                                                                                                                              id :: a -> a
                                                                                                                              

                                                                                                                              Looks like a type to me… but that’s just Haskell’s type system. Are you trying to make a more general claim?

                                                                                                                              1. 3

                                                                                                                                The more general claim is that it’s idiotic to continue the static vs. dynamic typing debate because it is a false dilemma. And it’s harmful to propagate it.

                                                                                                                                If you take the role of a dynamic typing zealot, it means you’ve discarded the study of logic and proof theory that could enhance your usual ways of reasoning about programming. You don’t need to stop using dynamic languages to reap the benefits, but once you’ve figured it out you want to use something like Prolog, Coq or Haskell more often.

                                                                                                                                If you go and be a static typing zealot, then you do some sort of a fallacy. It involves putting some particular set of deduction rules to a pedestal and discard all methods of reasoning outside of whatever framework you’ve decided to use. Just like the author asserts how everything, including assembly language and C, has to have a well-defined type, even if the type was useless by failing to convey what the program is trying to solve. There are tons of logical systems and frameworks and nobody has decided that we should stick to intuitionistic logic.

                                                                                                                                Effectively both stances discard the study of logic in one form or an another. It literally harms scientific progress and the motion from theory into practice.

                                                                                                                            1. 2

                                                                                                                              This is worrying. Do I have to start looking at fastmail alternatives? Any recommendations?

                                                                                                                              1. 2

                                                                                                                                How do you think this will affect fastmail, specifically? They do not provide encrypted email, and already will comply with Australian government search orders.

                                                                                                                                1. 2

                                                                                                                                  I’m not an expert, but I don’t see how it would change anything for Fastmail, they already give the govt. anything they ask for(that’s legal).

                                                                                                                                2. 1

                                                                                                                                  There is kolabnow.com (Kolab hosted) also Migadu.com, all hosted in the nice European countries that tend to care about privacy. There are Hosting providers for Zimbra, Horde, etc out there as well.

                                                                                                                                  1. 1

                                                                                                                                    I stumbled upon proton mail the other day. Not tried it though: https://en.wikipedia.org/wiki/ProtonMail

                                                                                                                                    If you like pain, you can host your own mail server.

                                                                                                                                    1. 1

                                                                                                                                      I stumbled upon proton mail the other day. Not tried it though: https://en.wikipedia.org/wiki/ProtonMail

                                                                                                                                      caveat emptor regarding protonmail.

                                                                                                                                      1. 1

                                                                                                                                        Who or what is Duke-Cohan?

                                                                                                                                        I’m a happy customer of ProtonMail, but I don’t like the pain of self-hosting unless it’s solely my own shit, and Google is a no-go zone.

                                                                                                                                        1. 1

                                                                                                                                          Oopsy!