1. 1

    The follow links don’t work, but it’s a cool idea!

    1. 1

      The author knows about the issue, so I guess it will be fixed soon:

      Also, since this is the early days, please know that there are downtimes when I’m desperately trying to fix things, probably right now!

    1. 3

      Huh. If it’s fine to create your own “mediation pipe” to satisfy the API, and then splice(in, pipe); splice(pipe, out) … why doesn’t the kernel support that itself?

      1. 1

        Maybe it’s because errors need to be reported on the pipe instead of on in or out?

        1. 4

          ActivityPub strikes me as the invention of people who believe that the internet = HTTP, and who know about JSON but not RFC822.

          Some of the example message bodies just look like JSON-ized SMTP headers, “inReplyTo” etc. It looks like it has a MIME-inspired “mediaType” attribute too, but does it allow only one media type per message?

          Can someone who is more familiar with ActivityPub give me the sales pitch about why existing protocols don’t suffice?

          1. 6

            RFC822 is ASCII only to begin with one of the biggest limitations of email related “standards”.

            Some 6.5 billion people around the globe use non-ascii charecters, and old standards only have layers of hacks upon them to support their usecases to some extent.

            Why not create new standards from the ground up for the current usecases? I’m not interested in ActivityPub curently, but I have some experience with email and related technologies, and it badly needs a redesign. It won’t happen as none of the parties capable to organise it is interested in it.

            1. 4

              My uninformed guess is that with the slow decline of email, there are more & better JSON parsers than there are MIME or email parsers. I would have made the same choice, but my reason would have revolved around JSON’s ability to store structured data, for future flexibility.

              1. 2

                HTTP Headers are the same format like MIME headers, browsers already have everything one would need for mail. Multipart documents (email attachments) are the same format like HTTP file uploads via form. There is a number of headers both share.

                1. 1

                  I think it comes down to tooling. Protocol A could be 10x as widely deployed as protocol B, but if protocol B has better libraries, I’ll give that more weight in my decision of which to use. I had to assemble a multipart MIME message for work a few weeks ago, and everything about the experience was inferior to “create a data structure and convert it to JSON”.

                  Coders are likely to pick the easiest path, if everything else is roughly equal.

              2. 1

                No reason, really. It’s a marketing effort, mostly.

                1. 1

                  SMTP is forever tainted by spam. ISPs like to block ports, spam filters like to eat mail from new unknown servers, etc.

                  Giving a pitch for Webmention instead of ActivityPub: Webmention requires the sender to publish an html page that actually links to the target URL. You can be stricter and require a valid microformats2 reply/like/repost/bookmark. That already stops old school pingback spam. For stronger protection, there are clever schemes based on “this non-spam domain you linked to has linked to me”.

              1. 11

                Sadly, this stuff has been well known since at least the famous “Peopleware” book of the 1980s, and certainly firmly ensconced into conventional wisdom by the zeros. Facts and new studies are not moving the needle on this one. (My pet theory: tech companies are much more risk-averse than they like to pretend, and doing anything different from your peers is too scary, especially if it costs more.)

                1. 3

                  tech companies are much more risk-averse than they like to pretend, and doing anything different from your peers is too scary

                  I think it’s more active than that. Tech companies want to emulate silicon valley startups. Silicon Valley startups are funded by VCs, who want to check on their investment. At some point, someone decided that open-air offices would 1- show thrifty spending, and 2- show people being busy. From that point on, everyone’s been doing open-air, even when it’s no longer #1 (I’m thinking of some significant remodel work that’s gone into some open-space workplaces, not to mention climate control costs or morale/productivity loss), nor #2 (when open-air offices are as large as football fields).

                  1. 3

                    It’s not just startups, although I bet most successful startups have open floor plans. Facebook and even Microsoft are moving or have moved to the model.

                    To quote Dan Luu, speaking of Peopleware:

                    This book seemed convincing when I read it in college. It even had all sorts of studies backing up what they said. No deadlines is better than having deadlines. Offices are better than cubicles. Basically all devs I talk to agree with this stuff.

                    But virtually every successful company is run the opposite way. Even Microsoft is remodeling buildings from individual offices to open plan layouts. Could it be that all of this stuff just doesn’t matter that much? If it really is that important, how come companies that are true believers, like Fog Creek, aren’t running roughshod over their competitors?

                    1. 3

                      I asked the Facilities people at Google about this last year and they told me why they subject us to open plan offices despite knowing we hate them. Apparently the #1 reason is flexibility. They move teams around frequently as they grow so being able to easily defrag space is their top priority.

                      1. 3

                        That’s long been a reason. I wonder if the answer is to try to make easy to build or tear down spaces that are more like actual offices. Especially with noise resistant material in between the “walls” maybe in form of squares or rectangles that can be shoved and stacked in there. I guess it would look like cubicles that go up to the ceiling with an outer wall, filler, and inner wall. If they have different shapes, windows could be another modular component in some of them if someone desires.

                        I wonder if anyone has tried something like this.

                      2. 1

                        I believe this is called survivor bias

                  1. 3

                    Does this apply to (a) businesses with customers resident in California, (b) businesses incorporated in California with customers anywhere, or (c) businesses incorporated in California with customers resident in California?

                    This seems tricky to do at the state level and should be federal…Similar to how GDPR was EU level.

                    1. 4

                      California has a long history of setting a higher (per-state) bar on things like vehicle emissions and mileage, forcing companies to either build two of everything (“49-state” cars) or follow the stricter set of rules for everything – usually they decide to do the latter. California’s lawmakers are probably hoping this will go the same way, since GDPR led the way.

                      1. 2

                        It seems they’re doing it to get the ball rolling so to speak. Europe, now California, who next?

                      1. 18

                        Would I like more Mastodon and ActivityPub propaganda without much real substance and cute elephant friend pictures, reiterating how great federation is? You bet your top dollar I would! Take my upvotes, Mastodon blog! And congrats on self-hosting your blog and moving the propaganda machine off Medium!

                        I’m @JordiGH@mathstodon.xyz in case anyone wants to say hi.

                        1. 6

                          “If Twitter shuts down, you’ll lose your followers. If Facebook shuts down, you’ll lose your friends. For some platforms, it’s not a question of “if”, but “when”. Such events are usually followed by a scrambling into a variety of different platforms, where you inevitably lose some people as you have to make a choice which one to stay on. This happened before. But it doesn’t have to happen again. Use the federated web. Join Mastodon.”

                          Adding to your comment, this is particularly such bullshit. More like you won’t have many friends, esp in real life, if you’re on Mastodon instead of those sites. The people on those major sites also use more than one. If one shuts down, they almost always have another way to contact them. The shutdowns also don’t usually happen overnight. There’s often time to see things are in decline. For instance, there’s plenty of posts about Twitter’s financial troubles where anyone with sense will have some other account on the side. It’s also amusing to see them saying “if X shuts down” after seeing a headline about Instagram alone being worth $100 billion, watching Facebook’s revenues grow over time despite all its threats, Slack doing what IRC isn’t in growth/profits, and knowing that previous shutdowns sent users to sites like Facebook instead of P2P software or Mastodon.

                          1. 5

                            It’s not Twitter’s finances that are making people leave; kinda the opposite.

                            1. 2

                              The article talked of companies disappearing. That can happen due to users leaving or financial reasons. Twitter’s money problems & mismanagement are a warning sign that should signal users to have another option. That’s all I was saying.

                              1. 2

                                Sorry, my reply was more opaque than I meant it to be. I think the toxic things Twitter is doing to chase a profit are why so many people are leaving. I don’t think most users cared about their financial situation other than to wonder why the service was free.

                                1. 2

                                  Oh OK. Yeah, that makes a lot of sense. It’s even more ridiculous when a company gets a nearly impossible amount of users, has tons of money to make off them, and just… lets them float away…

                                  You’d think rational self-interest wouldve made them take action sooner. Not how it often works in practice, though.

                                2. 1

                                  I thought Twitter was doing well lately

                                  1. 1

                                    It might be. I read it a while back. Just using it as an example warning sign.

                          1. 2

                            Whoa that means this year is the 50th anniversary of that paper!

                            1. 7

                              Meanwhile, a friend of mine just got a response on a PR. Written in nice polite professional prose: “Sure, we can merge that, thanks for this improvement, but would you first please …” and that’s where it spun off into senselessness. It was all politely written, no swearing, but made no sense.

                              Linus will swear. And read the code. And understand both the code and what you tried to do, and have a decent idea of what’s achievable.

                              It seems to me that some people really mind the swearing, and do not realise that everyone has faults. Complaining about swearing isn’t a step towards faultless people, it’s effectively pushing us towards other faults, the ones that get no complaints.

                              1. 18

                                Why is there a dichotomy? Why would reducing rudeness be “pushing us towards other faults”?

                                1. 0

                                  Suppose you are to choose a team leader. You can choose a team leader, but you have to choose from among the people who are there, none of whom are saints. One will swear like a sailor, another doesn’t care about documentation, a third suffers from Rambling Meeting Syndrome. Disallowing one of them for one fault doesn’t happen in a vacuum, because “none of the three” is not an option.

                                2. 21

                                  Removing or reducing one fault doesn’t necessarily imply the rise of another. Someone can both say nonsense and be a jerk at the same time, but I’d rather they just say nonsense.

                                  I’ve certainly responded to PRs with nonsense occasionally. It happens, because I didn’t read closely enough or just missed something. But I at least try not to be jerk.

                                  Should people be jerks? I’d rather not. At least try not to.

                                  Should people say nonsense? Again, I’d rather not.

                                  The world doesn’t have to be seen as black or white. People can ask others to not be a jerk without simultaneously endorsing the rise of nonsense.

                                  1. 17

                                    It seems to me that some people really mind the swearing

                                    I’m from New England. I swear a lot.

                                    That said, Torvald’s candor is unacceptable. There’s no reason to devolve into namecalling (“moron”, “brain damaged”) that’s clear intent is to hurt feelings.

                                    So no, I don’t mind the word “fuck.” However, I really fucking mind it when people insult other’s intelligence with the intentionality of hurting their feelings. Who would want to work on a project like that? I know that I wouldn’t.

                                    1. 9

                                      Pretty sure being crude and being nonsensical both garner complaints, for example, you’re complaining about your friend’s nonsensical PR response, and the author here is complaining about Linus’s crude and belittling language.

                                      While understanding the issues is much appreciated, so is not being a jerk. Technical skills have brought us this far, but soft skills will help bring us to a future where more and more people are willing to get involved with open-source (and other) projects.

                                      1. 13

                                        I don’t think the article is about swearing. It’s about verbal abuse and how to convey the same information without attacking the recipient.

                                        1. 5

                                          You hit the nail on the head. I feel like we’re moving rapidly towards a culture of, “Let’s all celebrate diversity and embrace inclusion, except for those who we disagree with who are just going to have to change their barbaric ways.”

                                          For every person who is offended by swear words (I mean, words of all things) there’s someone else who reads a phrase like, “this is pure and utter bullshit,” who thinks, man, this person is really passionate about what they’re talking about, maybe they have something interesting to say.

                                        1. 1

                                          Ironically (for someone working for archive.org), 4am’s blog is almost impossible to find on search. But they have posted many highly entertaining and informative articles about the process, including this one about Burgertime: https://ia800209.us.archive.org/8/items/BurgerTime4amCrack/BurgerTime%20(4am%20crack).txt

                                          (the ridiculous url may be part of the reason they’re hard to search for) :(

                                          1. 1

                                            Just to make this not all snark at poor security, here’s an idea I came up with in a few minutes for how to make a smart lock that’s actually secure:

                                            Lock will Bluetooth pair with anything that asks, but requires a long random key to open. The key is in a QR code and printed as a number on a couple of sturdy slips of paper that are in the package the lock comes with. You download the app, scan the QR code, and you can open the lock. Anyone else can get the app and pair with the lock, but can’t open it because they have no way to get the code.

                                            I’m not a security expert, and I only spent a few minutes on this, so it may have some holes. But it’s definitely better than what this smart lock is actually doing.

                                            1. 2

                                              That sounds good, there are so many other issues that you’d need to address too like preventing replay attacks, customers who switch phones and have lost the QR code, how would you manage temporary keys?, etc

                                              1. 1

                                                It requires more work than this, although most consumer hardware manufacturers are so clueless, they’re not aware of the problem at all, much less how difficult it is. I work for a company that sells security solutions for IoT (Afero) and I could tell you stories that would make your toenails fall off.

                                                1. 1

                                                  Replay attacks.

                                                  Like, what happens if I sniff your traffic and play it back once I pair?

                                                  A system where both I and the lock talk to another service that generates one-time tokens would probably help.

                                                1. 3

                                                  This article is mostly fluff, but I did laugh out loud at the line about “hacker news and reddit commenters are smarter than average”.

                                                  1. 1

                                                    I’m pretty sure that they are. Not because of some virtue that belonging to those communities inherently confers, but because of pre-selection for slightly-above-average invested people.

                                                    HN is sketchier though, while I was active there it seemed to have more business people than actual industry workers…

                                                    1. 1

                                                      I can see why you would think that, based only on the messages that are posted, without being able to see the ones that weren’t. But this is more like an iceberg: the smartest and most experienced take one look at these communities and nope away forever. (Yes, I already know what that says about me.) :)

                                                  1. 1

                                                    I think the arguments in this article would apply better to C, and I agree that everyone should tinker with C a bit to gain some historical grounding.

                                                    I would dis-recommend C++ precisely because it changes every few years, creating a moving target, but never deprecating or removing the failed experiments of older versions. It’s ended up having too many (often conflicting) features to allow depth of understanding. Modern concepts are “nailed on” but not integrated well, muddying the waters for beginners.

                                                    To learn modern concepts, you should play with a modern language. To learn the foundations, C has a much smaller syntax and set of concepts to deal with.

                                                    1. 2

                                                      Sorry to be a broken record about this, but can you post the link to the transcript too, for those of us who don’t watch videos?

                                                      1. 1

                                                        I couldn’t find the transcript and don’t know what apple typically does as far as posting transcripts but the page does at least include the slides used in the presentation - I’m not the poster, just wanted to make the slides easier to find.

                                                        1. 1

                                                          The transcript is not available yet. Apple has made the transcripts available for previous WWDCs (e.g. https://developer.apple.com/videos/play/wwdc2017/305/ - see the “Transcript” tab), so it’ll probably pop up in a few weeks. Sorry!

                                                        1. 2

                                                          This looks really cool! My main requirement for a package indexing site would be search, though. Maybe I’m unusual.

                                                          1. 3

                                                            Does anyone know why this needed to be in a PDF image? Looks like it would make a great article!

                                                            1. 18

                                                              I find it easier to write in latex. Don’t judge me.

                                                              1. 4

                                                                Have you tried pandoc to convert LaTeX source into HTML (or any other format you want)? Would seem to be a compromise, unless you use some esoteric LaTeX commands.

                                                                But at the same time I can’t really understand how a typesetting macro set would be easier to use than something like markdown…

                                                                1. 3

                                                                  I can’t really understand how a typesetting macro set would be easier to use than something like markdown…

                                                                  Why not both? (I wish it were easier to run this locally.)

                                                                  1. 3

                                                                    That in turn reminds me of Org-mode, since it too can embed LaTeX codes, source code and has multiple export facilities. But you have to be an Emacs user to get it’s full potential.

                                                                    1. 2

                                                                      Oh, I didn’t know that; I’ve been meaning to try it, since I’m evaluating Spacemacs/evil-mode anyway. Thanks!

                                                                    2. 1

                                                                      Interesting program and Koka language is really interesting. Thanks

                                                                    3. 1

                                                                      my experiments with pandoc did not produce good results.

                                                                      My preference for xelatex is not defensible. It’s just a habit.

                                                                    4. 2

                                                                      I don’t know enough about latex to judge it, but if your output was a document instead of an image, it would be a lot easier to read on reader devices (like tablets)!

                                                                  1. 2

                                                                    This is @robey’s article he talked about here. One commenter asked how he did the diagrams. They do look really good. What tool did you use, robey?

                                                                    1. 3

                                                                      IIRC, I used omnigraffle. These days I use sketch but I still highly recommend omnigraffle!

                                                                    1. 1

                                                                      I’m not a kotlin fan either, but many of the features they’re complaining about (like name: type instead of type name) are “consensus syntax” that most new languages use.

                                                                      1. 8

                                                                        The binary adder in this article has the advantage of being straightforward but tedious. For those interested in a deeper dive, I wrote about clever adder techniques of the 1970s a few years ago here: https://robey.lag.net/2012/11/07/how-to-add-numbers-1.html

                                                                        1. 2

                                                                          you should submitted those articles here.

                                                                          1. 2

                                                                            This is really, really good.

                                                                          1. 5

                                                                            I had way too much fun paring the binary down to under 1KB, so thought it would be fun to show off. This is a tool for the very limited use-case of writing your own bare metal kernel (to replace Linux) on a raspi 3.

                                                                            1. 3

                                                                              Nice work. How small did you get it in the end, and did you use any interesting tricks to get the size down?

                                                                              1. 4

                                                                                I wish there were some clever tricks; it might have made an interesting blog post. Since I wrote the code in 32-bit arm first, the biggest help was things that improved in arm64: CRC-32 and integer division are now built-in instructions, which saved me two routines.

                                                                                Other tricks:

                                                                                • “Re-roll” any loops and use subroutines heavily. Don’t paste code or unroll anything. A modern CPU can do thousands of instructions per byte to the UART at 115200bps, so the extra branches won’t matter.
                                                                                • Don’t obey calling conventions near the leaf nodes (which for me, was the entire program). I noted that a leaf routine trashed registers x0 to x3, then made the calling routine (one step higher in the call-stack) use registers x4 - x6 instead. ret {reg} helps a lot here, because the routine prelude can be just mov x7, lr. Effectively, arm64’s generous register file became my stack.
                                                                                • My favorite trick was using extr to rotate a word and add an incoming byte in a single instruction, in uart_read_u32. This is pretty much what extr is for, so it’s not especially clever. I just had to spend an hour reading through the instruction descriptions, looking for interesting ones that might help.
                                                                                • The loss of conditional execution was rough, but it felt like csel and tbz really do cover most of the use cases. In particular, tbz can be used to make a tight loop if you’re counting up to a power of 2.
                                                                                1. 1

                                                                                  Thanks for the update, sounds interesting and well worth a blog post to me.

                                                                              2. 1

                                                                                Have you done this? Specifically writing your own bare metal kernel?

                                                                                1. 1

                                                                                  I’ve… started. :) I got pretty far in rust in 32-bit mode before deciding I should jump into the deep end and go full 64-bit. I’ll definitely post my progress to github as I get things working.

                                                                                  1. 1

                                                                                    Very cool! What’s the goal? By that I mean, what features are you looking for in your as yet to be completed kernel?

                                                                                    1. 2

                                                                                      My loose goal, besides “fun”, is to build a non-posix unikernel that boots into a text environment reminiscent of the old Apple II. No real goal behind that, though it would be nice to get USB & wifi working.

                                                                                      1. 2

                                                                                        Are you familiar with the Ultibo unikernel for the Pi, which provides a neat Wirthian environment implemented completely in Object Pascal?

                                                                                        1. 2

                                                                                          I am now! Very cool! :)

                                                                                        2. 1

                                                                                          SO interesting to me how many people are hearkening back to that time!

                                                                                          Makes me think of tic-80 - https://tic.computer/

                                                                                          Sounds like a fun project. Good luck!

                                                                                          1. 1

                                                                                            There were even TIC-80 demos at Revision this year. This placed #10.

                                                                                            1. 1

                                                                                              Very cool! tic-80 is so much fun. it really evokes the feeling of working in that 8 bit, self contained, everything IMMEDIATELY accessible space while using a modern language, syntax and tools.

                                                                                1. 13

                                                                                  The author has a pretty explicit bias toward Earley parsing, and using LL(k) over LR(k) methods (which I mostly share). There’s a wealth of information here, but he’s missing a couple other significant strands of generalized parsing research outside of Marpa’s particular Earley variant (perhaps because they don’t fit within his narrative).

                                                                                  GLR (Generalized LR):

                                                                                  • Tomita, LR parsers for natural languages, 1984.
                                                                                  • Tomita, Efficient parsing for natural language, 1986.
                                                                                  • Scott and Johnstone, Generalised bottom up parsers with reduced stack activity, 2005. This segues into their later work with Earley and GLL.

                                                                                  GLL (Generalized LL):

                                                                                  • Scott and Johnstone, GLL Parsing, 2010.
                                                                                  • Afroozeh and Izmaylova, Faster, Practical GLL Parsing, 2015.

                                                                                  Outside of Earley, GLL seems like a very practical generalized parsing approach. Instaparse is one implementation.

                                                                                  Earley / Shared Packed Parse Forests:

                                                                                  • Scott, SPPF-style parsing from Earley recognisers, 2008 (and several related papers by Scott and Johnstone). Note: I’ve never been able to get the approach described in the paper implemented correctly, and not for lack of trying.

                                                                                  Earley Intersection Parsing (not sure if there’s a canonical name for this):

                                                                                  • Bar-Hillel, Perles, and Shamir, On formal properties of simple phrase structure grammars in Zeitschrift für Phonetik, Sprachwissenschaft und Kommunikationsforschung, 1961. Proves some important results about intersecting automata and context free grammars.
                                                                                  • Chapter 13 (“Parsing as Intersection”) of Grune and Jacobs (also cited elsewhere in his timeline), particularly 13.4 (pgs. 437-439): This describes Bar-Hillel’s Intersection Parsing approach using contemporary CS & parsing terminology, and then suggests combining it with Earley parsing. While the basic intersection parsing method produces an astonishing amount of dead-end nodes to garbage-collect later, Earley parsers limit searching to productions reachable from the start symbol. If the completer is modified to produce non-terminals in the intersection parsing format (which is easy), intersection parsing nicely handles producing the parse forest (with structural sharing, when ambiguity produces multiple derivations).

                                                                                  I’ve been working on a C library for Earley Intersection parsing, and an awk-like, pattern-match-directed language based on it, but working on testing them thoroughly tends to lead me to working on theft instead.

                                                                                  1. 3

                                                                                    Thanks for details about alternatives.

                                                                                    I have played around with Bison’s GLR option and have almost managed to create a C parser that does not require special handling of typedefs in the lexer (ambiguities still appearing at runtime for a few constructs).

                                                                                    Grune and Jacobs’ Parsing Techniques - A Practical Guide is sitting in a pile waiting to be read (the first edition was manageable, the second looks daunting)

                                                                                    1. 1

                                                                                      I have a print copy of the second, and it’s never struct me as daunting – reading it cover to cover, perhaps, but it’s well suited to dipping into for just specific techniques, with a well-curated bibliography for further details.

                                                                                    2. 2

                                                                                      packrattle is another example of a GLL parser.

                                                                                      The times I’ve read about Earley parsers, they seem like the same solution as GLL, but attacking the problem from a different angle. Hopefully someone in academia will eventually prove that.

                                                                                      1. 2

                                                                                        I’m particularly interested in Earley parsers because some use cases of mine assume ambiguity (reverse-engineering binaries, scraping damaged data), and the chart that Earley parsers build up supports several operations beyond just recognizing & building parse trees:

                                                                                        • what terminals/nonterminals would advance the current parse(s) in progress? (i.e., autocomplete)
                                                                                        • if we don’t assume where the starting position is, what overlapping instruction encodings are there?
                                                                                        • if we inserted a fake nonterminal here, would enough of the parse have completed to match any instances of (some structure) between here and there?
                                                                                        • incremental re-parsing of changing input, since earlier columns in the chart are constant once the parser has moved on.
                                                                                      2. 2

                                                                                        Thank you for adding this. I understand why everybody is excited about Earley parsing, though I personally don’t feel sufficiently grounded in it to share that excitement, but at the very least other lines of research deserve to be mentioned.