1. 5

    Is there a single Fuchsia user yet? When does that happen? Is this targeted at IOT? (I’ve heard that thrown around as a reason for its existence) or is this a take on what computers could be if you stripped away all the legacy?

    1. 16

      or is this a take on what computers could be if you stripped away all the legacy?

      I think that’s the technical ambition.
      A worth ambition, I would add, since I pursuit it too (if in a completely different direction).

      But the political reason is pretty obvious: to build a GPL-free system that can get rid of Linus & friends.
      Under Google’s own control.

      1. 2

        I’m a pretty heavy GPL advocate, but this at least is open source.

        1. 5

          Sadly this doesn’t mean much.

          Beyond my own experience with open source operating systems developed by Google employees, I’ve seen in the past that a liberal license means nothing when it comes to the power of such large companies.

          They have plenty of ways to discourage developers from using the freedoms that the license provides.

          The careful wording in the PATENTS file is probably one of these.

          Another could be unusual build infrastructure or huge build times (as in the case of Chromium).

          Indeed I do not remember any abandoned project from such companies that was later taken from a community (like in the case of open office, for example), but I’m happy to be corrected in this regard.

          1. 3

            I take your point regarding the reality that all such agreements rely on parties, particularly the more powerful ones, acting in good faith, and stories like the one you link about HarveyOS certainly should discourage contributing to non-GPLed (or similarly Freely licensed) projects. Or, one may engage with them that way by presuming that your contributions will be appropriated and unremarked. The ability to simply choose to fork the project initially and never give back, though, is something that somewhat mitigates that in the case of BSD, even as it goes against the FSF pro-sharing ethos.

            I’m not going to get into the patent release; I acknowledge your point but don’t think using that vector is likely. Same, I guess, with the technical discouragements like the build time in Chromium, because that’s extremely legit as a critique of the consequential impacts of that software. I do not think there’s an abstract philosophical defense against any of the points you raise, in the context of an FSF philosophy.

            But as long as we’re getting consequentialist here, there are reasons to believe that the interests of Google and the People are aligned on this project. I think their ambition is to have a Zircon-based OS running on devices from the scale of phones and watches on up to a datacenter-sized supercomputer, seamlessly participating in distributed computation, and the only way to achieve that is to make it as ubiquitous as possible. In order to do that, it needs to be developer-friendly, because as an operating system, it needs to appeal to people like us (technical users, developers, etc.), whereas something like Chromium doesn’t. Regarding my thesis about their ambition, from https://fuchsia.googlesource.com/docs/+/master/the-book/life_of_an_open.md (emphasis added here):

            Once the message has been transmitted from the client’s side of the channel, it lives in the server’s side of the channel, waiting to be read. The server is identified by “whoever holds the handle to the other end of the channel” – it may live in the same (or a different) process as the client, use the same (or a different) runtime than the client, and be written in the same (or a different language) than the client. By using an agreed-upon wire-format, the interprocess dependencies are bottlenecked at the thin communication layer that occurs over channels.

            I have seen good operating systems be BSD-licensed (for example, *BSD). I’m pretty stoked about this project.

            1. 7

              Well, from a technical point of view, I’m pretty sure it will be a great piece of software.
              And given how primitive are the mainstream operating systems right now, any piece of research is welcome for me.

              But I do not think that “the interests of Google and the People are aligned on this project”.
              I trust the Fuchsia developers’ skills, but Google is a huge corporation: its interests evolve independently from any ethical consideration, and its open source software is always a strategical marketing tool.

              Take Chromium, for example: it’s a great software that pushed the web forward (and JavaScript abuse with it, but this is another story), but its main purpose was to defeat Microsoft.
              I was naively fooled to think that the creation of Google Chrome Frame was a sort of philanthropic effort to helps those people who were stuck to IE. It was not.
              Indeed IE still sucks and many people are still forced to use it, but Google abandoned GCF when they won the war they cared about.

              While I still think that Microsoft purposely tried to slowdown the web for its own interests, Google was not better, just smarter: Google was able to dress its marketing strategies as philanthropic gifts, for years.

              Now they are doing the same with Fuchsia.

              If they will success, they will spread an os that is just “formerly” open source, but they will lead the development both technically and politically. And you will see a lot of effort from other companies too to make this happen. Just like such companies invest in LLVM, another great piece of software that is designed to defeat the most important piece of code controlled by the Free Software Foundation: GCC.

              As of today, indeed, I don’t think big companies are scared by the “virality” of the GPL.

              They are much, much more scared by the message it carries: programming is a powerful political act. Probably one of the most powerful.

              Had this realization to mix with the consciousness of how primitive is our field, they would completely loose control of their programmers (that they already try to keep under strict control with various methods).

              So just like Microsoft did not really understand what was happening with Chrome, now everybody looks at Fuchsia just for its technical design.

              But don’t get fooled, the enemy now is the true hacker’s culture that people like Richard Stallman, Linus Torvalds and Theo de Raadt, despite their differences (or maybe exactly for their differences), represent.

              1. 3

                I can’t argue with you, and even wrote similarly last year: https://blog.joeardent.net/2017/01/say-no-to-corporate-friendly-licenses/

                1. 1

                  Nice read.

                  We live in a world where a group of eight people have control of more economic means than the poorest four billion people, and the power disparity between our corporate masters and regular humans is unimaginably vast. There’s very little that you or I can do, but we do have one ace up our sleeves: we write software, and software increases our leverage. So don’t give that leverage to the leviathans trying to commoditize you.

                  I really think that we, as hackers, should deeply reflect on the power that we have.

                  And on the responsibility that comes with it.

              2. 2

                I’ve been interested in distributed, capability-based systems for a while, since EROS/CoyotOS in the early 2000s, and Zircon/Fuschia seem like the best shot at embodying the spirit of those systems, as in, eg, http://www.capros.org/overview.html

              3. 1

                How do huge build times restrict developers’ freedoms? Does Gentoo suffer the same fate?

                1. 2

                  Well, in 2015 I was asked by one of our customers (a large multinational bank) to evaluate the fork Google Chrome Frame when Google retired its support.
                  They used it over IE8 for their intranet (a pretty secure environment, AFAICT) and just wanted someone to call in case of functional bugs for a couple of years.
                  The budget they were ready to pay for this request was pretty high, and the technical skill of my team were (and are) pretty amazing.

                  If I remember correctly, it took 3 days to get the build complete on my desktop.

                  Managers were already pretty scared by the risks (that for the first time in their lives, they were over-estimating), so when I said this, they incredibly decided to refuse such opportunity.

                  Explaining that we just needed another server for the continuous build, was not enough.

                  This despite the Chromium license.

                  1. 2

                    It sucks that you lost a customer and I don’t want to get into the details of your build setup, but maybe the long build times are just inherent to a large C++ application? I understand your argument that a company with deep pockets can afford to work on more complex software faster, but was compiling Chromium that much slower than a comparable non-profit project like Firefox?

                    1. 1

                      Also, I was implicitly (sorry) replying to your comparison with Gentoo.

                      While maybe the Gentoo team have a continuous build setup comparable to that of Google, few people need to fork Gentoo because they have been locked in.
                      And even those who do, would not build it frequently as a whole.

                      Forking a single complex application is a completely different matter.

                      1. 0

                        It sucks that you lost a customer

                        Well… it’s still one of our best customers!
                        But we lost a pretty interesting work (both economically and technically).

                        I don’t want to get into the details of your build setup

                        Frankly I do not remember much. It was the first and last time I had to build Google Chrome Frame.
                        But I remember it was not an easy task: I remember an unusual number of WTF (why the hell they did it so?)

                        I understand your argument that a company with deep pockets can afford to work on more complex software faster, but was compiling Chromium that much slower than a comparable non-profit project like Firefox?

                        Honestly, I didn’t try.
                        But as far as I can read, there is still a huge factor, between Firefox and Chrome.

                        The point however is that people should realize that Open Source from big companies is just marketing.

                        At best, they do not really care about external developers (as opposite to Free Software).

                        Sometimes they just look for minions that work for free for them.

                        At worst, they are fooling developers to give up their freedom (and power).
                        And they are pretty good at this: I’ve talked with many smart developers that were not aware of how much they were working against their own long term interests.

                        Note that the problem is not the license and while I value collaboration over competition, alternatives are always a good thing. The problem is who controls the projects for real.

                        1. 2

                          I am familiar with the high level of WTF for building Chromium (and AOSP too, right?).

                          Do you think it has anything to do with this? https://lobste.rs/s/mbufwv/some_software_cannot_be_used_at_google

                          make is GPL. So is autoconf. Etc?

                          1. 1

                            Maybe. Frankly I cannot say.

                            I build my OS with a few disposable Go scripts. I will replace Go with a simpler general purpose language asap, but I will not use GNU Make or Autoconf (nor cmake or worse shits) because I want to see how far you can follow a minimalist approach.

                            However building Jeanne, from source to the first drawterm connection, requires just a handful of commands documented in the README.

                            And Jehanne is pure research.

                            But I think that if I were going to write a Free Software browser, I would use the most battle tested and well known tools, exactly to minimize WTFs.

                            In practice, it depends on what you want to do.

                            If your goal is to create a distributed system that replace current computing techniques from the ground up, you probably do not care much with compatibility: your gift to the world is the brave innovation that breaks every conventions.

                            On the other hand, if you want to create a Free Software for the current world you should focus on that and maximize for developer’s friendliness.

                            But what if you want to spread a new browser to break a monopoly while keeping the full control of that software?

                            Open source to the rescue!

            2. 8

              I think, like many of Google’s other “extra” projects, it’s mainly a hedge. It makes sure they can’t be boxed in in the future if something unfavorable happens with Linux, just like Android was created mainly to ensure they wouldn’t ever be at the mercy of other mobile OS makers choosing to switch away from the Google ecosystem. Like Android, if it turns out to be a pretty good OS that becomes massively popular, that’s a bonus for them. If it turns out to be a disaster, they can abandon it, it’s pocket change for them.

              1. 4

                My guess is that Google found that both ChromeOS and Android aren’t enough for business users, and are going all in on developing an alternative to Windows and macOS. Instead of .NET/C#, or Cocoa/Obj-C, you have Flutter/dart.

                I’d expect a series of notebooks, eventually, that are similar, hardware wise to Chromebook Pixel, running Fuschia, that can be members of the “Google Apps domain controller” but have a traditional set of applications that don’t just run in a browser.

                1. 4

                  If you look on YouTube, you’ll see people actually running the graphical/desktop Fuchsia, either in VMs or on Google hardware. A lot of stuff is broken, but it gives you an idea of what the graphical environment will eventually look like.

                  1. 1

                    The plan to strip away all the legacy from unix was apparently numbered nine, which fell off with its parent company.

                    It’s good that google is carrying it over with a bucket of bright paint. Namespace is great. FIDL is really just two letter more than fd. I really hope google will carry it through. No one want a plan b.

                    Once Fuchsia achieves acme, I will be happy to run it in qemu.

                  1. 3
                    1. 3

                      I’d be curious to know if there is a color scheme with a continuous mapping from light to dark.

                      1. 1

                        A lot of these linter plugins appear to be licensed under the AGPL, so heads up if your organization has concerns about this license.

                        1. 3

                          Please limit the size of pull requests under 300 lines, otherwise it would be rather hard to review the code. If you have a big feature to add, please consider splitting it into multiple pull requests.

                          This is amazing to have in a contributions guide, and I wish it was more prevalent. I think developers believe splitting things into small chunks slows things down and adds unnecessary overhead, but given how faster small PRs are reviewed they’re usually landed earlier and more frequently than their bulkier counterparts. Developers initially hate breaking up PRs so it’s good to have this sentiment front and center.

                          1. 3

                            I love jq but I find its syntax impossible to understand. Every time I use it I look up examples for even basic things. Maybe it makes more sense if you know javascript? I’m not saying I could design something better, just that I’ve struggled with it far more than I expected.

                            1. 5

                              I also have to look things up all the time with jq, takes a while to write scripts, but when they’re done, they work great. For exploratory stuff however, https://github.com/tomnomnom/gron is much simpler

                            1. 2

                              I feel like a lot of the author’s concerns (interruptibility, urgency) can be addressed by putting into place the recommendations of The Asshole Filter. Specifically:

                              • Encourage people to use low-interruption methods like tickets, email or bugs (whatever you or your team has declared ‘the right way’ to be contacted).
                              • If anyone attempts to priority-bump in an inappropriate manner, refuse to help until they go through the proper channel. “Sure I can help, can you please file a ticket so we can track this?” etc.
                              • Actively check and respond to items submitted ‘the right way.’ This is critical – people need to feel that the best way to get what they need is to go through the appropriate channel. As soon as people get the impression that tickets/emails/bugs are never checked, they’ll attempt to priority-bump and then you’re back to square one.
                              1. 23

                                I am often a dick, I am often difficult

                                Turns out this matters more than you’d think.

                                1. 2

                                  I am curious why SmartBear would change their own graph…

                                  1. 20

                                    I’ve worked on a team with twenty minute builds (also a C++ application), and found that all sorts of practices are rejected because “build times”. Test-first? I’m not waiting twenty minutes to see my failed test, so I’ll get it working then I’ll add tests. Actually, it looks like this is hard to test, I’ll skip it just this once. Continuous integration? I’m not twiddling my thumbs waiting for Jenkins, I’ll merge then fix any problems later. Pay down some technical debt? Why would I spend half a day moving the build from 20 mins to 19.5 mins? No thanks.

                                    On the other hand, who’s doing clean builds? If you change one file then your incremental build is, based on your estimate, around a third of a second. That’s acceptable. And Google presumably have some computers to throw at the problem to use distributed compilation and artefact caches. It’s possible that Google does not have compilation time problems, but does have data centre management and CI scale problems.

                                    1. 7

                                      There’s also the issue that some changes require recompilation of almost everything. Want to disable debug mode such that debug assertions don’t crash Chromium? Well, time to find something else to do for the day and let the build server compile chromium overnight. Let’s hope no issues come up during the compilation, because then maybe only half the project was compiled, and most of tomorrow goes to waiting for it to compile too.

                                      You’re probably right that Google has a ton of compute power to throw at the problem though.

                                      1. 6

                                        Our build system keeps two build trees, one for debug and one for release.

                                        1. 6

                                          Google has a ton of compute power to throw at the problem though.

                                          It does, and is how Chromium works around the problem – super beefy development machines.

                                        2. 1

                                          Test First in C++ can be beneficial. It can bias you toward making new abstractions with fewer dependencies rather than hacking things onto existing structures - because the latter gives you those 20 minute compile times. Better to have something that can just compile as cpp/h pair that stands alone with only a few includes.

                                          1. 1

                                            we have even longer build cycles, but we’ve learned the hard way enough times that a bug caught in CI is worth 10 caught in production, easily.

                                            Being patient and making sure that test tooling is always improving is also super important. Tests shouldn’t be written in a throwaway manner, because that makes the experience even that more miserable.

                                          1. 17

                                            If only json had allowed trailing commas in lists and maps.

                                            1. 9

                                              And /* comments! */

                                              1. 3

                                                And 0x... hex notation…

                                                1. 3

                                                  Please no. If you want structured configs, use yaml. JSON is not supposed to contain junk, it’s a wire format.

                                                  1. 4

                                                    But YAML is an incredibly complex and truth be told, rather surprising format. Every time I get it, I convert it to JSON and go on with my life. The tooling and support for JSON is a lot better, I think YAMLs place is on the sidelines of history.

                                                    1. 4

                                                      it’s a wire format

                                                      If it’s a wire format not designed to be easily read by humans, why use a textual representation instead of binary?

                                                      If it’s a wire format designed to be easily read by humans, why not add convenience for said humans?

                                                      1. 1

                                                        Things don’t have to be black and white, and they don’t even have to be specifically designed to be something. I can’t know what Douglas Crockford was thinking when he proposed JSON, but the fact is that since then it did become popular as a data interchange format. It means it was good enough and better than the alternatives at the time. And is still has its niche despite a wide choice of alternatives along the spectrum.

                                                        What I’m saying is that adding comments is not essential a sure-fire way to make it better. It’s a trade-off, with a glaring disadvantage of being backwards incompatible. Which warrants my “please no”.

                                                    2. 1

                                                      http://hjson.org/ is handy for human-edited config files.

                                                      1. 1
                                                      2. 5

                                                        The solutions exist!

                                                        https://github.com/json5/json5

                                                        I don’t know why it’s not more popular, especially among go people.

                                                        There is also http://json-schema.org/

                                                        1. 3

                                                          I had to do a bunch of message validation in a node.js app a while ago. Although as Tim Bray says the spec’s pretty impenetrable and the various libraries inconsistent, once I’d got my head round JSON Schema and settled on ajv as a validator, it really helped out. Super easy to dynamically generate per message-type handler functions from the schema.

                                                          1. 2

                                                            One rather serious problem with json5 is its lack of unicode.

                                                          2. 3

                                                            I think this only show that JSON has chosen tradeoff that make it more geared to be edited by software, but has the advantage of being human editable/readable for debugging. JSON as config is not appropriate. There is so many more appropriate format (toml, yaml or even ini come to mind), why would you pick the one that doesn’t allows comments and nice sugar such as trailing commas or multiline string. I like how kubernetes does use YAML as its configuration files, but seems to work internally with JSON.

                                                            1. 8

                                                              IMO YAML is not human-friendly, being whitespace-sensitive. TOML isn’t great for nesting entries.

                                                              Sad that JSON made an effort to be human-friendly but missed that last 5% that everyone wants. Now we have a dozen JSON supersets which add varying levels of complexity on top.

                                                              1. 11

                                                                “anything whitespace sensitive is not human friendly” is a pretty dubious claim

                                                                1. 5

                                                                  Solution: XML.

                                                                  Not even being ironic here. It has everything you’d want.

                                                                  1. 5

                                                                    And a metric ton of stuff you do not want! (Not to mention…what humans find XML friendly?)

                                                                    This endless cycle of reinvention of S-expressions with slightly different syntax depresses me. (And yeah, I did it too.)

                                                                    1. -5

                                                                      Triggered.

                                                                      1. 13

                                                                        Keep this shit off lobsters.

                                                              1. 2

                                                                It’s a bit long, but buried in the document is this:

                                                                Well-defined strict priorities, communicated to everyone, can get you to launch 2x-4x faster than just letting engineers work on whatever seemed important at the time.

                                                                1. 7

                                                                  A friend of mine is Québécois and lives in a French-speaking town on the border of English-speaking Ottawa. Using Google maps navigation is apparently a huge pain there, because the text-to-speech doesn’t know how to flip between languages mid-trip.

                                                                  1. 4

                                                                    It also cannot cope with Dutch street/place names when giving instruction in English. It really mangles things like Lage Rijndijk and Willem de Zwijgerlaan.

                                                                  1. 3

                                                                    Dhall looks really interesting. After being burned by turing-complete configuration languages in previous projects, I’m glad someone has taken the time to tackle the problem seriously.

                                                                    1. 32

                                                                      His stance is laid out more clearly later in the thread.

                                                                      People should basically always feel like they can update their kernel and simply not have to worry about it.

                                                                      I refuse to introduce “you can only update the kernel if you also update that other program” kind of limitations. If the kernel used to work for you, the rule is that it continues to work for you.

                                                                      And I seriously will refuse to take code from people who do not understand and honor this very simple rule.

                                                                      1. 23

                                                                        Also relevant is John Johansen’s response.

                                                                        1. 29

                                                                          What a difference between his first post and this one. In the first one he comes off like a colossally toxic asshat. I know this is no surprise to anyway, but still. That kind of behavior is not OK. Period.

                                                                          This post on the other hand is clear headed and explanatory. It lays out the rules and why it’s important to follow them.

                                                                          Maybe Linus just needs a 1h send buffer? :)

                                                                          1. 2

                                                                            “That behavior is not OK” is equivalent to “I am offended”, for this case.

                                                                            For all types of behavior, you can always find someone that thinks it is not OK. Should it matter? It would be severly limiting for everyone on a place like the Internet.

                                                                            1. 19

                                                                              It’s not “I am offended”, but rather probably 95% of people would be offended if they would hear something like this headed their way. Linus probably forgot how it’s like to hear this level of toxic communication because nobody speaks with him like that. I know his “ideology” behind his behavior (he talked about this several times), but honestly saying such “sh**” to people is low, and most people are above that, that’s why he stands out.

                                                                              1. 3

                                                                                Personally this power relationship is why I’m against BDFLs once a project reaches a certain size.

                                                                                1. 1

                                                                                  I agree in principle. In practice I have to wonder - what are the alternatives? Design by committee has some well known flaws :)

                                                                                2. -1

                                                                                  Do you even know what toxic means ?

                                                                                  1. 8

                                                                                    Toxic means that it is in some way damaging to a relationship between two individuals, groups, etc. In this case it is indeed toxic because it seeks to gain in some goal at the cost of the relationship with the submitters. Toxic isn’t strictly bad, sometimes a goal is so important that you need to break the relationship, however you should always choose the least toxic strategy that will ensure success. After all who knows when you’re going to need those people’s help in the future.

                                                                                    In summary, dark_grimoire seems to have a correct understanding of toxic, and mytrile does not which I assume is why they are being downvoted.

                                                                                3. 22

                                                                                  It would be severly limiting

                                                                                  It’s already limiting though – many people silently stop contributing when they receive messages like this or never consider contributing in the first place. This means the negative impact is hidden. Since it’s hidden, it becomes much easier to defend the status quo when an alternative might result in a better kernel.

                                                                                  1. 6

                                                                                    By the same logic, the positive impact is also hidden. Because it is conceivable that without these messages, the kernel might have imploded upon itself, and the prevention of said implosion is doubtlessly positive.

                                                                                    If you are going to argue with hidden stuff then it goes both ways.

                                                                                    1. 10

                                                                                      Do you really believe that it’s not possible to enforce rules and maintain high standards without calling people idiots, their contributions garbage, and so on?

                                                                                      I can certainly believe the parent comment, as it’s something I hear regularly, from people who decide not to get involved in projects/make further contributions/pursue opportunities at companies/etc because of things like this. FWIW, one of my friends can be found in the kernel CREDITS, and decided to walk away because of the LKML.

                                                                                      1. 6

                                                                                        it is conceivable that without these messages, the kernel might have imploded upon itself

                                                                                        As a counterpoint, I’ve worked on a project that has a similar code size, customer reach, and zero-tolerance stance on security and stability bugs as the Linux kernel: Chromium. Chromium does not have anywhere near the level of abusive discourse on its mailing list as the LKML, and it has not imploded on itself as you have suggested. So the burden of proof is on the abusive language to show it is needed and not the other way around.

                                                                                    2. 7

                                                                                      I disagree. I am not offended by his behavior, I find it to be unacceptable by virtue of the fact that I feel human beings should treat each other with a modicum of respect. Linus’s communications very often do not meet that standard. Hence from my book they do not represent an acceptable way to treat people, especially people volunteering to donate time to an open source project.

                                                                                    3. -5

                                                                                      Who are you to say what’s OK?

                                                                                      1. 16

                                                                                        He can certainly say what’s OK and NOT OK in his opinion.

                                                                                    1. 7

                                                                                      A while back we had an effort to make a ‘constitution’ which resulted in the text linked here. Whether we use that particular text or not, I still think we need some kind of official posting guidelines for the site (a stance which the OP disagrees with). I can see two main uses: clarity on moderation for cases like this one, and as a guide for newcomers to the community who want to know what is acceptable and unacceptable here.

                                                                                      Right now the entirety of the written guidance is on tagging, not posting.

                                                                                      1. 12

                                                                                        (a stance which the OP disagreed with at the time).

                                                                                        I still stand by my previous comments on the subject. Quoting myself from the linked thread:

                                                                                        I never saw substantial abuse in this community and if that happened we have moderators in place to take care of it.

                                                                                        Personally if I have to go through a document like the suggested draft before commenting/submitting an URL I would be not contributing to this community.

                                                                                        We are not a bunch of kids and the transparent moderation policy is all we need to keep things in line.

                                                                                        I am currently exercising the transparent moderation policy to know why a comment thread was censored and why down-voting the thread was not sufficient enough.

                                                                                        1. 6

                                                                                          I still stand by my previous comments

                                                                                          (I didn’t mean to imply you didn’t now, there wasn’t a good way to phrase it neutrally. I’ll update the original comment to reflect that)

                                                                                          1. 7

                                                                                            No worries, I didn’t misunderstand your intention - just voiced my current stance in case anyone wondered.

                                                                                      1. 1

                                                                                        Does this also break motions on verbs? Losing d<N>j seems like a big loss if you use relative numbering.

                                                                                        1. 1

                                                                                          I turned off relative numbering – as you point out, it’s not very useful if you don’t have hjkl. With absolute numbering you can jump wherever using G. d142G works just as well as d5j or whatever.

                                                                                          1. 17

                                                                                            The wikipedia comparison is pretty good, but it’s still lacking. Some things that are important to me in a serialisation protocol:

                                                                                            • Binary: As soon as you open the door to text you get questions about encoding and whitespace, and it becomes very difficult to process efficiently. The edge cases will haunt you in a way they never will with a binary protocol.
                                                                                            • Self-describing: It should be possible for a program to read an arbitrary object (unlike XDR, Protocol Buffers, etc)
                                                                                            • Efficiency: JSON/MessagePack/XML are out, but so is DER (ASN.1) because integers are variable length
                                                                                            • Explicit references/cycles (e.g. plain JSON, but not Capt’nProto or PHP’s serialize)
                                                                                            • Lots of types: ASN.1 has the right idea here, but it still falls short.
                                                                                            • Unambiguous: Fuck MessagePack. Seriously.

                                                                                            On the subject of types: k/q supports booleans, guids, bytes, shorts(16bit), ints(32bit), longs(64bit), real(32bit), float(64bit), characters, symbols, timestamps, months, dates, timespans (big interval), minutes, seconds, times, all as arrays or as scalars. It also supports enumerated types, plain/untyped lists, and can serialise functions (since the language is functional). None of the blog-poster’s suggestions can stand up to the kdb ipc/protocol, so clearly we needed at least one more protocol, but now what?

                                                                                            Something else I’m thinking about are capabilities/verified cookies. I don’t know if these can/should be encoded into the IPC (I tried this for a while in my ad server), but there was a clear advantage in having the protocol decoder abort early, so maybe a semantic layer should exist where the programmer can resolve references or cookies (however if you do it as a separate pass of the data, you’ll have efficiency problems again).

                                                                                            I think that if you can get away with an existing format, you should use it because you get to inherit all of the tooling that goes with it, but dogma that suggests serialisation is a solved problem is completely and obviously wrong.

                                                                                            1. 9

                                                                                              Cycles are hard to use safely. In my opinion, it’s much better to encode them explicitly when you need them (not that often) than to include them in the format itself. Other than that, I agree completely.

                                                                                              It is also important for the format to have a canonical form for when you deal with cryptography. Also, not having various lengths of numeric data types that are all treated differently is a great boon for current scripting languages.

                                                                                              Have you seen RFC 7049: Concise Binary Object Representation (CBOR)? It has JSON semantics with additional support for chunked transfers and sub-typing (interpret this string as a date or something).

                                                                                              RFC 8152: CBOR Object Signing and Encryption (COSE) also sounds promising. I believe that it’s time for DER and the whole ASN.1 world to go.

                                                                                              1. 7

                                                                                                I was surprised not to see CBOR in the list of formats, it’s actually an incredibly elegant encoding which can be efficiently decoded and provides a huge amount of flexibility at the same time (and is sensible enough to leave space to add more things if they become necessary). Haskell’s serialise and cborg libraries have adopted it, and I hope these will become the canonical serialisation format for Haskell data, replacing the really ad hoc and less efficient encoding currently offered by the binary and cereal packages.

                                                                                                CBOR is a protocol done right, with standardisation and even IANA registry for tags and other stuff. It’s also part of the COAP REST-for-IoT-but-efficient standard (thing - I’m not familiar enough with exactly what COAP does).

                                                                                                Edit: video describing the protocol and how it’s likely to be used in Haskell https://youtu.be/60gUaOuZZsE

                                                                                                1. 0

                                                                                                  What do you mean by “JSON semantics”? JSON has really terrible semantics, especially around numbers.

                                                                                                  1. 5

                                                                                                    I should have said JSON-compatible semantics.

                                                                                                    Most of the types in CBOR have direct analogs in JSON. All JSON values, once decoded, directly map into one or more CBOR values.

                                                                                                    The conversion from CBOR to JSON is lossy. CBOR supports limited-size integers, arbitrary-precision integers and floats. It also has support for NaN and Infinities.

                                                                                                    1. 1

                                                                                                      By this definition, is there any serialization format that is not JSON-compatible?

                                                                                                2. 1

                                                                                                  Always great to see another k programmer around. Been several years but man that was a trip.