1. 11

    Recall what happened: we decided to represent characters of text, including punctuation and whitespace, using fixed numbers (a practical, though rather dubious, decision; again, a product of early computing)

    Around here is where he lost me, how else are we supposed to do it? As he admits, computers just manipulate bits, if you want to represent anything else you’ll need a mapping from those bits to your character set.

    1. 2

      There’s no term for it, but the alternative is structured data: a “binary” AST with data at the leafs, and if that data is text it has to be with explicit mentioning of the character table.

      1. 1

        Everything just becomes a single node containing a blob of text again.

        Messaging actually shows promise, as does networking, but I think the former is more likely than the latter (despite much more energy being put into the latter by researchers)

        1. 2

          Why would everything become a single node with a blob of text? This is evidently true when it would have to work with systems that were primarily made to deal with unstructured text files, but that’s the circular argument addressed in the article.

          Messaging and networking are different solutions for problems not mentioned in the article. Not sure how it would help with the “build and destroy the world” issue.

          1. 6

            We get blobs of text because of serialisation, something we need to do to stream out to spinning platters made of rust, or to beam waves over a wire.

            We often prefer to choose the serialisation format because it’s always faster than a general purpose serialisation format one might try for baking down that “AST”: Remember, even if we have no other opinion we can choose from S-expressions and XML and JSON and ASN.1 and PkZip and so on. Each with different disadvantages.

            And once you serialise, you might as well freeze them someplace. Maybe a hierarchy. This thing is called a file system, and those frozen blobs are called files.

            Messaging and networking are a way to build a platform that doesn’t have a filesystem of files. They aren’t mentioned in the article, but then: no solutions are really offered by the article.

            1. 1

              Messaging and networking are a way to build a platform that doesn’t have a filesystem of files. They aren’t mentioned in the article, but then: no solutions are really offered by the article.

              This is interesting, what do you mean by using messaging as a substitute for a filesystem, what would that look like?

              1. 2

                iOS does something like this (awkwardly; through a blessed but ad hoc mechanism). You send a message to another app- and ask it send a message back to you.

                One obvious use is storing things that we used to store in files, like photos and preference and music, but we can also use it to authenticate (who are you), to authorise (do you allow this), to purchase, and perhaps for other things.

                Urbit is exploring some of these themes in a much more grand scale, but is so much less a “complete” operating environment at this point to teach us what computing will be like in this way.

                HDCP/HDMI has another (limited) use of this where you can play a (protected) video at an x/y/w/h without revealing the bits.

                The Mother of All Demos hinted at some of this with their collaborative single user super computer.

                And so on.

    1. 23

      Kinda late on UNIX bashing bandwagon :)

      Also, Windows owes more of it’s legacy to VMS.

      1. 10

        It does, but users don’t use any of the VMSy goodness in Windows: To them it’s just another shitty UNIX clone, with everything being a file or a program (which is also a file). I think that’s the point.

        Programmers rarely even use the VMSy goodness, especially if they also want their stuff to work on Mac. They treat Windows as a kind of retarded UNIX cousin (which is a shame because the API is better; IOCP et al)

        Sysadmins often struggle with Windows because of all the things underneath that aren’t files.

        Message/Object operating systems are interesting, but for the most part (OS/2, BeOS, QNX) they, for the most part, degraded into this “everything is a file” nonsense…

        Until they got rid of the shared filesystem: iOS finally required messaging for applications to communicate on their own, and while it’s been rocky, it’s starting to paint a picture to the next generation who will finally make an operating system without files.

        1. 10

          If we talk user experiences, it’s more a CP/M clone than anything. Generations later, Windows still smells COMMAND.COM.

          1. 6

            yes, the bowels are VMS, the visible stuff going out is CP/M

            1. 3

              Bowels is a good metaphor. There’s good stuff in Windows, but you’ve got to put on a shoulder length glove and grab a vat of crisco before you can find any of it.

          2. 10

            I think you’re being a little bit harsh. End-users definitely don’t grok the VMSy goodness; I agree. And maybe the majority of developers don’t, either (though I doubt the majority of Linux devs grok journald v. syslogs, really understand how to use /proc, grok Linux namespaces, etc.). But I’ve worked with enough Windows shops to promise you that a reasonable number of Windows developers do get the difference.

            That said, I have a half-finished book from a couple years ago, tentatively called Windows Is Not Linux, which dove into a lot of the, “okay, I know you want to do $x because that’s how you did it on Linux, and doing $x on Windows stinks, so you think Windows stinks, but let me walk you through $y and explain to you why it’s at least as good as the Linux way even though it’s different,” specifically because I got fed up with devs saying Windows was awful when they didn’t get how to use it. Things in that bucket included not remoting in to do syswork (use WMI/WinRM), not doing raw text munging unless you actually have to (COM from VBScript/PowerShell are your friends), adapting to the UAC model v. the sudo model, etc. The Windows way can actually be very nice, but untraining habits is indeed hard.

            1. 6

              I don’t disagree with any of that (except maybe that I’m being harsh), but if you parse what I’m saying as “Windows is awful” then it’s because my indelicate tone has been read into instead of my words.

              The point of the article is that those differences are superficial, and mean so very little to the mental model of use and implementation as to make no difference: IOCP is just threads and epoll, and epoll is just IOCP and fifos. Yes, IOCP is better, but I desperately want to see something new in how I use an operating system.

              I’ve been doing things roughly the same way for nearly four decades, despite the fact that I’ve done Microsoft/IBM for a decade, Linux since Slackware 1.1 (Unix since tapes of SCO), Common Lisp (of all things) for a decade, and OSX for nearly that long. They’re all the same, and that point is painfully clear to anyone who has actually used these things at a high level: I edit files, I copy files, I run programs. Huzzah.

              But: It’s also obvious to me who has gone into the bowels of these systems as well: I wrote winback which was for a long time the only tools for doing online Windows backups of standalone exchange servers and domain controllers; I’m the author of (perhaps) the fastest Linux webserver; I wrote ml a Linux emulator for OSX; I worked on ECL adding principally CL exceptions to streams and the Slime implementation. And so on.

              So: I understand what you mean when you say Windows is not Linux, but I also understand what the author means when he says they’re the same.

              1. 2

                That actually makes a ton of sense. Can I ask what would qualify as meaningfully different for you? Oberon, maybe? Or a version of Windows where WinRT was front-and-center from the kernel level upwards?

                1. 2

                  I didn’t use the term “meaningfully different”, so I might be interpreting your question you too broadly.

                  When I used VMS, I never “made a backup” before I changed a file. That’s really quite powerful.

                  The Canon Cat had “pages” you would scroll through. Like other forth environments, if you named any of your blocks/documents it was so you could search [leap] for them, not because you had hierarchy.

                  I also think containers are very interesting. The encapsulation of the application seems to massively change the way we use them. Like the iOS example, they don’t seem to need “files” since the files live inside the container/app. This poses some risk for data portability. There are other problems.

                  I never used Oberon or WinRT enough to feel as comfortable commenting about them as I do about some of these other examples.

              2. 2

                If it’s any motivation I would love to read this book.

                Do you know of any books or posts I could read in the meantime? I’m very open to the idea that Windows is nice if you know which tools and mental models to use, but kind of by definition I’m not sure what to Google to find them :)

                1. 4

                  I’ve just been hesitant because I worked in management for two years after I started the book (meaning my information atrophied), and now I don’t work with Windows very much. So, unfortunately, I don’t immediately have a great suggestion for you. Yeah, you could read Windows Internals 6, which is what I did when I was working on the book, but that’s 2000+ pages, and most of it honestly isn’t relevant for a normal developer.

                  That said, if you’ve got specific questions, I’d love to hear them. Maybe there’s a tl;dr blog post hiding in them, where I could salvage some of my work without completing the entire book.

              3. 7

                but users don’t use any of the VMSy goodness in Windows: To them it’s just another shitty UNIX clone, with everything being a file or a program (which is also a file). I think that’s the point.

                Most users don’t know anything about UNIX and can’t use it. On the UI side, pre-NT Windows was a Mac knockoff mixed with MSDOS which was based on a DOS they got from a third party. Microsoft even developed software for Apple in that time. Microsoft’s own users had previously learned MSDOS menu and some commands. Then, they had a nifty UI like Apple’s running on MSDOS. Then, Microsoft worked with IBM to make a new OS/2 with its philosophy. Then, Microsoft acquired OpenVMS team, made new kernel, and a new GUI w/ wizard-based configuration of services vs command line, text, and pipes like in UNIX.

                So, historically, internally, layperson-facing, and administration, Windows is a totally different thing than UNIX. Hence, the difficulty moving Windows users to UNIX when it’s a terminal OS with X Windows vs some Windows-style stuff like Gnome or KDE.

                You’re also overstating the everything is a file by conflating OS’s that store programs or something in files vs those like UNIX or Plan 9 that use file metaphor for about everything. It’s a false equivalence: from what I remember, you don’t get your running processes in Windows by reading the filesystem since they don’t use that metaphor or API. It’s object based with API calls specific to different categories. Different philosophy.

                1. 3

                  Bitsavers has some internal emails from DEC at the time of David Cutler’s departure.

                  I have linked to a few of them.

                  David Cutler’s team at DECwest was working on Mica (an operating system) for PRISM (a RISC CPU architecture). PRISM was canceled in June of 1988. Cutler resigned in August of 1988 and 8 other DECwest alumni followed him at Microsoft.

              4. 5

                I have my paper copy of The Unix Hater’s Handbook always close at hand (although I’m missing the barf bag, sad to say).

                1. 5

                  I always wanted to ask the author of The Unix Hater’s Handbook if he’s using Mac OS X


                  1. 5

                    It was edited by Simson Garfinkel, who co-wrote Building Cocoa Applications: a step-by-step guide. Which was sort of a “port” of Nextstep Programming Step One: object-oriented applications

                    Or, in other words, “yes” :)

                    1. 2

                      Add me to the list curious about what they ended up using. The hoaxers behind UNIX admitted they’ve been coding in Pascal on Macs. Maybe it’s what the rest were using if not Common LISP on Macs.

                  2. 7

                    Beat me to it. Author is full of it right when saying Windows is built on UNIX. Microsoft stealing, cloning, and improving OpenVMS into Windows NT is described here. This makes the Linux zealots’ parodies about a VMS desktop funnier given one destroyed Linux in desktop market. So, we have VMS and UNIX family trees going in parallel with the UNIX tree having more branches.

                    1. 4

                      The author doesn’t say Windows is built on Unix.

                      1. 5

                        “we are forced to choose from: Windows, Apple, Other (which I shall refer to as “Linux” despite it technically being more specific). All of these are built around the same foundational concepts, those of Unix.”

                        Says it’s built on the foundational concepts of UNIX. It’s built on a combo of DOS, OS/2, OpenVMS, and Microsoft concepts they called the NT kernel. The only thing UNIX-like was the networking stack they got from Spider Systems. They’ve since rewritten their networking stack from what I heard.

                        1. 4

                          Says it’s built on the foundational concepts of UNIX.

                          I don’t see any reason to disagree with that.

                          The only thing UNIX-like …

                          I don’t think that’s a helpful definition of “unix-like”.

                          It’s got files. Everything is a file. Windows might even be a better UNIX than Linux (since UNC)

                          Cutler might not have liked UNIX very much, but Windows NT ended up UNIX anyway because none of that VMS-goodness (Versions, types, streams, clusters) ended up in the hands of Users.

                          1. 10

                            It’s got files. Everything is a file.

                            Windows is object-based. It does have files which are another object. The files come from MULTICS which UNIX also copied in some ways. Even the name was a play on it: UNICS. I think Titan invented the access permissions. The internal model with its subsystems were more like microkernel designs running OS emulators as processes. They did their own thing for most of the rest with the Win32 API and registry. Again, not quite how a UNIX programming guide teaches you to do things. They got clustering later, too, with them and Oracle using the distributed, lock approach from OpenVMS.

                            Windows and UNIX are very different in approach to architecture. They’re different in how developer is expected to build individual apps and compose them. It wasn’t even developed on UNIX: they used OS/2 workstations for that. There’s no reason to say Windows is ground in the UNIX philosophy. It’s a lie.

                            “Windows NT ended up UNIX anyway because none of that VMS-goodness (Versions, types, streams, clusters) ended up in the hands of Users.”

                            I don’t know what you’re saying here. Neither VMS nor Windows teams intended to do anything for UNIX users. They took their own path except for networking for obvious reasons. UNIX users actively resisted Microsoft tech, too. Especially BSD and Linux users that often hated them. They’d reflexively do the opposite of Microsoft except when making knockoffs of their key products like Office to get desktop users.

                            1. 3

                              Windows is object-based.

                              Consider what methods of that “object” a program like Microsoft Word must be calling besides “ReadFile” and “WriteFile”.

                              That the kernel supports more methods is completely pointless. Users don’t interact with it. Programmers avoid it. Sysadmins don’t understand it and get it wrong.

                              I don’t know what you’re saying here.

                              That is clear, and yet you’re insisting I’m wrong.

                              1. 3

                                Except, that’s completely wrong.

                                I just started Word and dumped a summary of its open handles by object type:

                                C:\WINDOWS\system32>handle -s -p WinWord.exe
                                Nthandle v4.11 - Handle viewer
                                Copyright (C) 1997-2017 Mark Russinovich
                                Sysinternals - www.sysinternals.com
                                Handle type summary:
                                  ALPC Port       : 33
                                  Desktop         : 1
                                  Directory       : 3
                                  DxgkSharedResource: 2
                                  DxgkSharedSyncObject: 1
                                  EtwRegistration : 324
                                  Event           : 431
                                  File            : 75
                                  IoCompletion    : 66
                                  IoCompletionReserve: 1
                                  IRTimer         : 8
                                  Key             : 171
                                  KeyedEvent      : 24
                                  Mutant          : 32
                                  Process         : 2
                                  Section         : 67
                                  Semaphore       : 108
                                  Thread          : 138
                                  Timer           : 7
                                  Token           : 3
                                  TpWorkerFactory : 4
                                  WaitCompletionPacket: 36
                                  WindowStation   : 2
                                Total handles: 1539

                                Each of these types is a distinct kernel object with its own characteristics and semantics. And yes, you do create and interact with them from user-space. Some of those will be abstracted by lower-level APIs, but many are directly created and managed by the application. You’ll note the number of open “files” is a very small minority of the total number of open handles.

                                Simple examples of non-file object types commonly manipulated from user-land include Mutants (CreateMutex) and Semaphores (CreateSemaphore). Perhaps the most prominent example is manipulating the Windows Registry; this entails opening “Key” objects, which per above are entirely distinct from regular files. See the MSDN Registry Functions reference.

                                1. 0

                                  None of these objects can exist on a disk; they cannot persist beyond shutdown, and do not have any representation beyond their instantaneous in-memory instance. When someone wants an “EtwRegistration” they’re creating it again and again.

                                  Did you even read the article? Or are you trolling?

                                  1. 3

                                    None of these objects can exist on a disk; they cannot persist beyond shutdown, and do not have any representation beyond their instantaneous in-memory instance. When someone wants an “EtwRegistration” they’re creating it again and again.

                                    Key objects do typically exist on disk. Albeit, the underlying datastore for the Registry is a series of files, but you never directly manipulate those files. In the same sense you may ask for C:\whatever.txt, you may ask for HKLM:\whatever. We need to somehow isolate the different persisted data streams, and that isolation mechanism is a file. That doesn’t mean you have to directly manipulate those files if the operating system provides higher-level abstractions. What exactly are you after?

                                    From the article:

                                    But in Unix land, this is a taboo. Binary files are opaque, say the Unix ideologues. They are hard to read and write. Instead, we use Text Files, for it is surely the path of true righteousness we have taken.

                                    The Windows Registry, which is a core part of the operating system, is completely counter to this. It’s a bunch of large binary files, precisely because Microsoft recognised storing all that configuration data in plain text files would be completely impractical. So you don’t open a text file and write to it, you open a Registry key, and store data in it using one of many predefined data types (REG_DWORD, etc…).

                                    Did you even read the article? Or are you trolling?

                                    It sounds like you’re not interested in a constructive and respectful dialogue. If you are, you should work on your approach.

                                    1. -3

                                      What exactly are you after?

                                      Just go read the article.

                                      It’s about whether basing our entire interactions with a computer on a specific reduction of verbs (read and write) is really exploring what the operating system can do for us.

                                      That is a very interesting subject to me.

                                      Some idiot took party to the idea that Windows basically “built on Unix” then back-pedalled it to be about whether it was based on the same “foundational” concepts, then chooses to narrowly and uniquely interpret “foundational” in a very different way than the article.

                                      Yes, windows has domains and registries and lots of directory services, but they all have the exact same “file” semantics.

                                      But now you’re responding to this strange interpretation of “foundational” because you didn’t read the article either. Or you’re a troll. I’m not sure which yet.

                                      Read the article. It’s not well written but it’s a very interesting idea.

                                      Each of these types is a distinct kernel object with its own characteristics and semantics

                                      Why do you bring this up in response to whether Windows is basically the same as Unix? Unix has lots of different kernel “types” all backed by “handles”. Some operations and semantics are shared by handles of different types, but some are distinct.

                                      I don’t understand why you think this is important at all.

                                      It sounds like you’re not interested in a constructive and respectful dialogue. If you are, you should work on your approach.

                                      Do you often jump into the middle of a conversation with “Except, that’s completely wrong?”

                                      Or are you only an asshole on the Internet?

                                      1. 4

                                        Or are you only an asshole on the Internet?

                                        I’m not in the habit of calling people “asshole” anywhere, Internet or otherwise. You’d honestly be more persuasive if you just made your points without the nasty attacks. I’ll leave it at that.

                              2. 2

                                networking for obvious reasons

                                Them being what? Is the BSD socket API really the ultimate networking abstraction?

                                1. 7

                                  The TCP/IP protocols were part of a UNIX. AT&T gave UNIX away for free. They spread together with early applications being built on UNIX. Anyone reusing the protocols or code will inherit some of what UNIX folks were doing. They were also the most mature networking stacks for that reason. It’s why re-using BSD stacks was popular among proprietary vendors. On top of the licensing.

                                  Edit: Tried to Google you a source talking about this. I found one that mentions it.

                    1. 1

                      This is a neat trick!

                      1. 2

                        I read the whitepaper last night, it’s worrying that SipHash is being used in this way. A second-preimage attack against SipHash could help speed up the low-memory algorithm and SipHash wasn’t explicitly designed to be secure against those when the secret key is known.

                        Zooko says it better than I can:

                        One detail that bothers me is that SipHash is being used in Cuckoo PoW in a way that the attacker gets to control all the inputs to SipHash, and that is not what SipHash was designed to resist. SipHash was designed to resist an attacker who doesn’t control — and actually doesn’t even know — the key. There’s a possibility (although it seems unlikely to me) that an attacker could exploit something about the way Cuckoo uses SipHash to find Cuckoo solutions faster than by treating SipHash as a random oracle.

                        1. 3

                          I love the idea of giving out bounties to anyone who can speed up the miners.

                          If you’ve found a speed-up it makes far more sense to claim the bounty now than to hope nobody else notices before a public chain launches and becomes valuable.

                          1. 8

                            Sort of an aside discussion, but the author’s choice to distribute the code as a Docker image: is that becoming a thing now?

                            I’m notorious among my peers for installing and trying everything under the sun, and usually having to blow out and reinstall my computer about once a year (usually coinciding with Apple’s release of an updated MacOS). Maybe I’m late to the party, but Docker images are a much cleaner way of distributing projects in a working environment, are they not?

                            1. 13

                              This feels like the kind of thing I’d be grumpy about if I were any older; software distribution is one of our oldest and therefore most-studied problems. Java tried to solve it with a universal runtime. Package managers try to solve it with an army of maintainers who manage dependencies. Giving up on all that and bundling the entirety of latex and all its dependencies (one of the intermediate images is 3 point 23 fucking gigs!) just to distribute a 279 line style file and make it easier to use feels… kind of excessive?

                              That said, I’m not old and grumpy and this is awesome. I kind of hope that this becomes a thing, it’s easy to install and easy to remove (and know that you’ve left no traces on your system) and this image will presumably be usable for a very long time.

                              EDIT: I wrote the above comment while I was waiting for the image to finish downloading. It’s now finished and the final image takes up 5.63GB of my disk space. I don’t mind for this one-off package but would certainly mind if this method of distribution started catching on. Maybe we should just all use nix?

                              1. 3

                                I wrote the above comment while I was waiting for the image to finish downloading. It’s now finished and the final image takes up 5.63GB of my disk space. I don’t mind for this one-off package but would certainly mind if this method of distribution started catching on. Maybe we should just all use nix?

                                Docker has some mechanisms for sharing significant parts of those images… at least if they’re created from the same base. The problem obviously is that people are free to do whatever, so that sharing is far from optimal.

                                1. 1

                                  Agreed, I assumed this was going to be something like a 200 python script with maybe 2 or 3 dependencies.

                                2. 4

                                  A docker image is the new curl|sh install method.

                                  Basically ignore any concerns about security, updates, ‘I want this shit now Ma.’

                                  1. 4

                                    A random docker image is less likely to fuck up your home dir, though.

                                    1. 2

                                      I’ve spent a lot more time working with the shell than Docker. I find this Docker image a lot easier to understand and verify than various things I’ve been told to curl | sh.

                                      1. 1

                                        Couldn’t you just download and verify a script with curl -o filename.sh http://domain.name/filename.sh? How does a random Docker image end up being easier to verify? With a script you can just read through it, and verify what it does. With a Docker image you basically have to trust an image from 2014 of an entire operating system.

                                        This honestly looks like one of the worst candidates for a Docker image. You have a tiny plaintext file which is all this is installing, and you are being told to download a multi gigabyte blob. I can understand why some people recommend using Docker for development, and running things and places you might not have control of the entire system, it here just seems unnecessary here.

                                        1. 1

                                          I don’t see how it’s installing just a style. It’s installing TeX, which is a big hairy package.

                                          When I pull down the install for rustup, I end up with a 360 line shell script, which isn’t super easy to verify. For haskell’s stack, it’s 720. I swear I’ve seen 1500 before.

                                      2. 1

                                        Agree re security (you could get hacked) but at least it won’t accidentally wipe your hard drive while trying to uninstall (as has happened a few timed I’m aware of).

                                      3. 3

                                        In this case especially, as the instructions to install and configure are pretty painful:


                                        Oh, there are none. But there is this:


                                        As an aside, the Docker image has a couple of features I’m quite proud of (in a small way).

                                        1. The default command of the container outputs help text.

                                        2. If the convert_images.sh script spots a Makefile, it runs it, eg:


                                        which reduces build time significantly if you have a lot of images.

                                        1. 4

                                          Just scrolling through that second link gives me anxiety; oh my god this is going to go wrong in fifty different ways. Getting it all together in a configured package (docker image) was pretty smart.

                                          1. 3

                                            I don’t know… Looking at the second link, the install instructions are actually fairly simple if you have TeX and the dependencies installed. Even if you don’t, like it’s just a LaTeX distribution, the tikz package, and the xcolor-solarized package.

                                            In which case the instructions are only:

                                            $ cd ${LATEX_ROOT}/texmf/tex/latex && git clone https://github.com/Jubobs/gitdags.git
                                            $ kpsewhich gitdags.sty   # Check to see if gitdags.sty can be seen by TeX

                                            I feel like an entire Docker container is a little overkill. Might be an OK way to try the software, especially if you don’t have a TeX distribution installed, but it wouldn’t be a good way to actually use it.

                                            1. 1

                                              From the link:

                                              ‘First, do NOT use apt-get to install. The best is to install TexLive from the Tex Users Groug (TUG).’

                                              1. 1

                                                Yeah like I said, the instructions are fairly simple if you have a TeX distribution Installed. If that version does happened to be from a distribution, I’m sure it works anyways - he did say the best way.

                                                If you don’t happen to have TeX installed, it’s not that complicated to install it from manually from TUG anyways.

                                              2. 1

                                                Looking at the second link, the install instructions are actually fairly simple if you have TeX and the dependencies installed

                                                Yeah, I already have TeX on my system, so I don’t really see what the problem is.

                                              3. 1

                                                The default command of the container outputs help text.

                                                I haven’t seen that before; for a “packaging” container (rather than an “app deployment” container) it’s a nice touch I’ll be copying, thanks :-)

                                              4. 2

                                                I’ve used this very package with nix on OS X, I think a docker image is… a bit over the top personally. It wasn’t that bad to install and setup compared to any other latex package.

                                              1. 6

                                                I did this once. After some research I ended up using DokuWiki. It worked pretty well, I stuck with it for a few years. I liked how the database was just the filesystem, my backups were just tarballs of the data directory.

                                                My main gripe, and the reason why I’ve moved over to google keep + workflowy, is that it was difficult to edit pages while using my phone.

                                                1. 2

                                                  That’s a good point about editing it on mobile, that might be a sore spot.

                                                  1. 2

                                                    I had the same problem recently, I got pretty annoyed by the editor on my iPhone’s Safari so I’ve started a side project recently to sync Markdown on Dropbox into DokuWiki (https://github.com/milanaleksic/notesforlife). There are many very good mobile Markdown editors that can store into Dropbox… Now I’m happy since dokuwiki backup is trivial, it has search… everything I need

                                                  1. 13

                                                    I don’t have any research papers off hand, but I would love to read some criticisms of the blockchain/cryptocurrencies (in terms of technology rather than economics).

                                                    1. 6

                                                      This piece is pretty good.

                                                      1. 13

                                                        No, that piece is actually uniquely terrible. I mean, if you want to simply read a piece which throws cold water then sure, read it, but you’re probably looking for informed pieces. The linked article is written by somebody who doesn’t understand the space; it makes claims which can be trivially disproved by reading the relevant literature. (For example, he seems to think that systems like Filecoin are proposing we store bulk data directly onto a blockchain)

                                                        This one is much better.

                                                        Attack of the 50 foot blockchain, both the book and the blog, are also great. They go less into technical details and more into pointing out the crazy claims some people are making and the crazy schemes they’re proposing.

                                                        1. 6

                                                          This one is much better.

                                                          That looks like it might be worth including on the list. I’ll try to review it tonight.

                                                          1. 6

                                                            I thought it was a pretty good piece. Main factual errors are: eight years not ten, and bitcoin transactions haven’t been four cents since early 2015.

                                                            But I think his central thesis is sound: all of this is a series of very impressive solutions to things that aren’t in fact the problem. Key quote:

                                                            In conversations with bitcoin entrepreneurs and investors and consultants, there was often a lack of knowledge or even interest in how the jobs were being done today or what the value to the end user was.

                                                            The more general problem is that “blockchain” the buzzword has less and less to do with the weird trick Satoshi Nakamoto came up with. e.g. Hey, Estonia’s all Blockchain now! Everything there is on blockchains! Is this Good News for Bitcoin?? well, not quite, because their esteemed Guardtime KSI Blockchain isn’t actually, er, a blockchain. That sort of thing.

                                                            1. 2

                                                              Oh, it’s cool that you’re here. First off, thanks for the book, it gave me a very lovely Saturday!

                                                              In conversations with bitcoin entrepreneurs and investors and consultants, there was often a lack of knowledge or even interest in how the jobs were being done today or what the value to the end user was.

                                                              If this was his main point I would agree that it was a decent piece, but all of his examples really are terrible. He focuses on b2c applications; almost all of his reasons are how consumers already have much better alternatives than anything blockchain-powered currently provides.

                                                              I agree, however (1) there’s much more to the world than just b2c, (2) b2c is usually built as a wrapper on top of unfriendly systems. Dropbox, one of his examples, is built on top of s3, which isn’t much friendlier than Filecoin would be, if it succeeds. Filecoin has the promise of being much cheaper though, so you can imagine, 5 years from now, using a Filecoin-powered-Dropbox which gives you all the same features but is less than half the cost. I know that doesn’t sound revolutionary, but before I lose your interest by claiming even bigger things I wonder if you agree the above is possible.

                                                              He then prematurely generalizes, and claims that these failings prove the entire class of blockchain protocols are doomed. Again, I totally agree that consumers prefer airline miles to whatever libertarians mean by “sound money”. However, I think things like storage markets, prediction markets, incentivized mesh networking… all of these take time to build; and the author of this piece seems to think that because he can’t imagine them then they must surely not exist.

                                                              The more general problem is that “blockchain” the buzzword has less and less to do with the weird trick Satoshi Nakamoto came up with.

                                                              This is, in some circles, completely true. It’s something Matt Levine has written about; blockchain is often code for “let’s get decision makers to approve fixing some of our infrastructure by giving the work a sexy name”. You’re reading your own opinions into his piece though, I don’t think he ever argues this :)

                                                              1. 6

                                                                I probably am reading my own opinions into his paper to a degree, then agreeing with myself, yes :-) I was reading it as a businessperson’s frustration with fountains of BS that never quite work out.

                                                                I do think, though, that eight years in, the burden of proof is firmly on the blockchain promoters. I have no patience any more for purported blockchain systems that don’t exist yet, because their track record of achieving existence is really bad.

                                                                Hypotheticals in the blockchain space are so cheap as to be worthless. “Could” or “will” are words meaning “doesn’t”.

                                                                The bitcoin press don’t really help in this. CoinDesk in particular never seems to have seen a hypothetical it doesn’t like; all you need is to make your thing slightly blockchainy and they’ll post a piece of journalistic stenography about it. Its failure to eventuate six months later will never be noted. This is a big problem because the mainstream media assume good faith, that this is specialist press rather than boosterism press …

                                                                Try this yourself! Take any of these hype-filled proposals and replace “could” with “doesn’t”. I’ve got here a blockchain in healthcare paper - peer-reviewed!! though in a journal that made Beall’s List - that has “could” 81 times, all for things that DON’T EXIST. Perhaps one day they will! But the fact that they DON’T EXIST is in fact an important point.

                                                                The Lightning Network is one personal bugbear on this point. I have had this argument, live on video with a bitcoin advocate (they didn’t put that bit up, for some reason … though I still have my video) where I pointed out bitcoin’s utter failure to scale and got back “but look, the Lightning Network! (laugh)” as if that was a slam-dunk refutation. I had to point out the fact that LN’s been coming any moment now in 2015, 2016, 2017 and for 2018, and the most important thing about it is that it DOESN’T EXIST. And, y’know, its repeated failure to happen might be relevant to its value as a solution to the problem.

                                                                (Rant time: I just had another go-round recently with LN advocates. I am pleased to say the actual devs are much better to talk to than the advocates. There is now LN code! The crippling flaw of the whole LN design remains the path-finding algorithm, to go from arbitrary node A to arbitrary node B - which is the same problem as trying to solve a UUCP bang-path. What was the best that the most motivated computer scientists and sysadmins of the 1980s could manage? Either compute it by hand, or download monthly maps of the entire network and pathalias it. And they all ran screaming for DNS as soon as they had proper Internet. So to do a proper mesh network, rather than just a few hubs closely resembling the Visa-Mastercard-PayPal triumvirate, LN actually requires new computer science. The present alpha code uses “broadcast every channel and transaction to the whole network”, which is I think O(n^2) and scales to literally tens of nodes. And then, even if it all worked, the economics of the LN can’t possibly work and would seize up immediately. But as far as I can tell, they literally don’t have a planned or anticipated economic model; certainly there’s nothing in years of dev list messages referencing any sort of thing they’re aiming for, which any reasonable person would think was something that might be needed … gah. Hyped vaporware. I tire of it.)

                                                                so yeah. I have no patience for hypotheticals in the blockchain space any more. Call me when the thing exists and works. There’s too much BS and too many BSers.

                                                                I did just ask an actual mathematician working in related areas to write a section-by-section takedown of that TimeCube-in-LaTeX white paper IOTA put up. I hope they have something I can link or post soon.

                                                                1. 2

                                                                  I do think, though, that eight years in, the burden of proof is firmly on the blockchain promoters. I have no patience any more for purported blockchain systems that don’t exist yet, because their track record of achieving existence is really bad.

                                                                  I really can’t argue. I do think that these things take time. Many of the potential “could” and “will” applications assume things like low transaction fees or a lightning network or decentralized storage. Until we figure out how to scale these things those applications are likely to be out of reach.

                                                                  I’ll have to begrudgingly agree with you if we go another 8 years without any big breakthroughs but for now I’m hopeful, ethresear.ch has some cool ideas which are just crazy enough to work. It took a surprisingly long time for the steam engine to become practical.

                                                                  And then, even if it all worked, the economics of the LN can’t possibly work and would seize up immediately

                                                                  Do you have any references I could go read? I have my doubts on Lightning Network, it feels like there need to be hubs, and the channels between those hubs need to be massive to hold all the payment volume, but that’s just my idle speculation. I’d love to see some math.

                                                                  I did just ask an actual mathematician working in related areas to write a section-by-section takedown of that TimeCube-in-LaTeX white paper IOTA put up. I hope they have something I can link or post soon.

                                                                  Yes, IOTA is an unmitigated disaster. I also would love to have something I can link to.

                                                                  1. 3

                                                                    No, I was trying to put one together, but that really requires the LN to be an actual thing that could be talked about. It’s hard to make specific and pointed critiques of a nonexistent network. I mean, proponents can easily say “that’s not a problem because (spurious justification)” because the network doesn’t exist yet

                                                                    Though here’s someone writing several obvious arguments against it. Also, Jorge Stolfi on Reddit (/u/jstolfi) has critiqued it at length and in detail.

                                                                    Whenever I’ve seen someone raise economic arguments, LN devs and proponents go silent. I’m also pretty sure you can get credit - the promise of money that would be as usable as money - out of it, though the notion is anathema to bitcoiners because credit is evil therefore doesn’t exist. Etc., and on.

                                                                    The LN is very un-blockchain in nature. One of the big things, actually, that worries me about the LN is that I don’t think there’s been a system like it in financial history. And people have tried just about everything, so that’s actually a point against it. I asked one finance history buff, and the only thing they could think of was Bretton-Woods and large quantities of gold. And B-W eventually faded away as people stopped pretending they cared about the actual gold.

                                                                    I’m seriously starting to think they were desperate for something, anything to deal with the Bitcoin transaction clog and seized upon this half-formed LN payment channels idea, didn’t think it through at all, and still haven’t.

                                                                    The LN on the testnet is pretty elaborate. Buuuut the testnet doesn’t have $50 transaction fees to set up a channel.

                                                                    1. 2

                                                                      Thanks for making your points in this thread. They’re well-stated and express some of the concerns I’ve had while evaluating various possibilities for the current project I’m working on, better than I could have.

                                                            2. 3

                                                              The first is pretty good esp on speed, flexibility, and energy use. If it’s on the list, one might want to include prior and alternative work that uses traditional tech with audit ledgers, byzantine databases, securities built on multiple stable commodities/currencies, nonprofits solving for-profit transaction issues via charter + contracts, and so on. Lots of possibilities that reuse everything from proven security or software methods to well-understood laws and business models.

                                                              Strange enough, echo chamber is so strong on bitcoin that anyone bringing up stuff like that just fixing centralized methods or decentralizing without blockchains gets pushed to bottom of thread. Meanwhile, the newly popular thing is still failing while the traditional methods still mostly work.

                                                              1. 3

                                                                Fairly sure David Gerard is a fellow crustacean. (He and I seem to end up in all the same internet places for some reason.)

                                                              2. 3

                                                                That’s more an argument about the business side than the core technology, though.

                                                            1. 15

                                                              Great idea! For formal methods, I thought this paper was a great start on criteria for deciding which, if any, formal methods to adopt in an industrial project. It shows strengths and weaknesses of a bunch of them with the best picks matching what experience reports Ive read pretty well.

                                                              1. 10

                                                                Wow, inlining the abstract:

                                                                Despite extensive evangelizing and demonstration of several success stories in safety-critical applications, formal methods are still not widely practiced in contemporary systems and software engineering. One of the main reasons for this situation is the absence of systematic guidelines and evaluation criteria that help software practitioners choose the right formal method for the problem at hand. In this article, we present a comprehensive set of criteria for evaluating and comparing different formal methods, based on a systematic literature review and decade-long personal experience with the application of formal methods in industrial projects. We argue that besides technical grounds (e.g., modeling capabilities and supported development phases), formal methods should also be evaluated from social and industrial perspectives. We also evaluate several state-based formal methods on the stipulated criteria.

                                                                This looks great!

                                                              1. 4

                                                                Oof… this was heavy. I got lost somewhere around the point where he introduces the decision table. :(

                                                                1. 3

                                                                  On a side note, decision tables were what John Nagle recommended for smart contracts. Easy for lay people to understand and for computers to analyze.

                                                                  1. 1

                                                                    Have you got a link to that anywhere, sounds interesting and my search engine Fu is failing me.

                                                                    1. 2

                                                                      I did some digging and this appears to be the comment in question

                                                                        1. 2

                                                                          Good find, I could only find other posts from @nickpsecurity repeating his statement :) Thanks!

                                                                        2. 1

                                                                          I dont have one to his comments but here’s Decision Tables:


                                                                          Also, Googling model-checking decision tables will give you at least one paper on using automated solvers with them.

                                                                    1. 13

                                                                      For grep -v there’s a much easier way to invert than the suggested

                                                                      awk '/something/ {next} 1'

                                                                      You can also use

                                                                      awk '!/something/'

                                                                      Am I wrong because that’s not in posix or something?

                                                                      1. 2

                                                                        $0 !~ /something/ is also shorter and very valid. But yours should be fine by POSIX, too:

                                                                        When an ERE token appears as an expression in any context other than as the right-hand of the ‘˜’ or “!˜” operator or as one of the built-in function arguments described below, the value of the resulting expression shall be the equivalent of: $0 ˜ /ere/

                                                                        Going by that, !/ere/ is !($0 ~ /ere/) as expected.

                                                                      1. [Comment removed by author]

                                                                        1. 3

                                                                          I hate to get into this because the whole point of the above comments by nickpsecurity and vyodaiken isn’t that political topics are intrinsically off-topic, it’s just that we can’t seem to get into them without devolving into sub-threads like this which are off-topic.

                                                                          However, I’m genuinely confused, how is the quote you mentioned a jab? Steve Bannon, who runs Breitbart, is in the white house and purportedly part of Trump’s inner circle. Watering hole attacks are well documented and given Bannon’s presence it sounds pretty likely Trump might at some point want to visit that site?

                                                                          1. [Comment removed by author]

                                                                            1. 5

                                                                              Breitbart is considered… to be bunch of fake news

                                                                              Breitbart does not rise to the level of fake news. Breitbart is Nazi propaganda. Claiming otherwise makes you look either incredibly bigoted or unbelievably uninformed; pick your poison.

                                                                              virtue signaling

                                                                              Nobody has ever used this phrase in good faith. If you intend to be arguing in good faith, make a comment that amounts to more than “people don’t like Nazis and that’s unfair”.

                                                                              at least he did not get owned yet

                                                                              How do you know?

                                                                              1. 6

                                                                                Breitbart does not rise to the level of fake news. Breitbart is Nazi propaganda.

                                                                                Proof? The Nazis were truly evil, claims like these trivialize the word.

                                                                        1. 8

                                                                          As a fellow “dead” blogger, I understand this decision, as much as I enjoyed his essays.

                                                                          There are a million things to hate about business programming culture, but one of them is that it’s very easy to get typecast to one thing that you’re good at (regardless of whether it’s what you’re best at) and then be limited by that archetype because, let’s face it, the business world runs on shallow impressions and on decisions made by less curious/capable/intelligent people than us. He advises against being typecast as a programmer, but being typecast as a programming blogger is even worse. It helps you gain visibility early in your career, but it becomes limiting later on, even if what you’ve put out there is of high quality.

                                                                          On the whole, I’m surprised that anyone does it (having done it). It’s career-limiting and extremely time-consuming, on the behalf of an industry that isn’t known for showing gratitude or returning favors. Given that most of us work on business problems of zero or negative social value, there’s also no higher purpose that can be put behind it.

                                                                          1. 3

                                                                            Can you elaborate on how it hurts your career? I’ve never before heard that.

                                                                            1. 1
                                                                              1. Everything on the Internet can be viewed by anyone. This includes prospective employers, and most employers want subordinate, consistent serfs and not independent thinkers.

                                                                              2. Antifascism makes enemies if you’re in the tech world. Some of those enemies have a lot of power and influence, as I’ve learned.

                                                                              3. Most people will judge you to be good at one thing only. It doesn’t matter if you’re actually good at several things. If that one thing is programming blogging, then they will assume that you’re a mediocre programmer and certainly not cut out to be a businessman. (I’m not saying that I want to be a businessman. I’m saying that most corporate employers don’t want to invest in people who aren’t aiming to be executives.)

                                                                              4. When you lose opportunities, you’re rarely aware that it’s happening at the time. You usually figure this stuff out 5 years later through various side channels.

                                                                          1. 1

                                                                            I would like to point out that the exponential chain reaction is not exclusive to nuclear fission. Chemical process can undergo chain reaction too.

                                                                            1. 1

                                                                              But saying “like Dynamite” is so 1870s.

                                                                            1. 2

                                                                              What don’t you like about goodreads?

                                                                              1. 3

                                                                                I’ve never used it seriously. What facilities does it offer to discover new books?

                                                                                1. 1

                                                                                  What facilities does it offer to discover new books?

                                                                                  It does not do anything for new books, especially when it comes to tech/programming books

                                                                              1. 1

                                                                                Can anyone explain how the very last example works?

                                                                                double NoOp(double a) {
                                                                                    return a;
                                                                                int main() {
                                                                                    double (*ReturnLastReturnValue)() = (double (*)())&NoOp;
                                                                                    double value = pow(0.99, 100.0);
                                                                                    double other_value = ReturnLastReturnValue();
                                                                                    printf("Value: %lf   Other value: %lf\n" value, other_value);
                                                                                1. 5

                                                                                  I didn’t read the full article, but I read enough to get the gist. I imagine that (in at least some calling conventions) the register via which the first double argument is passed into the function is also the same register to which the returned double would subsequently be written. By casting the function to a function type with no arguments, in all likelihood the compiler will generate program text that does nothing to the double-shaped register in question. Calling it immediately after a function that stored a return value in that same double register means that you’ll have basically done “nothing” and will re-read the (untouched) register to get the same return value.

                                                                                1. 1

                                                                                  Does anyone know why it took so long? Chrome was released 8 years ago, and by this schedule Firefox won’t reach parity with how that first version worked for another 2 years. This can’t have been a technical issue, what made them decide it wasn’t worth moving over?

                                                                                  1. 6

                                                                                    The Firefox code base is massive, and steeped with assumptions of a single process. Re-writing all that code is a huge undertaking, the largest in Firefox’s history. Not to mention the entire addon ecosystem that Mozilla doesn’t even have control over. So actually yes, it is largely a technical issue. That being said, e10s was de-prioritized in favour of Firefox OS for a year or two, which in hindsight was a mistake.

                                                                                  1. 9

                                                                                    The pg-hackers mailing list has a pretty honest self-assessment of this article: https://www.postgresql.org/message-id/579795DF.10502%40commandprompt.com

                                                                                    1. 2

                                                                                      The responses in the mailing list (if you’re willing to click through a lot of them) are pretty good.

                                                                                      Not only are they honest about what the shortcomings of PostgreSQL they even start digging into how to fix it. Open Source for the win.

                                                                                    1. 4

                                                                                      A few months ago I moved to Turkey in order to learn and become fluent in Turkish. When I’m not working I’m reading a book or watching a movie or struggling to express myself at a bar. It’s an incredibly cool language with a grammar rigid enough that it feels like programming. I’m slowly working on a blog post describing some of it’s coolest features.

                                                                                      1. 2

                                                                                        Looking forward to this blog post! I’m learning Turkish via Duolingo.

                                                                                      1. 1

                                                                                        I know close to no assembly, so this is mostly a guess, but from reading the python file it first passes the program to https://github.com/xoreaxeaxeax/movfuscator which transforms the program into only mov instructions (move a constant into a register, read from the address in this register, write to the address in this register plus the offset in this other register). That part is already mind-blowing, but seems to be possible because a turing machine can represent your program and a turing machine is just a FSM with a transition table, both of which involve nothing more than reading/writing data. More is explained here: http://www.cl.cam.ac.uk/~sd601/papers/mov.pdf (for real, read this. It’s amazing)

                                                                                        The python program then performs a few transformations which seem to revolve around simplifying the arguments to the mov calls and playing with the addressing to make the next step possible: it pushes all the arguments into a data structure and emits a small loop which “interprets” those arguments. The loop just reads the next set of arguments, calls “mov” with them, then repeats.