1. 1

    AppEngine is the tee for deployment.

    1. 2

      AppEngine and Glitch for hosted, sure, but I think that the equivalent for native software is a bit missing at the moment. The old answer would’ve been Visual Basic or HyperCard, but I’m not aware of anything filling those niches right now. PowerPoint with VBScript might honestly be closest.

      1. 1

        I would say that for a little GUI that just has to work somehow, Lazarus (Free Pascal Compiler based IDE following most of the Delphi design decisions) exists and works and is pretty simple to start using. It can also be used in more advanced ways, but that’s optional.

    1. 4

      If one wants to learn concepts, isn’t it better to learn DVCS concepts first?

      (And for that I would recommend reading arbitrary Git, Mercurial, Fossil and Monotone tutorials in parallel, so that the differences and similarities are more visible)

      1. 4

        It looks like Bochs also has x86_64 emulation; and if you cannot use JIT or direct execution on the host CPU, maybe it is an option to look into.

          1. 1

            Can’t an attacker just replace the hash with their malicious hash?

            1. 1

              There’s only one hash. Most curl attacks use the user-agent, timing attacks, etc., so if the returned script is malformed or malicious, the hash would not match whatever’s advertised on the website. This is only applicable when you read the script before piping it to sh. If you pipe scripts without reading, it’s a lost case and there’s no way to stop anybody.

            2. 1

              Is there any threat model where curl-hashpipe-sh is safer than straight curl-sh (with HTTPS and basic partial-content precautions)?

              1. 1

                It makes sense when your browser’s connection to the package website is trustworthy but the connection you’re curling from isn’t trustworthy.

                Which, like, when does that happen? I put my browsers on jank WiFi more often than my servers, and if I can’t trust my server’s upstream why do I trust the install image or RAM?

              2. 1

                I started writing something similar a while ago but never finished it: https://github.com/zimbatm/curlsh

                The tricky bit is that because curl and bash are available almost everywhere, they are being used for bootstrapping. So that tool would also have to be distributed widely.

              1. 4

                You gave examples of “bad” commands, which is good, but I didn’t see any examples of “good” commands. This would make it difficult for the offenders to fix if they don’t know how to fix it.

                1. 5

                  Safe, trustworthy software distribution is hard, and the mechanisms to do it right are diverse across platforms (unlike curl | sh …).

                  The Right Way is usually something along the lines of…

                  • Provide a self-contained package appropriate for the user’s platform
                  • Along with a hash that the user can use to check the download
                  • Or better yet, signed with a key which is verifiably owned by the developer
                  • And in the best case, distributed and vetted by the platform’s official package manager or App Store

                  All of which greatly improve’s user confidence in your software, but is a ton of work and probably a lot slower than just curling that shell script.

                  (fwiw, I’m the kind of person who will build and sign an rpm to distribute a one-file script, but I get why people find the whole process tedious…)

                  1. 1

                    But then, hashes hosted on the same server as the file to download do not protect against any malicious action, RPM or DEB is less likely to be inspected before installation (that already runs arbitrary code) than a plain curl-to-sh script that can be downloaded, and fraction of users that will meaningfully verify signing key identity via independent means is low enough that Yarn providing signatures at all more or less saturates the real benefits.

                    Oh, and by the way Yarn provides Debian packages, which is actually worse than their curl-to-sh option because adding their repository makes you vulnerable to future infrastructure intrusions, not just currently ongoing ones.

                    Judging from «it has green padlock, it is safe!» around HTTPS, things like co-hosted hash can make the overall situation slightly worse.

                  2. 1

                    I don’t think there is a good command of the form curl ... | bash. It’s always a way to get a short term boost in downloads while polluting the commons for everyone.

                    https://en.wikipedia.org/wiki/Tragedy_of_the_commons

                    1. 0

                      You engage with trustworthy distributions to have your software packaged.

                    1. 7

                      OK, I’ll get the cite training rolling with something easy to trump. Here’s Haskell two years ago. But I bet someone here can find an article about doing this in COBOL in 1964 or something.

                      For the details of the headline’s claim, looks like that starts around 28:10 in the video.

                      1. 3

                        Kitsune did it for C in OOPSLA 2012 and cites other “recent” systems, at least one from 2005. No doubt that one cites older works in turn.

                        1. 3

                          Initially, I was also kinda surprised by the “first” claim, given that I remember something like that in MSVC++ in some olden times, and given that I find his achievement super interesting and valuable even without the “first” claim. (In particular, I believe “hot code reloading” is sorely missing among “recent” non-interpreted/VM languages, and even in C++ it didn’t seem to spread outside MSVC++.)

                          But after viewing the presentation, and seeing that he does in fact reference the /hotpatch option of MSVC++, I’m starting to think he may be trying to highlight the ease of use of the feature, with close to no “special support” needed in one’s app, as well as its extensiveness/depth (as in “fully supporting all features of the language”). Notably, glancing through the Haskell link you provided, I seem to understand it requires some special code at pre-determined “decoupling points”, as well as that it has some important limitations (e.g.: “The main binary cannot persist any reference to code or data originating from the shared object, because that will prevent the GC from unloading the object.”). He doesn’t mention specifics, but I believe he claims that the /hotpatch was also limited in some important ways. In contrast, I believe he suggests his solution implemented for Nim is much easier to use and not limited in any important ways.

                          Personally, I think it might be worth considering if his achievement is not unlike how Dropbox was “not much more” than FTP ;)

                          1. 3

                            There are some approaches in Wikipedia’s Dynamic software updating and Hot swapping. Their references might have some older approaches.

                            Common Lisp and Smalltalk were obvious counterpoints if they allowed for bytecode. I couldn’t remember if CL was native or bytecode for hot swapping. The CL programmers on HN are saying it could do native hot swapping.

                            1. 2

                              I think CMUCL has been able to produce native code for decades (and Common Lisp specification requires hot swapping). Right now there are definitely multiple Common Lisp implementations that produce native code.

                          1. 2

                            Are these specifically made for this competition or are they found in the wild?

                            1. 3

                              Well, the expectation is that the submissions are written for the competition; and the code has to solve the posted task (which is not hard, but reduces the likely number of preexisting eligible code).

                              1. 3

                                The expectation is that they are made for the competition. We do hope that the teardowns can provide educational value, however, since we try to pick as winners code that is “bad” in a way we could imagine seeing for real (not just obfuscated, golf, etc)

                                1. 1

                                  Oh, cool. If someone did commented braces for real, I would be so terrified lol

                              1. 7

                                Could somebody who has watched the video explain how «natively» is defined? I would consider some implementation of Common Lisp natively compiled, and they have code reloading in runtime.

                                1. 8

                                  Seems to be used in the sense of “there are some hacks to accomplish this with c++ but they aren’t well integrated”.

                                1. 8

                                  The attackers did not get in through a security flaw in Matrix itself, but via an outdated Jenkins.

                                  The nice thing about Matrix is that it is federated (much like e-mail is); there’s no reason to use the matrix.org server. Instead, you can use your own trusted home server. Another reason to use your own is that the Matrix main server tends to be pretty overloaded so everything is quite slow.

                                  1. 3

                                    I mean, it doesn’t saying anything about the quality of the Matrix codebase as such, but some things do make you wonder about the level of understanding that the people working on it bring to it:

                                    Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

                                    … and the corresponding Issues stop just short of overtly saying that that account had root access on all servers. The picture painted by that combination of a handful of facts makes me really wary…

                                    1. 0

                                      It looks like the core parts of the protocol (including E2E encryption) could now be under takeover-in-progress by French national security agencies.

                                      New Vector doesn’t look likely to say no to one more reasonably-looking paid change request from France, and also not likely to find security implications if there is a nation-state effort to hide them. Some of the incentives are good, though (it was promised that French goverment agencies will have an option of limited external federation with mainline installations; the current public story even claims that the agencies will run the mainline server code).

                                      For some bugfixes in Synapse, it is worrying the the pre-fix state was widely deployed for months…

                                      1. 3

                                        So, speaking as the project lead for Matrix, this really isn’t true. ANSSI (the french cybersecurity agency) have not made any code contributions to the E2E crypto or the protocol itself. If they did make contributions, we’d audit them ourselves incredibly carefully, if we even accepted them at all. So far we’ve minimised the number of contributors to libolm (the E2E library) to a very small set of individuals.

                                        To be clear: we would be utterly stupid to sabotage Matrix by somehow giving any entity (whether that’s France, or New Vector, or Ericsson or whoever) a competitive advantage, no matter how much they offered to pay us. We aren’t building Matrix out of short-termist greed, but long-term altruism - to build an open global standard for comms. And we are not naive, and are very aware that some parties might try to sneak in trojan horses, and will do everything to fight them. We’ve spent a lot of time trying to codify this into the governance of the Matrix.org Foundation over at https://github.com/matrix-org/matrix-doc/blob/matthew/msc1779/proposals/1779-open-governance.md.

                                        Now, in terms of some of Synapse’s long-lived pre-1.0 bugs being concerning (e.g. state resets; fungible event IDs)… this is true. But we were fixing these independently of the French deployment as part of our existing 1.0 roadmap. The only difference is that we knew we had to have them fixed in time for France to go live. The actual design and solution and code was written entirely by us, and France does run off the same public synapse tree as everyone else and so got them at the same time with no privileged access.

                                        So, TL;DR: there is categorically not a takeover-in-progress, and it’s very unfortunate if it seems that way from the outside. Instead, DINSIC have been amazingly good open source citizens, and I only wish all government ministries operated like this.

                                        1. 1

                                          I am indeed surprised that none of the bugs that looked like hotfixes (and persisted for months) were from ANSSI audit. Interesting, thanks for commenting about that.

                                          I only consider the threat model of them managing to find and explain a real problem in a way that leads to a natural fix that has unintended complicated implications — but hopefully they don’t want to create a risk for something they will also be expected to run.

                                          I actually hoped they already started pushing the bug reports — a fresh set of qualified reviewers who have an actual motivation to find out if the protocol works has a good chance of being useful.

                                          Sorry for not wording some parts of my comment well, and thanks for the clarifications that they haven’t yet gave you the results of their (hopefully already ongoing) audit.

                                          1. 3

                                            So to be clear - there have been at least 3 audits, but primarily around the infrastructure of the deployment rather than the E2EE in matrix, mainly because Matrix is obviously still developing rapidly. Once there’s a full audit on the Matrix side I’d push for it to be public, just like our US-government-funded E2EE audit from 2016.

                                      1. 2

                                        Ugh. I’ve thought about opening up the Jenkins server on one of my open source projects to the world, but hesitate due to crap like this. Even though I update it regularly, the risk seems high just to have public build statuses (which I could probably just proxy through something else).

                                        I also hate how you have you mount the underlying machine’s docker socket to allow docker build agents. Surely there’s got to be a better user-mode docker solution.

                                        1. 2

                                          Yeah, that’s unfortunate. Maybe you could script it to generate some static artifacts and then drop them somewhere web accessible?

                                    1. 2

                                      I guess the oldest code I have written lives in my .vimrc and provides some bindings. Argh, now that I am reminded I am tempted to recompile a litle puzzle game I wrote approximately at the same time (clone of a preexisting one, but with undo for misclicks) and play it a bit… That should be 2005 at the latest.

                                      A tool for customisable data acquisition from an ADC system that still works is probably ~2007 version, ~2003 version got obsoleted by ISA interfaces being hard to come by — but I think some code is shared between the two.

                                      A web system (coordinating education events, and processing the records, mostly) with a database name ending in 2010 (a rewrite sharing some code with 2009 version) is definitely still running. And handles hundreds of things per year, with a lot of participants (and reports). I know, I sometimes chat with the current maintainer and tell her what was the logic of the code nobody normally has to touch nowadays. But some functionality seems to have become «dark functionality» — it requires some knowledge to reach (only useful for a few qualified people anyway, and one can request a report of load-creating size), so either it entered the workflow (maybe as a yearly task), or people forgot it is even there. Sometimes I remind people it exists (in the context of something they want to do).

                                      1. 6

                                        There is a comparison to Plan9 and Hurd, but not to GenodeOS (which superficially seems more similar).

                                        I am not sure if Fuchsia brings anything actually good compared to Genode (and it would be interesting to see); of course, the adoption question is drivers, and Fuchsia will get adoption if Google forces manufacturers to support it, now without any problems around closed-source drivers.

                                        1. 8

                                          I heard more BeOS influence, due to the Be alumni working on it.

                                          1. 6

                                            And we don’t get a BeOS comparison, either.

                                            A bit like explaining Git while being careful to never compare it to preexisting DVCSes, only to Monotone — on the other hand, that approach usually works…

                                            (And I guess running Android apps on Fuchsia also means that security changes will matter more in terms of protecting manufacturer from consumers actually owning the device, than in providing consumers with securities, as most of the user-facing security problems are apps being happy to leak their entire state — although indeed it might also make it simpler for a custm build to provide fake sensitive data)

                                            1. 1

                                              Well, with no global file system, it should be easier for apps to have their own private storage to protect the customer from greedy data scouting.

                                              1. 2

                                                Hard to say how many vulnerabilities are exploited as single-app takeovers (these will probably stay in Fuchsia).

                                                On the other hand, cross-application interoperation becomes harder and users lose the ability to have file managers…

                                                (And some kind of cross-app message-passing will probably formally exist — inherited from intents — but will continue to require too much opt-in from apps to be widely useful — like OpenIntents that apparently didn’t even take off among apps explicitly wishing to ship in F-Droid)

                                              2. 1

                                                The speaker isn’t actually working on the OS so perhaps wasn’t aware of those comparisons to be made.

                                              3. 2

                                                If Fushia internals are remotely comparable to Be’s KernelKit then it’s a great architecture. I wrote an embedded kernel in grad school using Be’s documented Kernel API, and it’s extremely well designed. The VFS is a piece of art*; I still have fond memories implementing vfs_walk(). It’s a shame BeOS’ architecture is not better studied.

                                                *not technically part of the kernel and well detailed in Practical Filesystem Design de D. Giampaolo.

                                                1. 4

                                                  It’s not really like BeOS, no; BeOS was a monokernel, and Fuchsia is a microkernel. There are some vague similarities, but not anything too major.

                                                  On the other hand, Haiku is a complete re-implementation of BeOS including the kernel, and IMO our APIs are even more beautiful than Be’s … but I am biased, of course. :)

                                                2. 1

                                                  And we don’t get a BeOS comparison, either.

                                                  A bit like explaining Git while being careful to never compare it to preexisting DVCSes, only to Monotone — on the other hand, that approach usually works…

                                                  (And I guess running Android apps on Fuchsia also means that security changes will matter more in terms of protecting manufacturer from consumers actually owning the device, than in providing consumers with securities, as most of the user-facing security problems are apps being happy to leak their entire state — although indeed it might also make it simpler for a custm build to provide fake sensitive data)

                                              1. 4

                                                It is time we did away with “publish or perish” and replace it with “publish and perish.” Nothing will be more blasphemous than writing a textbook that anyone can go out and buy. Those who commit such a crime will face expulsion and reprisals.

                                                Hppasus, a Pythagorean, was executed by his fellows for publicly revealing the existence of irrational numbers.

                                                (Okay, so it’s an apocryphal story, but since we don’t really know what happened, I choose to believe the more entertaining version.)

                                                1. 2

                                                  If it happenned, the fact that irrational numbers are problematic for Pythagorean religious teachings might have played a role — it’s not just a random discovery he let out, it’s a very inconvenient one.

                                                1. 1

                                                  Phone: K9

                                                  Laptop: fdm → (custom scripts) → PostgreSQL → (custom FUSE tool — QueryFS) → Vim → mimedecode.py & ripmime & lynx –dump as needed

                                                  My preferred protocol is POP3 (because I do not want syncing the «read» status between phone and desktop), I do use webmail from time to time, mainly to mark SPAM/not-SPAM for on-server spam filters.

                                                  (considering migration from lynx –dump to a custom HTML dumper that I now have)

                                                  1. 6

                                                    https://i.imgur.com/9BK7QwL.png

                                                    StumpWM. On the right: Vim with a grabbed textified copy of this page. I do read more of the web like that than in the browser. I indeed skimmed the list of the descriptions there and only started the browser (on the left) to upload an image to Imgur and write a comment here. I use Vim (and do not use Emacs).

                                                    On the left — my Firefox setup. Minimal UI, tab bar auto-hidden even with multiple tabs (but there are just Lobsters and Imgur because I start a separate ns-isolated Firefox instance for every browsing tabs). Orange stripes in the address bar show that Marionette is enabled in this instance (I always enable it for script control of the browser — enabling Javascript as needed, dumping the pages that need some manual Javascript navigation, etc).

                                                    On the bottom — my system information xterm. WiFi state, email count (fresh/deferred/long-term-deferred), web stream status (fresh/backlog), IP adresses, free RAM (free/free+caches), battery state, date, UTC and non-UTC time. (there are sometimes more timezones, and they are sometimes marked with 3-letter place codes). Email and web streams are fetched in the background and listed in a database. Then they are read in Vim via QueryFS that represents an SQL query to list the fresh entries as a directory of symlinks (deleting a symlink marks the entry as read).

                                                    1. 4

                                                      Hi5, StumpWM buddy!

                                                      Here’s my desktop at work: https://i.imgur.com/cFUfqlG.png although it isn’t really showing work stuff but most of my virtual desktops have the same layout. I’ve got an ultrawide screen which I love but I could really use another monitor for keeping chat always visible.

                                                      Void Linux, StumpWM, Emacs + Slime + Evil (Vim keybindings), rxvt with unicode, Firefox (I wish it wasn’t bright white but it’s just too much of a hassle). Working on a Common Lisp version of my ex-boss’ @ahu Hello DNS.

                                                      As can be seen in the upper-right two sync services are running: pCloud and Dropbox. I transferred to pCloud a good while ago because of its price but had to revert to Dropbox for really important files like the password manager because I got too many sync conflicts. Still pCloud is really nice as a remote NAS.

                                                      1. 1

                                                        How would an extra monitor help? If you have an ultrawide monitor, you probably do have some space you could allocate for the IM (or not?). In StumpWM group switching is global anyway. That’s why I just use tagging instead…

                                                        1. 2

                                                          Oh darn, that’s true. I forgot about the group switching being global since I was still using i3 when on multiple monitors. Well, I’ll find a way.

                                                          Anyway, I’m pretty set in my ways on where I want my chat and it’s on a separate monitor :-)

                                                    1. 9

                                                      pushcx desktop

                                                      Basically the same as 2015: I no longer have the bookmarks bar turned on in Firefox; I added caffiene-indicator to the systray for disabling the screensaver; I toned down my vim colorscheme to one based on jcs’s but with the LightItalic from Operator Mono for constants and comments. Still thinking of moving to xmonad, especially since July when awesomewm made some breaking changes to config files.

                                                      1. 3

                                                        DWM is the bestest… Awesome is a fork of it, but breaks things :D

                                                        1. 2

                                                          I recommend Xmonad, ive been using it for a lot of years and it never lets me down. make sure to add xmobar as well for a nice info bar.

                                                          Ill post my setup later today.

                                                          1. 2

                                                            What’s the extension on the left of firefox?

                                                            1. 1

                                                              Tree-Style Tab. I find the grouping of tabs by parent and being able to open and close folders very useful.

                                                              1. 1

                                                                How do you shrink down the main horizontal firefox tab bar?

                                                                1. 1

                                                                  Either Tree-Style Tab removes it or I dragged it around in the usual Firefox appearance customizer.

                                                                  1. 1

                                                                    Hmm. Tree-Style Tabs doesn’t seem to change the main tab bar for me, and I can’t figure out a way to change it in the customizer.

                                                            2. 2

                                                              We had a nice chat about fonts in #lobsters in response to this thread and there were a bunch of mono italics I hadn’t heard of before: @zpojqwfejwfhiunz shared Dank Mono, @flyingfisch shared Plex, and @zdsmith shared PragmataPro. None cursive like Operator, but very nice fonts (except for the f in Dank mono).

                                                              1. 1

                                                                I can’t find any way to view the full size image of any of these imgur posts… and the page is full of random memes.

                                                                1. 1

                                                                  If you load them in the browser, open the link twice (first time works as a redirect to the image page, but once you load the image page, further requests for the image itself give you indeed the image, regardless of the Referer: field). I use wget and Geeqie, so for me they load fine the first time.

                                                              1. 1

                                                                Many of the first run behaviours of Firefox can be disabled by pre-seeding preferences, in fact. Then, I wouldn’t trust nothing to leak through $HOME — previously, Flash plugin saved data did leak through a completely different path than Firefox profile.

                                                                I prefer to pre-seed configuratoin, run Firefox once to prepare a profile (second start is faster than the first), then run new instances with no access to the true /home/ using fresh copies of the prepared profile.

                                                                1. 2

                                                                  Our team of five neural networks, OpenAI Five, has started to defeat amateur human teams at Dota 2.

                                                                  OpenAI Five plays 180 years worth of games against itself every day, learning via self-play. It trains using a scaled-up version of Proximal Policy Optimization running on 256 GPUs and 128,000 CPU cores

                                                                  If that doesn’t ease your fears about an impending AI apocalypse, I don’t know what will.

                                                                  1. 5

                                                                    That actually makes my AI fears worse. But that’s because they are not the exact stereotypical AI fears.

                                                                    What the article says is: if you can afford ten times more computing resources, you get better chances to achieve superhuman resources than by using novel approaches. Train once, run cheaply. So, capital matters, labor qualification doesn’t, economoies of scale with huge front-up costs and small recurring costs. That’s how you get badly broken monopoly markets, no? And of course then it breaks because someone spends on servers and not on human time to make sure nothing stupid happens.

                                                                    Yes, OpenAI on its own will probably try to release something already trained and with predictable failure risks; and this is likely to improve the situation overall — I like what they want to do and what they do, I am just afraid of what they find (and thanks to them for disclosing it).

                                                                    1. 4

                                                                      That’s a very good, and frightening point. My comment was mainly in regards to the side of it that smells like mind boggling computational waste in addition to the staggering amount of experience such models require to achieve sub-human (less than expert) performance. Personally, I think ANNs are very cool, and quite elegant. But I feel like they are treated as the ultimate hammer, and every domain of learning is a nail. Examples such as this post show that they scale less than optimally. And from an algorithmic perspective there has to be a more concise, more computationally efficient way to go about learning to perform a task.

                                                                      1. 2
                                                                        1. Yes, it takes a lot to get anything out of it, then if you just continue doing the same you get better results than you expected, and the road from median player to expert player is not always that large of a percentage increase in expenses on machine training.

                                                                        2. Comparing effort-to-mastery between humans and machines is hard, because the expert play is a product of many millions of hours of human play (and discussion on forums and in chats).

                                                                        2a. Yes, it would be intersting if centaur chess or centaur go would take off… A 1kW desktop and a human against a 10kW server. Especially interesting given that game 4 by AlphaGo was lost to a human strategy normally too tiring to keep track of — so humans never play it against each other — but it is unclear whether a weak computer could help a human in using such strategies.

                                                                        1. One of the useful properties of cryptocurrencies is to show that humans care little about computational waste per se… I mean, some people understood it after looking at smartphones and comparing them to pre-iPhone smartphones and PDAs, but some people needed a clearer explanation.

                                                                        2. Analysing algorithmic perspective requires qualified humans, and good tools for these humans. And some of our tools seem to have degraded (as tools of information processing) with time… (I mean, it’s almost painful to read Engelbart’s book on Intelligence Augmentation and see that some modern tools are worse than the prototypes his team had).

                                                                      2. 1

                                                                        and thanks to them for disclosing it

                                                                        I’m afraid they will just disclose peanuts, to engage people and spread trust in AI.

                                                                        It’s basically marketing.

                                                                        1. 1

                                                                          Well, they already tell what they tried and what didn’t work, which is unfortunately already above the median…

                                                                          I meant that I am already afraid of then things they find and disclose now.

                                                                      3. 2

                                                                        Evidence they can learn in an environment where they start with nothing, the rules change a lot, they have to pick up new concepts quickly based on old ones, malicious people try to destroy them, or possessing common sense. What the machines are used for or doing vs what we do on a regular basis are really, really different.

                                                                        1. 1

                                                                          It might not even matter if OpenAI can eventually make machine learning work in adversarial environment, though (I think this post might even be weak evidence that they will be able to do it at some point, given enough computational resources spent), as someone will still cut the corners and give a vulnerable agent direct control over something with significant negative externalities of failure.

                                                                      1. 7

                                                                        I have done some audio programming, and am studying engineering, so I guess I have some knowledge about it. There are many who are better than me, though. I hope this isn’t too mathematical, but you need to have some grasp on differentiation, integration, complex numbers and linear algebra anyway. Here’s a ‘short’ overview of the basics:

                                                                        First of all, you need to know what happens when an analog, continuous signal is converted to digital data and back. The A->D direction is called sampling. The amount of times the data can be read out per second (the sampling rate) and the accuracy (bit depth) are limited for obvious reasons, and this needs to be taken into account.

                                                                        Secondly, analysing a signal in the time domain doesn’t yield much interesting information, it’s much more useful to look analyse the frequencies in the signal instead.

                                                                        Fourier’s theorem states that every signal can be represented as a sum of (co)sines. Getting the amplitude of a given freqency is done through the Fourier transform (F(omega) = integrate(lambda t: f(t) * e^-omega*j*t, 0, infinity)). It works a bit like the following:

                                                                        1. Draw the function on a long ribbon
                                                                        2. Twist the ribbon along its longest axis, with an angle proportional to the desired frequency you want the amplitude of (multiplying f(t) by e^-omega*j*t, omega is the pulsation of the desired frequency, i.e. omega = 2pi*f, and j is the imaginary unit. j is used more often than i in engineering.)
                                                                        3. Now smash it flat. In the resulting (complex) plane, take the average of all the points (i.e. complex numbers). (This is the integration step.)
                                                                        4. The sines will cancel out themselves, except for the one with the desired freqency. The resulting complex number’s magnitude is the amplitude of the sine, and its angle is the sine’s phase.

                                                                        (Note: the Fourier transform is also known as the Laplace transform, when substituting omega*j with s (or p, or z, they’re “implicitely” complex variables), and as the Z-transform, when dealing with discrete signals. It’s still basically the same, though, and I’ll be using the terms pretty much interchangably. The Laplace transform is also used when analyzing linear differential equations, which is, under the hood, what we’re doing here anyway. If you really want to understand most/everything, you need to grok the Laplace transform first, and how it’s used to deal with differential equations.)

                                                                        Now, doing a Fourier transform (and an inverse afterwards) can be costly, so it’s better to use the information gained from a Fourier transform while writing code that modifies a signal (i.e. amplifies some frequencies while attenuating others, or adding a delay, etc.), and works only (or most of the time) in the time domain. Components like these are often called filters.

                                                                        Filters are linear systems (they can be nonlinear as well, but that complicates things). They are best thought of components that scale, add, or delay signals, combined like this. (A z^-1-box is a delay of one sample, the Z-transform of f(t-1) is equal to the Z-transform of f(t), divided by z.)

                                                                        If the system is linear, such a diagram can be ‘transformed’ into a bunch of matrix multiplications (A, B, C and D are matrices):

                                                                        • state [t+1] = A*state[t] + B*input[t]
                                                                        • output[t ] = C*state[t] + D*input[t]

                                                                        with state[t] a vector containing the state of the delays at t.

                                                                        Analyzing them happens as follows:

                                                                        1. Take the Z-transform of the input signal (Z{x(t)}=X(z)) and the output signal (Z{y(t)}=Y(z)).
                                                                        2. The proportion between Y and X is a (rational) function in z, the transfer function H(z).
                                                                        3. Now find the zeros of the numerator and denominator. The zeros of the latter are called the poles, signals of (or near to) that frequency are amplified. Zeros of the numerator are (boringly) called zeros, and they attenuate signals. These poles and zeros are also related to the eigenvectors and -values of the matrix A.

                                                                        However, if the poles are outside of the unit circle, the system is ‘unstable’: the output will grow exponentially (i.e. “explode”). If the pole is complex or negative, the output will oscillate a little (this corresponds to complex eigenvalues, and complex solutions to the characteristic equation of the linear differential equation).

                                                                        What most often is done, though, is making filters using some given poles and zeros. Then you just need to perform the steps in reverse direction.

                                                                        Finally, codecs simply use that knowledge to throw away uninteresting stuff. (Eg. data is stored in the frequency domain, and very soft sines, or sines outside the audible range are discarded. With images and video, it’s the same thing but in two dimensions.) I don’t know anything specific about them, though, so you should look up some stuff about them yourself.


                                                                        Hopefully, this wasn’t too overwhelming :). I suggest reading Yehar’s DSP tutorial for the braindead to get some more information (but it doesn’t become too technical), and you can use the Audio EQ Cookbook if you want to implement some filters. [This is a personal mirror, as the original seems to be down - 509.]

                                                                        There’s also a copy of Think DSP lying on my HDD, but I never read it, so I don’t know if it’s any good.

                                                                        1. 3

                                                                          The amount of times the data can be read out per second (the sampling rate) and the accuracy (bit depth) are limited for obvious reasons

                                                                          Interesting post. I wanted to highlight this part where you say it’s limited for “obvious reasons.” It’s probably better to explain that since it might not be obvious to folks trained to think transistors are free, the CPU’s are doing billions of ops a second, and everything is working instantly down to nanosecond scale. “How could such machines not see and process about everything?” I thought. What I learned studying hardware design at a high-level, esp on the tools and processes, was that the digital cells appeared to be asleep a good chunk of the time. From a software guy’s view, it’s like the clock signal comes as a wave, starts lighting them up to do their thing, leaves, and then they’re doing nothing. Whereas, the analog circuits worked non-stop. If it’s a sensor, it’s like the digital circuits kept closing their eyes periodically where they’d miss stuff. The analog circuits never blinked.

                                                                          After that, the ADC and DAC tutorials would explain how the system would go from continouous to discrete using the choppers or whatever. My interpretation was the digital cells were grabbing a snapshot of the electrical state as bit-based input kind of like requesting a picture of what a fast-moving database contains. It might even change a bit between cycles. I’m still not sure about that part since I didn’t learn it hands on where I could experiment. So, they’d have to design it to work with whatever its sampling rate/size was. Also, the mixed-signal people told me they’d do some components in analog specifically to take advantage of full-speed, non-blinking, and/or low-energy operation. Especially non-blinking, though, for detecting things like electrical problems that can negatively impact the digital chips. Analog could respond faster, too. Some entire designs like control systems or at least checking systems in safety-critical stuck with analog since the components directly implemented mathematical functions well-understood in terms of signal processing. More stuff could go wrong in a complex, digital chip they’d say. Maybe they just understood the older stuff better, too.

                                                                          So, that’s some of what I learned dipping my toes into this stuff. I don’t do hardware development or anything. I did find all of that really enlightening when looking at the ways hardware might fail or be subverted. That the digital stuff was an illusion built on lego-like, analog circuits was pretty mind-blowing. The analog wasn’t dead: it just got tamed into a regular, synthesizable, and manageable form that was then deployed all over the place. Many of the SoC’s still had to have analog components for signal processing and/or power competitiveness, though.

                                                                          1. 3

                                                                            You’re right, of course. On the other hand, I intended to make it a bit short (even though it didn’t work out as intended). I don’t know much about how CPUs work, though, I’m only in my first year.

                                                                            I remember an exercise during maths class in what’s probably the equivalent of middle or early high school, where multiple people were measuring the sea level at certain intervals. To one, the level remained flat, while for the other, it was wildly fluctuating, while to a third person, it was only slightly so, and at a different frequency.

                                                                            Because of the reasons you described, the ADC can’t keep up when the signal’s frequency is above half the sampling frequency (i.e. the Nyqvist frequency).

                                                                            (Interestingly, this causes the Fourier transform of the signal to be ‘reflected’ at the Nyqvist frequency. There’s a graph that makes this clear, but I can’t find it. Here’s a replacement I quickly hacked together using Inkscape. [Welp, the text is jumping around a little. I’m too tired to fix it.])

                                                                            The “changing a bit between cycles” might happen because the conversion doesn’t happen instantaneously, so the value can change during the conversion as well. Or, when converting multiple values that should happen “instantaneously” (such as taking a picture), the last part will be converted a little bit later than the first part, which sounds analogous to screen tearing to me. Then again, I might be wrong.


                                                                            P.S. I’ll take “interesting” as a compliment, I just finished my last exam when I wrote that, so I’m a little tired now. Some errors are very probably lurking in my replies.

                                                                            1. 3

                                                                              I’ll take “interesting” as a complimen

                                                                              You were trying to explain some hard concepts. I enjoy reading these summaries since I’m an outsider to these fields. I learn lots of stuff by reading and comparing explanations from both students and veterans. Yeah, it was a compliment for the effort. :)

                                                                          2. 3

                                                                            Even though I learned about the Fourier transformation in University this video gave me a new intuition: https://www.youtube.com/watch?v=spUNpyF58BY

                                                                            1. 2

                                                                              Thanks very much for your detailed reply :). The math doesn’t scare me, it’s just very rusty for me since a lot of what I do doesn’t have as much pure math in it.

                                                                              I appreciate the time you put into it.

                                                                              1. 2

                                                                                Speaking specifically of Fourier transform: it behaves well for infinite signals and for whole numbers of periods of strictly periodic signals.

                                                                                But in reality the period usually doesn’t divide the finite fragment we have (and also there are different components with different periods). If we ignore this, we effectively multiply the signal by a rectangle function (0… 1 in the interval… 0…) — and Fourier transform converts pointwise multiplication into convolution (an operation similar to blur). Having hard edges is bad, so the rectangle has a rather bad spectrum with large amplitudes pretty far from zero, and it is better to avoid convolution with that — this would mix rather strongly even frequencies very far from each other.

                                                                                This is the reason why window functions are used: the signal is multipled by something that goes smoothly to zero at the edges. A good window has a Fourier transform that falls very quickly as you go away from zero, but this usually requires the spectrum to have high intensity on a wide band near zero. This tradeoff means that if you want less leak between vastly different frequencies, you need to mix similar frequencies more. It is also one of the illustrations of the reason why a long recording is needed to separate close frequencies.

                                                                              1. 1

                                                                                Sculpt is a general-purpose OS on top of the Genode microkernel.

                                                                                They seem to be porting some specific driver subsystems from Linux and BSD (with some driver isolation measures).

                                                                                1. 31

                                                                                  at this point most browsers are OS’s that run (and build) on other OS’s:

                                                                                  • language runtime - multiple checks
                                                                                  • graphic subsystem - check
                                                                                  • networking - check
                                                                                  • interaction with peripherals (sound, location, etc) - check
                                                                                  • permissions - for users, pages, sites, and more.

                                                                                  And more importantly, is there any (important to the writers) advantage to them becoming smaller? Security maybe?

                                                                                  1. 11

                                                                                    Browsers rarely link out the system. FF/Chromium have their own PNG decodes, JPEG decodes, AV codecs, memory allocators or allocation abstraction layers, etc. etc.

                                                                                    It bothers me everything is now shipping as an electron app. Do we really need every single app to have the footprint of a modern browser? Can we at least limit them to the footprint of Firefox2?

                                                                                    1. 10

                                                                                      but if you limit it to the footprint of firefox2 then computers might be fast enough. (a problem)

                                                                                      1. 2

                                                                                        New computers are no longer faster than old computers at the same cost, though – moore’s law ended in 2005 and consumer stuff has caught up with the lag. So, the only speed-up from replacement is from clearing out bloat, not from actual hardware improvements in processing speed.

                                                                                        (Maybe secondary storage speed will have a big bump, if you’re moving from hard disk to SSD, but that only happens once.)

                                                                                        1. 3

                                                                                          moore’s law ended in 2005 and consumer stuff has caught up with the lag. So, the only speed-up from replacement is from clearing out bloat, not from actual hardware improvements in processing speed.

                                                                                          Are you claiming there have been no speedups due to better pipelining, out-of-order/speculative execution, larger caches, multicore, hyperthreading, and ASIC acceleration of common primitives? And the benchmarks magazines post showing newer stuff outperforming older stuff were all fabricated? I’d find those claims unbelievable.

                                                                                          Also, every newer system I had was faster past 2005. I recently had to use an older backup. Much slower. Finally, performance isn’t the only thing to consider: the newer, process nodes use less energy and have smaller chips.

                                                                                          1. 2

                                                                                            I’m slightly overstating the claim. Performance increases have dropped to incremental from exponential, and are associated with piecemeal attempts to chase performance increase goals that once were a straightforward result of increased circuit density through optimization tricks that can only really be done once.

                                                                                            Once we’ve picked all the low-hanging fruit (simple optimization tricks with major & general impact) we’ll need to start seriously milking performance out of multicore and other features that actually require the involvement of application developers. (Multicore doesn’t affect performance at all for single-threaded applications or fully-synchronous applications that happen to have multiple threads – in other words, everything an unschooled developer is prepared to write, unless they happen to be mostly into unix shell scripting or something.)

                                                                                            Moore’s law isn’t all that matters, no. But, it matters a lot with regard to whether or not we can reasonably expect to defend practices like electron apps on the grounds that we can maintain current responsiveness while making everything take more cycles. The era where the same slow code can be guaranteed to run faster on next year’s machine without any effort on the part of developers is over.

                                                                                            As a specific example: I doubt that even in ten years, a low-end desktop PC will be able to run today’s version of slack with reasonable performance. There is no discernible difference in its performance between my two primary machines (both low-end desktop PCs, one from 2011 and one from 2017). There isn’t a perpetually rising tide that makes all code more performant anymore, and the kind of bookkeeping that most web apps spend their cycles in doesn’t have specialized hardware accelerators the way matrix arithmetic does.

                                                                                            1. 5

                                                                                              Performance increases have dropped to incremental from exponential, and are associated with piecemeal attempts to chase performance increase goals that once were a straightforward result of increased circuit density through optimization tricks that can only really be done once.

                                                                                              I agree with that totally.

                                                                                              “Multicore doesn’t affect performance at all for single-threaded applications “

                                                                                              Although largely true, people often forget a way multicore can boost single-threaded performance: simply letting the single-threaded app have more time on CPU core since other stuff is running on another. Some OS’s, esp RTOS’s, let you control which cores apps run on specifically to utilize that. I’m not sure if desktop OS’s have good support for this right now, though. I haven’t tried it in a while.

                                                                                              “There isn’t a perpetually rising tide that makes all code more performant anymore, and the kind of bookkeeping that most web apps spend their cycles in doesn’t have specialized hardware accelerators the way matrix arithmetic does.”

                                                                                              Yeah, all the ideas I have for it are incremental. The best illustration of where rest of gains might come from is Cavium’s Octeon line. They have offloading engines for TCP/IP, compression, crypto, string ops, and so on. On rendering side, Firefox is switching to GPU’s which will take time to fully utilize. On Javascript side, maybe JIT’s could have a small, dedicated core. So, there’s still room for speeding Web up in hardware. Just not Moore’s law without developer effort like you were saying.

                                                                                    2. 9

                                                                                      Although you partly covered it, I’d say “execution of programs” is good wording for JavaScript since it matches browser and OS usage. There’s definitely advantages to them being smaller. A guy I knew even deleted a bunch of code out of his OS and Firefox to achieve that on top of a tiny, backup image. Dude had a WinXP system full of working apps that fit on one CD-R.

                                                                                      Far as secure browsers, I’d start with designs from high-assurance security bringing in mainstream components carefully. Some are already doing that. An older one inspired Chrome’s architecture. I have a list in this comment. I’ll also note that there were few of these because high-assurance security defaulted on just putting a browser in a dedicated partition that isolated it from other apps on top of security-focused kernels. One browser per domain of trust. Also common were partitioning network stacks and filesystems that limited effect of one partition using them on others. QubesOS and GenodeOS are open-source software that support these with QubesOS having great usability/polish and GenodeOS architecturally closer to high-security designs.

                                                                                      1. 6

                                                                                        Are there simpler browsers optimised for displaying plain ol’ hyperlinked HTML documents, and also support modern standards? I don’t really need 4 tiers of JIT and whatnot for web apps to go fast, since I don’t use them.

                                                                                        1. 12

                                                                                          I’ve always thought one could improve on a Dillo-like browser for that. I also thought compile-time programming might make various components in browsers optional where you could actually tune it to amount of code or attack surface you need. That would require lots of work for mainstream stuff, though. A project like Dillo might pull it off, though.

                                                                                          1. 10
                                                                                            1. 3

                                                                                              Oh yeah, I have that on a Raspberry Pi running RISC OS. It’s quite nice! I didn’t realise it runs on so many other platforms. Unfortunately it only crashes on my main machine, I will investigate. Thanks for reminding me that it exists.

                                                                                              1. 2

                                                                                                Fascinating; how had I never heard of this before?

                                                                                                Or maybe I had and just assumed it was a variant of suckless surf? https://surf.suckless.org/

                                                                                                Looks promising. I wonder how it fares on keyboard control in particular.

                                                                                                1. 1

                                                                                                  Aw hell; they don’t even have TLS set up correctly on https://netsurf-browser.org

                                                                                                  Does not exactly inspire confidence. Plus there appears to be no keyboard shortcut for switching tabs?

                                                                                                  Neat idea; hope they get it into a usable state in the future.

                                                                                                2. 1

                                                                                                  AFAIK, it doesn’t support “modern” non-standards.

                                                                                                  But it doesn’t support Javascript either, so it’s way more secure of mainstream ones.

                                                                                                3. 8

                                                                                                  No. Modern web standards are too complicated to implement in a simple manner.

                                                                                                  1. 3

                                                                                                    Either KHTML or Links is what you’d like. KHTML would probably be the smallest browser you could find with a working, modern CSS, javascript and HTML5 engine. Links only does HTML <=4.0 (including everything implied by its <img> tag, but not CSS).

                                                                                                    1. 2

                                                                                                      I’m pretty sure KHTML was taken to a farm upstate years ago, and replaced with WebKit or Blink.

                                                                                                      1. 6

                                                                                                        It wasn’t “replaced”, Konqueror supports all KHTML-based backends including WebKit, WebEngine (chromium) and KHTML. KHTML still works relatively well to show modern web pages according to HTML5 standards and fits OP’s description perfectly. Konqueror allows you to choose your browser engine per tab, and even switch on the fly which I think is really nice, although this means loading all engines that you’re currently using in memory.

                                                                                                        I wouldn’t say development is still very active, but it’s still supported in the KDE frameworks, they still make sure that it builds at least, along with the occasional bug fix. Saying that it was replaced is an overstatement. Although most KDE distributions do ship other browsers by default, if any, and I’m pretty sure Falkon is set to become KDE’s browser these days, which is basically an interface for WebEngine.

                                                                                                    2. 2

                                                                                                      A growing part of my browsing is now text-mode browsing. Maybe you could treat full graphical browsing as an exception and go to the minimum footprint most of the time…

                                                                                                  2. 4

                                                                                                    And more importantly, is there any (important to the writers) advantage to them becoming smaller? Security maybe?

                                                                                                    user choice. rampant complexity has restricted your options to 3 rendering engines, if you want to function in the modern world.

                                                                                                    1. 3

                                                                                                      When reimplementing malloc and testing it out on several applications, I found out that Firefox ( at the time, I don’t know if this is still true) had its own internal malloc. It was allocating a big chunk of memory at startup and then managing it itself.

                                                                                                      Back in the time I thought this was a crazy idea for a browser but in fact, it follows exactly the idea of your comment!

                                                                                                      1. 3

                                                                                                        Firefox uses a fork of jemalloc by default.

                                                                                                        1. 2

                                                                                                          IIRC this was done somewhere between Firefox 3 and Firefox 4 and was a huge speed boost. I can’t find a source for that claim though.

                                                                                                          Anyway, there are good reasons Firefox uses its own malloc.

                                                                                                          Edit: apparently I’m bored and/or like archeology, so I traced back the introduction of jemalloc to this hg changeset. This changeset is present in the tree for Mozilla 1.9.1 but not Mozilla 1.8.0. That would seem to indicate that jemalloc landed in the 3.6 cycle, although I’m not totally sure because the changeset description indicates that the real history is in CVS.

                                                                                                      2. 3

                                                                                                        In my daily job, this week I’m working on patching a modern Javascript application to run on older browsers (IE10, IE9 and IE8+ GCF 12).

                                                                                                        The hardest problems are due the different implementation details of same origin policy.
                                                                                                        The funniest problem has been one of the used famework that used “native” as variable name: when people speak about the good parts in Javascript I know they don’t know what they are talking about.

                                                                                                        BTW, if browser complexity address a real problem (instead of being a DARPA weapon to get control of foreign computers), such problem is the distribution of computation among long distances.

                                                                                                        Such problem was not addressed well enough by operating systems, despite some mild attempts, such as Microsoft’s CIFS.

                                                                                                        This is partially a protocol issue, as both NFS, SMB and 9P were designed with local network in mind.

                                                                                                        However, IMHO browsers OS are not the proper solution to the issue: they are designed for different goals, and they cannot discontinue such goals without loosing market share (unless they retain such share with weird marketing practices as Microsoft did years ago with IE on Windows and Google is currently doing with Chrome on Android).

                                                                                                        We need better protocols and better distributed operating systems.

                                                                                                        Unfortunately it’s not easy to create them.
                                                                                                        (Disclaimer: browsers as platforms for os and javascript’s ubiquity are among the strongest reasons that make me spend countless nights hacking an OS)