1. 4

    Has anyone done analysis on the scaling of these models? Is a 10x model 10x better? More? Less?

    1. 8

      Yes. See Scaling Laws for Neural Language Models. It follows power law and scaling is pretty exact.

      10x more parameters result in 16% reduction of loss. Exponent is 0.076.

      1. 3

        Important caveat: provided your dataset is large (and clean) enough, see Figure 9 from the paper.

        “For large [dataset size] D, performance is a straight power law in model [parameter count] N . For a smaller fixed D, performance stops improving as N increases and the model begins to overfit.”

        1. 1

          Thank you for the link to the paper. I need to read a bit into the topic to understand it better.

          Maybe there is a market related reason why training of larger data has not happened e.g.

          • the solution has been kept in house - this has happened with GPT-3, as I understand, so probably is not the case
          • the increase in model performance is no longer attractive when it comes to the cost achieving it
          • there is not demand on the market for the price a GPT-N+ solution would bring

          Out of cusiosity: What interesting applications have come out of GPT-3 based technology?

          1. 1

            That’s fantastic. Thank you much!

        1. 7

          Took a bit to get used to the new UI, but I like it.

          1. 10

            AOSP’s Linux days appear numbered, progress on Fuchsia has been fast.

            1. 1

              Maybe, but possibly only for the android ecosystem. I doubt that the server world will move away from that. And any company that wants to have a collaboration is better of using linux than some google OS which allows them to have an edge over many functions (by simply keeping them private, so everyone has their own extensions).

              1. 10

                AOSP is the Android Open Source Project. https://source.android.com/

                1. 2

                  AOSP doesn’t include anything required to run something you’d call android for normal users

                  • location
                  • sync
                  • playstore
                  • playstore APIs required for many apps (see location,sync..)
                  • google camera (no it won’t run unless modified to do so on approved hardware/os variations)

                  This is why people are using things like microG or opengapps on their custom ROMs.

                  Google explicitly got sued under anti-trust because they dictate the “kind” of android you can use when you want a real android and not some useless thing that’s running as much apps as ubuntu phone.

                  And let’s not talk about their driver stack, where every vendor ships their own crippled variation of android with their own binary blobs you have to use - if you can even download them by yourself and not have to compile their own version of android.

                  1. 5

                    I think the person you originally replied to was referring to the usage of the Linux kernel in the Android project and suggesting that the days that Linux will continue to be used as the base for the Android operating system are numbered.

                    I’m aware of the things you’ve mentioned here, but I’m not sure how they are related to the original point.

                    1. 2

                      Yeah you’re right, totally missed that

                2. 8

                  is better of using linux than some google OS

                  But what if… Fuchsia was actually much less work to write and maintain drivers for, due to drivers running in userspace and hitting stable APIs and ABIs?

                  1. 5

                    That might be a nice side effect (I know little about Fuschia internals other than some former Be people were working on it), but my suspicion is Google is in it primarily for the non-copyleft.

                    1. 12

                      I think you have it exactly backwards. Getting rid of copyleft is a nice bonus, but the thing that seems to be causing actual problems is HW vendors refusing to forward-port their drivers to new kernel versions, whether or not they throw some source code over the wall. With a stable driver/kernel interface, this would not be a problem.

                      1. 3

                        I think you under-estimate the ability of vendors to produce crap. Kernel-land vs user-land and APIs are very secondary. As soon as they can write crap, they’ll write crap. Sure the crap will be more isolated but it won’t work better and with less incentive to produce working stuff, it might even end up worse. I don’t know how things would turn out but “works well in practice, not only in demos” doesn’t seem to have been the goal of vendors so far.

                        1. 3

                          Sure the crap will be more isolated but it won’t work better and with less incentive to produce working stuff

                          That’s the purpose of isolation though, isn’t it? To allow folks to write crap and not have it mess up the entire kernel. It’s accepting the human nature of writing bad code and trying to contain the damage.

                          1. 3

                            To play devil’s advocate: reducing penalties for writing crap incentivizes developers to write crap, so more crap will appear.

                            To be clear, I’m entirely in favor of better OSes - but usually, when you incentivize something, people will do it. Ideally, we’d have developers trying hard to write good software and OSes that compensate for their mistakes - but in order to do that, we’ll need to incentivize them properly.

                            1. 1

                              Yup, that’s the purpose. But if that’s the camera driver and it’s crap, you’re going to have troubles taking pictures and that will cripple more than one app. I’m wary they’ll manage to make things worse. Not that the OS would make that more prevalent, but that these vendors are really really terrible.

                              1. 3

                                It sounds like the problem for Google is that the vendor makes it work once and then abandons it, so upgrading the kernel becomes hard.

                                If it never worked in the first place, it wouldn’t matter whether the kernel upgrade broke it, because it never worked in the first place. The problem Google seems to be facing is that it did work at one point, but keeping it working is a PITA due to the lack of encapsulation/isolation.

                            2. 2

                              When running in unprivileged mode, drivers are this much easier to both debug and reverse engineer. This is on top of all other advantages.

                              It’s a win-win situation.

                          2. 4

                            I believe Fuchsia is MIT licensed. https://fuchsia.googlesource.com/fuchsia/

                            1. 7

                              Yeah, not copyleft like GPLv2 is.

                              1. 2

                                Sorry, my brain read non-copyleft as proprietary for some reason.

                              2. 3

                                looks like 2-clause BSD (aka “Simplified BSD License” or “FreeBSD License”): https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/LICENSE

                                EDIT: With a litigation revoking patent grant! https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/PATENTS
                                Not sure why they didn’t just use apache-2 license?

                              3. 4

                                but my suspicion is Google is in it primarily for the non-copyleft.

                                You would be correct.

                              4. 3

                                It’s ultimately up to Google whether Fuchsia has stable ABI. I do think the biggest liability is the fact that this is after all a Google project - and they choose to kill it regardless of how good or beloved it is (eg: Reader)

                            1. 3

                              As someone who hasn’t tried any of the Beam languages; what’s the package ecosystem like? Can Gleam consume elixir/erlang packages?

                              1. 6

                                Yes, there should be no problem. However there may be some additional work to be done in Gleam to provide typing for such functions.

                                Here you have docs for that. It should be also perfectly possible to call Gleam functions from other BEAM languages as well.

                                1. 2

                                  Thanks to the great work of Elixir and Erlang there’s a good package ecosystem which can be used by all the languages on the VM. The website has details https://hex.pm/

                                1. 3

                                  Such a shame. I only know Dan from his work and from his fantastic defcon 22 talk. He will be missed.

                                  Defcon talk: https://www.youtube.com/watch?v=xneBjc8z0DE

                                  1. 2

                                    Thank you for the link. I wasn’t familiar with his work before today, but that talk was very interesting.

                                    As a sidenote, does anyone know the (scooby doo?) shirt he’s wearing?

                                  1. 16

                                    I personally like the path that gnome is on. I get that not everyone does and that’s fine, but ever since gnome 3 I’ve really fallen in love with gnome.

                                    1. 2

                                      I used to be very anti-gnome (“you’re turning my desktop into a phone?!”), but I’ve been using gnome for like three months because I wanted to actually give it a chance after using contrarian window managers for like a decade. I threw on dash to panel, made it very un-gnomey, etc. Gnome 40 came out and broke basically every extension, but I didn’t have time to transition to anything else at the time, so I was forced to just run with them turned off. With nothing more than the Yaru-remix shell/gtk/application theme running I started to really like how default gnome 40 looks and feels with just one theme and a few extra keyboard shortcuts turned on. Pretty sure even after all the recommended extensions start getting fixed I might stick with this.

                                      1. 2

                                        The problem is that their hold over GTK means that every other DE is forced to either follow suit in whatever Gnome comes up with, or invest in expensive workarounds (see CSD).

                                        I encourage Gnome devs to do whatever they want, but as a non-Gnome-user it gets tiring to be at the receiving end of their “innovations” without having much choice. I just want to be left alone.

                                        1. 5

                                          It’s far from only GTK. Their hold permeates the stack and slowly but surely forces more and more of these things to fit a certain mold as their ‘vision’ (OSandroidX with glaucoma) can only really be fulfilled with strong coupling across a wide range of desktop-system services. What little competition is left will be forced to write adapters (eudev, elogind, …) until they architecturally become more or less the same but lagging behind, accept obscurity, or run out of steam entirely and join the retro-computing trend.

                                      1. 23

                                        Openwrt all the way

                                        1. 2

                                          I’ve used openwrt in the past for single router/AP setups, but as far as I’m aware for larger properties it wouldn’t be enough, unless I’m misunderstanding something. Is it possible to use OpenWRT with multiple APs?

                                          1. 3

                                            It is possible, either as an 802.11s mesh or with a number of wired access points set up in bridge mode. I’m currently using the latter and it works fine.

                                          2. 2

                                            Same here, openwrt as main router and few dumb ap for wireless

                                            1. 3

                                              What do you have for a dumb AP?

                                              I’m in the market for something that I can broadcast two ssids (guest and home) and have them on separate vlans.

                                            2. 2

                                              With what kind of hardware?

                                              1. 3

                                                Not the OP, but in my case a NetGear R7800. Does 802.11ac, has dual radios so you can run 2.4GHz & 5Ghz simultaneously. 4+1 gigabit ethernet ports with a half decent switch behind them that can do tagged vlans.

                                                1. 1

                                                  I’m still using an old tplink archer c7. Probably gonna do an upgrade in the next year or so to get wifi 6. Pretty sure it was something like $80 back in 2014 or 2015.

                                                  1. 1

                                                    Not the OP, but I use a Linksys WRT1900ACS. A tad pricy, or was when I got it, but the wifi is good, has native support for OpenWRT, and and it’s fast enough to handle gigabit fiber.

                                                1. 3

                                                  I switched to a logitech mx keys last year and I’m very happy with it. I did the mechanical keys thing for a bit, but it’s not for me.

                                                  1. 2

                                                    Logitech MX Keys is my favorite keyboard as well! I also use the MX Master 3 mouse.

                                                  1. 8

                                                    [ Disclaimer: I have no knowledge of AMD roadmaps ]

                                                    I don’t find it at all surprising that AMD is developing an Arm[1] chip. They’ve been an Arm licensee for ages and already use Arm cores in some places (e.g. in the platform security processor). I’d be quite surprised if they hadn’t had a group working on fitting an Arm front end to their cores for a while. That said, there’s a big difference between ‘working on X’ and ‘shipping X as a product’. There’s a big gap between ‘AMD developing an Arm core internally so that they have leverage with Intel when they renew cross-licensing deals’ and ‘AMD plans on shipping an Arm laptop part’. I’d love to know which of these it actually is. Apple kept their x86 implementation of OS X around for around a decade before the Intel switch, to use in negotiations with IBM and Motorola / FreeScale. It took a change in the competitive landscape outside of their control before they shipped it as a product.

                                                    [1] Minor aside: Arm redid their branding a few years back and their style guide now recommends that you write it as Arm not ARM. It originally stood for Acorn RISC Machines, then Advanced RISC Machines, but it was just ARM Holdings for a while and they’ve dropped the term ‘RISC’ from everything as well. They now refer to the Arm architectures as ‘load-store architectures’, not as RISC. The instruction sets for both AArch32 and AArch64 are pretty massive, but they are orthogonal and everything is added because a compiler / OS actually can make use of it, unlike traditional CISC cores. It makes me chortle a bit when I read articles that talk about ARM RISC cores.

                                                    1. 2

                                                      That said, there’s a big difference between ‘working on X’ and ‘shipping X as a product’.

                                                      They did have an Arm SoC as a product, the Opteron A1100, that could be purchased.

                                                      1. 2

                                                        Could it? I thought it never went beyond pre-purchase/demo.

                                                        1. 2

                                                          It could albeit briefly. There was a generally available board on 96boards.org

                                                          http://armdevices.net/2015/11/16/amd-huskyboard-96boards-enterprise-edition-explained-by-jon-masters-of-red-hat/

                                                          1. 2

                                                            A more popular product was the SoftIron Overdrive 1000/3000

                                                      2. 1

                                                        Going off on a tangent…

                                                        The name load-store architecture makes sense to me for the arm instruction set, but the name has always made me wonder about its counterparts. Someone invented that name for one class within a classification, presumably because the classification made sense as a way to separate CPU architectures into top-level classes. What is that classification and what are other other classes?

                                                        1. 1

                                                          The other class is CISC where you have instructions that tell the CPU to load a value, modify it and store it back (eg. x86: addl $3, 4)

                                                          It never got a “fancy” name, and I guess the load-store naming only appeared because “reduced instruction set” was hard to say with a straight face when talking about an architecture with 1000+ instructions (e.g. ARM) because reduced can mean both “functionally reduced instructions” (e.g. load-store architecture) and “reduced number of instructions” (what RISC originally was as well, but only incidental).

                                                      1. 4

                                                        I wonder if the fedora server project will adopt a longer support cadence after this.

                                                        1. 2

                                                          Fedora is also a RedHat project, so probably won’t.

                                                          1. 2

                                                            Yes, but it seems a little independent. The messaging out of the fedora project is all about being community driven at least.

                                                            1. 1

                                                              It’s also bleeding edge, latest versions and released every 6 months. Changing this would be pretty much worthless for RH?

                                                        1. 10

                                                          comparable in performance to AMD’s “Excavator” cores used in the A8-7680 Godavari architecture. Of course, this demonstrates how far behind Longsoon is from TSMC and Intel in performance, speed, and efficiency of its latest node.

                                                          That’s a mainstream CPU from just two years ago, and probably fairly representative of what people are actually using today. I don’t get this implied assumption that every chip maker is supposed to be competing for the fastest chips. No one is expecting Volvo to be competing with Ferrari.

                                                          Due to this laptop being in the Chinese market, Windows is not supported at all. It only runs Chinese “domestic operating systems” that are typically modified versions of Linux.

                                                          I’m going to bet that the MIPS architecture has more to that than “the laptop being in the Chinese market”.

                                                          1. 0

                                                            No one is expecting Volvo to be competing with Ferrari.

                                                            My inner Volvo fanboy is crying.

                                                            1. 4

                                                              At least with your Volvo you get to drive more routes than just between your home and your service technician.

                                                              1. 1

                                                                Your inner Volvo fanboy should be aware of the Volvo 740 Turbo ad comparing a 740 Turbo Wagon with a Lamborghini pulling a trailer, see http://volvolady.com/some-vintage-volvo-ads/attachment/740-turbo-wagon/

                                                                1. 1

                                                                  Yeah Volvo in the 80s and early 90s had brilliant marketing :)

                                                              2. 0

                                                                That’s a mainstream CPU from just two years ago, and probably fairly representative of what people are actually using today.

                                                                That particular sku launched in 2018 with the uarch having been launched in 2015. Further Excavator was pretty poor at launch. I don’t think it’s really representative of what people are using.

                                                              1. 2

                                                                Looks neat. Word of warning for anyone trying to run this on a monolith; pysa wants tons of ram. I let my run get to 80GB before killing it.

                                                                1. 3

                                                                  I also find the linux tag misleading.

                                                                  1. 15

                                                                    Nice of them to delete the data

                                                                    1. 12

                                                                      Right? Imagine if they had exfiltrated everything; this could be much worse. This seems like it might be a greyhat effort to get people to secure their databases, especially given the whole metaphor of cats dropping things off of tables and filling the empty air with their meows. That’s just too funny to be an accident.

                                                                    1. 6

                                                                      I’m so happy to see thunderbird development continuing.

                                                                      1. 4

                                                                        Tempting, but I’m curious about a few things:

                                                                        • It mentions OpenGL ES 3.2 support. The official ARM drivers support ES 3.2 but only for a few select situations (mostly Android), and getting them to actually function on arbitrary linux distros is nontrivial. What’s the graphics support actually like?
                                                                        • Does it annoyingly route almost all network and disk I/O through USB like the RPi does?
                                                                        1. 3

                                                                          I just ordered one. If you like I can run some tests for you when it comes in. I had a C2 years ago and at least for that device network and disk were routed to the SOC.

                                                                          1. 1

                                                                            If you feel like it. If you get graphics working well then glxinfo | grep OpenGL should have the info I want.

                                                                            1. 2

                                                                              Here you go. This is with Ubuntu 20.04 pre-installed default from their emmc module. The vendor string might be due to me doing export DISPLAY=:0 to get around the display not being available in my headless setup.

                                                                              root@odroid:~# glxinfo | grep OpenGL
                                                                              OpenGL vendor string: VMware, Inc.
                                                                              OpenGL renderer string: llvmpipe (LLVM 9.0.1, 128 bits)
                                                                              OpenGL core profile version string: 3.3 (Core Profile) Mesa 20.0.4
                                                                              OpenGL core profile shading language version string: 3.30
                                                                              OpenGL core profile context flags: (none)
                                                                              OpenGL core profile profile mask: core profile
                                                                              OpenGL core profile extensions:
                                                                              OpenGL version string: 3.1 Mesa 20.0.4
                                                                              OpenGL shading language version string: 1.40
                                                                              OpenGL context flags: (none)
                                                                              OpenGL extensions:
                                                                              OpenGL ES profile version string: OpenGL ES 3.1 Mesa 20.0.4
                                                                              OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10
                                                                              OpenGL ES profile extensions:
                                                                              
                                                                              1. 1

                                                                                Thanks! That’s certainly… something, though it kind of raises more questions than answers. VMWare and llvmpipe suggest it’s a software OpenGL renderer, which means it’s emulating a GPU instead of actually using the hardware available. If the setup is headless, without any X server running and no display plugged in, then that may or may not be influencing it. If you’re doing X forwarding then it often can’t do much of any use with the GPU hardware anyway.

                                                                                Interpreting this is always kinda a black art on Linux, since almost all GPU drivers use Mesa to some degree or another, whether they’re binary blobs or open source drivers, and while Mesa is pretty good at choosing the best driver it has available, I’ve yet to find a way to get it to explain to me why it’s choosing a particular driver and which ones it’s choosing among. I mostly just rely on distro packages to do the correct black arts for me. If anyone has any pointers, I’d love to hear more.

                                                                                1. 1

                                                                                  Well, turns out that it’s the same with an hdmi cable plugged in and displaying out to a TV. No idea why it shows up as VMWare ¯_(ツ)_/¯

                                                                                  1. 1

                                                                                    Glad to help. I’ll give glxinfo another go once I get an hdmi cable and get this hooked up to my TV.

                                                                            2. 3

                                                                              Does it annoyingly route almost all network and disk I/O through USB like the RPi does?

                                                                              I’m not sure, but does this help? https://wiki.odroid.com/_detail/odroid-c4/c4_blockdiagram_rev0.4.png?id=odroid-c4%3Aodroid-c4

                                                                              1. 2

                                                                                It does help, if I’m reading it correctly. Looks like the GigE and SD go through their own interfaces instead of piggybacking off of USB. Thanks!

                                                                            1. 4

                                                                              I’m so curious what openbsd devs are going to do with this (great) patch

                                                                              1. 5

                                                                                Review it.

                                                                                1. 2

                                                                                  Me too. It’ll probably be the most interesting patch review round for wireguard.

                                                                                1. 1

                                                                                  Any word on when the recordings get posted?

                                                                                  1. 2

                                                                                    we’re processing them, which will take a few weeks :) we’ll tweet out and put them on our website when they’re ready: http://bangbangcon.com/recordings.html

                                                                                    in the meantime, the links thirdtruck posted allow you to see the whole stream

                                                                                    1. 1

                                                                                      Awesome, thanks!

                                                                                  1. 2

                                                                                    Well that’s a fascinating development. Thanks for sharing.