1. 2

    Wouldn’t parMap :: Strategy b -> (a -> b) -> [a] -> [b] be better fit for this example, instead of using the infix notation?

    On a related note: Is it true that future versions of GHC want to make maps and related functions parallel by default? I remember reading about it maybe 1 1/2 years ago, but I don’t think it was an authoritative source?

    1. 2

      parMap would achieve the same thing and is probably more idiomatic. I suppose I went with the more verbose version because I wanted to show how the primitives compose into something larger.

      The idea of automatic parallelization is interesting. My initial reaction is that just parallelizing everything would probably slow things down due to overhead (starting a spark is cheap but not free). Perhaps you could come up with some heuristics that would work well though.

      1. 1

        Are you sure you’re not thinking of data parallel haskell? I don’t see that mentioned in the linked page.

        https://wiki.haskell.org/GHC/Data_Parallel_Haskell

        Which is mostly abandoned now.

      1. 2

        Work: well its work, and this week is fixing merge conflicts for stuff, so exciting

        Home: Been debugging building ghc 8.2.2 on armhf for alpine linux on/off the past few months. Will keep working on that, but discovered I wasn’t the only one with the issue. So have a ticket open on that front.

        Also porting ghc to aarch64 and i586, but encountering other fun issues unrelated to ghc, aka the cross compilation stuff in alpine linux is broken for seemingly just me somehow.

        1. 3

          Here’s what I’ve been doing for 15 years. In .bashrc:

          unset HISTFILE HISTFILESIZE
          export HISTSIZE=1000000
          export HIST_LOGFILE="$HOME/.log/shell/$(date +"%Y")/$(date +"%m-%d-%H-%M-%S")-$(hostname)-$(tty |perl -pwe 's,/,_,g')"
          mkdir -p $(dirname $HIST_LOGFILE)
          export HISTTIMEFORMAT="%Y-%m-%d-%H-%M-%S "
          trap "history -w $HIST_LOGFILE" EXIT
          

          Sometime in the past I switched to Zsh and started saving complete timestamps for each command. In .zshrc:

          setopt extended_history hist_find_no_dups
          unset HISTFILE
          export SAVEHIST=1000000
          export HISTSIZE=$SAVEHIST
          export HIST_LOGFILE=$HOME/.log/shell/`date +"%Y"`/$(date +"%m-%d-%H-%M-%S")-$(hostname)-$(test $TMUX && echo "tmux-")$(tty |perl -pwe 's,/,_,g')
          mkdir -p $(dirname $HIST_LOGFILE)
          save_history() {
            fc -l -n -t "%Y-%m-%d %H:%M:%S" -D -i 1 >! $HIST_LOGFILE.tmp && mv $HIST_LOGFILE.tmp $HIST_LOGFILE  # -d doesn't include seconds
          }
          precmd_functions+=(save_history)
          history() {
            fc -l -n $* 1
          }
          

          Basically I turn off the default location for history to keep windows from clobbering each other. Each session gets its own private history file. To search for commands I use grep. To combine history from servers I periodically rsync. Everything is always private to guard against the password issue.

          My priority is maintaining an audit trail of what I did in experiments. So YMMV. But if this works for you, it’s a lot fewer moving parts than OP.

          1. 2

            I just do this with zsh, it handles merging multiple histories in parallel with a few set options. I’ve never really wanted to combine history between servers however. Just have common history amongst shells. I prefer sharing the history as I end up in one shell and want to use a command I just typed in another.

            export HISTSIZE=5000
            export SAVEHIST=${HISTSIZE}
            export HISTFILE=~/.zsh_history
            setopt append_history
            setopt extended_history
            setopt hist_reduce_blanks
            setopt hist_no_store
            setopt hist_ignore_dups
            setopt histignorespace
            setopt share_history
            setopt inc_append_history
            

            I could probably sort out saving a history file a day but really can’t be arsed to bother, don’t see much value in that.

          1. 40

            Almost anyone just reads the mail, but not the mail it was a reply to or the discussion that comes out of it.

            What makes you think this is true? I always read the surrounding thread.

            I have found many of these links very interesting and feel they are legitimate posts for Lobsters.

            1. 6

              General experience with the discussions that burst out below. Exceptions make the rule, as they say.

              Also, in the linked example, a lot of what Theo was referring to was in topics a couple of days away, so establishing context isn’t always “click previous, click followup”.

              1. 8

                Exceptions make the rule, as they say.

                No. “Exceptio probat regulam in casibus non exceptis” means that if there is an explicit rule regarding exceptions then there exists an implicit rule from which those exceptions are derived.

                1. 2

                  I saw the pattern you were talking about. It seemed part of a larger one being really eager to speak but lazy about reading. I second your proposal of no mailing list threads unless it’s an independent write-up with context included so people see big picture at a glance.

                  1. 2

                    I agree and disagree. I agree that if we’re going to ban deep linking into threads then sure a write up would be ok.

                    But, I would rather posit a second option, which amounts to when linking to a thread we link in a comment to the specific post that is at issue with the link to the top of the thread as the article link. Thus trying to encourage everyone to read the context that surrounds the post and allowing everyone to come to their own conclusions as to the post in its original context.

                    I say this as while I’d love independent write-ups, noticing how every write up or news post can skew things ever so slightly I would rather encourage everyone to read through the original source material.

                    I would propose the same rule for forum posts, I wouldn’t expect write ups in all cases and think that requirement would just discourage discussion over the actual issue at hand, if any.

                    1. 2

                      “I say this as while I’d love independent write-ups, noticing how every write up or news post can skew things ever so slightly I would rather encourage everyone to read through the original source material.”

                      This is a good point. The Dalmore memo discussions were a good example of that.

                2. 1

                  Personally, I try to get the context but I’ll admit that I can get a bit lazy with it. And my reflex is also a bit to do meta-discussions on communication.

                  Unlike something like a Github issue, a bit more effort is required to get the full context. Not a huge amount, of course, but enough. Not to mention double-quoting and whatnot confusing people.

                  Perhaps linking to the first e-mail in the chain will force people to read through stuff? Not sure how doable that is

                1. 9

                  There’s an incredible lengthy reply in this thread, which is completely made up.

                  The thing is that to run those 1 & 0, it has to, technically, store them in a physical way so that it can be passed through to what’s next. As it’s 0 & 1, it’s not encrypted or protected. It’s pure raw data. The encryption and protection are usually done after the data has passed through the processor… by a task handled by the processor (ironically). Now, what they have “found” (which is false. it’s has been known since the 80’s) is that it’s possible to access this raw data by force feeding some 0 & 1 to the processor which can be hidden in anything and makes it start an hidden small software which, for example, could send a copy of the raw data through the web.

                  Fascinating.

                  1. 8

                    It’s not just completely made up, it’s gibberish.

                    1. 3

                      This almost sounds like it was written by some AI…

                      1. 3

                        Looks more like a markov chain to me.

                    2. 1

                      I saw hints of the truth in there which I thought were pretty funny. Like the bit about force feeding 1s and 0s I assumed was referring to specially crafted instructions to starve the CPU cache or trick the branch predictor or something. Hilarious.

                      Permalink for those who want it: https://www.epicgames.com/fortnite/forums/news/announcements/132642-epic-services-stability-update?p=132713#post132713

                    1. 3

                      magit is phenomenal. I’d use emacs just for this even if I didn’t use emacs.

                      1. 5

                        magit+org mode+tramp are my trio of reasons to use emacs. Super happy I converted from vi/vim, it can be a curve at times but well worth it.

                        1. 1

                          I personally find markdown-mode a great, and underrated markdown editor too. It’s not quite a drop-in org-mode replacement, but if one has to write markdown, even nonstandard with TeX-math extentions, it’s the best choice I know of, that’s at least reasonably lightweight. And AucTeX is also always mentionable, if one uses LaTeX.

                          1. 1

                            yeah, helm is one of those amazing things too. I wish more things had helm like interactions. with spacemacs and evil mode it makes moving from vim much easier.

                            1. 1

                              I use ivy mode personally, helm was a bit too slow for my taste. Ivy mode is super nice, way lighter weight than helm.

                        1. 13

                          Historically, C++ has had this problem due to its compilation model. Every class has to include the header of classes it uses (transitively) unless you use some gimmicks or bias toward C++‘s equivalent of interfaces. When dependencies aren’t managed well, this can lead to N^2 build-time.

                          Java fixed this by making the compiler aware of the .class files (making it a linker in a way), so no there’s need to be aware of transitive dependences. A big impetus for Go was to make build time rational with its interface scheme.

                          People have tried everything in C++ including taking all files into memory at once to avoid disk I/O. But the language still pushes toward compile time work with compute intensive template meta-programming, etc. It’s a long story. I’ve spent years breaking dependencies in C++ to aid testing. In my experience C is far more tractable this way.

                          1. 8

                            Isn’t that part of why modules are being introduced?

                            https://clang.llvm.org/docs/Modules.html

                            Aka it allows the compiler the ability to treat things a lot more granular and not have to recompile the world due to textual changes to unrelated header files.

                            1. 8

                              Yes. I wish them luck. It’s sad that build-times spin out of control in C++ unless you apply diligence. This isn’t true for many other languages.

                          1. 6

                            The horror, I would try to statically link EVERYTHING. At least the syscalls are stable.

                            1. 6

                              Yep, I just use musl libc and static link everything into the binary. Simpler than all the effort here to use a libc designed to be static linked.. Just need to make sure the kernels you run on for threaded stuff is recent enough, iirc 2.6.16-19ish is when those stabilized. But for most stuff its unlikely to matter.

                              Irony is the binaries are smaller than the glibc binaries linked against libc. glibc is not my favorite libc.

                              1. 3

                                Unfortunately that doesn’t work for games, you can’t statically link things like X/libGL/ALSA.

                                There’s a blogpost about building bins that only use the system dlopen/dlsym but that seems like a hassle. If you use GLFW and want to try it out you’ll need to modify it to use dlopened X (GLFW did change to do this for many parts of X quite recently), not sure about SDL. On the bright side there are already free tools to generate the dlopen/dlsym mess for OpenGL (glad is the best one), and the code to dlopen ALSA is pretty short.

                                I assume the glibc dynamic linker is too massive and annoying to statically link, and the musl one can’t load glibc libraries so is useless in practice, so for now you can’t do anything about the libdl dependency. (Now I have the go ahead from my boss to sell my hobby projects, I wonder if people would pay for a solution to this?)

                                1. 4

                                  The author of this blog post is a gamedev working on dhewm3 (Doom 3 port) and yquake2 (Quake 2 port). The post itself touches on SDL, OpenGL and X/Wayland. I might be not seeing something but what makes you believe the article can’t be applied to games (I believe the author did this work to build one of the doom/quake ports that way - but I’m only guessing here).

                                  1. 5

                                    I’m responding to ac, not the article!

                                    My point is that you can’t statically link everything, you have to dynamically load certain things, and to do that you have to dynamically load the dynamic loader.

                                    1. 3

                                      For dhewm3 and yq2 this is no problem: It’s open source and we don’t provide Linux binaries ourselves (and if we did an old compiler should work just fine).

                                      I’ve needed this for a commercial closed source game that uses some C++11 features and can’t be built with GCC 4.7

                                    2. 4

                                      (I’m the author of the blog post)
                                      You’re right, statically linking is no option for games (and probably not for anything else that needs to interact with X11 or Wayland).
                                      Also note that glibc is not the problem here - usually linking against an old version of it works just fine, and if you want to optionally use a more recent feature, you can still dlsym() it. (It’s annoying that it’s not easy to just point the compiler/linker to a glibc version to use, but that it uses the system version instead - however using a chroot for building release binaries is workable).
                                      The problems are mostly libstdc++ and libgcc (though maybe that one can actually be statically linked, not sure). This is because some systemlibs you need to interact with (at least libGL) use libstdc++, and if you statically link a different version than the one on the users system, things explode. Same problem if you just bundle and unconditionally use (e.g. via rpath $ORIGIN) a version of libstdc++ and the libGL (or whatever other lib) needs a newer version than the one you bundled.
                                      Of course all this is detailed in the article.. :-P

                                      By the way, SDL2 indeed uses dlopen() and dlsym() to interact with most system libs (especially the display and audio specific ones, probably not glibc..). This is pretty awesome: The same libSDL2-2.0.so.0 can support X11 and wayland, and so can your game executable (unless you do X11 specific stuff in addition to what SDL2 offers).

                                    3. 2

                                      That’s what I do using MUSL. Seems to work great.

                                    1. 5

                                      Why does every headline mention “MINIX based” now? MINIX is not what’s bad about it >_<

                                      1. 4

                                        On the other hand if “Minix” becomes synonymous with “that spyware OS” in public perception maybe it will get Tannenbaum to realize the whole “BSD licensing offers the users maximum freedom” schtick is hilariously backwards.

                                        1. 4

                                          Tannenbaums statement about the news MINIX was being used continued to support the BSD license. He only wished that Intel had told him, cause it would have been cool to know.

                                          1. 0

                                            He also said “this bit of news reaffirms my view that the Berkeley license provides the maximum amount of freedom to potential users” which is just utterly delusional.

                                            1. 12

                                              Intel is the user. Intel got maximum freedom. I don’t see what is ambiguous about that.

                                              Unless, are you are recursively applying a broader concept of freedom to all users potentially affected by consequences of the license choice? Because that’s an entirely separate matter altogether. And highly subjective.

                                              1. 7

                                                It seems to me that I am the user of my CPU and all the software on it, Intel is just the manufacturer. At least legally, morally, and in terms of how GP used the term. Unfortunately there may be an argument that Intel is a user in reality.

                                                1. 5

                                                  You’re a user of your CPU as a complete product, not of MINIX. Intel is the user of MINIX, they are the ones the license applies to. Legally speaking, that much is completely unambiguous.

                                                  1. 3

                                                    Intel is the distributor of Minix, they require a license to create copies and distribute it because that’s how copyright works.

                                                    I do not need a license for my use of Minix because my use does not create a copy or otherwise implicate copyright law. That does not mean I am not the user, legally or otherwise. Just as I am the user of a book when I read it, but I do not require a license to do so.

                                                    1. 4

                                                      Precisely, Intel is the user of the license. That was the intended meaning of user in Tannenbaum’s letter.

                                                    2. 1

                                                      Hmmm. Just like I may be a user of a Dell Machine with Windows on it? But in that case, the license applies to me.

                                                      MINIX’s license applies equally to source and binary forms, so the “user” is actually quite questionable. Technically, it seems that the license is passed on to me, the user of the CPU, since the copyright notice and such must be distributed along with a binary distribution…

                                                      So, I guess I could legally redistribute my changes to the binary blob that Intel puts on the CPU, too, no? Of course, provided my changes are also released under a BSD license. (Difficulty of this aside)

                                                      Disclaimer: IANAL.

                                                      1. 4

                                                        Hmmm. Just like I may be a user of a Dell Machine with Windows on it? But in that case, the license applies to me.

                                                        Totally different. Intel is licensing MINIX as a component. Dell is an authorized reseller of Windows, i.e. they are authorized to sell you a Windows license along with their product. While they probably license Windows for their employees, they don’t license Windows as a part of the product. Their terms of resale allow them to modify the original distribution, to include drivers and so on. Also crapware, apparently.

                                                        MINIX’s license applies equally to source and binary forms, so the “user” is actually quite questionable.

                                                        No, it isn’t. The user of the license is the distributor. The license notice must be attached. That’s all.

                                                        Technically, it seems that the license is passed on to me, the user of the CPU, since the copyright notice and such must be distributed along with a binary distribution…

                                                        A notice of license use does not constitute a license grant. That is, if I buy a software library A and include it in my product, that does not transitively grant a license for A to all of my users. That would be absurd.

                                                        So, I guess I could legally redistribute my changes to the binary blob that Intel puts on the CPU, too, no? Of course, provided my changes are also released under a BSD license.

                                                        BSD does not require you release derivatives under a BSD license, only that you attach the original license in the documentation or other materials provided with your derivation. And you could only redistribute changes if Intel’s license for their blob permits redistribution.

                                                2. 3

                                                  He also said “this bit of news reaffirms my view that the Berkeley license provides the maximum amount of freedom to potential users” which is just utterly delusional.

                                                  How is it delusional exactly? Seems to be correct, the freedom for intel to use it in the way they see fit is preserved.

                                                  1. 2

                                                    BSD is about developer freedom whereas GPL is about end user freedom. For Tannenbaum to claim otherwise is bizarre. He should be arguing for the license he chose on its merits.

                                                    1. 5

                                                      What are you talking about? He said BSD provides maximum freedom to potential users. And the next sentence makes it extremely clear he is referring to developers like Intel, not end users.

                                                      So as far as I’m aware Tannenbaum never made any such claim. He certainly didn’t in his open letter to Intel.

                                                3. 1

                                                  Intel is the user here, and Intel indeed got maximum freedom. I see nothing wrong with that.

                                                  What kind of license would you suggest?

                                                  GPL/copyleft? Wouldn’t do anything, Intel would just reluctantly post an uninteresting dump of MINIX source with the tiny modifications they made to make it fit in a small device.

                                                  “Don’t use this for evil” licenses? They’re just broken. Incompatible with any existing FOSS licenses. No one will use your code.

                                                  1. 12

                                                    GPL/copyleft? Wouldn’t do anything

                                                    The whole point of the tivoization clauses of the GPLv3 is to make it illegal to prevent the end user from modifying this kind of thing and allowing them to remove user-hostile behavior.

                                                  2. 1

                                                    BSD forbids the use of the original name without author’s permission, so the license is fine. But no license is going to protect you from FUD and journalist misinterpretation.

                                                1. 10

                                                  I don’t really see a lot of smaller open source projects having their own LTS releases.

                                                  What I see is them suffering from trying to support their ecosystem’s LTS releases. When CentOS ships with a Python that the Python core team has dropped, I’m stuck supporting it in my packages, because there are users there (and they blame me, not CentOS, for their troubles if I drop support).

                                                  1. 2

                                                    I don’t understand CentOS, is enterprise really so inflexible with a shorter release cycle?

                                                    1. 12

                                                      Yes. Change is bad. Especially when you have scary SLAs (that is, downtime on your end costs your company thousands of dollars per minute) you tend to be very careful about what and when you upgrade, especially if things are working (if it ain’t broke, don’t fix it).

                                                      1. 1

                                                        I wonder why we don’t make better software / devops to handle change. Maybe the pain to change once in a while is less than to roll with tighter LTS windows?

                                                        1. 7

                                                          Because starting to use a new methodology to handle change is a change on its own. And so a new technology can only climb the scale relatively slowly (so many projects half our size have used this technology that we can as well run a small trial). This means that some importants kinds of feedback are received with a timescale of years, not weeks…

                                                          1. 4

                                                            Exactly, its not that enterprises don’t want to change its that change in and of itself is hard. It is also expensive in time, which means money. Which basically means: keep things as static as possible to minimize when things break. If you have N changes amongst M things, debugging what truly broke is non trivial. Reducing the scope of a regression is probably the number one motivator of never upgrading things.

                                                            An example, at work we modify the linux kernel for $REASONS, needless to say, testing this is just plain hard. Random fixes to one part of the kernel can drastically alter how often say the OOM killer triggers. Sometimes you don’t see issues until several weeks of beating the crap out of things. When the feedback cycle is literally a month, I am not sure one could argue that we want more change to be possible.

                                                            I don’t see much of a way to improve the situation beyond just sucking it up and accepting that certain changes cannot be rushed without triggering unknown unknowns. Even with multi week testing you still might miss regressions.

                                                            This is a different case entirely than making an update to a web application and restarting.

                                                            1. 2

                                                              First of all, thanks for a nice presentation of this kind of issues.

                                                              This is a different case entirely than making an update to a web application and restarting.

                                                              I am not sure what you mean here, because a lot of web applications have a lot of state, and a lot of inner structure, and a lot of rare events affecting the behaviour. I don’t want to deny that your case is more complicated than many, I just want to say that your post doesn’t convey that it is qualitatively different as opposed to quantitatively.

                                                              I am not sure one could argue that we want more change to be possible.

                                                              What you might have wanted, is comparing throwing more hardware at the problem (so that you can run more code in a less brittle part of the system) with continuing with the current situation. And then there would be questions of managing deployments and their reproducibility, possiibility or impossibility of redundancy, fault isolation, etc. I guess in your specific case the current situation is optimal by some parameters.

                                                              Then of course, the author of the original post might effectively have an interest — opposing to yours — in making your current situation more expensive to maintain (this could be linked to a change that might make something they want less expensive to get). Or maybe not.

                                                              How much do the things discussed in the original post apply to your situation by the way? Do you try to cherry-pick fixes or to stabilize an environment with minimal necessary minor-version upgrades?

                                                              1. 3

                                                                I am not sure what you mean here, because a lot of web applications have a lot of state, and a lot of inner structure, and a lot of rare events affecting the behaviour. I don’t want to deny that your case is more complicated than many, I just want to say that your post doesn’t convey that it is qualitatively different as opposed to quantitatively.

                                                                I’m not entirely sure I can reasonably argue that kernel hacking isn’t qualitatively different from say web application development but here goes. Mind that some of this is going to be specific to the use cases I encounter and thus can be considered an edge case, however edge cases are always great for challenging assumptions you may not realize you had.

                                                                Lets take the case of doing 175 deployments in one day that another commenter linked. For a web application, there are relatively easy ways of doing updates with minimal impact to end users. This is mostly possible as the overall stack is so far removed from the hardware, its relatively trivial to do. Mind you I’m not trying to discount the difficulty but overall it amounts to some sort of HA or load balancing say via dns, haproxy, etc… to handle flipping a switch from old version to new.

                                                                One might also have an in application way to do A/B version flips in place in the application as well, whatever the case here, the ability to update is in lots of ways a feature of the application space.

                                                                A con to this very feature is that restarting the application and deploying a new version inherently destroys the state the application is in. Aka: lets say you have a memory bug, restarting it fixes it magically but you upgrade so often you never notice it. This is a case where I am almost 99% sure that any user space developer would catch bugs if they were to run their application for longer than a month. Now I doubt that will happen but its something to contemplate. The ability to do rapid updates is a two edged sword.

                                                                Now lets compare to the kernel. Lets take a trivial idea like adding a 64 bit pointer to the skb buffer. Easy right? Shouldn’t impact a thing, its just 64 bits what is 64 bits of memory amongst friends? Well a lot it turns out, lets say you’re running network traffic at 10Gb/s all the while where you have a user space application using up as much memory as it can. Probably overcommitting memory as well just to be annoying. Debugging why this application triggers the OOM killer after a simple change like I described is definitely non trivial. The other problem is you need to trigger the exact circumstances to hit the bug. And worst of all it can often be a confluence of bugs that trigger it. Aka some network driver will leak a byte every so often once some queue is over a certain size, meaning you have to run stuff a long time to get to that state again.

                                                                I’m using a singular example but I could give others where the filesystem can similarly play into the same stats.

                                                                Note, since I’m talking about linux lets review the things that a kernel update cannot reasonably do, namely update in place. This severely limits how a user space application can be run and for how long. Lets say this user space application can’t be shutdown without some effect on the users end goal. Unreasonable? Sure, but note that a lot of runtime processes are not designed with rapid updates and things like checkpointing so they can be re-run from a point in time snapshot. And despite things like ksplice to update the kernel in place, it has…. limitations to trying to update things. Limitations relating to struct layout tend to cause things to go boom.

                                                                In my aforementioned case, struct layout and the impact on memory can also severely change how well user space code runs. Say you add another byte to a struct that was at 32bytes of memory already. Now you’re requiring 40bytes of memory per struct. This means that its likely your’e wasting 24 bytes of memory and hurting caching of data in the processor in ways you might not know. Lets say you decide to make it a pointer, now you’re hitting memory differently and also causing changes to the overall behavior of how everything runs.

                                                                I’m only scratching the surface here, but I’m not sure how one can arrive at kernel development isn’t qualitatively different state wise than a web application. I’m not denigrating web app developers either here, but I don’t know of many web app developers worrying about adding a single byte to a struct as the performance impact causes more cache invalidation and making things ever so slower for what a user space process sees. They both involve managing state, but making changes in the kernel can be frustrating when a simple 3 line change can cause odd space leaks in how user applications run. If you’re wondering why Linus is such a stickler about breaking user space, its because its really easy to do.

                                                                I also wish I could magically trip up every heisenbug related to long running processes abusing the scheduler, vm subsystem, filesystem, and network but much like any programmer, bugs at the boundary are hard to replicate. Its also hard to debug when all you’ve got is a memory image of the state of the kernel when things broke. What happened leading up to that is normally the important part but entirely gone.

                                                                What you might have wanted, is comparing throwing more hardware at the problem (so that you can run more code in a less brittle part of the system) with continuing with the current situation. And then there would be questions of managing deployments and their reproducibility, possiibility or impossibility of redundancy, fault isolation, etc. I guess in your specific case the current situation is optimal by some parameters.

                                                                Not sure its optimal but a bit of a casus belli in that if you have to run a suite of user land programs that have been known to trigger bad behavior, and run them for a month straight to be overly certain things aren’t broken, throwing more hardware at it doesn’t make the baby any faster. Just like throwing 9 women at the making a baby problem won’t make a baby any faster, sometimes the time it takes to know things work just can’t be reduced. You can test more in parallel at once sure, but even then, you run into cost issues for the hardware.

                                                                How much do the things discussed in the original post apply to your situation by the way? Do you try to cherry-pick fixes or to stabilize an environment with minimal necessary minor-version upgrades?

                                                                Pretty much that, cherry-pick changes as needed, and stick to a single kernel revision. Testing is mostly done on major version changes, aka upgrading from version N to version M reapply the changes and let things loose to see what the tree shaking finds on the ground. Then debugging what might have introduced the bug and fixing that along with more testing.

                                                                Generally though the month long runs tend to be freaks of nature bugs. But god are they horrible to debug.

                                                                Hopefully that helps explain my vantage point a bit. If its unconvincing feel free to ask for more clarification. Its hard to get too specific due to legal reasons but I’ll try to do as well as I can. Lets just say, I envy every user space application as to the debugging tools they have. I wish to god the kernel had something like rr to debug back in time to watch a space leak as an example.

                                                                1. 1

                                                                  Thanks a lot.

                                                                  Sorry for a poor word choice — I meant that the end-goal problems you solve are on a continuum with no bright cutoffs that passes through the tasks currently solved by the most complicated web systems, by other user-space systems, by embedded development (let’s say small enough to have no use for FS), other kinds of kernel development etc. There are no clear borders, and there are large overlaps and crazy outliers. I guess if you said «orders of magnitude», I would just agree.

                                                                  On the other hand, poor word choice is the most efficient way to make people tell interesting things…

                                                                  I think a large subset of examples you gave actually confirm the point I have failed to express.

                                                                  Deploying web applications doesn’t have to reset the process, it is just that many large systems now throw enough hardware to reset the entire OS instance. Reloading parts of the code inside the web application works fine, unless a library leaks an fd on some rare operations and the server process fails a week later. Restarting helps, that’s true. Redeploying a new set of instances takes more resources, needs to be separately maintained, but allows to shrug off some other problems (many of which you have enumerated).

                                                                  And persistent state management still requires effort for web apps, but less than before more resources were thrown at it.

                                                                  I do want to hope that at some point kernel debugging (yes, device drivers excluded, that’s true) by running Bochs-style CPU emulator under rr becomes feasible. After all, this is a question of throwing resources at the problem…

                                                                  1. 1

                                                                    Deploying web applications doesn’t have to reset the process, it is just that many large systems now throw enough hardware to reset the entire OS instance. Reloading parts of the code inside the web application works fine, unless a library leaks an fd on some rare operations and the server process fails a week later. Restarting helps, that’s true. Redeploying a new set of instances takes more resources, needs to be separately maintained, but allows to shrug off some other problems (many of which you have enumerated).

                                                                    Correct, but this all depends upon the application. A binary for example would necessarily have to be restarted somehow, even if it means re exec()‘ing the process to get at the new code. Unless you’re going to dynamically load in symbols on something like a HUP it seems a bit simpler to just do a load balanced type setup and bleed off connections then restart and let connections trickle back in. But I don’t know I’m not really a web guy. :)

                                                                    I do want to hope that at some point kernel debugging (yes, device drivers excluded, that’s true) by running Bochs-style CPU emulator under rr becomes feasible. After all, this is a question of throwing resources at the problem…

                                                                    I highly doubt that will ever happen, but I wish it would. But qemu/bochs etc… all have issues with perfect emulation of cpus sadly.

                                                          2. 2

                                                            It’s not like we don’t have the software. GitHub deployed to production 175 times in one day back in 2012. Tech product companies often do continuous deployment, with gradual rollout of both app versions across servers and features across user accounts, all that cool stuff.

                                                            The “enterprise” world is just not designed for change, and no one seems to be changing that yet.

                                                          3. 1

                                                            if it ain’t broke, don’t fix it

                                                            And if it isn’t seriously affecting profitability yet, it ain’t broke.

                                                            Even if there are known unpatched vulnerabilities which expose people you have a duty to to increased risk.

                                                        2. 1

                                                          The article recommendation seems to be to create a separate branch for LTS backports (so that the new development can initially happen easier) and (maybe gradually) handing over most of the control of the backports. Unless these users are a significant share of the project contributors already (regardless of form of contribution).

                                                          Whether this recommendation is aligned with your motivation for the project is another question.

                                                        1. 6

                                                          I love how in the end he says the BSD license brings maximum freedom to users. As far as I know, the code isn’t published anywhere that’s inside the chip. As a user of Intel, this isn’t exactly “more freedom”. Great example why we need the GPL.

                                                          1. 1

                                                            The user here is Intel, and BSD indeed is more free in this regard. If it weren’t this it would just have likely been some embedded os they bought. The GPL wouldn’t have improved anything in this regard.

                                                            1. 2

                                                              You are wrong that the user of the software is intel. Intel is the “reader” of the source code and we are the “users” of the software the model generates. The same way a book doesn’t have “user” but a “reader”, software only has a “user” when it is running on hardware.

                                                              Why is the alternative buying an embedded OS? Let’s pretend MINIX was GPL. What if the economics were better to just release the source of a modified MINIX vs buy an embedded OS. I don’t buy the “If it’s not BSD, it’s proprietary” argument.

                                                              1. 1

                                                                You are wrong that the user of the software is intel. Intel is the “reader” of the source code and we are the “users” of the software the model generates.

                                                                No the users of intel are the component buyers generally. Individual developer/power users aren’t significant enough to matter economically. You’re conflating your use of intel devices as being the user of the intel device itself. This is no different than a car, you aren’t the user of the bosch speed controller in say a volkswagen, that was volkswagen and bosch that had a business relationship. Directly all you have control over is the overall package.

                                                                Why is the alternative buying an embedded OS?

                                                                Ref tivo and the wrt situation. Even if a gpl os was used, and source released, it will inexorably end up useless in that there will be no reasonable way to update or install updates on the devices in question. In many regards GPL os’s will only encourage this happening more.

                                                                And as a BSD license fan, I encourage it. The goal is to make good code available, not push an agenda on how my code be used or renumerated.

                                                                1. 1

                                                                  Your example is flawed since the firmware can be updated on these Intel chips including the ME stuff. If the Bosch controller can be changed, then I am certainly a user.

                                                                  It’s kind of like the repair/hacker movement in hardware too. Make shit that can be changed by the people using said shit. iFixit should rate electronics not just on physical repair-ability, but software repair-ability too.

                                                                  1. 1

                                                                    Not all embedded devices are designed nor intended to be updated however. Take the case where a one time write flash chip is used.

                                                                    That Bosch controller just may not be changed without changing other things. In these scenarios we are third party users not first.

                                                          1. 7

                                                            There’s a dimension that I haven’t seen mentioned yet.

                                                            Communication norms differ wildly across cultures. For instance, people in The Netherlands speak bluntly, to the point of seeming offensive to people from the US. Contrariwise, people from the US may seem like they constantly beat around the bush and won’t just say what they mean.

                                                            Now, I don’t know of a culture where saying “you should be ashamed of this” is considered acceptable form. But a statement like, “You’re wrong. Here’s how,” can come off as rude or belittling to people from cultures like the US and Canada, where direct criticism is not used. I’ve seen the reverse too, where “That’s bollocks” was routine business comms.

                                                            1. 2

                                                              You even see this distinction within the US. I notice a lot of northeastern individuals have a more direct approach to discussion than compared to west coast people.

                                                            1. 4

                                                              or just Stop Using Git To Deploy, period, full stop.

                                                              1. 5

                                                                More, stop using a VCS to deploy. Git or otherwise is inconsequential.

                                                                1. 5

                                                                  or just Stop Using Git To Deploy, period, full stop.

                                                                  Agreed. I was appalled when I realized the author’s point was that s/pull/fetch + reset/g.

                                                                1. 8

                                                                  Boxed values vs. unboxed. I think most people understand C here so lets put it with those concepts. It helps to understand the subject of the link.

                                                                  A boxed value:

                                                                  typedef struct { int number; } box;
                                                                  box* number = box_integer(5);
                                                                  

                                                                  Unboxed value:

                                                                  int number = 5;
                                                                  

                                                                  The box_integer allocates a place from heap where to write the integer, whereas the lower writes an integer directly. The implementation and the rules may vary here.

                                                                  How the data is stored in memory matters a lot because tighter it is packed, more information the processor can load to work on at once. Being able to work on an array of unboxed integers vs. boxed values can triple the performance. List strategies in PyPy rely on that difference, by attempting to use simpler list representations when it can.

                                                                  1. 4

                                                                    Memory layout control is one thing I particuarly find interesting in the Jai programming language.

                                                                    https://inductive.no/jai/

                                                                  1. 6

                                                                    I’ve never done pairing at that level of intensity - all day every day. Doesn’t sound like fun, and I can definitely see feeling the way the author did. I think it might be useful to do an hour or two a day at most.

                                                                    Related - I spend a lot of time on things that paring doesn’t sound so good for, including researching possible solutions, writing docs, reviewing other peoples’ code, troubleshooting, setting up meetings to build consensus on possible solutions, etc.

                                                                    1. 6

                                                                      I gotta be honest, the thought of 8 hours of pair programming a day is probably my definition of hell. Maybe for a while but daily sounds evil.

                                                                    1. 4

                                                                      I was always impressed that the 2nd generation iPod Shuffle used its headphone jack for both data transfer and charging. I wonder if that would be viable for charging any device that actually had to power a display.

                                                                      1. 1

                                                                        At 100mah, I doubt it, that battery is tiny in its size.

                                                                      1. 7

                                                                        As someone out of the loop to a degree, what impact, if any, will this have for a “regular” user? Aka:

                                                                        • Will I have to update my password?
                                                                        • Will I have to update my username?
                                                                        • Will my username carry over or do I need to get reinvited (bit unclear on this but asking stupid questions to be silly clear)?
                                                                        • Will I have to do anything with my email address used for this site?
                                                                        • What impact if any is there for users that have been banned? Are their bans via ip/email/etc…? (this one is more of a curiosity as to bans over the migration and if there is any persistence for any bans)

                                                                        Thanks for any clarifications in advance, I didn’t see anything relating to the above which I presume means “situation normal, nothing to see here” but want to be sure.

                                                                        1. 6

                                                                          Nothing changes for you unless you fear I’m going to sell your email to spammers or try to crack your password hash. You don’t need to do anything.

                                                                          Bans are not being rescinded. They’re against an account and (to my knowledge) so far no one has attempted to get a new account to evade a ban.

                                                                          Happy to be really really explicit about this. I want the migration and administration change to be as seamless as possible.

                                                                          1. 6

                                                                            Don’t forget 2FA credentials, for users who have them set up. (I haven’t got around to it yet, despite advocating for the feature, so I’m not even sure if they’re TOTP or U2F…)

                                                                            1. 4

                                                                              These are great questions. You will not have to update your password, username, or email address. If you have an account now it will be migrated as-is. This also means banned users will remain banned users.

                                                                              While the website is moving to new hardware, the software and database will be wholly migrated, save the exceptions explained in this post–the most significant being the deletion of all private messages. As a reader, commentor, or link submitter you won’t notice any change–the migration is largely focused on ‘plumbing.’

                                                                            1. 5

                                                                              Ubuntu breaking itself with updates while still being less secure than openbsd is what made me switch to Openbsd stable on my laptop. Everything just works, even months later, and was surprisingly easy to setup (Openbsd was a breeze for me compared to arch linux).

                                                                              1. 1

                                                                                Going from ubuntu to openbsd! What a jump haha How was the transition? I always imagine it very hard to make.

                                                                                1. 5

                                                                                  I have been all over the place … something like ubuntu -> arch -> fedora -> debian -> ubuntu -> debian -> ubuntu -> openbsd.

                                                                                  I just practised installing openBSD once in a VM to make sure I could get i3 working and after that there was no problem. The older I get, the more I appreciate things that don’t change under your feet.

                                                                                  1. 5

                                                                                    The older I get, the more I appreciate things that don’t change under your feet.

                                                                                    So, there’s this OS called “Debian”… :-)

                                                                                    1. 9

                                                                                      I tried updating a Debian stable machine that I had not touched for six months. it blew up in my face. I’ve never had that happen with OpenBSD.

                                                                                      ever

                                                                                      1. 3

                                                                                        As a counterpoint, I’ve had Debian machines that have gone through 10+ years of upgrades without problem. For example, i’m currently in the process of retiring a VPS that was first installed in 2005 (it’s only being retired as it’s still running a 32-bit userland, has become too much of a snowflake and needs to be rebuilt using configuration management tools).

                                                                                        That said, I’ve never had a problem with the OpenBSD upgrade procedure either :)

                                                                                        1. 1

                                                                                          Agreed, debian stable is ok if you stick in the same stable version. Upgrading between stable releases can be… problematic.

                                                                                          With openbsd its mostly just a matter of reading release notes to see what config files need to be looked at. I’ve never had a linux distro be as straight forward as openbsd in this regard. And that is why it runs all my routing duties.

                                                                                          1. 1

                                                                                            Regular security or point release upgrades never break, so I imagine you’re talking of an upgrade to a new major release. Do you remember which version you’ve tried to upgrade to and what went wrong exactly?

                                                                                  1. 9

                                                                                    Neat, also interesting to see people using ghc on alpine linux. I ported it originally so i could get static pandoc binaries. Glad its useful.

                                                                                    1. 2

                                                                                      I’ve been programming haskell on alpine for about a week now, havent had an issue yet.

                                                                                      1. 1

                                                                                        Thanks for your work on this. We’d been using alpine for deploying ruby stuff for a while, but your work has allowed us to begin moving towards deploying haskell too.

                                                                                      1. 2

                                                                                        Looks like it’s more useful as a test bed for helping Clang/LLVM catch up with GCC.

                                                                                        1. 7

                                                                                          It depends on what you mean by “catch up”, my understanding is that many of the issues are nonstandard extensions, not compliance.

                                                                                          1. 8

                                                                                            Yep its primarily the use of “gccisms” like variable length arrays not at the end of a struct. (well not VLA’s exactly but whatever, its odd to me). Which I consider a misfeature in gcc, and bad to use in the kernel anyway. Why would you write c like this anyway?

                                                                                            int fn(int len) {
                                                                                              struct {
                                                                                                 int evilincarnate[len];
                                                                                                 int value;
                                                                                              } thing;
                                                                                              return thing.value;
                                                                                            }