1. 3

    This reminds me of Corkscrew, a tool for tunneling ssh through http/s proxies. I needed to use this in a similar environment where the only egress allowed was through an http proxy.

    https://github.com/bryanpkc/corkscrew/

    1. 1

      I picked up a Protectli FW4B for use on my home network, which includes VPN use. They seem to be a lesser known brand but it’s been great for me. Ships with no OS and I installed pfSense on it.

      1. 1

        Looks like a nice product but it’s out of my budget.

      1. 8

        While Dwarf Fortress looks and plays like an old game, its resource footprint is far from it. Get your fortress’s population up to around 70 and fps drops are unavoidable on even the newest and fastest processors. Note that in Dwarf Fortress fps is actually the rate at which the game’s logical ticks are performed, not graphical updates.

        Many mods have been created to cope with this problem, such as fastdwarf or teledwarf which causes your dwarves to walk faster or teleport to their destinations instead of computing a path and walking down it.

        1. 4

          The universe of Dwarf Fortress attempts to self-correct localised time dilation by throwing goblin hordes and mythical creatures at the problem.

          Fastdwarf and teledwarf seem like interesting workarounds. They effectively add ‘frameskip’ to the entity logic, if you want to think of it that way.

          1. 3

            Thank you for the explanation, I had no idea why this might be challenging.

          1. 1

            I used to keep documentation in Markdown files alongside related source and tools - this was an operations type role. It was nice way to keep the docs easily locatable (using foo_tool.py? there’s a foo_too.md right next to it) as well as the obvious benefits that git’s history provides.

            I’ve since moved to a different company that doesn’t have a standardized practice, and as a result docs have ended up in various wiki tools and microsites. It’s really awful.

            1. 6

              I’ve been learning C for the past couple months by writing a web-based IRC client. The web facing bits are python, but some of the underlying services (the IRC connection itself and message cache) are. The stack is C + ZeroMQ + Cap’n Proto + Python - It’s been a blast!

              1. 3

                Napkin calculation: it’s 624,197,820,790 bytes, spread across 4,073,468 files. That’s about 153,235 bytes per file. Assuming the average length of a line is around ~40 characters, and knowing that Windows takes two bytes to store each character (because they use UTF-16 for legacy reasons) and it’s a bit extra in needing two characters to represent a line ending, we can assume about 100 bytes per line. That would mean the average code file has ~1,532 lines, and the whole monorepo has about 6,241,978,207 lines (that’s 6 billion lines of code).

                4 million files is a lot. The linux kernel supports far more architectures and possibly has more drivers built in, and it only has 61,725 files with an average length of ~414 lines. So, I think there’s a chance that there’s something wrong with the way they counted the files, for example, if it’s a git repository that they counted files in .git/ as well? Not quite sure.

                1. 14

                  [this includes] source code, test files, build tools, everything The build tools and test fixtures probably make up a sizable portion, so that would definitely skew the total

                  1. 1

                    Does the repo include graphical assets too?

                  2. 6

                    more than a half million folders containing the code for every component making up the OS workstation and server products and all their editions, tools, and associated developement kits

                    You cannot compare their entire stack with another family’s vanilla kernel alone. Pull in all of GNU userland, GCC, an IDE, some browser, and so on… maybe that will help the numbers starting to compare.

                  1. 1

                    Overlaps quite strongly with what systemd does….

                    What would you say are the pros and cons of Orderly vs systemd?

                    1. 2

                      I use runit instead of systemd at backupbox.io for faster boot times and smaller virtual machine images. The big problem is runit doesn’t support ordered setup/teardown without writing ad hoc scripts.

                      Orderly is one layer I am going to use in conjunction with runit to provide these things in a more modular way than systemd.*

                      Another strength of orderly is that it is quite good for development environments when you are coding multiple servers and want to restart/stop them in groups while testing.*

                      *In theory, but I literally just wrote it, so we will see what needs to change.

                      1. 1

                        backupbox.io

                        Hmm. Took a brief poke around your site…. vaguely reminds me of my favourite backup on to usb pen drive tool… http://zbackup.org/

                        I presume you’ve met up with zbackup?

                        1. 1

                          Haven’t tried zbackup, but I have played with quite a few tools. Will need to look into it, thank you.

                        2. 1

                          Ok, so if I understand you correctly you’re adding a bunch of systemd like functionality for alternative init systems.

                          faster boot times

                          Interesting.

                          My experience has been systemd has done a lot to speed up boot times because it parallelizes everything that can be and hence runs a lot faster on multicores.

                          Are you using single core machines?

                          when you are coding multiple servers and want to restart/stop them in groups while testing.*

                          Systemd is quite handed for doing exactly that, plus can force’em to “play nice” about using resources. https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/

                          However cgroups are not a systemd thing but a kernel thing which you could probably also use.

                          1. 6

                            My experience has been systemd has done a lot to speed up boot times because it parallelizes everything that can be and hence runs a lot faster on multicores.

                            Well, I am using a custom linux image inside single core VM’s. Each vm only runs ssh, a fuse filesystem mount, audit daemon and getty. The extra work involved in initializing systemd is actually less than just starting those directly.

                            Actually, after removing systemd I was able to get the boot down from ~5 seconds to 0.5 seconds. Though it is difficult to measure precisely, because I also switched away from a ‘buildroot’ based filesystem.

                            The main problem is some ordering requirements between the audit daemon, ssh and the filesystem, especially if the file system fuse daemon crashes.

                            However cgroups are not a systemd thing but a kernel thing which you could probably also use.

                            Yeah, I just use them directly from runit.

                            1. 1

                              My experience has been systemd has done a lot to speed up boot times because it parallelizes everything that can be and hence runs a lot faster on multicores.

                              Compared to what? Init? To be clear, systemd vs $other has been beaten to death and I’m not trying to rehash it. Upstart has parallelized job starts since its inception so I’m curious what systemd is doing differently.

                              1. 2

                                Compared to whatever ubuntu had on the LTS prior to systemd… I forget exactly what.

                                I just noticed, wow… this is booting humanly noticeably faster. I suspect it’s a combo of parallelizing and accurately tracking dependencies and starting them as/when needed, but I never rolled forward and back and measured to track down what exactly.

                                If I remember correctly the sneaky ureadahead trick pre-dated systemd so I don’t think it was that.

                                Although we also shifted a single core openembedded system over to systemd from init.d and they seem to boot faster as well, probably because some of the I/O could happen while the CPU was doing useful stuff.

                        1. 6

                          I loved the look of the recently-posted Endlessh project so much that I’m going to integrate it with fail2ban and iptables to redirect banned attackers into the tarpit.

                          1. 1

                            Apparently we have similar ideas. I just threw an SSH tarpit together earlier and was looking at how to integrate it.

                          1. 15

                            The easiest way to solve the problem is to increase the size of your gp2 volume.

                            While this is true, there’s another way that will give you even more iops for less (or zero!) additional cost. Many EC2 instances come with a local SSD. For example the i3.large instance type - which is fairly small, just 2 cores and 16GB ram - includes a 475GB NVMe SSD. You can perform tens of thousands of iops on this disk easily.

                            Obviously, since this SSD is local it’s contents are lost if your instance is stopped for any reason, like hardware failure.

                            1. 3

                              Also worth noting there’s more options like this since the introduction of the new generation instances with “d” designators, like c5d and m5d, which have local nvme storage, and might be a good balance between general purpose compute while still having local storage. The i-type hosts are “I/O optimised” which solves the storage problem but might leave you without much for the actual build tasks.

                              1. 2

                                Thanks for the idea, noted in the article.

                            1. 6

                              This is definitely a problem with custom allocators, but using a custom allocator and also be leveraged to detect memory bugs. AFL does this:

                              Libdislocator is an abusive allocator that can be loaded as a drop-in replacement for the libc implementation via LD_PRELOAD or AFL_LD_PRELOAD.

                              It’s in no way AFL-specific, but it should play pretty well with the fuzzer. Basically, when loaded alongside with any dynamically linked binary (source not needed, but static binaries won’t work), it behaves in a way that maximizes the odds of triggering heap corruption issues in the targeted code:

                              1. It allocates all buffers so that they are immediately adjacent to a subsequent PROT_NONE page, causing most off-by-one reads and writes to immediately segfault,
                              1. It adds a canary immediately below the allocated buffer, to catch writes to negative offsets (won’t catch reads, though),
                              1. It sets the memory returned by malloc() to garbage values, improving the odds of crashing when the target accesses uninitialized data,
                              1. It sets freed memory to PROT_NONE and does not actually reuse it, causing most use-after-free bugs to segfault right away,
                              1. It forces all realloc() calls to return a new address - and sets PROT_NONE on the original block. This catches use-after-realloc bugs,
                              1. It checks for calloc() overflows and can cause soft or hard failures of alloc requests past a configurable memory limit (AFL_LD_LIMIT_MB, AFL_LD_HARD_LIMIT).

                              https://groups.google.com/forum/#!topic/afl-users/RW4RF6x9aBc

                              1. 4

                                This idea seems like a different way of expressing cyclomatic complexity.

                                1. 1
                                  memset(ptr, sizeof(*ptr), 0);
                                  

                                  Shouldn’t this (the first snippet) be this instead?

                                  memset(ptr, 0, sizeof(*ptr));
                                  
                                  1. 6

                                    (that’s the point)

                                    1. 4

                                      I believe that is indeed the basis of the motivating example.

                                      1. 1

                                        One of the *sans, maybe asan with replace_intrin ? will catch this error.

                                      1. 3

                                        I’m sinking time into some personal projects to improve “developer life” in my vm lab - a local apt/rpm/pypi mirror and some automation around that.

                                        1. 1

                                          I’d love a container for proxy/mirrors of each. I have never had great results with apt mirrors. I set them up in anger, but then they go sour.

                                        1. 8

                                          I just looked at the docs and I liked a lot of the ideas (the interface (ish) class based views, for instance), but the idea of passing mutable request and response objects and not returning anything really rubes me the wrong way, for some reason I can’t properly explain yet.

                                          I also like how it supports async, though, and has a built-in solution to production deploy as well.

                                          1. 4

                                            A lot of HTTP framework start with the end-point taking a request as input and return the response. This works beautifully until streaming responses need to be used.

                                            In reality a HTTP request looks more like this (read headers) -> (maybe read request body) -> (send headers) -> (maybe write response body). If the response object is returned it’s hard to control what is going into the body. It could be a long-poll for example. Or streaming from another data source and with a cancel event. There are a lot of scenarios here.

                                            1. 2

                                              I’m generalising, but fewer of your endpoints are going to need streaming so why would you hurt the common case in favour of an uncommon one?

                                              In molten I’ve implemented streaming via generators, which is exactly the opposite approach: all endpoints are functions of a request that return a response and streaming endpoints are just a special case over response types.

                                              1. 1

                                                If it’s a “HTTP Service Framework” I expect it to handle more complicated things than render HTML pages.

                                              2. 1

                                                Hmm, I see. Makes sense. So, probably the request object is not fully loaded, then? As in, the body hasn’t been read, it will be read only on demand?

                                                Anyway, now I’m curious to see how this could play with async/await =)

                                                1. 1

                                                  I’m curious too, I am mainly reflecting my experience of using Go

                                                2. 1

                                                  This works beautifully until streaming responses need to be used.

                                                  It can work well even then. You just need to have a mechanism that allows attaching a generator function as the response body (framework checks response body for callable perhaps?).

                                                  psuedocode-ish:

                                                  @api.route("/hodor")
                                                  def get_hodor(req):
                                                      def somefunc():
                                                        for x in range(1,100):
                                                          yield "hodor"
                                                          sleep(1)
                                                  
                                                      resp = make_response()
                                                      resp.body = somefunc 
                                                      return resp
                                                  
                                                    1. 1

                                                      Sure it can work but it becomes quickly awkward when mixing multiple streams together in my opinion. Now half of the handler logic is running in another function. I prefer to handle all of the logic in the handler body, this also allows for better destructor (close) handling.

                                                      Let’s say I want to implement a Server-Sent Event endpoint. It needs to handle the following cases:

                                                      • subscribe to a redis channel, if another request is already using that channel, share the subscription
                                                      • if the redis connection gets disconnected, re-connect without losing the client
                                                      • unsubscribe if all clients are gone
                                                      • send keep-alive if no message has been sent for 60 seconds
                                                1. 9

                                                  I have been a mac user since the beige-toasters. There is no chance I would buy a computer with “DRM” crap preventing me from repairing it.

                                                  1. 2

                                                    Didn’t the beige toasters require an Apple-licensed tool to open or was that an urban legend spread in the besieged Amiga community?

                                                    1. 5

                                                      It’s correct, they used Torx screws for the case: teardown.

                                                      1. 3

                                                        I just bought a torx screwdriver from Lowe’s last week. I’d hardly consider that to be locking users out of their hardware.

                                                        1. 3

                                                          They were hard to come by in 1984.

                                                          1. 3

                                                            Yet infinitely easier to bypass than software DRM.

                                                            1. 2

                                                              I started repairing Macs in 89 and had several torx screwdrivers so by then, they were easy to get.

                                                              1. 1

                                                                Good point.

                                                            2. 3

                                                              Tri-Point and Pentalobe are two common currently examples of this.

                                                              In highschool/college I worked at an authorized repair shop for Apple. Torx are great honestly. They are easier to work with and less likely to strip than Philips. Tripoint & Pentalobe are a nightmare to work with, which is why they are often not used internally.

                                                              1. 3

                                                                Torx is just a better type of allen bolt

                                                                1. 3

                                                                  Indeed, there’s valid reasons to use Torx.

                                                          1. 0

                                                            So far I’ve only found one solution that is actually robust. Which is to manually check that the value is not nil before actually using it.

                                                            This seems reasonable to me. If anything, I’d consider knowing how and when to use this kind of check a part of language competency knowledge as it is how Go was designed.

                                                            1. 9

                                                              We expect people to be competent enough to not crash their cars, but we still put seatbelts in.

                                                              That’s perhaps a bad analogy, because most people would say that there are scenarios where you being involved in a car crash wasn’t your fault. (My former driver’s ed teacher would disagree, but that’s another post.) However, the point remains that mistakes happen, and can remain undiscovered for a disturbingly long period of time. Putting it all down to competence is counter to what we’ve learned about what happens with software projects, whether we want it to happen or not.

                                                              1. 9

                                                                I wish more languages had patterns. Haskell example:

                                                                data Named = Named {Name :: Text} deriving Show
                                                                
                                                                greeting :: Maybe Named -> Text
                                                                greeting (Just thing) = "Hello " + (Name thing)
                                                                greeting _ = ""
                                                                

                                                                You still have to implement each pattern, but it’s so much easier, especially since the compiler will warn you when you miss one.

                                                                1. 3

                                                                  Swift does this well with Optionals

                                                                  1. 5

                                                                    You can even use an optional type in C++. It’s been a part of the Boost library for a while and was added to the language itself in C++17.

                                                                    1. 4

                                                                      You can do anything in C++ but most libraries and people don’t. The point is to make these features integral.

                                                                      1. 1

                                                                        It’s in the standard library now so I think it’s integral.

                                                                        1. 4

                                                                          If it’s not returned as a rule and not as an exception throughout the standard library it doesn’t matter though. C++, both the stdlib and the wider ecosystem, rely primarily on error handling outside of the type-system, as do many languages with even more integrated Maybe types

                                                                    2. 2

                                                                      Yep. Swift has nil, and by default no type can hold a nil. You have to annotate them with ? (or ! if you just don’t care, see below).

                                                                      var x: Int = nil // error
                                                                      var x: Int? = nil // ok
                                                                      

                                                                      It’s unwrapped with either if let or guard let

                                                                      if let unwrapped_x = x {
                                                                          print("x is \(x)") 
                                                                      } else {
                                                                          print("x was nil")
                                                                      }
                                                                      
                                                                      guard let unwrapped_x = x else {
                                                                          print("x was nil")
                                                                          return
                                                                      }
                                                                      

                                                                      Guard expects that you leave the surrounding block if the check fails.

                                                                      You can also force the unwraps with !.

                                                                      let x_str = "3"
                                                                      let x = Int(x_str)! // would crash at run-time if the conversion wouldn't succeed
                                                                      

                                                                      Then there’s implicit unwraps, which are pretty much like Java objects in the sense that if the object is nil when you try to use it, you get a run-time crash.

                                                                      let x: Int! = nil
                                                                      
                                                                  2. 7

                                                                    Hey, I’m the author of the post. And indeed that does work, which is why I’m doing that currently. However, like I try to explain further in the post this has quite some downsides. The main one is that it can be easily forgotten. The worst part of which is that if you did forget, you will likely find out only by a runtime panic. Which if you have some bad luck will occur in production. The point I try to make is that it would be nice to have this be a compile time failure.

                                                                    1. 1

                                                                      Sure, and that point came across. I think you’d agree that language shortcomings - and certainly this one - are generally excused (by the language itself) by what I mentioned?

                                                                  1. 2

                                                                    I own two of the motherboards mentioned (the second ASRock Rack one - actually, the variant with a C2750 CPU) and have suffered 100% failure rate. More, actually depending how you want to factor in the one that was DOA. FWIW, Intel had a defect in their C2000 series CPUs and my boards were manufactured during this time, but I don’t actually know what the failure cause was.

                                                                    ASRock has also failed to keep the Java-based IPKVM client up to date and signed in a way where Java is able to run it. I forget the specifics but for some time it was completely unusable due to applet signing issues.

                                                                    1. 5

                                                                      Funny to mention nested functions. GCC does allow them, and people passionately hate their existence because they require making the stack executable which is highly detrimental for security.

                                                                      1. 1

                                                                        Doesn’t this mention of feature vs security loop back to the author’s original point?

                                                                      1. 20

                                                                        I was in the middle of a long-winded comment, but I don’t have the time to point out all the problems. Quite simply, the article doesn’t make any sense. There is a term for this: it’s not even wrong.

                                                                        In order to be “wrong”, arguments have to be based in some sort of reality, but just reach the incorrect conclusion. None of the author’s arguments make any sense, so the article cannot be judged as being “right” or “wrong”, because it’s not based in reality.

                                                                        1. 7

                                                                          I somewhat agree with this but maybe not for the same reasons. Many of the points ignore obvious reasons that are still valid today. Such as:

                                                                          Regarding print statements:

                                                                          Oh, wait a minute, this is still the only way we can get anything done in the world of programming!

                                                                          But what about debuggers? Tcpdump-like tools? These are much better than debugging via printing.

                                                                          Our ancestors bequeathed to us cd, ps, mkdir, chown, grep, awk, fsck, nroff, malloc, stdio, creat, ioctl, errno, EBADF, ESRCH, ENXIO, SIGSEGV, et al.

                                                                          What technical field is not besieged by jargon with meaning not obvious to the unfamiliar? Likewise, all of these can be alised to more obvious terms, anything you desire, for only the cost of character space in your code or configs. So why don’t we see more of this? The author suggests it it because keyboards were physically painful to type on in the past.

                                                                          You know what hurts? RSI. Silly point aside, the time savings are absolutely still relevant today.

                                                                          The big problem with text-as-ASCII (or Unicode, or whatever) is that it only has structure for the humans who read it, not the computer.

                                                                          Is this really a problem? Haven’t we solved inefficiencies relating to this with various types of preprocessing such as, well, code preprocessing? Not to mention compilation? Code aside, is the minimal overhead of parsing a config file when your daemon happens to restart that big? Maybe I’m being pedantic but aren’t scripting languages generally “slow” to execute yet quick to write, and the inverse for binary programs?

                                                                          I feel that the author ignores obvious reasoning (obvious in that the alternative is, while not sugar-coated, certainly more modern than 70s Unix. This seems to fit every point in this article and, at the risk of ad hominem, paints a picture of a naive computer enthusiast.

                                                                        1. 15

                                                                          I’m using 1Password with local sync over the built-in web server. 1Passwords also supports syncing via Dropbox, iCloud and, most recently, 1Passwords’s own servers. I want nothing of that but it means that I can’t use 1Password on Linux. What’s great about 1Password is that it is highly polished - using it adds very little friction. I understand that I could manage all passwords encrypted in Git but the integration would never be as good and there is a lot of risk that this would somehow not be as secure as it sounds.

                                                                          1. 5

                                                                            I recently switched to Enpass, which is a conceptual clone of 1Password. Reason for switching was Linux support.

                                                                            1. 7

                                                                              This is a closed-source app that has not yet received a lot of scrutiny. Using it for truly sensitive information requires quite a bit of trust. They claim to use sqlite with encryption – which I would trust but of course, there is a lot of code around it that would have to be trusted as well.

                                                                              1. 2

                                                                                The first thing it tried to do when I ran it was reach out to Google Analytics. I said enough of that, and promptly uninstalled it.

                                                                              2. 2

                                                                                1Password (at least the hosted version) has linux support via both 1password-x and 1password-cli. I quite enjoy the CLI and generally find that it has a better user experience than LastPass.