1. 3

    As a company, we are looking to move from being tightly coupled to Amazon AWS to a more agnostic approach where we can deploy our platform to different cloud providers (this is not a technical requirement at first, but needed by the business).

    The obvious approach for achieving such outcome is to go with Kubernetes; for the past two weeks, I have been diving in the documentation of various tools including Kubernetes (+ Kustomise), Helm, ArgoCD, Ingresses (Istio, Nginx), etc. etc. I have found the amount of information to be overwhelming. We are pretty happy with our current pipeline which deploys on three separate environments (Staging/QA/Production) in Amazon ECS; the move to Kubernetes and GitOps already sound like a big endeavour, with a lot of decisions to be made on tooling and pipelines, and that’s frankly frightening.

    1. 1

      My company uses kubernetes and has a similar business requirement to be cloud agnostic. We use all of the hosted clusters, but there is still a crazy amount of complexity going on. Despite a dedicated team and some deep experience, we run into issues fairly often, especially when trying to spin up new services. Once a service is set up its fairly robust, but getting new things deployed is a massive pain.

      All of this is to say unless you really need it, I would try to avoid the complexity. I primarily work on the backend, so I don’t interact with the devops work super often, but every time I do its just layers upon layers of abstractions. Even the experts at our company have trouble.

      You can be cloud agnostic without k8s + co., and there are alternatives like nomad that I have heard good things about. But yeah, there is a crazy amount to learn, and even once you have things running there is a crazy amount to debug. Troubleshooting also becomes 2x harder.

      1. 1

        Thanks for your comment. It confirms my concerns regarding the complexity of a solution like Kubernetes for a small sized company. My main concern at this stage is how to get started since the most basic setup seems to involve many different tools, and supporting multiple environments like we do today involve adding even more complexity.

        I have also heard very good feedback on Nomad, but we need to think of future recruitments. There is no doubt that Kubernetes has won the container orchestration, and the number of potential knowledgeable / expert candidates would be significantly higher with Kubernetes vs Nomad (even if the latter is more suitable for our needs).

        1. 1

          You’re right, there are numerous tools. I think for getting started you can forgo things like helm and flux, and stick with raw k8s manifests. Helm is a pretty attrocious templating solution in my opinion, and we have run into a number of bugs in what should be a really simple program, so I’d argue you don’t ever need it. Even with just k8s manifests there is a lot to learn, but at least its just one tool rather than 5 or 6.

          You will have to do what is best for your situation, so definitely take everything with a grain of salt. One argument I would have for recruitments is that usually the popular technology has a bigger pool of talent, but the average quality of that talent is worse off. Personally I think startups should use niche but powerful tech rather than popular tech, since the applicant pool will self filter. Hiring takes a long time and a bad hire is 2x worse than missing out on a good hire at a small size.

          Just food for thought! Wish you all the best in your endeavors.

          1. 1

            I agree with your comment on niche technologies unlocking a pool of experts; the counterpart to this argument is that these people may cost a lot of money to acquire and retain, since they will be in demand. Having a large pool of candidates means that you, indeed, you will have more junior candidates, but it’s also an opportunity for people to grow in your company and for building a diverse team that can grow with your organisation.

            That being said, I will have definitely have a look and build a small POC with it.

      2. 1

        Author here. I wrote this other piece about this specific choice/challenge: https://zwischenzugs.com/2019/03/25/aws-vs-k8s-is-the-new-windows-vs-linux/

        1. 1

          Interesting read, thank you very much. The infographic at the end describes my feeling as a newcomer in the Kubernetes world; it feels that the best practices are not yet fully established so the ecosystem is super diverse and full of products of varying quality.

          PS: I am one of those people who were playing Linux in its early days! I remember (not very fondly) the kernel panics following plugging an USB device (especially DSL modems, Linux loved those!)

        2. 1

          Disclaimer: I work for Google on what I would call a k8s “adjacent” product where we are heavily invested in the k8s ecosystem, but not part of it.

          I think the k8s ecosystem is pretty Wild West as there is so much, and it’s impossible to figure out which tool is best-of-class. I think this is a common situation for “new” technologies. k8s is basically a cloud low-level operating system at this point, and there needs to be layers on top. Some good abstractions for some use cases do exist now, e.g. GCP Cloud Run, but if you’re determined on being cloud agnostic, it’s going to be a hard road until each cloud has comparable products. I don’t spend time in AWS/Azure land as I have my own job to do, but I do not think they have a Cloud Run-esque solution yet.

          Do you have to be cloud agnostic? If it’s for super high 99.999% reliability then yeah, that’s your only realistic option. If it’s for having an escape ramp if you want to switch to a different provider for some reason, then I think you could get away with just building your Docker images, and having scaffolding around the single provider you’re invested in. Retooling to a new provider wouldn’t be simple, but it would be an order months, not order years, issue, in my estimation.

          But I’ve never done this so don’t take my word for it.

        1. 4

          Does anyone here have experience with Flux? I’m curious if this mitigates the problems highlighted here. We’ve felt these pains, and we’re currently investigating Flux as a potential remedy.

          1. 1

            I worked indirectly with Flux, it’s a pretty decent system to build on. We chose it at my last job because we were managing a large amount of separate, fully-isolated clusters, all running applications on a common Rails platform. These had a lot of similarities in terms of how they’re deployed, but also had enough differences wherein the infrastructure would need to change on a per-client basis. Flux helped us codify those differences and made it so applying changes to all clusters (such as k8s upgrades, security fixes, or general non-invasive improvements) could be done easily and all at once. For our purposes, I have no complaints…it was a great little tool in our toolbelt.

            The only real downside is that it’s relatively simple, and you need to build your own tooling around it in order to get things done. No “point-and-click” stuff, but our infra team was full of competent developers who knew their way around Git, Kubernetes, and AWS. That said, I think the GitOps Toolkit that Flux is based on has some promise if you want to build that kind of tooling and user interface(s) out for your clients.

            1. 1

              Author here. Flux is part of one of the choices - how to reconcile the declarative code in source with the target environment.

              In that space you have the choice of scripting kubernetes updates in your CICD tool, or using an ArgoCD/Flux operator/agent to reconcile the state. ArgoCD is winning out over Flux where we work, mostly because it historically handled multi-tenant use cases better, and has a useful GUI. There is convergence in this space as their makers are collaborating on an ‘engine’ that both will rest on in future.

            1. 1

              If you enjoyed Turn This Ship Around, then you’ll “love” the Taylorist take The Goal and it’s modern agile retelling The Phoenix Project.

              1. 2

                Yeah, I loved the Goal. Didn’t think much of The Phoenix Project, having already read The Goal.

                1. 3

                  Sorry for being obtuse, but I’m confused by the quotes around “love” and this reply. Are these worth reading (either for good reasons, or for anti-exanple reasons?)

                  1. 3

                    I think those were not scare quotes, but quotes for emphasis. See also http://www.unnecessaryquotes.com/

                2. 1

                  Those are fiction though, this book is not, so this book seems more credible.

                1. 1

                  This isn’t convincing to read after

                  ignore the noise, and go to the signal

                  Not trying to be rude, but why should we believe the author’s summaries are better than those that confused him? I will take a look at the book though.

                  1. 2

                    You shouldn’t!

                  1. 3

                    This is a great doc, even though I was a bit upset by some generalizations about security folks :P

                    Now I wonder if we should rework our “what to bring and how to prepare for a security review” doc. So far we only got a template to fill out… So much for setting expectations 😅

                    1. 3

                      Thanks, and fair point. I am looking at things there very much from a ‘why are they saying no?’ approach, but I hope it helps bridge the gap in the end.

                      1. 1

                        Yeah, I totally get it. I just wasn’t aware of how scary that gap can be. Oops.

                    1. 53

                      All I wanted to do was help a company ship its domain-specific software faster so I looked into Docker.

                      Years later I’m working in infrastructure for banks trying to set up a microservices platform.

                      I can’t remember what business logic looks like anymore.

                      1. 0

                        Lucky.

                      1. 19

                        @zwischenzugs A script on your website (specifically, the fifth one in the <body>) is producing spammy and misleading pop-ups like the one at this link: http://www.creep.world/static/lps/u6Fs3j2D/ It also looks like your site is letting ads load arbitrary iframes.

                        1. 3

                          The site is owned and run by wordpress. I’ve already reported this to them, will try again.

                        1. 2

                          I thought podman used buildah for building, so I’m surprised there was a performance difference.

                          1. 1

                            Yeah, that’s right - I’m trying to figure out what my problem was now…

                          1. 1

                            Nice one, thank you! Minor grammatical issue “…that the readline library give you in…” give should be gives.

                            1. 2

                              Aside; I have this in my .zshrc;

                              # MOTD - reminder info on readline/emacs/zle binds
                              function echo_color() {
                                printf "\033[0;90m$1\033[0m\n"
                              }
                              echo_color "  c-b  Move backward            c-f  Move forward"
                              echo_color "  c-p  Move previous (up)       c-n  Move next (down)"
                              echo_color "  c-a  Jump to line beginning   c-e  Jump to line end"
                              echo_color "  c-h  Delete backward          c-d  Delete forward"
                              echo_color "  c-u  Delete entire line       c-k  Delete to end of line"
                              
                              1. 1

                                Thanks, updated.

                              1. 6

                                Another good one is "\C-x\C-e": edit-and-execute-command.

                                Just found out about this one recently from a coworker. It opens up $EDITOR for writing commands. I’ve been using it to edit stuff with a bunch of arguments so I can spread them across multiple lines and rearrange things more easily.

                                1. 2

                                  This is the greatest think I’ve learned in … a while. I don’t write a lot of one-off scripts and tend to just have a bunch of commands joined on one line with semicolons. This will help a bunch.

                                  Thanks, so much!

                                  1. 0

                                    Nice!

                                  1. 6

                                    My message of choice I’d display at random (1/1000 probability):

                                    If you are reading this, you have been in a coma for almost 20 years because of a car accident. We are trying a new technique. We do not know where this message will end up in your dream, but we hope we are getting through. Please wake up.

                                    1. 2

                                      That. Is. Genius.

                                      1. 2

                                        That is also the main plot device in ******* (no spoilers), a novel by Philip K. Dick.

                                    1. 4

                                      heh, I once made something similar for python: https://github.com/adtac/exterminate/

                                      fun stuff

                                      1. 1

                                        https://github.com/adtac/exterminate/

                                        I love it. Though far too ‘nice’ to give it a name like ‘exterminate’…

                                      1. 7

                                        I can definitely see Joe’s point, and far from me to claim that I know one thousandth of what he knows. HOWEVER. Every time he says stuff like that it seems to be in reference to telephony/network/low level infrastructure related code, and I think this is an unfair comparison. The kind of code that he wrote pretty much his whole life has significantly different requirements from web apps or SaaS backends, or mobile apps.

                                        1. 4

                                          I think if you look at what he said, he makes that point himself:

                                          ’Code I wrote 5 years ago with external dependencies often fails. ’

                                          I presume that code is more ‘modern’ in context than the low-level code. His point isn’t that he did it better, but that dependencies are a risk to running software for a long time, and if that’s your aim, try and avoid them whatever your domain.

                                          A lot of web app or SaaS code simply won’t last, so I guess it doesn’t matter so much for them.

                                        1. 32

                                          This is a pretty spot-on example of why I stopped using Twitter; the first reply is someone telling Joe Armstrong that he’s never coded beyond hello world.

                                          1. 9

                                            What our Twitter reply is saying is of course Joe Armstrong is past Hello World, he’s saying that it’s impossible to be without dependencies and this is just “old guy” advice. Joe Armstrong is choosing to have his cake and eat it too chastising dependencies while the real world revolves around them.

                                            I think this example is anecdotal to Mr. Armstrong and that as a practice I’d rather people use their judgement. The argument stinks to me of handwaving and of the old class of developers who’d rather not learn new stuff.

                                            If you’re Google you can vert. up all you like. Don’t act like any old place but large ones can put up with an engineer trying to argue this in reality.

                                            1. 6

                                              My guess is that most languages wrap system IO routines, which means that even a “hello world” isn’t truly dependency free.

                                              Now, if Chuck Moore was making this statement, I’d fully believe it. He made a career out of building things to the exact spec necessary—the only dependencies were often encoded in the CPU (and other hardware)’s silicon.

                                              1. 6

                                                That’s not all! In his later years, Chuck Moore build an entire VLSI CAD system in Forth so he could design his own CPUs. Some of these were even fabbed. There might be a few left for sale.

                                                Sounds extreme, but it’s true. That’s why we call him “Chuck Mooris”.

                                                1. 5

                                                  He’s the kind of programmer that “full stack” should refer to. An elite level where you can do everything from apps to full-custom silicon.

                                                  1. 4

                                                    @technomancy and I have a t-shirt that’s been in the works for a while. It’s totally my fault that it isn’t currently available (as it’s been done for ages), but it celebrates Chuck in a “witty” way. This might just be the nudge that gets me to follow through with the rest. ;)

                                              2. 15

                                                I’m glad that person made that comment, actually, even if it was a bit rude.

                                                When I first read Joe Armstrong’s statement “Of course - I try to write code with zero dependencies - code I wrote 25 years ago with zero dependencies still works today.”, my first thought was something along the lines of “hm, yeah, that’s sensible advice from Joe Armstrong, and he’s a well-known name he probably knows what he’s talking about”.

                                                Then some random guy told him publicly that he was wrong in an un-nuanced way.

                                                And I thought to myself, well, hm, that’s a rude way to put it - but yes, actually, “A software system simply cannot be built without components based on other components whether they are soft or hard dependencies.” has some merit to it as well. Maybe Joe Armstrong isn’t being completely literal when he says that’s he’s written telephony code for Ericsson base stations that’s run for 25 years with absolutely zero dependencies other than the compiler. Maybe some dependencies are getting “snuck in” via the compiler, or the hardware drivers for those base stations, or in changed hardware itself, even if the Erlang code running in the VM has been around for 25 years unchanged. Maybe the code he’s talking about that’s remained unchanged is itself a dependency for other code that changes more frequently - certainly there have been plenty of updates of the capabilities of the phone system over the past 25 years, and I doubt that Joe Armstrong’s quarter-century-old code now needs absolutely no additional augmentation now. Maybe telephony is a special domain, and eschewing dependencies is a good fit for solving telephony problems but a bad fit for, say, creating new pieces of software that do things undreamt-of 25 years ago.

                                                And none of this is to say that Armstrong’s points about dependencies increasing fragility and causing code bloat are invalid. He’s 100% right to point out that it’s bad to add 200K of CSS in order to make a button look pretty. But maybe doing that is the least bad of several trade-offs (if you couldn’t use bloated CSS libraries, maybe web design would look terrible, or maybe a bunch of useful websites would just never have been built). My own opinion is that things like 200K CSS files for one button are a local but not global minimum - the entire web is highly path-dependent, and relies on a bunch of inefficient hacks to make it usable as a software distribution platform, but there’s also a huge breaking-backwards-compatibility cost in moving away from the web and towards a better software ecosystem that does the same things as the web but doesn’t encourage 200K CSS libraries. Maybe webassembly is a small step towards climbing out of that local minimum to something globally better.

                                                1. 10

                                                  Should we really need a prompt to think that way, though? Whenever I hear anything I imagine the counter position as a test. It’s a way of getting at the nuance. I like to think that it’s something we should foster as an engineering mindset.

                                                  1. 12

                                                    I recently learned about the principle of charity: https://en.m.wikipedia.org/wiki/Principle_of_charity. Most online discussion and tools could use a lot more of it.

                                                  2. 10

                                                    The argument would be helped if the original responder didn’t descend to ‘ball-licking’ level: https://twitter.com/ethericlights/status/1075531837286555648

                                                    1. 7

                                                      “zero dependencies” but he does have the OTP system to support all this. Not like he’s rewriting the TCP socket logic in every program.

                                                      I think this is an argument for the seaparation of thought process between standard libraries and external dependencies. Standard libraries have very large backwards compat requirements in general, so relying on that is really not an issue. So having a standard library that is vast and covers a lot of non-controversial but tedious stuff (building HTTP requests, parsing JSON if that’s a thing, listening to sockets) can mean that you won’t feel the need to pull in external dependencies for the most part.

                                                      And what he said about vendoring in stuff and ripping out what you don’t need is… pretty good advice for lots of usages! Though libraries get added improvements, if you vendor in the libraries you can just change the stuff up to your specific use cases without much issue.

                                                      But yeah “I write a bunch of programs with no dependencies” kinda precludes a lot of stuff if you were doing something in C (for example). But zero-dep Python/Erlang(OTP) seems really doable in general

                                                      1. 3

                                                        Broadly agree, but zero-dependency C is often pretty doable too IME, for the kind of things where C shines. And even if you do need dependencies, well-known C libraries tend to be more stable than e.g. the nodejs ecosystem.

                                                  1. 9

                                                    This is a good starting point. My list of 6 ways to “level up” would look something like this:

                                                    • NSE scripts and LUA engine - I fully agree on this one. It’s actually crazy the amount you can do with it and I think it’s one of the underrated parts of nmap. I’ve started writing rules on the fly to try and deal with firewalls and other things that get in the way.
                                                    • Timing options - One major thing that I think bites people when starting to get deeper into nmap is “scans taking too long”. The moment I realized that some of the defaults are a little too aggressive was the moment my options started to grow and fix some of my pain points. Reducing retries, reducing timeout to an acceptable amount for the type of network, and changing version detection intensity are all things I suggest when dealing with say a /8 and UDP scans. Also look into timing templates, I haven’t been fully happy with mine yet, but it’s a WIP that might fix some of my pain points.
                                                    • Contextualize - Are you on link local? Use ARP/NDP/etc. Got a firewall being a jerk and telling you all ports are up? Switch to full connect scans (honestly I do this a ton now).
                                                    • OS detection sucks - I almost always don’t do OS detection these days, so many routes and “security devices” change the results that it’s just going to slow you down. The only exception is nmap IPv6 and NDP, I am actually trying to isolate the bug, but I can’t seem to get NDP ICMP to work properly without the “OS Detection”
                                                    • Interacting with the XML/Grep output - this is key to my survival, I don’t think I could conduct a pentest without some heavy parsing. Also this teaches you to be weary about certain unnamed script outputs.
                                                    • Spoofing - This is much less common for the vast majority of users, but I’ve run into a few situations recently where I could ARP poison and then use nmap spoofing options to trick clients into accepting my UDP packets. It’s tricky and I still want to re-lab the stuff to write it up, but there aren’t many tools that do it as easily.
                                                    1. 2

                                                      You should write that up. I’d read it. I’m just someone trying to improve my networking skills in my own time, rather than a pentester (hence aiming it at casual users).

                                                      1. 8

                                                        I’m trying to build this up:

                                                        https://therunbooks.com/

                                                        barely anything there at the moment, but this might give a flavour:

                                                        https://therunbooks.com/doku.php?id=networking:dns-lookup-failure

                                                        Looking for help :)

                                                        1. 1

                                                          This is a cool idea. I’ll try and remember it as I deal with problems in the future.

                                                          1. 1

                                                            Thanks! It feels like a lonely furrow, and it might end up being only useful to me :) So any feedback welcome.

                                                        1. 3

                                                          Um. Following this link I got redirected to some kind of spam website, that was blocked by my browser.

                                                          Edit: clicked through a bunch more times to try and reproduce, got something slightly different:

                                                          1. 2

                                                            I’ve seen this on compromised WordPress sites before. If it’s the same as what I investigated previously, they do something like push the spam/ad/etc. to 1% of traffic and that makes it difficult to inspect/discover.

                                                            1. 1

                                                              Does it say Comcast in there? Could that be targeted to that connection?

                                                              1. 1

                                                                That’s… worrying. It’s a bog-standard wordpress site. What happens if you go to https://zwischenzugs.com?

                                                                1. 1

                                                                  I clicked through a dozen times and nothing happened. It definitely didn’t happen every time on the original link either.

                                                                  1. 11

                                                                    Looks like it’s a malicious ad coming in. Hard to say which ad network it came from, since the site is loading an obscene number of them…

                                                              1. 0

                                                                There are so many…

                                                                tmux - meant I could ‘pick up where I left off’ when working on the move and crappy internet connections

                                                                vim - super fast and available everywhere