Does anyone here have experience with Flux? I’m curious if this mitigates the problems highlighted here. We’ve felt these pains, and we’re currently investigating Flux as a potential remedy.
I worked indirectly with Flux, it’s a pretty decent system to build on. We chose it at my last job because we were managing a large amount of separate, fully-isolated clusters, all running applications on a common Rails platform. These had a lot of similarities in terms of how they’re deployed, but also had enough differences wherein the infrastructure would need to change on a per-client basis. Flux helped us codify those differences and made it so applying changes to all clusters (such as k8s upgrades, security fixes, or general non-invasive improvements) could be done easily and all at once. For our purposes, I have no complaints…it was a great little tool in our toolbelt.
The only real downside is that it’s relatively simple, and you need to build your own tooling around it in order to get things done. No “point-and-click” stuff, but our infra team was full of competent developers who knew their way around Git, Kubernetes, and AWS. That said, I think the GitOps Toolkit that Flux is based on has some promise if you want to build that kind of tooling and user interface(s) out for your clients.
Author here. Flux is part of one of the choices - how to reconcile the declarative code in source with the target environment.
In that space you have the choice of scripting kubernetes updates in your CICD tool, or using an ArgoCD/Flux operator/agent to reconcile the state. ArgoCD is winning out over Flux where we work, mostly because it historically handled multi-tenant use cases better, and has a useful GUI. There is convergence in this space as their makers are collaborating on an ‘engine’ that both will rest on in future.
As a company, we are looking to move from being tightly coupled to Amazon AWS to a more agnostic approach where we can deploy our platform to different cloud providers (this is not a technical requirement at first, but needed by the business).
The obvious approach for achieving such outcome is to go with Kubernetes; for the past two weeks, I have been diving in the documentation of various tools including Kubernetes (+ Kustomise), Helm, ArgoCD, Ingresses (Istio, Nginx), etc. etc. I have found the amount of information to be overwhelming. We are pretty happy with our current pipeline which deploys on three separate environments (Staging/QA/Production) in Amazon ECS; the move to Kubernetes and GitOps already sound like a big endeavour, with a lot of decisions to be made on tooling and pipelines, and that’s frankly frightening.
My company uses kubernetes and has a similar business requirement to be cloud agnostic. We use all of the hosted clusters, but there is still a crazy amount of complexity going on. Despite a dedicated team and some deep experience, we run into issues fairly often, especially when trying to spin up new services. Once a service is set up its fairly robust, but getting new things deployed is a massive pain.
All of this is to say unless you really need it, I would try to avoid the complexity. I primarily work on the backend, so I don’t interact with the devops work super often, but every time I do its just layers upon layers of abstractions. Even the experts at our company have trouble.
You can be cloud agnostic without k8s + co., and there are alternatives like nomad that I have heard good things about. But yeah, there is a crazy amount to learn, and even once you have things running there is a crazy amount to debug. Troubleshooting also becomes 2x harder.
Thanks for your comment. It confirms my concerns regarding the complexity of a solution like Kubernetes for a small sized company. My main concern at this stage is how to get started since the most basic setup seems to involve many different tools, and supporting multiple environments like we do today involve adding even more complexity.
I have also heard very good feedback on Nomad, but we need to think of future recruitments. There is no doubt that Kubernetes has won the container orchestration, and the number of potential knowledgeable / expert candidates would be significantly higher with Kubernetes vs Nomad (even if the latter is more suitable for our needs).
You’re right, there are numerous tools. I think for getting started you can forgo things like helm and flux, and stick with raw k8s manifests. Helm is a pretty attrocious templating solution in my opinion, and we have run into a number of bugs in what should be a really simple program, so I’d argue you don’t ever need it. Even with just k8s manifests there is a lot to learn, but at least its just one tool rather than 5 or 6.
You will have to do what is best for your situation, so definitely take everything with a grain of salt. One argument I would have for recruitments is that usually the popular technology has a bigger pool of talent, but the average quality of that talent is worse off. Personally I think startups should use niche but powerful tech rather than popular tech, since the applicant pool will self filter. Hiring takes a long time and a bad hire is 2x worse than missing out on a good hire at a small size.
Just food for thought! Wish you all the best in your endeavors.
I agree with your comment on niche technologies unlocking a pool of experts; the counterpart to this argument is that these people may cost a lot of money to acquire and retain, since they will be in demand. Having a large pool of candidates means that you, indeed, you will have more junior candidates, but it’s also an opportunity for people to grow in your company and for building a diverse team that can grow with your organisation.
That being said, I will have definitely have a look and build a small POC with it.
Author here. I wrote this other piece about this specific choice/challenge: https://zwischenzugs.com/2019/03/25/aws-vs-k8s-is-the-new-windows-vs-linux/
Interesting read, thank you very much.
The infographic at the end describes my feeling as a newcomer in the Kubernetes world; it feels that the best practices are not yet fully established so the ecosystem is super diverse and full of products of varying quality.
PS: I am one of those people who were playing Linux in its early days! I remember (not very fondly) the kernel panics following plugging an USB device (especially DSL modems, Linux loved those!)
Disclaimer: I work for Google on what I would call a k8s “adjacent” product where we are heavily invested in the k8s ecosystem, but not part of it.
I think the k8s ecosystem is pretty Wild West as there is so much, and it’s impossible to figure out which tool is best-of-class. I think this is a common situation for “new” technologies. k8s is basically a cloud low-level operating system at this point, and there needs to be layers on top. Some good abstractions for some use cases do exist now, e.g. GCP Cloud Run, but if you’re determined on being cloud agnostic, it’s going to be a hard road until each cloud has comparable products. I don’t spend time in AWS/Azure land as I have my own job to do, but I do not think they have a Cloud Run-esque solution yet.
Do you have to be cloud agnostic? If it’s for super high 99.999% reliability then yeah, that’s your only realistic option. If it’s for having an escape ramp if you want to switch to a different provider for some reason, then I think you could get away with just building your Docker images, and having scaffolding around the single provider you’re invested in. Retooling to a new provider wouldn’t be simple, but it would be an order months, not order years, issue, in my estimation.
But I’ve never done this so don’t take my word for it.