I’m giving a presentation about new web security headers, Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy, etc. at a small Berlin OWASP meetup and I have officially over-polished and over-prepared my slides.
So, ehm, let me know if you have a conference near Germany[1] that wants a presentation about just that :)
[1] I want to stop flying. Nearby includes a night train, but not an airplane.
Cheers to that!
I don’t know for how long you’ve been doing this, but be prepared to have people look at you as if you said something completely alien when you mention this though.
Last week I gave up on Kubernetes after a week of trying to upgrade it [0], and decided that my team would do our simple web service hosting on a single instance. I looked on AWS and the biggest instance these days has 448 cores and 24TB of RAM so I am confident we will not into a problem scaling vertically.
I started with k8s because I wanted to learn it, and the plan was to deploy a whole raft of containers and pipelines into it. But then the data team went off and did their own thing, so my little k8s cluster never got past half a dozen containers.
I’m migrating us off k8s and I am very much looking forward to the reduced complexity.
[0] I will distribute some of the blame to Pulumi which has heretofore been very reliable, but fell apart tracking the cluster state, and also to myself for possibly not Doing It Right or even knowing how to Do It Right.
Don’t sleep on ECS and in particular ECS Fargate. The ECS model is essentially Docker Compose in the Cloud; which is convenient for running production-like clusters locally. If you’re already in on Docker containers, it’s not much additional overhead. Moreover, there are already-baked deployment patterns available in the AWS CDK.
Thanks, I will take a look at ECS, given that Docker Compose is what I was planning to use on my EC2 instance. I reached for “just use an EC2 instance” because I didn’t want to wade through docs for yet another AWS service, but yours is the second positive comment I’ve heard about ECS.
Everything I read about Kubernetes suggests that it’s one of those things you have to really invest time into to learn how to Do It Right. That in and of itself makes me suspicious when I know that I could get very very far scaling vertically. If something is going to be complex and difficult to the extent it has a reputation for that then it’s something that I feel should be absolutely required instead of added because “why not?” or someone wants it on their CV.
Everything I read about Kubernetes suggests that it’s one of those things you have to really invest time into to learn how to Do It Right.
Yeah, there’s a lot to it. I thought we’d grow into it, and I wanted to have it in place so that when containers started to proliferate we’d have a properly configured home for them.
I should say that k8s has been very reliable until it came to upgrade time. Upgrades are hairy and a lot of people say to just launch a new cluster and migrate over to it. It has the nice side-effect of testing that your IaC is solid and you can launch clusters and deploy to them. And to be fair, it’s easy to launch a new cluster. My real trouble was with getting Pulumi to track the state of two clusters, and Pulumi getting epically confused when one cluster went away. It’s the first issue I’ve had with Pulumi that was definitely not user error. That is not k8s fault at all, but it made me take a step back and look at all the k8s config code (don’t get me started on configuring ALBs!) and wonder why I needed all of it.
One of the big strengths of k8s is the ability to pack nodes full of containers so that resources are fully utilized. Totally makes sense when you have a data center with a limited amount of hardware that only changes slowly. But there are so many instance types on AWS that I can find an instance that’s just right for any particular workload. I don’t need the distinction between pods and nodes, and I don’t need all the resource management stuff.
I don’t really know what’s going on in the data infrastructure - I think it’s Kafka (presumably AWS hosted) and various machine learning containers. Triton, ONNX… those are some words I’ve heard recently 😆.
I’m employed mostly to be a fullstack engineer. I can do infrastructure and data engineering, too, but I can’t do all of those things and also get any app development done. The data engineer we hired is very smart, and I talked to him enough in the early days to be confident that he’s set things up at least as well as I would have.
Cool, thanks. Yeah, Kafka really seems to be the industry norm now. We use it as well. Works marvelously until you run into max message size limits, still trying to figure out the best way around that…
Life: Hopefully getting healthy so I can go to FOSSDEM by train from Prague. I already got tickets from Berlin to Brussel’s. It’s gonna be a long trip. I went once before by budget airline but I love trains so I’m trying something new despite the expence.
Work: Trying to combine rathole and Caddy to make an autoconfiguring proxy which will allow me to run jupyterlab anywhere (think behind a NAT or firewall that blocks incomming connections but not outgoing ones) and access it in the browser. The reason is that in my day job we are building streaming data pipelines in jupyterlab and there are some things we can only easilly test ‘close to the data’.
I looked into it and didn’t find a sleeper, I’m going two hops during the day from Berlin, and from Prague to Berlin the day before. It’s gonna be a lot of train, but I like train so yay!
The option i went with at work was a Cloudflare tunnel protected by Cloudflare access which provides an oAuth layer on top to plugin Google/Okta etc. This mainly because we were already using Cloudflare to block a bunch of self-hosted internal apps. Tailscale could be another option.
The built in access controll sounds nice, but I’m somehow picky about having controll over my stack and not feeling vendor locked so I chose a different route.
Working on image-based Linux with NixOS, which I think is quite an exciting match.
NixOS can build disk images with systemd-repart for a long time, and now also has support for building UKIs with ukify. It’s just perfect for software appliances, you can get reproducible, declarative image builds with loads of available packages while securing with measured boot and FDE. Will probably try to write a blog post on the topic once I’m done.
I’m working on a similar stack with Guix, which doesn’t have systemd or even support a seperate /boot partition OOTB– measured boot will be a stretch goal, to say the least. Have a growing appreciation for all those systemd utils. I’d be really interested in seeing or hearing more about how things come together for you!
Setting up Home Assistant. The pi 4 I got for it just came in the mail today so I flashed HAOS, installed ZigBee2MQTT, set up my grand collection of 3 devices and it’s looking pretty neat already. The on/off switch for the thermostat I had to add support for doesn’t show up on the dashboard so I’ll try to fix that.
I’m also trying to coalesce all the ideas I have for a programming language into something coherent, reading some compiler books and looking at other interesting languages that do something similar. I’m aiming for a “simple low-level ML”, but then ah, don’t you kinda want to stick linear types into it? And what about ad hoc polymorphism? And aren’t effect systems pretty neat? …though I think I’ll try to get a basic version without any of these things first, and then try to see how they fit together.
Also, tackling Bitterblack Isle in Dragon’s Dogma. I put off this game for way too long and I’m really enjoying it!
At work I’m refactoring some gRPC code and trying to think of a way to manage protobuf files and compiled code between different services and repos in a way that makes it:
Easy to consume when you’re a client; if a repo wants to call a gRPC service it shouldn’t have to generate the client code by itself.
Quick to iterate on when you’re a server; we have a script that watches the source files (which are mounted as a volume in a docker container) and automatically restarts the service when something is changed, and it also re-generates the protobuf code when the .proto files are changed.
Low maintenance if you don’t touch the gRPC code; right now we have a separate repo which is linked to the main one as a git submodule, with the generated code stored (but not committed) outside of the submodule, and the teams that handles this regularly gets people reporting errors because they forget to run git submodule update after a pull.
Does anybody have suggestions on how to deal with this?
Probably AnotherLevel, though I might also try some later versions! I’m not too familiar with early WMs, since my Linux days started with GNOME 2 in Ubuntu 8.04, so I definitely do want to try pre-GNOME Red Hat.
I’m thinking Red Hat 6 would be a good match to my hardware, but I’m having trouble finding an ISO online. I ordered an original CD from eBay, we’ll see if it arrives this week.
I might try to set up a VM with Bcachefs & see if it’s something I would like considering its in the Linux kernel as of 6.7. Really wish it had scrub, send, & multiple keys for encryption tho.
Working on building a command line remote LaTeX build service. Sometimes I want to write a document in a local editor and build it without installing all of TeXLive.
Becoming a father!!!!
My daughter appears to be very comfortable where she is, so we are having labor induction on Wednesday.
This is the best “What are you doing this week” post I’ve read in a while now :-D. Congratulations!
Congratulations!
Congratulations! Wonderful news!
congrats!
Congrats!
Best wishes to the baby, the mom, and you!
Yes, congratulations!
Congrats and all the best!
Practicing piano! I was just asked yesterday to accompany a choir that performs later this week. so I’ve got a lot of practice ahead of me.
oh, good luck and have fun!
I’m giving a presentation about new web security headers,
Cross-Origin-Opener-Policy,Cross-Origin-Embedder-Policy, etc. at a small Berlin OWASP meetup and I have officially over-polished and over-prepared my slides.So, ehm, let me know if you have a conference near Germany[1] that wants a presentation about just that :)
[1] I want to stop flying. Nearby includes a night train, but not an airplane.
Cheers to that! I don’t know for how long you’ve been doing this, but be prepared to have people look at you as if you said something completely alien when you mention this though.
Solo parenting Tue-Fri which means pretty much everything else will go on the back burner for those days.
Last week I gave up on Kubernetes after a week of trying to upgrade it [0], and decided that my team would do our simple web service hosting on a single instance. I looked on AWS and the biggest instance these days has 448 cores and 24TB of RAM so I am confident we will not into a problem scaling vertically.
I started with k8s because I wanted to learn it, and the plan was to deploy a whole raft of containers and pipelines into it. But then the data team went off and did their own thing, so my little k8s cluster never got past half a dozen containers.
I’m migrating us off k8s and I am very much looking forward to the reduced complexity.
[0] I will distribute some of the blame to Pulumi which has heretofore been very reliable, but fell apart tracking the cluster state, and also to myself for possibly not Doing It Right or even knowing how to Do It Right.
Don’t sleep on ECS and in particular ECS Fargate. The ECS model is essentially Docker Compose in the Cloud; which is convenient for running production-like clusters locally. If you’re already in on Docker containers, it’s not much additional overhead. Moreover, there are already-baked deployment patterns available in the AWS CDK.
Thanks, I will take a look at ECS, given that Docker Compose is what I was planning to use on my EC2 instance. I reached for “just use an EC2 instance” because I didn’t want to wade through docs for yet another AWS service, but yours is the second positive comment I’ve heard about ECS.
Everything I read about Kubernetes suggests that it’s one of those things you have to really invest time into to learn how to Do It Right. That in and of itself makes me suspicious when I know that I could get very very far scaling vertically. If something is going to be complex and difficult to the extent it has a reputation for that then it’s something that I feel should be absolutely required instead of added because “why not?” or someone wants it on their CV.
Yeah, there’s a lot to it. I thought we’d grow into it, and I wanted to have it in place so that when containers started to proliferate we’d have a properly configured home for them.
I should say that k8s has been very reliable until it came to upgrade time. Upgrades are hairy and a lot of people say to just launch a new cluster and migrate over to it. It has the nice side-effect of testing that your IaC is solid and you can launch clusters and deploy to them. And to be fair, it’s easy to launch a new cluster. My real trouble was with getting Pulumi to track the state of two clusters, and Pulumi getting epically confused when one cluster went away. It’s the first issue I’ve had with Pulumi that was definitely not user error. That is not k8s fault at all, but it made me take a step back and look at all the k8s config code (don’t get me started on configuring ALBs!) and wonder why I needed all of it.
One of the big strengths of k8s is the ability to pack nodes full of containers so that resources are fully utilized. Totally makes sense when you have a data center with a limited amount of hardware that only changes slowly. But there are so many instance types on AWS that I can find an instance that’s just right for any particular workload. I don’t need the distinction between pods and nodes, and I don’t need all the resource management stuff.
It helps with scaling but it also provides a bunch of supervision and resiliency stuff.
It’s not the only container cluster manager though.
May I ask what kind of pipelines you’re running and what stack you use for them?
I don’t really know what’s going on in the data infrastructure - I think it’s Kafka (presumably AWS hosted) and various machine learning containers. Triton, ONNX… those are some words I’ve heard recently 😆.
I’m employed mostly to be a fullstack engineer. I can do infrastructure and data engineering, too, but I can’t do all of those things and also get any app development done. The data engineer we hired is very smart, and I talked to him enough in the early days to be confident that he’s set things up at least as well as I would have.
Cool, thanks. Yeah, Kafka really seems to be the industry norm now. We use it as well. Works marvelously until you run into max message size limits, still trying to figure out the best way around that…
Life: Hopefully getting healthy so I can go to FOSSDEM by train from Prague. I already got tickets from Berlin to Brussel’s. It’s gonna be a long trip. I went once before by budget airline but I love trains so I’m trying something new despite the expence.
Work: Trying to combine rathole and Caddy to make an autoconfiguring proxy which will allow me to run jupyterlab anywhere (think behind a NAT or firewall that blocks incomming connections but not outgoing ones) and access it in the browser. The reason is that in my day job we are building streaming data pipelines in jupyterlab and there are some things we can only easilly test ‘close to the data’.
Hope you get well soon. Isn’t there a new sleeper service for Berlin-Brussels now? Are you taking that or what’s your route?
I looked into it and didn’t find a sleeper, I’m going two hops during the day from Berlin, and from Prague to Berlin the day before. It’s gonna be a lot of train, but I like train so yay!
Not sure if you mean “didn’t find one that suited your plans” but this is the service I was thinking of https://www.b-europe.com/EN/Trains/European-Sleeper
Anyway, have a good trip!
Oh, that’s awesome! I’ll have to check that out for the way back.
The option i went with at work was a Cloudflare tunnel protected by Cloudflare access which provides an oAuth layer on top to plugin Google/Okta etc. This mainly because we were already using Cloudflare to block a bunch of self-hosted internal apps. Tailscale could be another option.
The built in access controll sounds nice, but I’m somehow picky about having controll over my stack and not feeling vendor locked so I chose a different route.
$WORK is super busy this week, so likely working and not much else (we’re hiring!).
If I have time, move the needle a bit on my little scripting language.
I’d like to get another round of Castle Panic in with the kids as well.
Starting a new job! Time to brush up on xir very, very rusty PHP skills.
A lot of interviews. And starting the migration from Jira Server to Cloud.
Working on image-based Linux with NixOS, which I think is quite an exciting match.
NixOS can build disk images with
systemd-repartfor a long time, and now also has support for building UKIs withukify. It’s just perfect for software appliances, you can get reproducible, declarative image builds with loads of available packages while securing with measured boot and FDE. Will probably try to write a blog post on the topic once I’m done.I’m working on a similar stack with Guix, which doesn’t have systemd or even support a seperate
/bootpartition OOTB– measured boot will be a stretch goal, to say the least. Have a growing appreciation for all those systemd utils. I’d be really interested in seeing or hearing more about how things come together for you!Setting up Home Assistant. The pi 4 I got for it just came in the mail today so I flashed HAOS, installed ZigBee2MQTT, set up my grand collection of 3 devices and it’s looking pretty neat already. The on/off switch for the thermostat I had to add support for doesn’t show up on the dashboard so I’ll try to fix that.
I’m also trying to coalesce all the ideas I have for a programming language into something coherent, reading some compiler books and looking at other interesting languages that do something similar. I’m aiming for a “simple low-level ML”, but then ah, don’t you kinda want to stick linear types into it? And what about ad hoc polymorphism? And aren’t effect systems pretty neat? …though I think I’ll try to get a basic version without any of these things first, and then try to see how they fit together.
Also, tackling Bitterblack Isle in Dragon’s Dogma. I put off this game for way too long and I’m really enjoying it!
At work I’m refactoring some gRPC code and trying to think of a way to manage protobuf files and compiled code between different services and repos in a way that makes it:
git submodule updateafter a pull.Does anybody have suggestions on how to deal with this?
I think I’m gonna tinker around with some old OSes on my Pentium III system. Think NT4, Solaris, early Red Hat, etc.
Nothing exciting going on at $WORK, just some under-the-hood resilience work.
Are we talking AnotherLevel early, or Bluecurve early?
According to
gitit’s been 92 days since I last touched my FVWM nostalgia config file and AnotherLevel was just so good!Probably AnotherLevel, though I might also try some later versions! I’m not too familiar with early WMs, since my Linux days started with GNOME 2 in Ubuntu 8.04, so I definitely do want to try pre-GNOME Red Hat.
I’m thinking Red Hat 6 would be a good match to my hardware, but I’m having trouble finding an ISO online. I ordered an original CD from eBay, we’ll see if it arrives this week.
I might try to set up a VM with Bcachefs & see if it’s something I would like considering its in the Linux kernel as of 6.7. Really wish it had
scrub,send, & multiple keys for encryption tho.Working on building a command line remote LaTeX build service. Sometimes I want to write a document in a local editor and build it without installing all of TeXLive.
Writing the incremental version of Prolly Tree, where I don’t have to build it all from scratch.