I’m still looking for a way to do local CI using vm’s. Indeed my 6 year old laptop is powerful enough to start vm’s for several Linux distro’s (old and new) and BSDs. I imagine it will work by spinning up a vm with qemu (or the vmm’s on the BSDs) and using SSH to clone my repo into it and running a command inside. There should probably be a mechanism to fetch and/or build latest vm images, and probably a pre-build step that installs toolchains needed for my project so it doesn’t have to happen each time and the test vm’s can be started more quickly.
I hope someone can point me to software doing this. I recently saw a post about nix(os) and starting Linux vm’s (don’t recall anything about BSDs), perhaps that’s the way?
I use QEMU to spin up the VMs. The serial console is passed to a socket file for initial output. QMP can be used for controlling the machine, but that isn’t fully implemented yet. I pass in cloud-init configuration to setup SSH. For a pre-build step my plan is to use QEMU’s snapshot functionality and create a qcow2 file.
You may also be interested in multipass which works similarly. I ran into some issues like templating not working that led me to build my own tool.
Nice, this is about what I had in mind. I also figured the disk snapshot could be used to good effect, especially for the quick ephemeral snapshots. I hadn’t thought about the console yet, QMP seems like a good mechanism.
It looks like you’re aiming to “up” a machine and ssh into it and to things, keeping it around. I was thinking about starting, running a build and stopping, and the vm’s being ephemeral. But keeping them around is a great feature too.
I’m not fond of the idea of having some place where prebuilt images are pulled from. I would really like getting base images from official sources and applying changes locally. E.g. the freebsd vm images (that are signed), for openbsd an iso and auto installer script may be needed. The “library of vm images” would instead be a library of scripts to create vm images locally. The scripts shouldn’t be too complicated, and easy to inspect.
I want to support ephemeral workflows too. The plan is to add a “run” command that starts a machine, runs a command, and then destroys the machine when the command exits.
For the images I am only pulling prebuilt images from official sources. There is no signature verification yet, only checksums. The official library is just a bunch of YAML files shipped with the binary. You can see them in the ./images directory. I’m only supporting images that use cloud-init, so install scripts are regular cloud_config YAML with the nocloud datasource.
I’m hoping to resume work on this project in the next 3-6 months. The project I am focusing on right now needs this one for testing.
The main argument against is that even if you assume good intentions, it won’t be as close to production as an hosted CI (e.g. database version, OS type and version, etc).
Lots of developers develop on macOS and deploy on Linux, and there’s tons of subtle difference between the two systems, such as case sensitivity of the filesystem, as well as default ordering just to give an example.
To me the point of CI isn’t to ensure devs ran the test suite before merging. It’s to provide an environment that will catch as many things as possible that a local run wouldn’t be able to catch.
To me the point of CI isn’t to ensure devs ran the test suite before merging.
I’m basically repeating my other comment but I’m amped up about how much I dislike this idea, probably because it would tank my productivity, and this was too good as example to pass up: the point of CI isn’t (just) to ensure I ran the test suite before merging - although that’s part of it, because what if I forgot? The bigger point, though, is to run the test suite so that I don’t have to.
I have a very, very low threshold for what’s acceptably fast for a test suite. Probably 5-10 seconds or less. If it’s slower than that, I’m simply not going to run the entire thing locally, basically ever. I’m gonna run the tests I care about, and then I’m going to push my changes and let CI either trigger auto-merge, or tell me if there’s other tests I should have cared about (oops!). In the meantime, I’m fully context switched away not even thinking about that PR, because the work is being done for me.
You’re definitely correct here but I think there are plenty of applications where you can like… just trust the intersection between app and os/arch is gonna work.
But now that I think about it, this is such a GH-bound project and like… any such app small enough in scope or value for this to be worth using can just use the free Actions minutes. Doubt they’d go over.
any such app small enough in scope or value for this to be worth using can just use the free Actions minutes.
Yes, that’s the biggest thing that doesn’t make sense to me.
I get the argument that hosted runners are quite weak compared to many developer machines, but if your test suite is small enough to be ran on a single machine, it can probably run about as fast if you parallelize your CI just a tiny bit.
With a fully containerized dev environment yes, that pretty much abolish the divergence in software configuration.
But there are more concern than just that. Does your app relies on some caches? Dependencies?
Where they in a clean state?
I know it’s a bit of an extreme example, but I spend a lot of time using bundle open and editing my gems to debug stuff, it’s not rare I forget to gem pristine after an investigation.
This can lead me to have tests that pass on my machine, and will never work elsewhere. There are millions of scenarios like this one.
I was once rejected from a job (partly) because the Dockerfile I wrote for my code assignment didn’t build on the assessor’s Apple Silicon Mac. I had developed and tested on my x86-64 Linux device. Considering how much server software is built with the same pair of configurations just with the roles switched around, I’d say they aren’t diminished enough.
Was just about to point this out. I’ve seen a lot of bugs in aarch64 Linux software that don’t exist in x86-64 Linux software. You can run a container built for a non-native architecture through Docker’s compatibility layer, but it’s a pretty noticeable performance hit.
One of the things that I like having a CI is the fact that it forces you to declare your dev environment programmatically. It means that you avoid the famous “works in my machine” issue because if tests works in your machine but not in CI, something is missing.
There are of course ways to avoid this issue, maybe if they enforced that all dev tests also run in a controlled environment (either via Docker or maybe something like testcontainers), but it needs more discipline.
This is by far the biggest plus side to CI. Missing external dependencies have bitten me before, but without CI, they’d bite me during deploy, rather than as a failed CI run. I’ve also run into issues specifically with native dependencies on Node, where it’d fetch the correct native dependency on my local machine, but fail to fetch it on CI, which likely means it would’ve failed in prod.
This is something “local CI” can check for. I’ve wanted this, so I added it to my build server tool (that normally runs on a remote machine) called ding. I’ll run something like “ding build make build” where “ding build” is the ci command, and “make build” is what it runs. It clones the current git repo into a temporary directory, and runs the command “make build” in it, sandboxed with bubblewrap.
The point still stands that you can forget to run the local CI.
Yes, but it’s plausible that some people might have implicit trust in cargo due to the configuration presumably being all-declarative (unlike make/configure which are basically known to be equivalent to arbitrary code execution).
Misplaced implicit trust is a very significant factor.
Surely this reduces to: Do not run any commands on untrusted projects.
Not completely. You can run “go build” on a Go project. It does not run any user-supplied code, only the compiler. Pretty neat. Such properties do not happen accidentally, it takes consideration/work (eg the toolchain fully handling builds).
Of course, next thing you often want to do is run whatever you’ve just compiled. Or run the tests, which can do anything.
Still, I like to access interact with untrusted code in a bubblewrap sandbox, using a simple script and config file that exposes just the essentials of my home dir.
go generate can for sure run arbitrary code, but so can any other code generation utility. Importantly, it’s never invoked by go build, which abides a stricter security model, as @mjl described above.
I think that the regrettable and perhaps noteworthy part is that, unlike make or configure, Cargo is not entirely built around the ability to execute arbitrary code. The ability for projects to give you arbitrary code to run is used very rarely, almost everything works declaratively. With some minor tweaks, it could be a system that makes it safe to run certain subcommands on untrusted projects, and this would only break a small number of crates.
Sandboxing macros doesn’t seem super useful if it’s opt-in. But I guess potentially there could be hope for this macro system taking off and becoming the mainstream one to the point where the dangerous macro system could be made opt-in?
With Fennel we sandbox the macros, and it works great, but we started very early with that; like only about a year after the macro system was really created. Making it safe after the fact would be very difficult, but worth it to be able to use static analysis tools safely on untrusted code!
The idea I see people calling for is to make it desirable to switch with things like:
Badging sandboxed macros in the package index and thus making competing crates which don’t opt into sandboxing look less desirable
If they’re implemented using Watt-style WebAssembly, then the option becomes available for Crates.io to offer reproducible builds of the source uploaded to them, similar to how Flathub works, as a way to improve from-scratch cargo build times.
Dylan Beatie suggested something like this for NuGet in his “Open Source, Open Mind: The cost of free software” talk, where he suggests this as an easy way to make sure personal users can still get stuff for free, but corporations can’t un-duly profit from open-source work.
I think the hope there was that the folks collecting the money would give some of it to the developers creating the software. I suspect this situation is about docker trying to get paid for their hosting services.
Remember docker previously tried to “sunset” free image hosting for FOSS projects, I think related to the costs. After outcry, that decision was reverted, but I think there is still a process in place for FOSS projects to publish.
Now they will be charging the users of the images instead, at least doesn’t put the burden on those producing the images.
The previous sunset attempt came just when I needed to publish docker images. So I set up a registry myself instead and have been worry-free since. I also now use fully qualified paths for docker hub images, starting with docker.io/, to not give any special treatment to images on docker hub.
For anyone whose employer runs things in AWS, it’s probably worth just using AWS for this too.
And they have a public mirror of basically everything Docker Hub has (at least, I’ve never found anything missing), and their free/unauthenticated tier quotas have always been a lot more generous. Generally you can just replace FROM some_image with FROM public.ecr.aws/docker/library/some_image and it works.
I also now use fully qualified paths for docker hub images, starting with docker.io/, to not give any special treatment to images on docker hub.
I use Podman for personal projects since I don’t have to justify my weird hippie choices to anyone and that actually forces this on you, which is kind of a nice way to build the habit.
not quite forced, you can set unqualified-search-registries in the config, but it comes with a big warning about how falling back on a search path (rather than using fully-qualified names always) is a crazy risk.
Yes, the distribution was different but the mechanism was largely the same. I’m not saying this is a good solution, or that I support it, but perhaps it’s a step in the right direction.
Let’s say you start out by writing a program that is exactly one hundred lines long, and you release your creation to the world. Every line in this program was written just once.
By the time I commit code, each line has often been already been written at least twice. From initial prototyping, getting new insights, refactoring, bug fixes, modifications to make it testable.
And then there is prototyped code that didn’t make it into a commit/release at all.
I’ve complained about similar to this on Mastodon. I did a year and a half at AWS and got dumped into their “Containers” (EKS, ECS, Fargate, etc) teams. I went from being an “old-school sysop” type like the author describes to having to learn Kubernetes and cloud absolutely from scratch
It was miserable and it was frankly absolutely traumatic participating in the AWS metrics death marches (naturally I got PIP’d and thrown overboard after just over a year)
Something someone pointed out that has been lost in the transition to the cloud is the satisfaction. Everything is just numbers on a screen. CPU, memory. Charts in Grafana etc
Back in the day there was a genuine sense of pride and workmanship in setting up a rack of servers, drilling right down to the OS, tuning all the knobs and pushing all the dials. Really optimizing the thing you’re working on to fit the workload it’s after
Cloud vendors want to treat compute as an ephemeral abstraction. Just throw your perfect little app into a docker container and punt it into a kubernetes void to figure out how to run it. If it dies because it’s buggy, let Kubernetes restart it! Then wrap it in layers of bubble wrap with logging and metrics and all the other Sisyphean layers of abstraction and misery until it’s an unmanageable boulder of mud, tape and tears and it takes an entire SRE team to push it up a hill
My roommate likened it to Marx’s concept of a worker being alienated from his labor. You’re so divorced from your own acts of creation or operation from the computer that everything just melts into pointless sludge
My roommate likened it to Marx’s concept of a worker being alienated from his labor.
I imagine it’s not just LIKE that, it IS that … but I haven’t read the right bits of Marx so I’m not sure. Can anyone tell me which bits of Marx to read please? (He wrote too much!)
Cloud vendors want to treat compute as an ephemeral abstraction. Just throw your perfect little app into a docker container and punt it into a kubernetes void to figure out how to run it.
That’s because everything is a stateless web app! For state (i.e. the hard part), those just talk to a managed service! RDS, S3, etc. Are you making a non-stateless/non-web app? You’re out of luck. Can’t have those anymore…
The basic problem with getaddrinfo() is that it isn’t a DNS API, it’s a name service API, and the name service switch might look up names using /etc/hosts, mDNS, NIS, NETBIOS, who knows what else. Oh and probably DNS too. So the results you get from getaddrinfo() are very much a lowest common denominator.
Applications that depend on DNS specifics should probably pull in a modern stub resolver library, and avoid the ancient BIND-derived res_* functions. There are a couple of problems with this idea:
It might not be possible to get the recursive server IP addresses out of the libc resolver. Implementations have diverged since the BIND 4 days, especially wrt IPv6 support. Some resolver libraries parse /etc/resolv.conf themselves and try to do so in a compatible manner. Check support for fun things like scoped link-local IPv6 addresses.
The system might have a DNS cache that you might accidentally bypass. If that cache provides recursive DNS service on a loopback address it might be shit. You might be better off using a proprietary API instead. (Which is basically what the article concluded.)
The second problem is that it’s synchronous. Looking in hosts is fast and so that’s fine. NETBIOS is probably on the local network, so might be fast. mDNS is link-local and mostly served from a local cache. DNS may miss in a local cache and require a chain of synchronous fetches from DNS servers somewhere in the world, which may take hundreds of ms.
If you want to do one name lookup, that’s fine. If you want to do more, that’s a problem because the only way to overlap them is to have a separate thread for each.
Edit: this is a special case of the problem that a lot of transparent RPC systems encounter. It’s really hard to write software with performance that you can reason about when the latency of a call may vary by 3-4 orders of magnitude.
Applications that depend on DNS specifics should probably pull in a modern stub resolver library
I agree but I don’t think there is one, at least in C land your options are:
dns.c which is a suspiciously large amount of code for a not very complicated protocol (10k LOC), has a difficult API, and hasn’t been updated in 10 years
c-ares is a ridiculous amount of code (50k LOC)
rip out musl’s resolver, it’s not a lot of code but it’s split across 400 files and it all looks like this
It might not be possible to get the recursive server IP addresses out of the libc resolver.
The system might have a DNS cache that you might accidentally bypass.
i wouldn’t mind systems (the OS) just providing a (good!) recursive resolver on the loopback ip and listing it in /etc/resolv.conf. in other words: the dns protocol is the api.
then everyone can use their own stub resolver library with an api of their choice. the OS can do “smart things” in its recursive resolver (like redirect requests for internal domains to some other internal name server; macos has some config options for this in /etc that i have enjoyed but haven’t seen elsewhere; the system resolver could also do doh, etc). these stub resolver libraries may also look at /etc/hosts, but the system loopback resolver could even serve records based on that file too (and hopefully from a new file in /etc in zone syntax). going through a loopback resolver prevents the issue mentioned in the article with caching responses.
btw, is the /etc/resolv.conf syntax still updated at times? so stub resolvers could make use of modern features (i’m thinking of do[thq]).
it’s a name service API, and the name service switch might look up names using /etc/hosts, mDNS, NIS, NETBIOS, who knows what else
i haven’t used getaddrinfo much, certainly not recently. nowadays i (pretend to) live in a Go world, using the pure go resolver (https://pkg.go.dev/net#hdr-Name_Resolution), without going through the system libc resolver. works well for me. i’ve never expected name resolution to return names from mdns/nis/netbios, and i’ve never knowingly used nsswitch (perhaps i’m missing something great?). unfortunately, the current standard library go resolver api has the same issues as mentioned in the article with not providing access to the ttl, or the dnssec status, or the cnames followed (and their dnssec statusses).
I mostly agree. I am confused how access to apache style logs is a not standard anymore.
I think the main thing that changed for the better is the customer isolation requirement. CPUs now support all kinds of useful virtualization thingies that make isolation better.
People have shifted to an incredible amount of complexity into the build environment ala vercel and yet completely refuse to think about hosting implementation.
I finally gave fly.io a try and turns out it is a modern take on cgi-bin. One gets proper control over routing requests, logs in somewhat standard cloud format and firecracker for isolation. Prices are not insane either. However while their stack checks all the cool-modern-features box, it is not sufficiently stable and not at all interoperable with other providers
Yeah I was interested in fly.io too, but they seem to not have their “state” figured out yet … It started out as an edge network, and then people wanted to host stateful apps, so they rolled out an SQL service that was apparently not well received.
I think I just want a modern Heroku … that also has a static file service, like Github Pages
I’m wondering if I use a new language like Inko, Zig, or YSH, how do I say print <p>Hello $name</p> and have it work?
Heroku has build packs for that I guess, and the servers are plain HTTP rather than CGI or uwsgi.
So I wonder if there is a protocol to drop in an arbitrary HTTP server or CGI, in any language.
(BTW I noticed in that comment you worked at Stripe – I remember I visited their offices in SOMA in 2012, I think shortly after they moved from Palo Alto (?). Because I did one of their Capture the Flag contests from Hacker News. I definitely realized it was an extremely talented team then!)
Lack of protocols: unfortunately, the industry seems to have converged on asking app developers to start an HTTP listener in a Docker container that contains all the dependencies needed for the app. This setup can run on any host that supports Docker, including Render, and it frees the platform from worrying about installing or managing application dependencies. I say ‘unfortunately’ because Docker isn’t great for developer ergonomics. As a result, providers who care about DX end up with some behind-the-scenes ‘dockerizing’ for certain languages and frameworks, similar to Render’s native environments, or Heroku’s buildpacks. My own preference is to meet developers where they are and support however they build their apps; Docker or otherwise. We’ll keep working on it.
Lack of stability: the Lindy effect is real in hosting, which is why AWS is still the default for most businesses. However, the hyperscaler usability tax is also very real, pushing people to other providers. You want a host that doesn’t lock you in to their cloud by building vendor-exclusive features like Vercel does with Next.js. This is where Docker can be handy: as long as you manage your own dependencies in a Dockerfile, you can simply take your container to the next host and have them run your app without modification.
Re: Stripe, that was a long time ago! Yes, we moved to SF (near 2nd and Mission) from Palo Alto in January 2012. I’m glad you found CTF fun. It was indeed a great group of people.
I agree Docker has DX issues, but regardless of whether the platform chooses Docker, there are other issues with the app/platform interface, like:
What port or socket does the container listen on, so it can be proxied?
How does the platform get web logs out?
How does the platform check if the app is ready to serve?
How are replicas started? i.e. if the app can dynamically provision apps
In my ideal world, the platform would support any Unix process that obeys a simple protocol – and that process may or may not be a Linux container, which may or may not be Docker
I feel like those are orthogonal concerns
I think this is where Heroku was going around when they were acquired. They started out with Ruby, and then started to make a polyglot and Unix-y platform.
So when I said I would like “a modern Heroku with a static file service”, I was talking about something Unix-y or “12 factor”, based on processes.
And I also liked that the build packs were open source - the app and platform also have an interface at build time, which should be specified.
I mentioned static files because Apache lets you seamlessly mix static and dynamic content – your .html can live next to your .php and .cgi. I think your docs rightly pointed out that this is missing from Heroku
But I looked over the Render docs a bit more, and it seems to be architected more towards getting started easily in common languages, rather than neutral protocols:
There are runtimes native to the Render platform, for Node.js / Bun, Python, Ruby, Go, Rust, and Elixir
e.g. no PHP support, which kinda reminds me that the Google App Engine team had problems adding PHP support for many years – there was definitely a strong language bias there too, for various technical reasons
Although I didn’t see exactly how you’re supposed to build a Docker container, it says Render automatically detects Dockerfiles at the root of your repo and suggests a Docker runtime for your app during service creation – and there there are also like 10 or 20 apps as special cases
I might try it just to “understand what I’m missing”, but from reading the docs, I imagine I’m not the kind of customer you’re targeting
Similar to what I wrote in the blog – Dreamhost is for PHP hosting, it’s not for Unix hosting. I can tell that from the docs and support
Likewise, I think Render is for app hosting, but not arbitrary apps/languages apparently. I’m happy to be corrected though! I’d like it if there are NO “native runtimes” – if every language is on equal footing (as mentioned in the blog post)
I always vendor Go dependencies, in part for reviewing the commit (though I admit I don’t always scrutinize that diff). If someone goes through the trouble of reviewing Go dependencies, I would expect they would just vendor the code. This is for a good part what vendoring is for.
Some time ago I thought about making a service that monitors the Go sum transparency log and tries fetching all modules as well. As a second verification. Could be good to fetch and compare periodically. But it would be lots of work, and will involve dealing with URls that disappear. Anyway, it would be good to find repo’s with tags that change. Nothing good comes of that.
The success of this attack relied on the design of the Go Module Proxy service, which prioritizes caching for performance and availability
I don’t think that’s the reason. It’s more that the attacker created a new repo with a name that looked like the original and tricked someone into imported it instead of the real thing.
The next evolution in these attacks may be the attacker also creating lots of unrelated repo’s that import their malicious package, and get them into pkg.go.dev. To raise the “imported by” count, potentially making it look more popular than the original.
This copycat could have been detected by its use of a similar github username as the original. For regular forks (under existing github accounts, without a name involving boltdb) repo’s with malicious modifications may not be as easy to detect. But they would probably more clearly say they are a fork, so developers would not as easily include them as dependency.
Carriage return, the malicious byte value…
Are there cases where it serves a useful purpose in contemporary computing? I only know of \r\n, where the \r is redundant (i.e. not useful).
Interesting that the new check for the carriage return gets a setting for overriding. I understand not wanting to break anyone’s setup, but my imagination is lacking for who would need it.
For these kinds of text-based protocols, it’s probably reasonable to reject any control character other than tab, newline and space (also no delete). And possibly complicate your code with recognizing and allowing \r\n.
This looks really great! It should go well with my nsdiff and nsnotifyd programs :-)
I wish these parts of the protocol got more love. I can think of a few reasons that they are usually not exposed:
DNS operators tend to be super careful about their primary servers, keeping them well protected from the public Internet, which conflicts with offering DNS UPDATE directly.
Authentication (TSIG) is usually seen as an internal security matter, not something you’d expose to your customers, so implementations have scaling problems.
What might help is some kind of DNS UPDATE proxy that can offload the TSIG key management and access control, to act as a DNS-level firewall between a hidden primary and the big bad net. I haven’t heard of such a thing existing.
Those reasons make sense, thanks. The proxy idea is interesting. I’m only allowing DNS UPDATE on internal IPs for the DNS servers I run for that reason.
Dnsclay could be a good starting point for a proxy that DNS operators could run themselves. The idea had crossed my mind too. The dnsclay admin web interface currently uses HTTP basic auth with a single admin password, with full access to all configured zones. But for DNS UPDATE/AXFR auth, the admin can already manage multiple TSIG and/or TLS pubkey credentials per zone. So it’s mostly the admin web interface that would need to get support for multiple users. Authentication would probably be integrated with some operator-internal auth provider that also returns the zones the user has access to. The “provider” interface that dnsclay uses to make DNS changes is pretty simple to implement, and a backend “dns updates” provider is already available.
Btw, I prefer mutual TLS pubkey authentication over TSIG. Just seems safer, simpler and easier to use, though perhaps doing mutual TLS isn’t easy in some programming environment (it is in Go).
Heh, I didn’t realise until after posting my original reply that of course dnsclay would be a great starting point for the proxy idea :-)
Yeah, TLS has a lot of advantages. I’m an old enough DNS user that in my head TLS still counts as new, and I’m slightly worried a hacking side-project might be required to bring tools up to spec. The key er principal er main thing I wanted to get from TLS was simpler zone transfer configuration, where in many cases access control by IP address is enough, but I want better transport security, and normal one-sided TLS server authentication is perfect. (puns not intended)
I’ve been doing something similar, a little more manual, but easy to reason about. I have a makefile target to start a postgres instance, which I keep running in a terminal during the day I work on the project. Something like this:
There’s also a target to initialize the database, something like this:
postgres-init:
mkdir tmp/pg
initdb -D tmp/pg
TestMain sets up a new template db with migration scripts run. Each test creates a new db based off that template db, with tests run in parallel.
The postgres instance is also used for local development, with some long-term data for testing. During development, especially with concurrent branches with conflicting schema migrations, I just backup/restore the whole tmp/pg directory. With the instance running for local development/manual testing, might as well use it for automated tests. At the end of the day I shut down the terminal with postgres. Conceptually very simple, and fast.
Some times ago, I wanted to formalize a way to inform about supported releases of a software via DNS: https://codeberg.org/forgejo-contrib/rvcodns (unfinished and more or less abandoned for now)
_release.example.org. 3600 IN TXT "v=1.2.3;answer=42;other_key=spaced=20value"
Note that I really think that informing about a new version should be apart from actually deploying the upgrade (which depends on the way the software is packaged: built from source, docker, package manager, gobuilds…)
interesting, we came to almost the same solution and rationale! i searched online at the time, but couldn’t find anyone announcing releases over dns, while it made so much sense to me. i’m sending you an email about this.
it looks like a compatible dns api endpoint could be implemented in the gopherwatch.org dns server.
Isn’t the call for main track talks and dev rooms closed already? It would maybe be better to link to the calls than the submission system, which doesn’t seem to list the deadlines.
It would maybe be better to link to the calls than the submission system, which doesn’t seem to list the deadlines.
This is the CFP for the devrooms, the deadline is 6th of December. Everything is unified under pretalx this year instead of the oldoldoldoldold FOSDEM CFP system.
I thought the call for dev rooms had closed. At least some dev rooms have been announced, the dev room talk submissions are through this system (for most devrooms, a few are doing their own thing). The call for stands is also open.
Indeed devrooms cfp is long closed, and devrooms have been assigned. There are always more requests for devrooms than available rooms, so good proposals do get turned down (with another good chance next year). There is no chance of still getting a devroom.
I’ve had the best experiences with conferences when doing a talk, can recommend. With fosdem, the nice thing is you can also help by organising a devroom. It’s actually a relatively easy way to organise any kind of gathering around open source topics. The fosdem organisers do most of the heavy lifting: making space available, getting everyone in one place, video recording & publishing.
Edit: To be clear, you can still submit a proposal for doing a talk at a devroom (until 30 November, schedules will be announced 15 December), but you cannot submit a proposal for organising a devroom anymore.
Nice find. I’m happy the editor I’m using (acme) makes these characters stand out.
As developer, it’s nice to have x-ray vision on funny unicode business. The webmail client I’m using highlights switching between unicode scripts/blocks (originally to spot idn homoglyph attempts). I’m using the (small) JS code to do that highlighting in more of my projects. Has been helpful finding (often accidental) weird unicode whitespace characters.
In my limited experience, DNSSEC is only usable when your resolver is allowed to downgrade to insecure DNS. It just breaks too often. Strict DNSSEC is a pain.
So for me, SSHFP isn’t really much protection against MITM.
How does “it” break? Can some domains not be resolved because they have some form of brokenness in their DNSSEC setup?
I’m using a DNSSEC verifying resolver on my laptop and servers, and haven’t run into any issues yet.
With extended dns errors (EDE), you get details about DNS failures. So if an issue arises, it should be relatively easy to diagnose. I am a bit surprised at how new EDE is, seems like a pretty basic requirement for diagnosing issues…
Good reminder, I’m not using SSHFP, but it’s easy enough to setup and use.
DNSSEC needs support from all resolvers so that signatures are passed along correctly and so that DS records are queried in the parent zone. There are sadly a lot of resolvers that still lack basic support for a 19-year-old standard.
Yeah, I wouldn’t rely on the resolvers received through dhcp on random networks to implement dnssec. I run unbound locally. No other resolvers should be involved then (only the authoritative name servers). A local dnssec resolver also makes it more reasonable for software (that reads /etc/resolv.conf) to trust the (often still unverified) connection to the resolver.
If a network I’m on would intercept dns requests (from unbound) towards authoritative dns servers, and would break dnssec-related records, then that would cause trouble. I just checked, and it turns out my local unbound forwards to unbound instances on servers, over a vpn (to resolve internal names). Perhaps my experience would be worse when connecting directly to authoritative name servers on random networks.
On servers, I haven’t seen any dns(sec)-related request/response mangling, and would just move elsewhere when that happens.
I’m honestly not sure how it broke; that’s part of the problem. After all my time troubleshooting, I eventually decided to go back to plain-old DNS and get on with my life.
Maybe the tooling is better these days, and next time I set it up, it’ll go more smoothly.
The thing that blew my mind with Plan9 is that the bindings are process-specific. So if you need a very specific process to have a very private /tmp, you can do that, and it changes nothing for all the other processes.
What I don’t get with that… all usages of /tmp I’ve had to deal with are with inter-process communication stuff, so if /tmp is isolated to a process then its utility seems hindered?
Unless people are thinking of a sort of layered file system based off your process tree…. but that gets you into a whole world of pain without the right abstraction
I am, indeed, talking about layering of FS namespaces: that is how Plan9 primarily operates. As for uses of /tmp, you should really use /run instead in most cases.
In Plan 9, you would set up the namespace of your process, e.g. for logins or some system service. You could use ramfs(4) on /tmp (https://9p.io/magic/man2html/4/ramfs), or bind or mount something disk/network-backed. You wouldn’t need a variable to find the temp dir: anyone can put anything at the fixed path /tmp.
With per-user /tmp most reasonable sharing of files/sockets will keep working, without most of the risk.
You wouldn’t need a variable to find the temp dir: anyone can put anything at the fixed path /tmp.
Right, but the /tmp process A sees is not the /tmp process B sees, to my understanding.
With per-user /tmp most reasonable sharing of files/sockets will keep working, without most of the risk.
This is a reasonable stop-gap solution for that problem, but using /tmp for files/sockets is inappropriate in the first place IMO. /tmp is not a great way to share files and in general there are better places for things like named pipes/sockets/etc. and tying their existence to the lifetime of the process group and simplifies the application code because it no longer needs something like an atexit to clean up after itself.
Oh, interesting! The multics one sounds like it’s being used for the kinds of things that are usually mmap(MAP_ANON) on unix. But I suppose if the process directory isn’t anonymous it can be used for file-based or shared memory IPC?
Multics had single-level storage, so all those files were considered memory mapped anyways. AFAIK (I haven’t used Multics much), there wasn’t much distinction between files and heap in memory; I guess the best analogy is if all files were mmap’d on Unix - making the MAP_ANON analogy as well work.
I’m more familar with i, which is also SLS, but persistent memory and single address space as well. On i, you can could just pass other processes pointers to things you have in QTEMP (or anywhere else, but objects in QTEMP are known by name only to that process), which can be not just files (which are also SQL tables), but also other things like data queues and user spaces (basically opaque byte arrays).
The fact that email still involves copy-pasting the entire email into the response is insane to me. How is this still the case with all this amazing technology at our disposal!? It truly baffles me…
On the other hand, you never run into the problem where a message’s context can be lost because its parents were deleted and you don’t need to perform remote lookups to find it.
yes, this is a bit silly. you could forward & reply to messages by just attaching the entire original full message as a multipart part. and have the receiving mail client render the original message just like it would do if received directly. i would love to know how good clients are at rendering full attached messages. i still have this on the todo-list for my mail client…
writing an initial html message isn’t necessarily too crazy: use only basic styling mechanisms while editing, then generate html and send it. it starts getting a bit overcomplicated and unrealistic when people start start replying to and editing html emails. email clients (already often not the most modern piece of technology) need to be full html wysiwyg editors…
though people often also do this with html messages and hilarity ensues.
my favorite issue is where (some version of) outlook collapses (hides) html content when it sees anything that looks a bit quoted (indeed, how do you detect which parts of an html message are quoted and which are not?). so that includes your inline replies. they even collapse if they don’t find any other content in a message! so people think they’re getting empty messages, plenty never realize there actually was a reply.
Or just don’t include the message being replied to because any non-crap client should allow easy access to the thread’s history.
doing this can be useful for context if you’re forwarding, or including new recipients in a reply.
but indeed. most replies can just start with new empty messages.
Yeah, forwarding is a different story and may be better attached as a separate part as you suggested. I’m sure you can still come up with creative attacks based on client differences. (For example use CSS or elements that are hidden in the “middleman”‘s client but visible in the final destination’s client.) But it would likely be more robust than the quoting mechanism. Especially if you work at the same place and may be mandated to use the same client.
This practice has been very valuable for us. As a recent example, we avoided an issue that affected several other QUIC implementations. It’s also been adopted into other AWS codebases such as s2n-tls and the Encryption SDK.
Nice, very thorough, and good idea to work based on MUST/SHOULD/etc statements. Will be keeping this in mind for future development.
Have you cross-referenced internet drafts too? I can imagine it works well enough with the text-snippet approach. Outdated snippets can be detected. My line-based approach won’t work for such changing text.
Curious: have you found problems with the specs while annotating? E.g. contradictions, missing requirements.
Have you cross-referenced internet drafts too? I can imagine it works well enough with the text-snippet approach. Outdated snippets can be detected. My line-based approach won’t work for such changing text.
Yes! This was incredibly useful while QUIC was still WIP and was a moving target. The main RFC had 34 revisions (!) and the protocol is spread across 4 RFCs in total. There’s also lots of references to other RFCs that we wanted to capture as well. Needless to say, it’s a very complex protocol to implement and for us duvet greatly helped in taming some of the complexity.
Curious: have you found problems with the specs while annotating? E.g. contradictions, missing requirements.
Good question… There’s been a few issues we’ve found in RFCs, though none in QUIC at this point.
I’m still looking for a way to do local CI using vm’s. Indeed my 6 year old laptop is powerful enough to start vm’s for several Linux distro’s (old and new) and BSDs. I imagine it will work by spinning up a vm with qemu (or the vmm’s on the BSDs) and using SSH to clone my repo into it and running a command inside. There should probably be a mechanism to fetch and/or build latest vm images, and probably a pre-build step that installs toolchains needed for my project so it doesn’t have to happen each time and the test vm’s can be started more quickly.
I hope someone can point me to software doing this. I recently saw a post about nix(os) and starting Linux vm’s (don’t recall anything about BSDs), perhaps that’s the way?
It isn’t complete, but this is a tool I am working on to spin up VMs locally for testing: https://github.com/stacktide/fog
I use QEMU to spin up the VMs. The serial console is passed to a socket file for initial output. QMP can be used for controlling the machine, but that isn’t fully implemented yet. I pass in cloud-init configuration to setup SSH. For a pre-build step my plan is to use QEMU’s snapshot functionality and create a qcow2 file.
You may also be interested in multipass which works similarly. I ran into some issues like templating not working that led me to build my own tool.
Nice, this is about what I had in mind. I also figured the disk snapshot could be used to good effect, especially for the quick ephemeral snapshots. I hadn’t thought about the console yet, QMP seems like a good mechanism.
It looks like you’re aiming to “up” a machine and ssh into it and to things, keeping it around. I was thinking about starting, running a build and stopping, and the vm’s being ephemeral. But keeping them around is a great feature too.
I’m not fond of the idea of having some place where prebuilt images are pulled from. I would really like getting base images from official sources and applying changes locally. E.g. the freebsd vm images (that are signed), for openbsd an iso and auto installer script may be needed. The “library of vm images” would instead be a library of scripts to create vm images locally. The scripts shouldn’t be too complicated, and easy to inspect.
Would like to hear more about your plans!
I want to support ephemeral workflows too. The plan is to add a “run” command that starts a machine, runs a command, and then destroys the machine when the command exits.
For the images I am only pulling prebuilt images from official sources. There is no signature verification yet, only checksums. The official library is just a bunch of YAML files shipped with the binary. You can see them in the
./imagesdirectory. I’m only supporting images that use cloud-init, so install scripts are regular cloud_config YAML with the nocloud datasource.I’m hoping to resume work on this project in the next 3-6 months. The project I am focusing on right now needs this one for testing.
Why not virtualbox?
I think this is a great idea, but I am anticipating folks explainIng why it isn’t.
The main argument against is that even if you assume good intentions, it won’t be as close to production as an hosted CI (e.g. database version, OS type and version, etc).
Lots of developers develop on macOS and deploy on Linux, and there’s tons of subtle difference between the two systems, such as case sensitivity of the filesystem, as well as default ordering just to give an example.
To me the point of CI isn’t to ensure devs ran the test suite before merging. It’s to provide an environment that will catch as many things as possible that a local run wouldn’t be able to catch.
I’m basically repeating my other comment but I’m amped up about how much I dislike this idea, probably because it would tank my productivity, and this was too good as example to pass up: the point of CI isn’t (just) to ensure I ran the test suite before merging - although that’s part of it, because what if I forgot? The bigger point, though, is to run the test suite so that I don’t have to.
I have a very, very low threshold for what’s acceptably fast for a test suite. Probably 5-10 seconds or less. If it’s slower than that, I’m simply not going to run the entire thing locally, basically ever. I’m gonna run the tests I care about, and then I’m going to push my changes and let CI either trigger auto-merge, or tell me if there’s other tests I should have cared about (oops!). In the meantime, I’m fully context switched away not even thinking about that PR, because the work is being done for me.
You’re definitely correct here but I think there are plenty of applications where you can like… just trust the intersection between app and os/arch is gonna work.
But now that I think about it, this is such a GH-bound project and like… any such app small enough in scope or value for this to be worth using can just use the free Actions minutes. Doubt they’d go over.
Yes, that’s the biggest thing that doesn’t make sense to me.
I get the argument that hosted runners are quite weak compared to many developer machines, but if your test suite is small enough to be ran on a single machine, it can probably run about as fast if you parallelize your CI just a tiny bit.
I wonder if those differences are diminished if everything runs on Docker
With a fully containerized dev environment yes, that pretty much abolish the divergence in software configuration.
But there are more concern than just that. Does your app relies on some caches? Dependencies?
Where they in a clean state?
I know it’s a bit of an extreme example, but I spend a lot of time using
bundle openand editing my gems to debug stuff, it’s not rare I forget togem pristineafter an investigation.This can lead me to have tests that pass on my machine, and will never work elsewhere. There are millions of scenarios like this one.
I was once rejected from a job (partly) because the Dockerfile I wrote for my code assignment didn’t build on the assessor’s Apple Silicon Mac. I had developed and tested on my x86-64 Linux device. Considering how much server software is built with the same pair of configurations just with the roles switched around, I’d say they aren’t diminished enough.
Was just about to point this out. I’ve seen a lot of bugs in aarch64 Linux software that don’t exist in x86-64 Linux software. You can run a container built for a non-native architecture through Docker’s compatibility layer, but it’s a pretty noticeable performance hit.
One of the things that I like having a CI is the fact that it forces you to declare your dev environment programmatically. It means that you avoid the famous “works in my machine” issue because if tests works in your machine but not in CI, something is missing.
There are of course ways to avoid this issue, maybe if they enforced that all dev tests also run in a controlled environment (either via Docker or maybe something like testcontainers), but it needs more discipline.
This is by far the biggest plus side to CI. Missing external dependencies have bitten me before, but without CI, they’d bite me during deploy, rather than as a failed CI run. I’ve also run into issues specifically with native dependencies on Node, where it’d fetch the correct native dependency on my local machine, but fail to fetch it on CI, which likely means it would’ve failed in prod.
Here’s one: if you forget to check in a file, this won’t catch it.
It checks if the repo is not dirty, so it shouldn’t.
This is something “local CI” can check for. I’ve wanted this, so I added it to my build server tool (that normally runs on a remote machine) called ding. I’ll run something like “ding build make build” where “ding build” is the ci command, and “make build” is what it runs. It clones the current git repo into a temporary directory, and runs the command “make build” in it, sandboxed with bubblewrap.
The point still stands that you can forget to run the local CI.
What’s to stop me from lying and making the gh api calls manually?
Surely this reduces to: Do not run any commands on untrusted projects.
As soon as you run
make,configure,meson, it’s all the same no matter what the project is implemented in.Yes, but it’s plausible that some people might have implicit trust in cargo due to the configuration presumably being all-declarative (unlike make/configure which are basically known to be equivalent to arbitrary code execution).
Misplaced implicit trust is a very significant factor.
Not completely. You can run “go build” on a Go project. It does not run any user-supplied code, only the compiler. Pretty neat. Such properties do not happen accidentally, it takes consideration/work (eg the toolchain fully handling builds).
Of course, next thing you often want to do is run whatever you’ve just compiled. Or run the tests, which can do anything.
Still, I like to access interact with untrusted code in a bubblewrap sandbox, using a simple script and config file that exposes just the essentials of my home dir.
Apparently just
cding into a git repo can execute attacker-controlled code:https://old.reddit.com/r/rust/comments/1j2i3s0/psa_do_not_run_any_cargo_commands_on_untrusted/mft1rbl/
You have the
go generatestep beforego build, that many projects use and can run arbitrary commands.go generatecan for sure run arbitrary code, but so can any other code generation utility. Importantly, it’s never invoked bygo build, which abides a stricter security model, as @mjl described above.I think that the regrettable and perhaps noteworthy part is that, unlike make or configure, Cargo is not entirely built around the ability to execute arbitrary code. The ability for projects to give you arbitrary code to run is used very rarely, almost everything works declaratively. With some minor tweaks, it could be a system that makes it safe to run certain subcommands on untrusted projects, and this would only break a small number of crates.
…and there’s also a low-priority intent to support opt-in sandboxing for things like procedural macros and possibly build scripts, inspired by watt.
Sandboxing macros doesn’t seem super useful if it’s opt-in. But I guess potentially there could be hope for this macro system taking off and becoming the mainstream one to the point where the dangerous macro system could be made opt-in?
With Fennel we sandbox the macros, and it works great, but we started very early with that; like only about a year after the macro system was really created. Making it safe after the fact would be very difficult, but worth it to be able to use static analysis tools safely on untrusted code!
The idea I see people calling for is to make it desirable to switch with things like:
cargo buildtimes.Dylan Beatie suggested something like this for NuGet in his “Open Source, Open Mind: The cost of free software” talk, where he suggests this as an easy way to make sure personal users can still get stuff for free, but corporations can’t un-duly profit from open-source work.
I think the hope there was that the folks collecting the money would give some of it to the developers creating the software. I suspect this situation is about docker trying to get paid for their hosting services.
Remember docker previously tried to “sunset” free image hosting for FOSS projects, I think related to the costs. After outcry, that decision was reverted, but I think there is still a process in place for FOSS projects to publish.
Now they will be charging the users of the images instead, at least doesn’t put the burden on those producing the images.
The previous sunset attempt came just when I needed to publish docker images. So I set up a registry myself instead and have been worry-free since. I also now use fully qualified paths for docker hub images, starting with docker.io/, to not give any special treatment to images on docker hub.
For anyone whose employer runs things in AWS, it’s probably worth just using AWS for this too.
And they have a public mirror of basically everything Docker Hub has (at least, I’ve never found anything missing), and their free/unauthenticated tier quotas have always been a lot more generous. Generally you can just replace
FROM some_imagewithFROM public.ecr.aws/docker/library/some_imageand it works.I use Podman for personal projects since I don’t have to justify my weird hippie choices to anyone and that actually forces this on you, which is kind of a nice way to build the habit.
not quite forced, you can set
unqualified-search-registriesin the config, but it comes with a big warning about how falling back on a search path (rather than using fully-qualified names always) is a crazy risk.Yes, the distribution was different but the mechanism was largely the same. I’m not saying this is a good solution, or that I support it, but perhaps it’s a step in the right direction.
By the time I commit code, each line has often been already been written at least twice. From initial prototyping, getting new insights, refactoring, bug fixes, modifications to make it testable.
And then there is prototyped code that didn’t make it into a commit/release at all.
I’ve complained about similar to this on Mastodon. I did a year and a half at AWS and got dumped into their “Containers” (EKS, ECS, Fargate, etc) teams. I went from being an “old-school sysop” type like the author describes to having to learn Kubernetes and cloud absolutely from scratch
It was miserable and it was frankly absolutely traumatic participating in the AWS metrics death marches (naturally I got PIP’d and thrown overboard after just over a year)
Something someone pointed out that has been lost in the transition to the cloud is the satisfaction. Everything is just numbers on a screen. CPU, memory. Charts in Grafana etc
Back in the day there was a genuine sense of pride and workmanship in setting up a rack of servers, drilling right down to the OS, tuning all the knobs and pushing all the dials. Really optimizing the thing you’re working on to fit the workload it’s after
Cloud vendors want to treat compute as an ephemeral abstraction. Just throw your perfect little app into a docker container and punt it into a kubernetes void to figure out how to run it. If it dies because it’s buggy, let Kubernetes restart it! Then wrap it in layers of bubble wrap with logging and metrics and all the other Sisyphean layers of abstraction and misery until it’s an unmanageable boulder of mud, tape and tears and it takes an entire SRE team to push it up a hill
My roommate likened it to Marx’s concept of a worker being alienated from his labor. You’re so divorced from your own acts of creation or operation from the computer that everything just melts into pointless sludge
I imagine it’s not just LIKE that, it IS that … but I haven’t read the right bits of Marx so I’m not sure. Can anyone tell me which bits of Marx to read please? (He wrote too much!)
That’s because everything is a stateless web app! For state (i.e. the hard part), those just talk to a managed service! RDS, S3, etc. Are you making a non-stateless/non-web app? You’re out of luck. Can’t have those anymore…
I’m still keeping my state in sight and control.
The basic problem with getaddrinfo() is that it isn’t a DNS API, it’s a name service API, and the name service switch might look up names using /etc/hosts, mDNS, NIS, NETBIOS, who knows what else. Oh and probably DNS too. So the results you get from getaddrinfo() are very much a lowest common denominator.
Applications that depend on DNS specifics should probably pull in a modern stub resolver library, and avoid the ancient BIND-derived
res_*functions. There are a couple of problems with this idea:It might not be possible to get the recursive server IP addresses out of the libc resolver. Implementations have diverged since the BIND 4 days, especially wrt IPv6 support. Some resolver libraries parse /etc/resolv.conf themselves and try to do so in a compatible manner. Check support for fun things like scoped link-local IPv6 addresses.
The system might have a DNS cache that you might accidentally bypass. If that cache provides recursive DNS service on a loopback address it might be shit. You might be better off using a proprietary API instead. (Which is basically what the article concluded.)
Yeah, it’s a mess.
The second problem is that it’s synchronous. Looking in hosts is fast and so that’s fine. NETBIOS is probably on the local network, so might be fast. mDNS is link-local and mostly served from a local cache. DNS may miss in a local cache and require a chain of synchronous fetches from DNS servers somewhere in the world, which may take hundreds of ms.
If you want to do one name lookup, that’s fine. If you want to do more, that’s a problem because the only way to overlap them is to have a separate thread for each.
Edit: this is a special case of the problem that a lot of transparent RPC systems encounter. It’s really hard to write software with performance that you can reason about when the latency of a call may vary by 3-4 orders of magnitude.
I agree but I don’t think there is one, at least in C land your options are:
I was thinking of something like adns, ldns, getdns
There’s a lot of fiddly parsing in a DNS implementation because of the many different RR types, so it’s hard to make it both small and easy to use.
Also, that musl code appears to have completely incorrect domain name parsing. musl’s stub resolver is very bad.
libunbound is just shy of 3KLOC:
i wouldn’t mind systems (the OS) just providing a (good!) recursive resolver on the loopback ip and listing it in /etc/resolv.conf. in other words: the dns protocol is the api. then everyone can use their own stub resolver library with an api of their choice. the OS can do “smart things” in its recursive resolver (like redirect requests for internal domains to some other internal name server; macos has some config options for this in /etc that i have enjoyed but haven’t seen elsewhere; the system resolver could also do doh, etc). these stub resolver libraries may also look at /etc/hosts, but the system loopback resolver could even serve records based on that file too (and hopefully from a new file in /etc in zone syntax). going through a loopback resolver prevents the issue mentioned in the article with caching responses.
btw, is the /etc/resolv.conf syntax still updated at times? so stub resolvers could make use of modern features (i’m thinking of do[thq]).
i haven’t used getaddrinfo much, certainly not recently. nowadays i (pretend to) live in a Go world, using the pure go resolver (https://pkg.go.dev/net#hdr-Name_Resolution), without going through the system libc resolver. works well for me. i’ve never expected name resolution to return names from mdns/nis/netbios, and i’ve never knowingly used nsswitch (perhaps i’m missing something great?). unfortunately, the current standard library go resolver api has the same issues as mentioned in the article with not providing access to the ttl, or the dnssec status, or the cnames followed (and their dnssec statusses).
This is how most local (usually caching) resolvers work. unbound, systemd-resolved, etc…
I agree that it would be good to make it a “standard” such that it is always expected to be configured.
I mostly agree. I am confused how access to apache style logs is a not standard anymore.
I think the main thing that changed for the better is the customer isolation requirement. CPUs now support all kinds of useful virtualization thingies that make isolation better.
People have shifted to an incredible amount of complexity into the build environment ala vercel and yet completely refuse to think about hosting implementation.
I finally gave fly.io a try and turns out it is a modern take on cgi-bin. One gets proper control over routing requests, logs in somewhat standard cloud format and firecracker for isolation. Prices are not insane either. However while their stack checks all the cool-modern-features box, it is not sufficiently stable and not at all interoperable with other providers
Yeah I was interested in fly.io too, but they seem to not have their “state” figured out yet … It started out as an edge network, and then people wanted to host stateful apps, so they rolled out an SQL service that was apparently not well received.
I think I just want a modern Heroku … that also has a static file service, like Github Pages
(shameless plug) Render is pretty much this. I’m the founder; happy to help.
Oh cool! The funny thing is that I linked one of your docs in the appendix of this post:
https://render.com/docs/render-vs-heroku-comparison
And that’s because I saw it on Hacker News a few days ago :-)
https://news.ycombinator.com/item?id=42833080
That doc was helpful, and helped me understand what Render is.
What do you think about the points in this section: https://oils.pub/blog/2025/02/shared-hosting.html#whats-wrong-with-cloud-hosting
I linked one of your docs in that section: https://render.com/docs/#quickstarts
I’m wondering if I use a new language like Inko, Zig, or YSH, how do I say print
<p>Hello $name</p>and have it work?Heroku has build packs for that I guess, and the servers are plain HTTP rather than CGI or uwsgi.
So I wonder if there is a protocol to drop in an arbitrary HTTP server or CGI, in any language.
(BTW I noticed in that comment you worked at Stripe – I remember I visited their offices in SOMA in 2012, I think shortly after they moved from Palo Alto (?). Because I did one of their Capture the Flag contests from Hacker News. I definitely realized it was an extremely talented team then!)
Lack of protocols: unfortunately, the industry seems to have converged on asking app developers to start an HTTP listener in a Docker container that contains all the dependencies needed for the app. This setup can run on any host that supports Docker, including Render, and it frees the platform from worrying about installing or managing application dependencies. I say ‘unfortunately’ because Docker isn’t great for developer ergonomics. As a result, providers who care about DX end up with some behind-the-scenes ‘dockerizing’ for certain languages and frameworks, similar to Render’s native environments, or Heroku’s buildpacks. My own preference is to meet developers where they are and support however they build their apps; Docker or otherwise. We’ll keep working on it.
Lack of stability: the Lindy effect is real in hosting, which is why AWS is still the default for most businesses. However, the hyperscaler usability tax is also very real, pushing people to other providers. You want a host that doesn’t lock you in to their cloud by building vendor-exclusive features like Vercel does with Next.js. This is where Docker can be handy: as long as you manage your own dependencies in a Dockerfile, you can simply take your container to the next host and have them run your app without modification.
Re: Stripe, that was a long time ago! Yes, we moved to SF (near 2nd and Mission) from Palo Alto in January 2012. I’m glad you found CTF fun. It was indeed a great group of people.
I agree Docker has DX issues, but regardless of whether the platform chooses Docker, there are other issues with the app/platform interface, like:
In my ideal world, the platform would support any Unix process that obeys a simple protocol – and that process may or may not be a Linux container, which may or may not be Docker
I feel like those are orthogonal concerns
I think this is where Heroku was going around when they were acquired. They started out with Ruby, and then started to make a polyglot and Unix-y platform.
https://12factor.net/ - https://lobste.rs/s/ksbcmq/twelve_factor_app_2011
So when I said I would like “a modern Heroku with a static file service”, I was talking about something Unix-y or “12 factor”, based on processes.
And I also liked that the build packs were open source - the app and platform also have an interface at build time, which should be specified.
I mentioned static files because Apache lets you seamlessly mix static and dynamic content – your
.htmlcan live next to your.phpand.cgi. I think your docs rightly pointed out that this is missing from HerokuBut I looked over the Render docs a bit more, and it seems to be architected more towards getting started easily in common languages, rather than neutral protocols:
I might try it just to “understand what I’m missing”, but from reading the docs, I imagine I’m not the kind of customer you’re targeting
Similar to what I wrote in the blog – Dreamhost is for PHP hosting, it’s not for Unix hosting. I can tell that from the docs and support
Likewise, I think Render is for app hosting, but not arbitrary apps/languages apparently. I’m happy to be corrected though! I’d like it if there are NO “native runtimes” – if every language is on equal footing (as mentioned in the blog post)
fyi, they seem to be working on a better sql service, to be launched within months. https://go.news.fly.io/deliveries/dgSN7QkDAOOaEeKaEQGUtLI1B0iFoL8oIlsSYqY=
(i’m not a user of fly)
Neat find. These “AI” checks about dodgy calls seem useful to have for dependencies. I wouldn’t mind having it locally.
Looks like an application from one person is importing the code: https://pkg.go.dev/github.com/boltdb-go/bolt?tab=importedby Aside: I think most maintained software using “boltdb” is actually using the maintained fork at https://pkg.go.dev/go.etcd.io/bbolt.
I always vendor Go dependencies, in part for reviewing the commit (though I admit I don’t always scrutinize that diff). If someone goes through the trouble of reviewing Go dependencies, I would expect they would just vendor the code. This is for a good part what vendoring is for.
Some time ago I thought about making a service that monitors the Go sum transparency log and tries fetching all modules as well. As a second verification. Could be good to fetch and compare periodically. But it would be lots of work, and will involve dealing with URls that disappear. Anyway, it would be good to find repo’s with tags that change. Nothing good comes of that.
I don’t think that’s the reason. It’s more that the attacker created a new repo with a name that looked like the original and tricked someone into imported it instead of the real thing.
The next evolution in these attacks may be the attacker also creating lots of unrelated repo’s that import their malicious package, and get them into pkg.go.dev. To raise the “imported by” count, potentially making it look more popular than the original.
This copycat could have been detected by its use of a similar github username as the original. For regular forks (under existing github accounts, without a name involving boltdb) repo’s with malicious modifications may not be as easy to detect. But they would probably more clearly say they are a fork, so developers would not as easily include them as dependency.
Carriage return, the malicious byte value… Are there cases where it serves a useful purpose in contemporary computing? I only know of \r\n, where the \r is redundant (i.e. not useful). Interesting that the new check for the carriage return gets a setting for overriding. I understand not wanting to break anyone’s setup, but my imagination is lacking for who would need it. For these kinds of text-based protocols, it’s probably reasonable to reject any control character other than tab, newline and space (also no delete). And possibly complicate your code with recognizing and allowing \r\n.
Terminals still interpret it as originally intended:
printf 'Now you see me\rNow you dont! \n'appears to print only the second message.The fact that software emulation of VT100-era terminals is still a part of contemporary computing is truly poetic – as in Vogon poetry.
And it goes even further: The video terminals were emulating teletypes - which had a real carriage to return.
And they don’t call it the ASCII character
BELfor nothing!This looks really great! It should go well with my
nsdiffandnsnotifydprograms :-)I wish these parts of the protocol got more love. I can think of a few reasons that they are usually not exposed:
DNS operators tend to be super careful about their primary servers, keeping them well protected from the public Internet, which conflicts with offering DNS UPDATE directly.
Authentication (TSIG) is usually seen as an internal security matter, not something you’d expose to your customers, so implementations have scaling problems.
What might help is some kind of DNS UPDATE proxy that can offload the TSIG key management and access control, to act as a DNS-level firewall between a hidden primary and the big bad net. I haven’t heard of such a thing existing.
Those reasons make sense, thanks. The proxy idea is interesting. I’m only allowing DNS UPDATE on internal IPs for the DNS servers I run for that reason.
Dnsclay could be a good starting point for a proxy that DNS operators could run themselves. The idea had crossed my mind too. The dnsclay admin web interface currently uses HTTP basic auth with a single admin password, with full access to all configured zones. But for DNS UPDATE/AXFR auth, the admin can already manage multiple TSIG and/or TLS pubkey credentials per zone. So it’s mostly the admin web interface that would need to get support for multiple users. Authentication would probably be integrated with some operator-internal auth provider that also returns the zones the user has access to. The “provider” interface that dnsclay uses to make DNS changes is pretty simple to implement, and a backend “dns updates” provider is already available.
Btw, I prefer mutual TLS pubkey authentication over TSIG. Just seems safer, simpler and easier to use, though perhaps doing mutual TLS isn’t easy in some programming environment (it is in Go).
Heh, I didn’t realise until after posting my original reply that of course dnsclay would be a great starting point for the proxy idea :-)
Yeah, TLS has a lot of advantages. I’m an old enough DNS user that in my head TLS still counts as new, and I’m slightly worried a hacking side-project might be required to bring tools up to spec. The
keyerprincipaler main thing I wanted to get from TLS was simpler zone transfer configuration, where in many cases access control by IP address is enough, but I want better transport security, and normal one-sided TLS server authentication is perfect. (puns not intended)I’ve been doing something similar, a little more manual, but easy to reason about. I have a makefile target to start a postgres instance, which I keep running in a terminal during the day I work on the project. Something like this:
There’s also a target to initialize the database, something like this:
TestMain sets up a new template db with migration scripts run. Each test creates a new db based off that template db, with tests run in parallel.
The postgres instance is also used for local development, with some long-term data for testing. During development, especially with concurrent branches with conflicting schema migrations, I just backup/restore the whole tmp/pg directory. With the instance running for local development/manual testing, might as well use it for automated tests. At the end of the day I shut down the terminal with postgres. Conceptually very simple, and fast.
I found the DNS part really interesting.
Some times ago, I wanted to formalize a way to inform about supported releases of a software via DNS: https://codeberg.org/forgejo-contrib/rvcodns (unfinished and more or less abandoned for now)
I also created a Go library to fetch such release information: https://code.pfad.fr/rvcodns/
Forgejo already uses DNS to indicate when a new version is available (this was the initial experiment to the proposal above): https://www.whatsmydns.net/#TXT/release.forgejo.org
Note that I really think that informing about a new version should be apart from actually deploying the upgrade (which depends on the way the software is packaged: built from source, docker, package manager, gobuilds…)
interesting, we came to almost the same solution and rationale! i searched online at the time, but couldn’t find anyone announcing releases over dns, while it made so much sense to me. i’m sending you an email about this.
it looks like a compatible dns api endpoint could be implemented in the gopherwatch.org dns server.
Does anyone know which software will/won’t automatically trust the new root anchor?
A quick search indicates systemd-resolverd (https://www.freedesktop.org/software/systemd/man/latest/dnssec-trust-anchors.d.html#Positive%20Trust%20Anchors), powerdns recursor (https://doc.powerdns.com/recursor/dnssec.html#trust-anchors) and dnsmasq may not support it.
Unbound, bind & knot resolvers do.
Isn’t the call for main track talks and dev rooms closed already? It would maybe be better to link to the calls than the submission system, which doesn’t seem to list the deadlines.
This is the CFP for the devrooms, the deadline is 6th of December. Everything is unified under pretalx this year instead of the oldoldoldoldold FOSDEM CFP system.
I thought the call for dev rooms had closed. At least some dev rooms have been announced, the dev room talk submissions are through this system (for most devrooms, a few are doing their own thing). The call for stands is also open.
Indeed devrooms cfp is long closed, and devrooms have been assigned. There are always more requests for devrooms than available rooms, so good proposals do get turned down (with another good chance next year). There is no chance of still getting a devroom.
The cfp’s from the devrooms are now going out on the fosdem mailing list: https://lists.fosdem.org/pipermail/fosdem/2024q4/thread.html
I’ve had the best experiences with conferences when doing a talk, can recommend. With fosdem, the nice thing is you can also help by organising a devroom. It’s actually a relatively easy way to organise any kind of gathering around open source topics. The fosdem organisers do most of the heavy lifting: making space available, getting everyone in one place, video recording & publishing.
Edit: To be clear, you can still submit a proposal for doing a talk at a devroom (until 30 November, schedules will be announced 15 December), but you cannot submit a proposal for organising a devroom anymore.
Nice find. I’m happy the editor I’m using (acme) makes these characters stand out. As developer, it’s nice to have x-ray vision on funny unicode business. The webmail client I’m using highlights switching between unicode scripts/blocks (originally to spot idn homoglyph attempts). I’m using the (small) JS code to do that highlighting in more of my projects. Has been helpful finding (often accidental) weird unicode whitespace characters.
In my limited experience, DNSSEC is only usable when your resolver is allowed to downgrade to insecure DNS. It just breaks too often. Strict DNSSEC is a pain.
So for me, SSHFP isn’t really much protection against MITM.
How does “it” break? Can some domains not be resolved because they have some form of brokenness in their DNSSEC setup? I’m using a DNSSEC verifying resolver on my laptop and servers, and haven’t run into any issues yet.
With extended dns errors (EDE), you get details about DNS failures. So if an issue arises, it should be relatively easy to diagnose. I am a bit surprised at how new EDE is, seems like a pretty basic requirement for diagnosing issues…
Good reminder, I’m not using SSHFP, but it’s easy enough to setup and use.
DNSSEC needs support from all resolvers so that signatures are passed along correctly and so that DS records are queried in the parent zone. There are sadly a lot of resolvers that still lack basic support for a 19-year-old standard.
Yeah, I wouldn’t rely on the resolvers received through dhcp on random networks to implement dnssec. I run unbound locally. No other resolvers should be involved then (only the authoritative name servers). A local dnssec resolver also makes it more reasonable for software (that reads /etc/resolv.conf) to trust the (often still unverified) connection to the resolver.
If a network I’m on would intercept dns requests (from unbound) towards authoritative dns servers, and would break dnssec-related records, then that would cause trouble. I just checked, and it turns out my local unbound forwards to unbound instances on servers, over a vpn (to resolve internal names). Perhaps my experience would be worse when connecting directly to authoritative name servers on random networks. On servers, I haven’t seen any dns(sec)-related request/response mangling, and would just move elsewhere when that happens.
I’m honestly not sure how it broke; that’s part of the problem. After all my time troubleshooting, I eventually decided to go back to plain-old DNS and get on with my life.
Maybe the tooling is better these days, and next time I set it up, it’ll go more smoothly.
Some OSes solvled this by having process-local temporary directories instead. Multics had them, as does IBM i (QTEMP).
Plan9 probably did too, considering the whole architecture was built around
mount(1)andbind(1).I disagree with the author somewhat. I don’t think a per-user /tmp would be an ideal fix. I think a per process group /tmp would be ideal
Correct. Plan 9 binds
$user/tmpto/tmpby default. If you want finer grained isolation, you can bind any other dir over/tmpas you desire.The thing that blew my mind with Plan9 is that the bindings are process-specific. So if you need a very specific process to have a very private /tmp, you can do that, and it changes nothing for all the other processes.
What I don’t get with that… all usages of
/tmpI’ve had to deal with are with inter-process communication stuff, so if/tmpis isolated to a process then its utility seems hindered?Unless people are thinking of a sort of layered file system based off your process tree…. but that gets you into a whole world of pain without the right abstraction
I am, indeed, talking about layering of FS namespaces: that is how Plan9 primarily operates. As for uses of
/tmp, you should really use/runinstead in most cases.In Plan 9, you would set up the namespace of your process, e.g. for logins or some system service. You could use ramfs(4) on /tmp (https://9p.io/magic/man2html/4/ramfs), or bind or mount something disk/network-backed. You wouldn’t need a variable to find the temp dir: anyone can put anything at the fixed path /tmp.
With per-user /tmp most reasonable sharing of files/sockets will keep working, without most of the risk.
Right, but the /tmp process A sees is not the /tmp process B sees, to my understanding.
This is a reasonable stop-gap solution for that problem, but using
/tmpfor files/sockets is inappropriate in the first place IMO. /tmp is not a great way to share files and in general there are better places for things like named pipes/sockets/etc. and tying their existence to the lifetime of the process group and simplifies the application code because it no longer needs something like an atexit to clean up after itself.Maybe. But if
/tmpnever existed and everything used$TMPDIRinstead, you could slice and dice to your heart’s content!You might need to find another place for per-login state, though, if named pipes and suchlike can’t go in
$HOME.Debian (and others) have libpam-tmpdir, that “sets
$TMPDIRand$TMPfor PAM sessions and sets the permissions quite tight.”Oh, interesting! The multics one sounds like it’s being used for the kinds of things that are usually
mmap(MAP_ANON)on unix. But I suppose if the process directory isn’t anonymous it can be used for file-based or shared memory IPC?Multics had single-level storage, so all those files were considered memory mapped anyways. AFAIK (I haven’t used Multics much), there wasn’t much distinction between files and heap in memory; I guess the best analogy is if all files were mmap’d on Unix - making the
MAP_ANONanalogy as well work.I’m more familar with i, which is also SLS, but persistent memory and single address space as well. On i, you can could just pass other processes pointers to things you have in QTEMP (or anywhere else, but objects in QTEMP are known by name only to that process), which can be not just files (which are also SQL tables), but also other things like data queues and user spaces (basically opaque byte arrays).
The fact that email still involves copy-pasting the entire email into the response is insane to me. How is this still the case with all this amazing technology at our disposal!? It truly baffles me…
On the other hand, you never run into the problem where a message’s context can be lost because its parents were deleted and you don’t need to perform remote lookups to find it.
Very true, and I think it has value, but there must be a better, more structured way to achieve this!
yes, this is a bit silly. you could forward & reply to messages by just attaching the entire original full message as a multipart part. and have the receiving mail client render the original message just like it would do if received directly. i would love to know how good clients are at rendering full attached messages. i still have this on the todo-list for my mail client…
writing an initial html message isn’t necessarily too crazy: use only basic styling mechanisms while editing, then generate html and send it. it starts getting a bit overcomplicated and unrealistic when people start start replying to and editing html emails. email clients (already often not the most modern piece of technology) need to be full html wysiwyg editors…
Or just don’t include the message being replied to because any non-crap client should allow easy access to the thread’s history.
Although now I am assuming people use decent clients which isn’t a good assumption.
Also inline quotes can be quite useful.
agreed. (:
though people often also do this with html messages and hilarity ensues. my favorite issue is where (some version of) outlook collapses (hides) html content when it sees anything that looks a bit quoted (indeed, how do you detect which parts of an html message are quoted and which are not?). so that includes your inline replies. they even collapse if they don’t find any other content in a message! so people think they’re getting empty messages, plenty never realize there actually was a reply.
doing this can be useful for context if you’re forwarding, or including new recipients in a reply. but indeed. most replies can just start with new empty messages.
Yeah, forwarding is a different story and may be better attached as a separate part as you suggested. I’m sure you can still come up with creative attacks based on client differences. (For example use CSS or elements that are hidden in the “middleman”‘s client but visible in the final destination’s client.) But it would likely be more robust than the quoting mechanism. Especially if you work at the same place and may be mandated to use the same client.
Top-posting won. We who assiduously edited the email we were replying to address multiple points lost the war.
When implementing s2n-quic we ended up building a very similar tool: https://github.com/awslabs/duvet.
Instead of referring to line numbers, you copy the actual text from the RFC and put it inline - see an example here. In addition to other things, the tool can generate a coverage report of the RFC that links the text to code and tests: https://dnglbrstg7yg.cloudfront.net/ac7bf906d1c883e22fd3ada80e42fe16db95b7da/compliance.html.
This practice has been very valuable for us. As a recent example, we avoided an issue that affected several other QUIC implementations. It’s also been adopted into other AWS codebases such as s2n-tls and the Encryption SDK.
Nice, very thorough, and good idea to work based on MUST/SHOULD/etc statements. Will be keeping this in mind for future development.
Have you cross-referenced internet drafts too? I can imagine it works well enough with the text-snippet approach. Outdated snippets can be detected. My line-based approach won’t work for such changing text.
Curious: have you found problems with the specs while annotating? E.g. contradictions, missing requirements.
Yes! This was incredibly useful while QUIC was still WIP and was a moving target. The main RFC had 34 revisions (!) and the protocol is spread across 4 RFCs in total. There’s also lots of references to other RFCs that we wanted to capture as well. Needless to say, it’s a very complex protocol to implement and for us duvet greatly helped in taming some of the complexity.
Good question… There’s been a few issues we’ve found in RFCs, though none in QUIC at this point.