Nice work! And it’s great you were able to document it like this. The various techniques AWS uses to make the Lambda abstraction actually practical to use are fascinating. Lambda is simple enough that a relative novice can just take it for granted, but if you have a bit more experience you think “wait a minute, that could never work!”, but if you have even more experience you realize what a huge investment is below the surface for it to do what it’s doing.
No. Mostly that’s because the details of the approach (dedupe, convergent encryption, etc) don’t make a great deal of sense outside the context of large-scale cloud infra, and each large-scale infra providers needs are a bit different.
I have always been frustrated with how poorly docker image registries work.This article explains the problem and a pretty cool solution.
I think this could be generalized using the open estargz image format https://github.com/containerd/stargz-snapshotter.
ps. This is my first submission here!
Very cool to see this here. I’d be happy to answer any questions folks have about this work (and there’s a ton more detail in our ATC’23 paper https://www.usenix.org/conference/atc23/presentation/brooker).
Nice work! And it’s great you were able to document it like this. The various techniques AWS uses to make the Lambda abstraction actually practical to use are fascinating. Lambda is simple enough that a relative novice can just take it for granted, but if you have a bit more experience you think “wait a minute, that could never work!”, but if you have even more experience you realize what a huge investment is below the surface for it to do what it’s doing.
Does Fargate also use this solution? If not, would it be practical for Fargate to do something similar?
As of today, Fargate uses Seekable OCI (https://aws.amazon.com/about-aws/whats-new/2023/07/aws-fargate-container-startup-seekable-oci/). I can’t say much about the future, but I would expect Fargate container loading to get faster and faster over time.
Is there an effort to standardize this? Seems like every container invocation in the a cloud or data center could benefit from this tech
No. Mostly that’s because the details of the approach (dedupe, convergent encryption, etc) don’t make a great deal of sense outside the context of large-scale cloud infra, and each large-scale infra providers needs are a bit different.
Seekable OCI (https://aws.amazon.com/about-aws/whats-new/2022/09/introducing-seekable-oci-lazy-loading-container-images/) works in a related problem space, and is open.