I'm familiar with Nixery and I think it's a really cool project. It's extremely close to what I'd like but it's not quite it - it requires either relying on nixery.dev to be online (unacceptable for production considering there are no availability guarantees) or running my own instance (which essentially means that I'm maintaining a type of registry).
Why is there a need for an image registry? Part of the beauty of Nix is that Nix benefits from remote binary caches, but they are not required. Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache if possible and builds from source if not (with the caveat that production nodes should basically never be building from source)?
(Also Nixery is GCE-only and we're on AWS but leave that aside).
> which essentially means that I'm maintaining a type of registry
Mhm, there's no state that can't be thrown away and recreated, so I'd argue the overhead of running it is much lower than a full-blown registry.
> Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache
It depends on where you want to do this - Kubernetes for example has lots of opinions about images and how they're downloaded, so just replacing the runtime wouldn't be enough.
Nixery is an incremental step towards the end-goal, but there's a lot of mindset shifting that needs to happen first I think.
Not OP, but a registry works anywhere, whereas a custom runtime doesn't (managed node groups in EKS, GKE, AKS etc).
Additionally, once you use a custom runtime, now you have to deal with multiple runtimes in your cluster. You can no longer easily just run pods, you have to ensure they run on the nodes with the runtime for the images you want.
https://github.com/google/nixery
I'm the author of this, and here is a talk about it: https://www.youtube.com/watch?v=pOI9H4oeXqA