I've recently moved my "personal infrastructure" from a docker-compose setup to a k3s setup, and ultimately I think k3s is better for most cases here.
FWIW, my docker-compose setup used https://github.com/nginx-proxy/nginx-proxy and it's letsencrypt companion image, which "automagically" handles adding new apps, new domains, and all ssl cert renewals, which is awesome. It was also relatively easy to start up a brand new fresh machine and re-deploy everything with a few commands.
I started down the route of using kubeadm, but then quickly switched to k3s and never looked back. It's now trivial to add more horsepower to my infrastructure without having to re-create everything (spin up a new EC2 machine, run one command to install k3s & attach to the cluster as a worker node). There's also some redundancy there, as any of my tiny ec2 boxes crashes, the apps will be moved to healthy boxes automatically. I'm also planning on digging out a few old Raspberry Pi's to attach as nodes from home (over a VPN) just for funsies.
Ultimately k8s certainly has a well earned reputation for having a steep learning curve, but once you get past that curve, managing a personal cluster using k3s is pretty trivial.
Supporting clustering is a nice plus over Docker Swarm, albeit at a cost of extra complexity and resource use, but I guess it comes down to whether YAGNI.
On another note, do you think k3s has any advantages over Docker Swarm, for the kind of small-scale setups we are discussing here?
I found k3s to be VERY noisy in logs - I definitely recommend log2ram if you want your SD card to last very long! (Or use different external storage). I had two Pi nodes with corrupted filesystems until I made the switch.
Awesome protip, thanks! I normally keep a rolling log history backed up to S3 but I'm thinking for these Pi nodes there's probably going to be literally nothing of consequence running on them, so this looks like an ideal solution!
Docker-compose isn't going to help you with Let's Encrypt, you're going to need to keep resolving that problem with each app you have or find some other way to tackle it, because you've picked a way to deploy containers, and don't have any kind of centralized cloud system at your back.
In my comments, I mention that the author could have used Kilo, which would have been a Kubernetes-native way to manage their WireGuard system, and to connect the Pi & their other systems to their existing K3S system.
I agree that docker-compose might be simpler, but there's a very very limited realm of concerns that that will ever serve, where-as Kubernetes's / the Cloud Native ambition is to manage everything you would need in your cloud. Whatever you need, should, ideally, be managable within the same framework.
DNS is another decent example, where Kubernetes will help you manage domain names, somewhat. Still work to be done there but there are some good starts. There's so many operators, all of which purport to let you manage these services in a "cloud native" way. We're still learning, getting better at it, but being able to manage all these thing semi-consistently, via the same tools, is a superpower. https://github.com/operator-framework/awesome-operators
Also just the question of short term wins vs long term use. You will not use docker-compose at your job. More and more people are going to be using Kubernetes to manage a wider and wider variety of systems & services, making more and more capabilities managed by Kubernetes.
If you run a single server, getting a Let's Encrypt cert is as easy as running a cron script. Then you just run a single instance of Nginx with the cert directory mounted as a volume. You will have to do a little extra work to maintain the nginx configuration to point to all your other containers, but it's generally just copy/pasting a block and changing the port to add a new service.
Kubernetes is cool but the only reason to run it on a single instance is because you want to.
Also docker-compose isn't used to host stuff in production very often but it is used to manage running local instances quite a bit. I wouldn't write it off as not worth learning.
Comments in this discussion are filled with similar sort of 'it's easy, just do it by hand' sort of advice.
Let's Encrypt is an example of one thing that you can either get practically for free in Kubernetes, or which you can keep fiddling & expanding your nginx configuration with, as you add domains & subdomains. Keep hoping your cron jobs are running as they need to.
But it's one example. And it's an easy example.
If I want a postgres database, it's pretty easy to set up postgres too. I can do it in under 30 minutes. But I should also set up read replicas. I should set up backups. Somehow I have to leave enough documentation so that the next time I need to operator on these systems, or if someone else does, they can follow my work & understand what's happening. I should probably set up monitoring on all these systems, so that if something goes wrong with a replica & lag raises, or if backups don't write, I get notified. I should add metrics so I can see problems as they are developing, increasing my ability to sleep soundly. There's a fractal web of concerns around running & operating software, and the classical path is to become ever more an expert on the niche. To keep diving in, trying to find your own way forward, pick your own stack of tools to help, and document document document test test test everything. Run some chaos engineering tests to make sure these systems you've strung together are going to not lose all the data if something some day does go wrong.
Kubernetes should be the default model for anyone who cares about themselves and/or running software effectively. A off-the-shelf postgres operator will let me define a Postgres resource, with as many replicas as I please, and backup schedules as I like. One can install k3s on one node or a dozen nodes, install the helm chart for a postgres-operator, create a manifest for a postgres database, and apply it, in under an hour. Whether you want a tiny small one node postgres, or to run a many replica geo-distributed postgres system, this is a better way to do it, that will yield more predictable results, that more engineers will be able to recognize & identify & work with the pieces of, than what would take me a week of time to set up. Because it flips the model. Rather than bottom-up engineering the pieces of the system, Kubernetes empowers me to define a top-down set of things I desire, and it lets the automation work to fulfill those asks. It take the grunt work out, paves the cowpaths.
And Kubernetes does it consistently. After getting a backed up, monitored, replica'ed postgres going in an hour, I can use the same skillset, the same patterns, to deploy Redis. And the next service after that.
Even if you are single node, perhaps you might want to use SQS, or S3, or soon Lambdas; maybe as a part of some JAM Stack. Well, with Kubernetes, you can manage that under the same roof too, whether you run Amazon's EKS, or your own Kubernetes at home, or any other Kubernetes. You can manage everything, consistently. https://aws.amazon.com/blogs/containers/aws-controllers-for-...
It's a different operational paradigm, and I think almost all operators would have a better experience, learning & benefiting from a top-down control system like Kubernetes. Using bottom-up engineering to deploy services seems appealing to a large amount of the seasoned engineers of today, it seems easier to them, but they don't get how powerful it is having automation at your back, they don't understand how much clearer operations are when all of your state is centralized & something you can work on with consistent tooling, and they haven't seen how much nicer it is when everything you have can be managed under one roof. They all think it's easy, learning this service, learning that service, that the services are all simple. But lacking this top down system of control, lacking something consolidated & centralized to mange everything, lacking the live-agents/controllers that are tirelessly working to keep things running, it's a huge loss. I hope future engineers can enjoy a much better constructed, much less piecemeal environment.
> Docker-compose isn't going to help you with Let's Encrypt, you're going to need to keep resolving that problem with each app you have
A very common pattern is to front containerised HTTP services with a proxy, most typically nginx, Haproxy or Traefik, and have them do TLS termination - so you only have to deal with setting up Let's Encrypt for the proxy. For Traefik, that's really simple to setup (in general I actually find Traefik quite fiddly to configure, but for Let's Encrypt it's simple). For nginx, there are numerous sidecar images available that handle it for you, such as nginx-proxy. I haven't used Haproxy, but I would imagine it's the same ease of setup.
> You will not use docker-compose at your job
I do, and I've seen it used quite on development projects, so that developers can reliably and consistently run services locally.
Traefik reverse proxy can be hosted using docker compose, which will deal with fronting each container and offloading ssl.
You can communicate between containers using hostnames. I personally have a container keep DDNS updated for my home setup.I'm not sure what else you mean by DNS.
I’ve got 5 nodes (one remote VPS, 3 Raspberry Pi’s, and a NAS at home) and I still just use Docker Compose. Compose is just way less verbose and far simpler.
The advantages of Kubernetes seem to be adding an overlay network, config and secrets management. None seem particularly useful for me though. Generally, I care which host a service I’m running is on, so tacking affinities to everything is additional work and doesn’t help with scalability. Since services are already pinned to a host, configuration is easy to manage by deploying it to the server. And, since most traffic is on the local network, an overlay is not really useful. I use a set of containers that form an SSH reverse proxy to expose some home services to remote services without exposing them to the public.
