I wait for something to break. Then I yell “God fucking dammit, I don’t have time for this right now!” and spend an hour triaging things before I just try docker down, pull, up and everything works.

Ansible is one way to “code local, push to remote(s)”. Can define so-called playbooks, which is basically just a script, to do reoccuring tasks like e.g. updates.

It’s good to get comfortable in the command line, including over ssh.

Especially considering ai and documentation along the way?

If I am going to ask AI or Google about something, I just do that to help me find the answer and then apply the answer myself. That way I learn, and can double check the AI isn’t hallucinating, at least on in obvious ways.

As for documentation, I keep a notes folder with detailed notes on manual configuration I’ve done, how and why, and the things I’ve learned along the way. I’ve found it’s both useful to help remember the things I’ve learned, and it is useful to go back to refer to.

I ssh into it twice a year to perform updates but otherwise i leave it alone. Server’s been chugging along for like a decade now.

Depends what you’re trying to do, really

If you’re talking about going in and restarting containers and checking logs, there are tools like Portainer that give a nice UI you can quickly use from your phone

If you’re talking about repetitive tasks, sometimes there’s a tool to automate it already (e.g. updates via watchtower, unattended-upgrades), and if not, you can write a script and possibly run it via a cronjob if that makes sense

If you’re talking about doing new stuff, you either stick with the SSH & shell commands, or you look into IaC tools like ansible as another reply mentioned. Another option is something like Nix where the setup is defined in config, so you make your changes there

self hosted gitlab. each server has a repo with all the docker configs and application configs. also have any scripts for the server itself on there as well, things like required libs, network configs, etc.

I also have a repo dedicated for let’s encrypt SSL that retrieves new certs every month. then on each server is an install script scheduled that pulls the certs down, installs them, and restarts any services automatically.

should anything go wrong, I have a siem monitor that will alert me that a service failed to start etc.

currently running four servers like this with varying degrees of complexity.

Ansible to deploy services in docker compose stacks and backups with resticprofile.

Tailscale service configs for HTTPS ingress and DNS resolution. Podman quadlets / podlets for everything except backup software and tailscale itself. I use the open version of VS code to edit config files in place, but I push them to a private git as well as having them backed up. Storage for app data, media, and configs (everything but OS) is via NFS shares from my gaming PC’s RAID volume. I can run a plenty of apps on an old thinkpad this way. After you configure the first few quadlets with tailscale, it gets real straightforward, but there is a learning curve. Yes, I use SSH, but it is the remote conmection and terminal in Code.

I deploy with ansible keys and install ssh with a keys-only config and only ansible access.

Then I run some standard installs and configs with ansible and all future updates to apt, apk and docker are done with one ansible playbook.

I don’t include the ansible host itself in the automation, nor my workstation, just to prevent everything from being broken at once if something goes wrong.

Around 50-50 between SSH and Portainer (although I’ve been meaning to migrate from Portainer to Arcane so I can have my compose files in git)

Maybe a bit more on the overkill side, but I use Talos Linux (declarative K8s distribution), then Terraform for initial setup and FluxCD for everything else (including VMs via Kubevirt). It’s a hell of an initial learning curve, but afterwards I just do my talosctl upgrade and upgrade-k8s from time to time, and don’t have to worry about anything else. Upside is, Kubernetes provides a unified, extensible API for everything, for example:

• reverse proxy via Ingress or Gateway-API
• firewall via NetworkPolicy
• even databases like Postgres through an Kubernetes Operator like CNPG, with a similar simplicity as with the big cloud providers (just a single yaml file with the specs like storage capacity, CPU and memory limits, backup target and schedule etc.)
• it is very scriptable (everything is managed via the API)
• automated image updates via either FluxCD itself or just dependabot/RenovateBot creating PRs to your GitOps repo

Downsides:

• you have to figure out persistent storage (CSI), which is slightly more complicated than just a single filesystem and manually specifying volume mounts like with docker, but if done right, you have a simple interface with powerful capabilities via the Kubernetes API
  • there are simple CSI implementations that just expose node local disks via LVM or ZFS, so if you just have a single node, or don’t care about replication, it’s fairly easy to get started
• initial learning curve is quite high, especially if you have no prior experience with container orchestration in general
• definitely overkill if you just want something simple that “just works”

I created git repos on my main workstation for each homelab server/service I maintain that keeps:

• documentation
• notes
• lessons learned
• scripts, configs
• runbooks
• backup details
• security audit details
• log items that need attention
• infrastructure

I just point a local LLM (offline model that runs on my workststion) into those repos and ask it to perform certain things on those servers. It can do things like update packages, install packages, make config changes, set/check permissions, read logs (and fix errors in real time), and check the health of the overall system.

I have it run pre backups before making changes, then post backups once its done.

Once changes are in place and everything is running okay, I ask it to update documentation in the repo and tag the release.

I use opencode that connects to a llama.cpp service. opencode lets me gate the AI so that any elevated commands that it needs to run (e.g. sudo or ssh), I have to approve it. It cant just go around making changes without permission.

I use Gitlab for all configurations ans then I have a Gitlab runner in my K3S cluster so I can deploy locally for free

set the server as git remote and push there directly. set hooks to deploy. I do that for a static site. Is my little iac solution. Search for alternatives to “infrastructure as code”. It can go from simple dockerfile to swarms or kubernetes…

It depends on what I’m doing I suppose. I run my main VM on Proxmox, so I mainly use its web interface for any VM work. For all the containers within, I’ve been really loving Dockhand lately, it’s just as easy to use as Portainer, it even checks for updates for me AND gives me a link for the release notes so I can decide whether to upgrade or not.

Most other things like file transfers or editing my Caddyfile I do through SSH. Can’t beat simplicity sometimes!