Have been working on my Grafana dashboard recently and am thinking about what else I could add. So far, I have:
- CPU usage and temperature
- Memory usage
- Network usage
- Backrest daily backup status
- SMART status of drives
- CPU and memory usage of containers
Anything else that you find useful? I have a few other ideas, but am curious what others have up. Is everything on one scrollable dashboard, or do you have it organized across multiple?
- grafana dashboards https://github.com/nodiscc/xsrv/tree/master/roles/monitoring/grafana/files
- victoriametrics alerting rules https://github.com/nodiscc/xsrv/tree/master/roles/monitoring/victoriametrics/files
I’d add backup age and size, not just status, like alert if the newest dump is older than ~26h or smaller than 1MB, a job can say success and still leave you a useless file. and disk fill with a threshold, not only the graph. learned that one the hard way, a table on my vps was getting a row every minute with no cleanup and one day the disk hit 100% and postgres just died for like 40 min, obviously nobody was looking at the dashboard right then lol. now 80% warns and 90% sends me an email, and uptime kuma checks the cert expiry from outside. I guess the dashboard is nice to look at but the stuff that matters should ping you
Size is a great idea, thanks. The problem with backup age is that Backrest doesn’t expose a metric for when the backup was attempted, so right now it’s just checking for failure status daily. That, of course, fails if backrest doesn’t attempt a backup for whatever reason. Chances of that happening, I don’t know, but I’ll have to look into implementing the metric (or label, I suppose) myself.
What I do for that is turn it around, instead of checking for a failure I alert on the absence of a success. backrest has hooks, so on snapshot success it pings a push monitor in uptime kuma (healthchecks.io works the same), and if no ping arrives in ~26h it alerts. That way it catches the backup that never ran, which is the one that scares me the most I guess
I have the homepage widget. Basic but does the job of CPU, memory, and storage.
I enabled docker integration by passing docker.sock to the hompage container to see per services memory usage. Nice for spotting memory hogs.
I forgot to look into that! I use glance for my homepage now, and I think they have a Grafana community widget that I could take a look at. I have a bunch of bookmarks of community widgets that I need to look at lol
I have a series of Tasmota esp32’s with Temp/Humidity/Barometric Pressure sensors around the house, attic, garage and outside. I scrape it into influxdb and have a Grafana dashboard setup.
How much does each sensor wind up costing, esp32 and all? That sounds really interesting. I think I have an esp32 laying around that I haven’t gotten to playing with yet.
I found a pack of 5 on Amazon for $20, for each of the sensors and the esp32’s… so $40 for 5. Probably cheaper in Ali or similar. The only negative is they need some calibration. The ones from Adafruit are better quality and already calibrated, but much more expensive.
I mean, I got the whole set of default dashboards, some for networking and one each for backups (k8up) and Postgres (CloudNativePG), but I don’t look at them often, usually only if something’s wrong. I also have logs available via Grafana, but I don’t have any dashboards for that, I just query what I need. If I ever have enough time and motivation, some things I’d put on a kind of overview dashboard would be in addition to your list:
- zpool health (scrub status, IO etc.)
- list of crashing pods
- list of warnings from FluxCD (e.g. application upgrade failures or invalid manifests)
- list of failing persistent volumes
I think aggregating logs would be a good choice, yeah. My homelab is distributed across 5 machines, so it’s been getting a little unruly to ssh into them all the time. I actually haven’t taken a look at premade dashboards - completely forgot those exist. I’ve just been writing the PromQL by hand to learn how it all works before adding premade things
Yeah, for most things, a premade dashboard exists, and if it isn’t fully to your liking, just customize it. Also for more professionally maintained metrics exporters like the node-exporter, there’s usually an official dashboard. But learning PromQL before relying on premade stuff is probably a good idea, so you know how to adapt them to your needs
I think it’s pretty cool in beszel you can set it up using docker so easily and pass the disks through for SMART monitoring and not have to have an extra solution.
Beszel is really cool! I was actually planning to set that up at first, but I really wanted to learn Grafana / Prometheus and have the flexibility to customize things how I wanted. That’s definitely my biggest annoyance with using Grafana - I have five machines setup, and each one has 3 containers for monitoring (cAdvisor, smartctl_exporter, node_exporter), so it got pretty tiring to put up so many containers.
- Uptime
- Days since last system update
- Days since last CVE scan
Ah, those are good ones. I’m a little too neglectful with updates, haha.