“Set up proper monitoring” sat on my homelab backlog longer than I’d like to admit. What finally made me want to work it was another project I actually want to build, which turned out to need real observability underneath it first. So I went looking into how to do that properly on Proxmox, and found that upgrading from 8 to 9 had already handed me the best part of it. As of VE 9, the pvestatd daemon pushes its own metrics straight out to any OpenTelemetry Protocol (OTLP) endpoint, every few seconds, without anything installed on the nodes to make it happen.
So that covers performance: how the nodes, guests, and storage are doing right now. It does not tell you which guests have no backup job, whether replication is healthy, or what state the cluster thinks a VM is in, because that data lives in the Proxmox API rather than in system stats. And it stays silent when corosync loses quorum or a Ceph OSD starts flapping, because those are log events and OTLP metrics are not logs.
Proxmox observability needs three data streams, and no single product covers all three. Performance metrics arrive on Proxmox’s native OTLP push. The management plane, meaning HA state, backup coverage, and replication health, has to be read out of the Proxmox API by pve-exporter. Logs leave each node through rsyslog and land in Alloy. That’s why this post has three parts. Everything ends up in a self-hosted Grafana stack.
My Observability Stack
First I had to decide on an observability stack. There are a ton of options ranged from cloud-hosted SaaS (Datadog, Grafana Cloud, New Relic) to fully self-hosted. I wanted to go fully self-hosted which made me immediately think of running the entire Grafana observability, so that is what I did!
Here is what I deployed using Docker Compose:
- Grafana is the visualization and dashboarding layer: dashboards, Explore, and alerting. Grafana itself stores nothing. It reads from Prometheus and Loki.
- Prometheus is the metrics backend. It stores time-series metric data, provides a query language (PromQL) for it, and receives metrics via remote_write from Alloy.
- Loki is the log aggregation backend. It stores logs indexed by labels (like
host="pmox1"orjob="syslog") and makes them queryable with LogQL. Loki is intentionally lightweight: it indexes labels, not full log content. - Grafana Alloy is the collection and pipeline layer. Alloy runs on TrueNAS as a container and acts as the central hub: it receives syslog from Proxmox nodes, receives OTLP metrics from Proxmox, scrapes exporters, and forwards everything to the right backend. Think of it as the plumbing between your infrastructure and your storage.
All of this runs on-prem. No data leaves the homelab, no subscription required, and the entire stack is open source.