Monitoring & Observability Stack

Prometheus is pull-based — it reaches out on a schedule to scrape /metrics endpoints. Datadog is push-based — agents send data outward to Datadog's cloud.

Pull is generally better for on-prem and internal infrastructure: if a scrape target goes silent, Prometheus immediately knows it's unreachable. In a push model, a dead agent just stops sending — and depending on configuration, that silence can be ambiguous. Pull also means the monitoring system controls the collection rate, not each individual agent.

Prometheus stores data as timestamp + metric name + value + labels. The format is optimized for data collected repeatedly over time — in this case, every 15 seconds per scrape target. It stores deltas rather than full snapshots, making it far more efficient than a relational database for this use case.

Retention is configured to 30 days. Metrics older than that are automatically dropped. For long-term storage at scale, Prometheus can remote-write to Thanos or Cortex — but that's out of scope for this deployment.

PromQL is Prometheus's query language. It's what powers both the Grafana dashboards and the alerting rules. Key patterns used in this deployment:

# Instant vector — current health of all nodes
up{job="proxmox-nodes"}

# Rate of change over 5-min window (required for counters)
rate(node_cpu_seconds_total[5m])

# Inline math — memory usage %
100 - ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)

The critical concept: many metrics are counters — they only ever go up (like total CPU seconds). You must use rate() to get meaningful per-second values. Graphing a raw counter just produces a monotonically increasing line.

node_exporter uses a one-endpoint-per-host pattern — each of the 3 Proxmox nodes runs its own exporter, and each is listed as a separate scrape target in prometheus.yml.

pve-exporter works differently: a single exporter instance proxies requests to all three Proxmox nodes via the REST API. Prometheus passes the target node as a URL parameter when scraping. This is parametric scraping — one exporter, multiple targets, driven by query parameters rather than separate endpoints. Understanding this distinction was necessary to configure the scrape jobs correctly.

Grafana alerts have a For: duration — the condition must be true for that long before the alert fires. This prevents false alarms from a single missed scrape or a momentary spike.

Critical alerts (node down, instance down) have a 2-minute pending period — fast enough to matter, long enough to filter out a single missed scrape. Warning alerts (disk, memory) use 10 minutes because a brief spike to 91% RAM doesn't warrant a notification.

Grafana sends alert notifications to Discord via webhook — a URL that accepts HTTP POST requests containing JSON. No user authentication required; the secret is embedded in the URL itself.

The same pattern works with PagerDuty, Slack, and Microsoft Teams — Grafana formats the payload, the receiving service handles display. Swapping Discord for something else is just a URL change.

Observability has three distinct data types, each answering a different question:

• Metrics (Prometheus) — "Is something wrong?" Numeric, low volume, highly efficient to store and query. Best for detecting that a problem exists.
• Logs (Loki) — "Why is it wrong?" Text events, high volume. Best for understanding the sequence of events that led to a problem. Loki is deployed but Promtail log shipping is Phase 2.
• Traces (Jaeger/Tempo) — "Where in the request chain did it fail?" Distributed request paths across microservices. Out of scope for this deployment — relevant for application-layer observability.

Most production incidents start with a metrics alert, then a log investigation. Traces become important when you're debugging distributed systems with multiple services calling each other.

docker-compose.yml is a definition, not the current reality. The running container reflects whatever was set when it was last started. Editing the file has no effect on a running container until it's recreated.

Always verify the running state directly rather than assuming the file matches:

docker inspect grafana | grep ROOT_URL

This came up when GF_SERVER_ROOT_URL was corrected in the compose file but the running container still had the old value. The container had never been restarted after the edit. Fixed with docker compose up -d grafana, then verified with docker inspect.

What Went Wrong:

The cat > file << 'EOF' heredoc syntax was unfamiliar. Wasn't clear what each operator was doing or whether the file was actually being written.

The Explanation:

cat > file << 'EOF'
...content...
EOF

> redirects cat's output into the file. << 'EOF' defines the input block — everything between the two EOF markers becomes cat's stdin. The result is identical to opening the file in a text editor. Both approaches produce the same file.

Lesson Learned:

Heredocs are standard shell syntax for writing multi-line content to a file without opening an editor. Common in scripts, common in runbooks, worth knowing cold.

What Appeared:

curl: (23) Failure writing output to destination

The Problem (and Why It Wasn't One):

This error appeared when running curl http://localhost:9100/metrics | head -20. It looks alarming but it's not an error — head closes the pipe after 20 lines while curl is still actively sending the full metrics payload. When the read end of the pipe closes, curl sees a broken pipe and exits with code 23.

node_exporter was healthy and returning data. The command worked correctly.

Lesson Learned:

Exit code 23 from curl is "Write error" — it means the destination stopped accepting data. In the context of piping to head, that's expected behavior. Distinguish between "curl couldn't connect" (codes 6, 7) and "curl was told to stop" (code 23).

What Happened:

The Instance Down alert rule used up == 0 as its PromQL query. During testing with all targets healthy, the alert preview showed "No data" — confusing because it made the rule look broken.

The Distinction:

up == 0 is a PromQL filter — it only returns time series where the value is exactly 0. When all instances are healthy (all up values are 1), the filter returns an empty result set. That's "No data" — not an error, just no matching series.

The fix: use up as the query and let Grafana's alert condition handle the threshold (IS BELOW 1). The query returns all instances; the condition evaluates them. These are separate concerns.

Lesson Learned:

PromQL filtering in the query and alert threshold evaluation in the condition are two different layers. Mixing them causes exactly this problem. Use the query to select the right time series; use the condition to define what "alerting" means.

What Happened:

Discord alert notifications contained the wrong Grafana URL — pointing to an old hostname instead of grafana.seggsy.co. The GF_SERVER_ROOT_URL environment variable in docker-compose.yml was already correct.

Root Cause:

The Grafana container was never restarted after the environment variable was updated. The running container still reflected the old value from its initial startup. Editing docker-compose.yml does nothing to a running container.

The Fix:

# Recreate container with updated env var
docker compose up -d grafana

# Verify the running container has the correct value
docker inspect grafana | grep ROOT_URL

Lesson Learned:

Always verify environment variables against the running container, not the compose file. The compose file is the desired state; docker inspect shows the actual state. This pattern — desired config vs. running reality — applies everywhere in infrastructure.