Observing a Smart Grid with NATS Monitoring and Prometheus

The scenario

A utility company streams smart-meter readings (usage, outage signals) through NATS. Operators need to trust the messaging layer itself — not just the application — is healthy: is the server up, are connections churning unexpectedly, is a subject backing up? This is what NATS's own monitoring surface (distinct from the smart-meter data itself) is for.

Step 1 — start NATS with monitoring enabled (it's on by default with `-m`)

terminal 1

docker run -d --name nats-server -p 4222:4222 -p 8222:8222 nats:latest -m 8222

terminal 2 — raw JSON, right away, no extra tooling

curl -s http://localhost:8222/varz | head -c 400
curl -s http://localhost:8222/connz
curl -s http://localhost:8222/subsz

/varz — server-wide stats: uptime, CPU/mem, total bytes in/out.
/connz — every current client connection.
/subsz — every current subscription and its subject.

Step 2 — simulate meter traffic

terminal 3 — a dashboard subscriber

nats sub "grid.meter.*.reading"

terminal 4 — several meters reporting

for i in $(seq 1 20); do
  nats pub "grid.meter.$i.reading" "{\"kwh\": $((RANDOM % 10))}"
done

terminal 2 — confirm the subscription count and message counters moved

curl -s http://localhost:8222/connz | grep -E '"in_msgs"|"subscriptions"'

Step 3 — scrape-able metrics with the Prometheus exporter

Raw JSON is fine for a one-off check, but operators need history, alerting, and dashboards — the official exporter translates :8222's endpoints into Prometheus's text format:

terminal 5 — run the exporter alongside the server

docker run -d --name nats-exporter \
  --link nats-server \
  -p 7777:7777 \
  natsio/prometheus-nats-exporter:latest \
  -varz -connz -subz -jsz=all http://nats-server:8222

terminal 2 — confirm Prometheus-format metrics are exposed

curl -s http://localhost:7777/metrics | grep gnatsd_varz_connections

Step 4 — the metrics that actually matter for grid telemetry

Metric	What it tells an operator
`gnatsd_varz_connections`	Sudden drop → meters/gateways disconnecting (network issue, cert expiry)
`gnatsd_varz_slow_consumers`	A subscriber can't keep up and is being disconnected — dashboard or ingestion lag
`gnatsd_varz_in_msgs` / `out_msgs` rate	Sanity-check expected meter reporting volume against actual — a silent drop suggests upstream meters stopped reporting, not a NATS problem
`gnatsd_connz_*` per-connection	Identify exactly which client (by IP/name) is generating unexpected load

Step 5 — JetStream-specific health, if meter readings are captured in a stream

If readings are also captured for billing/history (see the stock-price and fleet-tracking articles for the JetStream pattern), also monitor stream-specific health:

terminal 2 — stream lag / consumer health

curl -s http://localhost:8222/jsz?streams=true

Watch each consumer's num_pending/num_ack_pending — a steadily growing num_ack_pending means a downstream billing worker is falling behind or stuck, distinct from a NATS server problem.

Troubleshooting

Symptom	Likely cause	Fix
`/varz` unreachable (`curl: (7) Failed to connect`)	Monitoring port not enabled, or a firewall blocks `8222`	Confirm the server was started with `-m 8222` (or `http_port: 8222` in config) and the port is published/reachable
`slow_consumers` count keeps climbing	A subscriber (e.g. the dashboard) is too slow to keep up with the publish rate and NATS disconnects it to protect the server	Speed up or scale out the slow subscriber (e.g. queue group it), or reduce publish rate/batch size, rather than ignoring the disconnects
Prometheus shows a gap in scraped metrics	Exporter container down/restarted, or Prometheus scrape config pointing at the wrong target	Check `docker ps`/`docker logs nats-exporter`, and confirm Prometheus's `scrape_configs` target matches the exporter's actual host:port
`num_ack_pending` grows steadily on a JetStream consumer	Downstream worker (e.g. billing) is stuck or too slow, not acking fast enough	Investigate the specific consumer's worker process — this is an application-level backlog, not a NATS outage

Recap

Tool	Answers
`/varz`, `/connz`, `/subsz`	Quick, no-extra-tooling snapshot of server/connection/subscription health
`prometheus-nats-exporter`	Turns those endpoints into scrapable, alertable, historical metrics
`slow_consumers`	The single most important early-warning metric for a subscriber falling behind
`/jsz` (JetStream)	Stream/consumer-specific lag, separate from general server health

The scenario#

Step 1 — start NATS with monitoring enabled (it's on by default with -m)#

Step 2 — simulate meter traffic#

Step 3 — scrape-able metrics with the Prometheus exporter#

Step 4 — the metrics that actually matter for grid telemetry#

Step 5 — JetStream-specific health, if meter readings are captured in a stream#

Troubleshooting#

Recap#