Multi-Region Telecom Messaging with NATS Clusters and Leaf Nodes

The scenario

A telecom operator runs infrastructure in multiple regions (US, EU). Each region needs its own local NATS cluster for low-latency, resilient local messaging — a region shouldn't go dark just because another region's link is down — but some events (billing reconciliation, cross-region roaming handoffs) must still reach subscribers anywhere.

Step 1 — a 3-node cluster, one region (US)

nats-us.conf

port: 4222
http_port: 8222
cluster {
  name: us-cluster
  port: 6222
  routes: [
    "nats-route://nats-us-2:6222"
    "nats-route://nats-us-3:6222"
  ]
}

terminal 1 — 3 containers on a shared docker network, config per node

docker network create nats-net
docker run -d --name nats-us-1 --network nats-net -p 4222:4222 \
  -v "$(pwd)/nats-us.conf:/etc/nats/nats-server.conf:ro" nats:latest -c /etc/nats/nats-server.conf
# repeat for nats-us-2 / nats-us-3 with matching per-node config (routes pointing at the other 2)

terminal 2 — verify the cluster formed

curl -s http://localhost:8222/routez | grep '"num_routes"'

A client publishing to nats-us-1 and a subscriber on nats-us-3 see each other instantly — the cluster replicates routing internally, so clients don't need to know or care which node they connect to.

Step 2 — the EU cluster, fully independent

Repeat Step 1 for eu-cluster, on its own docker network/hosts. At this point, US and EU are two completely separate NATS deployments — a message published in US never reaches an EU subscriber, which is intentional: regions stay isolated and resilient to each other's outages by default.

Step 3 — bridging the two with a leaf node

A leaf node lets one deployment extend specific subjects into another without merging the two clusters into one (which would create a much larger blast radius for any single failure):

nats-us-1-leaf-hub.conf (excerpt, added to one US node)

leafnodes {
  port: 7422
}

nats-eu-1-leaf.conf (excerpt, added to one EU node)

leafnodes {
  remotes: [
    { url: "nats-leaf://nats-us-1:7422" }
  ]
}

terminal 3 — restart the two nodes with leaf config layered in, then verify

curl -s http://localhost:8222/leafz

Step 4 — prove cross-region delivery works only for what's needed

terminal 4 — an EU subscriber, on the EU cluster

nats sub "roaming.handoff.>" --server="nats://localhost:4222" # EU node

terminal 5 — a US publisher, on the US cluster

nats pub roaming.handoff.eu-subscriber-42 '{"event":"customer roamed to EU tower"}' --server="nats://localhost:4222" # US node

The EU subscriber receives it — the leaf node forwarded the subject across the region boundary — while purely local subjects on either side (e.g. us.internal.diagnostics) stay confined to their own cluster unless explicitly configured to cross the leaf link too.

Troubleshooting

Symptom	Likely cause	Fix
Cluster nodes never form a cluster (`num_routes` stays 0)	`routes` in config point to the wrong hostnames/ports, or nodes can't reach each other on the docker network	Confirm all nodes share the same `--network`, and that `cluster.routes` on each node lists the other nodes correctly
Leaf node never connects (`/leafz` shows nothing)	Firewall/port mismatch on `leafnodes.port`, or `remotes.url` typo	Check both sides' logs (`docker logs`) for leaf connection attempts/errors, and confirm the port is reachable across regions
Cross-region subjects don't arrive	Leaf node link is up, but neither side actually publishes/subscribes to a subject that needs to cross it	Leaf nodes forward based on actual subscription interest — confirm a subscriber genuinely exists on the receiving side for that subject
One region's outage affects the other	Leaf node reconnect logic aside, this typically means the "regions" aren't actually separate clusters (e.g. accidentally configured as one big cluster via `cluster.routes` instead of a leaf node)	Double check region boundaries use `leafnodes`, not `cluster.routes`, between regions — clustering is for nodes within one resilience domain

Recap

Concept	Takeaway
Cluster (`cluster.routes`)	Multiple nodes acting as one logical server within a region — resilient to a single node's failure
Leaf node (`leafnodes`)	Bridges two independent deployments for specific cross-boundary subjects, without merging their failure domains
Region isolation by default	A leaf link down doesn't take down either region's local messaging
`/routez`, `/leafz`	The monitoring endpoints to confirm clustering/leaf topology is actually as configured

The scenario#

Step 1 — a 3-node cluster, one region (US)#

Step 2 — the EU cluster, fully independent#

Step 3 — bridging the two with a leaf node#

Step 4 — prove cross-region delivery works only for what's needed#

Troubleshooting#

Recap#