Primary and backup nodes
Smart Router has two tiers of upstream nodes:
- Primaries, under
direct-rpc, serve all traffic. - Backups, under
backup-direct-rpc, are an emergency fallback. A request reaches a backup only when no primary can take that request.
The tier is chosen again on every request and on every attempt inside a request. Nothing switches a router "over to backup": the next request starts with the primaries again. What does carry over between requests is the state of each node: whether it is blocked, switched off after repeated failures, or held off after a rate limit.
direct-rpc: # primaries: all traffic
- name: "own-node"
chain-id: "ETH1"
api-interface: "jsonrpc"
node-urls:
- url: "https://eth.internal.example:8545"
backup-direct-rpc: # backups: only when no primary can take the request
- name: "paid-provider"
chain-id: "ETH1"
api-interface: "jsonrpc"
node-urls:
- url: "https://eth-mainnet.paid.example/v2/KEY"
Every listener needs at least one primary for its chain and interface. A router with backups but no primaries does not start.
When a backup gets traffic
For each attempt, the router first picks from the primaries. A primary drops out of that pick for any of the reasons below. When every primary has dropped out, the attempt goes to a backup.
| A primary is... | Does that move the attempt to a backup? |
|---|---|
| already asked by this request (answered with an error, or still running) | Yes, once every primary has been asked |
| blocked, because every one of its URLs was switched off after repeated failures | Yes |
missing the add-on or extension the request needs (for example debug or archive) |
Yes, on the first attempt. See Methods only a backup serves |
behind the chain tip, for a request at latest |
Yes |
excluded by the caller with lava-providers-block |
Yes |
| not connected, or at its session limit | Yes |
rate-limited (held off after a 429) |
No. A held-off primary is skipped only while another primary is ready. When every primary is held off, the one whose hold-off ends soonest is used. See Rate-limited upstreams |
| slow | Not directly. After the per-attempt window the router asks the next primary while the slow attempt keeps running. A backup is asked only once every primary is busy on the request |
named by lava-select-provider |
No. A pinned request never goes to a backup. See Pinning |
A URL is switched off after --bench-after consecutive failures (default 50). A provider is blocked when all of its URLs are switched off. Only failures that are the node's fault count: a rate limit, an unsupported method or a caller mistake does not.
Errors: the retry limit can end a request before any backup
A request stops when it has seen more than --set-relay-retry-limit errors (default 2), counting errors from every node, rate limits included. The limit does not know a backup is still untried. So when every primary fails at the same time, a backup is reached only if the limit allows as many errors as there are primaries:
| Primaries | All fail fast (5xx, refused connection, node error), default limit 2 |
|---|---|
| 1 or 2 | the backup is asked next and can answer |
| 3 or more | the request stops after the third error. No backup is asked, and the caller gets a node's error (see What the caller gets) |
This is how the router is designed: the limit bounds how much one request costs. It changes once the failing primaries are blocked. A blocked primary is not selected at all, so later requests go straight to the backups. That takes --bench-after failures per URL.
If you run three or more primaries and want an error burst to reach the backups right away, raise --set-relay-retry-limit to at least the number of primaries. Each extra retry is one more upstream call.
A non-retryable error, such as execution reverted or a malformed request, ends the request at once on any tier. See Retry.
Quiet nodes: the window and the budget
A primary that never answers is not an error, so it does not count against the retry limit. Instead, attempts go out one per-attempt window apart, and a backup is asked after every primary has had its window:
t = 0 primary 1 (quiet, keeps running)
t = 1 × W primary 2 (quiet, keeps running)
t = 2 × W primary 3 (quiet, keeps running)
t = 3 × W backup 1 answers
For N primaries and a window W, a backup answers about N × W after the start, if that is still inside the request's budget. The default window floor is 1s (--min-relay-timeout; heavier methods get a longer window) and the default budget is 30s (--default-processing-timeout). With those defaults, for a typical read, three quiet primaries reach a backup after about 3 seconds. If you raise the window, for example to 10s, three quiet primaries use up the 30-second budget and the request ends with no backup asked.
Keep N × W below the budget. A caller can also shorten the window for one request with lava-relay-timeout.
Writes and batches
- A write (for example
eth_sendRawTransaction) is sent to every primary at once. Backups are added to that broadcast only with--stateful-to-backup, which is off by default. Without the flag, a write still goes to a single backup when no primary can be selected at all. With--strategy costthere is no broadcast: the write goes to one node. - A JSON-RPC batch gets no retry and no failover by default (
--disable-batch-request-retry). It can still be served by a backup on its first attempt, when no primary can take it. See Batch requests.
When traffic goes back to the primaries
Because the tier is chosen per request, traffic returns to a primary as soon as that primary can be selected again:
| What took the primary out | When it comes back |
|---|---|
| it was busy or failed on one request | at once: the next request starts with the primaries |
| its URLs were switched off after repeated failures | the background prober (every --probe-loop-interval, default 5s) switches a URL back on after 3 healthy probes in a row. The URL then gets a short trial: 3 new failures switch it off again, not another 50 |
| it was rate-limited | when the hold-off expires (30 seconds at first, doubling for repeated 429s), or as soon as it answers a request |
| it was behind the tip | when its block moves forward |
At each epoch boundary (--epoch-duration, default 15m) the router re-checks every node. A blocked node is let back in only when its probe passes.
Among backups, the node selection optimizer picks one per attempt, so a retry asks a different backup. While the primaries are healthy, backups get no client traffic. The exceptions are writes with --stateful-to-backup and read-your-writes reads. The prober still checks backups in the background.
Methods only a backup serves
A common setup puts an expensive archive or debug node only in the backup tier, so ordinary traffic stays on your own nodes. Declare the add-on on the backup's URL:
backup-direct-rpc:
- name: "paid-archive"
chain-id: "ETH1"
api-interface: "jsonrpc"
node-urls:
- url: "https://eth-archive.paid.example/v2/KEY"
addons: ["archive", "debug"]
The router's list of supported add-ons is built from both tiers. When no primary declares an add-on, a request that needs it skips the primaries and goes straight to a backup that declares it, on the first attempt. No primary has to fail first.
| Request | Where it goes |
|---|---|
an add-on method only the backup declares, for example debug_traceTransaction |
the backup, on the first attempt |
a read of an old block, deeper than the spec's archive rule (the router adds archive itself) |
the backup, on the first attempt |
a request with lava-extension: archive |
the backup, on the first attempt |
| an ordinary request | the primaries |
Once an attempt has gone to an archive node, the request is never downgraded. If the archive backup fails, the retry goes to another archive node, or the request ends with the archive node's error. A switched-off archive backup is lent to the request as a last chance rather than skipped.
When the archive backup is not in rotation
The router decides that archive is offered from the config file, before it checks the nodes. If the only node tagged archive is out of rotation, because it was unreachable at startup or failed its archive check, an archive request is answered by an ordinary primary. A deep-block read gets the pruned node's error or an empty result. A request with lava-extension: archive currently gets that answer without Lava-Extension-Unavailable. This is a known issue and will be fixed to set the header. Until then, run smartrouter health --include-backup after each deploy to confirm the archive backup passed its checks.
Things to know about this setup:
- The router adds
archiveonly when it can read the block. Methods that name a block or transaction by hash, such aseth_getTransactionReceiptordebug_traceTransaction, and EIP-1898 block objects are not routed to archive automatically. Sendlava-extension: archivefor those. - An archive backup still takes ordinary failover traffic. When every primary is out, a plain request can be served by any backup, including the archive one. For an add-on collection such as
debugyou can prevent that withstandalone-addons: trueon the URL: that URL then serves only its add-ons.standalone-addonshas no effect onarchive, becausearchiveis an extension of the normal API rather than a separate set of methods. - Check that the backup passed its checks. Backups are verified at startup the same way primaries are, add-on by add-on. A backup that fails its
archiveordebugcheck does not serve that add-on. Runsmartrouter health <config> --include-backup(without--include-backup,healthchecks primaries only). - When no node in either tier declares an add-on, a request for it fails with
No Providers For Addonand no node is contacted. When no node declares thearchiveextension, the request is served by an ordinary node, and a caller who asked for it withlava-extensiongetsLava-Extension-Unavailable: archiveon the reply.
What the caller gets when neither tier can serve
The reply depends on whether any node produced an answer. If one did, the router returns it with the usual response headers. If none did, the reply is a router error with no Lava-* headers. The request id is then only in the body, at error.data.guid.
These are JSON-RPC replies. REST and gRPC carry the same outcome in their own error format.
| Situation | HTTP status | Body |
|---|---|---|
| every node answered with a node error inside HTTP 200 | 200 |
the node's error, unchanged. lava-identified-node-error: true |
| every node returned HTTP 5xx | the node's status | the node's body when it is a JSON-RPC error; otherwise a router error with data.upstream_status and data.upstream_body |
| every node refused or dropped the connection | 502 |
no answer from upstream: … |
every node was rate-limited (429) |
503 |
a router error. The node's Retry-After is not forwarded |
| no node answered before the budget ended | 500 |
context deadline exceeded, no Lava-* headers |
| a write that no node answered | 500 |
transaction status unclear: …. The write may already be on chain, so check before you resend it |
| no node could be selected at all | 500 |
No pairings available., no Lava-* headers |
| an add-on no node declares | 500 |
No Providers For Addon, no Lava-* headers |
an extension no node declares (lava-extension: archive) |
200 |
an ordinary node's answer, with Lava-Extension-Unavailable |
Before it gives up on blocked nodes, the router lends one switched-off URL to the request as a last chance: blocked primaries first, then blocked backups. So when everything is blocked, the caller usually gets that node's own answer or error rather than No pairings available..
No header says that a backup served a reply. Lava-Provider-Address names the node, so to see a backup you have to know which names are backups. When the request was retried, the header lists every node asked, in order, and the node whose answer was returned is last. See Observability.
How failover works with other features
Pinning
lava-select-provider names a primary. A backup cannot be pinned: naming one fails the request with Selected provider not available. A pin also never falls through to the backups. If the pinned primary is blocked or unknown, the request fails without trying any other node. If the pinned primary was asked and returned an error, the retry goes through normal selection, which can reach a backup.
Sticky sessions
A sticky session binds a session to a primary. Backups are never bound.
- When the bound primary is blocked, or this request already tried it, the binding is dropped and the session is bound to another primary.
- When no primary is left, sticky requests are served by the backups without a binding. Two calls in the same session can land on different backups.
- With
--shared-state, a binding shared across router pods is kept even when its primary is down, and requests in that session fail withSelected provider not availableuntil the binding expires, about two epochs after it was made. Requests without the sticky header are served by the other primaries or the backups in the meantime. During such an outage a new session can also be bound to a primary that is down; this is a known issue. - After the primaries recover, the next sticky request is bound to a primary again. It may be a different one from before.
WebSocket and gRPC subscriptions
Subscriptions fail over differently from requests:
- One upstream per subscribe. A new subscription picks one upstream, primaries first. If it cannot connect to that upstream, the subscribe fails and the client gets the error; the router does not try another node within the same subscribe. Clients should retry a failed subscribe.
- Backups only when no primary is registered. A new subscription goes to a backup only when the router has no primary for the chain in rotation. A primary whose URLs are down at runtime stays in rotation for subscriptions until the next re-check removes it, which takes up to two epochs (30 minutes at the default
--epoch-duration). Until then new subscriptions keep going to it and fail, while ordinary requests are already served by the backups. - Same upstream for a WebSocket connection. Later subscriptions on the same WebSocket connection go to the upstream the connection first used, even if it has since gone down. Open a new connection to subscribe elsewhere.
- No move back. A subscription stays on its upstream for its whole life. One opened on a backup does not move to a primary when the primaries recover. If the upstream drops the connection, the router reconnects to the same upstream; if that fails, the subscription ends and the client has to subscribe again.
Cross-validation
Cross-validation draws only on the primaries. When the primaries that can be selected are fewer than the policy needs, the request is refused at once with insufficient-capacity. Backups are not added to make up the number. A policy whose max-participants equals your number of primaries is therefore refused while any one primary is blocked.
Cache and read-your-writes
- The cache is checked before any node is picked. A cached answer is served even when every node in both tiers is down (
Lava-Provider-Address: Cached). Answers from a backup are cached exactly like answers from a primary. - Read-your-writes (
--read-your-writes-window, default30s): for a short time after a write, a read that names the written transaction or its sender prefers the node that accepted the write, including a backup. It is a preference for the first attempt only, and it applies within one router pod.
Recipes
A paid provider as backup only
Put your own nodes under direct-rpc and the paid provider under backup-direct-rpc, as in the example at the top. Normal traffic, including writes, stays on your nodes. Then check:
- The retry limit. With three or more primaries, raise
--set-relay-retry-limitto the number of primaries if an error burst should reach the paid provider at once. Otherwise it is reached only after the primaries are blocked. - The window. Keep
N × --min-relay-timeoutbelow--default-processing-timeout, so quiet primaries leave time for a backup. - Writes. Leave
--stateful-to-backupoff unless writes should also go to the paid provider while your nodes are up. - Archive and debug. If the paid provider is also your only archive or debug node, those requests go to it on every call. See the next recipe.
Archive only on the backup
Declare archive (and debug, trace) only on the backup's URL, as in Methods only a backup serves. Do not set archive on a primary that is pruned. Then:
- Deep-block reads and
lava-extension: archiverequests go to the backup, and nothing else does while your primaries are healthy. - Send
lava-extension: archivefor by-hash methods and EIP-1898 block objects. - Run
smartrouter health <config> --include-backupafter a deploy, and confirm the backup passed itsarchivecheck.
Observability
| Signal | What it tells you |
|---|---|
Lava-Provider-Address on a reply |
the node that answered. Compare it with your backup names |
rpc_endpoint_total_relays_serviced{endpoint_id="<backup name>"} |
requests each backup served |
smartrouter_csm_blocked_providers |
how many primaries are blocked. When it reaches your primary count, the backups carry the traffic |
smartrouter_endpoint_serving_tier |
2 primaries registered, 1 only backups registered, 0 none. It follows registration at startup and at each epoch, not runtime failover: it stays 2 while blocked primaries leave all traffic to the backups |
smartrouter health <config> --include-backup |
whether each backup URL and add-on passes its checks |
See the Metrics reference for labels.