Retry
When an upstream returns a retryable error, Smart Router rotates to a different node and tries again. The pool is filtered to nodes that haven't already failed for this relay; the RPC Node selection policy picks the next candidate.
What's retryable
The error classifier (protocol/common/error_registry.go) assigns every failure a coded classification with a retryable flag. The flag is the primary signal: a non-retryable classification short-circuits retries for every terminal error — connection failures, malformed requests, and chain-level rejections alike.
| Layer | Examples | Behaviour |
|---|---|---|
PROTOCOL_* (connection / session) |
network timeout, connection reset, node unavailable | mostly retryable — rotate node |
NODE_* (node response) |
5xx upstream, rate limited, node syncing → retryable; method not found / unimplemented → terminal | per-code; unsupported methods are zero-CU and cached |
CHAIN_* (execution / state) |
eth_call reverted, out of gas, nonce too low → terminal; block/tx not found, state pruned → retryable |
per-code |
USER_* (bad request) |
malformed JSON, invalid params, invalid address | always terminal; charges normal CU |
The full code tables and per-code retryability are in Error codes.
Same-response retries are deduplicated: if two nodes return the identical response, Smart Router won't burn a third attempt looking for a different answer. The dedup is a hash cache in protocol/lavaprotocol/relay_retries_manager.go.
Limits
| Limit | Value | Notes |
|---|---|---|
| Error-retry limit | --set-relay-retry-limit (default 2) |
Errors tolerated before the relay gives up. 0 disables retries entirely. This is the knob you tune. |
| Hard attempt ceiling | 10 |
Hardcoded constant MaximumNumberOfTickerRelayRetries — an upper bound on total attempts including ticker-driven hedges, separate from the error-retry limit and not exposed as a flag. |
| Overall budget | --default-processing-timeout (default 30s) |
Ends retries even if attempts remain. |
| Per-attempt budget | --min-relay-timeout floor, or lava-relay-timeout header |
Retries get the same per-attempt timeout. |
The cap you actually control is --set-relay-retry-limit: the error-retry path stops
after that many errors (default 2). The hardcoded 10 is only the ceiling the
ticker/hedge path can reach — most relays stop far sooner.
Turning retries down or off
Retries are on by default, tolerating --set-relay-retry-limit errors (default 2). Change that with the flag:
--set-relay-retry-limit 0— disable retries: the first error surfaces immediately.--set-relay-retry-limit 5— tolerate more errors before giving up.
This is a global startup flag, not a per-relay control — there's no per-request header to disable retry for a single call.
When retries don't help
Some classes of failure look retryable on the wire but won't recover by switching nodes — for example:
- A bad request (malformed JSON, unknown method) — always terminal.
- A consensus-level error returned by every healthy node (the chain itself rejected it).
- A method your nodes genuinely don't support.
The classifier handles these cases without burning the budget.
Rate-limited upstreams
A 429 is retryable — the relay rotates to another node like any other retryable
error — but the node that said it is treated differently from one that failed. A
rate-limited node is healthy and busy, so the attempt is scored as neither a failure nor
a success (no availability or latency sample), and the node is held off: selection
skips it for the rest of this relay and for later relays until the hold-off expires.
- How long. If the upstream sent
Retry-After, the hold-off is at least that long (capped at 1h). Otherwise it starts at 30s and doubles on each consecutive429from the same URL, capped at 30m. Up to 20% jitter is added so a fleet held off by the same vendor does not come back in one burst. - Account-wide caps. When two different URLs of the same provider are held off at once, the whole provider is held off on every chain — a vendor cap is usually per-account, and per-URL hold-offs alone would keep hammering the account through its other chains.
- Any answer clears it. Once the node answers a request — success or a genuine error — its hold-off and strike count are dropped.
- You still get an answer. If every candidate is held off, the one that expires
soonest is used anyway; the router never synthesizes a
429to the client, andRetry-Afteris never forwarded. Alava-select-providerpin and existing sticky sessions bypass the hold-off — an explicit ask outranks it.
The same hold-off covers spec re-verification, recovery probes, and WebSocket subscriptions, so a node that said stop is not re-probed on a fixed cadence either.
Pinning to one node
The lava-select-provider header pins the request to a specific upstream. If that upstream fails, retry kicks in on the rest of the pool — pinning isn't a way to disable retry. See Directives.
Observability
| Metric | Meaning |
|---|---|
smartrouter_retries_total |
retry attempts triggered (beyond the first try) |
smartrouter_retries_success_total / smartrouter_retries_failed_total |
retried requests that succeeded / failed |
smartrouter_retry_attempts |
histogram of attempts per retried request (buckets 1…10) |
smartrouter_rate_limit_holdoffs_total |
hold-off events by provider and event (recorded / escalated / cleared) — the signal that an upstream is rate-limiting you |
smartrouter_rate_limit_holdoff_seconds |
histogram of applied hold-off durations, per provider |
| Tracing | each retry attempt is a span; correlate via the trace ID in response headers |
See the Metrics reference for labels and types, and Rate-limit hold-off for the hold-off pair.