Skip to content

Metrics

Every metric the Smart Router exposes over Prometheus, with its type, labels, and meaning. Metrics are defined under protocol/metrics/, except the rate-limit hold-off pair, which the registry that owns the events emits itself from protocol/holdoff/metrics.go.

The classified-error counter smartrouter_errors_total is broken down by error name and category; see Error codes for the full taxonomy of names it can carry.

Already running the dashboard overlay? It ships pre-built panels over these series — this page is the underlying reference and the recipes for rolling your own.

What to watch

If you only graph a handful of things, graph these:

Question Metric(s)
Is the router up and serving? smartrouter_overall_health, smartrouter_total_relays_serviced
Are requests failing? smartrouter_total_errored, smartrouter_errors_total (by error_category / retryable)
How fast is it? smartrouter_end_to_end_latency_milliseconds (histogram → p50/p95/p99)
Is a node degrading? rpc_endpoint_overall_health, rpc_endpoint_end_to_end_latency_milliseconds, rpc_endpoint_latest_block
How much load does the router itself put on my node? rpc_endpoint_tracker_requests_total (by kind) — the router's own polling, separate from the relays you sent it; rpc_endpoint_tracker_gate_skips_total (by source) — the polls it chose not to send
Is failover working hard? smartrouter_retries_total, smartrouter_hedge_total
Is an upstream rate-limiting us? smartrouter_rate_limit_holdoffs_total (by provider / event), smartrouter_rate_limit_holdoff_seconds
Is the cache earning its keep? smartrouter_cache_success_total / smartrouter_cache_requests_total
What do batched calls look like? smartrouter_requests_total{method=~"batch:.*"}, smartrouter_batch_size

Querying — PromQL recipes

Copy-paste starting points (adjust labels / windows to taste).

# Request error ratio (last 5m)
sum(rate(smartrouter_total_errored[5m]))
  / sum(rate(smartrouter_total_relays_serviced[5m]))

# p99 end-to-end latency, per chain (ms)
histogram_quantile(0.99,
  sum by (spec, le) (rate(smartrouter_end_to_end_latency_milliseconds_bucket[5m])))

# Cache hit ratio (both tiers; add cache_tier="primary" to scope it)
sum(rate(smartrouter_cache_success_total[5m]))
  / sum(rate(smartrouter_cache_requests_total[5m]))

# Is the secondary cache broken, or just cold?
sum by (outcome) (rate(smartrouter_cache_failed_total{cache_tier="secondary"}[5m]))

# Retry rate per relay (how often the first attempt isn't enough)
sum(rate(smartrouter_retries_total[5m]))
  / sum(rate(smartrouter_total_relays_serviced[5m]))

# Non-retryable (terminal) errors only — usually client/chain problems, not infra
sum by (error_name) (rate(smartrouter_errors_total{retryable="false"}[5m]))

# Nodes currently unhealthy
rpc_endpoint_overall_health == 0

# Per-node request share (is selection lopsided?)
sum by (provider_address) (rate(smartrouter_requests_total[5m]))

# Upstreams currently tripping their rate-limit caps
sum by (provider) (increase(smartrouter_rate_limit_holdoffs_total{event="recorded"}[15m])) > 0

# Busiest batch shapes, and how big those batches run
topk(10, sum by (method) (rate(smartrouter_requests_total{method=~"batch:.*"}[5m])))
histogram_quantile(0.95,
  sum by (spec, le) (rate(smartrouter_batch_size_bucket[5m])))

# Error ratio per batch shape (which shapes are failing?)
sum by (method) (rate(smartrouter_requests_failed_total{method=~"batch:.*"}[5m]))
  / sum by (method) (rate(smartrouter_requests_total{method=~"batch:.*"}[5m]))

Suggested alerts

Starting points — tune thresholds to your traffic.

Alert Expression sketch Why
Router unhealthy smartrouter_overall_health == 0 for 1m No endpoint is serving.
High error ratio error-ratio recipe > 0.05 for 5m Something broke upstream or in config.
Latency regression p99 recipe > 2000 for 10m Tail latency degraded.
Node down rpc_endpoint_overall_health == 0 for 5m A configured upstream is out.
Cache cold hit-ratio recipe < 0.2 for 30m Cache misconfigured or bypassed — cost/latency risk.
RESP cache backend unreachable smartrouter_resp_cache_connected == 0 for 5m Redis/Valkey is down or rejecting auth. Relays keep succeeding via upstreams, so nothing else alerts — but you are paying for every read.
Post-finality divergence increase(smartrouter_cross_validation_mismatch_total{finality="finalized"}[15m]) > 0 A node disagreed on finalized data — high-signal correctness alert.
Vendor-wide rate limit increase(smartrouter_rate_limit_holdoffs_total{event="escalated"}[1h]) > 0 Two of a provider's URLs were held off at once, so the router stopped asking that provider on every chain — you are over an account-wide cap, not a per-endpoint one.
Batch shapes merging increase(smartrouter_batch_signature_overflow_total[1h]) > 0 Clients send more batch shapes than the breakdown can name — shapes are collapsing into batch:other.

Exposition

Metrics are served in Prometheus text format from an HTTP server started by the metrics manager.

Path Format Description
/metrics Prometheus All registered metrics (promhttp.Handler())
/metrics/overall-health text 200 Health status OK if ≥1 endpoint is healthy, else 503 Unhealthy. Fail-closed at boot: turns 200 once endpoint setup has verified a provider, and health transitions publish immediately. While a chain is unhealthy the router re-probes it every --relays-health-unhealthy-interval (default 15s).
/metrics/health-overall text Alias of the above (backward-compat path)
/readyz text Alias of /metrics/overall-health for a Kubernetes readiness probe — pulls the pod from the Service without restarting it.
/livez text Always 200 while the process serves HTTP — liveness only; provider health is deliberately not consulted.

Configuration

Flag / env Default Effect
--metrics-listen-address disabled Address to expose Prometheus metrics on, e.g. :7779 or localhost:7779. The literal disabled turns the metrics server off entirely.
--optimizer-qos-sampling-interval 1s How often the optimizer-QoS sampler refreshes the rpc_optimizer_selection_score gauge and — when --usage-otel-enabled is set — emits optimizer_qos events to the OTel usage pipeline.
# enable, then scrape
smartrouter ... --metrics-listen-address ":7779"
curl http://localhost:7779/metrics

The flag name and the disabled sentinel live in flags.go.

Optimizer scores are always on

The optimizer-QoS client is created unconditionally, so rpc_optimizer_selection_score is populated on /metrics regardless of telemetry config. Remote shipping of these reports flows through the OTel usage pipeline (--usage-otel-enabled), not a dedicated push address.

Conventions

  • Latency histograms all share the same buckets, in milliseconds (buckets.go): 1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000.
  • Attempt histograms (retry_attempts, hedge_attempts) use integer buckets 1…10.
  • Common labels:
    • spec — chain spec id (e.g. ETH1, LAV1).
    • apiInterfacejsonrpc, tendermintrpc, rest, grpc.
    • endpoint_id — the configured upstream RPC endpoint.
    • provider_address — node the relay was routed to.
    • method — RPC method name. Batch requests carry a batch signature instead, e.g. batch:eth_call+eth_getBalance.
    • function — the RPC method / REST path the relay invoked, as carried by the latency histograms and relay counters (same values as method; sum by (...) over it gives the aggregate). Requests to methods outside the spec appear as Default-<method>.
  • Boolean gauges encode 1 = true / healthy / present, 0 = false / unhealthy / absent.
  • Protocol version gauges encode major*1e6 + minor*1e3 + patch.
  • Metrics are registered with registerOrReuse, which returns the already-registered collector instead of panicking on a duplicate — so re-registration (e.g. across test runs sharing the default registry) is safe.

Counting semantics — relays vs client requests

Verified against a live router with a controlled load; these distinctions matter for any dashboard or alert built on the request counters:

  • smartrouter_requests_total counts relays, not client requests. A cross-validated request increments it once per participant, cache-served requests appear under provider_address="Cached", and the router's own chain-tracker / verification traffic lands in it even with zero client load. Use it for per-provider and relay-level views.
  • smartrouter_end_to_end_latency_milliseconds_count increments exactly once per client request (cache hits included, internal probes excluded) — it is the honest source for "requests served" and RPS, and its function label doubles as the per-method breakdown.
  • requests_success_total is transport-level success. An upstream that answers with a JSON-RPC error object (invalid params, method not found, execution reverted) still counts as a successful relay — the error is recorded separately in node_errors_total and the classified smartrouter_errors_total. total − success therefore measures transport/routing failures, not "callers who got an error".
  • Consistency counters: consistency_total = reads that enforced a minimum seen block, consistency_success_total = checks that passed, and consistency_failed_total = stale responses actually caught. Lazily-registered families (retries, hedge, cache, cross-validation, ws, classified errors) are absent from /metrics until the feature first fires — an absent family means zero events, not missing instrumentation.

Smart Router metrics

These are the metrics specific to running as a Smart Router, defined in smartrouter_metrics_manager.go. They split into endpoint-scoped (rpc_endpoint_*) and router-scoped (smartrouter_*) families.

Endpoint-scoped — rpc_endpoint_*

Metric Type Labels Description
rpc_endpoint_total_relays_serviced Counter spec, apiInterface, endpoint_id, function Relays successfully served by this endpoint.
rpc_endpoint_total_errored Counter spec, apiInterface, endpoint_id, function Errored relays for this endpoint.
rpc_endpoint_requests_in_flight Gauge spec, apiInterface, endpoint_id, function Relays currently in flight to this endpoint.
rpc_endpoint_end_to_end_latency_milliseconds Histogram spec, apiInterface, endpoint_id, function End-to-end latency per function for this endpoint.
rpc_endpoint_overall_health Gauge spec, apiInterface, endpoint_id Endpoint health (1 healthy / 0 unhealthy).
rpc_endpoint_overall_health_breakdown Gauge spec, apiInterface Aggregate health per chain/interface.
rpc_endpoint_selection_score Gauge spec, apiInterface, endpoint_id, score_type Selection scores by score_type (availability / latency / sync / stake / composite).
rpc_endpoint_latest_block Gauge spec, apiInterface, endpoint_id Latest block reported by the endpoint.
rpc_endpoint_fetch_latest_fails Counter spec, apiInterface, endpoint_id Latest-block fetch failures. An event counter, not a request counter.
rpc_endpoint_fetch_block_fails Counter spec, apiInterface, endpoint_id Failed specific-block fetches.
rpc_endpoint_fetch_latest_success Counter spec, apiInterface, endpoint_id New-block detections by the chain tracker, not successful requests.
rpc_endpoint_fetch_block_success Counter spec, apiInterface, endpoint_id Successful specific-block fetches.
rpc_endpoint_tracker_requests_total Counter spec, apiInterface, endpoint_id, kind Requests the chain tracker actually sent upstream, by kindlatest_block, or block_hash (zero unless --enable-fork-detection). The only metric that measures tracker request volume.
rpc_endpoint_tracker_gate_skips_total Counter spec, apiInterface, endpoint_id, source Poll ticks the tracker skipped because the upstream's tip was already fresh, by sourcerelay (served traffic in the last block time) or peer (another replica's poll, shared via --shared-state). The complement of requests_total: together they account for every tick.
rpc_endpoint_tracker_gate_errors_total Counter spec, apiInterface, endpoint_id, op Failed calls the fleet gate made to the cache's observation store, by opfetch or publish. A failed fetch falls back to a local poll; a failed publish costs a peer one real poll. Rises when the sidecar predates the observation RPC.

On a multi-replica deployment with --shared-state, the fleet-wide sum(rate(rpc_endpoint_tracker_requests_total{kind="latest_block"}[5m])) for an upstream should settle near a single replica's worth, and source="peer" skips should be non-zero on every replica. peer stuck at zero with the flag on means one of three things: the router is on the RESP backend, which has no peer gate at all; the replicas are not reaching the same cache; or the cache is reached but the calls fail, for example a sidecar older than the router. rpc_endpoint_tracker_gate_errors_total tells the last apart from the other two — a failing store raises it, a fleet with nothing to share leaves it flat.

Two caveats when reading the chain-tracker counters:

  • fetch_latest_success can read higher with fork detection off. With the flag on, a failed block-hash fetch aborts the poll cycle before the new-block callback fires, so endpoints with flaky hash fetches under-count detections; with it off, every detected block is recorded. A rise after disabling the flag is recovered counting, not extra load.
  • endpoint_id holds a provider name, and several providers can share one physical URL. The tracker polls per provider, so a bare sum() over rpc_endpoint_tracker_requests_total counts one physical request once per provider sharing that URL. When estimating the load on a metered node, keep the endpoint_id breakdown (or dedupe by URL) instead of summing blindly.

Optimizer

Metric Type Labels Description
rpc_optimizer_selection_score Gauge spec, endpoint_id, score_type Periodic optimizer selection score per node, by score_type.

Router-scoped — smartrouter_*

Core relay & health

Metric Type Labels Description
smartrouter_total_relays_serviced Counter spec, apiInterface, function Relays served by the router.
smartrouter_total_errored Counter spec, apiInterface, function Errored relays.
smartrouter_end_to_end_latency_milliseconds Histogram spec, apiInterface, function Router-level end-to-end latency.
smartrouter_overall_health Gauge Overall router health (1 / 0).
smartrouter_overall_health_breakdown Gauge spec, apiInterface Per-chain/interface health.
smartrouter_latest_block Gauge spec, apiInterface Latest block known to the router.
smartrouter_protocol_version Gauge version Encoded protocol version.

WebSocket

Metric Type Labels Description
smartrouter_ws_connections_active Gauge spec, apiInterface Active WebSocket connections.
smartrouter_ws_subscriptions_total Counter spec, apiInterface Total WebSocket subscription requests.
smartrouter_ws_subscription_errors_total Counter spec, apiInterface Failed WebSocket subscription requests.

Request breakdown

requests_total = success + failed. read/write partition by statefulness; debug_trace and archive are orthogonal addon flags; batch is mutually exclusive with read/write.

Metric Type Labels Description
smartrouter_requests_total Counter spec, apiInterface, provider_address, method All requests.
smartrouter_requests_success_total Counter spec, apiInterface, provider_address, method Successful requests.
smartrouter_requests_failed_total Counter spec, apiInterface, provider_address, method Failed requests.
smartrouter_requests_read_total Counter spec, apiInterface, provider_address, method Read (stateless) requests.
smartrouter_requests_write_total Counter spec, apiInterface, provider_address, method Write (stateful) requests.
smartrouter_requests_debug_trace_total Counter spec, apiInterface, provider_address, method Debug/trace addon requests.
smartrouter_requests_archive_total Counter spec, apiInterface, provider_address, method Archive requests.
smartrouter_requests_batch_total Counter spec, apiInterface, provider_address, method Batch requests.

Batch requests

A batch has no single method, so its method label carries a batch signature: the sorted set of distinct sub-methods, prefixed batch:.

[eth_call ×30, eth_getBalance]   →   method="batch:eth_call+eth_getBalance"
[eth_call ×3]                    →   method="batch:eth_call"
[eth_call]                       →   method="eth_call"

The signature identifies the shape of a batch, not its contents in order: two batches with the same set of methods share a series regardless of how the elements were ordered or how many times each repeated. Element count lives in smartrouter_batch_size, which keeps the label space bounded by the number of method combinations your clients send rather than the number of batches they send.

Single-method requests are unaffected — method="eth_call" means one eth_call. So method=~"batch:.*" and method!~"batch:.*" split batch from single traffic cleanly, and because the signature rides the normal method label, every per-method view (success rate, latency, per-provider share) works per batch shape too.

Cache hits included: a batch served from the router's cache keeps its signature and, like every cached relay, lands under provider_address="Cached" (see counting semantics) — so a per-provider slice of a batch shape lists that pseudo-provider alongside the real nodes, and its sizes are observed in smartrouter_batch_size like any other batch.

A one-element batch produces the same label as a plain single request and is not counted in smartrouter_batch_size; smartrouter_requests_batch_total still identifies it as a batch.

Metric Type Labels Description
smartrouter_batch_size Histogram spec, apiInterface Sub-requests per batch (buckets 2, 3, 5, 10, 25, 50, 100, 250, 500).
smartrouter_batch_signature_overflow_total Counter spec, reason Batches whose signature landed in batch:other. reason="cap": the chain reached its limit of 64 distinct signatures. reason="wide": one batch mixed more than 8 distinct methods.

While smartrouter_batch_signature_overflow_total is zero, the batch-shape breakdown is complete. Once it climbs, some shapes are being merged into batch:other — the other series stay accurate, but the breakdown no longer names every shape.

Errors

Metric Type Labels Description
smartrouter_node_errors_total Counter spec, apiInterface, provider_address, method Node errors returned by endpoints.
smartrouter_protocol_errors_total Counter spec, apiInterface, provider_address, method Protocol/transport errors (connection/session failures).

Retries

Metric Type Labels Description
smartrouter_retries_total Counter spec, apiInterface, method Retry attempts triggered (beyond the first try).
smartrouter_retries_success_total Counter spec, apiInterface, method Retried requests that succeeded.
smartrouter_retries_failed_total Counter spec, apiInterface, method Retried requests that failed.
smartrouter_retry_attempts Histogram spec, apiInterface, method Attempts per retried request (buckets 1…10).

Rate-limit hold-off

When an upstream answers 429 the router stops asking it for a while instead of retrying into the limit — see Rate-limited upstreams for the behaviour. These two series are emitted by the hold-off registry itself, so every path that talks upstream (relays, spec re-verification, recovery probes, WebSocket subscriptions) is covered. A rate-limited relay is released with no QoS sample in either direction, so a vendor cap does not show up in rpc_endpoint_selection_score; this pair is the direct signal that an upstream is refusing you for load.

Metric Type Labels Description
smartrouter_rate_limit_holdoffs_total Counter provider, event Hold-off registry events. event is the closed set recorded (a 429 held an endpoint off), escalated (the hold-off widened to the whole provider — counted once on the transition, not per refresh), cleared (an answer dropped a standing penalty — not counted when there was nothing to clear).
smartrouter_rate_limit_holdoff_seconds Histogram provider Applied hold-off duration per recorded event, in seconds (buckets 15, 30, 60, 120, 300, 600, 1800, 3600 — not the shared millisecond latency buckets). Shows how long upstreams' Retry-After values are against the exponential default.

provider is the configured provider name on the relay, probe, and re-verification paths. The WebSocket-subscription path has no provider name and keys the registry by node URL; those keys are reduced to scheme://host before they become a label, because node URLs can embed API keys in their path or query and a credential must never reach a Prometheus series.

# how long upstreams are telling us to wait (p90), per provider
histogram_quantile(0.9,
  sum by (provider, le) (rate(smartrouter_rate_limit_holdoff_seconds_bucket[1h])))

Consistency

Metric Type Labels Description
smartrouter_consistency_total Counter spec, apiInterface, method Requests enforcing consistency (seenBlock).
smartrouter_consistency_success_total Counter spec, apiInterface, method Consistency-enforced requests that succeeded.
smartrouter_consistency_failed_total Counter spec, apiInterface, method Consistency-enforced requests that failed.

Hedging

Metric Type Labels Description
smartrouter_hedge_total Counter spec, apiInterface, method Hedge (batch-ticker) relays sent.
smartrouter_hedge_success_total Counter spec, apiInterface, method Hedged requests that succeeded.
smartrouter_hedge_failed_total Counter spec, apiInterface, method Hedged requests that failed.
smartrouter_hedge_attempts Histogram spec, apiInterface, method Hedge relays per request (buckets 1…10).

Cross-validation

Metric Type Labels Description
smartrouter_cross_validation_requests_total Counter spec, apiInterface, method Cross-validated requests.
smartrouter_cross_validation_success_total Counter spec, apiInterface, method Requests that reached consensus.
smartrouter_cross_validation_failed_total Counter spec, apiInterface, method Requests that failed to reach consensus.
smartrouter_cross_validation_provider_agreements_total Counter spec, apiInterface, method, provider_address Times a node agreed with consensus.
smartrouter_cross_validation_provider_disagreements_total Counter spec, apiInterface, method, provider_address Times a node disagreed with consensus.
smartrouter_cross_validation_mismatch_total Counter spec, apiInterface, method, group, finality Content outliers by group: one increment per distinct outlier group per successful deterministic cross-validation request (a response whose SHA256(reply.data) diverged from the reached consensus). Only emitted when a quorum was reached. finality is finalized / not_finalized / unknown; post-finality divergence is the high-signal alert. Bounded cardinality (keyed by operator-defined group, not node address).
smartrouter_cross_validation_failures_total Counter spec, apiInterface, method, reason Failures by reason — the by-reason breakdown of cross_validation_failed_total (which stays unlabeled, so existing dashboards are unaffected). reason is a closed enum: quorum-time no-agreement / insufficient-responses / diversity-unmet / group-quorum-unmet, or request-time structural insufficient-capacity / insufficient-groups. Separates a structural failure (client should fall back) from a quorum disagreement (a retry may help).

Cache

Metric Type Labels Description
smartrouter_cache_requests_total Counter spec, apiInterface, method, cache_tier Cache lookup attempts per tier (primary | secondary). A tier that is unconfigured, disconnected, or bypassed emits nothing for that request.
smartrouter_cache_success_total Counter spec, apiInterface, method, cache_tier Cache hits per tier.
smartrouter_cache_failed_total Counter spec, apiInterface, method, cache_tier, outcome Non-hit lookups, split by the closed enum outcome = miss (clean not-found) | error (transport/server error) | timeout (per-lookup budget exceeded).
smartrouter_cache_latency_milliseconds Histogram spec, apiInterface, method, cache_tier Cache lookup latency, observed on every attempted lookup — hits and non-hits.

Per tier, cache_requests_total = cache_success_total + sum without (outcome) (cache_failed_total). Note the sum without — recovering that identity takes an explicit aggregation across outcome.

The cache_tier="secondary" series exist only when a secondary cache is configured.

Dashboard migration

These series previously had no cache_tier label, _failed_total had no outcome, and the latency histogram was observed on hits only. Aggregating queries (sum by (spec, method)) keep working; exact label matchers must add cache_tier="primary". Latency-based alerts should expect non-hit observations, which typically lower percentiles — misses return faster than hits — while timeouts now appear as a bounded tail instead of being invisible.

RESP cache backend

Present only when the router runs against a RESP-compatible (Redis/Valkey) cache backend. The shared smartrouter_cache_* series above keep firing unchanged regardless of backend; they can't distinguish a backend error or timeout from a clean miss — these can, and they are the alerting surface for cache degradation.

Metric Type Labels Description
smartrouter_resp_cache_failed_total Counter op, kind Backend-level operation failures (never clean misses). op = get | set; kind = error (unreachable / protocol error) | timeout (budget exceeded — saturation reads differently from an outage).
smartrouter_resp_cache_connection_errors_total Counter Failed background health probes (PING, every 10s).
smartrouter_resp_cache_connected Gauge 1 while the last health probe succeeded, 0 after a failure. Reachability transitions are also logged; steady state stays quiet.
smartrouter_resp_cache_pool_total_conns Gauge Connections held by the client pool(s) — write and read summed when the read/write split is configured.
smartrouter_resp_cache_pool_idle_conns Gauge Idle pool connections.
smartrouter_resp_cache_pool_stale_conns Gauge Stale connections removed from the pool.

A failing backend never fails relays: lookups degrade to misses within the caller's budget and requests proceed to the upstreams. Alert on smartrouter_resp_cache_connected == 0 or a smartrouter_resp_cache_failed_total rate, not on request errors.

CSM state-store sizes (diagnostics)

Expose otherwise black-box Consumer-Session-Manager state so integration tests can assert /debug/reset-all emptied each store. All drop to 0 after a reset.

Metric Type Labels Description
smartrouter_csm_blocked_providers Gauge spec, apiInterface Size of the previous-epoch blocked-providers store.
smartrouter_csm_blocked_backup_providers Gauge spec, apiInterface Size of the blocked-backup-providers store.
smartrouter_csm_sticky_sessions Gauge spec, apiInterface Live sticky-session affinities.
smartrouter_csm_reported_providers Gauge spec, apiInterface Size of the reported-providers register.

Shared metrics

Classified errors — smartrouter_errors_*

Defined in error_metrics.go. The error_name and error_category labels come from the classifier registry — see Error codes for the full set of names and their retryability.

Metric Type Labels Description
smartrouter_errors_total Counter error_name, error_category, retryable, chain_id Errors classified by name, category, retryability, and chain.