Skip to content

Metrics ​

Overview ​

Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via a Prometheus HTTP endpoint:

toml
[tiles.metric]
    prometheus_listen_port = 7999
sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...

WARNING

Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.

There are three metric types reported by Firedancer, following the Prometheus data model:

  • counter — A cumulative metric representing a monotonically increasing counter.
  • gauge — A single numerical value that can go arbitrarily up or down.
  • histogram — Samples observations like packet sizes and counts them in buckets.

Durations are reported in integer nanoseconds (*_nanos), not seconds.

There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).

sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0

These link related metrics have four labels, which are to identify the link the metric is for:

  • kind — The name of the tile consuming from the link.
  • kind_id — The tile index of the tile which is consuming from the link.
  • link_kind — The name of the link being consumed.
  • link_kind_id — The link index of the link which is being consumed.
MetricTypeDescription
link_​frag_​consumedcounterFragments consumed by the link reader
link_​frag_​consumed_​bytescounterBytes read by the link consumer
link_​frag_​filteredcounterFragments filtered and not consumed
link_​frag_​filtered_​bytescounterBytes read by the link consumer that were filtered
link_​link_​polling_​overruncounterTimes the link was overrun while polling
link_​frag_​polling_​overruncounterFragments not processed because the link was overrun while polling
link_​link_​reading_​overruncounterTimes the link was overrun while reading metadata
link_​frag_​reading_​overruncounterFragments not processed because the link was overrun while reading
link_​backpressure_​blamedcounterProducer housekeeping intervals in which the producer was backpressured and this reliable consumer had the least credit across its outs; sampled, not an event count

All Tiles ​

Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:

sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268

# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23

All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:

  • kind — The tile name the metric is being reported for.
  • kind_id — The tile index of the tile which is reporting the metric.
MetricTypeDescription
tile_​pidgaugeProcess ID of the tile
tile_​tidgaugeThread ID of the tile. Same as Pid in production, but might differ in development
tile_​last_​cpu_​indexgaugeCPU the tile last ran on
tile_​context_​switch_​involuntarycounterInvoluntary context switches
tile_​context_​switch_​voluntarycounterVoluntary context switches
tile_​page_​fault_​majorcounterMajor page faults
tile_​page_​fault_​minorcounterMinor page faults
tile_​stategaugeTile lifecycle state: 0=booting, 1=running, 2=shutdown
tile_​heartbeat_​timegaugeLast wall-clock time the tile heartbeated
tile_​is_​backpressuredgaugeWhether the tile is currently backpressured
tile_​backpressurecounterTimes the tile waited for one or more consumers to catch up to resume publishing
tile_​regime_​duration_​nanos
{tile_​regime="caught_​up_​housekeeping"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Housekeeping)
tile_​regime_​duration_​nanos
{tile_​regime="processing_​housekeeping"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Processing + Housekeeping)
tile_​regime_​duration_​nanos
{tile_​regime="backpressure_​housekeeping"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Backpressure + Housekeeping)
tile_​regime_​duration_​nanos
{tile_​regime="caught_​up_​prefrag"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Prefrag)
tile_​regime_​duration_​nanos
{tile_​regime="processing_​prefrag"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Processing + Prefrag)
tile_​regime_​duration_​nanos
{tile_​regime="backpressure_​prefrag"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Backpressure + Prefrag)
tile_​regime_​duration_​nanos
{tile_​regime="caught_​up_​postfrag"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Postfrag)
tile_​regime_​duration_​nanos
{tile_​regime="processing_​postfrag"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Processing + Postfrag)
tile_​regime_​duration_​nanos
{tile_​regime="caught_​up_​sleeping"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Sleeping)
tile_​regime_​duration_​nanos
{tile_​regime="backpressure_​sleeping"}
counterTile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Backpressure + Sleeping)
tile_​thread_​duration_​nanos
{thread_​regime="wait"}
counterThread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (Wait (task was runnable but not scheduled))
tile_​thread_​duration_​nanos
{thread_​regime="idle"}
counterThread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (Idle (task was not runnable))
tile_​thread_​duration_​nanos
{thread_​regime="user"}
counterThread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (User (task was scheduled and executing in user mode))
tile_​thread_​duration_​nanos
{thread_​regime="system"}
counterThread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (System (task was scheduled and executing in kernel mode))
tile_​thread_​duration_​nanos
{thread_​regime="interrupt"}
counterThread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only))
tile_​park_​attempted
{park_​result="ring"}
counterTimes the tile tried to park while idle (efficient mode), by result (Woken by a doorbell ring)
tile_​park_​attempted
{park_​result="deadline"}
counterTimes the tile tried to park while idle (efficient mode), by result (Deadline or park cap lapsed)
tile_​park_​attempted
{park_​result="pending"}
counterTimes the tile tried to park while idle (efficient mode), by result (Pre-park recheck found work, did not park)
tile_​device_​irqcounterDevice IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
tile_​tlb_​shootdowncounterTLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
tile_​timer_​irqcounterLocal timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there

Genesi Tile ​

MetricTypeDescription

Ipecho Tile ​

MetricTypeDescription
ipecho_​shred_​versiongaugeShred version used by the validator
ipecho_​conn_​opengaugeOpen connections to the ipecho service
ipecho_​conn_​closed
{conn_​close_​result="accepted"}
counterConnections to the ipecho service made and closed, by outcome (Closed normally)
ipecho_​conn_​closed
{conn_​close_​result="error"}
counterConnections to the ipecho service made and closed, by outcome (Closed abnormally)
ipecho_​conn_​rx_​bytescounterBytes received on connections to the ipecho service
ipecho_​conn_​tx_​bytescounterBytes sent on connections to the ipecho service
ipecho_​port_​check_​udp_​txcounterPort check UDP packets sent to the joiner's UDP ports
ipecho_​port_​check_​tcp_​attempted
{port_​check_​tcp_​result="connected"}
counterPort check TCP connections to the joiner's TCP ports, by result (Port check connections connected to the joiner's port)
ipecho_​port_​check_​tcp_​attempted
{port_​check_​tcp_​result="failed"}
counterPort check TCP connections to the joiner's TCP ports, by result (Port check connections refused or errored)
ipecho_​port_​check_​tcp_​attempted
{port_​check_​tcp_​result="timed_​out"}
counterPort check TCP connections to the joiner's TCP ports, by result (Port check connections timed out)
ipecho_​port_​check_​tcp_​attempted
{port_​check_​tcp_​result="socket_​failed"}
counterPort check TCP connections to the joiner's TCP ports, by result (Port check connections not attempted because the socket could not be created)
ipecho_​port_​check_​tcp_​attempted
{port_​check_​tcp_​result="rejected_​per_​ip"}
counterPort check TCP connections to the joiner's TCP ports, by result (Port check connections rejected because too many were in flight for a particular IP)
ipecho_​port_​check_​tcp_​attempted
{port_​check_​tcp_​result="evicted"}
counterPort check TCP connections to the joiner's TCP ports, by result (Port check connections evicted because there were too many in flight connections)
ipecho_​port_​check_​tcp_​in_​flightgaugePort check TCP connections in flight

Snapct Tile ​

MetricTypeDescription
snapct_​load_​stategaugeSnapshot load controller state: 0=init (Initializing), 1=waiting_for_peers, 2=waiting_for_peers_incremental (Waiting for peers (incremental)), 3=collecting_peers, 4=collecting_peers_incremental (Collecting peers (incremental)), 5=reading_full_file (Reading full snapshot file), 6=flushing_full_file_fini (Full file read, draining downstream), 7=flushing_full_file_done (Full file loaded, draining downstream), 8=flushing_full_file_reset (Full file load aborted, resetting downstream), 9=reading_incremental_file (Reading incremental snapshot file), 10=flushing_incremental_file_fini (Incremental file read, draining downstream), 11=flushing_incremental_file_done (Incremental file loaded, draining downstream), 12=flushing_incremental_file_reset (Incremental file load aborted, resetting downstream), 13=reading_full_http (Downloading full snapshot), 14=flushing_full_http_fini (Full download read, draining downstream), 15=flushing_full_http_done (Full download loaded, draining downstream), 16=flushing_full_http_reset (Full download aborted, resetting downstream), 17=reading_incremental_http (Downloading incremental snapshot), 18=flushing_incremental_http_fini (Incremental download read, draining downstream), 19=flushing_incremental_http_done (Incremental download loaded, draining downstream), 20=flushing_incremental_http_reset (Incremental download aborted, resetting downstream), 21=shutdown (Snapshot load complete)
snapct_​read_​progress_​bytes
{snap_​type="full"}
gaugeSnapshot file bytes read (download or local file) in the current attempt, by type; resets on retry (Full snapshot)
snapct_​read_​progress_​bytes
{snap_​type="incremental"}
gaugeSnapshot file bytes read (download or local file) in the current attempt, by type; resets on retry (Incremental snapshot)
snapct_​read_​expected_​bytes
{snap_​type="full"}
gaugeSize of each snapshot file being loaded; may change if the load restarts (Full snapshot)
snapct_​read_​expected_​bytes
{snap_​type="incremental"}
gaugeSize of each snapshot file being loaded; may change if the load restarts (Incremental snapshot)
snapct_​save_​progress_​bytes
{snap_​type="full"}
gaugeBytes of each downloaded snapshot saved to the snapshot directory so far; resets if the load restarts (Full snapshot)
snapct_​save_​progress_​bytes
{snap_​type="incremental"}
gaugeBytes of each downloaded snapshot saved to the snapshot directory so far; resets if the load restarts (Incremental snapshot)
snapct_​load_​retried
{snap_​type="full"}
counterSnapshot downloads retried, by snapshot type (Full snapshot)
snapct_​load_​retried
{snap_​type="incremental"}
counterSnapshot downloads retried, by snapshot type (Incremental snapshot)
snapct_​predicted_​incremental_​slotgaugeSlot of the incremental snapshot replay is predicted to start from, 0 if none chosen; may change if the load restarts

Snapld Tile ​

MetricTypeDescription
snapld_​stream_​stategaugeSnapshot loader state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown

Snapdc Tile ​

MetricTypeDescription
snapdc_​stream_​stategaugeSnapshot decompressor state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
snapdc_​compressed_​progress_​bytes
{snap_​type="full"}
gaugeCompressed bytes of each snapshot read in the current attempt; sum across snapdc tiles (Full snapshot)
snapdc_​compressed_​progress_​bytes
{snap_​type="incremental"}
gaugeCompressed bytes of each snapshot read in the current attempt; sum across snapdc tiles (Incremental snapshot)
snapdc_​decompressed_​progress_​bytes
{snap_​type="full"}
gaugeDecompressed bytes of each snapshot produced in the current attempt; sum across snapdc tiles (Full snapshot)
snapdc_​decompressed_​progress_​bytes
{snap_​type="incremental"}
gaugeDecompressed bytes of each snapshot produced in the current attempt; sum across snapdc tiles (Incremental snapshot)

Snapin Tile ​

MetricTypeDescription
snapin_​stream_​stategaugeSnapshot inserter state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
snapin_​parse_​progress_​bytes
{snap_​type="full"}
gaugeDecompressed bytes of each snapshot this tile parsed in the current attempt; every snapin tile parses the whole stream, so take the min across tiles (Full snapshot)
snapin_​parse_​progress_​bytes
{snap_​type="incremental"}
gaugeDecompressed bytes of each snapshot this tile parsed in the current attempt; every snapin tile parses the whole stream, so take the min across tiles (Incremental snapshot)
snapin_​disk_​written_​bytescounterBytes this tile has written to the accounts database file
snapin_​account_​progress
{snapin_​account_​result="new"}
gaugeAccounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry (New account inserted)
snapin_​account_​progress
{snapin_​account_​result="replaced"}
gaugeAccounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry (Replaced an older duplicate inserted earlier)
snapin_​account_​progress
{snapin_​account_​result="ignored"}
gaugeAccounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry (Dropped because an inserted duplicate was newer)
snapin_​account_​processedcounterAccounts processed across all snapshots
snapin_​account_​batch_​processedcounterAccount batches processed across all snapshots

Admin Tile ​

MetricTypeDescription

Netlnk Tile ​

MetricTypeDescription
netlnk_​rx_​buffer_​overflowedcounterNetlink receive buffer overflows (ENOBUFS); each forces a full resync
netlnk_​link_​table_​syncedcounterFull link table syncs
netlnk_​route_​table_​syncedcounterFull route table syncs
netlnk_​update_​processed
{netlink_​message_​type="link"}
counterNetlink live updates processed, by message type (Link)
netlnk_​update_​processed
{netlink_​message_​type="neighbor"}
counterNetlink live updates processed, by message type (Neighbor Table Entry)
netlnk_​update_​processed
{netlink_​message_​type="ipv4_​route"}
counterNetlink live updates processed, by message type (IPv4 Route Table Entry)
netlnk_​interface_​countgaugeNetwork interfaces in the interface table
netlnk_​neighbor_​probe_​requested
{neighbor_​probe_​result="sent"}
counterNeighbor solicit requests from net tiles, by result (Neighbor solicit sent to the kernel)
netlnk_​neighbor_​probe_​requested
{neighbor_​probe_​result="in_​flight"}
counterNeighbor solicit requests from net tiles, by result (Kernel already resolving this neighbor)
netlnk_​neighbor_​probe_​requested
{neighbor_​probe_​result="table_​full"}
counterNeighbor solicit requests from net tiles, by result (Neighbor table had no room for a placeholder entry)
netlnk_​neighbor_​probe_​requested
{neighbor_​probe_​result="send_​failed"}
counterNeighbor solicit requests from net tiles, by result (Netlink send failed)
netlnk_​neighbor_​probe_​requested
{neighbor_​probe_​result="host_​rate_​limited"}
counterNeighbor solicit requests from net tiles, by result (Per-host probe rate limit hit)
netlnk_​neighbor_​probe_​requested
{neighbor_​probe_​result="global_​rate_​limited"}
counterNeighbor solicit requests from net tiles, by result (Global probe rate limit hit)

Net Tile ​

MetricTypeDescription
net_​pkt_​rxcounterPackets successfully published to output links
net_​pkt_​rx_​bytescounterBytes of packets successfully published to output links
net_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="malformed"}
counterReceived packets dropped, by reason (Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present)
net_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="dst_​ip_​mismatch"}
counterReceived packets dropped, by reason (Destination IP does not match the bind address)
net_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="dst_​port_​unbound"}
counterReceived packets dropped, by reason (No output link for the destination UDP port)
net_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="gre_​invalid"}
counterReceived packets dropped, by reason (GRE tunnel peer or headers invalid)
net_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="gre_​unconfigured"}
counterReceived packets dropped, by reason (GRE packet with no GRE tunnel configured)
net_​rx_​fill_​ring_​stalledcounterTimes RX stalled because the fill ring had no room; the packet stays in the RX ring and is retried (kernel drops are XdpRxRingFullDropped)
net_​rx_​buffer_​usedgaugeReceive buffers currently in use
net_​rx_​buffer_​freegaugeReceive buffers currently free
net_​pkt_​tx_​submittedcounterPacket transmit jobs submitted
net_​pkt_​tx_​completedcounterPacket transmit jobs marked as completed by the kernel
net_​pkt_​tx_​submitted_​bytescounterBytes of packets submitted for transmission
net_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="invalid"}
counterTransmit jobs dropped, by reason (Invalid packet headers, or GRE packet exceeding the MTU)
net_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="source_​ip_​missing"}
counterTransmit jobs dropped, by reason (No source IP address chosen)
net_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="ring_​full"}
counterTransmit jobs dropped, by reason (TX ring full or no TX buffer available)
net_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="gre_​no_​route"}
counterTransmit jobs dropped, by reason (GRE tunnel addresses missing or outer route is itself tunnelled)
net_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="socket_​send_​failure"}
counterTransmit jobs dropped, by reason (Socket send failed or failed to send the entire packet)
net_​pkt_​tx_​routed
{pkt_​tx_​route_​result="fast_​path"}
counterTransmit jobs accepted after route resolution, by result (Userland routing succeeded and selected the configured provider)
net_​pkt_​tx_​routed
{pkt_​tx_​route_​result="slow_​path"}
counterTransmit jobs accepted after route resolution, by result (Failed userland routing and selected the kernel fallback)
net_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="no_​route"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (No matching route)
net_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="unsupported_​route"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Unsupported route type)
net_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="interface_​missing"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface not available)
net_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="unsupported_​interface"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface type not supported)
net_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="no_​neighbor"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Next-hop neighbor unresolved or inactive)
net_​tx_​buffer_​usedgaugeTransmit buffers currently in use
net_​tx_​buffer_​freegaugeTransmit buffers currently free
net_​xsk_​syscall_​txcounterXSK sendto syscalls dispatched; in busy-poll mode each also counts in XskSyscallRx
net_​xsk_​syscall_​rxcounterXSK recvmsg syscalls dispatched
net_​xdp_​rx_​other_​droppedcounterDropped for other reasons (xdp_statistics_v0.rx_dropped)
net_​xdp_​rx_​invalid_​descriptor_​droppedcounterDropped due to invalid descriptor (xdp_statistics_v0.rx_invalid_descs)
net_​xdp_​tx_​invalid_​descriptor_​droppedcounterDropped due to invalid descriptor (xdp_statistics_v0.tx_invalid_descs)
net_​xdp_​rx_​ring_​full_​droppedcounterDropped due to rx ring being full (xdp_statistics_v1.rx_ring_full)
net_​xdp_​rx_​fill_​ring_​emptycounterTimes the kernel found the fill ring empty (xdp_statistics_v1.rx_fill_ring_empty_descs)
net_​xdp_​tx_​ring_​emptycounterTimes the kernel found the TX ring empty (xdp_statistics_v1.tx_ring_empty_descs)
net_​gre_​pkt_​rxcounterGRE packets decapsulated and published; subset of PktRx
net_​gre_​pkt_​tx_​submittedcounterGRE packet transmit jobs submitted; subset of PktTxSubmitted
net_​pkt_​tx_​slow_​path_​submittedcounterPackets submitted to a kernel socket on the slow path; subset of PktTxSubmitted
net_​route_​count
{route_​table_​type="local"}
gaugeIPv4 routes installed in the forwarding table (Local)
net_​route_​count
{route_​table_​type="main"}
gaugeIPv4 routes installed in the forwarding table (Main)
net_​vlan_​pkt_​rxcounterVLAN-tagged packets with the tag stripped in software and published; subset of PktRx

Sock Tile ​

MetricTypeDescription
sock_​syscall_​tx
{sendmmsg_​result="success"}
countersendmmsg syscalls dispatched, by result (Success)
sock_​syscall_​tx
{sendmmsg_​result="buffer_​full"}
countersendmmsg syscalls dispatched, by result (ENOBUFS, EAGAIN error)
sock_​syscall_​tx
{sendmmsg_​result="permission"}
countersendmmsg syscalls dispatched, by result (EPERM error (blocked by netfilter))
sock_​syscall_​tx
{sendmmsg_​result="unreachable"}
countersendmmsg syscalls dispatched, by result (ENETUNREACH, EHOSTUNREACH error)
sock_​syscall_​tx
{sendmmsg_​result="down"}
countersendmmsg syscalls dispatched, by result (ENONET, ENETDOWN, EHOSTDOWN error)
sock_​syscall_​tx
{sendmmsg_​result="other"}
countersendmmsg syscalls dispatched, by result (Unrecognized error code)
sock_​syscall_​rxcounterrecvmmsg syscalls dispatched
sock_​pkt_​rxcounterPackets received
sock_​pkt_​txcounterPackets sent
sock_​pkt_​tx_​failedcounterPackets that failed to send
sock_​pkt_​tx_​bytescounterBytes of packets successfully sent
sock_​pkt_​rx_​bytescounterUDP payload bytes received plus 42 bytes of synthesized Ethernet/IPv4/UDP headers per packet

Quic Tile ​

MetricTypeDescription
quic_​txn_​reassembly_​evictedcounterTxns evicted from reassembly before completing (txn_reassembly_count too small)
quic_​txn_​reassembly_​startedcounterFragmented txn receive ops started
quic_​txn_​reassembly_​pendinggaugeFragmented txn reassemblies in progress
quic_​txn_​reassembly_​chunk_​rxcounterTxn stream chunks received for reassembly
quic_​txn_​reassembly_​gap_​droppedcounterTxn stream chunks dropped because of a data gap
quic_​txn_​reassembly_​duplicate_​droppedcounterTxn stream chunks dropped as duplicates of a completed stream
quic_​txn_​rx
{tpu_​rx_​type="udp"}
counterTransactions published to verify, by TPU ingress path; excludes all *Dropped (TPU/UDP)
quic_​txn_​rx
{tpu_​rx_​type="quic_​fast"}
counterTransactions published to verify, by TPU ingress path; excludes all *Dropped (TPU/QUIC unfragmented)
quic_​txn_​rx
{tpu_​rx_​type="quic_​frag"}
counterTransactions published to verify, by TPU ingress path; excludes all *Dropped (TPU/QUIC fragmented)
quic_​txn_​reassembly_​abandonedcounterTxn reassemblies abandoned because the connection was lost
quic_​txn_​undersize_​droppedcounterTxns received via QUIC dropped because they were too small
quic_​txn_​oversize_​droppedcounterTxns received via QUIC dropped because they were too large
quic_​udp_​txn_​undersize_​droppedcounterPackets received on the non-QUIC port that were too small to be a valid IP packet
quic_​udp_​txn_​oversize_​droppedcounterPackets received on the non-QUIC port that were too large to be a valid transaction
quic_​pkt_​rxcounterUDP datagrams received (one may carry several coalesced QUIC packets)
quic_​pkt_​rx_​bytescounterBytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers
quic_​pkt_​txcounterUDP datagrams sent
quic_​pkt_​tx_​bytescounterBytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers
quic_​conn_​used
{quic_​conn_​state="handshake"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Handshaking peer)
quic_​conn_​used
{quic_​conn_​state="handshake_​complete"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Handshake complete, confirming with peer)
quic_​conn_​used
{quic_​conn_​state="active"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Active connection)
quic_​conn_​used
{quic_​conn_​state="peer_​close"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Peer requested close)
quic_​conn_​used
{quic_​conn_​state="abort"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection terminating due to error)
quic_​conn_​used
{quic_​conn_​state="close_​pending"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection is closing)
quic_​conn_​used
{quic_​conn_​state="dead"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection about to be freed)
quic_​conn_​freegaugeConnection slots free
quic_​conn_​capacitygaugeConnection slots
quic_​conn_​createdcounterConnections created
quic_​conn_​closedcounterConnections gracefully closed
quic_​conn_​abortedcounterConnections aborted
quic_​conn_​timed_​outcounterConnections timed out
quic_​conn_​retriedcounterConnections established with retry
quic_​conn_​no_​slot_​rejectedcounterConnections that failed to create due to lack of slots
quic_​conn_​retry_​rejectedcounterRetries rejected: an invalid or expired token as server, a late or invalid Retry packet as client
quic_​conn_​retry_​expired_​rejectedcounterInitials with an expired Retry token, also counted in ConnRetryRejected. Each gets an INVALID_TOKEN close so the client reconnects
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="undersize"}
counterReceived packets dropped, by reason (QUIC packet too small)
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="oversize"}
counterReceived packets dropped, by reason (QUIC packet too large)
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="src_​ip_​mismatch"}
counterReceived packets dropped, by reason (Wrong source IP)
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="header_​invalid"}
counterReceived packets dropped, by reason (Invalid QUIC header)
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="net_​header_​invalid"}
counterReceived packets dropped, by reason (Invalid IPv4 or UDP header)
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="version_​unsupported"}
counterReceived packets dropped, by reason (Version negotiation or unsupported version)
quic_​pkt_​dropped
{quic_​pkt_​drop_​reason="duplicate"}
counterReceived packets dropped, by reason (Packet number already received or too old)
quic_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="initial"}
counterPackets with an unknown connection ID, by packet type (Initial)
quic_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="retry"}
counterPackets with an unknown connection ID, by packet type (Retry)
quic_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="handshake"}
counterPackets with an unknown connection ID, by packet type (Handshake)
quic_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="one_​rtt"}
counterPackets with an unknown connection ID, by packet type (1-RTT)
quic_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="success"}
counterAttempts to acquire packet metadata for transmission, by result (Success)
quic_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="pool_​empty"}
counterAttempts to acquire packet metadata for transmission, by result (Packet metadata pool empty)
quic_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="conn_​limit"}
counterAttempts to acquire packet metadata for transmission, by result (Connection reached its in-flight frame limit)
quic_​initial_​pkt_​rx
{initial_​token_​type="none"}
counterInitial packets received, by token type (No token)
quic_​initial_​pkt_​rx
{initial_​token_​type="retry_​token"}
counterInitial packets received, by token type (fd_quic retry token)
quic_​initial_​pkt_​rx
{initial_​token_​type="other"}
counterInitial packets received, by token type (Token of another length)
quic_​handshake_​createdcounterHandshake flows created
quic_​handshake_​pool_​full_​rejectedcounterHandshakes dropped due to alloc fail
quic_​handshake_​evictedcounterHandshakes dropped due to eviction
quic_​stream_​rxcounterStream receive events
quic_​stream_​rx_​bytescounterStream payload bytes received
quic_​frame_​rx
{quic_​frame_​type="unknown"}
counterQUIC frames received, by frame type (Unknown frame type)
quic_​frame_​rx
{quic_​frame_​type="ack"}
counterQUIC frames received, by frame type (ACK frame)
quic_​frame_​rx
{quic_​frame_​type="reset_​stream"}
counterQUIC frames received, by frame type (RESET_STREAM frame)
quic_​frame_​rx
{quic_​frame_​type="stop_​sending"}
counterQUIC frames received, by frame type (STOP_SENDING frame)
quic_​frame_​rx
{quic_​frame_​type="crypto"}
counterQUIC frames received, by frame type (CRYPTO frame)
quic_​frame_​rx
{quic_​frame_​type="new_​token"}
counterQUIC frames received, by frame type (NEW_TOKEN frame)
quic_​frame_​rx
{quic_​frame_​type="stream"}
counterQUIC frames received, by frame type (STREAM frame)
quic_​frame_​rx
{quic_​frame_​type="max_​data"}
counterQUIC frames received, by frame type (MAX_DATA frame)
quic_​frame_​rx
{quic_​frame_​type="max_​stream_​data"}
counterQUIC frames received, by frame type (MAX_STREAM_DATA frame)
quic_​frame_​rx
{quic_​frame_​type="max_​streams"}
counterQUIC frames received, by frame type (MAX_STREAMS frame)
quic_​frame_​rx
{quic_​frame_​type="data_​blocked"}
counterQUIC frames received, by frame type (DATA_BLOCKED frame)
quic_​frame_​rx
{quic_​frame_​type="stream_​data_​blocked"}
counterQUIC frames received, by frame type (STREAM_DATA_BLOCKED frame)
quic_​frame_​rx
{quic_​frame_​type="streams_​blocked"}
counterQUIC frames received, by frame type (STREAMS_BLOCKED(bidi) frame)
quic_​frame_​rx
{quic_​frame_​type="new_​conn_​id"}
counterQUIC frames received, by frame type (NEW_CONN_ID frame)
quic_​frame_​rx
{quic_​frame_​type="retire_​conn_​id"}
counterQUIC frames received, by frame type (RETIRE_CONN_ID frame)
quic_​frame_​rx
{quic_​frame_​type="path_​challenge"}
counterQUIC frames received, by frame type (PATH_CHALLENGE frame)
quic_​frame_​rx
{quic_​frame_​type="path_​response"}
counterQUIC frames received, by frame type (PATH_RESPONSE frame)
quic_​frame_​rx
{quic_​frame_​type="conn_​close_​quic"}
counterQUIC frames received, by frame type (CONN_CLOSE(transport) frame)
quic_​frame_​rx
{quic_​frame_​type="conn_​close_​app"}
counterQUIC frames received, by frame type (CONN_CLOSE(app) frame)
quic_​frame_​rx
{quic_​frame_​type="handshake_​done"}
counterQUIC frames received, by frame type (HANDSHAKE_DONE frame)
quic_​frame_​rx
{quic_​frame_​type="ping"}
counterQUIC frames received, by frame type (PING frame)
quic_​frame_​rx
{quic_​frame_​type="padding"}
counterQUIC frames received, by frame type (PADDING frame)
quic_​frame_​rx
{quic_​frame_​type="datagram"}
counterQUIC frames received, by frame type (DATAGRAM frame)
quic_​pkt_​ack_​processed
{ack_​result="no_​pkt_​number"}
counterReceived packets processed for acknowledgement, by result (Packet had no packet number to acknowledge)
quic_​pkt_​ack_​processed
{ack_​result="new_​range"}
counterReceived packets processed for acknowledgement, by result (New ACK range)
quic_​pkt_​ack_​processed
{ack_​result="merged"}
counterReceived packets processed for acknowledgement, by result (Merged into existing ACK range)
quic_​pkt_​ack_​processed
{ack_​result="queue_​full"}
counterReceived packets processed for acknowledgement, by result (Out of buffers)
quic_​pkt_​ack_​processed
{ack_​result="cancelled"}
counterReceived packets processed for acknowledgement, by result (ACK suppressed by handler)
quic_​service_​duration_​nanoshistogramDuration of one fd_quic_service call that processed a due connection timer
quic_​pkt_​rx_​duration_​nanoshistogramDuration of processing one received packet
quic_​frame_​parse_​failedcounterQUIC frames that failed to parse
quic_​pkt_​decrypt_​failed
{quic_​encryption_​level="initial"}
counterPackets that failed decryption, by encryption level (Initial)
quic_​pkt_​decrypt_​failed
{quic_​encryption_​level="early"}
counterPackets that failed decryption, by encryption level (Early data)
quic_​pkt_​decrypt_​failed
{quic_​encryption_​level="handshake"}
counterPackets that failed decryption, by encryption level (Handshake)
quic_​pkt_​decrypt_​failed
{quic_​encryption_​level="app"}
counterPackets that failed decryption, by encryption level (App data)
quic_​pkt_​no_​key_​dropped
{quic_​encryption_​level="initial"}
counterPackets dropped because the decryption key was missing, by encryption level (Initial)
quic_​pkt_​no_​key_​dropped
{quic_​encryption_​level="early"}
counterPackets dropped because the decryption key was missing, by encryption level (Early data)
quic_​pkt_​no_​key_​dropped
{quic_​encryption_​level="handshake"}
counterPackets dropped because the decryption key was missing, by encryption level (Handshake)
quic_​pkt_​no_​key_​dropped
{quic_​encryption_​level="app"}
counterPackets dropped because the decryption key was missing, by encryption level (App data)
quic_​retry_​pkt_​txcounterQUIC Retry packets sent
quic_​pkt_​retransmitted
{quic_​encryption_​level="initial"}
counterQUIC packets retransmitted, by encryption level (Initial)
quic_​pkt_​retransmitted
{quic_​encryption_​level="early"}
counterQUIC packets retransmitted, by encryption level (Early data)
quic_​pkt_​retransmitted
{quic_​encryption_​level="handshake"}
counterQUIC packets retransmitted, by encryption level (Handshake)
quic_​pkt_​retransmitted
{quic_​encryption_​level="app"}
counterQUIC packets retransmitted, by encryption level (App data)

Bundle Tile ​

MetricTypeDescription
bundle_​txn_​rxcounterTransactions received, including transactions within bundles
bundle_​standalone_​txn_​rxcounterTransactions received outside of a bundle
bundle_​protobuf_​rx_​bytescounterBytes from received gRPC protobuf payloads
bundle_​bundle_​rxcounterBundles received
bundle_​client_​failed
{bundle_​client_​failure_​reason="decode"}
counterBundle client failures, by cause (Protobuf decode or encode failed)
bundle_​client_​failed
{bundle_​client_​failure_​reason="transport"}
counterBundle client failures, by cause (gRPC transport failed)
bundle_​client_​failed
{bundle_​client_​failure_​reason="no_​fee_​info"}
counterBundle client failures, by cause (Bundle dropped due to missing fee info)
bundle_​heap_​capacity_​bytesgaugeWorkspace heap capacity
bundle_​heap_​free_​bytesgaugeApproximate free space in workspace
bundle_​ping_​ackedcounterHTTP/2 PINGs acknowledged by server
bundle_​conn_​stategaugeBlock engine connection state; Sleeping means closed on purpose because the next leader slot is far away: 0=disconnected, 1=connecting, 2=connected, 3=sleeping
bundle_​last_​rtt_​nanosgaugeLatest block engine HTTP/2 PING round trip, 0 if none since connect
bundle_​rtt_​smoothed_​nanosgaugeSmoothed block engine HTTP/2 PING round trip (RFC 9002), 0 if none since connect
bundle_​rtt_​deviation_​nanosgaugeMean deviation of block engine HTTP/2 PING round trips (RFC 9002 rttvar), 0 if none since connect
bundle_​message_​rx_​delay_​nanoshistogramLocal wall clock minus server timestamp per block engine message; includes clock skew, negative clamped to 0
bundle_​txn_​buffer_​full_​droppedcounterTransactions from the block engine dropped because the verify tiles were too slow to keep up

Verify Tile ​

MetricTypeDescription
verify_​txn_​processed
{verify_​result="success"}
counterTransactions processed, by result (Transaction verified successfully)
verify_​txn_​processed
{verify_​result="bundle_​peer_​failed"}
counterTransactions processed, by result (Another transaction in the bundle failed)
verify_​txn_​processed
{verify_​result="parse_​failed"}
counterTransactions processed, by result (Transaction failed to parse)
verify_​txn_​processed
{verify_​result="duplicate"}
counterTransactions processed, by result (Transaction was a duplicate)
verify_​txn_​processed
{verify_​result="signature_​failed"}
counterTransactions processed, by result (Transaction failed signature verification)

Dedup Tile ​

MetricTypeDescription
dedup_​txn_​processed
{dedup_​result="success"}
counterTransactions processed, by result (Transaction was not a duplicate)
dedup_​txn_​processed
{dedup_​result="bundle_​peer_​failed"}
counterTransactions processed, by result (Another transaction in the bundle failed)
dedup_​txn_​processed
{dedup_​result="duplicate"}
counterTransactions processed, by result (Transaction was a duplicate)

Resolv Tile ​

MetricTypeDescription
resolv_​txn_​dropped
{resolv_​txn_​drop_​reason="no_​bank"}
counterTransactions dropped, by reason (No bank was available)
resolv_​txn_​dropped
{resolv_​txn_​drop_​reason="blockhash_​expired"}
counterTransactions dropped, by reason (Blockhash expired)
resolv_​txn_​dropped
{resolv_​txn_​drop_​reason="bundle_​peer_​failed"}
counterTransactions dropped, by reason (Another transaction in the bundle failed)
resolv_​stash_​txn_​insertedcounterTransactions with an unknown blockhash added to the stash
resolv_​stash_​txn_​removed
{stash_​remove_​reason="published"}
counterStashed transactions removed, by reason (Published once its blockhash became known)
resolv_​stash_​txn_​removed
{stash_​remove_​reason="evicted"}
counterStashed transactions removed, by reason (Evicted as the oldest entry to make room for a new one)
resolv_​stash_​txn_​removed
{stash_​remove_​reason="failed"}
counterStashed transactions removed, by reason (Removed without publishing because LUT resolution failed or no bank was available)
resolv_​txn_​lut_​resolve_​attempted
{lut_​resolve_​result="invalid_​lookup_​index"}
counterAddress lookup table resolution attempts, by result (The transaction referenced an index in a LUT that didn't exist)
resolv_​txn_​lut_​resolve_​attempted
{lut_​resolve_​result="account_​uninitialized"}
counterAddress lookup table resolution attempts, by result (The account referenced as a LUT hasn't been initialized)
resolv_​txn_​lut_​resolve_​attempted
{lut_​resolve_​result="invalid_​account_​data"}
counterAddress lookup table resolution attempts, by result (The account referenced as a LUT couldn't be parsed)
resolv_​txn_​lut_​resolve_​attempted
{lut_​resolve_​result="invalid_​account_​owner"}
counterAddress lookup table resolution attempts, by result (The account referenced as a LUT wasn't owned by the ALUT program ID)
resolv_​txn_​lut_​resolve_​attempted
{lut_​resolve_​result="account_​not_​found"}
counterAddress lookup table resolution attempts, by result (The account referenced as a LUT couldn't be found)
resolv_​txn_​lut_​resolve_​attempted
{lut_​resolve_​result="success"}
counterAddress lookup table resolution attempts, by result (Resolved successfully)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le512_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le2_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le8_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le32_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le1_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
resolv_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le10_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (0-128 B)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le512_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (129-512 B)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le2_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le8_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (2-8 KiB)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le32_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (8-32 KiB)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (32-128 KiB)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le1_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB)
resolv_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le10_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (1-10 MiB)
resolv_​accdb_​account_​waitedcounterAccounts that had to wait for another reader to finish loading them into the cache
resolv_​accdb_​disk_​read_​bytescounterBytes read from the account database file
resolv_​accdb_​disk_​readcounterReads from the account database file
resolv_​accdb_​copied_​bytescounterAccount data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded

Pack Tile ​

MetricTypeDescription
pack_​microblock_​schedule_​duration_​nanoshistogramDuration of scheduling one microblock
pack_​microblock_​schedule_​empty_​duration_​nanoshistogramDuration of discovering that no transaction is schedulable
pack_​txn_​insert_​duration_​nanoshistogramDuration of inserting one transaction into the pending pool
pack_​microblock_​complete_​duration_​nanoshistogramDuration of marking one microblock as complete
pack_​txn_​per_​microblockhistogramTransactions in a scheduled microblock, including both votes and non-votes
pack_​txn_​rxcounterTransactions received from resolv, including bundle members
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="instr_​too_​many_​accounts"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction has an instruction that references too many accounts)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="nonce_​conflict"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Bundle with two conflicting durable nonce transactions)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="account_​blocklisted"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction uses an account on the account blocklist)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="bundle_​blocklisted"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction uses an account on the bundle blocklist)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="invalid_​nonce"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction is an invalid durable nonce transaction)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="write_​sysvar"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction tries to write to a sysvar)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="estimation_​failed"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Estimating compute cost or fee failed)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="duplicate_​account"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction included an account address twice)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="too_​many_​accounts"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction tried to load too many accounts)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="too_​many_​cu"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction requests too many CUs)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="expired"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction already expired)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="lut_​unsupported"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction loads accounts from a lookup table, which is not supported here)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="unaffordable"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Fee payer's balance below transaction fee)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="duplicate"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Pack aware of transaction with same signature)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="nonce_​priority_​too_​low"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="priority_​too_​low"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction's fee was too low given its compute unit requirement and other competing transactions)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="nonvote_​add"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction added to pending transactions)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="nonvote_​replace"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Transaction replaced a lower priority transaction)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="nonce_​nonvote_​add"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Durable nonce transaction added to pending transactions)
pack_​txn_​insert_​attempted
{pack_​txn_​insert_​result="nonce_​nonvote_​replace"}
counterInsert attempts, by result; bundle members are counted individually with the bundle's result (Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="none"}
counterTime pack spent, by scheduling regime (No microblock: not leader, no transactions, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn"}
counterTime pack spent, by scheduling regime (No microblock: not leader, transactions available, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="execle"}
counterTime pack spent, by scheduling regime (No microblock: not leader, no transactions, execle available)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​execle"}
counterTime pack spent, by scheduling regime (No microblock: not leader, transactions available, execle available)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="leader"}
counterTime pack spent, by scheduling regime (No microblock: leader, no transactions, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​leader"}
counterTime pack spent, by scheduling regime (No microblock: leader, transactions available, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="execle_​leader"}
counterTime pack spent, by scheduling regime (No microblock: leader, no transactions, execle available)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​execle_​leader"}
counterTime pack spent, by scheduling regime (No microblock: leader, transactions available, execle available (all conflicting))
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: not leader, no transactions, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: not leader, transactions available, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="execle_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: not leader, no transactions, execle available)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​execle_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: not leader, transactions available, execle available)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="leader_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: leader, no transactions, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​leader_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: leader, transactions available, no available execle)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="execle_​leader_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: leader, no transactions, execle available)
pack_​schedule_​regime_​duration_​nanos
{pack_​schedule_​regime="txn_​execle_​leader_​microblock"}
counterTime pack spent, by scheduling regime (Scheduled a microblock: leader, transactions available, execle available)
pack_​txn_​dropped
{pack_​txn_​drop_​reason="expired"}
counterTransactions dropped from pack after insertion, by reason (TTL expired)
pack_​txn_​dropped
{pack_​txn_​drop_​reason="already_​executed"}
counterTransactions dropped from pack after insertion, by reason (Already executed in the replay or leader pipeline)
pack_​txn_​dropped
{pack_​txn_​drop_​reason="partial_​bundle"}
counterTransactions dropped from pack after insertion, by reason (Part of a partial bundle)
pack_​txn_​evictedcounterTransactions evicted from pack to make room for others, including the bundle initializer transaction removed when a leader slot ends
pack_​txn_​pending_​used
{pending_​txn_​type="regular"}
gaugePending transactions in pack's pool, by type (Transactions in the main treap)
pack_​txn_​pending_​used
{pending_​txn_​type="vote"}
gaugePending transactions in pack's pool, by type (Deprecated, always 0 since Firedancer v26.11.)
pack_​txn_​pending_​used
{pending_​txn_​type="conflicting"}
gaugePending transactions in pack's pool, by type (Transactions that write to a hotly-contended account)
pack_​txn_​pending_​used
{pending_​txn_​type="bundle"}
gaugePending transactions in pack's pool, by type (Transactions that are part of a bundle)
pack_​txn_​pending_​used
{pending_​txn_​type="next_​auction"}
gaugePending transactions in pack's pool, by type (Transactions that are currently held waiting for the next auction)
pack_​txn_​pending_​capacitygaugeMaximum pending transactions pack can hold
pack_​txn_​pending_​smallest_​costgaugeLower bound on the cost of the cheapest pending transaction, 0 if none pending
pack_​block_​microblock_​limit_​reachedcounterTimes pack did not schedule a microblock because the block microblock limit was reached
pack_​block_​byte_​limit_​reachedcounterTimes pack did not schedule a microblock because the block byte limit was reached
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="taken"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack included the transaction in the microblock)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="cu_​limit"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have exceeded the block CU limit)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="conflict_​fast"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because of an account conflict found by the fast bitvector check)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="byte_​limit"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have exceeded the block data size limit)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="alloc_​limit"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have exceeded the data allocation limit)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="write_​cost"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="conflict_​slow"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because of an account conflict found by the full check)
pack_​txn_​schedule_​attempted
{pack_​txn_​schedule_​result="write_​cost_​deferred"}
counterTimes a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it exceeded the per-account block write cost limit too many times before)
pack_​bundle_​crank_​attempted
{bundle_​crank_​result="up_​to_​date"}
counterTimes pack checked whether the bundle tip state needed cranking, by outcome (On-chain state was already up to date)
pack_​bundle_​crank_​attempted
{bundle_​crank_​result="inserted"}
counterTimes pack checked whether the bundle tip state needed cranking, by outcome (Inserted an initializer bundle to update the on-chain state)
pack_​bundle_​crank_​attempted
{bundle_​crank_​result="creation_​failed"}
counterTimes pack checked whether the bundle tip state needed cranking, by outcome (Tried to insert an initializer bundle to update the on-chain state, but creation failed)
pack_​bundle_​crank_​attempted
{bundle_​crank_​result="insertion_​failed"}
counterTimes pack checked whether the bundle tip state needed cranking, by outcome (Tried to insert an initializer bundle to update the on-chain state, but insertion failed)
pack_​pre_​rotation_​bundle_​pendinggaugeBundles that arrived while not leader that pack is holding
pack_​pre_​rotation_​bundle_​insertedcounterBundles that arrived while not leader that pack kept
pack_​pre_​rotation_​bundle_​rejectedcounterBundles that arrived while not leader that pack dropped because it was full of ones that declare at least as large a tip
pack_​pre_​rotation_​bundle_​evictedcounterBundles that arrived while not leader that pack dropped for one that declares a larger tip
pack_​pre_​rotation_​bundle_​scheduledcounterBundles that arrived while not leader that pack scheduled
pack_​auction_​startedcounterBatch auctions started
pack_​auction_​cancelledcounterBatch auctions skipped because the block was already at their CU limit
pack_​auction_​ended
{pack_​auction_​end_​reason="limit"}
counterBatch auctions ended, by reason (An auction ended because the block reached the auction's CU limit)
pack_​auction_​ended
{pack_​auction_​end_​reason="restarted"}
counterBatch auctions ended, by reason (An auction ended because a new auction was started)
pack_​auction_​ended
{pack_​auction_​end_​reason="final"}
counterBatch auctions ended, by reason (An auction ended because the final 10ms of our leader rotation began)
pack_​auction_​ended
{pack_​auction_​end_​reason="slot_​end"}
counterBatch auctions ended, by reason (An auction ended because the leader slot ended)
pack_​auction_​duration_​nanoshistogramDuration that batch auctions ran for
pack_​block_​costgaugeCost units used in the current block, 0 if not packing a block
pack_​cost_​scheduled_​per_​blockhistogramCost units scheduled for each block pack produced; can exceed the block limit because of rebates
pack_​cu_​rebated_​per_​blockhistogramCompute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses
pack_​cost_​net_​per_​blockhistogramNet cost units (scheduled minus rebated) in each block pack produced
pack_​cost_​per_​block_​pcthistogramPercent of max_cost_per_block used by each block pack produced

Execle Tile ​

MetricTypeDescription
execle_​txn_​processed
{txn_​result="success"}
counterTransactions loaded and executed, by result (Transaction executed successfully)
execle_​txn_​processed
{txn_​result="instruction_​error"}
counterTransactions loaded and executed, by result (An error occurred while processing an instruction)
execle_​txn_​processed
{txn_​result="account_​not_​found"}
counterTransactions loaded and executed, by result (The transaction fee payer address was not found)
execle_​txn_​processed
{txn_​result="program_​account_​not_​found"}
counterTransactions loaded and executed, by result (A program account referenced by the transaction was not found)
execle_​txn_​processed
{txn_​result="insufficient_​funds_​for_​fee"}
counterTransactions loaded and executed, by result (The transaction fee payer did not have balance to pay the fee)
execle_​txn_​processed
{txn_​result="invalid_​account_​for_​fee"}
counterTransactions loaded and executed, by result (The transaction fee payer account is not owned by the system program, or has data that is not a nonce)
execle_​txn_​processed
{txn_​result="already_​processed"}
counterTransactions loaded and executed, by result (The transaction has already been processed in a recent block)
execle_​txn_​processed
{txn_​result="blockhash_​not_​found"}
counterTransactions loaded and executed, by result (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value)
execle_​txn_​processed
{txn_​result="invalid_​program_​for_​execution"}
counterTransactions loaded and executed, by result (A program account referenced by the transaction was no executable.)
execle_​txn_​processed
{txn_​result="address_​lookup_​table_​not_​found"}
counterTransactions loaded and executed, by result (The transaction references an ALUT account that does not exist or is inactive)
execle_​txn_​processed
{txn_​result="invalid_​address_​lookup_​table_​owner"}
counterTransactions loaded and executed, by result (The transaction references an ALUT account that is not owned by the ALUT program account)
execle_​txn_​processed
{txn_​result="invalid_​address_​lookup_​table_​data"}
counterTransactions loaded and executed, by result (The transaction references an ALUT account that contains data which is not a valid ALUT)
execle_​txn_​processed
{txn_​result="invalid_​address_​lookup_​table_​index"}
counterTransactions loaded and executed, by result (The transaction references an account offset from the ALUT which does not exist)
execle_​txn_​processed
{txn_​result="max_​loaded_​accounts_​data_​size_​exceeded"}
counterTransactions loaded and executed, by result (The total account data size of the loaded accounts exceeds the consensus limit)
execle_​txn_​processed
{txn_​result="duplicate_​instruction"}
counterTransactions loaded and executed, by result (A compute budget program instruction was invoked more than once)
execle_​txn_​processed
{txn_​result="invalid_​loaded_​accounts_​data_​size_​limit"}
counterTransactions loaded and executed, by result (The compute budget program was invoked and set the loaded accounts data size to zero)
execle_​txn_​processed
{txn_​result="nonce_​already_​advanced"}
counterTransactions loaded and executed, by result (The transaction references a nonce account that is already advanced)
execle_​txn_​processed
{txn_​result="nonce_​advance_​failed"}
counterTransactions loaded and executed, by result (The transaction is a nonce transaction but the advance instruction was not valid or failed)
execle_​txn_​processed
{txn_​result="nonce_​wrong_​blockhash"}
counterTransactions loaded and executed, by result (The transaction is a nonce transaction but the blockhash is not the correct one)
execle_​txn_​processed
{txn_​result="unsupported_​version"}
counterTransactions loaded and executed, by result (The transaction is a message version that is not supported by the current feature set)
execle_​txn_​processed
{txn_​result="sanitize_​failure"}
counterTransactions loaded and executed, by result (The process for sanitizing the transaction failed)
execle_​txn_​processed
{txn_​result="account_​in_​use"}
counterTransactions loaded and executed, by result (The transaction conflicts with another transaction in the microblock.)
execle_​txn_​processed
{txn_​result="account_​loaded_​twice"}
counterTransactions loaded and executed, by result (The transaction references the same account twice)
execle_​txn_​processed
{txn_​result="signature_​failure"}
counterTransactions loaded and executed, by result (The transaction had an invalid signature)
execle_​txn_​processed
{txn_​result="too_​many_​account_​locks"}
counterTransactions loaded and executed, by result (The transaction references too many accounts.)
execle_​txn_​processed
{txn_​result="insufficient_​funds_​for_​rent"}
counterTransactions loaded and executed, by result (The transaction would leave an account with a lower balance than the rent-exempt minimum)
execle_​txn_​processed
{txn_​result="unbalanced_​transaction"}
counterTransactions loaded and executed, by result (The total referenced account lamports before and after the transaction was unbalanced)
execle_​txn_​processed
{txn_​result="bundle_​peer"}
counterTransactions loaded and executed, by result (The transaction was part of a bundle and an earlier transaction in the bundle failed)
execle_​txn_​processed_​by_​landing
{txn_​landing_​result="success"}
counterTransactions processed, by whether and how they landed in the block (Transaction landed and executed successfully)
execle_​txn_​processed_​by_​landing
{txn_​landing_​result="fees_​only"}
counterTransactions processed, by whether and how they landed in the block (Transaction landed, but was fees only and did not execute)
execle_​txn_​processed_​by_​landing
{txn_​landing_​result="failed"}
counterTransactions processed, by whether and how they landed in the block (Transaction landed, but failed to execute)
execle_​txn_​processed_​by_​landing
{txn_​landing_​result="dropped"}
counterTransactions processed, by whether and how they landed in the block (Transaction did not land)
execle_​txn_​processed_​by_​version
{txn_​version="legacy"}
counterTransactions processed, by transaction version (Legacy transaction format)
execle_​txn_​processed_​by_​version
{txn_​version="v0"}
counterTransactions processed, by transaction version (Version 0 transaction format)
execle_​txn_​processed_​by_​version
{txn_​version="v1"}
counterTransactions processed, by transaction version (Version 1 transaction format)
execle_​instruction_​executedcounterTop-level instructions executed
execle_​cpi_​executedcounterCross-program invocations executed
execle_​account_​lthash_​skippedcounterCommitted writable accounts left byte-identical by their transaction, so their in-band lthash update was skipped
execle_​bank_​switchedcounterCache invalidations due to transaction execution from a different block
execle_​cu_​consumedcounterCompute units consumed by executed transactions
execle_​txn_​regime_​duration_​nanos
{txn_​regime="load"}
counterTime per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction checks and account loading)
execle_​txn_​regime_​duration_​nanos
{txn_​regime="execute"}
counterTime per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction execution, including VM setup)
execle_​txn_​regime_​duration_​nanos
{txn_​regime="commit"}
counterTime per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction result commit)
execle_​vm_​regime_​duration_​nanos
{vm_​regime="setup"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup)
execle_​vm_​regime_​duration_​nanos
{vm_​regime="commit"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit)
execle_​vm_​regime_​duration_​nanos
{vm_​regime="setup_​cpi"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup (CPI))
execle_​vm_​regime_​duration_​nanos
{vm_​regime="commit_​cpi"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit (CPI))
execle_​vm_​regime_​duration_​nanos
{vm_​regime="interpreter"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM interpreter execution)
execle_​progcache_​looked_​upcounterProgram cache lookups; equals Hit + Miss + lookups of invalid program accounts
execle_​progcache_​misscounterProgram cache misses, each followed by an insert or a spill
execle_​progcache_​waitedcounterProgram cache lookups that waited on another tile's in-flight load of the same program
execle_​progcache_​insert_​retriedcounterProgram cache inserts that found no free slot in their size class on the first try and retried after eviction
execle_​progcache_​inserted_​bytescounterBytes inserted into the program cache
execle_​progcache_​spilled_​bytescounterBytes loaded into program cache spill memory
execle_​progcache_​evicted_​bytescounterBytes evicted from the program cache by insert eviction sweeps
execle_​progcache_​duration_​nanoscounterTime in program cache lookups, including waits on peer loads and insert, eviction and ELF load on a miss; superset of ProgcacheLoadDurationNanos
execle_​progcache_​loadedcounterProgram ELF loads and verifications for the program cache, including failed verifications and spill-backed loads
execle_​progcache_​load_​duration_​nanoscounterTime spent loading programs
execle_​progcache_​hit
{progcache_​class="le128_​kib"}
counterProgram cache hits, by size class (0-128 KiB)
execle_​progcache_​hit
{progcache_​class="le512_​kib"}
counterProgram cache hits, by size class (128-512 KiB)
execle_​progcache_​hit
{progcache_​class="le1_​mib"}
counterProgram cache hits, by size class (512 KiB-1 MiB)
execle_​progcache_​hit
{progcache_​class="le2_​mib"}
counterProgram cache hits, by size class (1-2 MiB)
execle_​progcache_​hit
{progcache_​class="le4_​mib"}
counterProgram cache hits, by size class (2-4 MiB)
execle_​progcache_​hit
{progcache_​class="le11_​mib"}
counterProgram cache hits, by size class (4-11 MiB)
execle_​progcache_​inserted
{progcache_​class="le128_​kib"}
counterPrograms inserted into the program cache, by size class (0-128 KiB)
execle_​progcache_​inserted
{progcache_​class="le512_​kib"}
counterPrograms inserted into the program cache, by size class (128-512 KiB)
execle_​progcache_​inserted
{progcache_​class="le1_​mib"}
counterPrograms inserted into the program cache, by size class (512 KiB-1 MiB)
execle_​progcache_​inserted
{progcache_​class="le2_​mib"}
counterPrograms inserted into the program cache, by size class (1-2 MiB)
execle_​progcache_​inserted
{progcache_​class="le4_​mib"}
counterPrograms inserted into the program cache, by size class (2-4 MiB)
execle_​progcache_​inserted
{progcache_​class="le11_​mib"}
counterPrograms inserted into the program cache, by size class (4-11 MiB)
execle_​progcache_​evicted
{progcache_​class="le128_​kib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (0-128 KiB)
execle_​progcache_​evicted
{progcache_​class="le512_​kib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (128-512 KiB)
execle_​progcache_​evicted
{progcache_​class="le1_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (512 KiB-1 MiB)
execle_​progcache_​evicted
{progcache_​class="le2_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (1-2 MiB)
execle_​progcache_​evicted
{progcache_​class="le4_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (2-4 MiB)
execle_​progcache_​evicted
{progcache_​class="le11_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (4-11 MiB)
execle_​progcache_​spilled
{progcache_​class="le128_​kib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (0-128 KiB)
execle_​progcache_​spilled
{progcache_​class="le512_​kib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (128-512 KiB)
execle_​progcache_​spilled
{progcache_​class="le1_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (512 KiB-1 MiB)
execle_​progcache_​spilled
{progcache_​class="le2_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (1-2 MiB)
execle_​progcache_​spilled
{progcache_​class="le4_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (2-4 MiB)
execle_​progcache_​spilled
{progcache_​class="le11_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (4-11 MiB)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le512_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le2_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le8_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le32_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le1_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
execle_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le10_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le128_​b"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le512_​b"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le2_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le8_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le32_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le128_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le1_​mib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
execle_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le10_​mib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le128_​b"}
counterAccounts evicted from the account database cache to make space, by cache size class (0-128 B)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le512_​b"}
counterAccounts evicted from the account database cache to make space, by cache size class (129-512 B)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le2_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (513 B-2 KiB)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le8_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (2-8 KiB)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le32_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (8-32 KiB)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le128_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (32-128 KiB)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le1_​mib"}
counterAccounts evicted from the account database cache to make space, by cache size class (128 KiB-1 MiB)
execle_​accdb_​account_​evicted
{accdb_​cache_​class="le10_​mib"}
counterAccounts evicted from the account database cache to make space, by cache size class (1-10 MiB)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le128_​b"}
counterNew account versions committed to the index, by destination cache size class (0-128 B)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le512_​b"}
counterNew account versions committed to the index, by destination cache size class (129-512 B)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le2_​kib"}
counterNew account versions committed to the index, by destination cache size class (513 B-2 KiB)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le8_​kib"}
counterNew account versions committed to the index, by destination cache size class (2-8 KiB)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le32_​kib"}
counterNew account versions committed to the index, by destination cache size class (8-32 KiB)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le128_​kib"}
counterNew account versions committed to the index, by destination cache size class (32-128 KiB)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le1_​mib"}
counterNew account versions committed to the index, by destination cache size class (128 KiB-1 MiB)
execle_​accdb_​account_​new_​committed
{accdb_​cache_​class="le10_​mib"}
counterNew account versions committed to the index, by destination cache size class (1-10 MiB)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le128_​b"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (0-128 B)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le512_​b"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (129-512 B)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le2_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (513 B-2 KiB)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le8_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (2-8 KiB)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le32_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (8-32 KiB)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le128_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (32-128 KiB)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le1_​mib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (128 KiB-1 MiB)
execle_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le10_​mib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (1-10 MiB)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (0-128 B)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le512_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (129-512 B)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le2_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le8_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (2-8 KiB)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le32_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (8-32 KiB)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (32-128 KiB)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le1_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB)
execle_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le10_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (1-10 MiB)
execle_​accdb_​account_​waitedcounterAccounts that had to wait for another reader to finish loading them into the cache
execle_​accdb_​batch_​acquiredcounterBatch acquire calls made by this tile
execle_​accdb_​cache_​reserve_​retriedcounterSpin retries while waiting to reserve cache lines for a transaction
execle_​accdb_​disk_​read_​bytescounterBytes read from the account database file
execle_​accdb_​disk_​readcounterReads from the account database file
execle_​accdb_​disk_​written_​bytescounterBytes written to the account database file
execle_​accdb_​disk_​writtencounterWrites to the account database file
execle_​accdb_​copied_​bytescounterAccount data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
execle_​txncache_​query_​misscounterTransaction cache queries by this tile that found a page not in memory
execle_​txncache_​insert_​misscounterTransaction cache inserts by this tile that found a page not in memory
execle_​txncache_​operation_​stalledcounterTransaction cache queries and inserts by this tile that fell back to the exclusive lock, stalling every tile; pages they load are read while holding it
execle_​txncache_​query_​duration_​nanoshistogramDuration of one transaction cache query
execle_​txncache_​insert_​duration_​nanoshistogramDuration of one transaction cache insert

Sign Tile ​

MetricTypeDescription
sign_​sign_​duration_​nanoshistogramDuration of signing a message

Shred Tile ​

MetricTypeDescription
shred_​batch_​abandonedcounterMicroblock batches abandoned because we switched slots without finishing the current slot
shred_​parent_​block_​id_​zeroedcounterLeader blocks whose parent block_id was unknown, so the chained merkle root was zeroed
shred_​batch_​size_​byteshistogramSize of each microblock batch that is shredded
shred_​microblock_​per_​batchhistogramMicroblocks in each microblock batch that is shredded
shred_​batch_​shred_​duration_​nanoshistogramDuration of shredding one microblock batch
shred_​add_​shred_​duration_​nanoshistogramDuration of verifying and processing one shred received from the network
shred_​store_​fec_​sync_​spill_​duration_​nanoshistogramDuration of synchronously spilling one FEC payload to the store
shred_​processed
{shred_​process_​result="own_​leader_​echoed"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Turbine shred for a slot we led, echoed back to us by a relay)
shred_​processed
{shred_​process_​result="leader_​unknown"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was for a slot whose leader is unknown)
shred_​processed
{shred_​process_​result="parse_​failed"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred parsing failed)
shred_​processed
{shred_​process_​result="equivocated"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was equivocated with another shred)
shred_​processed
{shred_​process_​result="rejected"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was invalid for one of many reasons)
shred_​processed
{shred_​process_​result="ignored"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was ignored because we had already reconstructed the FEC set, or was for an old slot)
shred_​processed
{shred_​process_​result="duplicate"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was a duplicate of a previously received valid shred of an incomplete FEC set)
shred_​processed
{shred_​process_​result="accepted"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred accepted into an incomplete FEC set)
shred_​processed
{shred_​process_​result="completed"}
counterShreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred accepted and resulted in a valid, complete FEC set)
shred_​fec_​incomplete_​evictedcounterIncomplete FEC sets evicted from the resolver to make room for new ones
shred_​initial_​rejectedcounterShreds rejected before any resources were allocated for the FEC set
shred_​unchained_​rejectedcounterShreds rejected because they're not chained merkle shreds
shred_​fec_​fatal_​rejectedcounterFEC sets rejected for reasons that cause the whole FEC set to become invalid
shred_​repair_​nonce_​failedcounterShreds that looked like repair responses but had an incorrect nonce
shred_​repair_​rxcounterRepair shreds received
shred_​repair_​rx_​bytescounterBytes received from network packets with repair shreds, including network headers
shred_​turbine_​rxcounterTurbine shreds received
shred_​turbine_​rx_​bytescounterBytes received from network packets with turbine shreds, including network headers
shred_​repair_​completion_​assistedcounterFEC sets that completed with at least one repair shred contributing
shred_​repair_​lead_​delay_​nanoshistogramEstimated time repair advanced FEC set completion: the delay from a repair-assisted completion until turbine had delivered as many shreds for that set as repair contributed. Counts arrivals rather than distinct indices, so turbine duplicates bias this low
shred_​store_​fec_​sync_​spilledcounterFEC payloads the shred tile spilled to the store synchronously because the store RAM cache was full
shred_​store_​fec_​sync_​spilled_​bytescounterFEC payload bytes the shred tile spilled to the store synchronously

Gossvf Tile ​

MetricTypeDescription
gossvf_​message_​rx
{gossvf_​message_​result="success_​pull_​request"}
counterGossip messages received, by verification result (Pull Request (success))
gossvf_​message_​rx
{gossvf_​message_​result="success_​pull_​response"}
counterGossip messages received, by verification result (Pull Response (success))
gossvf_​message_​rx
{gossvf_​message_​result="success_​push"}
counterGossip messages received, by verification result (Push (success))
gossvf_​message_​rx
{gossvf_​message_​result="success_​prune"}
counterGossip messages received, by verification result (Prune (success))
gossvf_​message_​rx
{gossvf_​message_​result="success_​ping"}
counterGossip messages received, by verification result (Ping (success))
gossvf_​message_​rx
{gossvf_​message_​result="success_​pong"}
counterGossip messages received, by verification result (Pong (success))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​unparseable"}
counterGossip messages received, by verification result (Unparseable)
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​not_​contact_​info"}
counterGossip messages received, by verification result (Pull Request (not contact info))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​loopback"}
counterGossip messages received, by verification result (Pull Request (loopback))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​inactive"}
counterGossip messages received, by verification result (Pull Request (inactive))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​wallclock"}
counterGossip messages received, by verification result (Pull Request (wallclock))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​signature"}
counterGossip messages received, by verification result (Pull Request (signature))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​shred_​version"}
counterGossip messages received, by verification result (Pull Request (shred version))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​request_​mask_​bits"}
counterGossip messages received, by verification result (Pull Request (mask bits))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​prune_​destination"}
counterGossip messages received, by verification result (Prune (destination))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​prune_​wallclock"}
counterGossip messages received, by verification result (Prune (wallclock))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​prune_​signature"}
counterGossip messages received, by verification result (Prune (signature))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​push_​loopback"}
counterGossip messages received, by verification result (Push (loopback))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​push_​no_​valid_​crds"}
counterGossip messages received, by verification result (Push (no valid crds))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​response_​loopback"}
counterGossip messages received, by verification result (Pull Response (loopback))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pull_​response_​no_​valid_​crds"}
counterGossip messages received, by verification result (Pull Response (no valid crds))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​ping_​signature"}
counterGossip messages received, by verification result (Ping (signature))
gossvf_​message_​rx
{gossvf_​message_​result="dropped_​pong_​signature"}
counterGossip messages received, by verification result (Pong (signature))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="success_​pull_​request"}
counterWire bytes of gossip messages received, by verification result (Pull Request (success))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="success_​pull_​response"}
counterWire bytes of gossip messages received, by verification result (Pull Response (success))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="success_​push"}
counterWire bytes of gossip messages received, by verification result (Push (success))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="success_​prune"}
counterWire bytes of gossip messages received, by verification result (Prune (success))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="success_​ping"}
counterWire bytes of gossip messages received, by verification result (Ping (success))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="success_​pong"}
counterWire bytes of gossip messages received, by verification result (Pong (success))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​unparseable"}
counterWire bytes of gossip messages received, by verification result (Unparseable)
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​not_​contact_​info"}
counterWire bytes of gossip messages received, by verification result (Pull Request (not contact info))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​loopback"}
counterWire bytes of gossip messages received, by verification result (Pull Request (loopback))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​inactive"}
counterWire bytes of gossip messages received, by verification result (Pull Request (inactive))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​wallclock"}
counterWire bytes of gossip messages received, by verification result (Pull Request (wallclock))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​signature"}
counterWire bytes of gossip messages received, by verification result (Pull Request (signature))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​shred_​version"}
counterWire bytes of gossip messages received, by verification result (Pull Request (shred version))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​request_​mask_​bits"}
counterWire bytes of gossip messages received, by verification result (Pull Request (mask bits))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​prune_​destination"}
counterWire bytes of gossip messages received, by verification result (Prune (destination))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​prune_​wallclock"}
counterWire bytes of gossip messages received, by verification result (Prune (wallclock))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​prune_​signature"}
counterWire bytes of gossip messages received, by verification result (Prune (signature))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​push_​loopback"}
counterWire bytes of gossip messages received, by verification result (Push (loopback))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​push_​no_​valid_​crds"}
counterWire bytes of gossip messages received, by verification result (Push (no valid crds))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​response_​loopback"}
counterWire bytes of gossip messages received, by verification result (Pull Response (loopback))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pull_​response_​no_​valid_​crds"}
counterWire bytes of gossip messages received, by verification result (Pull Response (no valid crds))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​ping_​signature"}
counterWire bytes of gossip messages received, by verification result (Ping (signature))
gossvf_​message_​rx_​bytes
{gossvf_​message_​result="dropped_​pong_​signature"}
counterWire bytes of gossip messages received, by verification result (Pong (signature))
gossvf_​crds_​rx
{gossvf_​crds_​result="verified_​pull_​response"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (success))
gossvf_​crds_​rx
{gossvf_​crds_​result="duplicate_​pull_​response"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (duplicate))
gossvf_​crds_​rx
{gossvf_​crds_​result="verified_​push"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (success))
gossvf_​crds_​rx
{gossvf_​crds_​result="duplicate_​push"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (duplicate))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​pull_​response_​signature"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (signature))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​pull_​response_​no_​contact_​info"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (origin no contact info))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​pull_​response_​shred_​version"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (origin shred version))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​pull_​response_​inactive"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (inactive))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​pull_​response_​wallclock"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (wallclock))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​push_​signature"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (signature))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​push_​no_​contact_​info"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (origin no contact info))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​push_​shred_​version"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (origin shred version))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​push_​inactive"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (inactive))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​push_​wallclock"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (wallclock))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​pull_​response_​deprecated"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (deprecated value type))
gossvf_​crds_​rx
{gossvf_​crds_​result="dropped_​push_​deprecated"}
counterCRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (deprecated value type))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="verified_​pull_​response"}
counterWire bytes of CRDS values received, by verification result (Pull Response (success))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="duplicate_​pull_​response"}
counterWire bytes of CRDS values received, by verification result (Pull Response (duplicate))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="verified_​push"}
counterWire bytes of CRDS values received, by verification result (Push (success))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="duplicate_​push"}
counterWire bytes of CRDS values received, by verification result (Push (duplicate))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​pull_​response_​signature"}
counterWire bytes of CRDS values received, by verification result (Pull Response (signature))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​pull_​response_​no_​contact_​info"}
counterWire bytes of CRDS values received, by verification result (Pull Response (origin no contact info))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​pull_​response_​shred_​version"}
counterWire bytes of CRDS values received, by verification result (Pull Response (origin shred version))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​pull_​response_​inactive"}
counterWire bytes of CRDS values received, by verification result (Pull Response (inactive))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​pull_​response_​wallclock"}
counterWire bytes of CRDS values received, by verification result (Pull Response (wallclock))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​push_​signature"}
counterWire bytes of CRDS values received, by verification result (Push (signature))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​push_​no_​contact_​info"}
counterWire bytes of CRDS values received, by verification result (Push (origin no contact info))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​push_​shred_​version"}
counterWire bytes of CRDS values received, by verification result (Push (origin shred version))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​push_​inactive"}
counterWire bytes of CRDS values received, by verification result (Push (inactive))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​push_​wallclock"}
counterWire bytes of CRDS values received, by verification result (Push (wallclock))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​pull_​response_​deprecated"}
counterWire bytes of CRDS values received, by verification result (Pull Response (deprecated value type))
gossvf_​crds_​rx_​bytes
{gossvf_​crds_​result="dropped_​push_​deprecated"}
counterWire bytes of CRDS values received, by verification result (Push (deprecated value type))

Gossip Tile ​

MetricTypeDescription
gossip_​ping_​peer_​count
{ping_​peer_​state="unpinged"}
gaugePeers tracked for ping/pong, by state (Peer is known but has not been pinged yet)
gossip_​ping_​peer_​count
{ping_​peer_​state="awaiting_​pong"}
gaugePeers tracked for ping/pong, by state (Peer was pinged but has not yet responded, or responded with an invalid pong)
gossip_​ping_​peer_​count
{ping_​peer_​state="valid"}
gaugePeers tracked for ping/pong, by state (Peer was pinged and is currently valid)
gossip_​ping_​peer_​count
{ping_​peer_​state="refreshing"}
gaugePeers tracked for ping/pong, by state (Peer was pinged and is currently valid, and is being refreshed with a new ping)
gossip_​pong_​processed
{pong_​result="accepted"}
counterPongs processed, by result (Pong accepted)
gossip_​pong_​processed
{pong_​result="staked"}
counterPongs processed, by result (Ignored because the peer is staked and needs no ping)
gossip_​pong_​processed
{pong_​result="untracked"}
counterPongs processed, by result (Ignored because the peer has no outstanding ping)
gossip_​pong_​processed
{pong_​result="address_​mismatch"}
counterPongs processed, by result (Ignored because it came from a different address than the one pinged)
gossip_​pong_​processed
{pong_​result="token_​mismatch"}
counterPongs processed, by result (Ignored because the token did not match)
gossip_​ping_​peer_​addedcounterPeers added to the ping tracker
gossip_​ping_​peer_​removed
{ping_​peer_​remove_​reason="evicted"}
counterPeers removed from the ping tracker, by reason (Evicted to make room for a new peer)
gossip_​ping_​peer_​removed
{ping_​peer_​remove_​reason="staked"}
counterPeers removed from the ping tracker, by reason (Became staked, no longer pinged)
gossip_​ping_​peer_​removed
{ping_​peer_​remove_​reason="stale"}
counterPeers removed from the ping tracker, by reason (Contact info not received for 60 s)
gossip_​ping_​peer_​removed
{ping_​peer_​remove_​reason="identity"}
counterPeers removed from the ping tracker, by reason (Became our identity)
gossip_​ping_​peer_​address_​changedcounterTracked peers whose gossip address changed; pong invalidated and reset to unpinged
gossip_​sign_​queue_​full_​droppedcounterMessages dropped because the maximum number of sign requests were in flight
gossip_​crds_​capacitygaugeCapacity of the data store
gossip_​crds_​used
{crds_​value_​type="contact_​info_​v1"}
gaugeEntries in the data store (Contact Info V1)
gossip_​crds_​used
{crds_​value_​type="vote"}
gaugeEntries in the data store (Vote)
gossip_​crds_​used
{crds_​value_​type="lowest_​slot"}
gaugeEntries in the data store (Lowest Slot)
gossip_​crds_​used
{crds_​value_​type="snapshot_​hashes"}
gaugeEntries in the data store (Snapshot Hashes)
gossip_​crds_​used
{crds_​value_​type="accounts_​hashes"}
gaugeEntries in the data store (Accounts Hashes)
gossip_​crds_​used
{crds_​value_​type="epoch_​slots"}
gaugeEntries in the data store (Epoch Slots)
gossip_​crds_​used
{crds_​value_​type="version_​v1"}
gaugeEntries in the data store (Version V1)
gossip_​crds_​used
{crds_​value_​type="version_​v2"}
gaugeEntries in the data store (Version V2)
gossip_​crds_​used
{crds_​value_​type="node_​instance"}
gaugeEntries in the data store (Node Instance)
gossip_​crds_​used
{crds_​value_​type="duplicate_​shred"}
gaugeEntries in the data store (Duplicate Shred)
gossip_​crds_​used
{crds_​value_​type="incremental_​snapshot_​hashes"}
gaugeEntries in the data store (Incremental Snapshot Hashes)
gossip_​crds_​used
{crds_​value_​type="contact_​info_​v2"}
gaugeEntries in the data store (Contact Info V2)
gossip_​crds_​used
{crds_​value_​type="restart_​last_​voted_​fork_​slots"}
gaugeEntries in the data store (Restart Last Voted Fork Slots)
gossip_​crds_​used
{crds_​value_​type="restart_​heaviest_​fork"}
gaugeEntries in the data store (Restart Heaviest Fork)
gossip_​crds_​expiredcounterEntries expired from the data store due to age
gossip_​crds_​evictedcounterEntries evicted from the data store to make space for new entries
gossip_​crds_​relay_​coalescedcounterReceived entry versions not relayed because a newer version of the same entry replaced them within the relay interval
gossip_​crds_​peer_​capacitygaugeCapacity for storing peers in the data store
gossip_​crds_​peer_​count
{crds_​peer_​stake_​class="staked"}
gaugePeers in the data store, by stake class (Peer has stake)
gossip_​crds_​peer_​count
{crds_​peer_​stake_​class="unstaked"}
gaugePeers in the data store, by stake class (Peer has no stake)
gossip_​crds_​peer_​stake_​lamportsgaugeStake of peers in the data store
gossip_​crds_​peer_​evictedcounterPeers evicted from the data store to make space for new peers
gossip_​crds_​purged_​capacitygaugeCapacity of the list of purged data store entries
gossip_​crds_​purged_​usedgaugePurged data store entries
gossip_​crds_​purged_​expiredcounterPurged entries expired from the purged list due to age
gossip_​crds_​purged_​evictedcounterEntries evicted from the purged list to make room
gossip_​crds_​upserted
{gossip_​crds_​result="inserted_​pull_​response"}
counterVerified CRDS values offered to the data store, by result (Pull Response (upserted))
gossip_​crds_​upserted
{gossip_​crds_​result="inserted_​push"}
counterVerified CRDS values offered to the data store, by result (Push (upserted))
gossip_​crds_​upserted
{gossip_​crds_​result="stale_​pull_​response"}
counterVerified CRDS values offered to the data store, by result (Pull Response (newer entry already present in table))
gossip_​crds_​upserted
{gossip_​crds_​result="duplicate_​pull_​response"}
counterVerified CRDS values offered to the data store, by result (Pull Response (duplicate))
gossip_​crds_​upserted
{gossip_​crds_​result="stale_​push"}
counterVerified CRDS values offered to the data store, by result (Push (newer entry already present in table))
gossip_​crds_​upserted
{gossip_​crds_​result="duplicate_​push"}
counterVerified CRDS values offered to the data store, by result (Push (duplicate))
gossip_​message_​tx
{gossip_​message_​type="pull_​request"}
counterGossip messages sent, by message type (Pull Request)
gossip_​message_​tx
{gossip_​message_​type="pull_​response"}
counterGossip messages sent, by message type (Pull Response)
gossip_​message_​tx
{gossip_​message_​type="push"}
counterGossip messages sent, by message type (Push)
gossip_​message_​tx
{gossip_​message_​type="prune"}
counterGossip messages sent, by message type (Prune)
gossip_​message_​tx
{gossip_​message_​type="ping"}
counterGossip messages sent, by message type (Ping)
gossip_​message_​tx
{gossip_​message_​type="pong"}
counterGossip messages sent, by message type (Pong)
gossip_​message_​tx_​bytes
{gossip_​message_​type="pull_​request"}
counterWire bytes sent in gossip messages, by message type (Pull Request)
gossip_​message_​tx_​bytes
{gossip_​message_​type="pull_​response"}
counterWire bytes sent in gossip messages, by message type (Pull Response)
gossip_​message_​tx_​bytes
{gossip_​message_​type="push"}
counterWire bytes sent in gossip messages, by message type (Push)
gossip_​message_​tx_​bytes
{gossip_​message_​type="prune"}
counterWire bytes sent in gossip messages, by message type (Prune)
gossip_​message_​tx_​bytes
{gossip_​message_​type="ping"}
counterWire bytes sent in gossip messages, by message type (Ping)
gossip_​message_​tx_​bytes
{gossip_​message_​type="pong"}
counterWire bytes sent in gossip messages, by message type (Pong)
gossip_​crds_​push_​tx
{crds_​value_​type="contact_​info_​v1"}
counterCRDS values sent in push messages, by value type (Contact Info V1)
gossip_​crds_​push_​tx
{crds_​value_​type="vote"}
counterCRDS values sent in push messages, by value type (Vote)
gossip_​crds_​push_​tx
{crds_​value_​type="lowest_​slot"}
counterCRDS values sent in push messages, by value type (Lowest Slot)
gossip_​crds_​push_​tx
{crds_​value_​type="snapshot_​hashes"}
counterCRDS values sent in push messages, by value type (Snapshot Hashes)
gossip_​crds_​push_​tx
{crds_​value_​type="accounts_​hashes"}
counterCRDS values sent in push messages, by value type (Accounts Hashes)
gossip_​crds_​push_​tx
{crds_​value_​type="epoch_​slots"}
counterCRDS values sent in push messages, by value type (Epoch Slots)
gossip_​crds_​push_​tx
{crds_​value_​type="version_​v1"}
counterCRDS values sent in push messages, by value type (Version V1)
gossip_​crds_​push_​tx
{crds_​value_​type="version_​v2"}
counterCRDS values sent in push messages, by value type (Version V2)
gossip_​crds_​push_​tx
{crds_​value_​type="node_​instance"}
counterCRDS values sent in push messages, by value type (Node Instance)
gossip_​crds_​push_​tx
{crds_​value_​type="duplicate_​shred"}
counterCRDS values sent in push messages, by value type (Duplicate Shred)
gossip_​crds_​push_​tx
{crds_​value_​type="incremental_​snapshot_​hashes"}
counterCRDS values sent in push messages, by value type (Incremental Snapshot Hashes)
gossip_​crds_​push_​tx
{crds_​value_​type="contact_​info_​v2"}
counterCRDS values sent in push messages, by value type (Contact Info V2)
gossip_​crds_​push_​tx
{crds_​value_​type="restart_​last_​voted_​fork_​slots"}
counterCRDS values sent in push messages, by value type (Restart Last Voted Fork Slots)
gossip_​crds_​push_​tx
{crds_​value_​type="restart_​heaviest_​fork"}
counterCRDS values sent in push messages, by value type (Restart Heaviest Fork)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="contact_​info_​v1"}
counterWire bytes of CRDS values sent in push messages, by value type (Contact Info V1)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="vote"}
counterWire bytes of CRDS values sent in push messages, by value type (Vote)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="lowest_​slot"}
counterWire bytes of CRDS values sent in push messages, by value type (Lowest Slot)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="snapshot_​hashes"}
counterWire bytes of CRDS values sent in push messages, by value type (Snapshot Hashes)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="accounts_​hashes"}
counterWire bytes of CRDS values sent in push messages, by value type (Accounts Hashes)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="epoch_​slots"}
counterWire bytes of CRDS values sent in push messages, by value type (Epoch Slots)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="version_​v1"}
counterWire bytes of CRDS values sent in push messages, by value type (Version V1)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="version_​v2"}
counterWire bytes of CRDS values sent in push messages, by value type (Version V2)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="node_​instance"}
counterWire bytes of CRDS values sent in push messages, by value type (Node Instance)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="duplicate_​shred"}
counterWire bytes of CRDS values sent in push messages, by value type (Duplicate Shred)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="incremental_​snapshot_​hashes"}
counterWire bytes of CRDS values sent in push messages, by value type (Incremental Snapshot Hashes)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="contact_​info_​v2"}
counterWire bytes of CRDS values sent in push messages, by value type (Contact Info V2)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="restart_​last_​voted_​fork_​slots"}
counterWire bytes of CRDS values sent in push messages, by value type (Restart Last Voted Fork Slots)
gossip_​crds_​push_​tx_​bytes
{crds_​value_​type="restart_​heaviest_​fork"}
counterWire bytes of CRDS values sent in push messages, by value type (Restart Heaviest Fork)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="contact_​info_​v1"}
counterCRDS values sent in pull response messages, by value type (Contact Info V1)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="vote"}
counterCRDS values sent in pull response messages, by value type (Vote)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="lowest_​slot"}
counterCRDS values sent in pull response messages, by value type (Lowest Slot)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="snapshot_​hashes"}
counterCRDS values sent in pull response messages, by value type (Snapshot Hashes)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="accounts_​hashes"}
counterCRDS values sent in pull response messages, by value type (Accounts Hashes)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="epoch_​slots"}
counterCRDS values sent in pull response messages, by value type (Epoch Slots)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="version_​v1"}
counterCRDS values sent in pull response messages, by value type (Version V1)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="version_​v2"}
counterCRDS values sent in pull response messages, by value type (Version V2)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="node_​instance"}
counterCRDS values sent in pull response messages, by value type (Node Instance)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="duplicate_​shred"}
counterCRDS values sent in pull response messages, by value type (Duplicate Shred)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="incremental_​snapshot_​hashes"}
counterCRDS values sent in pull response messages, by value type (Incremental Snapshot Hashes)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="contact_​info_​v2"}
counterCRDS values sent in pull response messages, by value type (Contact Info V2)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="restart_​last_​voted_​fork_​slots"}
counterCRDS values sent in pull response messages, by value type (Restart Last Voted Fork Slots)
gossip_​crds_​pull_​response_​tx
{crds_​value_​type="restart_​heaviest_​fork"}
counterCRDS values sent in pull response messages, by value type (Restart Heaviest Fork)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="contact_​info_​v1"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Contact Info V1)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="vote"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Vote)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="lowest_​slot"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Lowest Slot)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="snapshot_​hashes"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Snapshot Hashes)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="accounts_​hashes"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Accounts Hashes)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="epoch_​slots"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Epoch Slots)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="version_​v1"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Version V1)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="version_​v2"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Version V2)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="node_​instance"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Node Instance)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="duplicate_​shred"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Duplicate Shred)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="incremental_​snapshot_​hashes"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Incremental Snapshot Hashes)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="contact_​info_​v2"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Contact Info V2)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="restart_​last_​voted_​fork_​slots"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Restart Last Voted Fork Slots)
gossip_​crds_​pull_​response_​tx_​bytes
{crds_​value_​type="restart_​heaviest_​fork"}
counterWire bytes of CRDS values sent in pull response messages, by value type (Restart Heaviest Fork)
gossip_​wait_​for_​supermajority_​staked_​peer_​onlinegaugeStaked peers with recent gossip activity
gossip_​wait_​for_​supermajority_​staked_​peer_​countgaugeStaked peers tracked while waiting for supermajority
gossip_​wait_​for_​supermajority_​stake_​online_​lamportsgaugeStake from peers with recent gossip activity
gossip_​wait_​for_​supermajority_​stake_​tracked_​lamportsgaugeStake tracked while waiting for supermajority
gossip_​wait_​for_​supermajority_​stategaugeWait-for-supermajority progress: 0=unset (Not started), 1=loading_stakes (Loading stakes from the snapshot), 2=waiting_for_stake (Waiting for enough stake to come online), 3=publishing (Publishing our contact info to start), 4=done_or_disabled (Done, or wait-for-supermajority is disabled)
gossip_​contact_​info_​client_​idgaugeClient id currently advertised in our ContactInfo

Repair Tile ​

MetricTypeDescription
repair_​pkt_​txcounterNetwork packets sent, including requests, pings and pongs
repair_​request_​tx
{repair_​request_​type="pong"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Pong)
repair_​request_​tx
{repair_​request_​type="window_​index"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Shred at an index)
repair_​request_​tx
{repair_​request_​type="highest_​window_​index"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Highest shred at or above an index)
repair_​request_​tx
{repair_​request_​type="orphan"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Ancestors of an orphan slot)
repair_​request_​tx
{repair_​request_​type="parent_​fec_​set_​count"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Parent and FEC set count of a block (Alpenglow))
repair_​request_​tx
{repair_​request_​type="fec_​set_​root"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Merkle root of a FEC set (Alpenglow))
repair_​request_​tx
{repair_​request_​type="window_​index_​for_​block_​id"}
counterSigned repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Shred at an index of a specific block (Alpenglow))
repair_​highest_​delivered_​slotgaugeHighest slot fully delivered to replay, 0 if none
repair_​highest_​received_​slotgaugeHighest slot a shred was received for, 0 if none
repair_​first_​turbine_​slotgaugeFirst turbine slot seen, the catchup target, 0 if none
repair_​catchup_​slotgaugeHighest slot past the window with a verified cert, the staked catchup target, 0 if none
repair_​block_​pendinggaugeBlocks in the eager, notar and final block treaps, waiting for discovery
repair_​request_​in_​flightgaugePending request attempts awaiting a response; a hedge is another attempt
repair_​fec_​deliveredcounterFEC sets published to replay
repair_​shred_​old_​ignoredcounterShreds received at or below the root, ignored
repair_​shred_​rx
{shred_​response_​match_​result="for_​block_​id"}
counterData shreds received as repair responses, by the pending request they matched (Response to a pending WindowIndexForBlockId request)
repair_​shred_​rx
{shred_​response_​match_​result="slot_​index"}
counterData shreds received as repair responses, by the pending request they matched (Response to a pending WindowIndex or HighestWindowIndex request)
repair_​shred_​rx
{shred_​response_​match_​result="unmatched"}
counterData shreds received as repair responses, by the pending request they matched (Matched no pending request: a duplicate, or an answer to an expired or evicted request)
repair_​shred_​rx
{shred_​response_​match_​result="eqvoc"}
counterData shreds received as repair responses, by the pending request they matched (Repair response that showed the slot's leader equivocating, dropped before matching; only unverified-nonce HighestWindowIndex or Orphan replies)
repair_​meta_​rxcounterAlpenglow repair metadata responses received
repair_​meta_​malformed_​rejectedcounterMetadata responses that failed to decode
repair_​meta_​unsolicited_​rejectedcounterMetadata responses matching no outstanding request
repair_​parent_​fec_​set_​count_​acceptedcounterParentFecSetCount responses accepted
repair_​parent_​fec_​set_​count_​failedcounterParentFecSetCount responses that failed verification
repair_​fec_​set_​root_​acceptedcounterFecSetRoot responses accepted
repair_​fec_​set_​root_​failedcounterFecSetRoot responses that failed verification
repair_​replay_​root_​advanced_​rxcounterRoot advanced messages received from replay
repair_​replay_​missing_​fec_​rxcounterMissing FEC messages received from replay, each a from-root reconsume
repair_​ping_​rejected
{ping_​reject_​reason="malformed"}
counterPings rejected, by reason (Failed to decode)
repair_​ping_​rejected
{ping_​reject_​reason="unknown_​peer"}
counterPings rejected, by reason (Sender is not a known peer)
repair_​ping_​rejected
{ping_​reject_​reason="signature_​failed"}
counterPings rejected, by reason (Signature did not verify)
repair_​ping_​rejected
{ping_​reject_​reason="sign_​unavailable"}
counterPings rejected, by reason (Pong not sent: signing halted for an identity switch, or no free sign slot)
repair_​ping_​rejected
{ping_​reject_​reason="address_​mismatch"}
counterPings rejected, by reason (Pong not sent: sender address differs from the peer's gossip address)
repair_​ping_​rejected
{ping_​reject_​reason="unsolicited"}
counterPings rejected, by reason (Pong not sent: no request was sent to the peer within the ping TTL)
repair_​response_​latency_​nanoshistogramTime from sending a repair request to receiving its response
repair_​retry_​delay_​nanoshistogramDelay scheduled before a request's next attempt: its peer's hedge time, 16.8 ms to 100 ms, 16.8 ms after an escape
repair_​timeout_​pendinggaugeTimeouts in the eager, notar and final timeout queues
repair_​timeout_​cancelledcounterTimeouts dropped because their block is gone or their work is done
repair_​timeout_​no_​peer_​deferredcounterTimeouts deferred because no peer could be picked
repair_​request_​hedgedcounterRequest attempts after the first for the same timer
repair_​request_​expiredcounterPending requests with no response after about 1 s (PENDING_TTL); a peer is banned for 10 s after 32 in a row, pending-pool evictions not counted
repair_​request_​by_​id_​expiredcounterWindowIndexForBlockId requests that expired unanswered, at or above the root
repair_​request_​stuck_​escapedcounterRetries sent to a random staked peer because 3 attempts in a row brought no new shred of the FEC set
repair_​request_​late_​retriedcounterRequest attempts sent after 3 or more earlier attempts for the same timer
repair_​escape_​peer_​hedge_​delay_​nanoshistogramUncapped hedge time (srtt+4*rttvar, 100 ms if unmeasured) of each staked peer an escaped retry was sent to
repair_​peer_​by_​id_​bannedcounterPeers banned for 10 s after 4 WindowIndexForBlockId requests in a row expired, whatever their other replies
repair_​ping_​probe_​txcounterRequests sent to elicit a ping from a peer with a new address
repair_​eager_​delay_​nanosgaugeCluster-wide wait for turbine before a missing FEC set is repaired (srtt+4*rttvar, clamped to 20-250 ms; 250 ms until sampled)
repair_​sign_​request_​pendinggaugeMessages in the sign pool waiting for a signature from the sign tiles
repair_​timeout_​no_​staked_​peer_​deferredcounterOf TimeoutNoPeerDeferred, retries of a block not known only from turbine, which pick only staked peers: every staked peer was banned, without an address or at its in-flight limit
repair_​timeout_​deferred
{timeout_​defer_​reason="turbine_​active"}
counterTimeouts put back without sending a request, by reason, one per deferral (HighestWindowIndex put back: turbine delivered a FEC set of the slot within the eager wait)
repair_​timeout_​deferred
{timeout_​defer_​reason="all_​received"}
counterTimeouts put back without sending a request, by reason, one per deferral (FEC set request put back: every data shred arrived, the set has not completed)
repair_​timeout_​queue_​full_​droppedcounterTimeouts dropped because their role's timeout queue was full; the block goes back to its treap and the work is discovered again
repair_​timeout_​queue_​full_​pausedcounterBlocks whose discovery stopped because their role's timeout queue filled; the block goes back to its treap and its remaining FEC sets are queued later
repair_​signing_​haltedcounterIdentity switches that halted signing; no request, ping or pong is signed until the switch completes and unhalts
repair_​timeout_​overdue_​nanosgaugeTime the earliest due timeout has waited past its due time when metrics were written, 0 if none is due; grows while signing is halted or fewer than 40 sign slots are free
repair_​first_​cert_​slotgaugeFirst certified slot in the window seen by a staked node, which starts its catchup, 0 if none yet

Rserve Tile ​

MetricTypeDescription
rserve_​request_​rx
{repair_​request_​type="pong"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Pong)
rserve_​request_​rx
{repair_​request_​type="window_​index"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Shred at an index)
rserve_​request_​rx
{repair_​request_​type="highest_​window_​index"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Highest shred at or above an index)
rserve_​request_​rx
{repair_​request_​type="orphan"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Ancestors of an orphan slot)
rserve_​request_​rx
{repair_​request_​type="parent_​fec_​set_​count"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Parent and FEC set count of a block (Alpenglow))
rserve_​request_​rx
{repair_​request_​type="fec_​set_​root"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Merkle root of a FEC set (Alpenglow))
rserve_​request_​rx
{repair_​request_​type="window_​index_​for_​block_​id"}
counterRequests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Shred at an index of a specific block (Alpenglow))
rserve_​request_​rx_​bytescounterBytes of incoming repair request payloads
rserve_​request_​malformed
{request_​malformed_​reason="too_​small"}
counterPackets dropped before parsing, by reason (Packet shorter than a message tag)
rserve_​request_​malformed
{request_​malformed_​reason="unsupported_​ping"}
counterPackets dropped before parsing, by reason (Ping sent to the repair server port)
rserve_​request_​malformed
{request_​malformed_​reason="unknown_​tag"}
counterPackets dropped before parsing, by reason (Unknown or unsupported message tag)
rserve_​request_​malformed
{request_​malformed_​reason="wrong_​size"}
counterPackets dropped before parsing, by reason (Message size does not match its tag)
rserve_​request_​malformed
{request_​malformed_​reason="unsupported_​ancestor_​hashes"}
counterPackets dropped before parsing, by reason (Ancestor hashes request, which is not served)
rserve_​request_​dropped
{request_​drop_​reason="signature_​failed"}
counterRequests dropped during validation, by reason (Signature failed to verify)
rserve_​request_​dropped
{request_​drop_​reason="from_​self"}
counterRequests dropped during validation, by reason (Sent by our own identity)
rserve_​request_​dropped
{request_​drop_​reason="wrong_​recipient"}
counterRequests dropped during validation, by reason (Addressed to another identity)
rserve_​request_​dropped
{request_​drop_​reason="timestamp_​skew"}
counterRequests dropped during validation, by reason (Timestamp outside the accepted window)
rserve_​request_​dropped
{request_​drop_​reason="pong_​token_​invalid"}
counterRequests dropped during validation, by reason (Pong token outdated or invalid)
rserve_​request_​dropped
{request_​drop_​reason="shred_​index_​invalid"}
counterRequests dropped during validation, by reason (Shred index past the end of a block)
rserve_​request_​dropped
{request_​drop_​reason="fec_​set_​index_​invalid"}
counterRequests dropped during validation, by reason (FEC set index unaligned or past the end of the block)
rserve_​request_​unanswered
{repair_​request_​type="pong"}
counterValid requests that could not be answered, by type (Pong)
rserve_​request_​unanswered
{repair_​request_​type="window_​index"}
counterValid requests that could not be answered, by type (Shred at an index)
rserve_​request_​unanswered
{repair_​request_​type="highest_​window_​index"}
counterValid requests that could not be answered, by type (Highest shred at or above an index)
rserve_​request_​unanswered
{repair_​request_​type="orphan"}
counterValid requests that could not be answered, by type (Ancestors of an orphan slot)
rserve_​request_​unanswered
{repair_​request_​type="parent_​fec_​set_​count"}
counterValid requests that could not be answered, by type (Parent and FEC set count of a block (Alpenglow))
rserve_​request_​unanswered
{repair_​request_​type="fec_​set_​root"}
counterValid requests that could not be answered, by type (Merkle root of a FEC set (Alpenglow))
rserve_​request_​unanswered
{repair_​request_​type="window_​index_​for_​block_​id"}
counterValid requests that could not be answered, by type (Shred at an index of a specific block (Alpenglow))
rserve_​pkt_​txcounterNetwork packets sent, including pings and responses
rserve_​response_​tx
{repair_​response_​type="ping"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Ping sent to a requester not yet in the ping cache)
rserve_​response_​tx
{repair_​response_​type="window_​index"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Shred at an index)
rserve_​response_​tx
{repair_​response_​type="highest_​window_​index"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Highest shred at or above an index)
rserve_​response_​tx
{repair_​response_​type="orphan"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Shred from an ancestor of an orphan slot)
rserve_​response_​tx
{repair_​response_​type="parent_​fec_​set_​count"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Parent and FEC set count of a block (Alpenglow))
rserve_​response_​tx
{repair_​response_​type="fec_​set_​root"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Merkle root of a FEC set (Alpenglow))
rserve_​response_​tx
{repair_​response_​type="window_​index_​for_​block_​id"}
counterResponse packets sent, by type; Orphan sends up to 11 per request (Shred at an index of a specific block (Alpenglow))
rserve_​response_​tx_​bytescounterPayload bytes sent in response packets
rserve_​disk_​read_​attempted
{disk_​read_​result="success"}
counterStore reads for repair responses, by result (Read served from the store)
rserve_​disk_​read_​attempted
{disk_​read_​result="miss"}
counterStore reads for repair responses, by result (Not in the store)
rserve_​disk_​read_​attempted
{disk_​read_​result="busy"}
counterStore reads for repair responses, by result (Index binding or ring cell changed during the read)
rserve_​disk_​read_​attempted
{disk_​read_​result="scan_​limit"}
counterStore reads for repair responses, by result (Cached highest-shred bound was no longer readable)
rserve_​disk_​read_​bytescounterShred bytes read from the store for repair responses
rserve_​shred_​usedgaugeShreds currently in the shreds database
rserve_​shred_​capacitygaugeShreds the shreds database can hold
rserve_​disk_​used_​bytesgaugeLogical bytes occupied by live wire shreds and spilled FEC payloads
rserve_​disk_​peak_​bytesgaugeLogical wire-ring high-water plus spill pages high-water
rserve_​disk_​write_​duration_​nanoshistogramDuration of persisting one accepted data shred
rserve_​disk_​shred_​writtencounterData shreds persisted to the repair store
rserve_​disk_​shred_​write_​failedcounterData shreds rejected by the repair store API
rserve_​disk_​shred_​written_​bytescounterBytes written by persisted data shreds
rserve_​store_​fec_​preevict_​duration_​nanoshistogramDuration of preevicting one FEC payload from the store RAM cache
rserve_​store_​fec_​preevictedcounterFEC payloads preevicted from the store RAM cache to disk
rserve_​store_​fec_​preevicted_​bytescounterFEC payload bytes preevicted from the store RAM cache to disk
rserve_​store_​fec_​cache_​freegaugeImmediately available FEC payload cache entries
rserve_​store_​fec_​cache_​capacitygaugeFEC payload cache capacity
rserve_​store_​fec_​cache_​free_​targetgaugeFree entries background preeviction tries to restore
rserve_​store_​fec_​cache_​free_​low_​watergaugeFree entry count below which background preeviction starts
rserve_​ping_​cache_​usedgaugeActive entries in the ping cache
rserve_​ping_​cache_​capacitygaugePing cache entry capacity
rserve_​ping_​cache_​evictedcounterPing cache entries evicted for a new peer (cache full); TTL expiries not counted

Replay Tile ​

MetricTypeDescription
replay_​identity_​balance_​lamportsgaugeIdentity account balance at the optimistically confirmed slot, or the finalized slot under Alpenglow
replay_​active_​stake_​lamportsgaugeOur active stake at the optimistically confirmed slot, or the finalized slot under Alpenglow
replay_​cluster_​active_​stake_​lamportsgaugeCluster active stake at the optimistically confirmed slot, or the finalized slot under Alpenglow
replay_​epoch_​creditsgaugeCredits our vote account earned this epoch, as of the optimistically confirmed slot, or the finalized slot under Alpenglow, where they are lamport-denominated reward credits
replay_​is_​vote_​account_​inadmissiblegaugeWhether our vote account's stake is not admitted: at the optimistically confirmed slot it fails the validator admission ticket filter (missing, below the V4 rent-exempt minimum, or not V4 with a BLS pubkey), or it passes but has not yet been admitted at an epoch boundary
replay_​last_​rewarded_​vote_​slotgaugeLatest slot for which our vote appears in a reward certificate, 0 if none observed
replay_​store_​query_​duration_​nanoshistogramDuration of ingesting one queried FEC set into the scheduler
replay_​store_​fec_​queriedcounterFEC sets queried from the store
replay_​store_​fec_​query_​missedcounterFEC set store queries that found no entry
replay_​store_​fec_​spilledcounterFEC payloads spilled from the store RAM cache to disk
replay_​store_​fec_​spilled_​bytescounterFEC payload bytes spilled from the store RAM cache to disk
replay_​store_​fec_​spill_​readcounterSpilled FEC payloads read from disk
replay_​store_​fec_​spill_​read_​bytescounterSpilled FEC payload bytes read from disk
replay_​store_​fec_​cache_​usedgaugeFEC payload slots in use in the store RAM cache
replay_​store_​fec_​cache_​capacitygaugeFEC payload capacity of the store RAM cache
replay_​store_​fec_​cache_​pinnedgaugeFEC payload views currently pinned against eviction
replay_​txncache_​page_​misscounterTransaction cache lookups under the read lock that found their page not resident
replay_​txncache_​page_​spilledcounterDirty transaction cache pages written to the spill file on eviction
replay_​txncache_​page_​spill_​readcounterTransaction cache pages read back from the spill file
replay_​txncache_​frame_​evictedcounterTransaction cache frames evicted to load another page
replay_​txncache_​spill_​duration_​nanoscounterTime in transaction cache spill file reads and writes
replay_​txncache_​lock_​read_​wait_​duration_​nanoscounterTime all tiles spent waiting to take the transaction cache shared lock
replay_​txncache_​lock_​write_​wait_​duration_​nanoscounterTime all tiles spent waiting to take the transaction cache exclusive lock
replay_​txncache_​page_​usedgaugeTransaction cache logical pages in use, resident or spilled
replay_​txncache_​page_​capacitygaugeMaximum transaction cache logical pages
replay_​txncache_​frame_​usedgaugeTransaction cache in-memory frames holding a page
replay_​txncache_​frame_​capacitygaugeTransaction cache in-memory frames, from runtime.transaction_cache_size_mib
replay_​stake_​delegation_​root_​usedgaugeRooted stake delegations held in memory
replay_​stake_​delegation_​root_​capacitygaugeRooted stake delegations that fit in memory, from runtime.stake_delegation_cache_cnt
replay_​stake_​delegation_​root_​disk_​countgaugeRooted stake delegations currently spilled to disk
replay_​stake_​delegation_​delta_​usedgaugeUnrooted stake delegation fork deltas held in memory
replay_​stake_​delegation_​delta_​capacitygaugeUnrooted stake delegation fork deltas that fit in memory
replay_​stake_​delegation_​delta_​disk_​countgaugeUnrooted stake delegation fork deltas currently spilled to disk
replay_​stake_​delegation_​root_​spilledcounterRooted stake delegations written to disk because the in-memory root pool was full
replay_​stake_​delegation_​delta_​spilledcounterStake delegation fork deltas written to disk because the in-memory delta pool was full
replay_​stake_​delegation_​disk_​readcounterReads from the stake delegation spill file by all tiles
replay_​stake_​delegation_​disk_​writtencounterWrites to the stake delegation spill file by all tiles
replay_​stake_​delegation_​disk_​duration_​nanoscounterTime all tiles spent in stake delegation spill file reads and writes
replay_​is_​caught_​upgaugeWhether replay has caught up to the cluster tip; set once per run, never cleared
replay_​root_​slotgaugeMost recent consensus root, 0 if none
replay_​root_​slot_​distancegaugeSlots between the current reset slot and our root
replay_​leader_​slotgaugeSlot at which we are currently leader, or 0 if none
replay_​next_​leader_​slotgaugeSlot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot
replay_​reset_​slotgaugeSlot replay last reset to (the newest replayed slot or parent-ready parent); can decrease
replay_​bank_​capacitygaugeMaximum number of banks we can have alive
replay_​bank_​usedgaugeBanks we currently have alive
replay_​last_​reassembly_​dequeued_​slotgaugeSlot of the FEC set most recently dequeued from reassembly for replay, 0 if none
replay_​last_​reassembly_​dequeued_​fec_​indexgaugeFEC set index of the FEC set most recently dequeued from reassembly for replay; read with LastReassemblyDequeuedSlot
replay_​sched_​active_​bank_​indexgaugeBank pool index the replay scheduler is dispatching work for, ULONG_MAX if none
replay_​sched_​last_​dispatch_​bank_​indexgaugeBank pool index the replay scheduler last dispatched work for, ULONG_MAX if none
replay_​sched_​last_​dispatch_​timegaugeWall-clock time the replay scheduler last dispatched work, 0 if never
replay_​sched_​staging_​lane_​usedgaugeStaging lanes currently occupied in the replay scheduler. Up to four staging lanes can be occupied at once
replay_​sched_​staging_​lane_​peakgaugeHigh watermark of staging lanes that were at one point occupied in the replay scheduler
replay_​sched_​staging_​lane_​head_​bank_​index
{staging_​lane="lane0"}
gaugeBank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 0)
replay_​sched_​staging_​lane_​head_​bank_​index
{staging_​lane="lane1"}
gaugeBank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 1)
replay_​sched_​staging_​lane_​head_​bank_​index
{staging_​lane="lane2"}
gaugeBank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 2)
replay_​sched_​staging_​lane_​head_​bank_​index
{staging_​lane="lane3"}
gaugeBank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 3)
replay_​sched_​txn_​pool_​usedgaugeTransaction pool entries held by the replay scheduler, from parse until release
replay_​sched_​txn_​pool_​capacitygaugeMaximum transactions the replay scheduler can hold
replay_​sched_​microblock_​pool_​usedgaugeMicroblocks available to be scheduled in the replay scheduler
replay_​sched_​microblock_​pool_​capacitygaugeMaximum microblocks the replay scheduler can hold
replay_​sched_​block_​pool_​usedgaugeBlocks alive in the replay scheduler
replay_​sched_​block_​pool_​capacitygaugeMaximum blocks the replay scheduler can track
replay_​sched_​block_​added
{sched_​block_​staging_​state="staged"}
counterBlocks added to the replay scheduler, by staging state (Added as staged)
replay_​sched_​block_​added
{sched_​block_​staging_​state="unstaged"}
counterBlocks added to the replay scheduler, by staging state (Added as unstaged)
replay_​sched_​block_​added
{sched_​block_​staging_​state="dead"}
counterBlocks added to the replay scheduler, by staging state (Added as the dead child of a dead block, never dispatched)
replay_​sched_​block_​replayedcounterBlocks the replay scheduler considers fully replayed
replay_​sched_​block_​abandonedcounterBlocks the replay scheduler abandoned for any reason
replay_​sched_​block_​rejectedcounterBlocks the replay scheduler found invalid while parsing or verifying PoH; also counted in SchedBlockAbandoned once drained; replay-tile verdicts not counted
replay_​sched_​block_​promotedcounterBlocks the replay scheduler promoted from unstaged to a staging lane
replay_​sched_​block_​demotedcounterBlocks the replay scheduler demoted from a staging lane to unstaged
replay_​sched_​txn_​promotedcounterTransactions the replay scheduler moved into a staging lane when it promoted their block
replay_​sched_​block_​deactivated
{sched_​deactivate_​reason="no_​child"}
counterTimes the replay scheduler stopped dispatching a block, by reason (A block fully replayed and the block activated next, if any, was not its child)
replay_​sched_​block_​deactivated
{sched_​deactivate_​reason="no_​work"}
counterTimes the replay scheduler stopped dispatching a block, by reason (A partially replayed block ran out of work to dispatch)
replay_​sched_​block_​deactivated
{sched_​deactivate_​reason="abandoned"}
counterTimes the replay scheduler stopped dispatching a block, by reason (A block was abandoned, other than by minority fork pruning)
replay_​sched_​block_​deactivated
{sched_​deactivate_​reason="minority_​fork"}
counterTimes the replay scheduler stopped dispatching a block, by reason (A block was pruned for being on a minority fork (not also counted as Abandoned))
replay_​sched_​block_​deactivated
{sched_​deactivate_​reason="spec_​yield"}
counterTimes the replay scheduler stopped dispatching a block, by reason (A block that could only speculate on out-of-band lthash additions gave way to another lane head's real work)
replay_​sched_​block_​deactivated
{sched_​deactivate_​reason="spec_​pool_​full"}
counterTimes the replay scheduler stopped dispatching a block, by reason (A partially replayed block owing out-of-band lthash additions had nothing to dispatch because the speculation pool had no free entry)
replay_​sched_​block_​inband_​pickedcounterBlocks that fell back to in-band lthash because every demotion reservation was in use
replay_​sched_​lane_​switchedcounterTimes the replay scheduler activated a block on a different fork than the previously active block
replay_​sched_​lane_​promotedcounterTimes the replay scheduler promoted a block onto a staging lane
replay_​sched_​lane_​demotedcounterTimes the replay scheduler demoted a block from a staging lane
replay_​sched_​fork_​observedcounterForks the replay scheduler has observed
replay_​sched_​txn_​lut_​resolve_​attempted
{sched_​lut_​resolve_​result="success"}
counterTransactions with address lookup tables the replay scheduler tried to resolve, by result (Address lookup tables expanded successfully)
replay_​sched_​txn_​lut_​resolve_​attempted
{sched_​lut_​resolve_​result="failed"}
counterTransactions with address lookup tables the replay scheduler tried to resolve, by result (Address lookup table expansion failed)
replay_​sched_​txn_​parsed_​abandonedcounterTransactions that were parsed but ended up being in abandoned blocks
replay_​sched_​txn_​executed_​abandonedcounterTransactions that were executed but ended up being in abandoned blocks
replay_​sched_​txn_​done_​abandonedcounterTransactions that were fully replayed but ended up being in abandoned blocks
replay_​sched_​txn_​in_​flight_​integral_​nanoscounterIntegral of in-flight transactions over time (transaction-nanoseconds); divide by SchedTxnBusyDurationNanos for the average in-flight count
replay_​sched_​txn_​busy_​duration_​nanoscounterTime the replay scheduler had at least one transaction in flight
replay_​sched_​txn_​idle_​duration_​nanoscounterTime the replay scheduler had no transaction in flight
replay_​sched_​txn_​parsedcounterTransactions the replay pipeline has parsed
replay_​sched_​txn_​executedcounterTransactions the replay pipeline has executed
replay_​sched_​txn_​signature_​verifiedcounterTransactions the replay pipeline has sigverified
replay_​sched_​txn_​poh_​mixedcounterTransactions the replay pipeline has performed PoH mixin on
replay_​sched_​txn_​donecounterTransactions that were fully replayed
replay_​sched_​microblock_​parsedcounterMicroblocks the replay pipeline has parsed
replay_​sched_​microblock_​hashedcounterMicroblocks the replay pipeline has fully hashed
replay_​sched_​microblock_​donecounterMicroblocks that were fully replayed
replay_​sched_​ingested_​bytescounterBytes the replay scheduler has ingested
replay_​sched_​trailing_​skipped_​bytescounterTrailing bytes of a block the replay scheduler ingested but did not parse
replay_​sched_​dropped_​bytescounterBytes the replay scheduler refused to ingest because the block was abandoned
replay_​sched_​fec_​ingestedcounterFEC sets the replay scheduler has been given
replay_​sched_​lthash_​write_​observed
{sched_​lthash_​write_​result="first_​write"}
counterWrites the replay scheduler saw for out-of-band lthash, by result (First write to the account since its last addition pseudo transaction, which makes the block owe it an addition)
replay_​sched_​lthash_​write_​observed
{sched_​lthash_​write_​result="coalesced"}
counterWrites the replay scheduler saw for out-of-band lthash, by result (Write to an account the block already owed an addition for)
replay_​sched_​lthash_​write_​observed_​by_​type
{sched_​lthash_​write_​type="parsed"}
counterWrites the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Declared writable accounts of parsed transactions, as handed to the dispatcher)
replay_​sched_​lthash_​write_​observed_​by_​type
{sched_​lthash_​write_​type="block_​start"}
counterWrites the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Accounts registered by start-of-block processing)
replay_​sched_​lthash_​write_​observed_​by_​type
{sched_​lthash_​write_​type="late_​lut"}
counterWrites the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Writable lookup table accounts registered at transaction completion)
replay_​sched_​lthash_​write_​observed_​by_​type
{sched_​lthash_​write_​type="demoted"}
counterWrites the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Accounts re-registered when their block was demoted)
replay_​sched_​lthash_​write_​observed_​by_​type
{sched_​lthash_​write_​type="spec_​given_​up"}
counterWrites the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Accounts re-registered when a speculative addition was given up)
replay_​sched_​lthash_​add_​createdcounterOut-of-band lthash addition pseudo transactions the dispatcher handed back
replay_​sched_​lthash_​add_​surfacedcounterAddition pseudo transactions the dispatcher handed out as ready
replay_​sched_​lthash_​add_​stale_​surfacedcounterAddition pseudo transactions surfaced after a later write made them stale, completed without hashing
replay_​sched_​lthash_​add_​dispatchedcounterAdditions handed to an exec tile
replay_​sched_​lthash_​add_​stale_​droppedcounterAddition results dropped because a later write made them stale while on the tile
replay_​sched_​lthash_​add_​appliedcounterAddition results added to a block's delta
replay_​sched_​lthash_​add_​undonecounterApplied additions subtracted back out after a later write
replay_​sched_​lthash_​spec_​createdcounterSpeculative additions created
replay_​sched_​lthash_​spec_​appliedcounterSpeculative additions applied; a subset of SchedLthashAddApplied
replay_​sched_​lthash_​spec_​invalidatedcounterSpeculative additions invalidated before they applied
replay_​sched_​lthash_​sub_​owedcounterLane map entries inserted, each owing one out-of-band lthash subtraction
replay_​sched_​lthash_​sub_​dispatchedcounterSubtractions handed to an exec tile
replay_​sched_​lthash_​sub_​appliedcounterSubtraction results subtracted from a block's delta
replay_​sched_​lthash_​account_​frozencounterAccounts whose out-of-band lthash subtraction and addition are in the delta of a block whose end task completed
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="add_​owed"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions owed by abandoned blocks, never created)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="add_​pending"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Pseudo transactions of abandoned blocks, never surfaced)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="add_​applied"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions in the delta of an abandoned or demoted block, net of undone)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="sub_​queued"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Subtractions of abandoned blocks, never dispatched)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="sub_​applied"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Subtractions in the delta of an abandoned or demoted block)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="add_​held"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions of abandoned blocks that were batched behind a transaction and never reached a tile)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="inband_​add_​owed"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions owed by blocks that picked in-band hashing, never created)
replay_​sched_​lthash_​work_​dropped
{sched_​lthash_​work_​type="inband_​sub_​queued"}
counterOut-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Subtractions of blocks that picked in-band hashing, never dispatched)
replay_​sched_​lthash_​task_​dispatched
{sched_​lthash_​task_​type="sub"}
counterOut-of-band lthash tasks handed to exec tiles, each a batch of the accounts SchedLthashSubDispatched and SchedLthashAddDispatched count, by type (Subtraction tasks, each a batch of accounts hashed on the parent fork)
replay_​sched_​lthash_​task_​dispatched
{sched_​lthash_​task_​type="add"}
counterOut-of-band lthash tasks handed to exec tiles, each a batch of the accounts SchedLthashSubDispatched and SchedLthashAddDispatched count, by type (Addition tasks, each a batch of accounts hashed on the block's fork)
replay_​sched_​lthash_​batch_​heldcounterOut-of-band lthash addition batches held because a transaction or standalone speculation surfaced behind them
replay_​sched_​lthash_​batch_​idle_​waitedcounterDispatch polls that held an out-of-band lthash addition batch because too few exec tiles were idle
replay_​bank_​frozencounterBanks frozen, including our own leader blocks
replay_​chain_​txn_​heightgaugeTransactions from genesis through the last replayed block, 0 if none; drops on a fork switch
replay_​fec_​poll_​skipped
{fec_​poll_​skip_​reason="sched_​full"}
counterReplay loop iterations that processed no FEC set, by reason (The scheduler was full)
replay_​fec_​poll_​skipped
{fec_​poll_​skip_​reason="leader_​freeze_​wait"}
counterReplay loop iterations that processed no FEC set, by reason (Waiting for PoH to end our leader block)
replay_​fec_​poll_​skipped
{fec_​poll_​skip_​reason="bank_​full"}
counterReplay loop iterations that processed no FEC set, by reason (The bank pool was full)
replay_​fec_​poll_​skipped
{fec_​poll_​skip_​reason="lane_​wait"}
counterReplay loop iterations that processed no FEC set, by reason (A block's first FEC set waited for a staging lane)
replay_​fec_​parent_​missing_​droppedcounterFEC sets dropped because replay could not resolve the block's parent (Alpenglow), which was dropped on a dead lineage, evicted with its tracking slot reused, or rooted past
replay_​fec_​rooted_​past_​skippedcounterRepair FEC sets of a slot at or below the published root, or of a block whose parent is below it, skipped rather than dropped: no redelivery can replay them
replay_​root_​advance_​deferredcounterTimes the storage root could not advance to the consensus root yet
replay_​leader_​window_​already_​producedcounterLeader windows given up because the cluster already produced one of their slots, eg. our own past slots replayed after a restart (Alpenglow)
replay_​leader_​window_​supersededcounterLeader windows given up after votor sent them because certificates already let a later window start (Alpenglow)
replay_​progcache_​rootedcounterProgram cache entries rooted
replay_​progcache_​class_​used
{progcache_​class="le128_​kib"}
gaugeProgram cache value slots in use, by size class (0-128 KiB)
replay_​progcache_​class_​used
{progcache_​class="le512_​kib"}
gaugeProgram cache value slots in use, by size class (128-512 KiB)
replay_​progcache_​class_​used
{progcache_​class="le1_​mib"}
gaugeProgram cache value slots in use, by size class (512 KiB-1 MiB)
replay_​progcache_​class_​used
{progcache_​class="le2_​mib"}
gaugeProgram cache value slots in use, by size class (1-2 MiB)
replay_​progcache_​class_​used
{progcache_​class="le4_​mib"}
gaugeProgram cache value slots in use, by size class (2-4 MiB)
replay_​progcache_​class_​used
{progcache_​class="le11_​mib"}
gaugeProgram cache value slots in use, by size class (4-11 MiB)
replay_​progcache_​class_​capacity
{progcache_​class="le128_​kib"}
gaugeProgram cache value capacity, by size class (0-128 KiB)
replay_​progcache_​class_​capacity
{progcache_​class="le512_​kib"}
gaugeProgram cache value capacity, by size class (128-512 KiB)
replay_​progcache_​class_​capacity
{progcache_​class="le1_​mib"}
gaugeProgram cache value capacity, by size class (512 KiB-1 MiB)
replay_​progcache_​class_​capacity
{progcache_​class="le2_​mib"}
gaugeProgram cache value capacity, by size class (1-2 MiB)
replay_​progcache_​class_​capacity
{progcache_​class="le4_​mib"}
gaugeProgram cache value capacity, by size class (2-4 MiB)
replay_​progcache_​class_​capacity
{progcache_​class="le11_​mib"}
gaugeProgram cache value capacity, by size class (4-11 MiB)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le512_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le2_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le8_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le32_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le1_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
replay_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le10_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le128_​b"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le512_​b"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le2_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le8_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le32_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le128_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le1_​mib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
replay_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le10_​mib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le128_​b"}
counterAccounts evicted from the account database cache to make space, by cache size class (0-128 B)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le512_​b"}
counterAccounts evicted from the account database cache to make space, by cache size class (129-512 B)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le2_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (513 B-2 KiB)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le8_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (2-8 KiB)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le32_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (8-32 KiB)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le128_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (32-128 KiB)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le1_​mib"}
counterAccounts evicted from the account database cache to make space, by cache size class (128 KiB-1 MiB)
replay_​accdb_​account_​evicted
{accdb_​cache_​class="le10_​mib"}
counterAccounts evicted from the account database cache to make space, by cache size class (1-10 MiB)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le128_​b"}
counterNew account versions committed to the index, by destination cache size class (0-128 B)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le512_​b"}
counterNew account versions committed to the index, by destination cache size class (129-512 B)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le2_​kib"}
counterNew account versions committed to the index, by destination cache size class (513 B-2 KiB)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le8_​kib"}
counterNew account versions committed to the index, by destination cache size class (2-8 KiB)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le32_​kib"}
counterNew account versions committed to the index, by destination cache size class (8-32 KiB)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le128_​kib"}
counterNew account versions committed to the index, by destination cache size class (32-128 KiB)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le1_​mib"}
counterNew account versions committed to the index, by destination cache size class (128 KiB-1 MiB)
replay_​accdb_​account_​new_​committed
{accdb_​cache_​class="le10_​mib"}
counterNew account versions committed to the index, by destination cache size class (1-10 MiB)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le128_​b"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (0-128 B)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le512_​b"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (129-512 B)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le2_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (513 B-2 KiB)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le8_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (2-8 KiB)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le32_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (8-32 KiB)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le128_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (32-128 KiB)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le1_​mib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (128 KiB-1 MiB)
replay_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le10_​mib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (1-10 MiB)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (0-128 B)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le512_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (129-512 B)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le2_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le8_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (2-8 KiB)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le32_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (8-32 KiB)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (32-128 KiB)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le1_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB)
replay_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le10_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (1-10 MiB)
replay_​accdb_​account_​waitedcounterAccounts that had to wait for another reader to finish loading them into the cache
replay_​accdb_​batch_​acquiredcounterBatch acquire calls made by this tile
replay_​accdb_​cache_​reserve_​retriedcounterSpin retries while waiting to reserve cache lines for a transaction
replay_​accdb_​disk_​read_​bytescounterBytes read from the account database file
replay_​accdb_​disk_​readcounterReads from the account database file
replay_​accdb_​disk_​written_​bytescounterBytes written to the account database file
replay_​accdb_​disk_​writtencounterWrites to the account database file
replay_​accdb_​copied_​bytescounterAccount data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
replay_​is_​runtime_​loadedgaugeWhether the runtime has loaded a bank and can replay

Execrp Tile ​

MetricTypeDescription
execrp_​is_​processinggaugeWhether the tile is processing a replay task
execrp_​txn_​processed
{txn_​result="success"}
counterTransactions loaded and executed, by result (Transaction executed successfully)
execrp_​txn_​processed
{txn_​result="instruction_​error"}
counterTransactions loaded and executed, by result (An error occurred while processing an instruction)
execrp_​txn_​processed
{txn_​result="account_​not_​found"}
counterTransactions loaded and executed, by result (The transaction fee payer address was not found)
execrp_​txn_​processed
{txn_​result="program_​account_​not_​found"}
counterTransactions loaded and executed, by result (A program account referenced by the transaction was not found)
execrp_​txn_​processed
{txn_​result="insufficient_​funds_​for_​fee"}
counterTransactions loaded and executed, by result (The transaction fee payer did not have balance to pay the fee)
execrp_​txn_​processed
{txn_​result="invalid_​account_​for_​fee"}
counterTransactions loaded and executed, by result (The transaction fee payer account is not owned by the system program, or has data that is not a nonce)
execrp_​txn_​processed
{txn_​result="already_​processed"}
counterTransactions loaded and executed, by result (The transaction has already been processed in a recent block)
execrp_​txn_​processed
{txn_​result="blockhash_​not_​found"}
counterTransactions loaded and executed, by result (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value)
execrp_​txn_​processed
{txn_​result="invalid_​program_​for_​execution"}
counterTransactions loaded and executed, by result (A program account referenced by the transaction was no executable.)
execrp_​txn_​processed
{txn_​result="address_​lookup_​table_​not_​found"}
counterTransactions loaded and executed, by result (The transaction references an ALUT account that does not exist or is inactive)
execrp_​txn_​processed
{txn_​result="invalid_​address_​lookup_​table_​owner"}
counterTransactions loaded and executed, by result (The transaction references an ALUT account that is not owned by the ALUT program account)
execrp_​txn_​processed
{txn_​result="invalid_​address_​lookup_​table_​data"}
counterTransactions loaded and executed, by result (The transaction references an ALUT account that contains data which is not a valid ALUT)
execrp_​txn_​processed
{txn_​result="invalid_​address_​lookup_​table_​index"}
counterTransactions loaded and executed, by result (The transaction references an account offset from the ALUT which does not exist)
execrp_​txn_​processed
{txn_​result="max_​loaded_​accounts_​data_​size_​exceeded"}
counterTransactions loaded and executed, by result (The total account data size of the loaded accounts exceeds the consensus limit)
execrp_​txn_​processed
{txn_​result="duplicate_​instruction"}
counterTransactions loaded and executed, by result (A compute budget program instruction was invoked more than once)
execrp_​txn_​processed
{txn_​result="invalid_​loaded_​accounts_​data_​size_​limit"}
counterTransactions loaded and executed, by result (The compute budget program was invoked and set the loaded accounts data size to zero)
execrp_​txn_​processed
{txn_​result="nonce_​already_​advanced"}
counterTransactions loaded and executed, by result (The transaction references a nonce account that is already advanced)
execrp_​txn_​processed
{txn_​result="nonce_​advance_​failed"}
counterTransactions loaded and executed, by result (The transaction is a nonce transaction but the advance instruction was not valid or failed)
execrp_​txn_​processed
{txn_​result="nonce_​wrong_​blockhash"}
counterTransactions loaded and executed, by result (The transaction is a nonce transaction but the blockhash is not the correct one)
execrp_​txn_​processed
{txn_​result="unsupported_​version"}
counterTransactions loaded and executed, by result (The transaction is a message version that is not supported by the current feature set)
execrp_​txn_​processed
{txn_​result="sanitize_​failure"}
counterTransactions loaded and executed, by result (The process for sanitizing the transaction failed)
execrp_​txn_​processed
{txn_​result="account_​in_​use"}
counterTransactions loaded and executed, by result (The transaction conflicts with another transaction in the microblock.)
execrp_​txn_​processed
{txn_​result="account_​loaded_​twice"}
counterTransactions loaded and executed, by result (The transaction references the same account twice)
execrp_​txn_​processed
{txn_​result="signature_​failure"}
counterTransactions loaded and executed, by result (The transaction had an invalid signature)
execrp_​txn_​processed
{txn_​result="too_​many_​account_​locks"}
counterTransactions loaded and executed, by result (The transaction references too many accounts.)
execrp_​txn_​processed
{txn_​result="insufficient_​funds_​for_​rent"}
counterTransactions loaded and executed, by result (The transaction would leave an account with a lower balance than the rent-exempt minimum)
execrp_​txn_​processed
{txn_​result="unbalanced_​transaction"}
counterTransactions loaded and executed, by result (The total referenced account lamports before and after the transaction was unbalanced)
execrp_​txn_​processed
{txn_​result="bundle_​peer"}
counterTransactions loaded and executed, by result (The transaction was part of a bundle and an earlier transaction in the bundle failed)
execrp_​txn_​processed_​by_​version
{txn_​version="legacy"}
counterTransactions processed, by transaction version (Legacy transaction format)
execrp_​txn_​processed_​by_​version
{txn_​version="v0"}
counterTransactions processed, by transaction version (Version 0 transaction format)
execrp_​txn_​processed_​by_​version
{txn_​version="v1"}
counterTransactions processed, by transaction version (Version 1 transaction format)
execrp_​signature_​checkedcounterEd25519 signatures checked, including failures
execrp_​poh_​hash_​computedcounterPoH SHA-256 iterations computed
execrp_​instruction_​executedcounterTop-level instructions executed
execrp_​cpi_​executedcounterCross-program invocations executed
execrp_​account_​lthash_​skippedcounterCommitted writable accounts left byte-identical by their transaction, so their in-band lthash update was skipped
execrp_​bank_​switchedcounterCache invalidations due to transaction execution from a different block
execrp_​txn_​regime_​duration_​nanos
{txn_​regime="load"}
counterTime per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction checks and account loading)
execrp_​txn_​regime_​duration_​nanos
{txn_​regime="execute"}
counterTime per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction execution, including VM setup)
execrp_​txn_​regime_​duration_​nanos
{txn_​regime="commit"}
counterTime per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction result commit)
execrp_​vm_​regime_​duration_​nanos
{vm_​regime="setup"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup)
execrp_​vm_​regime_​duration_​nanos
{vm_​regime="commit"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit)
execrp_​vm_​regime_​duration_​nanos
{vm_​regime="setup_​cpi"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup (CPI))
execrp_​vm_​regime_​duration_​nanos
{vm_​regime="commit_​cpi"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit (CPI))
execrp_​vm_​regime_​duration_​nanos
{vm_​regime="interpreter"}
counterTime in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM interpreter execution)
execrp_​cu_​consumedcounterCompute units consumed by executed transactions
execrp_​progcache_​looked_​upcounterProgram cache lookups; equals Hit + Miss + lookups of invalid program accounts
execrp_​progcache_​misscounterProgram cache misses, each followed by an insert or a spill
execrp_​progcache_​waitedcounterProgram cache lookups that waited on another tile's in-flight load of the same program
execrp_​progcache_​insert_​retriedcounterProgram cache inserts that found no free slot in their size class on the first try and retried after eviction
execrp_​progcache_​inserted_​bytescounterBytes inserted into the program cache
execrp_​progcache_​spilled_​bytescounterBytes loaded into program cache spill memory
execrp_​progcache_​evicted_​bytescounterBytes evicted from the program cache by insert eviction sweeps
execrp_​progcache_​duration_​nanoscounterTime in program cache lookups, including waits on peer loads and insert, eviction and ELF load on a miss; superset of ProgcacheLoadDurationNanos
execrp_​progcache_​loadedcounterProgram ELF loads and verifications for the program cache, including failed verifications and spill-backed loads
execrp_​progcache_​load_​duration_​nanoscounterTime spent loading programs
execrp_​progcache_​hit
{progcache_​class="le128_​kib"}
counterProgram cache hits, by size class (0-128 KiB)
execrp_​progcache_​hit
{progcache_​class="le512_​kib"}
counterProgram cache hits, by size class (128-512 KiB)
execrp_​progcache_​hit
{progcache_​class="le1_​mib"}
counterProgram cache hits, by size class (512 KiB-1 MiB)
execrp_​progcache_​hit
{progcache_​class="le2_​mib"}
counterProgram cache hits, by size class (1-2 MiB)
execrp_​progcache_​hit
{progcache_​class="le4_​mib"}
counterProgram cache hits, by size class (2-4 MiB)
execrp_​progcache_​hit
{progcache_​class="le11_​mib"}
counterProgram cache hits, by size class (4-11 MiB)
execrp_​progcache_​inserted
{progcache_​class="le128_​kib"}
counterPrograms inserted into the program cache, by size class (0-128 KiB)
execrp_​progcache_​inserted
{progcache_​class="le512_​kib"}
counterPrograms inserted into the program cache, by size class (128-512 KiB)
execrp_​progcache_​inserted
{progcache_​class="le1_​mib"}
counterPrograms inserted into the program cache, by size class (512 KiB-1 MiB)
execrp_​progcache_​inserted
{progcache_​class="le2_​mib"}
counterPrograms inserted into the program cache, by size class (1-2 MiB)
execrp_​progcache_​inserted
{progcache_​class="le4_​mib"}
counterPrograms inserted into the program cache, by size class (2-4 MiB)
execrp_​progcache_​inserted
{progcache_​class="le11_​mib"}
counterPrograms inserted into the program cache, by size class (4-11 MiB)
execrp_​progcache_​evicted
{progcache_​class="le128_​kib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (0-128 KiB)
execrp_​progcache_​evicted
{progcache_​class="le512_​kib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (128-512 KiB)
execrp_​progcache_​evicted
{progcache_​class="le1_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (512 KiB-1 MiB)
execrp_​progcache_​evicted
{progcache_​class="le2_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (1-2 MiB)
execrp_​progcache_​evicted
{progcache_​class="le4_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (2-4 MiB)
execrp_​progcache_​evicted
{progcache_​class="le11_​mib"}
counterProgram cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (4-11 MiB)
execrp_​progcache_​spilled
{progcache_​class="le128_​kib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (0-128 KiB)
execrp_​progcache_​spilled
{progcache_​class="le512_​kib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (128-512 KiB)
execrp_​progcache_​spilled
{progcache_​class="le1_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (512 KiB-1 MiB)
execrp_​progcache_​spilled
{progcache_​class="le2_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (1-2 MiB)
execrp_​progcache_​spilled
{progcache_​class="le4_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (2-4 MiB)
execrp_​progcache_​spilled
{progcache_​class="le11_​mib"}
counterPrograms loaded into spill memory because no cache slot could be freed, by size class (4-11 MiB)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le512_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le2_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le8_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le32_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le1_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
execrp_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le10_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le128_​b"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le512_​b"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le2_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le8_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le32_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le128_​kib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le1_​mib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
execrp_​accdb_​account_​writable_​acquired
{accdb_​cache_​class="le10_​mib"}
counterWritable accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le128_​b"}
counterAccounts evicted from the account database cache to make space, by cache size class (0-128 B)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le512_​b"}
counterAccounts evicted from the account database cache to make space, by cache size class (129-512 B)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le2_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (513 B-2 KiB)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le8_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (2-8 KiB)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le32_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (8-32 KiB)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le128_​kib"}
counterAccounts evicted from the account database cache to make space, by cache size class (32-128 KiB)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le1_​mib"}
counterAccounts evicted from the account database cache to make space, by cache size class (128 KiB-1 MiB)
execrp_​accdb_​account_​evicted
{accdb_​cache_​class="le10_​mib"}
counterAccounts evicted from the account database cache to make space, by cache size class (1-10 MiB)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le128_​b"}
counterNew account versions committed to the index, by destination cache size class (0-128 B)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le512_​b"}
counterNew account versions committed to the index, by destination cache size class (129-512 B)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le2_​kib"}
counterNew account versions committed to the index, by destination cache size class (513 B-2 KiB)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le8_​kib"}
counterNew account versions committed to the index, by destination cache size class (2-8 KiB)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le32_​kib"}
counterNew account versions committed to the index, by destination cache size class (8-32 KiB)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le128_​kib"}
counterNew account versions committed to the index, by destination cache size class (32-128 KiB)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le1_​mib"}
counterNew account versions committed to the index, by destination cache size class (128 KiB-1 MiB)
execrp_​accdb_​account_​new_​committed
{accdb_​cache_​class="le10_​mib"}
counterNew account versions committed to the index, by destination cache size class (1-10 MiB)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le128_​b"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (0-128 B)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le512_​b"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (129-512 B)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le2_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (513 B-2 KiB)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le8_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (2-8 KiB)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le32_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (8-32 KiB)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le128_​kib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (32-128 KiB)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le1_​mib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (128 KiB-1 MiB)
execrp_​accdb_​account_​overwrite_​committed
{accdb_​cache_​class="le10_​mib"}
counterAccount versions committed over an existing version of the same fork and generation, by destination cache size class (1-10 MiB)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (0-128 B)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le512_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (129-512 B)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le2_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le8_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (2-8 KiB)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le32_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (8-32 KiB)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (32-128 KiB)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le1_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB)
execrp_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le10_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (1-10 MiB)
execrp_​accdb_​account_​waitedcounterAccounts that had to wait for another reader to finish loading them into the cache
execrp_​accdb_​batch_​acquiredcounterBatch acquire calls made by this tile
execrp_​accdb_​cache_​reserve_​retriedcounterSpin retries while waiting to reserve cache lines for a transaction
execrp_​accdb_​disk_​read_​bytescounterBytes read from the account database file
execrp_​accdb_​disk_​readcounterReads from the account database file
execrp_​accdb_​disk_​written_​bytescounterBytes written to the account database file
execrp_​accdb_​disk_​writtencounterWrites to the account database file
execrp_​accdb_​copied_​bytescounterAccount data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
execrp_​txncache_​query_​misscounterTransaction cache queries by this tile that found a page not in memory
execrp_​txncache_​insert_​misscounterTransaction cache inserts by this tile that found a page not in memory
execrp_​txncache_​operation_​stalledcounterTransaction cache queries and inserts by this tile that fell back to the exclusive lock, stalling every tile; pages they load are read while holding it
execrp_​txncache_​query_​duration_​nanoshistogramDuration of one transaction cache query
execrp_​txncache_​insert_​duration_​nanoshistogramDuration of one transaction cache insert

Accdb Tile ​

MetricTypeDescription
accdb_​account_​usedgaugeAccount versions in the index; every unrooted fork version counts
accdb_​account_​capacitygaugeAccount versions the index can hold
accdb_​account_​relocatedcounterAccounts moved in the account database during compaction
accdb_​disk_​allocated_​bytesgaugeAllocated size of the account database file
accdb_​disk_​consumed_​bytesgaugeBytes of the account database file written at least once, excluding never-written partition tail space
accdb_​disk_​used_​bytesgaugeBytes of the account database file holding live data
accdb_​is_​compactinggaugeWhether a compaction is currently in progress
accdb_​compaction_​requestedcounterCompactions requested on the account database
accdb_​compaction_​completedcounterCompactions fully completed on the account database, less than or equal to the requested amount
accdb_​account_​relocated_​bytescounterBytes moved in the account database during compaction
accdb_​account_​preevicted
{accdb_​cache_​class="le128_​b"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (0-128 B)
accdb_​account_​preevicted
{accdb_​cache_​class="le512_​b"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (129-512 B)
accdb_​account_​preevicted
{accdb_​cache_​class="le2_​kib"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (513 B-2 KiB)
accdb_​account_​preevicted
{accdb_​cache_​class="le8_​kib"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (2-8 KiB)
accdb_​account_​preevicted
{accdb_​cache_​class="le32_​kib"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (8-32 KiB)
accdb_​account_​preevicted
{accdb_​cache_​class="le128_​kib"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (32-128 KiB)
accdb_​account_​preevicted
{accdb_​cache_​class="le1_​mib"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (128 KiB-1 MiB)
accdb_​account_​preevicted
{accdb_​cache_​class="le10_​mib"}
counterDirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (1-10 MiB)
accdb_​disk_​read_​bytescounterBytes read from the account database file by compaction and writeback
accdb_​disk_​written_​bytescounterBytes written to the account database file by compaction and writeback
accdb_​writeback_​writtencounterWrite syscalls that wrote evicted cache lines back to the account database file
accdb_​compaction_​writtencounterWrites that relocated a batch of records during compaction
accdb_​account_​version_​removedcounterAccount versions removed from the account database index
accdb_​cache_​class_​used
{accdb_​cache_​class="le128_​b"}
gaugeCache lines in use, by size class (0-128 B)
accdb_​cache_​class_​used
{accdb_​cache_​class="le512_​b"}
gaugeCache lines in use, by size class (129-512 B)
accdb_​cache_​class_​used
{accdb_​cache_​class="le2_​kib"}
gaugeCache lines in use, by size class (513 B-2 KiB)
accdb_​cache_​class_​used
{accdb_​cache_​class="le8_​kib"}
gaugeCache lines in use, by size class (2-8 KiB)
accdb_​cache_​class_​used
{accdb_​cache_​class="le32_​kib"}
gaugeCache lines in use, by size class (8-32 KiB)
accdb_​cache_​class_​used
{accdb_​cache_​class="le128_​kib"}
gaugeCache lines in use, by size class (32-128 KiB)
accdb_​cache_​class_​used
{accdb_​cache_​class="le1_​mib"}
gaugeCache lines in use, by size class (128 KiB-1 MiB)
accdb_​cache_​class_​used
{accdb_​cache_​class="le10_​mib"}
gaugeCache lines in use, by size class (1-10 MiB)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le128_​b"}
gaugeCache line capacity, by size class (0-128 B)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le512_​b"}
gaugeCache line capacity, by size class (129-512 B)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le2_​kib"}
gaugeCache line capacity, by size class (513 B-2 KiB)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le8_​kib"}
gaugeCache line capacity, by size class (2-8 KiB)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le32_​kib"}
gaugeCache line capacity, by size class (8-32 KiB)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le128_​kib"}
gaugeCache line capacity, by size class (32-128 KiB)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le1_​mib"}
gaugeCache line capacity, by size class (128 KiB-1 MiB)
accdb_​cache_​class_​capacity
{accdb_​cache_​class="le10_​mib"}
gaugeCache line capacity, by size class (1-10 MiB)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le128_​b"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (0-128 B)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le512_​b"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (129-512 B)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le2_​kib"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (513 B-2 KiB)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le8_​kib"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (2-8 KiB)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le32_​kib"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (8-32 KiB)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le128_​kib"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (32-128 KiB)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le1_​mib"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (128 KiB-1 MiB)
accdb_​cache_​class_​reserved
{accdb_​cache_​class="le10_​mib"}
gaugeCache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (1-10 MiB)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le128_​b"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (0-128 B)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le512_​b"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (129-512 B)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le2_​kib"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (513 B-2 KiB)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le8_​kib"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (2-8 KiB)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le32_​kib"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (8-32 KiB)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le128_​kib"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (32-128 KiB)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le1_​mib"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (128 KiB-1 MiB)
accdb_​is_​cache_​class_​reservation_​tracked
{accdb_​cache_​class="le10_​mib"}
gaugeWhether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (1-10 MiB)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le128_​b"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (0-128 B)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le512_​b"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (129-512 B)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le2_​kib"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (513 B-2 KiB)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le8_​kib"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (2-8 KiB)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le32_​kib"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (8-32 KiB)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le128_​kib"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (32-128 KiB)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le1_​mib"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (128 KiB-1 MiB)
accdb_​cache_​class_​free_​target
{accdb_​cache_​class="le10_​mib"}
gaugeFree cache lines the background preevict pass tries to restore, by size class (1-10 MiB)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le128_​b"}
gaugeFree cache line count below which the background preevict pass starts, by size class (0-128 B)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le512_​b"}
gaugeFree cache line count below which the background preevict pass starts, by size class (129-512 B)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le2_​kib"}
gaugeFree cache line count below which the background preevict pass starts, by size class (513 B-2 KiB)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le8_​kib"}
gaugeFree cache line count below which the background preevict pass starts, by size class (2-8 KiB)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le32_​kib"}
gaugeFree cache line count below which the background preevict pass starts, by size class (8-32 KiB)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le128_​kib"}
gaugeFree cache line count below which the background preevict pass starts, by size class (32-128 KiB)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le1_​mib"}
gaugeFree cache line count below which the background preevict pass starts, by size class (128 KiB-1 MiB)
accdb_​cache_​class_​free_​low_​water
{accdb_​cache_​class="le10_​mib"}
gaugeFree cache line count below which the background preevict pass starts, by size class (1-10 MiB)
accdb_​progcache_​preevicted_​bytescounterBytes reclaimed from the program cache by the background preevict pass
accdb_​progcache_​preevicted
{progcache_​class="le128_​kib"}
counterProgram cache slots reclaimed by the background preevict pass, by size class (0-128 KiB)
accdb_​progcache_​preevicted
{progcache_​class="le512_​kib"}
counterProgram cache slots reclaimed by the background preevict pass, by size class (128-512 KiB)
accdb_​progcache_​preevicted
{progcache_​class="le1_​mib"}
counterProgram cache slots reclaimed by the background preevict pass, by size class (512 KiB-1 MiB)
accdb_​progcache_​preevicted
{progcache_​class="le2_​mib"}
counterProgram cache slots reclaimed by the background preevict pass, by size class (1-2 MiB)
accdb_​progcache_​preevicted
{progcache_​class="le4_​mib"}
counterProgram cache slots reclaimed by the background preevict pass, by size class (2-4 MiB)
accdb_​progcache_​preevicted
{progcache_​class="le11_​mib"}
counterProgram cache slots reclaimed by the background preevict pass, by size class (4-11 MiB)

Diag Tile ​

MetricTypeDescription
diag_​bundle_​stategaugeBundle subsystem state: 0=disconnected (All bundle tiles disconnected), 1=connecting (At least one bundle tile connecting, none connected or sleeping), 2=connected (At least one bundle tile connected), 3=sleeping (At least one bundle tile sleeping, none connected), 4=disabled (No bundle tiles configured)
diag_​vote_​stategaugeVote subsystem state: 0=disabled (Non-voting or no votor tile), 1=not_started (Votor tile not running or no votes cast yet), 2=delinquent (Vote distance exceeds threshold or vote stalled), 3=voting (Voting normally), 4=inadmissible (Voting, but the vote account fails the validator admission ticket filter)
diag_​replay_​stategaugeReplay subsystem state: 0=disabled (No replay tile), 1=not_started (Replay tile not running or slots are zero), 2=behind (Replay lagging behind turbine or reset slot stalled), 3=running (Replay keeping up)
diag_​turbine_​stategaugeTurbine subsystem state: 0=disabled (No shred or replay tiles), 1=not_started (Tiles not all running or turbine slot is zero), 2=stalled (Turbine slot not advancing), 3=repair_outpacing (Repair byte throughput exceeds turbine), 4=running (Turbine receiving normally)
diag_​builder_​stategaugeExternal block builder subsystem state: 0=disabled (No block builder configured), 1=disconnected (Block builder disconnected), 2=connecting (Block builder connection in progress), 3=unhealthy (Connected, but the block builder is not in a usable state), 4=connected (Block builder connected and healthy)
diag_​system_​device_​irqcounterDevice IRQs across all CPUs
diag_​device_​irq_​on_​tile_​cpucounterDevice IRQs on CPUs dedicated to pinned tiles; subset of SystemDeviceIrq
diag_​system_​softirq
{softirq_​type="net"}
counterSoft-IRQs across all CPUs, by softirq type (NET_TX, NET_RX)
diag_​system_​softirq
{softirq_​type="disk"}
counterSoft-IRQs across all CPUs, by softirq type (BLOCK)
diag_​system_​softirq
{softirq_​type="other"}
counterSoft-IRQs across all CPUs, by softirq type (e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...)
diag_​softirq_​on_​tile_​cpu
{softirq_​type="net"}
counterSoft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq (NET_TX, NET_RX)
diag_​softirq_​on_​tile_​cpu
{softirq_​type="disk"}
counterSoft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq (BLOCK)
diag_​softirq_​on_​tile_​cpu
{softirq_​type="other"}
counterSoft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq (e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...)
diag_​cpu_​package_​millicelsiusgaugeHottest CPU package temperature (k10temp Tctl or coretemp Package), 0 if no sensor
diag_​nic_​millicelsiusgaugeHottest NIC ASIC temperature (mlx5 or ice), 0 if no sensor
diag_​nvme_​millicelsiusgaugeHottest NVMe composite temperature, 0 if no sensor or the kernel lacks CONFIG_NVME_HWMON
diag_​cpu_​power_​microwattsgaugeCPU package power from the powercap (RAPL) energy counters, averaged over 5 s, 0 if unavailable
diag_​system_​power_​microwattsgaugeWhole-system power draw from the ACPI power meter, averaged over its interval, 0 if no meter
diag_​memory_​ecc_​correctedcounterCorrected memory ECC errors across all memory controllers since boot (EDAC), 0 if firmware handles errors instead
diag_​memory_​ecc_​uncorrectedcounterUncorrected memory ECC errors across all memory controllers since boot (EDAC), 0 if firmware handles errors instead
diag_​system_​load_​centi
{load_​average_​type="minute1"}
gaugeLoad average over each window, times 100 (/proc/loadavg has two decimals) (1 minute)
diag_​system_​load_​centi
{load_​average_​type="minute5"}
gaugeLoad average over each window, times 100 (/proc/loadavg has two decimals) (5 minutes)
diag_​system_​load_​centi
{load_​average_​type="minute15"}
gaugeLoad average over each window, times 100 (/proc/loadavg has two decimals) (15 minutes)
diag_​thread_​runnable_​countgaugeRunnable threads on the host (/proc/loadavg)
diag_​system_​cpu_​duration_​nanos
{system_​cpu_​state="user"}
counterCPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (User (including nice))
diag_​system_​cpu_​duration_​nanos
{system_​cpu_​state="system"}
counterCPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (System)
diag_​system_​cpu_​duration_​nanos
{system_​cpu_​state="idle"}
counterCPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Idle)
diag_​system_​cpu_​duration_​nanos
{system_​cpu_​state="io_​wait"}
counterCPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Idle waiting on IO)
diag_​system_​cpu_​duration_​nanos
{system_​cpu_​state="interrupt"}
counterCPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Hardirq and softirq handlers)
diag_​system_​cpu_​duration_​nanos
{system_​cpu_​state="steal"}
counterCPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Stolen by the hypervisor)
diag_​memory_​available_​bytesgaugeHost memory available without swapping (/proc/meminfo available)
diag_​memory_​free_​bytesgaugeHost memory not in use at all (/proc/meminfo free)
diag_​swap_​in_​bytescounterBytes swapped in from disk since boot (pswpin)
diag_​swap_​out_​bytescounterBytes swapped out to disk since boot (pswpout)
diag_​system_​page_​fault_​majorcounterMajor page faults across the host since boot (pgmajfault)
diag_​oom_​killedcounterProcesses killed by the kernel OOM killer since boot (oom_kill)
diag_​cpu_​pressure_​duration_​nanoscounterTime at least one task stalled waiting for CPU since boot (PSI some), 0 if the kernel lacks PSI
diag_​memory_​pressure_​duration_​nanoscounterTime at least one task stalled on memory reclaim or swap since boot (PSI some), 0 if the kernel lacks PSI
diag_​memory_​pressure_​full_​duration_​nanoscounterTime all non-idle tasks stalled on memory together since boot (PSI full), a subset of MemoryPressureDurationNanos
diag_​io_​pressure_​duration_​nanoscounterTime at least one task stalled on IO since boot (PSI some), 0 if the kernel lacks PSI
diag_​io_​pressure_​full_​duration_​nanoscounterTime all non-idle tasks stalled on IO together since boot (PSI full), a subset of IoPressureDurationNanos
diag_​accounts_​disk_​read_​bytescounterBytes read from the accounts disk since boot (/sys/block stat sectors read), 0 if not on a block device
diag_​accounts_​disk_​written_​bytescounterBytes written to the accounts disk since boot (/sys/block stat sectors written), 0 if not on a block device
diag_​accounts_​disk_​writtencounterWrite requests the accounts disk completed since boot (/sys/block stat writes completed), 0 if not on a block device
diag_​accounts_​disk_​write_​duration_​nanoscounterTime write requests to the accounts disk spent from submission to completion since the diag tile started, summed over requests (/sys/block stat write ticks), 0 if not on a block device
diag_​accounts_​disk_​busy_​duration_​nanoscounterTime the accounts disk had at least one request in flight since the diag tile started (/sys/block stat io_ticks), 0 if not on a block device
diag_​accounts_​disk_​request_​in_​flight_​integral_​nanoscounterIntegral of requests in flight at the accounts disk over time since the diag tile started (request-nanoseconds, /sys/block stat time_in_queue); divide by elapsed time for the average queue depth, 0 if not on a block device
diag_​accounts_​disk_​request_​in_​flightgaugeRequests the accounts disk had in flight at the last sample (/sys/block stat in_flight), 0 if not on a block device

Event Tile ​

MetricTypeDescription
event_​conn_​stategaugeTelemetry server connection state: 0=disconnected, 1=connecting, 2=authenticating, 3=connected
event_​queue_​usedgaugeEvents in the queue, sent-but-unacknowledged plus unsent
event_​queue_​unsent_​countgaugeEvents in the event queue not yet sent to the event service
event_​queue_​evictedcounterOldest events overwritten because the queue was full
event_​queue_​used_​bytesgaugeBytes used in the event queue
event_​queue_​capacity_​bytesgaugeEvent queue capacity
event_​event_​txcounterEvents written to the stream, including resends after reconnect
event_​ack_​rxcounterAcknowledgements received; one ack may retire many events
event_​last_​acked_​noncegaugeNonce of the most recently acknowledged event, 0 if none
event_​stream_​tx_​bytescounterBytes sent to the event service
event_​stream_​rx_​bytescounterBytes received from the event service
event_​auth_​failedcounterAuthentication failures with the event service
event_​response_​malformed_​rejectedcounterMalformed responses received from the event service
event_​conn_​attemptedcounterConnection attempts to the event service
event_​handshake_​timed_​outcounterAuthentication handshakes that timed out
event_​credit_​stalledcounterReconnects forced because the event service stopped granting HTTP/2 flow-control credit while data was pending

Gui Tile ​

MetricTypeDescription
gui_​conn_​opengaugeOpen HTTP connections to the GUI service, excluding connections upgraded to WebSocket
gui_​websocket_​conn_​opengaugeOpen WebSocket connections to the GUI service
gui_​websocket_​frame_​txcounterWebSocket frames sent to all connections to the GUI service
gui_​websocket_​frame_​rxcounterWebSocket frames received from all connections to the GUI service
gui_​conn_​tx_​bytescounterBytes sent on all connections to the GUI service
gui_​conn_​rx_​bytescounterBytes received on all connections to the GUI service
gui_​disk_​allocated_​bytesgaugeLogical bytes allocated by the GUI database backing file
gui_​db_​used_​bytes
{gui_​db_​type="scheduler_​counts"}
gaugeBytes occupied by records present in the store ring (scheduler_counts)
gui_​db_​used_​bytes
{gui_​db_​type="tile_​timers"}
gaugeBytes occupied by records present in the store ring (tile_timers)
gui_​db_​used_​bytes
{gui_​db_​type="shred_​events"}
gaugeBytes occupied by records present in the store ring (shred_events)
gui_​db_​used_​bytes
{gui_​db_​type="txn_​start"}
gaugeBytes occupied by records present in the store ring (txn_start)
gui_​db_​used_​bytes
{gui_​db_​type="txn_​end"}
gaugeBytes occupied by records present in the store ring (txn_end)
gui_​db_​used_​bytes
{gui_​db_​type="tower"}
gaugeBytes occupied by records present in the store ring (Retired: no longer a database, always zero)
gui_​db_​used_​bytes
{gui_​db_​type="slot"}
gaugeBytes occupied by records present in the store ring (slot)
gui_​db_​used_​bytes
{gui_​db_​type="leader_​slot"}
gaugeBytes occupied by records present in the store ring (leader_slot)
gui_​db_​used_​bytes
{gui_​db_​type="epoch"}
gaugeBytes occupied by records present in the store ring (epoch)
gui_​db_​used_​bytes
{gui_​db_​type="tile_​stats"}
gaugeBytes occupied by records present in the store ring (tile_stats)
gui_​db_​used_​bytes
{gui_​db_​type="txn_​waterfall"}
gaugeBytes occupied by records present in the store ring (txn_waterfall)
gui_​db_​used_​bytes
{gui_​db_​type="timeline_​day"}
gaugeBytes occupied by records present in the store ring (timeline_day)
gui_​db_​used_​bytes
{gui_​db_​type="replay_​txn"}
gaugeBytes occupied by records present in the store ring (replay_txn)
gui_​db_​used_​bytes
{gui_​db_​type="timeline_​tile_​day"}
gaugeBytes occupied by records present in the store ring (timeline_tile_day)
gui_​db_​used_​bytes
{gui_​db_​type="fec_​events"}
gaugeBytes occupied by records present in the store ring (fec_events)
gui_​db_​used_​bytes
{gui_​db_​type="fec_​event_​archive"}
gaugeBytes occupied by records present in the store ring (fec_event_archive)
gui_​db_​allocated_​bytes
{gui_​db_​type="scheduler_​counts"}
gaugeBytes of regions owned by each database ring; grows on demand (scheduler_counts)
gui_​db_​allocated_​bytes
{gui_​db_​type="tile_​timers"}
gaugeBytes of regions owned by each database ring; grows on demand (tile_timers)
gui_​db_​allocated_​bytes
{gui_​db_​type="shred_​events"}
gaugeBytes of regions owned by each database ring; grows on demand (shred_events)
gui_​db_​allocated_​bytes
{gui_​db_​type="txn_​start"}
gaugeBytes of regions owned by each database ring; grows on demand (txn_start)
gui_​db_​allocated_​bytes
{gui_​db_​type="txn_​end"}
gaugeBytes of regions owned by each database ring; grows on demand (txn_end)
gui_​db_​allocated_​bytes
{gui_​db_​type="tower"}
gaugeBytes of regions owned by each database ring; grows on demand (Retired: no longer a database, always zero)
gui_​db_​allocated_​bytes
{gui_​db_​type="slot"}
gaugeBytes of regions owned by each database ring; grows on demand (slot)
gui_​db_​allocated_​bytes
{gui_​db_​type="leader_​slot"}
gaugeBytes of regions owned by each database ring; grows on demand (leader_slot)
gui_​db_​allocated_​bytes
{gui_​db_​type="epoch"}
gaugeBytes of regions owned by each database ring; grows on demand (epoch)
gui_​db_​allocated_​bytes
{gui_​db_​type="tile_​stats"}
gaugeBytes of regions owned by each database ring; grows on demand (tile_stats)
gui_​db_​allocated_​bytes
{gui_​db_​type="txn_​waterfall"}
gaugeBytes of regions owned by each database ring; grows on demand (txn_waterfall)
gui_​db_​allocated_​bytes
{gui_​db_​type="timeline_​day"}
gaugeBytes of regions owned by each database ring; grows on demand (timeline_day)
gui_​db_​allocated_​bytes
{gui_​db_​type="replay_​txn"}
gaugeBytes of regions owned by each database ring; grows on demand (replay_txn)
gui_​db_​allocated_​bytes
{gui_​db_​type="timeline_​tile_​day"}
gaugeBytes of regions owned by each database ring; grows on demand (timeline_tile_day)
gui_​db_​allocated_​bytes
{gui_​db_​type="fec_​events"}
gaugeBytes of regions owned by each database ring; grows on demand (fec_events)
gui_​db_​allocated_​bytes
{gui_​db_​type="fec_​event_​archive"}
gaugeBytes of regions owned by each database ring; grows on demand (fec_event_archive)
gui_​db_​free_​bytesgaugeBytes of free regions not owned by any database ring
gui_​db_​record_​used
{gui_​db_​type="scheduler_​counts"}
gaugeRecords present in the store ring (scheduler_counts)
gui_​db_​record_​used
{gui_​db_​type="tile_​timers"}
gaugeRecords present in the store ring (tile_timers)
gui_​db_​record_​used
{gui_​db_​type="shred_​events"}
gaugeRecords present in the store ring (shred_events)
gui_​db_​record_​used
{gui_​db_​type="txn_​start"}
gaugeRecords present in the store ring (txn_start)
gui_​db_​record_​used
{gui_​db_​type="txn_​end"}
gaugeRecords present in the store ring (txn_end)
gui_​db_​record_​used
{gui_​db_​type="tower"}
gaugeRecords present in the store ring (Retired: no longer a database, always zero)
gui_​db_​record_​used
{gui_​db_​type="slot"}
gaugeRecords present in the store ring (slot)
gui_​db_​record_​used
{gui_​db_​type="leader_​slot"}
gaugeRecords present in the store ring (leader_slot)
gui_​db_​record_​used
{gui_​db_​type="epoch"}
gaugeRecords present in the store ring (epoch)
gui_​db_​record_​used
{gui_​db_​type="tile_​stats"}
gaugeRecords present in the store ring (tile_stats)
gui_​db_​record_​used
{gui_​db_​type="txn_​waterfall"}
gaugeRecords present in the store ring (txn_waterfall)
gui_​db_​record_​used
{gui_​db_​type="timeline_​day"}
gaugeRecords present in the store ring (timeline_day)
gui_​db_​record_​used
{gui_​db_​type="replay_​txn"}
gaugeRecords present in the store ring (replay_txn)
gui_​db_​record_​used
{gui_​db_​type="timeline_​tile_​day"}
gaugeRecords present in the store ring (timeline_tile_day)
gui_​db_​record_​used
{gui_​db_​type="fec_​events"}
gaugeRecords present in the store ring (fec_events)
gui_​db_​record_​used
{gui_​db_​type="fec_​event_​archive"}
gaugeRecords present in the store ring (fec_event_archive)
gui_​db_​record_​allocated
{gui_​db_​type="scheduler_​counts"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (scheduler_counts)
gui_​db_​record_​allocated
{gui_​db_​type="tile_​timers"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (tile_timers)
gui_​db_​record_​allocated
{gui_​db_​type="shred_​events"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (shred_events)
gui_​db_​record_​allocated
{gui_​db_​type="txn_​start"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (txn_start)
gui_​db_​record_​allocated
{gui_​db_​type="txn_​end"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (txn_end)
gui_​db_​record_​allocated
{gui_​db_​type="tower"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (Retired: no longer a database, always zero)
gui_​db_​record_​allocated
{gui_​db_​type="slot"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (slot)
gui_​db_​record_​allocated
{gui_​db_​type="leader_​slot"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (leader_slot)
gui_​db_​record_​allocated
{gui_​db_​type="epoch"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (epoch)
gui_​db_​record_​allocated
{gui_​db_​type="tile_​stats"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (tile_stats)
gui_​db_​record_​allocated
{gui_​db_​type="txn_​waterfall"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (txn_waterfall)
gui_​db_​record_​allocated
{gui_​db_​type="timeline_​day"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (timeline_day)
gui_​db_​record_​allocated
{gui_​db_​type="replay_​txn"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (replay_txn)
gui_​db_​record_​allocated
{gui_​db_​type="timeline_​tile_​day"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (timeline_tile_day)
gui_​db_​record_​allocated
{gui_​db_​type="fec_​events"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (fec_events)
gui_​db_​record_​allocated
{gui_​db_​type="fec_​event_​archive"}
gaugeRecord capacity of regions owned by each database ring; grows on demand (fec_event_archive)
gui_​db_​kv_​queried
{gui_​db_​type="scheduler_​counts"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (scheduler_counts)
gui_​db_​kv_​queried
{gui_​db_​type="tile_​timers"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (tile_timers)
gui_​db_​kv_​queried
{gui_​db_​type="shred_​events"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (shred_events)
gui_​db_​kv_​queried
{gui_​db_​type="txn_​start"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (txn_start)
gui_​db_​kv_​queried
{gui_​db_​type="txn_​end"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (txn_end)
gui_​db_​kv_​queried
{gui_​db_​type="tower"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (Retired: no longer a database, always zero)
gui_​db_​kv_​queried
{gui_​db_​type="slot"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (slot)
gui_​db_​kv_​queried
{gui_​db_​type="leader_​slot"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (leader_slot)
gui_​db_​kv_​queried
{gui_​db_​type="epoch"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (epoch)
gui_​db_​kv_​queried
{gui_​db_​type="tile_​stats"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (tile_stats)
gui_​db_​kv_​queried
{gui_​db_​type="txn_​waterfall"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (txn_waterfall)
gui_​db_​kv_​queried
{gui_​db_​type="timeline_​day"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (timeline_day)
gui_​db_​kv_​queried
{gui_​db_​type="replay_​txn"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (replay_txn)
gui_​db_​kv_​queried
{gui_​db_​type="timeline_​tile_​day"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (timeline_tile_day)
gui_​db_​kv_​queried
{gui_​db_​type="fec_​events"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (fec_events)
gui_​db_​kv_​queried
{gui_​db_​type="fec_​event_​archive"}
counterKey-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (fec_event_archive)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="scheduler_​counts"}
counterTime-series records appended, by database (scheduler_counts)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="tile_​timers"}
counterTime-series records appended, by database (tile_timers)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="shred_​events"}
counterTime-series records appended, by database (shred_events)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="txn_​start"}
counterTime-series records appended, by database (txn_start)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="txn_​end"}
counterTime-series records appended, by database (txn_end)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="tower"}
counterTime-series records appended, by database (Retired: no longer a database, always zero)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="slot"}
counterTime-series records appended, by database (slot)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="leader_​slot"}
counterTime-series records appended, by database (leader_slot)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="epoch"}
counterTime-series records appended, by database (epoch)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="tile_​stats"}
counterTime-series records appended, by database (tile_stats)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="txn_​waterfall"}
counterTime-series records appended, by database (txn_waterfall)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="timeline_​day"}
counterTime-series records appended, by database (timeline_day)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="replay_​txn"}
counterTime-series records appended, by database (replay_txn)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="timeline_​tile_​day"}
counterTime-series records appended, by database (timeline_tile_day)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="fec_​events"}
counterTime-series records appended, by database (fec_events)
gui_​db_​time_​series_​record_​appended
{gui_​db_​type="fec_​event_​archive"}
counterTime-series records appended, by database (fec_event_archive)
gui_​db_​time_​series_​scanned
{gui_​db_​type="scheduler_​counts"}
counterTime-series scans, by database (scheduler_counts)
gui_​db_​time_​series_​scanned
{gui_​db_​type="tile_​timers"}
counterTime-series scans, by database (tile_timers)
gui_​db_​time_​series_​scanned
{gui_​db_​type="shred_​events"}
counterTime-series scans, by database (shred_events)
gui_​db_​time_​series_​scanned
{gui_​db_​type="txn_​start"}
counterTime-series scans, by database (txn_start)
gui_​db_​time_​series_​scanned
{gui_​db_​type="txn_​end"}
counterTime-series scans, by database (txn_end)
gui_​db_​time_​series_​scanned
{gui_​db_​type="tower"}
counterTime-series scans, by database (Retired: no longer a database, always zero)
gui_​db_​time_​series_​scanned
{gui_​db_​type="slot"}
counterTime-series scans, by database (slot)
gui_​db_​time_​series_​scanned
{gui_​db_​type="leader_​slot"}
counterTime-series scans, by database (leader_slot)
gui_​db_​time_​series_​scanned
{gui_​db_​type="epoch"}
counterTime-series scans, by database (epoch)
gui_​db_​time_​series_​scanned
{gui_​db_​type="tile_​stats"}
counterTime-series scans, by database (tile_stats)
gui_​db_​time_​series_​scanned
{gui_​db_​type="txn_​waterfall"}
counterTime-series scans, by database (txn_waterfall)
gui_​db_​time_​series_​scanned
{gui_​db_​type="timeline_​day"}
counterTime-series scans, by database (timeline_day)
gui_​db_​time_​series_​scanned
{gui_​db_​type="replay_​txn"}
counterTime-series scans, by database (replay_txn)
gui_​db_​time_​series_​scanned
{gui_​db_​type="timeline_​tile_​day"}
counterTime-series scans, by database (timeline_tile_day)
gui_​db_​time_​series_​scanned
{gui_​db_​type="fec_​events"}
counterTime-series scans, by database (fec_events)
gui_​db_​time_​series_​scanned
{gui_​db_​type="fec_​event_​archive"}
counterTime-series scans, by database (fec_event_archive)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="scheduler_​counts"}
counterTime-series records returned by scans, by database (scheduler_counts)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="tile_​timers"}
counterTime-series records returned by scans, by database (tile_timers)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="shred_​events"}
counterTime-series records returned by scans, by database (shred_events)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="txn_​start"}
counterTime-series records returned by scans, by database (txn_start)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="txn_​end"}
counterTime-series records returned by scans, by database (txn_end)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="tower"}
counterTime-series records returned by scans, by database (Retired: no longer a database, always zero)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="slot"}
counterTime-series records returned by scans, by database (slot)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="leader_​slot"}
counterTime-series records returned by scans, by database (leader_slot)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="epoch"}
counterTime-series records returned by scans, by database (epoch)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="tile_​stats"}
counterTime-series records returned by scans, by database (tile_stats)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="txn_​waterfall"}
counterTime-series records returned by scans, by database (txn_waterfall)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="timeline_​day"}
counterTime-series records returned by scans, by database (timeline_day)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="replay_​txn"}
counterTime-series records returned by scans, by database (replay_txn)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="timeline_​tile_​day"}
counterTime-series records returned by scans, by database (timeline_tile_day)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="fec_​events"}
counterTime-series records returned by scans, by database (fec_events)
gui_​db_​time_​series_​record_​read
{gui_​db_​type="fec_​event_​archive"}
counterTime-series records returned by scans, by database (fec_event_archive)
gui_​db_​record_​evicted
{gui_​db_​type="scheduler_​counts"}
counterRecords evicted from the database, by database (scheduler_counts)
gui_​db_​record_​evicted
{gui_​db_​type="tile_​timers"}
counterRecords evicted from the database, by database (tile_timers)
gui_​db_​record_​evicted
{gui_​db_​type="shred_​events"}
counterRecords evicted from the database, by database (shred_events)
gui_​db_​record_​evicted
{gui_​db_​type="txn_​start"}
counterRecords evicted from the database, by database (txn_start)
gui_​db_​record_​evicted
{gui_​db_​type="txn_​end"}
counterRecords evicted from the database, by database (txn_end)
gui_​db_​record_​evicted
{gui_​db_​type="tower"}
counterRecords evicted from the database, by database (Retired: no longer a database, always zero)
gui_​db_​record_​evicted
{gui_​db_​type="slot"}
counterRecords evicted from the database, by database (slot)
gui_​db_​record_​evicted
{gui_​db_​type="leader_​slot"}
counterRecords evicted from the database, by database (leader_slot)
gui_​db_​record_​evicted
{gui_​db_​type="epoch"}
counterRecords evicted from the database, by database (epoch)
gui_​db_​record_​evicted
{gui_​db_​type="tile_​stats"}
counterRecords evicted from the database, by database (tile_stats)
gui_​db_​record_​evicted
{gui_​db_​type="txn_​waterfall"}
counterRecords evicted from the database, by database (txn_waterfall)
gui_​db_​record_​evicted
{gui_​db_​type="timeline_​day"}
counterRecords evicted from the database, by database (timeline_day)
gui_​db_​record_​evicted
{gui_​db_​type="replay_​txn"}
counterRecords evicted from the database, by database (replay_txn)
gui_​db_​record_​evicted
{gui_​db_​type="timeline_​tile_​day"}
counterRecords evicted from the database, by database (timeline_tile_day)
gui_​db_​record_​evicted
{gui_​db_​type="fec_​events"}
counterRecords evicted from the database, by database (fec_events)
gui_​db_​record_​evicted
{gui_​db_​type="fec_​event_​archive"}
counterRecords evicted from the database, by database (fec_event_archive)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="scheduler_​counts"}
counterEviction passes that removed at least one record, by database (scheduler_counts)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="tile_​timers"}
counterEviction passes that removed at least one record, by database (tile_timers)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="shred_​events"}
counterEviction passes that removed at least one record, by database (shred_events)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="txn_​start"}
counterEviction passes that removed at least one record, by database (txn_start)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="txn_​end"}
counterEviction passes that removed at least one record, by database (txn_end)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="tower"}
counterEviction passes that removed at least one record, by database (Retired: no longer a database, always zero)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="slot"}
counterEviction passes that removed at least one record, by database (slot)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="leader_​slot"}
counterEviction passes that removed at least one record, by database (leader_slot)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="epoch"}
counterEviction passes that removed at least one record, by database (epoch)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="tile_​stats"}
counterEviction passes that removed at least one record, by database (tile_stats)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="txn_​waterfall"}
counterEviction passes that removed at least one record, by database (txn_waterfall)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="timeline_​day"}
counterEviction passes that removed at least one record, by database (timeline_day)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="replay_​txn"}
counterEviction passes that removed at least one record, by database (replay_txn)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="timeline_​tile_​day"}
counterEviction passes that removed at least one record, by database (timeline_tile_day)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="fec_​events"}
counterEviction passes that removed at least one record, by database (fec_events)
gui_​db_​eviction_​pass_​completed
{gui_​db_​type="fec_​event_​archive"}
counterEviction passes that removed at least one record, by database (fec_event_archive)
gui_​db_​region_​claimed
{gui_​db_​type="scheduler_​counts"}
counterRegions claimed from the free list, by database (scheduler_counts)
gui_​db_​region_​claimed
{gui_​db_​type="tile_​timers"}
counterRegions claimed from the free list, by database (tile_timers)
gui_​db_​region_​claimed
{gui_​db_​type="shred_​events"}
counterRegions claimed from the free list, by database (shred_events)
gui_​db_​region_​claimed
{gui_​db_​type="txn_​start"}
counterRegions claimed from the free list, by database (txn_start)
gui_​db_​region_​claimed
{gui_​db_​type="txn_​end"}
counterRegions claimed from the free list, by database (txn_end)
gui_​db_​region_​claimed
{gui_​db_​type="tower"}
counterRegions claimed from the free list, by database (Retired: no longer a database, always zero)
gui_​db_​region_​claimed
{gui_​db_​type="slot"}
counterRegions claimed from the free list, by database (slot)
gui_​db_​region_​claimed
{gui_​db_​type="leader_​slot"}
counterRegions claimed from the free list, by database (leader_slot)
gui_​db_​region_​claimed
{gui_​db_​type="epoch"}
counterRegions claimed from the free list, by database (epoch)
gui_​db_​region_​claimed
{gui_​db_​type="tile_​stats"}
counterRegions claimed from the free list, by database (tile_stats)
gui_​db_​region_​claimed
{gui_​db_​type="txn_​waterfall"}
counterRegions claimed from the free list, by database (txn_waterfall)
gui_​db_​region_​claimed
{gui_​db_​type="timeline_​day"}
counterRegions claimed from the free list, by database (timeline_day)
gui_​db_​region_​claimed
{gui_​db_​type="replay_​txn"}
counterRegions claimed from the free list, by database (replay_txn)
gui_​db_​region_​claimed
{gui_​db_​type="timeline_​tile_​day"}
counterRegions claimed from the free list, by database (timeline_tile_day)
gui_​db_​region_​claimed
{gui_​db_​type="fec_​events"}
counterRegions claimed from the free list, by database (fec_events)
gui_​db_​region_​claimed
{gui_​db_​type="fec_​event_​archive"}
counterRegions claimed from the free list, by database (fec_event_archive)
gui_​db_​region_​released
{gui_​db_​type="scheduler_​counts"}
counterRegions returned to the free list, by database (scheduler_counts)
gui_​db_​region_​released
{gui_​db_​type="tile_​timers"}
counterRegions returned to the free list, by database (tile_timers)
gui_​db_​region_​released
{gui_​db_​type="shred_​events"}
counterRegions returned to the free list, by database (shred_events)
gui_​db_​region_​released
{gui_​db_​type="txn_​start"}
counterRegions returned to the free list, by database (txn_start)
gui_​db_​region_​released
{gui_​db_​type="txn_​end"}
counterRegions returned to the free list, by database (txn_end)
gui_​db_​region_​released
{gui_​db_​type="tower"}
counterRegions returned to the free list, by database (Retired: no longer a database, always zero)
gui_​db_​region_​released
{gui_​db_​type="slot"}
counterRegions returned to the free list, by database (slot)
gui_​db_​region_​released
{gui_​db_​type="leader_​slot"}
counterRegions returned to the free list, by database (leader_slot)
gui_​db_​region_​released
{gui_​db_​type="epoch"}
counterRegions returned to the free list, by database (epoch)
gui_​db_​region_​released
{gui_​db_​type="tile_​stats"}
counterRegions returned to the free list, by database (tile_stats)
gui_​db_​region_​released
{gui_​db_​type="txn_​waterfall"}
counterRegions returned to the free list, by database (txn_waterfall)
gui_​db_​region_​released
{gui_​db_​type="timeline_​day"}
counterRegions returned to the free list, by database (timeline_day)
gui_​db_​region_​released
{gui_​db_​type="replay_​txn"}
counterRegions returned to the free list, by database (replay_txn)
gui_​db_​region_​released
{gui_​db_​type="timeline_​tile_​day"}
counterRegions returned to the free list, by database (timeline_tile_day)
gui_​db_​region_​released
{gui_​db_​type="fec_​events"}
counterRegions returned to the free list, by database (fec_events)
gui_​db_​region_​released
{gui_​db_​type="fec_​event_​archive"}
counterRegions returned to the free list, by database (fec_event_archive)
gui_​db_​record_​dropped
{gui_​db_​type="scheduler_​counts"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (scheduler_counts)
gui_​db_​record_​dropped
{gui_​db_​type="tile_​timers"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (tile_timers)
gui_​db_​record_​dropped
{gui_​db_​type="shred_​events"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (shred_events)
gui_​db_​record_​dropped
{gui_​db_​type="txn_​start"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (txn_start)
gui_​db_​record_​dropped
{gui_​db_​type="txn_​end"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (txn_end)
gui_​db_​record_​dropped
{gui_​db_​type="tower"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (Retired: no longer a database, always zero)
gui_​db_​record_​dropped
{gui_​db_​type="slot"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (slot)
gui_​db_​record_​dropped
{gui_​db_​type="leader_​slot"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (leader_slot)
gui_​db_​record_​dropped
{gui_​db_​type="epoch"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (epoch)
gui_​db_​record_​dropped
{gui_​db_​type="tile_​stats"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (tile_stats)
gui_​db_​record_​dropped
{gui_​db_​type="txn_​waterfall"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (txn_waterfall)
gui_​db_​record_​dropped
{gui_​db_​type="timeline_​day"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (timeline_day)
gui_​db_​record_​dropped
{gui_​db_​type="replay_​txn"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (replay_txn)
gui_​db_​record_​dropped
{gui_​db_​type="timeline_​tile_​day"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (timeline_tile_day)
gui_​db_​record_​dropped
{gui_​db_​type="fec_​events"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (fec_events)
gui_​db_​record_​dropped
{gui_​db_​type="fec_​event_​archive"}
counterRecords dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (fec_event_archive)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="scheduler_​counts"}
counterWrites that evicted records before succeeding, by database (scheduler_counts)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="tile_​timers"}
counterWrites that evicted records before succeeding, by database (tile_timers)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="shred_​events"}
counterWrites that evicted records before succeeding, by database (shred_events)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="txn_​start"}
counterWrites that evicted records before succeeding, by database (txn_start)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="txn_​end"}
counterWrites that evicted records before succeeding, by database (txn_end)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="tower"}
counterWrites that evicted records before succeeding, by database (Retired: no longer a database, always zero)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="slot"}
counterWrites that evicted records before succeeding, by database (slot)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="leader_​slot"}
counterWrites that evicted records before succeeding, by database (leader_slot)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="epoch"}
counterWrites that evicted records before succeeding, by database (epoch)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="tile_​stats"}
counterWrites that evicted records before succeeding, by database (tile_stats)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="txn_​waterfall"}
counterWrites that evicted records before succeeding, by database (txn_waterfall)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="timeline_​day"}
counterWrites that evicted records before succeeding, by database (timeline_day)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="replay_​txn"}
counterWrites that evicted records before succeeding, by database (replay_txn)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="timeline_​tile_​day"}
counterWrites that evicted records before succeeding, by database (timeline_tile_day)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="fec_​events"}
counterWrites that evicted records before succeeding, by database (fec_events)
gui_​db_​forced_​evict_​write_​completed
{gui_​db_​type="fec_​event_​archive"}
counterWrites that evicted records before succeeding, by database (fec_event_archive)

Metric Tile ​

MetricTypeDescription
metric_​boot_​timegaugeWall-clock time the validator started
metric_​conn_​opengaugeOpen HTTP connections to the Prometheus endpoint
metric_​conn_​tx_​bytescounterBytes sent in responses from the Prometheus endpoint
metric_​conn_​rx_​bytescounterBytes received in requests to the Prometheus endpoint

Rpc Tile ​

MetricTypeDescription
rpc_​request_​duration_​nanoshistogramTime to handle one HTTP request or WebSocket message, excluding socket I/O
rpc_​request_​handled
{rpc_​method="genesis"}
counterRPC requests handled, by method, before parameter validation (GET /genesis.tar.bz2)
rpc_​request_​handled
{rpc_​method="unknown"}
counterRPC requests handled, by method, before parameter validation (Unknown method)
rpc_​request_​handled
{rpc_​method="get_​account_​info"}
counterRPC requests handled, by method, before parameter validation (getAccountInfo)
rpc_​request_​handled
{rpc_​method="get_​balance"}
counterRPC requests handled, by method, before parameter validation (getBalance)
rpc_​request_​handled
{rpc_​method="get_​block_​height"}
counterRPC requests handled, by method, before parameter validation (getBlockHeight)
rpc_​request_​handled
{rpc_​method="get_​cluster_​nodes"}
counterRPC requests handled, by method, before parameter validation (getClusterNodes)
rpc_​request_​handled
{rpc_​method="get_​epoch_​info"}
counterRPC requests handled, by method, before parameter validation (getEpochInfo)
rpc_​request_​handled
{rpc_​method="get_​genesis_​hash"}
counterRPC requests handled, by method, before parameter validation (getGenesisHash)
rpc_​request_​handled
{rpc_​method="get_​health"}
counterRPC requests handled, by method, before parameter validation (getHealth)
rpc_​request_​handled
{rpc_​method="get_​identity"}
counterRPC requests handled, by method, before parameter validation (getIdentity)
rpc_​request_​handled
{rpc_​method="get_​inflation_​governor"}
counterRPC requests handled, by method, before parameter validation (getInflationGovernor)
rpc_​request_​handled
{rpc_​method="get_​latest_​blockhash"}
counterRPC requests handled, by method, before parameter validation (getLatestBlockhash)
rpc_​request_​handled
{rpc_​method="get_​minimum_​balance_​for_​rent_​exemption"}
counterRPC requests handled, by method, before parameter validation (getMinimumBalanceForRentExemption)
rpc_​request_​handled
{rpc_​method="get_​multiple_​accounts"}
counterRPC requests handled, by method, before parameter validation (getMultipleAccounts)
rpc_​request_​handled
{rpc_​method="get_​slot"}
counterRPC requests handled, by method, before parameter validation (getSlot)
rpc_​request_​handled
{rpc_​method="get_​transaction_​count"}
counterRPC requests handled, by method, before parameter validation (getTransactionCount)
rpc_​request_​handled
{rpc_​method="get_​version"}
counterRPC requests handled, by method, before parameter validation (getVersion)
rpc_​request_​handled
{rpc_​method="get_​max_​retransmit_​slot"}
counterRPC requests handled, by method, before parameter validation (getMaxRetransmitSlot)
rpc_​request_​handled
{rpc_​method="get_​slot_​leader"}
counterRPC requests handled, by method, before parameter validation (getSlotLeader)
rpc_​request_​handled
{rpc_​method="get_​slot_​leaders"}
counterRPC requests handled, by method, before parameter validation (getSlotLeaders)
rpc_​request_​handled
{rpc_​method="get_​leader_​schedule"}
counterRPC requests handled, by method, before parameter validation (getLeaderSchedule)
rpc_​request_​handled
{rpc_​method="get_​ag_​genesis_​cert"}
counterRPC requests handled, by method, before parameter validation (getAgGenesisCert)
rpc_​request_​handled
{rpc_​method="slot_​subscribe"}
counterRPC requests handled, by method, before parameter validation (slotSubscribe)
rpc_​request_​handled
{rpc_​method="slot_​unsubscribe"}
counterRPC requests handled, by method, before parameter validation (slotUnsubscribe)
rpc_​request_​handled
{rpc_​method="vote_​subscribe"}
counterRPC requests handled, by method, before parameter validation (voteSubscribe)
rpc_​request_​handled
{rpc_​method="vote_​unsubscribe"}
counterRPC requests handled, by method, before parameter validation (voteUnsubscribe)
rpc_​request_​handled
{rpc_​method="unimplemented"}
counterRPC requests handled, by method, before parameter validation (Known method not implemented by Firedancer)
rpc_​request_​handled
{rpc_​method="invalid"}
counterRPC requests handled, by method, before parameter validation (Malformed JSON-RPC request)
rpc_​request_​handled
{rpc_​method="health"}
counterRPC requests handled, by method, before parameter validation (GET /health)
rpc_​conn_​opengaugeOpen HTTP connections to the RPC service
rpc_​websocket_​conn_​opengaugeOpen WebSocket connections to the RPC service
rpc_​websocket_​subscriber_​count
{rpc_​notification_​type="vote"}
gaugeWebSocket connections subscribed to each notification type (vote)
rpc_​websocket_​subscriber_​count
{rpc_​notification_​type="slot"}
gaugeWebSocket connections subscribed to each notification type (slot)
rpc_​websocket_​notification_​published
{rpc_​notification_​type="vote"}
counterNotifications published to at least one subscriber, by notification type (vote)
rpc_​websocket_​notification_​published
{rpc_​notification_​type="slot"}
counterNotifications published to at least one subscriber, by notification type (slot)
rpc_​websocket_​notification_​tx
{rpc_​notification_​type="vote"}
counterNotifications sent, counting each subscriber, by notification type (vote)
rpc_​websocket_​notification_​tx
{rpc_​notification_​type="slot"}
counterNotifications sent, counting each subscriber, by notification type (slot)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le512_​b"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le2_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le8_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le32_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le128_​kib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le1_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB)
rpc_​accdb_​account_​readonly_​acquired
{accdb_​cache_​class="le10_​mib"}
counterReadonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (0-128 B)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le512_​b"}
counterAccounts not found in the account database cache and read from disk, by cache size class (129-512 B)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le2_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le8_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (2-8 KiB)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le32_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (8-32 KiB)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le128_​kib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (32-128 KiB)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le1_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB)
rpc_​accdb_​account_​cache_​missed
{accdb_​cache_​class="le10_​mib"}
counterAccounts not found in the account database cache and read from disk, by cache size class (1-10 MiB)
rpc_​accdb_​account_​waitedcounterAccounts that had to wait for another reader to finish loading them into the cache
rpc_​accdb_​disk_​read_​bytescounterBytes read from the account database file
rpc_​accdb_​disk_​readcounterReads from the account database file
rpc_​accdb_​copied_​bytescounterAccount data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded

Mlx5 Tile ​

MetricTypeDescription
mlx5_​pkt_​rxcounterPackets successfully published to output links
mlx5_​pkt_​rx_​bytescounterBytes of packets successfully published to output links
mlx5_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="malformed"}
counterReceived packets dropped, by reason (Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present)
mlx5_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="dst_​ip_​mismatch"}
counterReceived packets dropped, by reason (Destination IP does not match the bind address)
mlx5_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="dst_​port_​unbound"}
counterReceived packets dropped, by reason (No output link for the destination UDP port)
mlx5_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="gre_​invalid"}
counterReceived packets dropped, by reason (GRE tunnel peer or headers invalid)
mlx5_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="gre_​unconfigured"}
counterReceived packets dropped, by reason (GRE packet with no GRE tunnel configured)
mlx5_​gre_​pkt_​rxcounterGRE packets decapsulated and published; subset of PktRx
mlx5_​vlan_​pkt_​rxcounterVLAN-tagged packets with the tag stripped in software and published; subset of PktRx
mlx5_​rx_​buffer_​usedgaugeRX buffers not currently posted to the NIC
mlx5_​rx_​buffer_​freegaugeRX buffers currently posted to the NIC
mlx5_​pkt_​tx_​completedcounterPackets completed by the NIC for transmission
mlx5_​pkt_​tx_​completed_​bytescounterBytes of packets completed by the NIC
mlx5_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="invalid"}
counterTransmit jobs dropped, by reason (Invalid packet headers, or GRE packet exceeding the MTU)
mlx5_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="source_​ip_​missing"}
counterTransmit jobs dropped, by reason (No source IP address chosen)
mlx5_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="ring_​full"}
counterTransmit jobs dropped, by reason (TX ring full or no TX buffer available)
mlx5_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="gre_​no_​route"}
counterTransmit jobs dropped, by reason (GRE tunnel addresses missing or outer route is itself tunnelled)
mlx5_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="socket_​send_​failure"}
counterTransmit jobs dropped, by reason (Socket send failed or failed to send the entire packet)
mlx5_​pkt_​tx_​routed
{pkt_​tx_​route_​result="fast_​path"}
counterTransmit jobs accepted after route resolution, by result (Userland routing succeeded and selected the configured provider)
mlx5_​pkt_​tx_​routed
{pkt_​tx_​route_​result="slow_​path"}
counterTransmit jobs accepted after route resolution, by result (Failed userland routing and selected the kernel fallback)
mlx5_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="no_​route"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (No matching route)
mlx5_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="unsupported_​route"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Unsupported route type)
mlx5_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="interface_​missing"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface not available)
mlx5_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="unsupported_​interface"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface type not supported)
mlx5_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="no_​neighbor"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Next-hop neighbor unresolved or inactive)
mlx5_​gre_​pkt_​tx_​submittedcounterGRE transmit jobs submitted to the NIC or a kernel socket
mlx5_​pkt_​tx_​slow_​path_​submittedcounterPackets submitted to a kernel socket on the slow path
mlx5_​tx_​buffer_​usedgaugeTX buffers pending or submitted to the NIC
mlx5_​tx_​buffer_​freegaugeTX buffers available for a new transmit job

Waker Tile ​

MetricTypeDescription
waker_​epoll_​wait_​returnedcounterepoll_pwait calls on the outer epoll set that returned, including timeouts; EINTR not counted
waker_​wake_​deliveredcounterClient wakes delivered (readiness fseq raised)

Mwaitx Tile ​

MetricTypeDescription
mwaitx_​nap_​takencounterHardware idle-wait naps on the doorbell line
mwaitx_​wake_​issued
{mwaitx_​wake_​cause="doorbell"}
counterfutex wakes issued to parked tiles, by cause (A producer rang the doorbell for a parked consumer)
mwaitx_​wake_​issued
{mwaitx_​wake_​cause="deadline"}
counterfutex wakes issued to parked tiles, by cause (A parked tile's deadline lapsed)
mwaitx_​wake_​issued
{mwaitx_​wake_​cause="sweep"}
counterfutex wakes issued to parked tiles, by cause (The verifying sweep found a pending frag whose doorbell raced or was lost)

Iavf Tile ​

MetricTypeDescription
iavf_​pkt_​rxcounterPackets successfully published to output links
iavf_​pkt_​rx_​bytescounterBytes of packets successfully published to output links
iavf_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="malformed"}
counterReceived packets dropped, by reason (Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present)
iavf_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="dst_​ip_​mismatch"}
counterReceived packets dropped, by reason (Destination IP does not match the bind address)
iavf_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="dst_​port_​unbound"}
counterReceived packets dropped, by reason (No output link for the destination UDP port)
iavf_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="gre_​invalid"}
counterReceived packets dropped, by reason (GRE tunnel peer or headers invalid)
iavf_​pkt_​rx_​dropped
{pkt_​rx_​drop_​reason="gre_​unconfigured"}
counterReceived packets dropped, by reason (GRE packet with no GRE tunnel configured)
iavf_​pkt_​rx_​no_​link_​droppedcounterPackets dropped because the configured VF or bond member is not accepting traffic
iavf_​gre_​pkt_​rxcounterGRE packets decapsulated and published; subset of PktRx
iavf_​vlan_​pkt_​rxcounterVLAN-tagged packets with the tag stripped in software and published; subset of PktRx
iavf_​rx_​buffer_​usedgaugeRX buffers not currently posted to the NIC
iavf_​rx_​buffer_​freegaugeRX buffers currently posted to the NIC
iavf_​pkt_​tx_​completedcounterPackets completed by the NIC for transmission
iavf_​pkt_​tx_​completed_​bytescounterBytes of packets completed by the NIC
iavf_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="invalid"}
counterTransmit jobs dropped, by reason (Invalid packet headers, or GRE packet exceeding the MTU)
iavf_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="source_​ip_​missing"}
counterTransmit jobs dropped, by reason (No source IP address chosen)
iavf_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="ring_​full"}
counterTransmit jobs dropped, by reason (TX ring full or no TX buffer available)
iavf_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="gre_​no_​route"}
counterTransmit jobs dropped, by reason (GRE tunnel addresses missing or outer route is itself tunnelled)
iavf_​pkt_​tx_​dropped
{pkt_​tx_​drop_​reason="socket_​send_​failure"}
counterTransmit jobs dropped, by reason (Socket send failed or failed to send the entire packet)
iavf_​pkt_​tx_​routed
{pkt_​tx_​route_​result="fast_​path"}
counterTransmit jobs accepted after route resolution, by result (Userland routing succeeded and selected the configured provider)
iavf_​pkt_​tx_​routed
{pkt_​tx_​route_​result="slow_​path"}
counterTransmit jobs accepted after route resolution, by result (Failed userland routing and selected the kernel fallback)
iavf_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="no_​route"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (No matching route)
iavf_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="unsupported_​route"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Unsupported route type)
iavf_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="interface_​missing"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface not available)
iavf_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="unsupported_​interface"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface type not supported)
iavf_​pkt_​tx_​slow_​path_​routed
{pkt_​tx_​slow_​path_​reason="no_​neighbor"}
counterTransmit jobs that failed userland routing and selected the kernel fallback, by reason (Next-hop neighbor unresolved or inactive)
iavf_​pkt_​tx_​no_​link_​droppedcounterTransmit jobs dropped because no configured VF or bond member can send traffic
iavf_​pkt_​tx_​loopback_​failedcounterLoopback transmit jobs dropped because the socket send failed or accepted only part of a batch
iavf_​gre_​pkt_​tx_​submittedcounterGRE transmit jobs submitted to the NIC or a kernel socket
iavf_​pkt_​tx_​slow_​path_​submittedcounterPackets submitted to a kernel socket on the slow path
iavf_​tx_​buffer_​usedgaugeTX buffers pending or submitted to the NIC
iavf_​tx_​buffer_​freegaugeTX buffers available for a new transmit job
iavf_​non_​fd_​pkt_​rxcounterPackets received by IAVF VFs that are not intended for Firedancer

Failov Tile ​

MetricTypeDescription

Snapmk Tile ​

MetricTypeDescription
snapmk_​job_​stategaugeSnapshot creation job state: 0=idle (Idle, waiting for a job), 1=start (Starting), 2=tar_headers (Writing tar headers), 3=manifest (Writing manifest), 4=accdb_cache (Writing cached accounts), 5=accdb_cache_flush (Flushing cached accounts), 6=accdb_cache_finish (Waiting for cached account flush), 7=accdb_disk (Writing on-disk accounts), 8=accdb_disk_flush (Flushing on-disk accounts), 9=accdb_disk_finish (Waiting for on-disk account flush), 10=accdb_delta (Writing incremental accounts), 11=accdb_delta_flush (Flushing incremental accounts), 12=accdb_delta_finish (Waiting for incremental account flush), 13=status_cache (Writing status cache), 14=eof_marker (Writing tar EOF marker), 15=done (Done, notifying replay), 16=fail (Failed, cleaning up), 17=sleep (Sleeping), 18=startup (Waiting for startup), 19=startup_burst (Publishing pre-existing snapshots)
snapmk_​snapshot_​started
{snap_​type="full"}
counterSnapshot creation jobs started, by snapshot type (Full snapshot)
snapmk_​snapshot_​started
{snap_​type="incremental"}
counterSnapshot creation jobs started, by snapshot type (Incremental snapshot)
snapmk_​snapshot_​finished
{snap_​type="full"}
counterSnapshot creation jobs finished successfully, by snapshot type (Full snapshot)
snapmk_​snapshot_​finished
{snap_​type="incremental"}
counterSnapshot creation jobs finished successfully, by snapshot type (Incremental snapshot)
snapmk_​last_​started_​slot
{snap_​type="full"}
gaugeSlot of the most recently started snapshot creation, by snapshot type, 0 if none since boot (Full snapshot)
snapmk_​last_​started_​slot
{snap_​type="incremental"}
gaugeSlot of the most recently started snapshot creation, by snapshot type, 0 if none since boot (Incremental snapshot)
snapmk_​last_​finished_​slot
{snap_​type="full"}
gaugeSlot of the most recently finished snapshot creation, by snapshot type, 0 if none since boot (Full snapshot)
snapmk_​last_​finished_​slot
{snap_​type="incremental"}
gaugeSlot of the most recently finished snapshot creation, by snapshot type, 0 if none since boot (Incremental snapshot)
snapmk_​compress_​consumed_​bytescounterUncompressed bytes fed to the compressor
snapmk_​disk_​written_​bytescounterCompressed bytes written to snapshot files
snapmk_​cache_​progress_​bytesgaugeBytes of the accounts database cache region scanned so far for the snapshot being created. Resets when a snapshot starts and holds at the total once the cached accounts phase finishes. Stays zero for an incremental snapshot, which does not scan the cache
snapmk_​cache_​expected_​bytesgaugeSize of the accounts database cache region a cached accounts phase scans
snapmk_​incremental_​account_​progressgaugeAccounts changed since the base snapshot that have been written so far for the incremental snapshot being created. Resets when a snapshot starts, and stays zero for a full snapshot, which has no changed account walk
snapmk_​incremental_​account_​expectedgaugeAccounts changed since the base snapshot, sampled when the incremental snapshot started; 0 for a full snapshot. Replay may record more changes while the job runs, so progress can exceed this
snapmk_​io_​blocked_​duration_​nanoscounterCumulative time blocked on I/O
snapmk_​compress_​duration_​nanoscounterCumulative time spent compressing data
snapmk_​incremental_​account_​usedgaugeDistinct accounts changed in rooted slots since the last full snapshot job started; saturates at IncrementalAccountCapacity
snapmk_​incremental_​account_​capacitygaugeMaximum changed accounts that can be tracked; once reached, no incremental snapshot can be created until the next full snapshot
snapmk_​disk_​allocated_​bytesgaugeLogical bytes allocated across managed snapshot files, including partial and retained files

Snapzp Tile ​

MetricTypeDescription
snapzp_​account_​compressedcounterAccounts compressed
snapzp_​compress_​consumed_​bytescounterUncompressed bytes fed to the compressor
snapzp_​disk_​written_​bytescounterCompressed bytes written to snapshot files
snapzp_​io_​blocked_​duration_​nanoscounterCumulative time blocked on I/O
snapzp_​compress_​duration_​nanoscounterCumulative time spent compressing accounts
snapzp_​cache_​read_​torncounterCached account reads torn by eviction, handed back to snapmk to un-claim

Snaprd Tile ​

MetricTypeDescription
snaprd_​disk_​read_​bytescounterBytes read from the account database file
snaprd_​export_​progress_​bytesgaugeAccount database bytes read so far for the current snapshot creation job
snaprd_​export_​expected_​bytesgaugeAccount database bytes to read for the current snapshot creation job
snaprd_​io_​blocked_​duration_​nanoscounterCumulative time blocked on I/O

Snapsv Tile ​

MetricTypeDescription
snapsv_​snapshot_​available
{snap_​type="full"}
gaugeSnapshot files currently available for download (Full snapshot)
snapsv_​snapshot_​available
{snap_​type="incremental"}
gaugeSnapshot files currently available for download (Incremental snapshot)
snapsv_​http_​request_​servedcounterHTTP requests served
snapsv_​conn_​opengaugeOpen HTTP connections to the snapshot service
snapsv_​http_​rx_​bytescounterBytes received in requests to the snapshot service
snapsv_​http_​tx_​bytescounterBytes sent in responses from the snapshot service

Backt Tile ​

MetricTypeDescription
backt_​end_​slotgaugeLast slot the backtest replays, 0 if unbounded
backt_​start_​slotgaugeSnapshot slot replayed from (replay begins at the next slot), 0 for genesis

Benchs Tile ​

MetricTypeDescription
benchs_​txn_​txcounterBenchmark transactions sent
benchs_​txn_​droppedcounterBenchmark transactions dropped because no QUIC connection was established or it could not send

Votor Tile ​

MetricTypeDescription
votor_​datagram_​rx
{datagram_​rx_​result="vote"}
counterInbound datagrams dispatched, by result (Datagram carried a vote and was handed to vote rx)
votor_​datagram_​rx
{datagram_​rx_​result="cert"}
counterInbound datagrams dispatched, by result (Datagram carried a cert and was handed to cert rx)
votor_​datagram_​rx
{datagram_​rx_​result="not_​ready"}
counterInbound datagrams dispatched, by result (Datagram arrived before the votor tile was initialized)
votor_​datagram_​rx
{datagram_​rx_​result="too_​small"}
counterInbound datagrams dispatched, by result (Datagram was too small to hold a version and a tag)
votor_​datagram_​rx
{datagram_​rx_​result="unknown_​tag"}
counterInbound datagrams dispatched, by result (Datagram tag was neither a vote nor a cert)
votor_​datagram_​rx
{datagram_​rx_​result="not_​admitted"}
counterInbound datagrams dispatched, by result (Datagram arrived on a connection not yet admitted, or already closed)
votor_​vote_​rx
{vote_​rx_​result="success"}
counterInbound votes processed, by result (Vote was handed to the pool and accepted)
votor_​vote_​rx
{vote_​rx_​result="bad_​size"}
counterInbound votes processed, by result (Vote was truncated or had trailing bytes)
votor_​vote_​rx
{vote_​rx_​result="bad_​encoding"}
counterInbound votes processed, by result (Vote had an invalid version, tag, or signature encoding)
votor_​vote_​rx
{vote_​rx_​result="unknown_​signer"}
counterInbound votes processed, by result (Sending connection had no authenticated identity)
votor_​vote_​rx
{vote_​rx_​result="not_​peer"}
counterInbound votes processed, by result (Sender was not in the peer set)
votor_​vote_​rx
{vote_​rx_​result="banned"}
counterInbound votes processed, by result (Sender is banned for a failed signature verification)
votor_​vote_​rx
{vote_​rx_​result="slot_​out_​of_​range"}
counterInbound votes processed, by result (Vote slot was either too old or too far in the future)
votor_​vote_​rx
{vote_​rx_​result="null_​block_​hash"}
counterInbound votes processed, by result (Vote block hash was null)
votor_​vote_​rx
{vote_​rx_​result="signature_​failed"}
counterInbound votes processed, by result (Vote signature was the identity point or outside the G2 subgroup)
votor_​vote_​rx
{vote_​rx_​result="unranked"}
counterInbound votes processed, by result (Sender was not a ranked validator in the vote slot's epoch)
votor_​vote_​rx
{vote_​rx_​result="shred_​version_​mismatched"}
counterInbound votes processed, by result (Vote was for a different shred version)
votor_​vote_​rx
{vote_​rx_​result="duplicate"}
counterInbound votes processed, by result (Voter had already cast this vote)
votor_​vote_​rx
{vote_​rx_​result="slashable"}
counterInbound votes processed, by result (Vote constitutes a slashable offence)
votor_​cert_​rx
{cert_​rx_​result="success"}
counterInbound certificates processed, by result (Cert was handed to the pool and accepted)
votor_​cert_​rx
{cert_​rx_​result="bad_​size"}
counterInbound certificates processed, by result (Cert was truncated or had trailing bytes)
votor_​cert_​rx
{cert_​rx_​result="bad_​encoding"}
counterInbound certificates processed, by result (Cert had an invalid version, tag, bitmap, or signature encoding)
votor_​cert_​rx
{cert_​rx_​result="shred_​version_​mismatched"}
counterInbound certificates processed, by result (Cert was for a different shred version)
votor_​cert_​rx
{cert_​rx_​result="unknown_​signer"}
counterInbound certificates processed, by result (Sending connection had no authenticated identity)
votor_​cert_​rx
{cert_​rx_​result="not_​peer"}
counterInbound certificates processed, by result (Sender was not in the peer set)
votor_​cert_​rx
{cert_​rx_​result="unranked"}
counterInbound certificates processed, by result (Sender was not a ranked validator in the cert slot's epoch)
votor_​cert_​rx
{cert_​rx_​result="slot_​out_​of_​range"}
counterInbound certificates processed, by result (Cert slot was either too old or too far in the future)
votor_​cert_​rx
{cert_​rx_​result="duplicate"}
counterInbound certificates processed, by result (Cert was already in the pool)
votor_​cert_​rx
{cert_​rx_​result="signature_​failed"}
counterInbound certificates processed, by result (Cert failed the stake threshold or the aggregate signature check)
votor_​cert_​rx
{cert_​rx_​result="banned"}
counterInbound certificates processed, by result (Sender is banned for a failed signature verification)
votor_​bls_​signature_​checkedcounterAggregate BLS signature checks of votes, certificates and reward certificates; a search for bad signers counts once
votor_​leader_​window_​supersededcounterLeader windows given up because certificates already let a later window start, eg. while catching up after a restart
votor_​cert_​tx_​unstaked_​skippedcounterCertificates and standstill votes not sent because we are not staked in their epoch
votor_​voting_​slot_​usedgaugeSlots tracked by the voting state machine
votor_​voting_​slot_​freegaugeFree entries in the voting state machine's slot table
votor_​highest_​final_​cert_​slotgaugeHighest slot with a final or fast-final cert seen by votor, 0 if none
votor_​pool_​slot_​usedgaugeSlots tracked by the vote and cert pool
votor_​pool_​slot_​freegaugeFree entries in the vote and cert pool's slot table
votor_​highest_​pool_​finalized_​slotgaugeHighest slot the pool has finalized (slow-final also needs the slot notarized), 0 if none
votor_​rankgaugeRank of this validator in the current epoch (0 is the highest rank), ULONG_MAX if unranked (unstaked)
votor_​client_​pkt_​rxcounterUDP datagrams received (one may carry several coalesced QUIC packets)
votor_​client_​pkt_​rx_​bytescounterBytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers
votor_​client_​pkt_​txcounterUDP datagrams sent
votor_​client_​pkt_​tx_​bytescounterBytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers
votor_​client_​conn_​used
{quic_​conn_​state="handshake"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Handshaking peer)
votor_​client_​conn_​used
{quic_​conn_​state="handshake_​complete"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Handshake complete, confirming with peer)
votor_​client_​conn_​used
{quic_​conn_​state="active"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Active connection)
votor_​client_​conn_​used
{quic_​conn_​state="peer_​close"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Peer requested close)
votor_​client_​conn_​used
{quic_​conn_​state="abort"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection terminating due to error)
votor_​client_​conn_​used
{quic_​conn_​state="close_​pending"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection is closing)
votor_​client_​conn_​used
{quic_​conn_​state="dead"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection about to be freed)
votor_​client_​conn_​freegaugeConnection slots free
votor_​client_​conn_​capacitygaugeConnection slots
votor_​client_​conn_​createdcounterConnections created
votor_​client_​conn_​closedcounterConnections gracefully closed
votor_​client_​conn_​abortedcounterConnections aborted
votor_​client_​conn_​timed_​outcounterConnections timed out
votor_​client_​conn_​no_​slot_​rejectedcounterConnections that failed to create due to lack of slots
votor_​client_​conn_​retry_​rejectedcounterRetries rejected: an invalid or expired token as server, a late or invalid Retry packet as client
votor_​client_​pkt_​decrypt_​failed
{quic_​encryption_​level="initial"}
counterPackets that failed decryption, by encryption level (Initial)
votor_​client_​pkt_​decrypt_​failed
{quic_​encryption_​level="early"}
counterPackets that failed decryption, by encryption level (Early data)
votor_​client_​pkt_​decrypt_​failed
{quic_​encryption_​level="handshake"}
counterPackets that failed decryption, by encryption level (Handshake)
votor_​client_​pkt_​decrypt_​failed
{quic_​encryption_​level="app"}
counterPackets that failed decryption, by encryption level (App data)
votor_​client_​pkt_​no_​key_​dropped
{quic_​encryption_​level="initial"}
counterPackets dropped because the decryption key was missing, by encryption level (Initial)
votor_​client_​pkt_​no_​key_​dropped
{quic_​encryption_​level="early"}
counterPackets dropped because the decryption key was missing, by encryption level (Early data)
votor_​client_​pkt_​no_​key_​dropped
{quic_​encryption_​level="handshake"}
counterPackets dropped because the decryption key was missing, by encryption level (Handshake)
votor_​client_​pkt_​no_​key_​dropped
{quic_​encryption_​level="app"}
counterPackets dropped because the decryption key was missing, by encryption level (App data)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="undersize"}
counterReceived packets dropped, by reason (QUIC packet too small)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="oversize"}
counterReceived packets dropped, by reason (QUIC packet too large)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="src_​ip_​mismatch"}
counterReceived packets dropped, by reason (Wrong source IP)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="header_​invalid"}
counterReceived packets dropped, by reason (Invalid QUIC header)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="net_​header_​invalid"}
counterReceived packets dropped, by reason (Invalid IPv4 or UDP header)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="version_​unsupported"}
counterReceived packets dropped, by reason (Version negotiation or unsupported version)
votor_​client_​pkt_​dropped
{quic_​pkt_​drop_​reason="duplicate"}
counterReceived packets dropped, by reason (Packet number already received or too old)
votor_​client_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="initial"}
counterPackets with an unknown connection ID, by packet type (Initial)
votor_​client_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="retry"}
counterPackets with an unknown connection ID, by packet type (Retry)
votor_​client_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="handshake"}
counterPackets with an unknown connection ID, by packet type (Handshake)
votor_​client_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="one_​rtt"}
counterPackets with an unknown connection ID, by packet type (1-RTT)
votor_​client_​pkt_​retransmitted
{quic_​encryption_​level="initial"}
counterQUIC packets retransmitted, by encryption level (Initial)
votor_​client_​pkt_​retransmitted
{quic_​encryption_​level="early"}
counterQUIC packets retransmitted, by encryption level (Early data)
votor_​client_​pkt_​retransmitted
{quic_​encryption_​level="handshake"}
counterQUIC packets retransmitted, by encryption level (Handshake)
votor_​client_​pkt_​retransmitted
{quic_​encryption_​level="app"}
counterQUIC packets retransmitted, by encryption level (App data)
votor_​client_​handshake_​createdcounterHandshake flows created
votor_​client_​handshake_​pool_​full_​rejectedcounterHandshakes dropped due to alloc fail
votor_​client_​handshake_​evictedcounterHandshakes dropped due to eviction
votor_​client_​frame_​rx
{quic_​frame_​type="unknown"}
counterQUIC frames received, by frame type (Unknown frame type)
votor_​client_​frame_​rx
{quic_​frame_​type="ack"}
counterQUIC frames received, by frame type (ACK frame)
votor_​client_​frame_​rx
{quic_​frame_​type="reset_​stream"}
counterQUIC frames received, by frame type (RESET_STREAM frame)
votor_​client_​frame_​rx
{quic_​frame_​type="stop_​sending"}
counterQUIC frames received, by frame type (STOP_SENDING frame)
votor_​client_​frame_​rx
{quic_​frame_​type="crypto"}
counterQUIC frames received, by frame type (CRYPTO frame)
votor_​client_​frame_​rx
{quic_​frame_​type="new_​token"}
counterQUIC frames received, by frame type (NEW_TOKEN frame)
votor_​client_​frame_​rx
{quic_​frame_​type="stream"}
counterQUIC frames received, by frame type (STREAM frame)
votor_​client_​frame_​rx
{quic_​frame_​type="max_​data"}
counterQUIC frames received, by frame type (MAX_DATA frame)
votor_​client_​frame_​rx
{quic_​frame_​type="max_​stream_​data"}
counterQUIC frames received, by frame type (MAX_STREAM_DATA frame)
votor_​client_​frame_​rx
{quic_​frame_​type="max_​streams"}
counterQUIC frames received, by frame type (MAX_STREAMS frame)
votor_​client_​frame_​rx
{quic_​frame_​type="data_​blocked"}
counterQUIC frames received, by frame type (DATA_BLOCKED frame)
votor_​client_​frame_​rx
{quic_​frame_​type="stream_​data_​blocked"}
counterQUIC frames received, by frame type (STREAM_DATA_BLOCKED frame)
votor_​client_​frame_​rx
{quic_​frame_​type="streams_​blocked"}
counterQUIC frames received, by frame type (STREAMS_BLOCKED(bidi) frame)
votor_​client_​frame_​rx
{quic_​frame_​type="new_​conn_​id"}
counterQUIC frames received, by frame type (NEW_CONN_ID frame)
votor_​client_​frame_​rx
{quic_​frame_​type="retire_​conn_​id"}
counterQUIC frames received, by frame type (RETIRE_CONN_ID frame)
votor_​client_​frame_​rx
{quic_​frame_​type="path_​challenge"}
counterQUIC frames received, by frame type (PATH_CHALLENGE frame)
votor_​client_​frame_​rx
{quic_​frame_​type="path_​response"}
counterQUIC frames received, by frame type (PATH_RESPONSE frame)
votor_​client_​frame_​rx
{quic_​frame_​type="conn_​close_​quic"}
counterQUIC frames received, by frame type (CONN_CLOSE(transport) frame)
votor_​client_​frame_​rx
{quic_​frame_​type="conn_​close_​app"}
counterQUIC frames received, by frame type (CONN_CLOSE(app) frame)
votor_​client_​frame_​rx
{quic_​frame_​type="handshake_​done"}
counterQUIC frames received, by frame type (HANDSHAKE_DONE frame)
votor_​client_​frame_​rx
{quic_​frame_​type="ping"}
counterQUIC frames received, by frame type (PING frame)
votor_​client_​frame_​rx
{quic_​frame_​type="padding"}
counterQUIC frames received, by frame type (PADDING frame)
votor_​client_​frame_​rx
{quic_​frame_​type="datagram"}
counterQUIC frames received, by frame type (DATAGRAM frame)
votor_​client_​frame_​parse_​failedcounterQUIC frames that failed to parse
votor_​client_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="success"}
counterAttempts to acquire packet metadata for transmission, by result (Success)
votor_​client_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="pool_​empty"}
counterAttempts to acquire packet metadata for transmission, by result (Packet metadata pool empty)
votor_​client_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="conn_​limit"}
counterAttempts to acquire packet metadata for transmission, by result (Connection reached its in-flight frame limit)
votor_​client_​pkt_​ack_​processed
{ack_​result="no_​pkt_​number"}
counterReceived packets processed for acknowledgement, by result (Packet had no packet number to acknowledge)
votor_​client_​pkt_​ack_​processed
{ack_​result="new_​range"}
counterReceived packets processed for acknowledgement, by result (New ACK range)
votor_​client_​pkt_​ack_​processed
{ack_​result="merged"}
counterReceived packets processed for acknowledgement, by result (Merged into existing ACK range)
votor_​client_​pkt_​ack_​processed
{ack_​result="queue_​full"}
counterReceived packets processed for acknowledgement, by result (Out of buffers)
votor_​client_​pkt_​ack_​processed
{ack_​result="cancelled"}
counterReceived packets processed for acknowledgement, by result (ACK suppressed by handler)
votor_​client_​service_​duration_​nanoshistogramDuration of one fd_quic_service call that processed a due connection timer
votor_​client_​pkt_​rx_​duration_​nanoshistogramDuration of processing one received packet
votor_​server_​pkt_​rxcounterUDP datagrams received (one may carry several coalesced QUIC packets)
votor_​server_​pkt_​rx_​bytescounterBytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers
votor_​server_​pkt_​txcounterUDP datagrams sent
votor_​server_​pkt_​tx_​bytescounterBytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers
votor_​server_​conn_​used
{quic_​conn_​state="handshake"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Handshaking peer)
votor_​server_​conn_​used
{quic_​conn_​state="handshake_​complete"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Handshake complete, confirming with peer)
votor_​server_​conn_​used
{quic_​conn_​state="active"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Active connection)
votor_​server_​conn_​used
{quic_​conn_​state="peer_​close"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Peer requested close)
votor_​server_​conn_​used
{quic_​conn_​state="abort"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection terminating due to error)
votor_​server_​conn_​used
{quic_​conn_​state="close_​pending"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection is closing)
votor_​server_​conn_​used
{quic_​conn_​state="dead"}
gaugeConnection slots in use, by state; sum + ConnFree = ConnCapacity (Connection about to be freed)
votor_​server_​conn_​freegaugeConnection slots free
votor_​server_​conn_​capacitygaugeConnection slots
votor_​server_​conn_​createdcounterConnections created
votor_​server_​conn_​closedcounterConnections gracefully closed
votor_​server_​conn_​abortedcounterConnections aborted
votor_​server_​conn_​timed_​outcounterConnections timed out
votor_​server_​conn_​no_​slot_​rejectedcounterConnections that failed to create due to lack of slots
votor_​server_​conn_​retry_​rejectedcounterRetries rejected: an invalid or expired token as server, a late or invalid Retry packet as client
votor_​server_​conn_​retry_​expired_​rejectedcounterInitials with an expired Retry token, also counted in ServerConnRetryRejected. Each gets an INVALID_TOKEN close so the client reconnects
votor_​server_​pkt_​decrypt_​failed
{quic_​encryption_​level="initial"}
counterPackets that failed decryption, by encryption level (Initial)
votor_​server_​pkt_​decrypt_​failed
{quic_​encryption_​level="early"}
counterPackets that failed decryption, by encryption level (Early data)
votor_​server_​pkt_​decrypt_​failed
{quic_​encryption_​level="handshake"}
counterPackets that failed decryption, by encryption level (Handshake)
votor_​server_​pkt_​decrypt_​failed
{quic_​encryption_​level="app"}
counterPackets that failed decryption, by encryption level (App data)
votor_​server_​pkt_​no_​key_​dropped
{quic_​encryption_​level="initial"}
counterPackets dropped because the decryption key was missing, by encryption level (Initial)
votor_​server_​pkt_​no_​key_​dropped
{quic_​encryption_​level="early"}
counterPackets dropped because the decryption key was missing, by encryption level (Early data)
votor_​server_​pkt_​no_​key_​dropped
{quic_​encryption_​level="handshake"}
counterPackets dropped because the decryption key was missing, by encryption level (Handshake)
votor_​server_​pkt_​no_​key_​dropped
{quic_​encryption_​level="app"}
counterPackets dropped because the decryption key was missing, by encryption level (App data)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="undersize"}
counterReceived packets dropped, by reason (QUIC packet too small)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="oversize"}
counterReceived packets dropped, by reason (QUIC packet too large)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="src_​ip_​mismatch"}
counterReceived packets dropped, by reason (Wrong source IP)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="header_​invalid"}
counterReceived packets dropped, by reason (Invalid QUIC header)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="net_​header_​invalid"}
counterReceived packets dropped, by reason (Invalid IPv4 or UDP header)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="version_​unsupported"}
counterReceived packets dropped, by reason (Version negotiation or unsupported version)
votor_​server_​pkt_​dropped
{quic_​pkt_​drop_​reason="duplicate"}
counterReceived packets dropped, by reason (Packet number already received or too old)
votor_​server_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="initial"}
counterPackets with an unknown connection ID, by packet type (Initial)
votor_​server_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="retry"}
counterPackets with an unknown connection ID, by packet type (Retry)
votor_​server_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="handshake"}
counterPackets with an unknown connection ID, by packet type (Handshake)
votor_​server_​pkt_​no_​conn_​dropped
{quic_​pkt_​type="one_​rtt"}
counterPackets with an unknown connection ID, by packet type (1-RTT)
votor_​server_​pkt_​retransmitted
{quic_​encryption_​level="initial"}
counterQUIC packets retransmitted, by encryption level (Initial)
votor_​server_​pkt_​retransmitted
{quic_​encryption_​level="early"}
counterQUIC packets retransmitted, by encryption level (Early data)
votor_​server_​pkt_​retransmitted
{quic_​encryption_​level="handshake"}
counterQUIC packets retransmitted, by encryption level (Handshake)
votor_​server_​pkt_​retransmitted
{quic_​encryption_​level="app"}
counterQUIC packets retransmitted, by encryption level (App data)
votor_​server_​handshake_​createdcounterHandshake flows created
votor_​server_​handshake_​pool_​full_​rejectedcounterHandshakes dropped due to alloc fail
votor_​server_​handshake_​evictedcounterHandshakes dropped due to eviction
votor_​server_​frame_​rx
{quic_​frame_​type="unknown"}
counterQUIC frames received, by frame type (Unknown frame type)
votor_​server_​frame_​rx
{quic_​frame_​type="ack"}
counterQUIC frames received, by frame type (ACK frame)
votor_​server_​frame_​rx
{quic_​frame_​type="reset_​stream"}
counterQUIC frames received, by frame type (RESET_STREAM frame)
votor_​server_​frame_​rx
{quic_​frame_​type="stop_​sending"}
counterQUIC frames received, by frame type (STOP_SENDING frame)
votor_​server_​frame_​rx
{quic_​frame_​type="crypto"}
counterQUIC frames received, by frame type (CRYPTO frame)
votor_​server_​frame_​rx
{quic_​frame_​type="new_​token"}
counterQUIC frames received, by frame type (NEW_TOKEN frame)
votor_​server_​frame_​rx
{quic_​frame_​type="stream"}
counterQUIC frames received, by frame type (STREAM frame)
votor_​server_​frame_​rx
{quic_​frame_​type="max_​data"}
counterQUIC frames received, by frame type (MAX_DATA frame)
votor_​server_​frame_​rx
{quic_​frame_​type="max_​stream_​data"}
counterQUIC frames received, by frame type (MAX_STREAM_DATA frame)
votor_​server_​frame_​rx
{quic_​frame_​type="max_​streams"}
counterQUIC frames received, by frame type (MAX_STREAMS frame)
votor_​server_​frame_​rx
{quic_​frame_​type="data_​blocked"}
counterQUIC frames received, by frame type (DATA_BLOCKED frame)
votor_​server_​frame_​rx
{quic_​frame_​type="stream_​data_​blocked"}
counterQUIC frames received, by frame type (STREAM_DATA_BLOCKED frame)
votor_​server_​frame_​rx
{quic_​frame_​type="streams_​blocked"}
counterQUIC frames received, by frame type (STREAMS_BLOCKED(bidi) frame)
votor_​server_​frame_​rx
{quic_​frame_​type="new_​conn_​id"}
counterQUIC frames received, by frame type (NEW_CONN_ID frame)
votor_​server_​frame_​rx
{quic_​frame_​type="retire_​conn_​id"}
counterQUIC frames received, by frame type (RETIRE_CONN_ID frame)
votor_​server_​frame_​rx
{quic_​frame_​type="path_​challenge"}
counterQUIC frames received, by frame type (PATH_CHALLENGE frame)
votor_​server_​frame_​rx
{quic_​frame_​type="path_​response"}
counterQUIC frames received, by frame type (PATH_RESPONSE frame)
votor_​server_​frame_​rx
{quic_​frame_​type="conn_​close_​quic"}
counterQUIC frames received, by frame type (CONN_CLOSE(transport) frame)
votor_​server_​frame_​rx
{quic_​frame_​type="conn_​close_​app"}
counterQUIC frames received, by frame type (CONN_CLOSE(app) frame)
votor_​server_​frame_​rx
{quic_​frame_​type="handshake_​done"}
counterQUIC frames received, by frame type (HANDSHAKE_DONE frame)
votor_​server_​frame_​rx
{quic_​frame_​type="ping"}
counterQUIC frames received, by frame type (PING frame)
votor_​server_​frame_​rx
{quic_​frame_​type="padding"}
counterQUIC frames received, by frame type (PADDING frame)
votor_​server_​frame_​rx
{quic_​frame_​type="datagram"}
counterQUIC frames received, by frame type (DATAGRAM frame)
votor_​server_​frame_​parse_​failedcounterQUIC frames that failed to parse
votor_​server_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="success"}
counterAttempts to acquire packet metadata for transmission, by result (Success)
votor_​server_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="pool_​empty"}
counterAttempts to acquire packet metadata for transmission, by result (Packet metadata pool empty)
votor_​server_​pkt_​meta_​acquire_​attempted
{pkt_​meta_​acquire_​result="conn_​limit"}
counterAttempts to acquire packet metadata for transmission, by result (Connection reached its in-flight frame limit)
votor_​server_​pkt_​ack_​processed
{ack_​result="no_​pkt_​number"}
counterReceived packets processed for acknowledgement, by result (Packet had no packet number to acknowledge)
votor_​server_​pkt_​ack_​processed
{ack_​result="new_​range"}
counterReceived packets processed for acknowledgement, by result (New ACK range)
votor_​server_​pkt_​ack_​processed
{ack_​result="merged"}
counterReceived packets processed for acknowledgement, by result (Merged into existing ACK range)
votor_​server_​pkt_​ack_​processed
{ack_​result="queue_​full"}
counterReceived packets processed for acknowledgement, by result (Out of buffers)
votor_​server_​pkt_​ack_​processed
{ack_​result="cancelled"}
counterReceived packets processed for acknowledgement, by result (ACK suppressed by handler)
votor_​server_​service_​duration_​nanoshistogramDuration of one fd_quic_service call that processed a due connection timer
votor_​server_​pkt_​rx_​duration_​nanoshistogramDuration of processing one received packet
votor_​vote_​history_​file_​writtencounterTimes the vote history file was written
votor_​wait_​to_​vote_​slotgaugeFirst slot votor may vote in, past the vote history file and the votes it sent

Motor Tile ​

MetricTypeDescription

Solcap Tile ​

MetricTypeDescription