Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via a Prometheus HTTP endpoint:
[tiles.metric]
prometheus_listen_port = 7999$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric representing a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
Durations are reported in integer nanoseconds (*_nanos), not seconds.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_frag_consumed | counter | Fragments consumed by the link reader |
| link_frag_consumed_bytes | counter | Bytes read by the link consumer |
| link_frag_filtered | counter | Fragments filtered and not consumed |
| link_frag_filtered_bytes | counter | Bytes read by the link consumer that were filtered |
| link_link_polling_overrun | counter | Times the link was overrun while polling |
| link_frag_polling_overrun | counter | Fragments not processed because the link was overrun while polling |
| link_link_reading_overrun | counter | Times the link was overrun while reading metadata |
| link_frag_reading_overrun | counter | Fragments not processed because the link was overrun while reading |
| link_backpressure_blamed | counter | Producer housekeeping intervals in which the producer was backpressured and this reliable consumer had the least credit across its outs; sampled, not an event count |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | Process ID of the tile |
| tile_tid | gauge | Thread ID of the tile. Same as Pid in production, but might differ in development |
| tile_last_cpu_index | gauge | CPU the tile last ran on |
| tile_context_switch_involuntary | counter | Involuntary context switches |
| tile_context_switch_voluntary | counter | Voluntary context switches |
| tile_page_fault_major | counter | Major page faults |
| tile_page_fault_minor | counter | Minor page faults |
| tile_state | gauge | Tile lifecycle state: 0=booting, 1=running, 2=shutdown |
| tile_heartbeat_time | gauge | Last wall-clock time the tile heartbeated |
| tile_is_backpressured | gauge | Whether the tile is currently backpressured |
| tile_backpressure | counter | Times the tile waited for one or more consumers to catch up to resume publishing |
| tile_regime_duration_nanos {tile_regime="caught_up_housekeeping"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="processing_housekeeping"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Processing + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="backpressure_housekeeping"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Backpressure + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="caught_up_prefrag"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Prefrag) |
| tile_regime_duration_nanos {tile_regime="processing_prefrag"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Processing + Prefrag) |
| tile_regime_duration_nanos {tile_regime="backpressure_prefrag"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Backpressure + Prefrag) |
| tile_regime_duration_nanos {tile_regime="caught_up_postfrag"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Postfrag) |
| tile_regime_duration_nanos {tile_regime="processing_postfrag"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Processing + Postfrag) |
| tile_regime_duration_nanos {tile_regime="caught_up_sleeping"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Caught up + Sleeping) |
| tile_regime_duration_nanos {tile_regime="backpressure_sleeping"} | counter | Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it (Backpressure + Sleeping) |
| tile_thread_duration_nanos {thread_regime="wait"} | counter | Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (Wait (task was runnable but not scheduled)) |
| tile_thread_duration_nanos {thread_regime="idle"} | counter | Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (Idle (task was not runnable)) |
| tile_thread_duration_nanos {thread_regime="user"} | counter | Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (User (task was scheduled and executing in user mode)) |
| tile_thread_duration_nanos {thread_regime="system"} | counter | Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (System (task was scheduled and executing in kernel mode)) |
| tile_thread_duration_nanos {thread_regime="interrupt"} | counter | Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU (Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)) |
| tile_park_attempted {park_result="ring"} | counter | Times the tile tried to park while idle (efficient mode), by result (Woken by a doorbell ring) |
| tile_park_attempted {park_result="deadline"} | counter | Times the tile tried to park while idle (efficient mode), by result (Deadline or park cap lapsed) |
| tile_park_attempted {park_result="pending"} | counter | Times the tile tried to park while idle (efficient mode), by result (Pre-park recheck found work, did not park) |
| tile_device_irq | counter | Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there |
| tile_tlb_shootdown | counter | TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there |
| tile_timer_irq | counter | Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there |
Genesi Tile
| Metric | Type | Description |
|---|
Ipecho Tile
| Metric | Type | Description |
|---|---|---|
| ipecho_shred_version | gauge | Shred version used by the validator |
| ipecho_conn_open | gauge | Open connections to the ipecho service |
| ipecho_conn_closed {conn_close_result="accepted"} | counter | Connections to the ipecho service made and closed, by outcome (Closed normally) |
| ipecho_conn_closed {conn_close_result="error"} | counter | Connections to the ipecho service made and closed, by outcome (Closed abnormally) |
| ipecho_conn_rx_bytes | counter | Bytes received on connections to the ipecho service |
| ipecho_conn_tx_bytes | counter | Bytes sent on connections to the ipecho service |
| ipecho_port_check_udp_tx | counter | Port check UDP packets sent to the joiner's UDP ports |
| ipecho_port_check_tcp_attempted {port_check_tcp_result="connected"} | counter | Port check TCP connections to the joiner's TCP ports, by result (Port check connections connected to the joiner's port) |
| ipecho_port_check_tcp_attempted {port_check_tcp_result="failed"} | counter | Port check TCP connections to the joiner's TCP ports, by result (Port check connections refused or errored) |
| ipecho_port_check_tcp_attempted {port_check_tcp_result="timed_out"} | counter | Port check TCP connections to the joiner's TCP ports, by result (Port check connections timed out) |
| ipecho_port_check_tcp_attempted {port_check_tcp_result="socket_failed"} | counter | Port check TCP connections to the joiner's TCP ports, by result (Port check connections not attempted because the socket could not be created) |
| ipecho_port_check_tcp_attempted {port_check_tcp_result="rejected_per_ip"} | counter | Port check TCP connections to the joiner's TCP ports, by result (Port check connections rejected because too many were in flight for a particular IP) |
| ipecho_port_check_tcp_attempted {port_check_tcp_result="evicted"} | counter | Port check TCP connections to the joiner's TCP ports, by result (Port check connections evicted because there were too many in flight connections) |
| ipecho_port_check_tcp_in_flight | gauge | Port check TCP connections in flight |
Snapct Tile
| Metric | Type | Description |
|---|---|---|
| snapct_load_state | gauge | Snapshot load controller state: 0=init (Initializing), 1=waiting_for_peers, 2=waiting_for_peers_incremental (Waiting for peers (incremental)), 3=collecting_peers, 4=collecting_peers_incremental (Collecting peers (incremental)), 5=reading_full_file (Reading full snapshot file), 6=flushing_full_file_fini (Full file read, draining downstream), 7=flushing_full_file_done (Full file loaded, draining downstream), 8=flushing_full_file_reset (Full file load aborted, resetting downstream), 9=reading_incremental_file (Reading incremental snapshot file), 10=flushing_incremental_file_fini (Incremental file read, draining downstream), 11=flushing_incremental_file_done (Incremental file loaded, draining downstream), 12=flushing_incremental_file_reset (Incremental file load aborted, resetting downstream), 13=reading_full_http (Downloading full snapshot), 14=flushing_full_http_fini (Full download read, draining downstream), 15=flushing_full_http_done (Full download loaded, draining downstream), 16=flushing_full_http_reset (Full download aborted, resetting downstream), 17=reading_incremental_http (Downloading incremental snapshot), 18=flushing_incremental_http_fini (Incremental download read, draining downstream), 19=flushing_incremental_http_done (Incremental download loaded, draining downstream), 20=flushing_incremental_http_reset (Incremental download aborted, resetting downstream), 21=shutdown (Snapshot load complete) |
| snapct_read_progress_bytes {snap_type="full"} | gauge | Snapshot file bytes read (download or local file) in the current attempt, by type; resets on retry (Full snapshot) |
| snapct_read_progress_bytes {snap_type="incremental"} | gauge | Snapshot file bytes read (download or local file) in the current attempt, by type; resets on retry (Incremental snapshot) |
| snapct_read_expected_bytes {snap_type="full"} | gauge | Size of each snapshot file being loaded; may change if the load restarts (Full snapshot) |
| snapct_read_expected_bytes {snap_type="incremental"} | gauge | Size of each snapshot file being loaded; may change if the load restarts (Incremental snapshot) |
| snapct_save_progress_bytes {snap_type="full"} | gauge | Bytes of each downloaded snapshot saved to the snapshot directory so far; resets if the load restarts (Full snapshot) |
| snapct_save_progress_bytes {snap_type="incremental"} | gauge | Bytes of each downloaded snapshot saved to the snapshot directory so far; resets if the load restarts (Incremental snapshot) |
| snapct_load_retried {snap_type="full"} | counter | Snapshot downloads retried, by snapshot type (Full snapshot) |
| snapct_load_retried {snap_type="incremental"} | counter | Snapshot downloads retried, by snapshot type (Incremental snapshot) |
| snapct_predicted_incremental_slot | gauge | Slot of the incremental snapshot replay is predicted to start from, 0 if none chosen; may change if the load restarts |
Snapld Tile
| Metric | Type | Description |
|---|---|---|
| snapld_stream_state | gauge | Snapshot loader state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown |
Snapdc Tile
| Metric | Type | Description |
|---|---|---|
| snapdc_stream_state | gauge | Snapshot decompressor state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown |
| snapdc_compressed_progress_bytes {snap_type="full"} | gauge | Compressed bytes of each snapshot read in the current attempt; sum across snapdc tiles (Full snapshot) |
| snapdc_compressed_progress_bytes {snap_type="incremental"} | gauge | Compressed bytes of each snapshot read in the current attempt; sum across snapdc tiles (Incremental snapshot) |
| snapdc_decompressed_progress_bytes {snap_type="full"} | gauge | Decompressed bytes of each snapshot produced in the current attempt; sum across snapdc tiles (Full snapshot) |
| snapdc_decompressed_progress_bytes {snap_type="incremental"} | gauge | Decompressed bytes of each snapshot produced in the current attempt; sum across snapdc tiles (Incremental snapshot) |
Snapin Tile
| Metric | Type | Description |
|---|---|---|
| snapin_stream_state | gauge | Snapshot inserter state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown |
| snapin_parse_progress_bytes {snap_type="full"} | gauge | Decompressed bytes of each snapshot this tile parsed in the current attempt; every snapin tile parses the whole stream, so take the min across tiles (Full snapshot) |
| snapin_parse_progress_bytes {snap_type="incremental"} | gauge | Decompressed bytes of each snapshot this tile parsed in the current attempt; every snapin tile parses the whole stream, so take the min across tiles (Incremental snapshot) |
| snapin_disk_written_bytes | counter | Bytes this tile has written to the accounts database file |
| snapin_account_progress {snapin_account_result="new"} | gauge | Accounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry (New account inserted) |
| snapin_account_progress {snapin_account_result="replaced"} | gauge | Accounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry (Replaced an older duplicate inserted earlier) |
| snapin_account_progress {snapin_account_result="ignored"} | gauge | Accounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry (Dropped because an inserted duplicate was newer) |
| snapin_account_processed | counter | Accounts processed across all snapshots |
| snapin_account_batch_processed | counter | Account batches processed across all snapshots |
Admin Tile
| Metric | Type | Description |
|---|
Netlnk Tile
| Metric | Type | Description |
|---|---|---|
| netlnk_rx_buffer_overflowed | counter | Netlink receive buffer overflows (ENOBUFS); each forces a full resync |
| netlnk_link_table_synced | counter | Full link table syncs |
| netlnk_route_table_synced | counter | Full route table syncs |
| netlnk_update_processed {netlink_message_type="link"} | counter | Netlink live updates processed, by message type (Link) |
| netlnk_update_processed {netlink_message_type="neighbor"} | counter | Netlink live updates processed, by message type (Neighbor Table Entry) |
| netlnk_update_processed {netlink_message_type="ipv4_route"} | counter | Netlink live updates processed, by message type (IPv4 Route Table Entry) |
| netlnk_interface_count | gauge | Network interfaces in the interface table |
| netlnk_neighbor_probe_requested {neighbor_probe_result="sent"} | counter | Neighbor solicit requests from net tiles, by result (Neighbor solicit sent to the kernel) |
| netlnk_neighbor_probe_requested {neighbor_probe_result="in_flight"} | counter | Neighbor solicit requests from net tiles, by result (Kernel already resolving this neighbor) |
| netlnk_neighbor_probe_requested {neighbor_probe_result="table_full"} | counter | Neighbor solicit requests from net tiles, by result (Neighbor table had no room for a placeholder entry) |
| netlnk_neighbor_probe_requested {neighbor_probe_result="send_failed"} | counter | Neighbor solicit requests from net tiles, by result (Netlink send failed) |
| netlnk_neighbor_probe_requested {neighbor_probe_result="host_rate_limited"} | counter | Neighbor solicit requests from net tiles, by result (Per-host probe rate limit hit) |
| netlnk_neighbor_probe_requested {neighbor_probe_result="global_rate_limited"} | counter | Neighbor solicit requests from net tiles, by result (Global probe rate limit hit) |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_pkt_rx | counter | Packets successfully published to output links |
| net_pkt_rx_bytes | counter | Bytes of packets successfully published to output links |
| net_pkt_rx_dropped {pkt_rx_drop_reason="malformed"} | counter | Received packets dropped, by reason (Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present) |
| net_pkt_rx_dropped {pkt_rx_drop_reason="dst_ip_mismatch"} | counter | Received packets dropped, by reason (Destination IP does not match the bind address) |
| net_pkt_rx_dropped {pkt_rx_drop_reason="dst_port_unbound"} | counter | Received packets dropped, by reason (No output link for the destination UDP port) |
| net_pkt_rx_dropped {pkt_rx_drop_reason="gre_invalid"} | counter | Received packets dropped, by reason (GRE tunnel peer or headers invalid) |
| net_pkt_rx_dropped {pkt_rx_drop_reason="gre_unconfigured"} | counter | Received packets dropped, by reason (GRE packet with no GRE tunnel configured) |
| net_rx_fill_ring_stalled | counter | Times RX stalled because the fill ring had no room; the packet stays in the RX ring and is retried (kernel drops are XdpRxRingFullDropped) |
| net_rx_buffer_used | gauge | Receive buffers currently in use |
| net_rx_buffer_free | gauge | Receive buffers currently free |
| net_pkt_tx_submitted | counter | Packet transmit jobs submitted |
| net_pkt_tx_completed | counter | Packet transmit jobs marked as completed by the kernel |
| net_pkt_tx_submitted_bytes | counter | Bytes of packets submitted for transmission |
| net_pkt_tx_dropped {pkt_tx_drop_reason="invalid"} | counter | Transmit jobs dropped, by reason (Invalid packet headers, or GRE packet exceeding the MTU) |
| net_pkt_tx_dropped {pkt_tx_drop_reason="source_ip_missing"} | counter | Transmit jobs dropped, by reason (No source IP address chosen) |
| net_pkt_tx_dropped {pkt_tx_drop_reason="ring_full"} | counter | Transmit jobs dropped, by reason (TX ring full or no TX buffer available) |
| net_pkt_tx_dropped {pkt_tx_drop_reason="gre_no_route"} | counter | Transmit jobs dropped, by reason (GRE tunnel addresses missing or outer route is itself tunnelled) |
| net_pkt_tx_dropped {pkt_tx_drop_reason="socket_send_failure"} | counter | Transmit jobs dropped, by reason (Socket send failed or failed to send the entire packet) |
| net_pkt_tx_routed {pkt_tx_route_result="fast_path"} | counter | Transmit jobs accepted after route resolution, by result (Userland routing succeeded and selected the configured provider) |
| net_pkt_tx_routed {pkt_tx_route_result="slow_path"} | counter | Transmit jobs accepted after route resolution, by result (Failed userland routing and selected the kernel fallback) |
| net_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="no_route"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (No matching route) |
| net_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="unsupported_route"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Unsupported route type) |
| net_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="interface_missing"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface not available) |
| net_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="unsupported_interface"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface type not supported) |
| net_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="no_neighbor"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Next-hop neighbor unresolved or inactive) |
| net_tx_buffer_used | gauge | Transmit buffers currently in use |
| net_tx_buffer_free | gauge | Transmit buffers currently free |
| net_xsk_syscall_tx | counter | XSK sendto syscalls dispatched; in busy-poll mode each also counts in XskSyscallRx |
| net_xsk_syscall_rx | counter | XSK recvmsg syscalls dispatched |
| net_xdp_rx_other_dropped | counter | Dropped for other reasons (xdp_statistics_v0.rx_dropped) |
| net_xdp_rx_invalid_descriptor_dropped | counter | Dropped due to invalid descriptor (xdp_statistics_v0.rx_invalid_descs) |
| net_xdp_tx_invalid_descriptor_dropped | counter | Dropped due to invalid descriptor (xdp_statistics_v0.tx_invalid_descs) |
| net_xdp_rx_ring_full_dropped | counter | Dropped due to rx ring being full (xdp_statistics_v1.rx_ring_full) |
| net_xdp_rx_fill_ring_empty | counter | Times the kernel found the fill ring empty (xdp_statistics_v1.rx_fill_ring_empty_descs) |
| net_xdp_tx_ring_empty | counter | Times the kernel found the TX ring empty (xdp_statistics_v1.tx_ring_empty_descs) |
| net_gre_pkt_rx | counter | GRE packets decapsulated and published; subset of PktRx |
| net_gre_pkt_tx_submitted | counter | GRE packet transmit jobs submitted; subset of PktTxSubmitted |
| net_pkt_tx_slow_path_submitted | counter | Packets submitted to a kernel socket on the slow path; subset of PktTxSubmitted |
| net_route_count {route_table_type="local"} | gauge | IPv4 routes installed in the forwarding table (Local) |
| net_route_count {route_table_type="main"} | gauge | IPv4 routes installed in the forwarding table (Main) |
| net_vlan_pkt_rx | counter | VLAN-tagged packets with the tag stripped in software and published; subset of PktRx |
Sock Tile
| Metric | Type | Description |
|---|---|---|
| sock_syscall_tx {sendmmsg_result="success"} | counter | sendmmsg syscalls dispatched, by result (Success) |
| sock_syscall_tx {sendmmsg_result="buffer_full"} | counter | sendmmsg syscalls dispatched, by result (ENOBUFS, EAGAIN error) |
| sock_syscall_tx {sendmmsg_result="permission"} | counter | sendmmsg syscalls dispatched, by result (EPERM error (blocked by netfilter)) |
| sock_syscall_tx {sendmmsg_result="unreachable"} | counter | sendmmsg syscalls dispatched, by result (ENETUNREACH, EHOSTUNREACH error) |
| sock_syscall_tx {sendmmsg_result="down"} | counter | sendmmsg syscalls dispatched, by result (ENONET, ENETDOWN, EHOSTDOWN error) |
| sock_syscall_tx {sendmmsg_result="other"} | counter | sendmmsg syscalls dispatched, by result (Unrecognized error code) |
| sock_syscall_rx | counter | recvmmsg syscalls dispatched |
| sock_pkt_rx | counter | Packets received |
| sock_pkt_tx | counter | Packets sent |
| sock_pkt_tx_failed | counter | Packets that failed to send |
| sock_pkt_tx_bytes | counter | Bytes of packets successfully sent |
| sock_pkt_rx_bytes | counter | UDP payload bytes received plus 42 bytes of synthesized Ethernet/IPv4/UDP headers per packet |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_txn_reassembly_evicted | counter | Txns evicted from reassembly before completing (txn_reassembly_count too small) |
| quic_txn_reassembly_started | counter | Fragmented txn receive ops started |
| quic_txn_reassembly_pending | gauge | Fragmented txn reassemblies in progress |
| quic_txn_reassembly_chunk_rx | counter | Txn stream chunks received for reassembly |
| quic_txn_reassembly_gap_dropped | counter | Txn stream chunks dropped because of a data gap |
| quic_txn_reassembly_duplicate_dropped | counter | Txn stream chunks dropped as duplicates of a completed stream |
| quic_txn_rx {tpu_rx_type="udp"} | counter | Transactions published to verify, by TPU ingress path; excludes all *Dropped (TPU/UDP) |
| quic_txn_rx {tpu_rx_type="quic_fast"} | counter | Transactions published to verify, by TPU ingress path; excludes all *Dropped (TPU/QUIC unfragmented) |
| quic_txn_rx {tpu_rx_type="quic_frag"} | counter | Transactions published to verify, by TPU ingress path; excludes all *Dropped (TPU/QUIC fragmented) |
| quic_txn_reassembly_abandoned | counter | Txn reassemblies abandoned because the connection was lost |
| quic_txn_undersize_dropped | counter | Txns received via QUIC dropped because they were too small |
| quic_txn_oversize_dropped | counter | Txns received via QUIC dropped because they were too large |
| quic_udp_txn_undersize_dropped | counter | Packets received on the non-QUIC port that were too small to be a valid IP packet |
| quic_udp_txn_oversize_dropped | counter | Packets received on the non-QUIC port that were too large to be a valid transaction |
| quic_pkt_rx | counter | UDP datagrams received (one may carry several coalesced QUIC packets) |
| quic_pkt_rx_bytes | counter | Bytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers |
| quic_pkt_tx | counter | UDP datagrams sent |
| quic_pkt_tx_bytes | counter | Bytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers |
| quic_conn_used {quic_conn_state="handshake"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Handshaking peer) |
| quic_conn_used {quic_conn_state="handshake_complete"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Handshake complete, confirming with peer) |
| quic_conn_used {quic_conn_state="active"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Active connection) |
| quic_conn_used {quic_conn_state="peer_close"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Peer requested close) |
| quic_conn_used {quic_conn_state="abort"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection terminating due to error) |
| quic_conn_used {quic_conn_state="close_pending"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection is closing) |
| quic_conn_used {quic_conn_state="dead"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection about to be freed) |
| quic_conn_free | gauge | Connection slots free |
| quic_conn_capacity | gauge | Connection slots |
| quic_conn_created | counter | Connections created |
| quic_conn_closed | counter | Connections gracefully closed |
| quic_conn_aborted | counter | Connections aborted |
| quic_conn_timed_out | counter | Connections timed out |
| quic_conn_retried | counter | Connections established with retry |
| quic_conn_no_slot_rejected | counter | Connections that failed to create due to lack of slots |
| quic_conn_retry_rejected | counter | Retries rejected: an invalid or expired token as server, a late or invalid Retry packet as client |
| quic_conn_retry_expired_rejected | counter | Initials with an expired Retry token, also counted in ConnRetryRejected. Each gets an INVALID_TOKEN close so the client reconnects |
| quic_pkt_dropped {quic_pkt_drop_reason="undersize"} | counter | Received packets dropped, by reason (QUIC packet too small) |
| quic_pkt_dropped {quic_pkt_drop_reason="oversize"} | counter | Received packets dropped, by reason (QUIC packet too large) |
| quic_pkt_dropped {quic_pkt_drop_reason="src_ip_mismatch"} | counter | Received packets dropped, by reason (Wrong source IP) |
| quic_pkt_dropped {quic_pkt_drop_reason="header_invalid"} | counter | Received packets dropped, by reason (Invalid QUIC header) |
| quic_pkt_dropped {quic_pkt_drop_reason="net_header_invalid"} | counter | Received packets dropped, by reason (Invalid IPv4 or UDP header) |
| quic_pkt_dropped {quic_pkt_drop_reason="version_unsupported"} | counter | Received packets dropped, by reason (Version negotiation or unsupported version) |
| quic_pkt_dropped {quic_pkt_drop_reason="duplicate"} | counter | Received packets dropped, by reason (Packet number already received or too old) |
| quic_pkt_no_conn_dropped {quic_pkt_type="initial"} | counter | Packets with an unknown connection ID, by packet type (Initial) |
| quic_pkt_no_conn_dropped {quic_pkt_type="retry"} | counter | Packets with an unknown connection ID, by packet type (Retry) |
| quic_pkt_no_conn_dropped {quic_pkt_type="handshake"} | counter | Packets with an unknown connection ID, by packet type (Handshake) |
| quic_pkt_no_conn_dropped {quic_pkt_type="one_rtt"} | counter | Packets with an unknown connection ID, by packet type (1-RTT) |
| quic_pkt_meta_acquire_attempted {pkt_meta_acquire_result="success"} | counter | Attempts to acquire packet metadata for transmission, by result (Success) |
| quic_pkt_meta_acquire_attempted {pkt_meta_acquire_result="pool_empty"} | counter | Attempts to acquire packet metadata for transmission, by result (Packet metadata pool empty) |
| quic_pkt_meta_acquire_attempted {pkt_meta_acquire_result="conn_limit"} | counter | Attempts to acquire packet metadata for transmission, by result (Connection reached its in-flight frame limit) |
| quic_initial_pkt_rx {initial_token_type="none"} | counter | Initial packets received, by token type (No token) |
| quic_initial_pkt_rx {initial_token_type="retry_token"} | counter | Initial packets received, by token type (fd_quic retry token) |
| quic_initial_pkt_rx {initial_token_type="other"} | counter | Initial packets received, by token type (Token of another length) |
| quic_handshake_created | counter | Handshake flows created |
| quic_handshake_pool_full_rejected | counter | Handshakes dropped due to alloc fail |
| quic_handshake_evicted | counter | Handshakes dropped due to eviction |
| quic_stream_rx | counter | Stream receive events |
| quic_stream_rx_bytes | counter | Stream payload bytes received |
| quic_frame_rx {quic_frame_type="unknown"} | counter | QUIC frames received, by frame type (Unknown frame type) |
| quic_frame_rx {quic_frame_type="ack"} | counter | QUIC frames received, by frame type (ACK frame) |
| quic_frame_rx {quic_frame_type="reset_stream"} | counter | QUIC frames received, by frame type (RESET_STREAM frame) |
| quic_frame_rx {quic_frame_type="stop_sending"} | counter | QUIC frames received, by frame type (STOP_SENDING frame) |
| quic_frame_rx {quic_frame_type="crypto"} | counter | QUIC frames received, by frame type (CRYPTO frame) |
| quic_frame_rx {quic_frame_type="new_token"} | counter | QUIC frames received, by frame type (NEW_TOKEN frame) |
| quic_frame_rx {quic_frame_type="stream"} | counter | QUIC frames received, by frame type (STREAM frame) |
| quic_frame_rx {quic_frame_type="max_data"} | counter | QUIC frames received, by frame type (MAX_DATA frame) |
| quic_frame_rx {quic_frame_type="max_stream_data"} | counter | QUIC frames received, by frame type (MAX_STREAM_DATA frame) |
| quic_frame_rx {quic_frame_type="max_streams"} | counter | QUIC frames received, by frame type (MAX_STREAMS frame) |
| quic_frame_rx {quic_frame_type="data_blocked"} | counter | QUIC frames received, by frame type (DATA_BLOCKED frame) |
| quic_frame_rx {quic_frame_type="stream_data_blocked"} | counter | QUIC frames received, by frame type (STREAM_DATA_BLOCKED frame) |
| quic_frame_rx {quic_frame_type="streams_blocked"} | counter | QUIC frames received, by frame type (STREAMS_BLOCKED(bidi) frame) |
| quic_frame_rx {quic_frame_type="new_conn_id"} | counter | QUIC frames received, by frame type (NEW_CONN_ID frame) |
| quic_frame_rx {quic_frame_type="retire_conn_id"} | counter | QUIC frames received, by frame type (RETIRE_CONN_ID frame) |
| quic_frame_rx {quic_frame_type="path_challenge"} | counter | QUIC frames received, by frame type (PATH_CHALLENGE frame) |
| quic_frame_rx {quic_frame_type="path_response"} | counter | QUIC frames received, by frame type (PATH_RESPONSE frame) |
| quic_frame_rx {quic_frame_type="conn_close_quic"} | counter | QUIC frames received, by frame type (CONN_CLOSE(transport) frame) |
| quic_frame_rx {quic_frame_type="conn_close_app"} | counter | QUIC frames received, by frame type (CONN_CLOSE(app) frame) |
| quic_frame_rx {quic_frame_type="handshake_done"} | counter | QUIC frames received, by frame type (HANDSHAKE_DONE frame) |
| quic_frame_rx {quic_frame_type="ping"} | counter | QUIC frames received, by frame type (PING frame) |
| quic_frame_rx {quic_frame_type="padding"} | counter | QUIC frames received, by frame type (PADDING frame) |
| quic_frame_rx {quic_frame_type="datagram"} | counter | QUIC frames received, by frame type (DATAGRAM frame) |
| quic_pkt_ack_processed {ack_result="no_pkt_number"} | counter | Received packets processed for acknowledgement, by result (Packet had no packet number to acknowledge) |
| quic_pkt_ack_processed {ack_result="new_range"} | counter | Received packets processed for acknowledgement, by result (New ACK range) |
| quic_pkt_ack_processed {ack_result="merged"} | counter | Received packets processed for acknowledgement, by result (Merged into existing ACK range) |
| quic_pkt_ack_processed {ack_result="queue_full"} | counter | Received packets processed for acknowledgement, by result (Out of buffers) |
| quic_pkt_ack_processed {ack_result="cancelled"} | counter | Received packets processed for acknowledgement, by result (ACK suppressed by handler) |
| quic_service_duration_nanos | histogram | Duration of one fd_quic_service call that processed a due connection timer |
| quic_pkt_rx_duration_nanos | histogram | Duration of processing one received packet |
| quic_frame_parse_failed | counter | QUIC frames that failed to parse |
| quic_pkt_decrypt_failed {quic_encryption_level="initial"} | counter | Packets that failed decryption, by encryption level (Initial) |
| quic_pkt_decrypt_failed {quic_encryption_level="early"} | counter | Packets that failed decryption, by encryption level (Early data) |
| quic_pkt_decrypt_failed {quic_encryption_level="handshake"} | counter | Packets that failed decryption, by encryption level (Handshake) |
| quic_pkt_decrypt_failed {quic_encryption_level="app"} | counter | Packets that failed decryption, by encryption level (App data) |
| quic_pkt_no_key_dropped {quic_encryption_level="initial"} | counter | Packets dropped because the decryption key was missing, by encryption level (Initial) |
| quic_pkt_no_key_dropped {quic_encryption_level="early"} | counter | Packets dropped because the decryption key was missing, by encryption level (Early data) |
| quic_pkt_no_key_dropped {quic_encryption_level="handshake"} | counter | Packets dropped because the decryption key was missing, by encryption level (Handshake) |
| quic_pkt_no_key_dropped {quic_encryption_level="app"} | counter | Packets dropped because the decryption key was missing, by encryption level (App data) |
| quic_retry_pkt_tx | counter | QUIC Retry packets sent |
| quic_pkt_retransmitted {quic_encryption_level="initial"} | counter | QUIC packets retransmitted, by encryption level (Initial) |
| quic_pkt_retransmitted {quic_encryption_level="early"} | counter | QUIC packets retransmitted, by encryption level (Early data) |
| quic_pkt_retransmitted {quic_encryption_level="handshake"} | counter | QUIC packets retransmitted, by encryption level (Handshake) |
| quic_pkt_retransmitted {quic_encryption_level="app"} | counter | QUIC packets retransmitted, by encryption level (App data) |
Bundle Tile
| Metric | Type | Description |
|---|---|---|
| bundle_txn_rx | counter | Transactions received, including transactions within bundles |
| bundle_standalone_txn_rx | counter | Transactions received outside of a bundle |
| bundle_protobuf_rx_bytes | counter | Bytes from received gRPC protobuf payloads |
| bundle_bundle_rx | counter | Bundles received |
| bundle_client_failed {bundle_client_failure_reason="decode"} | counter | Bundle client failures, by cause (Protobuf decode or encode failed) |
| bundle_client_failed {bundle_client_failure_reason="transport"} | counter | Bundle client failures, by cause (gRPC transport failed) |
| bundle_client_failed {bundle_client_failure_reason="no_fee_info"} | counter | Bundle client failures, by cause (Bundle dropped due to missing fee info) |
| bundle_heap_capacity_bytes | gauge | Workspace heap capacity |
| bundle_heap_free_bytes | gauge | Approximate free space in workspace |
| bundle_ping_acked | counter | HTTP/2 PINGs acknowledged by server |
| bundle_conn_state | gauge | Block engine connection state; Sleeping means closed on purpose because the next leader slot is far away: 0=disconnected, 1=connecting, 2=connected, 3=sleeping |
| bundle_last_rtt_nanos | gauge | Latest block engine HTTP/2 PING round trip, 0 if none since connect |
| bundle_rtt_smoothed_nanos | gauge | Smoothed block engine HTTP/2 PING round trip (RFC 9002), 0 if none since connect |
| bundle_rtt_deviation_nanos | gauge | Mean deviation of block engine HTTP/2 PING round trips (RFC 9002 rttvar), 0 if none since connect |
| bundle_message_rx_delay_nanos | histogram | Local wall clock minus server timestamp per block engine message; includes clock skew, negative clamped to 0 |
| bundle_txn_buffer_full_dropped | counter | Transactions from the block engine dropped because the verify tiles were too slow to keep up |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_txn_processed {verify_result="success"} | counter | Transactions processed, by result (Transaction verified successfully) |
| verify_txn_processed {verify_result="bundle_peer_failed"} | counter | Transactions processed, by result (Another transaction in the bundle failed) |
| verify_txn_processed {verify_result="parse_failed"} | counter | Transactions processed, by result (Transaction failed to parse) |
| verify_txn_processed {verify_result="duplicate"} | counter | Transactions processed, by result (Transaction was a duplicate) |
| verify_txn_processed {verify_result="signature_failed"} | counter | Transactions processed, by result (Transaction failed signature verification) |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_txn_processed {dedup_result="success"} | counter | Transactions processed, by result (Transaction was not a duplicate) |
| dedup_txn_processed {dedup_result="bundle_peer_failed"} | counter | Transactions processed, by result (Another transaction in the bundle failed) |
| dedup_txn_processed {dedup_result="duplicate"} | counter | Transactions processed, by result (Transaction was a duplicate) |
Resolv Tile
| Metric | Type | Description |
|---|---|---|
| resolv_txn_dropped {resolv_txn_drop_reason="no_bank"} | counter | Transactions dropped, by reason (No bank was available) |
| resolv_txn_dropped {resolv_txn_drop_reason="blockhash_expired"} | counter | Transactions dropped, by reason (Blockhash expired) |
| resolv_txn_dropped {resolv_txn_drop_reason="bundle_peer_failed"} | counter | Transactions dropped, by reason (Another transaction in the bundle failed) |
| resolv_stash_txn_inserted | counter | Transactions with an unknown blockhash added to the stash |
| resolv_stash_txn_removed {stash_remove_reason="published"} | counter | Stashed transactions removed, by reason (Published once its blockhash became known) |
| resolv_stash_txn_removed {stash_remove_reason="evicted"} | counter | Stashed transactions removed, by reason (Evicted as the oldest entry to make room for a new one) |
| resolv_stash_txn_removed {stash_remove_reason="failed"} | counter | Stashed transactions removed, by reason (Removed without publishing because LUT resolution failed or no bank was available) |
| resolv_txn_lut_resolve_attempted {lut_resolve_result="invalid_lookup_index"} | counter | Address lookup table resolution attempts, by result (The transaction referenced an index in a LUT that didn't exist) |
| resolv_txn_lut_resolve_attempted {lut_resolve_result="account_uninitialized"} | counter | Address lookup table resolution attempts, by result (The account referenced as a LUT hasn't been initialized) |
| resolv_txn_lut_resolve_attempted {lut_resolve_result="invalid_account_data"} | counter | Address lookup table resolution attempts, by result (The account referenced as a LUT couldn't be parsed) |
| resolv_txn_lut_resolve_attempted {lut_resolve_result="invalid_account_owner"} | counter | Address lookup table resolution attempts, by result (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolv_txn_lut_resolve_attempted {lut_resolve_result="account_not_found"} | counter | Address lookup table resolution attempts, by result (The account referenced as a LUT couldn't be found) |
| resolv_txn_lut_resolve_attempted {lut_resolve_result="success"} | counter | Address lookup table resolution attempts, by result (Resolved successfully) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le128_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le512_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le2_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le8_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le32_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le128_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le1_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| resolv_accdb_account_readonly_acquired {accdb_cache_class="le10_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le128_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (0-128 B) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le512_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (129-512 B) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le2_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le8_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (2-8 KiB) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le32_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (8-32 KiB) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le128_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (32-128 KiB) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le1_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB) |
| resolv_accdb_account_cache_missed {accdb_cache_class="le10_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (1-10 MiB) |
| resolv_accdb_account_waited | counter | Accounts that had to wait for another reader to finish loading them into the cache |
| resolv_accdb_disk_read_bytes | counter | Bytes read from the account database file |
| resolv_accdb_disk_read | counter | Reads from the account database file |
| resolv_accdb_copied_bytes | counter | Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_microblock_schedule_duration_nanos | histogram | Duration of scheduling one microblock |
| pack_microblock_schedule_empty_duration_nanos | histogram | Duration of discovering that no transaction is schedulable |
| pack_txn_insert_duration_nanos | histogram | Duration of inserting one transaction into the pending pool |
| pack_microblock_complete_duration_nanos | histogram | Duration of marking one microblock as complete |
| pack_txn_per_microblock | histogram | Transactions in a scheduled microblock, including both votes and non-votes |
| pack_txn_rx | counter | Transactions received from resolv, including bundle members |
| pack_txn_insert_attempted {pack_txn_insert_result="instr_too_many_accounts"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction has an instruction that references too many accounts) |
| pack_txn_insert_attempted {pack_txn_insert_result="nonce_conflict"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Bundle with two conflicting durable nonce transactions) |
| pack_txn_insert_attempted {pack_txn_insert_result="account_blocklisted"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction uses an account on the account blocklist) |
| pack_txn_insert_attempted {pack_txn_insert_result="bundle_blocklisted"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction uses an account on the bundle blocklist) |
| pack_txn_insert_attempted {pack_txn_insert_result="invalid_nonce"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction is an invalid durable nonce transaction) |
| pack_txn_insert_attempted {pack_txn_insert_result="write_sysvar"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction tries to write to a sysvar) |
| pack_txn_insert_attempted {pack_txn_insert_result="estimation_failed"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Estimating compute cost or fee failed) |
| pack_txn_insert_attempted {pack_txn_insert_result="duplicate_account"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction included an account address twice) |
| pack_txn_insert_attempted {pack_txn_insert_result="too_many_accounts"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction tried to load too many accounts) |
| pack_txn_insert_attempted {pack_txn_insert_result="too_many_cu"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction requests too many CUs) |
| pack_txn_insert_attempted {pack_txn_insert_result="expired"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction already expired) |
| pack_txn_insert_attempted {pack_txn_insert_result="lut_unsupported"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction loads accounts from a lookup table, which is not supported here) |
| pack_txn_insert_attempted {pack_txn_insert_result="unaffordable"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Fee payer's balance below transaction fee) |
| pack_txn_insert_attempted {pack_txn_insert_result="duplicate"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Pack aware of transaction with same signature) |
| pack_txn_insert_attempted {pack_txn_insert_result="nonce_priority_too_low"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce) |
| pack_txn_insert_attempted {pack_txn_insert_result="priority_too_low"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_txn_insert_attempted {pack_txn_insert_result="nonvote_add"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction added to pending transactions) |
| pack_txn_insert_attempted {pack_txn_insert_result="nonvote_replace"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Transaction replaced a lower priority transaction) |
| pack_txn_insert_attempted {pack_txn_insert_result="nonce_nonvote_add"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Durable nonce transaction added to pending transactions) |
| pack_txn_insert_attempted {pack_txn_insert_result="nonce_nonvote_replace"} | counter | Insert attempts, by result; bundle members are counted individually with the bundle's result (Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="none"} | counter | Time pack spent, by scheduling regime (No microblock: not leader, no transactions, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn"} | counter | Time pack spent, by scheduling regime (No microblock: not leader, transactions available, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="execle"} | counter | Time pack spent, by scheduling regime (No microblock: not leader, no transactions, execle available) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_execle"} | counter | Time pack spent, by scheduling regime (No microblock: not leader, transactions available, execle available) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="leader"} | counter | Time pack spent, by scheduling regime (No microblock: leader, no transactions, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_leader"} | counter | Time pack spent, by scheduling regime (No microblock: leader, transactions available, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="execle_leader"} | counter | Time pack spent, by scheduling regime (No microblock: leader, no transactions, execle available) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_execle_leader"} | counter | Time pack spent, by scheduling regime (No microblock: leader, transactions available, execle available (all conflicting)) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: not leader, no transactions, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: not leader, transactions available, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="execle_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: not leader, no transactions, execle available) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_execle_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: not leader, transactions available, execle available) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="leader_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: leader, no transactions, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_leader_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: leader, transactions available, no available execle) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="execle_leader_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: leader, no transactions, execle available) |
| pack_schedule_regime_duration_nanos {pack_schedule_regime="txn_execle_leader_microblock"} | counter | Time pack spent, by scheduling regime (Scheduled a microblock: leader, transactions available, execle available) |
| pack_txn_dropped {pack_txn_drop_reason="expired"} | counter | Transactions dropped from pack after insertion, by reason (TTL expired) |
| pack_txn_dropped {pack_txn_drop_reason="already_executed"} | counter | Transactions dropped from pack after insertion, by reason (Already executed in the replay or leader pipeline) |
| pack_txn_dropped {pack_txn_drop_reason="partial_bundle"} | counter | Transactions dropped from pack after insertion, by reason (Part of a partial bundle) |
| pack_txn_evicted | counter | Transactions evicted from pack to make room for others, including the bundle initializer transaction removed when a leader slot ends |
| pack_txn_pending_used {pending_txn_type="regular"} | gauge | Pending transactions in pack's pool, by type (Transactions in the main treap) |
| pack_txn_pending_used {pending_txn_type="vote"} | gauge | Pending transactions in pack's pool, by type (Deprecated, always 0 since Firedancer v26.11.) |
| pack_txn_pending_used {pending_txn_type="conflicting"} | gauge | Pending transactions in pack's pool, by type (Transactions that write to a hotly-contended account) |
| pack_txn_pending_used {pending_txn_type="bundle"} | gauge | Pending transactions in pack's pool, by type (Transactions that are part of a bundle) |
| pack_txn_pending_used {pending_txn_type="next_auction"} | gauge | Pending transactions in pack's pool, by type (Transactions that are currently held waiting for the next auction) |
| pack_txn_pending_capacity | gauge | Maximum pending transactions pack can hold |
| pack_txn_pending_smallest_cost | gauge | Lower bound on the cost of the cheapest pending transaction, 0 if none pending |
| pack_block_microblock_limit_reached | counter | Times pack did not schedule a microblock because the block microblock limit was reached |
| pack_block_byte_limit_reached | counter | Times pack did not schedule a microblock because the block byte limit was reached |
| pack_txn_schedule_attempted {pack_txn_schedule_result="taken"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack included the transaction in the microblock) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="cu_limit"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="conflict_fast"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because of an account conflict found by the fast bitvector check) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="byte_limit"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="alloc_limit"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have exceeded the data allocation limit) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="write_cost"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="conflict_slow"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because of an account conflict found by the full check) |
| pack_txn_schedule_attempted {pack_txn_schedule_result="write_cost_deferred"} | counter | Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times (Pack skipped the transaction because it exceeded the per-account block write cost limit too many times before) |
| pack_bundle_crank_attempted {bundle_crank_result="up_to_date"} | counter | Times pack checked whether the bundle tip state needed cranking, by outcome (On-chain state was already up to date) |
| pack_bundle_crank_attempted {bundle_crank_result="inserted"} | counter | Times pack checked whether the bundle tip state needed cranking, by outcome (Inserted an initializer bundle to update the on-chain state) |
| pack_bundle_crank_attempted {bundle_crank_result="creation_failed"} | counter | Times pack checked whether the bundle tip state needed cranking, by outcome (Tried to insert an initializer bundle to update the on-chain state, but creation failed) |
| pack_bundle_crank_attempted {bundle_crank_result="insertion_failed"} | counter | Times pack checked whether the bundle tip state needed cranking, by outcome (Tried to insert an initializer bundle to update the on-chain state, but insertion failed) |
| pack_pre_rotation_bundle_pending | gauge | Bundles that arrived while not leader that pack is holding |
| pack_pre_rotation_bundle_inserted | counter | Bundles that arrived while not leader that pack kept |
| pack_pre_rotation_bundle_rejected | counter | Bundles that arrived while not leader that pack dropped because it was full of ones that declare at least as large a tip |
| pack_pre_rotation_bundle_evicted | counter | Bundles that arrived while not leader that pack dropped for one that declares a larger tip |
| pack_pre_rotation_bundle_scheduled | counter | Bundles that arrived while not leader that pack scheduled |
| pack_auction_started | counter | Batch auctions started |
| pack_auction_cancelled | counter | Batch auctions skipped because the block was already at their CU limit |
| pack_auction_ended {pack_auction_end_reason="limit"} | counter | Batch auctions ended, by reason (An auction ended because the block reached the auction's CU limit) |
| pack_auction_ended {pack_auction_end_reason="restarted"} | counter | Batch auctions ended, by reason (An auction ended because a new auction was started) |
| pack_auction_ended {pack_auction_end_reason="final"} | counter | Batch auctions ended, by reason (An auction ended because the final 10ms of our leader rotation began) |
| pack_auction_ended {pack_auction_end_reason="slot_end"} | counter | Batch auctions ended, by reason (An auction ended because the leader slot ended) |
| pack_auction_duration_nanos | histogram | Duration that batch auctions ran for |
| pack_block_cost | gauge | Cost units used in the current block, 0 if not packing a block |
| pack_cost_scheduled_per_block | histogram | Cost units scheduled for each block pack produced; can exceed the block limit because of rebates |
| pack_cu_rebated_per_block | histogram | Compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses |
| pack_cost_net_per_block | histogram | Net cost units (scheduled minus rebated) in each block pack produced |
| pack_cost_per_block_pct | histogram | Percent of max_cost_per_block used by each block pack produced |
Execle Tile
| Metric | Type | Description |
|---|---|---|
| execle_txn_processed {txn_result="success"} | counter | Transactions loaded and executed, by result (Transaction executed successfully) |
| execle_txn_processed {txn_result="instruction_error"} | counter | Transactions loaded and executed, by result (An error occurred while processing an instruction) |
| execle_txn_processed {txn_result="account_not_found"} | counter | Transactions loaded and executed, by result (The transaction fee payer address was not found) |
| execle_txn_processed {txn_result="program_account_not_found"} | counter | Transactions loaded and executed, by result (A program account referenced by the transaction was not found) |
| execle_txn_processed {txn_result="insufficient_funds_for_fee"} | counter | Transactions loaded and executed, by result (The transaction fee payer did not have balance to pay the fee) |
| execle_txn_processed {txn_result="invalid_account_for_fee"} | counter | Transactions loaded and executed, by result (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) |
| execle_txn_processed {txn_result="already_processed"} | counter | Transactions loaded and executed, by result (The transaction has already been processed in a recent block) |
| execle_txn_processed {txn_result="blockhash_not_found"} | counter | Transactions loaded and executed, by result (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) |
| execle_txn_processed {txn_result="invalid_program_for_execution"} | counter | Transactions loaded and executed, by result (A program account referenced by the transaction was no executable.) |
| execle_txn_processed {txn_result="address_lookup_table_not_found"} | counter | Transactions loaded and executed, by result (The transaction references an ALUT account that does not exist or is inactive) |
| execle_txn_processed {txn_result="invalid_address_lookup_table_owner"} | counter | Transactions loaded and executed, by result (The transaction references an ALUT account that is not owned by the ALUT program account) |
| execle_txn_processed {txn_result="invalid_address_lookup_table_data"} | counter | Transactions loaded and executed, by result (The transaction references an ALUT account that contains data which is not a valid ALUT) |
| execle_txn_processed {txn_result="invalid_address_lookup_table_index"} | counter | Transactions loaded and executed, by result (The transaction references an account offset from the ALUT which does not exist) |
| execle_txn_processed {txn_result="max_loaded_accounts_data_size_exceeded"} | counter | Transactions loaded and executed, by result (The total account data size of the loaded accounts exceeds the consensus limit) |
| execle_txn_processed {txn_result="duplicate_instruction"} | counter | Transactions loaded and executed, by result (A compute budget program instruction was invoked more than once) |
| execle_txn_processed {txn_result="invalid_loaded_accounts_data_size_limit"} | counter | Transactions loaded and executed, by result (The compute budget program was invoked and set the loaded accounts data size to zero) |
| execle_txn_processed {txn_result="nonce_already_advanced"} | counter | Transactions loaded and executed, by result (The transaction references a nonce account that is already advanced) |
| execle_txn_processed {txn_result="nonce_advance_failed"} | counter | Transactions loaded and executed, by result (The transaction is a nonce transaction but the advance instruction was not valid or failed) |
| execle_txn_processed {txn_result="nonce_wrong_blockhash"} | counter | Transactions loaded and executed, by result (The transaction is a nonce transaction but the blockhash is not the correct one) |
| execle_txn_processed {txn_result="unsupported_version"} | counter | Transactions loaded and executed, by result (The transaction is a message version that is not supported by the current feature set) |
| execle_txn_processed {txn_result="sanitize_failure"} | counter | Transactions loaded and executed, by result (The process for sanitizing the transaction failed) |
| execle_txn_processed {txn_result="account_in_use"} | counter | Transactions loaded and executed, by result (The transaction conflicts with another transaction in the microblock.) |
| execle_txn_processed {txn_result="account_loaded_twice"} | counter | Transactions loaded and executed, by result (The transaction references the same account twice) |
| execle_txn_processed {txn_result="signature_failure"} | counter | Transactions loaded and executed, by result (The transaction had an invalid signature) |
| execle_txn_processed {txn_result="too_many_account_locks"} | counter | Transactions loaded and executed, by result (The transaction references too many accounts.) |
| execle_txn_processed {txn_result="insufficient_funds_for_rent"} | counter | Transactions loaded and executed, by result (The transaction would leave an account with a lower balance than the rent-exempt minimum) |
| execle_txn_processed {txn_result="unbalanced_transaction"} | counter | Transactions loaded and executed, by result (The total referenced account lamports before and after the transaction was unbalanced) |
| execle_txn_processed {txn_result="bundle_peer"} | counter | Transactions loaded and executed, by result (The transaction was part of a bundle and an earlier transaction in the bundle failed) |
| execle_txn_processed_by_landing {txn_landing_result="success"} | counter | Transactions processed, by whether and how they landed in the block (Transaction landed and executed successfully) |
| execle_txn_processed_by_landing {txn_landing_result="fees_only"} | counter | Transactions processed, by whether and how they landed in the block (Transaction landed, but was fees only and did not execute) |
| execle_txn_processed_by_landing {txn_landing_result="failed"} | counter | Transactions processed, by whether and how they landed in the block (Transaction landed, but failed to execute) |
| execle_txn_processed_by_landing {txn_landing_result="dropped"} | counter | Transactions processed, by whether and how they landed in the block (Transaction did not land) |
| execle_txn_processed_by_version {txn_version="legacy"} | counter | Transactions processed, by transaction version (Legacy transaction format) |
| execle_txn_processed_by_version {txn_version="v0"} | counter | Transactions processed, by transaction version (Version 0 transaction format) |
| execle_txn_processed_by_version {txn_version="v1"} | counter | Transactions processed, by transaction version (Version 1 transaction format) |
| execle_instruction_executed | counter | Top-level instructions executed |
| execle_cpi_executed | counter | Cross-program invocations executed |
| execle_account_lthash_skipped | counter | Committed writable accounts left byte-identical by their transaction, so their in-band lthash update was skipped |
| execle_bank_switched | counter | Cache invalidations due to transaction execution from a different block |
| execle_cu_consumed | counter | Compute units consumed by executed transactions |
| execle_txn_regime_duration_nanos {txn_regime="load"} | counter | Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction checks and account loading) |
| execle_txn_regime_duration_nanos {txn_regime="execute"} | counter | Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction execution, including VM setup) |
| execle_txn_regime_duration_nanos {txn_regime="commit"} | counter | Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction result commit) |
| execle_vm_regime_duration_nanos {vm_regime="setup"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup) |
| execle_vm_regime_duration_nanos {vm_regime="commit"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit) |
| execle_vm_regime_duration_nanos {vm_regime="setup_cpi"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup (CPI)) |
| execle_vm_regime_duration_nanos {vm_regime="commit_cpi"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit (CPI)) |
| execle_vm_regime_duration_nanos {vm_regime="interpreter"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM interpreter execution) |
| execle_progcache_looked_up | counter | Program cache lookups; equals Hit + Miss + lookups of invalid program accounts |
| execle_progcache_miss | counter | Program cache misses, each followed by an insert or a spill |
| execle_progcache_waited | counter | Program cache lookups that waited on another tile's in-flight load of the same program |
| execle_progcache_insert_retried | counter | Program cache inserts that found no free slot in their size class on the first try and retried after eviction |
| execle_progcache_inserted_bytes | counter | Bytes inserted into the program cache |
| execle_progcache_spilled_bytes | counter | Bytes loaded into program cache spill memory |
| execle_progcache_evicted_bytes | counter | Bytes evicted from the program cache by insert eviction sweeps |
| execle_progcache_duration_nanos | counter | Time in program cache lookups, including waits on peer loads and insert, eviction and ELF load on a miss; superset of ProgcacheLoadDurationNanos |
| execle_progcache_loaded | counter | Program ELF loads and verifications for the program cache, including failed verifications and spill-backed loads |
| execle_progcache_load_duration_nanos | counter | Time spent loading programs |
| execle_progcache_hit {progcache_class="le128_kib"} | counter | Program cache hits, by size class (0-128 KiB) |
| execle_progcache_hit {progcache_class="le512_kib"} | counter | Program cache hits, by size class (128-512 KiB) |
| execle_progcache_hit {progcache_class="le1_mib"} | counter | Program cache hits, by size class (512 KiB-1 MiB) |
| execle_progcache_hit {progcache_class="le2_mib"} | counter | Program cache hits, by size class (1-2 MiB) |
| execle_progcache_hit {progcache_class="le4_mib"} | counter | Program cache hits, by size class (2-4 MiB) |
| execle_progcache_hit {progcache_class="le11_mib"} | counter | Program cache hits, by size class (4-11 MiB) |
| execle_progcache_inserted {progcache_class="le128_kib"} | counter | Programs inserted into the program cache, by size class (0-128 KiB) |
| execle_progcache_inserted {progcache_class="le512_kib"} | counter | Programs inserted into the program cache, by size class (128-512 KiB) |
| execle_progcache_inserted {progcache_class="le1_mib"} | counter | Programs inserted into the program cache, by size class (512 KiB-1 MiB) |
| execle_progcache_inserted {progcache_class="le2_mib"} | counter | Programs inserted into the program cache, by size class (1-2 MiB) |
| execle_progcache_inserted {progcache_class="le4_mib"} | counter | Programs inserted into the program cache, by size class (2-4 MiB) |
| execle_progcache_inserted {progcache_class="le11_mib"} | counter | Programs inserted into the program cache, by size class (4-11 MiB) |
| execle_progcache_evicted {progcache_class="le128_kib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (0-128 KiB) |
| execle_progcache_evicted {progcache_class="le512_kib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (128-512 KiB) |
| execle_progcache_evicted {progcache_class="le1_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (512 KiB-1 MiB) |
| execle_progcache_evicted {progcache_class="le2_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (1-2 MiB) |
| execle_progcache_evicted {progcache_class="le4_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (2-4 MiB) |
| execle_progcache_evicted {progcache_class="le11_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (4-11 MiB) |
| execle_progcache_spilled {progcache_class="le128_kib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (0-128 KiB) |
| execle_progcache_spilled {progcache_class="le512_kib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (128-512 KiB) |
| execle_progcache_spilled {progcache_class="le1_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (512 KiB-1 MiB) |
| execle_progcache_spilled {progcache_class="le2_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (1-2 MiB) |
| execle_progcache_spilled {progcache_class="le4_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (2-4 MiB) |
| execle_progcache_spilled {progcache_class="le11_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (4-11 MiB) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le128_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le512_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le2_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le8_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le32_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le128_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le1_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| execle_accdb_account_readonly_acquired {accdb_cache_class="le10_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le128_b"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le512_b"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le2_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le8_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le32_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le128_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le1_mib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="le10_mib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| execle_accdb_account_evicted {accdb_cache_class="le128_b"} | counter | Accounts evicted from the account database cache to make space, by cache size class (0-128 B) |
| execle_accdb_account_evicted {accdb_cache_class="le512_b"} | counter | Accounts evicted from the account database cache to make space, by cache size class (129-512 B) |
| execle_accdb_account_evicted {accdb_cache_class="le2_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (513 B-2 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="le8_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (2-8 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="le32_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (8-32 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="le128_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (32-128 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="le1_mib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (128 KiB-1 MiB) |
| execle_accdb_account_evicted {accdb_cache_class="le10_mib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (1-10 MiB) |
| execle_accdb_account_new_committed {accdb_cache_class="le128_b"} | counter | New account versions committed to the index, by destination cache size class (0-128 B) |
| execle_accdb_account_new_committed {accdb_cache_class="le512_b"} | counter | New account versions committed to the index, by destination cache size class (129-512 B) |
| execle_accdb_account_new_committed {accdb_cache_class="le2_kib"} | counter | New account versions committed to the index, by destination cache size class (513 B-2 KiB) |
| execle_accdb_account_new_committed {accdb_cache_class="le8_kib"} | counter | New account versions committed to the index, by destination cache size class (2-8 KiB) |
| execle_accdb_account_new_committed {accdb_cache_class="le32_kib"} | counter | New account versions committed to the index, by destination cache size class (8-32 KiB) |
| execle_accdb_account_new_committed {accdb_cache_class="le128_kib"} | counter | New account versions committed to the index, by destination cache size class (32-128 KiB) |
| execle_accdb_account_new_committed {accdb_cache_class="le1_mib"} | counter | New account versions committed to the index, by destination cache size class (128 KiB-1 MiB) |
| execle_accdb_account_new_committed {accdb_cache_class="le10_mib"} | counter | New account versions committed to the index, by destination cache size class (1-10 MiB) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le128_b"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (0-128 B) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le512_b"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (129-512 B) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le2_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (513 B-2 KiB) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le8_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (2-8 KiB) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le32_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (8-32 KiB) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le128_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (32-128 KiB) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le1_mib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (128 KiB-1 MiB) |
| execle_accdb_account_overwrite_committed {accdb_cache_class="le10_mib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (1-10 MiB) |
| execle_accdb_account_cache_missed {accdb_cache_class="le128_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (0-128 B) |
| execle_accdb_account_cache_missed {accdb_cache_class="le512_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (129-512 B) |
| execle_accdb_account_cache_missed {accdb_cache_class="le2_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB) |
| execle_accdb_account_cache_missed {accdb_cache_class="le8_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (2-8 KiB) |
| execle_accdb_account_cache_missed {accdb_cache_class="le32_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (8-32 KiB) |
| execle_accdb_account_cache_missed {accdb_cache_class="le128_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (32-128 KiB) |
| execle_accdb_account_cache_missed {accdb_cache_class="le1_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB) |
| execle_accdb_account_cache_missed {accdb_cache_class="le10_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (1-10 MiB) |
| execle_accdb_account_waited | counter | Accounts that had to wait for another reader to finish loading them into the cache |
| execle_accdb_batch_acquired | counter | Batch acquire calls made by this tile |
| execle_accdb_cache_reserve_retried | counter | Spin retries while waiting to reserve cache lines for a transaction |
| execle_accdb_disk_read_bytes | counter | Bytes read from the account database file |
| execle_accdb_disk_read | counter | Reads from the account database file |
| execle_accdb_disk_written_bytes | counter | Bytes written to the account database file |
| execle_accdb_disk_written | counter | Writes to the account database file |
| execle_accdb_copied_bytes | counter | Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded |
| execle_txncache_query_miss | counter | Transaction cache queries by this tile that found a page not in memory |
| execle_txncache_insert_miss | counter | Transaction cache inserts by this tile that found a page not in memory |
| execle_txncache_operation_stalled | counter | Transaction cache queries and inserts by this tile that fell back to the exclusive lock, stalling every tile; pages they load are read while holding it |
| execle_txncache_query_duration_nanos | histogram | Duration of one transaction cache query |
| execle_txncache_insert_duration_nanos | histogram | Duration of one transaction cache insert |
Sign Tile
| Metric | Type | Description |
|---|---|---|
| sign_sign_duration_nanos | histogram | Duration of signing a message |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_batch_abandoned | counter | Microblock batches abandoned because we switched slots without finishing the current slot |
| shred_parent_block_id_zeroed | counter | Leader blocks whose parent block_id was unknown, so the chained merkle root was zeroed |
| shred_batch_size_bytes | histogram | Size of each microblock batch that is shredded |
| shred_microblock_per_batch | histogram | Microblocks in each microblock batch that is shredded |
| shred_batch_shred_duration_nanos | histogram | Duration of shredding one microblock batch |
| shred_add_shred_duration_nanos | histogram | Duration of verifying and processing one shred received from the network |
| shred_store_fec_sync_spill_duration_nanos | histogram | Duration of synchronously spilling one FEC payload to the store |
| shred_processed {shred_process_result="own_leader_echoed"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Turbine shred for a slot we led, echoed back to us by a relay) |
| shred_processed {shred_process_result="leader_unknown"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was for a slot whose leader is unknown) |
| shred_processed {shred_process_result="parse_failed"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred parsing failed) |
| shred_processed {shred_process_result="equivocated"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was equivocated with another shred) |
| shred_processed {shred_process_result="rejected"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was invalid for one of many reasons) |
| shred_processed {shred_process_result="ignored"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was ignored because we had already reconstructed the FEC set, or was for an old slot) |
| shred_processed {shred_process_result="duplicate"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred was a duplicate of a previously received valid shred of an incomplete FEC set) |
| shred_processed {shred_process_result="accepted"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred accepted into an incomplete FEC set) |
| shred_processed {shred_process_result="completed"} | counter | Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_incomplete_evicted | counter | Incomplete FEC sets evicted from the resolver to make room for new ones |
| shred_initial_rejected | counter | Shreds rejected before any resources were allocated for the FEC set |
| shred_unchained_rejected | counter | Shreds rejected because they're not chained merkle shreds |
| shred_fec_fatal_rejected | counter | FEC sets rejected for reasons that cause the whole FEC set to become invalid |
| shred_repair_nonce_failed | counter | Shreds that looked like repair responses but had an incorrect nonce |
| shred_repair_rx | counter | Repair shreds received |
| shred_repair_rx_bytes | counter | Bytes received from network packets with repair shreds, including network headers |
| shred_turbine_rx | counter | Turbine shreds received |
| shred_turbine_rx_bytes | counter | Bytes received from network packets with turbine shreds, including network headers |
| shred_repair_completion_assisted | counter | FEC sets that completed with at least one repair shred contributing |
| shred_repair_lead_delay_nanos | histogram | Estimated time repair advanced FEC set completion: the delay from a repair-assisted completion until turbine had delivered as many shreds for that set as repair contributed. Counts arrivals rather than distinct indices, so turbine duplicates bias this low |
| shred_store_fec_sync_spilled | counter | FEC payloads the shred tile spilled to the store synchronously because the store RAM cache was full |
| shred_store_fec_sync_spilled_bytes | counter | FEC payload bytes the shred tile spilled to the store synchronously |
Gossvf Tile
| Metric | Type | Description |
|---|---|---|
| gossvf_message_rx {gossvf_message_result="success_pull_request"} | counter | Gossip messages received, by verification result (Pull Request (success)) |
| gossvf_message_rx {gossvf_message_result="success_pull_response"} | counter | Gossip messages received, by verification result (Pull Response (success)) |
| gossvf_message_rx {gossvf_message_result="success_push"} | counter | Gossip messages received, by verification result (Push (success)) |
| gossvf_message_rx {gossvf_message_result="success_prune"} | counter | Gossip messages received, by verification result (Prune (success)) |
| gossvf_message_rx {gossvf_message_result="success_ping"} | counter | Gossip messages received, by verification result (Ping (success)) |
| gossvf_message_rx {gossvf_message_result="success_pong"} | counter | Gossip messages received, by verification result (Pong (success)) |
| gossvf_message_rx {gossvf_message_result="dropped_unparseable"} | counter | Gossip messages received, by verification result (Unparseable) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_not_contact_info"} | counter | Gossip messages received, by verification result (Pull Request (not contact info)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_loopback"} | counter | Gossip messages received, by verification result (Pull Request (loopback)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_inactive"} | counter | Gossip messages received, by verification result (Pull Request (inactive)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_wallclock"} | counter | Gossip messages received, by verification result (Pull Request (wallclock)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_signature"} | counter | Gossip messages received, by verification result (Pull Request (signature)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_shred_version"} | counter | Gossip messages received, by verification result (Pull Request (shred version)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_request_mask_bits"} | counter | Gossip messages received, by verification result (Pull Request (mask bits)) |
| gossvf_message_rx {gossvf_message_result="dropped_prune_destination"} | counter | Gossip messages received, by verification result (Prune (destination)) |
| gossvf_message_rx {gossvf_message_result="dropped_prune_wallclock"} | counter | Gossip messages received, by verification result (Prune (wallclock)) |
| gossvf_message_rx {gossvf_message_result="dropped_prune_signature"} | counter | Gossip messages received, by verification result (Prune (signature)) |
| gossvf_message_rx {gossvf_message_result="dropped_push_loopback"} | counter | Gossip messages received, by verification result (Push (loopback)) |
| gossvf_message_rx {gossvf_message_result="dropped_push_no_valid_crds"} | counter | Gossip messages received, by verification result (Push (no valid crds)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_response_loopback"} | counter | Gossip messages received, by verification result (Pull Response (loopback)) |
| gossvf_message_rx {gossvf_message_result="dropped_pull_response_no_valid_crds"} | counter | Gossip messages received, by verification result (Pull Response (no valid crds)) |
| gossvf_message_rx {gossvf_message_result="dropped_ping_signature"} | counter | Gossip messages received, by verification result (Ping (signature)) |
| gossvf_message_rx {gossvf_message_result="dropped_pong_signature"} | counter | Gossip messages received, by verification result (Pong (signature)) |
| gossvf_message_rx_bytes {gossvf_message_result="success_pull_request"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (success)) |
| gossvf_message_rx_bytes {gossvf_message_result="success_pull_response"} | counter | Wire bytes of gossip messages received, by verification result (Pull Response (success)) |
| gossvf_message_rx_bytes {gossvf_message_result="success_push"} | counter | Wire bytes of gossip messages received, by verification result (Push (success)) |
| gossvf_message_rx_bytes {gossvf_message_result="success_prune"} | counter | Wire bytes of gossip messages received, by verification result (Prune (success)) |
| gossvf_message_rx_bytes {gossvf_message_result="success_ping"} | counter | Wire bytes of gossip messages received, by verification result (Ping (success)) |
| gossvf_message_rx_bytes {gossvf_message_result="success_pong"} | counter | Wire bytes of gossip messages received, by verification result (Pong (success)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_unparseable"} | counter | Wire bytes of gossip messages received, by verification result (Unparseable) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_not_contact_info"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (not contact info)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_loopback"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_inactive"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (inactive)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_wallclock"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_signature"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (signature)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_shred_version"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (shred version)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_request_mask_bits"} | counter | Wire bytes of gossip messages received, by verification result (Pull Request (mask bits)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_prune_destination"} | counter | Wire bytes of gossip messages received, by verification result (Prune (destination)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_prune_wallclock"} | counter | Wire bytes of gossip messages received, by verification result (Prune (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_prune_signature"} | counter | Wire bytes of gossip messages received, by verification result (Prune (signature)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_push_loopback"} | counter | Wire bytes of gossip messages received, by verification result (Push (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_push_no_valid_crds"} | counter | Wire bytes of gossip messages received, by verification result (Push (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_response_loopback"} | counter | Wire bytes of gossip messages received, by verification result (Pull Response (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pull_response_no_valid_crds"} | counter | Wire bytes of gossip messages received, by verification result (Pull Response (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_ping_signature"} | counter | Wire bytes of gossip messages received, by verification result (Ping (signature)) |
| gossvf_message_rx_bytes {gossvf_message_result="dropped_pong_signature"} | counter | Wire bytes of gossip messages received, by verification result (Pong (signature)) |
| gossvf_crds_rx {gossvf_crds_result="verified_pull_response"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (success)) |
| gossvf_crds_rx {gossvf_crds_result="duplicate_pull_response"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (duplicate)) |
| gossvf_crds_rx {gossvf_crds_result="verified_push"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (success)) |
| gossvf_crds_rx {gossvf_crds_result="duplicate_push"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (duplicate)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_pull_response_signature"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (signature)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_pull_response_no_contact_info"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (origin no contact info)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_pull_response_shred_version"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (origin shred version)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_pull_response_inactive"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (inactive)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_pull_response_wallclock"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (wallclock)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_push_signature"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (signature)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_push_no_contact_info"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (origin no contact info)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_push_shred_version"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (origin shred version)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_push_inactive"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (inactive)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_push_wallclock"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (wallclock)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_pull_response_deprecated"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Pull Response (deprecated value type)) |
| gossvf_crds_rx {gossvf_crds_result="dropped_push_deprecated"} | counter | CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails (Push (deprecated value type)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="verified_pull_response"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="duplicate_pull_response"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (duplicate)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="verified_push"} | counter | Wire bytes of CRDS values received, by verification result (Push (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="duplicate_push"} | counter | Wire bytes of CRDS values received, by verification result (Push (duplicate)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_pull_response_signature"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_pull_response_no_contact_info"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_pull_response_shred_version"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_pull_response_inactive"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_pull_response_wallclock"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_push_signature"} | counter | Wire bytes of CRDS values received, by verification result (Push (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_push_no_contact_info"} | counter | Wire bytes of CRDS values received, by verification result (Push (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_push_shred_version"} | counter | Wire bytes of CRDS values received, by verification result (Push (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_push_inactive"} | counter | Wire bytes of CRDS values received, by verification result (Push (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_push_wallclock"} | counter | Wire bytes of CRDS values received, by verification result (Push (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_pull_response_deprecated"} | counter | Wire bytes of CRDS values received, by verification result (Pull Response (deprecated value type)) |
| gossvf_crds_rx_bytes {gossvf_crds_result="dropped_push_deprecated"} | counter | Wire bytes of CRDS values received, by verification result (Push (deprecated value type)) |
Gossip Tile
| Metric | Type | Description |
|---|---|---|
| gossip_ping_peer_count {ping_peer_state="unpinged"} | gauge | Peers tracked for ping/pong, by state (Peer is known but has not been pinged yet) |
| gossip_ping_peer_count {ping_peer_state="awaiting_pong"} | gauge | Peers tracked for ping/pong, by state (Peer was pinged but has not yet responded, or responded with an invalid pong) |
| gossip_ping_peer_count {ping_peer_state="valid"} | gauge | Peers tracked for ping/pong, by state (Peer was pinged and is currently valid) |
| gossip_ping_peer_count {ping_peer_state="refreshing"} | gauge | Peers tracked for ping/pong, by state (Peer was pinged and is currently valid, and is being refreshed with a new ping) |
| gossip_pong_processed {pong_result="accepted"} | counter | Pongs processed, by result (Pong accepted) |
| gossip_pong_processed {pong_result="staked"} | counter | Pongs processed, by result (Ignored because the peer is staked and needs no ping) |
| gossip_pong_processed {pong_result="untracked"} | counter | Pongs processed, by result (Ignored because the peer has no outstanding ping) |
| gossip_pong_processed {pong_result="address_mismatch"} | counter | Pongs processed, by result (Ignored because it came from a different address than the one pinged) |
| gossip_pong_processed {pong_result="token_mismatch"} | counter | Pongs processed, by result (Ignored because the token did not match) |
| gossip_ping_peer_added | counter | Peers added to the ping tracker |
| gossip_ping_peer_removed {ping_peer_remove_reason="evicted"} | counter | Peers removed from the ping tracker, by reason (Evicted to make room for a new peer) |
| gossip_ping_peer_removed {ping_peer_remove_reason="staked"} | counter | Peers removed from the ping tracker, by reason (Became staked, no longer pinged) |
| gossip_ping_peer_removed {ping_peer_remove_reason="stale"} | counter | Peers removed from the ping tracker, by reason (Contact info not received for 60 s) |
| gossip_ping_peer_removed {ping_peer_remove_reason="identity"} | counter | Peers removed from the ping tracker, by reason (Became our identity) |
| gossip_ping_peer_address_changed | counter | Tracked peers whose gossip address changed; pong invalidated and reset to unpinged |
| gossip_sign_queue_full_dropped | counter | Messages dropped because the maximum number of sign requests were in flight |
| gossip_crds_capacity | gauge | Capacity of the data store |
| gossip_crds_used {crds_value_type="contact_info_v1"} | gauge | Entries in the data store (Contact Info V1) |
| gossip_crds_used {crds_value_type="vote"} | gauge | Entries in the data store (Vote) |
| gossip_crds_used {crds_value_type="lowest_slot"} | gauge | Entries in the data store (Lowest Slot) |
| gossip_crds_used {crds_value_type="snapshot_hashes"} | gauge | Entries in the data store (Snapshot Hashes) |
| gossip_crds_used {crds_value_type="accounts_hashes"} | gauge | Entries in the data store (Accounts Hashes) |
| gossip_crds_used {crds_value_type="epoch_slots"} | gauge | Entries in the data store (Epoch Slots) |
| gossip_crds_used {crds_value_type="version_v1"} | gauge | Entries in the data store (Version V1) |
| gossip_crds_used {crds_value_type="version_v2"} | gauge | Entries in the data store (Version V2) |
| gossip_crds_used {crds_value_type="node_instance"} | gauge | Entries in the data store (Node Instance) |
| gossip_crds_used {crds_value_type="duplicate_shred"} | gauge | Entries in the data store (Duplicate Shred) |
| gossip_crds_used {crds_value_type="incremental_snapshot_hashes"} | gauge | Entries in the data store (Incremental Snapshot Hashes) |
| gossip_crds_used {crds_value_type="contact_info_v2"} | gauge | Entries in the data store (Contact Info V2) |
| gossip_crds_used {crds_value_type="restart_last_voted_fork_slots"} | gauge | Entries in the data store (Restart Last Voted Fork Slots) |
| gossip_crds_used {crds_value_type="restart_heaviest_fork"} | gauge | Entries in the data store (Restart Heaviest Fork) |
| gossip_crds_expired | counter | Entries expired from the data store due to age |
| gossip_crds_evicted | counter | Entries evicted from the data store to make space for new entries |
| gossip_crds_relay_coalesced | counter | Received entry versions not relayed because a newer version of the same entry replaced them within the relay interval |
| gossip_crds_peer_capacity | gauge | Capacity for storing peers in the data store |
| gossip_crds_peer_count {crds_peer_stake_class="staked"} | gauge | Peers in the data store, by stake class (Peer has stake) |
| gossip_crds_peer_count {crds_peer_stake_class="unstaked"} | gauge | Peers in the data store, by stake class (Peer has no stake) |
| gossip_crds_peer_stake_lamports | gauge | Stake of peers in the data store |
| gossip_crds_peer_evicted | counter | Peers evicted from the data store to make space for new peers |
| gossip_crds_purged_capacity | gauge | Capacity of the list of purged data store entries |
| gossip_crds_purged_used | gauge | Purged data store entries |
| gossip_crds_purged_expired | counter | Purged entries expired from the purged list due to age |
| gossip_crds_purged_evicted | counter | Entries evicted from the purged list to make room |
| gossip_crds_upserted {gossip_crds_result="inserted_pull_response"} | counter | Verified CRDS values offered to the data store, by result (Pull Response (upserted)) |
| gossip_crds_upserted {gossip_crds_result="inserted_push"} | counter | Verified CRDS values offered to the data store, by result (Push (upserted)) |
| gossip_crds_upserted {gossip_crds_result="stale_pull_response"} | counter | Verified CRDS values offered to the data store, by result (Pull Response (newer entry already present in table)) |
| gossip_crds_upserted {gossip_crds_result="duplicate_pull_response"} | counter | Verified CRDS values offered to the data store, by result (Pull Response (duplicate)) |
| gossip_crds_upserted {gossip_crds_result="stale_push"} | counter | Verified CRDS values offered to the data store, by result (Push (newer entry already present in table)) |
| gossip_crds_upserted {gossip_crds_result="duplicate_push"} | counter | Verified CRDS values offered to the data store, by result (Push (duplicate)) |
| gossip_message_tx {gossip_message_type="pull_request"} | counter | Gossip messages sent, by message type (Pull Request) |
| gossip_message_tx {gossip_message_type="pull_response"} | counter | Gossip messages sent, by message type (Pull Response) |
| gossip_message_tx {gossip_message_type="push"} | counter | Gossip messages sent, by message type (Push) |
| gossip_message_tx {gossip_message_type="prune"} | counter | Gossip messages sent, by message type (Prune) |
| gossip_message_tx {gossip_message_type="ping"} | counter | Gossip messages sent, by message type (Ping) |
| gossip_message_tx {gossip_message_type="pong"} | counter | Gossip messages sent, by message type (Pong) |
| gossip_message_tx_bytes {gossip_message_type="pull_request"} | counter | Wire bytes sent in gossip messages, by message type (Pull Request) |
| gossip_message_tx_bytes {gossip_message_type="pull_response"} | counter | Wire bytes sent in gossip messages, by message type (Pull Response) |
| gossip_message_tx_bytes {gossip_message_type="push"} | counter | Wire bytes sent in gossip messages, by message type (Push) |
| gossip_message_tx_bytes {gossip_message_type="prune"} | counter | Wire bytes sent in gossip messages, by message type (Prune) |
| gossip_message_tx_bytes {gossip_message_type="ping"} | counter | Wire bytes sent in gossip messages, by message type (Ping) |
| gossip_message_tx_bytes {gossip_message_type="pong"} | counter | Wire bytes sent in gossip messages, by message type (Pong) |
| gossip_crds_push_tx {crds_value_type="contact_info_v1"} | counter | CRDS values sent in push messages, by value type (Contact Info V1) |
| gossip_crds_push_tx {crds_value_type="vote"} | counter | CRDS values sent in push messages, by value type (Vote) |
| gossip_crds_push_tx {crds_value_type="lowest_slot"} | counter | CRDS values sent in push messages, by value type (Lowest Slot) |
| gossip_crds_push_tx {crds_value_type="snapshot_hashes"} | counter | CRDS values sent in push messages, by value type (Snapshot Hashes) |
| gossip_crds_push_tx {crds_value_type="accounts_hashes"} | counter | CRDS values sent in push messages, by value type (Accounts Hashes) |
| gossip_crds_push_tx {crds_value_type="epoch_slots"} | counter | CRDS values sent in push messages, by value type (Epoch Slots) |
| gossip_crds_push_tx {crds_value_type="version_v1"} | counter | CRDS values sent in push messages, by value type (Version V1) |
| gossip_crds_push_tx {crds_value_type="version_v2"} | counter | CRDS values sent in push messages, by value type (Version V2) |
| gossip_crds_push_tx {crds_value_type="node_instance"} | counter | CRDS values sent in push messages, by value type (Node Instance) |
| gossip_crds_push_tx {crds_value_type="duplicate_shred"} | counter | CRDS values sent in push messages, by value type (Duplicate Shred) |
| gossip_crds_push_tx {crds_value_type="incremental_snapshot_hashes"} | counter | CRDS values sent in push messages, by value type (Incremental Snapshot Hashes) |
| gossip_crds_push_tx {crds_value_type="contact_info_v2"} | counter | CRDS values sent in push messages, by value type (Contact Info V2) |
| gossip_crds_push_tx {crds_value_type="restart_last_voted_fork_slots"} | counter | CRDS values sent in push messages, by value type (Restart Last Voted Fork Slots) |
| gossip_crds_push_tx {crds_value_type="restart_heaviest_fork"} | counter | CRDS values sent in push messages, by value type (Restart Heaviest Fork) |
| gossip_crds_push_tx_bytes {crds_value_type="contact_info_v1"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Contact Info V1) |
| gossip_crds_push_tx_bytes {crds_value_type="vote"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Vote) |
| gossip_crds_push_tx_bytes {crds_value_type="lowest_slot"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Lowest Slot) |
| gossip_crds_push_tx_bytes {crds_value_type="snapshot_hashes"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Snapshot Hashes) |
| gossip_crds_push_tx_bytes {crds_value_type="accounts_hashes"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Accounts Hashes) |
| gossip_crds_push_tx_bytes {crds_value_type="epoch_slots"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Epoch Slots) |
| gossip_crds_push_tx_bytes {crds_value_type="version_v1"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Version V1) |
| gossip_crds_push_tx_bytes {crds_value_type="version_v2"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Version V2) |
| gossip_crds_push_tx_bytes {crds_value_type="node_instance"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Node Instance) |
| gossip_crds_push_tx_bytes {crds_value_type="duplicate_shred"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Duplicate Shred) |
| gossip_crds_push_tx_bytes {crds_value_type="incremental_snapshot_hashes"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Incremental Snapshot Hashes) |
| gossip_crds_push_tx_bytes {crds_value_type="contact_info_v2"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Contact Info V2) |
| gossip_crds_push_tx_bytes {crds_value_type="restart_last_voted_fork_slots"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Restart Last Voted Fork Slots) |
| gossip_crds_push_tx_bytes {crds_value_type="restart_heaviest_fork"} | counter | Wire bytes of CRDS values sent in push messages, by value type (Restart Heaviest Fork) |
| gossip_crds_pull_response_tx {crds_value_type="contact_info_v1"} | counter | CRDS values sent in pull response messages, by value type (Contact Info V1) |
| gossip_crds_pull_response_tx {crds_value_type="vote"} | counter | CRDS values sent in pull response messages, by value type (Vote) |
| gossip_crds_pull_response_tx {crds_value_type="lowest_slot"} | counter | CRDS values sent in pull response messages, by value type (Lowest Slot) |
| gossip_crds_pull_response_tx {crds_value_type="snapshot_hashes"} | counter | CRDS values sent in pull response messages, by value type (Snapshot Hashes) |
| gossip_crds_pull_response_tx {crds_value_type="accounts_hashes"} | counter | CRDS values sent in pull response messages, by value type (Accounts Hashes) |
| gossip_crds_pull_response_tx {crds_value_type="epoch_slots"} | counter | CRDS values sent in pull response messages, by value type (Epoch Slots) |
| gossip_crds_pull_response_tx {crds_value_type="version_v1"} | counter | CRDS values sent in pull response messages, by value type (Version V1) |
| gossip_crds_pull_response_tx {crds_value_type="version_v2"} | counter | CRDS values sent in pull response messages, by value type (Version V2) |
| gossip_crds_pull_response_tx {crds_value_type="node_instance"} | counter | CRDS values sent in pull response messages, by value type (Node Instance) |
| gossip_crds_pull_response_tx {crds_value_type="duplicate_shred"} | counter | CRDS values sent in pull response messages, by value type (Duplicate Shred) |
| gossip_crds_pull_response_tx {crds_value_type="incremental_snapshot_hashes"} | counter | CRDS values sent in pull response messages, by value type (Incremental Snapshot Hashes) |
| gossip_crds_pull_response_tx {crds_value_type="contact_info_v2"} | counter | CRDS values sent in pull response messages, by value type (Contact Info V2) |
| gossip_crds_pull_response_tx {crds_value_type="restart_last_voted_fork_slots"} | counter | CRDS values sent in pull response messages, by value type (Restart Last Voted Fork Slots) |
| gossip_crds_pull_response_tx {crds_value_type="restart_heaviest_fork"} | counter | CRDS values sent in pull response messages, by value type (Restart Heaviest Fork) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="contact_info_v1"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Contact Info V1) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="vote"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Vote) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="lowest_slot"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Lowest Slot) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="snapshot_hashes"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Snapshot Hashes) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="accounts_hashes"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Accounts Hashes) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="epoch_slots"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Epoch Slots) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="version_v1"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Version V1) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="version_v2"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Version V2) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="node_instance"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Node Instance) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="duplicate_shred"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Duplicate Shred) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="incremental_snapshot_hashes"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Incremental Snapshot Hashes) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="contact_info_v2"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Contact Info V2) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="restart_last_voted_fork_slots"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Restart Last Voted Fork Slots) |
| gossip_crds_pull_response_tx_bytes {crds_value_type="restart_heaviest_fork"} | counter | Wire bytes of CRDS values sent in pull response messages, by value type (Restart Heaviest Fork) |
| gossip_wait_for_supermajority_staked_peer_online | gauge | Staked peers with recent gossip activity |
| gossip_wait_for_supermajority_staked_peer_count | gauge | Staked peers tracked while waiting for supermajority |
| gossip_wait_for_supermajority_stake_online_lamports | gauge | Stake from peers with recent gossip activity |
| gossip_wait_for_supermajority_stake_tracked_lamports | gauge | Stake tracked while waiting for supermajority |
| gossip_wait_for_supermajority_state | gauge | Wait-for-supermajority progress: 0=unset (Not started), 1=loading_stakes (Loading stakes from the snapshot), 2=waiting_for_stake (Waiting for enough stake to come online), 3=publishing (Publishing our contact info to start), 4=done_or_disabled (Done, or wait-for-supermajority is disabled) |
| gossip_contact_info_client_id | gauge | Client id currently advertised in our ContactInfo |
Repair Tile
| Metric | Type | Description |
|---|---|---|
| repair_pkt_tx | counter | Network packets sent, including requests, pings and pongs |
| repair_request_tx {repair_request_type="pong"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Pong) |
| repair_request_tx {repair_request_type="window_index"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Shred at an index) |
| repair_request_tx {repair_request_type="highest_window_index"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Highest shred at or above an index) |
| repair_request_tx {repair_request_type="orphan"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Ancestors of an orphan slot) |
| repair_request_tx {repair_request_type="parent_fec_set_count"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Parent and FEC set count of a block (Alpenglow)) |
| repair_request_tx {repair_request_type="fec_set_root"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Merkle root of a FEC set (Alpenglow)) |
| repair_request_tx {repair_request_type="window_index_for_block_id"} | counter | Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt (Shred at an index of a specific block (Alpenglow)) |
| repair_highest_delivered_slot | gauge | Highest slot fully delivered to replay, 0 if none |
| repair_highest_received_slot | gauge | Highest slot a shred was received for, 0 if none |
| repair_first_turbine_slot | gauge | First turbine slot seen, the catchup target, 0 if none |
| repair_catchup_slot | gauge | Highest slot past the window with a verified cert, the staked catchup target, 0 if none |
| repair_block_pending | gauge | Blocks in the eager, notar and final block treaps, waiting for discovery |
| repair_request_in_flight | gauge | Pending request attempts awaiting a response; a hedge is another attempt |
| repair_fec_delivered | counter | FEC sets published to replay |
| repair_shred_old_ignored | counter | Shreds received at or below the root, ignored |
| repair_shred_rx {shred_response_match_result="for_block_id"} | counter | Data shreds received as repair responses, by the pending request they matched (Response to a pending WindowIndexForBlockId request) |
| repair_shred_rx {shred_response_match_result="slot_index"} | counter | Data shreds received as repair responses, by the pending request they matched (Response to a pending WindowIndex or HighestWindowIndex request) |
| repair_shred_rx {shred_response_match_result="unmatched"} | counter | Data shreds received as repair responses, by the pending request they matched (Matched no pending request: a duplicate, or an answer to an expired or evicted request) |
| repair_shred_rx {shred_response_match_result="eqvoc"} | counter | Data shreds received as repair responses, by the pending request they matched (Repair response that showed the slot's leader equivocating, dropped before matching; only unverified-nonce HighestWindowIndex or Orphan replies) |
| repair_meta_rx | counter | Alpenglow repair metadata responses received |
| repair_meta_malformed_rejected | counter | Metadata responses that failed to decode |
| repair_meta_unsolicited_rejected | counter | Metadata responses matching no outstanding request |
| repair_parent_fec_set_count_accepted | counter | ParentFecSetCount responses accepted |
| repair_parent_fec_set_count_failed | counter | ParentFecSetCount responses that failed verification |
| repair_fec_set_root_accepted | counter | FecSetRoot responses accepted |
| repair_fec_set_root_failed | counter | FecSetRoot responses that failed verification |
| repair_replay_root_advanced_rx | counter | Root advanced messages received from replay |
| repair_replay_missing_fec_rx | counter | Missing FEC messages received from replay, each a from-root reconsume |
| repair_ping_rejected {ping_reject_reason="malformed"} | counter | Pings rejected, by reason (Failed to decode) |
| repair_ping_rejected {ping_reject_reason="unknown_peer"} | counter | Pings rejected, by reason (Sender is not a known peer) |
| repair_ping_rejected {ping_reject_reason="signature_failed"} | counter | Pings rejected, by reason (Signature did not verify) |
| repair_ping_rejected {ping_reject_reason="sign_unavailable"} | counter | Pings rejected, by reason (Pong not sent: signing halted for an identity switch, or no free sign slot) |
| repair_ping_rejected {ping_reject_reason="address_mismatch"} | counter | Pings rejected, by reason (Pong not sent: sender address differs from the peer's gossip address) |
| repair_ping_rejected {ping_reject_reason="unsolicited"} | counter | Pings rejected, by reason (Pong not sent: no request was sent to the peer within the ping TTL) |
| repair_response_latency_nanos | histogram | Time from sending a repair request to receiving its response |
| repair_retry_delay_nanos | histogram | Delay scheduled before a request's next attempt: its peer's hedge time, 16.8 ms to 100 ms, 16.8 ms after an escape |
| repair_timeout_pending | gauge | Timeouts in the eager, notar and final timeout queues |
| repair_timeout_cancelled | counter | Timeouts dropped because their block is gone or their work is done |
| repair_timeout_no_peer_deferred | counter | Timeouts deferred because no peer could be picked |
| repair_request_hedged | counter | Request attempts after the first for the same timer |
| repair_request_expired | counter | Pending requests with no response after about 1 s (PENDING_TTL); a peer is banned for 10 s after 32 in a row, pending-pool evictions not counted |
| repair_request_by_id_expired | counter | WindowIndexForBlockId requests that expired unanswered, at or above the root |
| repair_request_stuck_escaped | counter | Retries sent to a random staked peer because 3 attempts in a row brought no new shred of the FEC set |
| repair_request_late_retried | counter | Request attempts sent after 3 or more earlier attempts for the same timer |
| repair_escape_peer_hedge_delay_nanos | histogram | Uncapped hedge time (srtt+4*rttvar, 100 ms if unmeasured) of each staked peer an escaped retry was sent to |
| repair_peer_by_id_banned | counter | Peers banned for 10 s after 4 WindowIndexForBlockId requests in a row expired, whatever their other replies |
| repair_ping_probe_tx | counter | Requests sent to elicit a ping from a peer with a new address |
| repair_eager_delay_nanos | gauge | Cluster-wide wait for turbine before a missing FEC set is repaired (srtt+4*rttvar, clamped to 20-250 ms; 250 ms until sampled) |
| repair_sign_request_pending | gauge | Messages in the sign pool waiting for a signature from the sign tiles |
| repair_timeout_no_staked_peer_deferred | counter | Of TimeoutNoPeerDeferred, retries of a block not known only from turbine, which pick only staked peers: every staked peer was banned, without an address or at its in-flight limit |
| repair_timeout_deferred {timeout_defer_reason="turbine_active"} | counter | Timeouts put back without sending a request, by reason, one per deferral (HighestWindowIndex put back: turbine delivered a FEC set of the slot within the eager wait) |
| repair_timeout_deferred {timeout_defer_reason="all_received"} | counter | Timeouts put back without sending a request, by reason, one per deferral (FEC set request put back: every data shred arrived, the set has not completed) |
| repair_timeout_queue_full_dropped | counter | Timeouts dropped because their role's timeout queue was full; the block goes back to its treap and the work is discovered again |
| repair_timeout_queue_full_paused | counter | Blocks whose discovery stopped because their role's timeout queue filled; the block goes back to its treap and its remaining FEC sets are queued later |
| repair_signing_halted | counter | Identity switches that halted signing; no request, ping or pong is signed until the switch completes and unhalts |
| repair_timeout_overdue_nanos | gauge | Time the earliest due timeout has waited past its due time when metrics were written, 0 if none is due; grows while signing is halted or fewer than 40 sign slots are free |
| repair_first_cert_slot | gauge | First certified slot in the window seen by a staked node, which starts its catchup, 0 if none yet |
Rserve Tile
| Metric | Type | Description |
|---|---|---|
| rserve_request_rx {repair_request_type="pong"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Pong) |
| rserve_request_rx {repair_request_type="window_index"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Shred at an index) |
| rserve_request_rx {repair_request_type="highest_window_index"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Highest shred at or above an index) |
| rserve_request_rx {repair_request_type="orphan"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Ancestors of an orphan slot) |
| rserve_request_rx {repair_request_type="parent_fec_set_count"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Parent and FEC set count of a block (Alpenglow)) |
| rserve_request_rx {repair_request_type="fec_set_root"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Merkle root of a FEC set (Alpenglow)) |
| rserve_request_rx {repair_request_type="window_index_for_block_id"} | counter | Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks (Shred at an index of a specific block (Alpenglow)) |
| rserve_request_rx_bytes | counter | Bytes of incoming repair request payloads |
| rserve_request_malformed {request_malformed_reason="too_small"} | counter | Packets dropped before parsing, by reason (Packet shorter than a message tag) |
| rserve_request_malformed {request_malformed_reason="unsupported_ping"} | counter | Packets dropped before parsing, by reason (Ping sent to the repair server port) |
| rserve_request_malformed {request_malformed_reason="unknown_tag"} | counter | Packets dropped before parsing, by reason (Unknown or unsupported message tag) |
| rserve_request_malformed {request_malformed_reason="wrong_size"} | counter | Packets dropped before parsing, by reason (Message size does not match its tag) |
| rserve_request_malformed {request_malformed_reason="unsupported_ancestor_hashes"} | counter | Packets dropped before parsing, by reason (Ancestor hashes request, which is not served) |
| rserve_request_dropped {request_drop_reason="signature_failed"} | counter | Requests dropped during validation, by reason (Signature failed to verify) |
| rserve_request_dropped {request_drop_reason="from_self"} | counter | Requests dropped during validation, by reason (Sent by our own identity) |
| rserve_request_dropped {request_drop_reason="wrong_recipient"} | counter | Requests dropped during validation, by reason (Addressed to another identity) |
| rserve_request_dropped {request_drop_reason="timestamp_skew"} | counter | Requests dropped during validation, by reason (Timestamp outside the accepted window) |
| rserve_request_dropped {request_drop_reason="pong_token_invalid"} | counter | Requests dropped during validation, by reason (Pong token outdated or invalid) |
| rserve_request_dropped {request_drop_reason="shred_index_invalid"} | counter | Requests dropped during validation, by reason (Shred index past the end of a block) |
| rserve_request_dropped {request_drop_reason="fec_set_index_invalid"} | counter | Requests dropped during validation, by reason (FEC set index unaligned or past the end of the block) |
| rserve_request_unanswered {repair_request_type="pong"} | counter | Valid requests that could not be answered, by type (Pong) |
| rserve_request_unanswered {repair_request_type="window_index"} | counter | Valid requests that could not be answered, by type (Shred at an index) |
| rserve_request_unanswered {repair_request_type="highest_window_index"} | counter | Valid requests that could not be answered, by type (Highest shred at or above an index) |
| rserve_request_unanswered {repair_request_type="orphan"} | counter | Valid requests that could not be answered, by type (Ancestors of an orphan slot) |
| rserve_request_unanswered {repair_request_type="parent_fec_set_count"} | counter | Valid requests that could not be answered, by type (Parent and FEC set count of a block (Alpenglow)) |
| rserve_request_unanswered {repair_request_type="fec_set_root"} | counter | Valid requests that could not be answered, by type (Merkle root of a FEC set (Alpenglow)) |
| rserve_request_unanswered {repair_request_type="window_index_for_block_id"} | counter | Valid requests that could not be answered, by type (Shred at an index of a specific block (Alpenglow)) |
| rserve_pkt_tx | counter | Network packets sent, including pings and responses |
| rserve_response_tx {repair_response_type="ping"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Ping sent to a requester not yet in the ping cache) |
| rserve_response_tx {repair_response_type="window_index"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Shred at an index) |
| rserve_response_tx {repair_response_type="highest_window_index"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Highest shred at or above an index) |
| rserve_response_tx {repair_response_type="orphan"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Shred from an ancestor of an orphan slot) |
| rserve_response_tx {repair_response_type="parent_fec_set_count"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Parent and FEC set count of a block (Alpenglow)) |
| rserve_response_tx {repair_response_type="fec_set_root"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Merkle root of a FEC set (Alpenglow)) |
| rserve_response_tx {repair_response_type="window_index_for_block_id"} | counter | Response packets sent, by type; Orphan sends up to 11 per request (Shred at an index of a specific block (Alpenglow)) |
| rserve_response_tx_bytes | counter | Payload bytes sent in response packets |
| rserve_disk_read_attempted {disk_read_result="success"} | counter | Store reads for repair responses, by result (Read served from the store) |
| rserve_disk_read_attempted {disk_read_result="miss"} | counter | Store reads for repair responses, by result (Not in the store) |
| rserve_disk_read_attempted {disk_read_result="busy"} | counter | Store reads for repair responses, by result (Index binding or ring cell changed during the read) |
| rserve_disk_read_attempted {disk_read_result="scan_limit"} | counter | Store reads for repair responses, by result (Cached highest-shred bound was no longer readable) |
| rserve_disk_read_bytes | counter | Shred bytes read from the store for repair responses |
| rserve_shred_used | gauge | Shreds currently in the shreds database |
| rserve_shred_capacity | gauge | Shreds the shreds database can hold |
| rserve_disk_used_bytes | gauge | Logical bytes occupied by live wire shreds and spilled FEC payloads |
| rserve_disk_peak_bytes | gauge | Logical wire-ring high-water plus spill pages high-water |
| rserve_disk_write_duration_nanos | histogram | Duration of persisting one accepted data shred |
| rserve_disk_shred_written | counter | Data shreds persisted to the repair store |
| rserve_disk_shred_write_failed | counter | Data shreds rejected by the repair store API |
| rserve_disk_shred_written_bytes | counter | Bytes written by persisted data shreds |
| rserve_store_fec_preevict_duration_nanos | histogram | Duration of preevicting one FEC payload from the store RAM cache |
| rserve_store_fec_preevicted | counter | FEC payloads preevicted from the store RAM cache to disk |
| rserve_store_fec_preevicted_bytes | counter | FEC payload bytes preevicted from the store RAM cache to disk |
| rserve_store_fec_cache_free | gauge | Immediately available FEC payload cache entries |
| rserve_store_fec_cache_capacity | gauge | FEC payload cache capacity |
| rserve_store_fec_cache_free_target | gauge | Free entries background preeviction tries to restore |
| rserve_store_fec_cache_free_low_water | gauge | Free entry count below which background preeviction starts |
| rserve_ping_cache_used | gauge | Active entries in the ping cache |
| rserve_ping_cache_capacity | gauge | Ping cache entry capacity |
| rserve_ping_cache_evicted | counter | Ping cache entries evicted for a new peer (cache full); TTL expiries not counted |
Replay Tile
| Metric | Type | Description |
|---|---|---|
| replay_identity_balance_lamports | gauge | Identity account balance at the optimistically confirmed slot, or the finalized slot under Alpenglow |
| replay_active_stake_lamports | gauge | Our active stake at the optimistically confirmed slot, or the finalized slot under Alpenglow |
| replay_cluster_active_stake_lamports | gauge | Cluster active stake at the optimistically confirmed slot, or the finalized slot under Alpenglow |
| replay_epoch_credits | gauge | Credits our vote account earned this epoch, as of the optimistically confirmed slot, or the finalized slot under Alpenglow, where they are lamport-denominated reward credits |
| replay_is_vote_account_inadmissible | gauge | Whether our vote account's stake is not admitted: at the optimistically confirmed slot it fails the validator admission ticket filter (missing, below the V4 rent-exempt minimum, or not V4 with a BLS pubkey), or it passes but has not yet been admitted at an epoch boundary |
| replay_last_rewarded_vote_slot | gauge | Latest slot for which our vote appears in a reward certificate, 0 if none observed |
| replay_store_query_duration_nanos | histogram | Duration of ingesting one queried FEC set into the scheduler |
| replay_store_fec_queried | counter | FEC sets queried from the store |
| replay_store_fec_query_missed | counter | FEC set store queries that found no entry |
| replay_store_fec_spilled | counter | FEC payloads spilled from the store RAM cache to disk |
| replay_store_fec_spilled_bytes | counter | FEC payload bytes spilled from the store RAM cache to disk |
| replay_store_fec_spill_read | counter | Spilled FEC payloads read from disk |
| replay_store_fec_spill_read_bytes | counter | Spilled FEC payload bytes read from disk |
| replay_store_fec_cache_used | gauge | FEC payload slots in use in the store RAM cache |
| replay_store_fec_cache_capacity | gauge | FEC payload capacity of the store RAM cache |
| replay_store_fec_cache_pinned | gauge | FEC payload views currently pinned against eviction |
| replay_txncache_page_miss | counter | Transaction cache lookups under the read lock that found their page not resident |
| replay_txncache_page_spilled | counter | Dirty transaction cache pages written to the spill file on eviction |
| replay_txncache_page_spill_read | counter | Transaction cache pages read back from the spill file |
| replay_txncache_frame_evicted | counter | Transaction cache frames evicted to load another page |
| replay_txncache_spill_duration_nanos | counter | Time in transaction cache spill file reads and writes |
| replay_txncache_lock_read_wait_duration_nanos | counter | Time all tiles spent waiting to take the transaction cache shared lock |
| replay_txncache_lock_write_wait_duration_nanos | counter | Time all tiles spent waiting to take the transaction cache exclusive lock |
| replay_txncache_page_used | gauge | Transaction cache logical pages in use, resident or spilled |
| replay_txncache_page_capacity | gauge | Maximum transaction cache logical pages |
| replay_txncache_frame_used | gauge | Transaction cache in-memory frames holding a page |
| replay_txncache_frame_capacity | gauge | Transaction cache in-memory frames, from runtime.transaction_cache_size_mib |
| replay_stake_delegation_root_used | gauge | Rooted stake delegations held in memory |
| replay_stake_delegation_root_capacity | gauge | Rooted stake delegations that fit in memory, from runtime.stake_delegation_cache_cnt |
| replay_stake_delegation_root_disk_count | gauge | Rooted stake delegations currently spilled to disk |
| replay_stake_delegation_delta_used | gauge | Unrooted stake delegation fork deltas held in memory |
| replay_stake_delegation_delta_capacity | gauge | Unrooted stake delegation fork deltas that fit in memory |
| replay_stake_delegation_delta_disk_count | gauge | Unrooted stake delegation fork deltas currently spilled to disk |
| replay_stake_delegation_root_spilled | counter | Rooted stake delegations written to disk because the in-memory root pool was full |
| replay_stake_delegation_delta_spilled | counter | Stake delegation fork deltas written to disk because the in-memory delta pool was full |
| replay_stake_delegation_disk_read | counter | Reads from the stake delegation spill file by all tiles |
| replay_stake_delegation_disk_written | counter | Writes to the stake delegation spill file by all tiles |
| replay_stake_delegation_disk_duration_nanos | counter | Time all tiles spent in stake delegation spill file reads and writes |
| replay_is_caught_up | gauge | Whether replay has caught up to the cluster tip; set once per run, never cleared |
| replay_root_slot | gauge | Most recent consensus root, 0 if none |
| replay_root_slot_distance | gauge | Slots between the current reset slot and our root |
| replay_leader_slot | gauge | Slot at which we are currently leader, or 0 if none |
| replay_next_leader_slot | gauge | Slot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot |
| replay_reset_slot | gauge | Slot replay last reset to (the newest replayed slot or parent-ready parent); can decrease |
| replay_bank_capacity | gauge | Maximum number of banks we can have alive |
| replay_bank_used | gauge | Banks we currently have alive |
| replay_last_reassembly_dequeued_slot | gauge | Slot of the FEC set most recently dequeued from reassembly for replay, 0 if none |
| replay_last_reassembly_dequeued_fec_index | gauge | FEC set index of the FEC set most recently dequeued from reassembly for replay; read with LastReassemblyDequeuedSlot |
| replay_sched_active_bank_index | gauge | Bank pool index the replay scheduler is dispatching work for, ULONG_MAX if none |
| replay_sched_last_dispatch_bank_index | gauge | Bank pool index the replay scheduler last dispatched work for, ULONG_MAX if none |
| replay_sched_last_dispatch_time | gauge | Wall-clock time the replay scheduler last dispatched work, 0 if never |
| replay_sched_staging_lane_used | gauge | Staging lanes currently occupied in the replay scheduler. Up to four staging lanes can be occupied at once |
| replay_sched_staging_lane_peak | gauge | High watermark of staging lanes that were at one point occupied in the replay scheduler |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane0"} | gauge | Bank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 0) |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane1"} | gauge | Bank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 1) |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane2"} | gauge | Bank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 2) |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane3"} | gauge | Bank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty (Staging lane 3) |
| replay_sched_txn_pool_used | gauge | Transaction pool entries held by the replay scheduler, from parse until release |
| replay_sched_txn_pool_capacity | gauge | Maximum transactions the replay scheduler can hold |
| replay_sched_microblock_pool_used | gauge | Microblocks available to be scheduled in the replay scheduler |
| replay_sched_microblock_pool_capacity | gauge | Maximum microblocks the replay scheduler can hold |
| replay_sched_block_pool_used | gauge | Blocks alive in the replay scheduler |
| replay_sched_block_pool_capacity | gauge | Maximum blocks the replay scheduler can track |
| replay_sched_block_added {sched_block_staging_state="staged"} | counter | Blocks added to the replay scheduler, by staging state (Added as staged) |
| replay_sched_block_added {sched_block_staging_state="unstaged"} | counter | Blocks added to the replay scheduler, by staging state (Added as unstaged) |
| replay_sched_block_added {sched_block_staging_state="dead"} | counter | Blocks added to the replay scheduler, by staging state (Added as the dead child of a dead block, never dispatched) |
| replay_sched_block_replayed | counter | Blocks the replay scheduler considers fully replayed |
| replay_sched_block_abandoned | counter | Blocks the replay scheduler abandoned for any reason |
| replay_sched_block_rejected | counter | Blocks the replay scheduler found invalid while parsing or verifying PoH; also counted in SchedBlockAbandoned once drained; replay-tile verdicts not counted |
| replay_sched_block_promoted | counter | Blocks the replay scheduler promoted from unstaged to a staging lane |
| replay_sched_block_demoted | counter | Blocks the replay scheduler demoted from a staging lane to unstaged |
| replay_sched_txn_promoted | counter | Transactions the replay scheduler moved into a staging lane when it promoted their block |
| replay_sched_block_deactivated {sched_deactivate_reason="no_child"} | counter | Times the replay scheduler stopped dispatching a block, by reason (A block fully replayed and the block activated next, if any, was not its child) |
| replay_sched_block_deactivated {sched_deactivate_reason="no_work"} | counter | Times the replay scheduler stopped dispatching a block, by reason (A partially replayed block ran out of work to dispatch) |
| replay_sched_block_deactivated {sched_deactivate_reason="abandoned"} | counter | Times the replay scheduler stopped dispatching a block, by reason (A block was abandoned, other than by minority fork pruning) |
| replay_sched_block_deactivated {sched_deactivate_reason="minority_fork"} | counter | Times the replay scheduler stopped dispatching a block, by reason (A block was pruned for being on a minority fork (not also counted as Abandoned)) |
| replay_sched_block_deactivated {sched_deactivate_reason="spec_yield"} | counter | Times the replay scheduler stopped dispatching a block, by reason (A block that could only speculate on out-of-band lthash additions gave way to another lane head's real work) |
| replay_sched_block_deactivated {sched_deactivate_reason="spec_pool_full"} | counter | Times the replay scheduler stopped dispatching a block, by reason (A partially replayed block owing out-of-band lthash additions had nothing to dispatch because the speculation pool had no free entry) |
| replay_sched_block_inband_picked | counter | Blocks that fell back to in-band lthash because every demotion reservation was in use |
| replay_sched_lane_switched | counter | Times the replay scheduler activated a block on a different fork than the previously active block |
| replay_sched_lane_promoted | counter | Times the replay scheduler promoted a block onto a staging lane |
| replay_sched_lane_demoted | counter | Times the replay scheduler demoted a block from a staging lane |
| replay_sched_fork_observed | counter | Forks the replay scheduler has observed |
| replay_sched_txn_lut_resolve_attempted {sched_lut_resolve_result="success"} | counter | Transactions with address lookup tables the replay scheduler tried to resolve, by result (Address lookup tables expanded successfully) |
| replay_sched_txn_lut_resolve_attempted {sched_lut_resolve_result="failed"} | counter | Transactions with address lookup tables the replay scheduler tried to resolve, by result (Address lookup table expansion failed) |
| replay_sched_txn_parsed_abandoned | counter | Transactions that were parsed but ended up being in abandoned blocks |
| replay_sched_txn_executed_abandoned | counter | Transactions that were executed but ended up being in abandoned blocks |
| replay_sched_txn_done_abandoned | counter | Transactions that were fully replayed but ended up being in abandoned blocks |
| replay_sched_txn_in_flight_integral_nanos | counter | Integral of in-flight transactions over time (transaction-nanoseconds); divide by SchedTxnBusyDurationNanos for the average in-flight count |
| replay_sched_txn_busy_duration_nanos | counter | Time the replay scheduler had at least one transaction in flight |
| replay_sched_txn_idle_duration_nanos | counter | Time the replay scheduler had no transaction in flight |
| replay_sched_txn_parsed | counter | Transactions the replay pipeline has parsed |
| replay_sched_txn_executed | counter | Transactions the replay pipeline has executed |
| replay_sched_txn_signature_verified | counter | Transactions the replay pipeline has sigverified |
| replay_sched_txn_poh_mixed | counter | Transactions the replay pipeline has performed PoH mixin on |
| replay_sched_txn_done | counter | Transactions that were fully replayed |
| replay_sched_microblock_parsed | counter | Microblocks the replay pipeline has parsed |
| replay_sched_microblock_hashed | counter | Microblocks the replay pipeline has fully hashed |
| replay_sched_microblock_done | counter | Microblocks that were fully replayed |
| replay_sched_ingested_bytes | counter | Bytes the replay scheduler has ingested |
| replay_sched_trailing_skipped_bytes | counter | Trailing bytes of a block the replay scheduler ingested but did not parse |
| replay_sched_dropped_bytes | counter | Bytes the replay scheduler refused to ingest because the block was abandoned |
| replay_sched_fec_ingested | counter | FEC sets the replay scheduler has been given |
| replay_sched_lthash_write_observed {sched_lthash_write_result="first_write"} | counter | Writes the replay scheduler saw for out-of-band lthash, by result (First write to the account since its last addition pseudo transaction, which makes the block owe it an addition) |
| replay_sched_lthash_write_observed {sched_lthash_write_result="coalesced"} | counter | Writes the replay scheduler saw for out-of-band lthash, by result (Write to an account the block already owed an addition for) |
| replay_sched_lthash_write_observed_by_type {sched_lthash_write_type="parsed"} | counter | Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Declared writable accounts of parsed transactions, as handed to the dispatcher) |
| replay_sched_lthash_write_observed_by_type {sched_lthash_write_type="block_start"} | counter | Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Accounts registered by start-of-block processing) |
| replay_sched_lthash_write_observed_by_type {sched_lthash_write_type="late_lut"} | counter | Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Writable lookup table accounts registered at transaction completion) |
| replay_sched_lthash_write_observed_by_type {sched_lthash_write_type="demoted"} | counter | Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Accounts re-registered when their block was demoted) |
| replay_sched_lthash_write_observed_by_type {sched_lthash_write_type="spec_given_up"} | counter | Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type (Accounts re-registered when a speculative addition was given up) |
| replay_sched_lthash_add_created | counter | Out-of-band lthash addition pseudo transactions the dispatcher handed back |
| replay_sched_lthash_add_surfaced | counter | Addition pseudo transactions the dispatcher handed out as ready |
| replay_sched_lthash_add_stale_surfaced | counter | Addition pseudo transactions surfaced after a later write made them stale, completed without hashing |
| replay_sched_lthash_add_dispatched | counter | Additions handed to an exec tile |
| replay_sched_lthash_add_stale_dropped | counter | Addition results dropped because a later write made them stale while on the tile |
| replay_sched_lthash_add_applied | counter | Addition results added to a block's delta |
| replay_sched_lthash_add_undone | counter | Applied additions subtracted back out after a later write |
| replay_sched_lthash_spec_created | counter | Speculative additions created |
| replay_sched_lthash_spec_applied | counter | Speculative additions applied; a subset of SchedLthashAddApplied |
| replay_sched_lthash_spec_invalidated | counter | Speculative additions invalidated before they applied |
| replay_sched_lthash_sub_owed | counter | Lane map entries inserted, each owing one out-of-band lthash subtraction |
| replay_sched_lthash_sub_dispatched | counter | Subtractions handed to an exec tile |
| replay_sched_lthash_sub_applied | counter | Subtraction results subtracted from a block's delta |
| replay_sched_lthash_account_frozen | counter | Accounts whose out-of-band lthash subtraction and addition are in the delta of a block whose end task completed |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="add_owed"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions owed by abandoned blocks, never created) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="add_pending"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Pseudo transactions of abandoned blocks, never surfaced) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="add_applied"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions in the delta of an abandoned or demoted block, net of undone) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="sub_queued"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Subtractions of abandoned blocks, never dispatched) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="sub_applied"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Subtractions in the delta of an abandoned or demoted block) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="add_held"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions of abandoned blocks that were batched behind a transaction and never reached a tile) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="inband_add_owed"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Additions owed by blocks that picked in-band hashing, never created) |
| replay_sched_lthash_work_dropped {sched_lthash_work_type="inband_sub_queued"} | counter | Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type (Subtractions of blocks that picked in-band hashing, never dispatched) |
| replay_sched_lthash_task_dispatched {sched_lthash_task_type="sub"} | counter | Out-of-band lthash tasks handed to exec tiles, each a batch of the accounts SchedLthashSubDispatched and SchedLthashAddDispatched count, by type (Subtraction tasks, each a batch of accounts hashed on the parent fork) |
| replay_sched_lthash_task_dispatched {sched_lthash_task_type="add"} | counter | Out-of-band lthash tasks handed to exec tiles, each a batch of the accounts SchedLthashSubDispatched and SchedLthashAddDispatched count, by type (Addition tasks, each a batch of accounts hashed on the block's fork) |
| replay_sched_lthash_batch_held | counter | Out-of-band lthash addition batches held because a transaction or standalone speculation surfaced behind them |
| replay_sched_lthash_batch_idle_waited | counter | Dispatch polls that held an out-of-band lthash addition batch because too few exec tiles were idle |
| replay_bank_frozen | counter | Banks frozen, including our own leader blocks |
| replay_chain_txn_height | gauge | Transactions from genesis through the last replayed block, 0 if none; drops on a fork switch |
| replay_fec_poll_skipped {fec_poll_skip_reason="sched_full"} | counter | Replay loop iterations that processed no FEC set, by reason (The scheduler was full) |
| replay_fec_poll_skipped {fec_poll_skip_reason="leader_freeze_wait"} | counter | Replay loop iterations that processed no FEC set, by reason (Waiting for PoH to end our leader block) |
| replay_fec_poll_skipped {fec_poll_skip_reason="bank_full"} | counter | Replay loop iterations that processed no FEC set, by reason (The bank pool was full) |
| replay_fec_poll_skipped {fec_poll_skip_reason="lane_wait"} | counter | Replay loop iterations that processed no FEC set, by reason (A block's first FEC set waited for a staging lane) |
| replay_fec_parent_missing_dropped | counter | FEC sets dropped because replay could not resolve the block's parent (Alpenglow), which was dropped on a dead lineage, evicted with its tracking slot reused, or rooted past |
| replay_fec_rooted_past_skipped | counter | Repair FEC sets of a slot at or below the published root, or of a block whose parent is below it, skipped rather than dropped: no redelivery can replay them |
| replay_root_advance_deferred | counter | Times the storage root could not advance to the consensus root yet |
| replay_leader_window_already_produced | counter | Leader windows given up because the cluster already produced one of their slots, eg. our own past slots replayed after a restart (Alpenglow) |
| replay_leader_window_superseded | counter | Leader windows given up after votor sent them because certificates already let a later window start (Alpenglow) |
| replay_progcache_rooted | counter | Program cache entries rooted |
| replay_progcache_class_used {progcache_class="le128_kib"} | gauge | Program cache value slots in use, by size class (0-128 KiB) |
| replay_progcache_class_used {progcache_class="le512_kib"} | gauge | Program cache value slots in use, by size class (128-512 KiB) |
| replay_progcache_class_used {progcache_class="le1_mib"} | gauge | Program cache value slots in use, by size class (512 KiB-1 MiB) |
| replay_progcache_class_used {progcache_class="le2_mib"} | gauge | Program cache value slots in use, by size class (1-2 MiB) |
| replay_progcache_class_used {progcache_class="le4_mib"} | gauge | Program cache value slots in use, by size class (2-4 MiB) |
| replay_progcache_class_used {progcache_class="le11_mib"} | gauge | Program cache value slots in use, by size class (4-11 MiB) |
| replay_progcache_class_capacity {progcache_class="le128_kib"} | gauge | Program cache value capacity, by size class (0-128 KiB) |
| replay_progcache_class_capacity {progcache_class="le512_kib"} | gauge | Program cache value capacity, by size class (128-512 KiB) |
| replay_progcache_class_capacity {progcache_class="le1_mib"} | gauge | Program cache value capacity, by size class (512 KiB-1 MiB) |
| replay_progcache_class_capacity {progcache_class="le2_mib"} | gauge | Program cache value capacity, by size class (1-2 MiB) |
| replay_progcache_class_capacity {progcache_class="le4_mib"} | gauge | Program cache value capacity, by size class (2-4 MiB) |
| replay_progcache_class_capacity {progcache_class="le11_mib"} | gauge | Program cache value capacity, by size class (4-11 MiB) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le128_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le512_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le2_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le8_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le32_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le128_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le1_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| replay_accdb_account_readonly_acquired {accdb_cache_class="le10_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le128_b"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le512_b"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le2_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le8_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le32_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le128_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le1_mib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="le10_mib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| replay_accdb_account_evicted {accdb_cache_class="le128_b"} | counter | Accounts evicted from the account database cache to make space, by cache size class (0-128 B) |
| replay_accdb_account_evicted {accdb_cache_class="le512_b"} | counter | Accounts evicted from the account database cache to make space, by cache size class (129-512 B) |
| replay_accdb_account_evicted {accdb_cache_class="le2_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (513 B-2 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="le8_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (2-8 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="le32_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (8-32 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="le128_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (32-128 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="le1_mib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (128 KiB-1 MiB) |
| replay_accdb_account_evicted {accdb_cache_class="le10_mib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (1-10 MiB) |
| replay_accdb_account_new_committed {accdb_cache_class="le128_b"} | counter | New account versions committed to the index, by destination cache size class (0-128 B) |
| replay_accdb_account_new_committed {accdb_cache_class="le512_b"} | counter | New account versions committed to the index, by destination cache size class (129-512 B) |
| replay_accdb_account_new_committed {accdb_cache_class="le2_kib"} | counter | New account versions committed to the index, by destination cache size class (513 B-2 KiB) |
| replay_accdb_account_new_committed {accdb_cache_class="le8_kib"} | counter | New account versions committed to the index, by destination cache size class (2-8 KiB) |
| replay_accdb_account_new_committed {accdb_cache_class="le32_kib"} | counter | New account versions committed to the index, by destination cache size class (8-32 KiB) |
| replay_accdb_account_new_committed {accdb_cache_class="le128_kib"} | counter | New account versions committed to the index, by destination cache size class (32-128 KiB) |
| replay_accdb_account_new_committed {accdb_cache_class="le1_mib"} | counter | New account versions committed to the index, by destination cache size class (128 KiB-1 MiB) |
| replay_accdb_account_new_committed {accdb_cache_class="le10_mib"} | counter | New account versions committed to the index, by destination cache size class (1-10 MiB) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le128_b"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (0-128 B) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le512_b"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (129-512 B) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le2_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (513 B-2 KiB) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le8_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (2-8 KiB) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le32_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (8-32 KiB) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le128_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (32-128 KiB) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le1_mib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (128 KiB-1 MiB) |
| replay_accdb_account_overwrite_committed {accdb_cache_class="le10_mib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (1-10 MiB) |
| replay_accdb_account_cache_missed {accdb_cache_class="le128_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (0-128 B) |
| replay_accdb_account_cache_missed {accdb_cache_class="le512_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (129-512 B) |
| replay_accdb_account_cache_missed {accdb_cache_class="le2_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB) |
| replay_accdb_account_cache_missed {accdb_cache_class="le8_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (2-8 KiB) |
| replay_accdb_account_cache_missed {accdb_cache_class="le32_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (8-32 KiB) |
| replay_accdb_account_cache_missed {accdb_cache_class="le128_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (32-128 KiB) |
| replay_accdb_account_cache_missed {accdb_cache_class="le1_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB) |
| replay_accdb_account_cache_missed {accdb_cache_class="le10_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (1-10 MiB) |
| replay_accdb_account_waited | counter | Accounts that had to wait for another reader to finish loading them into the cache |
| replay_accdb_batch_acquired | counter | Batch acquire calls made by this tile |
| replay_accdb_cache_reserve_retried | counter | Spin retries while waiting to reserve cache lines for a transaction |
| replay_accdb_disk_read_bytes | counter | Bytes read from the account database file |
| replay_accdb_disk_read | counter | Reads from the account database file |
| replay_accdb_disk_written_bytes | counter | Bytes written to the account database file |
| replay_accdb_disk_written | counter | Writes to the account database file |
| replay_accdb_copied_bytes | counter | Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded |
| replay_is_runtime_loaded | gauge | Whether the runtime has loaded a bank and can replay |
Execrp Tile
| Metric | Type | Description |
|---|---|---|
| execrp_is_processing | gauge | Whether the tile is processing a replay task |
| execrp_txn_processed {txn_result="success"} | counter | Transactions loaded and executed, by result (Transaction executed successfully) |
| execrp_txn_processed {txn_result="instruction_error"} | counter | Transactions loaded and executed, by result (An error occurred while processing an instruction) |
| execrp_txn_processed {txn_result="account_not_found"} | counter | Transactions loaded and executed, by result (The transaction fee payer address was not found) |
| execrp_txn_processed {txn_result="program_account_not_found"} | counter | Transactions loaded and executed, by result (A program account referenced by the transaction was not found) |
| execrp_txn_processed {txn_result="insufficient_funds_for_fee"} | counter | Transactions loaded and executed, by result (The transaction fee payer did not have balance to pay the fee) |
| execrp_txn_processed {txn_result="invalid_account_for_fee"} | counter | Transactions loaded and executed, by result (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) |
| execrp_txn_processed {txn_result="already_processed"} | counter | Transactions loaded and executed, by result (The transaction has already been processed in a recent block) |
| execrp_txn_processed {txn_result="blockhash_not_found"} | counter | Transactions loaded and executed, by result (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) |
| execrp_txn_processed {txn_result="invalid_program_for_execution"} | counter | Transactions loaded and executed, by result (A program account referenced by the transaction was no executable.) |
| execrp_txn_processed {txn_result="address_lookup_table_not_found"} | counter | Transactions loaded and executed, by result (The transaction references an ALUT account that does not exist or is inactive) |
| execrp_txn_processed {txn_result="invalid_address_lookup_table_owner"} | counter | Transactions loaded and executed, by result (The transaction references an ALUT account that is not owned by the ALUT program account) |
| execrp_txn_processed {txn_result="invalid_address_lookup_table_data"} | counter | Transactions loaded and executed, by result (The transaction references an ALUT account that contains data which is not a valid ALUT) |
| execrp_txn_processed {txn_result="invalid_address_lookup_table_index"} | counter | Transactions loaded and executed, by result (The transaction references an account offset from the ALUT which does not exist) |
| execrp_txn_processed {txn_result="max_loaded_accounts_data_size_exceeded"} | counter | Transactions loaded and executed, by result (The total account data size of the loaded accounts exceeds the consensus limit) |
| execrp_txn_processed {txn_result="duplicate_instruction"} | counter | Transactions loaded and executed, by result (A compute budget program instruction was invoked more than once) |
| execrp_txn_processed {txn_result="invalid_loaded_accounts_data_size_limit"} | counter | Transactions loaded and executed, by result (The compute budget program was invoked and set the loaded accounts data size to zero) |
| execrp_txn_processed {txn_result="nonce_already_advanced"} | counter | Transactions loaded and executed, by result (The transaction references a nonce account that is already advanced) |
| execrp_txn_processed {txn_result="nonce_advance_failed"} | counter | Transactions loaded and executed, by result (The transaction is a nonce transaction but the advance instruction was not valid or failed) |
| execrp_txn_processed {txn_result="nonce_wrong_blockhash"} | counter | Transactions loaded and executed, by result (The transaction is a nonce transaction but the blockhash is not the correct one) |
| execrp_txn_processed {txn_result="unsupported_version"} | counter | Transactions loaded and executed, by result (The transaction is a message version that is not supported by the current feature set) |
| execrp_txn_processed {txn_result="sanitize_failure"} | counter | Transactions loaded and executed, by result (The process for sanitizing the transaction failed) |
| execrp_txn_processed {txn_result="account_in_use"} | counter | Transactions loaded and executed, by result (The transaction conflicts with another transaction in the microblock.) |
| execrp_txn_processed {txn_result="account_loaded_twice"} | counter | Transactions loaded and executed, by result (The transaction references the same account twice) |
| execrp_txn_processed {txn_result="signature_failure"} | counter | Transactions loaded and executed, by result (The transaction had an invalid signature) |
| execrp_txn_processed {txn_result="too_many_account_locks"} | counter | Transactions loaded and executed, by result (The transaction references too many accounts.) |
| execrp_txn_processed {txn_result="insufficient_funds_for_rent"} | counter | Transactions loaded and executed, by result (The transaction would leave an account with a lower balance than the rent-exempt minimum) |
| execrp_txn_processed {txn_result="unbalanced_transaction"} | counter | Transactions loaded and executed, by result (The total referenced account lamports before and after the transaction was unbalanced) |
| execrp_txn_processed {txn_result="bundle_peer"} | counter | Transactions loaded and executed, by result (The transaction was part of a bundle and an earlier transaction in the bundle failed) |
| execrp_txn_processed_by_version {txn_version="legacy"} | counter | Transactions processed, by transaction version (Legacy transaction format) |
| execrp_txn_processed_by_version {txn_version="v0"} | counter | Transactions processed, by transaction version (Version 0 transaction format) |
| execrp_txn_processed_by_version {txn_version="v1"} | counter | Transactions processed, by transaction version (Version 1 transaction format) |
| execrp_signature_checked | counter | Ed25519 signatures checked, including failures |
| execrp_poh_hash_computed | counter | PoH SHA-256 iterations computed |
| execrp_instruction_executed | counter | Top-level instructions executed |
| execrp_cpi_executed | counter | Cross-program invocations executed |
| execrp_account_lthash_skipped | counter | Committed writable accounts left byte-identical by their transaction, so their in-band lthash update was skipped |
| execrp_bank_switched | counter | Cache invalidations due to transaction execution from a different block |
| execrp_txn_regime_duration_nanos {txn_regime="load"} | counter | Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction checks and account loading) |
| execrp_txn_regime_duration_nanos {txn_regime="execute"} | counter | Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction execution, including VM setup) |
| execrp_txn_regime_duration_nanos {txn_regime="commit"} | counter | Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases (Transaction result commit) |
| execrp_vm_regime_duration_nanos {vm_regime="setup"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup) |
| execrp_vm_regime_duration_nanos {vm_regime="commit"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit) |
| execrp_vm_regime_duration_nanos {vm_regime="setup_cpi"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM setup (CPI)) |
| execrp_vm_regime_duration_nanos {vm_regime="commit_cpi"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM commit (CPI)) |
| execrp_vm_regime_duration_nanos {vm_regime="interpreter"} | counter | Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute (VM interpreter execution) |
| execrp_cu_consumed | counter | Compute units consumed by executed transactions |
| execrp_progcache_looked_up | counter | Program cache lookups; equals Hit + Miss + lookups of invalid program accounts |
| execrp_progcache_miss | counter | Program cache misses, each followed by an insert or a spill |
| execrp_progcache_waited | counter | Program cache lookups that waited on another tile's in-flight load of the same program |
| execrp_progcache_insert_retried | counter | Program cache inserts that found no free slot in their size class on the first try and retried after eviction |
| execrp_progcache_inserted_bytes | counter | Bytes inserted into the program cache |
| execrp_progcache_spilled_bytes | counter | Bytes loaded into program cache spill memory |
| execrp_progcache_evicted_bytes | counter | Bytes evicted from the program cache by insert eviction sweeps |
| execrp_progcache_duration_nanos | counter | Time in program cache lookups, including waits on peer loads and insert, eviction and ELF load on a miss; superset of ProgcacheLoadDurationNanos |
| execrp_progcache_loaded | counter | Program ELF loads and verifications for the program cache, including failed verifications and spill-backed loads |
| execrp_progcache_load_duration_nanos | counter | Time spent loading programs |
| execrp_progcache_hit {progcache_class="le128_kib"} | counter | Program cache hits, by size class (0-128 KiB) |
| execrp_progcache_hit {progcache_class="le512_kib"} | counter | Program cache hits, by size class (128-512 KiB) |
| execrp_progcache_hit {progcache_class="le1_mib"} | counter | Program cache hits, by size class (512 KiB-1 MiB) |
| execrp_progcache_hit {progcache_class="le2_mib"} | counter | Program cache hits, by size class (1-2 MiB) |
| execrp_progcache_hit {progcache_class="le4_mib"} | counter | Program cache hits, by size class (2-4 MiB) |
| execrp_progcache_hit {progcache_class="le11_mib"} | counter | Program cache hits, by size class (4-11 MiB) |
| execrp_progcache_inserted {progcache_class="le128_kib"} | counter | Programs inserted into the program cache, by size class (0-128 KiB) |
| execrp_progcache_inserted {progcache_class="le512_kib"} | counter | Programs inserted into the program cache, by size class (128-512 KiB) |
| execrp_progcache_inserted {progcache_class="le1_mib"} | counter | Programs inserted into the program cache, by size class (512 KiB-1 MiB) |
| execrp_progcache_inserted {progcache_class="le2_mib"} | counter | Programs inserted into the program cache, by size class (1-2 MiB) |
| execrp_progcache_inserted {progcache_class="le4_mib"} | counter | Programs inserted into the program cache, by size class (2-4 MiB) |
| execrp_progcache_inserted {progcache_class="le11_mib"} | counter | Programs inserted into the program cache, by size class (4-11 MiB) |
| execrp_progcache_evicted {progcache_class="le128_kib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (0-128 KiB) |
| execrp_progcache_evicted {progcache_class="le512_kib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (128-512 KiB) |
| execrp_progcache_evicted {progcache_class="le1_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (512 KiB-1 MiB) |
| execrp_progcache_evicted {progcache_class="le2_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (1-2 MiB) |
| execrp_progcache_evicted {progcache_class="le4_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (2-4 MiB) |
| execrp_progcache_evicted {progcache_class="le11_mib"} | counter | Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile) (4-11 MiB) |
| execrp_progcache_spilled {progcache_class="le128_kib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (0-128 KiB) |
| execrp_progcache_spilled {progcache_class="le512_kib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (128-512 KiB) |
| execrp_progcache_spilled {progcache_class="le1_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (512 KiB-1 MiB) |
| execrp_progcache_spilled {progcache_class="le2_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (1-2 MiB) |
| execrp_progcache_spilled {progcache_class="le4_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (2-4 MiB) |
| execrp_progcache_spilled {progcache_class="le11_mib"} | counter | Programs loaded into spill memory because no cache slot could be freed, by size class (4-11 MiB) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le128_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le512_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le2_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le8_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le32_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le128_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le1_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| execrp_accdb_account_readonly_acquired {accdb_cache_class="le10_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le128_b"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le512_b"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le2_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le8_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le32_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le128_kib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le1_mib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="le10_mib"} | counter | Writable accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| execrp_accdb_account_evicted {accdb_cache_class="le128_b"} | counter | Accounts evicted from the account database cache to make space, by cache size class (0-128 B) |
| execrp_accdb_account_evicted {accdb_cache_class="le512_b"} | counter | Accounts evicted from the account database cache to make space, by cache size class (129-512 B) |
| execrp_accdb_account_evicted {accdb_cache_class="le2_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (513 B-2 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="le8_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (2-8 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="le32_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (8-32 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="le128_kib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (32-128 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="le1_mib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_evicted {accdb_cache_class="le10_mib"} | counter | Accounts evicted from the account database cache to make space, by cache size class (1-10 MiB) |
| execrp_accdb_account_new_committed {accdb_cache_class="le128_b"} | counter | New account versions committed to the index, by destination cache size class (0-128 B) |
| execrp_accdb_account_new_committed {accdb_cache_class="le512_b"} | counter | New account versions committed to the index, by destination cache size class (129-512 B) |
| execrp_accdb_account_new_committed {accdb_cache_class="le2_kib"} | counter | New account versions committed to the index, by destination cache size class (513 B-2 KiB) |
| execrp_accdb_account_new_committed {accdb_cache_class="le8_kib"} | counter | New account versions committed to the index, by destination cache size class (2-8 KiB) |
| execrp_accdb_account_new_committed {accdb_cache_class="le32_kib"} | counter | New account versions committed to the index, by destination cache size class (8-32 KiB) |
| execrp_accdb_account_new_committed {accdb_cache_class="le128_kib"} | counter | New account versions committed to the index, by destination cache size class (32-128 KiB) |
| execrp_accdb_account_new_committed {accdb_cache_class="le1_mib"} | counter | New account versions committed to the index, by destination cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_new_committed {accdb_cache_class="le10_mib"} | counter | New account versions committed to the index, by destination cache size class (1-10 MiB) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le128_b"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (0-128 B) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le512_b"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (129-512 B) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le2_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (513 B-2 KiB) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le8_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (2-8 KiB) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le32_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (8-32 KiB) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le128_kib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (32-128 KiB) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le1_mib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_overwrite_committed {accdb_cache_class="le10_mib"} | counter | Account versions committed over an existing version of the same fork and generation, by destination cache size class (1-10 MiB) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le128_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (0-128 B) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le512_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (129-512 B) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le2_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le8_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (2-8 KiB) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le32_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (8-32 KiB) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le128_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (32-128 KiB) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le1_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_cache_missed {accdb_cache_class="le10_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (1-10 MiB) |
| execrp_accdb_account_waited | counter | Accounts that had to wait for another reader to finish loading them into the cache |
| execrp_accdb_batch_acquired | counter | Batch acquire calls made by this tile |
| execrp_accdb_cache_reserve_retried | counter | Spin retries while waiting to reserve cache lines for a transaction |
| execrp_accdb_disk_read_bytes | counter | Bytes read from the account database file |
| execrp_accdb_disk_read | counter | Reads from the account database file |
| execrp_accdb_disk_written_bytes | counter | Bytes written to the account database file |
| execrp_accdb_disk_written | counter | Writes to the account database file |
| execrp_accdb_copied_bytes | counter | Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded |
| execrp_txncache_query_miss | counter | Transaction cache queries by this tile that found a page not in memory |
| execrp_txncache_insert_miss | counter | Transaction cache inserts by this tile that found a page not in memory |
| execrp_txncache_operation_stalled | counter | Transaction cache queries and inserts by this tile that fell back to the exclusive lock, stalling every tile; pages they load are read while holding it |
| execrp_txncache_query_duration_nanos | histogram | Duration of one transaction cache query |
| execrp_txncache_insert_duration_nanos | histogram | Duration of one transaction cache insert |
Accdb Tile
| Metric | Type | Description |
|---|---|---|
| accdb_account_used | gauge | Account versions in the index; every unrooted fork version counts |
| accdb_account_capacity | gauge | Account versions the index can hold |
| accdb_account_relocated | counter | Accounts moved in the account database during compaction |
| accdb_disk_allocated_bytes | gauge | Allocated size of the account database file |
| accdb_disk_consumed_bytes | gauge | Bytes of the account database file written at least once, excluding never-written partition tail space |
| accdb_disk_used_bytes | gauge | Bytes of the account database file holding live data |
| accdb_is_compacting | gauge | Whether a compaction is currently in progress |
| accdb_compaction_requested | counter | Compactions requested on the account database |
| accdb_compaction_completed | counter | Compactions fully completed on the account database, less than or equal to the requested amount |
| accdb_account_relocated_bytes | counter | Bytes moved in the account database during compaction |
| accdb_account_preevicted {accdb_cache_class="le128_b"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (0-128 B) |
| accdb_account_preevicted {accdb_cache_class="le512_b"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (129-512 B) |
| accdb_account_preevicted {accdb_cache_class="le2_kib"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (513 B-2 KiB) |
| accdb_account_preevicted {accdb_cache_class="le8_kib"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (2-8 KiB) |
| accdb_account_preevicted {accdb_cache_class="le32_kib"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (8-32 KiB) |
| accdb_account_preevicted {accdb_cache_class="le128_kib"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (32-128 KiB) |
| accdb_account_preevicted {accdb_cache_class="le1_mib"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (128 KiB-1 MiB) |
| accdb_account_preevicted {accdb_cache_class="le10_mib"} | counter | Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted (1-10 MiB) |
| accdb_disk_read_bytes | counter | Bytes read from the account database file by compaction and writeback |
| accdb_disk_written_bytes | counter | Bytes written to the account database file by compaction and writeback |
| accdb_writeback_written | counter | Write syscalls that wrote evicted cache lines back to the account database file |
| accdb_compaction_written | counter | Writes that relocated a batch of records during compaction |
| accdb_account_version_removed | counter | Account versions removed from the account database index |
| accdb_cache_class_used {accdb_cache_class="le128_b"} | gauge | Cache lines in use, by size class (0-128 B) |
| accdb_cache_class_used {accdb_cache_class="le512_b"} | gauge | Cache lines in use, by size class (129-512 B) |
| accdb_cache_class_used {accdb_cache_class="le2_kib"} | gauge | Cache lines in use, by size class (513 B-2 KiB) |
| accdb_cache_class_used {accdb_cache_class="le8_kib"} | gauge | Cache lines in use, by size class (2-8 KiB) |
| accdb_cache_class_used {accdb_cache_class="le32_kib"} | gauge | Cache lines in use, by size class (8-32 KiB) |
| accdb_cache_class_used {accdb_cache_class="le128_kib"} | gauge | Cache lines in use, by size class (32-128 KiB) |
| accdb_cache_class_used {accdb_cache_class="le1_mib"} | gauge | Cache lines in use, by size class (128 KiB-1 MiB) |
| accdb_cache_class_used {accdb_cache_class="le10_mib"} | gauge | Cache lines in use, by size class (1-10 MiB) |
| accdb_cache_class_capacity {accdb_cache_class="le128_b"} | gauge | Cache line capacity, by size class (0-128 B) |
| accdb_cache_class_capacity {accdb_cache_class="le512_b"} | gauge | Cache line capacity, by size class (129-512 B) |
| accdb_cache_class_capacity {accdb_cache_class="le2_kib"} | gauge | Cache line capacity, by size class (513 B-2 KiB) |
| accdb_cache_class_capacity {accdb_cache_class="le8_kib"} | gauge | Cache line capacity, by size class (2-8 KiB) |
| accdb_cache_class_capacity {accdb_cache_class="le32_kib"} | gauge | Cache line capacity, by size class (8-32 KiB) |
| accdb_cache_class_capacity {accdb_cache_class="le128_kib"} | gauge | Cache line capacity, by size class (32-128 KiB) |
| accdb_cache_class_capacity {accdb_cache_class="le1_mib"} | gauge | Cache line capacity, by size class (128 KiB-1 MiB) |
| accdb_cache_class_capacity {accdb_cache_class="le10_mib"} | gauge | Cache line capacity, by size class (1-10 MiB) |
| accdb_cache_class_reserved {accdb_cache_class="le128_b"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (0-128 B) |
| accdb_cache_class_reserved {accdb_cache_class="le512_b"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (129-512 B) |
| accdb_cache_class_reserved {accdb_cache_class="le2_kib"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (513 B-2 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="le8_kib"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (2-8 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="le32_kib"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (8-32 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="le128_kib"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (32-128 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="le1_mib"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (128 KiB-1 MiB) |
| accdb_cache_class_reserved {accdb_cache_class="le10_mib"} | gauge | Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class (1-10 MiB) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le128_b"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (0-128 B) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le512_b"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (129-512 B) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le2_kib"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (513 B-2 KiB) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le8_kib"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (2-8 KiB) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le32_kib"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (8-32 KiB) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le128_kib"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (32-128 KiB) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le1_mib"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (128 KiB-1 MiB) |
| accdb_is_cache_class_reservation_tracked {accdb_cache_class="le10_mib"} | gauge | Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved (1-10 MiB) |
| accdb_cache_class_free_target {accdb_cache_class="le128_b"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (0-128 B) |
| accdb_cache_class_free_target {accdb_cache_class="le512_b"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (129-512 B) |
| accdb_cache_class_free_target {accdb_cache_class="le2_kib"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (513 B-2 KiB) |
| accdb_cache_class_free_target {accdb_cache_class="le8_kib"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (2-8 KiB) |
| accdb_cache_class_free_target {accdb_cache_class="le32_kib"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (8-32 KiB) |
| accdb_cache_class_free_target {accdb_cache_class="le128_kib"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (32-128 KiB) |
| accdb_cache_class_free_target {accdb_cache_class="le1_mib"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (128 KiB-1 MiB) |
| accdb_cache_class_free_target {accdb_cache_class="le10_mib"} | gauge | Free cache lines the background preevict pass tries to restore, by size class (1-10 MiB) |
| accdb_cache_class_free_low_water {accdb_cache_class="le128_b"} | gauge | Free cache line count below which the background preevict pass starts, by size class (0-128 B) |
| accdb_cache_class_free_low_water {accdb_cache_class="le512_b"} | gauge | Free cache line count below which the background preevict pass starts, by size class (129-512 B) |
| accdb_cache_class_free_low_water {accdb_cache_class="le2_kib"} | gauge | Free cache line count below which the background preevict pass starts, by size class (513 B-2 KiB) |
| accdb_cache_class_free_low_water {accdb_cache_class="le8_kib"} | gauge | Free cache line count below which the background preevict pass starts, by size class (2-8 KiB) |
| accdb_cache_class_free_low_water {accdb_cache_class="le32_kib"} | gauge | Free cache line count below which the background preevict pass starts, by size class (8-32 KiB) |
| accdb_cache_class_free_low_water {accdb_cache_class="le128_kib"} | gauge | Free cache line count below which the background preevict pass starts, by size class (32-128 KiB) |
| accdb_cache_class_free_low_water {accdb_cache_class="le1_mib"} | gauge | Free cache line count below which the background preevict pass starts, by size class (128 KiB-1 MiB) |
| accdb_cache_class_free_low_water {accdb_cache_class="le10_mib"} | gauge | Free cache line count below which the background preevict pass starts, by size class (1-10 MiB) |
| accdb_progcache_preevicted_bytes | counter | Bytes reclaimed from the program cache by the background preevict pass |
| accdb_progcache_preevicted {progcache_class="le128_kib"} | counter | Program cache slots reclaimed by the background preevict pass, by size class (0-128 KiB) |
| accdb_progcache_preevicted {progcache_class="le512_kib"} | counter | Program cache slots reclaimed by the background preevict pass, by size class (128-512 KiB) |
| accdb_progcache_preevicted {progcache_class="le1_mib"} | counter | Program cache slots reclaimed by the background preevict pass, by size class (512 KiB-1 MiB) |
| accdb_progcache_preevicted {progcache_class="le2_mib"} | counter | Program cache slots reclaimed by the background preevict pass, by size class (1-2 MiB) |
| accdb_progcache_preevicted {progcache_class="le4_mib"} | counter | Program cache slots reclaimed by the background preevict pass, by size class (2-4 MiB) |
| accdb_progcache_preevicted {progcache_class="le11_mib"} | counter | Program cache slots reclaimed by the background preevict pass, by size class (4-11 MiB) |
Diag Tile
| Metric | Type | Description |
|---|---|---|
| diag_bundle_state | gauge | Bundle subsystem state: 0=disconnected (All bundle tiles disconnected), 1=connecting (At least one bundle tile connecting, none connected or sleeping), 2=connected (At least one bundle tile connected), 3=sleeping (At least one bundle tile sleeping, none connected), 4=disabled (No bundle tiles configured) |
| diag_vote_state | gauge | Vote subsystem state: 0=disabled (Non-voting or no votor tile), 1=not_started (Votor tile not running or no votes cast yet), 2=delinquent (Vote distance exceeds threshold or vote stalled), 3=voting (Voting normally), 4=inadmissible (Voting, but the vote account fails the validator admission ticket filter) |
| diag_replay_state | gauge | Replay subsystem state: 0=disabled (No replay tile), 1=not_started (Replay tile not running or slots are zero), 2=behind (Replay lagging behind turbine or reset slot stalled), 3=running (Replay keeping up) |
| diag_turbine_state | gauge | Turbine subsystem state: 0=disabled (No shred or replay tiles), 1=not_started (Tiles not all running or turbine slot is zero), 2=stalled (Turbine slot not advancing), 3=repair_outpacing (Repair byte throughput exceeds turbine), 4=running (Turbine receiving normally) |
| diag_builder_state | gauge | External block builder subsystem state: 0=disabled (No block builder configured), 1=disconnected (Block builder disconnected), 2=connecting (Block builder connection in progress), 3=unhealthy (Connected, but the block builder is not in a usable state), 4=connected (Block builder connected and healthy) |
| diag_system_device_irq | counter | Device IRQs across all CPUs |
| diag_device_irq_on_tile_cpu | counter | Device IRQs on CPUs dedicated to pinned tiles; subset of SystemDeviceIrq |
| diag_system_softirq {softirq_type="net"} | counter | Soft-IRQs across all CPUs, by softirq type (NET_TX, NET_RX) |
| diag_system_softirq {softirq_type="disk"} | counter | Soft-IRQs across all CPUs, by softirq type (BLOCK) |
| diag_system_softirq {softirq_type="other"} | counter | Soft-IRQs across all CPUs, by softirq type (e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...) |
| diag_softirq_on_tile_cpu {softirq_type="net"} | counter | Soft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq (NET_TX, NET_RX) |
| diag_softirq_on_tile_cpu {softirq_type="disk"} | counter | Soft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq (BLOCK) |
| diag_softirq_on_tile_cpu {softirq_type="other"} | counter | Soft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq (e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...) |
| diag_cpu_package_millicelsius | gauge | Hottest CPU package temperature (k10temp Tctl or coretemp Package), 0 if no sensor |
| diag_nic_millicelsius | gauge | Hottest NIC ASIC temperature (mlx5 or ice), 0 if no sensor |
| diag_nvme_millicelsius | gauge | Hottest NVMe composite temperature, 0 if no sensor or the kernel lacks CONFIG_NVME_HWMON |
| diag_cpu_power_microwatts | gauge | CPU package power from the powercap (RAPL) energy counters, averaged over 5 s, 0 if unavailable |
| diag_system_power_microwatts | gauge | Whole-system power draw from the ACPI power meter, averaged over its interval, 0 if no meter |
| diag_memory_ecc_corrected | counter | Corrected memory ECC errors across all memory controllers since boot (EDAC), 0 if firmware handles errors instead |
| diag_memory_ecc_uncorrected | counter | Uncorrected memory ECC errors across all memory controllers since boot (EDAC), 0 if firmware handles errors instead |
| diag_system_load_centi {load_average_type="minute1"} | gauge | Load average over each window, times 100 (/proc/loadavg has two decimals) (1 minute) |
| diag_system_load_centi {load_average_type="minute5"} | gauge | Load average over each window, times 100 (/proc/loadavg has two decimals) (5 minutes) |
| diag_system_load_centi {load_average_type="minute15"} | gauge | Load average over each window, times 100 (/proc/loadavg has two decimals) (15 minutes) |
| diag_thread_runnable_count | gauge | Runnable threads on the host (/proc/loadavg) |
| diag_system_cpu_duration_nanos {system_cpu_state="user"} | counter | CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (User (including nice)) |
| diag_system_cpu_duration_nanos {system_cpu_state="system"} | counter | CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (System) |
| diag_system_cpu_duration_nanos {system_cpu_state="idle"} | counter | CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Idle) |
| diag_system_cpu_duration_nanos {system_cpu_state="io_wait"} | counter | CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Idle waiting on IO) |
| diag_system_cpu_duration_nanos {system_cpu_state="interrupt"} | counter | CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Hardirq and softirq handlers) |
| diag_system_cpu_duration_nanos {system_cpu_state="steal"} | counter | CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution (Stolen by the hypervisor) |
| diag_memory_available_bytes | gauge | Host memory available without swapping (/proc/meminfo available) |
| diag_memory_free_bytes | gauge | Host memory not in use at all (/proc/meminfo free) |
| diag_swap_in_bytes | counter | Bytes swapped in from disk since boot (pswpin) |
| diag_swap_out_bytes | counter | Bytes swapped out to disk since boot (pswpout) |
| diag_system_page_fault_major | counter | Major page faults across the host since boot (pgmajfault) |
| diag_oom_killed | counter | Processes killed by the kernel OOM killer since boot (oom_kill) |
| diag_cpu_pressure_duration_nanos | counter | Time at least one task stalled waiting for CPU since boot (PSI some), 0 if the kernel lacks PSI |
| diag_memory_pressure_duration_nanos | counter | Time at least one task stalled on memory reclaim or swap since boot (PSI some), 0 if the kernel lacks PSI |
| diag_memory_pressure_full_duration_nanos | counter | Time all non-idle tasks stalled on memory together since boot (PSI full), a subset of MemoryPressureDurationNanos |
| diag_io_pressure_duration_nanos | counter | Time at least one task stalled on IO since boot (PSI some), 0 if the kernel lacks PSI |
| diag_io_pressure_full_duration_nanos | counter | Time all non-idle tasks stalled on IO together since boot (PSI full), a subset of IoPressureDurationNanos |
| diag_accounts_disk_read_bytes | counter | Bytes read from the accounts disk since boot (/sys/block stat sectors read), 0 if not on a block device |
| diag_accounts_disk_written_bytes | counter | Bytes written to the accounts disk since boot (/sys/block stat sectors written), 0 if not on a block device |
| diag_accounts_disk_written | counter | Write requests the accounts disk completed since boot (/sys/block stat writes completed), 0 if not on a block device |
| diag_accounts_disk_write_duration_nanos | counter | Time write requests to the accounts disk spent from submission to completion since the diag tile started, summed over requests (/sys/block stat write ticks), 0 if not on a block device |
| diag_accounts_disk_busy_duration_nanos | counter | Time the accounts disk had at least one request in flight since the diag tile started (/sys/block stat io_ticks), 0 if not on a block device |
| diag_accounts_disk_request_in_flight_integral_nanos | counter | Integral of requests in flight at the accounts disk over time since the diag tile started (request-nanoseconds, /sys/block stat time_in_queue); divide by elapsed time for the average queue depth, 0 if not on a block device |
| diag_accounts_disk_request_in_flight | gauge | Requests the accounts disk had in flight at the last sample (/sys/block stat in_flight), 0 if not on a block device |
Event Tile
| Metric | Type | Description |
|---|---|---|
| event_conn_state | gauge | Telemetry server connection state: 0=disconnected, 1=connecting, 2=authenticating, 3=connected |
| event_queue_used | gauge | Events in the queue, sent-but-unacknowledged plus unsent |
| event_queue_unsent_count | gauge | Events in the event queue not yet sent to the event service |
| event_queue_evicted | counter | Oldest events overwritten because the queue was full |
| event_queue_used_bytes | gauge | Bytes used in the event queue |
| event_queue_capacity_bytes | gauge | Event queue capacity |
| event_event_tx | counter | Events written to the stream, including resends after reconnect |
| event_ack_rx | counter | Acknowledgements received; one ack may retire many events |
| event_last_acked_nonce | gauge | Nonce of the most recently acknowledged event, 0 if none |
| event_stream_tx_bytes | counter | Bytes sent to the event service |
| event_stream_rx_bytes | counter | Bytes received from the event service |
| event_auth_failed | counter | Authentication failures with the event service |
| event_response_malformed_rejected | counter | Malformed responses received from the event service |
| event_conn_attempted | counter | Connection attempts to the event service |
| event_handshake_timed_out | counter | Authentication handshakes that timed out |
| event_credit_stalled | counter | Reconnects forced because the event service stopped granting HTTP/2 flow-control credit while data was pending |
Gui Tile
| Metric | Type | Description |
|---|---|---|
| gui_conn_open | gauge | Open HTTP connections to the GUI service, excluding connections upgraded to WebSocket |
| gui_websocket_conn_open | gauge | Open WebSocket connections to the GUI service |
| gui_websocket_frame_tx | counter | WebSocket frames sent to all connections to the GUI service |
| gui_websocket_frame_rx | counter | WebSocket frames received from all connections to the GUI service |
| gui_conn_tx_bytes | counter | Bytes sent on all connections to the GUI service |
| gui_conn_rx_bytes | counter | Bytes received on all connections to the GUI service |
| gui_disk_allocated_bytes | gauge | Logical bytes allocated by the GUI database backing file |
| gui_db_used_bytes {gui_db_type="scheduler_counts"} | gauge | Bytes occupied by records present in the store ring (scheduler_counts) |
| gui_db_used_bytes {gui_db_type="tile_timers"} | gauge | Bytes occupied by records present in the store ring (tile_timers) |
| gui_db_used_bytes {gui_db_type="shred_events"} | gauge | Bytes occupied by records present in the store ring (shred_events) |
| gui_db_used_bytes {gui_db_type="txn_start"} | gauge | Bytes occupied by records present in the store ring (txn_start) |
| gui_db_used_bytes {gui_db_type="txn_end"} | gauge | Bytes occupied by records present in the store ring (txn_end) |
| gui_db_used_bytes {gui_db_type="tower"} | gauge | Bytes occupied by records present in the store ring (Retired: no longer a database, always zero) |
| gui_db_used_bytes {gui_db_type="slot"} | gauge | Bytes occupied by records present in the store ring (slot) |
| gui_db_used_bytes {gui_db_type="leader_slot"} | gauge | Bytes occupied by records present in the store ring (leader_slot) |
| gui_db_used_bytes {gui_db_type="epoch"} | gauge | Bytes occupied by records present in the store ring (epoch) |
| gui_db_used_bytes {gui_db_type="tile_stats"} | gauge | Bytes occupied by records present in the store ring (tile_stats) |
| gui_db_used_bytes {gui_db_type="txn_waterfall"} | gauge | Bytes occupied by records present in the store ring (txn_waterfall) |
| gui_db_used_bytes {gui_db_type="timeline_day"} | gauge | Bytes occupied by records present in the store ring (timeline_day) |
| gui_db_used_bytes {gui_db_type="replay_txn"} | gauge | Bytes occupied by records present in the store ring (replay_txn) |
| gui_db_used_bytes {gui_db_type="timeline_tile_day"} | gauge | Bytes occupied by records present in the store ring (timeline_tile_day) |
| gui_db_used_bytes {gui_db_type="fec_events"} | gauge | Bytes occupied by records present in the store ring (fec_events) |
| gui_db_used_bytes {gui_db_type="fec_event_archive"} | gauge | Bytes occupied by records present in the store ring (fec_event_archive) |
| gui_db_allocated_bytes {gui_db_type="scheduler_counts"} | gauge | Bytes of regions owned by each database ring; grows on demand (scheduler_counts) |
| gui_db_allocated_bytes {gui_db_type="tile_timers"} | gauge | Bytes of regions owned by each database ring; grows on demand (tile_timers) |
| gui_db_allocated_bytes {gui_db_type="shred_events"} | gauge | Bytes of regions owned by each database ring; grows on demand (shred_events) |
| gui_db_allocated_bytes {gui_db_type="txn_start"} | gauge | Bytes of regions owned by each database ring; grows on demand (txn_start) |
| gui_db_allocated_bytes {gui_db_type="txn_end"} | gauge | Bytes of regions owned by each database ring; grows on demand (txn_end) |
| gui_db_allocated_bytes {gui_db_type="tower"} | gauge | Bytes of regions owned by each database ring; grows on demand (Retired: no longer a database, always zero) |
| gui_db_allocated_bytes {gui_db_type="slot"} | gauge | Bytes of regions owned by each database ring; grows on demand (slot) |
| gui_db_allocated_bytes {gui_db_type="leader_slot"} | gauge | Bytes of regions owned by each database ring; grows on demand (leader_slot) |
| gui_db_allocated_bytes {gui_db_type="epoch"} | gauge | Bytes of regions owned by each database ring; grows on demand (epoch) |
| gui_db_allocated_bytes {gui_db_type="tile_stats"} | gauge | Bytes of regions owned by each database ring; grows on demand (tile_stats) |
| gui_db_allocated_bytes {gui_db_type="txn_waterfall"} | gauge | Bytes of regions owned by each database ring; grows on demand (txn_waterfall) |
| gui_db_allocated_bytes {gui_db_type="timeline_day"} | gauge | Bytes of regions owned by each database ring; grows on demand (timeline_day) |
| gui_db_allocated_bytes {gui_db_type="replay_txn"} | gauge | Bytes of regions owned by each database ring; grows on demand (replay_txn) |
| gui_db_allocated_bytes {gui_db_type="timeline_tile_day"} | gauge | Bytes of regions owned by each database ring; grows on demand (timeline_tile_day) |
| gui_db_allocated_bytes {gui_db_type="fec_events"} | gauge | Bytes of regions owned by each database ring; grows on demand (fec_events) |
| gui_db_allocated_bytes {gui_db_type="fec_event_archive"} | gauge | Bytes of regions owned by each database ring; grows on demand (fec_event_archive) |
| gui_db_free_bytes | gauge | Bytes of free regions not owned by any database ring |
| gui_db_record_used {gui_db_type="scheduler_counts"} | gauge | Records present in the store ring (scheduler_counts) |
| gui_db_record_used {gui_db_type="tile_timers"} | gauge | Records present in the store ring (tile_timers) |
| gui_db_record_used {gui_db_type="shred_events"} | gauge | Records present in the store ring (shred_events) |
| gui_db_record_used {gui_db_type="txn_start"} | gauge | Records present in the store ring (txn_start) |
| gui_db_record_used {gui_db_type="txn_end"} | gauge | Records present in the store ring (txn_end) |
| gui_db_record_used {gui_db_type="tower"} | gauge | Records present in the store ring (Retired: no longer a database, always zero) |
| gui_db_record_used {gui_db_type="slot"} | gauge | Records present in the store ring (slot) |
| gui_db_record_used {gui_db_type="leader_slot"} | gauge | Records present in the store ring (leader_slot) |
| gui_db_record_used {gui_db_type="epoch"} | gauge | Records present in the store ring (epoch) |
| gui_db_record_used {gui_db_type="tile_stats"} | gauge | Records present in the store ring (tile_stats) |
| gui_db_record_used {gui_db_type="txn_waterfall"} | gauge | Records present in the store ring (txn_waterfall) |
| gui_db_record_used {gui_db_type="timeline_day"} | gauge | Records present in the store ring (timeline_day) |
| gui_db_record_used {gui_db_type="replay_txn"} | gauge | Records present in the store ring (replay_txn) |
| gui_db_record_used {gui_db_type="timeline_tile_day"} | gauge | Records present in the store ring (timeline_tile_day) |
| gui_db_record_used {gui_db_type="fec_events"} | gauge | Records present in the store ring (fec_events) |
| gui_db_record_used {gui_db_type="fec_event_archive"} | gauge | Records present in the store ring (fec_event_archive) |
| gui_db_record_allocated {gui_db_type="scheduler_counts"} | gauge | Record capacity of regions owned by each database ring; grows on demand (scheduler_counts) |
| gui_db_record_allocated {gui_db_type="tile_timers"} | gauge | Record capacity of regions owned by each database ring; grows on demand (tile_timers) |
| gui_db_record_allocated {gui_db_type="shred_events"} | gauge | Record capacity of regions owned by each database ring; grows on demand (shred_events) |
| gui_db_record_allocated {gui_db_type="txn_start"} | gauge | Record capacity of regions owned by each database ring; grows on demand (txn_start) |
| gui_db_record_allocated {gui_db_type="txn_end"} | gauge | Record capacity of regions owned by each database ring; grows on demand (txn_end) |
| gui_db_record_allocated {gui_db_type="tower"} | gauge | Record capacity of regions owned by each database ring; grows on demand (Retired: no longer a database, always zero) |
| gui_db_record_allocated {gui_db_type="slot"} | gauge | Record capacity of regions owned by each database ring; grows on demand (slot) |
| gui_db_record_allocated {gui_db_type="leader_slot"} | gauge | Record capacity of regions owned by each database ring; grows on demand (leader_slot) |
| gui_db_record_allocated {gui_db_type="epoch"} | gauge | Record capacity of regions owned by each database ring; grows on demand (epoch) |
| gui_db_record_allocated {gui_db_type="tile_stats"} | gauge | Record capacity of regions owned by each database ring; grows on demand (tile_stats) |
| gui_db_record_allocated {gui_db_type="txn_waterfall"} | gauge | Record capacity of regions owned by each database ring; grows on demand (txn_waterfall) |
| gui_db_record_allocated {gui_db_type="timeline_day"} | gauge | Record capacity of regions owned by each database ring; grows on demand (timeline_day) |
| gui_db_record_allocated {gui_db_type="replay_txn"} | gauge | Record capacity of regions owned by each database ring; grows on demand (replay_txn) |
| gui_db_record_allocated {gui_db_type="timeline_tile_day"} | gauge | Record capacity of regions owned by each database ring; grows on demand (timeline_tile_day) |
| gui_db_record_allocated {gui_db_type="fec_events"} | gauge | Record capacity of regions owned by each database ring; grows on demand (fec_events) |
| gui_db_record_allocated {gui_db_type="fec_event_archive"} | gauge | Record capacity of regions owned by each database ring; grows on demand (fec_event_archive) |
| gui_db_kv_queried {gui_db_type="scheduler_counts"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (scheduler_counts) |
| gui_db_kv_queried {gui_db_type="tile_timers"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (tile_timers) |
| gui_db_kv_queried {gui_db_type="shred_events"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (shred_events) |
| gui_db_kv_queried {gui_db_type="txn_start"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (txn_start) |
| gui_db_kv_queried {gui_db_type="txn_end"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (txn_end) |
| gui_db_kv_queried {gui_db_type="tower"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (Retired: no longer a database, always zero) |
| gui_db_kv_queried {gui_db_type="slot"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (slot) |
| gui_db_kv_queried {gui_db_type="leader_slot"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (leader_slot) |
| gui_db_kv_queried {gui_db_type="epoch"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (epoch) |
| gui_db_kv_queried {gui_db_type="tile_stats"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (tile_stats) |
| gui_db_kv_queried {gui_db_type="txn_waterfall"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (txn_waterfall) |
| gui_db_kv_queried {gui_db_type="timeline_day"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (timeline_day) |
| gui_db_kv_queried {gui_db_type="replay_txn"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (replay_txn) |
| gui_db_kv_queried {gui_db_type="timeline_tile_day"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (timeline_tile_day) |
| gui_db_kv_queried {gui_db_type="fec_events"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (fec_events) |
| gui_db_kv_queried {gui_db_type="fec_event_archive"} | counter | Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database (fec_event_archive) |
| gui_db_time_series_record_appended {gui_db_type="scheduler_counts"} | counter | Time-series records appended, by database (scheduler_counts) |
| gui_db_time_series_record_appended {gui_db_type="tile_timers"} | counter | Time-series records appended, by database (tile_timers) |
| gui_db_time_series_record_appended {gui_db_type="shred_events"} | counter | Time-series records appended, by database (shred_events) |
| gui_db_time_series_record_appended {gui_db_type="txn_start"} | counter | Time-series records appended, by database (txn_start) |
| gui_db_time_series_record_appended {gui_db_type="txn_end"} | counter | Time-series records appended, by database (txn_end) |
| gui_db_time_series_record_appended {gui_db_type="tower"} | counter | Time-series records appended, by database (Retired: no longer a database, always zero) |
| gui_db_time_series_record_appended {gui_db_type="slot"} | counter | Time-series records appended, by database (slot) |
| gui_db_time_series_record_appended {gui_db_type="leader_slot"} | counter | Time-series records appended, by database (leader_slot) |
| gui_db_time_series_record_appended {gui_db_type="epoch"} | counter | Time-series records appended, by database (epoch) |
| gui_db_time_series_record_appended {gui_db_type="tile_stats"} | counter | Time-series records appended, by database (tile_stats) |
| gui_db_time_series_record_appended {gui_db_type="txn_waterfall"} | counter | Time-series records appended, by database (txn_waterfall) |
| gui_db_time_series_record_appended {gui_db_type="timeline_day"} | counter | Time-series records appended, by database (timeline_day) |
| gui_db_time_series_record_appended {gui_db_type="replay_txn"} | counter | Time-series records appended, by database (replay_txn) |
| gui_db_time_series_record_appended {gui_db_type="timeline_tile_day"} | counter | Time-series records appended, by database (timeline_tile_day) |
| gui_db_time_series_record_appended {gui_db_type="fec_events"} | counter | Time-series records appended, by database (fec_events) |
| gui_db_time_series_record_appended {gui_db_type="fec_event_archive"} | counter | Time-series records appended, by database (fec_event_archive) |
| gui_db_time_series_scanned {gui_db_type="scheduler_counts"} | counter | Time-series scans, by database (scheduler_counts) |
| gui_db_time_series_scanned {gui_db_type="tile_timers"} | counter | Time-series scans, by database (tile_timers) |
| gui_db_time_series_scanned {gui_db_type="shred_events"} | counter | Time-series scans, by database (shred_events) |
| gui_db_time_series_scanned {gui_db_type="txn_start"} | counter | Time-series scans, by database (txn_start) |
| gui_db_time_series_scanned {gui_db_type="txn_end"} | counter | Time-series scans, by database (txn_end) |
| gui_db_time_series_scanned {gui_db_type="tower"} | counter | Time-series scans, by database (Retired: no longer a database, always zero) |
| gui_db_time_series_scanned {gui_db_type="slot"} | counter | Time-series scans, by database (slot) |
| gui_db_time_series_scanned {gui_db_type="leader_slot"} | counter | Time-series scans, by database (leader_slot) |
| gui_db_time_series_scanned {gui_db_type="epoch"} | counter | Time-series scans, by database (epoch) |
| gui_db_time_series_scanned {gui_db_type="tile_stats"} | counter | Time-series scans, by database (tile_stats) |
| gui_db_time_series_scanned {gui_db_type="txn_waterfall"} | counter | Time-series scans, by database (txn_waterfall) |
| gui_db_time_series_scanned {gui_db_type="timeline_day"} | counter | Time-series scans, by database (timeline_day) |
| gui_db_time_series_scanned {gui_db_type="replay_txn"} | counter | Time-series scans, by database (replay_txn) |
| gui_db_time_series_scanned {gui_db_type="timeline_tile_day"} | counter | Time-series scans, by database (timeline_tile_day) |
| gui_db_time_series_scanned {gui_db_type="fec_events"} | counter | Time-series scans, by database (fec_events) |
| gui_db_time_series_scanned {gui_db_type="fec_event_archive"} | counter | Time-series scans, by database (fec_event_archive) |
| gui_db_time_series_record_read {gui_db_type="scheduler_counts"} | counter | Time-series records returned by scans, by database (scheduler_counts) |
| gui_db_time_series_record_read {gui_db_type="tile_timers"} | counter | Time-series records returned by scans, by database (tile_timers) |
| gui_db_time_series_record_read {gui_db_type="shred_events"} | counter | Time-series records returned by scans, by database (shred_events) |
| gui_db_time_series_record_read {gui_db_type="txn_start"} | counter | Time-series records returned by scans, by database (txn_start) |
| gui_db_time_series_record_read {gui_db_type="txn_end"} | counter | Time-series records returned by scans, by database (txn_end) |
| gui_db_time_series_record_read {gui_db_type="tower"} | counter | Time-series records returned by scans, by database (Retired: no longer a database, always zero) |
| gui_db_time_series_record_read {gui_db_type="slot"} | counter | Time-series records returned by scans, by database (slot) |
| gui_db_time_series_record_read {gui_db_type="leader_slot"} | counter | Time-series records returned by scans, by database (leader_slot) |
| gui_db_time_series_record_read {gui_db_type="epoch"} | counter | Time-series records returned by scans, by database (epoch) |
| gui_db_time_series_record_read {gui_db_type="tile_stats"} | counter | Time-series records returned by scans, by database (tile_stats) |
| gui_db_time_series_record_read {gui_db_type="txn_waterfall"} | counter | Time-series records returned by scans, by database (txn_waterfall) |
| gui_db_time_series_record_read {gui_db_type="timeline_day"} | counter | Time-series records returned by scans, by database (timeline_day) |
| gui_db_time_series_record_read {gui_db_type="replay_txn"} | counter | Time-series records returned by scans, by database (replay_txn) |
| gui_db_time_series_record_read {gui_db_type="timeline_tile_day"} | counter | Time-series records returned by scans, by database (timeline_tile_day) |
| gui_db_time_series_record_read {gui_db_type="fec_events"} | counter | Time-series records returned by scans, by database (fec_events) |
| gui_db_time_series_record_read {gui_db_type="fec_event_archive"} | counter | Time-series records returned by scans, by database (fec_event_archive) |
| gui_db_record_evicted {gui_db_type="scheduler_counts"} | counter | Records evicted from the database, by database (scheduler_counts) |
| gui_db_record_evicted {gui_db_type="tile_timers"} | counter | Records evicted from the database, by database (tile_timers) |
| gui_db_record_evicted {gui_db_type="shred_events"} | counter | Records evicted from the database, by database (shred_events) |
| gui_db_record_evicted {gui_db_type="txn_start"} | counter | Records evicted from the database, by database (txn_start) |
| gui_db_record_evicted {gui_db_type="txn_end"} | counter | Records evicted from the database, by database (txn_end) |
| gui_db_record_evicted {gui_db_type="tower"} | counter | Records evicted from the database, by database (Retired: no longer a database, always zero) |
| gui_db_record_evicted {gui_db_type="slot"} | counter | Records evicted from the database, by database (slot) |
| gui_db_record_evicted {gui_db_type="leader_slot"} | counter | Records evicted from the database, by database (leader_slot) |
| gui_db_record_evicted {gui_db_type="epoch"} | counter | Records evicted from the database, by database (epoch) |
| gui_db_record_evicted {gui_db_type="tile_stats"} | counter | Records evicted from the database, by database (tile_stats) |
| gui_db_record_evicted {gui_db_type="txn_waterfall"} | counter | Records evicted from the database, by database (txn_waterfall) |
| gui_db_record_evicted {gui_db_type="timeline_day"} | counter | Records evicted from the database, by database (timeline_day) |
| gui_db_record_evicted {gui_db_type="replay_txn"} | counter | Records evicted from the database, by database (replay_txn) |
| gui_db_record_evicted {gui_db_type="timeline_tile_day"} | counter | Records evicted from the database, by database (timeline_tile_day) |
| gui_db_record_evicted {gui_db_type="fec_events"} | counter | Records evicted from the database, by database (fec_events) |
| gui_db_record_evicted {gui_db_type="fec_event_archive"} | counter | Records evicted from the database, by database (fec_event_archive) |
| gui_db_eviction_pass_completed {gui_db_type="scheduler_counts"} | counter | Eviction passes that removed at least one record, by database (scheduler_counts) |
| gui_db_eviction_pass_completed {gui_db_type="tile_timers"} | counter | Eviction passes that removed at least one record, by database (tile_timers) |
| gui_db_eviction_pass_completed {gui_db_type="shred_events"} | counter | Eviction passes that removed at least one record, by database (shred_events) |
| gui_db_eviction_pass_completed {gui_db_type="txn_start"} | counter | Eviction passes that removed at least one record, by database (txn_start) |
| gui_db_eviction_pass_completed {gui_db_type="txn_end"} | counter | Eviction passes that removed at least one record, by database (txn_end) |
| gui_db_eviction_pass_completed {gui_db_type="tower"} | counter | Eviction passes that removed at least one record, by database (Retired: no longer a database, always zero) |
| gui_db_eviction_pass_completed {gui_db_type="slot"} | counter | Eviction passes that removed at least one record, by database (slot) |
| gui_db_eviction_pass_completed {gui_db_type="leader_slot"} | counter | Eviction passes that removed at least one record, by database (leader_slot) |
| gui_db_eviction_pass_completed {gui_db_type="epoch"} | counter | Eviction passes that removed at least one record, by database (epoch) |
| gui_db_eviction_pass_completed {gui_db_type="tile_stats"} | counter | Eviction passes that removed at least one record, by database (tile_stats) |
| gui_db_eviction_pass_completed {gui_db_type="txn_waterfall"} | counter | Eviction passes that removed at least one record, by database (txn_waterfall) |
| gui_db_eviction_pass_completed {gui_db_type="timeline_day"} | counter | Eviction passes that removed at least one record, by database (timeline_day) |
| gui_db_eviction_pass_completed {gui_db_type="replay_txn"} | counter | Eviction passes that removed at least one record, by database (replay_txn) |
| gui_db_eviction_pass_completed {gui_db_type="timeline_tile_day"} | counter | Eviction passes that removed at least one record, by database (timeline_tile_day) |
| gui_db_eviction_pass_completed {gui_db_type="fec_events"} | counter | Eviction passes that removed at least one record, by database (fec_events) |
| gui_db_eviction_pass_completed {gui_db_type="fec_event_archive"} | counter | Eviction passes that removed at least one record, by database (fec_event_archive) |
| gui_db_region_claimed {gui_db_type="scheduler_counts"} | counter | Regions claimed from the free list, by database (scheduler_counts) |
| gui_db_region_claimed {gui_db_type="tile_timers"} | counter | Regions claimed from the free list, by database (tile_timers) |
| gui_db_region_claimed {gui_db_type="shred_events"} | counter | Regions claimed from the free list, by database (shred_events) |
| gui_db_region_claimed {gui_db_type="txn_start"} | counter | Regions claimed from the free list, by database (txn_start) |
| gui_db_region_claimed {gui_db_type="txn_end"} | counter | Regions claimed from the free list, by database (txn_end) |
| gui_db_region_claimed {gui_db_type="tower"} | counter | Regions claimed from the free list, by database (Retired: no longer a database, always zero) |
| gui_db_region_claimed {gui_db_type="slot"} | counter | Regions claimed from the free list, by database (slot) |
| gui_db_region_claimed {gui_db_type="leader_slot"} | counter | Regions claimed from the free list, by database (leader_slot) |
| gui_db_region_claimed {gui_db_type="epoch"} | counter | Regions claimed from the free list, by database (epoch) |
| gui_db_region_claimed {gui_db_type="tile_stats"} | counter | Regions claimed from the free list, by database (tile_stats) |
| gui_db_region_claimed {gui_db_type="txn_waterfall"} | counter | Regions claimed from the free list, by database (txn_waterfall) |
| gui_db_region_claimed {gui_db_type="timeline_day"} | counter | Regions claimed from the free list, by database (timeline_day) |
| gui_db_region_claimed {gui_db_type="replay_txn"} | counter | Regions claimed from the free list, by database (replay_txn) |
| gui_db_region_claimed {gui_db_type="timeline_tile_day"} | counter | Regions claimed from the free list, by database (timeline_tile_day) |
| gui_db_region_claimed {gui_db_type="fec_events"} | counter | Regions claimed from the free list, by database (fec_events) |
| gui_db_region_claimed {gui_db_type="fec_event_archive"} | counter | Regions claimed from the free list, by database (fec_event_archive) |
| gui_db_region_released {gui_db_type="scheduler_counts"} | counter | Regions returned to the free list, by database (scheduler_counts) |
| gui_db_region_released {gui_db_type="tile_timers"} | counter | Regions returned to the free list, by database (tile_timers) |
| gui_db_region_released {gui_db_type="shred_events"} | counter | Regions returned to the free list, by database (shred_events) |
| gui_db_region_released {gui_db_type="txn_start"} | counter | Regions returned to the free list, by database (txn_start) |
| gui_db_region_released {gui_db_type="txn_end"} | counter | Regions returned to the free list, by database (txn_end) |
| gui_db_region_released {gui_db_type="tower"} | counter | Regions returned to the free list, by database (Retired: no longer a database, always zero) |
| gui_db_region_released {gui_db_type="slot"} | counter | Regions returned to the free list, by database (slot) |
| gui_db_region_released {gui_db_type="leader_slot"} | counter | Regions returned to the free list, by database (leader_slot) |
| gui_db_region_released {gui_db_type="epoch"} | counter | Regions returned to the free list, by database (epoch) |
| gui_db_region_released {gui_db_type="tile_stats"} | counter | Regions returned to the free list, by database (tile_stats) |
| gui_db_region_released {gui_db_type="txn_waterfall"} | counter | Regions returned to the free list, by database (txn_waterfall) |
| gui_db_region_released {gui_db_type="timeline_day"} | counter | Regions returned to the free list, by database (timeline_day) |
| gui_db_region_released {gui_db_type="replay_txn"} | counter | Regions returned to the free list, by database (replay_txn) |
| gui_db_region_released {gui_db_type="timeline_tile_day"} | counter | Regions returned to the free list, by database (timeline_tile_day) |
| gui_db_region_released {gui_db_type="fec_events"} | counter | Regions returned to the free list, by database (fec_events) |
| gui_db_region_released {gui_db_type="fec_event_archive"} | counter | Regions returned to the free list, by database (fec_event_archive) |
| gui_db_record_dropped {gui_db_type="scheduler_counts"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (scheduler_counts) |
| gui_db_record_dropped {gui_db_type="tile_timers"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (tile_timers) |
| gui_db_record_dropped {gui_db_type="shred_events"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (shred_events) |
| gui_db_record_dropped {gui_db_type="txn_start"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (txn_start) |
| gui_db_record_dropped {gui_db_type="txn_end"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (txn_end) |
| gui_db_record_dropped {gui_db_type="tower"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (Retired: no longer a database, always zero) |
| gui_db_record_dropped {gui_db_type="slot"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (slot) |
| gui_db_record_dropped {gui_db_type="leader_slot"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (leader_slot) |
| gui_db_record_dropped {gui_db_type="epoch"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (epoch) |
| gui_db_record_dropped {gui_db_type="tile_stats"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (tile_stats) |
| gui_db_record_dropped {gui_db_type="txn_waterfall"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (txn_waterfall) |
| gui_db_record_dropped {gui_db_type="timeline_day"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (timeline_day) |
| gui_db_record_dropped {gui_db_type="replay_txn"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (replay_txn) |
| gui_db_record_dropped {gui_db_type="timeline_tile_day"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (timeline_tile_day) |
| gui_db_record_dropped {gui_db_type="fec_events"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (fec_events) |
| gui_db_record_dropped {gui_db_type="fec_event_archive"} | counter | Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database (fec_event_archive) |
| gui_db_forced_evict_write_completed {gui_db_type="scheduler_counts"} | counter | Writes that evicted records before succeeding, by database (scheduler_counts) |
| gui_db_forced_evict_write_completed {gui_db_type="tile_timers"} | counter | Writes that evicted records before succeeding, by database (tile_timers) |
| gui_db_forced_evict_write_completed {gui_db_type="shred_events"} | counter | Writes that evicted records before succeeding, by database (shred_events) |
| gui_db_forced_evict_write_completed {gui_db_type="txn_start"} | counter | Writes that evicted records before succeeding, by database (txn_start) |
| gui_db_forced_evict_write_completed {gui_db_type="txn_end"} | counter | Writes that evicted records before succeeding, by database (txn_end) |
| gui_db_forced_evict_write_completed {gui_db_type="tower"} | counter | Writes that evicted records before succeeding, by database (Retired: no longer a database, always zero) |
| gui_db_forced_evict_write_completed {gui_db_type="slot"} | counter | Writes that evicted records before succeeding, by database (slot) |
| gui_db_forced_evict_write_completed {gui_db_type="leader_slot"} | counter | Writes that evicted records before succeeding, by database (leader_slot) |
| gui_db_forced_evict_write_completed {gui_db_type="epoch"} | counter | Writes that evicted records before succeeding, by database (epoch) |
| gui_db_forced_evict_write_completed {gui_db_type="tile_stats"} | counter | Writes that evicted records before succeeding, by database (tile_stats) |
| gui_db_forced_evict_write_completed {gui_db_type="txn_waterfall"} | counter | Writes that evicted records before succeeding, by database (txn_waterfall) |
| gui_db_forced_evict_write_completed {gui_db_type="timeline_day"} | counter | Writes that evicted records before succeeding, by database (timeline_day) |
| gui_db_forced_evict_write_completed {gui_db_type="replay_txn"} | counter | Writes that evicted records before succeeding, by database (replay_txn) |
| gui_db_forced_evict_write_completed {gui_db_type="timeline_tile_day"} | counter | Writes that evicted records before succeeding, by database (timeline_tile_day) |
| gui_db_forced_evict_write_completed {gui_db_type="fec_events"} | counter | Writes that evicted records before succeeding, by database (fec_events) |
| gui_db_forced_evict_write_completed {gui_db_type="fec_event_archive"} | counter | Writes that evicted records before succeeding, by database (fec_event_archive) |
Metric Tile
| Metric | Type | Description |
|---|---|---|
| metric_boot_time | gauge | Wall-clock time the validator started |
| metric_conn_open | gauge | Open HTTP connections to the Prometheus endpoint |
| metric_conn_tx_bytes | counter | Bytes sent in responses from the Prometheus endpoint |
| metric_conn_rx_bytes | counter | Bytes received in requests to the Prometheus endpoint |
Rpc Tile
| Metric | Type | Description |
|---|---|---|
| rpc_request_duration_nanos | histogram | Time to handle one HTTP request or WebSocket message, excluding socket I/O |
| rpc_request_handled {rpc_method="genesis"} | counter | RPC requests handled, by method, before parameter validation (GET /genesis.tar.bz2) |
| rpc_request_handled {rpc_method="unknown"} | counter | RPC requests handled, by method, before parameter validation (Unknown method) |
| rpc_request_handled {rpc_method="get_account_info"} | counter | RPC requests handled, by method, before parameter validation (getAccountInfo) |
| rpc_request_handled {rpc_method="get_balance"} | counter | RPC requests handled, by method, before parameter validation (getBalance) |
| rpc_request_handled {rpc_method="get_block_height"} | counter | RPC requests handled, by method, before parameter validation (getBlockHeight) |
| rpc_request_handled {rpc_method="get_cluster_nodes"} | counter | RPC requests handled, by method, before parameter validation (getClusterNodes) |
| rpc_request_handled {rpc_method="get_epoch_info"} | counter | RPC requests handled, by method, before parameter validation (getEpochInfo) |
| rpc_request_handled {rpc_method="get_genesis_hash"} | counter | RPC requests handled, by method, before parameter validation (getGenesisHash) |
| rpc_request_handled {rpc_method="get_health"} | counter | RPC requests handled, by method, before parameter validation (getHealth) |
| rpc_request_handled {rpc_method="get_identity"} | counter | RPC requests handled, by method, before parameter validation (getIdentity) |
| rpc_request_handled {rpc_method="get_inflation_governor"} | counter | RPC requests handled, by method, before parameter validation (getInflationGovernor) |
| rpc_request_handled {rpc_method="get_latest_blockhash"} | counter | RPC requests handled, by method, before parameter validation (getLatestBlockhash) |
| rpc_request_handled {rpc_method="get_minimum_balance_for_rent_exemption"} | counter | RPC requests handled, by method, before parameter validation (getMinimumBalanceForRentExemption) |
| rpc_request_handled {rpc_method="get_multiple_accounts"} | counter | RPC requests handled, by method, before parameter validation (getMultipleAccounts) |
| rpc_request_handled {rpc_method="get_slot"} | counter | RPC requests handled, by method, before parameter validation (getSlot) |
| rpc_request_handled {rpc_method="get_transaction_count"} | counter | RPC requests handled, by method, before parameter validation (getTransactionCount) |
| rpc_request_handled {rpc_method="get_version"} | counter | RPC requests handled, by method, before parameter validation (getVersion) |
| rpc_request_handled {rpc_method="get_max_retransmit_slot"} | counter | RPC requests handled, by method, before parameter validation (getMaxRetransmitSlot) |
| rpc_request_handled {rpc_method="get_slot_leader"} | counter | RPC requests handled, by method, before parameter validation (getSlotLeader) |
| rpc_request_handled {rpc_method="get_slot_leaders"} | counter | RPC requests handled, by method, before parameter validation (getSlotLeaders) |
| rpc_request_handled {rpc_method="get_leader_schedule"} | counter | RPC requests handled, by method, before parameter validation (getLeaderSchedule) |
| rpc_request_handled {rpc_method="get_ag_genesis_cert"} | counter | RPC requests handled, by method, before parameter validation (getAgGenesisCert) |
| rpc_request_handled {rpc_method="slot_subscribe"} | counter | RPC requests handled, by method, before parameter validation (slotSubscribe) |
| rpc_request_handled {rpc_method="slot_unsubscribe"} | counter | RPC requests handled, by method, before parameter validation (slotUnsubscribe) |
| rpc_request_handled {rpc_method="vote_subscribe"} | counter | RPC requests handled, by method, before parameter validation (voteSubscribe) |
| rpc_request_handled {rpc_method="vote_unsubscribe"} | counter | RPC requests handled, by method, before parameter validation (voteUnsubscribe) |
| rpc_request_handled {rpc_method="unimplemented"} | counter | RPC requests handled, by method, before parameter validation (Known method not implemented by Firedancer) |
| rpc_request_handled {rpc_method="invalid"} | counter | RPC requests handled, by method, before parameter validation (Malformed JSON-RPC request) |
| rpc_request_handled {rpc_method="health"} | counter | RPC requests handled, by method, before parameter validation (GET /health) |
| rpc_conn_open | gauge | Open HTTP connections to the RPC service |
| rpc_websocket_conn_open | gauge | Open WebSocket connections to the RPC service |
| rpc_websocket_subscriber_count {rpc_notification_type="vote"} | gauge | WebSocket connections subscribed to each notification type (vote) |
| rpc_websocket_subscriber_count {rpc_notification_type="slot"} | gauge | WebSocket connections subscribed to each notification type (slot) |
| rpc_websocket_notification_published {rpc_notification_type="vote"} | counter | Notifications published to at least one subscriber, by notification type (vote) |
| rpc_websocket_notification_published {rpc_notification_type="slot"} | counter | Notifications published to at least one subscriber, by notification type (slot) |
| rpc_websocket_notification_tx {rpc_notification_type="vote"} | counter | Notifications sent, counting each subscriber, by notification type (vote) |
| rpc_websocket_notification_tx {rpc_notification_type="slot"} | counter | Notifications sent, counting each subscriber, by notification type (slot) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le128_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (0-128 B) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le512_b"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (129-512 B) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le2_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (513 B-2 KiB) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le8_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (2-8 KiB) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le32_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (8-32 KiB) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le128_kib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (32-128 KiB) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le1_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (128 KiB-1 MiB) |
| rpc_accdb_account_readonly_acquired {accdb_cache_class="le10_mib"} | counter | Readonly accounts acquired from the account database, by cache size class of the account's current data size (1-10 MiB) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le128_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (0-128 B) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le512_b"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (129-512 B) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le2_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (513 B-2 KiB) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le8_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (2-8 KiB) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le32_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (8-32 KiB) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le128_kib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (32-128 KiB) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le1_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (128 KiB-1 MiB) |
| rpc_accdb_account_cache_missed {accdb_cache_class="le10_mib"} | counter | Accounts not found in the account database cache and read from disk, by cache size class (1-10 MiB) |
| rpc_accdb_account_waited | counter | Accounts that had to wait for another reader to finish loading them into the cache |
| rpc_accdb_disk_read_bytes | counter | Bytes read from the account database file |
| rpc_accdb_disk_read | counter | Reads from the account database file |
| rpc_accdb_copied_bytes | counter | Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded |
Mlx5 Tile
| Metric | Type | Description |
|---|---|---|
| mlx5_pkt_rx | counter | Packets successfully published to output links |
| mlx5_pkt_rx_bytes | counter | Bytes of packets successfully published to output links |
| mlx5_pkt_rx_dropped {pkt_rx_drop_reason="malformed"} | counter | Received packets dropped, by reason (Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present) |
| mlx5_pkt_rx_dropped {pkt_rx_drop_reason="dst_ip_mismatch"} | counter | Received packets dropped, by reason (Destination IP does not match the bind address) |
| mlx5_pkt_rx_dropped {pkt_rx_drop_reason="dst_port_unbound"} | counter | Received packets dropped, by reason (No output link for the destination UDP port) |
| mlx5_pkt_rx_dropped {pkt_rx_drop_reason="gre_invalid"} | counter | Received packets dropped, by reason (GRE tunnel peer or headers invalid) |
| mlx5_pkt_rx_dropped {pkt_rx_drop_reason="gre_unconfigured"} | counter | Received packets dropped, by reason (GRE packet with no GRE tunnel configured) |
| mlx5_gre_pkt_rx | counter | GRE packets decapsulated and published; subset of PktRx |
| mlx5_vlan_pkt_rx | counter | VLAN-tagged packets with the tag stripped in software and published; subset of PktRx |
| mlx5_rx_buffer_used | gauge | RX buffers not currently posted to the NIC |
| mlx5_rx_buffer_free | gauge | RX buffers currently posted to the NIC |
| mlx5_pkt_tx_completed | counter | Packets completed by the NIC for transmission |
| mlx5_pkt_tx_completed_bytes | counter | Bytes of packets completed by the NIC |
| mlx5_pkt_tx_dropped {pkt_tx_drop_reason="invalid"} | counter | Transmit jobs dropped, by reason (Invalid packet headers, or GRE packet exceeding the MTU) |
| mlx5_pkt_tx_dropped {pkt_tx_drop_reason="source_ip_missing"} | counter | Transmit jobs dropped, by reason (No source IP address chosen) |
| mlx5_pkt_tx_dropped {pkt_tx_drop_reason="ring_full"} | counter | Transmit jobs dropped, by reason (TX ring full or no TX buffer available) |
| mlx5_pkt_tx_dropped {pkt_tx_drop_reason="gre_no_route"} | counter | Transmit jobs dropped, by reason (GRE tunnel addresses missing or outer route is itself tunnelled) |
| mlx5_pkt_tx_dropped {pkt_tx_drop_reason="socket_send_failure"} | counter | Transmit jobs dropped, by reason (Socket send failed or failed to send the entire packet) |
| mlx5_pkt_tx_routed {pkt_tx_route_result="fast_path"} | counter | Transmit jobs accepted after route resolution, by result (Userland routing succeeded and selected the configured provider) |
| mlx5_pkt_tx_routed {pkt_tx_route_result="slow_path"} | counter | Transmit jobs accepted after route resolution, by result (Failed userland routing and selected the kernel fallback) |
| mlx5_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="no_route"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (No matching route) |
| mlx5_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="unsupported_route"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Unsupported route type) |
| mlx5_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="interface_missing"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface not available) |
| mlx5_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="unsupported_interface"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface type not supported) |
| mlx5_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="no_neighbor"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Next-hop neighbor unresolved or inactive) |
| mlx5_gre_pkt_tx_submitted | counter | GRE transmit jobs submitted to the NIC or a kernel socket |
| mlx5_pkt_tx_slow_path_submitted | counter | Packets submitted to a kernel socket on the slow path |
| mlx5_tx_buffer_used | gauge | TX buffers pending or submitted to the NIC |
| mlx5_tx_buffer_free | gauge | TX buffers available for a new transmit job |
Waker Tile
| Metric | Type | Description |
|---|---|---|
| waker_epoll_wait_returned | counter | epoll_pwait calls on the outer epoll set that returned, including timeouts; EINTR not counted |
| waker_wake_delivered | counter | Client wakes delivered (readiness fseq raised) |
Mwaitx Tile
| Metric | Type | Description |
|---|---|---|
| mwaitx_nap_taken | counter | Hardware idle-wait naps on the doorbell line |
| mwaitx_wake_issued {mwaitx_wake_cause="doorbell"} | counter | futex wakes issued to parked tiles, by cause (A producer rang the doorbell for a parked consumer) |
| mwaitx_wake_issued {mwaitx_wake_cause="deadline"} | counter | futex wakes issued to parked tiles, by cause (A parked tile's deadline lapsed) |
| mwaitx_wake_issued {mwaitx_wake_cause="sweep"} | counter | futex wakes issued to parked tiles, by cause (The verifying sweep found a pending frag whose doorbell raced or was lost) |
Iavf Tile
| Metric | Type | Description |
|---|---|---|
| iavf_pkt_rx | counter | Packets successfully published to output links |
| iavf_pkt_rx_bytes | counter | Bytes of packets successfully published to output links |
| iavf_pkt_rx_dropped {pkt_rx_drop_reason="malformed"} | counter | Received packets dropped, by reason (Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present) |
| iavf_pkt_rx_dropped {pkt_rx_drop_reason="dst_ip_mismatch"} | counter | Received packets dropped, by reason (Destination IP does not match the bind address) |
| iavf_pkt_rx_dropped {pkt_rx_drop_reason="dst_port_unbound"} | counter | Received packets dropped, by reason (No output link for the destination UDP port) |
| iavf_pkt_rx_dropped {pkt_rx_drop_reason="gre_invalid"} | counter | Received packets dropped, by reason (GRE tunnel peer or headers invalid) |
| iavf_pkt_rx_dropped {pkt_rx_drop_reason="gre_unconfigured"} | counter | Received packets dropped, by reason (GRE packet with no GRE tunnel configured) |
| iavf_pkt_rx_no_link_dropped | counter | Packets dropped because the configured VF or bond member is not accepting traffic |
| iavf_gre_pkt_rx | counter | GRE packets decapsulated and published; subset of PktRx |
| iavf_vlan_pkt_rx | counter | VLAN-tagged packets with the tag stripped in software and published; subset of PktRx |
| iavf_rx_buffer_used | gauge | RX buffers not currently posted to the NIC |
| iavf_rx_buffer_free | gauge | RX buffers currently posted to the NIC |
| iavf_pkt_tx_completed | counter | Packets completed by the NIC for transmission |
| iavf_pkt_tx_completed_bytes | counter | Bytes of packets completed by the NIC |
| iavf_pkt_tx_dropped {pkt_tx_drop_reason="invalid"} | counter | Transmit jobs dropped, by reason (Invalid packet headers, or GRE packet exceeding the MTU) |
| iavf_pkt_tx_dropped {pkt_tx_drop_reason="source_ip_missing"} | counter | Transmit jobs dropped, by reason (No source IP address chosen) |
| iavf_pkt_tx_dropped {pkt_tx_drop_reason="ring_full"} | counter | Transmit jobs dropped, by reason (TX ring full or no TX buffer available) |
| iavf_pkt_tx_dropped {pkt_tx_drop_reason="gre_no_route"} | counter | Transmit jobs dropped, by reason (GRE tunnel addresses missing or outer route is itself tunnelled) |
| iavf_pkt_tx_dropped {pkt_tx_drop_reason="socket_send_failure"} | counter | Transmit jobs dropped, by reason (Socket send failed or failed to send the entire packet) |
| iavf_pkt_tx_routed {pkt_tx_route_result="fast_path"} | counter | Transmit jobs accepted after route resolution, by result (Userland routing succeeded and selected the configured provider) |
| iavf_pkt_tx_routed {pkt_tx_route_result="slow_path"} | counter | Transmit jobs accepted after route resolution, by result (Failed userland routing and selected the kernel fallback) |
| iavf_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="no_route"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (No matching route) |
| iavf_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="unsupported_route"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Unsupported route type) |
| iavf_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="interface_missing"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface not available) |
| iavf_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="unsupported_interface"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Interface type not supported) |
| iavf_pkt_tx_slow_path_routed {pkt_tx_slow_path_reason="no_neighbor"} | counter | Transmit jobs that failed userland routing and selected the kernel fallback, by reason (Next-hop neighbor unresolved or inactive) |
| iavf_pkt_tx_no_link_dropped | counter | Transmit jobs dropped because no configured VF or bond member can send traffic |
| iavf_pkt_tx_loopback_failed | counter | Loopback transmit jobs dropped because the socket send failed or accepted only part of a batch |
| iavf_gre_pkt_tx_submitted | counter | GRE transmit jobs submitted to the NIC or a kernel socket |
| iavf_pkt_tx_slow_path_submitted | counter | Packets submitted to a kernel socket on the slow path |
| iavf_tx_buffer_used | gauge | TX buffers pending or submitted to the NIC |
| iavf_tx_buffer_free | gauge | TX buffers available for a new transmit job |
| iavf_non_fd_pkt_rx | counter | Packets received by IAVF VFs that are not intended for Firedancer |
Failov Tile
| Metric | Type | Description |
|---|
Snapmk Tile
| Metric | Type | Description |
|---|---|---|
| snapmk_job_state | gauge | Snapshot creation job state: 0=idle (Idle, waiting for a job), 1=start (Starting), 2=tar_headers (Writing tar headers), 3=manifest (Writing manifest), 4=accdb_cache (Writing cached accounts), 5=accdb_cache_flush (Flushing cached accounts), 6=accdb_cache_finish (Waiting for cached account flush), 7=accdb_disk (Writing on-disk accounts), 8=accdb_disk_flush (Flushing on-disk accounts), 9=accdb_disk_finish (Waiting for on-disk account flush), 10=accdb_delta (Writing incremental accounts), 11=accdb_delta_flush (Flushing incremental accounts), 12=accdb_delta_finish (Waiting for incremental account flush), 13=status_cache (Writing status cache), 14=eof_marker (Writing tar EOF marker), 15=done (Done, notifying replay), 16=fail (Failed, cleaning up), 17=sleep (Sleeping), 18=startup (Waiting for startup), 19=startup_burst (Publishing pre-existing snapshots) |
| snapmk_snapshot_started {snap_type="full"} | counter | Snapshot creation jobs started, by snapshot type (Full snapshot) |
| snapmk_snapshot_started {snap_type="incremental"} | counter | Snapshot creation jobs started, by snapshot type (Incremental snapshot) |
| snapmk_snapshot_finished {snap_type="full"} | counter | Snapshot creation jobs finished successfully, by snapshot type (Full snapshot) |
| snapmk_snapshot_finished {snap_type="incremental"} | counter | Snapshot creation jobs finished successfully, by snapshot type (Incremental snapshot) |
| snapmk_last_started_slot {snap_type="full"} | gauge | Slot of the most recently started snapshot creation, by snapshot type, 0 if none since boot (Full snapshot) |
| snapmk_last_started_slot {snap_type="incremental"} | gauge | Slot of the most recently started snapshot creation, by snapshot type, 0 if none since boot (Incremental snapshot) |
| snapmk_last_finished_slot {snap_type="full"} | gauge | Slot of the most recently finished snapshot creation, by snapshot type, 0 if none since boot (Full snapshot) |
| snapmk_last_finished_slot {snap_type="incremental"} | gauge | Slot of the most recently finished snapshot creation, by snapshot type, 0 if none since boot (Incremental snapshot) |
| snapmk_compress_consumed_bytes | counter | Uncompressed bytes fed to the compressor |
| snapmk_disk_written_bytes | counter | Compressed bytes written to snapshot files |
| snapmk_cache_progress_bytes | gauge | Bytes of the accounts database cache region scanned so far for the snapshot being created. Resets when a snapshot starts and holds at the total once the cached accounts phase finishes. Stays zero for an incremental snapshot, which does not scan the cache |
| snapmk_cache_expected_bytes | gauge | Size of the accounts database cache region a cached accounts phase scans |
| snapmk_incremental_account_progress | gauge | Accounts changed since the base snapshot that have been written so far for the incremental snapshot being created. Resets when a snapshot starts, and stays zero for a full snapshot, which has no changed account walk |
| snapmk_incremental_account_expected | gauge | Accounts changed since the base snapshot, sampled when the incremental snapshot started; 0 for a full snapshot. Replay may record more changes while the job runs, so progress can exceed this |
| snapmk_io_blocked_duration_nanos | counter | Cumulative time blocked on I/O |
| snapmk_compress_duration_nanos | counter | Cumulative time spent compressing data |
| snapmk_incremental_account_used | gauge | Distinct accounts changed in rooted slots since the last full snapshot job started; saturates at IncrementalAccountCapacity |
| snapmk_incremental_account_capacity | gauge | Maximum changed accounts that can be tracked; once reached, no incremental snapshot can be created until the next full snapshot |
| snapmk_disk_allocated_bytes | gauge | Logical bytes allocated across managed snapshot files, including partial and retained files |
Snapzp Tile
| Metric | Type | Description |
|---|---|---|
| snapzp_account_compressed | counter | Accounts compressed |
| snapzp_compress_consumed_bytes | counter | Uncompressed bytes fed to the compressor |
| snapzp_disk_written_bytes | counter | Compressed bytes written to snapshot files |
| snapzp_io_blocked_duration_nanos | counter | Cumulative time blocked on I/O |
| snapzp_compress_duration_nanos | counter | Cumulative time spent compressing accounts |
| snapzp_cache_read_torn | counter | Cached account reads torn by eviction, handed back to snapmk to un-claim |
Snaprd Tile
| Metric | Type | Description |
|---|---|---|
| snaprd_disk_read_bytes | counter | Bytes read from the account database file |
| snaprd_export_progress_bytes | gauge | Account database bytes read so far for the current snapshot creation job |
| snaprd_export_expected_bytes | gauge | Account database bytes to read for the current snapshot creation job |
| snaprd_io_blocked_duration_nanos | counter | Cumulative time blocked on I/O |
Snapsv Tile
| Metric | Type | Description |
|---|---|---|
| snapsv_snapshot_available {snap_type="full"} | gauge | Snapshot files currently available for download (Full snapshot) |
| snapsv_snapshot_available {snap_type="incremental"} | gauge | Snapshot files currently available for download (Incremental snapshot) |
| snapsv_http_request_served | counter | HTTP requests served |
| snapsv_conn_open | gauge | Open HTTP connections to the snapshot service |
| snapsv_http_rx_bytes | counter | Bytes received in requests to the snapshot service |
| snapsv_http_tx_bytes | counter | Bytes sent in responses from the snapshot service |
Backt Tile
| Metric | Type | Description |
|---|---|---|
| backt_end_slot | gauge | Last slot the backtest replays, 0 if unbounded |
| backt_start_slot | gauge | Snapshot slot replayed from (replay begins at the next slot), 0 for genesis |
Benchs Tile
| Metric | Type | Description |
|---|---|---|
| benchs_txn_tx | counter | Benchmark transactions sent |
| benchs_txn_dropped | counter | Benchmark transactions dropped because no QUIC connection was established or it could not send |
Votor Tile
| Metric | Type | Description |
|---|---|---|
| votor_datagram_rx {datagram_rx_result="vote"} | counter | Inbound datagrams dispatched, by result (Datagram carried a vote and was handed to vote rx) |
| votor_datagram_rx {datagram_rx_result="cert"} | counter | Inbound datagrams dispatched, by result (Datagram carried a cert and was handed to cert rx) |
| votor_datagram_rx {datagram_rx_result="not_ready"} | counter | Inbound datagrams dispatched, by result (Datagram arrived before the votor tile was initialized) |
| votor_datagram_rx {datagram_rx_result="too_small"} | counter | Inbound datagrams dispatched, by result (Datagram was too small to hold a version and a tag) |
| votor_datagram_rx {datagram_rx_result="unknown_tag"} | counter | Inbound datagrams dispatched, by result (Datagram tag was neither a vote nor a cert) |
| votor_datagram_rx {datagram_rx_result="not_admitted"} | counter | Inbound datagrams dispatched, by result (Datagram arrived on a connection not yet admitted, or already closed) |
| votor_vote_rx {vote_rx_result="success"} | counter | Inbound votes processed, by result (Vote was handed to the pool and accepted) |
| votor_vote_rx {vote_rx_result="bad_size"} | counter | Inbound votes processed, by result (Vote was truncated or had trailing bytes) |
| votor_vote_rx {vote_rx_result="bad_encoding"} | counter | Inbound votes processed, by result (Vote had an invalid version, tag, or signature encoding) |
| votor_vote_rx {vote_rx_result="unknown_signer"} | counter | Inbound votes processed, by result (Sending connection had no authenticated identity) |
| votor_vote_rx {vote_rx_result="not_peer"} | counter | Inbound votes processed, by result (Sender was not in the peer set) |
| votor_vote_rx {vote_rx_result="banned"} | counter | Inbound votes processed, by result (Sender is banned for a failed signature verification) |
| votor_vote_rx {vote_rx_result="slot_out_of_range"} | counter | Inbound votes processed, by result (Vote slot was either too old or too far in the future) |
| votor_vote_rx {vote_rx_result="null_block_hash"} | counter | Inbound votes processed, by result (Vote block hash was null) |
| votor_vote_rx {vote_rx_result="signature_failed"} | counter | Inbound votes processed, by result (Vote signature was the identity point or outside the G2 subgroup) |
| votor_vote_rx {vote_rx_result="unranked"} | counter | Inbound votes processed, by result (Sender was not a ranked validator in the vote slot's epoch) |
| votor_vote_rx {vote_rx_result="shred_version_mismatched"} | counter | Inbound votes processed, by result (Vote was for a different shred version) |
| votor_vote_rx {vote_rx_result="duplicate"} | counter | Inbound votes processed, by result (Voter had already cast this vote) |
| votor_vote_rx {vote_rx_result="slashable"} | counter | Inbound votes processed, by result (Vote constitutes a slashable offence) |
| votor_cert_rx {cert_rx_result="success"} | counter | Inbound certificates processed, by result (Cert was handed to the pool and accepted) |
| votor_cert_rx {cert_rx_result="bad_size"} | counter | Inbound certificates processed, by result (Cert was truncated or had trailing bytes) |
| votor_cert_rx {cert_rx_result="bad_encoding"} | counter | Inbound certificates processed, by result (Cert had an invalid version, tag, bitmap, or signature encoding) |
| votor_cert_rx {cert_rx_result="shred_version_mismatched"} | counter | Inbound certificates processed, by result (Cert was for a different shred version) |
| votor_cert_rx {cert_rx_result="unknown_signer"} | counter | Inbound certificates processed, by result (Sending connection had no authenticated identity) |
| votor_cert_rx {cert_rx_result="not_peer"} | counter | Inbound certificates processed, by result (Sender was not in the peer set) |
| votor_cert_rx {cert_rx_result="unranked"} | counter | Inbound certificates processed, by result (Sender was not a ranked validator in the cert slot's epoch) |
| votor_cert_rx {cert_rx_result="slot_out_of_range"} | counter | Inbound certificates processed, by result (Cert slot was either too old or too far in the future) |
| votor_cert_rx {cert_rx_result="duplicate"} | counter | Inbound certificates processed, by result (Cert was already in the pool) |
| votor_cert_rx {cert_rx_result="signature_failed"} | counter | Inbound certificates processed, by result (Cert failed the stake threshold or the aggregate signature check) |
| votor_cert_rx {cert_rx_result="banned"} | counter | Inbound certificates processed, by result (Sender is banned for a failed signature verification) |
| votor_bls_signature_checked | counter | Aggregate BLS signature checks of votes, certificates and reward certificates; a search for bad signers counts once |
| votor_leader_window_superseded | counter | Leader windows given up because certificates already let a later window start, eg. while catching up after a restart |
| votor_cert_tx_unstaked_skipped | counter | Certificates and standstill votes not sent because we are not staked in their epoch |
| votor_voting_slot_used | gauge | Slots tracked by the voting state machine |
| votor_voting_slot_free | gauge | Free entries in the voting state machine's slot table |
| votor_highest_final_cert_slot | gauge | Highest slot with a final or fast-final cert seen by votor, 0 if none |
| votor_pool_slot_used | gauge | Slots tracked by the vote and cert pool |
| votor_pool_slot_free | gauge | Free entries in the vote and cert pool's slot table |
| votor_highest_pool_finalized_slot | gauge | Highest slot the pool has finalized (slow-final also needs the slot notarized), 0 if none |
| votor_rank | gauge | Rank of this validator in the current epoch (0 is the highest rank), ULONG_MAX if unranked (unstaked) |
| votor_client_pkt_rx | counter | UDP datagrams received (one may carry several coalesced QUIC packets) |
| votor_client_pkt_rx_bytes | counter | Bytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers |
| votor_client_pkt_tx | counter | UDP datagrams sent |
| votor_client_pkt_tx_bytes | counter | Bytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers |
| votor_client_conn_used {quic_conn_state="handshake"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Handshaking peer) |
| votor_client_conn_used {quic_conn_state="handshake_complete"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Handshake complete, confirming with peer) |
| votor_client_conn_used {quic_conn_state="active"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Active connection) |
| votor_client_conn_used {quic_conn_state="peer_close"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Peer requested close) |
| votor_client_conn_used {quic_conn_state="abort"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection terminating due to error) |
| votor_client_conn_used {quic_conn_state="close_pending"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection is closing) |
| votor_client_conn_used {quic_conn_state="dead"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection about to be freed) |
| votor_client_conn_free | gauge | Connection slots free |
| votor_client_conn_capacity | gauge | Connection slots |
| votor_client_conn_created | counter | Connections created |
| votor_client_conn_closed | counter | Connections gracefully closed |
| votor_client_conn_aborted | counter | Connections aborted |
| votor_client_conn_timed_out | counter | Connections timed out |
| votor_client_conn_no_slot_rejected | counter | Connections that failed to create due to lack of slots |
| votor_client_conn_retry_rejected | counter | Retries rejected: an invalid or expired token as server, a late or invalid Retry packet as client |
| votor_client_pkt_decrypt_failed {quic_encryption_level="initial"} | counter | Packets that failed decryption, by encryption level (Initial) |
| votor_client_pkt_decrypt_failed {quic_encryption_level="early"} | counter | Packets that failed decryption, by encryption level (Early data) |
| votor_client_pkt_decrypt_failed {quic_encryption_level="handshake"} | counter | Packets that failed decryption, by encryption level (Handshake) |
| votor_client_pkt_decrypt_failed {quic_encryption_level="app"} | counter | Packets that failed decryption, by encryption level (App data) |
| votor_client_pkt_no_key_dropped {quic_encryption_level="initial"} | counter | Packets dropped because the decryption key was missing, by encryption level (Initial) |
| votor_client_pkt_no_key_dropped {quic_encryption_level="early"} | counter | Packets dropped because the decryption key was missing, by encryption level (Early data) |
| votor_client_pkt_no_key_dropped {quic_encryption_level="handshake"} | counter | Packets dropped because the decryption key was missing, by encryption level (Handshake) |
| votor_client_pkt_no_key_dropped {quic_encryption_level="app"} | counter | Packets dropped because the decryption key was missing, by encryption level (App data) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="undersize"} | counter | Received packets dropped, by reason (QUIC packet too small) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="oversize"} | counter | Received packets dropped, by reason (QUIC packet too large) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="src_ip_mismatch"} | counter | Received packets dropped, by reason (Wrong source IP) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="header_invalid"} | counter | Received packets dropped, by reason (Invalid QUIC header) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="net_header_invalid"} | counter | Received packets dropped, by reason (Invalid IPv4 or UDP header) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="version_unsupported"} | counter | Received packets dropped, by reason (Version negotiation or unsupported version) |
| votor_client_pkt_dropped {quic_pkt_drop_reason="duplicate"} | counter | Received packets dropped, by reason (Packet number already received or too old) |
| votor_client_pkt_no_conn_dropped {quic_pkt_type="initial"} | counter | Packets with an unknown connection ID, by packet type (Initial) |
| votor_client_pkt_no_conn_dropped {quic_pkt_type="retry"} | counter | Packets with an unknown connection ID, by packet type (Retry) |
| votor_client_pkt_no_conn_dropped {quic_pkt_type="handshake"} | counter | Packets with an unknown connection ID, by packet type (Handshake) |
| votor_client_pkt_no_conn_dropped {quic_pkt_type="one_rtt"} | counter | Packets with an unknown connection ID, by packet type (1-RTT) |
| votor_client_pkt_retransmitted {quic_encryption_level="initial"} | counter | QUIC packets retransmitted, by encryption level (Initial) |
| votor_client_pkt_retransmitted {quic_encryption_level="early"} | counter | QUIC packets retransmitted, by encryption level (Early data) |
| votor_client_pkt_retransmitted {quic_encryption_level="handshake"} | counter | QUIC packets retransmitted, by encryption level (Handshake) |
| votor_client_pkt_retransmitted {quic_encryption_level="app"} | counter | QUIC packets retransmitted, by encryption level (App data) |
| votor_client_handshake_created | counter | Handshake flows created |
| votor_client_handshake_pool_full_rejected | counter | Handshakes dropped due to alloc fail |
| votor_client_handshake_evicted | counter | Handshakes dropped due to eviction |
| votor_client_frame_rx {quic_frame_type="unknown"} | counter | QUIC frames received, by frame type (Unknown frame type) |
| votor_client_frame_rx {quic_frame_type="ack"} | counter | QUIC frames received, by frame type (ACK frame) |
| votor_client_frame_rx {quic_frame_type="reset_stream"} | counter | QUIC frames received, by frame type (RESET_STREAM frame) |
| votor_client_frame_rx {quic_frame_type="stop_sending"} | counter | QUIC frames received, by frame type (STOP_SENDING frame) |
| votor_client_frame_rx {quic_frame_type="crypto"} | counter | QUIC frames received, by frame type (CRYPTO frame) |
| votor_client_frame_rx {quic_frame_type="new_token"} | counter | QUIC frames received, by frame type (NEW_TOKEN frame) |
| votor_client_frame_rx {quic_frame_type="stream"} | counter | QUIC frames received, by frame type (STREAM frame) |
| votor_client_frame_rx {quic_frame_type="max_data"} | counter | QUIC frames received, by frame type (MAX_DATA frame) |
| votor_client_frame_rx {quic_frame_type="max_stream_data"} | counter | QUIC frames received, by frame type (MAX_STREAM_DATA frame) |
| votor_client_frame_rx {quic_frame_type="max_streams"} | counter | QUIC frames received, by frame type (MAX_STREAMS frame) |
| votor_client_frame_rx {quic_frame_type="data_blocked"} | counter | QUIC frames received, by frame type (DATA_BLOCKED frame) |
| votor_client_frame_rx {quic_frame_type="stream_data_blocked"} | counter | QUIC frames received, by frame type (STREAM_DATA_BLOCKED frame) |
| votor_client_frame_rx {quic_frame_type="streams_blocked"} | counter | QUIC frames received, by frame type (STREAMS_BLOCKED(bidi) frame) |
| votor_client_frame_rx {quic_frame_type="new_conn_id"} | counter | QUIC frames received, by frame type (NEW_CONN_ID frame) |
| votor_client_frame_rx {quic_frame_type="retire_conn_id"} | counter | QUIC frames received, by frame type (RETIRE_CONN_ID frame) |
| votor_client_frame_rx {quic_frame_type="path_challenge"} | counter | QUIC frames received, by frame type (PATH_CHALLENGE frame) |
| votor_client_frame_rx {quic_frame_type="path_response"} | counter | QUIC frames received, by frame type (PATH_RESPONSE frame) |
| votor_client_frame_rx {quic_frame_type="conn_close_quic"} | counter | QUIC frames received, by frame type (CONN_CLOSE(transport) frame) |
| votor_client_frame_rx {quic_frame_type="conn_close_app"} | counter | QUIC frames received, by frame type (CONN_CLOSE(app) frame) |
| votor_client_frame_rx {quic_frame_type="handshake_done"} | counter | QUIC frames received, by frame type (HANDSHAKE_DONE frame) |
| votor_client_frame_rx {quic_frame_type="ping"} | counter | QUIC frames received, by frame type (PING frame) |
| votor_client_frame_rx {quic_frame_type="padding"} | counter | QUIC frames received, by frame type (PADDING frame) |
| votor_client_frame_rx {quic_frame_type="datagram"} | counter | QUIC frames received, by frame type (DATAGRAM frame) |
| votor_client_frame_parse_failed | counter | QUIC frames that failed to parse |
| votor_client_pkt_meta_acquire_attempted {pkt_meta_acquire_result="success"} | counter | Attempts to acquire packet metadata for transmission, by result (Success) |
| votor_client_pkt_meta_acquire_attempted {pkt_meta_acquire_result="pool_empty"} | counter | Attempts to acquire packet metadata for transmission, by result (Packet metadata pool empty) |
| votor_client_pkt_meta_acquire_attempted {pkt_meta_acquire_result="conn_limit"} | counter | Attempts to acquire packet metadata for transmission, by result (Connection reached its in-flight frame limit) |
| votor_client_pkt_ack_processed {ack_result="no_pkt_number"} | counter | Received packets processed for acknowledgement, by result (Packet had no packet number to acknowledge) |
| votor_client_pkt_ack_processed {ack_result="new_range"} | counter | Received packets processed for acknowledgement, by result (New ACK range) |
| votor_client_pkt_ack_processed {ack_result="merged"} | counter | Received packets processed for acknowledgement, by result (Merged into existing ACK range) |
| votor_client_pkt_ack_processed {ack_result="queue_full"} | counter | Received packets processed for acknowledgement, by result (Out of buffers) |
| votor_client_pkt_ack_processed {ack_result="cancelled"} | counter | Received packets processed for acknowledgement, by result (ACK suppressed by handler) |
| votor_client_service_duration_nanos | histogram | Duration of one fd_quic_service call that processed a due connection timer |
| votor_client_pkt_rx_duration_nanos | histogram | Duration of processing one received packet |
| votor_server_pkt_rx | counter | UDP datagrams received (one may carry several coalesced QUIC packets) |
| votor_server_pkt_rx_bytes | counter | Bytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers |
| votor_server_pkt_tx | counter | UDP datagrams sent |
| votor_server_pkt_tx_bytes | counter | Bytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers |
| votor_server_conn_used {quic_conn_state="handshake"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Handshaking peer) |
| votor_server_conn_used {quic_conn_state="handshake_complete"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Handshake complete, confirming with peer) |
| votor_server_conn_used {quic_conn_state="active"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Active connection) |
| votor_server_conn_used {quic_conn_state="peer_close"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Peer requested close) |
| votor_server_conn_used {quic_conn_state="abort"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection terminating due to error) |
| votor_server_conn_used {quic_conn_state="close_pending"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection is closing) |
| votor_server_conn_used {quic_conn_state="dead"} | gauge | Connection slots in use, by state; sum + ConnFree = ConnCapacity (Connection about to be freed) |
| votor_server_conn_free | gauge | Connection slots free |
| votor_server_conn_capacity | gauge | Connection slots |
| votor_server_conn_created | counter | Connections created |
| votor_server_conn_closed | counter | Connections gracefully closed |
| votor_server_conn_aborted | counter | Connections aborted |
| votor_server_conn_timed_out | counter | Connections timed out |
| votor_server_conn_no_slot_rejected | counter | Connections that failed to create due to lack of slots |
| votor_server_conn_retry_rejected | counter | Retries rejected: an invalid or expired token as server, a late or invalid Retry packet as client |
| votor_server_conn_retry_expired_rejected | counter | Initials with an expired Retry token, also counted in ServerConnRetryRejected. Each gets an INVALID_TOKEN close so the client reconnects |
| votor_server_pkt_decrypt_failed {quic_encryption_level="initial"} | counter | Packets that failed decryption, by encryption level (Initial) |
| votor_server_pkt_decrypt_failed {quic_encryption_level="early"} | counter | Packets that failed decryption, by encryption level (Early data) |
| votor_server_pkt_decrypt_failed {quic_encryption_level="handshake"} | counter | Packets that failed decryption, by encryption level (Handshake) |
| votor_server_pkt_decrypt_failed {quic_encryption_level="app"} | counter | Packets that failed decryption, by encryption level (App data) |
| votor_server_pkt_no_key_dropped {quic_encryption_level="initial"} | counter | Packets dropped because the decryption key was missing, by encryption level (Initial) |
| votor_server_pkt_no_key_dropped {quic_encryption_level="early"} | counter | Packets dropped because the decryption key was missing, by encryption level (Early data) |
| votor_server_pkt_no_key_dropped {quic_encryption_level="handshake"} | counter | Packets dropped because the decryption key was missing, by encryption level (Handshake) |
| votor_server_pkt_no_key_dropped {quic_encryption_level="app"} | counter | Packets dropped because the decryption key was missing, by encryption level (App data) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="undersize"} | counter | Received packets dropped, by reason (QUIC packet too small) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="oversize"} | counter | Received packets dropped, by reason (QUIC packet too large) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="src_ip_mismatch"} | counter | Received packets dropped, by reason (Wrong source IP) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="header_invalid"} | counter | Received packets dropped, by reason (Invalid QUIC header) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="net_header_invalid"} | counter | Received packets dropped, by reason (Invalid IPv4 or UDP header) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="version_unsupported"} | counter | Received packets dropped, by reason (Version negotiation or unsupported version) |
| votor_server_pkt_dropped {quic_pkt_drop_reason="duplicate"} | counter | Received packets dropped, by reason (Packet number already received or too old) |
| votor_server_pkt_no_conn_dropped {quic_pkt_type="initial"} | counter | Packets with an unknown connection ID, by packet type (Initial) |
| votor_server_pkt_no_conn_dropped {quic_pkt_type="retry"} | counter | Packets with an unknown connection ID, by packet type (Retry) |
| votor_server_pkt_no_conn_dropped {quic_pkt_type="handshake"} | counter | Packets with an unknown connection ID, by packet type (Handshake) |
| votor_server_pkt_no_conn_dropped {quic_pkt_type="one_rtt"} | counter | Packets with an unknown connection ID, by packet type (1-RTT) |
| votor_server_pkt_retransmitted {quic_encryption_level="initial"} | counter | QUIC packets retransmitted, by encryption level (Initial) |
| votor_server_pkt_retransmitted {quic_encryption_level="early"} | counter | QUIC packets retransmitted, by encryption level (Early data) |
| votor_server_pkt_retransmitted {quic_encryption_level="handshake"} | counter | QUIC packets retransmitted, by encryption level (Handshake) |
| votor_server_pkt_retransmitted {quic_encryption_level="app"} | counter | QUIC packets retransmitted, by encryption level (App data) |
| votor_server_handshake_created | counter | Handshake flows created |
| votor_server_handshake_pool_full_rejected | counter | Handshakes dropped due to alloc fail |
| votor_server_handshake_evicted | counter | Handshakes dropped due to eviction |
| votor_server_frame_rx {quic_frame_type="unknown"} | counter | QUIC frames received, by frame type (Unknown frame type) |
| votor_server_frame_rx {quic_frame_type="ack"} | counter | QUIC frames received, by frame type (ACK frame) |
| votor_server_frame_rx {quic_frame_type="reset_stream"} | counter | QUIC frames received, by frame type (RESET_STREAM frame) |
| votor_server_frame_rx {quic_frame_type="stop_sending"} | counter | QUIC frames received, by frame type (STOP_SENDING frame) |
| votor_server_frame_rx {quic_frame_type="crypto"} | counter | QUIC frames received, by frame type (CRYPTO frame) |
| votor_server_frame_rx {quic_frame_type="new_token"} | counter | QUIC frames received, by frame type (NEW_TOKEN frame) |
| votor_server_frame_rx {quic_frame_type="stream"} | counter | QUIC frames received, by frame type (STREAM frame) |
| votor_server_frame_rx {quic_frame_type="max_data"} | counter | QUIC frames received, by frame type (MAX_DATA frame) |
| votor_server_frame_rx {quic_frame_type="max_stream_data"} | counter | QUIC frames received, by frame type (MAX_STREAM_DATA frame) |
| votor_server_frame_rx {quic_frame_type="max_streams"} | counter | QUIC frames received, by frame type (MAX_STREAMS frame) |
| votor_server_frame_rx {quic_frame_type="data_blocked"} | counter | QUIC frames received, by frame type (DATA_BLOCKED frame) |
| votor_server_frame_rx {quic_frame_type="stream_data_blocked"} | counter | QUIC frames received, by frame type (STREAM_DATA_BLOCKED frame) |
| votor_server_frame_rx {quic_frame_type="streams_blocked"} | counter | QUIC frames received, by frame type (STREAMS_BLOCKED(bidi) frame) |
| votor_server_frame_rx {quic_frame_type="new_conn_id"} | counter | QUIC frames received, by frame type (NEW_CONN_ID frame) |
| votor_server_frame_rx {quic_frame_type="retire_conn_id"} | counter | QUIC frames received, by frame type (RETIRE_CONN_ID frame) |
| votor_server_frame_rx {quic_frame_type="path_challenge"} | counter | QUIC frames received, by frame type (PATH_CHALLENGE frame) |
| votor_server_frame_rx {quic_frame_type="path_response"} | counter | QUIC frames received, by frame type (PATH_RESPONSE frame) |
| votor_server_frame_rx {quic_frame_type="conn_close_quic"} | counter | QUIC frames received, by frame type (CONN_CLOSE(transport) frame) |
| votor_server_frame_rx {quic_frame_type="conn_close_app"} | counter | QUIC frames received, by frame type (CONN_CLOSE(app) frame) |
| votor_server_frame_rx {quic_frame_type="handshake_done"} | counter | QUIC frames received, by frame type (HANDSHAKE_DONE frame) |
| votor_server_frame_rx {quic_frame_type="ping"} | counter | QUIC frames received, by frame type (PING frame) |
| votor_server_frame_rx {quic_frame_type="padding"} | counter | QUIC frames received, by frame type (PADDING frame) |
| votor_server_frame_rx {quic_frame_type="datagram"} | counter | QUIC frames received, by frame type (DATAGRAM frame) |
| votor_server_frame_parse_failed | counter | QUIC frames that failed to parse |
| votor_server_pkt_meta_acquire_attempted {pkt_meta_acquire_result="success"} | counter | Attempts to acquire packet metadata for transmission, by result (Success) |
| votor_server_pkt_meta_acquire_attempted {pkt_meta_acquire_result="pool_empty"} | counter | Attempts to acquire packet metadata for transmission, by result (Packet metadata pool empty) |
| votor_server_pkt_meta_acquire_attempted {pkt_meta_acquire_result="conn_limit"} | counter | Attempts to acquire packet metadata for transmission, by result (Connection reached its in-flight frame limit) |
| votor_server_pkt_ack_processed {ack_result="no_pkt_number"} | counter | Received packets processed for acknowledgement, by result (Packet had no packet number to acknowledge) |
| votor_server_pkt_ack_processed {ack_result="new_range"} | counter | Received packets processed for acknowledgement, by result (New ACK range) |
| votor_server_pkt_ack_processed {ack_result="merged"} | counter | Received packets processed for acknowledgement, by result (Merged into existing ACK range) |
| votor_server_pkt_ack_processed {ack_result="queue_full"} | counter | Received packets processed for acknowledgement, by result (Out of buffers) |
| votor_server_pkt_ack_processed {ack_result="cancelled"} | counter | Received packets processed for acknowledgement, by result (ACK suppressed by handler) |
| votor_server_service_duration_nanos | histogram | Duration of one fd_quic_service call that processed a due connection timer |
| votor_server_pkt_rx_duration_nanos | histogram | Duration of processing one received packet |
| votor_vote_history_file_written | counter | Times the vote history file was written |
| votor_wait_to_vote_slot | gauge | First slot votor may vote in, past the vote history file and the votes it sent |
Motor Tile
| Metric | Type | Description |
|---|
Solcap Tile
| Metric | Type | Description |
|---|