Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via a Prometheus HTTP endpoint:
toml
[tiles.metric]
prometheus_listen_port = 7999sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric representing a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_frag_consumed | counter | Fragments consumed by the link reader |
| link_frag_consumed_bytes | counter | Bytes read by the link consumer |
| link_frag_filtered | counter | Fragments filtered and not consumed |
| link_frag_filtered_bytes | counter | Bytes read by the link consumer that were filtered |
| link_link_polling_overrun | counter | Times the link was overrun while polling |
| link_frag_polling_overrun | counter | Fragments not processed because the link was overrun while polling |
| link_link_reading_overrun | counter | Times the link was overrun while reading metadata |
| link_frag_reading_overrun | counter | Fragments not processed because the link was overrun while reading |
| link_slow | counter | Times the consumer was detected as slow, rate limiting the producer |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | Process ID of the tile |
| tile_tid | gauge | Thread ID of the tile. Same as Pid in production, but might differ in development |
| tile_last_cpu | gauge | Index of the CPU last executed on |
| tile_context_switch_involuntary | counter | Involuntary context switches |
| tile_context_switch_voluntary | counter | Voluntary context switches |
| tile_page_fault_major | counter | Major page faults |
| tile_page_fault_minor | counter | Minor page faults |
| tile_status | gauge | 0=booting, 1=running, 2=shutdown |
| tile_heartbeat_timestamp_nanos | gauge | Last UNIX timestamp the tile heartbeated, in nanoseconds |
| tile_in_backpressure | gauge | Whether the tile is currently backpressured, 1 or 0 |
| tile_backpressure | counter | Times the tile waited for one or more consumers to catch up to resume publishing |
| tile_regime_duration_nanos {tile_regime="caught_up_housekeeping"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Caught up + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="processing_housekeeping"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Processing + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="backpressure_housekeeping"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Backpressure + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="caught_up_prefrag"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Caught up + Prefrag) |
| tile_regime_duration_nanos {tile_regime="processing_prefrag"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Processing + Prefrag) |
| tile_regime_duration_nanos {tile_regime="backpressure_prefrag"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Backpressure + Prefrag) |
| tile_regime_duration_nanos {tile_regime="caught_up_postfrag"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Caught up + Postfrag) |
| tile_regime_duration_nanos {tile_regime="processing_postfrag"} | counter | Mutually exclusive and exhaustive duration the tile spent in each regime, in nanoseconds (Processing + Postfrag) |
| tile_cpu_duration_nanos {cpu_regime="wait"} | counter | CPU time spent in each CPU regime, in nanoseconds (Wait (task was runnable but not scheduled)) |
| tile_cpu_duration_nanos {cpu_regime="idle"} | counter | CPU time spent in each CPU regime, in nanoseconds (Idle (task was not runnable)) |
| tile_cpu_duration_nanos {cpu_regime="user"} | counter | CPU time spent in each CPU regime, in nanoseconds (User (task was scheduled and executing in user mode)) |
| tile_cpu_duration_nanos {cpu_regime="system"} | counter | CPU time spent in each CPU regime, in nanoseconds (System (task was scheduled and executing in kernel mode)) |
| tile_irq_preempted | counter | Times the tile was interrupted by an IRQ (fixed tiles only) |
| tile_tlb_shootdown | counter | TLB shootdowns observed on the tile CPU (fixed tiles only) |
Genesi Tile
| Metric | Type | Description |
|---|
Ipecho Tile
| Metric | Type | Description |
|---|---|---|
| ipecho_current_shred_version | gauge | Current shred version used by the validator |
| ipecho_conn_active | gauge | Active connections to the ipecho service |
| ipecho_conn_closed {conn_close_result="ok"} | counter | Connections to the ipecho service made and closed, by outcome (Closed normally) |
| ipecho_conn_closed {conn_close_result="error"} | counter | Connections to the ipecho service made and closed, by outcome (Closed abnormally) |
| ipecho_bytes_read | counter | Bytes read from all connections to the ipecho service |
| ipecho_bytes_written | counter | Bytes written to all connections to the ipecho service |
Snapct Tile
| Metric | Type | Description |
|---|---|---|
| snapct_state | gauge | 0=init, 1=waiting for peers, 2=waiting for peers (incremental), 3=collecting peers, 4=collecting peers (incremental), 5=reading full file, 6=flushing full file fini, 7=flushing full file done, 8=flushing full file reset, 9=reading incremental file, 10=flushing incremental file fini, 11=flushing incremental file done, 12=flushing incremental file reset, 13=reading full http, 14=flushing full http fini, 15=flushing full http done, 16=flushing full http reset, 17=reading incremental http, 18=flushing incremental http fini, 19=flushing incremental http done, 20=flushing incremental http reset, 21=shutdown |
| snapct_full_bytes_read | gauge | Bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_full_bytes_written | gauge | Bytes written so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_full_size_bytes | gauge | Total size of the full snapshot file. Might change if snapshot load is aborted and restarted |
| snapct_full_retry | gauge | Retries of the full snapshot download so far because the peer was too slow |
| snapct_incremental_bytes_read | gauge | Bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_incremental_bytes_written | gauge | Bytes written so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_incremental_size_bytes | gauge | Total size of the incremental snapshot file. Might change if snapshot load is aborted and restarted |
| snapct_incremental_retry | gauge | Retries of the incremental snapshot download so far because the peer was too slow |
| snapct_predicted_slot | gauge | Predicted slot from which replay starts after snapshot loading finishes. Might change if snapshot load is aborted and restarted |
| snapct_ssl_alloc_failed | counter | SSL allocation errors encountered |
Snapld Tile
| Metric | Type | Description |
|---|---|---|
| snapld_state | gauge | 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown |
| snapld_ssl_alloc_failed | counter | SSL allocation errors encountered |
Snapdc Tile
| Metric | Type | Description |
|---|---|---|
| snapdc_state | gauge | 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown |
| snapdc_full_compressed_bytes_read | gauge | Bytes read so far from the compressed full snapshot file. Might decrease if snapshot load is aborted and restarted |
| snapdc_full_decompressed_bytes_written | gauge | Bytes decompressed so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapdc_incremental_compressed_bytes_read | gauge | Bytes read so far from the compressed incremental snapshot file. Might decrease if snapshot load is aborted and restarted |
| snapdc_incremental_decompressed_bytes_written | gauge | Bytes decompressed so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
Snapin Tile
| Metric | Type | Description |
|---|---|---|
| snapin_state | gauge | 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown |
| snapin_full_bytes_read | gauge | Bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapin_incremental_bytes_read | gauge | Bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapin_account_loaded | gauge | Accounts seen during snapshot loading. Includes duplicates. Resets if snapshot load restarts |
| snapin_account_replaced | gauge | Previously inserted accounts replaced by a later duplicate. Resets if snapshot load restarts |
| snapin_account_ignored | gauge | Stale duplicate accounts dropped because a previously inserted account was newer. Resets if snapshot load restarts |
| snapin_account_processed | counter | Accounts processed across all snapshots (inserted or otherwise) |
| snapin_account_batch_processed | counter | Account batches processed across all snapshots (parallelism indicator) |
Snapwr Tile
| Metric | Type | Description |
|---|---|---|
| snapwr_full_bytes_read | gauge | Number of decompressed snapshot bytes consumed from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapwr_incremental_bytes_read | gauge | Number of decompressed snapshot bytes consumed from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapwr_bytes_written | gauge | Number of bytes written to the accounts database on disk. Monotonically increasing across snapshot loads. |
| snapwr_accounts_written | gauge | Number of accounts written to the accounts database on disk. Might decrease if snapshot load is aborted and restarted |
Netlnk Tile
| Metric | Type | Description |
|---|---|---|
| netlnk_event_dropped | counter | Netlink drop events caught |
| netlnk_link_full_sync | counter | Full link table syncs done |
| netlnk_route_full_sync | counter | Full route table syncs done |
| netlnk_update_processed {netlink_message="link"} | counter | Netlink live updates processed (Link) |
| netlnk_update_processed {netlink_message="neighbor"} | counter | Netlink live updates processed (Neighbor Table Entry) |
| netlnk_update_processed {netlink_message="ipv4_route"} | counter | Netlink live updates processed (IPv4 Route Table Entry) |
| netlnk_interface_count | gauge | Network interfaces |
| netlnk_route_count {route_table="local"} | gauge | IPv4 routes (Local) |
| netlnk_route_count {route_table="main"} | gauge | IPv4 routes (Main) |
| netlnk_neighbor_probe_sent | counter | Neighbor solicit requests sent to kernel |
| netlnk_neighbor_probe_failed | counter | Neighbor solicit requests that failed to send (kernel too slow) |
| netlnk_neighbor_probe_rate_limit_host | counter | Neighbor solicit requests that exceeded the per-host rate limit |
| netlnk_neighbor_probe_rate_limit_global | counter | Neighbor solicit requests that exceeded the global rate limit |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_pkt_rx | counter | Packets received |
| net_pkt_rx_bytes | counter | Bytes received (including Ethernet header) |
| net_pkt_rx_undersize | counter | Incoming packets dropped due to being too small |
| net_pkt_rx_fill_ring_full | counter | Incoming packets dropped due to fill ring being full |
| net_pkt_rx_backpressure | counter | Incoming packets dropped due to backpressure |
| net_rx_buffer_busy | gauge | Receive buffers currently busy |
| net_rx_buffer_idle | gauge | Receive buffers currently idle |
| net_pkt_tx_submitted | counter | Packet transmit jobs submitted |
| net_pkt_tx_completed | counter | Packet transmit jobs marked as completed by the kernel |
| net_pkt_tx_bytes | counter | Bytes transmitted (including Ethernet header) |
| net_pkt_tx_no_route | counter | Packet transmit jobs dropped due to route failure |
| net_pkt_tx_no_neighbor | counter | Packet transmit jobs dropped due to unresolved neighbor |
| net_pkt_tx_ring_full | counter | Packet transmit jobs dropped due to XDP TX ring full or missing completions |
| net_tx_buffer_busy | gauge | Transmit buffers currently busy |
| net_tx_buffer_idle | gauge | Transmit buffers currently idle |
| net_xsk_syscall_tx | counter | XSK sendto syscalls dispatched |
| net_xsk_syscall_rx | counter | XSK recvmsg syscalls dispatched |
| net_xdp_rx_other_dropped | counter | Dropped for other reasons (xdp_statistics_v0.rx_dropped) |
| net_xdp_rx_invalid_descriptor | counter | Dropped due to invalid descriptor (xdp_statistics_v0.rx_invalid_descs) |
| net_xdp_tx_invalid_descriptor | counter | Dropped due to invalid descriptor (xdp_statistics_v0.tx_invalid_descs) |
| net_xdp_rx_ring_full | counter | Dropped due to rx ring being full (xdp_statistics_v1.rx_ring_full) |
| net_xdp_rx_fill_ring_empty | counter | Failed to retrieve item from fill ring because it was empty (xdp_statistics_v1.rx_fill_ring_empty_descs) |
| net_xdp_tx_ring_empty | counter | Failed to retrieve item from tx ring because it was empty (xdp_statistics_v1.tx_ring_empty_descs) |
| net_gre_pkt_rx | counter | Valid GRE packets received |
| net_gre_pkt_rx_invalid | counter | Invalid GRE packets received |
| net_gre_pkt_rx_ignored | counter | GRE packets received but ignored |
| net_gre_pkt_tx_submitted | counter | GRE packet transmit jobs submitted |
| net_gre_pkt_tx_no_route | counter | GRE packet transmit jobs dropped due to route failure |
| net_pkt_rx_src_invalid | counter | Incoming packets dropped due to invalid source IP address |
Sock Tile
| Metric | Type | Description |
|---|---|---|
| sock_syscall_tx {socket_error="no_error"} | counter | sendmmsg syscalls dispatched (No error) |
| sock_syscall_tx {socket_error="slow"} | counter | sendmmsg syscalls dispatched (ENOBUFS, EAGAIN error) |
| sock_syscall_tx {socket_error="permission"} | counter | sendmmsg syscalls dispatched (EPERM error (blocked by netfilter)) |
| sock_syscall_tx {socket_error="unreachable"} | counter | sendmmsg syscalls dispatched (ENETUNREACH, EHOSTUNREACH error) |
| sock_syscall_tx {socket_error="down"} | counter | sendmmsg syscalls dispatched (ENONET, ENETDOWN, EHOSTDOWN error) |
| sock_syscall_tx {socket_error="other"} | counter | sendmmsg syscalls dispatched (Unrecognized error code) |
| sock_syscall_rx | counter | recvmmsg syscalls dispatched |
| sock_pkt_rx | counter | Packets received |
| sock_pkt_tx | counter | Packets sent |
| sock_pkt_tx_failed | counter | Packets that failed to send |
| sock_pkt_tx_bytes | counter | Bytes transmitted (including Ethernet header) |
| sock_pkt_rx_bytes | counter | Bytes received (including Ethernet header) |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_txn_overrun | counter | Txns overrun before reassembled (too small txn_reassembly_count) |
| quic_txn_reassembly_started | counter | Fragmented txn receive ops started |
| quic_txn_reassembly_active | gauge | Fragmented txn receive ops currently active |
| quic_frag_rx | counter | Txn frags received |
| quic_frag_gap | counter | Txn frags dropped due to data gap |
| quic_frag_duplicate | counter | Txn frags dropped due to duplicate (stream already completed) |
| quic_txn_rx {tpu_rx_type="udp"} | counter | Txns received via TPU (TPU/UDP) |
| quic_txn_rx {tpu_rx_type="quic_fast"} | counter | Txns received via TPU (TPU/QUIC unfragmented) |
| quic_txn_rx {tpu_rx_type="quic_frag"} | counter | Txns received via TPU (TPU/QUIC fragmented) |
| quic_txn_abandoned | counter | Txns abandoned because a connection was lost |
| quic_txn_undersize | counter | Txns received via QUIC dropped because they were too small |
| quic_txn_oversize | counter | Txns received via QUIC dropped because they were too large |
| quic_legacy_txn_undersize | counter | Packets received on the non-QUIC port that were too small to be a valid IP packet |
| quic_legacy_txn_oversize | counter | Packets received on the non-QUIC port that were too large to be a valid transaction |
| quic_pkt_rx | counter | IP packets received |
| quic_pkt_rx_bytes | counter | Bytes received (including IP, UDP, QUIC headers) |
| quic_pkt_tx | counter | IP packets sent |
| quic_pkt_tx_bytes | counter | Bytes sent (including IP, UDP, QUIC headers) |
| quic_conn_in_use | gauge | QUIC connection slots currently in use (allocated from connection create until free, including handshaking connections) |
| quic_conn_state {quic_conn_state="invalid"} | gauge | QUIC connections in each state (Freed) |
| quic_conn_state {quic_conn_state="handshake"} | gauge | QUIC connections in each state (Handshaking peer) |
| quic_conn_state {quic_conn_state="handshake_complete"} | gauge | QUIC connections in each state (Handshake complete, confirming with peer) |
| quic_conn_state {quic_conn_state="active"} | gauge | QUIC connections in each state (Active connection) |
| quic_conn_state {quic_conn_state="peer_close"} | gauge | QUIC connections in each state (Peer requested close) |
| quic_conn_state {quic_conn_state="abort"} | gauge | QUIC connections in each state (Connection terminating due to error) |
| quic_conn_state {quic_conn_state="close_pending"} | gauge | QUIC connections in each state (Connection is closing) |
| quic_conn_state {quic_conn_state="dead"} | gauge | QUIC connections in each state (Connection about to be freed) |
| quic_conn_created | counter | Connections created |
| quic_conn_closed | counter | Connections gracefully closed |
| quic_conn_aborted | counter | Connections aborted |
| quic_conn_timed_out | counter | Connections timed out |
| quic_conn_retried | counter | Connections established with retry |
| quic_conn_error_no_slots | counter | Connections that failed to create due to lack of slots |
| quic_conn_error_retry_failed | counter | Connections that failed during retry (e.g. invalid token) |
| quic_pkt_no_conn {quic_pkt_handle="initial"} | counter | Packets with an unknown connection ID (Initial) |
| quic_pkt_no_conn {quic_pkt_handle="retry"} | counter | Packets with an unknown connection ID (Retry) |
| quic_pkt_no_conn {quic_pkt_handle="handshake"} | counter | Packets with an unknown connection ID (Handshake) |
| quic_pkt_no_conn {quic_pkt_handle="one_rtt"} | counter | Packets with an unknown connection ID (1-RTT) |
| quic_pkt_src_invalid | counter | Packets dropped due to a wrong source IP |
| quic_frame_meta_acquired {frame_tx_alloc_result="success"} | counter | Attempts to acquire QUIC frame metadata (Success) |
| quic_frame_meta_acquired {frame_tx_alloc_result="fail_empty_pool"} | counter | Attempts to acquire QUIC frame metadata (PktMetaPoolEmpty) |
| quic_frame_meta_acquired {frame_tx_alloc_result="fail_connection_max"} | counter | Attempts to acquire QUIC frame metadata (ConnMaxedInflightFrames) |
| quic_initial_pkt_rx {quic_initial_token_length="zero"} | counter | Initial packets grouped by token length (No token) |
| quic_initial_pkt_rx {quic_initial_token_length="fd_quic_length"} | counter | Initial packets grouped by token length (fd_quic retry token length) |
| quic_initial_pkt_rx {quic_initial_token_length="invalid_length"} | counter | Initial packets grouped by token length (Invalid token length) |
| quic_handshake_created | counter | Handshake flows created |
| quic_handshake_error_alloc_fail | counter | Handshakes dropped due to alloc fail |
| quic_handshake_evicted | counter | Handshakes dropped due to eviction |
| quic_stream_rx | counter | Stream receive events |
| quic_stream_rx_bytes | counter | Stream payload bytes received |
| quic_frame_rx {quic_frame_type="unknown"} | counter | QUIC frames received (Unknown frame type) |
| quic_frame_rx {quic_frame_type="ack"} | counter | QUIC frames received (ACK frame) |
| quic_frame_rx {quic_frame_type="reset_stream"} | counter | QUIC frames received (RESET_STREAM frame) |
| quic_frame_rx {quic_frame_type="stop_sending"} | counter | QUIC frames received (STOP_SENDING frame) |
| quic_frame_rx {quic_frame_type="crypto"} | counter | QUIC frames received (CRYPTO frame) |
| quic_frame_rx {quic_frame_type="new_token"} | counter | QUIC frames received (NEW_TOKEN frame) |
| quic_frame_rx {quic_frame_type="stream"} | counter | QUIC frames received (STREAM frame) |
| quic_frame_rx {quic_frame_type="max_data"} | counter | QUIC frames received (MAX_DATA frame) |
| quic_frame_rx {quic_frame_type="max_stream_data"} | counter | QUIC frames received (MAX_STREAM_DATA frame) |
| quic_frame_rx {quic_frame_type="max_streams"} | counter | QUIC frames received (MAX_STREAMS frame) |
| quic_frame_rx {quic_frame_type="data_blocked"} | counter | QUIC frames received (DATA_BLOCKED frame) |
| quic_frame_rx {quic_frame_type="stream_data_blocked"} | counter | QUIC frames received (STREAM_DATA_BLOCKED frame) |
| quic_frame_rx {quic_frame_type="streams_blocked"} | counter | QUIC frames received (STREAMS_BLOCKED(bidi) frame) |
| quic_frame_rx {quic_frame_type="new_connection_id"} | counter | QUIC frames received (NEW_CONN_ID frame) |
| quic_frame_rx {quic_frame_type="retire_connection_id"} | counter | QUIC frames received (RETIRE_CONN_ID frame) |
| quic_frame_rx {quic_frame_type="path_challenge"} | counter | QUIC frames received (PATH_CHALLENGE frame) |
| quic_frame_rx {quic_frame_type="path_response"} | counter | QUIC frames received (PATH_RESPONSE frame) |
| quic_frame_rx {quic_frame_type="connection_close_quic"} | counter | QUIC frames received (CONN_CLOSE(transport) frame) |
| quic_frame_rx {quic_frame_type="connection_close_app"} | counter | QUIC frames received (CONN_CLOSE(app) frame) |
| quic_frame_rx {quic_frame_type="handshake_done"} | counter | QUIC frames received (HANDSHAKE_DONE frame) |
| quic_frame_rx {quic_frame_type="ping"} | counter | QUIC frames received (PING frame) |
| quic_frame_rx {quic_frame_type="padding"} | counter | QUIC frames received (PADDING frame) |
| quic_ack_tx {quic_ack_tx="noop"} | counter | ACK events (Non-ACK-eliciting packet) |
| quic_ack_tx {quic_ack_tx="new"} | counter | ACK events (New ACK range) |
| quic_ack_tx {quic_ack_tx="merged"} | counter | ACK events (Merged into existing ACK range) |
| quic_ack_tx {quic_ack_tx="drop"} | counter | ACK events (Out of buffers) |
| quic_ack_tx {quic_ack_tx="cancel"} | counter | ACK events (ACK suppressed by handler) |
| quic_service_duration_seconds | histogram | Duration spent in service |
| quic_rx_duration_seconds | histogram | Duration spent processing packets |
| quic_frame_parse_failed | counter | QUIC frames that failed to parse |
| quic_pkt_crypto_failed {quic_encryption_level="initial"} | counter | Packets that failed decryption (Initial) |
| quic_pkt_crypto_failed {quic_encryption_level="early"} | counter | Packets that failed decryption (Early data) |
| quic_pkt_crypto_failed {quic_encryption_level="handshake"} | counter | Packets that failed decryption (Handshake) |
| quic_pkt_crypto_failed {quic_encryption_level="app"} | counter | Packets that failed decryption (App data) |
| quic_pkt_no_key {quic_encryption_level="initial"} | counter | Packets that failed decryption due to missing key (Initial) |
| quic_pkt_no_key {quic_encryption_level="early"} | counter | Packets that failed decryption due to missing key (Early data) |
| quic_pkt_no_key {quic_encryption_level="handshake"} | counter | Packets that failed decryption due to missing key (Handshake) |
| quic_pkt_no_key {quic_encryption_level="app"} | counter | Packets that failed decryption due to missing key (App data) |
| quic_pkt_net_header_invalid | counter | Packets dropped due to weird IP or UDP header |
| quic_pkt_header_invalid | counter | Packets dropped due to weird QUIC header |
| quic_pkt_undersize | counter | QUIC packets dropped due to being too small |
| quic_pkt_oversize | counter | QUIC packets dropped due to being too large |
| quic_pkt_rx_version_negotiation | counter | QUIC version negotiation packets received |
| quic_pkt_tx_retry | counter | QUIC Retry packets sent |
| quic_pkt_tx_retransmitted {quic_encryption_level="initial"} | counter | QUIC packets retransmitted (Initial) |
| quic_pkt_tx_retransmitted {quic_encryption_level="early"} | counter | QUIC packets retransmitted (Early data) |
| quic_pkt_tx_retransmitted {quic_encryption_level="handshake"} | counter | QUIC packets retransmitted (Handshake) |
| quic_pkt_tx_retransmitted {quic_encryption_level="app"} | counter | QUIC packets retransmitted (App data) |
Bundle Tile
| Metric | Type | Description |
|---|---|---|
| bundle_txn_rx | counter | Transactions received, including transactions within bundles |
| bundle_pkt_rx | counter | Packets received |
| bundle_protobuf_rx_bytes | counter | Bytes from received gRPC protobuf payloads |
| bundle_bundle_rx | counter | Bundles received |
| bundle_conn_error {bundle_error="protobuf"} | counter | gRPC errors encountered (Protobuf decode/encode error) |
| bundle_conn_error {bundle_error="transport"} | counter | gRPC errors encountered (Transport error) |
| bundle_conn_error {bundle_error="timeout"} | counter | gRPC errors encountered (I/O timeout) |
| bundle_conn_error {bundle_error="no_fee_info"} | counter | gRPC errors encountered (Bundle dropped due to missing fee info) |
| bundle_conn_error {bundle_error="ssl_alloc"} | counter | gRPC errors encountered (OpenSSL alloc fail) |
| bundle_heap_size_bytes | gauge | Workspace heap size |
| bundle_heap_free_bytes | gauge | Approximate free space in workspace |
| bundle_shredstream_heartbeat_sent | counter | ShredStream heartbeats successfully sent |
| bundle_ping_acked | counter | HTTP/2 PINGs acknowledged by server |
| bundle_state | gauge | 0=disconnected, 1=connecting, 2=connected, 3=sleeping |
| bundle_rtt_sample_nanos | gauge | Latest RTT sample at scrape time, in nanoseconds |
| bundle_rtt_smoothed_nanos | gauge | RTT moving average, in nanoseconds |
| bundle_rtt_variance_nanos | gauge | RTT variance, in nanoseconds |
| bundle_message_rx_delay_nanos | histogram | Message receive delay from bundle server to bundle client, in nanoseconds |
| bundle_txn_pending | gauge | Transactions buffered and waiting to be published |
| bundle_txn_buffer_full | counter | Transactions dropped because the pending transaction buffer was full (should never happen) |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_txn_result {verify_tile_result="success"} | counter | Transaction results through verify tile (Transaction verified successfully) |
| verify_txn_result {verify_tile_result="bundle_peer_failure"} | counter | Transaction results through verify tile (Peer transaction in the bundle failed) |
| verify_txn_result {verify_tile_result="parse_failure"} | counter | Transaction results through verify tile (Transaction failed to parse) |
| verify_txn_result {verify_tile_result="dedup_failure"} | counter | Transaction results through verify tile (Transaction failed deduplication) |
| verify_txn_result {verify_tile_result="verify_failure"} | counter | Transaction results through verify tile (Transaction failed signature verification) |
| verify_vote_gossip_rx | counter | Simple vote transactions received over gossip instead of via the normal TPU path |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_txn_result {dedup_tile_result="success"} | counter | Transaction results through dedup tile (Transaction deduplicated successfully) |
| dedup_txn_result {dedup_tile_result="bundle_peer_failure"} | counter | Transaction results through dedup tile (Peer transaction in the bundle failed) |
| dedup_txn_result {dedup_tile_result="dedup_failure"} | counter | Transaction results through dedup tile (Transaction failed deduplication) |
| dedup_vote_gossip_rx | counter | Simple vote transactions received over gossip instead of via the normal TPU path |
Resolv Tile
| Metric | Type | Description |
|---|---|---|
| resolv_txn_no_bank | counter | Transactions dropped because the bank was not available |
| resolv_stash_operation {resolve_stash_operation="inserted"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolv_stash_operation {resolve_stash_operation="overrun"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolv_stash_operation {resolve_stash_operation="published"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolv_stash_operation {resolve_stash_operation="removed"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories) |
| resolv_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolv_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolv_lut_resolved {lut_resolve_result="account_not_found"} | counter | Address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolv_lut_resolved {lut_resolve_result="success"} | counter | Address lookup tables resolved (Resolved successfully) |
| resolv_blockhash_expired | counter | Transactions that failed to resolve because the blockhash was expired |
| resolv_txn_bundle_peer_failed | counter | Transactions that failed to resolve because a peer transaction in the bundle failed |
| resolv_accdb_account_acquired {accdb_cache_class="class0"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (0-128 B) |
| resolv_accdb_account_acquired {accdb_cache_class="class1"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (129-512 B) |
| resolv_accdb_account_acquired {accdb_cache_class="class2"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| resolv_accdb_account_acquired {accdb_cache_class="class3"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (2-8 KiB) |
| resolv_accdb_account_acquired {accdb_cache_class="class4"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (8-32 KiB) |
| resolv_accdb_account_acquired {accdb_cache_class="class5"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (32-128 KiB) |
| resolv_accdb_account_acquired {accdb_cache_class="class6"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| resolv_accdb_account_acquired {accdb_cache_class="class7"} | counter | Number of accounts read from the account database (one per address lookup table account fetched), attributed to the cache size class of the account's current data size (1-10 MiB) |
| resolv_accdb_account_not_found {accdb_cache_class="class0"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (0-128 B) |
| resolv_accdb_account_not_found {accdb_cache_class="class1"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (129-512 B) |
| resolv_accdb_account_not_found {accdb_cache_class="class2"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (513 B-2 KiB) |
| resolv_accdb_account_not_found {accdb_cache_class="class3"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (2-8 KiB) |
| resolv_accdb_account_not_found {accdb_cache_class="class4"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (8-32 KiB) |
| resolv_accdb_account_not_found {accdb_cache_class="class5"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (32-128 KiB) |
| resolv_accdb_account_not_found {accdb_cache_class="class6"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (128 KiB-1 MiB) |
| resolv_accdb_account_not_found {accdb_cache_class="class7"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (1-10 MiB) |
| resolv_accdb_account_waited | counter | Number of accounts that had to wait for a concurrent writer to publish a disk offset before being read |
| resolv_accdb_batch_acquired | counter | Number of fd_accdb_acquire (batch acquire) calls made by this tile |
| resolv_accdb_bytes_read | counter | Number of bytes read from the account database |
| resolv_accdb_read_operation | counter | Number of read operations performed on the account database |
| resolv_accdb_bytes_copied | counter | Number of bytes copied out of the account database cache on a cache hit |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_schedule_microblock_duration_seconds | histogram | Duration of scheduling one microblock |
| pack_no_schedule_microblock_duration_seconds | histogram | Duration of discovering that there are no schedulable transactions |
| pack_insert_transaction_duration_seconds | histogram | Duration of inserting one transaction into the pool of available transactions |
| pack_complete_microblock_duration_seconds | histogram | Duration of the computation associated with marking one microblock as complete |
| pack_txn_per_microblock | histogram | Transactions in a scheduled microblock, including both votes and non-votes |
| pack_vote_per_microblock | histogram | Simple vote transactions in a scheduled microblock |
| pack_txn_normal_rx | counter | Transactions received via the normal TPU path |
| pack_txn_inserted {pack_txn_insert_return="instr_acct_cnt"} | counter | Result of inserting a transaction into the pack object (Transaction has an instruction that references too many accounts) |
| pack_txn_inserted {pack_txn_insert_return="nonce_conflict"} | counter | Result of inserting a transaction into the pack object (Bundle with two conflicting durable nonce transactions) |
| pack_txn_inserted {pack_txn_insert_return="acct_blocklist"} | counter | Result of inserting a transaction into the pack object (Transaction uses an account on the account blocklist) |
| pack_txn_inserted {pack_txn_insert_return="bundle_blacklist"} | counter | Result of inserting a transaction into the pack object (Transaction is a vote in a bundle or uses an account on the bundle blacklist) |
| pack_txn_inserted {pack_txn_insert_return="invalid_nonce"} | counter | Result of inserting a transaction into the pack object (Transaction is an invalid durable nonce transaction) |
| pack_txn_inserted {pack_txn_insert_return="write_sysvar"} | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) |
| pack_txn_inserted {pack_txn_insert_return="estimation_fail"} | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) |
| pack_txn_inserted {pack_txn_insert_return="duplicate_account"} | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) |
| pack_txn_inserted {pack_txn_insert_return="too_many_accounts"} | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) |
| pack_txn_inserted {pack_txn_insert_return="too_large"} | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) |
| pack_txn_inserted {pack_txn_insert_return="expired"} | counter | Result of inserting a transaction into the pack object (Transaction already expired) |
| pack_txn_inserted {pack_txn_insert_return="addr_lut"} | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) |
| pack_txn_inserted {pack_txn_insert_return="unaffordable"} | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) |
| pack_txn_inserted {pack_txn_insert_return="duplicate"} | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) |
| pack_txn_inserted {pack_txn_insert_return="nonce_priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce) |
| pack_txn_inserted {pack_txn_insert_return="priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_txn_inserted {pack_txn_insert_return="nonvote_add"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) |
| pack_txn_inserted {pack_txn_insert_return="vote_add"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) |
| pack_txn_inserted {pack_txn_insert_return="nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) |
| pack_txn_inserted {pack_txn_insert_return="vote_replace"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) |
| pack_txn_inserted {pack_txn_insert_return="nonce_nonvote_add"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction added to pending transactions) |
| pack_txn_inserted {pack_txn_insert_return="unused"} | counter | Result of inserting a transaction into the pack object (Unused because durable nonce transactions can't be simple votes) |
| pack_txn_inserted {pack_txn_insert_return="nonce_nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce) |
| pack_state_duration_nanos {pack_timing_state="no_txn_no_execle_no_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: not leader, no transactions, no available execle) |
| pack_state_duration_nanos {pack_timing_state="txn_no_execle_no_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: not leader, transactions available, no available execle) |
| pack_state_duration_nanos {pack_timing_state="no_txn_execle_no_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: not leader, no transactions, execle available) |
| pack_state_duration_nanos {pack_timing_state="txn_execle_no_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: not leader, transactions available, execle available) |
| pack_state_duration_nanos {pack_timing_state="no_txn_no_execle_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: leader, no transactions, no available execle) |
| pack_state_duration_nanos {pack_timing_state="txn_no_execle_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: leader, transactions available, no available execle) |
| pack_state_duration_nanos {pack_timing_state="no_txn_execle_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: leader, no transactions, execle available) |
| pack_state_duration_nanos {pack_timing_state="txn_execle_leader_no_microblock"} | counter | Time spent in each state, in nanoseconds (No microblock: leader, transactions available, execle available (all conflicting)) |
| pack_state_duration_nanos {pack_timing_state="no_txn_no_execle_no_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: not leader, no transactions, no available execle) |
| pack_state_duration_nanos {pack_timing_state="txn_no_execle_no_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: not leader, transactions available, no available execle) |
| pack_state_duration_nanos {pack_timing_state="no_txn_execle_no_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: not leader, no transactions, execle available) |
| pack_state_duration_nanos {pack_timing_state="txn_execle_no_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: not leader, transactions available, execle available) |
| pack_state_duration_nanos {pack_timing_state="no_txn_no_execle_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: leader, no transactions, no available execle) |
| pack_state_duration_nanos {pack_timing_state="txn_no_execle_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: leader, transactions available, no available execle) |
| pack_state_duration_nanos {pack_timing_state="no_txn_execle_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: leader, no transactions, execle available) |
| pack_state_duration_nanos {pack_timing_state="txn_execle_leader_microblock"} | counter | Time spent in each state, in nanoseconds (Scheduled a microblock: leader, transactions available, execle available) |
| pack_txn_extra_dropped | counter | Transactions dropped from the extra transaction storage because it was full |
| pack_txn_extra_inserted | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full |
| pack_txn_extra_retrieved | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage |
| pack_txn_expired | counter | Transactions deleted from pack because their TTL expired |
| pack_txn_deleted | counter | Transactions dropped from pack because they were requested to be deleted |
| pack_txn_already_executed | counter | Transactions dropped from pack because they were already executed (in either the replay or leader pipeline) |
| pack_txn_partial_bundle | counter | Transactions dropped from pack because they were part of a partial bundle |
| pack_txn_available {available_txn_type="all"} | gauge | Pending transactions in pack's pool that are available to be scheduled (All transactions in any treap) |
| pack_txn_available {available_txn_type="regular"} | gauge | Pending transactions in pack's pool that are available to be scheduled (Non-votes in the main treap) |
| pack_txn_available {available_txn_type="votes"} | gauge | Pending transactions in pack's pool that are available to be scheduled (Simple votes) |
| pack_txn_available {available_txn_type="conflicting"} | gauge | Pending transactions in pack's pool that are available to be scheduled (Non-votes that write to a hotly-contended account) |
| pack_txn_available {available_txn_type="bundles"} | gauge | Pending transactions in pack's pool that are available to be scheduled (Transactions that are part of a bundle) |
| pack_txn_pending_capacity | gauge | Maximum number of pending transactions that pack can consider. Fixed at Firedancer startup but a useful reference for TxnAvailable |
| pack_txn_pending_smallest_cu | gauge | Lower bound on the smallest non-vote transaction (in cost units) that is immediately available for scheduling |
| pack_microblock_per_block_limit_reached | counter | Times pack did not pack a microblock because the microblocks-per-block limit had been reached |
| pack_data_per_block_limit_reached | counter | Times pack did not pack a microblock because it reached the data per block limit at the start of trying to schedule a microblock |
| pack_txn_scheduled {pack_txn_schedule="taken"} | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) |
| pack_txn_scheduled {pack_txn_schedule="cu_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_txn_scheduled {pack_txn_schedule="fast_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) |
| pack_txn_scheduled {pack_txn_schedule="byte_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_txn_scheduled {pack_txn_schedule="alloc_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the data allocation limit) |
| pack_txn_scheduled {pack_txn_schedule="write_cost"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_txn_scheduled {pack_txn_schedule="slow_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) |
| pack_txn_scheduled {pack_txn_schedule="defer_skip"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction it previously exceeded the per-account block write cost limit too many times) |
| pack_bundle_crank_result {bundle_crank_result="not_needed"} | counter | Result of considering whether bundle cranks are needed (On-chain state in the correct state) |
| pack_bundle_crank_result {bundle_crank_result="inserted"} | counter | Result of considering whether bundle cranks are needed (Inserted an initializer bundle to update the on-chain state) |
| pack_bundle_crank_result {bundle_crank_result="creation_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but creation failed) |
| pack_bundle_crank_result {bundle_crank_result="insertion_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but insertion failed) |
| pack_block_cu_consumed | gauge | Cost units consumed in the current block, or 0 if pack is not currently packing a block |
| pack_cu_scheduled_per_block | histogram | Cost units scheduled for each block pack produced. Can be higher than the block limit because of returned CUs |
| pack_cu_rebated_per_block | histogram | Compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses |
| pack_cu_net_per_block | histogram | Net cost units (scheduled - rebated) in each block pack produced |
| pack_cu_pct | histogram | Percent of the total block cost limit used for each block pack produced |
| pack_txn_delete_missed | counter | Attempts to delete a transaction that wasn't found |
| pack_txn_delete_hit | counter | Attempts to delete a transaction that was found and deleted |
Execle Tile
| Metric | Type | Description |
|---|---|---|
| execle_txn_result {transaction_result="success"} | counter | Result of loading and executing a transaction (Transaction executed successfully) |
| execle_txn_result {transaction_result="instruction_error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) |
| execle_txn_result {transaction_result="account_not_found"} | counter | Result of loading and executing a transaction (The transaction fee payer address was not found) |
| execle_txn_result {transaction_result="program_account_not_found"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was not found) |
| execle_txn_result {transaction_result="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction (The transaction fee payer did not have balance to pay the fee) |
| execle_txn_result {transaction_result="invalid_account_for_fee"} | counter | Result of loading and executing a transaction (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) |
| execle_txn_result {transaction_result="already_processed"} | counter | Result of loading and executing a transaction (The transaction has already been processed in a recent block) |
| execle_txn_result {transaction_result="blockhash_not_found"} | counter | Result of loading and executing a transaction (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) |
| execle_txn_result {transaction_result="invalid_program_for_execution"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was no executable. TODO: No longer needed with SIMD-0162) |
| execle_txn_result {transaction_result="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that does not exist or is inactive) |
| execle_txn_result {transaction_result="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that is not owned by the ALUT program account) |
| execle_txn_result {transaction_result="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that contains data which is not a valid ALUT) |
| execle_txn_result {transaction_result="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction (The transaction references an account offset from the ALUT which does not exist) |
| execle_txn_result {transaction_result="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction (The total account data size of the loaded accounts exceeds the consensus limit) |
| execle_txn_result {transaction_result="duplicate_instruction"} | counter | Result of loading and executing a transaction (A compute budget program instruction was invoked more than once) |
| execle_txn_result {transaction_result="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction (The compute budget program was invoked and set the loaded accounts data size to zero) |
| execle_txn_result {transaction_result="nonce_already_advanced"} | counter | Result of loading and executing a transaction (The transaction references a nonce account that is already advanced) |
| execle_txn_result {transaction_result="nonce_advance_failed"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the advance instruction was not valid or failed) |
| execle_txn_result {transaction_result="nonce_wrong_blockhash"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the blockhash is not the correct one) |
| execle_txn_result {transaction_result="sanitize_failure"} | counter | Result of loading and executing a transaction (The process for sanitizing the transaction failed) |
| execle_txn_result {transaction_result="account_in_use"} | counter | Result of loading and executing a transaction (The transaction conflicts with another transaction in the microblock. TODO: No longer possible with smart dispatcher) |
| execle_txn_result {transaction_result="account_loaded_twice"} | counter | Result of loading and executing a transaction (The transaction references the same account twice) |
| execle_txn_result {transaction_result="signature_failure"} | counter | Result of loading and executing a transaction (The transaction had an invalid signature) |
| execle_txn_result {transaction_result="too_many_account_locks"} | counter | Result of loading and executing a transaction (The transaction references too many accounts. TODO: No longer possible with smart dispatcher) |
| execle_txn_result {transaction_result="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction (The transaction would leave an account with a lower balance than the rent-exempt minimum) |
| execle_txn_result {transaction_result="unbalanced_transaction"} | counter | Result of loading and executing a transaction (The total referenced account lamports before and after the transaction was unbalanced) |
| execle_txn_result {transaction_result="bundle_peer"} | counter | Result of loading and executing a transaction (The transaction was part of a bundle and an earlier transaction in the bundle failed) |
| execle_txn_landed {transaction_landed="landed_success"} | counter | Whether a transaction landed in the block or not (Transaction landed) |
| execle_txn_landed {transaction_landed="landed_fees_only"} | counter | Whether a transaction landed in the block or not (Transaction landed, but was fees only and did not execute) |
| execle_txn_landed {transaction_landed="landed_failed"} | counter | Whether a transaction landed in the block or not (Transaction landed, but failed to execute) |
| execle_txn_landed {transaction_landed="unlanded"} | counter | Whether a transaction landed in the block or not (Transaction did not land) |
| execle_cu_executed | counter | Estimated compute units executed since tile start |
| execle_txn_regime_duration_nanos {txn_regime="setup"} | counter | Mutually exclusive and exhaustive duration spent in transaction execution regimes, in nanoseconds (Transaction setup) |
| execle_txn_regime_duration_nanos {txn_regime="exec"} | counter | Mutually exclusive and exhaustive duration spent in transaction execution regimes, in nanoseconds (Transaction execution (includes VM setup/execution)) |
| execle_txn_regime_duration_nanos {txn_regime="commit"} | counter | Mutually exclusive and exhaustive duration spent in transaction execution regimes, in nanoseconds (Transaction result commit) |
| execle_vm_regime_duration_nanos {vm_regime="setup"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM setup) |
| execle_vm_regime_duration_nanos {vm_regime="commit"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM commit) |
| execle_vm_regime_duration_nanos {vm_regime="setup_cpi"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM setup (CPI)) |
| execle_vm_regime_duration_nanos {vm_regime="commit_cpi"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM commit (CPI)) |
| execle_vm_regime_duration_nanos {vm_regime="interpreter"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM interpreter execution) |
| execle_accdb_account_acquired {accdb_cache_class="class0"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| execle_accdb_account_acquired {accdb_cache_class="class1"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| execle_accdb_account_acquired {accdb_cache_class="class2"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| execle_accdb_account_acquired {accdb_cache_class="class3"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| execle_accdb_account_acquired {accdb_cache_class="class4"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| execle_accdb_account_acquired {accdb_cache_class="class5"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| execle_accdb_account_acquired {accdb_cache_class="class6"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| execle_accdb_account_acquired {accdb_cache_class="class7"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class0"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class1"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class2"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class3"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class4"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class5"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class6"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| execle_accdb_account_writable_acquired {accdb_cache_class="class7"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| execle_accdb_account_evicted {accdb_cache_class="class0"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (0-128 B) |
| execle_accdb_account_evicted {accdb_cache_class="class1"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (129-512 B) |
| execle_accdb_account_evicted {accdb_cache_class="class2"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (513 B-2 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="class3"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (2-8 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="class4"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (8-32 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="class5"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (32-128 KiB) |
| execle_accdb_account_evicted {accdb_cache_class="class6"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (128 KiB-1 MiB) |
| execle_accdb_account_evicted {accdb_cache_class="class7"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (1-10 MiB) |
| execle_accdb_account_committed_new {accdb_cache_class="class0"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| execle_accdb_account_committed_new {accdb_cache_class="class1"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| execle_accdb_account_committed_new {accdb_cache_class="class2"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| execle_accdb_account_committed_new {accdb_cache_class="class3"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| execle_accdb_account_committed_new {accdb_cache_class="class4"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| execle_accdb_account_committed_new {accdb_cache_class="class5"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| execle_accdb_account_committed_new {accdb_cache_class="class6"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| execle_accdb_account_committed_new {accdb_cache_class="class7"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class0"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class1"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class2"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class3"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class4"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class5"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class6"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| execle_accdb_account_committed_overwrite {accdb_cache_class="class7"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| execle_accdb_account_not_found {accdb_cache_class="class0"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (0-128 B) |
| execle_accdb_account_not_found {accdb_cache_class="class1"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (129-512 B) |
| execle_accdb_account_not_found {accdb_cache_class="class2"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (513 B-2 KiB) |
| execle_accdb_account_not_found {accdb_cache_class="class3"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (2-8 KiB) |
| execle_accdb_account_not_found {accdb_cache_class="class4"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (8-32 KiB) |
| execle_accdb_account_not_found {accdb_cache_class="class5"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (32-128 KiB) |
| execle_accdb_account_not_found {accdb_cache_class="class6"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (128 KiB-1 MiB) |
| execle_accdb_account_not_found {accdb_cache_class="class7"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (1-10 MiB) |
| execle_accdb_account_waited | counter | Number of accounts that were needed for transaction execution but were concurrently loading into cache by another transaction, causing the transaction to wait |
| execle_accdb_batch_acquired | counter | Number of fd_accdb_acquire (batch acquire) calls made by this tile |
| execle_accdb_acquire_failed | counter | Number of times we failed to acquire the cache lines needed for a transaction, and had to spin loop waiting |
| execle_accdb_bytes_read | counter | Number of bytes read from the account database |
| execle_accdb_read_operation | counter | Number of read operations performed on the account database |
| execle_accdb_bytes_written | counter | Number of bytes written to the account database |
| execle_accdb_write_operation | counter | Number of write operations performed on the account database |
| execle_accdb_bytes_copied | counter | Number of bytes copied within the account database |
Poh Tile
| Metric | Type | Description |
|---|
Sign Tile
| Metric | Type | Description |
|---|---|---|
| sign_sign_duration_seconds | histogram | Duration of signing a message |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_contact_info_per_message | histogram | Contact infos in the cluster contact info message |
| shred_microblock_abandoned | counter | Microblocks abandoned because we switched slots without finishing the current slot |
| shred_block_id_invalid | counter | Times a block was created with unknown parent block_id |
| shred_batch_size_bytes | histogram | Size of each microblock batch that is shredded, in bytes |
| shred_microblock_per_batch | histogram | Microblocks in each microblock batch that is shredded |
| shred_shredding_duration_seconds | histogram | Duration of producing one FEC set from the shredder |
| shred_add_shred_duration_seconds | histogram | Duration of verifying and processing one shred received from the network |
| shred_shred_processed {shred_processing_result="bad_slot"} | counter | Result of processing a shred from the network (Shred was for a slot for which we don't know the leader) |
| shred_shred_processed {shred_processing_result="parse_failed"} | counter | Result of processing a shred from the network (Shred parsing failed) |
| shred_shred_processed {shred_processing_result="equivocated"} | counter | Result of processing a shred from the network (Shred was equivocated with another shred) |
| shred_shred_processed {shred_processing_result="rejected"} | counter | Result of processing a shred from the network (Shred was invalid for one of many reasons) |
| shred_shred_processed {shred_processing_result="ignored"} | counter | Result of processing a shred from the network (Shred was ignored because we had already reconstructed the FEC set, or was for an old slot) |
| shred_shred_processed {shred_processing_result="duplicate"} | counter | Result of processing a shred from the network (Shred was a duplicate of a previously received valid shred of an incomplete FEC set) |
| shred_shred_processed {shred_processing_result="okay"} | counter | Result of processing a shred from the network (Shred accepted to an incomplete FEC set) |
| shred_shred_processed {shred_processing_result="completes"} | counter | Result of processing a shred from the network (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_spilled | counter | FEC sets spilled because they didn't complete in time and we needed space |
| shred_shred_initial_rejected | counter | Shreds rejected before any resources were allocated for the FEC set |
| shred_shred_unchained_rejected | counter | Shreds rejected because they're not chained merkle shreds |
| shred_fec_fatal_rejected | counter | FEC sets rejected for reasons that cause the whole FEC set to become invalid |
| shred_nonce_invalid | counter | Shreds received that seemed to be repair responses but had an incorrect nonce |
| shred_shred_repair_rx | counter | Repair shreds received |
| shred_shred_repair_rx_bytes | counter | Bytes received from network packets with repair shreds, including network headers |
| shred_shred_turbine_rx | counter | Turbine shreds received |
| shred_shred_turbine_rx_bytes | counter | Bytes received from network packets with turbine shreds, including network headers |
Gossvf Tile
| Metric | Type | Description |
|---|---|---|
| gossvf_message_rx {gossvf_message_outcome="success_pull_request"} | counter | Gossip messages processed (Pull Request (success)) |
| gossvf_message_rx {gossvf_message_outcome="success_pull_response"} | counter | Gossip messages processed (Pull Response (success)) |
| gossvf_message_rx {gossvf_message_outcome="success_push"} | counter | Gossip messages processed (Push (success)) |
| gossvf_message_rx {gossvf_message_outcome="success_prune"} | counter | Gossip messages processed (Prune (success)) |
| gossvf_message_rx {gossvf_message_outcome="success_ping"} | counter | Gossip messages processed (Ping (success)) |
| gossvf_message_rx {gossvf_message_outcome="success_pong"} | counter | Gossip messages processed (Pong (success)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_unparseable"} | counter | Gossip messages processed (Unparseable) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_not_contact_info"} | counter | Gossip messages processed (Pull Request (not contact info)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_loopback"} | counter | Gossip messages processed (Pull Request (loopback)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_inactive"} | counter | Gossip messages processed (Pull Request (inactive)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_wallclock"} | counter | Gossip messages processed (Pull Request (wallclock)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_signature"} | counter | Gossip messages processed (Pull Request (signature)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_shred_version"} | counter | Gossip messages processed (Pull Request (shred version)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_request_mask_bits"} | counter | Gossip messages processed (Pull Request (mask bits)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_prune_destination"} | counter | Gossip messages processed (Prune (destination)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_prune_wallclock"} | counter | Gossip messages processed (Prune (wallclock)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_prune_signature"} | counter | Gossip messages processed (Prune (signature)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_push_loopback"} | counter | Gossip messages processed (Push (loopback)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_push_no_valid_crds"} | counter | Gossip messages processed (Push (no valid crds)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_response_loopback"} | counter | Gossip messages processed (Pull Response (loopback)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pull_response_no_valid_crds"} | counter | Gossip messages processed (Pull Response (no valid crds)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_ping_signature"} | counter | Gossip messages processed (Ping (signature)) |
| gossvf_message_rx {gossvf_message_outcome="dropped_pong_signature"} | counter | Gossip messages processed (Pong (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pull_request"} | counter | Wire bytes of gossip messages processed (Pull Request (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pull_response"} | counter | Wire bytes of gossip messages processed (Pull Response (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_push"} | counter | Wire bytes of gossip messages processed (Push (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_prune"} | counter | Wire bytes of gossip messages processed (Prune (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_ping"} | counter | Wire bytes of gossip messages processed (Ping (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pong"} | counter | Wire bytes of gossip messages processed (Pong (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_unparseable"} | counter | Wire bytes of gossip messages processed (Unparseable) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_not_contact_info"} | counter | Wire bytes of gossip messages processed (Pull Request (not contact info)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_loopback"} | counter | Wire bytes of gossip messages processed (Pull Request (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_inactive"} | counter | Wire bytes of gossip messages processed (Pull Request (inactive)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_wallclock"} | counter | Wire bytes of gossip messages processed (Pull Request (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_signature"} | counter | Wire bytes of gossip messages processed (Pull Request (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_shred_version"} | counter | Wire bytes of gossip messages processed (Pull Request (shred version)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_mask_bits"} | counter | Wire bytes of gossip messages processed (Pull Request (mask bits)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_destination"} | counter | Wire bytes of gossip messages processed (Prune (destination)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_wallclock"} | counter | Wire bytes of gossip messages processed (Prune (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_signature"} | counter | Wire bytes of gossip messages processed (Prune (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_push_loopback"} | counter | Wire bytes of gossip messages processed (Push (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_push_no_valid_crds"} | counter | Wire bytes of gossip messages processed (Push (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_response_loopback"} | counter | Wire bytes of gossip messages processed (Pull Response (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_response_no_valid_crds"} | counter | Wire bytes of gossip messages processed (Pull Response (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_ping_signature"} | counter | Wire bytes of gossip messages processed (Ping (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pong_signature"} | counter | Wire bytes of gossip messages processed (Pong (signature)) |
| gossvf_crds_rx {gossvf_crds_outcome="success_pull_response"} | counter | CRDS values processed (Pull Response (success)) |
| gossvf_crds_rx {gossvf_crds_outcome="success_push"} | counter | CRDS values processed (Push (success)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_pull_response_duplicate"} | counter | CRDS values processed (Pull Response (duplicate)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_pull_response_signature"} | counter | CRDS values processed (Pull Response (signature)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_pull_response_origin_no_contact_info"} | counter | CRDS values processed (Pull Response (origin no contact info)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_pull_response_origin_shred_version"} | counter | CRDS values processed (Pull Response (origin shred version)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_pull_response_inactive"} | counter | CRDS values processed (Pull Response (inactive)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_pull_response_wallclock"} | counter | CRDS values processed (Pull Response (wallclock)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_push_signature"} | counter | CRDS values processed (Push (signature)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_push_origin_no_contact_info"} | counter | CRDS values processed (Push (origin no contact info)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_push_origin_shred_version"} | counter | CRDS values processed (Push (origin shred version)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_push_inactive"} | counter | CRDS values processed (Push (inactive)) |
| gossvf_crds_rx {gossvf_crds_outcome="dropped_push_wallclock"} | counter | CRDS values processed (Push (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="success_pull_response"} | counter | Wire bytes of CRDS values processed (Pull Response (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="success_push"} | counter | Wire bytes of CRDS values processed (Push (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_duplicate"} | counter | Wire bytes of CRDS values processed (Pull Response (duplicate)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_signature"} | counter | Wire bytes of CRDS values processed (Pull Response (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_origin_no_contact_info"} | counter | Wire bytes of CRDS values processed (Pull Response (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_origin_shred_version"} | counter | Wire bytes of CRDS values processed (Pull Response (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_inactive"} | counter | Wire bytes of CRDS values processed (Pull Response (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_wallclock"} | counter | Wire bytes of CRDS values processed (Pull Response (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_signature"} | counter | Wire bytes of CRDS values processed (Push (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_origin_no_contact_info"} | counter | Wire bytes of CRDS values processed (Push (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_origin_shred_version"} | counter | Wire bytes of CRDS values processed (Push (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_inactive"} | counter | Wire bytes of CRDS values processed (Push (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_wallclock"} | counter | Wire bytes of CRDS values processed (Push (wallclock)) |
Gossip Tile
| Metric | Type | Description |
|---|---|---|
| gossip_ping_tracker_capacity | gauge | Peer tracking capacity of the ping tracker |
| gossip_ping_tracked {ping_tracked_type="unpinged"} | gauge | Peers being tracked for ping/pong (Peer is known but has not been pinged yet) |
| gossip_ping_tracked {ping_tracked_type="invalid"} | gauge | Peers being tracked for ping/pong (Peer was pinged but has not yet responded, or responded with an invalid pong) |
| gossip_ping_tracked {ping_tracked_type="valid"} | gauge | Peers being tracked for ping/pong (Peer was pinged and is currently valid) |
| gossip_ping_tracked {ping_tracked_type="valid_refreshing"} | gauge | Peers being tracked for ping/pong (Peer was pinged and is currently valid, and is being refreshed with a new ping) |
| gossip_ping_tracker_pong_result {pong_response="staked"} | counter | Result of processing a pong response to a ping (Pong ignored as node is staked) |
| gossip_ping_tracker_pong_result {pong_response="entrypoint"} | counter | Result of processing a pong response to a ping (Pong ignored as node is an entrypoint) |
| gossip_ping_tracker_pong_result {pong_response="untracked"} | counter | Result of processing a pong response to a ping (Pong ignored as node has no outstanding ping) |
| gossip_ping_tracker_pong_result {pong_response="address"} | counter | Result of processing a pong response to a ping (Pong ignored as responding node doesn't match pinged address) |
| gossip_ping_tracker_pong_result {pong_response="token"} | counter | Result of processing a pong response to a ping (Pong ignored as token was not valid) |
| gossip_ping_tracker_pong_result {pong_response="success"} | counter | Result of processing a pong response to a ping (Pong accepted) |
| gossip_ping_tracker_evicted | counter | Peers evicted from the ping tracker to make space for new peers |
| gossip_ping_tracker_added | counter | Peers ever tracked for ping/pong |
| gossip_ping_tracker_stake_changed | counter | Times a tracked peer was removed from tracking because it became staked |
| gossip_ping_tracker_address_changed | counter | Times a tracked peer was removed from tracking because its gossip address changed |
| gossip_crds_capacity | gauge | Capacity of the data store |
| gossip_crds_occupied {crds_value="contact_info_v1"} | gauge | Entries in the data store (Contact Info V1) |
| gossip_crds_occupied {crds_value="vote"} | gauge | Entries in the data store (Vote) |
| gossip_crds_occupied {crds_value="lowest_slot"} | gauge | Entries in the data store (Lowest Slot) |
| gossip_crds_occupied {crds_value="snapshot_hashes"} | gauge | Entries in the data store (Snapshot Hashes) |
| gossip_crds_occupied {crds_value="accounts_hashes"} | gauge | Entries in the data store (Accounts Hashes) |
| gossip_crds_occupied {crds_value="epoch_slots"} | gauge | Entries in the data store (Epoch Slots) |
| gossip_crds_occupied {crds_value="version_v1"} | gauge | Entries in the data store (Version V1) |
| gossip_crds_occupied {crds_value="version_v2"} | gauge | Entries in the data store (Version V2) |
| gossip_crds_occupied {crds_value="node_instance"} | gauge | Entries in the data store (Node Instance) |
| gossip_crds_occupied {crds_value="duplicate_shred"} | gauge | Entries in the data store (Duplicate Shred) |
| gossip_crds_occupied {crds_value="incremental_snapshot_hashes"} | gauge | Entries in the data store (Incremental Snapshot Hashes) |
| gossip_crds_occupied {crds_value="contact_info_v2"} | gauge | Entries in the data store (Contact Info V2) |
| gossip_crds_occupied {crds_value="restart_last_voted_fork_slots"} | gauge | Entries in the data store (Restart Last Voted Fork Slots) |
| gossip_crds_occupied {crds_value="restart_heaviest_fork"} | gauge | Entries in the data store (Restart Heaviest Fork) |
| gossip_crds_expired | counter | Entries expired from the data store due to age |
| gossip_crds_evicted | counter | Entries evicted from the data store to make space for new entries |
| gossip_crds_peer_capacity | gauge | Capacity for storing peers in the data store |
| gossip_crds_peer_staked | gauge | Staked peers in the data store |
| gossip_crds_peer_unstaked | gauge | Unstaked peers in the data store |
| gossip_crds_peer_stake | gauge | Total visible stake in the data store, in lamports |
| gossip_crds_peer_evicted | counter | Peers evicted from the data store to make space for new peers |
| gossip_crds_purged_capacity | gauge | Capacity of the list of purged data store entries |
| gossip_crds_purged_occupied | gauge | Purged data store entries |
| gossip_crds_purged_expired | counter | Purged entries expired from the purged list due to age |
| gossip_crds_purged_evicted | counter | Purged entries evicted from the data store to make space for new entries |
| gossip_crds_rx {gossip_crds_outcome="upserted_pull_response"} | counter | Outcome of incoming CRDS messages (Pull Response (upserted)) |
| gossip_crds_rx {gossip_crds_outcome="upserted_push"} | counter | Outcome of incoming CRDS messages (Push (upserted)) |
| gossip_crds_rx {gossip_crds_outcome="dropped_pull_response_stale"} | counter | Outcome of incoming CRDS messages (Pull Response (newer entry already present in table)) |
| gossip_crds_rx {gossip_crds_outcome="dropped_pull_response_duplicate"} | counter | Outcome of incoming CRDS messages (Pull Response (duplicate)) |
| gossip_crds_rx {gossip_crds_outcome="dropped_push_stale"} | counter | Outcome of incoming CRDS messages (Push (newer entry already present in table)) |
| gossip_crds_rx {gossip_crds_outcome="dropped_push_duplicate"} | counter | Outcome of incoming CRDS messages (Push (duplicate)) |
| gossip_message_tx {gossip_message="pull_request"} | counter | Gossip messages sent (Pull Request) |
| gossip_message_tx {gossip_message="pull_response"} | counter | Gossip messages sent (Pull Response) |
| gossip_message_tx {gossip_message="push"} | counter | Gossip messages sent (Push) |
| gossip_message_tx {gossip_message="prune"} | counter | Gossip messages sent (Prune) |
| gossip_message_tx {gossip_message="ping"} | counter | Gossip messages sent (Ping) |
| gossip_message_tx {gossip_message="pong"} | counter | Gossip messages sent (Pong) |
| gossip_message_tx_bytes {gossip_message="pull_request"} | counter | Wire bytes sent in gossip messages (Pull Request) |
| gossip_message_tx_bytes {gossip_message="pull_response"} | counter | Wire bytes sent in gossip messages (Pull Response) |
| gossip_message_tx_bytes {gossip_message="push"} | counter | Wire bytes sent in gossip messages (Push) |
| gossip_message_tx_bytes {gossip_message="prune"} | counter | Wire bytes sent in gossip messages (Prune) |
| gossip_message_tx_bytes {gossip_message="ping"} | counter | Wire bytes sent in gossip messages (Ping) |
| gossip_message_tx_bytes {gossip_message="pong"} | counter | Wire bytes sent in gossip messages (Pong) |
| gossip_crds_push_tx {crds_value="contact_info_v1"} | counter | CRDS values sent in push messages (Contact Info V1) |
| gossip_crds_push_tx {crds_value="vote"} | counter | CRDS values sent in push messages (Vote) |
| gossip_crds_push_tx {crds_value="lowest_slot"} | counter | CRDS values sent in push messages (Lowest Slot) |
| gossip_crds_push_tx {crds_value="snapshot_hashes"} | counter | CRDS values sent in push messages (Snapshot Hashes) |
| gossip_crds_push_tx {crds_value="accounts_hashes"} | counter | CRDS values sent in push messages (Accounts Hashes) |
| gossip_crds_push_tx {crds_value="epoch_slots"} | counter | CRDS values sent in push messages (Epoch Slots) |
| gossip_crds_push_tx {crds_value="version_v1"} | counter | CRDS values sent in push messages (Version V1) |
| gossip_crds_push_tx {crds_value="version_v2"} | counter | CRDS values sent in push messages (Version V2) |
| gossip_crds_push_tx {crds_value="node_instance"} | counter | CRDS values sent in push messages (Node Instance) |
| gossip_crds_push_tx {crds_value="duplicate_shred"} | counter | CRDS values sent in push messages (Duplicate Shred) |
| gossip_crds_push_tx {crds_value="incremental_snapshot_hashes"} | counter | CRDS values sent in push messages (Incremental Snapshot Hashes) |
| gossip_crds_push_tx {crds_value="contact_info_v2"} | counter | CRDS values sent in push messages (Contact Info V2) |
| gossip_crds_push_tx {crds_value="restart_last_voted_fork_slots"} | counter | CRDS values sent in push messages (Restart Last Voted Fork Slots) |
| gossip_crds_push_tx {crds_value="restart_heaviest_fork"} | counter | CRDS values sent in push messages (Restart Heaviest Fork) |
| gossip_crds_push_tx_bytes {crds_value="contact_info_v1"} | counter | Wire bytes of CRDS sent in push messages (Contact Info V1) |
| gossip_crds_push_tx_bytes {crds_value="vote"} | counter | Wire bytes of CRDS sent in push messages (Vote) |
| gossip_crds_push_tx_bytes {crds_value="lowest_slot"} | counter | Wire bytes of CRDS sent in push messages (Lowest Slot) |
| gossip_crds_push_tx_bytes {crds_value="snapshot_hashes"} | counter | Wire bytes of CRDS sent in push messages (Snapshot Hashes) |
| gossip_crds_push_tx_bytes {crds_value="accounts_hashes"} | counter | Wire bytes of CRDS sent in push messages (Accounts Hashes) |
| gossip_crds_push_tx_bytes {crds_value="epoch_slots"} | counter | Wire bytes of CRDS sent in push messages (Epoch Slots) |
| gossip_crds_push_tx_bytes {crds_value="version_v1"} | counter | Wire bytes of CRDS sent in push messages (Version V1) |
| gossip_crds_push_tx_bytes {crds_value="version_v2"} | counter | Wire bytes of CRDS sent in push messages (Version V2) |
| gossip_crds_push_tx_bytes {crds_value="node_instance"} | counter | Wire bytes of CRDS sent in push messages (Node Instance) |
| gossip_crds_push_tx_bytes {crds_value="duplicate_shred"} | counter | Wire bytes of CRDS sent in push messages (Duplicate Shred) |
| gossip_crds_push_tx_bytes {crds_value="incremental_snapshot_hashes"} | counter | Wire bytes of CRDS sent in push messages (Incremental Snapshot Hashes) |
| gossip_crds_push_tx_bytes {crds_value="contact_info_v2"} | counter | Wire bytes of CRDS sent in push messages (Contact Info V2) |
| gossip_crds_push_tx_bytes {crds_value="restart_last_voted_fork_slots"} | counter | Wire bytes of CRDS sent in push messages (Restart Last Voted Fork Slots) |
| gossip_crds_push_tx_bytes {crds_value="restart_heaviest_fork"} | counter | Wire bytes of CRDS sent in push messages (Restart Heaviest Fork) |
| gossip_crds_pull_response_tx {crds_value="contact_info_v1"} | counter | CRDS values sent in pull response messages (Contact Info V1) |
| gossip_crds_pull_response_tx {crds_value="vote"} | counter | CRDS values sent in pull response messages (Vote) |
| gossip_crds_pull_response_tx {crds_value="lowest_slot"} | counter | CRDS values sent in pull response messages (Lowest Slot) |
| gossip_crds_pull_response_tx {crds_value="snapshot_hashes"} | counter | CRDS values sent in pull response messages (Snapshot Hashes) |
| gossip_crds_pull_response_tx {crds_value="accounts_hashes"} | counter | CRDS values sent in pull response messages (Accounts Hashes) |
| gossip_crds_pull_response_tx {crds_value="epoch_slots"} | counter | CRDS values sent in pull response messages (Epoch Slots) |
| gossip_crds_pull_response_tx {crds_value="version_v1"} | counter | CRDS values sent in pull response messages (Version V1) |
| gossip_crds_pull_response_tx {crds_value="version_v2"} | counter | CRDS values sent in pull response messages (Version V2) |
| gossip_crds_pull_response_tx {crds_value="node_instance"} | counter | CRDS values sent in pull response messages (Node Instance) |
| gossip_crds_pull_response_tx {crds_value="duplicate_shred"} | counter | CRDS values sent in pull response messages (Duplicate Shred) |
| gossip_crds_pull_response_tx {crds_value="incremental_snapshot_hashes"} | counter | CRDS values sent in pull response messages (Incremental Snapshot Hashes) |
| gossip_crds_pull_response_tx {crds_value="contact_info_v2"} | counter | CRDS values sent in pull response messages (Contact Info V2) |
| gossip_crds_pull_response_tx {crds_value="restart_last_voted_fork_slots"} | counter | CRDS values sent in pull response messages (Restart Last Voted Fork Slots) |
| gossip_crds_pull_response_tx {crds_value="restart_heaviest_fork"} | counter | CRDS values sent in pull response messages (Restart Heaviest Fork) |
| gossip_crds_pull_response_tx_bytes {crds_value="contact_info_v1"} | counter | Wire bytes of CRDS sent in pull response messages (Contact Info V1) |
| gossip_crds_pull_response_tx_bytes {crds_value="vote"} | counter | Wire bytes of CRDS sent in pull response messages (Vote) |
| gossip_crds_pull_response_tx_bytes {crds_value="lowest_slot"} | counter | Wire bytes of CRDS sent in pull response messages (Lowest Slot) |
| gossip_crds_pull_response_tx_bytes {crds_value="snapshot_hashes"} | counter | Wire bytes of CRDS sent in pull response messages (Snapshot Hashes) |
| gossip_crds_pull_response_tx_bytes {crds_value="accounts_hashes"} | counter | Wire bytes of CRDS sent in pull response messages (Accounts Hashes) |
| gossip_crds_pull_response_tx_bytes {crds_value="epoch_slots"} | counter | Wire bytes of CRDS sent in pull response messages (Epoch Slots) |
| gossip_crds_pull_response_tx_bytes {crds_value="version_v1"} | counter | Wire bytes of CRDS sent in pull response messages (Version V1) |
| gossip_crds_pull_response_tx_bytes {crds_value="version_v2"} | counter | Wire bytes of CRDS sent in pull response messages (Version V2) |
| gossip_crds_pull_response_tx_bytes {crds_value="node_instance"} | counter | Wire bytes of CRDS sent in pull response messages (Node Instance) |
| gossip_crds_pull_response_tx_bytes {crds_value="duplicate_shred"} | counter | Wire bytes of CRDS sent in pull response messages (Duplicate Shred) |
| gossip_crds_pull_response_tx_bytes {crds_value="incremental_snapshot_hashes"} | counter | Wire bytes of CRDS sent in pull response messages (Incremental Snapshot Hashes) |
| gossip_crds_pull_response_tx_bytes {crds_value="contact_info_v2"} | counter | Wire bytes of CRDS sent in pull response messages (Contact Info V2) |
| gossip_crds_pull_response_tx_bytes {crds_value="restart_last_voted_fork_slots"} | counter | Wire bytes of CRDS sent in pull response messages (Restart Last Voted Fork Slots) |
| gossip_crds_pull_response_tx_bytes {crds_value="restart_heaviest_fork"} | counter | Wire bytes of CRDS sent in pull response messages (Restart Heaviest Fork) |
| gossip_wait_for_supermajority_staked_peer_online | gauge | Staked peers with recent gossip activity |
| gossip_wait_for_supermajority_staked_peer_total | gauge | Staked peers being tracked |
| gossip_wait_for_supermajority_stake_online | gauge | Lamports of stake from peers with recent gossip activity |
| gossip_wait_for_supermajority_stake_total | gauge | Total lamports of stake being tracked |
| gossip_wait_for_supermajority_state | gauge | 1=init, 2=wait, 3=publish, 4=done |
Repair Tile
| Metric | Type | Description |
|---|---|---|
| repair_pkt_tx | counter | Network packets sent, including reqs, pings, pongs, etc |
| repair_request_tx {repair_sent_request_type="needed_window"} | counter | Client messages sent, by type (Need Window) |
| repair_request_tx {repair_sent_request_type="needed_highest_window"} | counter | Client messages sent, by type (Need Highest Window) |
| repair_request_tx {repair_sent_request_type="needed_orphan"} | counter | Client messages sent, by type (Need Orphans) |
| repair_request_tx {repair_sent_request_type="pong"} | counter | Client messages sent, by type (Pong) |
| repair_slot_highest_repaired | gauge | Highest slot up to which we have fully repaired |
| repair_slot_current | gauge | Our view of the current cluster slot, max slot received |
| repair_shred_old | counter | Shreds received that were older than the root |
| repair_slot_last_requested | gauge | Last slot we requested a single shred for |
| repair_orphan_last_requested | gauge | Last slot we requested an orphan for |
| repair_request_inflight | gauge | Requests currently in the inflight queue, excluding orphans and highest window requests |
| repair_peer_requested | counter | Peers requested |
| repair_sign_tile_unavailable | counter | Times no sign tiles were available to send a request |
| repair_eager_threshold_exceeded | counter | Times we passed the eager repair threshold |
| repair_shred_rerequested | counter | Times we re-requested a shred from the inflights queue |
| repair_ping_malformed | counter | Malformed pings received |
| repair_ping_unknown_peer | counter | Pings received from an unknown peer |
| repair_ping_signature_failed | counter | Pings whose signature we failed to verify |
| repair_slot_complete_duration_seconds | histogram | Time it took to complete a slot |
| repair_response_latency_nanos | histogram | Time it took to receive a repair request response, in nanoseconds |
| repair_sign_duration_seconds | histogram | Duration of signing a message |
| repair_block_evicted | counter | Blocks evicted from the forest |
| repair_block_insert_failed | counter | Blocks we failed to insert into the forest due to failed eviction |
| repair_slot_last_evicted | gauge | Most recent slot evicted from forest |
| repair_slot_last_eviction_cause | gauge | Slot that caused the most recent eviction from forest |
| repair_slot_last_insert_failed | gauge | Most recent slot we failed to insert into the forest |
| repair_chain_verify_failed | counter | Times any block failed chained merkle verification |
| repair_slot_last_chain_verify_failed | gauge | Most recent slot that failed chained merkle verification |
Rserve Tile
| Metric | Type | Description |
|---|---|---|
| rserve_received_request_count {rserve_request_types="pong"} | counter | Total repair requests received by type, before any validation (Pong) |
| rserve_received_request_count {rserve_request_types="window_index"} | counter | Total repair requests received by type, before any validation (Window Index) |
| rserve_received_request_count {rserve_request_types="highest_window_index"} | counter | Total repair requests received by type, before any validation (Highest Window Index) |
| rserve_received_request_count {rserve_request_types="orphan"} | counter | Total repair requests received by type, before any validation (Orphan) |
| rserve_received_request_bytes | counter | Total bytes of incoming repair request payloads |
| rserve_received_malformed_count {rserve_malformed_types="too_small"} | counter | Packets dropped for being malformed, broken down by reason (Too Small) |
| rserve_received_malformed_count {rserve_malformed_types="ping"} | counter | Packets dropped for being malformed, broken down by reason (Ping) |
| rserve_received_malformed_count {rserve_malformed_types="unknown_tag"} | counter | Packets dropped for being malformed, broken down by reason (Unknown Tag) |
| rserve_received_malformed_count {rserve_malformed_types="wrong_size"} | counter | Packets dropped for being malformed, broken down by reason (Wrong Size) |
| rserve_received_malformed_count {rserve_malformed_types="ancestor_hashes"} | counter | Packets dropped for being malformed, broken down by reason (Ancestor Hashes) |
| rserve_total_pkt_count | counter | How many network packets we have sent, including pings, responses, etc |
| rserve_sent_response_types {rserve_sent_response_types="ping"} | counter | What types of response messages are we sending (Ping) |
| rserve_sent_response_types {rserve_sent_response_types="window"} | counter | What types of response messages are we sending (Window) |
| rserve_sent_response_types {rserve_sent_response_types="highest_window"} | counter | What types of response messages are we sending (Highest Window) |
| rserve_sent_response_types {rserve_sent_response_types="orphan"} | counter | What types of response messages are we sending (Orphan) |
| rserve_sent_response_bytes | counter | Total payload bytes sent in response packets |
| rserve_sent_ping_back_count | counter | How many ping-back packets we sent to nodes not yet in the ping cache |
| rserve_missed_response_types {rserve_sent_response_types="ping"} | counter | What types of response messages could we not fulfill (Ping) |
| rserve_missed_response_types {rserve_sent_response_types="window"} | counter | What types of response messages could we not fulfill (Window) |
| rserve_missed_response_types {rserve_sent_response_types="highest_window"} | counter | What types of response messages could we not fulfill (Highest Window) |
| rserve_missed_response_types {rserve_sent_response_types="orphan"} | counter | What types of response messages could we not fulfill (Orphan) |
| rserve_failed_sigverify | counter | How many times we failed to verify the signature of a request |
| rserve_failed_own_key | counter | How many requests we've received that were sent by us |
| rserve_failed_invalid_token | counter | How many pong requests we've received with outdated or invalid tokens |
| rserve_failed_not_for_us | counter | How many requests we've received that were not intended for us |
| rserve_failed_outdated | counter | How many requests we've received that had outdated timestamps |
| rserve_failed_invalid_shred_index | counter | How many requests we've received that had invalid shred indices |
| rserve_failed_ping_cache_lookup | counter | Requests from nodes not in the ping cache, which triggered a ping-back instead of a response |
| rserve_shreds_current | gauge | The number of shreds currently in the shreds database |
| rserve_shreds_max | gauge | Total capacity of shreds that can be stored in the shreds database |
| rserve_disk_current_bytes | gauge | The number of bytes currently used on disk by the database |
| rserve_disk_allocated_bytes | gauge | The current size of the database file on disk |
| rserve_ping_cache_entries | counter | How many active entries do we have in the ping cache |
| rserve_ping_cache_evictions | counter | How many entries we've evicted from the ping cache |
Replay Tile
| Metric | Type | Description |
|---|---|---|
| replay_store_query_acquired | counter | Store slock acquires for query |
| replay_store_query_released | counter | Store slock releases for query |
| replay_store_query_wait_seconds | histogram | Time spent waiting to acquire the slock for query |
| replay_store_query_work_seconds | histogram | Time spent working before releasing the slock for query |
| replay_store_queried | counter | Queries |
| replay_store_query_missing | counter | Queries where merkle root was missing |
| replay_store_query_merkle_root_sample | gauge | First 8 bytes of most recently queried merkle root |
| replay_store_query_missing_merkle_root_sample | gauge | First 8 bytes of most recently queried missing merkle root |
| replay_root_slot | gauge | Slot at which our node has most recently rooted |
| replay_root_distance | gauge | Distance in slots between our current root and the current reset slot |
| replay_leader_slot | gauge | Slot at which we are currently leader, or 0 if none |
| replay_next_leader_slot | gauge | Slot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot |
| replay_reset_slot | gauge | Slot at which we last reset the replay stage, or 0 if unknown |
| replay_bank_live_max | gauge | Maximum number of banks we can have alive |
| replay_bank_live | gauge | Banks we currently have alive |
| replay_reassembly_free | gauge | Free FEC sets in the reassembly queue |
| replay_reassembly_latest_slot | gauge | Slot of the latest FEC set in the reassembly queue that can be replayed |
| replay_reassembly_latest_fec_index | gauge | FEC set index of the latest FEC set in the reassembly queue that can be replayed |
| replay_sched_active_bank_index | gauge | Bank that the replay scheduler is currently dispatching work for |
| replay_sched_last_dispatch_bank_index | gauge | Bank that the replay scheduler last dispatched work for |
| replay_sched_last_dispatch_timestamp_nanos | gauge | Timestamp when the replay scheduler last dispatched work, in nanoseconds |
| replay_sched_staging_lane_occupied | gauge | Staging lanes currently occupied in the replay scheduler. Up to four staging lanes can be occupied at once |
| replay_sched_staging_lane_occupied_watermark | gauge | High watermark of staging lanes that were at one point occupied in the replay scheduler |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane0"} | gauge | Bank at the head of each staging lane of the replay dispatcher (Staging lane 0) |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane1"} | gauge | Bank at the head of each staging lane of the replay dispatcher (Staging lane 1) |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane2"} | gauge | Bank at the head of each staging lane of the replay dispatcher (Staging lane 2) |
| replay_sched_staging_lane_head_bank_index {staging_lane="lane3"} | gauge | Bank at the head of each staging lane of the replay dispatcher (Staging lane 3) |
| replay_sched_txn_pool_occupied | gauge | Transactions available to be scheduled in the replay scheduler |
| replay_sched_txn_pool_size | gauge | Maximum number of transactions the replay scheduler can consider. Fixed at Firedancer startup but a useful reference for SchedTxnPoolOccupied |
| replay_sched_microblock_pool_occupied | gauge | Microblocks available to be scheduled in the replay scheduler |
| replay_sched_microblock_pool_size | gauge | Maximum number of microblocks the replay scheduler can consider. Fixed at Firedancer startup but a useful reference for SchedMicroblockPoolOccupied |
| replay_sched_block_pool_occupied | gauge | Blocks alive in the replay scheduler |
| replay_sched_block_pool_size | gauge | Maximum number of blocks the replay scheduler can keep track of. Fixed at Firedancer startup but a useful reference for SchedBlockPoolOccupied |
| replay_sched_block_added {sched_block_staging="staged"} | counter | Blocks added to the replay scheduler, by staging state (Added as staged) |
| replay_sched_block_added {sched_block_staging="unstaged"} | counter | Blocks added to the replay scheduler, by staging state (Added as unstaged) |
| replay_sched_block_replayed | counter | Blocks the replay scheduler considers fully replayed |
| replay_sched_block_abandoned | counter | Blocks the replay scheduler abandoned for any reason |
| replay_sched_block_rejected | counter | Blocks the replay scheduler considers bad |
| replay_sched_block_promoted | counter | Blocks the replay scheduler promoted from unstaged to a staging lane |
| replay_sched_block_demoted | counter | Blocks the replay scheduler demoted from a staging lane to unstaged |
| replay_sched_deactivate {sched_deactivate_reason="no_child"} | counter | Times the replay scheduler stopped dispatching, by reason (A block fully replayed and there was no child block available yet) |
| replay_sched_deactivate {sched_deactivate_reason="no_work"} | counter | Times the replay scheduler stopped dispatching, by reason (A partially replayed block ran out of work to dispatch) |
| replay_sched_deactivate {sched_deactivate_reason="abandoned"} | counter | Times the replay scheduler stopped dispatching, by reason (A block got abandoned for any reason) |
| replay_sched_deactivate {sched_deactivate_reason="minority"} | counter | Times the replay scheduler stopped dispatching, by reason (A block got abandoned for being on a minority fork) |
| replay_sched_lane_switched | counter | Times the replay scheduler switched the lane it is dispatching from |
| replay_sched_lane_promoted | counter | Times the replay scheduler promoted one or more blocks onto a staging lane |
| replay_sched_lane_demoted | counter | Times the replay scheduler demoted one or more blocks from a staging lane |
| replay_sched_fork_observed | counter | Forks the replay scheduler has observed |
| replay_sched_alut {sched_alut_result="success"} | counter | Transactions with address lookup tables the replay scheduler tried to expand, by outcome (Address lookup tables expanded successfully) |
| replay_sched_alut {sched_alut_result="failed"} | counter | Transactions with address lookup tables the replay scheduler tried to expand, by outcome (Address lookup table expansion failed) |
| replay_sched_txn_parsed_abandoned | counter | Transactions that were parsed but ended up being in abandoned blocks |
| replay_sched_txn_executed_abandoned | counter | Transactions that were executed but ended up being in abandoned blocks |
| replay_sched_txn_done_abandoned | counter | Transactions that were fully replayed but ended up being in abandoned blocks |
| replay_sched_weighted_in_flight | counter | Cumulative transactions in-flight from the replay scheduler's point of view, weighted by the duration of each transaction. Useful as the dividend for SchedWeightedInFlightDurationNanos |
| replay_sched_weighted_in_flight_duration_nanos | counter | Duration over which SchedWeightedInFlight was accumulated, in nanoseconds. Use as divisor to get the average number of in-flight transactions |
| replay_sched_none_in_flight_duration_nanos | counter | Duration the replay scheduler had no transaction in-flight, in nanoseconds. Useful in a ratio against SchedWeightedInFlightDurationNanos to see how often the replay transaction execution pipeline was idle |
| replay_sched_txn_parsed | counter | Transactions the replay pipeline has parsed |
| replay_sched_txn_executed | counter | Transactions the replay pipeline has executed |
| replay_sched_txn_signature_verified | counter | Transactions the replay pipeline has sigverified |
| replay_sched_txn_poh_mixed | counter | Transactions the replay pipeline has performed PoH mixin on |
| replay_sched_txn_done | counter | Transactions that were fully replayed |
| replay_sched_microblock_parsed | counter | Microblocks the replay pipeline has parsed |
| replay_sched_microblock_hashed | counter | Microblocks the replay pipeline has fully hashed |
| replay_sched_microblock_done | counter | Microblocks that were fully replayed |
| replay_sched_bytes_ingested | counter | Bytes the replay scheduler has ingested |
| replay_sched_bytes_ingested_padding | counter | Bytes the replay scheduler ingested but did not parse for being padding |
| replay_sched_bytes_dropped | counter | Bytes the replay scheduler refused to ingest because the block is considered abandoned |
| replay_sched_fec_ingested | counter | FEC sets the replay scheduler has been given |
| replay_slot_replayed | counter | Slots replayed successfully or leader slots packed and shredded successfully |
| replay_txn_processed | counter | Transactions processed overall on the current fork |
| replay_fec_sched_full | counter | Times sched is full and a FEC set can't be processed |
| replay_fec_reassembly_empty | counter | Times reassembly is empty and a FEC set can't be processed |
| replay_fec_leader_bid_wait | counter | Times replay is blocked by the PoH tile not sending an end of leader message |
| replay_fec_bank_full | counter | Times banks are full and a FEC set can't be processed |
| replay_storage_root_behind | counter | Times the storage root is behind the consensus root and can't be advanced |
| replay_progcache_rooted | counter | Program cache entries rooted |
| replay_progcache_gc_root | counter | Number of program cache entries garbage collected while rooting |
| replay_progcache_free_partition | gauge | Free program cache heap partitions (indicates fragmentation) |
| replay_progcache_free_bytes | gauge | Free bytes in the program cache heap |
| replay_progcache_size_bytes | gauge | Total size of the program cache heap |
| replay_progcache_free_partition_max_bytes | gauge | Largest free heap partition in program cache |
| replay_progcache_used_partition_median_bytes | gauge | Median used heap partition size in program cache |
| replay_progcache_used_partition_mean_bytes | gauge | Mean used heap partition size in program cache |
| replay_progcache_duration_seconds | counter | Time spent doing program cache tasks, in seconds |
| replay_accdb_account_acquired {accdb_cache_class="class0"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| replay_accdb_account_acquired {accdb_cache_class="class1"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| replay_accdb_account_acquired {accdb_cache_class="class2"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| replay_accdb_account_acquired {accdb_cache_class="class3"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| replay_accdb_account_acquired {accdb_cache_class="class4"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| replay_accdb_account_acquired {accdb_cache_class="class5"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| replay_accdb_account_acquired {accdb_cache_class="class6"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| replay_accdb_account_acquired {accdb_cache_class="class7"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class0"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class1"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class2"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class3"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class4"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class5"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class6"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| replay_accdb_account_writable_acquired {accdb_cache_class="class7"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| replay_accdb_account_evicted {accdb_cache_class="class0"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (0-128 B) |
| replay_accdb_account_evicted {accdb_cache_class="class1"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (129-512 B) |
| replay_accdb_account_evicted {accdb_cache_class="class2"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (513 B-2 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="class3"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (2-8 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="class4"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (8-32 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="class5"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (32-128 KiB) |
| replay_accdb_account_evicted {accdb_cache_class="class6"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (128 KiB-1 MiB) |
| replay_accdb_account_evicted {accdb_cache_class="class7"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (1-10 MiB) |
| replay_accdb_account_committed_new {accdb_cache_class="class0"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| replay_accdb_account_committed_new {accdb_cache_class="class1"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| replay_accdb_account_committed_new {accdb_cache_class="class2"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| replay_accdb_account_committed_new {accdb_cache_class="class3"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| replay_accdb_account_committed_new {accdb_cache_class="class4"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| replay_accdb_account_committed_new {accdb_cache_class="class5"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| replay_accdb_account_committed_new {accdb_cache_class="class6"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| replay_accdb_account_committed_new {accdb_cache_class="class7"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class0"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class1"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class2"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class3"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class4"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class5"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class6"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| replay_accdb_account_committed_overwrite {accdb_cache_class="class7"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| replay_accdb_account_not_found {accdb_cache_class="class0"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (0-128 B) |
| replay_accdb_account_not_found {accdb_cache_class="class1"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (129-512 B) |
| replay_accdb_account_not_found {accdb_cache_class="class2"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (513 B-2 KiB) |
| replay_accdb_account_not_found {accdb_cache_class="class3"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (2-8 KiB) |
| replay_accdb_account_not_found {accdb_cache_class="class4"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (8-32 KiB) |
| replay_accdb_account_not_found {accdb_cache_class="class5"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (32-128 KiB) |
| replay_accdb_account_not_found {accdb_cache_class="class6"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (128 KiB-1 MiB) |
| replay_accdb_account_not_found {accdb_cache_class="class7"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (1-10 MiB) |
| replay_accdb_account_waited | counter | Number of accounts that were needed for transaction execution but were concurrently loading into cache by another transaction, causing the transaction to wait |
| replay_accdb_batch_acquired | counter | Number of fd_accdb_acquire (batch acquire) calls made by this tile |
| replay_accdb_acquire_failed | counter | Number of times we failed to acquire the cache lines needed for a transaction, and had to spin loop waiting |
| replay_accdb_bytes_read | counter | Number of bytes read from the account database |
| replay_accdb_read_operation | counter | Number of read operations performed on the account database |
| replay_accdb_bytes_written | counter | Number of bytes written to the account database |
| replay_accdb_write_operation | counter | Number of write operations performed on the account database |
| replay_accdb_bytes_copied | counter | Number of bytes copied within the account database |
| replay_root_account_duration_seconds | histogram | Time spent updating the rooted account store (one sample per block, normalized by account count) |
| replay_root_duration_seconds {root_phase="copy"} | counter | Time spent rooting accounts, in seconds (Copying account data) |
| replay_root_duration_seconds {root_phase="garbage_collect"} | counter | Time spent rooting accounts, in seconds (Garbage collecting old account data) |
| replay_runtime_status | gauge | 0=initializing, 1=loaded |
Execrp Tile
| Metric | Type | Description |
|---|---|---|
| execrp_processing | gauge | Whether the tile is currently processing a task (executing, sigverifying, or PoH hashing), either 1 or 0 |
| execrp_txn_result {transaction_result="success"} | counter | Result of loading and executing a transaction (Transaction executed successfully) |
| execrp_txn_result {transaction_result="instruction_error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) |
| execrp_txn_result {transaction_result="account_not_found"} | counter | Result of loading and executing a transaction (The transaction fee payer address was not found) |
| execrp_txn_result {transaction_result="program_account_not_found"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was not found) |
| execrp_txn_result {transaction_result="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction (The transaction fee payer did not have balance to pay the fee) |
| execrp_txn_result {transaction_result="invalid_account_for_fee"} | counter | Result of loading and executing a transaction (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) |
| execrp_txn_result {transaction_result="already_processed"} | counter | Result of loading and executing a transaction (The transaction has already been processed in a recent block) |
| execrp_txn_result {transaction_result="blockhash_not_found"} | counter | Result of loading and executing a transaction (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) |
| execrp_txn_result {transaction_result="invalid_program_for_execution"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was no executable. TODO: No longer needed with SIMD-0162) |
| execrp_txn_result {transaction_result="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that does not exist or is inactive) |
| execrp_txn_result {transaction_result="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that is not owned by the ALUT program account) |
| execrp_txn_result {transaction_result="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that contains data which is not a valid ALUT) |
| execrp_txn_result {transaction_result="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction (The transaction references an account offset from the ALUT which does not exist) |
| execrp_txn_result {transaction_result="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction (The total account data size of the loaded accounts exceeds the consensus limit) |
| execrp_txn_result {transaction_result="duplicate_instruction"} | counter | Result of loading and executing a transaction (A compute budget program instruction was invoked more than once) |
| execrp_txn_result {transaction_result="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction (The compute budget program was invoked and set the loaded accounts data size to zero) |
| execrp_txn_result {transaction_result="nonce_already_advanced"} | counter | Result of loading and executing a transaction (The transaction references a nonce account that is already advanced) |
| execrp_txn_result {transaction_result="nonce_advance_failed"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the advance instruction was not valid or failed) |
| execrp_txn_result {transaction_result="nonce_wrong_blockhash"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the blockhash is not the correct one) |
| execrp_txn_result {transaction_result="sanitize_failure"} | counter | Result of loading and executing a transaction (The process for sanitizing the transaction failed) |
| execrp_txn_result {transaction_result="account_in_use"} | counter | Result of loading and executing a transaction (The transaction conflicts with another transaction in the microblock. TODO: No longer possible with smart dispatcher) |
| execrp_txn_result {transaction_result="account_loaded_twice"} | counter | Result of loading and executing a transaction (The transaction references the same account twice) |
| execrp_txn_result {transaction_result="signature_failure"} | counter | Result of loading and executing a transaction (The transaction had an invalid signature) |
| execrp_txn_result {transaction_result="too_many_account_locks"} | counter | Result of loading and executing a transaction (The transaction references too many accounts. TODO: No longer possible with smart dispatcher) |
| execrp_txn_result {transaction_result="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction (The transaction would leave an account with a lower balance than the rent-exempt minimum) |
| execrp_txn_result {transaction_result="unbalanced_transaction"} | counter | Result of loading and executing a transaction (The total referenced account lamports before and after the transaction was unbalanced) |
| execrp_txn_result {transaction_result="bundle_peer"} | counter | Result of loading and executing a transaction (The transaction was part of a bundle and an earlier transaction in the bundle failed) |
| execrp_signature_verified | counter | Ed25519 signature verification jobs executed |
| execrp_poh_hashed | counter | PoH SHA-256 calls executed |
| execrp_txn_regime_duration_nanos {txn_regime="setup"} | counter | Mutually exclusive and exhaustive duration spent in transaction execution regimes, in nanoseconds (Transaction setup) |
| execrp_txn_regime_duration_nanos {txn_regime="exec"} | counter | Mutually exclusive and exhaustive duration spent in transaction execution regimes, in nanoseconds (Transaction execution (includes VM setup/execution)) |
| execrp_txn_regime_duration_nanos {txn_regime="commit"} | counter | Mutually exclusive and exhaustive duration spent in transaction execution regimes, in nanoseconds (Transaction result commit) |
| execrp_vm_regime_duration_nanos {vm_regime="setup"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM setup) |
| execrp_vm_regime_duration_nanos {vm_regime="commit"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM commit) |
| execrp_vm_regime_duration_nanos {vm_regime="setup_cpi"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM setup (CPI)) |
| execrp_vm_regime_duration_nanos {vm_regime="commit_cpi"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM commit (CPI)) |
| execrp_vm_regime_duration_nanos {vm_regime="interpreter"} | counter | Mutually exclusive and exhaustive duration spent in virtual machine execution regimes, in nanoseconds (VM interpreter execution) |
| execrp_cu_executed | counter | Estimated compute units executed since tile start |
| execrp_progcache_lookup | counter | Program cache lookups |
| execrp_progcache_hit | counter | Program cache hits |
| execrp_progcache_miss | counter | Program cache misses |
| execrp_progcache_oom_heap | counter | Program cache out-of-memory events (heap) |
| execrp_progcache_oom_desc | counter | Program cache out-of-memory events (descriptor table) |
| execrp_progcache_fill | counter | Program cache insertions |
| execrp_progcache_fill_bytes | counter | Bytes inserted into program cache |
| execrp_progcache_spill | counter | Program cache spills (OOM fallback mechanism) |
| execrp_progcache_spill_bytes | counter | Bytes spilled from program cache (OOM fallback mechanism) |
| execrp_progcache_eviction | counter | Program cache evictions |
| execrp_progcache_eviction_bytes | counter | Bytes evicted from program cache |
| execrp_progcache_duration_seconds | counter | Time spent on program cache operations, in seconds |
| execrp_progcache_load_duration_seconds | counter | Time spent loading programs, in seconds |
| execrp_accdb_account_acquired {accdb_cache_class="class0"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| execrp_accdb_account_acquired {accdb_cache_class="class1"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| execrp_accdb_account_acquired {accdb_cache_class="class2"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| execrp_accdb_account_acquired {accdb_cache_class="class3"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| execrp_accdb_account_acquired {accdb_cache_class="class4"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| execrp_accdb_account_acquired {accdb_cache_class="class5"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| execrp_accdb_account_acquired {accdb_cache_class="class6"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| execrp_accdb_account_acquired {accdb_cache_class="class7"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class0"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class1"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class2"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class3"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class4"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class5"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class6"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| execrp_accdb_account_writable_acquired {accdb_cache_class="class7"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| execrp_accdb_account_evicted {accdb_cache_class="class0"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (0-128 B) |
| execrp_accdb_account_evicted {accdb_cache_class="class1"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (129-512 B) |
| execrp_accdb_account_evicted {accdb_cache_class="class2"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (513 B-2 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="class3"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (2-8 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="class4"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (8-32 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="class5"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (32-128 KiB) |
| execrp_accdb_account_evicted {accdb_cache_class="class6"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_evicted {accdb_cache_class="class7"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (1-10 MiB) |
| execrp_accdb_account_committed_new {accdb_cache_class="class0"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| execrp_accdb_account_committed_new {accdb_cache_class="class1"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| execrp_accdb_account_committed_new {accdb_cache_class="class2"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| execrp_accdb_account_committed_new {accdb_cache_class="class3"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| execrp_accdb_account_committed_new {accdb_cache_class="class4"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| execrp_accdb_account_committed_new {accdb_cache_class="class5"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| execrp_accdb_account_committed_new {accdb_cache_class="class6"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_committed_new {accdb_cache_class="class7"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class0"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class1"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class2"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class3"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class4"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class5"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class6"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_committed_overwrite {accdb_cache_class="class7"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| execrp_accdb_account_not_found {accdb_cache_class="class0"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (0-128 B) |
| execrp_accdb_account_not_found {accdb_cache_class="class1"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (129-512 B) |
| execrp_accdb_account_not_found {accdb_cache_class="class2"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (513 B-2 KiB) |
| execrp_accdb_account_not_found {accdb_cache_class="class3"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (2-8 KiB) |
| execrp_accdb_account_not_found {accdb_cache_class="class4"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (8-32 KiB) |
| execrp_accdb_account_not_found {accdb_cache_class="class5"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (32-128 KiB) |
| execrp_accdb_account_not_found {accdb_cache_class="class6"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (128 KiB-1 MiB) |
| execrp_accdb_account_not_found {accdb_cache_class="class7"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (1-10 MiB) |
| execrp_accdb_account_waited | counter | Number of accounts that were needed for transaction execution but were concurrently loading into cache by another transaction, causing the transaction to wait |
| execrp_accdb_batch_acquired | counter | Number of fd_accdb_acquire (batch acquire) calls made by this tile |
| execrp_accdb_acquire_failed | counter | Number of times we failed to acquire the cache lines needed for a transaction, and had to spin loop waiting |
| execrp_accdb_bytes_read | counter | Number of bytes read from the account database |
| execrp_accdb_read_operation | counter | Number of read operations performed on the account database |
| execrp_accdb_bytes_written | counter | Number of bytes written to the account database |
| execrp_accdb_write_operation | counter | Number of write operations performed on the account database |
| execrp_accdb_bytes_copied | counter | Number of bytes copied within the account database |
Accdb Tile
| Metric | Type | Description |
|---|---|---|
| accdb_account_count | gauge | Total number of accounts currently in the account database |
| accdb_account_capacity | gauge | Total capacity of accounts that can be stored in the account database |
| accdb_account_relocated | counter | Total number of accounts moved in the account database during compaction |
| accdb_disk_allocated_bytes | gauge | Total allocated size of the account database on disk |
| accdb_disk_current_bytes | gauge | Total written-through size of the account database on disk, excluding never-written partition tail space |
| accdb_disk_used_bytes | gauge | Total size of the account database on disk that is currently used |
| accdb_in_compaction | gauge | 1 if a compaction is currently in progress, 0 if not |
| accdb_compaction_requested | counter | Total number of compactions requested on the account database |
| accdb_compaction_completed | counter | Total number of compactions fully completed on the account database, less than or equal to the requested amount |
| accdb_account_relocated_bytes | counter | Total number of bytes moved in the account database during compaction |
| accdb_account_preevicted {accdb_cache_class="class0"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (0-128 B) |
| accdb_account_preevicted {accdb_cache_class="class1"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (129-512 B) |
| accdb_account_preevicted {accdb_cache_class="class2"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (513 B-2 KiB) |
| accdb_account_preevicted {accdb_cache_class="class3"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (2-8 KiB) |
| accdb_account_preevicted {accdb_cache_class="class4"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (8-32 KiB) |
| accdb_account_preevicted {accdb_cache_class="class5"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (32-128 KiB) |
| accdb_account_preevicted {accdb_cache_class="class6"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (128 KiB-1 MiB) |
| accdb_account_preevicted {accdb_cache_class="class7"} | counter | Number of accounts preemptively evicted (written back) from the account database cache by the background preevict pass, broken down by cache size class (1-10 MiB) |
| accdb_bytes_read | counter | Number of bytes read from the account database |
| accdb_bytes_written | counter | Number of bytes written to the account database |
| accdb_write_operation | counter | Number of write operations performed on the account database |
| accdb_copy_operation | counter | Number of in-place copy_file_range operations performed on the account database during compaction |
| accdb_account_deleted | counter | Number of accounts deleted from the account database |
| accdb_cache_class_used {accdb_cache_class="class0"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (0-128 B) |
| accdb_cache_class_used {accdb_cache_class="class1"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (129-512 B) |
| accdb_cache_class_used {accdb_cache_class="class2"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (513 B-2 KiB) |
| accdb_cache_class_used {accdb_cache_class="class3"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (2-8 KiB) |
| accdb_cache_class_used {accdb_cache_class="class4"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (8-32 KiB) |
| accdb_cache_class_used {accdb_cache_class="class5"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (32-128 KiB) |
| accdb_cache_class_used {accdb_cache_class="class6"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (128 KiB-1 MiB) |
| accdb_cache_class_used {accdb_cache_class="class7"} | gauge | Number of slots currently occupied in the account database cache, broken down by size class (1-10 MiB) |
| accdb_cache_class_max {accdb_cache_class="class0"} | gauge | Total slot capacity of the account database cache, broken down by size class (0-128 B) |
| accdb_cache_class_max {accdb_cache_class="class1"} | gauge | Total slot capacity of the account database cache, broken down by size class (129-512 B) |
| accdb_cache_class_max {accdb_cache_class="class2"} | gauge | Total slot capacity of the account database cache, broken down by size class (513 B-2 KiB) |
| accdb_cache_class_max {accdb_cache_class="class3"} | gauge | Total slot capacity of the account database cache, broken down by size class (2-8 KiB) |
| accdb_cache_class_max {accdb_cache_class="class4"} | gauge | Total slot capacity of the account database cache, broken down by size class (8-32 KiB) |
| accdb_cache_class_max {accdb_cache_class="class5"} | gauge | Total slot capacity of the account database cache, broken down by size class (32-128 KiB) |
| accdb_cache_class_max {accdb_cache_class="class6"} | gauge | Total slot capacity of the account database cache, broken down by size class (128 KiB-1 MiB) |
| accdb_cache_class_max {accdb_cache_class="class7"} | gauge | Total slot capacity of the account database cache, broken down by size class (1-10 MiB) |
| accdb_cache_class_reserved {accdb_cache_class="class0"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (0-128 B) |
| accdb_cache_class_reserved {accdb_cache_class="class1"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (129-512 B) |
| accdb_cache_class_reserved {accdb_cache_class="class2"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (513 B-2 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="class3"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (2-8 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="class4"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (8-32 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="class5"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (32-128 KiB) |
| accdb_cache_class_reserved {accdb_cache_class="class6"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (128 KiB-1 MiB) |
| accdb_cache_class_reserved {accdb_cache_class="class7"} | gauge | Number of slots currently reserved by in-flight acquires (cache_class_used), or ULONG_MAX when reservation tracking is disabled for the class (1-10 MiB) |
| accdb_cache_class_target_used {accdb_cache_class="class0"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (0-128 B) |
| accdb_cache_class_target_used {accdb_cache_class="class1"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (129-512 B) |
| accdb_cache_class_target_used {accdb_cache_class="class2"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (513 B-2 KiB) |
| accdb_cache_class_target_used {accdb_cache_class="class3"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (2-8 KiB) |
| accdb_cache_class_target_used {accdb_cache_class="class4"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (8-32 KiB) |
| accdb_cache_class_target_used {accdb_cache_class="class5"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (32-128 KiB) |
| accdb_cache_class_target_used {accdb_cache_class="class6"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (128 KiB-1 MiB) |
| accdb_cache_class_target_used {accdb_cache_class="class7"} | gauge | Target used-slot count for the account database cache (max - cache_free_target). When used exceeds this, the background preevict pass tries to bring used back down to this level. (1-10 MiB) |
| accdb_cache_class_low_water_used {accdb_cache_class="class0"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (0-128 B) |
| accdb_cache_class_low_water_used {accdb_cache_class="class1"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (129-512 B) |
| accdb_cache_class_low_water_used {accdb_cache_class="class2"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (513 B-2 KiB) |
| accdb_cache_class_low_water_used {accdb_cache_class="class3"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (2-8 KiB) |
| accdb_cache_class_low_water_used {accdb_cache_class="class4"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (8-32 KiB) |
| accdb_cache_class_low_water_used {accdb_cache_class="class5"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (32-128 KiB) |
| accdb_cache_class_low_water_used {accdb_cache_class="class6"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (128 KiB-1 MiB) |
| accdb_cache_class_low_water_used {accdb_cache_class="class7"} | gauge | Used-slot count at which the background preevict pass kicks in (max - cache_free_low_water). When used exceeds this, preevicts start. (1-10 MiB) |
Tower Tile
| Metric | Type | Description |
|---|---|---|
| tower_frag_ignored | counter | replay_slot_completed frags we ignored |
| tower_slot_last_ignored | gauge | Most recent ignored replay_slot_completed frag |
| tower_frag_eqvoc_detected | counter | replay_slot_completed frags we detect as equivocations |
| tower_slot_last_eqvoc | gauge | Most recent equivocating replay_slot_completed frag |
| tower_replay_slot | gauge | Most recently replayed slot, ULONG_MAX if nothing replayed yet. Not monotonically increasing |
| tower_vote_slot | gauge | Highest voted slot in the local tower, ULONG_MAX if haven't voted. Monotonically increasing |
| tower_reset_slot | gauge | Most recent reset slot, ULONG_MAX if no reset yet. Not monotonically increasing |
| tower_root_slot | gauge | Highest rooted slot, ULONG_MAX if no root yet. Monotonically increasing |
| tower_init_slot | gauge | Init slot, either the snapshot or genesis slot. Set once and does not change |
| tower_frag_not_ready_dropped | counter | Frags dropped because they arrived before the tower tile was initialized |
| tower_eqvoc_proof {eqvoc_proof_result="success"} | counter | Equivocation proofs, by validation outcome (Valid equivocation proof) |
| tower_eqvoc_proof {eqvoc_proof_result="error"} | counter | Equivocation proofs, by validation outcome (Proof failed validation (gossip-only)) |
| tower_ghost_vote {ghost_vote_result="success"} | counter | Result of counting a vote towards ghost (Vote was counted towards ghost) |
| tower_ghost_vote {ghost_vote_result="not_voted"} | counter | Result of counting a vote towards ghost (Skipped because voter hasn't voted) |
| tower_ghost_vote {ghost_vote_result="too_old"} | counter | Result of counting a vote towards ghost (Slot was behind the root) |
| tower_ghost_vote {ghost_vote_result="already_voted"} | counter | Result of counting a vote towards ghost (Slot was not newer than previous vote) |
| tower_hard_fork_vote {hard_fork_vote_result="success"} | counter | Result of counting a vote towards hard fork detection (Vote was counted, no hard fork agreement reached) |
| tower_hard_fork_vote {hard_fork_vote_result="success_matched"} | counter | Result of counting a vote towards hard fork detection (52%+ of stake agreed on our bank hash) |
| tower_hard_fork_vote {hard_fork_vote_result="mismatched"} | counter | Result of counting a vote towards hard fork detection (52%+ of stake agreed on a different bank hash than ours (we hard forked)) |
| tower_hard_fork_vote {hard_fork_vote_result="unknown_voter"} | counter | Result of counting a vote towards hard fork detection (Voter not in voter set) |
| tower_hard_fork_vote {hard_fork_vote_result="already_voted"} | counter | Result of counting a vote towards hard fork detection (Voter already voted for this block_id) |
| tower_hard_fork_vote {hard_fork_vote_result="too_old"} | counter | Result of counting a vote towards hard fork detection (Slot not newer than previous) |
| tower_hard_fork_matched_slot | gauge | Highest slot where 52%+ of stake agreed on our bank hash |
| tower_hard_fork_mismatched_slot | gauge | Highest slot where 52%+ of stake agreed on a different bank hash than ours (we hard forked) |
| tower_fork_decision {tower_fork_decision="no_vote_not_recent"} | counter | Result of deciding which fork to vote/reset on (Haven't voted yet and best block is not recent enough to vote (can't vote)) |
| tower_fork_decision {tower_fork_decision="empty_tower_vote"} | counter | Result of deciding which fork to vote/reset on (Haven't voted yet, voting for best block (can vote)) |
| tower_fork_decision {tower_fork_decision="ancestor_rollback"} | counter | Result of deciding which fork to vote/reset on (Rollback to an ancestor of our prev vote (can't vote)) |
| tower_fork_decision {tower_fork_decision="sibling_confirmed"} | counter | Result of deciding which fork to vote/reset on (Duplicate sibling got confirmed (can vote)) |
| tower_fork_decision {tower_fork_decision="same_fork"} | counter | Result of deciding which fork to vote/reset on (Same fork as prev vote (can vote)) |
| tower_fork_decision {tower_fork_decision="switch_pass"} | counter | Result of deciding which fork to vote/reset on (Prev vote was on a different fork, but we are allowed to switch (can vote)) |
| tower_fork_decision {tower_fork_decision="switch_fail"} | counter | Result of deciding which fork to vote/reset on (Prev vote was on a different fork, and we are not allowed to switch (can't vote)) |
| tower_vote_gate {tower_vote_gate="voted"} | counter | Result of the vote gating checks after a fork was selected (Passed all vote checks and cast a vote) |
| tower_vote_gate {tower_vote_gate="lockout_fail"} | counter | Result of the vote gating checks after a fork was selected (Locked out (can't vote)) |
| tower_vote_gate {tower_vote_gate="threshold_fail"} | counter | Result of the vote gating checks after a fork was selected (Did not pass threshold check (can't vote)) |
| tower_vote_gate {tower_vote_gate="propagated_fail"} | counter | Result of the vote gating checks after a fork was selected (Prev leader block did not propagate (can't vote)) |
| tower_vote_gate {tower_vote_gate="no_candidate"} | counter | Result of the vote gating checks after a fork was selected (Fork decision did not produce a votable block) |
| tower_vote_txn {vote_txn_result="success"} | counter | Result of processing a vote txn (per txn) (Vote txn passed per-txn validation and reached slot counting (per-slot outcomes tracked by VoteSlotResult)) |
| tower_vote_txn {vote_txn_result="not_simple_vote"} | counter | Result of processing a vote txn (per txn) (Txn was not a simple vote transaction) |
| tower_vote_txn {vote_txn_result="bad_deser"} | counter | Result of processing a vote txn (per txn) (Vote txn failed to deserialize) |
| tower_vote_txn {vote_txn_result="not_tower_sync"} | counter | Result of processing a vote txn (per txn) (Vote txn was not a TowerSync instruction) |
| tower_vote_txn {vote_txn_result="empty_tower"} | counter | Result of processing a vote txn (per txn) (Vote txn had an empty tower (validator hasn't voted)) |
| tower_vote_txn {vote_txn_result="bad_tower"} | counter | Result of processing a vote txn (per txn) (Vote txn deserialized but tower was invalid) |
| tower_vote_txn {vote_txn_result="unknown_block_id"} | counter | Result of processing a vote txn (per txn) (Last vote had a null block id) |
| tower_vote_txn {vote_txn_result="too_old"} | counter | Result of processing a vote txn (per txn) (Last vote slot was behind the votes root) |
| tower_vote_txn {vote_txn_result="not_staked"} | counter | Result of processing a vote txn (per txn) (Unable to query stake for vote slot) |
| tower_vote_txn {vote_txn_result="bad_signer"} | counter | Result of processing a vote txn (per txn) (Vote txn was not signed by the authorized voter for the vote account at the vote epoch) |
| tower_vote_slot_counted {vote_slot_result="success"} | counter | Result of counting an individual vote slot from a vote txn (per slot) (Vote slot was counted) |
| tower_vote_slot_counted {vote_slot_result="too_new"} | counter | Result of counting an individual vote slot from a vote txn (per slot) (Vote slot was too far ahead of the votes root) |
| tower_vote_slot_counted {vote_slot_result="unknown_voter"} | counter | Result of counting an individual vote slot from a vote txn (per slot) (Vote account was not in the voter set) |
| tower_vote_slot_counted {vote_slot_result="already_voted"} | counter | Result of counting an individual vote slot from a vote txn (per slot) (Voter already voted for this slot) |
| tower_vote_slot_counted {vote_slot_result="too_old"} | counter | Result of counting an individual vote slot from a vote txn (per slot) (Vote slot was behind the votes root) |
| tower_vote_slot_counted {vote_slot_result="unknown_slot"} | counter | Result of counting an individual vote slot from a vote txn (per slot) (Vote slot was for a slot we haven't replayed) |
| tower_vote_intermediate_gate {vote_intermediate_gate="proceed"} | counter | Whether the last vote slot's replay state allowed counting the intermediate vote slots (per txn that reached slot counting) (Last vote slot was replayed and matched, intermediate vote slots will be counted) |
| tower_vote_intermediate_gate {vote_intermediate_gate="unknown_slot"} | counter | Whether the last vote slot's replay state allowed counting the intermediate vote slots (per txn that reached slot counting) (Last vote slot was not replayed, intermediate vote slots skipped) |
| tower_vote_intermediate_gate {vote_intermediate_gate="unknown_block_id"} | counter | Whether the last vote slot's replay state allowed counting the intermediate vote slots (per txn that reached slot counting) (Last vote slot block id did not match, intermediate vote slots skipped) |
| tower_accdb_account_acquired {accdb_cache_class="class0"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| tower_accdb_account_acquired {accdb_cache_class="class1"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| tower_accdb_account_acquired {accdb_cache_class="class2"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| tower_accdb_account_acquired {accdb_cache_class="class3"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| tower_accdb_account_acquired {accdb_cache_class="class4"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| tower_accdb_account_acquired {accdb_cache_class="class5"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| tower_accdb_account_acquired {accdb_cache_class="class6"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| tower_accdb_account_acquired {accdb_cache_class="class7"} | counter | Number of accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class0"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class1"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class2"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class3"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class4"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class5"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class6"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| tower_accdb_account_writable_acquired {accdb_cache_class="class7"} | counter | Number of writable accounts acquired from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| tower_accdb_account_evicted {accdb_cache_class="class0"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (0-128 B) |
| tower_accdb_account_evicted {accdb_cache_class="class1"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (129-512 B) |
| tower_accdb_account_evicted {accdb_cache_class="class2"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (513 B-2 KiB) |
| tower_accdb_account_evicted {accdb_cache_class="class3"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (2-8 KiB) |
| tower_accdb_account_evicted {accdb_cache_class="class4"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (8-32 KiB) |
| tower_accdb_account_evicted {accdb_cache_class="class5"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (32-128 KiB) |
| tower_accdb_account_evicted {accdb_cache_class="class6"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (128 KiB-1 MiB) |
| tower_accdb_account_evicted {accdb_cache_class="class7"} | counter | Number of accounts evicted from the account database cache to make space, broken down by cache size class (1-10 MiB) |
| tower_accdb_account_committed_new {accdb_cache_class="class0"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| tower_accdb_account_committed_new {accdb_cache_class="class1"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| tower_accdb_account_committed_new {accdb_cache_class="class2"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| tower_accdb_account_committed_new {accdb_cache_class="class3"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| tower_accdb_account_committed_new {accdb_cache_class="class4"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| tower_accdb_account_committed_new {accdb_cache_class="class5"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| tower_accdb_account_committed_new {accdb_cache_class="class6"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| tower_accdb_account_committed_new {accdb_cache_class="class7"} | counter | Number of new (non-overwrite) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class0"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (0-128 B) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class1"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (129-512 B) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class2"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (513 B-2 KiB) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class3"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (2-8 KiB) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class4"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (8-32 KiB) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class5"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (32-128 KiB) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class6"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (128 KiB-1 MiB) |
| tower_accdb_account_committed_overwrite {accdb_cache_class="class7"} | counter | Number of overwrite (existing fork+generation) account versions committed to the index, broken down by destination cache size class (1-10 MiB) |
| tower_accdb_account_not_found {accdb_cache_class="class0"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (0-128 B) |
| tower_accdb_account_not_found {accdb_cache_class="class1"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (129-512 B) |
| tower_accdb_account_not_found {accdb_cache_class="class2"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (513 B-2 KiB) |
| tower_accdb_account_not_found {accdb_cache_class="class3"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (2-8 KiB) |
| tower_accdb_account_not_found {accdb_cache_class="class4"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (8-32 KiB) |
| tower_accdb_account_not_found {accdb_cache_class="class5"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (32-128 KiB) |
| tower_accdb_account_not_found {accdb_cache_class="class6"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (128 KiB-1 MiB) |
| tower_accdb_account_not_found {accdb_cache_class="class7"} | counter | Number of accounts that were needed for transaction execution but were not found in the account database cache, broken down by cache size class (1-10 MiB) |
| tower_accdb_account_waited | counter | Number of accounts that were needed for transaction execution but were concurrently loading into cache by another transaction, causing the transaction to wait |
| tower_accdb_batch_acquired | counter | Number of fd_accdb_acquire (batch acquire) calls made by this tile |
| tower_accdb_acquire_failed | counter | Number of times we failed to acquire the cache lines needed for a transaction, and had to spin loop waiting |
| tower_accdb_bytes_read | counter | Number of bytes read from the account database |
| tower_accdb_read_operation | counter | Number of read operations performed on the account database |
| tower_accdb_bytes_written | counter | Number of bytes written to the account database |
| tower_accdb_write_operation | counter | Number of write operations performed on the account database |
| tower_accdb_bytes_copied | counter | Number of bytes copied within the account database |
Txsend Tile
| Metric | Type | Description |
|---|---|---|
| txsend_pkt_rx | counter | QUIC packets received |
| txsend_pkt_rx_bytes | counter | Bytes received via QUIC |
| txsend_pkt_tx | counter | QUIC packets sent |
| txsend_pkt_tx_bytes | counter | Bytes sent via QUIC |
| txsend_pkt_tx_retry | counter | QUIC Retry packets sent |
| txsend_conn_in_use | gauge | QUIC connection slots currently in use (allocated from connection create until free, including handshaking connections) |
| txsend_conn_state {quic_conn_state="invalid"} | gauge | QUIC connections in each state (Freed) |
| txsend_conn_state {quic_conn_state="handshake"} | gauge | QUIC connections in each state (Handshaking peer) |
| txsend_conn_state {quic_conn_state="handshake_complete"} | gauge | QUIC connections in each state (Handshake complete, confirming with peer) |
| txsend_conn_state {quic_conn_state="active"} | gauge | QUIC connections in each state (Active connection) |
| txsend_conn_state {quic_conn_state="peer_close"} | gauge | QUIC connections in each state (Peer requested close) |
| txsend_conn_state {quic_conn_state="abort"} | gauge | QUIC connections in each state (Connection terminating due to error) |
| txsend_conn_state {quic_conn_state="close_pending"} | gauge | QUIC connections in each state (Connection is closing) |
| txsend_conn_state {quic_conn_state="dead"} | gauge | QUIC connections in each state (Connection about to be freed) |
| txsend_conn_created | counter | Connections created |
| txsend_conn_closed | counter | Connections gracefully closed |
| txsend_conn_aborted | counter | Connections aborted |
| txsend_conn_timed_out | counter | Connections timed out |
| txsend_conn_retried | counter | Connections established with retry |
| txsend_conn_error_no_slots | counter | Connections that failed to create due to lack of slots |
| txsend_conn_error_retry_failed | counter | Connections that failed during retry (e.g. invalid token) |
| txsend_pkt_crypto_failed {quic_encryption_level="initial"} | counter | Packets that failed decryption (Initial) |
| txsend_pkt_crypto_failed {quic_encryption_level="early"} | counter | Packets that failed decryption (Early data) |
| txsend_pkt_crypto_failed {quic_encryption_level="handshake"} | counter | Packets that failed decryption (Handshake) |
| txsend_pkt_crypto_failed {quic_encryption_level="app"} | counter | Packets that failed decryption (App data) |
| txsend_pkt_no_key {quic_encryption_level="initial"} | counter | Packets that failed decryption due to missing key (Initial) |
| txsend_pkt_no_key {quic_encryption_level="early"} | counter | Packets that failed decryption due to missing key (Early data) |
| txsend_pkt_no_key {quic_encryption_level="handshake"} | counter | Packets that failed decryption due to missing key (Handshake) |
| txsend_pkt_no_key {quic_encryption_level="app"} | counter | Packets that failed decryption due to missing key (App data) |
| txsend_pkt_no_conn {quic_pkt_handle="initial"} | counter | Packets with an unknown connection ID (Initial) |
| txsend_pkt_no_conn {quic_pkt_handle="retry"} | counter | Packets with an unknown connection ID (Retry) |
| txsend_pkt_no_conn {quic_pkt_handle="handshake"} | counter | Packets with an unknown connection ID (Handshake) |
| txsend_pkt_no_conn {quic_pkt_handle="one_rtt"} | counter | Packets with an unknown connection ID (1-RTT) |
| txsend_pkt_src_invalid | counter | Packets dropped due to a wrong source IP |
| txsend_pkt_tx_alloc_failed | counter | Packet transmit jobs dropped due to allocation failure |
| txsend_pkt_net_header_invalid | counter | Packets dropped due to weird IP or UDP header |
| txsend_pkt_header_invalid | counter | Packets dropped due to weird QUIC header |
| txsend_pkt_undersize | counter | QUIC packets dropped due to being too small |
| txsend_pkt_oversize | counter | QUIC packets dropped due to being too large |
| txsend_pkt_rx_version_negotiation | counter | QUIC version negotiation packets received |
| txsend_pkt_tx_retransmitted {quic_encryption_level="initial"} | counter | QUIC packets retransmitted (Initial) |
| txsend_pkt_tx_retransmitted {quic_encryption_level="early"} | counter | QUIC packets retransmitted (Early data) |
| txsend_pkt_tx_retransmitted {quic_encryption_level="handshake"} | counter | QUIC packets retransmitted (Handshake) |
| txsend_pkt_tx_retransmitted {quic_encryption_level="app"} | counter | QUIC packets retransmitted (App data) |
| txsend_handshake_created | counter | Handshake flows created |
| txsend_handshake_error_alloc_fail | counter | Handshakes dropped due to alloc fail |
| txsend_handshake_evicted | counter | Handshakes dropped due to eviction |
| txsend_stream_rx | counter | Stream receive events |
| txsend_stream_rx_bytes | counter | Stream payload bytes received |
| txsend_frame_rx {quic_frame_type="unknown"} | counter | QUIC frames received (Unknown frame type) |
| txsend_frame_rx {quic_frame_type="ack"} | counter | QUIC frames received (ACK frame) |
| txsend_frame_rx {quic_frame_type="reset_stream"} | counter | QUIC frames received (RESET_STREAM frame) |
| txsend_frame_rx {quic_frame_type="stop_sending"} | counter | QUIC frames received (STOP_SENDING frame) |
| txsend_frame_rx {quic_frame_type="crypto"} | counter | QUIC frames received (CRYPTO frame) |
| txsend_frame_rx {quic_frame_type="new_token"} | counter | QUIC frames received (NEW_TOKEN frame) |
| txsend_frame_rx {quic_frame_type="stream"} | counter | QUIC frames received (STREAM frame) |
| txsend_frame_rx {quic_frame_type="max_data"} | counter | QUIC frames received (MAX_DATA frame) |
| txsend_frame_rx {quic_frame_type="max_stream_data"} | counter | QUIC frames received (MAX_STREAM_DATA frame) |
| txsend_frame_rx {quic_frame_type="max_streams"} | counter | QUIC frames received (MAX_STREAMS frame) |
| txsend_frame_rx {quic_frame_type="data_blocked"} | counter | QUIC frames received (DATA_BLOCKED frame) |
| txsend_frame_rx {quic_frame_type="stream_data_blocked"} | counter | QUIC frames received (STREAM_DATA_BLOCKED frame) |
| txsend_frame_rx {quic_frame_type="streams_blocked"} | counter | QUIC frames received (STREAMS_BLOCKED(bidi) frame) |
| txsend_frame_rx {quic_frame_type="new_connection_id"} | counter | QUIC frames received (NEW_CONN_ID frame) |
| txsend_frame_rx {quic_frame_type="retire_connection_id"} | counter | QUIC frames received (RETIRE_CONN_ID frame) |
| txsend_frame_rx {quic_frame_type="path_challenge"} | counter | QUIC frames received (PATH_CHALLENGE frame) |
| txsend_frame_rx {quic_frame_type="path_response"} | counter | QUIC frames received (PATH_RESPONSE frame) |
| txsend_frame_rx {quic_frame_type="connection_close_quic"} | counter | QUIC frames received (CONN_CLOSE(transport) frame) |
| txsend_frame_rx {quic_frame_type="connection_close_app"} | counter | QUIC frames received (CONN_CLOSE(app) frame) |
| txsend_frame_rx {quic_frame_type="handshake_done"} | counter | QUIC frames received (HANDSHAKE_DONE frame) |
| txsend_frame_rx {quic_frame_type="ping"} | counter | QUIC frames received (PING frame) |
| txsend_frame_rx {quic_frame_type="padding"} | counter | QUIC frames received (PADDING frame) |
| txsend_frame_parse_failed | counter | QUIC frames that failed to parse |
| txsend_frame_meta_acquired {frame_tx_alloc_result="success"} | counter | Attempts to acquire QUIC frame metadata (Success) |
| txsend_frame_meta_acquired {frame_tx_alloc_result="fail_empty_pool"} | counter | Attempts to acquire QUIC frame metadata (PktMetaPoolEmpty) |
| txsend_frame_meta_acquired {frame_tx_alloc_result="fail_connection_max"} | counter | Attempts to acquire QUIC frame metadata (ConnMaxedInflightFrames) |
| txsend_ack_tx {quic_ack_tx="noop"} | counter | ACK events (Non-ACK-eliciting packet) |
| txsend_ack_tx {quic_ack_tx="new"} | counter | ACK events (New ACK range) |
| txsend_ack_tx {quic_ack_tx="merged"} | counter | ACK events (Merged into existing ACK range) |
| txsend_ack_tx {quic_ack_tx="drop"} | counter | ACK events (Out of buffers) |
| txsend_ack_tx {quic_ack_tx="cancel"} | counter | ACK events (ACK suppressed by handler) |
| txsend_service_duration_seconds | histogram | Duration spent in service |
| txsend_rx_duration_seconds | histogram | Duration spent processing packets |
Diag Tile
| Metric | Type | Description |
|---|---|---|
| diag_bundle_status | gauge | Precise status of the bundle subsystem: 0=disabled (no bundle tiles configured), 1=disconnected (all bundle tiles disconnected), 2=connecting (at least one bundle tile connecting, none connected or sleeping), 3=connected (at least one bundle tile connected), 4=sleeping (at least one bundle tile sleeping, none connected) |
| diag_vote_status | gauge | Precise status of the vote subsystem: 0=disabled (non-voting or no tower tile), 1=not started (tower tile not running or no votes cast yet), 2=delinquent (vote distance exceeds threshold or vote stalled), 3=voting (voting normally) |
| diag_replay_status | gauge | Precise status of the replay subsystem: 0=disabled (no replay tile), 1=not started (replay tile not running or slots are zero), 2=behind (replay lagging behind turbine or reset slot stalled), 3=running (replay keeping up) |
| diag_turbine_status | gauge | Precise status of the turbine subsystem: 0=disabled (no shred or replay tiles), 1=not started (tiles not all running or turbine slot is zero), 2=stalled (turbine slot not advancing), 3=repair outpacing (repair byte throughput exceeds turbine), 4=running (turbine receiving normally) |
| diag_device_irq | counter | Number of device IRQs across all CPUs |
| diag_device_irq_undesired | counter | Number of device hard IRQs that stole CPU time from fixed tiles |
| diag_softirq {softirq="net"} | counter | Number of soft-IRQs across all CPUs (NET_TX, NET_RX) |
| diag_softirq {softirq="disk"} | counter | Number of soft-IRQs across all CPUs (BLOCK) |
| diag_softirq {softirq="other"} | counter | Number of soft-IRQs across all CPUs (e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...) |
| diag_softirq_undesired {softirq="net"} | counter | Number of soft-IRQs that stole CPU time from fixed tiles (NET_TX, NET_RX) |
| diag_softirq_undesired {softirq="disk"} | counter | Number of soft-IRQs that stole CPU time from fixed tiles (BLOCK) |
| diag_softirq_undesired {softirq="other"} | counter | Number of soft-IRQs that stole CPU time from fixed tiles (e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...) |
Event Tile
| Metric | Type | Description |
|---|---|---|
| event_conn_state | gauge | 0=disconnected, 1=connecting, 2=connected |
| event_queue_depth | gauge | Total events in the event queue (sent-but-unacknowledged plus unsent) |
| event_queue_unsent | gauge | Events in the event queue not yet sent to the event service |
| event_queue_dropped | counter | Events dropped because the event queue was full |
| event_queue_bytes_used | gauge | Bytes used in the event queue |
| event_queue_bytes_capacity | gauge | Total capacity of the event queue, in bytes |
| event_sent | counter | Events sent to the event service |
| event_acked | counter | Events acknowledged by the event service |
| event_last_acked_id | gauge | Event id (nonce) of the most recently acknowledged event |
| event_bytes_written | counter | Bytes written to the event service |
| event_bytes_read | counter | Bytes read from the event service |
| event_auth_failed | counter | Authentication failures with the event service |
| event_invalid_message | counter | Malformed messages received from the event service |
| event_conn_attempt | counter | Connection attempts to the event service |
| event_handshake_timeout | counter | Authentication handshake timeouts with the event service |
Gui Tile
| Metric | Type | Description |
|---|---|---|
| gui_conn_active | gauge | Active HTTP connections to the GUI service, excluding connections that have been upgraded to a WebSocket connection |
| gui_websocket_conn_active | gauge | Active WebSocket connections to the GUI service |
| gui_websocket_frame_tx | counter | WebSocket frames sent to all connections to the GUI service |
| gui_websocket_frame_rx | counter | WebSocket frames received from all connections to the GUI service |
| gui_bytes_written | counter | Bytes written to all connections to the GUI service |
| gui_bytes_read | counter | Bytes read from all connections to the GUI service |
Metric Tile
| Metric | Type | Description |
|---|---|---|
| metric_boot_timestamp_nanos | gauge | Timestamp when validator was started, in nanoseconds since epoch |
| metric_conn_active | gauge | Active HTTP connections to the Prometheus endpoint |
| metric_bytes_written | counter | Bytes written to all responses on the Prometheus endpoint |
| metric_bytes_read | counter | Bytes read from all requests to the Prometheus endpoint |
Rpc Tile
| Metric | Type | Description |
|---|---|---|
| rpc_request_duration_seconds | histogram | Duration spent in service |
| rpc_request_served {rpc_method="genesis"} | counter | Number of RPC requests served (GET /genesis.tar.bz2) |
| rpc_request_served {rpc_method="unknown"} | counter | Number of RPC requests served (Unknown or unsupported method) |
| rpc_request_served {rpc_method="getAccountInfo"} | counter | Number of RPC requests served (getAccountInfo) |
| rpc_request_served {rpc_method="getBalance"} | counter | Number of RPC requests served (getBalance) |
| rpc_request_served {rpc_method="getBlockHeight"} | counter | Number of RPC requests served (getBlockHeight) |
| rpc_request_served {rpc_method="getClusterNodes"} | counter | Number of RPC requests served (getClusterNodes) |
| rpc_request_served {rpc_method="getEpochInfo"} | counter | Number of RPC requests served (getEpochInfo) |
| rpc_request_served {rpc_method="getGenesisHash"} | counter | Number of RPC requests served (getGenesisHash) |
| rpc_request_served {rpc_method="getHealth"} | counter | Number of RPC requests served (getHealth) |
| rpc_request_served {rpc_method="getIdentity"} | counter | Number of RPC requests served (getIdentity) |
| rpc_request_served {rpc_method="getInflationGovernor"} | counter | Number of RPC requests served (getInflationGovernor) |
| rpc_request_served {rpc_method="getLatestBlockhash"} | counter | Number of RPC requests served (getLatestBlockhash) |
| rpc_request_served {rpc_method="getMinimumBalanceForRentExemption"} | counter | Number of RPC requests served (getMinimumBalanceForRentExemption) |
| rpc_request_served {rpc_method="getMultipleAccounts"} | counter | Number of RPC requests served (getMultipleAccounts) |
| rpc_request_served {rpc_method="getSlot"} | counter | Number of RPC requests served (getSlot) |
| rpc_request_served {rpc_method="getTransactionCount"} | counter | Number of RPC requests served (getTransactionCount) |
| rpc_request_served {rpc_method="getVersion"} | counter | Number of RPC requests served (getVersion) |
| rpc_conn_active | gauge | The number of active HTTP connections to the RPC service |
| rpc_websocket_conn_active | gauge | The number of active WebSocket connections to the RPC service |
| rpc_websocket_subscription_active {rpc_event_type="vote"} | gauge | The number of active WebSocket subscriptions to the RPC service, broken down by subscription type (vote) |
| rpc_websocket_subscription_active {rpc_event_type="slot"} | gauge | The number of active WebSocket subscriptions to the RPC service, broken down by subscription type (slot) |
| rpc_websocket_event_unique_sent {rpc_event_type="vote"} | counter | Number of unique WebSocket events sent by the RPC service (vote) |
| rpc_websocket_event_unique_sent {rpc_event_type="slot"} | counter | Number of unique WebSocket events sent by the RPC service (slot) |
| rpc_websocket_event_sent {rpc_event_type="vote"} | counter | Number of WebSocket events sent by the RPC service across all subscriptions (vote) |
| rpc_websocket_event_sent {rpc_event_type="slot"} | counter | Number of WebSocket events sent by the RPC service across all subscriptions (slot) |
| rpc_accdb_account_acquired {accdb_cache_class="class0"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (0-128 B) |
| rpc_accdb_account_acquired {accdb_cache_class="class1"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (129-512 B) |
| rpc_accdb_account_acquired {accdb_cache_class="class2"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (513 B-2 KiB) |
| rpc_accdb_account_acquired {accdb_cache_class="class3"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (2-8 KiB) |
| rpc_accdb_account_acquired {accdb_cache_class="class4"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (8-32 KiB) |
| rpc_accdb_account_acquired {accdb_cache_class="class5"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (32-128 KiB) |
| rpc_accdb_account_acquired {accdb_cache_class="class6"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (128 KiB-1 MiB) |
| rpc_accdb_account_acquired {accdb_cache_class="class7"} | counter | Number of accounts read from the account database, attributed to the cache size class of the account's current data size (1-10 MiB) |
| rpc_accdb_account_not_found {accdb_cache_class="class0"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (0-128 B) |
| rpc_accdb_account_not_found {accdb_cache_class="class1"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (129-512 B) |
| rpc_accdb_account_not_found {accdb_cache_class="class2"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (513 B-2 KiB) |
| rpc_accdb_account_not_found {accdb_cache_class="class3"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (2-8 KiB) |
| rpc_accdb_account_not_found {accdb_cache_class="class4"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (8-32 KiB) |
| rpc_accdb_account_not_found {accdb_cache_class="class5"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (32-128 KiB) |
| rpc_accdb_account_not_found {accdb_cache_class="class6"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (128 KiB-1 MiB) |
| rpc_accdb_account_not_found {accdb_cache_class="class7"} | counter | Number of accounts that were not found in the account database cache and had to be read from disk, broken down by cache size class (1-10 MiB) |
| rpc_accdb_account_waited | counter | Number of accounts that had to wait for a concurrent writer to publish a disk offset before being read |
| rpc_accdb_batch_acquired | counter | Number of fd_accdb_acquire (batch acquire) calls made by this tile |
| rpc_accdb_bytes_read | counter | Number of bytes read from the account database |
| rpc_accdb_read_operation | counter | Number of read operations performed on the account database |
| rpc_accdb_bytes_copied | counter | Number of bytes copied out of the account database cache on a cache hit |
Resolh Tile
| Metric | Type | Description |
|---|---|---|
| resolh_txn_no_bank | counter | Transactions dropped because the bank was not available |
| resolh_stash_operation {resolve_stash_operation="inserted"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolh_stash_operation {resolve_stash_operation="overrun"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolh_stash_operation {resolve_stash_operation="published"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolh_stash_operation {resolve_stash_operation="removed"} | counter | Operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories) |
| resolh_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolh_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolh_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolh_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolh_lut_resolved {lut_resolve_result="account_not_found"} | counter | Address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolh_lut_resolved {lut_resolve_result="success"} | counter | Address lookup tables resolved (Resolved successfully) |
| resolh_blockhash_expired | counter | Transactions that failed to resolve because the blockhash was expired |
| resolh_txn_bundle_peer_failed | counter | Transactions that failed to resolve because a peer transaction in the bundle failed |
Bank Tile
| Metric | Type | Description |
|---|---|---|
| bank_txn_sanitize_failed | counter | Transactions that failed to sanitize |
| bank_txn_not_executed_failed | counter | Transactions that did not execute. Different from transactions that fail to execute, which make it onto the chain |
| bank_txn_load_address_table {lut_resolve_result="invalid_lookup_index"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The transaction referenced an index in a LUT that didn't exist) |
| bank_txn_load_address_table {lut_resolve_result="account_uninitialized"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT hasn't been initialized) |
| bank_txn_load_address_table {lut_resolve_result="invalid_account_data"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT couldn't be parsed) |
| bank_txn_load_address_table {lut_resolve_result="invalid_account_owner"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| bank_txn_load_address_table {lut_resolve_result="account_not_found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT couldn't be found) |
| bank_txn_load_address_table {lut_resolve_result="success"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (Resolved successfully) |
| bank_txn_result {transaction_error="success"} | counter | Result of loading and executing a transaction (Success) |
| bank_txn_result {transaction_error="account_in_use"} | counter | Result of loading and executing a transaction (An account is already being processed in another transaction in a way that does not support parallelism) |
| bank_txn_result {transaction_error="account_loaded_twice"} | counter | Result of loading and executing a transaction (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys) |
| bank_txn_result {transaction_error="account_not_found"} | counter | Result of loading and executing a transaction (Attempt to debit an account but found no record of a prior credit) |
| bank_txn_result {transaction_error="program_account_not_found"} | counter | Result of loading and executing a transaction (Attempt to load a program that does not exist) |
| bank_txn_result {transaction_error="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction) |
| bank_txn_result {transaction_error="invalid_account_for_fee"} | counter | Result of loading and executing a transaction (This account may not be used to pay transaction fees) |
| bank_txn_result {transaction_error="already_processed"} | counter | Result of loading and executing a transaction (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack) |
| bank_txn_result {transaction_error="blockhash_not_found"} | counter | Result of loading and executing a transaction (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded) |
| bank_txn_result {transaction_error="instruction_error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) |
| bank_txn_result {transaction_error="call_chain_too_deep"} | counter | Result of loading and executing a transaction (Loader call chain is too deep) |
| bank_txn_result {transaction_error="missing_signature_for_fee"} | counter | Result of loading and executing a transaction (Transaction requires a fee but has no signature present) |
| bank_txn_result {transaction_error="invalid_account_index"} | counter | Result of loading and executing a transaction (Transaction contains an invalid account reference) |
| bank_txn_result {transaction_error="signature_failure"} | counter | Result of loading and executing a transaction (Transaction did not pass signature verification) |
| bank_txn_result {transaction_error="invalid_program_for_execution"} | counter | Result of loading and executing a transaction (This program may not be used for executing instructions) |
| bank_txn_result {transaction_error="sanitize_failure"} | counter | Result of loading and executing a transaction (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked) |
| bank_txn_result {transaction_error="cluster_maintenance"} | counter | Result of loading and executing a transaction (Transactions are currently disabled due to cluster maintenance) |
| bank_txn_result {transaction_error="account_borrow_outstanding"} | counter | Result of loading and executing a transaction (Transaction processing left an account with an outstanding borrowed reference) |
| bank_txn_result {transaction_error="would_exceed_max_block_cost_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max Block Cost Limit) |
| bank_txn_result {transaction_error="unsupported_version"} | counter | Result of loading and executing a transaction (Transaction version is unsupported) |
| bank_txn_result {transaction_error="invalid_writable_account"} | counter | Result of loading and executing a transaction (Transaction loads a writable account that cannot be written) |
| bank_txn_result {transaction_error="would_exceed_max_account_cost_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max account limit within the block) |
| bank_txn_result {transaction_error="would_exceed_account_data_block_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed account data limit within the block) |
| bank_txn_result {transaction_error="too_many_account_locks"} | counter | Result of loading and executing a transaction (Transaction locked too many accounts) |
| bank_txn_result {transaction_error="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction (Address lookup table not found) |
| bank_txn_result {transaction_error="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction (Attempted to lookup addresses from an account owned by the wrong program) |
| bank_txn_result {transaction_error="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction (Attempted to lookup addresses from an invalid account) |
| bank_txn_result {transaction_error="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction (Address table lookup uses an invalid index) |
| bank_txn_result {transaction_error="invalid_rent_paying_account"} | counter | Result of loading and executing a transaction (Transaction leaves an account with a lower balance than rent-exempt minimum) |
| bank_txn_result {transaction_error="would_exceed_max_vote_cost_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max Vote Cost Limit) |
| bank_txn_result {transaction_error="would_exceed_account_data_total_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed total account data limit) |
| bank_txn_result {transaction_error="duplicate_instruction"} | counter | Result of loading and executing a transaction (Transaction contains a duplicate instruction that is not allowed) |
| bank_txn_result {transaction_error="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction (Transaction results in an account with insufficient funds for rent) |
| bank_txn_result {transaction_error="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction (Transaction exceeded max loaded accounts data size cap) |
| bank_txn_result {transaction_error="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction (LoadedAccountsDataSizeLimit set for transaction must be greater than 0) |
| bank_txn_result {transaction_error="resanitization_needed"} | counter | Result of loading and executing a transaction (Sanitized transaction differed before/after feature activation. Needs to be resanitized) |
| bank_txn_result {transaction_error="program_execution_temporarily_restricted"} | counter | Result of loading and executing a transaction (Program execution is temporarily restricted on an account) |
| bank_txn_result {transaction_error="unbalanced_transaction"} | counter | Result of loading and executing a transaction (The total balance before the transaction does not equal the total balance after the transaction) |
| bank_txn_result {transaction_error="program_cache_hit_max_limit"} | counter | Result of loading and executing a transaction (The total program cache size hit the maximum allowed limit) |
| bank_txn_result {transaction_error="commit_cancelled"} | counter | Result of loading and executing a transaction (The process for committing the transaction was cancelled internally) |
| bank_txn_result {transaction_error="bundle_peer"} | counter | Result of loading and executing a transaction (Transaction is part of a bundle and one of the peer transactions failed) |
| bank_txn_processing_failed | counter | Transactions for which the processing stage failed and won't land on chain |
| bank_txn_fee_only | counter | Transactions that will land on chain but without executing |
| bank_txn_executed {txn_execute_result="success"} | counter | Transactions that execute on chain, by outcome (Executed on chain and succeeded) |
| bank_txn_executed {txn_execute_result="failed"} | counter | Transactions that execute on chain, by outcome (Executed on chain but failed) |
| bank_txn_cost_model_undercount | counter | Transactions that used more CUs than the cost model should have permitted them to |
Pohh Tile
| Metric | Type | Description |
|---|---|---|
| pohh_begin_leader_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the bank |
| pohh_first_microblock_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock |
| pohh_slot_done_delay_seconds | histogram | Delay between when we become leader in a slot and when we finish the slot |
| pohh_bundle_initialize_delay_seconds | histogram | Delay in starting the slot caused by loading the information needed to generate the bundle crank transactions |
Store Tile
| Metric | Type | Description |
|---|---|---|
| store_txn_inserted | counter | Transactions produced while we were leader in the shreds that have been inserted so far |
Backt Tile
| Metric | Type | Description |
|---|---|---|
| backt_final_slot | gauge | Slot after which the backtest will complete |
| backt_start_slot | gauge | Slot at which the backtest started |
Benchs Tile
| Metric | Type | Description |
|---|---|---|
| benchs_txn_tx | counter | Benchmark transactions sent |
Guih Tile
| Metric | Type | Description |
|---|---|---|
| guih_conn_active | gauge | Active HTTP connections to the GUI service, excluding connections that have been upgraded to a WebSocket connection |
| guih_websocket_conn_active | gauge | Active WebSocket connections to the GUI service |
| guih_websocket_frame_tx | counter | WebSocket frames sent to all connections to the GUI service |
| guih_websocket_frame_rx | counter | WebSocket frames received from all connections to the GUI service |
| guih_bytes_written | counter | Bytes written to all connections to the GUI service |
| guih_bytes_read | counter | Bytes read from all connections to the GUI service |