Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via a Prometheus HTTP endpoint:
toml
[tiles.metric]
prometheus_listen_port = 7999sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric representing a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_consumed_count | counter | The number of times the link reader has consumed a fragment |
| link_consumed_size_bytes | counter | The total number of bytes read by the link consumer |
| link_filtered_count | counter | The number of fragments that were filtered and not consumed |
| link_filtered_size_bytes | counter | The total number of bytes read by the link consumer that were filtered |
| link_overrun_polling_count | counter | The number of times the link has been overrun while polling |
| link_overrun_polling_frag_count | counter | The number of fragments the link has not processed because it was overrun while polling |
| link_overrun_reading_count | counter | The number of input overruns detected while reading metadata by the consumer |
| link_overrun_reading_frag_count | counter | The number of fragments the link has not processed because it was overrun while reading |
| link_slow_count | counter | The number of times the consumer was detected as rate limiting consumer by the producer |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | The process ID of the tile |
| tile_tid | gauge | The thread ID of the tile. Always the same as the Pid in production, but might be different in development |
| tile_last_cpu | gauge | Index of the CPU last executed on |
| tile_context_switch_involuntary_count | counter | The number of involuntary context switches |
| tile_context_switch_voluntary_count | counter | The number of voluntary context switches |
| tile_page_fault_major_count | counter | The number of major page faults |
| tile_page_fault_minor_count | counter | The number of minor page faults |
| tile_status | gauge | The current status of the tile. 0 is booting, 1 is running. 2 is shutdown |
| tile_heartbeat | gauge | The last UNIX timestamp in nanoseconds that the tile heartbeated |
| tile_in_backpressure | gauge | Whether the tile is currently backpressured or not, either 1 or 0 |
| tile_backpressure_count | counter | Number of times the tile has had to wait for one of more consumers to catch up to resume publishing |
| tile_regime_duration_nanos {tile_regime="caught_up_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Caught up + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="processing_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Processing + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="backpressure_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Backpressure + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="caught_up_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Caught up + Prefrag) |
| tile_regime_duration_nanos {tile_regime="processing_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Processing + Prefrag) |
| tile_regime_duration_nanos {tile_regime="backpressure_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Backpressure + Prefrag) |
| tile_regime_duration_nanos {tile_regime="caught_up_postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Caught up + Postfrag) |
| tile_regime_duration_nanos {tile_regime="processing_postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes (Processing + Postfrag) |
| tile_cpu_duration_nanos {cpu_regime="wait"} | counter | CPU time spent in each CPU regime (Wait (task was runnable but not scheduled)) |
| tile_cpu_duration_nanos {cpu_regime="idle"} | counter | CPU time spent in each CPU regime (Idle (task was not runnable)) |
| tile_cpu_duration_nanos {cpu_regime="user"} | counter | CPU time spent in each CPU regime (User (task was scheduled and executing in user mode)) |
| tile_cpu_duration_nanos {cpu_regime="system"} | counter | CPU time spent in each CPU regime (System (task was scheduled and executing in kernel mode)) |
Genesi Tile
| Metric | Type | Description |
|---|
Ipecho Tile
| Metric | Type | Description |
|---|---|---|
| ipecho_current_shred_version | gauge | The current shred version used by the validator |
| ipecho_connection_count | gauge | The number of active connections to the ipecho service |
| ipecho_connections_closed_ok | counter | The number of connections to the ipecho service that have been made and closed normally |
| ipecho_connections_closed_error | counter | The number of connections to the ipecho service that have been made and closed abnormally |
| ipecho_bytes_read | counter | The total number of bytes read from all connections to the ipecho service |
| ipecho_bytes_written | counter | The total number of bytes written to all connections to the ipecho service |
Snapct Tile
| Metric | Type | Description |
|---|---|---|
| snapct_state | gauge | State of the tile |
| snapct_full_num_retries | counter | Number of times we aborted and retried full snapshot download because the peer was too slow |
| snapct_incremental_num_retries | counter | Number of times we aborted and retried incremental snapshot download because the peer was too slow |
| snapct_full_bytes_read | gauge | Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_full_bytes_written | gauge | Number of bytes written so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_full_bytes_total | gauge | Total size of the full snapshot file. Might change if snapshot load is aborted and restarted |
| snapct_full_download_retries | gauge | Number of times we retried the full snapshot download because the peer was too slow |
| snapct_incremental_bytes_read | gauge | Number of bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_incremental_bytes_written | gauge | Number of bytes written so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_incremental_bytes_total | gauge | Total size of the incremental snapshot file. Might change if snapshot load is aborted and restarted |
| snapct_incremental_download_retries | gauge | Number of times we retried the incremental snapshot download because the peer was too slow |
| snapct_predicted_slot | gauge | The predicted slot from which replay starts after snapshot loading finishes. Might change if snapshot load is aborted and restarted |
| snapct_ssl_alloc_errors | counter | Number of SSL allocation errors encountered. |
Snapld Tile
| Metric | Type | Description |
|---|---|---|
| snapld_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapld_ssl_alloc_errors | counter | Number of SSL allocation errors encountered. |
Snapdc Tile
| Metric | Type | Description |
|---|---|---|
| snapdc_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapdc_full_compressed_bytes_read | gauge | Number of bytes read so far from the compressed full snapshot file. Might decrease if snapshot load is aborted and restarted |
| snapdc_full_decompressed_bytes_written | gauge | Number of bytes decompressed so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapdc_incremental_compressed_bytes_read | gauge | Number of bytes read so far from the compressed incremental snapshot file. Might decrease if snapshot load is aborted and restarted |
| snapdc_incremental_decompressed_bytes_written | gauge | Number of bytes decompressed so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
Snapin Tile
| Metric | Type | Description |
|---|---|---|
| snapin_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapin_full_bytes_read | gauge | Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapin_incremental_bytes_read | gauge | Number of bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapin_accounts_loaded | gauge | Number of accounts seen during snapshot loading. Includes duplicates. Resets if snapshot load restarts |
| snapin_accounts_replaced | gauge | Number of previously inserted accounts replaced by a later duplicate. Resets if snapshot load restarts |
| snapin_accounts_ignored | gauge | Number of stale duplicate accounts dropped because a previously inserted account was newer. Resets if snapshot load restarts |
Snapwr Tile
| Metric | Type | Description |
|---|---|---|
| snapwr_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 4=SHUTDOWN |
| snapwr_file_capacity_bytes | gauge | Account database file capacity in bytes |
| snapwr_file_used_bytes | gauge | Current number of bytes used in the account database file |
Snapwh Tile
| Metric | Type | Description |
|---|---|---|
| snapwh_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 4=SHUTDOWN |
Snapla Tile
| Metric | Type | Description |
|---|---|---|
| snapla_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapla_full_accounts_hashed | gauge | Number of accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
| snapla_incremental_accounts_hashed | gauge | Number of accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
Snapls Tile
| Metric | Type | Description |
|---|---|---|
| snapls_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapls_full_accounts_hashed | gauge | Number of accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
| snapls_incremental_accounts_hashed | gauge | Number of accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
Snapwm Tile
| Metric | Type | Description |
|---|---|---|
| snapwm_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapwm_accounts_loaded | gauge | Number of accounts seen during snapshot loading. Includes duplicates. Resets if snapshot load restarts |
| snapwm_accounts_replaced | gauge | Number of previously inserted accounts replaced by a later duplicate. Resets if snapshot load restarts |
| snapwm_accounts_ignored | gauge | Number of stale duplicate accounts dropped because a previously inserted account was newer. Resets if snapshot load restarts |
| snapwm_accounts_active | gauge | Current number of accounts in index. Resets if snapshot load restarts |
Snaplh Tile
| Metric | Type | Description |
|---|---|---|
| snaplh_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snaplh_full_accounts_hashed | gauge | Number of accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
| snaplh_incremental_accounts_hashed | gauge | Number of accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
Snaplv Tile
| Metric | Type | Description |
|---|---|---|
| snaplv_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snaplv_full_duplicate_accounts_hashed | gauge | Number of duplicate accounts hashed for the full snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
| snaplv_incremental_duplicate_accounts_hashed | gauge | Number of duplicate accounts hashed for the incremental snapshot during snapshot loading. Might decrease if snapshot load is aborted and restarted |
Netlnk Tile
| Metric | Type | Description |
|---|---|---|
| netlnk_drop_events | counter | Number of netlink drop events caught |
| netlnk_link_full_syncs | counter | Number of full link table syncs done |
| netlnk_route_full_syncs | counter | Number of full route table syncs done |
| netlnk_updates {netlink_msg="link"} | counter | Number of netlink live updates processed (Link) |
| netlnk_updates {netlink_msg="neigh"} | counter | Number of netlink live updates processed (Neighbor Table Entry) |
| netlnk_updates {netlink_msg="ipv4_route"} | counter | Number of netlink live updates processed (IPv4 Route Table Entry) |
| netlnk_interface_count | gauge | Number of network interfaces |
| netlnk_route_count {route_table="local"} | gauge | Number of IPv4 routes (Local) |
| netlnk_route_count {route_table="main"} | gauge | Number of IPv4 routes (Main) |
| netlnk_neigh_probe_sent | counter | Number of neighbor solicit requests sent to kernel |
| netlnk_neigh_probe_fails | counter | Number of neighbor solicit requests that failed to send (kernel too slow) |
| netlnk_neigh_probe_rate_limit_host | counter | Number of neighbor solicit that exceeded the per-host rate limit |
| netlnk_neigh_probe_rate_limit_global | counter | Number of neighbor solicit that exceeded the global rate limit |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_rx_pkt_cnt | counter | Packet receive count |
| net_rx_bytes_total | counter | Total number of bytes received (including Ethernet header) |
| net_rx_undersz_cnt | counter | Number of incoming packets dropped due to being too small |
| net_rx_fill_blocked_cnt | counter | Number of incoming packets dropped due to fill ring being full |
| net_rx_backpressure_cnt | counter | Number of incoming packets dropped due to backpressure |
| net_rx_busy_cnt | gauge | Number of receive buffers currently busy |
| net_rx_idle_cnt | gauge | Number of receive buffers currently idle |
| net_tx_submit_cnt | counter | Number of packet transmit jobs submitted |
| net_tx_complete_cnt | counter | Number of packet transmit jobs marked as completed by the kernel |
| net_tx_bytes_total | counter | Total number of bytes transmitted (including Ethernet header) |
| net_tx_route_fail_cnt | counter | Number of packet transmit jobs dropped due to route failure |
| net_tx_neighbor_fail_cnt | counter | Number of packet transmit jobs dropped due to unresolved neighbor |
| net_tx_full_fail_cnt | counter | Number of packet transmit jobs dropped due to XDP TX ring full or missing completions |
| net_tx_busy_cnt | gauge | Number of transmit buffers currently busy |
| net_tx_idle_cnt | gauge | Number of transmit buffers currently idle |
| net_xsk_tx_wakeup_cnt | counter | Number of XSK sendto syscalls dispatched |
| net_xsk_rx_wakeup_cnt | counter | Number of XSK recvmsg syscalls dispatched |
| net_xdp_rx_dropped_other | counter | Dropped for other reasons (xdp_statistics_v0.rx_dropped) |
| net_xdp_rx_invalid_descs | counter | Dropped due to invalid descriptor (xdp_statistics_v0.rx_invalid_descs) |
| net_xdp_tx_invalid_descs | counter | Dropped due to invalid descriptor (xdp_statistics_v0.tx_invalid_descs) |
| net_xdp_rx_ring_full | counter | Dropped due to rx ring being full (xdp_statistics_v1.rx_ring_full) |
| net_xdp_rx_fill_ring_empty_descs | counter | Failed to retrieve item from fill ring (xdp_statistics_v1.rx_fill_ring_empty_descs) |
| net_xdp_tx_ring_empty_descs | counter | Failed to retrieve item from tx ring (xdp_statistics_v1.tx_ring_empty_descs) |
| net_rx_gre_cnt | counter | Number of valid GRE packets received |
| net_rx_gre_invalid_cnt | counter | Number of invalid GRE packets received |
| net_rx_gre_ignored_cnt | counter | Number of received but ignored GRE packets |
| net_tx_gre_cnt | counter | Number of GRE packet transmit jobs submitted |
| net_tx_gre_route_fail_cnt | counter | Number of GRE packets transmit jobs dropped due to route failure |
| net_rx_src_addr_invalid_cnt | counter | Number of incoming packets dropped due to invalid source IP address |
Sock Tile
| Metric | Type | Description |
|---|---|---|
| sock_syscalls_sendmmsg {sock_err="no_error"} | counter | Number of sendmmsg syscalls dispatched (No error) |
| sock_syscalls_sendmmsg {sock_err="slow"} | counter | Number of sendmmsg syscalls dispatched (ENOBUFS, EAGAIN error) |
| sock_syscalls_sendmmsg {sock_err="perm"} | counter | Number of sendmmsg syscalls dispatched (EPERM error (blocked by netfilter)) |
| sock_syscalls_sendmmsg {sock_err="unreach"} | counter | Number of sendmmsg syscalls dispatched (ENETUNREACH, EHOSTUNREACH error) |
| sock_syscalls_sendmmsg {sock_err="down"} | counter | Number of sendmmsg syscalls dispatched (ENONET, ENETDOWN, EHOSTDOWN error) |
| sock_syscalls_sendmmsg {sock_err="other"} | counter | Number of sendmmsg syscalls dispatched (Unrecognized error code) |
| sock_syscalls_recvmmsg | counter | Number of recvmsg syscalls dispatched |
| sock_rx_pkt_cnt | counter | Number of packets received |
| sock_tx_pkt_cnt | counter | Number of packets sent |
| sock_tx_drop_cnt | counter | Number of packets failed to send |
| sock_tx_bytes_total | counter | Total number of bytes transmitted (including Ethernet header) |
| sock_rx_bytes_total | counter | Total number of bytes received (including Ethernet header) |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_txns_overrun | counter | Count of txns overrun before reassembled (too small txn_reassembly_count) |
| quic_txn_reasms_started | counter | Count of fragmented txn receive ops started |
| quic_txn_reasms_active | gauge | Number of fragmented txn receive ops currently active |
| quic_frags_ok | counter | Count of txn frags received |
| quic_frags_gap | counter | Count of txn frags dropped due to data gap |
| quic_frags_dup | counter | Count of txn frags dropped due to dup (stream already completed) |
| quic_txns_received {tpu_recv_type="udp"} | counter | Count of txns received via TPU (TPU/UDP) |
| quic_txns_received {tpu_recv_type="quic_fast"} | counter | Count of txns received via TPU (TPU/QUIC unfragmented) |
| quic_txns_received {tpu_recv_type="quic_frag"} | counter | Count of txns received via TPU (TPU/QUIC fragmented) |
| quic_txns_abandoned | counter | Count of txns abandoned because a conn was lost |
| quic_txn_undersz | counter | Count of txns received via QUIC dropped because they were too small |
| quic_txn_oversz | counter | Count of txns received via QUIC dropped because they were too large |
| quic_legacy_txn_undersz | counter | Count of packets received on the non-QUIC port that were too small to be a valid IP packet |
| quic_legacy_txn_oversz | counter | Count of packets received on the non-QUIC port that were too large to be a valid transaction |
| quic_received_packets | counter | Number of IP packets received |
| quic_received_bytes | counter | Total bytes received (including IP, UDP, QUIC headers) |
| quic_sent_packets | counter | Number of IP packets sent |
| quic_sent_bytes | counter | Total bytes sent (including IP, UDP, QUIC headers) |
| quic_connections_alloc | gauge | The number of currently allocated QUIC connections |
| quic_connections_state {quic_conn_state="invalid"} | gauge | The number of QUIC connections in each state (Freed) |
| quic_connections_state {quic_conn_state="handshake"} | gauge | The number of QUIC connections in each state (Handshaking peer) |
| quic_connections_state {quic_conn_state="handshake_complete"} | gauge | The number of QUIC connections in each state (Handshake complete, confirming with peer) |
| quic_connections_state {quic_conn_state="active"} | gauge | The number of QUIC connections in each state (Active connection) |
| quic_connections_state {quic_conn_state="peer_close"} | gauge | The number of QUIC connections in each state (Peer requested close) |
| quic_connections_state {quic_conn_state="abort"} | gauge | The number of QUIC connections in each state (Connection terminating due to error) |
| quic_connections_state {quic_conn_state="close_pending"} | gauge | The number of QUIC connections in each state (Connection is closing) |
| quic_connections_state {quic_conn_state="dead"} | gauge | The number of QUIC connections in each state (Connection about to be freed) |
| quic_connections_created | counter | The total number of connections that have been created |
| quic_connections_closed | counter | Number of connections gracefully closed |
| quic_connections_aborted | counter | Number of connections aborted |
| quic_connections_timed_out | counter | Number of connections timed out |
| quic_connections_retried | counter | Number of connections established with retry |
| quic_connection_error_no_slots | counter | Number of connections that failed to create due to lack of slots |
| quic_connection_error_retry_fail | counter | Number of connections that failed during retry (e.g. invalid token) |
| quic_pkt_no_conn {quic_pkt_handle="initial"} | counter | Number of packets with an unknown connection ID (Initial) |
| quic_pkt_no_conn {quic_pkt_handle="retry"} | counter | Number of packets with an unknown connection ID (Retry) |
| quic_pkt_no_conn {quic_pkt_handle="handshake"} | counter | Number of packets with an unknown connection ID (Handshake) |
| quic_pkt_no_conn {quic_pkt_handle="one_rtt"} | counter | Number of packets with an unknown connection ID (1-RTT) |
| quic_frame_tx_alloc {frame_tx_alloc_result="success"} | counter | Results of attempts to acquire QUIC frame metadata (Success) |
| quic_frame_tx_alloc {frame_tx_alloc_result="fail_empty_pool"} | counter | Results of attempts to acquire QUIC frame metadata (PktMetaPoolEmpty) |
| quic_frame_tx_alloc {frame_tx_alloc_result="fail_conn_max"} | counter | Results of attempts to acquire QUIC frame metadata (ConnMaxedInflightFrames) |
| quic_initial_token_len {quic_initial_token_len="zero"} | counter | Number of Initial packets grouped by token length (No token) |
| quic_initial_token_len {quic_initial_token_len="fd_quic_len"} | counter | Number of Initial packets grouped by token length (fd_quic retry token length) |
| quic_initial_token_len {quic_initial_token_len="invalid_len"} | counter | Number of Initial packets grouped by token length (Invalid token length) |
| quic_handshakes_created | counter | Number of handshake flows created |
| quic_handshake_error_alloc_fail | counter | Number of handshakes dropped due to alloc fail |
| quic_handshake_evicted | counter | Number of handshakes dropped due to eviction |
| quic_stream_received_events | counter | Number of stream RX events |
| quic_stream_received_bytes | counter | Total stream payload bytes received |
| quic_received_frames {quic_frame_type="unknown"} | counter | Number of QUIC frames received (Unknown frame type) |
| quic_received_frames {quic_frame_type="ack"} | counter | Number of QUIC frames received (ACK frame) |
| quic_received_frames {quic_frame_type="reset_stream"} | counter | Number of QUIC frames received (RESET_STREAM frame) |
| quic_received_frames {quic_frame_type="stop_sending"} | counter | Number of QUIC frames received (STOP_SENDING frame) |
| quic_received_frames {quic_frame_type="crypto"} | counter | Number of QUIC frames received (CRYPTO frame) |
| quic_received_frames {quic_frame_type="new_token"} | counter | Number of QUIC frames received (NEW_TOKEN frame) |
| quic_received_frames {quic_frame_type="stream"} | counter | Number of QUIC frames received (STREAM frame) |
| quic_received_frames {quic_frame_type="max_data"} | counter | Number of QUIC frames received (MAX_DATA frame) |
| quic_received_frames {quic_frame_type="max_stream_data"} | counter | Number of QUIC frames received (MAX_STREAM_DATA frame) |
| quic_received_frames {quic_frame_type="max_streams"} | counter | Number of QUIC frames received (MAX_STREAMS frame) |
| quic_received_frames {quic_frame_type="data_blocked"} | counter | Number of QUIC frames received (DATA_BLOCKED frame) |
| quic_received_frames {quic_frame_type="stream_data_blocked"} | counter | Number of QUIC frames received (STREAM_DATA_BLOCKED frame) |
| quic_received_frames {quic_frame_type="streams_blocked"} | counter | Number of QUIC frames received (STREAMS_BLOCKED(bidi) frame) |
| quic_received_frames {quic_frame_type="new_conn_id"} | counter | Number of QUIC frames received (NEW_CONN_ID frame) |
| quic_received_frames {quic_frame_type="retire_conn_id"} | counter | Number of QUIC frames received (RETIRE_CONN_ID frame) |
| quic_received_frames {quic_frame_type="path_challenge"} | counter | Number of QUIC frames received (PATH_CHALLENGE frame) |
| quic_received_frames {quic_frame_type="path_response"} | counter | Number of QUIC frames received (PATH_RESPONSE frame) |
| quic_received_frames {quic_frame_type="conn_close_quic"} | counter | Number of QUIC frames received (CONN_CLOSE(transport) frame) |
| quic_received_frames {quic_frame_type="conn_close_app"} | counter | Number of QUIC frames received (CONN_CLOSE(app) frame) |
| quic_received_frames {quic_frame_type="handshake_done"} | counter | Number of QUIC frames received (HANDSHAKE_DONE frame) |
| quic_received_frames {quic_frame_type="ping"} | counter | Number of QUIC frames received (PING frame) |
| quic_received_frames {quic_frame_type="padding"} | counter | Number of QUIC frames received (PADDING frame) |
| quic_ack_tx {quic_ack_tx="noop"} | counter | ACK events (Non-ACK-eliciting packet) |
| quic_ack_tx {quic_ack_tx="new"} | counter | ACK events (New ACK range) |
| quic_ack_tx {quic_ack_tx="merged"} | counter | ACK events (Merged into existing ACK range) |
| quic_ack_tx {quic_ack_tx="drop"} | counter | ACK events (Out of buffers) |
| quic_ack_tx {quic_ack_tx="cancel"} | counter | ACK events (ACK suppressed by handler) |
| quic_service_duration_seconds | histogram | Duration spent in service |
| quic_receive_duration_seconds | histogram | Duration spent processing packets |
| quic_frame_fail_parse | counter | Number of QUIC frames failed to parse |
| quic_pkt_crypto_failed {quic_enc_level="initial"} | counter | Number of packets that failed decryption (Initial) |
| quic_pkt_crypto_failed {quic_enc_level="early"} | counter | Number of packets that failed decryption (Early data) |
| quic_pkt_crypto_failed {quic_enc_level="handshake"} | counter | Number of packets that failed decryption (Handshake) |
| quic_pkt_crypto_failed {quic_enc_level="app"} | counter | Number of packets that failed decryption (App data) |
| quic_pkt_no_key {quic_enc_level="initial"} | counter | Number of packets that failed decryption due to missing key (Initial) |
| quic_pkt_no_key {quic_enc_level="early"} | counter | Number of packets that failed decryption due to missing key (Early data) |
| quic_pkt_no_key {quic_enc_level="handshake"} | counter | Number of packets that failed decryption due to missing key (Handshake) |
| quic_pkt_no_key {quic_enc_level="app"} | counter | Number of packets that failed decryption due to missing key (App data) |
| quic_pkt_net_header_invalid | counter | Number of packets dropped due to weird IP or UDP header |
| quic_pkt_quic_header_invalid | counter | Number of packets dropped due to weird QUIC header |
| quic_pkt_undersz | counter | Number of QUIC packets dropped due to being too small |
| quic_pkt_oversz | counter | Number of QUIC packets dropped due to being too large |
| quic_pkt_verneg | counter | Number of QUIC version negotiation packets received |
| quic_retry_sent | counter | Number of QUIC Retry packets sent |
| quic_pkt_retransmissions {quic_enc_level="initial"} | counter | Number of QUIC packets that retransmitted (Initial) |
| quic_pkt_retransmissions {quic_enc_level="early"} | counter | Number of QUIC packets that retransmitted (Early data) |
| quic_pkt_retransmissions {quic_enc_level="handshake"} | counter | Number of QUIC packets that retransmitted (Handshake) |
| quic_pkt_retransmissions {quic_enc_level="app"} | counter | Number of QUIC packets that retransmitted (App data) |
Bundle Tile
| Metric | Type | Description |
|---|---|---|
| bundle_transaction_received | counter | Total count of transactions received, including transactions within bundles |
| bundle_packet_received | counter | Total count of packets received |
| bundle_proto_received_bytes | counter | Total count of bytes from received grpc protobuf payloads |
| bundle_bundle_received | counter | Total count of bundles received |
| bundle_errors {bundle_error="protobuf"} | counter | Number of gRPC errors encountered (Protobuf decode/encode error) |
| bundle_errors {bundle_error="transport"} | counter | Number of gRPC errors encountered (Transport error) |
| bundle_errors {bundle_error="timeout"} | counter | Number of gRPC errors encountered (I/O timeout) |
| bundle_errors {bundle_error="no_fee_info"} | counter | Number of gRPC errors encountered (Bundle dropped due to missing fee info) |
| bundle_errors {bundle_error="ssl_alloc"} | counter | Number of gRPC errors encountered (OpenSSL alloc fail) |
| bundle_heap_size | gauge | Workspace heap size |
| bundle_heap_free_bytes | gauge | Approx free space in workspace |
| bundle_shredstream_heartbeats | counter | Number of ShredStream heartbeats successfully sent |
| bundle_keepalives | counter | Number of HTTP/2 PINGs acknowledged by server |
| bundle_connected | gauge | 1 if connected to the bundle server, 0 if not |
| bundle_rtt_sample | gauge | Latest RTT sample at scrape time (nanoseconds) |
| bundle_rtt_smoothed | gauge | RTT moving average (nanoseconds) |
| bundle_rtt_var | gauge | RTT variance (nanoseconds) |
| bundle_message_rx_delay_nanos | histogram | Message receive delay in nanoseconds from bundle server to bundle client |
| bundle_pending_transactions | gauge | Number of transactions buffered and waiting to be published |
| bundle_transaction_dropped_backpressure | counter | Transactions dropped because the pending transaction buffer was full (should never happen) |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_transaction_result {verify_tile_result="success"} | counter | Count of transaction results through verify tile (Transaction verified successfully) |
| verify_transaction_result {verify_tile_result="bundle_peer_failure"} | counter | Count of transaction results through verify tile (Peer transaction in the bundle failed) |
| verify_transaction_result {verify_tile_result="parse_failure"} | counter | Count of transaction results through verify tile (Transaction failed to parse) |
| verify_transaction_result {verify_tile_result="dedup_failure"} | counter | Count of transaction results through verify tile (Transaction failed deduplication) |
| verify_transaction_result {verify_tile_result="verify_failure"} | counter | Count of transaction results through verify tile (Transaction failed signature verification) |
| verify_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_transaction_result {dedup_tile_result="success"} | counter | Count of transaction results through dedup tile (Transaction deduplicated successfully) |
| dedup_transaction_result {dedup_tile_result="bundle_peer_failure"} | counter | Count of transaction results through dedup tile (Peer transaction in the bundle failed) |
| dedup_transaction_result {dedup_tile_result="dedup_failure"} | counter | Count of transaction results through dedup tile (Transaction failed deduplication) |
| dedup_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Resolv Tile
| Metric | Type | Description |
|---|---|---|
| resolv_no_bank_drop | counter | Count of transactions dropped because the bank was not available |
| resolv_stash_operation {resolve_stash_operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolv_stash_operation {resolve_stash_operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolv_stash_operation {resolve_stash_operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolv_stash_operation {resolve_stash_operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories) |
| resolv_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolv_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolv_lut_resolved {lut_resolve_result="account_not_found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolv_lut_resolved {lut_resolve_result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) |
| resolv_blockhash_expired | counter | Count of transactions that failed to resolve because the blockhash was expired |
| resolv_transaction_bundle_peer_failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_schedule_microblock_duration_seconds | histogram | Duration of scheduling one microblock |
| pack_no_sched_microblock_duration_seconds | histogram | Duration of discovering that there are no schedulable transactions |
| pack_insert_transaction_duration_seconds | histogram | Duration of inserting one transaction into the pool of available transactions |
| pack_complete_microblock_duration_seconds | histogram | Duration of the computation associated with marking one microblock as complete |
| pack_total_transactions_per_microblock_count | histogram | Count of transactions in a scheduled microblock, including both votes and non-votes |
| pack_votes_per_microblock_count | histogram | Count of simple vote transactions in a scheduled microblock |
| pack_normal_transaction_received | counter | Count of transactions received via the normal TPU path |
| pack_transaction_inserted {pack_txn_insert_return="instr_acct_cnt"} | counter | Result of inserting a transaction into the pack object (Transaction has an instruction that references too many accounts) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_conflict"} | counter | Result of inserting a transaction into the pack object (Bundle with two conflicting durable nonce transactions) |
| pack_transaction_inserted {pack_txn_insert_return="bundle_blacklist"} | counter | Result of inserting a transaction into the pack object (Transaction uses an account on the bundle blacklist) |
| pack_transaction_inserted {pack_txn_insert_return="invalid_nonce"} | counter | Result of inserting a transaction into the pack object (Transaction is an invalid durable nonce transaction) |
| pack_transaction_inserted {pack_txn_insert_return="write_sysvar"} | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) |
| pack_transaction_inserted {pack_txn_insert_return="estimation_fail"} | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) |
| pack_transaction_inserted {pack_txn_insert_return="duplicate_account"} | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) |
| pack_transaction_inserted {pack_txn_insert_return="too_many_accounts"} | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) |
| pack_transaction_inserted {pack_txn_insert_return="too_large"} | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) |
| pack_transaction_inserted {pack_txn_insert_return="expired"} | counter | Result of inserting a transaction into the pack object (Transaction already expired) |
| pack_transaction_inserted {pack_txn_insert_return="addr_lut"} | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) |
| pack_transaction_inserted {pack_txn_insert_return="unaffordable"} | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) |
| pack_transaction_inserted {pack_txn_insert_return="duplicate"} | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce) |
| pack_transaction_inserted {pack_txn_insert_return="priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_transaction_inserted {pack_txn_insert_return="nonvote_add"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="vote_add"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) |
| pack_transaction_inserted {pack_txn_insert_return="vote_replace"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_nonvote_add"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="unused"} | counter | Result of inserting a transaction into the pack object (Unused because durable nonce transactions can't be simple votes) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce) |
| pack_metric_timing {pack_timing_state="no_txn_no_execle_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and wasn't leader) |
| pack_metric_timing {pack_timing_state="txn_no_execle_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, but wasn't leader or had hit a limit) |
| pack_metric_timing {pack_timing_state="no_txn_execle_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, had execle but wasn't leader) |
| pack_metric_timing {pack_timing_state="txn_execle_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, had execle but wasn't leader) |
| pack_metric_timing {pack_timing_state="no_txn_no_execle_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and was leader but had no available execle) |
| pack_metric_timing {pack_timing_state="txn_no_execle_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, was leader, but had no available execle) |
| pack_metric_timing {pack_timing_state="no_txn_execle_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had available execle but no transactions) |
| pack_metric_timing {pack_timing_state="txn_execle_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had execle and transactions available but couldn't schedule anything non-conflicting) |
| pack_metric_timing {pack_timing_state="no_txn_no_execle_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="txn_no_execle_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="no_txn_execle_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="txn_execle_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="no_txn_no_execle_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all execle were busy) |
| pack_metric_timing {pack_timing_state="txn_no_execle_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all execle were busy) |
| pack_metric_timing {pack_timing_state="no_txn_execle_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock and now has no transactions) |
| pack_metric_timing {pack_timing_state="txn_execle_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock) |
| pack_transaction_dropped_from_extra | counter | Transactions dropped from the extra transaction storage because it was full |
| pack_transaction_inserted_to_extra | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full |
| pack_transaction_inserted_from_extra | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage |
| pack_transaction_expired | counter | Transactions deleted from pack because their TTL expired |
| pack_transaction_deleted | counter | Transactions dropped from pack because they were requested to be deleted |
| pack_transaction_already_executed | counter | Transactions dropped from pack because they were already executed (in either the replay or leader pipeline) |
| pack_transaction_dropped_partial_bundle | counter | Transactions dropped from pack because they were part of a partial bundle |
| pack_available_transactions {avail_txn_type="all"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (All transactions in any treap) |
| pack_available_transactions {avail_txn_type="regular"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes in the main treap) |
| pack_available_transactions {avail_txn_type="votes"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Simple votes) |
| pack_available_transactions {avail_txn_type="conflicting"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes that write to a hotly-contended account) |
| pack_available_transactions {avail_txn_type="bundles"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Transactions that are part of a bundle) |
| pack_pending_transactions_heap_size | gauge | The maximum number of pending transactions that pack can consider. This value is fixed at Firedancer startup but is a useful reference for AvailableTransactions |
| pack_smallest_pending_transaction | gauge | A lower bound on the smallest non-vote transaction (in cost units) that is immediately available for scheduling |
| pack_microblock_per_block_limit | counter | The number of times pack did not pack a microblock because the limit on microblocks/block had been reached |
| pack_data_per_block_limit | counter | The number of times pack did not pack a microblock because it reached the data per block limit at the start of trying to schedule a microblock |
| pack_transaction_schedule {pack_txn_schedule="taken"} | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) |
| pack_transaction_schedule {pack_txn_schedule="cu_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_transaction_schedule {pack_txn_schedule="fast_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) |
| pack_transaction_schedule {pack_txn_schedule="byte_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_transaction_schedule {pack_txn_schedule="alloc_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the data allocation limit) |
| pack_transaction_schedule {pack_txn_schedule="write_cost"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_transaction_schedule {pack_txn_schedule="slow_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) |
| pack_transaction_schedule {pack_txn_schedule="defer_skip"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction it previously exceeded the per-account block write cost limit too many times) |
| pack_bundle_crank_status {bundle_crank_result="not_needed"} | counter | Result of considering whether bundle cranks are needed (On-chain state in the correct state) |
| pack_bundle_crank_status {bundle_crank_result="inserted"} | counter | Result of considering whether bundle cranks are needed (Inserted an initializer bundle to update the on-chain state) |
| pack_bundle_crank_status {bundle_crank_result="creation_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but creation failed) |
| pack_bundle_crank_status {bundle_crank_result="insertion_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but insertion failed) |
| pack_cus_consumed_in_block | gauge | The number of cost units consumed in the current block, or 0 if pack is not currently packing a block |
| pack_cus_scheduled | histogram | The number of cost units scheduled for each block pack produced. This can be higher than the block limit because of returned CUs. |
| pack_cus_rebated | histogram | The number of compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses. |
| pack_cus_net | histogram | The net number of cost units (scheduled - rebated) in each block pack produced. |
| pack_cus_pct | histogram | The percent of the total block cost limit used for each block pack produced. |
| pack_delete_missed | counter | Count of attempts to delete a transaction that wasn't found |
| pack_delete_hit | counter | Count of attempts to delete a transaction that was found and deleted |
Execle Tile
| Metric | Type | Description |
|---|---|---|
| execle_transaction_result {transaction_result="success"} | counter | Result of loading and executing a transaction (Transaction executed successfully) |
| execle_transaction_result {transaction_result="instructon_error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) |
| execle_transaction_result {transaction_result="account_not_found"} | counter | Result of loading and executing a transaction (The transaction fee payer address was not found) |
| execle_transaction_result {transaction_result="program_account_not_found"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was not found) |
| execle_transaction_result {transaction_result="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction (The transaction fee payer did not have balance to pay the fee) |
| execle_transaction_result {transaction_result="invalid_account_for_fee"} | counter | Result of loading and executing a transaction (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) |
| execle_transaction_result {transaction_result="already_processed"} | counter | Result of loading and executing a transaction (The transaction has already been processed in a recent block) |
| execle_transaction_result {transaction_result="blockhash_not_found"} | counter | Result of loading and executing a transaction (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) |
| execle_transaction_result {transaction_result="invalid_program_for_execution"} | counter | Result of loading and executing a transaction (A program account referenced by the transaction was no executable. TODO: No longer needed with SIMD-0162) |
| execle_transaction_result {transaction_result="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that does not exist or is inactive) |
| execle_transaction_result {transaction_result="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that is not owned by the ALUT program account) |
| execle_transaction_result {transaction_result="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction (The transaction references an ALUT account that contains data which is not a valid ALUT) |
| execle_transaction_result {transaction_result="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction (The transaction references an account offset from the ALUT which does not exist) |
| execle_transaction_result {transaction_result="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction (The total account data size of the loaded accounts exceeds the consensus limit) |
| execle_transaction_result {transaction_result="duplicate_instruction"} | counter | Result of loading and executing a transaction (A compute budget program instruction was invoked more than once) |
| execle_transaction_result {transaction_result="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction (The compute budget program was invoked and set the loaded accounts data size to zero) |
| execle_transaction_result {transaction_result="nonce_already_advanced"} | counter | Result of loading and executing a transaction (The transaction references a nonce account that is already advanced) |
| execle_transaction_result {transaction_result="nonce_advance_failed"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the advance instruction was not valid or failed) |
| execle_transaction_result {transaction_result="nonce_wrong_blockhash"} | counter | Result of loading and executing a transaction (The transaction is a nonce transaction but the blockhash is not the correct one) |
| execle_transaction_result {transaction_result="account_in_use"} | counter | Result of loading and executing a transaction (The transaction conflicts with another transaction in the microblock. TODO: No longer possible with smart dispatcher) |
| execle_transaction_result {transaction_result="account_loaded_twice"} | counter | Result of loading and executing a transaction (The transaction references the same account twice) |
| execle_transaction_result {transaction_result="signature_failure"} | counter | Result of loading and executing a transaction (The transaction had an invalid signature) |
| execle_transaction_result {transaction_result="too_many_account_locks"} | counter | Result of loading and executing a transaction (The transaction references too many accounts. TODO: No longer possible with smart dispatcher) |
| execle_transaction_result {transaction_result="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction (The transaction would leave an account with a lower balance than the rent-exempt minimum) |
| execle_transaction_result {transaction_result="unbalanced_transaction"} | counter | Result of loading and executing a transaction (The total referenced account lamports before and after the transaction was unbalanced) |
| execle_transaction_result {transaction_result="bundle_peer"} | counter | Result of loading and executing a transaction (The transaction was part of a bundle and an earlier transaction in the bundle failed) |
| execle_transaction_landed {transaction_landed="landed_success"} | counter | Whether a transaction landed in the block or not (Transaction landed) |
| execle_transaction_landed {transaction_landed="landed_fees_only"} | counter | Whether a transaction landed in the block or not (Transaction landed, but was fees only and did not execute) |
| execle_transaction_landed {transaction_landed="landed_failed"} | counter | Whether a transaction landed in the block or not (Transaction landed, but failed to execute) |
| execle_transaction_landed {transaction_landed="unlanded"} | counter | Whether a transaction landed in the block or not (Transaction did not land) |
| execle_compute_units_total | counter | Estimated number of compute units executed since tile start |
Poh Tile
| Metric | Type | Description |
|---|
Sign Tile
| Metric | Type | Description |
|---|---|---|
| sign_sign_duration_seconds | histogram | Duration of signing a message |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_cluster_contact_info_cnt | histogram | Number of contact infos in the cluster contact info message |
| shred_microblocks_abandoned | counter | The number of microblocks that were abandoned because we switched slots without finishing the current slot |
| shred_invalid_block_id | counter | The number of times a block was created with unknown parent block_id |
| shred_batch_sz | histogram | The size (in bytes) of each microblock batch that is shredded |
| shred_batch_microblock_cnt | histogram | The number of microblocks in each microblock batch that is shredded |
| shred_shredding_duration_seconds | histogram | Duration of producing one FEC set from the shredder |
| shred_add_shred_duration_seconds | histogram | Duration of verifying and processing one shred received from the network |
| shred_shred_processed {shred_processing_result="bad_slot"} | counter | The result of processing a shred from the network (Shred was for a slot for which we don't know the leader) |
| shred_shred_processed {shred_processing_result="parse_failed"} | counter | The result of processing a shred from the network (Shred parsing failed) |
| shred_shred_processed {shred_processing_result="equivoc"} | counter | The result of processing a shred from the network (Shred was equivocated with another shred) |
| shred_shred_processed {shred_processing_result="rejected"} | counter | The result of processing a shred from the network (Shred was invalid for one of many reasons) |
| shred_shred_processed {shred_processing_result="ignored"} | counter | The result of processing a shred from the network (Shred was ignored because we had already reconstructed the FEC set, or was for an old slot) |
| shred_shred_processed {shred_processing_result="duplicate"} | counter | The result of processing a shred from the network (Shred was a duplicate of a previously received valid shred of an incomplete FEC set) |
| shred_shred_processed {shred_processing_result="okay"} | counter | The result of processing a shred from the network (Shred accepted to an incomplete FEC set) |
| shred_shred_processed {shred_processing_result="completes"} | counter | The result of processing a shred from the network (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_set_spilled | counter | The number of FEC sets that were spilled because they didn't complete in time and we needed space |
| shred_shred_rejected_initial | counter | The number of shreds that were rejected before any resources were allocated for the FEC set |
| shred_shred_rejected_unchained | counter | The number of shreds that were rejected because they're not chained merkle shreds |
| shred_fec_rejected_fatal | counter | The number of FEC sets that were rejected for reasons that cause the whole FEC set to become invalid |
| shred_bad_nonce | counter | The number of shreds we received that seemed to be repair responses but had an incorrect nonce |
| shred_shred_repair_rcv | counter | The number of times we received a repair shred |
| shred_shred_repair_rcv_bytes | counter | The number bytes received from network packets with repair shreds. Bytes include network headers |
| shred_shred_turbine_rcv | counter | The number of times we received a turbine shred |
| shred_shred_turbine_rcv_bytes | counter | The number bytes received from network packets with turbine shreds. Bytes include network headers |
Gossvf Tile
| Metric | Type | Description |
|---|---|---|
| gossvf_message_rx_count {gossvf_message_outcome="success_pull_request"} | counter | Number of gossip messages processed (Pull Request (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_pull_response"} | counter | Number of gossip messages processed (Pull Response (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_push"} | counter | Number of gossip messages processed (Push (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_prune"} | counter | Number of gossip messages processed (Prune (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_ping"} | counter | Number of gossip messages processed (Ping (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_pong"} | counter | Number of gossip messages processed (Pong (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_unparseable"} | counter | Number of gossip messages processed (Unparseable) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_not_contact_info"} | counter | Number of gossip messages processed (Pull Request (not contact info)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_loopback"} | counter | Number of gossip messages processed (Pull Request (loopback)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_inactive"} | counter | Number of gossip messages processed (Pull Request (inactive)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_wallclock"} | counter | Number of gossip messages processed (Pull Request (wallclock)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_signature"} | counter | Number of gossip messages processed (Pull Request (signature)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_shred_version"} | counter | Number of gossip messages processed (Pull Request (shred version)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_mask_bits"} | counter | Number of gossip messages processed (Pull Request (mask bits)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_prune_destination"} | counter | Number of gossip messages processed (Prune (destination)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_prune_wallclock"} | counter | Number of gossip messages processed (Prune (wallclock)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_prune_signature"} | counter | Number of gossip messages processed (Prune (signature)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_push_no_valid_crds"} | counter | Number of gossip messages processed (Push (no valid crds)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_response_no_valid_crds"} | counter | Number of gossip messages processed (Pull Response (no valid crds)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_ping_signature"} | counter | Number of gossip messages processed (Ping (signature)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pong_signature"} | counter | Number of gossip messages processed (Pong (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pull_request"} | counter | Total wire bytes of gossip messages processed (Pull Request (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pull_response"} | counter | Total wire bytes of gossip messages processed (Pull Response (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_push"} | counter | Total wire bytes of gossip messages processed (Push (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_prune"} | counter | Total wire bytes of gossip messages processed (Prune (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_ping"} | counter | Total wire bytes of gossip messages processed (Ping (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pong"} | counter | Total wire bytes of gossip messages processed (Pong (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_unparseable"} | counter | Total wire bytes of gossip messages processed (Unparseable) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_not_contact_info"} | counter | Total wire bytes of gossip messages processed (Pull Request (not contact info)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_loopback"} | counter | Total wire bytes of gossip messages processed (Pull Request (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_inactive"} | counter | Total wire bytes of gossip messages processed (Pull Request (inactive)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_wallclock"} | counter | Total wire bytes of gossip messages processed (Pull Request (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_signature"} | counter | Total wire bytes of gossip messages processed (Pull Request (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_shred_version"} | counter | Total wire bytes of gossip messages processed (Pull Request (shred version)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_mask_bits"} | counter | Total wire bytes of gossip messages processed (Pull Request (mask bits)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_destination"} | counter | Total wire bytes of gossip messages processed (Prune (destination)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_wallclock"} | counter | Total wire bytes of gossip messages processed (Prune (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_signature"} | counter | Total wire bytes of gossip messages processed (Prune (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_push_no_valid_crds"} | counter | Total wire bytes of gossip messages processed (Push (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_response_no_valid_crds"} | counter | Total wire bytes of gossip messages processed (Pull Response (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_ping_signature"} | counter | Total wire bytes of gossip messages processed (Ping (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pong_signature"} | counter | Total wire bytes of gossip messages processed (Pong (signature)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="success_pull_response"} | counter | Number of CRDS values processed (Pull Response (success)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="success_push"} | counter | Number of CRDS values processed (Push (success)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_duplicate"} | counter | Number of CRDS values processed (Pull Response (duplicate)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_signature"} | counter | Number of CRDS values processed (Pull Response (signature)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_origin_no_contact_info"} | counter | Number of CRDS values processed (Pull Response (origin no contact info)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_origin_shred_version"} | counter | Number of CRDS values processed (Pull Response (origin shred version)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_inactive"} | counter | Number of CRDS values processed (Pull Response (inactive)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_wallclock"} | counter | Number of CRDS values processed (Pull Response (wallclock)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_signature"} | counter | Number of CRDS values processed (Push (signature)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_origin_no_contact_info"} | counter | Number of CRDS values processed (Push (origin no contact info)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_origin_shred_version"} | counter | Number of CRDS values processed (Push (origin shred version)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_inactive"} | counter | Number of CRDS values processed (Push (inactive)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_wallclock"} | counter | Number of CRDS values processed (Push (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="success_pull_response"} | counter | Total wire bytes of CRDS values processed (Pull Response (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="success_push"} | counter | Total wire bytes of CRDS values processed (Push (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_duplicate"} | counter | Total wire bytes of CRDS values processed (Pull Response (duplicate)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_signature"} | counter | Total wire bytes of CRDS values processed (Pull Response (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_origin_no_contact_info"} | counter | Total wire bytes of CRDS values processed (Pull Response (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_origin_shred_version"} | counter | Total wire bytes of CRDS values processed (Pull Response (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_inactive"} | counter | Total wire bytes of CRDS values processed (Pull Response (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_wallclock"} | counter | Total wire bytes of CRDS values processed (Pull Response (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_signature"} | counter | Total wire bytes of CRDS values processed (Push (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_origin_no_contact_info"} | counter | Total wire bytes of CRDS values processed (Push (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_origin_shred_version"} | counter | Total wire bytes of CRDS values processed (Push (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_inactive"} | counter | Total wire bytes of CRDS values processed (Push (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_wallclock"} | counter | Total wire bytes of CRDS values processed (Push (wallclock)) |
Gossip Tile
| Metric | Type | Description |
|---|---|---|
| gossip_ping_tracker_capacity | counter | The peer tracking capacity of the ping tracker |
| gossip_ping_tracker_count {ping_tracked_type="unpinged"} | gauge | The number of peers being tracked for ping/pong (Peer is known but has not been pinged yet) |
| gossip_ping_tracker_count {ping_tracked_type="invalid"} | gauge | The number of peers being tracked for ping/pong (Peer was pinged but has not yet responded, or responded with an invalid pong) |
| gossip_ping_tracker_count {ping_tracked_type="valid"} | gauge | The number of peers being tracked for ping/pong (Peer was pinged and is currently valid) |
| gossip_ping_tracker_count {ping_tracked_type="valid_refreshing"} | gauge | The number of peers being tracked for ping/pong (Peer was pinged and is currently valid, and is being refreshed with a new ping) |
| gossip_ping_tracker_pong_result {pong_response="staked"} | counter | The result of processing a pong response to a ping (Pong ignored as node is staked) |
| gossip_ping_tracker_pong_result {pong_response="entrypoint"} | counter | The result of processing a pong response to a ping (Pong ignored as node is an entrypoint) |
| gossip_ping_tracker_pong_result {pong_response="untracked"} | counter | The result of processing a pong response to a ping (Pong ignored as node has no outstanding ping) |
| gossip_ping_tracker_pong_result {pong_response="address"} | counter | The result of processing a pong response to a ping (Pong ignored as responding node doesn't match pinged address) |
| gossip_ping_tracker_pong_result {pong_response="token"} | counter | The result of processing a pong response to a ping (Pong ignored as token was not valid) |
| gossip_ping_tracker_pong_result {pong_response="success"} | counter | The result of processing a pong response to a ping (Pong accepted) |
| gossip_ping_tracker_evicted_count | counter | The number of peers evicted from the ping tracker to make space for new peers |
| gossip_ping_tracked_count | counter | The total number of peers ever tracked for ping/pong |
| gossip_ping_tracker_stake_changed_count | counter | The number of times a tracked peer was removed from tracking as it became staked |
| gossip_ping_tracker_address_changed_count | counter | The number of times a tracked peer was removed from tracking as it's gossip address changed |
| gossip_crds_capacity | gauge | The capacity of the data store |
| gossip_crds_count {crds_value="contact_info_v1"} | gauge | The number of entries in the data store (Contact Info V1) |
| gossip_crds_count {crds_value="vote"} | gauge | The number of entries in the data store (Vote) |
| gossip_crds_count {crds_value="lowest_slot"} | gauge | The number of entries in the data store (Lowest Slot) |
| gossip_crds_count {crds_value="snapshot_hashes"} | gauge | The number of entries in the data store (Snapshot Hashes) |
| gossip_crds_count {crds_value="accounts_hashes"} | gauge | The number of entries in the data store (Accounts Hashes) |
| gossip_crds_count {crds_value="epoch_slots"} | gauge | The number of entries in the data store (Epoch Slots) |
| gossip_crds_count {crds_value="version_v1"} | gauge | The number of entries in the data store (Version V1) |
| gossip_crds_count {crds_value="version_v2"} | gauge | The number of entries in the data store (Version V2) |
| gossip_crds_count {crds_value="node_instance"} | gauge | The number of entries in the data store (Node Instance) |
| gossip_crds_count {crds_value="duplicate_shred"} | gauge | The number of entries in the data store (Duplicate Shred) |
| gossip_crds_count {crds_value="incremental_snapshot_hashes"} | gauge | The number of entries in the data store (Incremental Snapshot Hashes) |
| gossip_crds_count {crds_value="contact_info_v2"} | gauge | The number of entries in the data store (Contact Info V2) |
| gossip_crds_count {crds_value="restart_last_voted_fork_slots"} | gauge | The number of entries in the data store (Restart Last Voted Fork Slots) |
| gossip_crds_count {crds_value="restart_heaviest_fork"} | gauge | The number of entries in the data store (Restart Heaviest Fork) |
| gossip_crds_expired_count | counter | The number of entries expired from the data store due to age |
| gossip_crds_evicted_count | counter | The number of entries evicted from the data store to make space for new entries |
| gossip_crds_peer_capacity | gauge | The capacity for storing peers in the data store |
| gossip_crds_peer_staked_count | gauge | The number of staked peers in the data store |
| gossip_crds_peer_unstaked_count | gauge | The number of unstaked peers in the data store |
| gossip_crds_peer_total_stake | gauge | The total visible stake in the data store, in lamports |
| gossip_crds_peer_evicted_count | counter | The number of peers evicted from the data store to make space for new peers |
| gossip_crds_purged_capacity | gauge | The capacity of the list of purged data store entries |
| gossip_crds_purged_count | gauge | The number of purged data store entries |
| gossip_crds_purged_expired_count | counter | The number of purged entries expired from the purged list due to age |
| gossip_crds_purged_evicted_count | counter | The number of purged entries evicted from the data store to make space for new entries |
| gossip_crds_rx_count {gossip_crds_outcome="upserted_pull_response"} | counter | Outcome of incoming CRDS messages (Pull Response (upserted)) |
| gossip_crds_rx_count {gossip_crds_outcome="upserted_push"} | counter | Outcome of incoming CRDS messages (Push (upserted)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_pull_response_stale"} | counter | Outcome of incoming CRDS messages (Pull Response (newer entry already present in table)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_pull_response_duplicate"} | counter | Outcome of incoming CRDS messages (Pull Response (duplicate)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_push_stale"} | counter | Outcome of incoming CRDS messages (Push (newer entry already present in table)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_push_duplicate"} | counter | Outcome of incoming CRDS messages (Push (duplicate)) |
| gossip_message_tx_count {gossip_message="pull_request"} | counter | Number of gossip messages sent (Pull Request) |
| gossip_message_tx_count {gossip_message="pull_response"} | counter | Number of gossip messages sent (Pull Response) |
| gossip_message_tx_count {gossip_message="push"} | counter | Number of gossip messages sent (Push) |
| gossip_message_tx_count {gossip_message="prune"} | counter | Number of gossip messages sent (Prune) |
| gossip_message_tx_count {gossip_message="ping"} | counter | Number of gossip messages sent (Ping) |
| gossip_message_tx_count {gossip_message="pong"} | counter | Number of gossip messages sent (Pong) |
| gossip_message_tx_bytes {gossip_message="pull_request"} | counter | Total wire bytes sent in gossip messages (Pull Request) |
| gossip_message_tx_bytes {gossip_message="pull_response"} | counter | Total wire bytes sent in gossip messages (Pull Response) |
| gossip_message_tx_bytes {gossip_message="push"} | counter | Total wire bytes sent in gossip messages (Push) |
| gossip_message_tx_bytes {gossip_message="prune"} | counter | Total wire bytes sent in gossip messages (Prune) |
| gossip_message_tx_bytes {gossip_message="ping"} | counter | Total wire bytes sent in gossip messages (Ping) |
| gossip_message_tx_bytes {gossip_message="pong"} | counter | Total wire bytes sent in gossip messages (Pong) |
| gossip_crds_tx_push_count {crds_value="contact_info_v1"} | counter | Number of CRDS values sent in push messages (Contact Info V1) |
| gossip_crds_tx_push_count {crds_value="vote"} | counter | Number of CRDS values sent in push messages (Vote) |
| gossip_crds_tx_push_count {crds_value="lowest_slot"} | counter | Number of CRDS values sent in push messages (Lowest Slot) |
| gossip_crds_tx_push_count {crds_value="snapshot_hashes"} | counter | Number of CRDS values sent in push messages (Snapshot Hashes) |
| gossip_crds_tx_push_count {crds_value="accounts_hashes"} | counter | Number of CRDS values sent in push messages (Accounts Hashes) |
| gossip_crds_tx_push_count {crds_value="epoch_slots"} | counter | Number of CRDS values sent in push messages (Epoch Slots) |
| gossip_crds_tx_push_count {crds_value="version_v1"} | counter | Number of CRDS values sent in push messages (Version V1) |
| gossip_crds_tx_push_count {crds_value="version_v2"} | counter | Number of CRDS values sent in push messages (Version V2) |
| gossip_crds_tx_push_count {crds_value="node_instance"} | counter | Number of CRDS values sent in push messages (Node Instance) |
| gossip_crds_tx_push_count {crds_value="duplicate_shred"} | counter | Number of CRDS values sent in push messages (Duplicate Shred) |
| gossip_crds_tx_push_count {crds_value="incremental_snapshot_hashes"} | counter | Number of CRDS values sent in push messages (Incremental Snapshot Hashes) |
| gossip_crds_tx_push_count {crds_value="contact_info_v2"} | counter | Number of CRDS values sent in push messages (Contact Info V2) |
| gossip_crds_tx_push_count {crds_value="restart_last_voted_fork_slots"} | counter | Number of CRDS values sent in push messages (Restart Last Voted Fork Slots) |
| gossip_crds_tx_push_count {crds_value="restart_heaviest_fork"} | counter | Number of CRDS values sent in push messages (Restart Heaviest Fork) |
| gossip_crds_tx_push_bytes {crds_value="contact_info_v1"} | counter | Total wire bytes of CRDS sent out in push messages (Contact Info V1) |
| gossip_crds_tx_push_bytes {crds_value="vote"} | counter | Total wire bytes of CRDS sent out in push messages (Vote) |
| gossip_crds_tx_push_bytes {crds_value="lowest_slot"} | counter | Total wire bytes of CRDS sent out in push messages (Lowest Slot) |
| gossip_crds_tx_push_bytes {crds_value="snapshot_hashes"} | counter | Total wire bytes of CRDS sent out in push messages (Snapshot Hashes) |
| gossip_crds_tx_push_bytes {crds_value="accounts_hashes"} | counter | Total wire bytes of CRDS sent out in push messages (Accounts Hashes) |
| gossip_crds_tx_push_bytes {crds_value="epoch_slots"} | counter | Total wire bytes of CRDS sent out in push messages (Epoch Slots) |
| gossip_crds_tx_push_bytes {crds_value="version_v1"} | counter | Total wire bytes of CRDS sent out in push messages (Version V1) |
| gossip_crds_tx_push_bytes {crds_value="version_v2"} | counter | Total wire bytes of CRDS sent out in push messages (Version V2) |
| gossip_crds_tx_push_bytes {crds_value="node_instance"} | counter | Total wire bytes of CRDS sent out in push messages (Node Instance) |
| gossip_crds_tx_push_bytes {crds_value="duplicate_shred"} | counter | Total wire bytes of CRDS sent out in push messages (Duplicate Shred) |
| gossip_crds_tx_push_bytes {crds_value="incremental_snapshot_hashes"} | counter | Total wire bytes of CRDS sent out in push messages (Incremental Snapshot Hashes) |
| gossip_crds_tx_push_bytes {crds_value="contact_info_v2"} | counter | Total wire bytes of CRDS sent out in push messages (Contact Info V2) |
| gossip_crds_tx_push_bytes {crds_value="restart_last_voted_fork_slots"} | counter | Total wire bytes of CRDS sent out in push messages (Restart Last Voted Fork Slots) |
| gossip_crds_tx_push_bytes {crds_value="restart_heaviest_fork"} | counter | Total wire bytes of CRDS sent out in push messages (Restart Heaviest Fork) |
| gossip_crds_tx_pull_response_count {crds_value="contact_info_v1"} | counter | Number of CRDS values sent in pull response messages (Contact Info V1) |
| gossip_crds_tx_pull_response_count {crds_value="vote"} | counter | Number of CRDS values sent in pull response messages (Vote) |
| gossip_crds_tx_pull_response_count {crds_value="lowest_slot"} | counter | Number of CRDS values sent in pull response messages (Lowest Slot) |
| gossip_crds_tx_pull_response_count {crds_value="snapshot_hashes"} | counter | Number of CRDS values sent in pull response messages (Snapshot Hashes) |
| gossip_crds_tx_pull_response_count {crds_value="accounts_hashes"} | counter | Number of CRDS values sent in pull response messages (Accounts Hashes) |
| gossip_crds_tx_pull_response_count {crds_value="epoch_slots"} | counter | Number of CRDS values sent in pull response messages (Epoch Slots) |
| gossip_crds_tx_pull_response_count {crds_value="version_v1"} | counter | Number of CRDS values sent in pull response messages (Version V1) |
| gossip_crds_tx_pull_response_count {crds_value="version_v2"} | counter | Number of CRDS values sent in pull response messages (Version V2) |
| gossip_crds_tx_pull_response_count {crds_value="node_instance"} | counter | Number of CRDS values sent in pull response messages (Node Instance) |
| gossip_crds_tx_pull_response_count {crds_value="duplicate_shred"} | counter | Number of CRDS values sent in pull response messages (Duplicate Shred) |
| gossip_crds_tx_pull_response_count {crds_value="incremental_snapshot_hashes"} | counter | Number of CRDS values sent in pull response messages (Incremental Snapshot Hashes) |
| gossip_crds_tx_pull_response_count {crds_value="contact_info_v2"} | counter | Number of CRDS values sent in pull response messages (Contact Info V2) |
| gossip_crds_tx_pull_response_count {crds_value="restart_last_voted_fork_slots"} | counter | Number of CRDS values sent in pull response messages (Restart Last Voted Fork Slots) |
| gossip_crds_tx_pull_response_count {crds_value="restart_heaviest_fork"} | counter | Number of CRDS values sent in pull response messages (Restart Heaviest Fork) |
| gossip_crds_tx_pull_response_bytes {crds_value="contact_info_v1"} | counter | Total wire bytes of CRDS sent out in pull response messages (Contact Info V1) |
| gossip_crds_tx_pull_response_bytes {crds_value="vote"} | counter | Total wire bytes of CRDS sent out in pull response messages (Vote) |
| gossip_crds_tx_pull_response_bytes {crds_value="lowest_slot"} | counter | Total wire bytes of CRDS sent out in pull response messages (Lowest Slot) |
| gossip_crds_tx_pull_response_bytes {crds_value="snapshot_hashes"} | counter | Total wire bytes of CRDS sent out in pull response messages (Snapshot Hashes) |
| gossip_crds_tx_pull_response_bytes {crds_value="accounts_hashes"} | counter | Total wire bytes of CRDS sent out in pull response messages (Accounts Hashes) |
| gossip_crds_tx_pull_response_bytes {crds_value="epoch_slots"} | counter | Total wire bytes of CRDS sent out in pull response messages (Epoch Slots) |
| gossip_crds_tx_pull_response_bytes {crds_value="version_v1"} | counter | Total wire bytes of CRDS sent out in pull response messages (Version V1) |
| gossip_crds_tx_pull_response_bytes {crds_value="version_v2"} | counter | Total wire bytes of CRDS sent out in pull response messages (Version V2) |
| gossip_crds_tx_pull_response_bytes {crds_value="node_instance"} | counter | Total wire bytes of CRDS sent out in pull response messages (Node Instance) |
| gossip_crds_tx_pull_response_bytes {crds_value="duplicate_shred"} | counter | Total wire bytes of CRDS sent out in pull response messages (Duplicate Shred) |
| gossip_crds_tx_pull_response_bytes {crds_value="incremental_snapshot_hashes"} | counter | Total wire bytes of CRDS sent out in pull response messages (Incremental Snapshot Hashes) |
| gossip_crds_tx_pull_response_bytes {crds_value="contact_info_v2"} | counter | Total wire bytes of CRDS sent out in pull response messages (Contact Info V2) |
| gossip_crds_tx_pull_response_bytes {crds_value="restart_last_voted_fork_slots"} | counter | Total wire bytes of CRDS sent out in pull response messages (Restart Last Voted Fork Slots) |
| gossip_crds_tx_pull_response_bytes {crds_value="restart_heaviest_fork"} | counter | Total wire bytes of CRDS sent out in pull response messages (Restart Heaviest Fork) |
| gossip_wfs_staked_peers_online | gauge | Number of staked peers with recent gossip activity |
| gossip_wfs_staked_peers_total | gauge | Total number of staked peers being tracked |
| gossip_wfs_stake_online | gauge | Lamports of stake from peers with recent gossip activity |
| gossip_wfs_stake_total | gauge | Total lamports of stake being tracked |
| gossip_wfs_state | gauge | Wait for supermajority state. 1=INIT, 2=WAIT, 3=PUBLISH, 4=DONE |
Repair Tile
| Metric | Type | Description |
|---|---|---|
| repair_total_pkt_count | counter | How many network packets we have sent, including reqs, pings, pongs, etc |
| repair_sent_pkt_types {repair_sent_request_types="needed_window"} | counter | What types of client messages are we sending (Need Window) |
| repair_sent_pkt_types {repair_sent_request_types="needed_highest_window"} | counter | What types of client messages are we sending (Need Highest Window) |
| repair_sent_pkt_types {repair_sent_request_types="needed_orphan"} | counter | What types of client messages are we sending (Need Orphans) |
| repair_sent_pkt_types {repair_sent_request_types="pong"} | counter | What types of client messages are we sending (Pong) |
| repair_repaired_slots | counter | Until which slots have we fully repaired |
| repair_current_slot | counter | Our view of the current cluster slot, max slot received |
| repair_request_peers | counter | How many peers have we requested |
| repair_sign_tile_unavail | counter | How many times no sign tiles were available to send request |
| repair_eager_repair_aggresses | counter | How many times we pass eager repair threshold |
| repair_rerequest_queue | counter | How many times we re-request a shred from the inflights queue |
| repair_malformed_ping | counter | How many times we received a malformed ping |
| repair_unknown_peer_ping | counter | How many times we received a ping from an unknown peer |
| repair_failed_sigverify_ping | counter | How many times we failed to verify the signature of a ping |
| repair_slot_complete_time | histogram | Time in seconds it took to complete a slot |
| repair_response_latency | histogram | Time in nanoseconds it took to receive a repair request response |
| repair_sign_duration_seconds | histogram | Duration of signing a message |
| repair_blk_evicted | counter | How many times we evicted a block from the forest |
| repair_blk_failed_insert | counter | How many times we failed to insert a block into the forest due to failed eviction |
Replay Tile
| Metric | Type | Description |
|---|---|---|
| replay_store_query_acquire | counter | Count of store slock acquires for query |
| replay_store_query_release | counter | Count of store slock releases for query |
| replay_store_query_wait | histogram | Time in seconds spent waiting to acquire the slock for query |
| replay_store_query_work | histogram | Time in seconds spent working before releasing the slock for query |
| replay_store_query_cnt | counter | Count of queries |
| replay_store_query_missing_cnt | counter | Count of queries where merkle root was missing |
| replay_store_query_mr | gauge | First 8 bytes of most recently queried merkle root |
| replay_store_query_missing_mr | gauge | First 8 bytes of most recently queried missing merkle root |
| replay_root_slot | gauge | The slot at which our node has most recently rooted |
| replay_root_distance | gauge | The distance in slots between our current root and the current reset slot |
| replay_leader_slot | gauge | The slot at which we are currently leader, or 0 if none |
| replay_next_leader_slot | gauge | The slot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot |
| replay_reset_slot | gauge | The slot at which we last reset the replay stage, or 0 if unknown |
| replay_max_live_banks | gauge | The maximum number of banks we can have alive |
| replay_live_banks | gauge | The number of banks we currently have alive |
| replay_reasm_free | gauge | The number of free FEC sets in the reassembly queue |
| replay_reasm_latest_slot | gauge | Slot of the latest FEC set in the reassembly queue that can be replayed |
| replay_reasm_latest_fec_idx | gauge | FEC set index of the latest FEC set in the reassembly queue that can be replayed |
| replay_sched_active_bank_idx | gauge | The bank that the replay scheduler is currently dispatching work for |
| replay_sched_last_dispatch_bank_idx | gauge | The bank that the replay scheduler last dispatched work for |
| replay_sched_last_dispatch_time_nanos | gauge | Timestamp when the replay scheduler last dispatched work |
| replay_sched_staging_lane_popcnt | gauge | The total number of staging lanes that are currently occupied in the replay scheduler. Up to four staging lanes can be occupied at once |
| replay_sched_staging_lane_popcnt_wmk | gauge | The high watermark number of staging lanes that were at one point occupied in the replay scheduler |
| replay_sched_staging_lane_head_bank_idx0 | gauge | The bank at the head of staging lane 0 of the replay dispatcher |
| replay_sched_staging_lane_head_bank_idx1 | gauge | The bank at the head of staging lane 1 of the replay dispatcher |
| replay_sched_staging_lane_head_bank_idx2 | gauge | The bank at the head of staging lane 2 of the replay dispatcher |
| replay_sched_staging_lane_head_bank_idx3 | gauge | The bank at the head of staging lane 3 of the replay dispatcher |
| replay_sched_txn_pool_popcnt | gauge | The total number of transactions that are available to be scheduled in the replay scheduler |
| replay_sched_txn_pool_size | gauge | The maximum number of transactions that the replay scheduler can consider. This value is fixed at Firedancer startup but is a useful reference for TxnPoolPopcnt |
| replay_sched_mblk_pool_popcnt | gauge | The total number of microblocks that are available to be scheduled in the replay scheduler |
| replay_sched_mblk_pool_size | gauge | The maximum number of microblocks that the replay scheduler can consider. This value is fixed at Firedancer startup but is a useful reference for MblkPoolPopcnt |
| replay_sched_block_pool_popcnt | gauge | The total number of blocks that are alive in the replay scheduler |
| replay_sched_block_pool_size | gauge | The maximum number of blocks that the replay scheduler can keep track of. This value is fixed at Firedancer startup but is a useful reference for BlockPoolPopcnt |
| replay_sched_block_added_staged | counter | Blocks added to the replay scheduler as staged |
| replay_sched_block_added_unstaged | counter | Blocks added to the replay scheduler as unstaged |
| replay_sched_block_replayed | counter | Blocks that the replay scheduler considers fully replayed |
| replay_sched_block_abandoned | counter | Blocks that the replay scheduler abandoned for any reason |
| replay_sched_block_bad | counter | Blocks that the replay scheduler considers bad |
| replay_sched_block_promoted | counter | Blocks that the replay scheduler promoted from unstaged to a staging lane |
| replay_sched_block_demoted | counter | Blocks that the replay scheduler demoted from a staging lane to unstaged |
| replay_sched_deactivate_no_child | counter | Number of times that the replay scheduler stopped dispatching because a block fully replayed and there was no child block available yet |
| replay_sched_deactivate_no_work | counter | Number of times that the replay scheduler stopped dispatching because a partially replayed block ran out of work to dispatch |
| replay_sched_deactivate_abandoned | counter | Number of times that the replay scheduler stopped dispatching because a block got abandoned for any reason |
| replay_sched_deactivate_minority | counter | Number of times that the replay scheduler stopped dispatching because a block got abandoned for being on a minority fork |
| replay_sched_lane_switch | counter | Number of times that the replay scheduler switched the lane that it is dispatching from |
| replay_sched_lane_promote | counter | Number of times that the replay scheduler promoted one or more blocks onto a staging lane |
| replay_sched_lane_demote | counter | Number of times that the replay scheduler demoted one or more blocks from a staging lane |
| replay_sched_fork_observed | counter | Number of forks that the replay scheduler has observed |
| replay_sched_alut_success | counter | Number of transactions with address lookup tables that the replay scheduler successfully expanded |
| replay_sched_alut_failure | counter | Number of transactions with address lookup tables that the replay scheduler failed to expand |
| replay_sched_txn_abandoned_parsed | counter | Number of transactions that were parsed but ended up being in abandoned blocks |
| replay_sched_txn_abandoned_exec | counter | Number of transactions that were executed but ended up being in abandoned blocks |
| replay_sched_txn_abandoned_done | counter | Number of transactions that were fully replayed but ended up being in abandoned blocks |
| replay_sched_weighted_in_flight | counter | Cumulative number of transactions that have been in-flight from the replay scheduler's point of view, weighted by the duration of each transaction. This value is useful as the dividend for TxnWeightedInFlightDuration |
| replay_sched_weighted_in_flight_duration | counter | Duration over which TxnWeightedInFlight was accumulated. Use this as divisor to get the average number of in-flight transactions |
| replay_sched_none_in_flight_duration | counter | Duration over which the replay scheduler had no transaction in-flight. This value is useful in a ratio against WeightedInFlightDuration to see how often the replay transaction execution pipeline was idle |
| replay_sched_txn_parsed | counter | Number of transactions that the replay pipeline has parsed |
| replay_sched_txn_exec | counter | Number of transactions that the replay pipeline has executed |
| replay_sched_txn_sigverify | counter | Number of transactions that the replay pipeline has sigverified |
| replay_sched_txn_mixin | counter | Number of transactions that the replay pipeline has performed PoH mixin |
| replay_sched_txn_done | counter | Number of transactions that were fully replayed |
| replay_sched_mblk_parsed | counter | Number of microblocks that the replay pipeline has parsed |
| replay_sched_mblk_hashed | counter | Number of microblocks that the replay pipeline has fully hashed |
| replay_sched_mblk_done | counter | Number of microblocks that were fully replayed |
| replay_sched_bytes_ingested | counter | Number of bytes that the replay scheduler has ingested |
| replay_sched_bytes_ingested_padding | counter | Number of bytes that the replay scheduler ingested but did not parse for being padding |
| replay_sched_bytes_dropped | counter | Number of bytes that the replay scheduler refused to ingest because the block is considered abandoned |
| replay_sched_fec | counter | Number of FEC sets that the replay scheduler has been given |
| replay_slots_total | counter | Count of slots replayed successfully or leader slots packed and shredded successfully |
| replay_transactions_total | counter | Count of transactions processed overall on the current fork |
| replay_sched_full | counter | Times where sched is full and a FEC set can't be processed |
| replay_reasm_empty | counter | Times where reasm is empty and a FEC set can't be processed |
| replay_leader_bid_wait | counter | Times where replay is blocked by the PoH tile not sending an end of leader message |
| replay_banks_full | counter | Times where banks are full and a FEC set can't be processed |
| replay_storage_root_behind | counter | Times where the storage root is behind the consensus root and can't be advanced |
| replay_accdb_created | counter | Number of account database records created |
| replay_accdb_reverted | counter | Number of account database records reverted |
| replay_accdb_rooted | counter | Number of account database entries rooted |
| replay_accdb_rooted_bytes | counter | Number of bytes in account database entries rooted (including overhead) |
| replay_accdb_gc_root | counter | Number of account database entries garbage collected |
| replay_accdb_reclaimed | counter | Number of account database entries reclaimed (deletion rooted) |
| replay_root_slot_duration_seconds | histogram | Time in seconds spent updating the rooted account store (one sample per block) |
| replay_root_account_duration_seconds | histogram | Time in seconds spent updating the rooted account store (one sample per block, normalized by account count) |
| replay_root_elapsed_seconds {root_phase="db"} | counter | Total time in seconds spent rooting accounts (Waiting on database server) |
| replay_root_elapsed_seconds {root_phase="copy"} | counter | Total time in seconds spent rooting accounts (Copying account data) |
| replay_root_elapsed_seconds {root_phase="gc"} | counter | Total time in seconds spent rooting accounts (Garbage collecting old account data) |
| replay_progcache_rooted | counter | Number of program cache entries rooted |
| replay_progcache_gc_root | counter | Number of program cache entries garbage collected while rooting |
| replay_progcache_free_parts | gauge | Number of program cache heap partitions free (indicates fragmentation) |
| replay_progcache_free_bytes | gauge | Program cache heap utilization (free bytes) |
| replay_progcache_size_bytes | gauge | Program cache heap utilization (total size) |
| replay_progcache_free_part_max_bytes | gauge | Largest free heap partition in program cache |
| replay_progcache_used_part_median_bytes | gauge | Median used heap partition size in program cache |
| replay_progcache_used_part_mean_bytes | gauge | Mean used heap partition size in program cache |
| replay_progcache_time_seconds | counter | Total time in seconds spent doing program cache tasks |
| replay_accdb_cache_free_parts | gauge | Number of account database cache heap partitions free (indicates fragmentation) |
| replay_accdb_cache_free_bytes | gauge | Account database cache heap utilization (free bytes) |
| replay_accdb_cache_size_bytes | gauge | Account database cache heap utilization (total size) |
| replay_accdb_cache_free_part_max_bytes | gauge | Largest free heap partition in account database cache |
| replay_accdb_cache_used_part_median_bytes | gauge | Median used heap partition size in account database cache |
| replay_accdb_cache_used_part_mean_bytes | gauge | Mean used heap partition size in account database cache |
Execrp Tile
| Metric | Type | Description |
|---|---|---|
| execrp_accdb_created | counter | Number of account database records created |
| execrp_txn_regime {txn_regime="setup"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction setup) |
| execrp_txn_regime {txn_regime="exec"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction execution (includes VM setup/execution)) |
| execrp_txn_regime {txn_regime="commit"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction result commit) |
| execrp_vm_regime {vm_regime="setup"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM setup) |
| execrp_vm_regime {vm_regime="commit"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM commit) |
| execrp_vm_regime {vm_regime="setup_cpi"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM setup (CPI)) |
| execrp_vm_regime {vm_regime="commit_cpi"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM commit (CPI)) |
| execrp_vm_regime {vm_regime="interpreter"} | counter | Mutually exclusive and exhaustive duration of time spent in virtual machine execution regimes (VM interpreter execution) |
| execrp_txn_account_changes {account_change="unchanged_nonexist"} | counter | Transaction account change event counters (Account did not exist before and still does not) |
| execrp_txn_account_changes {account_change="created"} | counter | Transaction account change event counters (Account created) |
| execrp_txn_account_changes {account_change="delete"} | counter | Transaction account change event counters (Account deleted) |
| execrp_txn_account_changes {account_change="modify"} | counter | Transaction account change event counters (Account modified) |
| execrp_txn_account_changes {account_change="unchanged"} | counter | Transaction account change event counters (Account unchanged) |
| execrp_compute_units_total | counter | Estimated number of compute units executed since tile start |
| execrp_progcache_lookups | counter | Program cache lookup counter |
| execrp_progcache_hits | counter | Program cache hit counter |
| execrp_progcache_misses | counter | Program cache miss counter |
| execrp_progcache_oom_heap | counter | Program cache out-of-memory event counter (heap) |
| execrp_progcache_oom_desc | counter | Program cache out-of-memory event counter (descriptor table) |
| execrp_progcache_fills | counter | Number of program cache insertions |
| execrp_progcache_fill_bytes | counter | Number of bytes inserted into program cache |
| execrp_progcache_spills | counter | Program cache spill counter (OOM fallback mechanism) |
| execrp_progcache_spill_bytes | counter | Number of bytes spilled from program cache (OOM fallback mechanism) |
| execrp_progcache_evictions | counter | Program cache eviction counter |
| execrp_progcache_eviction_bytes | counter | Number of bytes evicted from program cache |
| execrp_progcache_duration_total_seconds | counter | Total time in seconds spent on program cache operations |
| execrp_progcache_duration_load_seconds | counter | Total time in seconds spent loading programs |
Accdb Tile
| Metric | Type | Description |
|---|---|---|
| accdb_accounts | gauge | Current number of accounts |
| accdb_read_ops {storage_type="shared_cache"} | counter | Total number of read operations (Record cache) |
| accdb_read_ops {storage_type="io_cache"} | counter | Total number of read operations (I/O layer cache) |
| accdb_read_ops {storage_type="file"} | counter | Total number of read operations (File access) |
| accdb_read_bytes {storage_type="shared_cache"} | counter | Total number of bytes read (Record cache) |
| accdb_read_bytes {storage_type="io_cache"} | counter | Total number of bytes read (I/O layer cache) |
| accdb_read_bytes {storage_type="file"} | counter | Total number of bytes read (File access) |
| accdb_write_ops {storage_type="shared_cache"} | counter | Total number of write operations (Record cache) |
| accdb_write_ops {storage_type="io_cache"} | counter | Total number of write operations (I/O layer cache) |
| accdb_write_ops {storage_type="file"} | counter | Total number of write operations (File access) |
| accdb_write_bytes {storage_type="shared_cache"} | counter | Total number of bytes written (Record cache) |
| accdb_write_bytes {storage_type="io_cache"} | counter | Total number of bytes written (I/O layer cache) |
| accdb_write_bytes {storage_type="file"} | counter | Total number of bytes written (File access) |
| accdb_file_capacity_bytes | gauge | Account database file capacity in bytes |
| accdb_file_used_bytes | gauge | Current number of bytes used in account database file |
| accdb_bstream_seq {bstream_seq="ancient"} | gauge | Current bstream sequence number (Blocks between ancient and past have been written and forgotten (no read, no write)) |
| accdb_bstream_seq {bstream_seq="past"} | gauge | Current bstream sequence number (Blocks between past and present have been written (read only)) |
| accdb_bstream_seq {bstream_seq="present"} | gauge | Current bstream sequence number (Blocks between present and future are being written (write only)) |
| accdb_bstream_seq {bstream_seq="future"} | gauge | Current bstream sequence number (Blocks between future and ancient have not been written (no read, no write)) |
| accdb_request_batches | counter | Number of request batches processed |
| accdb_requests {vinyl_request="acquire"} | counter | Number of requests processed (Acquire record) |
| accdb_requests {vinyl_request="release"} | counter | Number of requests processed (Release record) |
| accdb_requests {vinyl_request="erase"} | counter | Number of requests processed (Erase record) |
| accdb_blocks {vinyl_blocks="pair"} | counter | Number of blocks written to bstream (Record) |
| accdb_blocks {vinyl_blocks="dead"} | counter | Number of blocks written to bstream (Record deletion) |
| accdb_blocks {vinyl_blocks="part"} | counter | Number of blocks written to bstream (Partition/divider) |
| accdb_garbage_bytes | gauge | |
| accdb_cum_gc_bytes | counter | Total number of record bytes that were garbage collected |
| accdb_account_index_remaining_free | gauge | Remaining free slots in the account database index (validator crashes when this number reaches zero) |
Tower Tile
| Metric | Type | Description |
|---|---|---|
| tower_ignored_cnt | counter | Number of replay_slot_completed frags we ignored |
| tower_ignored_slot | gauge | Most recent ignored replay_slot_completed frag |
| tower_eqvoc_cnt | counter | Number of replay_slot_completed frags we detect as equivocations |
| tower_eqvoc_slot | gauge | Most recent equivocating replay_slot_completed frag |
| tower_replay_slot | gauge | Replay slot |
| tower_vote_slot | gauge | Most recent vote slot, ULONG_MAX if no vote cast |
| tower_reset_slot | gauge | Reset slot |
| tower_root_slot | gauge | Root slot |
| tower_init_slot | gauge | Init slot |
| tower_txn_bad_deser | counter | Vote txn failed to deserialize |
| tower_txn_bad_tower | counter | Vote txn deserialized but tower was invalid |
| tower_txn_not_tower_sync | counter | Vote txn was not a TowerSync instruction |
| tower_txn_empty_tower | counter | Vote txn had an empty tower (validator hasn't voted) |
| tower_txn_not_ready | counter | Vote txn arrived before votes root was initialized |
| tower_eqvoc_success_merkle | counter | Merkle root conflict |
| tower_eqvoc_success_meta | counter | Coding metadata conflict |
| tower_eqvoc_success_last | counter | Last shred index conflict |
| tower_eqvoc_success_overlap | counter | Overlapping FEC set conflict |
| tower_eqvoc_success_chained | counter | Chained merkle root conflict |
| tower_eqvoc_err_serde | counter | Invalid serialization |
| tower_eqvoc_err_slot | counter | Shreds were for different slots |
| tower_eqvoc_err_version | counter | Either shred had wrong shred version |
| tower_eqvoc_err_type | counter | Wrong shred type (must be chained merkle) |
| tower_eqvoc_err_merkle | counter | Failed to derive merkle root |
| tower_eqvoc_err_signature | counter | Failed to sig verify |
| tower_eqvoc_err_chunk_cnt | counter | num_chunks != FD_EQVOC_CHUNK_CNT |
| tower_eqvoc_err_chunk_idx | counter | chunk_index >= FD_EQVOC_CHUNK_CNT |
| tower_eqvoc_err_chunk_len | counter | chunk_len does not match expected length for chunk_index |
| tower_eqvoc_err_ignored_from | counter | Unrecognized from address |
| tower_eqvoc_err_ignored_slot | counter | Slot older than root or unable to derive leader schedule |
| tower_eqvoc_proof_constructed | counter | Number of duplicate proofs we constructed from shreds |
| tower_eqvoc_proof_verified | counter | Number of duplicate proofs we verified from gossip |
| tower_ghost_not_voted | counter | Ghost vote skipped because voter hasn't voted |
| tower_ghost_too_old | counter | Ghost vote slot was behind the root |
| tower_ghost_already_voted | counter | Ghost vote slot was not newer than previous vote |
| tower_hfork_unknown_vtr | counter | Hfork voter not in voter set |
| tower_hfork_already_voted | counter | Hfork voter already voted for this block_id |
| tower_hfork_too_old | counter | Hfork vote slot not newer than previous |
| tower_hfork_matched_slot | gauge | Highest slot where 52%+ of stake agreed on our bank hash |
| tower_hfork_mismatched_slot | gauge | Highest slot where 52%+ of stake agreed on a different bank hash than ours (we hard forked) |
| tower_ancestor_rollback | counter | Rollback to an ancestor of our prev vote (can't vote) |
| tower_sibling_confirmed | counter | Duplicate sibling got confirmed (can't vote) |
| tower_same_fork | counter | Same fork as prev vote (can vote) |
| tower_switch_pass | counter | Prev vote was on a different fork, but we are allowed to switch (can vote) |
| tower_switch_fail | counter | Prev vote was on a different fork, and we are not allowed to switch (can't vote) |
| tower_lockout_fail | counter | Locked out (can't vote) |
| tower_threshold_fail | counter | Did not pass threshold check (can't vote) |
| tower_propagated_fail | counter | Prev leader block did not propagate (can't vote) |
| tower_votes_too_old | counter | Vote slot was behind the votes root |
| tower_votes_too_new | counter | Vote slot was too far ahead of the votes root |
| tower_votes_unknown_vtr | counter | Vote account was not in the voter set |
| tower_votes_already_voted | counter | Voter already voted for this slot |
| tower_votes_unknown_slot | counter | Vote txn was for a slot we haven't replayed |
| tower_votes_unknown_block_id | counter | Vote txn was for a block id we don't recognize |
Txsend Tile
| Metric | Type | Description |
|---|---|---|
| txsend_received_packets | counter | Total count of QUIC packets received |
| txsend_received_bytes | counter | Total bytes received via QUIC |
| txsend_sent_packets | counter | Total count of QUIC packets sent |
| txsend_sent_bytes | counter | Total bytes sent via QUIC |
| txsend_retry_sent | counter | Total count of QUIC retry packets sent |
| txsend_connections_alloc | gauge | Number of currently allocated QUIC connections |
| txsend_connections_state {quic_conn_state="invalid"} | gauge | Number of QUIC connections in each state (Freed) |
| txsend_connections_state {quic_conn_state="handshake"} | gauge | Number of QUIC connections in each state (Handshaking peer) |
| txsend_connections_state {quic_conn_state="handshake_complete"} | gauge | Number of QUIC connections in each state (Handshake complete, confirming with peer) |
| txsend_connections_state {quic_conn_state="active"} | gauge | Number of QUIC connections in each state (Active connection) |
| txsend_connections_state {quic_conn_state="peer_close"} | gauge | Number of QUIC connections in each state (Peer requested close) |
| txsend_connections_state {quic_conn_state="abort"} | gauge | Number of QUIC connections in each state (Connection terminating due to error) |
| txsend_connections_state {quic_conn_state="close_pending"} | gauge | Number of QUIC connections in each state (Connection is closing) |
| txsend_connections_state {quic_conn_state="dead"} | gauge | Number of QUIC connections in each state (Connection about to be freed) |
| txsend_connections_created | counter | Total count of QUIC connections created |
| txsend_connections_closed | counter | Total count of QUIC connections closed |
| txsend_connections_aborted | counter | Total count of QUIC connections aborted |
| txsend_connections_timed_out | counter | Total count of QUIC connections timed out |
| txsend_connections_retried | counter | Total count of QUIC connections retried |
| txsend_connection_error_no_slots | counter | Total count of connection errors due to no slots |
| txsend_connection_error_retry_fail | counter | Total count of connection retry failures |
| txsend_pkt_crypto_failed {quic_enc_level="initial"} | counter | Total count of packets with crypto failures (Initial) |
| txsend_pkt_crypto_failed {quic_enc_level="early"} | counter | Total count of packets with crypto failures (Early data) |
| txsend_pkt_crypto_failed {quic_enc_level="handshake"} | counter | Total count of packets with crypto failures (Handshake) |
| txsend_pkt_crypto_failed {quic_enc_level="app"} | counter | Total count of packets with crypto failures (App data) |
| txsend_pkt_no_key {quic_enc_level="initial"} | counter | Total count of packets with no key (Initial) |
| txsend_pkt_no_key {quic_enc_level="early"} | counter | Total count of packets with no key (Early data) |
| txsend_pkt_no_key {quic_enc_level="handshake"} | counter | Total count of packets with no key (Handshake) |
| txsend_pkt_no_key {quic_enc_level="app"} | counter | Total count of packets with no key (App data) |
| txsend_pkt_no_conn {quic_pkt_handle="initial"} | counter | Total count of packets with no connection (Initial) |
| txsend_pkt_no_conn {quic_pkt_handle="retry"} | counter | Total count of packets with no connection (Retry) |
| txsend_pkt_no_conn {quic_pkt_handle="handshake"} | counter | Total count of packets with no connection (Handshake) |
| txsend_pkt_no_conn {quic_pkt_handle="one_rtt"} | counter | Total count of packets with no connection (1-RTT) |
| txsend_pkt_tx_alloc_fail | counter | Total count of packet TX allocation failures |
| txsend_pkt_net_header_invalid | counter | Total count of packets with invalid network headers |
| txsend_pkt_quic_header_invalid | counter | Total count of packets with invalid QUIC headers |
| txsend_pkt_undersz | counter | Total count of undersized packets |
| txsend_pkt_oversz | counter | Total count of oversized packets |
| txsend_pkt_verneg | counter | Total count of version negotiation packets |
| txsend_pkt_retransmissions {quic_enc_level="initial"} | counter | Total count of QUIC packet retransmissions (Initial) |
| txsend_pkt_retransmissions {quic_enc_level="early"} | counter | Total count of QUIC packet retransmissions (Early data) |
| txsend_pkt_retransmissions {quic_enc_level="handshake"} | counter | Total count of QUIC packet retransmissions (Handshake) |
| txsend_pkt_retransmissions {quic_enc_level="app"} | counter | Total count of QUIC packet retransmissions (App data) |
| txsend_handshakes_created | counter | Total count of QUIC handshakes created |
| txsend_handshake_error_alloc_fail | counter | Total count of handshake allocation failures |
| txsend_handshake_evicted | counter | Total count of handshakes evicted |
| txsend_stream_received_events | counter | Total count of stream events received |
| txsend_stream_received_bytes | counter | Total bytes received via streams |
| txsend_received_frames {quic_frame_type="unknown"} | counter | Total count of QUIC frames received (Unknown frame type) |
| txsend_received_frames {quic_frame_type="ack"} | counter | Total count of QUIC frames received (ACK frame) |
| txsend_received_frames {quic_frame_type="reset_stream"} | counter | Total count of QUIC frames received (RESET_STREAM frame) |
| txsend_received_frames {quic_frame_type="stop_sending"} | counter | Total count of QUIC frames received (STOP_SENDING frame) |
| txsend_received_frames {quic_frame_type="crypto"} | counter | Total count of QUIC frames received (CRYPTO frame) |
| txsend_received_frames {quic_frame_type="new_token"} | counter | Total count of QUIC frames received (NEW_TOKEN frame) |
| txsend_received_frames {quic_frame_type="stream"} | counter | Total count of QUIC frames received (STREAM frame) |
| txsend_received_frames {quic_frame_type="max_data"} | counter | Total count of QUIC frames received (MAX_DATA frame) |
| txsend_received_frames {quic_frame_type="max_stream_data"} | counter | Total count of QUIC frames received (MAX_STREAM_DATA frame) |
| txsend_received_frames {quic_frame_type="max_streams"} | counter | Total count of QUIC frames received (MAX_STREAMS frame) |
| txsend_received_frames {quic_frame_type="data_blocked"} | counter | Total count of QUIC frames received (DATA_BLOCKED frame) |
| txsend_received_frames {quic_frame_type="stream_data_blocked"} | counter | Total count of QUIC frames received (STREAM_DATA_BLOCKED frame) |
| txsend_received_frames {quic_frame_type="streams_blocked"} | counter | Total count of QUIC frames received (STREAMS_BLOCKED(bidi) frame) |
| txsend_received_frames {quic_frame_type="new_conn_id"} | counter | Total count of QUIC frames received (NEW_CONN_ID frame) |
| txsend_received_frames {quic_frame_type="retire_conn_id"} | counter | Total count of QUIC frames received (RETIRE_CONN_ID frame) |
| txsend_received_frames {quic_frame_type="path_challenge"} | counter | Total count of QUIC frames received (PATH_CHALLENGE frame) |
| txsend_received_frames {quic_frame_type="path_response"} | counter | Total count of QUIC frames received (PATH_RESPONSE frame) |
| txsend_received_frames {quic_frame_type="conn_close_quic"} | counter | Total count of QUIC frames received (CONN_CLOSE(transport) frame) |
| txsend_received_frames {quic_frame_type="conn_close_app"} | counter | Total count of QUIC frames received (CONN_CLOSE(app) frame) |
| txsend_received_frames {quic_frame_type="handshake_done"} | counter | Total count of QUIC frames received (HANDSHAKE_DONE frame) |
| txsend_received_frames {quic_frame_type="ping"} | counter | Total count of QUIC frames received (PING frame) |
| txsend_received_frames {quic_frame_type="padding"} | counter | Total count of QUIC frames received (PADDING frame) |
| txsend_frame_fail_parse | counter | Total count of frame parse failures |
| txsend_frame_tx_alloc {frame_tx_alloc_result="success"} | counter | Results of attempts to acquire QUIC frame metadata (Success) |
| txsend_frame_tx_alloc {frame_tx_alloc_result="fail_empty_pool"} | counter | Results of attempts to acquire QUIC frame metadata (PktMetaPoolEmpty) |
| txsend_frame_tx_alloc {frame_tx_alloc_result="fail_conn_max"} | counter | Results of attempts to acquire QUIC frame metadata (ConnMaxedInflightFrames) |
| txsend_ack_tx {quic_ack_tx="noop"} | counter | Total count of ACK frames transmitted (Non-ACK-eliciting packet) |
| txsend_ack_tx {quic_ack_tx="new"} | counter | Total count of ACK frames transmitted (New ACK range) |
| txsend_ack_tx {quic_ack_tx="merged"} | counter | Total count of ACK frames transmitted (Merged into existing ACK range) |
| txsend_ack_tx {quic_ack_tx="drop"} | counter | Total count of ACK frames transmitted (Out of buffers) |
| txsend_ack_tx {quic_ack_tx="cancel"} | counter | Total count of ACK frames transmitted (ACK suppressed by handler) |
| txsend_service_duration_seconds | histogram | Duration spent in service |
| txsend_receive_duration_seconds | histogram | Duration spent processing packets |
Diag Tile
| Metric | Type | Description |
|---|---|---|
| diag_bundle_health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy bundle subsystem means at least one bundle tile currently zhas an active connection to the block engine server |
| diag_vote_health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy vote subsystem means the client has cast at least one vote in both the last 60 seconds and last 150 slots (before the currently replay slot) |
| diag_replay_health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy replay subsystem means that the largest fully-processed replay slot on the chosen consensus fork is within 12 slots of the current turbine slot |
| diag_turbine_health | gauge | 0=unhealthy, 1=healthy, 2=disabled. A healthy turbine subsystem means that the largest slot associated with received turbine shreds has not stalled for 12 seconds, and also that the average replay ingress traffic exceeds the average ingress repair traffic over the past 12 seconds |
Event Tile
| Metric | Type | Description |
|---|---|---|
| event_connection_state | gauge | 0 = disconnected, 1 = connecting, 2 = connected |
| event_event_queue_count | gauge | The current number of events in the event queue waiting to be sent to the event service |
| event_event_queue_drops | counter | The total number of events dropped because the event queue was full |
| event_event_queue_bytes_used | gauge | The current number of bytes used in the event queue |
| event_event_queue_bytes_capacity | gauge | The total capacity in bytes of the event queue |
| event_events_sent | counter | The total number of events sent to the event service |
| event_events_acked | counter | The total number of events acknowledged by the event service |
| event_bytes_written | counter | The total number of bytes written to the event service |
| event_bytes_read | counter | The total number of bytes read from the event service |
Gui Tile
| Metric | Type | Description |
|---|---|---|
| gui_connection_count | gauge | The number of active http connections to the GUI service, excluding connections that have been upgraded to a WebSocket connection |
| gui_websocket_connection_count | gauge | The number of active websocket connections to the GUI service |
| gui_websocket_frames_sent | counter | The total number of websocket frames sent to all connections to the GUI service |
| gui_websocket_frames_received | counter | The total number of websocket frames received from all connections to the GUI service |
| gui_bytes_written | counter | The total number of bytes written to all connections to the GUI service |
| gui_bytes_read | counter | The total number of bytes read from all connections to the GUI service |
Metric Tile
| Metric | Type | Description |
|---|---|---|
| metric_boot_timestamp_nanos | gauge | Timestamp when validator was started (nanoseconds since epoch) |
| metric_connection_count | gauge | The number of active http connections to the Prometheus endpoint |
| metric_bytes_written | counter | The total number of bytes written to all responses on the Prometheus endpoint |
| metric_bytes_read | counter | The total number of bytes read from all requests to the Prometheus endpoint |
Rpc Tile
| Metric | Type | Description |
|---|
Resolh Tile
| Metric | Type | Description |
|---|---|---|
| resolh_no_bank_drop | counter | Count of transactions dropped because the bank was not available |
| resolh_stash_operation {resolve_stash_operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolh_stash_operation {resolve_stash_operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolh_stash_operation {resolve_stash_operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolh_stash_operation {resolve_stash_operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories) |
| resolh_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolh_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolh_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolh_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolh_lut_resolved {lut_resolve_result="account_not_found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolh_lut_resolved {lut_resolve_result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) |
| resolh_blockhash_expired | counter | Count of transactions that failed to resolve because the blockhash was expired |
| resolh_transaction_bundle_peer_failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed |
Bank Tile
| Metric | Type | Description |
|---|---|---|
| bank_transaction_sanitize_failure | counter | Number of transactions that failed to sanitize |
| bank_transaction_not_executed_failure | counter | Number of transactions that did not execute. This is different than transactions which fail to execute, which make it onto the chain |
| bank_transaction_load_address_tables {lut_resolve_result="invalid_lookup_index"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The transaction referenced an index in a LUT that didn't exist) |
| bank_transaction_load_address_tables {lut_resolve_result="account_uninitialized"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT hasn't been initialized) |
| bank_transaction_load_address_tables {lut_resolve_result="invalid_account_data"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT couldn't be parsed) |
| bank_transaction_load_address_tables {lut_resolve_result="invalid_account_owner"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| bank_transaction_load_address_tables {lut_resolve_result="account_not_found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (The account referenced as a LUT couldn't be found) |
| bank_transaction_load_address_tables {lut_resolve_result="success"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported (Resolved successfully) |
| bank_transaction_result {transaction_error="success"} | counter | Result of loading and executing a transaction (Success) |
| bank_transaction_result {transaction_error="account_in_use"} | counter | Result of loading and executing a transaction (An account is already being processed in another transaction in a way that does not support parallelism) |
| bank_transaction_result {transaction_error="account_loaded_twice"} | counter | Result of loading and executing a transaction (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys) |
| bank_transaction_result {transaction_error="account_not_found"} | counter | Result of loading and executing a transaction (Attempt to debit an account but found no record of a prior credit) |
| bank_transaction_result {transaction_error="program_account_not_found"} | counter | Result of loading and executing a transaction (Attempt to load a program that does not exist) |
| bank_transaction_result {transaction_error="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction) |
| bank_transaction_result {transaction_error="invalid_account_for_fee"} | counter | Result of loading and executing a transaction (This account may not be used to pay transaction fees) |
| bank_transaction_result {transaction_error="already_processed"} | counter | Result of loading and executing a transaction (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack) |
| bank_transaction_result {transaction_error="blockhash_not_found"} | counter | Result of loading and executing a transaction (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded) |
| bank_transaction_result {transaction_error="instruction_error"} | counter | Result of loading and executing a transaction (An error occurred while processing an instruction) |
| bank_transaction_result {transaction_error="call_chain_too_deep"} | counter | Result of loading and executing a transaction (Loader call chain is too deep) |
| bank_transaction_result {transaction_error="missing_signature_for_fee"} | counter | Result of loading and executing a transaction (Transaction requires a fee but has no signature present) |
| bank_transaction_result {transaction_error="invalid_account_index"} | counter | Result of loading and executing a transaction (Transaction contains an invalid account reference) |
| bank_transaction_result {transaction_error="signature_failure"} | counter | Result of loading and executing a transaction (Transaction did not pass signature verification) |
| bank_transaction_result {transaction_error="invalid_program_for_execution"} | counter | Result of loading and executing a transaction (This program may not be used for executing instructions) |
| bank_transaction_result {transaction_error="sanitize_failure"} | counter | Result of loading and executing a transaction (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked) |
| bank_transaction_result {transaction_error="cluster_maintenance"} | counter | Result of loading and executing a transaction (Transactions are currently disabled due to cluster maintenance) |
| bank_transaction_result {transaction_error="account_borrow_outstanding"} | counter | Result of loading and executing a transaction (Transaction processing left an account with an outstanding borrowed reference) |
| bank_transaction_result {transaction_error="would_exceed_max_block_cost_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max Block Cost Limit) |
| bank_transaction_result {transaction_error="unsupported_version"} | counter | Result of loading and executing a transaction (Transaction version is unsupported) |
| bank_transaction_result {transaction_error="invalid_writable_account"} | counter | Result of loading and executing a transaction (Transaction loads a writable account that cannot be written) |
| bank_transaction_result {transaction_error="would_exceed_max_account_cost_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max account limit within the block) |
| bank_transaction_result {transaction_error="would_exceed_account_data_block_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed account data limit within the block) |
| bank_transaction_result {transaction_error="too_many_account_locks"} | counter | Result of loading and executing a transaction (Transaction locked too many accounts) |
| bank_transaction_result {transaction_error="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction (Address lookup table not found) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction (Attempted to lookup addresses from an account owned by the wrong program) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction (Attempted to lookup addresses from an invalid account) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction (Address table lookup uses an invalid index) |
| bank_transaction_result {transaction_error="invalid_rent_paying_account"} | counter | Result of loading and executing a transaction (Transaction leaves an account with a lower balance than rent-exempt minimum) |
| bank_transaction_result {transaction_error="would_exceed_max_vote_cost_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed max Vote Cost Limit) |
| bank_transaction_result {transaction_error="would_exceed_account_data_total_limit"} | counter | Result of loading and executing a transaction (Transaction would exceed total account data limit) |
| bank_transaction_result {transaction_error="duplicate_instruction"} | counter | Result of loading and executing a transaction (Transaction contains a duplicate instruction that is not allowed) |
| bank_transaction_result {transaction_error="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction (Transaction results in an account with insufficient funds for rent) |
| bank_transaction_result {transaction_error="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction (Transaction exceeded max loaded accounts data size cap) |
| bank_transaction_result {transaction_error="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction (LoadedAccountsDataSizeLimit set for transaction must be greater than 0) |
| bank_transaction_result {transaction_error="resanitization_needed"} | counter | Result of loading and executing a transaction (Sanitized transaction differed before/after feature activation. Needs to be resanitized) |
| bank_transaction_result {transaction_error="program_execution_temporarily_restricted"} | counter | Result of loading and executing a transaction (Program execution is temporarily restricted on an account) |
| bank_transaction_result {transaction_error="unbalanced_transaction"} | counter | Result of loading and executing a transaction (The total balance before the transaction does not equal the total balance after the transaction) |
| bank_transaction_result {transaction_error="program_cache_hit_max_limit"} | counter | Result of loading and executing a transaction (The total program cache size hit the maximum allowed limit) |
| bank_transaction_result {transaction_error="commit_cancelled"} | counter | Result of loading and executing a transaction (The process for committing the transaction was cancelled internally) |
| bank_transaction_result {transaction_error="bundle_peer"} | counter | Result of loading and executing a transaction (Transaction is part of a bundle and one of the peer transactions failed) |
| bank_processing_failed | counter | Count of transactions for which the processing stage failed and won't land on chain |
| bank_fee_only_transactions | counter | Count of transactions that will land on chain but without executing |
| bank_executed_failed_transactions | counter | Count of transactions that execute on chain but failed |
| bank_successful_transactions | counter | Count of transactions that execute on chain and succeed |
| bank_cost_model_undercount | counter | Count of transactions that used more CUs than the cost model should have permitted them to |
Pohh Tile
| Metric | Type | Description |
|---|---|---|
| pohh_begin_leader_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the bank. |
| pohh_first_microblock_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock. |
| pohh_slot_done_delay_seconds | histogram | Delay between when we become leader in a slot and when we finish the slot. |
| pohh_bundle_initialize_delay_seconds | histogram | Delay in starting the slot caused by loading the information needed to generate the bundle crank transactions |
Store Tile
| Metric | Type | Description |
|---|---|---|
| store_transactions_inserted | counter | Count of transactions produced while we were leader in the shreds that have been inserted so far |
Backt Tile
| Metric | Type | Description |
|---|---|---|
| backt_final_slot | gauge | The slot after which the backtest will complete |
| backt_start_slot | gauge | The slot at which the backtest started |
Benchs Tile
| Metric | Type | Description |
|---|---|---|
| benchs_transactions_sent | counter | Number of benchmark packets sent |