Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via. a Prometheus HTTP endpoint:
toml
[tiles.metric]
prometheus_listen_port = 7999sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric representing a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_slow_count | counter | The number of times the consumer was detected as rate limiting consumer by the producer. |
| link_consumed_count | counter | The number of times the link reader has consumed a fragment. |
| link_consumed_size_bytes | counter | The total number of bytes read by the link consumer. |
| link_filtered_count | counter | The number of fragments that were filtered and not consumed. |
| link_filtered_size_bytes | counter | The total number of bytes read by the link consumer that were filtered. |
| link_overrun_polling_count | counter | The number of times the link has been overrun while polling. |
| link_overrun_polling_frag_count | counter | The number of fragments the link has not processed because it was overrun while polling. |
| link_overrun_reading_count | counter | The number of input overruns detected while reading metadata by the consumer. |
| link_overrun_reading_frag_count | counter | The number of fragments the link has not processed because it was overrun while reading. |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | The process ID of the tile. |
| tile_tid | gauge | The thread ID of the tile. Always the same as the Pid in production, but might be different in development. |
| tile_context_switch_involuntary_count | counter | The number of involuntary context switches. |
| tile_context_switch_voluntary_count | counter | The number of voluntary context switches. |
| tile_status | gauge | The current status of the tile. 0 is booting, 1 is running. 2 is shutdown |
| tile_heartbeat | gauge | The last UNIX timestamp in nanoseconds that the tile heartbeated. |
| tile_in_backpressure | gauge | Whether the tile is currently backpressured or not, either 1 or 0. |
| tile_backpressure_count | counter | Number of times the tile has had to wait for one of more consumers to catch up to resume publishing. |
| tile_regime_duration_nanos {tile_regime="caught_up_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="processing_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="backpressure_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Backpressure + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="caught_up_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Prefrag) |
| tile_regime_duration_nanos {tile_regime="processing_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Prefrag) |
| tile_regime_duration_nanos {tile_regime="backpressure_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Backpressure + Prefrag) |
| tile_regime_duration_nanos {tile_regime="caught_up_postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Postfrag) |
| tile_regime_duration_nanos {tile_regime="processing_postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Postfrag) |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_rx_pkt_cnt | counter | Packet receive count. |
| net_rx_bytes_total | counter | Total number of bytes received (including Ethernet header). |
| net_rx_undersz_cnt | counter | Number of incoming packets dropped due to being too small. |
| net_rx_fill_blocked_cnt | counter | Number of incoming packets dropped due to fill ring being full. |
| net_rx_backpressure_cnt | counter | Number of incoming packets dropped due to backpressure. |
| net_rx_busy_cnt | gauge | Number of receive buffers currently busy. |
| net_rx_idle_cnt | gauge | Number of receive buffers currently idle. |
| net_tx_submit_cnt | counter | Number of packet transmit jobs submitted. |
| net_tx_complete_cnt | counter | Number of packet transmit jobs marked as completed by the kernel. |
| net_tx_bytes_total | counter | Total number of bytes transmitted (including Ethernet header). |
| net_tx_route_fail_cnt | counter | Number of packet transmit jobs dropped due to route failure. |
| net_tx_neighbor_fail_cnt | counter | Number of packet transmit jobs dropped due to unresolved neighbor. |
| net_tx_full_fail_cnt | counter | Number of packet transmit jobs dropped due to XDP TX ring full or missing completions. |
| net_tx_busy_cnt | gauge | Number of transmit buffers currently busy. |
| net_tx_idle_cnt | gauge | Number of transmit buffers currently idle. |
| net_xsk_tx_wakeup_cnt | counter | Number of XSK sendto syscalls dispatched. |
| net_xsk_rx_wakeup_cnt | counter | Number of XSK recvmsg syscalls dispatched. |
| net_xdp_rx_dropped_other | counter | xdp_statistics_v0.rx_dropped: Dropped for other reasons |
| net_xdp_rx_invalid_descs | counter | xdp_statistics_v0.rx_invalid_descs: Dropped due to invalid descriptor |
| net_xdp_tx_invalid_descs | counter | xdp_statistics_v0.tx_invalid_descs: Dropped due to invalid descriptor |
| net_xdp_rx_ring_full | counter | xdp_statistics_v1.rx_ring_full: Dropped due to rx ring being full |
| net_xdp_rx_fill_ring_empty_descs | counter | xdp_statistics_v1.rx_fill_ring_empty_descs: Failed to retrieve item from fill ring |
| net_xdp_tx_ring_empty_descs | counter | xdp_statistics_v1.tx_ring_empty_descs: Failed to retrieve item from tx ring |
| net_rx_gre_cnt | counter | Number of valid GRE packets received |
| net_rx_gre_invalid_cnt | counter | Number of invalid GRE packets received |
| net_rx_gre_ignored_cnt | counter | Number of received but ignored GRE packets |
| net_tx_gre_cnt | counter | Number of GRE packet transmit jobs submitted |
| net_tx_gre_route_fail_cnt | counter | Number of GRE packets transmit jobs dropped due to route failure |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_txns_overrun | counter | Count of txns overrun before reassembled (too small txn_reassembly_count). |
| quic_txn_reasms_started | counter | Count of fragmented txn receive ops started. |
| quic_txn_reasms_active | gauge | Number of fragmented txn receive ops currently active. |
| quic_frags_ok | counter | Count of txn frags received |
| quic_frags_gap | counter | Count of txn frags dropped due to data gap |
| quic_frags_dup | counter | Count of txn frags dropped due to dup (stream already completed) |
| quic_txns_received {tpu_recv_type="udp"} | counter | Count of txns received via TPU. (TPU/UDP) |
| quic_txns_received {tpu_recv_type="quic_fast"} | counter | Count of txns received via TPU. (TPU/QUIC unfragmented) |
| quic_txns_received {tpu_recv_type="quic_frag"} | counter | Count of txns received via TPU. (TPU/QUIC fragmented) |
| quic_txns_abandoned | counter | Count of txns abandoned because a conn was lost. |
| quic_txn_undersz | counter | Count of txns received via QUIC dropped because they were too small. |
| quic_txn_oversz | counter | Count of txns received via QUIC dropped because they were too large. |
| quic_legacy_txn_undersz | counter | Count of packets received on the non-QUIC port that were too small to be a valid IP packet. |
| quic_legacy_txn_oversz | counter | Count of packets received on the non-QUIC port that were too large to be a valid transaction. |
| quic_received_packets | counter | Number of IP packets received. |
| quic_received_bytes | counter | Total bytes received (including IP, UDP, QUIC headers). |
| quic_sent_packets | counter | Number of IP packets sent. |
| quic_sent_bytes | counter | Total bytes sent (including IP, UDP, QUIC headers). |
| quic_connections_alloc | gauge | The number of currently allocated QUIC connections. |
| quic_connections_state {quic_conn_state="invalid"} | gauge | The number of QUIC connections in each state. (freed) |
| quic_connections_state {quic_conn_state="handshake"} | gauge | The number of QUIC connections in each state. (handshaking peer) |
| quic_connections_state {quic_conn_state="handshake_complete"} | gauge | The number of QUIC connections in each state. (handshake complete, confirming with peer) |
| quic_connections_state {quic_conn_state="active"} | gauge | The number of QUIC connections in each state. (active connection) |
| quic_connections_state {quic_conn_state="peer_close"} | gauge | The number of QUIC connections in each state. (peer requested close) |
| quic_connections_state {quic_conn_state="abort"} | gauge | The number of QUIC connections in each state. (connection terminating due to error) |
| quic_connections_state {quic_conn_state="close_pending"} | gauge | The number of QUIC connections in each state. (connection is closing) |
| quic_connections_state {quic_conn_state="dead"} | gauge | The number of QUIC connections in each state. (connection about to be freed) |
| quic_connections_created | counter | The total number of connections that have been created. |
| quic_connections_closed | counter | Number of connections gracefully closed. |
| quic_connections_aborted | counter | Number of connections aborted. |
| quic_connections_timed_out | counter | Number of connections timed out. |
| quic_connections_retried | counter | Number of connections established with retry. |
| quic_connection_error_no_slots | counter | Number of connections that failed to create due to lack of slots. |
| quic_connection_error_retry_fail | counter | Number of connections that failed during retry (e.g. invalid token). |
| quic_pkt_no_conn {quic_pkt_handle="initial"} | counter | Number of packets with an unknown connection ID. (initial) |
| quic_pkt_no_conn {quic_pkt_handle="retry"} | counter | Number of packets with an unknown connection ID. (retry) |
| quic_pkt_no_conn {quic_pkt_handle="handshake"} | counter | Number of packets with an unknown connection ID. (handshake) |
| quic_pkt_no_conn {quic_pkt_handle="one_rtt"} | counter | Number of packets with an unknown connection ID. (1-RTT) |
| quic_frame_tx_alloc {frame_tx_alloc_result="success"} | counter | Results of attempts to acquire QUIC frame metadata. (Success) |
| quic_frame_tx_alloc {frame_tx_alloc_result="fail_empty_pool"} | counter | Results of attempts to acquire QUIC frame metadata. (PktMetaPoolEmpty) |
| quic_frame_tx_alloc {frame_tx_alloc_result="fail_conn_max"} | counter | Results of attempts to acquire QUIC frame metadata. (ConnMaxedInflightFrames) |
| quic_initial_token_len {quic_initial_token_len="zero"} | counter | Number of Initial packets grouped by token length. (No token) |
| quic_initial_token_len {quic_initial_token_len="fd_quic_len"} | counter | Number of Initial packets grouped by token length. (fd_quic retry token length) |
| quic_initial_token_len {quic_initial_token_len="invalid_len"} | counter | Number of Initial packets grouped by token length. (Invalid token length) |
| quic_handshakes_created | counter | Number of handshake flows created. |
| quic_handshake_error_alloc_fail | counter | Number of handshakes dropped due to alloc fail. |
| quic_handshake_evicted | counter | Number of handshakes dropped due to eviction. |
| quic_stream_received_events | counter | Number of stream RX events. |
| quic_stream_received_bytes | counter | Total stream payload bytes received. |
| quic_received_frames {quic_frame_type="unknown"} | counter | Number of QUIC frames received. (Unknown frame type) |
| quic_received_frames {quic_frame_type="ack"} | counter | Number of QUIC frames received. (ACK frame) |
| quic_received_frames {quic_frame_type="reset_stream"} | counter | Number of QUIC frames received. (RESET_STREAM frame) |
| quic_received_frames {quic_frame_type="stop_sending"} | counter | Number of QUIC frames received. (STOP_SENDING frame) |
| quic_received_frames {quic_frame_type="crypto"} | counter | Number of QUIC frames received. (CRYPTO frame) |
| quic_received_frames {quic_frame_type="new_token"} | counter | Number of QUIC frames received. (NEW_TOKEN frame) |
| quic_received_frames {quic_frame_type="stream"} | counter | Number of QUIC frames received. (STREAM frame) |
| quic_received_frames {quic_frame_type="max_data"} | counter | Number of QUIC frames received. (MAX_DATA frame) |
| quic_received_frames {quic_frame_type="max_stream_data"} | counter | Number of QUIC frames received. (MAX_STREAM_DATA frame) |
| quic_received_frames {quic_frame_type="max_streams"} | counter | Number of QUIC frames received. (MAX_STREAMS frame) |
| quic_received_frames {quic_frame_type="data_blocked"} | counter | Number of QUIC frames received. (DATA_BLOCKED frame) |
| quic_received_frames {quic_frame_type="stream_data_blocked"} | counter | Number of QUIC frames received. (STREAM_DATA_BLOCKED frame) |
| quic_received_frames {quic_frame_type="streams_blocked"} | counter | Number of QUIC frames received. (STREAMS_BLOCKED(bidi) frame) |
| quic_received_frames {quic_frame_type="new_conn_id"} | counter | Number of QUIC frames received. (NEW_CONN_ID frame) |
| quic_received_frames {quic_frame_type="retire_conn_id"} | counter | Number of QUIC frames received. (RETIRE_CONN_ID frame) |
| quic_received_frames {quic_frame_type="path_challenge"} | counter | Number of QUIC frames received. (PATH_CHALLENGE frame) |
| quic_received_frames {quic_frame_type="path_response"} | counter | Number of QUIC frames received. (PATH_RESPONSE frame) |
| quic_received_frames {quic_frame_type="conn_close_quic"} | counter | Number of QUIC frames received. (CONN_CLOSE(transport) frame) |
| quic_received_frames {quic_frame_type="conn_close_app"} | counter | Number of QUIC frames received. (CONN_CLOSE(app) frame) |
| quic_received_frames {quic_frame_type="handshake_done"} | counter | Number of QUIC frames received. (HANDSHAKE_DONE frame) |
| quic_received_frames {quic_frame_type="ping"} | counter | Number of QUIC frames received. (PING frame) |
| quic_received_frames {quic_frame_type="padding"} | counter | Number of QUIC frames received. (PADDING frame) |
| quic_ack_tx {quic_ack_tx="noop"} | counter | ACK events (non-ACK-eliciting packet) |
| quic_ack_tx {quic_ack_tx="new"} | counter | ACK events (new ACK range) |
| quic_ack_tx {quic_ack_tx="merged"} | counter | ACK events (merged into existing ACK range) |
| quic_ack_tx {quic_ack_tx="drop"} | counter | ACK events (out of buffers) |
| quic_ack_tx {quic_ack_tx="cancel"} | counter | ACK events (ACK suppressed by handler) |
| quic_service_duration_seconds | histogram | Duration spent in service |
| quic_receive_duration_seconds | histogram | Duration spent processing packets |
| quic_frame_fail_parse | counter | Number of QUIC frames failed to parse. |
| quic_pkt_crypto_failed {quic_enc_level="initial"} | counter | Number of packets that failed decryption. (initial) |
| quic_pkt_crypto_failed {quic_enc_level="early"} | counter | Number of packets that failed decryption. (early data) |
| quic_pkt_crypto_failed {quic_enc_level="handshake"} | counter | Number of packets that failed decryption. (handshake) |
| quic_pkt_crypto_failed {quic_enc_level="app"} | counter | Number of packets that failed decryption. (app data) |
| quic_pkt_no_key {quic_enc_level="initial"} | counter | Number of packets that failed decryption due to missing key. (initial) |
| quic_pkt_no_key {quic_enc_level="early"} | counter | Number of packets that failed decryption due to missing key. (early data) |
| quic_pkt_no_key {quic_enc_level="handshake"} | counter | Number of packets that failed decryption due to missing key. (handshake) |
| quic_pkt_no_key {quic_enc_level="app"} | counter | Number of packets that failed decryption due to missing key. (app data) |
| quic_pkt_net_header_invalid | counter | Number of packets dropped due to weird IP or UDP header. |
| quic_pkt_quic_header_invalid | counter | Number of packets dropped due to weird QUIC header. |
| quic_pkt_undersz | counter | Number of QUIC packets dropped due to being too small. |
| quic_pkt_oversz | counter | Number of QUIC packets dropped due to being too large. |
| quic_pkt_verneg | counter | Number of QUIC version negotiation packets received. |
| quic_retry_sent | counter | Number of QUIC Retry packets sent. |
| quic_pkt_retransmissions {quic_enc_level="initial"} | counter | Number of QUIC packets that retransmitted. (initial) |
| quic_pkt_retransmissions {quic_enc_level="early"} | counter | Number of QUIC packets that retransmitted. (early data) |
| quic_pkt_retransmissions {quic_enc_level="handshake"} | counter | Number of QUIC packets that retransmitted. (handshake) |
| quic_pkt_retransmissions {quic_enc_level="app"} | counter | Number of QUIC packets that retransmitted. (app data) |
Bundle Tile
| Metric | Type | Description |
|---|---|---|
| bundle_transaction_received | counter | Total count of transactions received, including transactions within bundles |
| bundle_packet_received | counter | Total count of packets received |
| bundle_bundle_received | counter | Total count of bundles received |
| bundle_errors {bundle_error="protobuf"} | counter | Number of gRPC errors encountered (Protobuf decode/encode error) |
| bundle_errors {bundle_error="transport"} | counter | Number of gRPC errors encountered (Transport error) |
| bundle_errors {bundle_error="timeout"} | counter | Number of gRPC errors encountered (I/O timeout) |
| bundle_errors {bundle_error="no_fee_info"} | counter | Number of gRPC errors encountered (Bundle dropped due to missing fee info) |
| bundle_errors {bundle_error="ssl_alloc"} | counter | Number of gRPC errors encountered (OpenSSL alloc fail) |
| bundle_heap_size | gauge | Workspace heap size |
| bundle_heap_free_bytes | gauge | Approx free space in workspace |
| bundle_shredstream_heartbeats | counter | Number of ShredStream heartbeats successfully sent |
| bundle_keepalives | counter | Number of HTTP/2 PINGs acknowledged by server |
| bundle_connected | gauge | 1 if connected to the bundle server, 0 if not |
| bundle_rtt_sample | gauge | Latest RTT sample at scrape time (nanoseconds) |
| bundle_rtt_smoothed | gauge | RTT moving average (nanoseconds) |
| bundle_rtt_var | gauge | RTT variance (nanoseconds) |
| bundle_message_rx_delay_nanos | histogram | Message receive delay in nanoseconds from bundle server to bundle client |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_transaction_bundle_peer_failure | counter | Count of transactions that failed to verify because a peer transaction in the bundle failed |
| verify_transaction_parse_failure | counter | Count of transactions that failed to parse |
| verify_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
| verify_transaction_verify_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
| verify_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_transaction_bundle_peer_failure | counter | Count of transactions that failed to dedup because a peer transaction in the bundle failed |
| dedup_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the dedup stage |
| dedup_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Resolv Tile
| Metric | Type | Description |
|---|---|---|
| resolv_no_bank_drop | counter | Count of transactions dropped because the bank was not available |
| resolv_stash_operation {resolve_stash_operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolv_stash_operation {resolve_stash_operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolv_stash_operation {resolve_stash_operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolv_stash_operation {resolve_stash_operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories.) |
| resolv_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolv_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolv_lut_resolved {lut_resolve_result="account_not_found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolv_lut_resolved {lut_resolve_result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) |
| resolv_blockhash_expired | counter | Count of transactions that failed to resolve because the blockhash was expired |
| resolv_transaction_bundle_peer_failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_schedule_microblock_duration_seconds | histogram | Duration of scheduling one microblock |
| pack_no_sched_microblock_duration_seconds | histogram | Duration of discovering that there are no schedulable transactions |
| pack_insert_transaction_duration_seconds | histogram | Duration of inserting one transaction into the pool of available transactions |
| pack_complete_microblock_duration_seconds | histogram | Duration of the computation associated with marking one microblock as complete |
| pack_total_transactions_per_microblock_count | histogram | Count of transactions in a scheduled microblock, including both votes and non-votes |
| pack_votes_per_microblock_count | histogram | Count of simple vote transactions in a scheduled microblock |
| pack_normal_transaction_received | counter | Count of transactions received via the normal TPU path |
| pack_transaction_inserted {pack_txn_insert_return="nonce_conflict"} | counter | Result of inserting a transaction into the pack object (Bundle with two conflicting durable nonce transactions) |
| pack_transaction_inserted {pack_txn_insert_return="bundle_blacklist"} | counter | Result of inserting a transaction into the pack object (Transaction uses an account on the bundle blacklist) |
| pack_transaction_inserted {pack_txn_insert_return="invalid_nonce"} | counter | Result of inserting a transaction into the pack object (Transaction is an invalid durable nonce transaction) |
| pack_transaction_inserted {pack_txn_insert_return="write_sysvar"} | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) |
| pack_transaction_inserted {pack_txn_insert_return="estimation_fail"} | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) |
| pack_transaction_inserted {pack_txn_insert_return="duplicate_account"} | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) |
| pack_transaction_inserted {pack_txn_insert_return="too_many_accounts"} | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) |
| pack_transaction_inserted {pack_txn_insert_return="too_large"} | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) |
| pack_transaction_inserted {pack_txn_insert_return="expired"} | counter | Result of inserting a transaction into the pack object (Transaction already expired) |
| pack_transaction_inserted {pack_txn_insert_return="addr_lut"} | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) |
| pack_transaction_inserted {pack_txn_insert_return="unaffordable"} | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) |
| pack_transaction_inserted {pack_txn_insert_return="duplicate"} | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce) |
| pack_transaction_inserted {pack_txn_insert_return="priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_transaction_inserted {pack_txn_insert_return="nonvote_add"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="vote_add"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) |
| pack_transaction_inserted {pack_txn_insert_return="vote_replace"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_nonvote_add"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="unused"} | counter | Result of inserting a transaction into the pack object (Unused because durable nonce transactions can't be simple votes) |
| pack_transaction_inserted {pack_txn_insert_return="nonce_nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and wasn't leader) |
| pack_metric_timing {pack_timing_state="txn_no_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, but wasn't leader or had hit a limit) |
| pack_metric_timing {pack_timing_state="no_txn_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, had banks but wasn't leader) |
| pack_metric_timing {pack_timing_state="txn_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, had banks but wasn't leader) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and was leader but had no available banks) |
| pack_metric_timing {pack_timing_state="txn_no_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, was leader, but had no available banks) |
| pack_metric_timing {pack_timing_state="no_txn_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had available banks but no transactions) |
| pack_metric_timing {pack_timing_state="txn_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had banks and transactions available but couldn't schedule anything non-conflicting) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="txn_no_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="no_txn_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="txn_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy) |
| pack_metric_timing {pack_timing_state="txn_no_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy) |
| pack_metric_timing {pack_timing_state="no_txn_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock and now has no transactions) |
| pack_metric_timing {pack_timing_state="txn_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock) |
| pack_transaction_dropped_from_extra | counter | Transactions dropped from the extra transaction storage because it was full |
| pack_transaction_inserted_to_extra | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full |
| pack_transaction_inserted_from_extra | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage |
| pack_transaction_expired | counter | Transactions deleted from pack because their TTL expired |
| pack_transaction_deleted | counter | Transactions dropped from pack because they were requested to be deleted |
| pack_transaction_dropped_partial_bundle | counter | Transactions dropped from pack because they were part of a partial bundle |
| pack_available_transactions {avail_txn_type="all"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (All transactions in any treap) |
| pack_available_transactions {avail_txn_type="regular"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes in the main treap) |
| pack_available_transactions {avail_txn_type="votes"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Simple votes) |
| pack_available_transactions {avail_txn_type="conflicting"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes that write to a hotly-contended account) |
| pack_available_transactions {avail_txn_type="bundles"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Transactions that are part of a bundle) |
| pack_pending_transactions_heap_size | gauge | The maximum number of pending transactions that pack can consider. This value is fixed at Firedancer startup but is a useful reference for AvailableTransactions. |
| pack_smallest_pending_transaction | gauge | A lower bound on the smallest non-vote transaction (in cost units) that is immediately available for scheduling |
| pack_microblock_per_block_limit | counter | The number of times pack did not pack a microblock because the limit on microblocks/block had been reached |
| pack_data_per_block_limit | counter | The number of times pack did not pack a microblock because it reached the data per block limit at the start of trying to schedule a microblock |
| pack_transaction_schedule {pack_txn_schedule="taken"} | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) |
| pack_transaction_schedule {pack_txn_schedule="cu_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_transaction_schedule {pack_txn_schedule="fast_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) |
| pack_transaction_schedule {pack_txn_schedule="byte_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_transaction_schedule {pack_txn_schedule="write_cost"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_transaction_schedule {pack_txn_schedule="slow_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) |
| pack_transaction_schedule {pack_txn_schedule="defer_skip"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction it previously exceeded the per-account block write cost limit too many times) |
| pack_bundle_crank_status {bundle_crank_result="not_needed"} | counter | Result of considering whether bundle cranks are needed (On-chain state in the correct state) |
| pack_bundle_crank_status {bundle_crank_result="inserted"} | counter | Result of considering whether bundle cranks are needed (Inserted an initializer bundle to update the on-chain state) |
| pack_bundle_crank_status {bundle_crank_result="creation_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but creation failed) |
| pack_bundle_crank_status {bundle_crank_result="insertion_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but insertion failed) |
| pack_cus_consumed_in_block | gauge | The number of cost units consumed in the current block, or 0 if pack is not currently packing a block |
| pack_cus_scheduled | histogram | The number of cost units scheduled for each block pack produced. This can be higher than the block limit because of returned CUs. |
| pack_cus_rebated | histogram | The number of compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses. |
| pack_cus_net | histogram | The net number of cost units (scheduled - rebated) in each block pack produced. |
| pack_cus_pct | histogram | The percent of the total block cost limit used for each block pack produced. |
| pack_delete_missed | counter | Count of attempts to delete a transaction that wasn't found |
| pack_delete_hit | counter | Count of attempts to delete a transaction that was found and deleted |
Bank Tile
| Metric | Type | Description |
|---|---|---|
| bank_transaction_sanitize_failure | counter | Number of transactions that failed to sanitize. |
| bank_transaction_not_executed_failure | counter | Number of transactions that did not execute. This is different than transactions which fail to execute, which make it onto the chain. |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="success"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (Success) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="slot_hashes_sysvar_not_found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The slot hashes syvar could not be found.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="account_not_found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account storing the address lookup table was deactivated or could not be found.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="invalid_account_owner"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account that owns the referenced lookup table is not the address lookup table program.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="invalid_account_data"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The data for the referenced address lookup table is malformed.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="invalid_index"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The referenced index in the address lookup table does not exist.) |
| bank_transaction_result {transaction_error="success"} | counter | Result of loading and executing a transaction. (Success) |
| bank_transaction_result {transaction_error="account_in_use"} | counter | Result of loading and executing a transaction. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_transaction_result {transaction_error="account_loaded_twice"} | counter | Result of loading and executing a transaction. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_transaction_result {transaction_error="account_not_found"} | counter | Result of loading and executing a transaction. (Attempt to debit an account but found no record of a prior credit.) |
| bank_transaction_result {transaction_error="program_account_not_found"} | counter | Result of loading and executing a transaction. (Attempt to load a program that does not exist.) |
| bank_transaction_result {transaction_error="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_transaction_result {transaction_error="invalid_account_for_fee"} | counter | Result of loading and executing a transaction. (This account may not be used to pay transaction fees.) |
| bank_transaction_result {transaction_error="already_processed"} | counter | Result of loading and executing a transaction. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_transaction_result {transaction_error="blockhash_not_found"} | counter | Result of loading and executing a transaction. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_transaction_result {transaction_error="instruction_error"} | counter | Result of loading and executing a transaction. (An error occurred while processing an instruction.) |
| bank_transaction_result {transaction_error="call_chain_too_deep"} | counter | Result of loading and executing a transaction. (Loader call chain is too deep.) |
| bank_transaction_result {transaction_error="missing_signature_for_fee"} | counter | Result of loading and executing a transaction. (Transaction requires a fee but has no signature present.) |
| bank_transaction_result {transaction_error="invalid_account_index"} | counter | Result of loading and executing a transaction. (Transaction contains an invalid account reference.) |
| bank_transaction_result {transaction_error="signature_failure"} | counter | Result of loading and executing a transaction. (Transaction did not pass signature verification.) |
| bank_transaction_result {transaction_error="invalid_program_for_execution"} | counter | Result of loading and executing a transaction. (This program may not be used for executing instructions.) |
| bank_transaction_result {transaction_error="sanitize_failure"} | counter | Result of loading and executing a transaction. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_transaction_result {transaction_error="cluster_maintenance"} | counter | Result of loading and executing a transaction. (Transactions are currently disabled due to cluster maintenance.) |
| bank_transaction_result {transaction_error="account_borrow_outstanding"} | counter | Result of loading and executing a transaction. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_transaction_result {transaction_error="would_exceed_max_block_cost_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed max Block Cost Limit.) |
| bank_transaction_result {transaction_error="unsupported_version"} | counter | Result of loading and executing a transaction. (Transaction version is unsupported.) |
| bank_transaction_result {transaction_error="invalid_writable_account"} | counter | Result of loading and executing a transaction. (Transaction loads a writable account that cannot be written.) |
| bank_transaction_result {transaction_error="would_exceed_max_account_cost_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed max account limit within the block.) |
| bank_transaction_result {transaction_error="would_exceed_account_data_block_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed account data limit within the block.) |
| bank_transaction_result {transaction_error="too_many_account_locks"} | counter | Result of loading and executing a transaction. (Transaction locked too many accounts.) |
| bank_transaction_result {transaction_error="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction. (Address lookup table not found.) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction. (Attempted to lookup addresses from an invalid account.) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction. (Address table lookup uses an invalid index.) |
| bank_transaction_result {transaction_error="invalid_rent_paying_account"} | counter | Result of loading and executing a transaction. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_transaction_result {transaction_error="would_exceed_max_vote_cost_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed max Vote Cost Limit.) |
| bank_transaction_result {transaction_error="would_exceed_account_data_total_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed total account data limit.) |
| bank_transaction_result {transaction_error="duplicate_instruction"} | counter | Result of loading and executing a transaction. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_transaction_result {transaction_error="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction. (Transaction results in an account with insufficient funds for rent.) |
| bank_transaction_result {transaction_error="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction. (Transaction exceeded max loaded accounts data size cap.) |
| bank_transaction_result {transaction_error="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_transaction_result {transaction_error="resanitization_needed"} | counter | Result of loading and executing a transaction. (Sanitized transaction differed before/after feature activation. Needs to be resanitized.) |
| bank_transaction_result {transaction_error="program_execution_temporarily_restricted"} | counter | Result of loading and executing a transaction. (Program execution is temporarily restricted on an account.) |
| bank_transaction_result {transaction_error="unbalanced_transaction"} | counter | Result of loading and executing a transaction. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_transaction_result {transaction_error="program_cache_hit_max_limit"} | counter | Result of loading and executing a transaction. (The total program cache size hit the maximum allowed limit.) |
| bank_transaction_result {transaction_error="commit_cancelled"} | counter | Result of loading and executing a transaction. (The process for committing the transaction was cancelled internally.) |
| bank_transaction_result {transaction_error="bundle_peer"} | counter | Result of loading and executing a transaction. (Transaction is part of a bundle and one of the peer transactions failed.) |
| bank_processing_failed | counter | Count of transactions for which the processing stage failed and won't land on chain |
| bank_fee_only_transactions | counter | Count of transactions that will land on chain but without executing |
| bank_executed_failed_transactions | counter | Count of transactions that execute on chain but failed |
| bank_successful_transactions | counter | Count of transactions that execute on chain and succeed |
| bank_cost_model_undercount | counter | Count of transactions that used more CUs than the cost model should have permitted them to |
Poh Tile
| Metric | Type | Description |
|---|---|---|
| poh_begin_leader_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the bank. |
| poh_first_microblock_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock. |
| poh_slot_done_delay_seconds | histogram | Delay between when we become leader in a slot and when we finish the slot. |
| poh_bundle_initialize_delay_seconds | histogram | Delay in starting the slot caused by loading the information needed to generate the bundle crank transactions |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_cluster_contact_info_cnt | histogram | Number of contact infos in the cluster contact info message |
| shred_microblocks_abandoned | counter | The number of microblocks that were abandoned because we switched slots without finishing the current slot |
| shred_invalid_block_id | counter | The number of times a block was created with unknown parent block_id |
| shred_batch_sz | histogram | The size (in bytes) of each microblock batch that is shredded |
| shred_batch_microblock_cnt | histogram | The number of microblocks in each microblock batch that is shredded |
| shred_shredding_duration_seconds | histogram | Duration of producing one FEC set from the shredder |
| shred_add_shred_duration_seconds | histogram | Duration of verifying and processing one shred received from the network |
| shred_shred_processed {shred_processing_result="bad_slot"} | counter | The result of processing a shred from the network (Shred was for a slot for which we don't know the leader) |
| shred_shred_processed {shred_processing_result="parse_failed"} | counter | The result of processing a shred from the network (Shred parsing failed) |
| shred_shred_processed {shred_processing_result="rejected"} | counter | The result of processing a shred from the network (Shred was invalid for one of many reasons) |
| shred_shred_processed {shred_processing_result="ignored"} | counter | The result of processing a shred from the network (Shred was ignored because we had already received or reconstructed it) |
| shred_shred_processed {shred_processing_result="okay"} | counter | The result of processing a shred from the network (Shred accepted to an incomplete FEC set) |
| shred_shred_processed {shred_processing_result="completes"} | counter | The result of processing a shred from the network (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_set_spilled | counter | The number of FEC sets that were spilled because they didn't complete in time and we needed space |
| shred_shred_rejected_initial | counter | The number of shreds that were rejected before any resources were allocated for the FEC set |
| shred_shred_rejected_unchained | counter | The number of shreds that were rejected because they're not chained merkle shreds |
| shred_fec_rejected_fatal | counter | The number of FEC sets that were rejected for reasons that cause the whole FEC set to become invalid |
| shred_force_complete_request | counter | The number of times we received a FEC force complete message |
| shred_force_complete_failure | counter | The number of times we failed to force complete a FEC set on request |
| shred_force_complete_success | counter | The number of times we successfully forced completed a FEC set on request |
| shred_shred_out_rcv | counter | The number of times we received a repair shred |
| shred_shred_turbine_rcv | counter | The number of times we received a turbine shred |
| shred_store_insert_wait | histogram | Time in seconds spent waiting for the store to insert a new FEC set |
| shred_store_insert_work | histogram | Time in seconds spent on inserting a new FEC set |
Store Tile
| Metric | Type | Description |
|---|---|---|
| store_transactions_inserted | counter | Count of transactions produced while we were leader in the shreds that have been inserted so far |
Sign Tile
| Metric | Type | Description |
|---|---|---|
| sign_sign_duration_seconds | histogram | Duration of signing a message |
Metric Tile
| Metric | Type | Description |
|---|---|---|
| metric_boot_timestamp_nanos | gauge | Timestamp when validator was started (nanoseconds since epoch) |
Replay Tile
| Metric | Type | Description |
|---|---|---|
| replay_store_link_wait | histogram | Time in seconds spent waiting for the store to link a new FEC set |
| replay_store_link_work | histogram | Time in seconds spent on linking a new FEC set |
| replay_store_read_wait | histogram | Time in seconds spent waiting for the store to read a FEC set |
| replay_store_read_work | histogram | Time in seconds spent on reading a FEC set |
| replay_store_publish_wait | histogram | Time in seconds spent waiting for the store to publish a new FEC set |
| replay_store_publish_work | histogram | Time in seconds spent on publishing a new FEC set |
| replay_root_slot | gauge | The slot at which our node has most recently rooted |
| replay_root_distance | gauge | The distance in slots between our current root and the current reset slot |
| replay_leader_slot | gauge | The slot at which we are currently leader, or 0 if none |
| replay_next_leader_slot | gauge | The slot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot |
| replay_reset_slot | gauge | The slot at which we last reset the replay stage, or 0 if unknown |
| replay_max_live_banks | gauge | The maximum number of banks we can have alive |
| replay_live_banks | gauge | The number of banks we currently have alive |
| replay_slots_total | counter | Count of slots replayed successfully |
| replay_transactions_total | counter | Count of transactions processed overall on the current fork |
Storei Tile
| Metric | Type | Description |
|---|---|---|
| storei_first_turbine_slot | gauge | |
| storei_current_turbine_slot | gauge |
Gossip Tile
| Metric | Type | Description |
|---|---|---|
| gossip_ping_tracker_capacity | counter | |
| gossip_ping_tracker_count {ping_tracked_type="unpinged"} | gauge | (Peer is known but has not been pinged yet) |
| gossip_ping_tracker_count {ping_tracked_type="invalid"} | gauge | (Peer was pinged but has not yet responded, or responded with an invalid pong) |
| gossip_ping_tracker_count {ping_tracked_type="valid"} | gauge | (Peer was pinged and is currently valid) |
| gossip_ping_tracker_count {ping_tracked_type="valid_refreshing"} | gauge | (Peer was pinged and is currently valid, and is being refreshed with a new ping) |
| gossip_ping_tracker_pong_result {pong_response="staked"} | counter | (Pong ignored as node is staked) |
| gossip_ping_tracker_pong_result {pong_response="entrypoint"} | counter | (Pong ignored as node is an entrypoint) |
| gossip_ping_tracker_pong_result {pong_response="untracked"} | counter | (Pong ignored as node has no outstanding ping) |
| gossip_ping_tracker_pong_result {pong_response="address"} | counter | (Pong ignored as responding node doesn't match pinged address) |
| gossip_ping_tracker_pong_result {pong_response="token"} | counter | (Pong ignored as token was not valid) |
| gossip_ping_tracker_pong_result {pong_response="success"} | counter | (Pong accepted) |
| gossip_ping_tracker_evicted_count | counter | |
| gossip_ping_tracked_count | counter | |
| gossip_ping_tracker_stake_changed_count | counter | |
| gossip_ping_tracker_address_changed_count | counter | |
| gossip_crds_capacity | gauge | |
| gossip_crds_count {crds_value="contact_info_v1"} | gauge | (Contact Info V1) |
| gossip_crds_count {crds_value="vote"} | gauge | (Vote) |
| gossip_crds_count {crds_value="lowest_slot"} | gauge | (Lowest Slot) |
| gossip_crds_count {crds_value="snapshot_hashes"} | gauge | (Snapshot Hashes) |
| gossip_crds_count {crds_value="accounts_hashes"} | gauge | (Accounts Hashes) |
| gossip_crds_count {crds_value="epoch_slots"} | gauge | (Epoch Slots) |
| gossip_crds_count {crds_value="version_v1"} | gauge | (Version V1) |
| gossip_crds_count {crds_value="version_v2"} | gauge | (Version V2) |
| gossip_crds_count {crds_value="node_instance"} | gauge | (Node Instance) |
| gossip_crds_count {crds_value="duplicate_shred"} | gauge | (Duplicate Shred) |
| gossip_crds_count {crds_value="incremental_snapshot_hashes"} | gauge | (Incremental Snapshot Hashes) |
| gossip_crds_count {crds_value="contact_info_v2"} | gauge | (Contact Info V2) |
| gossip_crds_count {crds_value="restart_last_voted_fork_slots"} | gauge | (Restart Last Voted Fork Slots) |
| gossip_crds_count {crds_value="restart_heaviest_fork"} | gauge | (Restart Heaviest Fork) |
| gossip_crds_expired_count | counter | |
| gossip_crds_evicted_count | counter | |
| gossip_crds_peer_capacity | gauge | |
| gossip_crds_peer_staked_count | gauge | |
| gossip_crds_peer_unstaked_count | gauge | |
| gossip_crds_peer_total_stake | gauge | |
| gossip_crds_peer_evicted_count | counter | |
| gossip_crds_purged_capacity | gauge | |
| gossip_crds_purged_count | gauge | |
| gossip_crds_purged_expired_count | counter | |
| gossip_crds_purged_evicted_count | counter | |
| gossip_contact_info_unrecognized_socket_tags | counter | |
| gossip_contact_info_ipv6 | counter | |
| gossip_crds_rx_count {gossip_crds_outcome="upserted_pull_response"} | counter | (Pull Response (upserted)) |
| gossip_crds_rx_count {gossip_crds_outcome="upserted_push"} | counter | (Push (upserted)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_pull_response_stale"} | counter | (Pull Response (newer entry already present in table)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_pull_response_wallclock"} | counter | (Pull Response (outside expiry window and no contact info entry)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_pull_response_duplicate"} | counter | (Pull Response (duplicate)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_push_stale"} | counter | (Push (newer entry already present in table)) |
| gossip_crds_rx_count {gossip_crds_outcome="dropped_push_duplicate"} | counter | (Push (duplicate)) |
| gossip_message_tx_count {gossip_message="pull_request"} | counter | (Pull Request) |
| gossip_message_tx_count {gossip_message="pull_response"} | counter | (Pull Response) |
| gossip_message_tx_count {gossip_message="push"} | counter | (Push) |
| gossip_message_tx_count {gossip_message="prune"} | counter | (Prune) |
| gossip_message_tx_count {gossip_message="ping"} | counter | (Ping) |
| gossip_message_tx_count {gossip_message="pong"} | counter | (Pong) |
| gossip_message_tx_bytes {gossip_message="pull_request"} | counter | (Pull Request) |
| gossip_message_tx_bytes {gossip_message="pull_response"} | counter | (Pull Response) |
| gossip_message_tx_bytes {gossip_message="push"} | counter | (Push) |
| gossip_message_tx_bytes {gossip_message="prune"} | counter | (Prune) |
| gossip_message_tx_bytes {gossip_message="ping"} | counter | (Ping) |
| gossip_message_tx_bytes {gossip_message="pong"} | counter | (Pong) |
| gossip_crds_tx_push_count {crds_value="contact_info_v1"} | counter | (Contact Info V1) |
| gossip_crds_tx_push_count {crds_value="vote"} | counter | (Vote) |
| gossip_crds_tx_push_count {crds_value="lowest_slot"} | counter | (Lowest Slot) |
| gossip_crds_tx_push_count {crds_value="snapshot_hashes"} | counter | (Snapshot Hashes) |
| gossip_crds_tx_push_count {crds_value="accounts_hashes"} | counter | (Accounts Hashes) |
| gossip_crds_tx_push_count {crds_value="epoch_slots"} | counter | (Epoch Slots) |
| gossip_crds_tx_push_count {crds_value="version_v1"} | counter | (Version V1) |
| gossip_crds_tx_push_count {crds_value="version_v2"} | counter | (Version V2) |
| gossip_crds_tx_push_count {crds_value="node_instance"} | counter | (Node Instance) |
| gossip_crds_tx_push_count {crds_value="duplicate_shred"} | counter | (Duplicate Shred) |
| gossip_crds_tx_push_count {crds_value="incremental_snapshot_hashes"} | counter | (Incremental Snapshot Hashes) |
| gossip_crds_tx_push_count {crds_value="contact_info_v2"} | counter | (Contact Info V2) |
| gossip_crds_tx_push_count {crds_value="restart_last_voted_fork_slots"} | counter | (Restart Last Voted Fork Slots) |
| gossip_crds_tx_push_count {crds_value="restart_heaviest_fork"} | counter | (Restart Heaviest Fork) |
| gossip_crds_tx_push_bytes {crds_value="contact_info_v1"} | counter | (Contact Info V1) |
| gossip_crds_tx_push_bytes {crds_value="vote"} | counter | (Vote) |
| gossip_crds_tx_push_bytes {crds_value="lowest_slot"} | counter | (Lowest Slot) |
| gossip_crds_tx_push_bytes {crds_value="snapshot_hashes"} | counter | (Snapshot Hashes) |
| gossip_crds_tx_push_bytes {crds_value="accounts_hashes"} | counter | (Accounts Hashes) |
| gossip_crds_tx_push_bytes {crds_value="epoch_slots"} | counter | (Epoch Slots) |
| gossip_crds_tx_push_bytes {crds_value="version_v1"} | counter | (Version V1) |
| gossip_crds_tx_push_bytes {crds_value="version_v2"} | counter | (Version V2) |
| gossip_crds_tx_push_bytes {crds_value="node_instance"} | counter | (Node Instance) |
| gossip_crds_tx_push_bytes {crds_value="duplicate_shred"} | counter | (Duplicate Shred) |
| gossip_crds_tx_push_bytes {crds_value="incremental_snapshot_hashes"} | counter | (Incremental Snapshot Hashes) |
| gossip_crds_tx_push_bytes {crds_value="contact_info_v2"} | counter | (Contact Info V2) |
| gossip_crds_tx_push_bytes {crds_value="restart_last_voted_fork_slots"} | counter | (Restart Last Voted Fork Slots) |
| gossip_crds_tx_push_bytes {crds_value="restart_heaviest_fork"} | counter | (Restart Heaviest Fork) |
| gossip_crds_tx_pull_response_count {crds_value="contact_info_v1"} | counter | (Contact Info V1) |
| gossip_crds_tx_pull_response_count {crds_value="vote"} | counter | (Vote) |
| gossip_crds_tx_pull_response_count {crds_value="lowest_slot"} | counter | (Lowest Slot) |
| gossip_crds_tx_pull_response_count {crds_value="snapshot_hashes"} | counter | (Snapshot Hashes) |
| gossip_crds_tx_pull_response_count {crds_value="accounts_hashes"} | counter | (Accounts Hashes) |
| gossip_crds_tx_pull_response_count {crds_value="epoch_slots"} | counter | (Epoch Slots) |
| gossip_crds_tx_pull_response_count {crds_value="version_v1"} | counter | (Version V1) |
| gossip_crds_tx_pull_response_count {crds_value="version_v2"} | counter | (Version V2) |
| gossip_crds_tx_pull_response_count {crds_value="node_instance"} | counter | (Node Instance) |
| gossip_crds_tx_pull_response_count {crds_value="duplicate_shred"} | counter | (Duplicate Shred) |
| gossip_crds_tx_pull_response_count {crds_value="incremental_snapshot_hashes"} | counter | (Incremental Snapshot Hashes) |
| gossip_crds_tx_pull_response_count {crds_value="contact_info_v2"} | counter | (Contact Info V2) |
| gossip_crds_tx_pull_response_count {crds_value="restart_last_voted_fork_slots"} | counter | (Restart Last Voted Fork Slots) |
| gossip_crds_tx_pull_response_count {crds_value="restart_heaviest_fork"} | counter | (Restart Heaviest Fork) |
| gossip_crds_tx_pull_response_bytes {crds_value="contact_info_v1"} | counter | (Contact Info V1) |
| gossip_crds_tx_pull_response_bytes {crds_value="vote"} | counter | (Vote) |
| gossip_crds_tx_pull_response_bytes {crds_value="lowest_slot"} | counter | (Lowest Slot) |
| gossip_crds_tx_pull_response_bytes {crds_value="snapshot_hashes"} | counter | (Snapshot Hashes) |
| gossip_crds_tx_pull_response_bytes {crds_value="accounts_hashes"} | counter | (Accounts Hashes) |
| gossip_crds_tx_pull_response_bytes {crds_value="epoch_slots"} | counter | (Epoch Slots) |
| gossip_crds_tx_pull_response_bytes {crds_value="version_v1"} | counter | (Version V1) |
| gossip_crds_tx_pull_response_bytes {crds_value="version_v2"} | counter | (Version V2) |
| gossip_crds_tx_pull_response_bytes {crds_value="node_instance"} | counter | (Node Instance) |
| gossip_crds_tx_pull_response_bytes {crds_value="duplicate_shred"} | counter | (Duplicate Shred) |
| gossip_crds_tx_pull_response_bytes {crds_value="incremental_snapshot_hashes"} | counter | (Incremental Snapshot Hashes) |
| gossip_crds_tx_pull_response_bytes {crds_value="contact_info_v2"} | counter | (Contact Info V2) |
| gossip_crds_tx_pull_response_bytes {crds_value="restart_last_voted_fork_slots"} | counter | (Restart Last Voted Fork Slots) |
| gossip_crds_tx_pull_response_bytes {crds_value="restart_heaviest_fork"} | counter | (Restart Heaviest Fork) |
Netlnk Tile
| Metric | Type | Description |
|---|---|---|
| netlnk_drop_events | counter | Number of netlink drop events caught |
| netlnk_link_full_syncs | counter | Number of full link table syncs done |
| netlnk_route_full_syncs | counter | Number of full route table syncs done |
| netlnk_updates {netlink_msg="link"} | counter | Number of netlink live updates processed (Link) |
| netlnk_updates {netlink_msg="neigh"} | counter | Number of netlink live updates processed (Neighbor Table Entry) |
| netlnk_updates {netlink_msg="ipv4_route"} | counter | Number of netlink live updates processed (IPv4 Route Table Entry) |
| netlnk_interface_count | gauge | Number of network interfaces |
| netlnk_route_count {route_table="local"} | gauge | Number of IPv4 routes (Local) |
| netlnk_route_count {route_table="main"} | gauge | Number of IPv4 routes (Main) |
| netlnk_neigh_probe_sent | counter | Number of neighbor solicit requests sent to kernel |
| netlnk_neigh_probe_fails | counter | Number of neighbor solicit requests that failed to send (kernel too slow) |
| netlnk_neigh_probe_rate_limit_host | counter | Number of neighbor solicit that exceeded the per-host rate limit |
| netlnk_neigh_probe_rate_limit_global | counter | Number of neighbor solicit that exceeded the global rate limit |
Sock Tile
| Metric | Type | Description |
|---|---|---|
| sock_syscalls_sendmmsg {sock_err="no_error"} | counter | Number of sendmmsg syscalls dispatched (No error) |
| sock_syscalls_sendmmsg {sock_err="slow"} | counter | Number of sendmmsg syscalls dispatched (ENOBUFS, EAGAIN error) |
| sock_syscalls_sendmmsg {sock_err="perm"} | counter | Number of sendmmsg syscalls dispatched (EPERM error (blocked by netfilter)) |
| sock_syscalls_sendmmsg {sock_err="unreach"} | counter | Number of sendmmsg syscalls dispatched (ENETUNREACH, EHOSTUNREACH error) |
| sock_syscalls_sendmmsg {sock_err="down"} | counter | Number of sendmmsg syscalls dispatched (ENONET, ENETDOWN, EHOSTDOWN error) |
| sock_syscalls_sendmmsg {sock_err="other"} | counter | Number of sendmmsg syscalls dispatched (Unrecognized error code) |
| sock_syscalls_recvmmsg | counter | Number of recvmsg syscalls dispatched |
| sock_rx_pkt_cnt | counter | Number of packets received |
| sock_tx_pkt_cnt | counter | Number of packets sent |
| sock_tx_drop_cnt | counter | Number of packets failed to send |
| sock_tx_bytes_total | counter | Total number of bytes transmitted (including Ethernet header). |
| sock_rx_bytes_total | counter | Total number of bytes received (including Ethernet header). |
Repair Tile
| Metric | Type | Description |
|---|---|---|
| repair_total_pkt_count | counter | How many network packets we have sent, including reqs, pings, pongs, etc. |
| repair_sent_pkt_types {repair_sent_request_types="needed_window"} | counter | What types of client messages are we sending (Need Window) |
| repair_sent_pkt_types {repair_sent_request_types="needed_highest_window"} | counter | What types of client messages are we sending (Need Highest Window) |
| repair_sent_pkt_types {repair_sent_request_types="needed_orphan"} | counter | What types of client messages are we sending (Need Orphans) |
| repair_sent_pkt_types {repair_sent_request_types="pong"} | counter | What types of client messages are we sending (Pong) |
| repair_repaired_slots | counter | Until which slots have we fully repaired |
| repair_current_slot | counter | Our view of the current cluster slot, max slot received |
| repair_request_peers | counter | How many peers have we requested |
| repair_sign_tile_unavail | counter | How many times no sign tiles were available to send request |
| repair_eager_repair_aggresses | counter | How many times we pass eager repair threshold |
| repair_slot_complete_time | histogram | Time in seconds it took to complete a slot |
| repair_response_latency | histogram | Time in nanoseconds it took to receive a repair request response |
| repair_sign_duration_seconds | histogram | Duration of signing a message |
Send Tile
| Metric | Type | Description |
|---|---|---|
| send_leader_not_found | counter | Total number of times slot leader not found |
| send_unstaked_ci | counter | Total number of times we received contact info for an unstaked node |
| send_ci_removed | counter | Total number of times we removed contact info |
| send_new_contact_info_quic_vote {new_contact_outcome="unroutable"} | counter | Total number of contact infos received and handled for QUIC Vote port (Skipped (unroutable)) |
| send_new_contact_info_quic_vote {new_contact_outcome="initialized"} | counter | Total number of contact infos received and handled for QUIC Vote port (Initialized) |
| send_new_contact_info_quic_vote {new_contact_outcome="changed"} | counter | Total number of contact infos received and handled for QUIC Vote port (Contact info changed) |
| send_new_contact_info_quic_vote {new_contact_outcome="no_change"} | counter | Total number of contact infos received and handled for QUIC Vote port (Contact info unchanged) |
| send_new_contact_info_quic_tpu {new_contact_outcome="unroutable"} | counter | Total number of contact infos received and handled for QUIC TPU port (Skipped (unroutable)) |
| send_new_contact_info_quic_tpu {new_contact_outcome="initialized"} | counter | Total number of contact infos received and handled for QUIC TPU port (Initialized) |
| send_new_contact_info_quic_tpu {new_contact_outcome="changed"} | counter | Total number of contact infos received and handled for QUIC TPU port (Contact info changed) |
| send_new_contact_info_quic_tpu {new_contact_outcome="no_change"} | counter | Total number of contact infos received and handled for QUIC TPU port (Contact info unchanged) |
| send_new_contact_info_udp_vote {new_contact_outcome="unroutable"} | counter | Total number of contact infos received and handled for UDP Vote port (Skipped (unroutable)) |
| send_new_contact_info_udp_vote {new_contact_outcome="initialized"} | counter | Total number of contact infos received and handled for UDP Vote port (Initialized) |
| send_new_contact_info_udp_vote {new_contact_outcome="changed"} | counter | Total number of contact infos received and handled for UDP Vote port (Contact info changed) |
| send_new_contact_info_udp_vote {new_contact_outcome="no_change"} | counter | Total number of contact infos received and handled for UDP Vote port (Contact info unchanged) |
| send_new_contact_info_udp_tpu {new_contact_outcome="unroutable"} | counter | Total number of contact infos received and handled for UDP TPU port (Skipped (unroutable)) |
| send_new_contact_info_udp_tpu {new_contact_outcome="initialized"} | counter | Total number of contact infos received and handled for UDP TPU port (Initialized) |
| send_new_contact_info_udp_tpu {new_contact_outcome="changed"} | counter | Total number of contact infos received and handled for UDP TPU port (Contact info changed) |
| send_new_contact_info_udp_tpu {new_contact_outcome="no_change"} | counter | Total number of contact infos received and handled for UDP TPU port (Contact info unchanged) |
| send_send_result_quic_vote {txn_send_result="success"} | counter | Total count of results from trying to send via QUIC Vote port (Success) |
| send_send_result_quic_vote {txn_send_result="no_ci"} | counter | Total count of results from trying to send via QUIC Vote port (No contact info) |
| send_send_result_quic_vote {txn_send_result="no_conn"} | counter | Total count of results from trying to send via QUIC Vote port (No QUIC connection) |
| send_send_result_quic_vote {txn_send_result="no_stream"} | counter | Total count of results from trying to send via QUIC Vote port (No QUIC stream) |
| send_send_result_quic_tpu {txn_send_result="success"} | counter | Total count of results from trying to send via QUIC TPU port (Success) |
| send_send_result_quic_tpu {txn_send_result="no_ci"} | counter | Total count of results from trying to send via QUIC TPU port (No contact info) |
| send_send_result_quic_tpu {txn_send_result="no_conn"} | counter | Total count of results from trying to send via QUIC TPU port (No QUIC connection) |
| send_send_result_quic_tpu {txn_send_result="no_stream"} | counter | Total count of results from trying to send via QUIC TPU port (No QUIC stream) |
| send_send_result_udp_vote {txn_send_result="success"} | counter | Total count of results from trying to send via UDP Vote port (Success) |
| send_send_result_udp_vote {txn_send_result="no_ci"} | counter | Total count of results from trying to send via UDP Vote port (No contact info) |
| send_send_result_udp_vote {txn_send_result="no_conn"} | counter | Total count of results from trying to send via UDP Vote port (No QUIC connection) |
| send_send_result_udp_vote {txn_send_result="no_stream"} | counter | Total count of results from trying to send via UDP Vote port (No QUIC stream) |
| send_send_result_udp_tpu {txn_send_result="success"} | counter | Total count of results from trying to send via UDP TPU port (Success) |
| send_send_result_udp_tpu {txn_send_result="no_ci"} | counter | Total count of results from trying to send via UDP TPU port (No contact info) |
| send_send_result_udp_tpu {txn_send_result="no_conn"} | counter | Total count of results from trying to send via UDP TPU port (No QUIC connection) |
| send_send_result_udp_tpu {txn_send_result="no_stream"} | counter | Total count of results from trying to send via UDP TPU port (No QUIC stream) |
| send_ensure_conn_result_quic_vote {send_ensure_conn_result="no_leader"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC Vote port (No QUIC connection) |
| send_ensure_conn_result_quic_vote {send_ensure_conn_result="no_ci"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC Vote port (No contact info) |
| send_ensure_conn_result_quic_vote {send_ensure_conn_result="new_connection"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC Vote port (Initiated connection) |
| send_ensure_conn_result_quic_vote {send_ensure_conn_result="conn_failed"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC Vote port (Connection failed) |
| send_ensure_conn_result_quic_vote {send_ensure_conn_result="connected"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC Vote port (Connection exists) |
| send_ensure_conn_result_quic_vote {send_ensure_conn_result="cooldown"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC Vote port (Connection cooldown) |
| send_ensure_conn_result_quic_tpu {send_ensure_conn_result="no_leader"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC TPU port (No QUIC connection) |
| send_ensure_conn_result_quic_tpu {send_ensure_conn_result="no_ci"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC TPU port (No contact info) |
| send_ensure_conn_result_quic_tpu {send_ensure_conn_result="new_connection"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC TPU port (Initiated connection) |
| send_ensure_conn_result_quic_tpu {send_ensure_conn_result="conn_failed"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC TPU port (Connection failed) |
| send_ensure_conn_result_quic_tpu {send_ensure_conn_result="connected"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC TPU port (Connection exists) |
| send_ensure_conn_result_quic_tpu {send_ensure_conn_result="cooldown"} | counter | Total count of results from trying to ensure a connection for a leader for QUIC TPU port (Connection cooldown) |
| send_handshake_complete {send_quic_ports="quic_vote"} | counter | Total number of times we completed a handshake (QUIC Vote port) |
| send_handshake_complete {send_quic_ports="quic_tpu"} | counter | Total number of times we completed a handshake (QUIC TPU port) |
| send_quic_conn_final {send_quic_ports="quic_vote"} | counter | Total number of times QUIC connection closed (QUIC Vote port) |
| send_quic_conn_final {send_quic_ports="quic_tpu"} | counter | Total number of times QUIC connection closed (QUIC TPU port) |
| send_staked_no_ci | gauge | Last counted # staked accounts without contact info |
| send_stale_ci | gauge | Last counted # staked accounts with stale contact info |
| send_received_packets | counter | Total count of QUIC packets received |
| send_received_bytes | counter | Total bytes received via QUIC |
| send_sent_packets | counter | Total count of QUIC packets sent |
| send_sent_bytes | counter | Total bytes sent via QUIC |
| send_retry_sent | counter | Total count of QUIC retry packets sent |
| send_connections_alloc | gauge | Number of currently allocated QUIC connections |
| send_connections_state {quic_conn_state="invalid"} | gauge | Number of QUIC connections in each state (freed) |
| send_connections_state {quic_conn_state="handshake"} | gauge | Number of QUIC connections in each state (handshaking peer) |
| send_connections_state {quic_conn_state="handshake_complete"} | gauge | Number of QUIC connections in each state (handshake complete, confirming with peer) |
| send_connections_state {quic_conn_state="active"} | gauge | Number of QUIC connections in each state (active connection) |
| send_connections_state {quic_conn_state="peer_close"} | gauge | Number of QUIC connections in each state (peer requested close) |
| send_connections_state {quic_conn_state="abort"} | gauge | Number of QUIC connections in each state (connection terminating due to error) |
| send_connections_state {quic_conn_state="close_pending"} | gauge | Number of QUIC connections in each state (connection is closing) |
| send_connections_state {quic_conn_state="dead"} | gauge | Number of QUIC connections in each state (connection about to be freed) |
| send_connections_created | counter | Total count of QUIC connections created |
| send_connections_closed | counter | Total count of QUIC connections closed |
| send_connections_aborted | counter | Total count of QUIC connections aborted |
| send_connections_timed_out | counter | Total count of QUIC connections timed out |
| send_connections_retried | counter | Total count of QUIC connections retried |
| send_connection_error_no_slots | counter | Total count of connection errors due to no slots |
| send_connection_error_retry_fail | counter | Total count of connection retry failures |
| send_pkt_crypto_failed {quic_enc_level="initial"} | counter | Total count of packets with crypto failures (initial) |
| send_pkt_crypto_failed {quic_enc_level="early"} | counter | Total count of packets with crypto failures (early data) |
| send_pkt_crypto_failed {quic_enc_level="handshake"} | counter | Total count of packets with crypto failures (handshake) |
| send_pkt_crypto_failed {quic_enc_level="app"} | counter | Total count of packets with crypto failures (app data) |
| send_pkt_no_key {quic_enc_level="initial"} | counter | Total count of packets with no key (initial) |
| send_pkt_no_key {quic_enc_level="early"} | counter | Total count of packets with no key (early data) |
| send_pkt_no_key {quic_enc_level="handshake"} | counter | Total count of packets with no key (handshake) |
| send_pkt_no_key {quic_enc_level="app"} | counter | Total count of packets with no key (app data) |
| send_pkt_no_conn {quic_pkt_handle="initial"} | counter | Total count of packets with no connection (initial) |
| send_pkt_no_conn {quic_pkt_handle="retry"} | counter | Total count of packets with no connection (retry) |
| send_pkt_no_conn {quic_pkt_handle="handshake"} | counter | Total count of packets with no connection (handshake) |
| send_pkt_no_conn {quic_pkt_handle="one_rtt"} | counter | Total count of packets with no connection (1-RTT) |
| send_pkt_tx_alloc_fail | counter | Total count of packet TX allocation failures |
| send_pkt_net_header_invalid | counter | Total count of packets with invalid network headers |
| send_pkt_quic_header_invalid | counter | Total count of packets with invalid QUIC headers |
| send_pkt_undersz | counter | Total count of undersized packets |
| send_pkt_oversz | counter | Total count of oversized packets |
| send_pkt_verneg | counter | Total count of version negotiation packets |
| send_pkt_retransmissions {quic_enc_level="initial"} | counter | Total count of QUIC packet retransmissions. (initial) |
| send_pkt_retransmissions {quic_enc_level="early"} | counter | Total count of QUIC packet retransmissions. (early data) |
| send_pkt_retransmissions {quic_enc_level="handshake"} | counter | Total count of QUIC packet retransmissions. (handshake) |
| send_pkt_retransmissions {quic_enc_level="app"} | counter | Total count of QUIC packet retransmissions. (app data) |
| send_handshakes_created | counter | Total count of QUIC handshakes created |
| send_handshake_error_alloc_fail | counter | Total count of handshake allocation failures |
| send_handshake_evicted | counter | Total count of handshakes evicted |
| send_stream_received_events | counter | Total count of stream events received |
| send_stream_received_bytes | counter | Total bytes received via streams |
| send_received_frames {quic_frame_type="unknown"} | counter | Total count of QUIC frames received (Unknown frame type) |
| send_received_frames {quic_frame_type="ack"} | counter | Total count of QUIC frames received (ACK frame) |
| send_received_frames {quic_frame_type="reset_stream"} | counter | Total count of QUIC frames received (RESET_STREAM frame) |
| send_received_frames {quic_frame_type="stop_sending"} | counter | Total count of QUIC frames received (STOP_SENDING frame) |
| send_received_frames {quic_frame_type="crypto"} | counter | Total count of QUIC frames received (CRYPTO frame) |
| send_received_frames {quic_frame_type="new_token"} | counter | Total count of QUIC frames received (NEW_TOKEN frame) |
| send_received_frames {quic_frame_type="stream"} | counter | Total count of QUIC frames received (STREAM frame) |
| send_received_frames {quic_frame_type="max_data"} | counter | Total count of QUIC frames received (MAX_DATA frame) |
| send_received_frames {quic_frame_type="max_stream_data"} | counter | Total count of QUIC frames received (MAX_STREAM_DATA frame) |
| send_received_frames {quic_frame_type="max_streams"} | counter | Total count of QUIC frames received (MAX_STREAMS frame) |
| send_received_frames {quic_frame_type="data_blocked"} | counter | Total count of QUIC frames received (DATA_BLOCKED frame) |
| send_received_frames {quic_frame_type="stream_data_blocked"} | counter | Total count of QUIC frames received (STREAM_DATA_BLOCKED frame) |
| send_received_frames {quic_frame_type="streams_blocked"} | counter | Total count of QUIC frames received (STREAMS_BLOCKED(bidi) frame) |
| send_received_frames {quic_frame_type="new_conn_id"} | counter | Total count of QUIC frames received (NEW_CONN_ID frame) |
| send_received_frames {quic_frame_type="retire_conn_id"} | counter | Total count of QUIC frames received (RETIRE_CONN_ID frame) |
| send_received_frames {quic_frame_type="path_challenge"} | counter | Total count of QUIC frames received (PATH_CHALLENGE frame) |
| send_received_frames {quic_frame_type="path_response"} | counter | Total count of QUIC frames received (PATH_RESPONSE frame) |
| send_received_frames {quic_frame_type="conn_close_quic"} | counter | Total count of QUIC frames received (CONN_CLOSE(transport) frame) |
| send_received_frames {quic_frame_type="conn_close_app"} | counter | Total count of QUIC frames received (CONN_CLOSE(app) frame) |
| send_received_frames {quic_frame_type="handshake_done"} | counter | Total count of QUIC frames received (HANDSHAKE_DONE frame) |
| send_received_frames {quic_frame_type="ping"} | counter | Total count of QUIC frames received (PING frame) |
| send_received_frames {quic_frame_type="padding"} | counter | Total count of QUIC frames received (PADDING frame) |
| send_frame_fail_parse | counter | Total count of frame parse failures |
| send_frame_tx_alloc {frame_tx_alloc_result="success"} | counter | Results of attempts to acquire QUIC frame metadata. (Success) |
| send_frame_tx_alloc {frame_tx_alloc_result="fail_empty_pool"} | counter | Results of attempts to acquire QUIC frame metadata. (PktMetaPoolEmpty) |
| send_frame_tx_alloc {frame_tx_alloc_result="fail_conn_max"} | counter | Results of attempts to acquire QUIC frame metadata. (ConnMaxedInflightFrames) |
| send_ack_tx {quic_ack_tx="noop"} | counter | Total count of ACK frames transmitted (non-ACK-eliciting packet) |
| send_ack_tx {quic_ack_tx="new"} | counter | Total count of ACK frames transmitted (new ACK range) |
| send_ack_tx {quic_ack_tx="merged"} | counter | Total count of ACK frames transmitted (merged into existing ACK range) |
| send_ack_tx {quic_ack_tx="drop"} | counter | Total count of ACK frames transmitted (out of buffers) |
| send_ack_tx {quic_ack_tx="cancel"} | counter | Total count of ACK frames transmitted (ACK suppressed by handler) |
| send_service_duration_seconds | histogram | Duration spent in service |
| send_receive_duration_seconds | histogram | Duration spent processing packets |
| send_sign_duration_nanos | histogram | Duration spent waiting for tls_cv signatures |
Snapct Tile
| Metric | Type | Description |
|---|---|---|
| snapct_state | gauge | State of the tile |
| snapct_full_num_retries | counter | Number of times we aborted and retried full snapshot download because the peer was too slow |
| snapct_incremental_num_retries | counter | Number of times we aborted and retried incremental snapshot download because the peer was too slow |
| snapct_full_bytes_read | gauge | Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_full_bytes_written | gauge | Number of bytes written so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_full_bytes_total | gauge | Total size of the full snapshot file. Might change if snapshot load is aborted and restarted |
| snapct_full_download_retries | gauge | Number of times we retried the full snapshot download because the peer was too slow |
| snapct_incremental_bytes_read | gauge | Number of bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_incremental_bytes_written | gauge | Number of bytes written so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapct_incremental_bytes_total | gauge | Total size of the incremental snapshot file. Might change if snapshot load is aborted and restarted |
| snapct_incremental_download_retries | gauge | Number of times we retried the incremental snapshot download because the peer was too slow |
| snapct_predicted_slot | gauge | The predicted slot from which replay starts after snapshot loading finishes. Might change if snapshot load is aborted and restarted |
| snapct_gossip_fresh_count | gauge | Number of fresh gossip peers seen when collecting gossip peers. |
| snapct_gossip_total_count | gauge | Number of total gossip peers seen when collecting gossip peers. |
Snapld Tile
| Metric | Type | Description |
|---|---|---|
| snapld_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
Snapdc Tile
| Metric | Type | Description |
|---|---|---|
| snapdc_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapdc_full_compressed_bytes_read | gauge | Number of bytes read so far from the compressed full snapshot file. Might decrease if snapshot load is aborted and restarted |
| snapdc_full_decompressed_bytes_read | gauge | Number of bytes read so far from the decompressed file. Might decrease if snapshot load is aborted and restarted |
| snapdc_incremental_compressed_bytes_read | gauge | Number of bytes read so far from the compressed incremental snapshot file. Might decrease if snapshot load is aborted and restarted |
| snapdc_incremental_decompressed_bytes_read | gauge | Number of bytes read so far from the decompressed incremental snapshot file. Might decrease if snapshot load is aborted and restarted |
Snapin Tile
| Metric | Type | Description |
|---|---|---|
| snapin_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 2=FINISHING, 3=ERROR, 4=SHUTDOWN |
| snapin_full_bytes_read | gauge | Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted |
| snapin_incremental_bytes_read | gauge | Number of bytes read so far from the incremental snapshot. Might decrease if snapshot load is aborted and restarted |
| snapin_accounts_inserted | gauge | Number of accounts inserted during snapshot loading. Might decrease if snapshot load is aborted and restarted |
Ipecho Tile
| Metric | Type | Description |
|---|---|---|
| ipecho_shred_version | gauge | The current shred version used by the validator |
| ipecho_connection_count | gauge | The number of active connections to the ipecho service |
| ipecho_connections_closed_ok | counter | The number of connections to the ipecho service that have been made and closed normally |
| ipecho_connections_closed_error | counter | The number of connections to the ipecho service that have been made and closed abnormally |
| ipecho_bytes_read | counter | The total number of bytes read from all connections to the ipecho service |
| ipecho_bytes_written | counter | The total number of bytes written to all connections to the ipecho service |
Gossvf Tile
| Metric | Type | Description |
|---|---|---|
| gossvf_message_rx_count {gossvf_message_outcome="success_pull_request"} | counter | (Pull Request (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_pull_response"} | counter | (Pull Response (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_push"} | counter | (Push (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_prune"} | counter | (Prune (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_ping"} | counter | (Ping (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="success_pong"} | counter | (Pong (success)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_unparseable"} | counter | (Unparseable) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_not_contact_info"} | counter | (Pull Request (not contact info)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_loopback"} | counter | (Pull Request (loopback)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_inactive"} | counter | (Pull Request (inactive)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_wallclock"} | counter | (Pull Request (wallclock)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_signature"} | counter | (Pull Request (signature)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_request_shred_version"} | counter | (Pull Request (shred version)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_prune_destination"} | counter | (Prune (destination)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_prune_wallclock"} | counter | (Prune (wallclock)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_prune_signature"} | counter | (Prune (signature)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_push_no_valid_crds"} | counter | (Push (no valid crds)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pull_response_no_valid_crds"} | counter | (Pull Response (no valid crds)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_ping_signature"} | counter | (Ping (signature)) |
| gossvf_message_rx_count {gossvf_message_outcome="dropped_pong_signature"} | counter | (Pong (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pull_request"} | counter | (Pull Request (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pull_response"} | counter | (Pull Response (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_push"} | counter | (Push (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_prune"} | counter | (Prune (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_ping"} | counter | (Ping (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="success_pong"} | counter | (Pong (success)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_unparseable"} | counter | (Unparseable) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_not_contact_info"} | counter | (Pull Request (not contact info)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_loopback"} | counter | (Pull Request (loopback)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_inactive"} | counter | (Pull Request (inactive)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_wallclock"} | counter | (Pull Request (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_signature"} | counter | (Pull Request (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_request_shred_version"} | counter | (Pull Request (shred version)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_destination"} | counter | (Prune (destination)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_wallclock"} | counter | (Prune (wallclock)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_prune_signature"} | counter | (Prune (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_push_no_valid_crds"} | counter | (Push (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pull_response_no_valid_crds"} | counter | (Pull Response (no valid crds)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_ping_signature"} | counter | (Ping (signature)) |
| gossvf_message_rx_bytes {gossvf_message_outcome="dropped_pong_signature"} | counter | (Pong (signature)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="success_pull_response"} | counter | (Pull Response (success)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="success_push"} | counter | (Push (success)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_duplicate"} | counter | (Pull Response (duplicate)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_signature"} | counter | (Pull Response (signature)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_relayer_no_contact_info"} | counter | (Pull Response (relayer no contact info)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_relayer_shred_version"} | counter | (Pull Response (relayer shred version)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_origin_no_contact_info"} | counter | (Pull Response (origin no contact info)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_origin_shred_version"} | counter | (Pull Response (origin shred version)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_pull_response_inactive"} | counter | (Pull Response (inactive)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_signature"} | counter | (Push (signature)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_relayer_no_contact_info"} | counter | (Push (relayer no contact info)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_relayer_shred_version"} | counter | (Push (relayer shred version)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_origin_no_contact_info"} | counter | (Push (origin no contact info)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_origin_shred_version"} | counter | (Push (origin shred version)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_inactive"} | counter | (Push (inactive)) |
| gossvf_crds_rx_count {gossvf_crds_outcome="dropped_push_wallclock"} | counter | (Push (wallclock)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="success_pull_response"} | counter | (Pull Response (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="success_push"} | counter | (Push (success)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_duplicate"} | counter | (Pull Response (duplicate)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_signature"} | counter | (Pull Response (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_relayer_no_contact_info"} | counter | (Pull Response (relayer no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_relayer_shred_version"} | counter | (Pull Response (relayer shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_origin_no_contact_info"} | counter | (Pull Response (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_origin_shred_version"} | counter | (Pull Response (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_pull_response_inactive"} | counter | (Pull Response (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_signature"} | counter | (Push (signature)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_relayer_no_contact_info"} | counter | (Push (relayer no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_relayer_shred_version"} | counter | (Push (relayer shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_origin_no_contact_info"} | counter | (Push (origin no contact info)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_origin_shred_version"} | counter | (Push (origin shred version)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_inactive"} | counter | (Push (inactive)) |
| gossvf_crds_rx_bytes {gossvf_crds_outcome="dropped_push_wallclock"} | counter | (Push (wallclock)) |
Bankf Tile
| Metric | Type | Description |
|---|---|---|
| bankf_transaction_result {transaction_result="success"} | counter | Result of loading and executing a transaction. (Transaction executed successfully) |
| bankf_transaction_result {transaction_result="instructon_error"} | counter | Result of loading and executing a transaction. (An error occurred while processing an instruction) |
| bankf_transaction_result {transaction_result="account_not_found"} | counter | Result of loading and executing a transaction. (The transaction fee payer address was not found) |
| bankf_transaction_result {transaction_result="program_account_not_found"} | counter | Result of loading and executing a transaction. (A program account referenced by the transaction was not found) |
| bankf_transaction_result {transaction_result="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction. (The transaction fee payer did not have balance to pay the fee) |
| bankf_transaction_result {transaction_result="invalid_account_for_fee"} | counter | Result of loading and executing a transaction. (The transaction fee payer account is not owned by the system program, or has data that is not a nonce) |
| bankf_transaction_result {transaction_result="already_processed"} | counter | Result of loading and executing a transaction. (The transaction has already been processed in a recent block) |
| bankf_transaction_result {transaction_result="blockhash_not_found"} | counter | Result of loading and executing a transaction. (The transaction references a blockhash that is not recent, or advances a nonce with the wrong value) |
| bankf_transaction_result {transaction_result="invalid_program_for_execution"} | counter | Result of loading and executing a transaction. (A program account referenced by the transaction was no executable. TODO: No longer needed with SIMD-0162) |
| bankf_transaction_result {transaction_result="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction. (The transaction references an ALUT account that does not exist or is inactive) |
| bankf_transaction_result {transaction_result="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction. (The transaction references an ALUT account that is not owned by the ALUT program account) |
| bankf_transaction_result {transaction_result="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction. (The transaction references an ALUT account that contains data which is not a valid ALUT) |
| bankf_transaction_result {transaction_result="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction. (The transaction references an account offset from the ALUT which does not exist) |
| bankf_transaction_result {transaction_result="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction. (The total account data size of the loaded accounts exceeds the consensus limit) |
| bankf_transaction_result {transaction_result="duplicate_instruction"} | counter | Result of loading and executing a transaction. (A compute budget program instruction was invoked more than once) |
| bankf_transaction_result {transaction_result="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction. (The compute budget program was invoked and set the loaded accounts data size to zero) |
| bankf_transaction_result {transaction_result="account_in_use"} | counter | Result of loading and executing a transaction. (The transaction conflicts with another transaction in the microblock. TODO: No longer possible with smart dispatcher) |
| bankf_transaction_result {transaction_result="account_loaded_twice"} | counter | Result of loading and executing a transaction. (The transaction references the same account twice) |
| bankf_transaction_result {transaction_result="signature_failure"} | counter | Result of loading and executing a transaction. (The transaction had an invalid signature) |
| bankf_transaction_result {transaction_result="too_many_account_locks"} | counter | Result of loading and executing a transaction. (The transaction references too many accounts. TODO: No longer possible with smart dispatcher) |
| bankf_transaction_result {transaction_result="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction. (The transaction would leave an account with a lower balance than the rent-exempt minimum) |
| bankf_transaction_result {transaction_result="unbalanced_transaction"} | counter | Result of loading and executing a transaction. (The total referenced account lamports before and after the transaction was unbalanced) |
| bankf_transaction_result {transaction_result="bundle_peer"} | counter | Result of loading and executing a transaction. (The transaction was part of a bundle and an earlier transaction in the bundle failed) |
Resolf Tile
| Metric | Type | Description |
|---|---|---|
| resolf_no_bank_drop | counter | Count of transactions dropped because the bank was not available |
| resolf_stash_operation {resolve_stash_operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolf_stash_operation {resolve_stash_operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolf_stash_operation {resolve_stash_operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolf_stash_operation {resolve_stash_operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories.) |
| resolf_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolf_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolf_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolf_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolf_lut_resolved {lut_resolve_result="account_not_found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolf_lut_resolved {lut_resolve_result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) |
| resolf_blockhash_expired | counter | Count of transactions that failed to resolve because the blockhash was expired |
| resolf_transaction_bundle_peer_failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed |
| resolf_db_races | counter | Number of database races encountered (diagnostic counter, not indicative of issues) |
Backt Tile
| Metric | Type | Description |
|---|---|---|
| backt_final_slot | gauge | The slot after which the backtest will complete |
| backt_start_slot | gauge | The slot at which the backtest started |
Exec Tile
| Metric | Type | Description |
|---|---|---|
| exec_progcache_misses | counter | Number of program cache misses |
| exec_progcache_hits | counter | Number of program cache hits |
| exec_progcache_fills | counter | Number of program cache insertions |
| exec_progcache_fill_tot_sz | counter | Total number of bytes inserted into program cache |
| exec_progcache_fill_fails | counter | Number of program cache load fails (tombstones inserted) |
| exec_progcache_dup_inserts | counter | Number of time two tiles raced to insert the same cache entry |
| exec_progcache_invalidations | counter | Number of program cache invalidations |
Snapwr Tile
| Metric | Type | Description |
|---|---|---|
| snapwr_state | gauge | State of the tile. 0=IDLE, 1=PROCESSING, 4=SHUTDOWN |
| snapwr_vinyl_bytes_written | gauge | Number of bytes written so far to the vinyl snapshot file. Might decrease if snapshot creation is aborted and restarted |