Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via. a Prometheus HTTP endpoint:
toml
[tiles.metric]
prometheus_listen_port = 7999sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric representing a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_slow_count | counter | The number of times the consumer was detected as rate limiting consumer by the producer. |
| link_consumed_count | counter | The number of times the link reader has consumed a fragment. |
| link_consumed_size_bytes | counter | The total number of bytes read by the link consumer. |
| link_filtered_count | counter | The number of fragments that were filtered and not consumed. |
| link_filtered_size_bytes | counter | The total number of bytes read by the link consumer that were filtered. |
| link_overrun_polling_count | counter | The number of times the link has been overrun while polling. |
| link_overrun_polling_frag_count | counter | The number of fragments the link has not processed because it was overrun while polling. |
| link_overrun_reading_count | counter | The number of input overruns detected while reading metadata by the consumer. |
| link_overrun_reading_frag_count | counter | The number of fragments the link has not processed because it was overrun while reading. |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | The process ID of the tile. |
| tile_tid | gauge | The thread ID of the tile. Always the same as the Pid in production, but might be different in development. |
| tile_context_switch_involuntary_count | counter | The number of involuntary context switches. |
| tile_context_switch_voluntary_count | counter | The number of voluntary context switches. |
| tile_status | gauge | The current status of the tile. 0 is booting, 1 is running. |
| tile_heartbeat | gauge | The last UNIX timestamp in nanoseconds that the tile heartbeated. |
| tile_in_backpressure | gauge | Whether the tile is currently backpressured or not, either 1 or 0. |
| tile_backpressure_count | counter | Number of times the tile has had to wait for one of more consumers to catch up to resume publishing. |
| tile_regime_duration_nanos {tile_regime="caught_up_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="processing_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="backpressure_housekeeping"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Backpressure + Housekeeping) |
| tile_regime_duration_nanos {tile_regime="caught_up_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Prefrag) |
| tile_regime_duration_nanos {tile_regime="processing_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Prefrag) |
| tile_regime_duration_nanos {tile_regime="backpressure_prefrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Backpressure + Prefrag) |
| tile_regime_duration_nanos {tile_regime="caught_up_postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Postfrag) |
| tile_regime_duration_nanos {tile_regime="processing_postfrag"} | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Postfrag) |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_rx_pkt_cnt | counter | Packet receive count. |
| net_rx_bytes_total | counter | Total number of bytes received (including Ethernet header). |
| net_rx_undersz_cnt | counter | Number of incoming packets dropped due to being too small. |
| net_rx_fill_blocked_cnt | counter | Number of incoming packets dropped due to fill ring being full. |
| net_rx_backpressure_cnt | counter | Number of incoming packets dropped due to backpressure. |
| net_rx_busy_cnt | gauge | Number of receive buffers currently busy. |
| net_rx_idle_cnt | gauge | Number of receive buffers currently idle. |
| net_tx_submit_cnt | counter | Number of packet transmit jobs submitted. |
| net_tx_complete_cnt | counter | Number of packet transmit jobs marked as completed by the kernel. |
| net_tx_bytes_total | counter | Total number of bytes transmitted (including Ethernet header). |
| net_tx_route_fail_cnt | counter | Number of packet transmit jobs dropped due to route failure. |
| net_tx_neighbor_fail_cnt | counter | Number of packet transmit jobs dropped due to unresolved neighbor. |
| net_tx_full_fail_cnt | counter | Number of packet transmit jobs dropped due to XDP TX ring full or missing completions. |
| net_tx_busy_cnt | gauge | Number of transmit buffers currently busy. |
| net_tx_idle_cnt | gauge | Number of transmit buffers currently idle. |
| net_xsk_tx_wakeup_cnt | counter | Number of XSK sendto syscalls dispatched. |
| net_xsk_rx_wakeup_cnt | counter | Number of XSK recvmsg syscalls dispatched. |
| net_xdp_rx_dropped_other | counter | xdp_statistics_v0.rx_dropped: Dropped for other reasons |
| net_xdp_rx_invalid_descs | counter | xdp_statistics_v0.rx_invalid_descs: Dropped due to invalid descriptor |
| net_xdp_tx_invalid_descs | counter | xdp_statistics_v0.tx_invalid_descs: Dropped due to invalid descriptor |
| net_xdp_rx_ring_full | counter | xdp_statistics_v1.rx_ring_full: Dropped due to rx ring being full |
| net_xdp_rx_fill_ring_empty_descs | counter | xdp_statistics_v1.rx_fill_ring_empty_descs: Failed to retrieve item from fill ring |
| net_xdp_tx_ring_empty_descs | counter | xdp_statistics_v1.tx_ring_empty_descs: Failed to retrieve item from tx ring |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_txns_overrun | counter | Count of txns overrun before reassembled (too small txn_reassembly_count). |
| quic_txn_reasms_started | counter | Count of fragmented txn receive ops started. |
| quic_txn_reasms_active | gauge | Number of fragmented txn receive ops currently active. |
| quic_frags_ok | counter | Count of txn frags received |
| quic_frags_gap | counter | Count of txn frags dropped due to data gap |
| quic_frags_dup | counter | Count of txn frags dropped due to dup (stream already completed) |
| quic_txns_received {tpu_recv_type="udp"} | counter | Count of txns received via TPU. (TPU/UDP) |
| quic_txns_received {tpu_recv_type="quic_fast"} | counter | Count of txns received via TPU. (TPU/QUIC unfragmented) |
| quic_txns_received {tpu_recv_type="quic_frag"} | counter | Count of txns received via TPU. (TPU/QUIC fragmented) |
| quic_txns_abandoned | counter | Count of txns abandoned because a conn was lost. |
| quic_txn_undersz | counter | Count of txns received via QUIC dropped because they were too small. |
| quic_txn_oversz | counter | Count of txns received via QUIC dropped because they were too large. |
| quic_legacy_txn_undersz | counter | Count of packets received on the non-QUIC port that were too small to be a valid IP packet. |
| quic_legacy_txn_oversz | counter | Count of packets received on the non-QUIC port that were too large to be a valid transaction. |
| quic_received_packets | counter | Number of IP packets received. |
| quic_received_bytes | counter | Total bytes received (including IP, UDP, QUIC headers). |
| quic_sent_packets | counter | Number of IP packets sent. |
| quic_sent_bytes | counter | Total bytes sent (including IP, UDP, QUIC headers). |
| quic_connections_active | gauge | The number of currently active QUIC connections. |
| quic_connections_created | counter | The total number of connections that have been created. |
| quic_connections_closed | counter | Number of connections gracefully closed. |
| quic_connections_aborted | counter | Number of connections aborted. |
| quic_connections_timed_out | counter | Number of connections timed out. |
| quic_connections_retried | counter | Number of connections established with retry. |
| quic_connection_error_no_slots | counter | Number of connections that failed to create due to lack of slots. |
| quic_connection_error_retry_fail | counter | Number of connections that failed during retry (e.g. invalid token). |
| quic_pkt_no_conn | counter | Number of packets with an unknown connection ID. |
| quic_frame_tx_alloc {frame_tx_alloc_result="success"} | counter | Results of attempts to acquire QUIC frame metadata. (Success) |
| quic_frame_tx_alloc {frame_tx_alloc_result="fail_empty_pool"} | counter | Results of attempts to acquire QUIC frame metadata. (PktMetaPoolEmpty) |
| quic_frame_tx_alloc {frame_tx_alloc_result="fail_conn_max"} | counter | Results of attempts to acquire QUIC frame metadata. (ConnMaxedInflightFrames) |
| quic_handshakes_created | counter | Number of handshake flows created. |
| quic_handshake_error_alloc_fail | counter | Number of handshakes dropped due to alloc fail. |
| quic_handshake_evicted | counter | Number of handshakes dropped due to eviction. |
| quic_stream_received_events | counter | Number of stream RX events. |
| quic_stream_received_bytes | counter | Total stream payload bytes received. |
| quic_received_frames {quic_frame_type="unknown"} | counter | Number of QUIC frames received. (Unknown frame type) |
| quic_received_frames {quic_frame_type="ack"} | counter | Number of QUIC frames received. (ACK frame) |
| quic_received_frames {quic_frame_type="reset_stream"} | counter | Number of QUIC frames received. (RESET_STREAM frame) |
| quic_received_frames {quic_frame_type="stop_sending"} | counter | Number of QUIC frames received. (STOP_SENDING frame) |
| quic_received_frames {quic_frame_type="crypto"} | counter | Number of QUIC frames received. (CRYPTO frame) |
| quic_received_frames {quic_frame_type="new_token"} | counter | Number of QUIC frames received. (NEW_TOKEN frame) |
| quic_received_frames {quic_frame_type="stream"} | counter | Number of QUIC frames received. (STREAM frame) |
| quic_received_frames {quic_frame_type="max_data"} | counter | Number of QUIC frames received. (MAX_DATA frame) |
| quic_received_frames {quic_frame_type="max_stream_data"} | counter | Number of QUIC frames received. (MAX_STREAM_DATA frame) |
| quic_received_frames {quic_frame_type="max_streams"} | counter | Number of QUIC frames received. (MAX_STREAMS frame) |
| quic_received_frames {quic_frame_type="data_blocked"} | counter | Number of QUIC frames received. (DATA_BLOCKED frame) |
| quic_received_frames {quic_frame_type="stream_data_blocked"} | counter | Number of QUIC frames received. (STREAM_DATA_BLOCKED frame) |
| quic_received_frames {quic_frame_type="streams_blocked"} | counter | Number of QUIC frames received. (STREAMS_BLOCKED(bidi) frame) |
| quic_received_frames {quic_frame_type="new_conn_id"} | counter | Number of QUIC frames received. (NEW_CONN_ID frame) |
| quic_received_frames {quic_frame_type="retire_conn_id"} | counter | Number of QUIC frames received. (RETIRE_CONN_ID frame) |
| quic_received_frames {quic_frame_type="path_challenge"} | counter | Number of QUIC frames received. (PATH_CHALLENGE frame) |
| quic_received_frames {quic_frame_type="path_response"} | counter | Number of QUIC frames received. (PATH_RESPONSE frame) |
| quic_received_frames {quic_frame_type="conn_close_quic"} | counter | Number of QUIC frames received. (CONN_CLOSE(transport) frame) |
| quic_received_frames {quic_frame_type="conn_close_app"} | counter | Number of QUIC frames received. (CONN_CLOSE(app) frame) |
| quic_received_frames {quic_frame_type="handshake_done"} | counter | Number of QUIC frames received. (HANDSHAKE_DONE frame) |
| quic_received_frames {quic_frame_type="ping"} | counter | Number of QUIC frames received. (PING frame) |
| quic_received_frames {quic_frame_type="padding"} | counter | Number of QUIC frames received. (PADDING frame) |
| quic_ack_tx {quic_ack_tx="noop"} | counter | ACK events (non-ACK-eliciting packet) |
| quic_ack_tx {quic_ack_tx="new"} | counter | ACK events (new ACK range) |
| quic_ack_tx {quic_ack_tx="merged"} | counter | ACK events (merged into existing ACK range) |
| quic_ack_tx {quic_ack_tx="drop"} | counter | ACK events (out of buffers) |
| quic_ack_tx {quic_ack_tx="cancel"} | counter | ACK events (ACK suppressed by handler) |
| quic_service_duration_seconds | histogram | Duration spent in service |
| quic_receive_duration_seconds | histogram | Duration spent receiving packets |
| quic_frame_fail_parse | counter | Number of QUIC frames failed to parse. |
| quic_pkt_crypto_failed {quic_enc_level="initial"} | counter | Number of packets that failed decryption. (initial) |
| quic_pkt_crypto_failed {quic_enc_level="early"} | counter | Number of packets that failed decryption. (early data) |
| quic_pkt_crypto_failed {quic_enc_level="handshake"} | counter | Number of packets that failed decryption. (handshake) |
| quic_pkt_crypto_failed {quic_enc_level="app"} | counter | Number of packets that failed decryption. (app data) |
| quic_pkt_no_key {quic_enc_level="initial"} | counter | Number of packets that failed decryption due to missing key. (initial) |
| quic_pkt_no_key {quic_enc_level="early"} | counter | Number of packets that failed decryption due to missing key. (early data) |
| quic_pkt_no_key {quic_enc_level="handshake"} | counter | Number of packets that failed decryption due to missing key. (handshake) |
| quic_pkt_no_key {quic_enc_level="app"} | counter | Number of packets that failed decryption due to missing key. (app data) |
| quic_pkt_net_header_invalid | counter | Number of packets dropped due to weird IP or UDP header. |
| quic_pkt_quic_header_invalid | counter | Number of packets dropped due to weird QUIC header. |
| quic_pkt_undersz | counter | Number of QUIC packets dropped due to being too small. |
| quic_pkt_oversz | counter | Number of QUIC packets dropped due to being too large. |
| quic_pkt_verneg | counter | Number of QUIC version negotiation packets received. |
| quic_retry_sent | counter | Number of QUIC Retry packets sent. |
| quic_pkt_retransmissions | counter | Number of QUIC packets that retransmitted. |
Bundle Tile
| Metric | Type | Description |
|---|---|---|
| bundle_transaction_received | counter | Total count of transactions received, including transactions within bundles |
| bundle_packet_received | counter | Total count of packets received |
| bundle_bundle_received | counter | Total count of bundles received |
| bundle_errors {bundle_error="protobuf"} | counter | Number of gRPC errors encountered (Protobuf decode/encode error) |
| bundle_errors {bundle_error="transport"} | counter | Number of gRPC errors encountered (Transport error) |
| bundle_errors {bundle_error="timeout"} | counter | Number of gRPC errors encountered (I/O timeout) |
| bundle_errors {bundle_error="no_fee_info"} | counter | Number of gRPC errors encountered (Bundle dropped due to missing fee info) |
| bundle_errors {bundle_error="ssl_alloc"} | counter | Number of gRPC errors encountered (OpenSSL alloc fail) |
| bundle_heap_size | gauge | Workspace heap size |
| bundle_heap_free_bytes | gauge | Approx free space in workspace |
| bundle_shredstream_heartbeats | counter | Number of ShredStream heartbeats successfully sent |
| bundle_keepalives | counter | Number of HTTP/2 PINGs acknowledged by server |
| bundle_connected | gauge | 1 if connected to the bundle server, 0 if not |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_transaction_bundle_peer_failure | counter | Count of transactions that failed to verify because a peer transaction in the bundle failed |
| verify_transaction_parse_failure | counter | Count of transactions that failed to parse |
| verify_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
| verify_transaction_verify_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_transaction_bundle_peer_failure | counter | Count of transactions that failed to dedup because a peer transaction in the bundle failed |
| dedup_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the dedup stage |
| dedup_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Resolv Tile
| Metric | Type | Description |
|---|---|---|
| resolv_no_bank_drop | counter | Count of transactions dropped because the bank was not available |
| resolv_stash_operation {resolve_stash_operation="inserted"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was added to the stash) |
| resolv_stash_operation {resolve_stash_operation="overrun"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was dropped because the stash was full) |
| resolv_stash_operation {resolve_stash_operation="published"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was published as the blockhash became known) |
| resolv_stash_operation {resolve_stash_operation="removed"} | counter | Count of operations that happened on the transaction stash (A transaction with an unknown blockhash was removed from the stash without publishing, due to a bad LUT resolved failure, or no bank. These errors are double counted with the respective metrics for those categories.) |
| resolv_lut_resolved {lut_resolve_result="invalid_lookup_index"} | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolv_lut_resolved {lut_resolve_result="account_uninitialized"} | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_data"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolv_lut_resolved {lut_resolve_result="invalid_account_owner"} | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolv_lut_resolved {lut_resolve_result="account_not_found"} | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolv_lut_resolved {lut_resolve_result="success"} | counter | Count of address lookup tables resolved (Resolved successfully) |
| resolv_blockhash_expired | counter | Count of transactions that failed to resolve because the blockhash was expired |
| resolv_transaction_bundle_peer_failure | counter | Count of transactions that failed to resolve because a peer transaction in the bundle failed |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_schedule_microblock_duration_seconds | histogram | Duration of scheduling one microblock |
| pack_no_sched_microblock_duration_seconds | histogram | Duration of discovering that there are no schedulable transactions |
| pack_insert_transaction_duration_seconds | histogram | Duration of inserting one transaction into the pool of available transactions |
| pack_complete_microblock_duration_seconds | histogram | Duration of the computation associated with marking one microblock as complete |
| pack_total_transactions_per_microblock_count | histogram | Count of transactions in a scheduled microblock, including both votes and non-votes |
| pack_votes_per_microblock_count | histogram | Count of simple vote transactions in a scheduled microblock |
| pack_normal_transaction_received | counter | Count of transactions received via the normal TPU path |
| pack_transaction_inserted {pack_txn_insert_return="bundle_blacklist"} | counter | Result of inserting a transaction into the pack object (Transaction uses an account on the bundle blacklist) |
| pack_transaction_inserted {pack_txn_insert_return="write_sysvar"} | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) |
| pack_transaction_inserted {pack_txn_insert_return="estimation_fail"} | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) |
| pack_transaction_inserted {pack_txn_insert_return="duplicate_account"} | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) |
| pack_transaction_inserted {pack_txn_insert_return="too_many_accounts"} | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) |
| pack_transaction_inserted {pack_txn_insert_return="too_large"} | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) |
| pack_transaction_inserted {pack_txn_insert_return="expired"} | counter | Result of inserting a transaction into the pack object (Transaction already expired) |
| pack_transaction_inserted {pack_txn_insert_return="addr_lut"} | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) |
| pack_transaction_inserted {pack_txn_insert_return="unaffordable"} | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) |
| pack_transaction_inserted {pack_txn_insert_return="duplicate"} | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) |
| pack_transaction_inserted {pack_txn_insert_return="priority"} | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_transaction_inserted {pack_txn_insert_return="nonvote_add"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="vote_add"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) |
| pack_transaction_inserted {pack_txn_insert_return="nonvote_replace"} | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) |
| pack_transaction_inserted {pack_txn_insert_return="vote_replace"} | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and wasn't leader) |
| pack_metric_timing {pack_timing_state="txn_no_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, but wasn't leader or had hit a limit) |
| pack_metric_timing {pack_timing_state="no_txn_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, had banks but wasn't leader) |
| pack_metric_timing {pack_timing_state="txn_bank_no_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, had banks but wasn't leader) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had no transactions available, and was leader but had no available banks) |
| pack_metric_timing {pack_timing_state="txn_no_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had transactions available, was leader, but had no available banks) |
| pack_metric_timing {pack_timing_state="no_txn_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had available banks but no transactions) |
| pack_metric_timing {pack_timing_state="txn_bank_leader_no_microblock"} | counter | Time in nanos spent in each state (Pack had banks and transactions available but couldn't schedule anything non-conflicting) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="txn_no_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="no_txn_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="txn_bank_no_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader) |
| pack_metric_timing {pack_timing_state="no_txn_no_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy) |
| pack_metric_timing {pack_timing_state="txn_no_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy) |
| pack_metric_timing {pack_timing_state="no_txn_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock and now has no transactions) |
| pack_metric_timing {pack_timing_state="txn_bank_leader_microblock"} | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock) |
| pack_transaction_dropped_from_extra | counter | Transactions dropped from the extra transaction storage because it was full |
| pack_transaction_inserted_to_extra | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full |
| pack_transaction_inserted_from_extra | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage |
| pack_transaction_expired | counter | Transactions deleted from pack because their TTL expired |
| pack_transaction_dropped_partial_bundle | counter | Transactions dropped from pack because they were part of a partial bundle |
| pack_available_transactions {avail_txn_type="all"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (All transactions in any treap) |
| pack_available_transactions {avail_txn_type="regular"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes in the main treap) |
| pack_available_transactions {avail_txn_type="votes"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Simple votes) |
| pack_available_transactions {avail_txn_type="conflicting"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Non-votes that write to a hotly-contended account) |
| pack_available_transactions {avail_txn_type="bundles"} | gauge | The total number of pending transactions in pack's pool that are available to be scheduled (Transactions that are part of a bundle) |
| pack_pending_transactions_heap_size | gauge | The maximum number of pending transactions that pack can consider. This value is fixed at Firedancer startup but is a useful reference for AvailableTransactions. |
| pack_smallest_pending_transaction | gauge | A lower bound on the smallest non-vote transaction (in cost units) that is immediately available for scheduling |
| pack_microblock_per_block_limit | counter | The number of times pack did not pack a microblock because the limit on microblocks/block had been reached |
| pack_data_per_block_limit | counter | The number of times pack did not pack a microblock because it reached the data per block limit at the start of trying to schedule a microblock |
| pack_transaction_schedule {pack_txn_schedule="taken"} | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) |
| pack_transaction_schedule {pack_txn_schedule="cu_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_transaction_schedule {pack_txn_schedule="fast_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) |
| pack_transaction_schedule {pack_txn_schedule="byte_limit"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_transaction_schedule {pack_txn_schedule="write_cost"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_transaction_schedule {pack_txn_schedule="slow_path"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) |
| pack_transaction_schedule {pack_txn_schedule="defer_skip"} | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction it previously exceeded the per-account block write cost limit too many times) |
| pack_bundle_crank_status {bundle_crank_result="not_needed"} | counter | Result of considering whether bundle cranks are needed (On-chain state in the correct state) |
| pack_bundle_crank_status {bundle_crank_result="inserted"} | counter | Result of considering whether bundle cranks are needed (Inserted an initializer bundle to update the on-chain state) |
| pack_bundle_crank_status {bundle_crank_result="creation_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but creation failed) |
| pack_bundle_crank_status {bundle_crank_result="insertion_failed"} | counter | Result of considering whether bundle cranks are needed (Tried to insert an initializer bundle to update the on-chain state, but insertion failed) |
| pack_cus_consumed_in_block | gauge | The number of cost units consumed in the current block, or 0 if pack is not currently packing a block |
| pack_cus_scheduled | histogram | The number of cost units scheduled for each block pack produced. This can be higher than the block limit because of returned CUs. |
| pack_cus_rebated | histogram | The number of compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses. |
| pack_cus_net | histogram | The net number of cost units (scheduled - rebated) in each block pack produced. |
| pack_cus_pct | histogram | The percent of the total block cost limit used for each block pack produced. |
| pack_delete_missed | counter | Count of attempts to delete a transaction that wasn't found |
| pack_delete_hit | counter | Count of attempts to delete a transaction that was found and deleted |
Bank Tile
| Metric | Type | Description |
|---|---|---|
| bank_transaction_sanitize_failure | counter | Number of transactions that failed to sanitize. |
| bank_transaction_not_executed_failure | counter | Number of transactions that did not execute. This is different than transactions which fail to execute, which make it onto the chain. |
| bank_precompile_verify_failure | counter | Number of transactions that failed precompile verification and thus will not execute. |
| bank_slot_acquire {bank_slot_acquire="success"} | counter | Result of acquiring a slot. (Success) |
| bank_slot_acquire {bank_slot_acquire="too_high"} | counter | Result of acquiring a slot. (Too high) |
| bank_slot_acquire {bank_slot_acquire="too_low"} | counter | Result of acquiring a slot. (Too low) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="success"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (Success) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="slot_hashes_sysvar_not_found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The slot hashes syvar could not be found.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="account_not_found"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account storing the address lookup table was deactivated or could not be found.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="invalid_account_owner"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account that owns the referenced lookup table is not the address lookup table program.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="invalid_account_data"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The data for the referenced address lookup table is malformed.) |
| bank_transaction_load_address_tables {bank_load_address_lookup_table="invalid_index"} | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The referenced index in the address lookup table does not exist.) |
| bank_transaction_result {transaction_error="success"} | counter | Result of loading and executing a transaction. (Success) |
| bank_transaction_result {transaction_error="account_in_use"} | counter | Result of loading and executing a transaction. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_transaction_result {transaction_error="account_loaded_twice"} | counter | Result of loading and executing a transaction. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_transaction_result {transaction_error="account_not_found"} | counter | Result of loading and executing a transaction. (Attempt to debit an account but found no record of a prior credit.) |
| bank_transaction_result {transaction_error="program_account_not_found"} | counter | Result of loading and executing a transaction. (Attempt to load a program that does not exist.) |
| bank_transaction_result {transaction_error="insufficient_funds_for_fee"} | counter | Result of loading and executing a transaction. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_transaction_result {transaction_error="invalid_account_for_fee"} | counter | Result of loading and executing a transaction. (This account may not be used to pay transaction fees.) |
| bank_transaction_result {transaction_error="already_processed"} | counter | Result of loading and executing a transaction. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_transaction_result {transaction_error="blockhash_not_found"} | counter | Result of loading and executing a transaction. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_transaction_result {transaction_error="instruction_error"} | counter | Result of loading and executing a transaction. (An error occurred while processing an instruction.) |
| bank_transaction_result {transaction_error="call_chain_too_deep"} | counter | Result of loading and executing a transaction. (Loader call chain is too deep.) |
| bank_transaction_result {transaction_error="missing_signature_for_fee"} | counter | Result of loading and executing a transaction. (Transaction requires a fee but has no signature present.) |
| bank_transaction_result {transaction_error="invalid_account_index"} | counter | Result of loading and executing a transaction. (Transaction contains an invalid account reference.) |
| bank_transaction_result {transaction_error="signature_failure"} | counter | Result of loading and executing a transaction. (Transaction did not pass signature verification.) |
| bank_transaction_result {transaction_error="invalid_program_for_execution"} | counter | Result of loading and executing a transaction. (This program may not be used for executing instructions.) |
| bank_transaction_result {transaction_error="sanitize_failure"} | counter | Result of loading and executing a transaction. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_transaction_result {transaction_error="cluster_maintenance"} | counter | Result of loading and executing a transaction. (Transactions are currently disabled due to cluster maintenance.) |
| bank_transaction_result {transaction_error="account_borrow_outstanding"} | counter | Result of loading and executing a transaction. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_transaction_result {transaction_error="would_exceed_max_block_cost_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed max Block Cost Limit.) |
| bank_transaction_result {transaction_error="unsupported_version"} | counter | Result of loading and executing a transaction. (Transaction version is unsupported.) |
| bank_transaction_result {transaction_error="invalid_writable_account"} | counter | Result of loading and executing a transaction. (Transaction loads a writable account that cannot be written.) |
| bank_transaction_result {transaction_error="would_exceed_max_account_cost_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed max account limit within the block.) |
| bank_transaction_result {transaction_error="would_exceed_account_data_block_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed account data limit within the block.) |
| bank_transaction_result {transaction_error="too_many_account_locks"} | counter | Result of loading and executing a transaction. (Transaction locked too many accounts.) |
| bank_transaction_result {transaction_error="address_lookup_table_not_found"} | counter | Result of loading and executing a transaction. (Address lookup table not found.) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_owner"} | counter | Result of loading and executing a transaction. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_data"} | counter | Result of loading and executing a transaction. (Attempted to lookup addresses from an invalid account.) |
| bank_transaction_result {transaction_error="invalid_address_lookup_table_index"} | counter | Result of loading and executing a transaction. (Address table lookup uses an invalid index.) |
| bank_transaction_result {transaction_error="invalid_rent_paying_account"} | counter | Result of loading and executing a transaction. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_transaction_result {transaction_error="would_exceed_max_vote_cost_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed max Vote Cost Limit.) |
| bank_transaction_result {transaction_error="would_exceed_account_data_total_limit"} | counter | Result of loading and executing a transaction. (Transaction would exceed total account data limit.) |
| bank_transaction_result {transaction_error="duplicate_instruction"} | counter | Result of loading and executing a transaction. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_transaction_result {transaction_error="insufficient_funds_for_rent"} | counter | Result of loading and executing a transaction. (Transaction results in an account with insufficient funds for rent.) |
| bank_transaction_result {transaction_error="max_loaded_accounts_data_size_exceeded"} | counter | Result of loading and executing a transaction. (Transaction exceeded max loaded accounts data size cap.) |
| bank_transaction_result {transaction_error="invalid_loaded_accounts_data_size_limit"} | counter | Result of loading and executing a transaction. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_transaction_result {transaction_error="resanitization_needed"} | counter | Result of loading and executing a transaction. (Sanitized transaction differed before/after feature activation. Needs to be resanitized.) |
| bank_transaction_result {transaction_error="program_execution_temporarily_restricted"} | counter | Result of loading and executing a transaction. (Program execution is temporarily restricted on an account.) |
| bank_transaction_result {transaction_error="unbalanced_transaction"} | counter | Result of loading and executing a transaction. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_transaction_result {transaction_error="program_cache_hit_max_limit"} | counter | Result of loading and executing a transaction. (The total program cache size hit the maximum allowed limit.) |
| bank_transaction_result {transaction_error="commit_cancelled"} | counter | Result of loading and executing a transaction. (The process for comitting the transaction was cancelled internaly.) |
| bank_transaction_result {transaction_error="bundle_peer"} | counter | Result of loading and executing a transaction. (Transaction is part of a bundle and one of the peer transactions failed.) |
| bank_processing_failed | counter | Count of transactions for which the processing stage failed and won't land on chain |
| bank_fee_only_transactions | counter | Count of transactions that will land on chain but without executing |
| bank_executed_failed_transactions | counter | Count of transactions that execute on chain but failed |
| bank_successful_transactions | counter | Count of transactions that execute on chain and succeed |
| bank_cost_model_undercount | counter | Count of transactions that used more CUs than the cost model should have permitted them to |
Poh Tile
| Metric | Type | Description |
|---|---|---|
| poh_begin_leader_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the bank. |
| poh_first_microblock_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock. |
| poh_slot_done_delay_seconds | histogram | Delay between when we become leader in a slot and when we finish the slot. |
| poh_bundle_initialize_delay_seconds | histogram | Delay in starting the slot caused by loading the information needed to generate the bundle crank transactions |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_cluster_contact_info_cnt | histogram | Number of contact infos in the cluster contact info message |
| shred_microblocks_abandoned | counter | The number of microblocks that were abandoned because we switched slots without finishing the current slot |
| shred_invalid_block_id | counter | The number of times a block was created with unknown parent block_id |
| shred_batch_sz | histogram | The size (in bytes) of each microblock batch that is shredded |
| shred_batch_microblock_cnt | histogram | The number of microblocks in each microblock batch that is shredded |
| shred_shredding_duration_seconds | histogram | Duration of producing one FEC set from the shredder |
| shred_add_shred_duration_seconds | histogram | Duration of verifying and processing one shred received from the network |
| shred_shred_processed {shred_processing_result="bad_slot"} | counter | The result of processing a thread from the network (Shred was for a slot for which we don't know the leader) |
| shred_shred_processed {shred_processing_result="parse_failed"} | counter | The result of processing a thread from the network (Shred parsing failed) |
| shred_shred_processed {shred_processing_result="rejected"} | counter | The result of processing a thread from the network (Shred was invalid for one of many reasons) |
| shred_shred_processed {shred_processing_result="ignored"} | counter | The result of processing a thread from the network (Shred was ignored because we had already received or reconstructed it) |
| shred_shred_processed {shred_processing_result="okay"} | counter | The result of processing a thread from the network (Shred accepted to an incomplete FEC set) |
| shred_shred_processed {shred_processing_result="completes"} | counter | The result of processing a thread from the network (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_set_spilled | counter | The number of FEC sets that were spilled because they didn't complete in time and we needed space |
| shred_shred_rejected_initial | counter | The number shreds that were rejected before any resources were allocated for the FEC set |
| shred_fec_rejected_fatal | counter | The number of FEC sets that were rejected for reasons that cause the whole FEC set to become invalid |
| shred_force_complete_request | counter | The number of times we recieved a FEC force complete message |
| shred_force_complete_failure | counter | The number of times we failed to force complete a FEC set on request |
| shred_force_complete_success | counter | The number of times we successfully forced completed a FEC set on request |
Store Tile
| Metric | Type | Description |
|---|---|---|
| store_transactions_inserted | counter | Count of transactions produced while we were leader in the shreds that have been inserted so far |
Replay Tile
| Metric | Type | Description |
|---|---|---|
| replay_slot | gauge | |
| replay_last_voted_slot | gauge |
Storei Tile
| Metric | Type | Description |
|---|---|---|
| storei_first_turbine_slot | gauge | |
| storei_current_turbine_slot | gauge |
Gossip Tile
| Metric | Type | Description |
|---|---|---|
| gossip_last_crds_push_contact_info_publish_timestamp_nanos | gauge | Time (in nanoseconds) of last CRDS Push ContactInfo message publish |
| gossip_mismatched_contact_info_shred_version | counter | Mismatched Contact Info Shred Version |
| gossip_ipv6_contact_info {peer_types="tvu"} | counter | IPv6 Contact Info (by peer type) (TVU) |
| gossip_ipv6_contact_info {peer_types="repair"} | counter | IPv6 Contact Info (by peer type) (Repair) |
| gossip_ipv6_contact_info {peer_types="voter"} | counter | IPv6 Contact Info (by peer type) (Voter) |
| gossip_zero_ipv4_contact_info {peer_types="tvu"} | counter | Zero IPv4 Contact Info (by peer type) (TVU) |
| gossip_zero_ipv4_contact_info {peer_types="repair"} | counter | Zero IPv4 Contact Info (by peer type) (Repair) |
| gossip_zero_ipv4_contact_info {peer_types="voter"} | counter | Zero IPv4 Contact Info (by peer type) (Voter) |
| gossip_peer_counts {peer_types="tvu"} | gauge | Number of peers of each type (TVU) |
| gossip_peer_counts {peer_types="repair"} | gauge | Number of peers of each type (Repair) |
| gossip_peer_counts {peer_types="voter"} | gauge | Number of peers of each type (Voter) |
| gossip_shred_version_zero | counter | Shred version zero |
| gossip_value_meta_size | gauge | Current size of the CRDS value metas map |
| gossip_value_vec_size | gauge | Current size of the CRDS value vector |
| gossip_received_packets | counter | Number of all gossip packets received |
| gossip_corrupted_messages | counter | Number of corrupted gossip messages received |
| gossip_received_gossip_messages {gossip_message="pull_request"} | counter | Number of gossip messages received (Pull Request) |
| gossip_received_gossip_messages {gossip_message="pull_response"} | counter | Number of gossip messages received (Pull Response) |
| gossip_received_gossip_messages {gossip_message="push"} | counter | Number of gossip messages received (Push) |
| gossip_received_gossip_messages {gossip_message="prune"} | counter | Number of gossip messages received (Prune) |
| gossip_received_gossip_messages {gossip_message="ping"} | counter | Number of gossip messages received (Ping) |
| gossip_received_gossip_messages {gossip_message="pong"} | counter | Number of gossip messages received (Pong) |
| gossip_received_unknown_message | counter | Number of gossip messages received that have an unknown discriminant |
| gossip_received_crds_push {crds_value="contact_info_v1"} | counter | Number of CRDS values received from push messages (Contact Info V1) |
| gossip_received_crds_push {crds_value="vote"} | counter | Number of CRDS values received from push messages (Vote) |
| gossip_received_crds_push {crds_value="lowest_slot"} | counter | Number of CRDS values received from push messages (Lowest Slot) |
| gossip_received_crds_push {crds_value="snapshot_hashes"} | counter | Number of CRDS values received from push messages (Snapshot Hashes) |
| gossip_received_crds_push {crds_value="accounts_hashes"} | counter | Number of CRDS values received from push messages (Accounts Hashes) |
| gossip_received_crds_push {crds_value="epoch_slots"} | counter | Number of CRDS values received from push messages (Epoch Slots) |
| gossip_received_crds_push {crds_value="version_v1"} | counter | Number of CRDS values received from push messages (Version V1) |
| gossip_received_crds_push {crds_value="version_v2"} | counter | Number of CRDS values received from push messages (Version V2) |
| gossip_received_crds_push {crds_value="node_instance"} | counter | Number of CRDS values received from push messages (Node Instance) |
| gossip_received_crds_push {crds_value="duplicate_shred"} | counter | Number of CRDS values received from push messages (Duplicate Shred) |
| gossip_received_crds_push {crds_value="incremental_snapshot_hashes"} | counter | Number of CRDS values received from push messages (Incremental Snapshot Hashes) |
| gossip_received_crds_push {crds_value="contact_info_v2"} | counter | Number of CRDS values received from push messages (Contact Info V2) |
| gossip_received_crds_push {crds_value="restart_last_voted_fork_slots"} | counter | Number of CRDS values received from push messages (Restart Last Voted Fork Slots) |
| gossip_received_crds_push {crds_value="restart_heaviest_fork"} | counter | Number of CRDS values received from push messages (Restart Heaviest Fork) |
| gossip_received_crds_pull {crds_value="contact_info_v1"} | counter | Number of CRDS values received from pull response messages (Contact Info V1) |
| gossip_received_crds_pull {crds_value="vote"} | counter | Number of CRDS values received from pull response messages (Vote) |
| gossip_received_crds_pull {crds_value="lowest_slot"} | counter | Number of CRDS values received from pull response messages (Lowest Slot) |
| gossip_received_crds_pull {crds_value="snapshot_hashes"} | counter | Number of CRDS values received from pull response messages (Snapshot Hashes) |
| gossip_received_crds_pull {crds_value="accounts_hashes"} | counter | Number of CRDS values received from pull response messages (Accounts Hashes) |
| gossip_received_crds_pull {crds_value="epoch_slots"} | counter | Number of CRDS values received from pull response messages (Epoch Slots) |
| gossip_received_crds_pull {crds_value="version_v1"} | counter | Number of CRDS values received from pull response messages (Version V1) |
| gossip_received_crds_pull {crds_value="version_v2"} | counter | Number of CRDS values received from pull response messages (Version V2) |
| gossip_received_crds_pull {crds_value="node_instance"} | counter | Number of CRDS values received from pull response messages (Node Instance) |
| gossip_received_crds_pull {crds_value="duplicate_shred"} | counter | Number of CRDS values received from pull response messages (Duplicate Shred) |
| gossip_received_crds_pull {crds_value="incremental_snapshot_hashes"} | counter | Number of CRDS values received from pull response messages (Incremental Snapshot Hashes) |
| gossip_received_crds_pull {crds_value="contact_info_v2"} | counter | Number of CRDS values received from pull response messages (Contact Info V2) |
| gossip_received_crds_pull {crds_value="restart_last_voted_fork_slots"} | counter | Number of CRDS values received from pull response messages (Restart Last Voted Fork Slots) |
| gossip_received_crds_pull {crds_value="restart_heaviest_fork"} | counter | Number of CRDS values received from pull response messages (Restart Heaviest Fork) |
| gossip_received_crds_duplicate_message_push {crds_value="contact_info_v1"} | counter | Number of duplicate CRDS values received from push messages (Contact Info V1) |
| gossip_received_crds_duplicate_message_push {crds_value="vote"} | counter | Number of duplicate CRDS values received from push messages (Vote) |
| gossip_received_crds_duplicate_message_push {crds_value="lowest_slot"} | counter | Number of duplicate CRDS values received from push messages (Lowest Slot) |
| gossip_received_crds_duplicate_message_push {crds_value="snapshot_hashes"} | counter | Number of duplicate CRDS values received from push messages (Snapshot Hashes) |
| gossip_received_crds_duplicate_message_push {crds_value="accounts_hashes"} | counter | Number of duplicate CRDS values received from push messages (Accounts Hashes) |
| gossip_received_crds_duplicate_message_push {crds_value="epoch_slots"} | counter | Number of duplicate CRDS values received from push messages (Epoch Slots) |
| gossip_received_crds_duplicate_message_push {crds_value="version_v1"} | counter | Number of duplicate CRDS values received from push messages (Version V1) |
| gossip_received_crds_duplicate_message_push {crds_value="version_v2"} | counter | Number of duplicate CRDS values received from push messages (Version V2) |
| gossip_received_crds_duplicate_message_push {crds_value="node_instance"} | counter | Number of duplicate CRDS values received from push messages (Node Instance) |
| gossip_received_crds_duplicate_message_push {crds_value="duplicate_shred"} | counter | Number of duplicate CRDS values received from push messages (Duplicate Shred) |
| gossip_received_crds_duplicate_message_push {crds_value="incremental_snapshot_hashes"} | counter | Number of duplicate CRDS values received from push messages (Incremental Snapshot Hashes) |
| gossip_received_crds_duplicate_message_push {crds_value="contact_info_v2"} | counter | Number of duplicate CRDS values received from push messages (Contact Info V2) |
| gossip_received_crds_duplicate_message_push {crds_value="restart_last_voted_fork_slots"} | counter | Number of duplicate CRDS values received from push messages (Restart Last Voted Fork Slots) |
| gossip_received_crds_duplicate_message_push {crds_value="restart_heaviest_fork"} | counter | Number of duplicate CRDS values received from push messages (Restart Heaviest Fork) |
| gossip_received_crds_duplicate_message_pull {crds_value="contact_info_v1"} | counter | Number of duplicate CRDS values received from pull response messages (Contact Info V1) |
| gossip_received_crds_duplicate_message_pull {crds_value="vote"} | counter | Number of duplicate CRDS values received from pull response messages (Vote) |
| gossip_received_crds_duplicate_message_pull {crds_value="lowest_slot"} | counter | Number of duplicate CRDS values received from pull response messages (Lowest Slot) |
| gossip_received_crds_duplicate_message_pull {crds_value="snapshot_hashes"} | counter | Number of duplicate CRDS values received from pull response messages (Snapshot Hashes) |
| gossip_received_crds_duplicate_message_pull {crds_value="accounts_hashes"} | counter | Number of duplicate CRDS values received from pull response messages (Accounts Hashes) |
| gossip_received_crds_duplicate_message_pull {crds_value="epoch_slots"} | counter | Number of duplicate CRDS values received from pull response messages (Epoch Slots) |
| gossip_received_crds_duplicate_message_pull {crds_value="version_v1"} | counter | Number of duplicate CRDS values received from pull response messages (Version V1) |
| gossip_received_crds_duplicate_message_pull {crds_value="version_v2"} | counter | Number of duplicate CRDS values received from pull response messages (Version V2) |
| gossip_received_crds_duplicate_message_pull {crds_value="node_instance"} | counter | Number of duplicate CRDS values received from pull response messages (Node Instance) |
| gossip_received_crds_duplicate_message_pull {crds_value="duplicate_shred"} | counter | Number of duplicate CRDS values received from pull response messages (Duplicate Shred) |
| gossip_received_crds_duplicate_message_pull {crds_value="incremental_snapshot_hashes"} | counter | Number of duplicate CRDS values received from pull response messages (Incremental Snapshot Hashes) |
| gossip_received_crds_duplicate_message_pull {crds_value="contact_info_v2"} | counter | Number of duplicate CRDS values received from pull response messages (Contact Info V2) |
| gossip_received_crds_duplicate_message_pull {crds_value="restart_last_voted_fork_slots"} | counter | Number of duplicate CRDS values received from pull response messages (Restart Last Voted Fork Slots) |
| gossip_received_crds_duplicate_message_pull {crds_value="restart_heaviest_fork"} | counter | Number of duplicate CRDS values received from pull response messages (Restart Heaviest Fork) |
| gossip_received_crds_drop {crds_drop_reason="success"} | counter | Number of CRDS values dropped on receive (Successfully processed CRDS (not dropped)) |
| gossip_received_crds_drop {crds_drop_reason="duplicate"} | counter | Number of CRDS values dropped on receive (Duplicate CRDS value) |
| gossip_received_crds_drop {crds_drop_reason="unknown_discriminant"} | counter | Number of CRDS values dropped on receive (Unknown discriminant) |
| gossip_received_crds_drop {crds_drop_reason="own_message"} | counter | Number of CRDS values dropped on receive (Own message) |
| gossip_received_crds_drop {crds_drop_reason="invalid_signature"} | counter | Number of CRDS values dropped on receive (Invalid signature) |
| gossip_received_crds_drop {crds_drop_reason="table_full"} | counter | Number of CRDS values dropped on receive (Table full) |
| gossip_received_crds_drop {crds_drop_reason="push_queue_full"} | counter | Number of CRDS values dropped on receive (Push queue full) |
| gossip_received_crds_drop {crds_drop_reason="invalid_gossip_port"} | counter | Number of CRDS values dropped on receive (Invalid gossip port) |
| gossip_received_crds_drop {crds_drop_reason="peer_table_full"} | counter | Number of CRDS values dropped on receive (Peer table full) |
| gossip_received_crds_drop {crds_drop_reason="inactives_queue_full"} | counter | Number of CRDS values dropped on receive (Inactives queue full) |
| gossip_received_crds_drop {crds_drop_reason="discarded_peer"} | counter | Number of CRDS values dropped on receive (Discarded peer) |
| gossip_received_crds_drop {crds_drop_reason="encoding_failed"} | counter | Number of CRDS values dropped on receive (Encoding failed) |
| gossip_push_crds {crds_value="contact_info_v1"} | counter | Number of CRDS values pushed (Contact Info V1) |
| gossip_push_crds {crds_value="vote"} | counter | Number of CRDS values pushed (Vote) |
| gossip_push_crds {crds_value="lowest_slot"} | counter | Number of CRDS values pushed (Lowest Slot) |
| gossip_push_crds {crds_value="snapshot_hashes"} | counter | Number of CRDS values pushed (Snapshot Hashes) |
| gossip_push_crds {crds_value="accounts_hashes"} | counter | Number of CRDS values pushed (Accounts Hashes) |
| gossip_push_crds {crds_value="epoch_slots"} | counter | Number of CRDS values pushed (Epoch Slots) |
| gossip_push_crds {crds_value="version_v1"} | counter | Number of CRDS values pushed (Version V1) |
| gossip_push_crds {crds_value="version_v2"} | counter | Number of CRDS values pushed (Version V2) |
| gossip_push_crds {crds_value="node_instance"} | counter | Number of CRDS values pushed (Node Instance) |
| gossip_push_crds {crds_value="duplicate_shred"} | counter | Number of CRDS values pushed (Duplicate Shred) |
| gossip_push_crds {crds_value="incremental_snapshot_hashes"} | counter | Number of CRDS values pushed (Incremental Snapshot Hashes) |
| gossip_push_crds {crds_value="contact_info_v2"} | counter | Number of CRDS values pushed (Contact Info V2) |
| gossip_push_crds {crds_value="restart_last_voted_fork_slots"} | counter | Number of CRDS values pushed (Restart Last Voted Fork Slots) |
| gossip_push_crds {crds_value="restart_heaviest_fork"} | counter | Number of CRDS values pushed (Restart Heaviest Fork) |
| gossip_push_crds_duplicate_message {crds_value="contact_info_v1"} | counter | Number of duplicate CRDS values inserted (internally) (Contact Info V1) |
| gossip_push_crds_duplicate_message {crds_value="vote"} | counter | Number of duplicate CRDS values inserted (internally) (Vote) |
| gossip_push_crds_duplicate_message {crds_value="lowest_slot"} | counter | Number of duplicate CRDS values inserted (internally) (Lowest Slot) |
| gossip_push_crds_duplicate_message {crds_value="snapshot_hashes"} | counter | Number of duplicate CRDS values inserted (internally) (Snapshot Hashes) |
| gossip_push_crds_duplicate_message {crds_value="accounts_hashes"} | counter | Number of duplicate CRDS values inserted (internally) (Accounts Hashes) |
| gossip_push_crds_duplicate_message {crds_value="epoch_slots"} | counter | Number of duplicate CRDS values inserted (internally) (Epoch Slots) |
| gossip_push_crds_duplicate_message {crds_value="version_v1"} | counter | Number of duplicate CRDS values inserted (internally) (Version V1) |
| gossip_push_crds_duplicate_message {crds_value="version_v2"} | counter | Number of duplicate CRDS values inserted (internally) (Version V2) |
| gossip_push_crds_duplicate_message {crds_value="node_instance"} | counter | Number of duplicate CRDS values inserted (internally) (Node Instance) |
| gossip_push_crds_duplicate_message {crds_value="duplicate_shred"} | counter | Number of duplicate CRDS values inserted (internally) (Duplicate Shred) |
| gossip_push_crds_duplicate_message {crds_value="incremental_snapshot_hashes"} | counter | Number of duplicate CRDS values inserted (internally) (Incremental Snapshot Hashes) |
| gossip_push_crds_duplicate_message {crds_value="contact_info_v2"} | counter | Number of duplicate CRDS values inserted (internally) (Contact Info V2) |
| gossip_push_crds_duplicate_message {crds_value="restart_last_voted_fork_slots"} | counter | Number of duplicate CRDS values inserted (internally) (Restart Last Voted Fork Slots) |
| gossip_push_crds_duplicate_message {crds_value="restart_heaviest_fork"} | counter | Number of duplicate CRDS values inserted (internally) (Restart Heaviest Fork) |
| gossip_push_crds_drop {crds_drop_reason="success"} | counter | Number of CRDS values dropped on push (Successfully processed CRDS (not dropped)) |
| gossip_push_crds_drop {crds_drop_reason="duplicate"} | counter | Number of CRDS values dropped on push (Duplicate CRDS value) |
| gossip_push_crds_drop {crds_drop_reason="unknown_discriminant"} | counter | Number of CRDS values dropped on push (Unknown discriminant) |
| gossip_push_crds_drop {crds_drop_reason="own_message"} | counter | Number of CRDS values dropped on push (Own message) |
| gossip_push_crds_drop {crds_drop_reason="invalid_signature"} | counter | Number of CRDS values dropped on push (Invalid signature) |
| gossip_push_crds_drop {crds_drop_reason="table_full"} | counter | Number of CRDS values dropped on push (Table full) |
| gossip_push_crds_drop {crds_drop_reason="push_queue_full"} | counter | Number of CRDS values dropped on push (Push queue full) |
| gossip_push_crds_drop {crds_drop_reason="invalid_gossip_port"} | counter | Number of CRDS values dropped on push (Invalid gossip port) |
| gossip_push_crds_drop {crds_drop_reason="peer_table_full"} | counter | Number of CRDS values dropped on push (Peer table full) |
| gossip_push_crds_drop {crds_drop_reason="inactives_queue_full"} | counter | Number of CRDS values dropped on push (Inactives queue full) |
| gossip_push_crds_drop {crds_drop_reason="discarded_peer"} | counter | Number of CRDS values dropped on push (Discarded peer) |
| gossip_push_crds_drop {crds_drop_reason="encoding_failed"} | counter | Number of CRDS values dropped on push (Encoding failed) |
| gossip_push_crds_queue_count | gauge | Number of CRDS values in the queue to be pushed |
| gossip_active_push_destinations | gauge | Number of active Push destinations |
| gossip_refresh_push_states_fail_count | counter | Number of failures whilst refreshing push states |
| gossip_pull_req_fail {pull_req_fail_reason="peer_not_in_actives"} | counter | Number of PullReq messages that failed (Number of PullReq messages from peers that are not in the active set) |
| gossip_pull_req_fail {pull_req_fail_reason="unresponsive_peer"} | counter | Number of PullReq messages that failed (Number of PullReq messages from a peer that hasn't responded to our ping message yet) |
| gossip_pull_req_fail {pull_req_fail_reason="pending_pool_full"} | counter | Number of PullReq messages that failed (Number of PullReq messages skipped due to the pending pool being full) |
| gossip_pull_req_fail {pull_req_fail_reason="encoding_failed"} | counter | Number of PullReq messages that failed (Number of PullReq messages skipped due to message encoding failed) |
| gossip_pull_req_bloom_filter {pull_req_bloom_filter_result="hit"} | counter | Result of the bloom filter check for a PullReq (Number of PullReq messages that hit the bloom filter) |
| gossip_pull_req_bloom_filter {pull_req_bloom_filter_result="miss"} | counter | Result of the bloom filter check for a PullReq (Number of PullReq messages that missed the bloom filter) |
| gossip_pull_req_resp_packets | gauge | Number of packets used to respond to a PullReq |
| gossip_prune_fail_count {prune_failure_reason="not_for_me"} | counter | Number of Prune messages that failed (Prune message not for me) |
| gossip_prune_fail_count {prune_failure_reason="sign_encoding_failed"} | counter | Number of Prune messages that failed (Prune message sign encoding failed) |
| gossip_prune_fail_count {prune_failure_reason="invalid_signature"} | counter | Number of Prune messages that failed (Prune message invalid signature) |
| gossip_make_prune_stale_entry | counter | Number of stale entries removed from the stats table while making prune messages |
| gossip_make_prune_high_duplicates | counter | Number of origins with high duplicate counts found while making prune messages |
| gossip_make_prune_requested_origins | gauge | Number of requested origins in the last prune message we made |
| gossip_make_prune_sign_data_encode_failed | counter | Number of times we failed to encode the sign data |
| gossip_sent_gossip_messages {gossip_message="pull_request"} | counter | Number of gossip messages sent (Pull Request) |
| gossip_sent_gossip_messages {gossip_message="pull_response"} | counter | Number of gossip messages sent (Pull Response) |
| gossip_sent_gossip_messages {gossip_message="push"} | counter | Number of gossip messages sent (Push) |
| gossip_sent_gossip_messages {gossip_message="prune"} | counter | Number of gossip messages sent (Prune) |
| gossip_sent_gossip_messages {gossip_message="ping"} | counter | Number of gossip messages sent (Ping) |
| gossip_sent_gossip_messages {gossip_message="pong"} | counter | Number of gossip messages sent (Pong) |
| gossip_sent_packets | counter | Number of Packets sent |
| gossip_send_ping_event {send_ping_event="actives_table_full"} | counter | Number of Ping messages sent with non-standard outcomes (Number of Ping messages we failed to send due to the Active Peers table being full) |
| gossip_send_ping_event {send_ping_event="actives_table_insert"} | counter | Number of Ping messages sent with non-standard outcomes (Number of Ping messages that cause an insert into the Active Peers table) |
| gossip_send_ping_event {send_ping_event="max_ping_count_exceeded"} | counter | Number of Ping messages sent with non-standard outcomes (Number of times we removed a peer from the Actives table, because it repeatedly failed to respond to a ping) |
| gossip_recv_ping_invalid_signature | counter | Number of times we received a Ping message with an invalid signature |
| gossip_recv_pong_event {recv_pong_event="new_peer"} | counter | Number of Pong messages processed with non-standard outcomes (Pong peer is not in table) |
| gossip_recv_pong_event {recv_pong_event="wrong_token"} | counter | Number of Pong messages processed with non-standard outcomes (Pong peer token mismatch) |
| gossip_recv_pong_event {recv_pong_event="invalid_signature"} | counter | Number of Pong messages processed with non-standard outcomes (Pong peer invalid signature) |
| gossip_recv_pong_event {recv_pong_event="expired"} | counter | Number of Pong messages processed with non-standard outcomes (Pong peer expired) |
| gossip_recv_pong_event {recv_pong_event="table_full"} | counter | Number of Pong messages processed with non-standard outcomes (Unable to insert peer due to table full) |
| gossip_gossip_peer_counts {gossip_peer_state="total"} | gauge | Number of gossip peers tracked (Total Peers Detected) |
| gossip_gossip_peer_counts {gossip_peer_state="active"} | gauge | Number of gossip peers tracked (Active) |
| gossip_gossip_peer_counts {gossip_peer_state="inactive"} | gauge | Number of gossip peers tracked (Inactive) |
Netlnk Tile
| Metric | Type | Description |
|---|---|---|
| netlnk_drop_events | counter | Number of netlink drop events caught |
| netlnk_link_full_syncs | counter | Number of full link table syncs done |
| netlnk_route_full_syncs | counter | Number of full route table syncs done |
| netlnk_updates {netlink_msg="link"} | counter | Number of netlink live updates processed (Link) |
| netlnk_updates {netlink_msg="neigh"} | counter | Number of netlink live updates processed (Neighbor Table Entry) |
| netlnk_updates {netlink_msg="ipv4_route"} | counter | Number of netlink live updates processed (IPv4 Route Table Entry) |
| netlnk_interface_count | gauge | Number of network interfaces |
| netlnk_route_count {route_table="local"} | gauge | Number of IPv4 routes (Local) |
| netlnk_route_count {route_table="main"} | gauge | Number of IPv4 routes (Main) |
| netlnk_neigh_probe_sent | counter | Number of neighbor solicit requests sent to kernel |
| netlnk_neigh_probe_fails | counter | Number of neighbor solicit requests that failed to send (kernel too slow) |
| netlnk_neigh_probe_rate_limit_host | counter | Number of neighbor solicit that exceeded the per-host rate limit |
| netlnk_neigh_probe_rate_limit_global | counter | Number of neighbor solicit that exceeded the global rate limit |
Sock Tile
| Metric | Type | Description |
|---|---|---|
| sock_syscalls_sendmmsg {sock_err="no_error"} | counter | Number of sendmmsg syscalls dispatched (No error) |
| sock_syscalls_sendmmsg {sock_err="slow"} | counter | Number of sendmmsg syscalls dispatched (ENOBUFS, EAGAIN error) |
| sock_syscalls_sendmmsg {sock_err="perm"} | counter | Number of sendmmsg syscalls dispatched (EPERM error (blocked by netfilter)) |
| sock_syscalls_sendmmsg {sock_err="unreach"} | counter | Number of sendmmsg syscalls dispatched (ENETUNREACH, EHOSTUNREACH error) |
| sock_syscalls_sendmmsg {sock_err="down"} | counter | Number of sendmmsg syscalls dispatched (ENONET, ENETDOWN, EHOSTDOWN error) |
| sock_syscalls_sendmmsg {sock_err="other"} | counter | Number of sendmmsg syscalls dispatched (Unrecognized error code) |
| sock_syscalls_recvmmsg | counter | Number of recvmsg syscalls dispatched |
| sock_rx_pkt_cnt | counter | Number of packets received |
| sock_tx_pkt_cnt | counter | Number of packets sent |
| sock_tx_drop_cnt | counter | Number of packets failed to send |
| sock_tx_bytes_total | counter | Total number of bytes transmitted (including Ethernet header). |
| sock_rx_bytes_total | counter | Total number of bytes received (including Ethernet header). |
Repair Tile
| Metric | Type | Description |
|---|---|---|
| repair_recv_clnt_pkt | counter | Now many client packets have we received |
| repair_recv_serv_pkt | counter | How many server packets have we received |
| repair_recv_serv_corrupt_pkt | counter | How many corrupt server packets have we received |
| repair_recv_serv_invalid_signature | counter | How many invalid signatures have we received |
| repair_recv_serv_full_ping_table | counter | Is our ping table full and causing packet drops |
| repair_recv_serv_pkt_types {repair_serv_pkt_types="pong"} | counter | Server messages received (Pong) |
| repair_recv_serv_pkt_types {repair_serv_pkt_types="window"} | counter | Server messages received (Window) |
| repair_recv_serv_pkt_types {repair_serv_pkt_types="highest_window"} | counter | Server messages received (Highest Window) |
| repair_recv_serv_pkt_types {repair_serv_pkt_types="orphan"} | counter | Server messages received (Orphan) |
| repair_recv_serv_pkt_types {repair_serv_pkt_types="unknown"} | counter | Server messages received (Unknown) |
| repair_recv_pkt_corrupted_msg | counter | How many corrupt messages have we received |
| repair_send_pkt_cnt | counter | How many packets have sent |
| repair_sent_pkt_types {repair_sent_request_types="needed_window"} | counter | What types of client messages are we sending (Need Window) |
| repair_sent_pkt_types {repair_sent_request_types="needed_highest_window"} | counter | What types of client messages are we sending (Need Highest Window) |
| repair_sent_pkt_types {repair_sent_request_types="needed_orphan"} | counter | What types of client messages are we sending (Need Orphans) |
Sender Tile
| Metric | Type | Description |
|---|---|---|
| sender_txns_sent_to_leader | counter | Total count of transactions sent to leader |
| sender_leader_sched_not_found | counter | Total count of times leader schedule not found |
| sender_leader_not_found | counter | Total count of times leader not found for given slot |
| sender_leader_contact_not_found | counter | Total count of times leader contact info not found |
| sender_leader_contact_nonroutable | counter | Total count of times leader contact is nonroutable |