Skip to content

firedancer Command Line Interface ​

The Firedancer binary firedancer contains many subcommands which can be run from the command line.

Commands that attach to a running validator fall into two groups. set-identity, get-identity, and add-authorized-voter are versioned and work across releases. Diagnostic commands like monitor, watch, and metrics read the validator's memory directly and must be run from the same binary the validator is running.

run ​

Runs the validator. This command does not exit until the validator does, an error in running the validator will be propagated to the exit code of the process. The command can be run with the capabilities listed below but it is suggested to run it as sudo. The command writes an abbreviated log output to stderr and nothing will be written to stdout.

ArgumentsDescription
--config <path>Path to a configuration TOML file to run the validator with
--standbyBoot as a failover standby using an unstaked identity, see the failover guide
Capabilities
CapabilityReason
CAP_NET_RAWcall socket(2) to bind to a raw socket for use by XDP
CAP_SYS_ADMINcall bpf(2) with the BPF_OBJ_GET command to initialize XDP
CAP_SYS_ADMINcall unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces
CAP_SETUIDcall setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID
CAP_SETGIDcall setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID
CAP_SYS_RESOURCEcall rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit
CAP_SYS_RESOURCEcall setpriority(2) to increase thread priorities. Not required if the process already has a nice value of -19
CAP_SYS_RESOURCEcall rlimit(2) to increase RLIMIT_NOFILE` to allow more open files for Agave. Not required if the resource limit is already high enough
CAP_NET_BIND_SERVICEcall bind(2) to bind to a privileged port for serving metrics. Only required if the bind port is below 1024
$ firedancer run --config config.toml
Log at "/tmp/fd-0.1.1_1987461_sun-ossdev-firedancer31_2026_10_20_16_32_23_068864834_GMT+00"
WARNING 10-20 16:32:23.089738 main      nohz-full.c(71): tile cpus 1-38 are not in the kernel nohz_full= set and will be interrupted by periodic timer ticks. For lower jitter, boot with nohz_full=1-38.
WARNING 10-20 16:32:24.188608 main      fd_topo.c(159): fd_topo_wksp_new(store) took 165 ms
NOTICE  10-20 16:32:24.306879 main      fd_bootinfo.c(168): booting validator fd1 version 0.1.1 (babe91c83ca) pid 1987461 with 55 tiles and 87 GiB memory at /mnt/.fd
NOTICE  10-20 16:32:24.362519 metric:0  fd_metric_tile.c(148): prometheus server listening at http://127.0.0.1:7999/metrics
WARNING 10-20 16:32:24.419250 votor:0   fd_votor_tile.c(2508): vote history file `/home/firedancer/.firedancer/fd1/vote_history-4Dm5AX6DruesFVJYK5ajk2Gyymqmscgx1PoXbYfQbGSY.bin` does not exist, creating it.  This identity is assumed to have never voted.
NOTICE  10-20 16:32:24.438850 gui:0     fd_gui_tile.c(857): gui server listening at http://127.0.0.1:80
NOTICE  10-20 16:32:24.442772 rpc:0     fd_rpc_tile.c(3155): rpc server listening at http://127.0.0.1:8899
NOTICE  10-20 16:32:25.290178 genesi:0  fd_genesi_tile.c(383): retrieved genesis /home/firedancer/.firedancer/fd1/genesis.bin from peer at http://109.94.99.177:8899/genesis.tar.bz2
NOTICE  10-20 16:32:25.454314 event:0   fd_event_client.c(687): connected to telemetry server https://events-in.firedancer.io:443
NOTICE  10-20 16:32:27.370526 snapct:0  fd_snapct_tile.c(756): downloading full snapshot from untrusted gossip peer 141vSYKGRPNGieSrGJy8EeDVBcbjSr6aWkimNgrNZ6xN
NOTICE  10-20 16:32:38.820601 snapct:0  fd_snapct_tile.c(756): downloading incremental snapshot from untrusted gossip peer 141vSYKGRPNGieSrGJy8EeDVBcbjSr6aWkimNgrNZ6xN
NOTICE  10-20 16:32:38.858957 snapin:0  fd_snapin_tile.c(389): loaded 89.3M accounts (9.2K dups) from snapshot in 14.466 seconds
NOTICE  10-20 16:32:41.805075 replay:0  fd_replay_tile.c(1313): caught up to cluster at slot 454774712 (19.3 seconds since boot)

monitor ​

Monitors a validator that is running locally on this machine. This is a low level performance monitor mostly useful for diagnosing throughput issues. The monitor takes over the controlling terminal and refreshes it many times a second with up to date information. You can exit the monitor by sending Ctrl+C or SIGINT.

ArgumentsDescription
--config <path>Path to a configuration TOML file to run the monitor with. This must be the same configuration file the validator was started with
Capabilities
CapabilityReason
CAP_SYS_ADMINcall unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces
CAP_SETUIDcall setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID
CAP_SETGIDcall setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID
CAP_SYS_RESOURCEcall rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit
$ firedancer monitor --config config.toml
snapshot for 2026-10-20 16:29:11.311695967 GMT+00 | Use TAB to switch panes
    tile |     tid |      stale | heart | nivcsw              | nvcsw               | in backp |           backp cnt |  % hkeep |  % wait  |  % backp | % finish
---------+---------+------------+-------+---------------------+---------------------+----------+---------------------+----------+----------+----------+----------
  netlnk | 1983482 |          - |     - |          1(     +0) |      92592(     +0) |   -(  -) |          0(     +0) |    0.002 |   99.998 |    0.000 |    0.000
     net | 1983506 |          - |     - |         29(     +0) |          8(     +0) |   -(  -) |          0(     +0) |    0.684 |   99.316 |    0.000 |    0.000
     net | 1983508 |          - |     - |         27(     +0) |          4(     +0) |   -(  -) |          0(     +0) |    0.678 |   99.322 |    0.000 |    0.000
  metric | 1983483 |          - |     - |          0(     +0) |      92626(     +0) |   -(  -) |          0(     +0) |    0.003 |   99.997 |    0.000 |    0.000
    diag | 1983484 |          - |     - |          0(     +0) |       4989(     +0) |   -(  -) |          0(     +0) |    0.003 |   99.997 |    0.000 |    0.000
  ipecho | 1983486 |          - |     - |          3(     +0) |      92511(     +0) |   -(  -) |          1(     +0) |    0.003 |   99.997 |    0.000 |    0.000
  gossvf | 1983512 |          - |     - |         27(     +0) |          3(     +0) |   -(  -) |          0(     +0) |    0.049 |   99.951 |    0.000 |    0.000
  gossip | 1983511 |          - |     - |         37(     +0) |          5(     +0) |   -(  -) |          0(     +0) |    0.092 |   99.897 |    0.000 |    0.011
   shred | 1983514 |          - |     - |         26(     +0) |          5(     +0) |   -(  -) |          0(     +0) |    0.086 |   99.914 |    0.000 |    0.000
  repair | 1983515 |          - |     - |         30(     +0) |          5(     +0) |   -(  -) |          0(     +0) |    0.086 |   99.914 |    0.000 |    0.000
  rserve | 1983517 |          - |     - |         61(     +0) |          5(     +0) |   -(  -) |          0(     +0) |    0.491 |   99.509 |    0.000 |    0.000
  replay | 1983528 |          - |     - |         27(     +0) |          5(     +0) |   -(  -) |          0(     +0) |    2.738 |   97.262 |    0.000 |    0.000
   accdb | 1983529 |          - |     - |         40(     +0) |         55(     +0) |   -(  -) |          0(     +0) |    0.018 |   99.982 |    0.000 |    0.000
  execrp | 1983531 |          - |     - |         27(     +0) |         64(     +0) |   -(  -) |          0(     +0) |    0.019 |   99.981 |    0.000 |    0.000
  execrp | 1983532 |          - |     - |         27(     +0) |         67(     +0) |   -(  -) |          0(     +0) |    0.020 |   99.980 |    0.000 |    0.000
  execrp | 1983534 |          - |     - |         26(     +0) |         65(     +0) |   -(  -) |          0(     +0) |    0.020 |   99.980 |    0.000 |    0.000
  execrp | 1983535 |          - |     - |         26(     +0) |         68(     +0) |   -(  -) |          0(     +0) |    0.020 |   99.980 |    0.000 |    0.000
  execrp | 1983538 |          - |     - |         29(     +0) |         60(     +0) |   -(  -) |          0(     +0) |    0.020 |   99.980 |    0.000 |    0.000
  execrp | 1983547 |          - |     - |         25(     +0) |         59(     +0) |   -(  -) |          0(     +0) |    0.020 |   99.980 |    0.000 |    0.000
  execrp | 1983555 |          - |     - |         25(     +0) |         63(     +0) |   -(  -) |          0(     +0) |    0.024 |   99.976 |    0.000 |    0.000
  execrp | 1983556 |          - |     - |         25(     +0) |         63(     +0) |   -(  -) |          0(     +0) |    0.020 |   99.980 |    0.000 |    0.000
  execrp | 1983554 |          - |     - |         26(     +0) |         58(     +0) |   -(  -) |          0(     +0) |    0.023 |   99.977 |    0.000 |    0.000
  execrp | 1983557 |          - |     - |         24(     +0) |         59(     +0) |   -(  -) |          0(     +0) |    0.022 |   99.978 |    0.000 |    0.000
   votor | 1983566 |          - |     - |         27(     +0) |          3(     +0) |   -(  -) |          0(     +0) |    0.110 |   99.890 |    0.000 |    0.000
    quic | 1983569 |          - |     - |         28(     +0) |         15(     +0) |   -(  -) |          0(     +0) |    0.004 |   99.996 |    0.000 |    0.000
  verify | 1983568 |          - |     - |         27(     +0) |         16(     +0) |   -(  -) |          0(     +0) |    6.555 |   93.445 |    0.000 |    0.000
  verify | 1983570 |          - |     - |         27(     +0) |         16(     +0) |   -(  -) |          0(     +0) |    6.543 |   93.457 |    0.000 |    0.000
  verify | 1983572 |          - |     - |         26(     +0) |         15(     +0) |   -(  -) |          0(     +0) |    6.625 |   93.375 |    0.000 |    0.000
   dedup | 1983573 |          - |     - |         27(     +0) |          8(     +0) |   -(  -) |          0(     +0) |    0.398 |   99.602 |    0.000 |    0.000
  resolv | 1983575 |          - |     - |         26(     +0) |         17(     +0) |   -(  -) |          0(     +0) |    0.102 |   99.898 |    0.000 |    0.000
    pack | 1983584 |          - |     - |         28(     +0) |         16(     +0) |   -(  -) |          0(     +0) |    0.101 |   15.631 |    0.000 |   84.268
  execle | 1983586 |          - |     - |         24(     +0) |         48(     +0) |   -(  -) |          0(     +0) |    0.057 |   99.943 |    0.000 |    0.000
  execle | 1983585 |          - |     - |         27(     +0) |         51(     +0) |   -(  -) |          0(     +0) |    0.058 |   99.942 |    0.000 |    0.000
   motor | 1983587 |          - |     - |         24(     +0) |         44(     +0) |   -(  -) |          0(     +0) |    0.044 |   99.956 |    0.000 |    0.000
    sign | 1983593 |          - |     - |         27(     +0) |          3(     +0) |   -(  -) |          0(     +0) |    0.045 |   99.955 |    0.000 |    0.000
    sign | 1983594 |          - |     - |         40(     +0) |          4(     +0) |   -(  -) |          0(     +0) |    0.017 |   99.983 |    0.000 |    0.000
     rpc | 1983595 |          - |     - |         75(     +0) |          3(     +0) |   -(  -) |          0(     +0) |    0.049 |   99.951 |    0.000 |    0.000
   admin | 1983489 |          - |     - |          0(     +0) |       4855(     +0) |   -(  -) |          0(     +0) |    0.002 |   99.998 |    0.000 |    0.000
   event | 1983598 |          - |     - |         52(     +0) |         23(     +0) |   -(  -) |          0(     +0) |    0.006 |   99.982 |    0.000 |    0.012
     gui | 1983599 |          - |     - |         27(     +0) |          3(     +0) |   -(  -) |          0(     +0) |    0.163 |   99.528 |    0.000 |    0.309
   waker | 1983490 |          - |     - |          1(     +0) |      10303(     +0) |   -(  -) |          0(     +0) |    0.003 |   99.997 |    0.000 |    0.000

configure ​

Configures the operating system so that it can run Firedancer. See the guide for more information. There are the following stages to each configure command:

  • iavf Configures VFs, clears and installs PF ntuple rules and loads the vfio-pci kernel module required by the iavf network provider.
  • mlx5 Loads the ib_uverbs kernel module required by the mlx5 network provider. Clears leftover XDP ntuple rules, resets the RSS indirection table and channel count to defaults.
  • hugetlbfs Reserves huge and gigantic pages for use by Firedancer and mounts huge page filesystems for then under a path in the configuration TOML file.
  • sysctl Set required kernel parameters.
  • hyperthreads Checks hyperthreaded pair for critical CPU cores.
  • bonding Prepares bonded network devices for XDP networking.
  • ethtool-channels Configures the number of channels on the network device.
  • ethtool-offloads Modify offload feature flags on the network device.
  • ethtool-loopback Disables UDP segmentation on the loopback device.
  • irq-affinity Removes Firedancer tile CPUs from configurable /proc/irq/*/smp_affinity masks.
  • irq-balance Configures the irqbalance daemon to avoid Firedancer tile CPUs. If irqbalance is not running, this stage is a no-op.
  • snapshots Prepares the snapshot download directory.
ArgumentsDescription
--config <path>Path to a configuration TOML file to configure the validator with. This must be the same configuration file the validator will be started with
toml
[hugetlbfs]
    mount_path = "/mnt/.fd"
[layout]
    net_tile_count = 2
[tiles]
    [net]
        interface = "ens3f0"

configure init <stage>... ​

Prepare the operating system environment to run Firedancer. This will reserve and mount the huge page filesystems, set the kernel parameters, and configure the number of combined channels on the network device.

Capabilities
CapabilityReason
rootincrease /proc/sys/vm/nr_hugepages and mount hugetlbfs filesystems. Only applies for the hugetlbfs stage
rootincrease network device channels with ethtool --set-channels. Only applies for the ethtool-channels stage
rootdisable network device offloads with ethtool --offload IFACE FEATURE off. Only applies for the ethtool-offloads stage
rootdisable network device tx-udp-segmentation with ethtool --offload lo tx-udp-segmentation off. Only applies for the ethtool-loopback stage
rootrestore network device defaults and load the ib_uverbs kernel module if needed
CAP_SYS_MODULEload the ib_uverbs kernel module with modprobe when the uverbs interface is unavailable
rootconfigure VFs and PF ntuple rules. Only applies for the iavf stage
CAP_SYS_MODULEload the vfio-pci kernel module with modprobe when the driver is unavailable. Only applies for the iavf stage
CAP_SYS_ADMINset kernel parameters in /proc/sys. Only applies for the sysctl stage
$ firedancer configure init all
NOTICE  hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE  hugetlbfs ... configuring
NOTICE  RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE  RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=6155141120`
NOTICE  RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE  RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=150323855360`
NOTICE  ethtool-channels ... unconfigured ... device `bond0` missing ethtool-channels network configuration
NOTICE  ethtool-channels ... configuring
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 start 2 equal 126`
NOTICE  RUN: `ethtool --config-ntuple enp1s0f0np0 flow-type udp4 dst-port 8003 src-ip 0.0.0.0 m 255.255.255.254 queue 0`
WARNING error configuring network device (enp1s0f0np0), ioctl(SIOCETHTOOL,ETHTOOL_SRXCLSRLINS) failed (95-ENOTSUP-operation not supported)
WARNING error configuring network device (bond0), rss_queue_mode "auto" attempted "dedicated" configuration but falling back to "simple".
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 2`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --config-nfc enp1s0f0np0 rx-flow-hash udp4 sdfn`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f1np1 combined 2`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE  RUN: `ethtool --config-nfc enp1s0f1np1 rx-flow-hash udp4 sdfn`
NOTICE  irq-affinity ... unconfigured ... found IRQs overlapping with Firedancer tile CPUs (IRQs: 1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,+45 more; tiles: )
NOTICE  irq-affinity ... configuring
NOTICE  kworkers ... unconfigured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
NOTICE  kworkers ... configuring
NOTICE  RUN: `echo "fffffff0,03fffff9,fffffff0,00000001" > /sys/devices/virtual/workqueue/cpumask`
NOTICE  kernel workqueue cpumask changed while writing `/sys/devices/virtual/workqueue/cpumask`: fffffe00,40000001,fffff800,00000001 -> fffffff0,03fffff9,fffffff0,00000001 (kernel 7.0.0-34-generic). 
NOTICE  cpuset ... unconfigured ... `/sys/fs/cgroup/fd1` does not exist
NOTICE  cpuset ... configuring
NOTICE  RUN: `echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control`
NOTICE  RUN: `mkdir /sys/fs/cgroup/fd1`
NOTICE  RUN: `echo "1-35,65-66,90-99" > /sys/fs/cgroup/fd1/cpuset.cpus`
NOTICE  RUN: `echo "isolated" > /sys/fs/cgroup/fd1/cpuset.cpus.partition`
NOTICE  console ... unconfigured ... console loglevel is 4; kernel messages render console work on the printing CPU
NOTICE  console ... configuring
NOTICE  RUN: `echo "0" > /sys/class/graphics/fbcon/cursor_blink`
NOTICE  RUN: `echo "2" > /proc/sys/kernel/printk`

configure check <stage>... ​

Check if the operating system environment is properly configured. Exits with a non-zero exit code if it is not, after printing relevant diagnostics to stderr.

$ firedancer configure check all
WARNING hugetlbfs ... not configured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
WARNING ethtool-channels ... not configured ... device `bond0` missing ethtool-channels network configuration
WARNING irq-affinity ... not configured ... insufficient permissions to write /proc/irq/0/smp_affinity
WARNING kworkers ... not configured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
WARNING cpuset ... not configured ... `/sys/fs/cgroup/fd1` does not exist
WARNING console ... not configured ... console loglevel is 4; kernel messages render console work on the printing CPU
ERR     failed to configure some stages

configure fini <stage>... ​

Remove any Firedancer specific operating system configuration still lingering. The hugetlbfs stage unmounts its filesystems and returns the reserved huge and gigantic pages to the kernel pool. The mlx5 stage clears leftover XDP ntuple rules and resets the RSS indirection table and channel count to defaults. The iavf stage clears PF ntuple rules and removes VF 0 if its PF has exactly one VF. PFs with multiple VFs retain all of them. The fini mode does not reduce sysctls that were earlier increased or restore previous network settings.

Capabilities
CapabilityReason
rootremove directories from /mnt, unmount hugetlbfs. Only applies for the hugetlbfs stage
rootrestore network device defaults
rootclear PF ntuple rules and remove VFs. Only applies for the iavf stage
$ firedancer configure fini all
NOTICE  snapshots ... finishing
NOTICE  console ... not configured ... console loglevel is 4; kernel messages render console work on the printing CPU
NOTICE  rcu-nocbs ... finishing
NOTICE  nohz-full ... finishing
NOTICE  cpuset ... not configured ... `/sys/fs/cgroup/fd1` does not exist
NOTICE  kworkers ... not configured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
NOTICE  irq-affinity ... not configured ... found IRQs overlapping with Firedancer tile CPUs (IRQs: 1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,+45 more; tiles: )
NOTICE  irq-balance ... finishing
NOTICE  ethtool-loopback ... finishing
NOTICE  ethtool-offloads ... finishing
NOTICE  ethtool-channels ... finishing
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f1np1 combined 128`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE  sysctl ... finishing
NOTICE  hugetlbfs ... finishing
NOTICE  RUN: `umount /mnt/.fd/.huge`
NOTICE  RUN: `rmdir /mnt/.fd/.huge`
NOTICE  RUN: `umount /mnt/.fd/.gigantic`
NOTICE  RUN: `rmdir /mnt/.fd/.gigantic`
NOTICE  RUN: `rmdir /mnt/.fd/.normal`
NOTICE  RUN: `rmdir /mnt/.fd`

version ​

Prints the current version of the validator to the standard output and exits. The command writes diagnostic messages from logs to stderr.

sh
$ firedancer version
26.10.3

shred-version ​

Prints the current shred version of the cluster being joined, according to the entrypoints, to standard output and exits. The command writes diagnostic messages from logs to stderr.

sh
$ firedancer shred-version
50093

metrics ​

Prints the current validator metrics to stdout. Metrics can typically be accessed via HTTP when the metric tile is enabled, but the command can be used even if the metrics server is not enabled, or the validator has crashed.

sh
$ firedancer metrics --config ~/config.toml
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="netlnk",kind_id="0"} 627750
tile_pid{kind="net",kind_id="0"} 627759

set-identity ​

Changes the identity key of a running validator. The <keypair> argument is required and must be the path to an Agave style identity.json keypair file. If the path is specified as - the key will instead be read from stdin.

NOTE

Between two Firedancer validators, Firedancer supports a native failover mechanism that is faster and safer than set-identity, and using set-identity between them is heavily discouraged. Use set-identity primarily for failing over between Agave and Firedancer.

It is not generally safe to call set-identity, as another validator might be running with the same identity, and if they both produce a block or vote concurrently, the validator may violate consensus and be subject to (future) slashing.

The validator will not change identity in the middle of a leader slot, and will wait until any in-progress leader slot completes before switching to the new identity. It is safe to call during or near a leader slot because of this wait.

The command exits successfully (with an exit code of 0) if the identity key was changed, otherwise it will fail and print diagnostic messages to stderr. Reasons for failure include being unable to load or verify the provided identity key.

If more than one validator is running, pass --name <name> to select one (see ps to list instances). If --config is given, the validator is instead located from the configuration file: only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator. Compatibility with the running validator is checked either way, and a version mismatch fails cleanly without changing anything. Pass the Agave-compatible vote history file of the new identity, signed by it. Without a file, the new identity is treated as never having voted. If the file is empty or invalid, the command will fail and the identity of the running validator will not be changed. Switching to the identity the validator is already running as is refused.

The validator writes vote_history-<identity>.bin in the [paths.vote_history] directory before each vote and each block it produces. This is an Agave-compatible vote history file and it protects against equivocation, both double voting and double producing blocks. A cluster restart, booting at the [consensus.wait_for_supermajority_at_slot] slot, discards the vote history, as in Agave.

Manual failover ​

Only one validator may run a staked identity at a time. To move an identity between two running validators:

  1. On the validator giving up the identity, run set-identity with an unstaked keypair. If failing over from Firedancer, it saves the identity's vote history in [paths.vote_history] as vote_history-<identity>.bin.bak. If the validator stopped without switching away, the identity's vote history is vote_history-<identity>.bin instead, and its .bak is not the identity's latest.
  2. Copy that file to the validator assuming the identity. For Agave, place it in the ledger directory as vote_history-<identity>.bin. For Firedancer, place it anywhere other than [paths.vote_history].
  3. Then, on the assuming validator, run set-identity with the identity keypair. If failing over to Firedancer, pass the vote history file with --vote-history-file <path>.

A Firedancer validator that restarts after switching away from the identity it booted as refuses to boot until its vote history file is back in [paths.vote_history]. Copy in the latest vote history file from the validator that last ran the identity.

A sample runbook, moving identity A from Firedancer validator primary to Firedancer validator standby, where each validator has its own unstaked keypair junk.json and both use the default [paths.vote_history]:

sh
# on primary
firedancer set-identity junk.json
scp ~/.firedancer/fd1/vote_history-A.bin.bak standby:/tmp/vote_history-A.bin

# on standby
firedancer set-identity A.json --vote-history-file /tmp/vote_history-A.bin

To move it back, run the same steps with the validators swapped.

ArgumentsDescription
<keypair>Path to a identity.json keypair file, or - to read the JSON formatted key from stdin
--name <name>Name of the validator instance to attach to, if more than one is running on this host
--config <path>Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator
--vote-history-file <path>Path to the vote history file saved by the validator previously running with the new identity. Without it, the new identity is treated as never having voted
$ sudo firedancer set-identity junk.json
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  validator identity key switched to 6jywfRwuaMztF6fqa2BVFjQEZQjyMHzdEViAkaqGZW4

get-identity ​

Prints the base58 encoded identity public key the running validator is currently using for gossip, voting, and block production. This may differ from [paths.identity_key] in the configuration file if the identity was changed at runtime with set-identity.

Like set-identity, the command discovers the running validator automatically when no --config is given.

The command exits successfully (with an exit code of 0) and prints the key to stdout if the identity was retrieved, otherwise it fails and prints diagnostic messages to stderr.

ArgumentsDescription
--name <name>Name of the validator instance to attach to, if more than one is running on this host
--config <path>Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator
$ sudo firedancer get-identity
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf

add-authorized-voter ​

Adds an authorized voter to the running validator. The <keypair> argument is required and must be the path to an Agave style voter.json keypair file. If the path is specified as - the key will instead be read from stdin.

With no arguments the command discovers the running validator on the host automatically. If more than one validator is running, pass --name <name> to select one (see ps to list instances). If --config is given, the validator is instead located from the configuration file: only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator. Compatibility with the running validator is checked either way, and a version mismatch fails cleanly without changing anything.

It is not generally safe to call add-authorized-voter, as another validator might be running with the same authorized voter and vote account. If they both vote concurrently, the validator may violate consensus and be subject to (future) slashing.

It is safe to call the command while the validator is running and voting as the client guarantees that votes will not be produced with the new authorized voter key until the key has been gracefully added to the running validator.

The command exits successfully (with an exit code of 0) if the authorized voter was added, otherwise it will fail and print diagnostic messages to stderr. Reasons for failure include the validator being unable to load or verify the provided authorized voter key, if the provided key is a duplicate that the validator is already using, or if there are too many authorized voters for the running validator (more than 16). With failover enabled, the key also cannot be the identity failover hands off.

ArgumentsDescription
<keypair>Path to a voter.json keypair file, or - to read the JSON formatted key from stdin
--name <name>Name of the validator instance to attach to, if more than one is running on this host
--config <path>Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator
$ sudo firedancer add-authorized-voter voter.json
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  authorized voter key AjEjeVFQZr1Xu3nnnGgZYb7zuDioqBz6GTctGeWiAkS4 added successfully

failover ​

Failover takes over a remote identity (and its voting) from an already-running validator and hands it off to this locally-running validator. Run this command on the machine that should take over, with the keypair of the identity to take. The remote validator will switch to an unstaked junk key.

See the failover guide for more details.

sh
firedancer failover promote  [<keypair>] [--name <name>] [--yes] [--address <address>] [--port <port>]
firedancer failover takeover [<keypair>] [--name <name>] [--yes] [--force] [--address <address>] [--port <port>]

failover promote is for planned failover. The machine running the identity cooperatively hands it over. failover takeover is for when the other machine is dead or unreachable, see taking the identity without the remote.

The <keypair> argument is the path to an Agave style identity.json keypair file of the identity to take. If the path is specified as - the key will instead be read from stdin. Without <keypair>, the command takes the identity in [paths.identity_key], or the last keypair passed into failover or set-identity since the validator started.

The command discovers the running validator on the host automatically. If more than one is running, pass --name <name> to select one (see ps). In a terminal it asks for confirmation unless --yes is given.

The local validator finds the address of the machine running the identity through gossip. Use --address to give its IPv4 address or host name instead, and --port if its failover listener does not use the default port 8010.

The command waits for the outcome and exits with a code for the state it leaves the validators in. A failure prints the result name and what it means. The failover guide lists every result with its exit code.

Exit codeMeaning
0Success, the local validator now runs the identity
1Any other failure, the outcome is likely unknown, check both validators with get-identity
2Rejected, no validator changed identity
3The outcome is unknown, check both validators with get-identity before anything else
4Nobody runs the identity, see taking the identity without the remote
128+sigInterrupted by a signal. An already-started handoff keeps running
$ sudo firedancer failover promote identity.json --yes
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  requesting handoff of the identity from the peer running it in gossip
NOTICE  handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
NOTICE  handoff 4142940312112305589: asked peer boot bdb26c31cb1b6921 for the identity (send 1)
NOTICE  handoff 4142940312112305589: adopting the vote history the peer gave us
NOTICE  handoff 4142940312112305589: asking the votor to adopt the vote history the peer gave us
NOTICE  handoff 4142940312112305589: the votor adopted the vote history the peer gave us, last vote at slot 389274617, no vote at or below slot 389274493
NOTICE  handoff 4142940312112305589: asked the admin tile to install the identity
NOTICE  promotion success (identity now 2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf at slot 389274618)
$ sudo firedancer failover promote identity.json --yes
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  requesting handoff of the identity from the peer running it in gossip
NOTICE  handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
NOTICE  handoff 4142940312112305589: asked peer boot bdb26c31cb1b6921 for the identity (send 1)
ERR     `failover promote` failed (replay_behind), the local validator's replay is too far behind the finality anchor of the vote history to adopt, no validator changed identity

Taking the identity without the remote ​

With failover takeover, the validator asks nicely for a handoff first. If that handoff fails, the failure reason is logged and then the local validator takes over the identity anyway, without coordinating with the other machine.

As a safeguard, takeover only proceeds if the other machine shows no signs of life (neither in gossip nor on the failover channel). With --force, takeover will still proceed even if the other machine appears to still be alive.

WARNING

A takeover skips the checks on the other machine and can leave votes and leader windows unaccounted for by starting from an empty history. Use failover takeover when the other machine is down, and --force only after ensuring every other machine with the identity cannot sign. Never run this on both machines at the same time. Please note that --force can result in two validators double voting.

$ sudo firedancer failover takeover identity.json --yes
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  requesting handoff of the identity from the peer running it in gossip
NOTICE  handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
WARNING the handoff failed (no_peer), no validator running the identity answered at the address before the deadline
NOTICE  taking the identity if the other machine shows no signs of life
NOTICE  operator promotion: adopting an empty history, the other machine showed no signs of life
NOTICE  operator promotion: asking the votor to adopt an empty history
NOTICE  operator promotion: the votor adopted an empty history, no last vote, no vote at or below slot 389274493
NOTICE  operator promotion: asked the admin tile to install the identity
NOTICE  promotion success (identity now 2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf, no earlier vote)
$ sudo firedancer failover takeover identity.json --yes --force
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  requesting handoff of the identity from the peer running it in gossip
NOTICE  handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
WARNING the handoff failed (timeout), the remote validator did not answer before the deadline
NOTICE  taking the identity without the other machine, --force skips all checks
NOTICE  operator promotion: adopting an empty history, --force skips the checks on the other machine
NOTICE  operator promotion: asking the votor to adopt an empty history
NOTICE  operator promotion: the votor adopted an empty history, no last vote, no vote at or below slot 389274493
NOTICE  operator promotion: asked the admin tile to install the identity
NOTICE  promotion success (identity now 2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf, no earlier vote)
ArgumentsDescription
promoteAsk the machine running the identity to cooperatively hand it over
takeoverLike promote, but if the handoff fails and the machine running the identity shows no signs of life, take the identity without coordinating with it
<keypair>Path to the identity.json keypair file of the identity to take, or - to read the JSON formatted key from stdin. Defaults to [paths.identity_key], or the last keypair given to failover or set-identity
--name <name>Name of the validator instance to attach to, if more than one is running on this host
--yesSkip the confirmation prompt
--forcetakeover only. Forcibly take the identity, even if the machine already running it may still be alive
--address <address>IPv4 address or host name of the machine running the identity, instead of the address found through gossip.
--port <port>Failover TCP port of the machine running the identity, 8010 as default

remove-all-authorized-voters ​

Removes all authorized voters from the running validator, including any seeded from [paths.authorized_voter_paths] at startup as well as any added at runtime with add-authorized-voter. After removal the validator can only sign votes for vote accounts whose authorized voter is the identity key.

WARNING

Unlike Agave, this command will still leave the validator in a possibly voting state and will continue producing signed vote transactions with the identity of the running validator.

The command is idempotent: removing when there are no authorized voters also succeeds. It exits successfully (with an exit code of 0) and prints All authorized voters removed.

The change is live only: it is not written back to the configuration file, so any voters listed in [paths.authorized_voter_paths] return on the validator's next restart. To drop them across restarts, also remove them from the configuration file.

ArgumentsDescription
--name <name>Name of the validator instance to attach to, if more than one is running on this host
--config <path>Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator
$ sudo firedancer remove-all-authorized-voters
NOTICE  attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE  All authorized voters removed

keys ​

keys pubkey <PATH> ​

Prints the base58 encoding of the public key in the file at <PATH> to the standard output and exits. The file at <PATH> should be an Agave style identity.json key file. The command writes diagnostic messages from logs to stderr.

sh
$ firedancer keys pubkey ~/.firedancer/fd1/identity.json
Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYF

keys new <PATH> ​

Creates a new keypair from the kernel random number generator and writes it to the file specified at <PATH>. The default user for the operation is the user running the command and should have write access to <PATH>. The user can be changed by specifying it in the TOML configuration file.

ArgumentsDescription
--configPath to a configuration TOML file which determines the user creating the file.
toml
user = "firedancer"
$ firedancer keys new ~/identity.json --config config.toml
NOTICE  successfully created keypair in /home/firedancer/identity.json

mem ​

Prints information about the memory requirements and the tile configuration and layout of the validator to stdout before exiting. The command writes diagnostic messages from logs to stderr.

Firedancer preallocates and locks all memory it needs from huge and gigantic page mounts before booting, and the hugetlbfs stage of firedancer configure will reserve the memory described here for exclusive use by Firedancer.

ArgumentsDescription
--configPath to a configuration TOML file to print memory usage information with
--sortList all memory allocations sorted by size in decreasing order, including a percentage of total and exact byte count
sh
$ firedancer mem --config config.toml
── Summary ─────────────────────────────────────────────────────────────────────
  Total Tiles              57
  Total Memory Locked      158 GiB + 915 MiB + 520 KiB  (170611187712 bytes)
  Required Gigantic Pages  154
  Required Huge Pages      2504
  Required Normal Pages    898

── Workspaces (102) ────────────────────────────────────────────────────────────
   ID        SIZE  NAME           PAGES  PAGE SZ   NUMA     FOOTPRINT         LOOSE
    0    36.0 MiB  metric            18  huge         0      35946496       1798144
    1     2.0 MiB  diag               1  huge         0        548864       1544192
[...]

── Objects (609) ───────────────────────────────────────────────────────────────
    ID        SIZE  WORKSPACE      OBJECT         WKSP        OFFSET  PROPERTIES
     0     1.0 MiB  net_gossip     mcache           25          4096  depth=32768
     1    64.0 MiB  net_gossip     dcache           25       1056768  depth=32768 burst=1 mtu=2048
[...]

── Links (120) ─────────────────────────────────────────────────────────────────
   ID        SIZE  NAME           KIND  WKSP     DEPTH        MTU  BURST
    0    64.0 MiB  gossip_net        0    25     32768       2048      1
    1    64.0 MiB  shred_net         0    26     32768       2048      1
[...]

── Tiles (57) ──────────────────────────────────────────────────────────────────
   ID       MLOCK  NAME           KIND  WKSP   CPU  NUMA  LINKS / OBJECTS
    0    34.0 MiB  netlnk            0    89   any     0  in=[-80, -81]  out=[79]  objs=[149:rw 150:rw 151:rw 152:rw 153:rw 154:rw 158:rw 157:ro 164:rw 163:ro 531:rw]
    1     1.3 GiB  net               0    88     1     0  in=[79, -1,  0, -2, -3, -4, -5]  out=[80, 82, 84, 86, 88, 90, 92]  objs=[155:rw 156:rw 157:rw 159:rw 151:ro 152:ro 153:ro 154:ro 160:rw 167:rw 160:rw 169:rw 160:rw 171:rw 160:rw 173:rw 160:rw 175:rw 160:rw 177:rw 160:rw 309:rw 2:ro 3:ro 311:rw 0:ro 1:ro 313:rw 4:ro 5:ro 315:rw 6:ro 7:ro 317:rw 8:ro 9:ro 337:rw 10:ro 11:ro]
[...]

ps ​

Lists validator instances on this host. Each row shows the instance name, the process ID of the validator supervisor, whether the validator is currently live or stale, its uptime, and the version and commit of the running build.

A stale entry means a validator was stopped or crashed. Stale entries are harmless and are cleaned up when the validator next starts, or can be removed with --clean.

The command exits successfully (with an exit code of 0) even if no validators are found.

ArgumentsDescription
--cleanRemove entries for validators that are no longer running
$ firedancer ps
NAME             PID      STATE   UPTIME     VERSION      COMMIT       MOUNT
fd1              2019871  live    2m         0.1.1        babe91c83ca  /mnt/.fd-ps1
fd2              2021957  live    29s        0.1.1        babe91c83ca  /mnt/.fd-ps2

wait ​

Waits for a window where it would be safe to stop the running validator before exiting, it does not actually stop the validator. Various conditions can be waited on, according to the supplied arguments:

  • The validator is caught up to the tip of the chain
  • There is sufficient time until our next leader slot
  • No full or incremental snapshot is currently being written
  • A new up to date incremental snapshot has been written
  • The cluster delinquent stake is below a certain threshold

Once the supplied checks pass, the command exits successfully with code 0. If the validator being waited on exits prematurely, or the command is interrupted, it fails with a non-zero code as follows:

Exit codeMeaning
0A safe window was found
2The validator exited while waiting
128+sigInterrupted by a signal, as a shell reports it (130 for SIGINT, 143 for SIGTERM)
1Any other failure

If run from a terminal, the command prints live diagnostic output, you can prevent this with --silent.

When it fails, the command prints diagnostic messages to stderr. Reasons for failure include no running validator being found, more than one running validator with nothing to select between them, the running validator being a different commit than this binary, a --config file that is not the one the validator was started with, or an idle gap that is larger than an epoch and so can never be satisfied.

With no arguments the command discovers the running validator on the host automatically. If more than one validator is running, pass --name <name> to select one (see ps to list instances). If --config is given, that resolved configuration is used to locate and attach to the validator, and its topology layout must match the running validator. Compatibility with the running validator is checked either way, and a version mismatch fails cleanly without changing anything.

ArgumentsDescription
--min-idle-slots <slots>Minimum number of idle slots required before the next leader slot. Default: 1500. Set to 0 to disable the gap check
--min-idle-seconds <seconds>Minimum idle time in seconds before the next leader slot. Converted to slots using the live slot duration. Mutually exclusive with --min-idle-slots. Set to 0 to disable the gap check
--max-delinquent-stake <percent>Maximum percentage of delinquent stake allowed, in range [0–100]. Default: 5. Set to 100 to disable the check
--skip-health-checkSkip the health check (replay caught-up status)
--skip-snapshot-checkSkip the snapshot check
--silentDo not print the live status panel
--name <name>Name of the validator instance to attach to, if more than one is running on this host
--config <path>Optional path to the configuration TOML file the validator was started with. Its resolved topology layout must match the running validator
$ sudo firedancer wait
NOTICE  waiting for a safe window (min-idle-slots=1500, max-delinquent-stake=5%, skip-health=no, skip-snapshot=no)
  health       ✓ caught up
  leader gap   ✗ leading slot 378
  snapshot     - skipped (periodic disabled)
  delinquent   ✓ 0% delinquent (max 5%)
  waiting... (24s elapsed)

$ sudo firedancer wait --min-idle-seconds 600 --max-delinquent-stake 5
NOTICE  waiting for a safe window (min-idle-seconds=600s, max-delinquent-stake=5%, skip-health=no, skip-snapshot=no)
  health       ✓ caught up
  leader gap   ✓ idle (no upcoming leader slots)
  snapshot     - skipped (periodic disabled)
  delinquent   ✓ ≤1% delinquent (max 5%)
NOTICE  success at slot 454885022 (health: caught up; leader gap: idle (no upcoming leader slots); snapshot: skipped (periodic disabled); delinquent: ≤1% delinquent (max 5%))