firedancer Command Line Interface
The Firedancer binary firedancer contains many subcommands which can be run from the command line.
Commands that attach to a running validator fall into two groups. set-identity, get-identity, and add-authorized-voter are versioned and work across releases. Diagnostic commands like monitor, watch, and metrics read the validator's memory directly and must be run from the same binary the validator is running.
run
Runs the validator. This command does not exit until the validator does, an error in running the validator will be propagated to the exit code of the process. The command can be run with the capabilities listed below but it is suggested to run it as sudo. The command writes an abbreviated log output to stderr and nothing will be written to stdout.
| Arguments | Description |
|---|---|
--config <path> | Path to a configuration TOML file to run the validator with |
--standby | Boot as a failover standby using an unstaked identity, see the failover guide |
Capabilities
| Capability | Reason |
|---|---|
CAP_NET_RAW | call socket(2) to bind to a raw socket for use by XDP |
CAP_SYS_ADMIN | call bpf(2) with the BPF_OBJ_GET command to initialize XDP |
CAP_SYS_ADMIN | call unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces |
CAP_SETUID | call setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID |
CAP_SETGID | call setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit |
CAP_SYS_RESOURCE | call setpriority(2) to increase thread priorities. Not required if the process already has a nice value of -19 |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_NOFILE` to allow more open files for Agave. Not required if the resource limit is already high enough |
CAP_NET_BIND_SERVICE | call bind(2) to bind to a privileged port for serving metrics. Only required if the bind port is below 1024 |
$ firedancer run --config config.toml
Log at "/tmp/fd-0.1.1_1987461_sun-ossdev-firedancer31_2026_10_20_16_32_23_068864834_GMT+00"
WARNING 10-20 16:32:23.089738 main nohz-full.c(71): tile cpus 1-38 are not in the kernel nohz_full= set and will be interrupted by periodic timer ticks. For lower jitter, boot with nohz_full=1-38.
WARNING 10-20 16:32:24.188608 main fd_topo.c(159): fd_topo_wksp_new(store) took 165 ms
NOTICE 10-20 16:32:24.306879 main fd_bootinfo.c(168): booting validator fd1 version 0.1.1 (babe91c83ca) pid 1987461 with 55 tiles and 87 GiB memory at /mnt/.fd
NOTICE 10-20 16:32:24.362519 metric:0 fd_metric_tile.c(148): prometheus server listening at http://127.0.0.1:7999/metrics
WARNING 10-20 16:32:24.419250 votor:0 fd_votor_tile.c(2508): vote history file `/home/firedancer/.firedancer/fd1/vote_history-4Dm5AX6DruesFVJYK5ajk2Gyymqmscgx1PoXbYfQbGSY.bin` does not exist, creating it. This identity is assumed to have never voted.
NOTICE 10-20 16:32:24.438850 gui:0 fd_gui_tile.c(857): gui server listening at http://127.0.0.1:80
NOTICE 10-20 16:32:24.442772 rpc:0 fd_rpc_tile.c(3155): rpc server listening at http://127.0.0.1:8899
NOTICE 10-20 16:32:25.290178 genesi:0 fd_genesi_tile.c(383): retrieved genesis /home/firedancer/.firedancer/fd1/genesis.bin from peer at http://109.94.99.177:8899/genesis.tar.bz2
NOTICE 10-20 16:32:25.454314 event:0 fd_event_client.c(687): connected to telemetry server https://events-in.firedancer.io:443
NOTICE 10-20 16:32:27.370526 snapct:0 fd_snapct_tile.c(756): downloading full snapshot from untrusted gossip peer 141vSYKGRPNGieSrGJy8EeDVBcbjSr6aWkimNgrNZ6xN
NOTICE 10-20 16:32:38.820601 snapct:0 fd_snapct_tile.c(756): downloading incremental snapshot from untrusted gossip peer 141vSYKGRPNGieSrGJy8EeDVBcbjSr6aWkimNgrNZ6xN
NOTICE 10-20 16:32:38.858957 snapin:0 fd_snapin_tile.c(389): loaded 89.3M accounts (9.2K dups) from snapshot in 14.466 seconds
NOTICE 10-20 16:32:41.805075 replay:0 fd_replay_tile.c(1313): caught up to cluster at slot 454774712 (19.3 seconds since boot)monitor
Monitors a validator that is running locally on this machine. This is a low level performance monitor mostly useful for diagnosing throughput issues. The monitor takes over the controlling terminal and refreshes it many times a second with up to date information. You can exit the monitor by sending Ctrl+C or SIGINT.
| Arguments | Description |
|---|---|
--config <path> | Path to a configuration TOML file to run the monitor with. This must be the same configuration file the validator was started with |
Capabilities
| Capability | Reason |
|---|---|
CAP_SYS_ADMIN | call unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces |
CAP_SETUID | call setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID |
CAP_SETGID | call setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit |
$ firedancer monitor --config config.toml
snapshot for 2026-10-20 16:29:11.311695967 GMT+00 | Use TAB to switch panes[0K
tile | tid | stale | heart | nivcsw | nvcsw | in backp | backp cnt | % hkeep | % wait | % backp | % finish[0K
---------+---------+------------+-------+---------------------+---------------------+----------+---------------------+----------+----------+----------+----------[0K
netlnk | 1983482 | - | - | 1( +0) | 92592( +0) | -( -) | 0( +0) | 0.002 | 99.998 | 0.000 | 0.000[0K
net | 1983506 | - | - | 29( +0) | 8( +0) | -( -) | 0( +0) | 0.684 | 99.316 | 0.000 | 0.000[0K
net | 1983508 | - | - | 27( +0) | 4( +0) | -( -) | 0( +0) | 0.678 | 99.322 | 0.000 | 0.000[0K
metric | 1983483 | - | - | 0( +0) | 92626( +0) | -( -) | 0( +0) | 0.003 | 99.997 | 0.000 | 0.000[0K
diag | 1983484 | - | - | 0( +0) | 4989( +0) | -( -) | 0( +0) | 0.003 | 99.997 | 0.000 | 0.000[0K
ipecho | 1983486 | - | - | 3( +0) | 92511( +0) | -( -) | 1( +0) | 0.003 | 99.997 | 0.000 | 0.000[0K
gossvf | 1983512 | - | - | 27( +0) | 3( +0) | -( -) | 0( +0) | 0.049 | 99.951 | 0.000 | 0.000[0K
gossip | 1983511 | - | - | 37( +0) | 5( +0) | -( -) | 0( +0) | 0.092 | 99.897 | 0.000 | 0.011[0K
shred | 1983514 | - | - | 26( +0) | 5( +0) | -( -) | 0( +0) | 0.086 | 99.914 | 0.000 | 0.000[0K
repair | 1983515 | - | - | 30( +0) | 5( +0) | -( -) | 0( +0) | 0.086 | 99.914 | 0.000 | 0.000[0K
rserve | 1983517 | - | - | 61( +0) | 5( +0) | -( -) | 0( +0) | 0.491 | 99.509 | 0.000 | 0.000[0K
replay | 1983528 | - | - | 27( +0) | 5( +0) | -( -) | 0( +0) | 2.738 | 97.262 | 0.000 | 0.000[0K
accdb | 1983529 | - | - | 40( +0) | 55( +0) | -( -) | 0( +0) | 0.018 | 99.982 | 0.000 | 0.000[0K
execrp | 1983531 | - | - | 27( +0) | 64( +0) | -( -) | 0( +0) | 0.019 | 99.981 | 0.000 | 0.000[0K
execrp | 1983532 | - | - | 27( +0) | 67( +0) | -( -) | 0( +0) | 0.020 | 99.980 | 0.000 | 0.000[0K
execrp | 1983534 | - | - | 26( +0) | 65( +0) | -( -) | 0( +0) | 0.020 | 99.980 | 0.000 | 0.000[0K
execrp | 1983535 | - | - | 26( +0) | 68( +0) | -( -) | 0( +0) | 0.020 | 99.980 | 0.000 | 0.000[0K
execrp | 1983538 | - | - | 29( +0) | 60( +0) | -( -) | 0( +0) | 0.020 | 99.980 | 0.000 | 0.000[0K
execrp | 1983547 | - | - | 25( +0) | 59( +0) | -( -) | 0( +0) | 0.020 | 99.980 | 0.000 | 0.000[0K
execrp | 1983555 | - | - | 25( +0) | 63( +0) | -( -) | 0( +0) | 0.024 | 99.976 | 0.000 | 0.000[0K
execrp | 1983556 | - | - | 25( +0) | 63( +0) | -( -) | 0( +0) | 0.020 | 99.980 | 0.000 | 0.000[0K
execrp | 1983554 | - | - | 26( +0) | 58( +0) | -( -) | 0( +0) | 0.023 | 99.977 | 0.000 | 0.000[0K
execrp | 1983557 | - | - | 24( +0) | 59( +0) | -( -) | 0( +0) | 0.022 | 99.978 | 0.000 | 0.000[0K
votor | 1983566 | - | - | 27( +0) | 3( +0) | -( -) | 0( +0) | 0.110 | 99.890 | 0.000 | 0.000[0K
quic | 1983569 | - | - | 28( +0) | 15( +0) | -( -) | 0( +0) | 0.004 | 99.996 | 0.000 | 0.000[0K
verify | 1983568 | - | - | 27( +0) | 16( +0) | -( -) | 0( +0) | 6.555 | 93.445 | 0.000 | 0.000[0K
verify | 1983570 | - | - | 27( +0) | 16( +0) | -( -) | 0( +0) | 6.543 | 93.457 | 0.000 | 0.000[0K
verify | 1983572 | - | - | 26( +0) | 15( +0) | -( -) | 0( +0) | 6.625 | 93.375 | 0.000 | 0.000[0K
dedup | 1983573 | - | - | 27( +0) | 8( +0) | -( -) | 0( +0) | 0.398 | 99.602 | 0.000 | 0.000[0K
resolv | 1983575 | - | - | 26( +0) | 17( +0) | -( -) | 0( +0) | 0.102 | 99.898 | 0.000 | 0.000[0K
pack | 1983584 | - | - | 28( +0) | 16( +0) | -( -) | 0( +0) | 0.101 | 15.631 | 0.000 | 84.268[0K
execle | 1983586 | - | - | 24( +0) | 48( +0) | -( -) | 0( +0) | 0.057 | 99.943 | 0.000 | 0.000[0K
execle | 1983585 | - | - | 27( +0) | 51( +0) | -( -) | 0( +0) | 0.058 | 99.942 | 0.000 | 0.000[0K
motor | 1983587 | - | - | 24( +0) | 44( +0) | -( -) | 0( +0) | 0.044 | 99.956 | 0.000 | 0.000[0K
sign | 1983593 | - | - | 27( +0) | 3( +0) | -( -) | 0( +0) | 0.045 | 99.955 | 0.000 | 0.000[0K
sign | 1983594 | - | - | 40( +0) | 4( +0) | -( -) | 0( +0) | 0.017 | 99.983 | 0.000 | 0.000[0K
rpc | 1983595 | - | - | 75( +0) | 3( +0) | -( -) | 0( +0) | 0.049 | 99.951 | 0.000 | 0.000[0K
admin | 1983489 | - | - | 0( +0) | 4855( +0) | -( -) | 0( +0) | 0.002 | 99.998 | 0.000 | 0.000[0K
event | 1983598 | - | - | 52( +0) | 23( +0) | -( -) | 0( +0) | 0.006 | 99.982 | 0.000 | 0.012[0K
gui | 1983599 | - | - | 27( +0) | 3( +0) | -( -) | 0( +0) | 0.163 | 99.528 | 0.000 | 0.309[0K
waker | 1983490 | - | - | 1( +0) | 10303( +0) | -( -) | 0( +0) | 0.003 | 99.997 | 0.000 | 0.000[0Kconfigure
Configures the operating system so that it can run Firedancer. See the guide for more information. There are the following stages to each configure command:
iavfConfigures VFs, clears and installs PF ntuple rules and loads thevfio-pcikernel module required by the iavf network provider.mlx5Loads theib_uverbskernel module required by the mlx5 network provider. Clears leftover XDP ntuple rules, resets the RSS indirection table and channel count to defaults.hugetlbfsReserves huge and gigantic pages for use by Firedancer and mounts huge page filesystems for then under a path in the configuration TOML file.sysctlSet required kernel parameters.hyperthreadsChecks hyperthreaded pair for critical CPU cores.bondingPrepares bonded network devices for XDP networking.ethtool-channelsConfigures the number of channels on the network device.ethtool-offloadsModify offload feature flags on the network device.ethtool-loopbackDisables UDP segmentation on the loopback device.irq-affinityRemoves Firedancer tile CPUs from configurable/proc/irq/*/smp_affinitymasks.irq-balanceConfigures the irqbalance daemon to avoid Firedancer tile CPUs. If irqbalance is not running, this stage is a no-op.snapshotsPrepares the snapshot download directory.
| Arguments | Description |
|---|---|
--config <path> | Path to a configuration TOML file to configure the validator with. This must be the same configuration file the validator will be started with |
[hugetlbfs]
mount_path = "/mnt/.fd"
[layout]
net_tile_count = 2
[tiles]
[net]
interface = "ens3f0"configure init <stage>...
Prepare the operating system environment to run Firedancer. This will reserve and mount the huge page filesystems, set the kernel parameters, and configure the number of combined channels on the network device.
Capabilities
| Capability | Reason |
|---|---|
root | increase /proc/sys/vm/nr_hugepages and mount hugetlbfs filesystems. Only applies for the hugetlbfs stage |
root | increase network device channels with ethtool --set-channels. Only applies for the ethtool-channels stage |
root | disable network device offloads with ethtool --offload IFACE FEATURE off. Only applies for the ethtool-offloads stage |
root | disable network device tx-udp-segmentation with ethtool --offload lo tx-udp-segmentation off. Only applies for the ethtool-loopback stage |
root | restore network device defaults and load the ib_uverbs kernel module if needed |
CAP_SYS_MODULE | load the ib_uverbs kernel module with modprobe when the uverbs interface is unavailable |
root | configure VFs and PF ntuple rules. Only applies for the iavf stage |
CAP_SYS_MODULE | load the vfio-pci kernel module with modprobe when the driver is unavailable. Only applies for the iavf stage |
CAP_SYS_ADMIN | set kernel parameters in /proc/sys. Only applies for the sysctl stage |
$ firedancer configure init all
NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE hugetlbfs ... configuring
NOTICE RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=6155141120`
NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=150323855360`
NOTICE ethtool-channels ... unconfigured ... device `bond0` missing ethtool-channels network configuration
NOTICE ethtool-channels ... configuring
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 start 2 equal 126`
NOTICE RUN: `ethtool --config-ntuple enp1s0f0np0 flow-type udp4 dst-port 8003 src-ip 0.0.0.0 m 255.255.255.254 queue 0`
WARNING error configuring network device (enp1s0f0np0), ioctl(SIOCETHTOOL,ETHTOOL_SRXCLSRLINS) failed (95-ENOTSUP-operation not supported)
WARNING error configuring network device (bond0), rss_queue_mode "auto" attempted "dedicated" configuration but falling back to "simple".
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-channels enp1s0f0np0 combined 2`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --config-nfc enp1s0f0np0 rx-flow-hash udp4 sdfn`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE RUN: `ethtool --set-channels enp1s0f1np1 combined 2`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE RUN: `ethtool --config-nfc enp1s0f1np1 rx-flow-hash udp4 sdfn`
NOTICE irq-affinity ... unconfigured ... found IRQs overlapping with Firedancer tile CPUs (IRQs: 1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,+45 more; tiles: )
NOTICE irq-affinity ... configuring
NOTICE kworkers ... unconfigured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
NOTICE kworkers ... configuring
NOTICE RUN: `echo "fffffff0,03fffff9,fffffff0,00000001" > /sys/devices/virtual/workqueue/cpumask`
NOTICE kernel workqueue cpumask changed while writing `/sys/devices/virtual/workqueue/cpumask`: fffffe00,40000001,fffff800,00000001 -> fffffff0,03fffff9,fffffff0,00000001 (kernel 7.0.0-34-generic).
NOTICE cpuset ... unconfigured ... `/sys/fs/cgroup/fd1` does not exist
NOTICE cpuset ... configuring
NOTICE RUN: `echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control`
NOTICE RUN: `mkdir /sys/fs/cgroup/fd1`
NOTICE RUN: `echo "1-35,65-66,90-99" > /sys/fs/cgroup/fd1/cpuset.cpus`
NOTICE RUN: `echo "isolated" > /sys/fs/cgroup/fd1/cpuset.cpus.partition`
NOTICE console ... unconfigured ... console loglevel is 4; kernel messages render console work on the printing CPU
NOTICE console ... configuring
NOTICE RUN: `echo "0" > /sys/class/graphics/fbcon/cursor_blink`
NOTICE RUN: `echo "2" > /proc/sys/kernel/printk`configure check <stage>...
Check if the operating system environment is properly configured. Exits with a non-zero exit code if it is not, after printing relevant diagnostics to stderr.
$ firedancer configure check all
WARNING hugetlbfs ... not configured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
WARNING ethtool-channels ... not configured ... device `bond0` missing ethtool-channels network configuration
WARNING irq-affinity ... not configured ... insufficient permissions to write /proc/irq/0/smp_affinity
WARNING kworkers ... not configured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
WARNING cpuset ... not configured ... `/sys/fs/cgroup/fd1` does not exist
WARNING console ... not configured ... console loglevel is 4; kernel messages render console work on the printing CPU
ERR failed to configure some stagesconfigure fini <stage>...
Remove any Firedancer specific operating system configuration still lingering. The hugetlbfs stage unmounts its filesystems and returns the reserved huge and gigantic pages to the kernel pool. The mlx5 stage clears leftover XDP ntuple rules and resets the RSS indirection table and channel count to defaults. The iavf stage clears PF ntuple rules and removes VF 0 if its PF has exactly one VF. PFs with multiple VFs retain all of them. The fini mode does not reduce sysctls that were earlier increased or restore previous network settings.
Capabilities
| Capability | Reason |
|---|---|
root | remove directories from /mnt, unmount hugetlbfs. Only applies for the hugetlbfs stage |
root | restore network device defaults |
root | clear PF ntuple rules and remove VFs. Only applies for the iavf stage |
$ firedancer configure fini all
NOTICE snapshots ... finishing
NOTICE console ... not configured ... console loglevel is 4; kernel messages render console work on the printing CPU
NOTICE rcu-nocbs ... finishing
NOTICE nohz-full ... finishing
NOTICE cpuset ... not configured ... `/sys/fs/cgroup/fd1` does not exist
NOTICE kworkers ... not configured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
NOTICE irq-affinity ... not configured ... found IRQs overlapping with Firedancer tile CPUs (IRQs: 1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,+45 more; tiles: )
NOTICE irq-balance ... finishing
NOTICE ethtool-loopback ... finishing
NOTICE ethtool-offloads ... finishing
NOTICE ethtool-channels ... finishing
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE RUN: `ethtool --set-channels enp1s0f1np1 combined 128`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE sysctl ... finishing
NOTICE hugetlbfs ... finishing
NOTICE RUN: `umount /mnt/.fd/.huge`
NOTICE RUN: `rmdir /mnt/.fd/.huge`
NOTICE RUN: `umount /mnt/.fd/.gigantic`
NOTICE RUN: `rmdir /mnt/.fd/.gigantic`
NOTICE RUN: `rmdir /mnt/.fd/.normal`
NOTICE RUN: `rmdir /mnt/.fd`version
Prints the current version of the validator to the standard output and exits. The command writes diagnostic messages from logs to stderr.
$ firedancer version
26.10.3shred-version
Prints the current shred version of the cluster being joined, according to the entrypoints, to standard output and exits. The command writes diagnostic messages from logs to stderr.
$ firedancer shred-version
50093metrics
Prints the current validator metrics to stdout. Metrics can typically be accessed via HTTP when the metric tile is enabled, but the command can be used even if the metrics server is not enabled, or the validator has crashed.
$ firedancer metrics --config ~/config.toml
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="netlnk",kind_id="0"} 627750
tile_pid{kind="net",kind_id="0"} 627759set-identity
Changes the identity key of a running validator. The <keypair> argument is required and must be the path to an Agave style identity.json keypair file. If the path is specified as - the key will instead be read from stdin.
NOTE
Between two Firedancer validators, Firedancer supports a native failover mechanism that is faster and safer than set-identity, and using set-identity between them is heavily discouraged. Use set-identity primarily for failing over between Agave and Firedancer.
It is not generally safe to call set-identity, as another validator might be running with the same identity, and if they both produce a block or vote concurrently, the validator may violate consensus and be subject to (future) slashing.
The validator will not change identity in the middle of a leader slot, and will wait until any in-progress leader slot completes before switching to the new identity. It is safe to call during or near a leader slot because of this wait.
The command exits successfully (with an exit code of 0) if the identity key was changed, otherwise it will fail and print diagnostic messages to stderr. Reasons for failure include being unable to load or verify the provided identity key.
If more than one validator is running, pass --name <name> to select one (see ps to list instances). If --config is given, the validator is instead located from the configuration file: only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator. Compatibility with the running validator is checked either way, and a version mismatch fails cleanly without changing anything. Pass the Agave-compatible vote history file of the new identity, signed by it. Without a file, the new identity is treated as never having voted. If the file is empty or invalid, the command will fail and the identity of the running validator will not be changed. Switching to the identity the validator is already running as is refused.
The validator writes vote_history-<identity>.bin in the [paths.vote_history] directory before each vote and each block it produces. This is an Agave-compatible vote history file and it protects against equivocation, both double voting and double producing blocks. A cluster restart, booting at the [consensus.wait_for_supermajority_at_slot] slot, discards the vote history, as in Agave.
Manual failover
Only one validator may run a staked identity at a time. To move an identity between two running validators:
- On the validator giving up the identity, run
set-identitywith an unstaked keypair. If failing over from Firedancer, it saves the identity's vote history in[paths.vote_history]asvote_history-<identity>.bin.bak. If the validator stopped without switching away, the identity's vote history isvote_history-<identity>.bininstead, and its.bakis not the identity's latest. - Copy that file to the validator assuming the identity. For Agave, place it in the ledger directory as
vote_history-<identity>.bin. For Firedancer, place it anywhere other than[paths.vote_history]. - Then, on the assuming validator, run
set-identitywith the identity keypair. If failing over to Firedancer, pass the vote history file with--vote-history-file <path>.
A Firedancer validator that restarts after switching away from the identity it booted as refuses to boot until its vote history file is back in [paths.vote_history]. Copy in the latest vote history file from the validator that last ran the identity.
A sample runbook, moving identity A from Firedancer validator primary to Firedancer validator standby, where each validator has its own unstaked keypair junk.json and both use the default [paths.vote_history]:
# on primary
firedancer set-identity junk.json
scp ~/.firedancer/fd1/vote_history-A.bin.bak standby:/tmp/vote_history-A.bin
# on standby
firedancer set-identity A.json --vote-history-file /tmp/vote_history-A.binTo move it back, run the same steps with the validators swapped.
| Arguments | Description |
|---|---|
<keypair> | Path to a identity.json keypair file, or - to read the JSON formatted key from stdin |
--name <name> | Name of the validator instance to attach to, if more than one is running on this host |
--config <path> | Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator |
--vote-history-file <path> | Path to the vote history file saved by the validator previously running with the new identity. Without it, the new identity is treated as never having voted |
$ sudo firedancer set-identity junk.json
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE validator identity key switched to 6jywfRwuaMztF6fqa2BVFjQEZQjyMHzdEViAkaqGZW4get-identity
Prints the base58 encoded identity public key the running validator is currently using for gossip, voting, and block production. This may differ from [paths.identity_key] in the configuration file if the identity was changed at runtime with set-identity.
Like set-identity, the command discovers the running validator automatically when no --config is given.
The command exits successfully (with an exit code of 0) and prints the key to stdout if the identity was retrieved, otherwise it fails and prints diagnostic messages to stderr.
| Arguments | Description |
|---|---|
--name <name> | Name of the validator instance to attach to, if more than one is running on this host |
--config <path> | Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator |
$ sudo firedancer get-identity
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTfadd-authorized-voter
Adds an authorized voter to the running validator. The <keypair> argument is required and must be the path to an Agave style voter.json keypair file. If the path is specified as - the key will instead be read from stdin.
With no arguments the command discovers the running validator on the host automatically. If more than one validator is running, pass --name <name> to select one (see ps to list instances). If --config is given, the validator is instead located from the configuration file: only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator. Compatibility with the running validator is checked either way, and a version mismatch fails cleanly without changing anything.
It is not generally safe to call add-authorized-voter, as another validator might be running with the same authorized voter and vote account. If they both vote concurrently, the validator may violate consensus and be subject to (future) slashing.
It is safe to call the command while the validator is running and voting as the client guarantees that votes will not be produced with the new authorized voter key until the key has been gracefully added to the running validator.
The command exits successfully (with an exit code of 0) if the authorized voter was added, otherwise it will fail and print diagnostic messages to stderr. Reasons for failure include the validator being unable to load or verify the provided authorized voter key, if the provided key is a duplicate that the validator is already using, or if there are too many authorized voters for the running validator (more than 16). With failover enabled, the key also cannot be the identity failover hands off.
| Arguments | Description |
|---|---|
<keypair> | Path to a voter.json keypair file, or - to read the JSON formatted key from stdin |
--name <name> | Name of the validator instance to attach to, if more than one is running on this host |
--config <path> | Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator |
$ sudo firedancer add-authorized-voter voter.json
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE authorized voter key AjEjeVFQZr1Xu3nnnGgZYb7zuDioqBz6GTctGeWiAkS4 added successfullyfailover
Failover takes over a remote identity (and its voting) from an already-running validator and hands it off to this locally-running validator. Run this command on the machine that should take over, with the keypair of the identity to take. The remote validator will switch to an unstaked junk key.
See the failover guide for more details.
firedancer failover promote [<keypair>] [--name <name>] [--yes] [--address <address>] [--port <port>]
firedancer failover takeover [<keypair>] [--name <name>] [--yes] [--force] [--address <address>] [--port <port>]failover promote is for planned failover. The machine running the identity cooperatively hands it over. failover takeover is for when the other machine is dead or unreachable, see taking the identity without the remote.
The <keypair> argument is the path to an Agave style identity.json keypair file of the identity to take. If the path is specified as - the key will instead be read from stdin. Without <keypair>, the command takes the identity in [paths.identity_key], or the last keypair passed into failover or set-identity since the validator started.
The command discovers the running validator on the host automatically. If more than one is running, pass --name <name> to select one (see ps). In a terminal it asks for confirmation unless --yes is given.
The local validator finds the address of the machine running the identity through gossip. Use --address to give its IPv4 address or host name instead, and --port if its failover listener does not use the default port 8010.
The command waits for the outcome and exits with a code for the state it leaves the validators in. A failure prints the result name and what it means. The failover guide lists every result with its exit code.
| Exit code | Meaning |
|---|---|
0 | Success, the local validator now runs the identity |
1 | Any other failure, the outcome is likely unknown, check both validators with get-identity |
2 | Rejected, no validator changed identity |
3 | The outcome is unknown, check both validators with get-identity before anything else |
4 | Nobody runs the identity, see taking the identity without the remote |
128+sig | Interrupted by a signal. An already-started handoff keeps running |
$ sudo firedancer failover promote identity.json --yes
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE requesting handoff of the identity from the peer running it in gossip
NOTICE handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
NOTICE handoff 4142940312112305589: asked peer boot bdb26c31cb1b6921 for the identity (send 1)
NOTICE handoff 4142940312112305589: adopting the vote history the peer gave us
NOTICE handoff 4142940312112305589: asking the votor to adopt the vote history the peer gave us
NOTICE handoff 4142940312112305589: the votor adopted the vote history the peer gave us, last vote at slot 389274617, no vote at or below slot 389274493
NOTICE handoff 4142940312112305589: asked the admin tile to install the identity
NOTICE promotion success (identity now 2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf at slot 389274618)$ sudo firedancer failover promote identity.json --yes
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE requesting handoff of the identity from the peer running it in gossip
NOTICE handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
NOTICE handoff 4142940312112305589: asked peer boot bdb26c31cb1b6921 for the identity (send 1)
ERR `failover promote` failed (replay_behind), the local validator's replay is too far behind the finality anchor of the vote history to adopt, no validator changed identityTaking the identity without the remote
With failover takeover, the validator asks nicely for a handoff first. If that handoff fails, the failure reason is logged and then the local validator takes over the identity anyway, without coordinating with the other machine.
As a safeguard, takeover only proceeds if the other machine shows no signs of life (neither in gossip nor on the failover channel). With --force, takeover will still proceed even if the other machine appears to still be alive.
WARNING
A takeover skips the checks on the other machine and can leave votes and leader windows unaccounted for by starting from an empty history. Use failover takeover when the other machine is down, and --force only after ensuring every other machine with the identity cannot sign. Never run this on both machines at the same time. Please note that --force can result in two validators double voting.
$ sudo firedancer failover takeover identity.json --yes
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE requesting handoff of the identity from the peer running it in gossip
NOTICE handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
WARNING the handoff failed (no_peer), no validator running the identity answered at the address before the deadline
NOTICE taking the identity if the other machine shows no signs of life
NOTICE operator promotion: adopting an empty history, the other machine showed no signs of life
NOTICE operator promotion: asking the votor to adopt an empty history
NOTICE operator promotion: the votor adopted an empty history, no last vote, no vote at or below slot 389274493
NOTICE operator promotion: asked the admin tile to install the identity
NOTICE promotion success (identity now 2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf, no earlier vote)$ sudo firedancer failover takeover identity.json --yes --force
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE requesting handoff of the identity from the peer running it in gossip
NOTICE handoff 4142940312112305589: requesting handoff from the machine running the identity at `192.0.2.10:8010` using gossip
WARNING the handoff failed (timeout), the remote validator did not answer before the deadline
NOTICE taking the identity without the other machine, --force skips all checks
NOTICE operator promotion: adopting an empty history, --force skips the checks on the other machine
NOTICE operator promotion: asking the votor to adopt an empty history
NOTICE operator promotion: the votor adopted an empty history, no last vote, no vote at or below slot 389274493
NOTICE operator promotion: asked the admin tile to install the identity
NOTICE promotion success (identity now 2vJXGJVGTxabY97CtBfby7KcHz3Pb2ZEX8aH6vLbqvTf, no earlier vote)| Arguments | Description |
|---|---|
promote | Ask the machine running the identity to cooperatively hand it over |
takeover | Like promote, but if the handoff fails and the machine running the identity shows no signs of life, take the identity without coordinating with it |
<keypair> | Path to the identity.json keypair file of the identity to take, or - to read the JSON formatted key from stdin. Defaults to [paths.identity_key], or the last keypair given to failover or set-identity |
--name <name> | Name of the validator instance to attach to, if more than one is running on this host |
--yes | Skip the confirmation prompt |
--force | takeover only. Forcibly take the identity, even if the machine already running it may still be alive |
--address <address> | IPv4 address or host name of the machine running the identity, instead of the address found through gossip. |
--port <port> | Failover TCP port of the machine running the identity, 8010 as default |
remove-all-authorized-voters
Removes all authorized voters from the running validator, including any seeded from [paths.authorized_voter_paths] at startup as well as any added at runtime with add-authorized-voter. After removal the validator can only sign votes for vote accounts whose authorized voter is the identity key.
WARNING
Unlike Agave, this command will still leave the validator in a possibly voting state and will continue producing signed vote transactions with the identity of the running validator.
The command is idempotent: removing when there are no authorized voters also succeeds. It exits successfully (with an exit code of 0) and prints All authorized voters removed.
The change is live only: it is not written back to the configuration file, so any voters listed in [paths.authorized_voter_paths] return on the validator's next restart. To drop them across restarts, also remove them from the configuration file.
| Arguments | Description |
|---|---|
--name <name> | Name of the validator instance to attach to, if more than one is running on this host |
--config <path> | Optional path to a configuration TOML file naming the validator to attach to. Only the name and [hugetlbfs.mount_path] values are used, and they must match the running validator |
$ sudo firedancer remove-all-authorized-voters
NOTICE attached to live validator fd1 version 0.1.1 (babe91c83ca) pid 1983468 up 3m at /mnt/.fd
NOTICE All authorized voters removedkeys
keys pubkey <PATH>
Prints the base58 encoding of the public key in the file at <PATH> to the standard output and exits. The file at <PATH> should be an Agave style identity.json key file. The command writes diagnostic messages from logs to stderr.
$ firedancer keys pubkey ~/.firedancer/fd1/identity.json
Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYFkeys new <PATH>
Creates a new keypair from the kernel random number generator and writes it to the file specified at <PATH>. The default user for the operation is the user running the command and should have write access to <PATH>. The user can be changed by specifying it in the TOML configuration file.
| Arguments | Description |
|---|---|
--config | Path to a configuration TOML file which determines the user creating the file. |
user = "firedancer"$ firedancer keys new ~/identity.json --config config.toml
NOTICE successfully created keypair in /home/firedancer/identity.jsonmem
Prints information about the memory requirements and the tile configuration and layout of the validator to stdout before exiting. The command writes diagnostic messages from logs to stderr.
Firedancer preallocates and locks all memory it needs from huge and gigantic page mounts before booting, and the hugetlbfs stage of firedancer configure will reserve the memory described here for exclusive use by Firedancer.
| Arguments | Description |
|---|---|
--config | Path to a configuration TOML file to print memory usage information with |
--sort | List all memory allocations sorted by size in decreasing order, including a percentage of total and exact byte count |
$ firedancer mem --config config.toml
── Summary ─────────────────────────────────────────────────────────────────────
Total Tiles 57
Total Memory Locked 158 GiB + 915 MiB + 520 KiB (170611187712 bytes)
Required Gigantic Pages 154
Required Huge Pages 2504
Required Normal Pages 898
── Workspaces (102) ────────────────────────────────────────────────────────────
ID SIZE NAME PAGES PAGE SZ NUMA FOOTPRINT LOOSE
0 36.0 MiB metric 18 huge 0 35946496 1798144
1 2.0 MiB diag 1 huge 0 548864 1544192
[...]
── Objects (609) ───────────────────────────────────────────────────────────────
ID SIZE WORKSPACE OBJECT WKSP OFFSET PROPERTIES
0 1.0 MiB net_gossip mcache 25 4096 depth=32768
1 64.0 MiB net_gossip dcache 25 1056768 depth=32768 burst=1 mtu=2048
[...]
── Links (120) ─────────────────────────────────────────────────────────────────
ID SIZE NAME KIND WKSP DEPTH MTU BURST
0 64.0 MiB gossip_net 0 25 32768 2048 1
1 64.0 MiB shred_net 0 26 32768 2048 1
[...]
── Tiles (57) ──────────────────────────────────────────────────────────────────
ID MLOCK NAME KIND WKSP CPU NUMA LINKS / OBJECTS
0 34.0 MiB netlnk 0 89 any 0 in=[-80, -81] out=[79] objs=[149:rw 150:rw 151:rw 152:rw 153:rw 154:rw 158:rw 157:ro 164:rw 163:ro 531:rw]
1 1.3 GiB net 0 88 1 0 in=[79, -1, 0, -2, -3, -4, -5] out=[80, 82, 84, 86, 88, 90, 92] objs=[155:rw 156:rw 157:rw 159:rw 151:ro 152:ro 153:ro 154:ro 160:rw 167:rw 160:rw 169:rw 160:rw 171:rw 160:rw 173:rw 160:rw 175:rw 160:rw 177:rw 160:rw 309:rw 2:ro 3:ro 311:rw 0:ro 1:ro 313:rw 4:ro 5:ro 315:rw 6:ro 7:ro 317:rw 8:ro 9:ro 337:rw 10:ro 11:ro]
[...]ps
Lists validator instances on this host. Each row shows the instance name, the process ID of the validator supervisor, whether the validator is currently live or stale, its uptime, and the version and commit of the running build.
A stale entry means a validator was stopped or crashed. Stale entries are harmless and are cleaned up when the validator next starts, or can be removed with --clean.
The command exits successfully (with an exit code of 0) even if no validators are found.
| Arguments | Description |
|---|---|
--clean | Remove entries for validators that are no longer running |
$ firedancer ps
NAME PID STATE UPTIME VERSION COMMIT MOUNT
fd1 2019871 live 2m 0.1.1 babe91c83ca /mnt/.fd-ps1
fd2 2021957 live 29s 0.1.1 babe91c83ca /mnt/.fd-ps2wait
Waits for a window where it would be safe to stop the running validator before exiting, it does not actually stop the validator. Various conditions can be waited on, according to the supplied arguments:
- The validator is caught up to the tip of the chain
- There is sufficient time until our next leader slot
- No full or incremental snapshot is currently being written
- A new up to date incremental snapshot has been written
- The cluster delinquent stake is below a certain threshold
Once the supplied checks pass, the command exits successfully with code 0. If the validator being waited on exits prematurely, or the command is interrupted, it fails with a non-zero code as follows:
| Exit code | Meaning |
|---|---|
0 | A safe window was found |
2 | The validator exited while waiting |
128+sig | Interrupted by a signal, as a shell reports it (130 for SIGINT, 143 for SIGTERM) |
1 | Any other failure |
If run from a terminal, the command prints live diagnostic output, you can prevent this with --silent.
When it fails, the command prints diagnostic messages to stderr. Reasons for failure include no running validator being found, more than one running validator with nothing to select between them, the running validator being a different commit than this binary, a --config file that is not the one the validator was started with, or an idle gap that is larger than an epoch and so can never be satisfied.
With no arguments the command discovers the running validator on the host automatically. If more than one validator is running, pass --name <name> to select one (see ps to list instances). If --config is given, that resolved configuration is used to locate and attach to the validator, and its topology layout must match the running validator. Compatibility with the running validator is checked either way, and a version mismatch fails cleanly without changing anything.
| Arguments | Description |
|---|---|
--min-idle-slots <slots> | Minimum number of idle slots required before the next leader slot. Default: 1500. Set to 0 to disable the gap check |
--min-idle-seconds <seconds> | Minimum idle time in seconds before the next leader slot. Converted to slots using the live slot duration. Mutually exclusive with --min-idle-slots. Set to 0 to disable the gap check |
--max-delinquent-stake <percent> | Maximum percentage of delinquent stake allowed, in range [0–100]. Default: 5. Set to 100 to disable the check |
--skip-health-check | Skip the health check (replay caught-up status) |
--skip-snapshot-check | Skip the snapshot check |
--silent | Do not print the live status panel |
--name <name> | Name of the validator instance to attach to, if more than one is running on this host |
--config <path> | Optional path to the configuration TOML file the validator was started with. Its resolved topology layout must match the running validator |
$ sudo firedancer wait
NOTICE waiting for a safe window (min-idle-slots=1500, max-delinquent-stake=5%, skip-health=no, skip-snapshot=no)
health ✓ caught up[K
leader gap ✗ leading slot 378[K
snapshot - skipped (periodic disabled)[K
delinquent ✓ 0% delinquent (max 5%)[K
waiting... (24s elapsed)[K
$ sudo firedancer wait --min-idle-seconds 600 --max-delinquent-stake 5
NOTICE waiting for a safe window (min-idle-seconds=600s, max-delinquent-stake=5%, skip-health=no, skip-snapshot=no)
health ✓ caught up[K
leader gap ✓ idle (no upcoming leader slots)[K
snapshot - skipped (periodic disabled)[K
delinquent ✓ ≤1% delinquent (max 5%)[K
NOTICE success at slot 454885022 (health: caught up; leader gap: idle (no upcoming leader slots); snapshot: skipped (periodic disabled); delinquent: ≤1% delinquent (max 5%))