fdctl Command Line Interface
The Firedancer binary fdctl contains many subcommands which can be run from the command line.
run
Runs the validator. This command does not exit until the validator does, an error in running the validator will be propagated to the exit code of the process. The command can be run with the capabilities listed below but it is suggested to run it as sudo. The command writes an abbreviated log output to stderr and nothing will be written to stdout.
| Arguments | Description |
|---|---|
--config <path> | Path to a configuration TOML file to run the validator with |
Capabilities
| Capability | Reason |
|---|---|
CAP_NET_RAW | call socket(2) to bind to a raw socket for use by XDP |
CAP_SYS_ADMIN | call bpf(2) with the BPF_OBJ_GET command to initialize XDP |
CAP_SYS_ADMIN | call unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces |
CAP_SETUID | call setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID |
CAP_SETGID | call setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit |
CAP_SYS_RESOURCE | call setpriority(2) to increase thread priorities. Not required if the process already has a nice value of -19 |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_NOFILE` to allow more open files for Agave. Not required if the resource limit is already high enough |
CAP_NET_BIND_SERVICE | call bind(2) to bind to a privileged port for serving metrics. Only required if the bind port is below 1024 |
$ fdctl run --config config.toml
Log at "/tmp/fd-0.0.0_755725_fd_fd_2024_07_01_03_24_27_470038344_GMT+00"
NOTICE 07-01 03:24:27.471949 755725 f0 main src/disco/topo/fd_topo.c(445):
SUMMARY
Total Tiles: 16
Total Memory Locked: 28145897472 bytes (26 GiB + 218 MiB + 20 KiB)
Required Gigantic Pages: 26
Required Huge Pages: 109
Required Normal Pages: 36
Required Gigantic Pages (NUMA node 0): 26
Required Huge Pages (NUMA node 0): 109
[...]
NOTICE 07-01 03:24:28.772209 755745 16 metric:0 src/disco/topo/fd_topo_run.c(30): booting tile metric:0 pid:755742 tid:755745
NOTICE 07-01 03:24:28.773617 755747 13 shred:0 src/disco/topo/fd_topo_run.c(30): booting tile shred:0 pid:755740 tid:755747
NOTICE 07-01 03:24:28.773798 755748 9 pack:0 src/disco/topo/fd_topo_run.c(30): booting tile pack:0 pid:755739 tid:755748
[...]monitor
Monitors a validator that is running locally on this machine. This is a low level performance monitor mostly useful for diagnosing throughput issues. The monitor takes over the controlling terminal and refreshes it many times a second with up to date information. You can exit the monitor by sending Ctrl+C or SIGINT.
| Arguments | Description |
|---|---|
--config <path> | Path to a configuration TOML file to run the monitor with. This must be the same configuration file the validator was started with |
Capabilities
| Capability | Reason |
|---|---|
CAP_SYS_ADMIN | call unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces |
CAP_SETUID | call setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID |
CAP_SETGID | call setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit |
$ fdctl monitor --config config.toml
snapshot for 2024-07-01 04:31:34.669413032 GMT+00
tile | pid | stale | heart | sig | in backp | backp cnt | % hkeep | % backp | % wait | % ovrnp | % ovrnr | % filt1 | % filt2 | % finish
---------+---------+------------+-------+------------+----------+---------------------+----------+----------+----------+----------+----------+----------+----------+----------
net | 763643 | - | - | run( run) | -( -) | 0( +0) | 42.300 | 0.000 | 57.700 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
quic | 763638 | - | - | run( run) | -( -) | 0( +0) | 0.331 | 0.000 | 99.669 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763641 | - | - | run( run) | -( -) | 0( +0) | 0.490 | 0.000 | 99.509 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763639 | - | - | run( run) | -( -) | 0( +0) | 0.488 | 0.000 | 99.417 | 0.000 | 0.000 | 0.000 | 0.000 | 0.095
verify | 763640 | - | - | run( run) | -( -) | 0( +0) | 0.483 | 0.000 | 99.516 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763644 | - | - | run( run) | -( -) | 0( +0) | 0.490 | 0.000 | 99.510 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763642 | - | - | run( run) | -( -) | 0( +0) | 0.489 | 0.000 | 99.511 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
dedup | 763634 | - | - | run( run) | -( -) | 0( +0) | 0.049 | 0.000 | 99.951 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
pack | 763635 | - | - | run( run) | -( -) | 0( +0) | 0.060 | 0.000 | 99.937 | 0.000 | 0.000 | 0.000 | 0.000 | 0.003
bank | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.287 | 0.000 | 99.584 | 0.000 | 0.000 | 0.000 | 0.000 | 0.129
bank | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.286 | 0.000 | 99.714 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
poh | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.463 | 0.000 | 99.517 | 0.000 | 0.000 | 0.000 | 0.018 | 0.002
shred | 763637 | - | - | run( run) | -( -) | 0( +0) | 0.404 | 0.000 | 99.519 | 0.000 | 0.000 | 0.000 | 0.000 | 0.077
store | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.293 | 0.000 | 99.447 | 0.000 | 0.000 | 0.000 | 0.000 | 0.260
sign | 763636 | - | - | run( run) | -( -) | 0( +0) | 0.239 | 0.000 | 99.729 | 0.000 | 0.000 | 0.000 | 0.000 | 0.032
metric | 763630 | - | - | run( run) | -( -) | 0( +0) | 100.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
link | tot TPS | tot bps | uniq TPS | uniq bps | ha tr% | uniq bw% | filt tr% | filt bw% | ovrnp cnt | ovrnr cnt | slow cnt | tx seq
------------------+----------+----------+----------+----------+----------+----------+----------+----------+---------------------+---------------------+---------------------+-------------------
quic->net | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
shred->net | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
net->quic | 15.4 | 44.4K | 15.4 | 44.4K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
sign->quic | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 15.4 | 39.3K | 15.4 | 39.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 16( +0)
verify->dedup | 15.4 | 43.3K | 15.4 | 43.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 16( +1)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0)
dedup->pack | 15.4 | 43.3K | 15.4 | 43.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
poh->pack | 15.4 | 39.3K | 15.4 | 39.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
poh->pack | 15.4 | 5907.3 | 15.4 | 5907.3 | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
pack->bank | 30.8 | 519.8K | 30.8 | 519.8K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 306( +3)
pack->bank | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 306( +3)
bank->poh | 30.8 | 529.7K | 30.8 | 529.7K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 229( +2)
bank->poh | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 0( +0)
poh->poh | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
pack->poh | 46.2 | 519.8K | 46.2 | 519.8K | 100.000 | 100.000 | 100.000 | 100.000 | 0( +0) | 0( +0) | 0( +0) | 306( +3)
net->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
poh->shred | 969.2 | 597.5K | 969.2 | 597.5K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 4930( +63)
poh->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
poh->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
sign->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
shred->store | 30.8 | 10.3M | 30.8 | 10.3M | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 154( +2)
quic->sign | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
shred->sign | 15.4 | 3938.2 | 15.4 | 3938.2 | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)configure
Configures the operating system so that it can run Firedancer. See the guide for more information. There are the following stages to each configure command:
hugetlbfsReserves huge and gigantic pages for use by Firedancer and mounts huge page filesystems for then under a path in the configuration TOML file.sysctlSet required kernel parameters.hyperthreadsDisables hyperthreaded pair for critical CPU cores.ethtool-channelsConfigures the number of channels on the network device.ethtool-groDisables generic receive offload (GRO) on the network device.ethtool-loopbackDisables UDP segmentation on the loopback device.
| Arguments | Description |
|---|---|
--config <path> | Path to a configuration TOML file to configure the validator with. This must be the same configuration file the validator will be started with |
[hugetlbfs]
mount_path = "/mnt/.fd"
[layout]
net_tile_count = 2
[tiles]
[net]
interface = "ens3f0"configure init <stage>...
Prepare the operating system environment to run Firedancer. This will reserve and mount the huge page filesystems, set the kernel parameters, and configure the number of combined channels on the network device.
Capabilities
| Capability | Reason |
|---|---|
root | increase /proc/sys/vm/nr_hugepages and mount hugetblfs filesystems. Only applies for the hugetlbfs stage |
root | increase network device channels with ethtool --set-channels. Only applies for the ethtool-channels stage |
root | disable network device generic-receive-offload (gro) with ethtool --offload IFACE generic-receive-offload off. Only applies for the ethtool-gro stage |
root | disable network device tx-udp-segmentation with ethtool --offload lo tx-udp-segmentation off. Only applies for the ethtool-loopback stage |
CAP_SYS_ADMIN | set kernel parameters in /proc/sys. Only applies for the sysctl stage |
$ fdctl configure init all
NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE hugetlbfs ... configuring
NOTICE RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=241172480`
NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=27917287424`
NOTICE sysctl ... unconfigured ... kernel parameter `/proc/sys/vm/max_map_count` is too low (got 65536 but expected at least 1000000)
NOTICE RUN: `echo "1000000" > /proc/sys/vm/max_map_count`
NOTICE RUN: `echo "1024000" > /proc/sys/fs/file-max`
NOTICE RUN: `echo "1024000" > /proc/sys/fs/nr_open`
NOTICE RUN: `echo "2" > /proc/sys/net/ipv4/conf/lo/rp_filter`
NOTICE RUN: `echo "1" > /proc/sys/net/ipv4/conf/lo/accept_local`
WARNING kernel parameter `/proc/sys/net/core/bpf_jit_enable` is too low (got 0 but expected at least 1). Proceeding but performance may be reduced.
WARNING kernel parameter `/proc/sys/kernel/numa_balancing` is set to 1, not the expected value of 0. Proceeding but performance may be reduced.
NOTICE hyperthreads ... unconfigured ... pack cpu 5 has hyperthread pair 37 which should be offline
NOTICE hyperthreads ... configuring
NOTICE RUN: `echo "0" > /sys/devices/system/cpu/cpu37/online`
NOTICE RUN: `echo "0" > /sys/devices/system/cpu/cpu40/online`
NOTICE ethtool-channels ... unconfigured ... device `ens3f0` does not have right number of channels (got 1 but expected 2)
NOTICE ethtool-channels ... configuring
NOTICE ethtool-channels ... RUN: `ethtool --set-channels ens3f0 combined 2`
NOTICE ethtool-gro ... unconfigured ... device `ens3f0` has generic-receive-offload enabled. Should be disabled
NOTICE ethtool-gro ... configuring
NOTICE ethtool-gro ... RUN: `ethtool --offload ens3f0 generic-receive-offload off`
NOTICE ethtool-loopback ... unconfigured ... device `lo` has tx-udp-segmentation enabled. Should be disabled
NOTICE ethtool-loopback ... configuring
NOTICE ethtool-loopback ... RUN: `ethtool --offload lo tx-udp-segmentation off`configure check <stage>...
Check if the operating system environment is properly configured. Exits with a non-zero exit code if it is not, after printing relevant diagnostics to stderr.
$ fdctl configure check all
WARNING hugetlbfs ... not configured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
WARNING sysctl ... not configured ... kernel parameter `/proc/sys/vm/max_map_count` is too low (got 65536 but expected at least 1000000)
WARNING ethtool-channels ... not configured ... device `ens3f0` does not have right number of channels (got 1 but expected 2)
WARNING ethtool-gro ... not configured ... device `ens3f0` has generic-receive-offload enabled. Should be disabled
WARNING ethtool-loopback ... not configured ... device `lo` has tx-udp-segmentation enabled. Should be disabled
ERR failed to configure some stagesconfigure fini <stage>...
Remove any Firedancer specific operating system configuration still lingering. This only unmounts the hugetlbfs stages and returns the reserved huge and gigantic pages to the kernel pool. It will not reduce sysctls that were earlier increased, or change the network channel count back as we no longer know what the original value was.
Capabilities
| Capability | Reason |
|---|---|
root | remove directories from /mnt, unmount hugetlbfs. Only applies for the hugetlbfs stage |
$ fdctl configure fini all
NOTICE ethtool ... finishing
NOTICE sysctl ... finishing
NOTICE hugetlbfs ... finishing
NOTICE RUN: `umount /mnt/.fd/.huge`
NOTICE RUN: `rmdir /mnt/.fd/.huge`
NOTICE RUN: `umount /mnt/.fd/.gigantic`
NOTICE RUN: `rmdir /mnt/.fd/.gigantic`
NOTICE RUN: `rmdir /mnt/.fd/.normal`
NOTICE RUN: `rmdir /mnt/.fd`version
Prints the current version of the validator to the standard output and exits. The command writes diagnostic messages from logs to stderr.
$ fdctl version
0.101.11814set-identity
Changes the identity key of a running validator. The <keypair> argument is required and must be the path to an Agave style identity.json keypair file. If the path is specified as - the key will instead be read from stdin.
WARNING
set-identity must be called with the configuration file you started the validator with, like fdctl set-identity --config <config.toml>, if the config argument is not provided, the command may not update the key on all tiles and your validator may start skipping slots.
It is not generally safe to call set-identity, as another validator might be running with the same identity, and if they both produce a block or vote concurrently, the validator may violate consensus and be subject to (future) slashing.
Best practice requires copying the tower.bin file from the prior to the new validator, to ensure that vote lockouts are respected.
The validator will not change identity in the middle of a leader slot, and will wait until any in-progress leader slot completes before switching to the new identity. It is safe to call during or near a leader slot because of this wait.
The command exits successfully (with an exit code of 0) if the identity key was changed, otherwise it will fail and print diagnostic messages to stderr. Reasons for failure include the validator being unable to open or load the tower, when --require-tower is specified, or being unable to load or verify the provided identity key.
Currently due to implementation limitations, the key can be partially changed if the set-identity command is cancelled (for example with Ctrl+C) while running. The next call to set-identity might need to provide the --force argument to succeed if this occurs, to reset this partial state and proceed with setting a new key.
| Arguments | Description |
|---|---|
<keypair> | Path to a identity.json keypair file, or - to read the JSON formatted key from stdin |
--config <path> | Path to a configuration TOML file of the validator to change identity for. This must be the same configuration file the validator was started with |
--require-tower | If specified, refuse to set the validator identity if saved tower state is not found |
--force | If a set-identity operation is abandoned part way through, you will need to specify --force to reset the validator key state when trying again |
$ sudo fdctl set-identity --config ~/config.toml ~/keys/validator-keypair.json
NOTICE Validator identity key switched to `4UCZB7zfquCVN7GafWETFVLTceNH3nm2mndyHeDuSggC`keys
keys pubkey <PATH>
Prints the base58 encoding of the public key in the file at <PATH> to the standard output and exits. The file at <PATH> should be an Agave style identity.json key file. The command writes diagnostic messages from logs to stderr.
$ fdctl keys pubkey ~/.firedancer/fd1/identity.json
Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYFkeys new <identity|vote>
Creates a new keypair from the kernel random number generator and writes it to the identity key path, or vote key path. The key path is retrieved from the configuration TOML file
| Arguments | Description |
|---|---|
--config | Path to a configuration TOML file which determines where the key is written. Either [consensus.identity_path] or [consensus.vote_account_path] for identity or vote arguments respectively |
[consensus]
identity_path = "/home/{user}/.fd/keys/identity.json"$ fdctl keys new vote --config config.toml
NOTICE successfully created keypair in `/home/fd/.fd/keys/identity.json`mem
Prints information about the memory requirements and the tile configuration and layout of the validator to stdout before exiting. The command writes diagnostic messages from logs to stderr.
Firedancer preallocates and locks all memory it needs from huge and gigantic page mounts before booting, and the hugetlbfs stage of fdctl configure will reserve the memory described here for exclusive use by Firedancer.
| Arguments | Description |
|---|---|
--config | Path to a configuration TOML file to print memory usage information with |
$ fdctl mem --config config.toml
SUMMARY
Total Tiles: 17
Total Memory Locked: 27088932864 bytes (25 GiB + 234 MiB + 20 KiB)
Required Gigantic Pages: 25
Required Huge Pages: 117
Required Normal Pages: 27
Required Gigantic Pages (NUMA node 0): 25
Required Huge Pages (NUMA node 0): 117
WORKSPACES
0 ( 1 GiB): net_quic page_cnt=1 page_sz=gigantic numa_idx=0 footprint=68173824 loose=1005563904
1 ( 1 GiB): net_shred page_cnt=1 page_sz=gigantic numa_idx=0 footprint=68173824 loose=1005563904
[...]
LINKS
0 ( 32 MiB): net_quic kind_id=0 wksp_id=0 depth=16384 mtu=2048 burst=1
1 ( 32 MiB): net_shred kind_id=0 wksp_id=1 depth=16384 mtu=2048 burst=1
[...]
TILES
0 ( 3 GiB): net kind_id=0 wksp_id=18 cpu_idx=1 out_link=-1 in=[-2, -3] out=[ 0, 1]
1 ( 3 GiB): quic kind_id=0 wksp_id=19 cpu_idx=2 out_link=4 in=[ 0, -21] out=[ 2, 20]
[...]