Firedancer

Appearance

Getting Started ​

Firedancer ​

This guide details building and running the Firedancer validator.

Hardware Requirements ​

Minimum

  • 24-Core AMD or Intel CPU @ >2.8GHz
  • 256GB RAM
  • 2TB PCI Gen3 NVME SSD (High TBW)

Recommended

  • 32-Core CPU @ >3GHz with AVX512 support
  • 512GB RAM with ECC memory
  • 1 Gigabit/s Network Bandwidth

Validator operators also refer to https://solanahcl.org/ which has a lot of useful information about hardware.

Installing ​

Prerequisites ​

Firedancer must be built from source and currently only supports building and running on Linux. Firedancer requires a recent Linux kernel, at least v4.18. This corresponds to Ubuntu 20.04, Fedora 29, Debian 11, or RHEL 8.

  • GCC 10 or newer
  • make

First, clone the source code with:

sh
$ git clone https://github.com/firedancer-io/firedancer.git
$ cd firedancer
$ git checkout v26.10.5 # Or the latest Firedancer release

Then you can run the deps.sh script to install system packages. These will be installed via the package manager on your system.

sh
$ ./deps.sh

Releases ​

Firedancer does not produce pre-built binaries and you must build from source, but Firedancer releases are made available as tags. The following naming convention is used,

  • main This should not be used. The main branch is bleeding edge and includes all Firedancer development and changes that could break Frankendancer.
  • vYY.MM.PATCH Full Firedancer releases.
  • v0.xxx.yyyyy Legacy Frankendancer releases.

Firedancer versioning has three components,

  • The major (year) and minor (month) versions identify the release line. A new release line starts every month, containing new features and performance improvements.
  • Patch versions contain bug fixes.
================= main branch =================
   \                         \
    \ v26.09.0                \ v26.10.0
     \                         \
      \ v26.09.1                \ v26.10.1
       \
        \ v26.09.2

Building ​

Once dependencies are installed, you can build Firedancer.

sh
$ make -j

You will need around 32GiB of available memory to build Firedancer. If you run out of memory compiling, make can return a variety of errors.

TIP

The Firedancer production validator is built as a single binary firedancer. You can start, stop, and monitor the Firedancer instance from this one program.

Firedancer automatically detects the hardware it is being built on and enables architecture specific instructions for maximum performance if possible. This means binaries built on one machine may not be able to run on another.

If you wish to target a lower machine architecture you can compile for a specific target by setting the MACHINE environment variable to one of the targets under config/.

sh
$ MACHINE=linux_gcc_x86_64 make -j

The default target is native, and compiled binaries will be placed in ./build/native/gcc/<compiler-version>/bin, with a hardlink to each at ./build/<name>.

Updating ​

If you checked out Firedancer using Git, run through these steps to check out a newer version, update dependencies, and rebuild binaries.

sh
git fetch --tags
git checkout v26.10.5
make -j

Running ​

Configuration ​

Firedancer has many configuration options which are discussed later. For now, we override only the essential options needed to start the validator on Testnet.

toml
user = "firedancer"

[gossip]
    entrypoints = [
      "entrypoint.testnet.solana.com:8001",
      "entrypoint2.testnet.solana.com:8001",
      "entrypoint3.testnet.solana.com:8001",
    ]

[paths]
    identity_key = "/home/firedancer/validator-keypair.json"
    vote_account = "/home/firedancer/vote-keypair.json"

[snapshots.sources.gossip]
    allow_any = false
    allow_list = [
        "5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on",
        "dDzy5SR3AXdYWVqbDEkVFdvSPCtS9ihF5kJkHCtXoFs",
        "Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN",
        "eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ",
        "9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv",
    ]

[tiles.rpc]
    rpc_listen_port = 8899

This configuration will cause Firedancer to run as the user firedancer on the local machine. The identity_key and vote_account should be Agave style keys, which can be generated using the firedancer keys subcommand. The vote_account can also be the public key of an existing vote account.

Additionally, this configuration enables the full RPC API at port 8899. Although the port will not be published to other validators in gossip, use a firewall to restrict access to this port for maximum security.

The Firedancer client reports metrics via a Prometheus listening at http://127.0.0.1:7999/metrics.

Permissions ​

There are two users involved in running Firedancer. The user that you launch firedancer with, and the user Firedancer switches to after it has started. The requirements for these users are very different:

  • The user Firedancer starts as is not specified in configuration, but is simply the user that launches the process. For most commands, including firedancer run and configure it needs to be root or have various capabilities described below to setup kernel bypass networking. It is recommended to simply use the root user when launching.

  • The user Firedancer switches to after it has booted up and performed privileged initialization. This is given by the user option in your configuration TOML file. Firedancer requires nothing from this user and it should be as minimally permissioned as possible. It should never be root or another superuser, and the user should not be present in the sudoers file or have any other privileges.

Only the firedancer run and monitor commands will switch to the non-privileged user, and other commands will run as the startup user until they complete. Most commands can be started with capabilities rather than as the root user, although this isn't recommended. If you are an advanced operator, you can see which capabilities are required for a command by running it unprivileged:

$ ./build/firedancer run --config ~/config.toml
WARNING run ... process requires capability `CAP_NET_RAW` to call `socket(2)` to bind to a raw socket for use by XDP
WARNING run ... process requires capability `CAP_SYS_ADMIN` to call `bpf(2)` with the `BPF_OBJ_GET` command to initialize XDP
WARNING run ... process requires capability `CAP_SYS_ADMIN` to call `unshare(2)` with `CLONE_NEWUSER` to sandbox the process in a user namespace
WARNING run ... process requires capability `CAP_NET_BIND_SERVICE` to call `bind(2)` to bind to a privileged port for serving the GUI
ERR     insufficient permissions to execute command `run`. It is recommended to start Firedancer as the root user, but you can also start it with the missing capabilities listed above. The program only needs to start with elevated permissions to do privileged operations at boot, and will immediately drop permissions and switch to the user specified in your configuration file once they are complete. Firedancer will not execute outside of the boot process as root, and will refuse to start if it cannot drop privileges. Firedancer needs to be started privileged to configure high performance networking with XDP.

For additional layers of defense against local privilege escalation, it is not suggested to setcap(8) the firedancer binary as this can create a larger attack surface.

Initialization ​

The validator uses some Linux features that must be enabled and configured before it can be started correctly. It is possible for advanced operators to do this configuration manually, but firedancer provides a command to check and automate this step.

WARNING

Running any firedancer configure command may make permanent changes to your system. You should be careful before running these commands on a production host.

The initialization steps are described in detail later. But plowing ahead at the moment:

sh
$ sudo ./build/firedancer configure init all --config ~/config.toml

You will be told what steps are performed:

NOTICE  hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE  hugetlbfs ... configuring
NOTICE  RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE  RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=6155141120`
NOTICE  RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE  RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=150323855360`
NOTICE  ethtool-channels ... unconfigured ... device `bond0` missing ethtool-channels network configuration
NOTICE  ethtool-channels ... configuring
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 start 2 equal 126`
NOTICE  RUN: `ethtool --config-ntuple enp1s0f0np0 flow-type udp4 dst-port 8003 src-ip 0.0.0.0 m 255.255.255.254 queue 0`
WARNING error configuring network device (enp1s0f0np0), ioctl(SIOCETHTOOL,ETHTOOL_SRXCLSRLINS) failed (95-ENOTSUP-operation not supported)
WARNING error configuring network device (bond0), rss_queue_mode "auto" attempted "dedicated" configuration but falling back to "simple".
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f0np0 combined 2`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE  RUN: `ethtool --config-nfc enp1s0f0np0 rx-flow-hash udp4 sdfn`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE  RUN: `ethtool --set-channels enp1s0f1np1 combined 2`
NOTICE  RUN: `ethtool --set-rxfh-indir enp1s0f1np1 default`
NOTICE  RUN: `ethtool --config-nfc enp1s0f1np1 rx-flow-hash udp4 sdfn`
NOTICE  irq-affinity ... unconfigured ... found IRQs overlapping with Firedancer tile CPUs (IRQs: 1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,+45 more; tiles: )
NOTICE  irq-affinity ... configuring
NOTICE  kworkers ... unconfigured ... kernel workqueue cpumask includes Firedancer tile CPUs or their isolated siblings 94
NOTICE  kworkers ... configuring
NOTICE  RUN: `echo "fffffff0,03fffff9,fffffff0,00000001" > /sys/devices/virtual/workqueue/cpumask`
NOTICE  kernel workqueue cpumask changed while writing `/sys/devices/virtual/workqueue/cpumask`: fffffe00,40000001,fffff800,00000001 -> fffffff0,03fffff9,fffffff0,00000001 (kernel 7.0.0-34-generic). 
NOTICE  cpuset ... unconfigured ... `/sys/fs/cgroup/fd1` does not exist
NOTICE  cpuset ... configuring
NOTICE  RUN: `echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control`
NOTICE  RUN: `mkdir /sys/fs/cgroup/fd1`
NOTICE  RUN: `echo "1-35,65-66,90-99" > /sys/fs/cgroup/fd1/cpuset.cpus`
NOTICE  RUN: `echo "isolated" > /sys/fs/cgroup/fd1/cpuset.cpus.partition`
NOTICE  console ... unconfigured ... console loglevel is 4; kernel messages render console work on the printing CPU
NOTICE  console ... configuring
NOTICE  RUN: `echo "0" > /sys/class/graphics/fbcon/cursor_blink`
NOTICE  RUN: `echo "2" > /proc/sys/kernel/printk`

It is strongly suggested to run the configure command when the system boots, and it needs to be run each time the system is rebooted.

Running ​

Finally, we can run Firedancer:

sh
$ sudo ./build/firedancer run --config ~/config.toml

Firedancer logs selected output to stderr and a more detailed log to a local file. Every tile in Firedancer runs in a separate process for security isolation, so you will see a complete process tree get launched.

sh
$ pstree 111819 -as
systemd --switched-root --system --deserialize=51
  └─sudo ./build/firedancer run --config ~/config.toml
      └─firedancer run --config ~/config.toml
          └─firedancer run --config ~/config.toml
              ├─accdb:0 run1 accdb 0 --pipe-fd 31 --config-fd 0
              ├─admin:0 run1 admin 0 --pipe-fd 9 --config-fd 0
              ├─dedup:0 run1 dedup 0 --pipe-fd 51 --config-fd 0
              ├─diag:0 run1 diag 0 --pipe-fd 7 --config-fd 0
              ├─event:0 run1 event 0 --pipe-fd 60 --config-fd 0
              ├─execle:0 run1 execle 0 --pipe-fd 54 --config-fd 0
              ├─execle:1 run1 execle 1 --pipe-fd 55 --config-fd 0
              ├─execrp:0 run1 execrp 0 --pipe-fd 32 --config-fd 0
              ├─execrp:1 run1 execrp 1 --pipe-fd 33 --config-fd 0
              ├─execrp:2 run1 execrp 2 --pipe-fd 34 --config-fd 0
              ├─execrp:3 run1 execrp 3 --pipe-fd 35 --config-fd 0
              ├─execrp:4 run1 execrp 4 --pipe-fd 36 --config-fd 0
              ├─execrp:5 run1 execrp 5 --pipe-fd 37 --config-fd 0
              ├─execrp:6 run1 execrp 6 --pipe-fd 38 --config-fd 0
              ├─execrp:7 run1 execrp 7 --pipe-fd 39 --config-fd 0
              ├─execrp:8 run1 execrp 8 --pipe-fd 40 --config-fd 0
              ├─execrp:9 run1 execrp 9 --pipe-fd 41 --config-fd 0
              ├─gossip:0 run1 gossip 0 --pipe-fd 26 --config-fd 0
              ├─gossvf:0 run1 gossvf 0 --pipe-fd 24 --config-fd 0
              ├─gossvf:1 run1 gossvf 1 --pipe-fd 25 --config-fd 0
              ├─gui:0 run1 gui 0 --pipe-fd 61 --config-fd 0
              ├─ipecho:0 run1 ipecho 0 --pipe-fd 8 --config-fd 0
              ├─metric:0 run1 metric 0 --pipe-fd 6 --config-fd 0
              ├─net:0 run1 net 0 --pipe-fd 11 --config-fd 0
              ├─net:1 run1 net 1 --pipe-fd 12 --config-fd 0
              ├─netlnk:0 run1 netlnk 0 --pipe-fd 5 --config-fd 0
              ├─pack:0 run1 pack 0 --pipe-fd 53 --config-fd 0
              ├─motor:0 run1 motor 0 --pipe-fd 56 --config-fd 0
              ├─quic:0 run1 quic 0 --pipe-fd 44 --config-fd 0
              ├─rotor:0 run1 rotor 0 --pipe-fd 28 --config-fd 0
              ├─replay:0 run1 replay 0 --pipe-fd 30 --config-fd 0
              ├─resolv:0 run1 resolv 0 --pipe-fd 52 --config-fd 0
              ├─rpc:0 run1 rpc 0 --pipe-fd 59 --config-fd 0
              ├─rserve:0 run1 rserve 0 --pipe-fd 29 --config-fd 0
              ├─shred:0 run1 shred 0 --pipe-fd 27 --config-fd 0
              ├─sign:0 run1 sign 0 --pipe-fd 57 --config-fd 0
              ├─sign:1 run1 sign 1 --pipe-fd 58 --config-fd 0
              ├─votor:0 run1 votor 0 --pipe-fd 42 --config-fd 0
              ├─verify:0 run1 verify 0 --pipe-fd 45 --config-fd 0
              ├─verify:1 run1 verify 1 --pipe-fd 46 --config-fd 0
              ├─verify:2 run1 verify 2 --pipe-fd 47 --config-fd 0
              ├─verify:3 run1 verify 3 --pipe-fd 48 --config-fd 0
              ├─verify:4 run1 verify 4 --pipe-fd 49 --config-fd 0
              ├─verify:5 run1 verify 5 --pipe-fd 50 --config-fd 0
              └─waker:0 run1 waker 0 --pipe-fd 10 --config-fd 0

If any of the processes dies or is killed it will bring all of the others down with it.

Networking ​

Firedancer uses AF_XDP, a Linux API for high performance networking. For more background see the kernel documentation.

Although AF_XDP works with any ethernet network interface, results may vary across drivers. Popular well tested drivers include:

  • ixgbe — Intel X540
  • i40e — Intel X710 series
  • ice — Intel E800 series

Firedancer installs an XDP program on the network interface [net.interface] and lo while it is running. This program redirects traffic on ports that Firedancer is listening on via AF_XDP. Traffic targeting any other applications (e.g. an SSH or HTTP server running on the system) passes through as usual. The XDP program is unloaded when the Firedancer process exits.

AF_XDP requires CAP_SYS_ADMIN and CAP_NET_RAW privileges. This is one of the reasons why Firedancer requires root permissions on Linux.

WARNING

Packets received and sent via AF_XDP will not appear under standard network monitoring tools like tcpdump.