Node operation

Running qubits-node: every subcommand and flag, what each combination of flags starts, the single-validator and multi-validator startup sequences, how the process is built, the mempool and signature cache, exits and shutdown, health, metrics, logs and resources. This is a reference for node operators and for auditors reading the node. Commands and log lines come from chains run with the 0.2.3 binaries. Where a limit is given, it is a constant in the code.

The binary

qubits-node init   --genesis FILE --data-dir DIR
qubits-node run    --data-dir DIR [options]
qubits-node era    fetch | verify | import

qubits-node --version prints qubits-node 0.2.3, and --help works at every level. There is no configuration file. Apart from the Rust runtime's own RUST_BACKTRACE, the only environment variable that matters is RUST_LOG.

init

Creates a chain database from a genesis file.

$ qubits-node init --genesis genesis.json --data-dir data
initialized chain `qubits-example` in data
genesis hash: 01481e2aa25b64045d1ef080cac50cffd33e2eb0cfa05e0e765bfd8587305517

A directory that holds a chain is refused: configuration: data already holds a chain. An invalid genesis is refused before the directory is touched.

run

Runs the node and its HTTP API. There are fourteen flags, one of them hidden.

Addresses are parsed as socket addresses, so a host name is refused: HOST:PORT must be an IP address and port.

The round timeout is round_timeout_ms * 3^failed / 2^failed in integers, with failed = min(round - high_qc.round - 1, 6), so the default tops out at 17,085 ms. One quarter of the same flag, 375 ms by default, is the grace before a leader sends a certificate and before a missing block is requested. See Consensus.

Timing has two more fields with no flag: tx_ttl_ms, 600,000, and regossip_ms, 3,000. They are fixed in the binary.

Era subcommands

Era files are described on Sync and history.

era fetch downloads eras from a node's HTTP API into a directory.

era verify checks files against a genesis file, from the genesis validators up. It needs no database, and it exits with 1 if any file is invalid.

era import takes era files into a stopped node's database. They are executed after the head, or put back below the oldest block held.

Roles

What run starts depends on the key, on whether the key is in the current validator set, and on --follow.

In every P2P row, a current validator with no --peer entry stops the start: {n} current validators have no --peer address.

Startup order

run does these in order. The HTTP server starts before catching up, so validators starting together can follow each other.

1. Set the panic hook and the log subscriber.
2. Node::open(data_dir).
3. Apply --prune-keep and --era-length.
4. Spawn the history thread, unless --history-keep all.
5. Read the key file and the password, and decrypt the key.
6. Start the Tokio runtime, bind --rpc-addr, and start serving.
7. For a key in the set with --follow, log catching up before joining consensus and run catch_up for at most 3,600 s. A failure logs a warning, and startup goes on.
8. For no key with --follow, call forward_to and spawn the follower thread.
9. For a key not in the set without --follow, exit.
10. Start the role from the table above. For P2P, resolve --peer entries, open the transport, and log accepting validator connections on {addr}.
11. Log serving HTTP API on http://{addr}, then wait for Ctrl-C or for the server to fail.

A single validator

This is the first-deployment shape. Blocks are produced by one key, there are no peers, and no P2P port is opened.

qubits key new --out validator.json
qubits key show --key validator.json      # the public key for genesis
qubits key new --out alice.json
qubits address --key alice.json           # the address to fund
# write genesis.json, see /docs/state#genesis
qubits-node init --genesis genesis.json --data-dir data
qubits-node run --data-dir data --validator-key validator.json --password-file pw

pw holds the key's password on its first line. A run logs:

2026-11-07T01:31:44.309250Z  INFO qubits_node::validator: consensus epoch started epoch=0 validators=1 index=0
2026-11-07T01:31:44.309335Z  INFO qubits_node: serving HTTP API on http://127.0.0.1:28545 chain=qubits-docs height=0 validator=true

Height stays 0 until a transaction arrives, because an idle chain makes no blocks.

Several validators

Genesis holds all validator keys, and every validator gets the same genesis file. Each runs init once on its own data directory. For four validators on 10.0.0.1 to 10.0.0.4, the first runs:

qubits-node run \
  --data-dir data \
  --validator-key validator.json --password-file pw \
  --rpc-addr 10.0.0.1:8545 --p2p-addr 10.0.0.1:9545 \
  --peer 1=10.0.0.2:9545 --peer 2=10.0.0.3:9545 --peer 3=10.0.0.4:9545 \
  --follow http://10.0.0.2:8545 \
  --follow http://10.0.0.3:8545 \
  --follow http://10.0.0.4:8545

The others run the same with their own addresses. Start them in any order. --follow needs each validator's --rpc-addr reachable from the others, and a validator with it syncs before voting. --peer indexes refer to genesis, so use hex keys for validators added later. The lower key dials the higher one. Both sides need the entry, because the peer list is also the list of keys a node accepts connections from. See Networking.

A four-validator start, seen from index 1:

INFO qubits_node: catching up before joining consensus
INFO qubits_node: accepting validator connections on 127.0.0.1:29601
INFO qubits_node::validator: consensus epoch started epoch=0 validators=4 index=1
INFO qubits_node: serving HTTP API on http://127.0.0.1:28601 chain=qubits-docs-4 height=0 validator=true
INFO qubits_node::p2p: validator connected peer=d7aed88a7406
INFO qubits_node::p2p: validator connected peer=71e0a5ec5bc4
INFO qubits_node::p2p: validator connected peer=6910d647238d
INFO qubits_node::validator: status height=0 round=1 epoch=0 peers=3 mempool=0 in_msgs=0 in_kb=0 out_msgs=0 out_kb=0

The chain needs a quorum of n minus f validators up to make progress, 3 of 4 here.

Validator set changes are on Validators and epochs. A key added later starts as a standby with --follow and a --peer for every current validator, and logs not a validator yet: following the chain until a validator-set change adds this key.

A validator's votes are recorded in consensus_safety in its data directory, synced before each vote leaves. A wiped or restored directory forgets them, and the key can then sign a second vote in a round it already voted in. Restart validators on their own directory.

An API node

An API node has no validator key and serves the same HTTP API.

qubits-node init --genesis genesis.json --data-dir api
qubits-node run --data-dir api --rpc-addr 127.0.0.1:8545 \
  --follow http://10.0.0.1:8545 --follow http://10.0.0.2:8545

It joins from a snapshot if the source is more than --snapshot-if-behind blocks ahead, and then replays. Transactions submitted to it go through the same checks as on a validator, then are POSTed to every source. An archive node sets --history-keep all, leaves out --prune-keep, and sets --snapshot-if-behind above the chain's height so that a fresh start replays from genesis instead of taking a snapshot. It needs a source with full history. Trust and verification are on Sync and history.

Structure

pub struct Node {
    store: Arc<Store>,
    genesis: Genesis,
    genesis_hash: Hash,
    mempool: Arc<Mutex<Mempool>>,
    signatures: Arc<SignatureCache>,
    gossip: Mutex<Option<Arc<dyn Transport>>>,
    forward: Mutex<Option<SyncSender<SignedTransaction>>>,
    keep_versions: Arc<AtomicU64>,
    era_length: AtomicU64,
    submits: Arc<tokio::sync::Semaphore>,
    metrics: Metrics,
}

pub struct Validator {
    node: Arc<Node>,
    key: SecretKey,
    timing: Timing,
    last_housekeeping: u64,
    last_status: u64,
    reported: Traffic,
    keeping_up: Option<KeepingUp>,
    transport: Arc<dyn Transport>,
    intake: Intake,
    current: Option<EpochEngine>,
    retired: Option<(EpochEngine, u64)>,
}

pub struct Timing {
    pub round_timeout_ms: u64,
    pub min_block_interval_ms: u64,
    pub tx_ttl_ms: u64,
    pub regossip_ms: u64,
}

Node is the state every role shares: store, mempool, signature cache, and the two ways out for a transaction, gossip and forwarding. Validator owns the consensus engine for the current epoch, and the previous one for 60 s (RETIRED_FOR_MS), which votes for nothing and writes nothing but helps laggards.

The engine is a state machine with no I/O of its own. The validator thread feeds it and carries out what it returns:

1. Wait for a message, for at most the time to the engine's next deadline and never more than 50 ms.
2. Hand a message to handle, which decodes it and routes it by epoch to the current or retired engine. On a timeout, call tick.
3. If the deadline has passed, call tick.
4. act carries out each Output in order. PersistSafety and PersistBlock are written with sync before later outputs run, so no vote leaves before the disk has it. Send and Broadcast go to the transport.
5. An error panics the thread on purpose.

tick logs status every 10 s, runs keep_up, updates the gauges, and every regossip_ms expires and re-gossips the mempool.

Threads

Mempool

The mempool is in memory, and a restart loses it. It keeps transactions in arrival order, and that is the only order: fees do not affect it.

Admission

Node::accept is the same for HTTP and gossip.

1. Take the mempool lock. admission refuses a transaction already in the pool (Duplicate), a full pool, or a sender at 64 (MempoolFull). Copy the sender's waiting transactions, and release the lock.
2. Look the id up in tx_index. An included transaction is Duplicate.
3. Read the head, and build a BlockEnv for height head + 1 at max(now, head.timestamp + 1).
4. Create an Executor on the store with the signature cache, without_contracts().
5. Apply the sender's waiting transactions, without checking their signatures again and ignoring failures, so the nonce and balance are what they will be.
6. apply the new transaction. A failure is Invalid.
7. Take the lock, run admission again, and insert.

Signatures are checked in step 6 with no lock held. ChainApp::propose, has_work and write_committed take the same lock on the consensus thread, and ML-DSA verification under it would hold up consensus. That is why admission runs twice: the pool can change between the steps.

without_contracts() means a contract call is checked for fee, nonce and signatures, and not run. Submitting must not run code for free.

Node::submit adds the transaction received log line, broadcasts to peers and forwards to sources.

A gossiped transaction takes Validator::handle to Intake::push, a try_send onto a queue, and thread tx-intake runs accept. A validator doesn't pass it on when it arrives. It reaches peers through re-gossip if it is still waiting.

Leaving the pool

A proposer takes the first 2 * max_block_transactions waiting transactions that aren't in uncommitted parents, and builds from those. See speculative execution.

Every regossip_ms a validator broadcasts up to 64 of the oldest transactions that have waited at least that long. An API node re-forwards the same way every 3 s. Forwarded transactions that sit in a mempool also clear from it when the node sees them in a committed block.

An expired transaction logs dropped expired transactions expired=1.

Signature cache

A validator checks a transaction when it arrives, and again in every block built on it. ML-DSA makes that the most expensive step of a block, so SignatureCache remembers what verified. One cache is shared by accept, ChainApp and apply_proven.

An entry is digest("qubits.sh 2026-11 signature cache", raw key ‖ message length as u64 LE ‖ message ‖ raw signature). It is a fact about a key, a message and a signature, so it never goes stale. Whether that key may sign for the account is still decided at execution. Only Domain::Transaction signatures that verified are remembered, and a failure is not.

When young reaches GENERATION, it replaces old, and a new young starts.

Executor::prewarm runs before each block executes:

1. For each transaction, find the policy that must sign: the account's auth or the reveal for an owner, vault.recovery for a recovery signature.
2. Collect (key, message, signature) jobs.
3. SignatureCache::prewarm drops the ones it knows. If fewer than 8 are left, or the machine has one thread, it returns.
4. Otherwise it splits them across available_parallelism scoped threads and verifies. Failures are left for execution to report.

This is only a head start. A transaction that changes a key makes a later one miss, and that one verifies in turn.

Exits and supervision

A panic in any thread exits the whole process, rather than keep serving the API without consensus. The hook prints the panic, then fatal: stopping the node, and exits with 101, so a supervisor must restart it. Storage failures in consensus panic on purpose. A real one, forced with a file size limit:

thread 'validator-6910d647' (929544) panicked at crates/qubits-node/src/validator.rs:702:25:
validator stopped: storage: IO error: While appending to file: p-v0/000011.log: File too large
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal: stopping the node

Panic output goes to stderr, logs to stdout. Every commit and every vote record is a synced batch, and a restarted validator resumes from ConsensusTip, SafetyRecord and pending_blocks, so a crash at any point leaves a consistent database. See Consensus.

Only one process can open a data directory. A second fails on RocksDB's lock: storage: IO error: While lock file: {dir}/LOCK: Resource temporarily unavailable. A port in use fails with binding {addr}.

Shutdown

tokio::signal::ctrl_c handles SIGINT only. On it, the node logs shutting down, drops the validator handle, which stops and joins its thread, then the follower and the history thread, and exits with 0.

SIGTERM has no handler. The process ends at once, with no shutting down line (measured: killed by signal 15). The database stays consistent, since it has to survive a crash anyway, but a clean stop needs SIGINT. Supervisors usually send SIGTERM, so set the stop signal, as in the systemd example.

Health and metrics

Both sit on --rpc-addr next to /v1.

GET /health answers ok {height} and a newline, or 500 if the head can't be read:

ok 131

It says the process serves and can read its head. It does not say consensus is moving, and neither does a flat height: an idle chain makes no blocks.

GET /metrics is Prometheus text. Each metric has a # TYPE line:

# TYPE qubits_height gauge
qubits_height 103
# TYPE qubits_peers gauge
qubits_peers 3
# TYPE qubits_messages_in_total counter
qubits_messages_in_total 2390

A qubits_peers below n - 1 shows a lost link even on an idle chain.

Logs

Logs go to stdout through tracing, as a UTC timestamp, a level, a target and fields, with ANSI colours only when stdout is a terminal. The filter comes from RUST_LOG, default info. era subcommands log too, so an era import shows committed block lines.

2026-11-07T01:31:44.461937Z  INFO qubits_node: transaction received id=8d772ab9e0738a4c60b154196e78ef1ea9abe4b747bedad11e7c7d5c0b6e3b25 sender=qb1mhjvmv2rfrw23yazwauh047rdrfw35dxw7lcxr8a9mkytfcwgghqrqmlgl nonce=0 fee=5432 bytes=5432
2026-11-07T01:31:44.625011Z  INFO qubits_node::app: committed block height=1 round=1 txs=40 kb=212 id=fe10b19f2b035705701de8d05afd95ddc3257bc811aa6cbf362695d2e61e7b74
2026-11-07T01:35:14.893031Z  INFO qubits_node::validator: status height=53 round=55 epoch=0 peers=3 mempool=40 in_msgs=202 in_kb=3315 out_msgs=204 out_kb=3538
2026-11-07T01:41:02.271363Z  INFO qubits_node::sync: following the chain height=357 mempool=0

There is one line per committed block and received transaction, and a summary every 10 s: status on validators, with traffic since the last line, and following the chain on API nodes. A standby logs no summary until it joins.

RUST_LOG=qubits_node=debug adds every message sent and received, by type, round and size. kind is one of proposal, certify, block, vote, timeout, sync, block-request and transaction. from and to are the first 6 bytes of the raw key.

DEBUG qubits_node::validator: received from=ead4f8458275 kind="proposal" round=1 epoch=0 bytes=220768
DEBUG qubits_node::validator: sent to="all" kind="vote" round=1 epoch=0 bytes=3367 peers=3
DEBUG qubits_node::validator: received from=d7aed88a7406 kind="vote" round=1 epoch=0 bytes=3367
DEBUG qubits_node::p2p: refused connection from=127.0.0.1:62742 error=not a configured validator

Plain RUST_LOG=debug turns on every dependency too.

Exposing the API

The server is axum::serve with no limit on HTTP connections and no header timeout, and it speaks plain HTTP and ws://. Run public API nodes behind a reverse proxy that adds TLS, connection limits and a timeout on reading headers, and keep --rpc-addr on a private address. The proxy also decides whether /metrics and /health are public, since they share the port with /v1.

What the node itself enforces:

POST /v1/contracts/{address}/view runs code for free, so it shares the permits. Routes are on the API page.

/v1/ws needs the proxy to pass WebSocket upgrades, and the proxy's TLS is what makes the address wss://. With nginx:

location /v1/ws {
    proxy_pass http://127.0.0.1:8545;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

--ws-limit counts all clients together, so one client could take every connection. Limit connections per address at the proxy.

Clocks

A block with a timestamp more than MAX_CLOCK_DRIFT_MS, 10,000 ms, ahead of the verifier's clock is refused, and timestamps are what valid_until_ms and vault delays are measured against. Keep validators' clocks in sync: a clock more than 10 s behind the proposer's stops that validator from voting on its blocks.

Resources

There are no published hardware requirements. What follows is computed from the code or was measured once, as stated.

Measured

Four validators on one machine, over loopback TLS: Apple M5 Max, 18 cores, 48 GiB. Release build, default flags, max_block_transactions 1,000. 9,600 transfers from 80 accounts were submitted over HTTP as fast as the API took them, a short burst and not a sustained rate.

The write-ahead log holds everything written, pending_blocks included, until RocksDB flushes it.

Sizes for planning

Those give, before database overhead:

An idle chain makes no blocks, so these are rates under load only. --history-keep bounds blocks and proofs, --prune-keep bounds old state, and an archive node with neither keeps growing.

Memory bounds

systemd

The flags below were run on a real node with other paths. The unit itself has not been loaded by systemd.

[Unit]
Description=Qubits validator
After=network-online.target
Wants=network-online.target

[Service]
User=qubits
Environment=RUST_LOG=info
ExecStart=/usr/local/bin/qubits-node run \
    --data-dir /var/lib/qubits \
    --validator-key /etc/qubits/validator.json \
    --password-file /etc/qubits/password \
    --rpc-addr 10.0.0.1:8545 \
    --p2p-addr 10.0.0.1:9545 \
    --peer 1=10.0.0.2:9545 \
    --peer 2=10.0.0.3:9545 \
    --peer 3=10.0.0.4:9545 \
    --follow http://10.0.0.2:8545 \
    --follow http://10.0.0.3:8545 \
    --follow http://10.0.0.4:8545 \
    --history-keep 7d
KillSignal=SIGINT
TimeoutStopSec=30
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

KillSignal=SIGINT is what makes a stop clean. Restart=on-failure covers exit 101, and a stop through systemd stays stopped.

Source

• chain/crates/qubits-node/src/main.rs
• chain/crates/qubits-node/src/lib.rs
• chain/crates/qubits-node/src/validator.rs
• chain/crates/qubits-node/src/mempool.rs
• chain/crates/qubits-node/src/metrics.rs
• chain/crates/qubits-node/src/app.rs
• chain/crates/qubits-node/src/rpc.rs
• chain/crates/qubits-node/src/history.rs
• chain/crates/qubits-node/src/network.rs
• chain/crates/qubits-node/src/p2p.rs
• chain/crates/qubits-types/src/signatures.rs
• chain/crates/qubits-types/src/lib.rs
• chain/crates/qubits-execution/src/lib.rs
• chain/crates/qubits-consensus/src/engine.rs