Sync and history

How a node gets the chain from other nodes without trusting them, how it joins from a state snapshot, what it drops as history expires, and how history moves between nodes as era files. This is a reference for node operators and auditors, down to what each step verifies and what each failure message means. Log lines and sizes here come from chains run with the 0.2.3 binaries, some with the hidden test flag --era-length 64 so that eras complete quickly.

Trust model

A node starts from its genesis file and trusts nothing else it reads over HTTP. Sources are the nodes named with --follow URL, and any number of them may lie.

What a follower relies on:

• The genesis file it was initialized with.
• At most f of the n = 3f + 1 validators of each epoch being faulty. A fresh node accepts any chain of epochs that verifies from the genesis validators.
• One reachable source that answers. Sources are not compared or voted on: a pass uses one source, and the next pass uses another only if this one failed.

A source cannot make a node store a block that has no valid proof, or state that doesn't hash to a certified root.

The follow loop

Follower holds the node, the source list, a next_source counter and an HTTP agent with a 60 s timeout per request. Replies are capped at MAX_REPLY_BYTES, 64 MiB. One pass is Follower::step, which adds 1 to next_source if the pass fails. The steps of a pass:

1. GET /v1/status from sources[next_source % len]. The genesis_hash must match this node's.
2. If the source's height is not above the head, stop: nothing to do.
3. If the head is 0, and the source is more than --snapshot-if-behind blocks ahead or its blocks_from is above 1, try a snapshot. If one was taken, the pass ends there. If the source offers none, go on.
4. If the source's blocks_from is above head + 1, fail: it no longer has the next block.
5. Set end to the lower of the source's height and the next multiple of 64 above the head.
6. GET /v1/blocks/{h}/raw for each height from head + 1 to end, with ?prove=true on the last. /v1/blocks/{h}/raw includes a proof if one is kept for that height, and prove=true builds one when it isn't.
7. Run apply_proven on the run of blocks.

A pass covers at most 64 blocks, and while the source is far ahead it ends on a height that has a kept proof.

Who drives it:

catch_up asks for the height, passes if behind, and sleeps 200 ms when a pass applied nothing. It returns an error at its timeout or after UNREACHABLE_AFTER, 30 s, of silence. At startup that error logs could not catch up; joining consensus anyway, and the validator goes on.

Verifying blocks

apply_proven(node, parent, parent_height, blocks, proofs) takes blocks in height order and applies only those it can prove. It reads the set and chain of the head's epoch (current_view) and calls committed_prefix, which for each block in turn does these checks:

1. The height is parent_height + 1 + i, or the run fails: expected height {h}, got {h'}.
2. block.parent is the id of the block before, or the run fails: block {h} does not extend the previous block.
3. If a proof came with the block, proof.tip must be the block's id and proof.verify(block.round, validators, chain) must pass. A proof that passes moves the proven prefix up to this block. A proof that fails is cleared and is no error.
4. If the block changes the validators, it must have its own good proof, because the next block is signed by others. Without one, the walk stops here. With one, the view switches to the new set, with chain set to this block's id.

A block counts as proven if it, or anything linked above it in the run, has a good proof. Blocks past the last good proof are dropped, and a later pass fetches them again. See finality proofs for why one proof covers its ancestors.

Then, for each block of the proven prefix, in order:

5. check_tx_root, or the pass fails.
6. execute_block_cached runs every transaction with the real rules. Blocks contain only valid transactions, so any failure fails the pass: block {h}: ....
7. write_committed prepares the tree, compares state_root and receipt_root, and commits in one synced batch. See writing state.

Every block is re-executed, so a certificate alone never changes a follower's state: a block with valid signatures from a quorum and an invalid transaction stops at step 6.

The ConsensusTip written with each block gets the certificate of the next proof at or above it in the run. /v1/blocks/{h}/raw returns { height, block, finality }, where block is the hex of borsh ChainBlock, and finality the hex of a borsh FinalityProof or null.

Snapshots

A fresh node far behind takes the state at a recent certified height instead of replaying every block. Below that height it holds no blocks and no state.

What a server offers

Node::snapshot_height:

1. regular = head - head % 64, the newest multiple of PROOF_EVERY.
2. epoch_start is the first height of the current epoch, and candidate is the higher of the two.
3. There is no snapshot if candidate is 0, is below pruned_before, or has no kept proof.

GET /v1/state/snapshot answers with that height, or 503 and no snapshot to serve:

pub struct SnapshotResponse {
    pub height: u64,
    pub state_root: Hash,
    pub block: String,    // hex of borsh ChainBlock
    pub finality: String, // hex of borsh FinalityProof
}

A node pruning fewer than 64 versions could fall below the candidate and offer nothing, but --prune-keep refuses values below 100, so a pruned node on the command line always has one.

Walking the epochs

The pivot block is signed by the validators of its epoch, and the only set a fresh node knows is genesis. So it walks every validator change from there.

1. GET /v1/state/snapshot. Treat 404 and 503 as "no snapshot".
2. GET /v1/epochs, which lists every epoch as { epoch, start, start_height }.
3. For each epoch above 0, GET /v1/blocks/{start_height}/raw. A reply without a proof fails: an epoch's first block comes without a proof.
4. The pivot's height and root must match the summary, or the snapshot block does not match its summary.
5. Run checked_pivot.

epoch_views builds the chain of sets. It starts with the genesis validators and chain set to the genesis hash. For each epoch i + 1, in order:

1. The summary must match the block: epoch is i + 1, start is the block's id, start_height is its height, and that height is above the previous epoch's.
2. The block must contain a SetValidators, have a good tx_root, and proof.tip must be its id.
3. proof.verify(block.round, validators, chain) must pass with the set of epoch i. This is the link: each set vouches for the next.
4. The last SetValidators in the block gives the next set, with chain set to the block's id.

checked_pivot then checks the pivot itself:

1. proof.tip is the pivot's id, and its tx_root is good. Otherwise the pivot block and its proof do not match.
2. Find the last epoch whose start_height is below the pivot's height: that set signed the pivot. A pivot at genesis fails.
3. proof.verify with that set, or the pivot's proof fails.
4. If the pivot changes the validators, an epoch must start exactly there, or the pivot changes the validators but no epoch starts there.
5. Return the epoch the pivot belongs to, and the list of epochs up to it.

Leaving epochs out of /v1/epochs doesn't help a source: a pivot signed by a later set then fails step 3.

Downloading the state

GET /v1/state/ranges?height=H&from=KEY&bytes=N returns application/octet-stream, the borsh of:

pub struct StateChunk {
    pub leaves: Vec<([u8; 32], Vec<u8>)>, // key hash, value; in key-hash order
    pub proof: Option<StateRangeProof>,
    pub done: bool,
}

from is 32 bytes of hex, default all zero. A leaf counts 32 bytes plus its value toward the limit. The limit is soft: the leaf that crosses it is still included, so asking for 1,500 bytes returned one leaf of 2,008 bytes and a proof, 2,314 bytes in all.

The follower walks it like this:

1. Store::begin_restore(height, state_root) refuses unless the head is 0 (a snapshot can only be restored into a fresh chain) and the height is above 0.
2. Start with from at 32 zero bytes.
3. Fetch a chunk, take chunk.next() as the next from, which is the last key hash plus 1, or nothing once done is set.
4. Restore::add_chunk(chunk).
5. Log restoring the state every 50 chunks.

add_chunk hands the leaves and the proof to JellyfishMerkleRestore:

1. Key hashes must rise strictly from the last one seen.
2. Each leaf goes into the partial tree, with its value hash computed locally.
3. The proof holds only right_siblings: what lies to the right of the path to the chunk's last leaf. The left siblings come from the leaves received so far, so a proof at chunk i covers chunks 0 to i. The root computed from the last leaf and both sets of siblings must be the certified root.
4. Finished nodes and values go to jmt_nodes and jmt_values through Sink, unsynced.

A lying source fails at its first bad chunk. It can't insert, change or leave out a leaf, because the root would differ. No trailing part of the tree can be omitted either, because the right siblings of the last leaf commit to it.

Finishing

Store::finish_restore(restore, block):

1. The block's height and root must match the restore.
2. inner.finish() writes what is left.
3. get_root_hash(version) is compared with the root once more.
4. The head must still be 0.
5. One synced batch sets latest_height, pruned_before and blocks_from to the pivot, and writes the block, its tx_index entries and its metadata.

Before step 5 the node is still at height 0, and nothing it restored counts. The metadata is the tip, the tip proof, finality/<pivot>, consensus_epoch and consensus_epochs.

After a snapshot the node holds:

Later passes replay from the pivot. qubits_snapshot_syncs_total goes up by one.

A join in the logs, with 44 leaves, on loopback:

INFO qubits_node::sync: taking the state from a snapshot height=320
INFO qubits_node::sync: state restored from a snapshot height=320 leaves=44 chunks=1
INFO qubits_node::app: committed block height=321 round=321 txs=40 kb=135 id=6a192319e94d958f...

Failure modes

A failed attempt is retried by the next pass, with the next source, from the start. The two caveats were reproduced against Store.

• The pivot is pruned or the source goes away while downloading. /v1/state/ranges answers 404, state at version {v} was pruned (kept from {p}), and the pass fails. On a node with --prune-keep 100 a pivot can be as little as 37 blocks from being pruned.
• Another source has another pivot. The next attempt starts over with that pivot.
• A source that holds no epoch blocks. A node that joined from a snapshot doesn't hold the blocks that ended earlier epochs, so on a chain that has changed validators, a joiner pointed only at it fails at GET /v1/blocks/{start_height}/raw. Put eras back on that node first, or follow a node with full history.

A failed attempt leaves its nodes and values in jmt_nodes and jmt_values. The markers don't move, so the node stays at height 0 and those versions are never read, until a retry finishes at a later pivot. Then a key that held a value at the failed pivot and was deleted before the new one still answers from jmt_values, while the tree proves it absent. Only contract storage keys are deleted, so this needs a contract. If taking the state from a snapshot is logged without a matching state restored from a snapshot, stop the node, delete the data directory and run init again.

If a chunk reaches its limit exactly on the last leaf live at the pivot, and jmt_values holds further key hashes that aren't live there, such as keys created after the pivot, done is false and the next chunk has no leaves and no proof. Restore::add_chunk refuses it with chunk without a proof. That attempt fails, and so does every retry at that pivot. The next pivot gets a different walk.

Keeping up while validating

A validator follows the chain through consensus, and its peers answer block requests from memory. The engine keeps blocks down to BLOCK_RETENTION, 256 blocks below the committed height, and answers with up to SYNC_BATCH, 64, ancestors. A validator further behind than that can't catch up through consensus. With --follow, it syncs over HTTP before that happens.

Validator::follow starts a KeepingUp with its own thread, keep-up, which polls remote_height every 2 s, so watching the sources never holds up consensus. On the validator thread, every 2 s, keep_up:

1. Computes behind = remote - head.
2. Counts a stuck check if behind is above 0 and the head has not moved. On an idle chain no traffic arrives to catch up from, so this is the only way to notice.
3. Sets far if behind is above 128.
4. Does nothing if behind is 0, or if this key is a member and neither far nor stuck.
5. Logs syncing from the follow sources at WARN with behind.
6. If far, runs catch_up(2, 300 s). Otherwise runs one pass.
7. On success, adds 1 to qubits_syncs_total. If the head moved, drops the current engine and opens the epoch again from the new tip.

A validator removed from the set has no engine, so it counts as a non-member and keeps syncing like this.

catch_up in step 6 runs on the validator thread, and the height it acts on is unverified. A source that reports a height it can't back makes a member with --follow stop running consensus until catch_up times out, 300 s at a time. Point --follow at nodes you run or trust to be up.

A key that is not in the set runs spawn_standby and needs --follow. It does a pass every 100 ms, drops anything arriving from peers, and checks after each pass whether the set now holds its key. Once it does, it starts a Validator, logs joined the validator set, and hands its Follower to Validator::follow. See Validators and epochs.

History expiry

--history-keep bounds how long a node keeps old blocks. The default is 7d, and all keeps everything. State is not affected: that is --prune-keep, see state pruning.

The value is all, or a count and a unit: 7d, 12h, 30m, 45s, for days, hours, minutes and seconds. The count must be above 0.

Unless the value is all, thread history runs on every node, validators included. After a run that deleted a full batch it goes again at once. Otherwise it waits PRUNE_EVERY.

prune_once computes cutoff = now - keep, and Store::prune_history walks from blocks_from upward, stopping at head - 128 or after 1,024 blocks. For each block held:

1. If its timestamp is at or above cutoff, stop. Everything from here on stays.
2. If its height is a multiple of the era length, or it changes the validators, keep it.
3. Otherwise delete it, with its tx_index entries, its receipts entries and finality/<height>.

blocks_from moves past everything looked at, kept or not, in one synced batch with the deletions. So it means "the oldest block from which on everything is held", and kept blocks sit below it.

Epoch ends hand the next validator set to joining nodes. Era ends anchor era files, and any range of blocks can be proven final from the next one. A checkpoint proof with 50 validators is about 113 KB, per 8,192 blocks.

/v1/status reports blocks_from. Fetching a dropped block answers 404, history before block {from} is not kept on this node, and a dropped transaction 404, unknown transaction. qubits_history_pruned_total counts blocks deleted.

Measured with --era-length 64 --history-keep 1s, at head 357:

INFO qubits_node::history: dropped old blocks deleted=64 from=229

blocks_from was 229, which is 357 - 128. Block 64, an era end, still answered, and block 65 answered 404.

Era files

An era file holds a run of blocks that can be checked against the genesis file alone, so history can live anywhere. An archive node hands them out, and a node that dropped its history takes them back.

Era index holds heights index * length + 1 to (index + 1) * length, with length 8,192. Era 0 is heights 1 to 8,192, and every era ends on a block whose proof is kept forever. The length is a chain-wide agreement. --era-length exists for tests, and every node of a chain must use the same value.

Layout

All borsh, integers little-endian. Blocks are streamed, never all held in memory.

Header
  magic         [u8; 4]   "QERA"
  version       u8        1
  genesis_hash  [u8; 32]
  first_height  u64
  count         u32       blocks in the file
Vec<EraEpoch>             u32 length, then the items
  epoch         u64
  block         ChainBlock
  proof         FinalityProof
count times
  block         ChainBlock
  proof         Option<FinalityProof>   1 tag byte, then the proof if the tag is 1

The header is 49 bytes. Vec<EraEpoch> holds every epoch above 0 that starts below first_height: the blocks and proofs a reader needs to get from the genesis set to the set in force at the start of the file. Each block carries the proof kept for its height, if there is one. The last block always has one.

The first bytes of a real era 0, from a chain with genesis hash 4203f664...:

00000000: 5145 5241 0142 03f6 6474 3dd9 48e3 0b61  QERA.B..dt=.H..a
00000010: 7a14 c320 a806 db4c d194 66fe efae a8f5  z.. ...L..f.....
00000020: ee5a 9f26 3101 0000 0000 0000 0040 0000  .Z.&1........@..
00000030: 0000 0000 0001 0000 0000 0000 0001 0000  ................

Two 64-block files from that chain, holding 1,200 and 1,160 transfers, were 4,256,574 and 4,039,298 bytes.

File names are {chain_id}-{first 8 hex of the genesis hash}-{index:05}.era, and for a part of an era {chain_id}-{hash}-{index:05}-to-{height}.era.

Serving

An era is listed if its first block is at or above blocks_from and the proof at its end is kept. A node that has lost part of an era doesn't offer it. ?to=H needs H inside the era and at or below the head, the era's first block held, and a proof kept at H, because a file must end on a proof. It answers 404 otherwise.

Commands

qubits-node era fetch --from URL --out DIR [ERA...]
qubits-node era fetch --from URL --out DIR --to HEIGHT
qubits-node era verify --genesis FILE FILES...
qubits-node era import --data-dir DIR FILES...

era fetch with no era numbers takes every era the source offers, and skips files already in DIR. Each file is written as {name}.part, synced, then renamed. There is no timeout. Flags are listed on the node page.

--to HEIGHT is for a node that holds blocks from HEIGHT on: it fetches the part of that era up to HEIGHT, then each complete era below, newest first. That is what a node that joined from a snapshot needs to get its history back, and HEIGHT is its pivot, which has a proof.

$ qubits-node era fetch --from http://127.0.0.1:28545 --out eras
qubits-docs-4203f664-00000.era: 4 MB
qubits-docs-4203f664-00001.era: 4 MB

$ qubits-node era fetch --from http://127.0.0.1:28545 --out eras
qubits-docs-4203f664-00000.era: already here
qubits-docs-4203f664-00001.era: already here

$ qubits-node era verify --genesis genesis.json eras/*.era
eras/qubits-docs-4203f664-00000.era: ok, blocks 1..=64, 1200 transactions, 1 proofs
eras/qubits-docs-4203f664-00001.era: ok, blocks 65..=128, 1160 transactions, 1 proofs

era::verify

era verify needs nothing but the genesis file. It exits with code 1 if any file fails, and prints INVALID: {reason} for that file.

1. Reader::open checks the magic and the version, and refuses count 0 or first_height 0.
2. The genesis hash in the header must match: another chain's era.
3. epoch_views walks Vec<EraEpoch> as in walking the epochs, from the genesis validators.
4. The set in force is the last epoch starting below first_height. A later epoch in the list fails: epochs after the era's start included.
5. For each block: the height must be the next one, parent must be the previous id, and check_tx_root must pass.
6. If the block has a proof, it must verify with the set in force.
7. A block that changes the validators must have a proof, and then the set in force changes.
8. The last block must have a proof: the last block has no proof.

Blocks are linked by hash, and block ids commit to their payloads, so the proof at the end covers the whole run. Verification does not execute anything or check transaction signatures. That takes state, and era import does it when appending.

Flipping one bit in a file gave:

bad.era: INVALID: block 1: decoding failed: tx_root does not match transactions
Error: some files failed

era::import

era import opens the database itself, so stop the node first. A running node makes it fail with storage: IO error: While lock file: {dir}/LOCK: Resource temporarily unavailable. Node::open needs the database initialized with the same genesis. It prints what it did per file, then height {h}, blocks held from {from}.

With first and last the file's heights, head the node's head and from its blocks_from:

Append goes through apply_proven, the path of verifying blocks, including full re-execution. It flushes at each block with a proof, and at 64 blocks at the latest. Blocks left unproven at the end fail with the era's last blocks are not provably final. It rebuilds a node from files alone: init, then import from era 0 up, oldest first. Blocks are logged as usual.

a-eras/qubits-docs-4203f664-00000.era: executed blocks 1..=64
a-eras/qubits-docs-4203f664-00001.era: executed blocks 65..=128
height 128, blocks held from 1

Put back gets history back under a node that started from a snapshot or dropped it. It does not execute: state below state_from is gone, and the blocks are only served.

1. Set up the same Checker as era::verify, from the file's epochs, and run it on every block.
2. Stream the blocks. Those below from go to blocks, with their tx_index entries and proofs, 64 at a time through Store::put_blocks. They are not visible yet: blocks_from hasn't moved.
3. Where the file overlaps blocks held, the first such block's id must equal the one held, or block {h} is not the one this node holds. That links the whole file to a block this node already trusts.
4. If the file ends right under from, the block held at from must have the file's last block as its parent, or the era does not link to block {from}.
5. set_blocks_from(first).

Any failure undoes the writes with Store::delete_blocks, leaving index entries that lead nowhere. Import newest first, because each file must reach up to the blocks already held.

$ qubits-node era import --data-dir api eras/qubits-docs-4203f664-00001.era eras/qubits-docs-4203f664-00000.era
eras/qubits-docs-4203f664-00001.era: put back blocks 65..=128
eras/qubits-docs-4203f664-00000.era: put back blocks 1..=64
height 131, blocks held from 1

After that, the node in this run reported blocks_from: 1 and offered eras 0 and 1 itself. state_from stayed 128, so a proof at height 5 answered state at that height was pruned.

Era files hold no receipts. After a put back, /v1/transactions/{id} finds a contract transaction but its receipt is null. Append re-executes, so it gets receipts back.

What failures look like

A follower blocked on keeps blocks only from does not recover by itself, and it logs that line about twice a second until the operator acts.

Source

• chain/crates/qubits-node/src/sync.rs
• chain/crates/qubits-node/src/history.rs
• chain/crates/qubits-node/src/era.rs
• chain/crates/qubits-node/src/validator.rs
• chain/crates/qubits-node/src/lib.rs
• chain/crates/qubits-node/src/rpc.rs
• chain/crates/qubits-node/src/main.rs
• chain/crates/qubits-node/tests/sync.rs
• chain/crates/qubits-state/src/store.rs
• chain/crates/qubits-consensus/src/engine.rs
• chain/crates/qubits-consensus/src/types.rs
• chain/crates/qubits-types/src/rpc.rs
• chain/crates/qubits-e2e/solo.py