Consensus

Qubits orders blocks with Pipelined Moonshot (Doidge et al., DSN 2024), written as a sans-I/O state machine in the qubits-consensus crate. This page specifies that implementation for protocol engineers and auditors: messages and certificates, the voting, commit and timeout rules, what must be durable and when, the bounds against faulty validators, and what has been checked and how. Validator set changes are on Validators and epochs. The transport below the engine is on Validator networking.

Parameters

Validators have equal weight. With n validators:

f      = floor((n - 1) / 3)    Byzantine validators tolerated
quorum = n - f                 signatures in a certificate

For n = 3f + 1 the quorum is 2f + 1. For other sizes it is larger: 50 validators tolerate 16 and need 34 signatures, not 33.

ValidatorSet::new in the engine accepts 1 to 65,535 distinct keys. The chain rules allow 1 to 50 (see Validators and epochs). Round is u64. ValidatorIndex is u16, the position of a key in the set.

Every engine runs against a chain hash: the genesis hash in epoch 0, and the id of the block that ended the previous epoch afterwards. Rounds start at 1 in every epoch. Round 0 exists only as the genesis certificate, QuorumCert::genesis(chain), which has no votes and names chain as its block.

grace_ms is round_timeout_ms / 4, 375 ms by default. It sets both the delay before a leader sends a certificate and the delay before a missing block is requested.

The good case

time     0          δ                 2δ                      3δ
B(r)     proposed   received, voted   QC(r) formed everywhere
B(r+1)              proposed          received, voted         QC(r+1) formed, B(r) final
B(r+2)                                proposed                received, voted

Votes are broadcast, so every validator counts a quorum from the votes it receives and forms each certificate itself. The leader of the next round proposes in the same step in which it votes, without waiting for a certificate. Blocks therefore carry no certificate on this path. A block follows every δ, and a block is final 3δ after it was proposed. The design doc records 2δ and 5δ for Jolteon, the consensus of v0.1.

ML-DSA signatures do not aggregate, so a certificate holds one 3.3 KB signature per signer. Keeping certificates off the wire in the good case is the reason for the design choices below: the one deviation from the paper, sparse finality proofs, and validators that never re-broadcast a certificate.

Data structures

All structures are encoded with borsh. A Hash is 32 bytes. A PublicKey or Signature is encoded as a Vec<u8> holding a one-byte scheme tag and the raw bytes, so an ML-DSA-65 key takes 1957 bytes (4 + 1 + 1952) and a signature 3314 bytes (4 + 1 + 3309).

Blocks

pub struct Header {
    pub round: Round,
    pub height: u64,
    pub parent: Hash,
    pub proposer: ValidatorIndex,
    pub voters: Vec<ValidatorIndex>,
    pub payload: Hash,
}

pub struct Block<P> {
    pub round: Round,
    pub height: u64,
    pub parent: Hash,
    pub proposer: ValidatorIndex,
    /// Validators the proposer last heard a vote from. Only picks leaders: nothing proves it, and
    /// a faulty proposer can add names but not remove the ones honest proposers reported.
    pub voters: Vec<ValidatorIndex>,
    pub payload: P,
}

A block's id is the hash of its Header, where payload is the hash of the borsh-encoded payload. The chain instantiates P as BlockPayload, so a ChainBlock is a Block<BlockPayload>.

Proposals

pub struct Proposal<P> {
    pub block: Block<P>,
    pub justify: Option<QuorumCert>,
    pub tc: Option<TimeoutCert>,
    pub signature: Signature,
}

The two options tell the three kinds of proposal apart. Any other combination is refused.

Votes and quorum certificates

pub enum VoteKind {
    Optimistic,
    Normal,
    Fallback,
}

pub struct Vote {
    pub block: Hash,
    pub round: Round,
    pub kind: VoteKind,
    pub voter: ValidatorIndex,
    pub signature: Signature,
}

pub struct QuorumCert {
    pub block: Hash,
    pub round: Round,
    pub kind: VoteKind,
    pub votes: Vec<(ValidatorIndex, Signature)>,
}

A certificate holds votes of one kind only: an optimistic quorum proves no timeout certificate exists for the round before, which the safety argument needs. QuorumCert::verify accepts round 0 only as the genesis form. Any other certificate needs at least a quorum of entries, in strictly ascending validator order, every one of which must verify. Entries beyond the quorum are allowed and are checked too.

Timeouts and timeout certificates

pub struct Timeout {
    pub round: Round,
    pub high_qc: QuorumCert,
    pub voter: ValidatorIndex,
    pub signature: Signature,
}

pub struct TimeoutCert {
    pub round: Round,
    pub entries: Vec<(ValidatorIndex, Round, Signature)>,
    pub high_qc: QuorumCert,
}

Each entry of a TimeoutCert is a validator, the round of the highest certificate it reported, and its signature. TimeoutCert::verify requires high_qc.round to equal the highest reported round, requires high_qc itself to verify, and requires a quorum of entries under the same rules as a QuorumCert. A faulty validator therefore cannot inflate a reported round without producing that certificate.

Finality proofs

pub struct FinalityProof {
    pub tip: Hash,
    pub path: Vec<Header>,
    pub qc: QuorumCert,
}

What is signed

Every signature is ML-DSA-65 under Domain::Consensus, FIPS 204 context qubits/consensus/v1, over the borsh encoding of one value of this private enum:

enum Signed<'a> {
    Proposal { chain: &'a Hash, block: &'a Hash, justify: Option<Hash>, tc: Option<Hash> },
    Vote { chain: &'a Hash, block: &'a Hash, round: Round, kind: VoteKind },
    Timeout { chain: &'a Hash, round: Round, high_qc_round: Round },
}

• chain binds every signature to one epoch.
• justify and tc are digests of the attached certificates, so the proposer's signature covers them. Stripping or swapping a certificate invalidates the proposal.
• kind is signed, so a vote of one kind cannot be relabelled as another.
• A timeout signs the round of its highest certificate, not the certificate. That is what lets TimeoutCert carry one high_qc and a list of rounds.

Encoded sizes

Measured by encoding real signed values. voters lists all n validators, the payload is one byte, every Timeout carries a full certificate, and a TimeoutCert has one quorum of entries.

A Vote is 3,357 bytes, 3,367 bytes in its NetMessage envelope. A Timeout carrying the genesis certificate is 3,369 bytes. A QuorumCert of q votes is 45 + 3316q bytes.

Proposing

Who may propose

slot decides what a validator may propose right now. In order:

1. If it voted for block b in its current round r and it leads round r + 1 on top of b, it may propose in round r + 1 on b, optimistically. This branch runs before any certificate for b exists.
2. If it already voted or timed out in its current round, it may not propose.
3. If it does not lead the current round on top of high_qc.block, it may not propose.
4. If high_qc.round + 1 == round, it may propose optimistically on high_qc.block.
5. If it holds a timeout certificate with tc.round + 1 == round, it may propose on high_qc.block with justify and tc: a fallback proposal.

try_propose then refuses to act while replaying at startup, if it already proposed in that round (proposed_in), before last_block_ms + min_block_interval_ms, or if App::has_work is false and there is no TC to carry. The last two cases set proposal_due so the attempt is repeated. If the parent is not known yet, the block is fetched instead.

A block's voters field lists the validators this proposer heard a vote from at a round of at least high_qc.round - 2. The list comes from a BTreeMap, so it is sorted.

The one deviation from the paper

In the paper, the leader also sends the certificate of the round before as a normal proposal. A certificate is a quorum of 3.3 KB signatures, so Qubits sends it only when the optimistic path did not get the round certified in time.

1. A leader that holds the certificate of round r - 1 and has proposed in round r arms normal_due (arm_normal).
2. The wait is grace_ms. It is 0 if its block does not extend high_qc.block (stale), or if timeout_round + 1 >= r with a non-zero timeout_round. A validator that timed out in the round before cannot vote optimistically, so the leader does not wait.
3. advance_round clears normal_due. A round certified within grace_ms sends no certificate at all.
4. When normal_due passes, normal_propose signs a normal proposal. If its block extends high_qc.block, it broadcasts Message::Certify { block, justify, signature }, which avoids sending megabytes of block again. The signature is the normal proposal's. If its block does not extend high_qc.block, it builds a new block and broadcasts a full proposal.
5. A receiver that holds the block rebuilds the Proposal from it and handles it as usual. One that does not processes the certificate and schedules a fetch of the block.

This is safe because it only delays a message. A run of this protocol is a run of the paper's protocol on a network that delivers that message late or not at all, and safety assumes nothing about timing. The SMT model lets the network withhold certificates forever.

Validators do not re-broadcast certificates either. The normal proposal stays in the protocol so that a locked validator can be made to vote again. A validator that misses a certificate catches up with the Sync it gets back for its timeout.

Receiving a proposal

on_proposal handles Message::Proposal, Message::BlockResponse and a rebuilt Message::Certify:

1. Compute the block id. The proposal is solicited if the engine is replaying or asked for this id.
2. Ignore it if the block is known and justify is empty, or if the block was refused and the proposal is not solicited.
3. Check the shape from the table above. Reject height <= committed_height, round <= committed_round, and an invalid proposer signature.
4. Verify tc and then justify. Process the certificate, then the TC, which can advance the round.
5. If the block is known, this was a normal or fallback proposal for it. Try to vote and stop.
6. Drop the block if its round is more than VOTE_WINDOW (64) ahead.
7. Record (round, proposer, justify.is_some()) in seen. A second block in that slot is dropped unless solicited.
8. If the parent is unknown, park the proposal in waiting. Request the parent at once if this proposal was solicited, otherwise after grace_ms.
9. Check height == parent.height + 1, round > parent.round, voters strictly ascending and below n, and proposer == leader(round, parent). Then call App::verify. On failure, add the id to refused and stop.
10. Record the block, emit Output::PersistBlock unless replaying, and update last_block_ms.
11. Try to commit, vote and propose. Re-queue children waiting for this block.

Voting rules

Engine::vote_kind holds the rules. They are figure 3 of the paper, and every clause is used by its safety proof.

fn vote_kind(&self, id: &Hash, p: &Proposal<P>) -> Option<VoteKind> {
    let (s, v) = (&self.safety, p.block.round);
    if v != self.round || s.timeout_round >= v {
        return None;
    }
    match (&p.justify, &p.tc) {
        (None, None) => {
            let clean = s.timeout_round == 0 || s.timeout_round + 1 < v;
            let locked = self.high_qc.round + 1 == v && self.high_qc.block == p.block.parent;
            (clean && locked && !s.voted_in(v)).then_some(VoteKind::Optimistic)
        }
        (Some(_), None) => {
            let conflicts = s.optimistic.is_some_and(|(r, opt)| r == v && opt != *id);
            (s.voted_round < v && !conflicts).then_some(VoteKind::Normal)
        }
        (Some(qc), Some(tc)) => (s.voted_round < v && qc.round >= tc.max_high_qc_round())
            .then_some(VoteKind::Fallback),
        (None, Some(_)) => None,
    }
}

voted_in(r) is true if voted_round >= r or the latest optimistic vote is at round r or above. Each clause below names the rule that switches it off in the SMT model. Without any one of them, the solver finds a fork.

1. v != self.round keeps votes in the current round. A block that arrives early waits in arrived and is voted for when the round is reached.
2. s.timeout_round >= v is the rule no vote after timing out. A timeout reports the validator's highest certificate, and what is built on a TC relies on that report. Having timed out in round v, a validator does not vote there or below. Tests: no_vote_after_timing_out_in_the_same_round, a_timeout_is_written_down_and_survives_a_restart.
3. clean is the rule optimistic needs a clean past: no timeout in round v - 1 or later. A quorum of optimistic votes in v and a TC for v - 1 then cannot both exist. Each holds at least n - 2f honest validators, and two such groups exceed the n - f honest validators, so one honest validator would have to be in both. timeout_round == 0 means it never timed out. Test: no_optimistic_vote_after_timing_out_in_the_round_before.
4. locked is the rule optimistic needs the lock. An optimistic proposal carries no certificate, so the voter must hold the parent's certificate itself, as its highest, from exactly the round before. Tests: an_optimistic_vote_waits_for_the_parents_certificate, an_optimistic_vote_needs_the_certificate_of_exactly_the_round_before.
5. !s.voted_in(v) is the rule one vote before optimistic: one optimistic vote in a round, and none after any other vote there or later.
6. s.voted_round < v is the rule one normal or fallback vote, shared by both kinds. An optimistic vote does not count against it.
7. conflicts is the rule normal respects optimistic. A normal vote may repeat the validator's optimistic vote on the same block, so that validators that voted either way can join one normal quorum. It may not name another block. Test: a_normal_vote_may_repeat_an_optimistic_one_but_not_contradict_it.
8. qc.round >= tc.max_high_qc_round() is the rule fallback needs the rank. A fallback block must extend a certificate at least as high as every certificate reported in the TC, so it cannot skip what an earlier round certified. A fallback vote may follow an optimistic vote in the same round, because a TC for the round before means no optimistic quorum exists in this one. Tests: a_fallback_vote_needs_a_tc_and_the_highest_certificate_in_it, a_fallback_vote_is_allowed_after_an_optimistic_vote_in_the_same_round.

Four more rules of the model live outside vote_kind:

Casting and counting votes

cast_vote updates optimistic or voted_round, copies high_qc into the safety state, emits Output::PersistSafety, signs, and broadcasts. Then the validator may propose on the block it voted for.

on_vote handles a received vote:

1. Ignore it if its round is more than VOTE_WINDOW ahead. The vote is not verified yet.
2. If the authenticated sender is the voter, update heard. A validator can report only itself this way.
3. Ignore it if round <= high_qc.round, or if the entry for (round, kind, block) already has a quorum.
4. Ignore it if the voter already voted in that round and kind for any block, or if the signature fails.
5. Store it. At exactly a quorum, build a QuorumCert and process it.

Processing a certificate raises high_qc if it is higher, schedules a fetch of its block if unknown, tries to commit, advances to qc.round + 1, and votes for any block waiting in the new round.

Commit and finality

Every ancestor of a certified block is certified too, because honest validators vote only after seeing the parent's certificate. A single certificate therefore speaks for its whole chain. The commit rule needs two consecutive rounds anywhere on the path below it:

committed ... tip ... a ... b ... certified
                      r    r+1     high_qc

Here b is a child of a with b.round == a.round + 1, and high_qc certifies a descendant of b or b itself. Then a and every ancestor of it are final.

try_commit walks from high_qc.block towards the committed block and stops at the first block whose round is its parent's plus one. It then commits that parent, the lower block of the highest pair. If a block on the way is unknown, or the walk reaches the committed height, nothing happens.

commit collects the blocks from the committed block to the target, asserts that each is above the committed height, and calls App::commit with the blocks in order and a FinalityProof { tip, path, qc }. path holds the headers from the child of tip up to the certified block, and qc is high_qc. A conflict with a committed block fails the assertion with safety violation, and in a node that panic exits the process.

FinalityProof::verify(tip_round, validators, chain) needs only the tip's round and the validator set. It checks:

1. Each header's parent links to the one before it, starting from tip.
2. Somewhere on that walk, a header's round is the previous round plus one. An empty path fails.
3. qc.block is the last header's id and qc.round its round.
4. qc verifies against validators and chain.

A node keeps proofs sparsely, because one per block at 50 validators and 10 blocks a second would cost about 100 GB a day. Proofs live under finality/<height> every PROOF_EVERY (64) blocks, at the block that ends an epoch, and for the tip. Node::finality_proof builds a proof for any height from the next one kept, by extending path. See Node and Sync.

Timeouts

A round's deadline is set when the round begins:

failed     = min(6, round - (high_qc.round + 1))
timeout_ms = round_timeout_ms * 3^failed / 2^failed     (integer division)

At the deadline, tick checks whether the chain is idle (see Pacing and the idle pause). If it is not, local_timeout runs:

1. If this round is new or high_qc is newer than the persisted one, update timeout_round and high_qc in the safety state and emit Output::PersistSafety.
2. Sign a Timeout for the round with the current high_qc and broadcast it.
3. Clear requested and fetching. Broadcast a BlockRequest for every parent that blocks are waiting for and for high_qc.block if unknown.
4. Set a new deadline. The timeout repeats until the round moves.

on_timeout handles a received timeout:

1. Verify the signature and the carried certificate, and process the certificate.
2. If t.round is below the current round, reply with Sync { high_qc, last_tc } to that validator and stop.
3. Keep only the newest timeout per validator.
4. If a quorum of timeouts exists for that round and no TC at that round or later is held, build a TimeoutCert. It is not sent: f + 1 timeouts make every honest validator time out too, so each builds its own.
5. Run join_timeouts.

Processing a TC updates last_tc, processes its high_qc, advances to tc.round + 1 and tries to propose.

join_timeouts sorts the newest timeout round of every validator in descending order and takes the entry at index f. If that round is at or past the current one and this validator has not timed out there, it advances to that round and times out itself. f + 1 timeouts at or past a round include an honest validator, so the round is safe to join. This exists because restarts lose TCs: without it, rounds could drift apart with no certificate to bring them together. The test rounds_follow_f_plus_one_timeouts_and_no_fewer shows one faulty validator moving nobody.

on_sync verifies and processes the carried certificate and any TC newer than the one held.

Leader rotation

Engine::leader(round, parent) depends on the chain being extended:

1. Walk from parent through at most REPUTATION_WINDOW (10) known blocks, parent included.
2. Collect each block's proposer and voters into a sorted set.
3. If the set has fewer members than a quorum, return round % n over all validators.
4. Otherwise return candidates[round % candidates.len()].

Crashed validators leave the set within the window and stop being chosen, and a validator that returns is heard again and leads again. voters is not authenticated, because it only widens the set. Two forks can have different leaders for the same round. A restarted node rebuilds the window from the last 10 blocks of the epoch through Start::recent.

Pacing and the idle pause

min_block_interval_ms paces the whole chain, because it is measured from the last block seen and not from the last block this validator proposed. last_block_ms is set when a validator builds a block and when it accepts one. It is a local rule, not a validity rule. Without it a single validator would loop forever, since its own vote completes the certificate.

An idle chain produces no blocks and no traffic. App::has_work(parent) gates two things:

• A proposal without a TC waits, and proposal_due retries after max(min_block_interval_ms, 20) ms (IDLE_POLL_MS is 20).
• At a deadline, idle pushes the deadline forward instead of timing out if nothing is parked in waiting, high_qc.round + 1 >= round, and has_work(high_qc.block) is false.

A proposal that carries a TC ignores has_work, because after a timeout a block is needed to carry the TC.

The default has_work returns true. ChainApp returns true while the mempool holds a transaction, while a validator change is uncommitted on the chain, or while an uncommitted ancestor has transactions or is unknown, so pipelined blocks keep coming until those commit.

Block requests

Certificates and children often arrive before a block, because votes are smaller and take other paths. A missing block is therefore requested only once overdue:

• fetch(id, from) schedules a request grace_ms from now, unless one is pending. tick sends it when due, unless the block arrived.
• request(id, from) sends BlockRequest at once, once per id, recorded in requested. It is used for parents of solicited blocks and at startup.
• The peer asked is the proposal's sender, or the first signer of the certificate that named the block (holder).

answer_block_request walks from the requested id through its ancestors, up to SYNC_BATCH (64) blocks, and sends them oldest first as BlockResponse, each as the proposal this validator stored. It stops at the first block it does not hold, and when the requester has used SERVE_BYTES_PER_SECOND (32 MiB) in the current wall-clock second. The bytes served are charged per validator.

A block this validator asked for is solicited, which lets it pass the seen gate. That is how a validator gets a second block from an equivocating leader once a certificate names that block.

Shards

shards.rs is transport only. A leader that sends a block to everyone uploads it n - 1 times, which caps a chain at a fraction of one uplink. Validator::spread instead cuts a large proposal into n Reed-Solomon shards with reed-solomon-simd 3.1.0:

k        = n - 2f       data shards; any k rebuild the message
recovery = 2f           recovery shards
len      = ceil(size / k), rounded up to a multiple of 2, at least 2

Honest validators alone hold enough shards: up to f can drop theirs and n - f >= k remain. And k > f, so f validators cannot rebuild a message alone.

pub struct Shard {
    pub round: Round,
    pub proposer: ValidatorIndex,
    pub root: Hash,
    pub size: u32,
    pub index: u16,
    pub proof: Vec<Hash>,
    pub data: Vec<u8>,
}

Shard i goes to validator i. A shard commits to a Merkle tree of all shards, padded with Hash::ZERO to the next power of two. root seals the tree's root with size, round and proposer, under these contexts:

shards::worthwhile(bytes, n) chooses the route for a proposal: n >= 4, bytes * (n - 1) > DIRECT_BYTES (6 MiB), and bytes at most MAX_SIZE (16 MiB). 6 MiB is about 50 ms at 1 Gbit/s, the time the extra hop costs on a network spanning continents. Message::BlockResponse always goes directly.

The leader sends each validator its shard and broadcasts its own shard to all peers. A validator that receives its own shard from the proposer passes it to all peers. The sizes below were measured with split. A shard frame is for an 8 MiB message, excluding the 4-byte frame prefix. Copies are multiples of the message size, for an 8 MiB message.

A relayer uploads (n - 1) / k copies, under 3.1 for every n from 4 to 100. A leader uploads twice that, because it sends n - 1 shards and then its own shard n - 1 times. The transport sends to every configured peer, so standby validators in the --peer list add to these numbers.

Assembler::add(from, shard, round) takes the authenticated sender and the current round. It refuses a shard:

• outside the window of 8 rounds behind (ROUNDS_BEHIND) to 64 rounds ahead (ROUNDS_AHEAD),
• from anyone but the proposer or the validator with that index, so a sender holds at most one slot per message,
• with a bad index, size or proof length, or a proof that does not reach root.

A shard is relayed once, only if it is this validator's own and came from the proposer, and for at most ROOTS_PER_ROUND (2) roots per proposer and round. With k shards the message is rebuilt once. Memory is limited to MAX_HELD_BYTES (128 MiB) per sender, and forget_before drops anything older than the window.

What comes out is an ordinary signed proposal. The node accepts only a NetMessage::Consensus holding a Message::Proposal, because a shard can hold anything its sender likes, shards inside shards too. The node hands it to the engine as coming from the proposer. A leader lying about shards can waste its own round and nothing else. The Merkle root does not prove the shards are a valid encoding, so a bad set fails to rebuild, and whoever cannot rebuild a block fetches it once a certificate names it.

Messages

pub enum Message<P> {
    Proposal(Proposal<P>),
    /// A normal proposal for a block already sent: the signature is the proposal's
    Certify { block: Hash, justify: QuorumCert, signature: Signature },
    Vote(Vote),
    Timeout(Timeout),
    Sync { high_qc: QuorumCert, last_tc: Option<TimeoutCert> },
    BlockRequest(Hash),
    BlockResponse(Proposal<P>),
}

On the wire each is wrapped in NetMessage::Consensus { epoch, message }, described on Validator networking.

Bounds against faulty validators

Round arithmetic on network input saturates or uses checked_add, so a message with Round::MAX is harmless (extreme_rounds_from_the_network_are_harmless).

Engine API

The engine does no I/O, starts no threads and reads no clock. Time and messages come in as arguments and everything it wants done comes out as Output values, which is what lets the tests run whole networks in simulated time.

impl<P, A> Engine<P, A>
where
    P: Clone + BorshSerialize,
    A: App<P>,
{
    pub fn new(cfg: Config, app: A, start: Start<P>, now_ms: u64) -> (Self, Vec<Output<P>>);
    pub fn handle(&mut self, from: ValidatorIndex, msg: Message<P>, now_ms: u64) -> Vec<Output<P>>;
    pub fn tick(&mut self, now_ms: u64) -> Vec<Output<P>>;
    pub fn next_deadline(&self) -> u64;
    pub fn round(&self) -> Round;
    pub fn committed(&self) -> (Hash, u64);
    pub fn high_qc(&self) -> &QuorumCert;
    pub fn app(&self) -> &A;
    pub fn app_mut(&mut self) -> &mut A;
    pub fn into_app(self) -> A;
}

pub struct Config {
    pub me: ValidatorIndex,
    pub key: SecretKey,
    pub validators: ValidatorSet,
    pub chain: Hash,
    pub round_timeout_ms: u64,
    pub min_block_interval_ms: u64,
}

pub enum Output<P> {
    /// must be durable before acting on any later output
    PersistSafety(SafetyState),
    /// must be durable too; hand back in `Start::blocks` after a restart
    PersistBlock(Proposal<P>),
    Send(ValidatorIndex, Message<P>),
    Broadcast(Message<P>),
    Committed { id: Hash, height: u64 },
}

The driver's obligations:

• from must be the authenticated sender. A value at or above n is dropped. heard, the waiting budget, the serving budget and shard slots all trust it.
• Outputs must be carried out in order.
• Broadcast goes to everyone else. The engine has already delivered the message to itself.
• tick must run by next_deadline(), the earliest of the round deadline, proposal_due, normal_due and the first scheduled fetch.
• now_ms need not be monotonic. The engine keeps the maximum it has seen.

The application plugs in through this trait:

pub trait App<P> {
    fn propose(&mut self, parent: &Hash, height: u64, round: Round, now_ms: u64) -> P;
    fn verify(&mut self, id: &Hash, block: &Block<P>, now_ms: u64) -> bool;
    fn commit(&mut self, chain: &[Final<'_, P>], proof: &FinalityProof);
    fn has_work(&self, _parent: &Hash) -> bool {
        true
    }
}

pub struct Final<'a, P> {
    pub id: Hash,
    pub block: &'a Block<P>,
}

commit receives consecutive blocks in ascending order, and proof.tip is the id of the last one. The engine is generic over P, so the tests run it on small integer payloads.

ChainApp in qubits-node is the chain's implementation:

Persistence and restart

pub struct SafetyState {
    /// highest round with a normal or fallback vote
    pub voted_round: Round,
    /// latest optimistic vote: a normal vote in that round may only repeat it
    pub optimistic: Option<(Round, Hash)>,
    /// highest round timed out in, no voting there or below
    pub timeout_round: Round,
    /// a restart must not report a lower QC, or a TC could skip a final block
    pub high_qc: Option<QuorumCert>,
}

pub struct Start<P> {
    pub committed: Hash,
    pub committed_height: u64,
    pub committed_round: Round,
    pub high_qc: QuorumCert,
    pub safety: SafetyState,
    pub recent: Vec<Recent>,
    pub blocks: Vec<Proposal<P>>,
}

pub struct Recent {
    pub id: Hash,
    pub parent: Hash,
    pub height: u64,
    pub round: Round,
    pub proposer: ValidatorIndex,
    pub voters: Vec<ValidatorIndex>,
}

PersistSafety and PersistBlock must be durable before the driver acts on any later output. The engine emits each before the message that depends on it:

In the node, Validator::act runs on the consensus thread and carries out outputs in order. Each PersistSafety and PersistBlock is a single RocksDB write with set_sync(true), under consensus_safety and in the pending-blocks column family. Output::Committed deletes pending blocks up to that height.

Engine::new rebuilds the engine:

1. Load recent into the block metadata, so that leaders come out as they did.
2. Take the higher of start.high_qc and safety.high_qc.
3. Replay start.blocks through on_proposal with replaying set: no votes, no proposals, no PersistBlock, and everything counts as solicited.
4. Request high_qc.block if it is still unknown.
5. Enter round max(high_qc.round + 1, voted_round, optimistic round, timeout_round).

A restarted validator resumes in the round it last voted in without skipping one, so it can still repeat an optimistic vote as a normal vote there. It rejects anything else in that round (a_restart_remembers_what_it_voted_for).

Votes, timeouts, TCs and heard are not persisted. Losing TCs is what join_timeouts repairs.

Verification

SMT model

model/check.py asks Z3 whether the voting rules allow a fork. A run is a set of event times, the protocol is constraints on them, and unsat means no schedule produces two conflicting final blocks within R rounds and n validators. It is a separate statement of the rules, not an analysis of the Rust code.

The model favours the attacker:

• Every leader is Byzantine, so any block on any block exists.
• The f Byzantine validators vote for everything and report any lock.
• The network delivers certificates to whoever it likes at any time, or never.
• Honest validators may enter any round at any time.
• Only the voting rules protect the chain.

Two blocks per round are modelled, because a fork is two chains and any other vote only ties its voter's hands. "Final" means what a FinalityProof shows: a certified child in the very next round.

The script then switches off each of the 11 rules in turn, at min(R, 5) rounds, and requires a fork. That shows both that the rule is needed and that the model can see the attack. Output of check.py 4 8 with Z3 4.15.0:

n=4 rounds=3: no fork possible (0.2 s)
n=4 rounds=4: no fork possible (0.4 s)
n=4 rounds=5: no fork possible (4.3 s)
n=4 rounds=6: no fork possible (13.0 s)
n=4 rounds=7: no fork possible (44.5 s)
n=4 rounds=8: no fork possible (83.4 s)
without 'optimistic needs a clean past': fork found (0.9 s)
without 'optimistic needs the lock': fork found (0.8 s)
without 'one vote before optimistic': fork found (0.6 s)
without 'normal respects optimistic': fork found (0.6 s)
without 'one normal or fallback vote': fork found (0.5 s)
without 'fallback needs the rank': fork found (0.7 s)
without 'no vote after timing out': fork found (0.9 s)
without 'timeouts report the lock': fork found (0.6 s)
without 'votes need the parent's certificate': fork found (0.5 s)
without 'final needs consecutive rounds': fork found (1.2 s)
without 'certificates hold one kind': fork found (1.2 s)

check.py 7 5 gives the same answers: no fork for n = 7 at 3, 4 and 5 rounds, and a fork without each rule. The result is bounded to those sizes and says nothing about liveness.

Unit tests

engine.rs has one test per voting rule, because chaos rarely reaches the orderings that exploit a missing rule. shards.rs covers rebuilding from any k shards, tampered and misplaced shards, relay limits and memory bounds. tests/certificates.rs tampers with real certificates: missing, duplicated, reordered and misattributed votes, relabelled kinds, wrong block, round, chain and domain, and inflated high_qc rounds.

Simulation

tests/simulation.rs runs whole networks in simulated time with real ML-DSA signatures, a 1000 ms round timeout and a 50 ms block interval. Messages take 10 to 100 ms by default. Every test with more than one validator checks that honest validators committed the same blocks.

Restarts

tests/restart.rs persists outputs the way a node must and reboots engines from that record:

• a_restart_cannot_fork_a_committed_block commits a block on one validator only, cuts it off, restarts the other three, and requires them to reach that height with the same block.
• the_chain_survives_every_validator_restarting_at_once drops every message in flight and restarts all four.
• repeated_restarts_stay_safe_and_live restarts validators in turn 30 times.

Chaos

tests/chaos.rs throws random faults at whole networks. Each scenario runs 40 strikes, 50 to 1500 ms apart, drawn from: one crash, a power cut of every validator with all messages lost, rebooting whatever is down, a random partition into up to 3 groups, healing it, 0 to 30% loss, delays of up to 900 ms, and one restart. Any process can also die halfway through carrying out its outputs, with a chance of 0.4% per batch, which exercises the persistence order.

Every commit checks that heights are consecutive, that no honest validator committed a different block at that height, and that the FinalityProof verifies. After the last strike the network heals, and validators more than 100 blocks behind take the committed chain, as a node with --follow does. The chain must then advance within 300 s of simulated time. QUBITS_CHAOS_SEEDS=200 gives a long run. Chaos does not reach the orderings that exploit a missing voting rule, which is why the unit tests and the model exist.

chain/crates/qubits-e2e/e2e.py repeats the fault scenarios with real processes over TCP and TLS, including partitions through proxies and a validator that sends only rubbish.

Wide-area figures

These are simulated numbers, not measurements of a deployed network.

tests/wan.rs runs real engines and real signatures over a simulated network. Its output:

  n    payload     block    final    final    throughput     upload avg     upload max
                  period      p50      p99
  4        0 KiB      82 ms    210 ms    384 ms      0.00 MB/s       1.2 Mbit/s       1.3 Mbit/s
  4       64 KiB      82 ms    210 ms    384 ms      0.80 MB/s       6.0 Mbit/s       6.3 Mbit/s
  4     1024 KiB      91 ms    226 ms    387 ms     11.53 MB/s      69.0 Mbit/s      71.6 Mbit/s
  4     8192 KiB     227 ms    499 ms    567 ms     36.91 MB/s     336.1 Mbit/s     349.5 Mbit/s
 10        0 KiB      93 ms    233 ms    398 ms      0.00 MB/s       2.9 Mbit/s       3.0 Mbit/s
 10       64 KiB      93 ms    233 ms    398 ms      0.71 MB/s       8.1 Mbit/s       8.7 Mbit/s
 10     1024 KiB     122 ms    313 ms    411 ms      8.60 MB/s     139.9 Mbit/s     145.7 Mbit/s
 10     8192 KiB     294 ms    741 ms    879 ms     28.52 MB/s     457.7 Mbit/s     464.2 Mbit/s
 25        0 KiB     111 ms    288 ms    413 ms      0.00 MB/s       6.0 Mbit/s       6.1 Mbit/s
 25       64 KiB     111 ms    288 ms    413 ms      0.59 MB/s      10.4 Mbit/s      11.1 Mbit/s
 25     1024 KiB     135 ms    356 ms    443 ms      7.76 MB/s     163.4 Mbit/s     164.7 Mbit/s
 25     8192 KiB     357 ms    912 ms   1013 ms     23.49 MB/s     469.3 Mbit/s     483.8 Mbit/s
 50        0 KiB     152 ms    370 ms    468 ms      0.00 MB/s       8.9 Mbit/s       9.2 Mbit/s
 50       64 KiB     152 ms    370 ms    468 ms      0.43 MB/s      12.3 Mbit/s      14.4 Mbit/s
 50     1024 KiB     192 ms    503 ms    578 ms      5.45 MB/s     123.2 Mbit/s     129.0 Mbit/s
 50     8192 KiB     357 ms    960 ms   1038 ms     23.49 MB/s     504.7 Mbit/s     541.9 Mbit/s

Reproduce it with cargo test --release -p qubits-consensus --test wan -- --ignored --nocapture. Its assumptions:

• Validators sit in three regions in turn, by i % 3. One-way latency is assumed: Frankfurt to Virginia 45 ms, Frankfurt to Tokyo 112 ms, Virginia to Tokyo 75 ms, 1 ms within a region.
• Each validator has a 1 Gbit/s uplink, modelled as one sending queue. Receiving is unlimited.
• There is no CPU time. Signing, verifying, execution and storage cost nothing, and payload is a number added to the size of a proposal, not transactions.
• round_timeout_ms is 2000 and min_block_interval_ms is 0, so block periods here are not limited by the 100 ms default.
• Each run lasts 6 s and the first second is discarded. Engines are polled every 5 ms.
• block period is the 5 s measured divided by the blocks validator 0 committed, and throughput is those blocks times payload, over 5 s.
• Latency is from the moment a block is made until a validator commits it, over all validators.
• The model charges a leader n - 1 shards and a relayer n - 2, at size / k + 300 bytes each. A node sends more: the leader also broadcasts its own shard, and a relayer also sends to the proposer. upload max at large payloads is therefore lower here than a node would use.

Source

• chain/crates/qubits-consensus/src/engine.rs
• chain/crates/qubits-consensus/src/types.rs
• chain/crates/qubits-consensus/src/shards.rs
• chain/crates/qubits-consensus/model/check.py
• chain/crates/qubits-consensus/tests/simulation.rs
• chain/crates/qubits-consensus/tests/restart.rs
• chain/crates/qubits-consensus/tests/chaos.rs
• chain/crates/qubits-consensus/tests/certificates.rs
• chain/crates/qubits-consensus/tests/wan.rs
• chain/crates/qubits-node/src/app.rs
• chain/crates/qubits-node/src/validator.rs
• chain/crates/qubits-node/src/lib.rs
• chain/crates/qubits-state/src/store.rs