Validator networking

Validators talk over direct TCP connections, one per pair, encrypted with TLS 1.3 using only the hybrid X25519MLKEM768 key exchange and authenticated by an ML-DSA signature over the TLS exporter. There is no discovery and no libp2p, because a fixed validator set needs neither. This page specifies the transport for node operators and auditors: connection setup, the handshake byte by byte, frames, limits, and transaction gossip. What travels over it is on Consensus, and followers use HTTP as described on Sync.

Path of a message

Engine output
  |  Validator::act: encode a NetMessage, send directly or as shards
  v
Transport::send or broadcast
  |  Outgoing::push: per-peer queue, 4096 frames, 256 MiB
  v
run_connection: write_frame, TLS, TCP
  |
  v
read_frame on the peer: at most 16 MiB, paused above 2 MiB unhandled
  |
  v
Inbox, then Validator::handle on the consensus thread

Transport

pub trait Transport: Send + Sync + 'static {
    fn send(&self, to: &PublicKey, bytes: Vec<u8>);
    fn broadcast(&self, bytes: Vec<u8>);
    fn connected(&self) -> usize {
        0
    }
}

pub type Envelope = (PublicKey, Frame);
pub type Inbox = Receiver<Envelope>;

Peers are named by public key and not by ValidatorIndex, because indices change between epochs. Both calls return nothing and never block. Delivery is best effort, and consensus is built to tolerate loss.

A Frame holds the received bytes and dereferences to [u8]. Dropping it releases its bytes from the sender's unhandled count.

Configuration

KEY is read in this order:

1. If it parses as a decimal number, it is a position in the genesis validators. This is always the genesis list, not the current set.
2. Otherwise it is the hex public key that qubits key show prints, with its scheme byte.

HOST must be an IP address. The value is parsed as a SocketAddr, so a host name fails with HOST:PORT must be an IP address and port.

The --peer list is the whole access list. It is fixed when the process starts, it may contain this node's own key, and it should contain validators expected to join later. A key outside the list cannot complete a handshake and has no send queue. At startup the node exits with N current validators have no --peer address if any current validator is missing. See Validators and epochs.

A node with no validator key opens no validator connections.

p2p::connect builds an Identity from the genesis hash, the validator key and the keys in --peer, binds the listener, and returns the transport, the inbox and the bound address.

Connections

There is one TCP connection per pair. The lower key dials the higher one, and an acceptor refuses a dialer whose key is higher than its own. PublicKey orders by scheme and then by raw bytes. The rule means two validators never race to open two connections.

For each peer the node runs one task that owns that peer's send queue:

• If this node dials, it loops over dial. After a failure it waits, starting at 100 ms, doubling up to 5 s, and the wait resets once connected.
• If this node accepts, it waits for the accept loop to hand it an authenticated stream, through a channel of 4.
• While connected, run_connection writes queued frames and a separate task reads. It ends when a write fails, when the read task ends, or when a newer accepted stream arrives. In the last case the task switches to the new stream and drops the old one.

The send queue belongs to the task, not to the connection, so frames queued while a peer is away go out when it returns. Both sides set TCP_NODELAY. connected() counts live connections and is what qubits_peers and the status line report.

TLS

The crypto provider lists only that key-exchange group, so a peer that does not offer it fails the TLS handshake in either direction. The tests a_client_without_the_hybrid_key_exchange_is_refused and a_server_without_the_hybrid_key_exchange_is_refused cover both.

Certificates are throwaway. rcgen generates one with its default key type, ECDSA P-256, and no trust rests on it: ExporterAuthenticated, the client's verifier, accepts any certificate. It still checks the TLS 1.3 CertificateVerify signature against the presented certificate, and it refuses TLS 1.2 signatures. Identity comes only from the hello that follows.

Handshake

Both sides run these structures after the TLS handshake:

struct Hello {
    key: PublicKey,
    signature: Signature,
}

struct HelloBytes<'a> {
    tag: &'static [u8],       // b"qubits p2p hello v1"
    chain: &'a Hash,          // the genesis hash
    dialer: bool,
    key: &'a PublicKey,
    exporter: &'a [u8; 32],
}

The whole exchange runs under HANDSHAKE_TIMEOUT (10 s):

1. The dialer connects over TCP and completes TLS.
2. Both sides derive a 32-byte exporter with export_keying_material, label EXPORTER-qubits-p2p-v1, no context.
3. The dialer signs HelloBytes { dialer: true, .. } with its validator key under Domain::Handshake, FIPS 204 context qubits/handshake/v1, and sends a Hello frame.
4. The acceptor reads one frame of at most MAX_HELLO_BYTES (8 KiB) and decodes it.
5. The acceptor checks that hello.key is in its --peer list. Only then does it spend time on the signature.
6. The acceptor verifies the signature with dialer: true, its own view of chain, and its own exporter.
7. The acceptor refuses the connection if the peer's key is higher than its own.
8. The acceptor sends its own Hello with dialer: false.
9. The dialer reads it, verifies it the same way with dialer: false, and requires the key to be the peer it meant to reach.
10. The stream goes to that peer's task.

What each signed field stops:

Byte layout of the signed message with an ML-DSA-65 key, 2045 bytes in all. Borsh integers are little-endian.

A Hello is 5271 bytes: 1957 for the key and 3314 for the signature, with Vec<u8> length prefixes. It goes in one frame, 5275 bytes with the frame prefix.

The acceptor logs these at debug level as refused connection, the dialer as dial failed.

Frames

Everything after TLS is framed. This length prefix is big-endian, unlike borsh integers.

offset  size     field
0       4        length, u32 big-endian
4       length   payload

The payload is a Hello during the handshake and a NetMessage afterwards. write_frame flushes after every frame. read_frame compares the length with its limit before allocating. A frame over the limit ends the read loop, and with it the connection. A frame within the limit that does not decode is dropped with a debug log, and the connection stays.

For scale, MAX_BLOCK_BYTES is 8 MiB and MAX_TX_BYTES is 288 KiB.

Messages

pub enum NetMessage {
    Consensus { epoch: u64, message: Message<BlockPayload> },
    Transaction(SignedTransaction),
    Shard { epoch: u64, shard: Shard },
}

The envelope adds 9 bytes to a consensus message. A vote is 3,367 bytes in its envelope, and a frame of it 3,371 bytes. Routing by epoch is on Validators and epochs.

Limits on what a peer can cost

Before authentication

The listener accepts anyone, so a connection must take two slots before any work is done on it. If either is unavailable, the socket is closed at once, before TLS.

Slots are released as soon as the handshake ends, either way, so established connections hold none. A signature is verified only for a key on the --peer list. An attacker that keeps all 64 slots filled from many addresses delays a validator's reconnection for as long as it does so. It does not touch established connections.

Reading

An authenticated peer is not read from while more than MAX_UNHANDLED_BYTES (2 MiB) of what it sent waits to be handled. The read task adds each frame's size to a counter for that connection, passes the frame to the inbox, and polls every 5 ms until the counter is back at or below the limit. Dropping a Frame after Validator::handle subtracts its size.

The limit is checked after a frame is read, so a peer can hold up to 2 MiB plus one frame, 18 MiB at most. TCP then pushes back on the sender. The inbox channel itself is unbounded, and this count is what bounds it, one sender at a time.

Sending

Each peer has a queue limited in frames and in bytes. Outgoing::push drops a frame when either limit is reached. Consensus never waits for a slow peer, and what is dropped is recovered through timeouts, Sync and block requests.

There is one queue per peer, first in first out, with no priority between kinds of message.

Gossip

Transactions spread in one hop, plus retries:

1. A node that accepts a transaction through POST /v1/transactions broadcasts it to all peers as NetMessage::Transaction. Validator::start registers the transport for this.
2. A receiver pushes it into Intake, a queue of INTAKE_QUEUE (1024) read by its own thread, tx-intake. Signature checks are too slow for the consensus thread. When the queue is full the transaction is dropped.
3. The thread calls Node::accept, which adds it to the mempool or refuses it. Accepting does not broadcast again.
4. Every regossip_ms (3 s), Validator::tick expires old transactions and rebroadcasts up to 64 of the oldest that have waited at least 3 s.

A transaction also expires at its valid_until_ms. Gossip carries no epoch and is accepted from any authenticated peer. A duplicate ends in SubmitError::Duplicate.

API nodes are not on this network. They pass submitted transactions to their --follow sources over HTTP, and repeat that every 3 s for up to 64 waiting transactions. See Sync.

Large proposals also travel as NetMessage::Shard, described on Consensus.

Observing it

Tests

chain/crates/qubits-e2e/e2e.py runs real processes through TCP proxies that add delay and cut links. Its byzantine phase uses a validator that passes the handshake with its real key and then sends only rubbish.

Source

• chain/crates/qubits-node/src/p2p.rs
• chain/crates/qubits-node/src/network.rs
• chain/crates/qubits-node/src/validator.rs
• chain/crates/qubits-node/src/lib.rs
• chain/crates/qubits-node/src/mempool.rs
• chain/crates/qubits-node/src/main.rs
• chain/crates/qubits-node/src/metrics.rs
• chain/crates/qubits-crypto/src/scheme.rs
• chain/crates/qubits-node/tests/p2p.rs
• chain/crates/qubits-node/tests/robustness.rs