A Rust framework for correct and performant distributed systems

Distributed systems deserve a native framework

Nearly every application today is a distributed system: services call other services, replicas coordinate state, and data flows across regions. Distribution is how modern software scales, survives failures, and stays close to its users.

Hydro is a Rust framework that treats distribution as a first-class concern. Instead of assembling a system from parts and relying on manual review to catch mistakes at their boundaries, you express, check, test, and deploy the whole distributed system as one program.

Today's frameworks make networks implicit

Most frameworks split a distributed system into single-machine programs that communicate through opaque RPC calls. The network—the part that makes your system distributed—is hidden inside client stubs and await points, invisible to the compiler and your tools.

Reordering, duplication, and partial failure all live in the gap between these files. Because the language cannot see the network, it cannot help you reason about what happens across machines.

node1/src/main.rs

let mut client = EchoClient::connect("http://node2:5000").await?;

// retries? reordering? failures? not visible here.
let reply = client.echo(Msg { text: "hello".into() }).await?;

node2/src/service.rs

impl Echo for EchoService {
    async fn echo(&self, req: Request<Msg>)
        -> Result<Response<Msg>, Status> {
        // where did this request come from? in what order?
        let text = req.into_inner().text;
        Ok(Response::new(Msg { text: text.to_uppercase() }))
    }
}

Hydro is global

Hydro is the first production framework with location-oriented programming: a single function can encapsulate logic spanning several machines. Distributed locations are captured in types, and sending data across the network is an explicit, type-checked operation.

These abstractions are zero-cost: Hydro compiles to the same networked binaries you would write by hand, and you retain full control over the network protocol, compute placement, and serialization format.

src/two_pc.rs

pub fn prepare_round<'a>(
    txns: Stream<Txn, Process<'a, Leader>>,
    leader: &Process<'a, Leader>,
    parts: &Cluster<'a, Participant>,
) -> KeyedStream<MemberId<Participant>, Vote, Process<'a, Leader>> {
    txns
        .broadcast(parts, TCP.fail_stop().bincode())
        // ⇒ Stream<Txn, Cluster<Participant>>, on every participant
        .map(q!(|txn| wal.prepare(txn)))
        .send(leader, TCP.fail_stop().bincode())
        // ⇒ every participant's vote, back on the leader
}

Hydro catches distributed bugs at compile time

Hydro encodes distributed behavior in the type system, end-to-end: every stream carries types that track ordering guarantees and retries, derived from your distributed logic.

Hydro uses these types to enforce eventual determinism: code whose result could be affected by timing or duplication does not compile. You discharge the obligation with an algebraic proof — idempotence, commutativity — that the compiler holds you to, or you scope the non-determinism explicitly with nondet!. Whole classes of distributed systems bugs become unrepresentable.

pub fn count_yes<'a>(
    votes: Stream<Vote, Cluster<'a, Participant>>,
    leader: &Process<'a, Leader>,
) -> Singleton<usize, Process<'a, Leader>> {
    votes
        // re-sent on reconnect, so no vote is ever lost…
        .send(leader, TCP.retry_on_fail().bincode())
        // ⇒ Stream<Vote, …, NoOrder, AtLeastOnce>
        .fold(q!(|| 0), q!(|n, v| if v == Vote::Yes { *n += 1 }))
error: `fold` requires `ExactlyOnce` delivery, but this stream is `AtLeastOnce`   = help: prove the closure is idempotent with `idempotent = manual_proof!(...)`, or acknowledge the non-determinism with `assume_retries(nondet!(…))`
}

Hydro lets you write distributed tests

The compiler and the simulator split the work: the type system guides you to eliminate sources of non-determinism, and the simulator exhaustively explores the nondet! points you kept — the only places non-determinism can live.

Because the simulator only focuses on nondet! points, it is incredibly efficient: entire distributed protocols can be exhaustively checked on your laptop. That turns assertions into guarantees about every possible execution. And when a test fails, you get the exact schedule that broke it, replayable deterministically instead of flaking.

let (vote_port, votes) = participants.sim_input();

let quorum = votes
    .send(&leader, TCP.fail_stop().bincode())
    .entries_partially_ordered(
        nondet!(/** votes from members interleave: NYY? YNY? YYN? */))
    .map(q!(|(_participant, vote)| vote))
    .limit(q!(2)) // BUG: 2 of 3 is a majority, not unanimity
    .sim_output();

flow.sim().with_cluster_size(&participants, 3).exhaustive(async || {
    vote_port.send(0, Vote::No); // this participant must veto
    vote_port.send(1, Vote::Yes);
    vote_port.send(2, Vote::Yes);

    let seen: Vec<_> = quorum.collect().await;
    assert!(seen.contains(&Vote::No)); // the veto must be heard
});

Research Backed. Production Ready.

Hydro has its roots in foundational distributed systems research at UC Berkeley, such as the CALM theorem. It is now co-led by a team at Berkeley and AWS, with contributions from the open-source community.

Hydro continues to lead the way with cutting-edge capabilities, such as automatically optimizing distributed protocols, while supporting production use with cloud integrations and observability tooling.

Get Started with Hydro