Skip to main content

Writing Simulation Tests

The Hydro simulator is a deterministic testing environment that explores the space of possible distributed executions. Unlike traditional unit tests that run a single execution path, the simulator systematically varies non-deterministic choices—batch boundaries, message ordering, and state snapshots—to find bugs that only manifest under specific interleavings.

Writing Tests​

A simulation test has three parts: setup, execution, and assertions. The setup creates a FlowBuilder and the distributed locations (processes and clusters). You then wire up your Hydro code and create simulation inputs and outputs using sim_input() and sim_output(). Here's a minimal example:

single_client_counter.rs
#[test]
fn test_counter_read_after_write() {
    let mut flow = FlowBuilder::new();
    let process = flow.process();

    let (inc_in_port, inc_requests) = process.sim_input();
    let (get_in_port, get_requests) = process.sim_input();

    let (inc_acks, get_responses) = single_client_counter_service(inc_requests, get_requests);

    let inc_out_port = inc_acks.sim_output();
    let get_out_port = get_responses.sim_output();

    flow.sim().exhaustive(async || {
        inc_in_port.send(());
        inc_out_port.assert_yields([()]).await;
        get_in_port.send(());
        get_out_port.assert_yields_only([1]).await;
    });
}

The execution happens inside the async closure passed to exhaustive(). This closure is called repeatedly—once for each distinct execution the simulator explores. Inside, you send messages using the input ports and make assertions about the outputs.

For ordered streams, use send() to send a single message or send_many() to send multiple messages in sequence. For unordered streams, use send_many_unordered() to send multiple messages that can be processed in any order. On the output side, you can check what messages appear using assertion methods like assert_yields() for ordered streams or assert_yields_unordered() for unordered streams. The _only variants additionally check that the stream ends after the expected messages.

The exhaustive() method explores all possible executions. This is feasible when the inputs are finite and the number of nondet! decision points is manageable. For complex protocols, use fuzz() instead (see Coverage-Guided Simulation for details).

Receiving Outputs and Quiescence​

Instead of asserting, you can also receive output messages directly. next() waits for and returns the next message (failing the test if no more messages can arrive), and collect_n(n) returns the next n messages. These are always safe to use in the middle of a test: waiting for a message only runs the simulation work needed to produce it.

Observing the absence of messages is more delicate, because the simulator must run all pending work (such as ticks with buffered inputs) to prove that no more messages can arrive—potentially running work that a later assertion expected to observe in an unfinished state. The simulator handles this depending on the API and mode:

  • The assertion methods (assert_no_more(), the assert_yields_only* variants, and collect_n_only(n)) first let the simulation settle: if it reaches quiescence with only deterministic work, the check is free and the test continues. Otherwise, under exhaustive() the search forks: one instance performs the check and ends there (explored first, so failures are attributed to the right assertion), while sibling instances skip the check and continue.
  • try_next() (returns Option) and collect() drain the simulation to quiescence when needed, in every mode. Because this may overrun pending work, they should be the last observations of a test: after forcing quiescence this way, sending more input and then attempting to receive output will panic.
  • Multi-phase tests that intentionally let the system settle between rounds of input (e.g. modeling timers that fire long after messages propagate) can insert an explicit hydro_lang::sim::quiesce().await phase barrier. This forces the simulation to quiesce and declares that later observations are meant to see the settled state, so the test may continue sending and receiving afterwards — at the cost of never exploring executions where the phases interleave (pair with a barrier-free test if those matter).

Restricting Explored Executions with continue_if!​

Sometimes an assertion only makes sense for a subset of executions—for example, when the property you want to check only holds if certain messages happened to arrive in the same batch. The hydro_lang::sim::continue_if! macro lets you express such preconditions: if the condition is false, the current simulation instance is stopped and discarded (it is not a test failure), and exploration moves on to the next execution. This is the same concept as assume in verification tools and property-based testing libraries (e.g. kani::assume or proptest's prop_assume!).

flow.sim().exhaustive(async || {
    in_send.send_many([1, 2]);
    let batch_sizes: Vec<usize> = out_recv.collect().await;
    // only check executions where both messages landed in the first batch
    continue_if!(batch_sizes.first() == Some(&2), "first batch was {:?}", batch_sizes);
    // ... assertions that rely on the assumption ...
});

Like assert!, continue_if! accepts an optional message with format arguments. Discarded instances are never recorded as fuzzing reproducers, and when logging is enabled (HYDRO_SIM_LOG=1 or during replays), the failed assumption is logged along with its location. Take care not to make the condition too restrictive—if most executions are discarded, the simulator wastes time exploring executions that are never checked.

When a test fails, the simulator prints a trace showing the decisions it made:

Running Tick
  | let request_batch = use::batch(get_requests, nondet!(/** we never observe batch boundaries */));
  |                         ^ releasing no items
  | let count_snapshot = use::atomic(current_count, nondet!(/** intentional, based on when the request came in */));
  |                          ^ releasing snapshot: 0

Running Tick
  | let request_batch = use::batch(get_requests, nondet!(/** we never observe batch boundaries */));
  |                         ^ releasing items: [()]
  | let count_snapshot = use::atomic(current_count, nondet!(/** intentional, based on when the request came in */));
  |                          ^ releasing unchanged snapshot: 0

thread ... panicked at src/single_client_counter.rs:50:
Stream yielded unexpected message: 0

The trace shows which tick is running (and which cluster member, if applicable), each nondet! decision point with the choice made, and what items were released or which snapshot was observed. The "releasing unchanged snapshot" message indicates the simulator chose to re-release a previous snapshot rather than advancing to a newer one—simulating the case where state updates lag behind.

Effective simulation testing relies on careful scoping and documentation:

  • Start simple. Test individual components before testing the full system. A test that sends one message and checks one response is easier to debug than one with complex interactions.
  • Document your nondet! markers. The explanation in each nondet! call appears in failure traces. Good explanations help you understand why a particular decision point exists and whether the non-determinism is acceptable.

Deterministic Exploration​

The simulator's power comes from systematically varying non-deterministic choices. Every nondet! marker in your code represents a decision point where the simulator makes different choices across executions.

Code without any nondet! markers has exactly one possible execution. If your test sends a fixed sequence of inputs and your code has no nondet! markers, the test completes in a single execution. This means you can write fast unit tests for deterministic transformations while reserving exhaustive testing for code that genuinely non-deterministic choices.

When you call sliced! with a use statement, the simulator makes a decision for each sliced input. Consider:

let get_response = sliced! {
    let request_batch = use::batch(get_requests, nondet!(/** ... */));
    // ...
};

If three requests arrive and get_requests is an ordered stream, the simulator might:

  • Release none initially, then all three later
  • Release all three in one batch
  • Release two, then one
  • ... and so on

Similarly, when snapshotting a Singleton, the simulator decides which version to observe:

let count_snapshot = use::snapshot(current_count, nondet!(/** ... */));

If the count has progressed through values 0 → 1 → 2, the simulator might snapshot any of these values. It can also re-release the same snapshot multiple times, simulating the case where state updates haven't propagated yet.

For unordered streams, the simulator has additional freedom. When batching elements from an unordered stream, it selects which elements to include but doesn't need to simulate all possible orderings within the batch. Since the batch is also unordered, testing different internal orderings would be redundant. Such optimizations are key to making exhaustive testing tractable.