Docs · The doctrine · Section 07
State, concurrency, and scaling
Scaling an agentic system depends less on the model than on state management. The rule is simple: the agent is a reducer with no hidden state, and the state lives outside. But "state" is not monolithic: it reads as three layers that must be distinguished, so as not to confuse what must endure with what must be able to be forgotten.
1 The agent as a stateless reducer
An agent turn is treated as a function that takes an input state and an event, and produces an output state, with no hidden internal memory between calls. Each turn acts only on the explicit context it is given. This makes executions reproducible, debuggable, and allows the service to be replicated without surprises.
2 Explicit state, in three layers
All interactions are recorded as an append-only event log (event sourcing), attached to a business entity. The technique is old: it is the heir of double-entry bookkeeping, codified by Luca Pacioli in 1494, the first ledger where nothing is ever erased: a correcting entry is added. But this log is not the agent's working memory: it is its cold source. Three layers stand apart.
- The interaction log is immutable: source of truth, the means of auditing and replaying a trajectory from the recorded outputs. Nothing in it is ever erased.
- Working memory (Section 06) is a derived view, a projection (CQRS) of this log and of the extractions that follow from it. It is the layer that forgets, densifies, and contextualizes itself, freely, because the log always reconciles it.
- Prompt provenance (Section 08) connects the two: at each inference, the fingerprint of what was actually injected is kept, in order to replay exactly what the model saw, even after forgetting.
Confusing these layers, making the log "both the memory and the audit", forces impossible compromises: either the agent is weighed down with multiple contexts, or audit data is destroyed in trying to lighten the memory. Separating them dissolves the dilemma.
Replaying does not mean re-calling the model. A model call is not deterministic: you re-read the recorded output, you do not re-run the model, with the output recorded by provenance. The flow of the trajectory, on the other hand, is deterministic and replays faithfully. Three uses stand apart: audit replay reconstructs a trajectory from outputs already recorded; resumption restarts from a checkpoint without replaying everything; replication, which is precisely what the absence of hidden state allows, has another instance carry the same explicit state. Temporal consistency, knowing which version of prompt and model produced which output, is carried by provenance.
You do not erase the truth, you cool it. The immutable log grows, but its hot portion stays bounded: old segments compacted and archived cold, retained for audit. Physical deletion is the exception, reserved for low-value derived data under a retention rule, and for mandated erasure only, rendering a piece of data unreadable without breaking the audit chain.
3 From single-instance to multi-instance
In a single instance, in-memory structures suffice for rate limits, idempotency, sessions, cache, and progress. These are precisely the structures that block replication. Moving them into a shared store is the move that opens up multi-instance, without rewriting the domain. That same shared store, as soon as the single-process boundary is crossed, hosts three things at once: the externalized state, the capability discovery registry, and the budget allocation registry.
The domain does not change, but the semantics do, and this move is not free. A local, atomic access becomes a network call, with its latency and its partial failures. Strong consistency gives way to eventual consistency at the global level: the state converges without being instantaneously shared. Session consistency, however, often remains required, read-your-writes, following the order of causality, which is not the same thing as pure eventual consistency. An operation that spans several processes is no longer a single transaction; it becomes a sequence of local steps, each equipped with a compensation action that undoes its effect if a later step fails. Causal order is preserved explicitly, an effect never precedes its cause, through per-entity sequence numbers or logical clocks; and idempotency must hold under concurrency, not only on retry, through idempotency keys and a deduplication store. This is the entry price of distribution: it is paid only when an objective trigger demands it, and with full knowledge of the cost.
Diagram, externalizing state to replicate
Concurrency and budgets
Several sub-agents can work in parallel; each has its own iteration budget, capped, and a global cap per root task bounds the whole. As long as everything fits in one process, this cap is held by an aggregate in-memory counter, atomic and free because the event loop serializes accesses, a guarantee that falls away as soon as you distribute. As soon as the sub-agents spread across processes, the per-token central counter becomes a bottleneck: you move to lease-based block allocation. The orchestrator hands out batches; each sub-agent keeps a local counter, asks again as it nears exhaustion, and returns what is unused, strict control without permanently locking the store. The batch is time-stamped: a sub-agent that goes down sees it expire and be released, rather than tying up the budget. Concurrency is useful for independent subtasks; it must not mask sequential reasoning, which, for its part, benefits from staying in a single thread.