The Hidden Cost of Multi-Agent Systems in Production: Loop Locks, Token Bankruptcy, and Failure Isolation

At 02:14 a research agent receives an empty result, repeats semantically identical searches, and is sent back by a reviewer. Nothing crashes, yet 37 minutes later one run has consumed 6.8 million tokens. The costly multi-agent failure is valid-looking work without progress.

Frameworks express the graph; production safety defines progress, budgets, repetition, delegation, and blast radius.

1. Infinite loops are not merely graph cycles

Cycles are legitimate while an invariant improves. Changed prose is not progress when normalized tools, arguments, evidence, and decisions repeat. LangGraph recursion limits and CrewAI iteration, time, RPM, retry, and guardrail limits are hard caps, not a complete loop policy.

2. Deterministic state

A transcript is not workflow state. Store typed, versioned control fields. The model proposes; deterministic policy authorizes. Replaying recorded nondeterministic results must choose the same transition. Keep large tool payloads behind immutable references and bounded summaries.

3. Concurrent state writes

Parallel agents commit against expected state_version. Reject stale writers unless an explicit field reducer defines a deterministic merge. Last-write-wins can silently discard correct evidence.

4. Layered circuit breakers

Combine graph steps, repeated action signatures, no-progress streak, token and cost budgets, deadline, delegation depth, child count, and tenant budget. Reserve usage before a call and reconcile actual usage; timeouts are not necessarily free.

5. Progress digests

Project domain invariants rather than hashing timestamps and prose. Research uses unresolved questions, canonical evidence, validated claims, conflicts, and artifact digest. Interpret unchanged progress with attempt outcome and retry class.

6. The quiet 400-dollar token leak

At 1,000 runs/day, 1 percent loop rate, 22 extra calls, 14k input and 1.2k output tokens, illustrative prices of 3/15 dollars per million yield 396 dollars/month. Track reserved, actual, and wasted usage by tenant, workflow, model, node, and tool.

7. p99 latency cost

End-to-end p99 is not the sum of step p99 values. Trace replay or correlated Monte Carlo must include retries and joins. In the representative model, controls move p99 from 168s to 43s while p50 rises from 18.4s to 19.1s.

8. Failure isolation and effects

Bulkhead model quotas, browsers, databases, sandboxes, outbound domains, and tenants with separate queues and deadlines. Checkpoints do not make external effects exactly-once; persist an intent and idempotency key, execute, then reconcile.

9. LangGraph and CrewAI

For LangGraph, keep recursion limit, add budget/progress state, deterministic admission, idempotent effects, interrupts, and explicit reducers. For CrewAI, combine agent limits and task guardrails with typed Flow budgets, progress checks, and delegation bounds.

10. Operations and chaos

Alert on no-progress, repetitions, burn rate, node and tenant p99, fan-out, conflicts, checkpoint latency, and unknown effects. Inject timeout-after-success, crash-after-checkpoint, concurrent writes, reordered args, prolonged 429, and interrupt resume.

Conclusion: autonomy is bounded execution

A production multi-agent system is a distributed workflow supervising costly nondeterministic workers. Version state, measure progress, enforce budgets, make effects idempotent, and isolate failures.

Continue with multi-agent boundaries, event-driven workers, and GIL/p99 capacity.

11. Separate retry budgets from loop budgets

A retry re-executes one node after a classified transient failure; a loop is a policy transition back to the same or another node. Track invocation attempt_no, error_class, available_at, and result certainty separately from run transition_count and semantic_action_count. A 429 retry need not increase the no-progress streak, but it still consumes wall-clock and usage budget.

Backoff is not a sleeping worker. Persist the next eligible time, release the lease, and let capacity serve another tenant. Honour valid retry-after as a lower bound, use full jitter, and refuse another attempt when the job deadline cannot contain its p95 duration and commit margin. This prevents retry storms from occupying worker slots or being misdiagnosed as agent reasoning loops.

12. Migrating state schemas and policy versions

Long-lived human interrupts make in-place state changes dangerous. Persist schema_version, policy_version, prompt_version, and tool_contract_version in every checkpoint. On resume, a deterministic migrator transforms old state or routes the run to its original worker revision. Asking a model to guess missing migration fields is not a deterministic migration.

In blue-green delivery, new runs enter revision B while revision A drains its own open runs. Metrics remain revision-scoped. Do not change prompts and breaker thresholds in the same experiment. A safe rollback includes compatible checkpoint readers, tool contracts, and effect reconcilers—not merely the application image.

13. Human review must also be a bounded queue

Sending every opened breaker to a human creates another bankruptcy queue. A review item carries reason, risk, consumed budget, last progress projection, and explicit choices. Apply SLA, maximum wait, and queue quotas by tenant and risk. Low-risk expired work may fail in a controlled way; financial or irreversible work must never auto-approve.

A human decision is a versioned command. The reviewer submits the state_version they observed; if state changed, the stale approval is rejected. Approve, edit, reject, and retry are distinct events with identity, time, rationale, and policy in the audit log. Human-in-the-loop then becomes an accountable part of the execution protocol.