Durable AI Agent Pipeline
A pluggable, streaming gateway for AI agents built entirely on managed services — user input survives any failure, turns dispatch exactly once, tokens stream live, and onboarding a new agent is a registry entry, not a deployment.
By Jai Ganesh
What is a durable AI agent pipeline?
A durable AI agent pipeline is the infrastructure that keeps agent work safe from failure: idempotent task dispatch, persisted turn state, and retry semantics wrapped around long-running inference calls. When a 30-second model call fails mid-stream, a transient error costs a retry — not the user's work. Built entirely on managed services (Cloud Tasks, Firestore, Redis Pub/Sub), it also makes agents pluggable: onboarding a new agent is a registry entry, not a redeployment.
The Problem
One chat interface, many AI agents — each owned by a different team, with its own model, prompt, and deployment lifecycle. The obvious monolith that imports all agent logic collapses at the organizational boundary: Team A cannot deploy without redeploying Team B, and a one-line prompt fix waits on a full release cycle.
The runtime problem is just as sharp: inference calls take 30+ seconds and occasionally fail. A plain request-response design loses the user's message when anything breaks mid-call, and users watching a spinner for half a minute assume it already has.
This pattern answers both with a durable, registry-driven pipeline: the gateway routes to agents it knows nothing about, every message is persisted before acknowledgment, dispatch is idempotent, and tokens stream to the browser as they are generated — all on managed services, with no message brokers or Kubernetes to operate.
Durability is also the quiet difference between an agent demo and an agent product. A demo can lose a message and nobody notices; a production system that loses user input even once loses something harder to rebuild — the user’s willingness to hand it real work. This pipeline treats “never lose input, always converge to a truthful state” as the platform contract everything else is built on.
The Architecture
How It Works
The gateway — routing, persistence, streaming; zero inference
A FastAPI service on Cloud Run. On each message it writes the user message and a turn document (status: processing) to Firestore, looks up the target agent in the registry, enqueues a Cloud Task, and returns 202 Accepted. From that moment the user's input is durable — whatever fails downstream, nothing is lost.
The agent registry
A Firestore collection with TTL-cached entries: worker URL, dispatch deadline, service account. Onboarding an agent means writing one registry document and granting one IAM role — no gateway code change, no redeployment, no cross-team coordination. Pluggability is an operational property, not a code property.
Idempotent dispatch via Cloud Tasks
The turn ID is the task name, so duplicate submissions collapse into one dispatch — exactly-once turn processing guaranteed by the queue's own semantics. Retries with backoff come free, and each task carries an OIDC token scoped to its target worker.
Per-team agent workers
Each agent is its own Cloud Run service owning its model, prompt, and release cadence. Workers publish tokens as they are generated and write the final answer and turn status back to Firestore.
The turn state machine
Every turn is a small Firestore document that moves through explicit states: processing when the gateway persists and enqueues, streaming when the worker begins publishing tokens, then complete with the final answer — or failed with a reason. The document is the single source of truth: the stream is a live projection of it, never the other way around. This is what makes recovery boring, in the best sense — a client that reconnects, a worker that retries, or a dashboard that audits all read the same state and reach the same conclusion. There is no “what actually happened?” to reconstruct from logs, because the turn document is what happened.
Live streaming via Redis Pub/Sub
Workers publish tokens to per-turn channels; the gateway subscribes and relays them to the browser over SSE. The user watches the answer being written within moments of sending — durability without the dead-air spinner.
The streaming handshake
The subtlest seam in the system: Redis Pub/Sub delivers only to subscribers present at publish time, and a fast worker can start emitting tokens before the gateway’s subscription lands — silently dropping the first words of the answer. The fix is ordering and buffering: the gateway subscribes to the per-turn channel before enqueuing the dispatch task, and workers buffer early tokens until the turn status confirms a listener. Getting this wrong doesn’t throw an error — it just truncates answers nondeterministically under load, which is why it nearly shipped broken. Distributed streaming bugs rarely announce themselves; they have to be designed out.
All managed services
Cloud Tasks, Firestore, Cloud Run, and Redis — no self-hosted brokers, no Kubernetes, no custom retry infrastructure. The durability guarantees come from service semantics, not from operational heroics.
Design Decisions
Persist before acknowledging — 202, not 200
Returning 202 only after the message and turn are written to Firestore makes durability the contract: a crash after acknowledgment costs a retry, never the user's input. The single most important guarantee in the system.
Turn ID as Cloud Task name
Idempotency by construction: the queue physically cannot dispatch the same turn twice. Cheaper and more reliable than any dedupe logic you would write yourself.
Registry-driven routing over compiled-in agents
The gateway routing to agents it has never heard of is what decouples team release cycles. Adding an agent is an operational task — write a document, grant a role, done. No PRs, no deployments, no coordination.
Separate the durability path from the streaming path
Firestore holds the truth; Redis carries the ephemeral token stream. If streaming hiccups, the completed turn still lands from state — the user experience degrades gracefully instead of the data being wrong.
OIDC service-to-service auth per worker
Each task carries a token scoped to its specific target worker, so agents cannot invoke each other laterally and a compromised worker does not inherit the platform's reach.
Managed services over self-hosted infrastructure
Every guarantee this pipeline needs — at-least-once delivery, retry with backoff, serverless scaling — is a native property of the managed stack. The team operates agents, not middleware.
Subscribe before dispatch, buffer at the worker
The token-drop race cannot be fixed with retries because nothing fails — delivery to zero subscribers is a success in Pub/Sub semantics. The only reliable fix is structural: guarantee the listener exists before the speaker can start, and make the worker tolerant of a late subscriber anyway. Belt and suspenders, because the failure mode is invisible in testing and constant at scale.
Trade-offs & Limits
- ›Race conditions live in the seams: a worker can start publishing tokens before the gateway subscribes to the channel. The streaming handshake needs deliberate design — this nearly broke the original build.
- ›Redis Pub/Sub is fire-and-forget: a briefly disconnected client misses tokens and must recover the full answer from Firestore state. Fine for chat; use Redis Streams if replay matters.
- ›Cloud vendor coupling: the pattern maps cleanly to AWS or Azure equivalents, but the idempotency and auth mechanics are service-specific and must be re-verified, not assumed.
- ›A registry of independently deployed workers needs governance — health checks, dispatch deadlines, and ownership metadata — or it accretes abandoned agents.
When To Use It
- ✓Multiple teams shipping agents behind one interface, each needing independent deploys
- ✓Long-running inference (30+ seconds) where losing user input on failure is unacceptable
- ✓Chat experiences that need live token streaming with durability behind it
- ✓Teams that want platform guarantees without operating brokers or Kubernetes
- ✓Regulated or audited contexts where every user message and agent response must be reconstructable from persisted state, not from logs
When Not To
- ✕A single agent owned by a single team — a monolith with SSE is simpler and fine
- ✕Sub-second, stateless completions where durability adds latency without value
- ✕Workflows needing multi-step orchestration across agents — this is a dispatch pipeline, not an orchestrator; pair it with the multi-agent pattern instead
Enterprise architect and independent AI consultant — I help teams take agentic systems from deck to production, with the governance story intact.