What We Learned Building a Durable AI Agent Pipeline on GCP

The Problem

We had a straightforward-sounding goal: let users chat with specialized AI agents through a single web interface. Each agent is owned by a different team. Each has its own model, prompt, and deployment lifecycle.

The obvious first instinct — one monolith that imports all agent logic — falls apart at the organizational boundary. Team A can’t deploy their agent without redeploying Team B’s. A hotfix to one agent’s prompt requires a full release cycle. And when an inference call takes 30+ seconds and occasionally fails, you need durability guarantees that a simple request-response architecture can’t provide.

We needed a gateway that routes to agents it knows nothing about, survives transient failures without losing user input, and streams tokens back to the browser in real time. We decided to build a proof-of-concept using only GCP managed services — no self-hosted message brokers, no Kubernetes, no custom retry infrastructure.

The Hypothesis

One sentence: a pluggable, durable AI agent pipeline can be built entirely on Cloud Tasks, Firestore, and Redis Pub/Sub, with live token streaming to a React frontend — and adding a new agent should require zero gateway redeployment.

“Pluggable” had a specific definition: onboarding an agent means writing one Firestore document (the registry entry) and granting one IAM role. That’s it.

What We Built

The architecture splits into two services with a clean contract between them.

The Gateway

A FastAPI service on Cloud Run. It owns routing, persistence, and streaming — but has zero inference logic. When a user sends a message:

1. 1.The gateway writes the user message and a turn document (status: processing) to Firestore.
2. 2.It looks up the target agent in a registry — a Firestore collection with TTL-cached entries containing the worker URL, dispatch deadline, and service account.
3. 3.It creates a Cloud Task with the turn ID as the task name (guaranteeing idempotency) and an OIDC token scoped to the target worker.
4. 4.It returns 202 Accepted. The user’s message is now durable.

For streaming, the gateway exposes an SSE endpoint. The browser connects to /stream/{turn_id} and receives token chunks as they arrive via Redis Pub/Sub. Redis is ephemeral and fire-and-forget — if the SSE connection drops, no data is lost because Firestore holds the completed message. The browser simply fetches the final state on reconnect.

The Worker

A separate Cloud Run service, locked behind IAM (not publicly accessible). It implements a simple contract we called Agent Contract v1:

1. 1.Accept POST /process with a turn payload
2. 2.Check idempotency (skip if the turn is already terminal)
3. 3.Stream inference via the LLM of your choice, publishing chunks to Redis
4. 4.On completion: persist the assistant message to Firestore, update the turn status, then publish the terminal event — in that order

The ordering constraint matters. If you publish done before persisting the message, any client that polls Firestore on receiving done will find… nothing. We learned this the hard way.

The Registry

The agent registry is the mechanism that makes the system pluggable. Each entry contains:

worker_url:          https://my-agent-xyz.run.app/process
dispatch_deadline_s: 900
invoker_sa:          agent-invoker@project.iam.gserviceaccount.com
audience:            https://my-agent-xyz.run.app
enabled:             true

The gateway caches this with a 60-second TTL and serves stale on error. Adding a new agent means: deploy your worker, write a registry doc, grant the invoker service account roles/run.invoker on your service. The gateway discovers it automatically. No config files, no deployments, no PRs to the gateway repo.

The Hardest Bugs Were Race Conditions

Every interesting bug we found during iteration was a race condition. Not one was caught by unit tests. Here are the four that taught us the most.

1. The Subscribe-After-Check Race

Our first SSE implementation checked Firestore for the turn status, then subscribed to Redis Pub/Sub. Seems logical — why subscribe if the turn is already done?

The problem: if the worker publishes the done event between the Firestore check and the Redis subscribe, the event is lost forever. The SSE connection hangs until timeout.

The fix was a subscribe-first pattern: subscribe to the Redis channel before checking Firestore. If the turn is already complete when you check, you unsubscribe and return the final state. If not, you’re already listening and can’t miss the event. The window is closed.

2. The Status Regression

Our gateway originally created the Cloud Task first, then wrote status: processing to Firestore. On a fast worker (or a retry), the worker could complete and write status: completed before the gateway wrote processing — regressing the turn back to an in-progress state.

The fix was trivial once diagnosed: write the status before enqueueing the task. But it took three days of “why does this turn say processing when the message is already there?” to find it.

3. The Optimistic UI Merge

The frontend renders user messages immediately (optimistic UI) and adds streaming assistant text as a synthetic message. When the stream completes, a background fetch retrieves the authoritative messages from the server and reconciles them with the optimistic state.

Our first reconciliation used content matching: if a fetched message has the same role and content as a synthetic one, discard the synthetic. This worked perfectly — until it didn’t. Streaming text occasionally differs from stored text by whitespace or a trailing token. The content match fails silently, and the user sees duplicate messages.

The fix: match synthetic messages by turn_id in addition to content. A synthetic message tagged with the same turn ID as a fetched message is always a duplicate, regardless of minor content differences.

4. The Queue Deadlock

The frontend implements a message queue: if the user sends messages while a turn is in progress, they’re queued and drained sequentially. The drain logic triggers on turn completion.

We had a fetchMessages() call in the drain path. If that call failed (network error, server hiccup), the Promise rejection went unhandled. The draining flag stayed true forever. Every subsequent message the user sent went into the queue and was never sent. The UI appeared functional — the input worked, the queued messages badge updated — but nothing ever reached the server.

The fix was a single .catch() that clears the draining flag. Five characters of code, three hours of debugging, and a valuable reminder that every Promise in a state machine needs an error path.

The Subtraction Finding

The most impactful architectural recommendation from this exercise had nothing to do with new code.

While assessing the existing production system that this POC would eventually feed into, we discovered it was running three separate LLM conversations per user message: one for the main agent turn, one to evaluate what the agent “learned,” and one to decide whether to notify other agents.

Two of these conversations were redundant. The main conversation already declared tool calls that captured the same information the evaluation conversations were re-deriving. The system was paying for three inference calls — using a large, expensive model — to do the work of one.

The highest-value recommendation was subtraction: collapse to a single conversation, cut inference cost by roughly two-thirds, and reduce per-turn latency from ~45 seconds to ~15. No new services, no new infrastructure. Just removing code.

Sometimes the best architecture decision is figuring out what to delete.

Frontend Patterns Worth Noting

A few patterns from the React frontend that proved more important than expected:

Batched SSE rendering. Naively calling setState on every SSE token chunk means a React re-render per token — potentially hundreds per second. Instead, chunks accumulate in a ref and flush to state on an 80ms interval. The UI stays smooth and CPU usage drops dramatically.

Start-event buffer reset. Cloud Tasks retries mean a worker might stream the same turn twice. When the frontend receives a start event (including retry attempts), it unconditionally clears the streaming buffer. Attempt 2 re-streams from scratch without leftover text from attempt 1 appearing in the UI. This single event handler made retries invisible to the user.

Cross-agent polling. The polling fallback (for when SSE connections drop) runs across all agent sessions, not just the currently visible one. If the user switches agents while a turn is in progress, the background poll catches the completion and reconciles the session state. Without this, switching agents and switching back could show a permanently “thinking” indicator.

Per-agent session isolation. Each agent gets its own chatId, message history, queue, and streaming state. Switching agents is instant — no re-fetch, no loading spinner, no lost context. Chat IDs persist to localStorage, so sessions survive page refreshes and the frontend re-fetches message history on mount.

What We’d Do Differently

Infrastructure as code from day one. We used a shell script for provisioning. It worked, but it’s not auditable, not reviewable, and not safe to run twice without reading it first. Terraform or Pulumi would have been the right call even for a POC.

Structured logging. We added console.log where it hurt and nothing where it didn’t. By the time we were debugging the subscribe-after-check race in production, we wished we had trace IDs correlating the gateway request, the Cloud Task, and the worker execution. Structured logging with a shared trace context should be non-negotiable infrastructure, not a “we’ll add it later” item.

Contract versioning. Agent Contract v1 is implicit — it exists in documentation and convention but not in code. A versioned schema (even a simple JSON Schema for the /process request/response) would make it possible to evolve the contract without breaking existing workers.

What stays

The managed-services-only stance was the right call. Cloud Tasks gave us durable dispatch, retries, and backpressure for free. Redis Pub/Sub gave us streaming without operating a WebSocket server. Firestore gave us persistence without a database migration story. We didn’t operate a single piece of stateful infrastructure.

The gateway/worker split was the right boundary. The gateway knows about routing, persistence, and presentation. The worker knows about inference. Neither knows about the other’s internals. That boundary held through every bug, every refactor, and every new agent we added.

And the registry pattern delivered exactly what we hoped: adding an agent is an operational task, not a development task. No PRs, no deployments, no coordination. Write a document, grant a role, done.

The best architecture work isn’t about what you build. It’s about what you make easy to change later.