I've Built Orchestration Layers Three Times. Here's the Pattern That Survives.

I’ve watched the same architectural mistake happen three times in twenty years. Each time, the technology was different. The pattern was identical.

The first time was microservices. Around 2014, every team at Huge started decomposing their monoliths into services. Within 18 months we had 80+ services and no coordinated way to trace a request across them. Every team had their own logging, their own error handling, their own deployment pipeline. We spent the next year building a service mesh and shared observability layer. The services were fine. The lack of a coordination layer was the problem.

The second time was API gateways. At Adidas, we had a dozen consumer-facing applications each managing their own API integrations. Authentication was handled differently by each team. Rate limiting was inconsistent. When we needed to change a backend contract, we had to coordinate across every consumer individually. We built a central API gateway. Same pattern: the individual components worked, the lack of a coordination layer created the chaos.

The third time is now. And the technology is AI agents.

The Agent Sprawl Problem

Every enterprise SaaS vendor shipped AI agents in the past 18 months. Salesforce has Agentforce. Adobe has Agent Orchestrator. ServiceNow has AI Agents. Microsoft has Copilot everywhere. Each one is optimized for its own platform, its own data model, its own workflow.

The pitch from every vendor is the same: “Our agent is the orchestrator. Everything flows through us.” The reality is that you have five vendors each claiming to be the central brain, and none of them can see what the others are doing.

The email agent doesn’t know the website agent just offered a discount. Budget caps set in the ad platform don’t propagate to the content personalization engine. Consent preferences captured in the CRM don’t apply to the chatbot. There is no unified trail that explains why a decision was made across the customer journey.

If you’ve been in enterprise technology long enough, this sounds familiar. It should. It’s personalization silos from 2015, repackaged with an AI label.

The Pattern That Survives

In every case — microservices, API gateways, and now agents — the solution was a coordination layer that sits above the individual components. Not replacing them. Orchestrating them.

The specific technology changes. The architecture pattern does not:

Individual components do what they’re good at. Salesforce manages CRM data. Adobe optimizes content. ServiceNow handles IT workflows. Each vendor’s agent stays inside its platform, executing domain-specific tasks with the data and capabilities it knows best.

A central layer handles coordination. One system decides what happens when a workflow spans multiple platforms. It passes shared context between agents. It enforces policies consistently — consent, budget, compliance — regardless of which vendor’s agent is executing. It maintains a single log that explains every decision.

The coordination layer owns context, policy, and the audit trail. This is the critical distinction. The individual agents are workers. The coordination layer is the orchestrator. Trying to make every vendor’s agent the orchestrator is how you get agent sprawl.

This isn’t theoretical. It’s the exact same pattern we implemented at scale with service meshes (Istio/Linkerd for microservices), API gateways (Kong/Apigee for API management), and observability platforms (Datadog/New Relic for distributed tracing). The technology label changes every five years. The architecture doesn’t.

Why This Time Is Harder

Agent orchestration is harder than service orchestration for one specific reason: agents make decisions, services don’t.

A microservice receives a request, processes it deterministically, and returns a response. The service mesh coordinates routing and observability, but the service itself doesn’t exercise judgment. An AI agent receives a task, reasons about how to accomplish it, and may take different actions each time. The orchestration layer now needs to manage not just routing and policy, but reasoning quality and decision accountability.

This means the coordination layer needs components that didn’t exist in the service mesh era:

A context layer that assembles the right information for each agent decision. Not a database — a retrieval and assembly pipeline that manages token budgets and relevance.

A policy engine that enforces business rules before and after agent actions. Not just authentication and rate limiting, but consent enforcement, budget caps, brand guidelines, and compliance constraints.

A run log that captures not just what happened, but why. When a regulator asks why a customer received a specific offer, the answer can’t be “the model decided.” It needs to be: “given this context, under these policies, the agent chose this action for these reasons.”

The Decision You Actually Need to Make

If your business processes are contained within one or two platforms, vendor-embedded agents are fine. Let Salesforce orchestrate within Salesforce. The complexity of an independent layer isn’t justified.

If your business processes span three or more platforms — and in any enterprise above 500 people, they do — you need an independent orchestration layer. Not because the vendor agents are bad at their jobs, but because nobody is coordinating between them.

The pragmatic answer for most enterprises is a hybrid: an independent orchestration layer that handles planning, coordination, policy enforcement, and the audit trail, while vendor agents execute domain-specific tasks as workers.

This is the same answer we arrived at with microservices (service mesh + individual services), with APIs (gateway + individual endpoints), and with observability (platform + individual agents). The pattern survives because the problem — coordinating independent components in a distributed system — is always the same problem.

What I’d Do If I Were Starting This Today

If I were the CTO of a mid-market enterprise with 5-10 SaaS platforms in my stack and a mandate to “do something with AI agents,” here’s the sequence:

1. Audit your agent landscape. How many AI agents are already running? Most CTOs I work with are surprised by the number. Marketing turned on three. IT turned on two. Sales has one. Nobody coordinated.

2. Map the cross-system workflows. Which business processes touch more than two platforms? Those are the processes where agent sprawl will cause the most damage.

3. Build the context layer before adding more agents. An agent with bad context is worse than no agent at all. Invest in a shared context pipeline that can serve any agent in your stack.

4. Start the policy engine small. Begin with consent enforcement and budget caps — the two policy types that cause the most damage when inconsistent. Expand to compliance and brand guidelines as you scale.

5. Instrument the run log from day one. Retrofitting observability after you have 15 agents in production is painful and expensive. I’ve done it. Don’t be me.

The architecture that survives is never the one that’s most sophisticated. It’s the one that coordinates the most effectively across independently evolving components. That was true with microservices. It was true with APIs. It’s true with agents.

For the full technical reference — including architecture diagrams, vendor comparison, and a decision framework — see the comprehensive agentic AI architecture guide on We The Flywheel.

To assess whether your current architecture is ready for agentic workloads, take the free AI readiness assessment. Architecture is consistently the lowest-scoring dimension — only 12% of engineering leaders rate theirs as AI-ready.