Reference architecture: enterprise agentic AI platform
A layered reference architecture for running agentic AI in a large enterprise — channels, agent orchestration, grounding, and the governance plane that makes it production-viable.
- agents
- architecture
- google-cloud
Most agentic AI initiatives stall not on model capability but on structure: a demo agent wired directly to a model endpoint, with no separation between orchestration, tools, data access, and governance. This reference architecture is the layered structure that survives contact with enterprise reality.
The architecture
Layer responsibilities
Channels own the user contract — web, internal tools, contact center, partner APIs. They hold no agent logic; they speak to the agent layer through one stable interface, which is what lets you change orchestration without renegotiating every integration.
Agent orchestration is where the system’s behavior lives. A root agent owns routing, session state, and memory; specialist task agents own bounded domains; tools are the only path to side effects. Keeping tools as first-class, independently-testable components — rather than prompt-embedded instructions — is the single highest-leverage structural decision.
Model layer treats models as replaceable infrastructure: versioned endpoints behind guardrails, never called directly from channel code. Model upgrades become configuration changes with an eval gate, not rewrites.
Enterprise systems and grounding split into transactional integrations (ERP, CRM, MES, ticketing) reached through tools, and the retrieval plane (warehouse, vector search, document stores) that grounds responses. The separation matters because their failure modes, latency budgets, and access controls differ completely.
Governance, evaluation and operations is cross-cutting by design, not a phase-two add-on: eval harnesses with golden datasets, guardrails and PII policy, tracing and cost telemetry, human-in-the-loop review for consequential actions, least-privilege identity for every tool, and CI/CD that treats prompts, tools, and agent definitions as deployable artifacts.
Where implementations go wrong
- Channel-to-model shortcuts. The demo calls the model directly; six months later every channel has its own divergent prompt stack.
- Tools without contracts. Side effects reachable only through prompt text cannot be tested, rate-limited, or audited.
- Governance as a gate at the end. If evals and guardrails aren’t in the loop from the first sprint, the platform that reaches the review board is not the platform that was reviewed.
Adapting it
The layer boundaries are the durable part; the products inside each box are not. On Google Cloud the natural mapping is ADK for orchestration, Gemini endpoints in the model layer, and BigQuery plus Vertex AI Search for grounding — but the same shape holds on any stack that can enforce the boundaries.