AI Model Waddle

AI Agent Architecture

A production blueprint for applied AI agents: control loops, context management, tool orchestration, policy gating, telemetry, and recovery mechanisms.

AI Agent Architecture

An AI agent is not simply a language model with prompt instructions. In production software, an agent is an applied control system that perceives environmental state, plans actions against explicit constraints, executes tools in external environments, evaluates outcomes, and self-heals when failures occur.

A production agent architecture decouples reasoning from execution, organizing the system into eight core subsystems.

Applied Architecture Diagram: The 8 Subsystems

                           ┌────────────────────────┐
                           │   1. Model Engine      │
                           │ (System 2 / Reasoning) │
                           └───────────┬────────────┘
                                       │
                                       ▼
┌──────────────────┐       ┌────────────────────────┐       ┌──────────────────┐
│ 2. Context/State │ <───> │ 4. Orchestration Loop  │ <───> │ 5. Policy Gates  │
│  Management      │       │  (Plan, ReAct, Steps)  │       │ & Safety Checks  │
└──────────────────┘       └───────────┬────────────┘       └──────────────────┘
                                       │
                                       ▼
┌──────────────────┐       ┌────────────────────────┐       ┌──────────────────┐
│  6. Telemetry &  │ <───> │   3. Tools & Actions   │ <───> │ 8. Recovery &    │
│  Observability   │       │ (APIs, CLI, Filesystem)│       │ Fallback Engine  │
└──────────────────┘       └───────────┬────────────┘       └──────────────────┘
                                       │
                                       ▼
                           ┌────────────────────────┐
                           │ 7. Evaluation & QA     │
                           │ (Tests + Factual Gate) │
                           └────────────────────────┘

The Eight Applied Subsystems

1. The Model Engine (Reasoning vs. Execution)

The model engine provides the intelligence primitive. Modern architectures increasingly employ a two-tier strategy:

  • System 2 / Reasoning Engine: High-capacity frontier reasoning models handle intake analysis, high-level planning, architectural trade-offs, and multi-file code synthesis.
  • System 1 / Bounded Evaluators: Low-latency, non-autoregressive decision models or fine-tuned classifiers handle discrete categorical tasks, such as intent triage, simple routing, or pre-execution policy checks.

2. Context & State Management

Language models have fixed context windows and degrade in reasoning accuracy as uncurated tokens accumulate. A robust state subsystem separates memory into distinct tiers:

  • Ephemeral Working State: The immediate turn conversation and active tool execution outputs.
  • Repository / Environment Context: Structured ASTs, symbol indices, and dependency graphs retrieved dynamically via targeted tools rather than dumped entirely into prompt history.
  • Context Compaction: Rolling summarization or message pruning mechanisms that trim stale execution outputs while preserving critical goals and error signatures.

3. Tools & Action Execution Layer

Agents interact with the world through structured interfaces. Production tool environments must provide:

  • Deterministic Protocols: Structured tool calling using JSON Schema, Model Context Protocol (MCP), or typed RPC endpoints.
  • Sandboxed Execution: Confining filesystem modifications and bash command execution to sandboxed containers or isolated git branches.
  • Atomic Operations: Preferring targeted, contiguous file edits and single commands over massive multi-file overhauls to simplify rollback.

4. Orchestration & Planning (The Control Loop)

The orchestrator drives the execution cycle. Production frameworks typically implement variants of the ReAct (Reason + Act) loop or DAG-based task runners:

  1. Intake & Task Decomposition: Deconstructing the primary goal into ordered subtasks.
  2. Step Budget Governors: Enforcing strict iteration ceilings (e.g. maximum 15 tool execution turns) to prevent infinite loops.
  3. Subagent Delegation: Spawning isolated child agents for deep codebase research, documentation verification, or database triage to protect the primary orchestrator's context window.

5. Policy & Safety Gates

Before any irreversible action is executed, it must pass through deterministic policy filters:

  • Permission Ceilings: Restricting commands that touch production databases, external network endpoints, or sensitive secret stores.
  • Human-in-the-Loop Tripwires: Halting execution and requesting explicit user approval for destructive changes, large file deletions, or external financial transactions.
  • Budget Tripwires: Tracking cumulative token usage and terminating loops if token expenditure exceeds preset limits.

6. Telemetry & Observability

Production agent systems require granular runtime tracing to enable post-mortem analysis:

  • Step-by-Step Trajectory Logging: Capturing exact tool call arguments, standard output, standard error, and latency per step.
  • Token Accounting: Tracking input tokens, visible output tokens, and billed reasoning tokens across every model invocation.
  • State Diffs: Recording git diffs after each tool execution to monitor workspace evolution.

7. Evaluation & Verification Gateways

Agents must verify their work using ground truth feedback rather than self-congratulatory model assertions:

  • Deterministic Validation: Automatically running compilers, static analysis linters, and unit test suites.
  • Factual Verification: Auditing generated configuration, documentation, or pricing against primary source documentation to ensure external consistency.

8. Recovery & Fallback Mechanisms

When actions fail or errors occur, the recovery subsystem intervenes:

  • Self-Healing Retries: Catching compiler or linter errors, injecting the exact error trace into context, and allowing the agent to attempt a targeted correction.
  • Strategy Reset: If an agent repeats identical failing actions across three consecutive turns, the orchestrator interrupts the loop, reverts the uncommitted git state, and forces the model to formulate an alternative plan.
  • Graceful Escalation: Falling back to human operators with a clear diagnostic summary when automatic recovery thresholds are exhausted.

Concrete Grounding: Two Production Implementations

Case 1: The Multi-Stage Coding Agent Workflow

In software development environments, agent architectures must enforce a strict separation between code execution and external validation.

As demonstrated in our Coding Agent Workflow case study, an agent operating in a development loop can successfully produce clean code while simultaneously introducing factual errors regarding vendor API pricing, tier allowances, and reset mechanics.

By structuring the agent architecture to enforce an independent Factual QA Gate (Subsystem 7) following code compilation, teams catch external discrepancies before deployment.

Case 2: Bounded Decision Layer & Scoped Latency

Not every step in an agent control loop requires calling a multi-second generative language model. In high-frequency workflows—such as classifying incoming user requests, routing tasks to specialized subagents, or evaluating safety guardrails—architectures can integrate non-autoregressive decision models.

In our 385-sample Banking77 experiment, TypeSafe Jev (jev-1.13.0) recorded 188ms median client-observed latency (p95: 320ms) and 79.48% top-1 accuracy at $0.0679 per 1,000 decisions. In contrast, Google Gemini 3.8 Flash (gemini-3.8-flash) structured output achieved 83.64% top-1 accuracy with a 1,204ms median latency (p95: 3,245ms) at $0.4570 per 1,000 decisions.

To review the raw measurement data and statistical significance tests, see our Jev vs. Structured Outputs benchmark report.

Architectural Best Practices for Engineering Teams

  1. Keep Tool Interfaces Small and Strongly Typed: Expose narrow, single-purpose tools with strict schema validation rather than monolithic, multi-purpose functions.
  2. Treat Git as the Ultimate Undo Buffer: Always run agent tasks in dedicated feature branches or isolated worktrees, committing only after all evaluation gates pass.
  3. Budget Context Explicitly: Do not allow unbounded tool output (e.g. 10,000-line log dumps) to be returned directly to the model. Truncate outputs and extract relevant snippets in application code.
  4. Decouple Policy from the Model: Implement safety constraints as deterministic code checks outside the model's prompt instructions.

Related Architecture and Workflow Guides

Study the end-to-end 10-stage lifecycle of production coding agents

workflow

Learn how to match specific software tasks to appropriate model and system roles

concept

Review the technical architecture and non-autoregressive primitives of TypeSafe Jev

entity