At AIEE.io, written notes and mental models were not enough to keep pace with agentic software engineering. So we built a verifiable ingestion and processing pipeline with one goal: stay current, precisely. It tracks papers, anchors concepts to primary sources, and ranks findings by a composite signal across key insights, cross-field reach, empirical claims, and recency. The Field Pulse is the digest that pipeline produces. Each issue surfaces the concepts crystallizing, the patterns stabilizing, and the papers worth your attention.

If you want to learn from engineers who were building production-grade AI/ML systems long before the LLM hype, subscribe and share.

Subscribe now

Find the previous pulse here!

State of the field

Behavioral alignment now has its first control primitives. Until recently the field could only name the ways coding agents go wrong; now it can steer them. The sharpest result is TACT: over a long task an agent tends to drift into overthinking or overacting, and TACT finds that this drift is a single measurable signal inside the model, so you can push against it while the agent runs, with no retraining, and it resolves 5.8pp more issues. Two blunter levers join it. Symbolic guardrails enforce most explicit rules (~74%) with plain deterministic checks instead of trusting the model to follow them. And the 5,000-run Claude Code study finds that telling an agent what not to do lifts its score (+13.8pp) while telling it what to do does not; even random rules help, so it is the constraint itself doing the work, not the agent’s grasp of the advice. No single unifying mechanism yet, but the corner finally has tools.

Two widely held beliefs no longer hold. First, more tokens do not buy a better answer: Bai et al. find accuracy peaks at a middle level of spend and then flattens, and that agents cannot predict their own token use. Second, multi-agent coding is not settling on one winning design. Talluri’s complexity-conditioned topology selection picks a different arrangement of agents depending on how hard the task is, and proves it stays within budget, making it a genuinely distinct architecture rather than the Planner/Generator/Evaluator triad renamed.

Two new failure surfaces open. Benchmarks keep getting harder: SWE-Cycle tests the entire issue-resolution loop rather than a single patch, and RepoZero asks an agent to rebuild a repository from scratch, where the best models reach only 30-55%. But the benchmarks themselves are now gameable. Chen et al. show that under user pressure agents learn to chase the public score instead of genuinely solving the task. And on the human side, Duma finds most agent-written pull requests are merged with little or no real human review, so “merged-but-flawed” is better described as “merged-but-unreviewed.”

Concepts crystallizing

The ideas the field keeps circling, and where each stands now. Behavioral alignment has moved from naming failure modes to its first control mechanisms. The convention of loadable spec files (AGENTS.md and the like) now has hard empirical backing. And workspace-scale evaluation has two new benchmarks raising the bar.

Behavioral Alignment in Coding Agents

Behavioral alignment has moved from naming failure modes to concrete control mechanisms: TACT (activation steering of agent drift), symbolic guardrails (non-neural policy enforcement), and negative-constraint priming (Guardrails Beat Guidance). The regulate problem now has levers, not just diagnoses.

Coding agents don’t just succeed or fail at tasks — they exhibit behavioral patterns under load that systematically deviate from instructions, evaluations, and stated values. Behavioral alignment is the sub-discipline that names these patterns, measures them, and asks whether the harness should intervene.

Key papers: Trustworthy AI Software Engineers · Harness Engineering for Coding Agent Users · Symbolic Guardrails for Domain-Specific Agents · Self-Attribution Bias · Agentic Harness Engineering · Beyond Resolution Rates

Loadable Spec Files

Guardrails Beat Guidance puts a hard empirical edge on the rule-file convention: across 5,000+ Claude Code runs, prohibitions help (+13.8pp) and positive guidance does not, and even random rules help. The effect is context priming, not semantic understanding.

A family of Markdown files in known locations that coding agents load at session start (or on demand) to acquire project-specific rules, knowledge, or constraints. The pattern shows up across multiple system properties — coding rules, visual identity, domain knowledge, workflow recipes — with the same shape:

Key papers: Linux Foundation Announces the Agentic AI Foundation (AAIF)… · AGENTS.md — A Simple, Open Format for Guiding Coding Agents · Best Practices for Claude Code · Effective Context Engineering for AI Agents · claude-code — Anthropic’s terminal coding agent reference… · continue — OSS Copilot alternative with custom slash…

Workspace-Level Agent Evaluation

The evaluation substrate keeps advancing: SWE-Cycle extends evaluation to the full issue-resolution loop on a bare repository, and RepoZero to from-scratch reproduction (30-55%). Both push past single-patch tasks toward workspace-scale mastery.

An agent’s effectiveness is not a property of the agent alone — it’s a property of the (agent × harness × workspace) triple. Workspace-level evaluation treats the filesystem-and-its-dependency-graph as a first-class evaluation substrate: how well does the agent retrieve across files, reason over implicit dependencies, and adapt its plan as it discovers structure? Tang et al.’s Workspace-Bench operationalizes this with 20,476 files across 5 worker profiles and 388 tasks, each scored against an explicit file-dependency graph. The headline gap — best agent 68.7% vs. human 80.7%, mean 47.4% — says current agents fail on long-range cross-file reasoning, not on individual operations.

Key papers: The Semi-Executable Stack · Agentic Harness Engineering · Workspace-Bench 1.0 · From Agent Loops to Structured Graphs

Patterns stabilizing

The recurring designs the field is converging on, and how each is holding up. The four-corner harness model has gained a fifth corner, regulate, with concrete mechanisms behind it. A new control-primitive pattern has emerged. And the Planner/Generator/Evaluator triad is weakening as multi-agent designs diverge rather than converge.

Harness 4-corner picture (+ regulate)

locate · name · evolve · optimize, and now a populated fifth corner: regulate. Once an empty slot, it now holds concrete mechanisms (TACT activation steering, symbolic guardrails, negative-constraint priming). The harness is an evolvable, optimizable, observable, and steerable artifact.

Key papers: The Semi-Executable Stack · Agentic Harness Engineering · HARBOR · TACT

Behavioral control primitive (regulate) ⭐ NEW

First mechanisms for the ‘regulate’ corner, split across two halves: internal/dynamics (TACT steers drift as a residual-stream direction at test time) and external/policy (symbolic guardrails enforce explicit rules without a model; negative-constraint priming shows prohibitions, not guidance, are the lever). No single unifying abstraction yet, but the corner is no longer empty.

Key papers: TACT · Symbolic Guardrails for Domain-Specific Agents · Guardrails Beat Guidance

Planner/Generator/Evaluator triad

Three-agent decomposition separating reasoning, generation, and verification. Still a real attractor, but the evidence now cuts against convergence: Talluri’s complexity-conditioned topology selection treats the topology itself as the variable (chosen per a retrieved structural-complexity vector, with a budget-conservation proof), evidence that multi-agent SWE is diverging into distinct architectures rather than collapsing onto one triad.

Key papers: Harness Design for Long-Running Application Development · Building Effective AI Coding Agents for the Terminal… · Retrieval-Conditioned Topology Selection with Provable…

Correctness-gated code evolution

Outer optimization loop where each iteration must pass a correctness oracle before becoming the new state. Works in narrow domains (logic synthesis, leetcode); generalizing to open-ended software requires weaker oracles, learned verifiers, or human-in-the-loop scaffolding.

Key papers: Autonomous Evolution of EDA Tools · Agentic Harness Engineering

Latest papers

The most consequential papers from this period.

1. TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering

Yuan Sui et al. · 2026 · arXiv:2605.05980 · Behavioral Alignment

Agent drift (overthinking/overacting) is a linearly separable direction in the residual stream; test-time activation steering lifts SWE-bench resolve rate +5.8pp. Coding agents degrade over long trajectories — agent drift — via two failure modes: overthinking (re-reasoning over information already held) and overacting (issuing tool calls without integrating recent observations).

• Long-horizon coding-agent failure decomposes into two named, recurring behavioral modes: overthinking and overacting.

• These modes are linearly separable in the residual stream (AUC ≈ 0.9) along two drift axes anchored at calibrated behavior.

• Drift can be detected and corrected before it surfaces as a behavioral failure — a test-time, training-free intervention.

• Steering yields concurrent gains in both quality (resolve rate +4.8–5.8pp) and efficiency (steps-to-resolve down to −26%).

2. Symbolic Guardrails for Domain-Specific Agents: Stronger Safety and Security Guarantees Without Sacrificing Utility

Yining Hong et al. · 2026 · arXiv:2604.15579 · Behavioral Alignment

74% of explicitly-specified agent policy requirements can be enforced by symbolic (non-neural) guardrails without sacrificing utility. Training-based methods and neural guardrails improve agent reliability but cannot guarantee it.

• Neural and training-based mitigations cannot provide guarantees; symbolic guardrails can, for a meaningful subset of policies.

• 85% of surveyed agent safety/security benchmarks (n=80) lack concrete, machine-checkable policies — they rely on high-level goals or common sense.

• Among policies that are specified, 74% of requirements are enforceable by symbolic guardrails, often with simple, low-cost mechanisms.

• Symbolic guardrails improve safety and security without sacrificing agent utility on τ²-Bench, CAR-bench, and MedAgentBench.

3. Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents

Xing Zhang et al. · 2026 · arXiv:2604.11088 · Behavioral Alignment

Negative constraints (prohibitions) improve coding-agent performance; positive directives (guidance) do not — even random rules help +13.8pp, evidencing context priming over semantic understanding. The first large-scale controlled study of agent rule files (.claude.md, .cursorrules, and the broader family of agent skills, plugin manifests, persona definitions): 679 rule files (25,532 rules) scraped from GitHub, 5,000+ Claude Code runs with Claude Opus 4.6 on SWE-bench Verified.

• Random rules and expert-curated rules deliver the same performance gain (+13.8pp), implying the content semantics matter less than the presence of rules.

• Every individually beneficial rule observed was a negative constraint (a prohibition); every individually harmful one was a positive directive (prescriptive guidance).

• The effect is consistent with “context priming” — the rule shifts the agent’s behavior distribution — rather than the agent semantically understanding and following the rule.

• Operationalizes the broad family of persistent agent config: rule files, skills, plugin manifests, persona definitions — directly relevant to the loadable-spec-files surface.

4. Retrieval-Conditioned Topology Selection with Provable Budget Conservation for Multi-Agent Code Generation

Abhijit Talluri et al. · 2026 · arXiv:2605.05657 · Multi-Agent Orchestration

Conditioning orchestration topology on a code-complexity vector cuts proxy-measured misrouting from 30.1% to 8.2% with provable budget conservation. Multi-agent codegen systems pick an orchestration topology without consulting the codebase, yet the optimal topology depends on the structural complexity of the code under modification.

• The orchestration topology should be conditioned on the structural complexity of the code being modified, extracted via retrieval from a hierarchical code index — not chosen statically.

• Complexity-conditioned routing reduces proxy-measured misrouting from 30.1% to 8.2%.

• A budget algebra with six-dimensional budget vectors yields provable budget conservation (structural-induction conservation theorem) under dynamic topology selection — a property neither complexity-conditioned routing nor resource algebras provides alone.

• Engineering claims: sub-millisecond DAG construction and linear tree-index scalability.

5. How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

Longju Bai et al. · 2026 · arXiv:2604.22750 · Cost Efficiency

Accuracy peaks at intermediate token cost and saturates higher; more tokens != better, and models can’t predict their own usage. First systematic study of token consumption in agentic coding tasks, analyzing trajectories from eight frontier LLMs on SWE-bench Verified.

• Agentic tasks consume ~1000x more tokens than code reasoning / code chat; input tokens (not output) drive cost.

• Token usage is highly stochastic — same task can vary by up to 30x in total tokens.

• Higher token usage does not translate into higher accuracy; accuracy peaks at intermediate cost and saturates at higher cost.

• Substantial cross-model efficiency gaps: Kimi-K2 and Claude-Sonnet-4.5 average 1.5M+ more tokens than GPT-5 on identical tasks.

6. These Aren’t the Reviews You’re Looking For: How Humans Review AI-Generated Pull Requests

Kacper Duma et al. · 2026 · arXiv:2605.02273 · Empirical PR Studies

Most AI-generated PRs get no human review; oversight is automation-mediated, so merge metrics overstate human scrutiny. Using the AIDev dataset, the authors compare code-review interactions on AI-generated vs human-authored PRs within the same repos.

• Most AI-generated PRs receive no review whatsoever.

• When AI PRs are reviewed, the review is dominated by AI agents, not humans.

• Human-authored PRs are more likely to get human-only review and direct human feedback.

• Reviews of AI PRs more often take the form of “automation-mediated interaction” — human involvement expressed through agent steering rather than standalone evaluation.

7. To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study

“Shota Sawada” et al. · 2026 · arXiv:2605.06464 · Empirical PR Studies

AI-generated files are maintained LESS often than human code; edits are feature extensions, not bug fixes. Empirical study using the AIDev dataset (1,000+ files, ~3,200 changes, 100 popular GitHub repos) comparing maintenance of AI-generated files vs human-authored code.

• AI-generated files receive less frequent maintenance than human-authored code, and when touched, updates affect only a small fraction of file size.

• The most frequent modifications to AI code are feature extensions; human-code updates skew toward bug fixes.

• Human developers perform the large majority of maintenance on AI-generated code.

• Reframes the “AI code has more defects” narrative toward “AI code has a different maintenance profile” — extension over repair.

Field anchors

The works the field keeps returning to: the references the rest of this body of work builds on.

• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses — Observability-driven automatic evolution of coding-agent harnesses lifts Terminal-Bench 2 pass@1 from 69.7 to 77.0 over 10 iterations, with transfer across model families.

• claude-code — Anthropic’s terminal coding agent reference harness — The vendor-controlled reference harness pattern: ships with editorial guidance (best-practices docs, harness-design posts) and a convention layer (AGENTS.md, skills, hooks) documented as a contract between agent and project.

• Linux Foundation Announces the Agentic AI Foundation (AAIF) — anchored by MCP, goose, and AGENTS.md — Linux Foundation forms AAIF anchoring MCP, goose, and AGENTS.md; convention layer of the agent stack shifts from vendor-controlled to multi-vendor governance.

• The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE — Six-ring conceptual reference model spanning intent → spec → code → test as a continuous semi-executable artifact ladder; locates where harness layers live.

• 12 Agentic Harness Patterns from Claude Code

Open questions

• Does a unifying behavioral control abstraction emerge (the “AHE for behavior”), or do internal-steering and external-policy primitives stay separate?

• Does TACT-style activation steering generalize past overthinking/overacting and survive stronger models + adversarial codebases?

• Do symbolic guardrails transfer from agent-general benchmarks to coding-specific rule files (AGENTS.md/CLAUDE.md as symbolically-enforceable, not prose)?

• Is the accuracy-cost interior optimum stable across models/task families, and can an agent find it online?

• What review substrate scales to agent-volume PRs without a human in every loop?

• env-automation took no in-window catches — is the thread genuinely cooling, or a discovery-query gap?

At AIEE.io, written notes and mental models were not enough to keep pace with agentic software engineering. So we built a verifiable ingestion and processing pipeline with one goal: stay current, precisely. It tracks papers, anchors concepts to primary sources, and ranks findings by a composite signal across key insights, cross-field reach, empirical claims, and recency. The Field Pulse is the digest that pipeline produces. Each issue surfaces the concepts crystallizing, the patterns stabilizing, and the papers worth your attention.

If you want to learn from engineers who were building production-grade AI/ML systems long before the LLM hype, subscribe and share.

Subscribe now

State of the field

Coding agents are moving from prompt-and-pray to engineered harnesses with observability-driven evolution (Lin AHE) and bounded BO optimization (Sengupta HARBOR). The bootstrap pulse opened a fifth corner the field had been ignoring: behavioral alignment, where five Jan-Apr papers name distinct failure modes (goal drift, self-monitoring biases, motivation-framing sensitivity, behavioral drift between LLMs, unmeasurable trustworthiness) that share a thread: behavior is a target of design, not just an emergent property of the model. The eval-substrate wave the May-6 framing called emergent has been quietly cresting since January (OctoBench, ABC-Bench, SWE-Refactor, CktEvo, FeatureBench predate Workspace-Bench / POSTCONDBENCH).

Concepts crystallizing

Selected concepts for this pulse.

Behavioral Alignment in Coding Agents — ⭐ NEW

NEW corner opened this pulse — 5 Jan-Apr papers (Saebo, Khullar, Wu, Mehtiyev, Aleti) anchor a fifth ‘regulate’ axis to the harness picture.

Coding agents don’t just succeed or fail at tasks — they exhibit behavioral patterns under load that systematically deviate from instructions, evaluations, and stated values. Behavioral alignment is the sub-discipline that names these patterns, measures them, and asks whether the harness should intervene.

Anchored by: Trustworthy AI Software Engineers · Harness Engineering for Coding Agent Users · Self-Attribution Bias · Agentic Harness Engineering · Beyond Resolution Rates · Asymmetric Goal Drift in Coding Agents Under Value Conflict · +2 more

Loadable Spec Files

Linux Foundation governance moment under AAIF (60K+ repos, 30+ tools, 170+ orgs) shifts the convention layer of the agent stack from vendor-controlled to multi-vendor governance.

A family of Markdown files in known locations that coding agents load at session start (or on demand) to acquire project-specific rules, knowledge, or constraints. The pattern shows up across multiple system properties — coding rules, visual identity, domain knowledge, workflow recipes — with the same shape:

Anchored by: Linux Foundation Announces the Agentic AI Foundation (AAIF)… · AGENTS.md — A Simple, Open Format for Guiding Coding Agents · Best Practices for Claude Code · Effective Context Engineering for AI Agents · CLAUDE.md, AGENTS.md & Copilot Instructions · LLM-Assisted Repository-Level Generation with Structured…

Workspace-Level Agent Evaluation

Workspace-Bench formalizes the eval substrate beyond task completion — filesystem and dependency graph as the testbed, not the harness.

An agent’s effectiveness is not a property of the agent alone — it’s a property of the (agent × harness × workspace) triple. Workspace-level evaluation treats the filesystem-and-its-dependency-graph as a first-class evaluation substrate: how well does the agent retrieve across files, reason over implicit dependencies, and adapt its plan as it discovers structure? Tang et al.’s Workspace-Bench operationalizes this with 20,476 files across 5 worker profiles and 388 tasks, each scored against an explicit file-dependency graph. The headline gap — best agent 68.7% vs. human 80.7%, mean 47.4% — says current agents fail on long-range cross-file reasoning, not on individual operations.

Anchored by: The Semi-Executable Stack · Agentic Harness Engineering · Workspace-Bench 1.0 · From Agent Loops to Structured Graphs

Subscribe now

Patterns stabilizing

Selected patterns for this pulse.

Harness 4-corner picture

locate · name · evolve · optimize — possibly a 5th corner: regulate. The harness as an evolvable, optimizable, observable artifact, not just runtime infrastructure.

Anchored by: The Semi-Executable Stack · Agentic Harness Engineering · HARBOR

Planner/Generator/Evaluator triad

Three-agent decomposition where reasoning, generation, and verification are separated. Structurally distinct from single-agent loops; converging on multi-vendor consensus (Anthropic Applied AI + OPENDEV + emerging multi-agent orchestration papers).

Anchored by: Harness Design for Long-Running Application Development · Building Effective AI Coding Agents for the Terminal…

Correctness-gated code evolution

Outer optimization loop where each iteration must pass a correctness oracle before becoming the new state. Works in narrow domains (logic synthesis, leetcode); generalizing to open-ended software requires weaker oracles, learned verifiers, or human-in-the-loop scaffolding.

Anchored by: Autonomous Evolution of EDA Tools · Agentic Harness Engineering

Top 7 papers

Ranked by an 8-signal share-worthiness composite: key-insight thread, vocabulary, cross-field reach, substantial My Take, empirical claims, log-inbound citations, wildcard, recency decay. Hooks are verbatim from each paper’s abstract. No LLM rewriting at render time.

1. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Jiahang Lin et al. · 2026 · arXiv:2604.25850 · Harness Design

Observability-driven automatic evolution of coding-agent harnesses lifts Terminal-Bench 2 pass@1 from 69.7 to 77.0 over 10 iterations, with transfer across model families. Treats the harness (runtime around a coding agent: tools, prompts, memory, controls) as a first-class engineered artifact and automates its evolution via an observability-driven loop with three pillars — component, experience, decision.

• The harness is the unit of optimization. Performance gains come from patching the harness (tools, prompts, controls) — not from retraining the model.

• Three observability pillars make evolution diagnostic instead of trial-and-error:

• AHE loop: instrument → execute on benchmark → diagnose failure modes via the three pillars → patch harness components → re-evaluate. Repeat to convergence / iteration budget.

• Result on Terminal-Bench 2: 69.7 → 77.0 pass@1 after 10 evolution iterations.

2. The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE

Robert Feldt et al. · 2026 · arXiv:2604.15468 · Harness Design

Six-ring conceptual reference model spanning intent → spec → code → test as a continuous semi-executable artifact ladder; locates where harness layers live. Conceptual keynote companion (arXiv:2604.15468, submitted 16 Apr 2026) arguing SE isn’t being replaced by LLM agents — the thing being engineered is expanding beyond executable code to semi-executable artifacts: natural language, tools, workflows, control mechanisms, and organizational routines enacted by human or probabilistic interpretation rather than deterministic execution.

• The AI threat narrative (”LLMs eat SE”) misreads the situation. The hard-won expertise isn’t losing value; the engineering surface is expanding.

• Semi-executable artifacts — natural language specs, tool configs, workflows, controls, organizational routines — now require engineering discipline even though they’re not deterministically executable.

• The Semi-Executable Stack spans six rings: executable artifacts → instructional artifacts → orchestrated execution → controls → operating logic → societal and institutional fit.

• Contributions, bottlenecks, and transitions can be located on the stack; each ring has neighbors it depends on.

3. HARBOR: Automated Harness Optimization

Biswa Sengupta et al. · 2026 · arXiv:2604.20938 · Harness Design

Bounded bayesian-optimization of fixed-component harness configurations; complement to AHE’s open-ended observability-driven evolution. A formalization of harness configuration as a constrained noisy Bayesian optimization problem over a mixed-variable, cost-heterogeneous flag space, plus a reference solver — HARBOR (Harness Axis-aligned Regularized Bayesian Optimization Routine) — built from a block-additive SAAS surrogate, multi-fidelity cost-aware acquisition, and TuRBO trust regions.

• Harness dominates the agent’s operational complexity, not the model. This is the load-bearing assertion — restated: in production agents, the model is one moving part; the harness is the surrounding 80%+ (compaction, caching, memory, trajectory reuse, speculative tools, sandbox glue).

• Manual flag stacking doesn’t scale past a handful of bits. Once the harness has more than ~6-8 binary flags (let alone continuous params), the combinatorial space exceeds what humans can grid-search by inspection. This is the practical motivation for automation.

• Constrained noisy Bayesian optimization is the right frame. Mixed-variable (binary + categorical + continuous), cost-heterogeneous (some flag combinations cost more inference dollars), and noisy (eval reward is stochastic). They formalize:

• Solver components: block-additive SAAS (Sparsity-Aware Sampling) surrogate for high-dim flag space; multi-fidelity cost-aware acquisition function; TuRBO trust-region restarts for non-convex landscape.

4. Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

Zirui Tang et al. · 2026 · arXiv:2605.03596 · Eval Substrate

Workspace-Bench 1.0 (388 tasks, 5 worker profiles, 20K files up to 20GB, 7K rubrics) shifts the eval axis from task completion to workspace mastery. Introduces Workspace-Bench, a benchmark for evaluating AI agents on tasks that require reasoning over large-scale, real-world file dependencies — the kind of cross-file retrieval, contextual reasoning, and adaptive decision-making a knowledge worker actually does.

• Workspace learning is underexplored. Prior agent benchmarks evaluate on pre-specified or synthesized files with shallow dependency structure; they don’t probe an agent’s ability to reason across a realistic worker’s filesystem.

• Scale of realism: 5 worker profiles, 74 file types, 20,476 files (up to 20 GB), 388 tasks, each with its own explicit file-dependency graph and a rubric set (7,399 total) covering cross-file retrieval, contextual reasoning, adaptive decision-making.

• Workspace-Bench-Lite = 100-task subset that preserves the distribution while reducing evaluation cost ~70% — a deliberate cost-control move that mirrors a recurring pattern across modern agent benchmarks.

• Empirical gap: 4 popular agent harnesses × 7 foundation models. Best 68.7%, average 47.4%, human 80.7% — large headroom; agents fail on long-range cross-file reasoning rather than on individual operations.

5. Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC

Cunxi Yu et al. · 2026 · arXiv:2604.15082 · Repo Codegen

Self-evolved logic-synthesis on million-line EDA codebases via correctness-gated outer loop; same skeleton as AHE applied to the target codebase rather than the harness. LLM agents autonomously rewrite the source code of ABC — the canonical open-source logic synthesis system — at full million-line integrated-codebase scale, and discover synthesis strategies that exceed human-designed heuristics on the standard EDA benchmark suites (ISCAS 85/89/99, VTR, EPFL, IWLS 2005).

• Million-line integrated-codebase scale. Agents reason about cross-file changes that compile and stay correct in a 20-year-old, performance-critical C codebase — not isolated functions like most LLM-code papers.

• Discovery, not implementation. The claim is that LLMs discover synthesis optimizations the human community didn’t write, not that they implement known optimizations faster. If defensible with concrete QoR deltas, one of the cleaner claims of LLM-driven scientific advance in a domain with mature priors.

• Programming guidance prompts. Meta-instructions about how to structure changes — encoded domain knowledge about synthesis software architecture. The mechanism that prevents random code mutation. Also the piece most likely to be hand-crafted in disguise.

• Correctness-gated evolution loop. Compile → validate correctness → score QoR on benchmarks → feedback → next iteration. Correctness is a hard gate; QoR is the optimization signal.

6. SWE-chat: Coding Agent Interactions From Real Users in the Wild

Joachim Baumann et al. · 2026 · arXiv:2604.20779 · Eval Substrate

SWE-Chat assembles a dataset of agent-user interactions during coding tasks; treated as dataset-existence citation per map’s wildcard slot. A 6,000-session, 63k-prompt, 355k-tool-call dataset of real coding-agent interactions collected from open-source developers in the wild — the first large-scale dataset of this kind.

• First large-scale wild dataset of coding-agent sessions. 6,000 sessions / 63k prompts / 355k tool calls; the scale is the contribution. Living/auto-updating pipeline is positioned as ongoing infrastructure.

• Bimodal usage: 41% vibe-coding (agent writes ~all committed code) + 23% human-only + ~36% mixed. The bimodality itself is the finding — implies there are two distinct user populations (or two distinct task modes), not a continuum of human-AI collaboration.

• 44% commit survival rate. Of all code an agent produces, less than half ends up in user commits. The remaining 56% is rejected, replaced, or modified beyond recognition.

• Agent code has more security vulnerabilities than human code in this dataset. The abstract asserts this but doesn’t quantify; presumably measured via static analyzers on committed code.

7. Insights into Security-Related AI-Generated Pull Requests

Md Fazle Rabbi et al. · 2026 · arXiv:2604.19965 · Empirical PR Studies

Empirical study of 33k+ AI-generated PRs / 675 security-related; merged-but-flawed PRs are load-bearing for agent-PR governance arguments. A descriptive empirical study of 33,000+ AI-generated pull requests filtered to 675 security-related submissions from agentic AI coders.

• AI security PRs concentrate on a small set of recurring weakness classes: regex inefficiencies, injection flaws (SQL/cmd/template), path traversal. The “small set” framing matters: it suggests AI security contributions are narrow — addressing easy-to-pattern-match vulnerability classes rather than deep architectural security issues.

• Many flawed AI PRs still get merged. This is the load-bearing finding for AIEE. Code review is not catching all the issues — review-as-gate is leaky for AI-generated security work.

• Rejection drivers are social/process, not technical. Inactivity (PR sits, no maintainer responds), missing test coverage (a process gate), and similar non-code factors dominate rejection reasons. Technical merit of the security fix often isn’t even the primary axis of decision.

• Commit-message quality decoupled from acceptance latency. Prior literature on human PRs shows commit message quality predicts acceptance speed; for AI PRs this signal weakens substantially. Possible interpretations: (a) AI commit messages are uniformly fluent so the signal saturates; (b) reviewers discount commit messages they suspect are LLM-generated; (c) the variance in AI commit quality is too low to discriminate.

What I’m watching for next pulse

• Does a unifying behavioral-alignment control primitive land? (the “AHE for behavior” gap)

• Do any of the multi-agent-orchestration patterns (SGAgent, RepoReviewer, AgentForge) get reframed as instances of Anthropic’s Planner/Generator/Evaluator pattern, or do they propose distinct architectures?

• Do new SWE-bench-successor benchmarks land, or does the wave cool? (signals whether eval-substrate is now over-served)

• Does cost-efficiency thicken? Currently a 1-source thread; a single new paper changes its shape considerably.

• Behavioral-alignment depth zoom via /notebook-autoresearch behavioral-alignment-in-coding-agents — should run before the next pulse so the field-pulse can render the depth-pass output (synthesis + new sources) alongside the bootstrap.

Articles 2 and 3 built the static half of this system. Information Architecture designed the routing: which agent handles which task, which knowledge loads. Quality Gates encoded the verification: deterministic chains that run before a commit lands, read-only reviewers that catch what unit-tests and linters can’t see. Both articles defined the static scaffolding of information and rules so that SWE agents can start producing high-quality outputs for us. This article is what puts that scaffolding in motion — the dynamic execution layer my agent teams actually use to ship work.

At the same time, I’ll take the chance to answer a question that has come up more than any other in my conversations these last few months: which MCPs and tools do you actually use in your Claude Code setup? This article is the complete answer — the full list, what each one closes, and how I wire them into real client work.

Before the inventory, though, this layer deserves a proper place inside the Cognitive Domain Engineering framework.

Execution Infrastructure is everything an agent uses to act on the world: the tools it calls directly (bash, file operations, MCP servers) and the platform mechanisms that wrap those calls (permission gates, hooks, the memory substrate underneath). Articles 2 and 3 decided which agent should act and when the action is allowed to land. This article is about how the action happens.

This whole layer exists to do one thing: drive coordination tax toward zero. Coordination tax is the work an agent spends translating between incompatible interfaces instead of doing the actual job — different error schemas, different retry semantics, different lifecycle assumptions. All of it has to be handled before the work can happen.

Coordination tax is infrastructure friction pretending to be cognitive work.

Before any MCP server enters the picture, though, Claude Code already arrives with a native execution vocabulary — a fixed set of tools baked in, available before a single external connection is configured. The next section is where that vocabulary gets named.

Subscribe now

The Native Execution Surface

On March 31, 2026, a source map surfaced inside the Claude Code npm package — noted publicly by Chaofan Shou (@Fried_rice) on X — and gave the community an unintended look at the inner workings of the harness. It revealed the details of the native execution surface: bash access, file operations, search, language server support, and the ability to spawn subagents.

The leak confirmed a deliberately narrow default surface: fewer than 20 tools, purpose-built rather than general. Bash for shell, but Read/Write/Edit for files (not cat/tee/sed), Grep/Glob for search (not raw shell pipes), Task for spawning subagents, TodoWrite for in-session planning. Each tool has structured arguments and predictable failure modes. Narrower surface than a POSIX shell, by design.

The native surface also includes full LSP support — go-to-definition, find references, symbol resolution across the codebase. The IDE-grade floor is there before any MCP loads.

The native layer also exposes hook checkpoints that fire around every tool call — PreToolUse before the action, PostToolUse after — plus session-lifecycle hooks (SessionStart, PostCompact, Stop) that wrap the conversation itself. I use them sparingly, on a tight discipline: load context, surface signal, never mutate memory or commit on my behalf. In my notebook vault, three small hooks load recent journal context at session start and surface a reminder when wiki content has changed — the writes still happen through Claude, with my eyes on them, never auto-mutated.

The next section shows what MCP adds on top.

My go-to MCPs and how they fit my client work

This is the tool shelf I actually use at AIEE to set agents in motion on client work. Each MCP server below earns its place by closing a specific loop — browser, design, presentation, sustained reasoning — that the native surface alone can’t reach.

Serena

Serena is the difference between an agent grepping its way through a 200-file service and an agent following the dependency graph the way I would in an IDE. With it loaded, the session has go-to-definition, find references, and symbol navigation natively. The time my teams spend looking for and reading code drops sharply, and they move a lot faster as a result.

Playwright

Playwright MCP gives the agent a browser. It navigates pages, clicks, fills forms, takes screenshots, captures network requests. Whenever one of my web UIs gets modified, I run a regression workflow that sends a QA specialist agent into the browser to look at the changes and audit against visual best practices. The agent does the visual tests while I go grab a coffee or start the next task.

Figma

Figma MCP makes my design files readable during a session. My agent teams read the Figma source directly to extract design tokens, build the design system, and mirror the source designs in code — no screenshots, no copy-pasted specs. I’ve also tried using it to generate wireframes from scratch, but that path still needs too much manual support to lean on; for now I’m using it for reading and understanding only, and that alone is extremely useful.

Gamma

Gamma MCP generates presentation-grade decks and pages from structured input. I run it alongside a set of Gamma-specific skills I’ve built — when I want a deck, one of my frontend agents walks me through a short set of definition questions (audience, tone, visual direction, length) before any generation happens. The output lands close to ship-ready because the visual decisions were made up front, not negotiated with the model after the fact.

Sequential thinking

Sequential thinking MCP keeps earlier decisions reachable across a long task. On a refactor that spans three or four services, the first decision the agent makes about the contract is usually the one it forgets six tool calls later — Sequential Thinking is what holds it in place. I keep it on by default for any large codebase where long files erode context fast. It doesn’t eliminate drift, but on tasks where drift is the main failure mode, it’s the right constraint.

Gemini for image generation

The one item on this list that isn’t an MCP — it reaches my sessions through a custom skill that calls the Gemini API directly, with a prompt-engineering layer I’ve tuned over time. Every image on my websites is generated this way: either an agent specialist runs the full flow autonomously, or I use prompts a Claude agent produces through the same skill.

For installation, follow the link on each tool above — the official docs carry the current setup instructions. If you want to see the agent team that puts these tools to work, take a look at the aiee-team repo.

Underneath all of this — the native surface, the MCPs, the agents that wield them — sits a memory substrate that doesn’t execute anything but holds everything together: shared knowledge, agent episodes, personal workspace, compiled skills. Article 5 is what that substrate looks like, and how 20 specialist agents write to it without drowning the signal.

Further Reading

• Bilgin Ibryam — 12 Agentic Harness Patterns from the Claude Code Leak — practitioner analysis of the March 2026 source map; useful companion to the Native Execution Surface section.

• Anthropic — Introducing the Model Context Protocol — origin announcement (November 2024) for readers who want the protocol’s history.

Series: Cognitive Domain Engineering

1. Cognitive Domain Engineering — A Framework for Self-Improving AI Systems

2. Information Architecture — Beyond Context Engineering

3. The Verification Chain Your SWE Agents Need

4. A modern execution stack for your SWE Agents: MCPs and native tools ← you are here

5. The Human as Training Signal (coming soon)

6. From Amnesia to Adaptation (coming soon)

Until very recently, building software required knowing how to write it. That changed fast. Today anyone who can articulate a product need in plain English can ship a working application, and millions are doing exactly that. The tools deliver: the code compiles, the tests pass, the demo works. The confusion starts when “it works on my laptop” meets real users, real traffic, and real attackers. Veracode found that 45% of applications containing AI-generated code carry at least one OWASP Top 10 vulnerability.¹ The code works. Whether it’s ready for production is a different question.

The tools write code that works. Whether that code is secure, scalable, and production-ready requires a verification layer that most workflows skip entirely. The gap is not in the generation. It is in the quality verification.

My own experience confirmed it. When I moved from generating code in a chat window to full agentic coding with Cursor at the end of 2024 and Claude Code at the beginning of 2025, I was blown away by how efficiently they could produce compiling code. But upon looking closely, any experienced engineer would have seen it: most of the code was overly complex for the tasks at hand, and the tests were tautological, raised to hit coverage in blind optimization of the only signal they had. The question was how. How do you encode “this test is meaningless” or “this architecture is overkill” into something a machine can evaluate? That encoding is what this article addresses.

Domain expertise is only useful if it is deterministically verifiable.

Quality gates don’t operate as single checks. They compose into layers, each catching a different class of failure.

Building the Verification Chain

In every human cognitive domain, outputs are verified before acting on them. Or at least we hope so (Article 1). Around the time this article was published, Artemis II launched: every component that carries astronauts to the Moon passed deterministic, exhaustive verification. Every bolt, every weld, every line of flight software. That is not randomness. That is engineering. The same principle scales down to every domain where outputs matter: the verification is either built into the process, or it is missing entirely.

In software engineering, the quality verification chain has traditionally been owned by human engineers: writing tests, running linters, checking types, reviewing diffs. Unit tests sit at the foundation of that chain. A well-designed test suite is a deterministic quality gate that lives alongside the code, runs on demand, and produces binary pass/fail outcomes any engineer or agent can interpret. With AI generating code at the speed and volume it does today, solid test coverage moves from good engineering practice to structural necessity. Without it, there is nothing to verify against. That is why 80% coverage is the minimum the chain enforces, and why it sits at the base rather than at the end.

A command file, a Justfile or a Makefile, encodes the verification chain as a single executable sequence that both human and synthetic engineers run identically, every time. Ours makes the complete validation pipeline one command, just validate-branch, covering format, lint, type-check, and test with coverage enforcement. The project’s CLAUDE.md instructs agents to run this validation before every commit. The agent writes code, runs the gates, reads the failures, fixes, and re-runs, an autonomous feedback loop that resolves most issues before a human ever sees the diff.

Pre-commit hooks are the safety net. If an engineer skips the validation step, or if an agent’s self-assessment drifts, the same pipeline runs automatically before every commit. A commit that fails format, lint, type-check, or test coverage never reaches the repository. The last deterministic checkpoint before code leaves the machine.

GitHub CI re-runs the identical deterministic checks on the repository’s own environment, catching the class of failure that passed on a developer’s machine and breaks on clean infrastructure.

The only reliable way to get quality from AI agents is to verify everything they produce, exhaustively and automatically.

What remains for the human at the end of the deterministic chain is the decision to push. The format is clean, the types check, the tests pass, coverage clears. The code is structurally sound. Whether it is semantically correct is the next section’s question.

Gates earn adoption through graduated severity. Phase 0 emits warnings only: surface the gap, orient the team to what the gate sees, let the organization understand the standard before enforcement begins. Phase 1 enforces CRITICAL findings. Phase 2 escalates to CRITICAL and HIGH. Demonstration first, mandate second.

With this chain in place, no agent commits code that fails tests, breaks type safety, or ships unformatted. No human needs to check those things manually. That alone is a significant upgrade. The question that remains: how do you verify that the tests are meaningful, or that the architecture is sound?

When Passing Tests Isn’t Enough

That judgment requires understanding the logic, reasoning about what the code is supposed to do and whether it does it well. Until recently, only human engineers could provide it, and they provided it at one moment every developer knows: the pull request review.

The claude-code-action mechanism bridges this gap at the pull request. It is a GitHub Action that triggers AI review on every PR. Set it up with claude /install-github-app or add the workflow YAML directly. Out of the box it reviews for code quality, bugs, performance, and security concerns. Customize the prompt field in the workflow to encode your own review criteria, and the action loads your project’s CLAUDE.md as context, scoping the review to the behavioral rules Article 2 established. The need is quantifiable: Stanford found that 41% of Copilot-generated functions contained security flaws, against 21% without AI assistance.²

The pattern extends further. Domain specialist agents like the aiee-security-engineer and the aiee-python-expert-engineer load targeted review instructions and run dynamically during the implementation cycle itself, catching tautological tests, architectural issues, and security vulnerabilities before the code ever reaches a PR. That dynamic review system is a later article’s subject. The agents, their review instructions, and the workflows that orchestrate them are part of aiee-team, an early version of a synthetic engineering team, fully functional and installable as a Claude Code plugin.

These reviewers declare read-only tools: Read, Grep, Glob. They can examine everything and modify nothing. The constraint is a design choice: it separates a reviewer’s verdict from a reviewer’s preference.

What does this catch that the deterministic chain misses? Consider tautological tests: the test verifies that a mock was called, not that any real behavior occurred. Coverage reports 100%. Every linter passes. The code is structurally perfect and semantically empty. A reviewer built to recognize that pattern catches it. A deterministic pipeline never will.

These criteria work for code because they map to observable indicators. But what about domains where the expertise being encoded resists quantification entirely?

Quality Gates for Content

This article moved through the same gated process before publication. The gate looks different, content instead of code, but the encoding method is identical.

“Does this sound AI-generated?” lives entirely in expert intuition until you decompose it. Start with what specifically triggers that judgment, then make each trigger measurable. Hedging overuse becomes a count: how many softening phrases per paragraph. Uniform sentence length becomes a variance metric — real writers mix short and long, generators produce even rhythm. Voice absence becomes the hardest check: does the text contain specific lived experience, confident opinions, personality? That last one blocks publication outright because revision cannot manufacture what was never there.

Once each indicator is observable, the categories compose into a gate — a content quality reviewer built on the same read-only pattern as the code domain specialists, scoring Technical Credibility at 50%, Engagement at 30%, Readability at 20%. Credibility carries the most weight because it is the primary failure mode for technical content. Two domains, two sets of criteria, one architectural pattern.

If an expert can describe what wrong looks like, a machine can verify it.

What Gates Unlock

That automation only works if the gate speaks clearly when it fails. When the failure is specific, the system can self-correct. When failure is ambiguous, the retry is a guess. The difference between the two is the difference between a system that improves and one that drifts.

Silent degradation, an agent that keeps attempting and quietly accumulates errors, is more dangerous than a visible escalation. The gate that stops bad work is valuable. The gate that teaches the system what bad work looks like is transformative. How that teaching happens across sessions, through execution hooks and memory, is the subject of the next articles in this series.

Two pillars are now in place. The information architecture delivers the right knowledge to the right process. The quality gates verify the output. Together, they give reasoning engines the scaffolding to self-correct. Any structured cognitive domain where experts can articulate what “wrong” looks like is a domain where this architecture applies. The remaining pillars, execution infrastructure, human orchestration, and memory, are what turn this from a verification system into a self-improving one.

References

¹ Veracode 2025 State of Software Security, as reported by IT Pro

² Do Users Write More Insecure Code with AI Assistants?, Stanford Internet Observatory, ACM CCS 2024

³ aiee-team reference implementation — agents, skills, and quality-gated workflows

⁴ aiee-backend workflow — five-phase gated implementation workflow

⁵ dev-standards skill — just validate-branch deterministic gate

Series: Cognitive Domain Engineering

1. Cognitive Domain Engineering — A Framework for Self-Improving AI Systems

2. Information Architecture — Beyond Context Engineering

3. The Verification Chain Your SWE Agents Need ← you are here

4. Execution Infrastructure (coming soon)

5. The Human as Training Signal (coming soon)

6. From Amnesia to Adaptation (coming soon)

Since mid-2025, context engineering has been seen as the dominant frame for thinking about Foundation Model powered system performance. Andrej Karpathy gave it a name.¹ Tobi Lutke gave it authority.² The discourse moved fast, and the framing was right as far as it went.³ Context engineering optimizes what goes into the window. That is a real and valuable discipline. It is also only half the problem.

The other half is the routing layer context engineering cannot see. Before a single token enters the context window, a prior decision has already been made: which agent handles this request, which rules apply, which tools are available, and which knowledge gets loaded. That decision is not prompt craft. Context engineering has no vocabulary for it. One concrete example: when a task spans three services, the question of which agent should own the work, and what project conventions that agent needs to load, is decided before any prompt is written. That decision is either designed or improvised. One is prompt craft. The other is systems architecture. The gap between them is where quality degrades.

“Routing precedes reasoning. Without it, reasoning fails.”

That failure showed up in my own work before I had words for it.

Subscribe now

Agentic SWE in Practice

I started using Claude Code two weeks after it launched — roughly a year ago as of this writing. Before that, Cursor and Codex were solidly integrated into my workflows, pushing code to production under manual supervision. Claude Code felt different, the fact that it could run in any terminal process blew my mind, so I committed early and went deep. Within a few hours I had a CLAUDE.md file with a specific set of rules and conventions for each project. Then, when implementing features spanning multiple services started becoming more frequent, I ended up manually updating the same conventions across project after project. The files bloated more as systems evolved. Development tasks inevitably started loading information that had nothing to do with the work at hand. The context window filled with noise, and the reasoning — and therefore the quality of the outputs — degraded with it. That degradation is also a cost: context thrashing burns tokens, but more critically it burns quality. Any coding agent/process reasoning over the wrong information produces the wrong answer, and no prompt refinement fixes a routing failure or missing information. The same routing decisions were being retyped by hand every session, encoded nowhere, enforced by nothing.

That failure is architectural: the model had the capacity, but the information reaching it was wrong. Too much noise, missing conventions, no routing. Article 1 established the cross-domain pattern: experts route information before reasoning, whether they’re lawyers scoping jurisdiction or engineers loading PR history. The pre-reasoning assembly work is the same cognitive process across every domain. It is also invisible, unmeasured, and performed manually every time. Naming the gap is the first step toward engineering a solution.

Information Architecture is the engineering discipline that designs this routing. Richard Saul Wurman coined the term in 1975⁴ to describe the structural design of information for human navigation: organizing it so people can find and use what they need. The same discipline applies to systems that imitate cognition: the agent needs to find and use the right information too. We are not borrowing the term loosely. We are applying the original concept to a new class of system. The distinction from context engineering matters here. Context engineering optimizes what goes into the window. Information Architecture designs the system that decides what gets routed there, and when, and to whom.

From this article forward, we build in a specific domain: software engineering. Claude Code serves as the foundational framework: the platform on which we construct information architecture. That model has four layers: Rules → Roles → Abilities → Workflows. Each layer is a component of a working system. They nest. They compose.

The reference implementation — aiee-team — demonstrates all four in a public, inspectable plugin you can load into any Claude Code workspace.

The foundation of that four-layer system is the one that’s easiest to overlook. Before roles, before skills, before workflows — there are rules.

Behavioral Rules: The Invariant Foundation

Every agent you build inherits assumptions. The question is whether those assumptions are designed or accidental.

Behavioral rules are the foundation: always-on, unconditional, applied identically whether the executor is a human engineer or an AI agent, regardless of the task at hand. They operate as invariants: once defined, they cannot be overridden by context the way defaults can. Decision priority ordering, git conventions, validation requirements: these apply regardless of what the agent was called for, regardless of what the user asked. That universality is the point.

The aiee-team reference implementation demonstrates this directly. skills/dev-standards/ is the only skill loaded by every agent in the system. SKILL.md defines the core rules: business requirements take precedence. Match existing patterns before inventing new ones. No commits without explicit user approval. reference.md extends these with detailed standards. This skill is the invariant foundation the entire system rests on.

“Rules without enforcement are suggestions.”

Human engineers know this pattern by heart (at least the ones developing under good practices). Format before commit. Lint before merge. Type-check before deploy. Test before release. These are the same quality gates that run in every CI/CD pipeline — the gates that prevent a careless change from reaching production. AI agents inherit the same pipeline. The aiee-toolset justfile makes this operational: just validate-branch runs the full sequence — format, lint, type-check, test — in the same order a CI/CD system would execute it. Pre-commit hooks enforce the same gates automatically on every commit. 80% test coverage minimum. mypy strict mode. ruff formatting. The same standards a human engineer runs before submitting a PR, applied to every agent output identically.

Claude Code’s configuration architecture makes layering native. User-level rules in ~/.claude/CLAUDE.md apply across every project the system touches. Project-level CLAUDE.md files specialize for the domain. Session-level context scopes further to the immediate task. Define the invariants once, at the right level. Let specialization happen below.

When role boundaries are undefined, the rules have nowhere to land. Every agent inherits the same behavior — not because the work demands it, but because there’s nothing to distinguish one from another. That’s the failure the second layer prevents.

Role Boundaries: Agents as Information Surfaces

Eric Evans introduced the Bounded Context in Domain-Driven Design⁵ to solve a specific problem: the same word means different things in different parts of a system. The fix was a boundary: a defined scope within which terms, rules, and models apply consistently. Cross that boundary, and you need an explicit translation.

The same problem appears in multi-agent systems. The fix is identical to Evans’s: define the boundary first. Agents are information boundaries, not capability configurations. An agent definition doesn’t control what the foundational framework can do. It controls what information reaches it.

The aiee-team public repository demonstrates this with three agents that share a single underlying model. agents/aiee-backend-engineer.md routes Python patterns, DDD principles, FastAPI conventions, and database modeling standards. agents/aiee-frontend-engineer.md routes Angular 21+ signals, standalone component patterns, and accessibility requirements. agents/aiee-security-engineer.md routes OWASP threat modeling frameworks, SOC 2 compliance controls, and GDPR privacy patterns. The full aiee-team plugin defines seven agents: seven distinct information surfaces from one system.

The mechanism is explicit in each agent definition. A skills: field in the frontmatter declares which knowledge packages reach this agent. A “When to Call” section tells the orchestrator which task types belong here. A “NOT For” section states the explicit exclusions. Every field in an agent definition is a routing rule. Read one and you know exactly what information will reach the reasoning engine when that agent is invoked.

“A Bounded Context delimits the applicability of a particular model.” — Eric Evans, Domain-Driven Design

A trained language model is exactly that: a bounded context.

Roles define information boundaries. But a boundary is only as useful as what fills it.

Specialized Abilities: Knowledge That Travels

That’s where skills come in.

Skills are reusable knowledge packages: domain expertise that can live inside a single role boundary or travel across several. unit-test-standards is a domain concern that the backend engineer, the frontend engineer, and the security engineer each load. Same knowledge package; three different information surfaces. Skills travel. Agent definitions stay put.

Progressive disclosure⁶ governs how skills are structured. SKILL.md is required: the mental model, decision criteria, and key patterns. reference.md is optional: detailed standards and edge cases, retrieved when the task demands it. Claude Code handles this natively: when an agent is invoked, it loads the SKILL.md summary automatically and retrieves the full reference only when the task requires it. dev-standards implements both tiers: 800 to 1,500 lines of deep reference available on demand, the summary always in context. Lite skills like compliance-frameworks carry SKILL.md alone: 150 to 300 lines of essential mental model without the full reference apparatus.

“The summary is always loaded. The detail is available on demand.”

The aiee-team public repository ships 15 skills organized across five domains: architecture standards (arch-python-modern, arch-angular-modern), language standards, infrastructure patterns, development practices, and accessibility. arch-python-modern is loaded by both the aiee-backend-engineer and the python-expert agent. Each skill is authored once and consumed wherever the domain knowledge applies.

An agent loaded with 15 complete skills at full depth would saturate its context before reasoning about the actual task. The same principle governing what reaches agents from the outside also governs what reaches agents from inside each skill.

Rules constrain. Roles scope. Abilities empower. The fourth layer does something different: it encodes not what agents know, but how they think.

Cognitive Workflows: Encoding How Experts Think

A cognitive workflow encodes the actual reasoning pattern of a skilled practitioner: when to run processes in parallel, when to enforce a gate before continuing, how to synthesize competing perspectives into a single coherent plan. The structure of the workflow is itself the design decision.

Because the orchestrator is itself a reasoning engine, cognitive workflows can dynamically select which specialists to invoke based on the nature of the task. The example below is static: the five phases and their participants are predefined. Dynamic orchestration, where the workflow adapts its structure to the problem, is the subject of a later article in this series. The aiee-team backend workflow makes this concrete.

commands/aiee-backend.md defines five phases:

• Phase 1: Implement. Produce the working code.

• Phase 2: Simultaneous expert review. A quality auditor examines code structure and maintainability, a Python expert validates type safety and modern idioms, a security engineer checks for vulnerabilities and OWASP compliance. All three reviewers run in parallel.

• Phase 3: Test enforcement gate. The work cannot advance without passing test coverage.

• Phase 4: Consolidation. The three review perspectives synthesize into a single coherent revision plan.

• Phase 5: Iteration. The cycle repeats up to three times before escalating.

Look at the full shape of the five phases. Phase 3 is a hard gate because some failures cannot be revised around. Phase 4 is synthesis because three reviewers produce three perspectives, not one. Phase 5 is bounded because iteration without a limit becomes drift. Each structural choice in the workflow encodes a specific judgment call.

That raises the question of where the four layers stop.

One Pattern at Every Scale

A directory structure is an information architecture. Every level of nesting is a layer boundary, and every CLAUDE.md is a behavioral contract. The same four-layer pattern that governs a single agent plugin governs an engineering hub spanning multiple clients and dozens of services.

Every directory level maps to a layer. Portfolio directories are role boundaries: agents working in Client A’s context see nothing of Client B’s. CLAUDE.md files at each level are behavioral rules, applied in cascade from hub to service. projects.yaml encodes routing knowledge: machine-readable decisions about which services handle which work. Sprint directories encode cognitive workflow state.

Master the four layers at the plugin level and you have a design pattern that scales to any organizational complexity without structural changes. One pattern governs an individual agent definition and an enterprise engineering hub with equal precision. The four layers don’t change at scale. The directories do.

What the four layers don’t answer is the next question: how do you know if it’s working? Structure without verification is a guess dressed as a system. Article 3 takes that question seriously.

References

¹ Andrej Karpathy, tweet on context engineering, June 2025

² Tobi Lutke, tweet on context engineering, June 2025

³ Effective Context Engineering for AI Agents, Anthropic Engineering, September 2025

⁴ Richard Saul Wurman coined “information architecture” at the 1975 AIA National Conference. See Wikipedia: Richard Saul Wurman

⁵ Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software, Addison-Wesley, 2003

⁶ Jakob Nielsen, Progressive Disclosure, Nielsen Norman Group, 2006

These professionals execute structured, repeatable cognitive processes that produce quality work:

1. Structure the right information for the task at hand

2. Evaluate output against domain-specific criteria

3. Use appropriate tools to execute the work

4. Maintain expert oversight to catch errors

5. Learn from outcomes to improve future performance

Today, these processes are driven entirely by humans. Engineers review code. Lawyers verify citations. Doctors check protocols. Financial analysts validate trades against risk thresholds. Teachers assess whether student work demonstrates understanding.

This is repeatable work. And repeatable work is automatable.

Subscribe now

The CDE Thesis

Given the right architecture, these cognitive processes can be automated, with humans remaining as supervisors rather than executors. This applies to structured cognitive work with deterministic verification criteria: domains where experts can articulate “good output” as testable rules. Creative synthesis, novel problem formulation, and judgment calls requiring ethical tradeoffs remain human domains. CDE automates the verification, not the vision.

The five pillars emerged from necessity, not vision. Once in place, the system worked.

This is production-tested. I solo-built a multi-service AI SaaS company (multiple backend services, a frontend dashboard, an AI assistant engine, and full cloud infrastructure) entirely through AI agents (botbrewers.ca). After three months of structured operation: 80% of database migrations require approval only at cutover (not per statement), deployment lead time dropped from 45 minutes of manual orchestration to 6 minutes of automated verification, and agent context reuse rose from 12% to 68% through Information Architecture improvements.

The same cognitive processes govern software engineering and apply across domains:

• A lawyer retrieving case precedent uses the same information structuring process as an engineer loading context for a code review

• A doctor evaluating treatment efficacy applies the same output validation process as a financial analyst checking trade compliance

• A teacher maintaining expert oversight of student work mirrors an engineer approving infrastructure changes

The difference between domains is what gets structured, what criteria define quality, which tools get used, what experts oversee, and what outcomes inform learning. The pattern is constant. These five processes map to five engineering pillars.

The Five Pillars

The framework decomposes human expertise into five engineering pillars. Each pillar targets a specific cognitive process: the mental work that domain experts perform to produce quality work.

Each pillar is also a functional component. Together they form a machine — the diagram above is its schematic. Requirements enter as raw material. Shipped product exits the other side. One component routes, one drives, one filters, one oversees, one learns. The machine is what all five become when operating together.

Pillar 1: Information Architecture

The intake manifold. Routes the right knowledge before the engine turns.

In the AIEE agent team, the backend engineer loads Python patterns, FastAPI architecture, and database schemas. The security engineer loads OWASP guidelines, threat models, and SOC 2 requirements. The frontend engineer loads Angular signals, Svelte components, and accessibility standards. Same codebase. Seven different information surfaces — each defined by what that agent needs to reason well⁵. That routing is Information Architecture.

Experts do this manually. A junior associate spends months learning which precedents apply in which jurisdiction. A doctor orders targeted tests based on symptoms before forming a diagnosis. Information Architecture encodes that routing so the system does it automatically — at the right time, without prompting.

Why it matters: The right context at the right time is the difference between an agent that reasons clearly and one that hallucinates. Article 2 builds this layer by layer: behavioral rules, role boundaries, reusable skills, and cognitive workflows.

Pillar 2: Quality Gates

The valve bank. Nothing bad advances.

Before any code reaches production in the Bot Brewers system, it passes through a five-layer verification chain: tests at ≥80% coverage, strict type checking, a security scan, a linting pass, and a read-only reviewer agent trained to detect tautological tests — tests that pass regardless of whether the code is actually correct. Each layer catches a different class of error. The composition is the insight, not any individual check.

Surgical teams use the same logic. Before an operation begins, every item on the WHO checklist gets verbally confirmed: correct patient, correct site, correct procedure, allergies cleared. Complications fell 36% and deaths fell 47% when this became standard⁶. The checklist doesn’t add surgical expertise — it makes verification deterministic.

Why it matters: Deterministic verification catches errors that expertise alone misses. Article 3 builds the full chain, including how to encode subjective criteria — like “does this read as AI-generated?” — into pass/fail gates.

Pillar 3: Execution Infrastructure

The engine block. The tools in motion.

Claude Code arrives with nine native tools: filesystem access, terminal, version control, language server, and search. MCP servers extend that surface. Playwright handles browser automation and visual regression testing. A direct database connection queries Postgres. GitHub manages pull requests. An agent can write code, run the test suite, verify the UI renders correctly in a browser, and open a PR — without switching tools or losing context. What used to require coordinating six separate systems becomes one workflow.

The coordination overhead in most domain work isn’t the expertise — it’s the tool-switching. A financial analyst moving between Bloomberg, spreadsheets, and compliance portals loses time to context-switching, not analysis. Execution Infrastructure eliminates that tax.

Why it matters: Unified execution lets agents act without coordination overhead. Article 4 maps the full execution surface: native tools, MCP servers, and the hooks that connect execution to verification.

Pillar 4: Human Orchestration

The operator console. Oversees, calibrates, decides when to ship.

In the Bot Brewers system, oversight operates at three levels. During development, every session runs through adaptive workflows — structured checkpoints where the agent proposes and the human approves before the work advances. Before anything merges, a human reviews the code at the PR level. Features that touch more than one service are integration-tested by a human before they reach production. The human doesn’t disappear — the role scales to the risk: in-session for decisions, PR-level for code, integration-level for cross-service changes.

Senior engineers apply the same logic with junior developers. They review every pull request. They test cross-service changes before they ship. The oversight is calibrated to the blast radius of the work, not distributed uniformly across every action.

Why it matters: Systems earn autonomy through demonstrated reliability. Article 5 shows how every human approval becomes a training signal that shapes future behavior.

Pillar 5: Memory & Adaptation

The ECU. The machine improves between runs.

The bb-backend-engineer agent records every meaningful session decision in an 800-token memory file: what it tried, what worked, what failed. Each week, the sync-learnings-agent reads those records, extracts durable patterns, and writes them into permanent skill files⁷. A session management bug found in December 2025 became a permanent rule in the arch-python-modern skill. Every agent that loads that skill won’t repeat the mistake. The system learns between sessions, not just within them.

Subscribe now

Human teams try to do this with postmortems and runbooks. The learning is inconsistent and disappears when people leave. Memory & Adaptation captures outcomes systematically and closes the loop between what the system does and what it learns from doing it.

Memory & Adaptation closes the loop: quality gates enable evaluation, and memory enables development. Together, they create systems that develop and evaluate themselves. Article 6 builds the full learning architecture, from episodic memory to the human-in-loop feedback cycle.

This is where human orchestration evolves from approval workflow to continuous calibration: reviewing aggregate outcomes, validating that the system’s improving judgment still reflects yours.

The Cross-Domain Pattern

These five pillars address the same cognitive work across every domain. Here’s how they manifest in law, medicine, finance, and software engineering today, and what CDE systematizes:

Sources: ¹ Legal Research Methods, Georgetown Law Library ² Clinical Reasoning in Medicine, NIH ³ Medication Safety Standards, ASHP Guidelines ⁴ Compliance Workflows in Financial Services, SEC Framework

Together, these five pillars form a unified framework for systematic cognitive automation. The synthesis required three conditions that didn’t coexist until recently: LLMs capable of judgment calls (not just pattern matching), tool-use frameworks that unify execution platforms, and deployment speed that makes iterative learning feasible⁸.

Share AI Enhanced Engineer

Why This Works Today

Years of engineering production-grade software provided the prerequisite expertise: defining what “good enough” means for a feature, deciding when to refactor versus ship, learning which architecture patterns prevented production fires. I needed quality assurance without a QA team, architecture decisions without a senior architect, deployment automation without a DevOps engineer. Knowing test-driven development prevented me from shipping authentication bugs that would have exposed customer data. Understanding eventual consistency kept me from building distributed race conditions into the assistant routing layer. The synthetic engineers needed my judgment on where complexity lives — that’s not knowledge you prompt-engineer into existence. I could drive a team of synthetic engineers because I knew where the road was.

Without that domain expertise orchestrating the system, you drive off a cliff.

Researchers have been chasing these problems for decades. Knowledge Engineering encoded expertise as rules constrained to known patterns⁹. RPA and Cognitive Automation automated deterministic tasks within defined boundaries¹⁰. AI Engineering, defined by practitioners like Chip Huyen, built model infrastructure — training pipelines, evaluation harnesses, deployment platforms — but operated at the wrong abstraction level. These fields each captured one aspect: rules, automation, or models. None targeted the cognitive layer where humans structure information, apply quality gates, orchestrate tools, maintain oversight, and learn from outcomes.

CDE targets what these fields all assumed was human: the cognitive layer where knowledge work happens.

Subscribe now

The insight: cognitive processes are the same across domains. What differs is the content, not the pattern.

The process pattern is universal. The implementations are domain-specific.

The framework synthesizes what prior fields attempted separately. Knowledge Engineering’s rules become adaptive feedback loops. RPA’s automation extends into judgment calls. AI Engineering’s model infrastructure gains structure for domain knowledge, quality standards, and production learning. The synthesis targets how domain expertise gets encoded, validated, executed, supervised, and improved.

The five pillars aren’t theory. They’re survival patterns from production reality.

Our Laboratory

Software engineering serves as my proving ground for Cognitive Domain Engineering. The system I described earlier, which now runs botbrewers.ca, operates with 40 specialized agents, 107 reusable skills, episodic memory that spans 7-day work cycles, and quality gates that enforce production standards before any code reaches customers.

While the Bot Brewers agent team and workflows remain private, the AIEE (AI-Enhanced Engineering) implementation demonstrates the same patterns publicly. Our agent team (specialists for architecture, backend, frontend, data, AI, and DevOps work) exemplifies the Information Architecture pillar (aiee.io/team). Our multi-phase workflows with approval gates illustrate the Human Orchestration pillar (aiee.io/workflows). These aren’t just documentation. They’re the operational system I use for technical content production.

The examples throughout this series draw from this implementation because software engineering offers something most domains lack: immediate feedback on failure. A broken authentication service announces itself instantly. A hallucinating legal AI might go undetected for months.

Software’s unforgiving nature makes it an ideal testbed for frameworks that claim to automate cognitive work.

The principles transcend their testing ground. The five pillars (Information Architecture, Quality Gates, Execution Infrastructure, Human Orchestration, Memory & Adaptation) apply anywhere humans perform structured cognitive work. A radiologist reviewing medical images follows the same pattern as a software architect reviewing system designs: ingest information, apply expertise, execute judgment, escalate edge cases, refine approach from outcomes.

This eight-article series unpacks the framework methodically. Articles 2 through 6 build each pillar with production-tested patterns from the software implementation: how to structure domain knowledge so agents can navigate it, where to place quality gates that catch errors without strangling velocity, what makes execution infrastructure reliable under production load, how human oversight scales beyond micromanagement, and how systems learn from their own operation. Article 7 examines why capable tools routinely make teams slower — the capability trap that derails most AI adoption efforts. Article 8 closes with what actually works in production.

By the series end, you’ll have a repeatable framework for identifying which cognitive processes in your domain can be automated, what infrastructure they require, and how to deploy them without triggering organizational antibodies.

References

⁵ Effective Context Engineering for AI Agents, Anthropic Engineering

⁶ A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population, Haynes et al., NEJM 2009

⁷ Memory in the Age of AI Agents (arXiv 2512.13564)

⁸ 2026 Agentic Coding Trends Report, Anthropic

⁹ Expert Systems: What Can We Learn from its Rise and Fall, The Unscalable Blog

¹⁰ RPA and Agentic AI: A Transformational Shift

This Series: Cognitive Domain Engineering

1. Cognitive Domain Engineering — A Blueprint for Self-Improving AI Systems ← you are here

2. Information Architecture — Beyond Context Engineering

3. The Verification Chain Your SWE Agents Need

4. SWE Agents in Motion (coming soon)

5. The Human as Training Signal (coming soon)

6. From Amnesia to Adaptation (coming soon)

7. The Capability Trap (coming soon)

8. What Actually Works in Production (coming soon)

After building aiee.io, botbrewers.ca, and several others this year, we noticed the same pattern repeating. Whether we used AI tools to vibe-code them or built them carefully by hand, the process was the same: set up the structure, customize the theme, validate responsive design, deploy. The initial architecture didn’t change much site-to-site.

So we built a template for ourselves. It automates the repetitive parts and lets us focus on what makes each site unique. We’re sharing it now because it might be useful if you’re building similar sites.

The template works with Claude Code, leveraging a subsystem of agents with specific skills to execute development workflows. These workflows (setup, customization, deployment) are encoded in natural language within the template itself. Here’s what that looks like:

Behind that output, the template just executed its own development workflow. Type /create-site, and the orchestrator runs a command that gathers details through interactive questions, then delegates to web-dev-agent for implementation and web-qa-agent for validation. The workflow asks you to choose a color theme and provide content, then the dev agent applies those choices as CSS variables and HTML structure. The QA agent validates accessibility and responsive design before preview. You describe what you want; the system develops and validates itself.

It’s a self-developing system: templates that know how to modify themselves.

Three Components That Make It Work

That three-minute build involves three layers working together: an orchestrator, skills, and agents.

The Orchestrator (.claude/CLAUDE.md) routes requests to the appropriate handler based on keywords and intent. Type /create-site and it invokes the website-setup skill. Type “deploy” and it invokes deployment-guide. Ask for custom changes like “add testimonials” and it calls web-dev-agent. The routing is explicit and debuggable. Each request maps to a specific skill or agent.

The Skills (.claude/skills/) handle common workflows. Six skills cover development and quality:

• website-setup - Interactive wizard for initial site creation

• color-theming - Brand customization with color picker

• deployment-guide - Walks you through SiteGround deployment

• accessibility-check - WCAG 2.1 AA validation

• seo-validation - Checks SEO and meta tags

• visual-test - Responsive design validation

Each skill is a focused conversation flow that asks questions, validates inputs, and hands off implementation to the agents.

The Agents (.claude/agents/) implement and review changes. Two agents coordinate:

• web-dev-agent - Implements HTML/CSS/JS changes

• web-qa-agent - Reviews for accessibility, SEO, performance

When you ask to “add a testimonials section”, the dev agent scans for structural patterns, generates the code, then the QA agent validates it before showing you the preview. Every change goes through this workflow: implementation → quality review → preview.

We deployed a site with broken mobile nav once. That’s when we added QA to the workflow.

After rebuilding similar patterns across aiee.io, botbrewers.ca, and metamathematics.ai, we realized we were spending 80% of our time on repetitive setup. Now we spend about 20% of our time developing the specialists that develop the solutions. The agents and skills encode those patterns so the template can apply them automatically.

Two Ways to Use This

This isn’t Webflow or a static site generator—you own the codebase and develop through conversation.

Click “Use this template” on GitHub to create your repository, clone it locally, then run claude. Three minutes later, you have a working site. No configuration files, no build steps, just conversation with Claude.

Build a site today. Three commands get you a working site. Describe what you want in conversation, make changes the same way. Works for static sites (no databases needed). Everything happens through natural language.

Learn the pattern for your own projects. The orchestrator shows real request routing in action. Read the agent prompts to understand how they preserve design patterns while implementing changes. Then apply what you learned to your own repeatable project types.

What This Won’t Build

• Complex applications with databases or backend logic

• E-commerce platforms with payment processing

• Sites requiring server-side authentication

• Prerequisites: Claude Code, basic understanding of web development

• Best for: Landing pages, portfolios, documentation sites, static marketing sites

Whether you’re shipping today or learning for tomorrow, the code is documented and ready.

Try It Yourself

You can build and deploy a production website by describing it in three sentences.

Here’s what that looks like:

1. Click “Use this template” at the repository

2. Create your new repository on GitHub

3. Clone and launch:

Type your idea, hit enter, and it starts asking questions. “What’s your site about?” “Who’s your audience?” “Brand colors?” The agent uses these answers as hard design constraints—vague inputs produce generic sites. Then the QA agent validates accessibility, SEO, and responsive design before showing you the preview at localhost:5174.

The QA agent catches real problems. When we tested this with a dark blue on black color scheme, it immediately flagged contrast failures that would have violated WCAG standards. The same test caught a layout break on mobile at 375px width. Missing alt text, layout breaks on small screens, each error with a line number and the specific standard it violated. Not vague warnings, actionable fixes.

Need a pricing section? Say what you want:

The dev agent builds it, the QA agent validates it, then you see the preview update. Every change (typo fixes, layout tweaks, new sections) goes through the same cycle. No broken states, no “forgot to test mobile” surprises.

Ready to deploy? Type “deploy” and the deployment-guide skill walks you through the GitHub Actions setup. Sub-minute deploys to SiteGround via rsync (after GitHub Actions builds).

The entire development workflow happens through conversation. Describe what you want and the agents handle implementation.

Beyond Websites

The same pattern handles backend APIs, data pipelines, and microservices.

Swap the domain-specific skills, keep the orchestrator. We tested this with a Python API template—ported the orchestrator in an afternoon. Swapped color-theming for endpoint-design, updated the agent’s FastAPI knowledge, and the rest just worked. Same coordination logic, different domain.

The pattern applies to repeatable projects:

• Web development (this template)

• Backend services (Python API template)

• Data pipelines (analytics template)

• Microservices (GCP template)

We built the website template first because it’s the clearest demonstration. Other project types use the same orchestrator pattern with domain-specific agents. We’re currently building autonomous templates for backend Python APIs and frontend Angular applications—same orchestration pattern, different domains.

Each template implements the orchestration patterns from Mastering Claude Code (multi-agent coordination) and AI Agents in Production (safe automation workflows). Click “Use this template” to create your own repo and see those patterns in production-ready code.

Use the template to create your repo, clone it locally, run claude, and watch the system build through conversation. Open .claude/ to see orchestrator, skills, and agents working together.

Use it to ship a website today. Study it to build your own autonomous systems tomorrow.

Sources

• Autonomous Website Template Repository

• AI Agents in Production Series - Agent architecture patterns

Agent testing breaks traditional assumptions. A production agent answering a single user query might make 7 sequential tool calls: search the knowledge base, retrieve pricing data, calculate discounts, verify inventory, check shipping rates, apply promotions, format the response. Run 10,000 tests in CI, and you’ve made 70,000 API calls. At $0.03 per reasoning step, that’s $2,100 per test suite execution.

Beyond cost, agents introduce a deeper complexity: non-deterministic execution paths. Each test run triggers the agent’s “Reasoning” loop: think about the query, select a tool, observe the result, think again, select another tool, repeat until done.

The path from query to answer isn’t predetermined: it emerges from the reasoning loop. Same query, different runs, different tool sequences.

Traditional software follows deterministic paths. Function A calls function B, which returns value C. You test A with mocked B, verify C appears correctly. Agents reason through possibilities, selecting each tool based on observations from prior steps.

Subscribe now

The Multi-Step Testing Challenge

Consider this minimal ReAct agent in LlamaIndex:

This agent might execute:

Same query, different paths, all producing valid answers. Now multiply by 10,000 test runs. At roughly $0.03 per reasoning step (averaging 500 input + 200 output tokens with GPT-4), those 3-4 steps cost $900-$1,200 per test suite execution. Run tests on every commit, several times daily across a team, and you’re burning thousands monthly—just on tests.

The traditional response is “mock your external dependencies.” Sure, mock the search_kb tool so it returns canned data. Mock get_pricing to return test values. You still need the LLM to decide which tools to call and when to stop. Without the reasoning engine, you can’t test the agent loop itself. You’re testing individual tools in isolation, not the multi-step orchestration that makes agents different from simple function calls.

This is the agent testing paradox: agents are valuable because they adapt their behavior based on observations, but that same adaptability makes deterministic testing expensive. You need confidence that your agent selects the right tools, handles errors gracefully, and terminates appropriately—without spending hundreds per test run.

That’s what reasoning trajectory mocking provides. But before we implement that pattern, we need to clarify what kind of tests we’re actually writing: because agents need two fundamentally different testing strategies.

Deterministic Tests vs Quality Evaluations

Agent systems are hybrid: deterministic software orchestrating non-deterministic LLM outputs. Each half needs its own testing strategy.

Deterministic Testing

When you mock the LLM and verify your agent calls the right tool with the right parameters? That’s deterministic testing. Unit tests check individual tool invocations and error handling. Integration tests verify service boundaries and API contracts. Same input, same output—every time. These run on every commit in CI/CD. This is the Test-Driven Development (TDD) world engineers know.

Quality & Safety Evaluations

These test the content the LLM produces for human consumption. Quality asks: Is this response helpful, accurate, well-formatted? Safety asks: Is it harmful, biased, or policy-violating?

Evaluations run in three modes:

1. Automatic evaluations: Built alongside the system, run continuously

2. Production monitoring: Sample live traffic, detect drift

3. Human SME review: Periodic expert assessment, discover edge cases

Evaluation-Driven Data Science (EDDS)

Just as TDD writes tests before code, EDDS defines evaluations before prompts. The workflow:

1. Design quality criteria (e.g., “answers cite specific policy sections”)

2. Build evaluation suite with curated examples

3. Iterate on prompts until scores improve

4. Deploy and monitor for drift

Hamel Husain’s “Your AI Product Needs Evals“ (2024) and Eugene Yan’s “Evaluation-Driven Development“ (2023) formalized this practice for production LLM systems. The parallel to TDD is exact: write your quality assertions first, then tune your system to pass them.

This article focuses on deterministic tests—the ones that validate software behavior and run on every commit. We’ll expand on quality evaluations and EDDS in a future article. Right now, we need to solve the testing paradox: how do we steer agent behavior without burning $900 per test run?

Reasoning Trajectory Mocking: Controlling the Reasoning Loop

A ReAct agent executing “What’s the refund policy?” makes a sequence of decisions: think → search → observe → think → respond. Traditional mocks can’t capture this behavior—they handle single calls, not multi-step paths.

Reasoning trajectory mocking pre-defines the exact sequence of LLM responses, letting you steer the entire path. Instead of mocking one API call, you mock the agent’s decision chain:

Each string in the chain corresponds to one decision step. The mock returns responses in order, advancing the agent along its pre-defined path. This maps directly to LlamaIndex’s ReActAgent protocol, where each response contains a thought-action pair.

Here’s a complete test defining the full reasoning loop:

The test defines the exact trajectory the agent will take. Step 1 triggers the search_kb tool. Step 2 uses the tool’s output to generate the final answer. You define what the agent “thinks” at each point along the path.

This same pattern scales to more complex workflows. A data analysis agent running query_database → parse_results → calculate_statistics → generate_chart follows the same structure: each chain entry defines one step, and the test verifies the exact sequence executed.

Cross-Framework Testing Patterns

This mocking pattern works across all three major agent frameworks. The concept stays the same—steering multi-step agent behavior—but neither LlamaIndex nor LangChain provide trajectory mocking out of the box. We built MockLLMWithChain and MockChatModelWithChain to fill this gap. PydanticAI is the exception: it ships with TestModel for structured output testing.

LlamaIndex requires a custom mock. We built MockLLMWithChain implementing the full LLM interface with streaming support:

LangGraph also lacks built-in trajectory mocking. We built MockChatModelWithChain extending LangChain's BaseChatModel with automatic tool call parsing:

Both custom mocks parse ReAct-style responses automatically. MockLLMWithChain includes streaming support for testing async chat interfaces where character-by-character rendering affects UX. MockChatModelWithChain converts Action patterns into native LangChain tool calls.

PydanticAI ships with built-in testing support. Its native TestModel handles structured output validation:

PydanticAI bypasses free-text parsing entirely. This approach shines when testing structured data extraction—no ReAct formatting, just type-checked outputs matching your Pydantic schema.

The harness components from Article 2.1 work identically across all three: planning loops, tool registries, and memory management. The mocks change. The architecture stays constant.

Testing Tool Invocations

Calling delete_user instead of update_user could destroy production data—syntactic correctness won’t save you.

Tool invocation tests verify three critical properties:

1. Tool selection - Did the agent choose the right tool?

2. Parameters - Did it pass correct arguments?

3. Call sequence - Did it use tools in the right order?

Here’s a basic tool invocation test:

Multi-step tool sequences require ordering validation:

Choose your testing pattern based on what you need to verify:

Semantic Assertions for Natural Language Arguments

LLMs rephrase tool arguments constantly, breaking exact string matching. An agent might search for “refund policy” or “policy for refunds”—semantically identical, but == fails. Semantic similarity provides robustness.

Use semantic assertions when:

• Tool arguments contain natural language queries

• Meaning matters more than exact wording

Stick with exact matching for structured data (IDs, numbers), code, or compliance-critical language.

These patterns test the “execute tools” pillar from Part 2. Tools don’t always succeed though. Let’s test what happens when they fail.

Testing Error Recovery

Production systems fail constantly. Search APIs timeout, rate limits kick in, and responses come back malformed. Production systems see tool failures in roughly 12% of agent interactions. Without proper handling, 68% result in confusing responses or infinite loops.

Trajectory mocking lets you inject failures at specific points and verify graceful degradation. You specify when tools fail, how they fail, and whether the agent adapts appropriately.

Tool Failure Injection

This test simulates a realistic failure scenario. The primary search times out, the agent recognizes the failure in its reasoning chain, and recovers using the fallback tool rather than crashing or looping.

Graceful Degradation

You’re testing the safety layer from Part 2: graceful degradation under cascading failures. The agent encounters rate limits, empty results, and finally succeeds with partial data. It methodically tries alternatives and returns the best available answer.

Error scenarios to test explicitly:

• Timeout: Operation exceeds time limit

• API error: 429, 500, 503 responses

• Empty result: Valid response, no data

• Exception: Unexpected failures

• Cascading failures: Multiple tools fail sequentially

Failure handling determines whether your agent helps users or frustrates them. Treat it as a core feature. Test it explicitly.

Implementation Roadmap

Start with your existing agents and build coverage incrementally.

Phase 1: Establish Baseline

Map what you already have:

• [ ] Identify agents using tools (grep for ReActAgent.from_tools or similar)

• [ ] Run agents with verbose logging to trace actual reasoning chains

• [ ] Document 3-5 most common tool sequences (search → retrieve → format)

Phase 2: Build Happy Path Coverage

• [ ] Write tool invocation tests for each tool your agents use

• [ ] Create chain fixtures for common behaviors (target 80% of observed sequences)

• [ ] Validate mocked chains produce expected final outputs

Phase 3: Add Error Resilience

• [ ] Add error injection tests (API failures, timeouts, invalid tool args)

• [ ] Test recovery paths: Does the agent retry? Fall back? Abort gracefully?

• [ ] Create fixtures for error scenarios

Phase 4: Measure Impact

• [ ] Track coverage: % of observed sequences tested vs seen in production

• [ ] Calculate cost savings: test_runs × avoided_API_calls × $0.03/call

• [ ] Document edge cases discovered during testing

Reference Implementation: See the complete agent testing suite with examples for LlamaIndex, LangGraph, and PydanticAI.

Testing agent reasoning doesn’t mean running expensive models on every commit. With trajectory mocking, you control thoughts, actions, and observations at each step—catching infinite loops before production, validating tool selection without hitting real APIs, and verifying error recovery without waiting for actual failures. The difference between a $2,100 test suite and a zero-cost test suite is precise control over multi-step agent behavior.

Next: This article covered deterministic testing fundamentals: reasoning trajectory mocking, tool invocation testing, and error recovery. Article 3.1 goes deeper into advanced patterns—trajectory validation, state machine testing, memory retention, and regression testing. Together, they give you complete coverage of the six canonical harness components from Article 2.1.

AI Agents in Production Series

1. The Foundations

2. The Three-Layer Architecture: The Harness, the Model, and the Loop

   1. The Harness Dissected: Inside the Agent Execution Engine

3. Deterministic Testing with Trajectory Mocking ← You are here

4. LlamaIndex vs PydanticAI vs LangGraph

5. Agent Observability: Traces, Evals, Alerts

6. 5 Agentic Design Patterns That Actually Work

It’s 2am and your agent is failing. You’re debugging, but which component broke? The reasoning engine misinterpreted context? Tool registry failed validation? State persistence corrupted between requests? You can’t troubleshoot an abstract “harness.” You need to know what’s inside.

Article 2 established the harness as the middle layer between your UI and the LLM. Now we’ll examine what you’re actually building: six canonical components that appear across every single implementation weather using an established framework or building from the ground up. The reasoning engine, planning & orchestration, tool registry, memory & context, state & persistence, and structured I/O. These aren’t framework features—they’re architectural responsibilities you’ll handle regardless of technology choices.

Understanding these components helps you isolate failures faster, evaluate frameworks better, and make informed tradeoffs between flexibility and complexity.

Let’s start with the component that makes all decisions: the reasoning engine.

Subscribe now

The Six Canonical Harness Components

1. Reasoning Engine

The reasoning engine is the LLM at the heart of your agent—the component that processes context, makes decisions, and generates responses. It’s what transforms user requests into tool calls, evaluates observations, and determines next steps.

The key architectural choice: harnesses are model-agnostic. You can swap OpenAI for Anthropic, Gemini for local models, without touching your agent logic. This abstraction matters in production, where model selection becomes a configuration decision driven by cost, latency, and capability requirements.

2. Planning & Orchestration

This component implements the agent loop: gather context → take action → verify work → repeat. It’s where “tools in a loop” becomes executable code.

The loop terminates based on stopping conditions: the agent declares completion, hits max_iterations, exhausts a cost budget, or exceeds error thresholds. This orchestration layer separates workflows (predetermined steps) from agents (dynamic decision-making). Production planning components also implement task decomposition—breaking complex goals into manageable subtasks that the reasoning engine can sequence adaptively.

3. Tool Registry

The tool registry manages the agent’s capabilities: registration, schema validation, execution, and error handling. Tool descriptions are critical—they’re what the LLM reads when deciding which capability to invoke.

Frameworks provide built-in tools (web search, document retrieval), but custom tools are where agents become domain-specific. The registry validates parameters before execution and catches runtime errors without crashing the agent loop. In production, tool execution often includes authorization checks—ensuring read-only operations by default and gating write operations behind explicit approval.

4. Memory & Context

Memory defines what the agent “knows” during execution. Working memory lives in the context window—the conversation history, system prompts, and accumulated observations. Long-term memory extends beyond the context window using RAG or vector stores.

Context accumulation creates a constraint: LLMs have token limits. When context approaches capacity, production harnesses implement compaction—automatic summarization of older messages to preserve relevance while staying within limits. Modern harnesses also support agentic search: loading context on demand rather than pre-loading everything. The agent decides when additional context is needed, retrieves it via tools, and incorporates it dynamically. This pattern reduces wasted tokens and improves reasoning quality by surfacing relevant information precisely when needed.

LlamaIndex has very complete memory apis, see here.

5. State & Persistence

Production agents need checkpointing—the ability to save execution state and resume after failures or interruptions. This component handles state snapshots, thread management, and recovery.

Thread management enables multiple concurrent conversations per agent—each user maintains independent state. Without persistence, a crash means starting over. Long-running tasks (report generation, multi-step analysis) become unreliable. Checkpointing also enables human-in-the-loop workflows: the agent pauses for approval, persists its state, and resumes when authorized.

6. Structured I/O

Structured I/O enforces type safety on agent inputs and outputs. The Pydantic pattern is increasingly standard across frameworks—define schemas, validate automatically, retry on failures.

When validation fails, frameworks retry automatically up to configured limits. This catches malformed outputs—missing fields, wrong types, hallucinated keys—before they reach your application. Structured outputs also enable composition: one agent’s validated output becomes another’s typed input, creating reliable multi-agent pipelines.

Observability as Feedback Loop

Observability isn’t a separate seventh component alongside the reasoning engine, planning, tool registry, memory, state, and structured I/O—it’s the nervous system that connects them all. OpenTelemetry frames it clearly: “Observability is an integral capability used as a feedback loop.” You don’t build a separate observability layer. You instrument each component to understand what’s happening inside the agent.

Instrument each component to track the signals that reveal agent behavior:

This telemetry gives you the traditional observability pillars: logging, metrics, and tracing. The non-deterministic nature of agents means you can’t debug with breakpoints alone.

High tool failure rates signal poor instruction design. Repeated validation errors reveal schema mismatches. Token spikes expose inefficient prompts. You have the possibility of building a dataset that drives continuous improvement through data, not guesswork. And this is the ultimate goal for a healthy self-improving system.

Production Checklist

You’ve built a harness. You’ve wired up its components. You’ve probably tested it against a happy-path query.

Now comes the hard part: hardening each piece so it survives contact with production. Walk through each component and verify the item applies to your setup. If it doesn’t exist yet, add it before you push.

Reasoning Engine:

• [ ] Model selection matches task complexity (GPT-4 for planning, 3.5 for simple lookups)

• [ ] Token limits configured per request (context + completion)

• [ ] Fallback model defined for rate limit or availability errors

Planning & Orchestration:

• [ ] max_iterations set to prevent runaway loops (10-15 typical)

• [ ] Cost budget enforced per session (total token spend)

• [ ] Timeout configured per iteration (30-60s)

• [ ] Early exit condition defined (goal reached, confidence threshold)

Tool Registry:

• [ ] Read-only tools by default (searches, lookups)

• [ ] Write operations gated behind approval or dry-run mode

• [ ] Tool execution timeout enforced (no hanging database queries)

• [ ] Tool error handling returns structured failures (not raw exceptions)

Memory & Context:

• [ ] Compaction threshold configured (summarize after N messages)

• [ ] PII scrubbing applied before logging conversation state

• [ ] Conversation length limit enforced (prevent unbounded context growth)

State & Persistence:

• [ ] Checkpointing enabled (save state after each iteration)

• [ ] Recovery path tested (resume from checkpoint after failure)

Structured I/O:

• [ ] Output schemas defined using Pydantic models

• [ ] Retry limits set for malformed LLM outputs (3 attempts typical)

• [ ] Fallback behavior defined when schema validation fails repeatedly

Observability:

• [ ] Logging enabled at each component (LLM calls, tool invocations, state transitions)

• [ ] Distributed tracing configured across tool calls and reasoning loops

• [ ] Cost and latency metrics collected per session

• [ ] Error rate alerting configured (when failure rate exceeds threshold)

If every box is checked, your harness is production-ready. If not, you know where to add guardrails next.

What’s Next

You now have the blueprint. Six canonical harness components—reasoning, planning, tools, memory, state, and I/O—provide the execution infrastructure every agent needs. Observability weaves through each, surfacing what reasoning occurs, why plans shift, and how tools perform.

These patterns transfer across frameworks, whether you’re in LangGraph, LlamaIndex workflows, or PydanticAI.

But understanding the harness creates a new problem: how do you test a system where execution paths change based on LLM decisions? In the next article, we solve this with Chain Based Mocking, bringing deterministic testing to non-deterministic reasoning. You’ll control exactly what the LLM returns, validate harness behavior end-to-end, and build confidence without burning budget on live API calls.

References

• Anthropic: Building Effective Agents

• OpenTelemetry: AI Agent Observability

• Microsoft: Semantic Kernel Agent Architecture

• arXiv: Agentic AI Frameworks Survey

AI Agents in Production Series

1. AI Agents in Production: The Foundations

2. The Three-Layer Architecture: The Harness, the Model, and the Loop

3. The Harness Dissected: Inside the Agent Execution Engine ← You are here

4. Testing Agent Reasoning with Mock Chains

5. LlamaIndex vs PydanticAI vs LangGraph

6. Agent Observability: Traces, Evals, Alerts

7. 5 Agentic Design Patterns That Actually Work

The theoretical definition was crystal and clear: a Foundation Model (LM) that runs tools in a loop to achieve a goal. The implementation taught me where that simplicity breaks down. Where does the loop logic live? How do I pass state between tool calls without serializing everything to JSON and praying? What happens when the model halts instead of calling the tool I expected, and how do I test this without burning through API credits?

The answer is a three-layer split that makes agents manageable: the harness (where you write code), the model (the reasoning black box you prompt), and the UI (how users interact with the system). This split maps directly to where your code lives and where things break. Most production failures start in the harness: mishandled state, incorrect tool routing, or prompt construction bugs.

We’ll explore each component, examine the control loop that connects them, and cover the four pillars that determine whether your agent succeeds or fails.

Let’s dive in.

Subscribe now

The Three-Layer Architecture

The harness is where you live as a developer. The UI is where your users live. The model is the “reasoning” black box between them.

The clean split looks like this:

Session state and presentation live in the UI layer. For a chat interface, this means managing conversation history, streaming the model’s thinking in real-time so users see progress, and presenting errors without exposing internal details. For a background job, this layer triggers the agent, stores results, and retries on failures. The boundary matters because the UI layer should know nothing about how tools work or how the loop executes. It just sends goals and receives updates. When you violate this, you end up debugging whether streaming failed because of a websocket issue or because tool execution timed out.

The harness is where your code lives. The loop controller implements the pattern you choose (ReAct, planning, chain-of-thought). The tool registry validates calls before execution, checks parameters, enforces timeouts, and logs every action. Memory management decides what context to include and what to discard when you hit token limits. The safety layer enforces cost budgets, rate limits, and approval gates for dangerous operations. Everything you control lives here. When production breaks at 2am, this is where you debug.

You send the model a system prompt, tool definitions, context, and history. It returns either a thought and an action or a final answer. You control what you send in and how you interpret what comes out. You can’t control the reasoning process itself. The model might choose a different tool than you expected, hallucinate parameters, or return a final answer before calling any tools. The harness handles all of these cases.

LlamaIndex and LangChain abstract some of this. LlamaIndex provides concrete implementations like ReActAgent and FunctionAgent that implement the harness layer with built-in tool validation and memory management. LangChain offers AgentExecutor with similar functionality plus LangSmith tracing, they specifically define it as “A chain managing an agent using tools” which aligns perfectly with the definition of harness we are building up. These frameworks save you from writing loop logic from scratch, but you still need to understand the layer separation. When things break, you need to know whether the issue is in how you’re presenting results (UI), how you’re constructing the prompt (harness), or whether the model just made a bad decision.

The harness orchestrates the core pattern: the agent loop.

The Agent Loop Pattern

The core pattern that defines agent behavior: reason, act, observe, repeat. The Foundation Model (presently, usually an LM) reasons about the current state, decides which tool to use, executes it, observes the result, and repeats until it either reaches the goal or hits a stopping condition.

Unbounded loops are terrifying. Early experimentation taught me this the hard way when a debugging agent burned through a week’s budget in 12 minutes (luckily this was formy own start-up). This was a deliberate test to see how far the agent could go without boundaries. Turns out, pretty far. It kept trying to fix a syntax error by running increasingly creative variations of the same broken code. The loop needs circuit breakers.

Here’s the minimal viable implementation:

def run_agent(goal: str, max_iterations: int = 10) -> str:
    context = [{”role”: “user”, “content”: goal}]

    for iteration in range(max_iterations):
        response = llm.complete(context, tools=available_tools)

        if response.is_final_answer:
            return response.content

        tool_result = execute_tool_safely(
            tool=response.tool_name,
            args=response.tool_args,
            timeout=5.0,
            budget=remaining_cost_budget
        )

        context.append({
            “role”: “tool”,
            “content”: f”Tool {response.tool_name} returned: {tool_result}”
        })

    raise MaxIterationsExceeded(”Agent couldn’t complete goal”)

Our concrete example in fm-app-toolkit shows how to implement a custom ReactAgent by inheriting from LlamaIndex’s BaseWorkflowAgent, but this core structure remains unchanged. The loop implements the fundamental capabilities every agent needs: reasoning about the current state, selecting appropriate actions, executing tools safely, and maintaining conversational context. Let’s examine each pillar.

The Four Pillars

The agentic loop I showed you maps directly to four dimensions that researchers at NVIDIA and in academic literature have identified as fundamental to agent architecture: perception, reasoning, action, and learning. Understanding this mapping helps you know where to focus your production hardening efforts.

Here’s how the loop breaks down:

In production, perception and action create the most operational burden. Perception failures cascade when context windows fill with irrelevant tool examples, causing agents to invoke `search_documents` when they needed `create_document`. The agent wasn’t broken—it just couldn’t see the right pattern in time. Action failures hit differently: a timeout on a write operation without idempotency keys creates retry storms. You deploy at 2pm and by 4pm you’re investigating why your database has 47 duplicate records.

Reasoning draws the most research attention, and once you establish solid prompts it becomes your most stable pillar. Learning is where costs compound silently. Store every interaction in your vector database without relevance filtering and you’ll watch retrieval costs triple while quality degrades because your agent can’t distinguish signal from noise.

The production priority is clear:

Common Anti-Patterns

The fastest way to burn through your integration testing budget? Unbounded loops. Set explicit max_iterations on every agent loop: typically 10-15 for ReAct patterns.

I watched a customer support agent consume 200k tokens trying to resolve an ambiguous request because the loop had no exit condition. This happened when I was prototyping the first customer service gents in the very early days of my start-up Bot Brewers and happened under strict testing conditions under observed scenarios. This is absolutely the situation you want to avoid in production for a fleet of agents at scale since it will bankrupt you, move carefully here.

# BAD: No max iterations
while not agent.is_done():
    agent.step()  # Could run forever, cost explodes

Log every step to tracing systems. When an agent misbehaves, you need its complete reasoning chain.

Send every thought, action, and observation to OpenTelemetry.

Weather (self-hosted) or LangSmith (managed). Without this, debugging becomes forensic archeology through API logs after the incident. I personally lile Arize Phoenix a lof for how it decomposes the traces that flow through your implementations to show them in a neat and easy to understand UI.

Start read-only by default. Write operations need explicit approval workflows. Early in production, a colleague’s agent had bash script execution enabled. It triggered a destructive operation during what should have been a read-only analysis task. Gate dangerous tools behind confirmation steps.

# BAD: Agent can delete production data
tools = [search_db, update_user, delete_user]  # No safeguards

Set cost budgets at the session level. Cap max_tokens, max_tool_calls, and fail gracefully when limits hit. Better to return “I couldn’t complete this” than let costs spiral unexpectedly. A single runaway session can burn through your daily budget.

Testing these patterns is its own challenge. We’ll cover mocks and validation strategies in Article 3.

The Cost Reality

Teams fall into the same trap: they prototype an agent that works beautifully on a dozen test cases, then get budget alerts when users start hammering it. At scale, the math changes fast.

The cost multiplier is real. A workflow that executes five hardcoded steps might cost $0.03 per run. Turn that into an agentic system, and you’re looking at $1.80 for the same task. The 10-50x jump comes from reasoning overhead. Where a workflow executes five steps sequentially, an agent might reason through 20 decisions: reading the task, evaluating which tool to use, examining the result, deciding whether to continue or pivot, and repeating. Each decision burns tokens. Context accumulates with every step. Tool calls multiply beyond the minimum needed.

Sometimes the cost is justified. When you’re debugging a production incident and can’t predict which logs matter, an agent that adapts its investigation is worth every token. Research tasks where the path depends on what you find. Complex analysis where the branching logic would take weeks to map out. Tasks where the alternative is human hours, not workflow minutes.

But that cost burden creates a testing problem: how do you validate a system that can take different paths every time without burning through API credits?

In the next article we will cover the details of deterministic testing for non-deterministic reasoning systems.

Stay tuned!

AI Agents in Production Series

1. The Foundations

2. The Three-Layer Architecture: The Harness, the Model, and the Loop ← You are here

   1. The Harness Dissected: Inside the Agent Execution Engine

3. Deterministic Testing with Trajectory Mocking

4. LlamaIndex vs PydanticAI vs LangGraph

5. Agent Observability: Traces, Evals, Alerts

6. 5 Agentic Design Patterns That Actually Work

I’ve lost count of the times someone showed me their “AI agent” and it turned out to be a chatbot with good marketing. The confusion is understandable: both use LLM and both respond in natural language, but the difference matters more than most people realize.

Here’s the quick test: Does it wait for a prompt, respond, and stop? That’s a chatbot. Does it receive a goal, plan steps, execute tools, evaluate results, and loop until completion? That’s an agent.

Agents are fundamentally different from the single-call LLM patterns that today dominate most implementations. Agents loop. They plan. They fail, retry, and adapt. They interact with external systems, manage state, and operate autonomously. The architecture changes. The testing changes. The whole operational playbook changes.

This new series of articles will cover the practical side of building agents for production, from foundational definitions to deployment checklists. Every article links to working code in fm-app-toolkit and agentic-design-patterns. No theory without implementation.

To understand agents, we need context (just like the agent itself :P). Let’s start engineering the context around agents so we can generate meaningful insights.

Subscribe now

The Software Evolution

Think about how search has evolved. Ten years ago, you’d match keywords against document titles. Five years ago, you’d compute embedding similarity to find semantically related content. Today, you might give a search application a goal in natural language (”Find me resources on async Python testing, specifically mocking strategies”), and it is capable of using an LLM (reasoning engine) to reformulate queries, search multiple sources, evaluate result quality, and asks clarifying questions.

This progression was intelligently defined by Andrej Karpathy as Software 1.0, 2.0, and 3.0.

Software 1.0 represents applications were the bussiness logic is writen entirely in source code, there are no trained models or vendor FMs used to power them. You write `if “python” in query: return docs_with_tag(”python”)` and the computer follows orders.

Software 2.0 are machine learning and deep learning models, the business logic is encoded in the weights and distributions of the models. You feed a neural network thousands of search queries and their clicked results, and it learns relevance patterns you couldn’t articulate. The intelligence lives in the training data. When you run `model.find_similar(embed(query), top_k=10)`, it returns ranked results.

Software 3.0 is where the paradigm shifts. The core is a reasoning engine (an LLM, FM to be more precise) that can understand context, plan actions, and make decisions at runtime. The software applications use these reasoning engines along with specific software implementations patterns to encode and enhance business logic in a way we had never seen before.

Here’s the key insight: Software 3.0 is defined by the use of a reasoning engine, not by how much autonomy the software application itself has.

The reasoning engine can work in two modes:

1. Execute a predefined workflow: Follow steps you’ve designed. Retrieve documents, generate a summary, format the output. The LLM handles each step, but you control the flow. Lower autonomy, more predictable.

2. Make autonomous decisions: Choose which tools to use, when to use them, what to do with results. The LLM plans its own execution path. Higher autonomy, more flexible.

A chatbot that completes a single prompt-response cycle is Software 3.0. A RAG system that retrieves documents and generates summaries is Software 3.0. A multi-agent system that coordinates specialists for hours is also Software 3.0. They all use the same reasoning engine. What differs is how much control you give the LLM over execution flow.

The spectrum from low to high autonomy isn’t about whether you’re using Software 3.0. You’re using it the moment you have a reasoning engine. The question is: how much decision-making authority does the LLM have?

The Autonomy Spectrum

All LLM-powered applications are Software 3.0. What differs is how much control you give the LLM over execution flow. This isn’t a binary choice between “AI” and “not AI.” It’s a spectrum, and where you land on it determines your debugging nightmares.

Harrison Chase articulated this taxonomy in ”What is a Cognitive Architecture?”, mapping LLM-powered systems to six levels based on what the LLM controls:

To see this graphically:

These last few years I’ve had the chance (I’m enormously grateful) to built different production-grade systems across this spectrum. At Coveo, our multi-tenant RAG based Q&A system sat at Level 2: chains of prompts with semantic search, deployed to enterprise clients worldwide. Later, building a job search agent for a Bounteous client (a US-based talent acquisition cloud platform that helps enterprises match candidates to opportunities), I jumped to Level 5. The LLM decided which databases to query, how to match skills to job requirements, when to ask clarifying questions. The debugging experience was completely different. Below Level 4, you’re tracing linear flows. Above it, you’re watching state transitions and emergent behavior unfold in real time.

Karpathy captured it perfectly at YC AI Startup School:

Start at low autonomy. Earn trust through iteration. A chatbot that works beats an autonomous agent that hallucinates.

So where do we draw the line? Simon Willison’s definition cut through the noise and became the production standard.

The Production Definition

I spent months watching the industry argue past each other about what “agent” meant. Academic papers talked about agents as autonomous entities in RL environments. Marketing teams slapped “agent” on anything with an LLM. Framework maintainers used it differently from each other. Everyone was building the same primitives with incompatible vocabularies.

Then Simon Willison published a definition that cut through the noise:

This was cleaner and directly implementable. It mapped directly to the code patterns everyone was already writing. Within weeks, Anthropic adopted it in their docs. OpenAI integrated it into their SDK. The major frameworks converged. Not because it was academically rigorous, but because it mapped to abstractions engineers were already building.

Three pieces make this work:

Tools are capabilities the agent can request. In a job search agent I built, that meant querying Snowflake for candidate history, running skill-matching algorithms against job requirements, and fetching real-time job postings from partner APIs. The LLM doesn’t execute these directly. It generates tool requests in a structured format (usually JSON), and your harness (the orchestration layer you write) executes them in the real world. I capped iterations at 10 using LlamaIndex’s `max_iterations` parameter: enough for complex searches, bounded enough to prevent runaway costs.

In a loop means the LLM sees the tool result and decides what to do next. The agent requests an action, the harness executes it, and the result feeds back into context. The agent iterates until it’s done. This is the key difference from a workflow: the execution path isn’t predetermined.

To achieve a goal means it runs until it’s done. The agent doesn’t loop forever. It’s working toward a specific outcome: answer a question, complete a task, satisfy a constraint. When it hits that outcome, the loop terminates.

Memory is built into this model. Short-term memory is the context window. Long-term memory is just another tool: read from vector DB, write to vector DB. Same loop, different capability.

This definition won because it’s verifiable. Does it run tools? Does it loop? Does it have a goal? No to any of those? It’s not an agent. It’s something else. And “something else” might be exactly what you need.

The real question: when do you want this loop?

Workflows vs Agents

I’ve watched teams rebrand perfectly good if-else logic as “AI agents” to impress stakeholders. Most production systems that claim to use agents are actually using workflows, and that’s not a weakness. It’s smart engineering.

Anthropic’s Building Effective Agents guide draws a useful distinction:

Workflows are LLMs orchestrated through predefined code paths. You control the decision tree. The LLM acts at specific steps, but you’ve mapped the territory. Cheaper (1-3 LLM calls), more reliable, less flexible. Use workflows for RAG Q&A, classification, anything with known paths.

Agents are LLM-directed. The model decides what to do next, which tools to use, when to stop. More expensive (4-15+ calls), less predictable, more flexible. Use agents for research, debugging, and exploratory analysis: tasks where you can’t map every branch upfront.

I learned this building multi-tenant systems for different clients across north America. More than once I deployed RAG systems that ingested documentation for different clients and served tailored answers to each client. The architecture was straightforward: semantic search handled retrieval, a simple chain of prompts handled generation. This workflow approach (Level 2) gave us predictable costs and reliable performance at scale. We could have built an agent system for more flexibility, but we didn’t need it. The workflow was powerful enough for sophisticated multi-tenant use cases without the complexity or cost of autonomous decision-making.

In production, most systems use hybrid patterns: workflows for the 80% case, agents for complex scenarios, humans for edge cases. The key is building systems with an autonomy dial that lets users turn it up when they need leverage, down when they need control. As Karpathy put it at YC AI Startup School: ”It’s less Iron Man robots and more Iron Man suits.”

Anthropic’s guidance: ”Start with workflows, add autonomy where needed.” If you can define the decision tree, use a workflow. Only introduce agent complexity when branching logic justifies the cost and the debugging effort.

Here’s the uncomfortable truth: most teams pick agents when they need workflows, burn 10x the budget learning the difference, and then face a harder question. Even when you do need an agent, how long until it’s production-ready?

The Reality Check

Your CEO just asked when the AI agents will be ready to handle customer support. Your VP read an article claiming “2025 is the year of agents.” At YC’s AI Startup School in June 2025, Karpathy addressed this timeline pressure directly: “When I see things like, ‘2025 is the year of agents,’ I get very concerned... this is the decade of agents.“

The gap between demo and production is vast. Your agent aces 90% of test cases in development, then fails spectacularly on edge cases that take 3-6 months of production traffic to surface. As one engineer crystallized it: ”demo is works.any(), product is works.all().” Working for a client in the US, I inherited an email generation system that hallucinated hireing campaigns and exhibited non-deterministic behavior across identical inputs. The fix wasn’t prompt engineering. It required rearchitecting the pipeline to eliminate the failure modes entirely. Some problems require structural solutions, not cleverer prompts.

LLMs exhibit what researchers call “jagged intelligence”: superhuman at code generation, unable to count letters reliably. This isn’t a bug you can patch. It’s the nature of the technology, which means your agent might handle complex API integrations flawlessly but fail on validation tasks you’d never think to test.

The solution? Deploy at low autonomy, accumulate edge cases in production, refine your prompts and evals, and gradually increase autonomy over quarters, not sprints. Keep humans in the loop, especially when mistakes have consequences.

AI Agents in Production Series

1. The Foundations ← You are here

2. The Three-Layer Architecture: The Harness, the Model, and the Loop

   1. The Harness Dissected: Inside the Agent Execution Engine

3. Deterministic Testing with Trajectory Mocking

4. LlamaIndex vs PydanticAI vs LangGraph

5. Agent Observability: Traces, Evals, Alerts

6. 5 Agentic Design Patterns That Actually Work

In the next article, we’ll unpack the three-layer architecture that makes agents practical to build: the **harness** (where you write code), the **model** (the black box you prompt), and the **loop pattern** that connects them.

References

- Software 3.0 — Andrej Karpathy’s framework for understanding LLM-based software

- What is a Cognitive Architecture? — Harrison Chase on autonomy levels

- Agents — Simon Willison’s production definition

- Building Effective Agents — Anthropic’s workflows vs agents guide

- YC AI Startup School— Karpathy on the “decade of agents”

📌 TL;DR: Unit testing AI applications traditionally means choosing between expensive API calls or no tests at all. We introduce custom testing abstractions that extend your framework's base classes, giving you deterministic tests that run instantly, cost nothing, and work in CI/CD without API keys. Transform your test suite from anxiety-inducing to confidence-building with patterns that work across any framework.

This is the fundamental challenge that stops most AI engineering teams cold when moving beyond prototypes. You've abstracted your data loading with the Repository pattern from our previous article, but now you need to validate that your agent's reasoning works, tool calls execute correctly, and edge cases don't spiral costs or mislead users.

The typical approach becomes a manual nightmare: deploy to staging, run scenarios, hope for reasonable responses. Each test costs real money. When behavior changes, teams waste hours determining if it's an improvement, regression, or just the LLM's different "mood" that day.

The breaking point comes when developers avoid tests entirely. Nobody wants to rack up hundreds in API costs validating a prompt tweak. When your team is afraid to test their code, something is fundamentally broken. Yet agentic systems are even trickier: non-deterministic reasoning loops, unpredictable tool calls, decision trees that branch differently each time.

Let’s dive in.

This article is part of the series:

• Part 1: Production AI systems: A reality check - Why production thinking beats prototype culture

• Part 2: Production AI Systems: The Data Loading Chaos - Data abstraction for testable AI systems

• Part 3.0: This article

  • Part 3.1: Deterministically Testing Agentic Systems - Coming next week

Subscribe now

The Testing Paradox

Just as we had data coupling killing our deployments in Article 2, we now face testing coupling that kills development velocity. The paradox is stark: the more thoroughly you want to test your agentic systems, the slower and more expensive your development becomes. Most teams find themselves trapped between two impossible choices: expensive, slow tests with real APIs or no automated testing at all.

The Test Pyramid Challenge for AI Systems

Before we dive into the specific problems, let's establish what we're trying to achieve. Mike Cohn introduced the test pyramid concept in "Succeeding with Agile", which provides a simple but powerful framework for organizing automated tests:

• Unit Tests (foundation): Fast, isolated tests of individual components

• Integration Tests (middle): Tests of component interactions

• End-to-End Tests (top): Full system tests through the user interface

The pyramid shape reflects both quantity and speed: lots of fast unit tests at the bottom, fewer slower integration tests in the middle, and just a handful of comprehensive end-to-end tests at the top. This foundation has guided testing strategy across the software industry for over a decade.

The problem?

Traditional unit tests assume deterministic behavior: given the same input, you get the same output. But when your "unit" includes an LLM call, this assumption crumbles. Every test run produces different responses, making traditional assertions impossible.

This forces most AI teams to skip unit testing entirely and rely heavily on expensive, slow integration and end-to-end tests. The result is an inverted pyramid that's expensive to run, slow to provide feedback, and brittle in CI/CD pipelines.

The Three Killers of AI Development Velocity

1. Cost Death by a Thousand Cuts

Every test that calls a real API costs money. A comprehensive test suite covering different agent scenarios quickly becomes expensive when each run consumes thousands of tokens. Run your tests multiple times during development, across your entire team, and suddenly you're looking at a significant monthly expense just for testing.

2. The Latency Tax

API calls add 1-3 seconds per test case. Your test suite now takes 2-3 minutes to complete. Compared to traditional unit tests that run in milliseconds, this destroys the tight feedback loop that makes test-driven development possible. Developers stop running tests after every change and batch them up, reducing the quality of debugging information when something breaks.

3. The Flaky Test Nightmare

Traditional testing assumes deterministic outputs, but LLMs are fundamentally probabilistic. The same prompt can return:

• "The answer is 42" one day

• "42 is the answer" the next day

• "The result is forty-two" on Friday

Your test expects "42" but gets "forty-two." Is this a regression? An improvement? Just random variation? Teams waste hours investigating "failures" that are actually just different-but-equivalent responses.

Why Traditional Mocking Fails

The instinct is to mock the API of our model provider directly. But this violates a key principle: don't mock what you don't own. When you mock external services directly, you couple your tests to implementation details you don't control.

Let's take OpenAI as an example for instance:

You end up testing your mocks, not your business logic. When your model provider updates their API response format, your tests still pass even though your application might break in production.

Custom Testing Abstractions

The question becomes: how do we build these custom implementations that solve the testing paradox? The answer lies in a breakthrough that transforms our biggest weakness—non-deterministic responses—into our greatest testing advantage.

Just as DocumentRepository.load_documents() (in our previous article) hides whether you're reading from local disk or GCS, our custom LLM abstractions hide whether responses come from live API calls or controlled test behavior. The breakthrough isn't about replacing LLMs—it's about controlling the interface.

The Perfect Abstraction Point

We'll be using LlamaIndex as our foundational framework to exemplify our testing strategies. LlamaIndex is designed with a clean engineering focus—it's very easy to extend and build on top of, making it perfect for demonstrating these patterns. The solution follows the same approach that saved our data pipelines: abstract away the complexity behind an interface you control.

LlamaIndex already provides this through their base LLM class:

View the complete LLM abstraction implementation

Every LLM integration in LlamaIndex (whether OpenAI, Anthropic, or local models) extends the same base class with consistent methods like chat() and complete(). This gives us the perfect abstraction point: we can create our own implementations that return predictable responses while maintaining full compatibility with the entire LlamaIndex ecosystem.

Just as our DocumentRepository abstracted away whether documents came from local disk or cloud storage, custom LLM implementations can abstract away whether responses come from live APIs or predefined test data.

Predictable Responses for Unpredictable Systems

The simplest way to understand this approach is through echo behavior. Our MockLLMEchoStream overrides the LLM's chat() and complete() methods with a simple but powerful logic: extract the last user message from the conversation and return it unchanged. No generation, no variation, just a perfect echo of what was sent in.

Here's the critical insight: when your RAG pipeline builds a complex prompt with retrieved context and sends it to the LLM, the mock echoes back that entire synthesized prompt, letting you see exactly what your pipeline constructed. This transforms the non-deterministic black box into a transparent, testable system:

See the full implementation here.

Practical Testing with Echo Behavior

This echo pattern enables comprehensive testing of your AI pipeline integration without any external dependencies:

The Power of Controlled Environments

The echo pattern reveals something important: most of your application logic doesn't actually depend on the LLM being "smart." It depends on the LLM returning text that your system can process. By controlling exactly what text gets returned, you can:

• Test prompt formatting: Ensure your prompts are constructed correctly

• Validate parsing logic: Confirm your code handles the expected response format

• Exercise error paths: See how your system behaves with unexpected inputs

• Performance test: Run thousands of iterations to find bottlenecks

When you're testing whether your agent correctly calls a calculator tool, you don't need GPT-4's reasoning abilities. You need a predictable response that triggers your tool-calling logic. Custom testing abstractions provide exactly that predictability.

Building Test Confidence

This approach transforms AI testing from anxiety-inducing to confidence-building. Instead of wondering whether a test failure indicates a real bug or just random LLM variation, you know exactly what the mock will return. Your tests become reliable indicators of your code quality, not your luck with API responses.

Unit Testing AI Components

Now that you understand the core principle, let's see how this transforms the way you test real AI systems. The echo pattern is just the beginning—here's how it scales to complex production workflows.

Building on our foundation from the previous section, let's see how custom testing abstractions enable comprehensive unit testing of AI components. Your DocumentRepository loaded test data deterministically; now your custom testing abstractions provide test responses deterministically.

Testing Complex AI Pipeline Integration

Most AI applications aren't just simple LLM calls—they're complex pipelines with multiple components that need to work together seamlessly. Consider a typical RAG pipeline: document indexing → retrieval → synthesis. With custom testing abstractions, you can validate this entire chain plus your response parsing logic—all without external dependencies:

This single test validates five critical integration points:

1. Document Indexing: Your documents are properly embedded and stored

2. Vector Retrieval: Relevant documents are found for the query

3. Prompt Construction: Retrieved context is correctly formatted into the prompt

4. LLM Integration: The query engine properly calls your LLM abstraction

5. Response Synthesis: The final response is correctly assembled

Because MockLLMEchoStream echoes back everything it receives, you can verify that your RAG pipeline correctly builds prompts with context, formats them properly, and handles the response—all deterministically. The same approach lets you test edge cases like empty retrievals, malformed documents, or parsing variations without a single API call.

The Same Code, Different Environments

Following our Repository pattern approach, you swap real LLMs for custom testing abstractions while keeping your application code unchanged:

This separation means developers can iterate rapidly with zero-cost testing, while production gets the full power of advanced models.

Building Comprehensive Test Suites

With custom testing abstractions, you can build test suites that actually run—maintaining our zero-cost, fast testing approach:

These tests run in milliseconds, cost nothing, and provide reliable feedback about your code quality. While at the same time will guarantee that you will cover for the most obvious error scenarios.

Advanced Patterns

These integration tests reveal something powerful: we're not just testing individual components anymore. We're building an entire testing ecosystem that can handle any AI workflow.

The custom testing abstractions we've explored represent just the beginning of what's possible. The key takeaway is that you can write any custom testing abstraction that serves your specific testing needs, following these same patterns.

Beyond Echo: A Universe of Testing Possibilities

Using the echo behavior we introduced demonstrates the core principle, but real-world testing demands more sophisticated approaches. You might need:

• Response sequence mocks for testing multi-step reasoning workflows

• Rule-based mocks that respond differently based on input patterns

• Conditional mocks that simulate different agent "personalities" or capabilities

• Error simulation mocks for testing failure recovery scenarios

Each follows the same pattern: extend LlamaIndex's base LLM class we established earlier, implement the required methods, and provide the specific behavior your tests need.

Preview: Testing Agent Reasoning Loops

Consider how you might test an agent's multi-step reasoning process. With predetermined response sequences, you can validate the entire thought→action→observation cycle:

This approach lets you validate that agents follow expected reasoning patterns, call tools with correct parameters, and integrate results properly—all critical for production agentic systems.

Our next article, "Deterministically Testing Agentic Systems," will provide comprehensive strategies for testing agent workflows, including response chains, reasoning validation, error simulation, and advanced mock patterns that handle the full spectrum of agentic behaviors.

The Complete Testing Foundation

The combination creates a comprehensive testing ecosystem:

• Repository pattern for data: Deterministic document loading (Article 2)

• Custom testing abstractions for responses: Deterministic LLM behavior (this article)

• Integration testing patterns: Validating complete system workflows

Together, these patterns transform AI development from expensive, anxiety-inducing guesswork into confident, rapid iteration cycles. You can test every component, every integration point, and every edge case without external dependencies or costs.

In our next article, "Deterministically Testing Agentic Systems," we'll dive deep into testing complex agent workflows, reasoning validation, and advanced mock patterns that handle the full spectrum of agentic behaviors.

Frequently Asked Questions

Q: How do you unit test LLM applications? A: Use custom testing abstractions that extend your framework's base LLM class to provide deterministic responses without API calls. Instead of mocking external APIs directly, create implementations that return predictable outputs while maintaining interface compatibility with your application code.

Q: What's the difference between mocking APIs and custom abstractions? A: Mocking APIs couples your tests to external interfaces you don't control—when the API changes, your mocks break. Custom abstractions provide a stable interface you own, allowing you to swap between test and production implementations without changing application code.

Q: Can this approach work with frameworks other than LlamaIndex? A: Yes, the same pattern works with LangChain, PydanticAI, and any framework with an LLM abstraction layer. The key is finding the base class or interface that all LLM implementations share, then creating your own test implementations.

Q: How much can I really save on testing costs? A: Teams typically see 95% cost reduction in testing expenses. A test suite that costs $300/month with real APIs can run for free with custom abstractions. Additionally, tests run 50x faster (milliseconds vs seconds), enabling more frequent testing.

Q: Do I still need any tests with real LLMs? A: Yes, keep a small set of end-to-end tests with real LLMs for final validation. Use the test pyramid approach: many unit tests with mocks, some integration tests with mocks, and a few E2E tests with real APIs.

Your Implementation Roadmap

Start small, prove value, then scale:

1. Create your first mock: Copy MockLLMEchoStream from above

2. Replace one expensive test: Pick your slowest, most costly test

3. Add environment switching:

4. Convert your test suite: Replace API calls with mocks

5. Deploy to CI/CD: Remove API keys, tests run on every PR

Results: Teams typically see 95% cost reduction, 50x faster tests, and actually run them.

The foundation is set. Time to build something remarkable on top of it.

References & Further Reading

This Series

• Part 1: Production AI systems: A reality check - Why production thinking beats prototype culture

• Part 2: Production AI Systems: The Data Loading Chaos - Data abstraction for testable AI systems

• Part 4: Deterministically Testing Agentic Systems - Coming next week

Code & Implementation

• FM App Toolkit GitHub Repository - Complete working code for all patterns in this article

• LlamaIndex Documentation - Framework documentation

• LlamaIndex LLM Base Class - The abstraction point we extend

Testing Fundamentals

• The Practical Test Pyramid - Martin Fowler's essential guide to test strategy

• Succeeding with Agile - Mike Cohn's original test pyramid concept

Connect & Discuss

• AI Enhanced Engineer Newsletter - Subscribe for weekly production AI insights

• GitHub Discussions - Share your testing patterns and questions

📌 TL;DR: Your AI experiments may begin with simple local file loading, but production quickly demands multiple data sources across different environments. To solve this, we introduce a key concept from Domain-Driven Design that creates a clean abstraction layer. With local and cloud backends behind the same interface, your data-access code works everywhere: tests run locally without credentials, development happens at zero cost, and deployment becomes as simple as changing an environment variable. By hiding data access complexity behind a simple interface, you can focus on building the intelligence layer instead of wrestling with infrastructure plumbing.

Usually data science experiments start like this: you dump some data files in a local folder, write a quick script to read them, and start iterating on your application. The loading logic is an afterthought, maybe just a one-liner pointing to your data directory. Why overcomplicate things when you're just trying to see if your approach even works for your use case?

Fast forward three weeks. The experiment showed promise, stakeholders are excited, and now it needs to run in production. Suddenly that one-liner spawns a dozen functions: load_from_local(), load_from_s3(), load_from_gcs(), load_from_staging_bucket(). Each environment needs different logic. Your code becomes a maze of if-statements checking environment variables.

Then product asks for a simple change: "Can we test with last month's data instead?"

Your heart sinks. That requires code changes, a new deployment, waiting for CI/CD, coordinating with DevOps. To change which data files you're reading. Something smells terribly wrong here.

And it is. You've coupled your AI logic to your infrastructure. Every data source change requires a code deployment. Every new environment needs custom loading logic. You're shipping code to change configuration.

The solution isn't new. It's been solving this exact problem in traditional software for decades, elegantly described by Harry Percival and Bob Gregory in their book Architecture Patterns with Python (also known as Cosmic Python).

Let’s adapt their Repository Pattern for our AI engineering field.

This article is part of the series:

• Part 1: Production AI systems: A reality check

• Part 2: This article.

• Part 3.0: Production AI Systems: The Unit Testing Paradox

  • Part 3.1: Deterministically Testing Agentic Systems - Coming next week

Subscribe now

The Coupling That Kills AI Projects

Building on our previous article’s (here) production reality check, data access represents the first major coupling point that derails AI/ML projects. Your application's quality depends entirely on the data it processes, but that data lives in different places across your development lifecycle.

In local development, you likely read from a ./data/ folder on your laptop. In CI/CD testing, you use temporary directories or mock files. Staging pulls from Google Cloud Storage with test data. Production uses a different GCS bucket with real data, probably with different access patterns and permissions.

Without proper abstraction, your code sprawls with environment checks. You've seen this pattern before, maybe even written it yourself:

This looks innocent enough with four environments. But real systems aren't this clean. You'll have regional deployments, disaster recovery environments, partner integrations, each with their own data sources and authentication patterns. That simple if-statement becomes a 200-line configuration nightmare that nobody fully understands.

The Repository Pattern from DDD

The solution comes from Eric Evans' Domain-Driven Design, but it's Harry Percival and Bob Gregory who made it practical for Python developers in their "Cosmic Python" book. The pattern is elegantly simple: hide data access behind an interface.

Your intelligence layer never knows or cares where data originates. It just calls repository.load_documents() and gets back Document objects ready for processing. The repository handles all the complexity of finding, loading, authenticating, and retrying.

This isn't just moving complexity around. It's isolating it. When GCS changes their API, you update one repository class, not fifty different loading functions scattered across your codebase. When you need to add S3 support, you create a new repository implementation without touching any business logic.

Why This Matters More for AI

Traditional applications might tolerate some infrastructure coupling. A bit of technical debt here, some duplicated code there. AI applications can't afford this luxury for four critical reasons.

Iteration velocity: You're constantly experimenting with different chunking strategies, embedding models, and retrieval approaches. Every experiment needs the same data loaded consistently. If loading logic is scattered everywhere, you can't iterate quickly.

Cost implications: Every test against production data costs money in embeddings and API calls. You need to seamlessly switch between free local data during development and expensive cloud data in production. Without abstraction, developers will skip tests to save money.

Non-determinism management: You need consistent test data to identify when behavior changes come from your code versus model updates. If every environment loads data differently, debugging becomes impossible. Was that failure from your new chunking logic or because staging loaded different files than your local tests?

Compliance requirements: Production data often contains PII that can't exist in development environments. You need different data sources for different environments, but the same processing logic everywhere. The repository pattern makes this natural rather than painful.

💡Production Reality box

The $18,000 Lesson: A startup's RAG system loaded all documents on every request. No caching, no repository pattern, just direct S3 calls. Each request loaded 10GB of documents, embedded them fresh, then answered. The monthly AWS bill: $18,000. The fix: Repository pattern with caching. New bill: $400. The lesson: Abstract early, optimize always.

Local Development Without Friction

Let's start with the simplest possible repository to understand the concept:

This abstract base class defines a contract: any repository must implement load_data() and return a DataFrame. Now you can create specific implementations:

The beauty is that your application logic doesn't care which repository you use. Different implementations can conform to the same interface, and you swap them based on your needs.

Adapting the Pattern for RAG Applications

To apply this exact same pattern to RAG use cases, we leverage LlamaIndex's document loading capabilities while maintaining the repository abstraction:

Now we implement a local filesystem version using LlamaIndex's SimpleDirectoryReader:

We're using LlamaIndex's SimpleDirectoryReader for robust file loading and Pydantic for fast type validation and modern constructor patterns. The @validate_call decorator ensures type safety at runtime, catching errors before they propagate through your pipeline.

Each parameter solves a real problem you'll face. recursive lets you organize documents in subdirectories without changing code. required_exts prevents accidentally loading that .DS_Store file that breaks your parser. num_files_limit keeps your laptop from running out of memory during development. exclude_hidden saves you from processing git files.

Working with the Sample Documents

The toolkit features three carefully selected blog posts from Chip Huyen that serve as ideal test data for our implementation. What makes these particularly valuable is that we'll soon be able to interact with this content through our chat interface, leveraging our ML engineering expertise to evaluate the quality of the system's responses.

These aren't arbitrary selections. They represent real GenAI engineering content covering concepts you'll encounter in production. The collection explores RAG system design and implementation patterns in depth, walking through the architectural decisions that make these systems work at scale. It explains four progressive levels of context enhancement, from basic retrieval to sophisticated personalization strategies. Additionally, it provides a comprehensive framework for evaluation metrics and safety guardrails, which are essential considerations for any production system.

Soon, we'll be able to engage with this knowledge conversationally through our custom implementation, transforming static content into an interactive learning experience.

The Development Workflow Advantage

Zero latency, zero cost development changes everything about how you work. You can change chunking strategies and see results instantly. Same files always produce same documents, giving you deterministic testing. You can work on planes, trains, anywhere without internet. Test a hundred variations without worrying about API charges.

The pattern extends naturally to building a complete RAG system:

Notice how build_rag_pipeline takes any DocumentRepository. It doesn't know or care whether documents come from local disk or cloud storage. This same function works unchanged in production with a GCP repository.

By the end of this article series, you'll have built a configurable RAG system to interact with AI engineering documents. The repository pattern is the foundation that makes this possible. But its immediate value is simpler: you can develop and test your entire application without cloud credentials, without internet access, and without spending a penny on API calls.

Seamless Cloud Transition

Moving to production doesn't require rewriting your application. The same interface you used locally now connects to Google Cloud Storage:

Notice the pattern here. Just like LocalDocumentRepository accepts a path parameter in load_documents(), so does GCPDocumentRepository. The location format changes from a local path to a GCS URI, but the method signature remains identical. Your build_rag_pipeline function from earlier works without modification.

The repository pattern has absorbed all the complexity of cloud storage access, parsing GCS URIs, handling authentication, and managing errors. Your business logic remains blissfully unaware that it's now reading from GCS instead of your laptop.

This isn't theoretical. These patterns come from real production systems that process millions of documents daily. The abstraction means that when GCS introduces changes to their API, you update one repository class, not hundreds of data loading functions scattered across your codebase.

Environment-Based Configuration

Real production systems need to adapt to their environment without code changes. The factory pattern makes this elegant:

Now deployment is just setting an environment variable and configuring the appropriate paths. No code changes to switch data sources. No redeployment to test with different documents. The same container image works in development, staging, and production.

This pattern scales beyond just local and GCS. Need S3 support? Add an S3Repository that accepts s3:// URIs. Azure Blob Storage? AzureRepository with azure:// paths. The interface remains constant while implementations proliferate. Your application logic never changes.

The Foundation of Deterministic AI Testing

The repository pattern transforms how you write unit tests for AI applications. Instead of mocking file systems, stubbing cloud storage clients, or skipping tests because "they need real data," you get clean, fast, deterministic tests that exercise your actual production code.

There's a principle from Cosmic Python that perfectly captures why this pattern works:

Percival and Gregory explain that when you mock external systems directly, you couple your tests to their complexity. If we mocked GCSReader or SimpleDirectoryReader throughout our tests, we'd be coupling to all the intricacies of these libraries. Instead, by creating our own DocumentRepository abstraction, we build a simple interface that we control completely.

The repository pattern forces us to build this simple abstraction over messy subsystems. DocumentRepository is much simpler than LlamaIndex's file readers or Google's storage clients. It exposes exactly what our application needs (loading documents) and nothing more. No arbitrary file system operations, no complex authentication flows, no vendor-specific APIs. Just load_documents() with a location parameter.

This approach gives us clarity, not magic. Speaking of which, if you find yourself reaching for MagicMock to test your AI pipelines, remember: the only magic you want in your tests is how quickly they run, not how they work. Real implementations of simple interfaces beat magical mocks of complex ones every time.

This has the same performance benefit as mocking the file system directly, but encourages us to think carefully about our design. By limiting access to our persistence layer, each component gets exactly what it needs. The service layer can request documents without knowing whether they come from disk or cloud storage. Tests can inject a local repository without touching any external APIs.

Traditional testing approaches force you to choose between incomplete coverage (mocking everything) or expensive, slow tests (hitting real services). The repository pattern eliminates this false choice. Your tests run the exact production logic, just with a different data source. No credentials, no network calls, no flaky failures from transient cloud issues.

Here's production code that works identically in tests and deployment:

Now watch how this same function works in tests:

This is the breakthrough: you're bringing determinism to non-deterministic applications. Your chunking logic, metadata extraction, document parsing, all tested with the exact code that runs in production. No divergence between test and production behavior.

Integration Testing When You Need It

The repository pattern doesn't mean you never test against real infrastructure. It means you choose when to do it:

This separation gives you control. PR checks run instantly without credentials. Developers iterate quickly with local tests that complete in seconds. Integration tests verify that your GCS configuration and permissions work correctly, but only when you explicitly run them. You control costs by choosing when to hit external services.

The pattern extends to your CI/CD pipeline. Your GitHub Actions workflow can run hundreds of tests on every commit without any cloud credentials. Only your deployment pipeline needs access to production resources. This isn't just about saving money on API calls. It's about development velocity. When tests run in seconds instead of minutes, developers actually run them. When tests don't require internet access, you can work anywhere. When tests are deterministic, you can trust their results.

Your Implementation Roadmap

The repository pattern isn't just about loading files. It's about building AI systems that can move from laptop to production without rewrites. It's about testing that doesn't require cloud credentials. It's about development velocity that doesn't sacrifice production reliability.

We started this article with a familiar pain: your simple data loading logic spawning into a maze of environment-specific functions. Now you have the solution. The same pattern that Percival and Gregory use to tame database complexity works beautifully for AI data pipelines.

Start simple:

1. Today: Refactor one data loading function to use the repository pattern

2. This week: Move your test documents to LocalDocumentRepository

3. Next sprint: Implement cloud repository for your staging environment

The immediate benefits:

• ✅ Tests run without credentials or internet

• ✅ Development happens at zero cost

• ✅ The same code works everywhere

• ✅ New developers are productive immediately

The long-term advantages:

• 🚀 Seamless scaling from prototype to production

• 🔧 Easy migration between cloud providers

• 🧪 Comprehensive testing without infrastructure

• 💰 Controlled costs through explicit boundaries

Remember the code we showed earlier, where process_documents() works identically with local and cloud repositories? That's not a demo. That's your production code. The three Chip Huyen blog posts in the samples directory? They're waiting for you to build your first RAG pipeline.

Every production AI system that successfully scales implements some version of this pattern. The teams that adopt it early save months of refactoring later. But more importantly, they ship faster because their developers aren't fighting their infrastructure.

Next in this series, we'll dive deep into testing patterns for AI applications. You've seen how the repository pattern enables deterministic tests. Article 3 will show you how to mock LLMs, test agent behaviors, and validate entire AI workflows without spending a penny on API calls.

The foundation is set. Your data pipeline is abstracted. Now let's build something remarkable on top of it.

Code Repository & Resources

All code examples from this article are available in the FM App Toolkit:

• Repository implementations: fm_app_toolkit/data_loading/

Further Reading

Core Concepts:

• Cosmic Python: Repository Pattern - The foundational text on repository pattern in Python

• Cosmic Python: Unit of Work Pattern - Where "Don't mock what you don't own" comes from

• Harry Percival's Testing Blog - Deep dives into Python testing patterns

• Bob Gregory on Abstractions - Practical posts on building maintainable Python systems

LlamaIndex Resources:

• SimpleDirectoryReader Documentation

• GCS Reader Documentation

Related Articles:

• Article 1: Production-First AI Systems Engineering

• Article 3: Testing AI Workflows (Coming Soon)

📌 TL;DR: Production AI requires different engineering patterns than traditional software. This article explains the three-layer AI stack, why research prototypes fail in production, and provides a practical framework for building reliable AI systems using engineering-first principles. You'll learn essential patterns for handling non-deterministic components and get a concrete playbook to start building today.

A senior engineer joins an AI team expecting to work on cutting-edge model architectures. Three months later, they're debugging data pipelines at 2 AM, frantically optimizing API costs that somehow exceeded the monthly AWS bill, and wrestling with tests that pass locally but fail in CI. The reason? The LLM decided to be slightly more verbose today.

This isn't a failure story. It's the reality of AI engineering at the application layer, where the theoretical elegance of foundation models meets the messy complexity of production systems. You're not training the next GPT-5. You're not optimizing CUDA kernels. You're building the critical bridge between breakthrough AI capabilities and actual business value, and it turns out that bridge needs more engineering than anyone warned you about.

The gap between "my RAG demo works" and "our AI system serves 10,000 users reliably" isn't about better prompts or fancier models. It's about understanding where you sit in the AI stack and building with the right patterns from day one.

This article is part of the series:

• Part 1: This article.

• Part 2: Production AI Systems: The Data Loading Chaos.

• Part 3.0: Production AI Systems: The Unit Testing Paradox.

  • Part 3.1: Deterministically Testing Agentic Systems - Coming next week

Subscribe now

The Three-Layer AI Stack

The modern AI ecosystem isn't a flat landscape—it's a carefully structured pyramid that Chip Huyen brilliantly articulated in her framework. Understanding this structure isn't academic; it fundamentally shapes how you approach building AI applications.

At the infrastructure layer, companies like NVIDIA, AWS, and Google Cloud provide the raw compute power. This is the domain of GPU clusters, distributed training frameworks, and petabyte-scale data pipelines. Unless you're at a hyperscaler or a foundation model company, you're consuming this layer, not building it. The key insight? You inherit both its capabilities and constraints—latency bounds, rate limits, regional availability, and yes, those eye-watering compute costs.

The model layer sits in the middle, where OpenAI, Anthropic, Google, and others train foundation models. These teams wrestle with transformer architectures, constitutional AI, and reinforcement learning from human feedback. They're pushing the boundaries of what's possible with language understanding and generation. As an application developer, these models are your primary building blocks—powerful, but opaque. You can fine-tune them, prompt them, even chain them together, but their core behaviors remain largely fixed.

Then there's the application layer, your domain. This is where abstract AI capabilities become concrete business solutions. It's where a language model becomes a customer service agent, a code reviewer, or a medical diagnostic assistant. Industry reports show that 67% of enterprises are building at this layer, up from just 23% two years ago. The explosion isn't just in quantity; it's in complexity.

Here's what makes this layer uniquely challenging: you're orchestrating services you don't control, with behaviors you can't fully predict, at costs that scale with every user interaction. A traditional web service might handle a million requests for pennies. Your AI service might spend $10 on those same requests, and that's before considering the cascade effects of retries on failures, context window management, and multi-step reasoning chains.

Your position in this ecosystem matters because it defines your constraints and opportunities. You can't make the model inherently smarter (that's the model layer's job), and you can't make GPUs cheaper (that's infrastructure). But you can build resilient systems that gracefully handle model uncertainties. You can create testing strategies that catch issues before they reach production. You can architect data pipelines that minimize API calls while maximizing relevance. Most importantly, you can establish patterns that make AI applications as reliable and maintainable as any other production system.

The Production Reality Check

The most dangerous moment in any AI project? When the prototype works perfectly.

A data scientist builds a RAG system in a Jupyter notebook. It retrieves accurately, responds coherently, and everyone's impressed. Six months later, that same system is hemorrhaging money in production, failing mysteriously, and requiring constant manual intervention. The notebook never had to handle concurrent users competing for vector database connections, documents exceeding context windows, or the same question getting different answers five minutes apart. Shankar et al. documented this exact phenomenon in their study on how ML systems behave unpredictably until they hit production.

This fundamental difference shapes everything about how we build AI systems. A research prototype might tolerate a 15% failure rate if the successes are spectacular. A production system serving customers needs 99.9% uptime, predictable costs, and graceful degradation when things go wrong.

The hidden iceberg of production AI reveals itself through a cascade of requirements:

• Malformed responses: Your LLM returns invalid JSON despite perfect prompts
• Service failures: Embedding services timeout during peak hours (>30s latency spikes)
• Cost explosions: That multi-agent system makes 47 API calls at $0.03 each = $1.41 per interaction
• Compliance requirements: Audit trails, PII scrubbing, response filtering for data protection
• Engineering wrapper: Connection pooling, exponential backoff, caching, request queuing
• Observability needs: Cost tracking per department, performance metrics, error rates

The 90/10 rule applies brutally: 10% of your code handles the actual AI logic, while 90% handles everything else. As Google's MLOps guidelines emphasize: "The real challenge isn't building an ML model, the challenge is building an integrated ML system and to continuously operate it in production."

Moving from a RAG demo to production meant adding connection pooling for vector database access, implementing exponential backoff for API retries, building a caching layer for embeddings, adding cost tracking per department, creating fallback responses for service outages, implementing request queuing for rate limit management, and building comprehensive observability across all components. The core RAG logic? Unchanged. The engineering around it? 10,000 lines of production code versus 50 lines of AI logic.

💡 Production Reality: The $72,000 Weekend

A team deployed their RAG system without retry limits. A bug caused infinite retries on failures. Each retry called GPT-4 with full context (8K tokens). The weekend bill: $72,000. The fix: Three lines of code for exponential backoff with max retries. The lesson: Production AI needs defensive engineering.

But here's what makes the application layer uniquely challenging: you're building deterministic systems on probabilistic foundations you don't control. Model providers update their systems without announcement, and your carefully crafted prompts suddenly return different schemas. Previously reliable workflows break when models become more or less verbose overnight. The same prompt to the same model yields different responses based on load balancing, temperature settings, or just inherent randomness in token sampling. You discover this when production starts failing in subtle ways your tests don't catch. As researchers at Carnegie Mellon's SEI note, these bugs are "rare, intermittent, and hard to reproduce"—the worst kind in production.

The cost structure breaks every assumption about system scaling. Traditional systems have marginal costs approaching zero. With AI applications, costs scale linearly with usage, and not gently. One team celebrated hitting 10,000 daily active users until they realized they were spending $3,000 per day on API calls. The math was simple: 10,000 users × 5 interactions × $0.06 per interaction. The solution required redesigning their entire interaction pattern.

The cruel irony? These challenges intensify as you succeed. More users mean more edge cases. Higher stakes mean stronger reliability requirements. Broader deployment means more diverse failure modes. The prototype that handled 100 friendly beta testers crumbles under 10,000 real users who paste entire novels into your carefully token-limited input fields. Google's seminal paper on ML technical debt warned us in 2015, but with foundation models, you're not just maintaining code; you're maintaining compatibility with constantly evolving external services you'll never control.

Building on Solid Foundations

After staring into the abyss of production complexity, here's the good news: you don't have to solve these problems from scratch. The chaos becomes manageable when you start with the right foundation.

Most AI projects begin backwards. A developer gets API access, writes some prompt experiments in a notebook, then tries to retrofit production practices later. Six months in, they're refactoring everything because the original structure can't support testing, deployment, or monitoring. The technical debt isn't just high interest; it's compounding daily.

The ai-base-template flips this sequence. Before you write a single prompt or make any API calls, you establish engineering discipline:

This structure solves the fundamental problem: AI projects need more engineering rigor, not less. The Makefile isn't just convenience; it's consistency. Every developer runs make validate-branch and gets the same formatting, linting, and test execution. No more "but it worked on my machine" when Ruff formatted differently on your colleague's setup.

Modern Python packaging through uv changes everything. Traditional pip and requirements.txt lead to dependency hell, especially with the rapidly evolving AI ecosystem. The pyproject.toml locks your dependencies precisely. When OpenAI releases a breaking change to their client library, your production environment stays stable. When you need to upgrade, it's a controlled, testable change, not a surprise at 3 AM.

The test markers might seem premature when you haven't written any AI code yet, but they establish the right habits:

You run make test-unit hundreds of times a day during development. These tests must be fast and deterministic. Functional tests validate complete workflows. Integration tests verify external connections. This separation becomes crucial when you add AI components. Your prompt parsing logic gets unit tests. Your RAG pipeline gets functional tests. Your API calls get integration tests that can be skipped in CI to save costs.

The pre-commit hooks enforce quality before code even reaches version control. Ruff formats your code. MyPy checks your types. The hooks catch the embarrassing mistakes before they become pull request comments. This matters more for AI code, where a missing type hint might mean you're passing a string to something expecting a list, causing mysterious failures three function calls deep.

GitHub Actions workflows complete the safety net. They run on every push, every pull request, without needing API keys or secrets. How? Because you've structured your code to be testable without external dependencies. The workflows validate formatting, run linting, execute tests, and ensure type safety. Your AI logic will plug into this existing infrastructure, inheriting all these quality gates automatically.

The template includes everything you need to start building immediately:

• 🐍 Python 3.12 with modern packaging via uv
• 🧪 Testing setup with pytest (unit, functional, integration markers)
• 🔧 Code quality with Ruff (formatting + linting) and MyPy (type checking)
• 📝 Type hints and Pydantic for data validation
• ⚡ Make commands for common development tasks
• 📓 Jupyter support for experimentation
• 🎯 Pre-commit hooks for quality gates
• 📦 ML-ready structure (just uncomment libraries in pyproject.toml)

The real power comes from cognitive load management. When every project starts the same way, when every developer knows where to find configurations, tests, and utilities, your team's mental energy focuses on solving AI problems rather than debating project structure. The boring decisions are made, documented, and automated.

This isn't about following arbitrary rules. Every aspect of ai-base-template exists because someone learned an expensive lesson. The Makefile exists because manual command sequences lead to skipped steps. The uv packaging exists because pip conflicts wasted days of debugging. The test structure exists because retrofitting tests onto unstructured code is nearly impossible.

Starting here means you're building on bedrock, not sand. Your AI components will slot into a proven structure. Your team will follow established patterns. Your production deployments will inherit battle-tested practices. Most importantly, when things go wrong (and they will), you'll have the observability, testing, and structure to fix them quickly.

Engineering for Uncertainty

With solid engineering foundations in place, you face a deeper challenge: how do you engineer systems when the core components behave probabilistically? Traditional software engineering assumes predictable components. Service A calls Service B and gets a deterministic response. AI applications shatter this assumption at every level.

The shift requires new architectural patterns, not because they're trendy, but because they're necessary for survival. These patterns aren't suggestions; they're requirements discovered through painful production experiences.

Configurability becomes critical when you realize hardcoded prompts are ticking time bombs. The prompt that works perfectly with GPT-4 today might produce garbage with tomorrow's model update. Your carefully tuned temperature setting becomes obsolete when Claude's behavior changes. Systems that bake these values into code require redeployment for every adjustment. The teams that survive build configuration systems that allow real-time adjustments without code changes. Model selection, prompt templates, temperature settings, retry logic, timeout values, all become runtime configurations.

Observability transforms from nice-to-have to existential necessity. When a traditional service fails, you have stack traces, error codes, and deterministic reproduction steps. When your AI system produces a nonsensical response, you have... what exactly? Without comprehensive logging of prompts, responses, token counts, latencies, and model versions, debugging becomes impossible. You need to know not just what the model said, but why it might have said it. Context windows, token limits, rate limit approaches, all invisible until you instrument them.

Testability seems impossible but becomes mandatory. How do you write unit tests for non-deterministic components? The answer isn't to skip testing; it's to architect for testability from the start. This means abstraction layers that allow mock implementations, interfaces that separate AI logic from API calls, and test harnesses that can validate behavior without requiring deterministic outputs. The team that says "we can't test AI code" is the team that will have a $72,000 weekend.

Abstraction saves you from vendor lock-in disasters. When OpenAI deprecates your model with 30 days notice, or when Anthropic's API goes down during Black Friday, your abstraction layer determines whether you have a migration path or a crisis. Teams that couple directly to provider SDKs learn this lesson expensively.

Graceful degradation keeps you online when AI fails. Because AI will fail. APIs go down. Rate limits hit. Costs spike. Systems architected for uncertainty have fallback paths: cached responses for common queries, simplified logic when the smart model is unavailable, human escalation when confidence is low.

The architecture challenge operates at two levels. At the system level, you're orchestrating services, managing queues, implementing caches, and handling distributed failures. At the unit level, you're managing prompts, parsing responses, handling retries, and validating outputs. Both levels demand equal attention. A perfectly designed microservice architecture means nothing if your prompt parsing fails silently.

📋 Pattern Checklist for Production AI:

• ✅ Configuration externalized from code

• ✅ Every AI interaction logged with context

• ✅ Unit tests that run without API calls

• ✅ Provider-agnostic interfaces

• ✅ Fallback paths for every AI component

• ✅ Cost tracking at interaction level

• ✅ Response validation and sanitization

The FM App Toolkit embodies these patterns, providing concrete implementations of these architectural principles. The upcoming articles will dive deep into each pattern, showing not just why they matter, but exactly how to implement them. But the principles themselves are universal. Whether you use our toolkit or build your own, these patterns form the foundation of production AI systems.

Getting Started

Theory without action is worthless. Here's your immediate path from reading to building.

Today, right now: Go to the ai-base-template repository and click "Use this template" to create your own repository. Don't overthink it. Don't wait for the perfect project idea. GitHub will create a fresh copy with all the structure ready to go. Run make init in your new repo. The fifteen minutes you invest now save weeks of refactoring later.

This week: Build something real, even if it's small. A prompt validator. A response parser. An embedding cache. The specific functionality matters less than establishing the patterns. Write tests first, even before your AI logic. Create that .env file for configurations. Run make validate-branch obsessively. Feel the rhythm of professional AI development.

Explore early: The FM App Toolkit repository contains battle-tested implementations of every pattern discussed here. Browse the code. See how DocumentRepository abstracts data access. Understand how MockLLMWithChain enables deterministic testing. Watch how SimpleReActAgent makes debugging possible. You don't need to master it all now, but familiarizing yourself with the patterns accelerates your learning.

Coming next: This series continues with deep dives into the critical patterns that make production AI possible. We'll explore data loading strategies that work across environments, testing approaches that catch bugs before they cost money, and architectural patterns that turn chaotic AI interactions into observable, debuggable systems. Each article builds on these foundations, taking you from concept to production-ready implementation.

The mindset shift starts now. You're not building an AI demo that impresses in meetings. You're engineering a system that serves real users, handles real load, and solves real problems. Every production AI system that works reliably follows these patterns. Every failure story stems from ignoring them.

Your AI engineering journey doesn't start with prompts or embeddings or agents. It starts with clicking "Use this template" and running make environment-create.

The future is probabilistic. Your engineering doesn't have to be.

References & Further Reading

Research & Academic Papers

• Carnegie Mellon SEI (2024). The Challenges of Testing in a Non-Deterministic World. Analysis of why non-deterministic systems make bugs rare and hard to reproduce.

• Google Cloud Architecture (2024). MLOps: Continuous delivery and automation pipelines in machine learning. Comprehensive guide to production ML operations.

• Shankar, S., et al. (2024). We Have No Idea How Models will Behave in Production until Production. Study on the experimental nature of ML systems moving from notebooks to production.

• Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. The seminal paper introducing ML technical debt concepts.

Books & Industry Insights

• Huyen, Chip (2024). AI Engineering. O'Reilly Media. Comprehensive framework for building AI systems.

• Huyen, Chip (2023). Building LLM applications for production. Practical insights on production LLM challenges.

Tools & Templates

• AI Base Template: Production-ready Python template with modern tooling for AI/ML projects

• FM App Toolkit: Battle-tested patterns and implementations for production AI applications

Additional Resources

• MLOps Community: Latest practices and challenges in deploying ML systems at scale

• Faubel, L., Schmid, K. & Eichelberger, H. (2023). MLOps Challenges in Industry 4.0. SN Computer Science. Analysis of MLOps challenges across industrial contexts.

We've all been there: You ask Claude or ChatGPT to generate some code, and it works great. But when you start a new chat the next day, the code is completely different. Same request, different output. Your prompts become novels. You're copying context between conversations, spending more time managing the AI than building.

There's a better way.

What if each AI interaction started with perfect context, knew your tech stack, and remembered your architectural decisions? This is exactly what specialized role-based prompts provide. Instead of one general-purpose AI that forgets everything between chats, you create dedicated AI specialists: experts in their domain who maintain your project context. Think of it as giving your AI a specific job title and expertise. For instance: a Systems Architect who only thinks about design, a Security Analyst who spots vulnerabilities, a Test Engineer who ensures quality, or a Technical Writer who creates clear documentation.

I discovered this approach while leading the delivery of features for fast-paced teams and later refined it building my own software company - playing every role from architect to DevOps. The impact was immediate: consistent output, faster development, better architecture. I went from prompt engineering back to software engineering.

In this article, I'll show you exactly how to build your own team of AI assistants using specialized role-based prompts. While my examples focus on software engineering, this same approach works across all domains - from product strategy and UX design to marketing and operations.

What You'll Learn

• Why specialized roles beat generic AI chat by 10x

• How to set up your AI development team in 5 minutes

• Real example: Building a production gateway in one morning

• Power patterns that multiply your productivity

• Your 7-day implementation playbook

🚀 Quick Start

Want to experience this immediately? Here's your 3-step quick start:

1. Clone the repository

2. Pick a role: Start with Systems Architect, Backend Engineer, or Code Reviewer

3. Create a project: In Claude or ChatGPT, paste the role as custom instructions

That's it. Your next coding session will be transformed. The rest of this article shows you why and how to maximize the impact.

Subscribe now

Setting Up Your AI Development Team

Let's get practical. The good news: it takes minutes to set up and works with tools you're already using.

Whether you're using Claude Projects, ChatGPT Projects, or just saving prompts in a text file, the process is the same:

1. Create a dedicated space for each core role

2. Add the role definition from the repository as instructions

3. Upload your context files: tech stack, coding standards, architecture docs

4. Start building with reliable AI assistance

Each platform offers unique advantages. Claude Projects excels with its 200K token context window and ability to include extensive documentation - I can upload entire codebases, architecture diagrams, and API specs that persist across all conversations. ChatGPT Projects organizes your work differently, letting you create separate spaces for different aspects of your development, with each project maintaining its own custom instructions and file uploads. Both platforms remember your context between sessions, eliminating the need to re-explain your tech stack every time. If you're starting out or working with budget constraints, the manual method - simply copying role prompts at the start of each conversation - delivers 80% of the value at zero cost.

Pro tip: Use descriptive names like "Architect - [YourProduct]" or "Backend - [YourProduct]". Clear naming saves mental overhead when you have multiple projects.

The key insight: The system matters more than the tool. Whether you're using paid features or copy-paste, specialized roles will transform how you work with AI.

Your Core AI Colleagues

Let me introduce you to the team members who help me build and ship features every week. You've seen the setup; now let's see them in action.

The Development Trinity

Every feature I build follows this pattern:

Systems Architect → Backend Engineer → Code Reviewer

Here's last week's example: building a production gateway service in one morning.

9:00 AM - Architecture Design

Me: I need a gateway service between our frontend widget and ML backend. 
Must handle WebSocket connections, JWT auth, rate limiting, and message streaming.

Systems Architect: [Designs FastAPI gateway with Redis pub/sub, PostgreSQL 
persistence, and clean separation of concerns]

10:00 AM - Implementation I take the architecture and start building. The Backend Engineer provides a complete ConnectionManager with Redis persistence and heartbeat handling. Copy, paste, test - it works on the first try.

11:00 AM - Debugging Reality This is Tier 1 in action. I hit a WebSocket timeout issue:

Me: Connections drop after exactly 30 seconds. Here's the error log...

Backend Engineer: The heartbeat coroutine lifecycle is the issue. 
[Provides specific async cleanup fix]

Apply fix, test again. Connections now stable.

2:00 PM - Final Review

Code Reviewer: Found issues:
- Redis connections not properly closed on shutdown
- Missing rate limit headers in HTTP responses  
- WebSocket error messages leak internal details

I manually apply each fix, re-test, and commit. Gateway ready for production in one morning.

The Quality Squadron

Beyond the core trinity, specialists ensure production readiness:

• Security Analyst: Found JWT validation bypass in edge case

• DevOps Engineer: Provided Kubernetes manifests and health checks

• Technical Writer: Created clear integration guide

A complete, deployable service with documentation. What would have taken days of research and trial-and-error was done before lunch.

Power Patterns That Actually Work

Building features is just the beginning. After months of refining this system, I've discovered patterns that multiply its effectiveness.

The Inception Technique

Here's the most powerful pattern I've discovered: making your AI colleagues evolve with your project. As your codebase grows and patterns emerge, your assistants should grow too. The Inception Technique makes this happen.

After a productive session where you've established new patterns or made architectural decisions, simply ask:

"Based on our conversation today, please generate an updated version of your 
role prompt that incorporates the patterns and decisions we've established. 
Include the specific technical choices, code patterns, and architectural 
decisions we've made."

Real example from last week:

Me: We've established several patterns in this session: using Redis sorted 
sets for rate limiting, WebSocket heartbeats every 30 seconds, and our 
specific error handling with custom exception classes. Please generate an 
updated Backend Engineer prompt that includes these patterns.

Backend Engineer: Here's an updated role definition that incorporates your 
established patterns:

[Generates enhanced prompt that includes:]
- Redis sorted sets as the default rate limiting approach
- 30-second WebSocket heartbeat pattern with cleanup handling
- Custom exception classes following your ErrorCode enum pattern
- Async context manager patterns for resource cleanup
- Your specific PostgreSQL connection pooling configuration

I then update my Claude Project or ChatGPT custom instructions with this evolved prompt. Now, every future conversation starts with these patterns baked in. No more reminding the AI about decisions made last week.

This technique is critical because:

• Your assistants grow with your codebase: They learn your patterns as you establish them

• Reliability becomes automatic: New patterns become part of the baseline

• Knowledge compounds: Each session builds on the last

I run this inception process weekly. My Backend Engineer role now knows dozens of project-specific patterns that would take paragraphs to explain each time.

The Handoff Technique

The secret to maintaining context across roles is explicit handoffs. When switching from architect to developer, or developer to reviewer, you need clean transitions. Here's my template:

"The [Previous Role] determined that [key decisions]. 
Based on this, please [specific request for new role]."

Real example from the gateway project:

Me: The Systems Architect designed a FastAPI gateway with Redis-backed 
WebSocket management and PostgreSQL for message persistence. Based on 
this architecture, please implement the message handler service with 
proper database transactions and Redis pub/sub for cross-instance 
communication.

Backend Engineer: [Implements exactly what was specified without questioning
the architectural decisions]

This simple pattern does three things:

• Prevents the new role from questioning established decisions

• Maintains architectural consistency

• Keeps each role focused on their specialty

Without handoffs, your Backend Engineer might suggest MongoDB when you've already committed to PostgreSQL. With handoffs, they implement exactly what was designed.

The Consultation Loop

When stuck, use rapid role switching:

1. Backend Engineer hits a problem

2. Quick switch to Debugger for analysis

3. Security Analyst validates the fix

4. Back to Backend Engineer for implementation

Keep consultations focused: "I'm seeing X error. Here's the trace. What's the cause?"

Custom Roles for Your Domain

The repository provides battle-tested roles, but your secret weapons are custom ones tailored to your stack. Here are three I created for this project:

• "Web Components Expert": Deep knowledge of shadow DOM, custom elements, and browser APIs. Saved hours when building the widget.

• "FastAPI Specialist": Optimizes Python async patterns, understands Starlette internals, and knows every Pydantic trick. Critical for the gateway.

• "Redis Architecture Advisor": Designs pub/sub patterns, caching strategies, and distributed state. Prevented major scaling issues.

Creating a custom role is straightforward:

1. Start with a similar role from the repository

2. Add domain-specific knowledge: Your tech choices, patterns, constraints

3. Test with real scenarios: Use it on actual problems

4. Iterate based on outputs: Refine until it's consistently excellent

Here's the template I use:

You are a [Role Name] specializing in [specific domain/technology].

Core Expertise:
- [Primary skill/knowledge area]
- [Secondary skill/knowledge area]
- [Specific tools/frameworks you use]

Project Context:
- Tech Stack: [Your specific technologies]
- Architecture Pattern: [Your patterns]
- Key Conventions: [Your coding standards]
- Common Patterns: [Patterns you've established]

When providing assistance, you:
- [Specific behavior 1]
- [Specific behavior 2]
- [Specific behavior 3]

Always remember:
- [Critical rule 1]
- [Critical rule 2]
- [Project-specific constraint]

Example of my FastAPI Specialist:

You are a FastAPI Backend Specialist with deep expertise in async Python.

Core Expertise:
- FastAPI framework and Starlette internals
- Python async/await patterns and asyncio
- Redis for caching and pub/sub
- PostgreSQL with SQLAlchemy

Project Context:
- Tech Stack: FastAPI, Redis, PostgreSQL, Pydantic v2
- Architecture Pattern: Clean architecture with dependency injection
- Key Conventions: Type hints on everything, async by default
- Common Patterns: Redis sorted sets for rate limiting, 30s WebSocket heartbeats

When providing assistance, you:
- Write production-ready code with proper error handling
- Use dependency injection for all services
- Include comprehensive type hints and docstrings

Always remember:
- Use connection pooling for Redis and PostgreSQL
- Handle WebSocket lifecycle properly with cleanup
- Custom exceptions inherit from our ErrorCode enum

The "FastAPI Specialist" started as the generic Backend Engineer role. I added specifics about async Python, WebSocket handling, and our particular use of Redis. Now it writes production-ready FastAPI code that follows all our patterns without being reminded.

Your First Week Playbook

You've seen the system in action. You understand the patterns. Now let's get you there. This isn't theory - it's the exact path I followed to transform my development workflow.

Days 1-2: Foundation

Start simple. Don't try to revolutionize everything at once.

1. Pick three core roles: Systems Architect, Backend Engineer, Code Reviewer

2. Set up your first Claude Project (or ChatGPT Project, or just bookmark the prompts)

3. Test with a simple feature: An API endpoint, a small refactor, a bug fix

4. Notice the difference: How much context did you NOT have to explain?

The goal isn't perfection. It's experiencing the power of specialized assistance firsthand.

Days 3-5: Build Something Real

Now apply it to actual work:

1. Choose a postponed feature - something you've been avoiding

2. Follow the trinity workflow: Architect → Engineer → Reviewer

3. Practice the handoff technique between roles

4. Use the inception technique to capture new patterns

This is where the magic happens. You'll catch yourself thinking "this would have taken all day" as you finish in two hours.

Days 6-7: Expand Your Team

Time to specialize:

1. Add the Quality Squadron: Security Analyst, Test Engineer, DevOps Engineer

2. Create your first custom role for your specific stack

3. Run inception on all roles to capture the week's learnings

4. Document your workflow - you're building a system now

By day 7, you should be moving at least twice as fast. Many developers find their productivity continues to improve as they refine their workflow.

Early Success Signals

You'll know the workflow is clicking when you notice:

• Complex tasks feel more manageable with specialized help

• You're catching issues earlier in the development cycle

• Your code has more consistent patterns across features

• You're tackling technical challenges you previously avoided

The transformation isn't just in speed - it's in confidence. You're not just coding faster; you're making better architectural decisions and building more maintainable systems.

When to Apply Your New Workflow

After months of daily use, I've learned where this approach delivers maximum value - and where it doesn't.

Where Role-Driven Development Shines:

• Learning new technologies: When I tackled Web Components, my AI colleagues taught me shadow DOM intricacies while building production code

• Architecture and design decisions: Multiple specialists debating trade-offs beats solo analysis every time

• Boilerplate and scaffolding: Auth systems, CRUD APIs, standard patterns - get them right the first time

• Code reviews and debugging: Fresh, specialized eyes catch issues you're blind to after hours of coding

• Documentation and tests: AI colleagues never skip these "boring" parts

When to Just Code:

• Five-line fixes: If you know exactly what to write, write it

• Performance-critical algorithms: Hand-optimize these sections

• Company-specific logic: Your AI doesn't know your business rules

• IDE refactoring: When built-in tools do the job perfectly

The key is using the right tool for each task. I still write plenty of code directly - but now it's the interesting code, not the boilerplate.

Ready to start? The repository is waiting. Your AI colleagues are ready to join your team. All that's missing is you.

The Path Forward

We started with a problem we've all faced: inconsistent AI responses, context lost between sessions, more time managing prompts than building. Now you've seen the solution in action - specialized AI roles that maintain context, follow patterns, and evolve with your project.

This isn't about replacing developers or chasing AI hype. It's about amplifying what you already do well. My gateway service wasn't built by AI alone - it was built by me orchestrating specialized assistants who understood my architecture, followed my patterns, and caught my mistakes.

The engineers who will thrive in the next decade aren't those waiting for perfect autonomous agents. They're the ones building today with the tools we have, mastering the art of AI collaboration while others are still writing prompt novels.

Every pattern you've learned here - handoffs, consultations, inception - will remain valuable as tools evolve. When we start working with Tier 2 and Tier 3 tools, you'll already know how to orchestrate AI teams effectively. Imagine these same specialized roles in Tier 2 tools like Cursor or Windsurf, where they can directly edit code, run tests, and iterate on solutions. Or in Tier 3, where agent teams collaborate autonomously while you sleep. But more importantly, you'll be shipping better software right now.

Your Next Move:

1. Clone the repository: github.com/ai-enhanced-engineer/ai-assistant-roles

2. Pick your first three roles: I recommend Systems Architect, Backend Engineer, and Code Reviewer

3. Set up your first project in Claude or ChatGPT (or just bookmark the prompts)

4. Build something this week - start small, see the difference

Remember: I built a deployable gateway service in one morning. Not because I'm exceptional, but because I had the right AI specialists helping me. You can do the same.

The future of engineering isn't human vs. AI. It's humans with AI colleagues building what neither could create alone. And that future is available to you right now.

Welcome to your new development team. They're ready when you are.

Not long ago, software development was a solitary craft: one brain, one keyboard, one line of code at a time.
Today, that landscape is shifting. AI is stepping beyond the hype. What was once a background utility or novelty feature is now embedding itself in the core development loop — drafting functions, refactoring files, even submitting pull requests.

What was once autocomplete has become collaboration.

Across the industry, developers are discovering new rhythms of work, new relationships with code, and new partners in the form of intelligent agents. These agents can suggest a fix, finish your thought, or take on an entire feature. They don’t replace developers — they extend them.

But not all AI-powered tools operate the same way.

A clear spectrum of interaction has emerged, defined by how much autonomy we give and how much control we retain. On one end is manual prompting, where the AI is more of a helpful notepad. In the middle is shared control, where agents know your code and respond to your intent. At the far end lies full automation — agents that act independently, offering their work as pull requests and expecting only a final review.

And here’s the crux: AI agents are not magic.
Without the right context, configuration, and constraints, they drift.
Even the most powerful systems fail without structure.

Recent research shows measurable productivity gains when AI agents are deployed effectively. But success does not come from delegation alone. It depends on engineering the environment in which these agents operate. Studies emphasize that the real challenge lies not only in building smarter agents, but in developing the scaffolding around them, including pipelines, testing infrastructure, and well-defined workflows that guide behavior and ensure quality.

This article is a map of that landscape. We’ll walk through the three tiers of AI-assisted development: from command-driven interfaces to autonomous collaborators and explore the hidden infrastructure that makes it all work.

Let’s begin.

Tier 1: Manual Copilot

Prompt, Paste, Repeat

Imagine you’re driving with a printed map. You stop at every intersection, trace your finger along the route, and decide your next move. Now and then, someone in the passenger seat offers advice — helpful, maybe even brilliant — but they can’t see the road, the weather, or your destination. You’re grateful for the insight, but every turn is still yours to make.

That’s what it’s like working with AI in its most basic form. Tools such as ChatGPT, Claude, or the autocomplete-only version of GitHub Copilot sit just outside your development environment. They are detached from your codebase and wait for you to copy and paste your problem into a prompt window.

They respond with suggestions, explanations, even full code blocks. And when they work, it feels like magic. But it is a manual kind of magic. You’re doing the prompting, the interpreting, the adapting. The AI has no awareness of your repository structure, your test suite, or your business logic. It knows only what you tell it, and it forgets as soon as the session ends.

Still, this mode offers powerful advantages:

• 🛠️ Maximum flexibility: You can use the AI for virtually anything, from writing regular expressions to reverse-engineering algorithms, across languages, platforms, or domains.

• 💡 Creative problem-solving: The dialogue often surfaces novel approaches or reminds you of tools and techniques you might have overlooked.

• 🧪 Safe experimentation: Nothing changes in your codebase unless you make it happen. The AI is completely out-of-band, so there’s no risk of unintended changes.

But the limitations show up just as clearly:

• ❌ No access to project context: The AI doesn’t know your imports, abstractions, or naming conventions. Every session is a blank slate.

• 🔁 Inefficient iteration: You constantly switch between environments, translating back and forth between the AI’s suggestions and your working code.

• 🧠 Heavy reliance on prompt quality: Great results demand great prompts. And interpreting the output often requires as much mental effort as writing the code yourself.

This is the manual copilot tier. It is fast, flexible, and fully under your control. But it doesn’t scale well to deep collaboration. There’s no shared memory, no architectural insight — just bursts of assistance that live in isolation.

Still, for early ideation, exploratory coding, or surgical problem-solving, this approach remains an essential part of the modern developer’s toolkit.

You’re still driving. But now, you’re not driving alone.

Tier 2: Controlled Autopilot

Context-Aware Collaboration

Some AI tools no longer live in isolation. They work alongside you, within your editor, watching what you write and responding with context-aware suggestions that feel like collaboration rather than command.

This is the promise of controlled autopilot — a class of AI development tools that operate with your codebase instead of outside it. Tools like Cursor, GitHub Copilot X, Devin, and Amazon CodeWhisperer are embedded in your IDE. They understand the files you’re editing, track your comments, and offer inline suggestions that align with your project’s actual structure.

These assistants aren’t simply completing lines of code. They are participating in your flow of thought.

Here’s how the interaction typically unfolds:

• ✍️ You describe a task in a chat interface or begin implementing a new function.

• 🤖 The AI scans the surrounding context, including test files, import paths, and naming conventions.

• 💬 It generates a proposed solution, often tailored to your framework or language.

• ✅ You accept, modify, or reject the suggestion, all without leaving your editor.

It’s not a fully automated experience, but it lifts the cognitive load. You no longer need to re-explain everything. You don’t have to copy and paste code between windows. The AI learns from what it sees in your open files — and often from the broader structure of your workspace.

Why does this matter?

Because this is where real collaboration begins:

• ⚡ Increased velocity: You move faster with fewer keystrokes and fewer decisions to micromanage.

• 🧠 Reduced repetition: Common patterns and boilerplate are handled automatically, freeing your attention.

• 🔍 Greater contextual accuracy: The agent is working with real information from your environment, not generic assumptions.

But even here, there are important limitations:

• 🚧 Partial understanding: The AI might grasp local context but miss architectural intent or project-wide patterns.

• 🔄 Misaligned contributions: It won’t know your team’s conventions or modular boundaries unless those are clearly encoded.

• 🧪 Human oversight remains essential: You still review, test, and approve. The agent doesn’t own responsibility — you do.

This level of integration marks a shift from prompting to partnering. You’re no longer instructing an assistant from afar. You’re working with a collaborator that can see the road with you — and help you stay on course, one suggestion at a time.

Tier 3: Autonomous Autopilot

Agents That Drive the Code

Some AI agents no longer wait for your input or sit quietly in the editor. They scan the codebase, understand your issues, plan full solutions, and deliver results directly as pull requests. These systems take initiative — and in doing so, they cross a new threshold in AI-assisted development.

This is the world of full autopilot: agents that act independently across the entire development lifecycle. Tools such as OpenAI Codex, AutoDev, GPT-Engineer, or custom LangChain-based agents don’t just suggest lines of code — they act like contributors with full-stack awareness.

What makes them distinct is their scope and autonomy. These agents proactively:

• 🧭 Parse the structure and semantics of your codebase

• 🛠️ Generate implementations across multiple files independently, including refactors, new features, or bug fixes

• 🧪 Run or integrate with your CI pipelines to validate changes

• 📬 Deliver structured outputs such as pull requests, commit messages, and changelogs

Under the right conditions, these systems resemble tireless junior developers — ones who never forget syntax, always follow instructions, and can work around the clock without losing focus.

The potential here is significant:

• 🔁 Automating repetitive tasks: From scaffolding features to updating deprecated patterns or generating tests, agents can handle the tedious work reliably.

• ⏩ Accelerating prototyping: Entire solutions can be generated from high-level prompts or loosely defined design documents.

• 🔍 Improving traceability: Combined with logging and formatting standards, they create cleaner, more auditable histories of change.

Unsurprisingly, companies are starting to integrate these tools into internal workflows. They shine especially in areas involving predictable, low-risk updates — domains where human effort is often wasted.

But this autonomy comes with real risk.

Without structure, these agents can easily produce chaos. An unbounded agent in a loosely defined repo is like a new hire given access without documentation, supervision, or guidelines.

To truly benefit from this model, teams must prepare meticulously. These agents require:

• 🧱 Well-organized, modular project structures: Consistency in patterns helps agents navigate and generate valid solutions.

• ✅ Clearly scoped tasks with testable targets: The more well-defined the unit of work, the more reliable the output.

• 🧪 Robust, fast test suites: These allow agents to self-validate and iteratively refine their output before submission.

• 🔒 Permissioning and sandboxing: Agents should be limited in what they can touch and where they can operate — for both safety and clarity.

These tools aren’t fire-and-forget. They require engineering discipline, ongoing evaluation, and a thoughtfully constrained operating environment.

With structure and oversight, these agents become valuable teammates. But without clear boundaries, they can easily become liabilities. Their success depends entirely on the clarity of the environment they are invited to operate within.

The Hidden Architecture Beneath the Agent

Would you onboard a new developer without documentation, guidance, or a test suite — and expect them to thrive?

Of course not.

Yet many teams make this very mistake when deploying AI agents. They ask for high-quality output from systems that have no map, no constraints, and no feedback loops. Instead of producing value, these agents generate noise. The issue isn’t the intelligence of the system — it’s the absence of structure.

AI agents don’t improve with freedom, they improve with clarity of structure. Without reliable inputs, feedback, and boundaries, their contributions are random at best.

For fully autonomous systems to deliver value, and for context-aware copilots to remain useful, they must operate within a well-defined environment. That environment isn’t incidental. It must be designed, maintained, and treated as part of the engineering discipline itself.

Here are the critical foundations:

• ✅ Documentation that teaches
  Your README.md, AGENTS.md, and architecture guides should not just orient humans. They should be structured with clarity and precision so that agents can parse workflows, naming patterns, and behavioral expectations.

• ✅ Reproducible test environments
  A functioning CI pipeline is essential. Whether it’s Pytest, Jest, or another framework, agents must receive test feedback to evaluate their work — and ideally, use it to iterate.

• ✅ Formatting and linting standards
  Agents don’t guess your code style. Tools like .editorconfig, Prettier, Black, Ruff, and ESLint encode your conventions. When enforced automatically, these tools help agents produce clean, conforming code that matches your standards — without extra effort.

• ✅ Type hints and schema contracts
  Typing systems like MyPy, Pydantic, and TypeScript provide structural boundaries. They reduce ambiguity and guide agents toward valid implementations by describing expectations clearly and explicitly.

• ✅ Clearly organized repositories
  Agents rely on navigable project layouts. Monolithic folders, dynamic imports, and unclear abstractions create confusion. Modular structure, clear boundaries, and readable dependencies create the foundation for intelligent automation.

These components may seem peripheral. They rarely appear in demos or product launches. But this invisible scaffolding is what transforms an AI from a risk into a reliable teammate.

In effect, infrastructure becomes your language for shaping agent behavior. It communicates what matters, what’s allowed, and what success looks like. When this language is absent or inconsistent, even the most advanced models are forced to guess — and they often guess wrong.

With the right infrastructure in place, AI agents do more than generate output. They learn how to work within your standards, operate within your boundaries, and adapt to your expectations.
They don’t just write code — they start behaving like part of the team.

Building the AI Playground: Freedom Through Constraints

To empower agents, you must limit their freedom.

That idea may seem contradictory at first. Isn’t the goal of AI autonomy to let the system figure things out on its own? Isn’t the promise about handing off the tedious parts and trusting the machine to deliver results?

In part, yes. But only within clearly defined boundaries.

The groundwork for effective agents lies in structure — as we explored in The Hidden Infrastructure. But principles alone aren’t enough. To safely empower agents, teams must enforce those principles through technical boundaries. That means giving agents not a blank canvas, but a sandboxed environment with visible edges, explicit permissions, and automated feedback. True autonomy doesn’t come from removing constraints — it comes from designing the right ones.

In this context, you aren’t granting freedom. You’re constructing a framework that channels effort into safe, repeatable, and reliable outputs. You’re building boundaries or more precisely, a kind of containment zone designed not to punish, but to enable.

This controlled space might look like a prison, but it’s one built for productivity, not confinement.

Here’s how that environment is typically structured:

• 🔐 Limited write access: Agents should only be able to modify specific areas of the codebase — such as src/, tests/, or isolated feature branches. Configuration files, infrastructure code, and secrets should remain inaccessible by default.

• 📁 Task-level configuration files: Tools like .cursor/.mdc, .copilot/config.json, or custom-defined schemas give explicit instructions. These files tell agents what actions are permitted and what constraints must be respected.

• 🧪 Test-driven feedback loops: A test suite is more than a validator — it’s a teacher. When an agent makes a change and receives a failing test, it receives implicit guidance. You can even design test-first tasks where the success criteria are clearly defined upfront.

• 📜 Immutable checkpoints: Use CI pipelines, branch protection, and code review protocols to define legal boundaries. Agents can generate proposals, but they cannot push directly to production without human oversight.

• 🕵️ Structured logging and traceability: Every agent action should be auditable. You should know what prompt initiated a change, which files were affected, what tests were executed, and what the results were. This visibility is essential for debugging and accountability.

Too much freedom leads to fragile systems. But the right constraints create space for intelligent agents to succeed — safely and predictably.

Teams already apply this mindset in high-stakes engineering: isolating risky processes, sandboxing untrusted code, and granting only the necessary permissions. These same practices apply to autonomous AI agents, not out of fear, but out of respect for complexity.

Ultimately, this isn’t about holding the AI back. It’s about setting it up to succeed — not by letting it roam, but by teaching it how to build within the lines.
Inside a controlled environment, the agent can explore, adapt, and thrive. Outside of it, reliability fades.

AI Agents as Teammates: The New Developer Paradigm

AI is no longer confined to the realm of passive utilities. It is stepping forward as an active participant in how we build software — and that evolution changes everything.

Throughout this article, we’ve traced the spectrum of AI-assisted development as it unfolds across three distinct tiers.

At the foundational level, manual copilots offer assistance on demand. You guide every step. The AI listens, interprets your prompt, and responds — but it has no visibility into your codebase. It’s helpful, but isolated — a silent partner without a map.

In the middle tier, we encounter controlled autopilot. These agents operate inside your IDE, drawing from your files, test suites, and naming conventions. They suggest solutions that align with your architecture and coding style. You’re still in charge, but now the AI begins to behave like a skilled co-pilot — one who understands your trajectory and offers meaningful support along the way.

At the highest level lies full autopilot. Here, agents autonomously generate features, refactor code, run tests, and present their work as structured pull requests. Your role shifts from executor to reviewer. The agent becomes a contributor — not just a tool, but a collaborator.

What becomes increasingly clear as we ascend this spectrum is that autonomy demands architecture. Their power lies not in what they can do unassisted, but in how well they operate inside the systems we engineer around them.

They are not magical, nor are they innately intelligent. Their effectiveness is a direct reflection of the environment we design around them: the organization of our repositories, the robustness of our test suites, the consistency of our formatting, and the explicitness of our development workflows.

Put simply, AI agents perform best not as replacements for developers, but as augmented teammates. Like junior engineers, they thrive with structured onboarding, steady mentorship, and continuous feedback. What they need is not blind trust — but intentional design.

As this new paradigm takes hold, the teams who succeed won’t just be the ones who adopt AI. They’ll be the ones who engineer for it — who recognize that deploying intelligent agents means building intelligent systems.

And so, the true future of software development is not a rivalry between humans and machines.

It is a collaboration: human and machine, working in tandem within environments that foster clarity, reproducibility, and shared responsibility.

That future isn’t on the horizon.
It’s already underway.