I was debugging an agent a few weeks ago when I hit a problem that made me realize something fundamental about the shift we’re undergoing. The script had run, consumed a hundred thousand tokens, and returned an answer. But the answer was wrong. Not catastrophically wrong, just subtly, dangerously off.

The issue wasn’t that the model was bad. The problem was that I had no idea what the agent had thought while producing that answer. Which tools had it called? What information had it retrieved? What reasoning path had it wandered down? I had the input and the output, but the middle, the actual decision-making process, was a black box.

This mirrors the challenge I described in Everything Becomes an Agent. If our future architecture is a mesh of interacting agents, we cannot afford for them to be inscrutable. A single black box is a mystery; a system of black boxes is chaos.

This is the Observability Gap, and it is the first wall you hit when you move from prototype to production. You can build a working agent in an afternoon. You can give it tools, wire up a nice ReAct loop, and watch it dazzle you. But the moment you rely on it for something that matters, you realize you’re flying blind.

How do you know if your agent is working well? And more importantly, how do you fix it when it’s not?

Earlier in this series, I wrote about building guardrails and the Policy Engine that keeps agents from doing dangerous things. Observability is the complement to those guardrails. Guardrails define the boundaries; observability tells you whether the agent is respecting them, struggling against them, or quietly finding ways around them. One without the other is incomplete. A guardrail you can’t monitor is just a hope.

The Chain of Thought Problem

When you’re building traditional software, debugging is an exercise in logic. You set breakpoints, inspect variables, and trace execution. The flow is deterministic: if Input A produces Output B today, it will produce Output B tomorrow.

Agents don’t work that way. The same input can produce wildly different outputs depending on which tools the agent decides to call, how it interprets the results, and what “thought” it generates in that split second. The agent’s logic isn’t written in code; it’s written in natural language, scattered across multiple LLM calls, tool invocations, and iterative refinements.

I learned this the hard way with my Podcast RAG system. I’d ask it a question about a specific episode, and sometimes it would nail it, pulling the exact segment and synthesizing a perfect answer. Other times, it would search with the wrong keywords, get back irrelevant chunks, and confidently synthesize nonsense.

The model wasn’t hallucinating in the traditional sense. It was following a process. But I couldn’t see that process, so I couldn’t fix it.

That experience taught me the most important lesson about production agents: the final answer is the least interesting part. What matters is the chain of thought that produced it, every tool call, every intermediate result, every reasoning trace. Think of it as a flight recorder. When the plane lands at the wrong airport, the only way to understand what went wrong is to replay the entire flight.

Four Layers of Seeing

When I started building that flight recorder, I realized that “log everything” isn’t actually a strategy. You need structure. Through trial and error, and by studying how platforms like Langfuse and Arize Phoenix approach the problem, I’ve come to think of agent observability as having four distinct layers.

The first is the reasoning layer: the agent’s internal monologue where it decomposes your request into sub-tasks. This is where you catch the subtle bugs. When my Podcast RAG agent searched for the wrong keywords, the failure wasn’t in the tool call itself (which returned a perfectly valid HTTP 200). The failure was in the reasoning that chose those keywords. Without visibility into the “Thought” step of the ReAct loop, that kind of error is indistinguishable from an external system failure.

The second is the execution layer: the actual tool calls, their arguments, and the raw results. This is where you catch a different class of bug, one that’s becoming increasingly important. Tool hallucination. Not the model making up facts in prose, but the model calling a tool that doesn’t exist (you provided shell_tool but the model confidently calls bash_tool), fabricating a file path that isn’t real, or passing a string to a parameter that expects an integer. These are operational failures that cascade. I’ve seen an agent confidently pass a hallucinated document ID to a retrieval tool, get back an error, and then re-hallucinate a different invalid ID rather than change strategy. You only catch this if you’re logging the schema validation at the boundary between the model and the tool.

The third is the state layer: the contents of the agent’s context window at each decision point. Agents are stateful creatures. Their behavior at step ten is shaped by everything that happened in steps one through nine. And context windows are not infinite. As verbose tool outputs accumulate, relevant information gets pushed further and further from the model’s attention, a phenomenon researchers call “context drift” or the “Lost in the Middle” effect. Snapshotting the context at critical decision points lets you “time travel” during debugging. You can see exactly what the agent could see when it made its bad call.

The fourth is the feedback layer: error codes, user corrections, and signals from any critic or evaluator models. This layer tells you whether the agent is actually learning from its environment within a session, or just ignoring failure signals and looping. In frameworks like Reflexion, this feedback is explicitly wired into the next reasoning step. Watching this layer is how you know if your self-correction mechanisms are actually correcting.

But capturing these four layers independently isn’t enough. You need to bundle them into sessions: discrete, self-contained records of a single task from the moment the user makes a request to the moment the agent delivers (or fails to deliver) its result. A session is your unit of analysis. It’s the difference between having a pile of timestamped log lines and having a story you can read from beginning to end. When something goes wrong, you don’t want to grep through millions of events hoping to reconstruct what happened. You want to pull up session #47832 and replay the agent’s entire decision-making journey: what it thought, what it tried, what it saw, and how it responded to each result along the way.

This session-level thinking changes how you build your infrastructure. Every trace, every tool call, every context snapshot gets tagged with a session ID. Your dashboards stop showing you aggregate metrics and start showing you individual narratives. You can sort sessions by outcome (success, failure, abandonment), by cost (token consumption), or by duration, and immediately drill into the ones that matter. It’s the observability equivalent of going from reading a box score to watching the game film.

Making It Concrete

Here’s what this looks like in practice. Suppose you ask your agent to “check my calendar and suggest a time for a meeting.”

Without observability, you see:

Input: "Check my calendar and suggest a time for a meeting"
Output: "How about Thursday at 2pm?"

With observability across all four layers, you see the mind at work:

[REASONING] User wants to schedule a meeting. I need to:
1. Check their calendar for availability
2. Consider team availability
3. Suggest an optimal time
[TOOL CALL] get_calendar(user_id="allen", days=7)
[TOOL RESULT] Returns 45 events over next 7 days
[STATE] Context window: 2,847 tokens used
[REASONING] Analyzing free slots. User has:
- Monday 2pm-4pm free
- Thursday 2pm-4pm free
- Friday all day booked
[TOOL CALL] get_team_availability()
[TOOL RESULT] Team members mostly available Thursday afternoon
[REASONING] Thursday 2pm works for both user and team.
[FEEDBACK] No errors. Response generated.
[RESPONSE] "How about Thursday at 2pm?"

Suddenly, the black box is transparent. If the suggestion is wrong, you can see exactly why. Maybe the calendar tool returned incomplete data. Maybe the team availability check failed silently. Maybe the agent’s definition of “optimal” means “soonest” rather than “best for focus time.”

This kind of visibility saved me countless hours when building Gemini Scribe. Users would report that the agent “didn’t understand” their request, which is about as useful as telling your mechanic “the car sounds funny.” But when I turned on debug logging and pulled up the console output, I could see exactly where the confusion happened, usually in how the agent interpreted the file context or which notes it decided were relevant. The fix was never a mystery once I could see the reasoning. All of this logging is to the developer console and off by default, which is an important distinction. You want observability for yourself as the builder, not surveillance of your users.

The Standards Are Coming

For my own production agents, I’ve settled on a layered approach. Structured logging captures every action in machine-parseable JSON. A unique trace ID stitches together every LLM call and tool invocation into a single narrative flow.

But we are also seeing the industry mature beyond “roll your own.” The critical development here is the adoption of the OpenTelemetry (OTel) standard for GenAI. The OTel community has published semantic conventions that define a standard schema for agent traces: things like gen_ai.system (which provider), gen_ai.request.model (which exact model version), gen_ai.tool.name (which tool was called), and gen_ai.usage.input_tokens (how many tokens were consumed at each step).

This matters because it means an agent built with LangChain in Python and an agent built with Semantic Kernel in C# can produce traces that look structurally identical. You can pipe both into the same Datadog or Langfuse dashboard and analyze them side by side. You aren’t locked into a proprietary debugging tool; you can stream your agent’s thoughts into the same infrastructure you use for the rest of your stack.

It also enables what I think of as “boundary tracing,” where you instrument the stable interfaces (the HTTP calls, the tool invocations) rather than hacking into the agent’s internal logic. You get visibility without coupling your observability to a specific framework. That’s important, because if there’s one thing I’ve learned building in this space, it’s that frameworks change fast.

If you’re wondering where to start, here’s my honest advice: don’t wait for the perfect stack. Start with structured JSON logs and a session ID that ties each task together end-to-end. That alone gives you something you can grep, filter, and replay. Once you outgrow that (and you will, faster than you expect), graduate to an OTel-based pipeline. The good news is that many agent frameworks are adding robust hook mechanisms that let you tap into the agent lifecycle (before and after tool calls, on reasoning steps, on errors) without modifying your core logic. These hooks make it straightforward to plug in your telemetry from the start. The key is to instrument early, even if you’re only logging to a local file. Retrofitting observability into an agent that’s already in production is significantly harder than building it in from the beginning.

The Price of Transparency

Here’s the tension no one wants to talk about: full observability is expensive.

Autonomous agents are verbose by nature. A single reasoning step might generate hundreds of tokens of internal monologue. A RAG retrieval might pull megabytes of document context. If you log the full payload for every transaction, your storage costs can rival the cost of the LLM inference itself. I’ve seen reports of evaluation runs consuming over 100 million tokens, with more than 60% of the cost attributed to hidden reasoning tokens.

In production, you need sampling strategies. The approach I’ve landed on borrows from traditional distributed systems. Keep 100% of traces that result in errors or negative user feedback, because every failure is a learning opportunity. Keep traces that exceed your latency threshold (P95 or P99), because slow agents are often stuck agents. And for everything else, a small random sample (1-5%) is enough to establish your baseline and spot trends.

For storage, I use a tiered approach. Recent and failed traces go into a fast database for immediate querying. Older successful traces get compressed and moved to cold storage, where they can be pulled back if needed for deeper analysis. It’s not glamorous, but it keeps costs manageable without sacrificing the ability to debug the things that matter. In my own setup, this sampling and tiering strategy keeps observability overhead to roughly 15-20% of my inference spend. Without it, I was on track to spend more on storing agent thoughts than on generating them.

Evaluation Beyond Unit Tests

Logging tells you what happened. Evaluation tells you if it was any good.

This is where agents diverge sharply from traditional software. You can’t write a unit test that asserts function(x) == y. The whole point of an agent is to make decisions, and decisions must be evaluated on quality, not just syntax.

As Gemini Scribe grew more capable, I had to develop a new kind of test suite. I track Task Success Rate (did the agent accomplish what the user asked?), Tool Use Accuracy (did it read the right files and use the right tools for the job?), and Efficiency (did it burn 50 steps to do a 2-step task?).

But here’s the number that keeps me up at night. Because agents are non-deterministic, a single run is statistically meaningless. You have to run the same evaluation multiple times and look at distributions. Researchers distinguish between Pass@k (the probability that at least one of k attempts succeeds) and Pass^k (the probability that all k attempts succeed). Pass@k measures potential. Pass^k measures reliability.

The math is sobering. If your agent has a 70% success rate on a single attempt, its Pass^3 (succeeding three times in a row) drops to about 34%. Scale that to a real workflow where the agent needs to perform ten sequential steps correctly, and even a 95% per-step success rate gives you only about a 60% chance of completing the full task. This is the compounding probability of failure, and it’s why “works most of the time” isn’t good enough for production.

This kind of evaluation framework pays for itself the moment a new model drops. When Google released Flash 2.0, I was excited about the cost savings, but would it perform as well as Pro? I ran my eval suite on the same tasks with both models, and the results were more nuanced than I expected. For simple tasks like reformatting text or fixing grammar, Flash was just as good. For complex multi-step reasoning, particularly in my Podcast RAG system, Pro was noticeably better. The eval suite gave me the data to keep Pro where it mattered.

Then Flash 3 came out, and the eval suite surprised me in the other direction. I ran the same benchmarks expecting similar trade-offs, but Flash 3 handled the Podcast RAG tasks so well that I moved the entire system off of 2.5 Pro. Without evals, I might have assumed the old trade-off still held and kept paying for a model I no longer needed. The point isn’t that one model is always better. The point is that you can’t know without measuring, and the landscape shifts under your feet with every release.

The real breakthrough in my own workflow came when I started using an agent to evaluate itself. I built a separate “Evaluation Agent” that reviews the logs of the “Worker Agent.” It scores performance based on a rubric I defined: did it confirm the action before executing? Was the response grounded in retrieved context? Was the tone appropriate?

This LLM-as-a-Judge pattern is powerful, but it comes with caveats. Research shows these evaluator models have their own biases, particularly a tendency to prefer longer answers regardless of quality and a bias toward their own outputs. To calibrate mine, I built a small “golden dataset” of traces that I graded by hand, then tuned the evaluator’s prompt until its scores matched mine. It’s not perfect, but it spots patterns I miss, like a tendency to over-rely on search when a simple calculation would do.

When Things Go Wrong

The research into agentic failure modes has identified three patterns that I see constantly in my own work.

The first is looping. The agent searches for “pricing,” gets no results, then searches for “pricing” again with exactly the same parameters. It’s stuck in a local optimum of reasoning, unable to update its strategy based on the observation that it failed. The simplest fix is a state hash: you hash the (Thought, Action, Observation) tuple at each step and check it against a sliding window of recent steps. If you see a repeat, you force the agent to try something different. For “soft” loops where the agent slightly rephrases but semantically repeats itself, embedding similarity between consecutive reasoning steps catches the pattern. And above all, production agents need circuit breakers: hard limits on steps, tool calls, or tokens per session. When the breaker trips, the agent escalates to a human rather than continuing to burn resources.

The second is tool hallucination. I mentioned this earlier, but it deserves its own spotlight. The most robust defense is constrained decoding, where libraries like Outlines or Instructor use the tool’s JSON schema to build a finite state machine that masks out invalid tokens during generation. If the schema expects an integer, the system sets the probability of all non-digit tokens to zero. It mathematically guarantees that the agent’s tool call will be valid. This moves validation from “check after the fact” to “ensure during generation,” which is a fundamentally better position. A practical note: full constrained decoding (the FSM approach) requires control over the inference engine, so it works with locally-hosted models or providers that expose logit-level access. If you’re calling a hosted API like Gemini or OpenAI, Instructor-style libraries can still enforce schema validation by wrapping the response in a Pydantic model and retrying on parse failure. It’s not as elegant as preventing bad tokens from ever being generated, but it catches the same class of errors.

The third is silent abandonment. The agent hits an ambiguity or a tool failure, and instead of trying an alternative, it politely apologizes and gives up. “I’m sorry, I couldn’t find that information.” This is often a side effect of RLHF training, where the model has learned that apologizing is a safe response to uncertainty. The Reflexion pattern combats this by forcing the agent to generate a self-critique when it fails (“I searched with the wrong term”) and storing that critique in a short-term memory buffer. The next reasoning step is conditioned on this reflection, pushing the agent to generate a new plan rather than surrender. Research shows this kind of “verbal reinforcement” can improve success rates on complex tasks from 80% to over 90%.

The Self-Improving System

Moving from prototype to production isn’t about adding features; it’s about shifting your mindset. A prototype proves that something can work. A production system proves that it works reliably, measurably, and transparently. But the real unlock comes when you realize that production isn’t the end of the development lifecycle. It’s the beginning of something more powerful.

Remember those sessions I mentioned, the bundled records of every task your agent attempts? Once you have a critical mass of them, you’re sitting on a goldmine. And this is where I think the story gets really interesting: you can point a different AI system at your session archive and ask it to find the patterns you’re missing.

I’ve started doing this with my own agents. The workflow is straightforward: I have a script that runs weekly, pulls the last seven days of sessions from my trace store, filters for failures and anything above P90 latency, and exports them as structured JSON. I then feed that batch to a separate, more capable evaluator model. Not the lightweight rubric-scorer I use for real-time evaluation, but a model with a broader mandate and a carefully written prompt: look across these sessions and tell me what you see. Where is the agent consistently struggling? Which tool calls tend to precede failures? Are there categories of user requests that reliably lead to abandonment or looping? I ask it to return its findings as a ranked list of patterns with supporting session IDs, so I can verify each observation myself.

The results have been genuinely surprising. The evaluator flagged a cluster of sessions where users were asking questions about the corpus itself, things like “how many of these podcasts are about guitars?” or “which shows cover AI the most?” The agent would gamely try to answer by searching transcripts, but it was never going to get there because I hadn’t indexed podcast descriptions. Each individual session just looked like a search that came up short. It was only in aggregate that the pattern became clear: users wanted to explore the collection, not just search within it. That finding led me to index descriptions as a new data source, and a whole category of previously failing queries started working.

This is what the industry calls the Data Flywheel: production data feeding back into development, continuously tightening the loop between user intent and agent capability. Your prompt logs become your reality check, revealing how users actually talk to your system versus how you imagined they would.When you cluster those real-world prompts (something as straightforward as embedding them and running HDBSCAN), you start finding these gaps systematically. That’s your roadmap for what to build next.

And the flywheel compounds. Better observability produces richer sessions. Richer sessions give the evaluator more to work with. Better evaluations lead to targeted improvements. Targeted improvements produce better outcomes, which produce more informative sessions. Each rotation makes the system a little smarter, a little more aligned with what users actually need.

To be clear: this isn’t the agent autonomously rewriting itself. I’m the one who reads the evaluator’s findings, verifies them against the session data, and decides what to change. Maybe I update a system prompt, add a new tool, or adjust a circuit breaker threshold. The AI surfaces the patterns; the human decides what to do about them. It’s the same human-on-the-loop philosophy I described in the last post, applied to the development cycle itself.

Together, these layers transform a clever demo into a system you can trust. Because in the age of agents, trust isn’t built on magic. It’s built on the ability to see the trick.

Throughout this series, we’ve been building up the theory: what agents are, how they think, what tools they need, how to keep them safe, and now how to make sure they’re actually working. In the next installment, I want to move from theory to practice. We’ll look at agents in the wild, real-world case studies in customer support, software development, and personal productivity, and what they tell us about how this technology is actually changing the way we work.

Welcome back to The Agentic Shift. Over the past eight installments, we’ve built our agent from the ground up, giving it a brain to think, memory to learn, a toolkit to act, instructions to follow, guardrails for safety, and a framework to build on. But there’s been an elephant in the room this whole time: our agent is alone.

I was sitting at my desk late last night, staring at three different windows on my monitor, feeling like a digital switchboard operator from the 1950s.

In one window, I had Helix, my text editor, where I was writing a Python script. In the second, I had a terminal running a deep research agent I’d built for Gemini CLI. In the third, I had a browser open to a documentation page.

Here’s the thing: Gemini CLI is brilliant, but it’s blind. It couldn’t see the code I had open in Helix. It couldn’t read the documentation in my browser. When it found a critical library update, I had to manually copy-paste the relevant code into the terminal. When I wanted it to understand an error, I had to copy-paste the stack trace. I was the glue, the slow, error-prone, context-losing glue.

We have spent this entire series building a digital Robinson Crusoe. In Part 1, we gave our agent a brain. In Part 4, we gave it tools. But watching my own workflow fragment into disjointed copy-paste loops, I realized we’ve hit a wall. We have built brilliant, isolated sparks of intelligence, but we haven’t built the wiring to connect them.

This fragmentation is the single biggest bottleneck in the agentic shift. But that is changing. We are witnessing the birth of the protocols that will turn these isolated islands into a network. We are moving from building agents to building the Internet of Agents.

The Struggle Before Standards

I tried to fix this myself, of course. We all have. I wrote brittle Python scripts to wrap my CLI tools. I tried building a mega-agent that had every possible API key hardcoded into its environment variables. I even built my own agentic TUI that explored many interesting ideas, but ultimately wasn’t the right solution.

My lowest moment came when I spent several evenings and weekends building an Electron-based AI research and writing application. The vision was grand: a unified workspace where I could query multiple AI models, organize research into projects, and write drafts with AI assistance, all in one window. I built a beautiful sidebar for project navigation, a markdown editor with live preview, a chat interface that could talk to Gemini, and a “sources” panel for managing references. By the time I stepped back to evaluate what I’d built, I had thousands of lines of TypeScript, a complex state management system, and an app that was slower than just using the terminal. Worse, it didn’t actually solve my problem. I still couldn’t get the AI to see what was in my other tools. I’d built a new silo, not a bridge. The repo still sits on my hard drive, unopened.

Every solution felt like a band-aid. The problem wasn’t that I couldn’t write the code; it was that I was trying to solve an ecosystem problem with a point solution.

The Anatomy of Connection

To solve this, we don’t just need “better agents.” We need a common language. The industry is converging on three distinct protocols, each solving a different layer of the communication stack: MCP for tools, ACP for interfaces, and A2A for collaboration.

Why three protocols instead of one? For the same reason the internet isn’t just “one protocol.” Think of it like the networking stack: TCP/IP handles reliable data transmission, HTTP handles document requests, and SMTP handles email. Each layer solves a distinct problem, and trying to collapse them into one mega-protocol would create an unmaintainable mess. The same logic applies here. MCP solves the “how do I use this tool?” problem. ACP solves the “how do I show this to a human?” problem. A2A solves the “how do I collaborate with another agent?” problem. They’re designed to compose, not compete.

The Internal Wiring of MCP

The Model Context Protocol (MCP), championed by Anthropic, represents the agent’s Internal Wiring. It answers the fundamental question: How does an agent perceive, act upon, and understand the world?

It’s easy to dismiss MCP as just “standardized tool calling,” but that misses the architectural shift. MCP creates a universal substrate for context, built on three distinct pillars. First, there are Resources, the agent’s sensory input that allows it to read data (files, logs, database rows) passively. Crucially, MCP supports subscriptions, meaning an agent can “watch” a log file and wake up the moment an error appears. Next are Tools, the agent’s hands, allowing for action: executing a SQL query, hitting an API, or writing a file. Finally, there are Prompts, perhaps the most overlooked feature, which allow domain experts to bake workflows directly into the server. A “Git Server” doesn’t just expose git commit; it can expose a generate_commit_message prompt that inherently knows your team’s style guide and grabs the current diff automatically.

Here is what that “handshake” looks like (from Anthropic’s MCP specification). It’s not magic; it’s a strict contract that turns an opaque binary into a discoverable capability:

{
  "jsonrpc": "2.0",
  "method": "tools/list",
  "result": {
    "tools": [
      {
        "name": "query_database",
        "description": "Execute a SELECT query against the local Postgres instance",
        "inputSchema": {
          "type": "object",
          "properties": {
            "sql": { "type": "string" }
          }
        }
      }
    ]
  }
}

Now, any agent (whether it’s running in Claude Desktop, Cursor, or a custom script) can “plug in” to my Postgres server and immediately know how to use it. It solves the N × M integration problem forever.

A skeptical reader might ask: “How is this different from REST or OpenAPI?” It’s a fair question. On the surface, MCP looks like “JSON-RPC with a schema,” and that’s not wrong. But the difference is what gets standardized. OpenAPI describes how to call an endpoint; MCP describes how an agent should understand and use a capability. The schema isn’t just for validation. It’s for reasoning. An MCP tool description is a prompt fragment that teaches the model when and why to use the tool, not just how.

But here’s where I need to offer some nuance, because protocol boosterism can obscure practical reality.

As Simon Willison observed in his year-end review, MCP’s explosive adoption may have been partly a timing accident. It launched right as models got reliable at tool-calling, leading some to confuse “MCP support” with “tool-calling ability.” More pointedly, he notes that for coding agents, “the best possible tool for any situation is Bash.” If your agent can run shell commands, it can use gh for GitHub, curl for APIs, and psql for databases, no MCP server required.

I’ve felt this myself. When I’m working in Gemini CLI, I rarely reach for an MCP server. The GitHub CLI (gh) is faster and more capable than any MCP wrapper I’ve tried. The same goes for git, docker, and most developer tools with good CLIs.

So when does MCP make sense? I see three clear cases. First, when there’s no CLI (for example with my MCP service for Google Workspace), since many SaaS products expose APIs but no command-line interface. An MCP server is the natural wrapper. Second, when you need subscriptions, since MCP’s ability to “watch” a resource and push updates to the agent is something CLIs can’t do cleanly. Third, when you’re crossing network boundaries, since an MCP server can run on a remote machine and expose capabilities securely, which is harder to orchestrate with raw shell access.

The real insight here is about context engineering. MCP servers bring along a lot of context for every tool (descriptions, schemas, the full capability surface). For some workflows, that richness is valuable. But Anthropic themselves acknowledged the overhead with their Skills mechanism, a simpler approach where a Skill is just a Markdown file in a folder, optionally with some executable scripts. Skills are lightweight and only load when needed. MCP and Skills aren’t competing; they’re different tools for different context budgets.

Giving the Agent a Seat at the Keyboard

If MCP is the agent’s internal wiring, the Agent Client Protocol (ACP) is its window to the world.

I like to think of this as the LSP (Language Server Protocol) moment for the agentic age. Before LSP, if you wanted to support a new language in an IDE, you had to write a custom parser for every single editor. It was a nightmare of N × M complexity. ACP solves the same problem for intelligence. It decouples the “brain” from the “UI.”

This is why the collaboration between Zed and Google is so critical. When Zed announced bring your own agent with Google Gemini CLI integration, they weren’t just shipping features. They were standardizing the interface between the client (the editor) and the server (the agent). Intelligence became swappable. I can run a local Gemini instance through the same UI that powers a remote Claude agent.

The core of ACP is Symmetry. It’s not just the editor sending prompts to the agent. Through ACP, an editor like Zed (the reference implementation) can tell the agent exactly where your cursor is, what files you have open, and even feed it the terminal output from a failed build. The agent, in turn, can request to edit a specific line or show you a diff for approval.

I’ve been seriously thinking about building ACP support for Obsidian. I already built Gemini Scribe, an agent that lives inside Obsidian for research and writing assistance, but it’s hardcoded to Gemini. With ACP, I could make Obsidian a universal agent host, letting users bring whatever intelligence they prefer into their knowledge management workflow.

This turns the editor into the ultimate guardrail. Because the agent communicates its intent through a standardized protocol, the editor can pause, show the user exactly what’s about to happen, and wait for that “Approve” click. It’s the infrastructure that makes autonomous coding safe.

But the real magic isn’t just safety; it’s ubiquity. ACP liberates the agent from the tool. It means you can bring your preferred intelligence to whatever surface helps you flow. We are already seeing the ecosystem explode beyond just Zed.

For the terminal die-hards, there is Toad, a framework dedicated entirely to running ACP agents in a unified CLI. And for the VIM crowd, the CodeCompanion project has brought full ACP support to Neovim. This is the promise of the protocol: write the agent once, and let the user decide if they want to interact with it in a modern GUI, a raw terminal, or a modal editor from the 90s. The intelligence remains the same; only the glass changes.

When Agents Meet Strangers

Finally, we have the “Internet” layer: Agent-to-Agent (A2A).

While MCP connects an agent to a thing, and ACP connects an agent to a person, A2A connects an agent to society. It addresses the “lonely agent” problem by establishing a standard for horizontal, peer-to-peer collaboration.

This protocol, pushed forward by Google and the Linux Foundation, introduces a profound shift in how we think about distributed systems: Opaque Execution.

In traditional software, if Service A talks to Service B, Service A needs to know exactly how to call the API. In A2A, my agent doesn’t care about the how; it cares about the goal. My “Travel Agent” can ask a “Calendar Agent” to “find a slot for a meeting,” without knowing if that Calendar Agent is running a simple SQL query, consulting a complex rules engine, or even asking a human secretary for help.

This negotiation happens through the Agent Card, a machine-readable identity file hosted at a standard /.well-known/agent.json endpoint. It solves the “Theory of Mind” gap, allowing one agent to understand the capabilities of another. Here’s what one looks like:

{
  "name": "Calendar Agent",
  "description": "Manages scheduling, finds available slots, and coordinates meetings across time zones.",
  "url": "https://calendar.example.com",
  "version": "1.0.0",
  "capabilities": {
    "streaming": true,
    "pushNotifications": true
  },
  "skills": [
    {
      "id": "find-meeting-slot",
      "name": "Find Meeting Slot",
      "description": "Given a list of participants and constraints, finds optimal meeting times.",
      "inputSchema": {
        "type": "object",
        "properties": {
          "participants": { "type": "array", "items": { "type": "string" } },
          "duration_minutes": { "type": "integer" },
          "preferred_time_range": { "type": "string" }
        }
      }
    }
  ],
  "authentication": {
    "schemes": ["oauth2", "api_key"]
  }
}


When my Travel Agent encounters a scheduling problem, it doesn’t need to know how the Calendar Agent works internally. It reads this card, understands the agent can “find meeting slots,” and delegates the task. The Calendar Agent might use Google Calendar, Outlook, or a custom database. My agent doesn’t care.

But the real breakthrough is the Task Lifecycle. A2A tasks aren’t just request-response loops; they are stateful, modeled as a finite state machine with well-defined transitions:

• Submitted: The task has been received but work hasn’t started.
• Working: The agent is actively processing the request.
• Input-Required: The agent needs clarification before continuing. This is the key innovation: the agent can pause, ask “Do you prefer aisle or window?”, and wait indefinitely.
• Completed: The task finished successfully.
• Failed: Something went wrong. The response includes an error message and optional retry hints.
• Canceled: The requesting agent (or human) aborted the task.

This state machine brings the asynchronous, messy reality of human collaboration to the machine world. A task might sit in Input-Required for hours while waiting for a human to respond. It might transition from Working to Failed and back to Working after a retry. The protocol handles all of this gracefully.

Finding Agents You Can Trust

But let’s not declare victory just yet. We are seeing the very beginning of this shift, and the “Internet of Agents” brings its own set of dangers.

As we move from tens of agents to millions, we face a massive Discovery Problem. In a global network of opaque execution, how do you find the right agent? And more importantly, how do you trust it?

It’s not enough to just connect. You need safety guarantees. You need to know that the “Travel Agent” you just hired isn’t going to hallucinate a non-refundable booking or, worse, exfiltrate your credit card data to a malicious third party.

This is the focus of recent research on multi-agent security, which highlights that protocol compliance is only the first step. We need mechanisms for Behavioral Verification, ensuring that an agent does what it says it does.

What does verification look like in practice? Today, it’s mostly manual and ad-hoc. You might:

• Audit the agent’s logs to see what actions it actually took versus what it claimed.
• Run it in a sandbox with fake data before trusting it with real resources.
• Require human approval for high-stakes actions (the “Human-in-the-Loop” pattern we explored in Part 6).
• Check reputation signals: who built this agent? What’s their track record?

But these are stopgaps. The dream is automated verification: cryptographic proofs that an agent behaved according to its advertised policy, or sandboxed execution environments that can mathematically guarantee an agent never accessed unauthorized data. We’re not there yet.

Whether the solution looks like a decentralized “Web of Trust” (where agents vouch for each other, like PGP key signing) or a centralized “App Store for Agents” (where a trusted authority vets and signs off on agents) remains to be seen. My bet is we’ll see both: curated marketplaces for enterprise use cases, and open registries for the long tail. But solving the discovery and safety problem is the only way we move from a toy ecosystem to a production economy.

The Foundation of the Future

What excites me most isn’t just the code. It’s the governance.

We have seen this movie before. In the early days of the web, proprietary browser wars threatened to fracture the internet. We risked a world where “This site only works in Internet Explorer” became the norm. We avoided that fate because of open standards.

The same risk exists for agents. We cannot afford a future where an “Anthropic Agent” refuses to talk to an “OpenAI Agent” that won’t talk to a “Google Agent.”

That is why the formation of the Agentic AI Foundation by the Linux Foundation is the most important news you might have missed. By bringing together AI pioneers like OpenAI and Anthropic alongside infrastructure giants like Google, Microsoft, and AWS under a neutral banner, we are ensuring that the “Internet of Agents” remains open. This foundation will oversee the development of protocols like A2A, ensuring they evolve as shared public utilities rather than walled gardens. It is the guarantee that the intelligence we build today will be able to talk to the intelligence we build tomorrow.

The New Architecture of Work

When we combine these three protocols, the fragmentation dissolves.

Imagine I am back in Zed (connected via ACP). I ask my coding agent to “Add a secure user profile page.” Zed sends my cursor context to the agent. The agent reaches for MCP to query my local database schema and understand the users table. Realizing this touches PII, it autonomously pings a “Security Guardrail Agent” via A2A to review the proposed code. Approval comes back, and my local agent writes the code directly into my buffer.

I didn’t switch windows once.

But what happens when things go wrong? Let’s say the Security Guardrail Agent rejects the code because it detected a SQL injection vulnerability. The A2A task transitions to Failed with a structured error: {"reason": "sql_injection_detected", "line": 42, "suggestion": "Use parameterized queries"}. My local agent receives this, understands the failure, and either fixes the issue automatically or surfaces it to me with context. The rejection isn’t a dead end; it’s a conversation.

Or imagine the MCP server for my database is unreachable. The agent doesn’t just hang. It receives a timeout error and can decide to retry, fall back to cached schema information, or ask me whether to proceed without database context. Robust failure handling is baked into the protocols, not bolted on as an afterthought.

Where We Are Today

I want to be honest about maturity. These protocols are real and shipping, but the ecosystem is young.

MCP is the most mature. Just about everything supports it now: coding tools, virtualization environments, editors, even mobile apps. There are hundreds of community MCP servers for everything from Notion to Kubernetes. If you want to try this today, MCP is the on-ramp.

ACP is newer but moving fast. Zed is the reference implementation, with Neovim (via CodeCompanion) and terminal clients (via Toad) close behind. There are also robust client APIs for many languages, making ACP an interesting interface for controlling local agentic applications. If your editor doesn’t support ACP yet, you’ll likely be using proprietary plugin APIs for now.

A2A is the most nascent. Google and partners announced it in mid-2025, and the specification is still evolving. There aren’t many production A2A deployments yet. Most multi-agent systems today use custom protocols or framework-specific solutions like CrewAI or LangGraph. But the spec is public, the governance is in place, and early adopters are building.

If you’re starting a project today, my advice is: use MCP for tool integration, use whatever your editor supports for the UI layer, and keep an eye on A2A for future multi-agent workflows. The pieces are coming together, but we’re still early.

And yet, this isn’t science fiction. The protocols are here today. The “Internet of Agents” is booting up, and for the first time, our digital Robinson Crusoes are finally getting a radio.

But a radio is only as good as the conversations it enables. In our next post, we’ll move from protocols to practice and explore what happens when agents don’t just connect, but actually collaborate: forming teams, delegating tasks, and solving problems no single agent could tackle alone.

Welcome back to The Agentic Shift. Over the past seven installments, we’ve carefully dissected the anatomy of an agent, peered into its different modes of thinking, mapped out its memory systems, examined its toolkit, learned how to guide its behavior, erected necessary safety guardrails, and tackled the challenge of managing its finite attention. We’ve essentially built a conceptual blueprint for an autonomous AI partner.

The ‘Simple Loop’ Fallacy

While some people will say that an agent is nothing more than a model running in a loop and using tools, if you implement your own agent you will find that there are a lot of details missing in that simple statement. Every model has its quirks—handling parallel tool calls, progressive context compression, or input context window management—all come with frustrations, and that’s before you start to deal with the unique features of each model SDK. Each time, you’ll find yourself spending 80% of your time on what one might call “undifferentiated heavy lifting” the complex but repetitive plumbing that every agent needs but that adds no unique value to your specific application.

This brings to mind something François Chollet said recently: to truly understand a concept, you have to “invent” it yourself. Understanding is an “active, high-agency, self-directed process of creating and debugging your own mental models”. And anyone who builds an agent from scratch has definitely been creating and debugging. The hands-on struggle teaches precisely where the real engineering challenges lie: state persistence across sessions, secure tool execution, intelligent context curation, and robust error handling.

The lesson is clear: build from scratch once to truly understand the fundamentals, but use a framework for everything after that. The real decision isn’t if you should use one, but which one to choose. This choice hinges on your specific needs—from single vs. multi-agent architectures to your existing cloud ecosystem, production timeline, and desired level of control. While my personal stack now favors the Google ADK for production and raw code for learning, I’ve learned the framework is just the scaffolding, not the building itself.

What Frameworks Actually Solve

The complexity frameworks address isn’t trivial. Based on my experience and research, they solve four fundamental challenges:

1. The Stateless → Stateful Transformation

LLM APIs are fundamentally stateless—each call has no memory of the previous one. Creating an agent that remembers your preferences, learns from interactions, and maintains context requires sophisticated external memory architecture, as we explored in Part 3. Frameworks provide battle-tested solutions, from simple conversation buffers to complex integrations with vector databases for semantic memory and knowledge graphs for entity relationships.

Take a customer service agent that needs to remember a user’s issue across multiple sessions. Without a framework, you’re writing database schemas, managing session state, implementing conversation history pruning, and building retrieval pipelines. Frameworks like LangGraph, however, handle much of this with just a few lines of configuration.

2. The Tool Orchestration Loop

Giving an agent “hands,” as we discussed in Part 4, means building a robust runtime that can generate machine-readable tool definitions, parse the LLM’s tool-calling decisions, validate and securely execute calls, handle errors gracefully, and feed results back into reasoning. I’ve written this loop several times. Each time, I discovered new edge cases. What happens when a tool times out? When the LLM hallucinates a non-existent function? When it tries to pass a string to an integer parameter? Frameworks have discovered these edge cases already and handle them elegantly.

3. The Context Window Crisis

As I explored in Part 7, context windows are finite and precious. Without active management, they fill with noise: old conversations, verbose tool outputs, redundant information. Frameworks offer automated strategies like recursive summarization and intelligent pruning that maintain signal while discarding noise.

4. The Security Minefield

An agent that can delete files, send emails, or execute code is a loaded weapon. The attack vectors we covered in Part 6 (prompt injection, tool manipulation, data exfiltration) are novel. Frameworks provide architectural patterns for sandboxing, human-in-the-loop approval, and policy enforcement that would take months to build from scratch.

Different Philosophies for Different Problems

The agent framework landscape isn’t just diverse; it’s philosophically fragmented. Each framework embodies a distinct worldview about how agents should be built and managed.

The Enterprise Architects: Google ADK and Microsoft’s Unified Framework

Google’s Agent Development Kit represents the “enterprise-first” philosophy. It treats agents as first-class software artifacts—with proper testing, versioning, and observability. The framework’s hierarchical multi-agent support is invaluable for scaling from a single agent to a team of specialists. The code can be verbose and the learning curve steep, but the production reliability is a key feature.

Microsoft’s newly unified Agent Framework (merging AutoGen’s innovation with Semantic Kernel’s enterprise features) takes a different approach: “conversation as computation.” Instead of explicit orchestration, agents collaborate through structured dialogue. It’s fascinating to watch, almost like a Slack channel where AI team members actually get work done.

The Developer Experience Champions: LangChain’s Evolution

LangChain’s journey mirrors the entire field’s maturation. It started with “chains”, linear sequences of operations that were intuitive but limited. The introduction of LangChain Expression Language (LCEL) formalized this into a powerful pipe syntax: prompt | model | parser.

But then came LangGraph, acknowledging what many developers learned the hard way: agents need cycles, not just chains. This directly relates to the cognitive patterns we discussed in Part 2, where simple linear “Plan-and-Execute” patterns give way to more complex, graph-based reasoning. LangGraph models workflows as stateful graphs where nodes are functions and edges are conditional logic. It’s more complex but infinitely more powerful, and has become a popular choice for developers who need fine-grained control over agent behavior.

The Minimalist’s Choice: OpenAI Agents SDK

OpenAI’s official open-source, lightweight framework takes a refreshingly minimal approach. It’s model-agnostic and provides just enough structure to be helpful without being prescriptive. Perfect for developers who want to build custom, multi-agent logic from the ground up without fighting framework opinions.

The Intuitive Collaborators: CrewAI’s Role-Playing Revolution

CrewAI took a radically different approach: what if we just described agents like job postings? You define a “Senior Research Analyst” with a goal and backstory, a “Technical Writer” with their own expertise, and let them collaborate naturally.

This model has proven to be remarkably effective for content creation pipelines, such as having a Researcher, Writer, and Editor collaborate on a blog post. The framework is designed to handle delegation, task management, and inter-agent communication transparently. In effect, you write what feels like HR documentation and get a functioning multi-agent system.

The Pythonic Pragmatists: Phidata

Phidata embodies “AI assistants as code”—clean, object-oriented Python with minimal magic. It’s a “batteries included” framework where an Assistant can be instantiated, configured with tools and a knowledge base, and deployed with a built-in UI.

For Retrieval-Augmented Generation (RAG) applications, Phidata is particularly well-suited. It is designed to handle vector database complexity, provides pre-built knowledge base classes, and manages the retrieval pipeline transparently.

Visual and Node-Based Builders: Democratizing Development

Platforms like Voiceflow, Botpress, and MindStudio represent a philosophy of visual programming for AI agents. They’re not just no-code—they’re thoughtfully designed for non-programmers to build sophisticated conversational agents, offering visual canvases, drag-and-drop logic, and built-in integrations. While you can hit walls when you need custom logic, many use cases don’t need it.

A powerful middle ground between these conversational builders and pure code exists with node-based automation platforms like n8n. These tools also use a visual, graph-based canvas, but are designed for complex data workflows, integrations, and backend logic, allowing developers to visually stitch together APIs, databases, and AI models in a way that is more robust than no-code and more accessible than a pure framework.

The Trade-offs: What You Give Up for Convenience

While frameworks accelerate development, they aren’t a free lunch, and it’s important to consider what you’re trading away. You are, in effect, trading the complexity of building from scratch for the complexity of learning a framework’s specific abstractions, such as internalizing LangGraph’s graph-based mental model or ADK’s enterprise patterns. There’s also the matter of framework overhead; these layers of abstraction can introduce performance hits or higher token usage. You might see simple tasks consume two to three times more tokens through a framework than in a hand-rolled implementation, depending on the approach. You also risk the “leaky abstraction” problem, where you eventually hit a wall that the framework’s design simply doesn’t fit. For instance, a developer might spend a week fighting a framework’s complex delegation logic when a simple round-robin task assignment is all that’s needed. Finally, in this breakneck-speed field, you’re betting on maturity and stability. APIs change, frameworks pivot, and what works today might be deprecated tomorrow—a phenomenon seen in LangChain’s multiple major architectural shifts over just two years.

Choosing Your Framework: A Practical Guide

My decision-making process for choosing a framework has evolved into a practical decision tree.

Start with Architecture: Single Agent or Multi-Agent?

For single, complex agents, your choice depends on your needs. LangGraph is ideal if you need explicit control over reasoning patterns, while the OpenAI Agents SDK is perfect if you want minimal abstractions. If your application is built around RAG, Phidata is a strong contender. For multi-agent systems, the options are philosophically different. CrewAI excels at intuitive, role-based teams, while the Microsoft Agent Framework is built for conversation-driven collaboration. For hierarchical, production-grade systems, Google ADK is the most robust choice.

Consider Your Constraints

Your production timeline is a major factor. If you’re prototyping this week, the rapid setup of CrewAI or LangChain is invaluable. If you’re building for a production launch next quarter, the enterprise-grade architecture of Google ADK or the Microsoft Agent Framework is a safer bet. For timelines somewhere in between, LangGraph or Phidata offer a balance of power and speed.

Team expertise also matters. Deep Python experience is a good fit for any code-first framework, but CrewAI’s natural language approach can be easier for mixed technical teams. For non-technical stakeholders, no-code platforms are the most accessible.

Finally, your choice depends heavily on your ecosystem and specific goals. If you’re building for production scale within Google Cloud, the Google ADK is the clear choice for its built-in Vertex AI integration and enterprise observability. Similarly, if your organization is built on Azure, the Microsoft Agent Framework provides native service integration and AutoGen’s powerful multi-agent patterns. For those needing rapid prototyping, LangChain and LangGraph offer a massive component library. If your goal is intuitive multi-agent collaboration, CrewAI’s role-based design is

remarkably effective, while the OpenAI Agents SDK is perfect for those who want minimal, clean abstractions. For visual, no-code workflow design, platforms like Voiceflow and Botpress democratize deployment; for node-based visual automation, platforms like n8n bridge the gap; and for a Python-native, full-stack experience, Phidata’s object-oriented approach is excellent.

My Personal Stack

For personal projects where I want complete control—like my Gemini Scribe agent for Obsidian, I still build from scratch. The entire agent is ~500 lines of typescript, perfectly tailored to my workflow. You can see exactly what’s happening at every step, and there’s no framework magic to debug when things go wrong.

But for more complex systems, Google ADK has become my go-to. My recent adh-cli TUI for Gemini is built with ADK, allowing me to spend more time thinking about the unique concepts I want to explore and less time on boilerplate of agent development.

The choice ultimately depends on your specific context and is a strategic decision that reflects and reinforces your approach to AI development.

Beyond The Scaffolding

Here’s what I’ve come to believe: the choice between frameworks isn’t about features—it’s about philosophy. Each framework embodies assumptions about how agents should think, collaborate, and evolve.

Adopting any framework provides immediate benefits: velocity to skip the boilerplate and focus on your unique logic; reliability from leveraging battle-tested patterns and error handling; community to tap into collective knowledge and shared solutions; and governance by enforcing architectural best practices automatically. The open-source nature of many of these frameworks means that even if you encounter a novel edge case, it’s likely you’re not alone, and a solution may already be in progress within the community.

But the real value is subtler. Frameworks don’t just accelerate development—they shape how you think about agents. LangGraph teaches you to model cognition as state machines. CrewAI makes you consider role-based decomposition. ADK asks you to think about production from day one.

The frameworks are the scaffolding necessary to build the next generation of intelligent applications. They’re transforming agent development from an artisanal craft into a systematic engineering discipline.

When One Agent Isn’t Enough

But what happens when a single agent, even one built on a sophisticated framework, hits its limits? Complex problems often require multiple perspectives, specialized expertise, and collaborative problem-solving.

In Part 9: Building an Agentic Team, we’ll explore the fascinating world of multi-agent systems. We’ll dive into orchestration patterns, examine how agents negotiate and delegate, and uncover the emergent behaviors that arise when AI agents work in teams.

If you thought managing one agent’s context window was challenging, wait until you see five agents trying to agree on a shared goal while maintaining their own specialized knowledge and constraints.

The future isn’t just agentic, it’s collaborative. And it’s already here.