How AI Agents Actually Work: Memory, Tools, Planning & Real-World Systems

There's a useful analogy that separates a chatbot from an AI agent. A chatbot is a very smart answering machine — it takes your question and returns a response. An AI agent is more like a very competent employee: you hand it a goal, and it figures out a plan, uses whatever tools it has access to, checks its work, and iterates until the job is done.

That distinction sounds simple. The engineering required to make it real is not. In 2025, AI agents have gone from research curiosities to production systems running inside some of the world's largest companies. 57% of enterprises now have AI agents running in production, according to G2's Enterprise AI Agents Report. Andrej Karpathy — founding member of OpenAI — declared that this will be the decade of AI agents.

So how do they actually work? Let's go inside the architecture.

"An AI agent is a system that can perceive its environment, plan actions, use external tools, and — crucially — a feedback loop that allows it to act, observe results, and correct itself."
— Wang et al., "A Survey on Large Language Model-based Autonomous Agents" (2023)

What Makes Something an AI Agent?

The academic literature identifies three components that transform a raw LLM into an agent: a planning module, a memory module, and a tool-use module. Without all three, you have a chatbot. With them, you have something that starts to look like autonomy.

A standard LLM call is a single round trip: you send input, you get output. An agent wraps that in a loop. The model reasons about what to do next, takes an action (usually calling a tool or an API), receives the result, updates its understanding, and goes again. This continues until the task is complete — or the agent decides it can't proceed without more information.

The key difference from a simple script or automation is that the agent's next step is not predetermined. It's generated by the model at runtime, based on what it has learned so far. This is what gives agents genuine flexibility in the face of unexpected situations — and what makes them so much harder to debug when things go wrong.

The ReAct Loop: Think → Act → Observe

The dominant architecture for modern AI agents is called ReAct (Reason + Act), introduced in a landmark ICLR 2023 paper by Yao et al. at Princeton and Google. It's elegant in its simplicity: the agent alternates between thinking and doing, grounding its reasoning in real-world observations at every step.

Here's what a single ReAct loop looks like in practice. Say an agent is given the task: "Find the latest revenue figure for NVIDIA and calculate what percentage of it is R&D spend."

Thought: I need NVIDIA's latest annual revenue and R&D expenditure. I'll search for the most recent earnings report.
Action: web_search("NVIDIA FY2025 annual revenue R&D expenditure")
Observation: [search returns: Revenue $130.5B, R&D $8.67B]
Thought: I have both numbers. R&D as % of revenue = 8.67/130.5 × 100 = 6.6%. I can now answer.
Final answer: NVIDIA's R&D spend was approximately 6.6% of its $130.5B revenue in FY2025.

Each Observation feeds directly into the next Thought, grounding the agent in real information rather than letting it hallucinate answers it doesn't know. This is ReAct's killer feature: it ties reasoning to evidence at every step.

The Four Layers of Agent Memory

Memory is what separates a stateless chatbot from an agent that can tackle multi-hour, multi-session tasks. Modern agent frameworks implement memory across four distinct layers, each with different scope and retrieval speed.

The most exciting development in 2025 is persistent agent memory — systems where an agent remembers decisions, preferences, and outcomes across completely separate sessions. Tools like Mem0 and LangGraph's checkpointing enable agents to build a genuine "experience" over time, becoming more effective with each interaction rather than resetting to zero on every conversation.

Tool Use: How Agents Interact With the World

An LLM alone can reason and generate text, but it can't take actions. Tools are what give agents hands. When a model wants to use a tool, it outputs a structured function call — essentially saying "I want to call search_web with query='NVIDIA revenue 2025'". The framework intercepts this, executes the function, and returns the result as the next observation.

The key to good tool use is the tool description. The model selects which tool to call based entirely on reading the tool's natural-language description. A well-written description — "Use this tool to get the current stock price for a given ticker symbol; do not use for historical prices" — is the difference between an agent that works reliably and one that calls the wrong function half the time.

Planning Strategies: Beyond ReAct

ReAct is the dominant pattern, but it's not the only one. As tasks grow more complex, different planning architectures make different tradeoffs between flexibility and efficiency.

Plan-and-Execute (ReWOO)

Instead of reasoning one step at a time, this approach generates a complete plan upfront, executes all tool calls in parallel, then synthesises the results into a final answer. The advantage: dramatically fewer LLM calls and the ability to parallelise work. The disadvantage: if an early step fails or returns unexpected data, the plan can't adapt mid-execution the way ReAct can.

Tree of Thoughts (ToT)

Rather than following one chain of reasoning, ToT generates multiple branches of thought simultaneously — like a chess engine exploring several lines — then evaluates and prunes them. Useful for tasks with many possible solution paths, but computationally expensive. Typically reserved for high-stakes decisions where accuracy outweighs speed.

Reflexion

Adds an explicit self-critique layer after each failed attempt. The agent writes a natural-language reflection ("I failed because my SQL query didn't account for NULL values") which is stored in memory and fed back on the next attempt. This mimics reinforcement learning — the agent gets better at a specific task over multiple tries without any weight updates.

Multi-Agent Systems: When One Agent Isn't Enough

A single agent can be overwhelmed by too many tools, too many domains, or tasks so long they exceed any context window. The solution is multi-agent systems — networks of specialised agents that delegate, collaborate, and check each other's work.

The most powerful recent development in multi-agent systems is the Agent-to-Agent (A2A) Protocol — a communication standard that lets agents built on completely different frameworks discover each other and collaborate. An Anthropic-built research agent can now delegate a coding task to an OpenAI-built code agent and receive the result, all through a standardised interface.

Real AI Agents in Production: 2025's Frontier

Claude Code — released and widely adopted by early 2026 — is arguably the most technically sophisticated coding agent in production. It operates in the user's own terminal, can read and write files across an entire codebase, run tests, spawn sub-agents for parallel subtasks, and work autonomously for hours on large engineering problems. It achieves 72.7% on OSWorld and 65.4% on Terminal-Bench 2.0, benchmarks that measure an agent's ability to operate a real computer.

OpenAI's Operator, powered by GPT-5, focuses on browser-based task completion — the kinds of tasks that previously required a human to click through a web interface. Book a flight, fill a government form, update a spreadsheet in a third-party app. By combining vision (seeing the screen) with action (clicking and typing), Operator represents a new class of agents that don't need API access to interact with software — they use the UI just like a human would.

Why Agents Fail — And How to Stop Them

For all their power, AI agents fail in predictable ways. Understanding these failure modes is essential for anyone building or deploying them in production.

Tool hallucination: The agent calls a tool with plausible-sounding but incorrect parameters — for example, passing a date in the wrong format or a ticker symbol that doesn't exist. Mitigation: strict JSON schema validation on all tool inputs, with error messages the agent can read and correct.

Infinite loops: An agent can get stuck repeating the same action when it doesn't understand why it's failing. Mitigation: hard step limits, and frameworks like LangGraph that enforce explicit state machines rather than open-ended loops.

Context window overflow: Long-running tasks can accumulate more observations than fit in the context window, causing the agent to "forget" earlier steps. Mitigation: summarisation between steps, persistent external memory, and hierarchical task decomposition.

Compounding errors: A small mistake in step 2 becomes a larger one by step 8. Mitigation: reflection checkpoints, human-in-the-loop approval gates for high-stakes actions, and evaluator agents that review outputs before they're committed.

What's Next: Towards Truly Autonomous Systems

The trajectory of AI agents in 2026 and beyond is clear: longer horizons, better memory, and tighter collaboration between specialised agents. Three developments stand out as the most consequential.

Computer Use at Scale: Models like Claude Opus 4.6 and GPT-5 can now see and control a computer screen directly. As this capability matures, agents will be able to operate any software — not just those with APIs — opening up entire industries that have been automation-resistant due to legacy systems.

Persistent, Learning Agents: The next frontier is agents that genuinely improve over time — not through retraining, but through accumulated experience stored in long-term memory. An agent that has completed 10,000 coding tasks should be measurably better at its 10,001st than it was on its first. Architectures like Reflexion and MemGPT are early steps toward this; 2026 will see it become standard.

Agent Networks as Infrastructure: Just as microservices replaced monolithic apps, collections of specialised agents are replacing monolithic AI systems. With A2A and MCP as the communication standards, we're moving toward an internet of agents — where a user's request is routed, decomposed, and fulfilled by a coordinated network of AI systems, most of them running entirely without human involvement in the loop.

Conclusion

AI agents are not magic. They are a specific architecture: an LLM in a loop, equipped with memory layers, a curated set of tools, and a planning strategy that determines how it sequences its reasoning. The ReAct pattern made agents practical. Multi-agent systems made them scalable. MCP and A2A are making them interoperable.

What's remarkable about 2025 is not that AI agents exist — it's that they are genuinely useful at tasks that previously required human expertise and hours of focused work. Claude Code rewriting a codebase overnight. Operator booking a complex multi-leg flight. Devin fixing a production bug while the engineer sleeps. These are not demos. They are products, running in production, at scale.

Practical Advice for Building Agents Today

If you're building an AI agent in 2026, the most important decision is not which model to use — it's how to define your tools. A well-described tool with clear boundaries ("use this to search the web for current information; do not use for mathematical calculations") will outperform a poorly-described one regardless of the underlying model. Invest as much time in tool design as in prompt design.

Start with the simplest architecture that solves your problem. A single ReAct agent with three to five well-chosen tools handles the vast majority of practical use cases without the complexity overhead of multi-agent orchestration. Add pipeline complexity — sub-agents, orchestrators, parallel execution — only when you have a specific bottleneck that justifies it. The most common mistake is building a distributed multi-agent system when a single agent with good tools would have been sufficient.

Observability is non-negotiable. Unlike a standard API call where you can inspect input and output, an agent's failure can be buried six steps deep in a chain of actions that individually looked reasonable. Logging every Thought, Action, and Observation — with timestamps and token counts — is the minimum viable safety net. Tools like LangSmith and Weights & Biases Weave make this straightforward to implement.

Finally: never give an agent an irreversible action without a human approval gate. Sending emails, committing to databases, making purchases — these actions cannot be undone if the agent misunderstands the task. The correct architecture for high-stakes agentic systems in 2026 is not full autonomy; it is full autonomy for reversible actions and human-in-the-loop for irreversible ones. Trust must be earned incrementally, task type by task type.

The next few years will determine how much of knowledge work can be handed to agents operating autonomously. If the current rate of capability improvement holds, the answer may be: most of it. But the teams that get there first won't be the ones who gave agents the most freedom — they'll be the ones who built the most reliable, observable, and correctly-scoped systems first.