AI Agent Architecture: Tools, State, Permissions, Retries & Human Review featured visual

An AI agent becomes an engineering system when it can do more than generate text. The moment it can query private data, invoke APIs, update records, send messages, create files, change infrastructure, or hand work to another agent, the architecture has to answer a different set of questions: what can it do, on whose authority, with what state, under which limits, and how does the system recover when something goes wrong?

This guide focuses on those production boundaries. It is not tied to one framework or model provider. The goal is to show how tools, state, permissions, retries, human review, observability, and orchestration fit together in a reliable agent system.

If you are still deciding whether an agent is the right pattern at all, start with our buyer guide for production AI systems and the broader AI systems engineering capability.

1. First decide whether the workload needs an agent

Not every multi-step application needs agentic orchestration. If a workflow is deterministic, well-structured, and can be expressed as a fixed sequence of validated steps, conventional software is usually easier to test, cheaper to operate, and simpler to secure.

Google Cloud's current architecture guidance recommends starting with a single agent when agentic behavior is genuinely needed, and only adding multi-agent complexity when distinct responsibilities, tools, or orchestration requirements justify it. That is a useful default because every additional autonomous component introduces more state, more failure modes, more latency, and more coordination overhead.

A practical rule is: use an agent when the system must interpret context, choose among actions, adapt its plan as new information arrives, or operate across tools where the next step cannot be fully predetermined in code.

2. Separate the model from the control plane

The model should not be the entire application. In a production architecture, the model proposes or selects actions, while the surrounding application enforces what is allowed to happen.

A useful control-plane decomposition includes:

Agent runtime: the loop that sends context to the model and processes its output;
Tool registry: the set of capabilities the agent may invoke;
Policy layer: rules for permissions, approvals, limits, and prohibited actions;
State layer: conversation state, workflow state, durable business state, and memory;
Execution layer: the services that actually perform tool calls and side effects;
Observability: traces, tool-call records, errors, latency, cost, and policy decisions;
Evaluation: automated and human checks that determine whether changes are safe to release.

This separation matters because prompts are not authorization systems. A system prompt can tell an agent not to delete a customer record, but the application should still prevent the deletion unless the agent has the required permission and, where appropriate, explicit human approval.

3. Treat tools as contracts, not plugins

Every tool exposed to an agent should have a narrow contract. The agent needs to know what the tool does, what arguments it accepts, what it returns, and what failure states mean. The application needs to validate those arguments before execution.

Good tool design usually means:

small, explicit schemas rather than a single generic "execute anything" endpoint;
validated enums, identifiers, dates, amounts, and resource references;
clear read-versus-write semantics;
bounded result sizes;
structured error responses;
timeouts and cancellation behavior;
stable identifiers for tracing and retries.

OpenAI's current agent guidance treats tools as a first-class part of the agent definition and emphasizes that application code still owns client-side function execution. That is an important architectural boundary: the model can request a tool call, but your system decides whether and how it executes.

For OSYSTIC's implementation approach, see AI agent and workflow automation.

4. Scope permissions to the user, agent, and action

Agents should not inherit broad infrastructure credentials simply because they need access to one workflow. Permissions should be scoped to the smallest set of actions and resources required for the task.

Think about authorization at three levels:

User authority: what is the requesting user allowed to do?
Agent authority: what subset of those actions is this agent allowed to perform?
Tool authority: what exact operation can this tool execute against which resource?

This avoids a common failure mode where a user with read-only access asks an agent a question, but the agent's service account has write access to the underlying system.

Google's recent agentic architecture guidance explicitly recommends workload-bound agent identity and least-privilege access for business-critical systems. The broader principle is platform-independent: an agent should have an identity that can be audited, constrained, and revoked independently.

5. Keep conversational state separate from durable business state

Agent systems usually need several kinds of state, and mixing them together creates reliability problems.

State type	Purpose	Reliability expectation
Conversation context	Recent messages and tool results needed for the current interaction.	Useful but not the source of truth.
Workflow state	Which step the agent is on, what is pending, what has completed, and what requires approval.	Durable enough to resume safely.
Business state	Orders, tickets, invoices, customer records, permissions, approvals.	Authoritative system of record.
Long-term memory	Preferences or distilled context that may help future interactions.	Explicitly governed and revisable.
Audit state	Who requested what, which tool ran, what changed, and why.	Durable and tamper-resistant enough for investigation.

Do not let conversational memory become the authoritative record of whether an action occurred. If an agent says "the refund was processed," the application should verify the transaction system rather than trusting remembered context.

6. Put human approval at side-effect boundaries

Human review should not be a cosmetic confirmation screen added at the end. It should sit immediately before actions where the cost of a wrong decision is materially higher than the cost of asking for approval.

Common approval boundaries include:

sending external communications;
issuing refunds or payments;
changing permissions;
deleting or publishing records;
modifying production systems;
making commitments on behalf of a company;
executing actions with legal, financial, or safety consequences.

Microsoft's Agent Framework describes human-in-the-loop tool approval as a workflow pause: the system emits an approval request and resumes only after the human response. OpenAI's current safety guidance similarly recommends keeping tool approvals enabled for sensitive operations. The architecture lesson is that approval must be part of the runtime state machine, not just an instruction in the prompt.

7. Design retries around side effects and idempotency

Retries are one of the most underestimated risks in agent systems. A timeout does not prove that an action failed. A tool may have completed successfully while the network response was lost.

Before automatically retrying a write action, the system should answer:

Did the previous attempt create a durable result?
Can the operation be safely repeated?
Is there an idempotency key or transaction identifier?
Can the system query the destination to confirm the current state?
Is the error transient or permanent?

OpenAI's agent recovery guidance explicitly warns that a failed turn may already have changed files or called external tools, and recommends checking completed effects before repeating work. Its session guidance also recommends reusing the same idempotency key for retries of the same logical message. Google Cloud's retry guidance likewise distinguishes transient failures from permanent errors and treats idempotency as a core factor in deciding whether an operation is safe to retry.

For side-effecting tools, a strong pattern is to persist a request identifier, execution status, destination identifier, and final result before telling the agent that the action is complete.

8. Bound the agent loop

An agent should not be able to continue indefinitely. Every production workflow needs stopping conditions.

Useful limits include:

maximum tool calls per run;
maximum retries per action;
maximum loop or refinement iterations;
wall-clock deadlines;
token or cost budgets;
maximum subagents or parallel tasks;
escalation after repeated uncertainty or failure.

Google's agent design guidance describes iterative loops that stop when the agent reaches a conclusive result, a configured maximum number of iterations, or an error that prevents continuation. Those boundaries are operational controls, not only cost controls: they prevent ambiguous tasks from turning into uncontrolled execution.

9. Use multi-agent systems only when responsibilities are genuinely distinct

Multi-agent architecture can help when one agent is overloaded with unrelated responsibilities, tools, or policies. It can also make a simple system much harder to operate.

Useful reasons to split agents include:

different specialists need different tools;
different tasks require different instructions or safety policies;
one coordinator should retain responsibility while specialists act as bounded capabilities;
a specialist should take ownership of a branch of work through an explicit handoff.

OpenAI's orchestration guidance distinguishes two useful patterns: a manager calling specialists as tools while retaining control, and handoffs where a specialist takes over the next part of the workflow. The important design question is ownership: who is responsible for the next user-facing decision and which context follows that transition?

Do not add agents merely because a workflow contains several steps. A deterministic service, queue, or ordinary function is often the better component.

10. Treat untrusted content as data, not instructions

Agentic systems often read webpages, emails, documents, tickets, code, and files. Any of those inputs can contain text that attempts to influence the agent's behavior.

OWASP's Agentic Security Initiative highlights the need for transparency and control across autonomous systems. OpenAI's current agent safety guidance recommends structuring workflows so untrusted data does not directly drive tool behavior and using structured extraction, guardrails, and approvals around sensitive actions.

Practical controls include:

separate trusted system instructions from retrieved content;
extract structured fields from untrusted documents before using them in actions;
validate tool arguments against application rules;
restrict which tools are available in high-risk contexts;
require approval before external side effects;
never expose raw secrets to the model when a scoped service can perform the action instead.

11. Trace the full decision path

When an agent gives a wrong answer, the root cause may be the model, a retrieval miss, the wrong tool choice, invalid tool arguments, stale state, a permission check, a downstream API, or a retry. Without traces, those failures are difficult to separate.

A production trace should make it possible to reconstruct:

which agent or subagent was active;
which model and instructions were used;
which tools were available;
which tools were called and with what validated arguments;
what each tool returned;
which approval or policy checks ran;
how long each stage took;
which failure or stopping condition ended the run.

OpenAI's current tracing documentation records model generations, agent spans, tool calls, tool results, errors, duration, and outcome status. Whatever platform you use, your observability layer should give you equivalent operational visibility.

See AI reliability and LLMOps for the evaluation and observability layer around production AI.

12. Evaluate workflows, not just final answers

For an agent, a correct final sentence is not enough if it took an unsafe path to get there.

Your evaluation set should include assertions about the workflow itself:

Was the correct tool selected?
Were unnecessary tools avoided?
Were permission checks respected?
Did the workflow pause for approval at the right boundary?
Did retries preserve idempotency?
Did the agent stop within its configured budget?
Did the result remain grounded in authoritative business data?
Was the final action actually completed in the destination system?

OpenAI's current programmatic tool-calling guidance recommends measuring not only final correctness but also tool calls, retries, recovery behavior, safety outcomes, approval requirements, latency, and cost. That is the right evaluation frame for any serious agent system.

13. A practical production reference architecture

A robust agent workflow can be summarized as the following sequence:

Authenticate the user. Establish the real user and organization context.
Load bounded context. Retrieve only the information the task needs and the user may access.
Run the agent. Give it narrow instructions, scoped tools, and explicit limits.
Validate proposed actions. Check schemas, policy, permissions, and resource ownership.
Pause for approval when required. Persist the pending action so the workflow can resume safely.
Execute through a service boundary. Run the tool with scoped credentials and an idempotency strategy.
Persist the result. Record durable outcome identifiers and workflow state.
Return the observation. Let the agent continue from verified tool results.
Trace and evaluate. Capture the path for debugging, monitoring, regression testing, and audit.

The model sits inside this architecture. It does not replace it.

14. Architecture red flags

Red flag	What it usually means
One agent has unrestricted access to every tool	Least privilege and responsibility boundaries are missing.
Prompt instructions are the only safety control	Application-level authorization and policy enforcement are weak.
Every tool failure is retried automatically	Duplicate side effects are likely.
Conversation memory is treated as the system of record	State can drift from real business systems.
No explicit loop or cost limits	Runaway execution and unpredictable latency are possible.
Human approval happens after the action	The approval is informational, not protective.
No trace of tool arguments and outcomes	Failures cannot be reliably investigated.
Multi-agent by default	Complexity is being added before a simpler design is proven insufficient.

15. Questions to answer before production

Which tools can this agent use, and why?
Which operations are read-only and which create side effects?
Whose permissions are enforced at execution time?
What state must survive a restart?
Which actions require approval?
How is duplicate execution prevented?
What is the retry policy for each tool?
What stops the agent loop?
What happens when a downstream system is partially available?
How do operators reconstruct a failed run?
Which workflow behaviors are covered by automated evaluations?
How can an administrator disable a tool, agent, or integration quickly?

If these questions do not have concrete answers, the system is probably not ready for autonomous production work.

Sources and further reading

For implementation help, review OSYSTIC's AI agent development service, AI reliability and LLMOps, or send a technical brief.

AI Agent Architecture: Tools, State, Permissions, Retries & Human Review