Fourteen operational checks that decide whether your agent reaches production or stalls in the demo graveyard.
The pattern repeats. A team wires up an agent, watches it complete a real task on the first afternoon, and books the launch. Six months later nothing has shipped and nobody agrees on why.
The reasons rarely involve the model. They involve the harness around it, the part that decides what the agent may touch, what it must prove before acting, and what happens when it gets something wrong at three in the morning. Those questions look like paperwork during the demo and become the whole project afterwards. This checklist is the operational discipline I bring to every agent build. It is short on purpose. Each item is the kind of check that, when skipped, costs months.
Write down, before any code, which decisions the agent may make without a human and which it may only propose. Refunds under fifty dollars, yes. Refunds over five hundred, propose only. Schema migrations, never. This single document does more to unblock a deployment than any model upgrade, because it converts an argument about trust into a list a compliance lead can sign. Teams that skip it discover the boundary during the first incident review, and by then the boundary is set by whoever is angriest in the room.
An agent is exactly as dangerous as the tools you hand it. Enumerate every tool, and for each one record what it can read, what it can write, and whether the write is reversible. Read-only tools are cheap to grant. Write tools need a reason. Tools that touch money, customer records, or production infrastructure need a reason and an approval gate. The common failure is granting a broad API credential because scoping it looked like an afternoon of work, then discovering the agent used it in a way nobody modelled.
Assume the agent will do the wrong thing with the right credentials. Run it where that is survivable. Separate execution environment, its own service account, network egress restricted to the hosts it actually needs, and a filesystem it cannot escape. Shell access and code execution deserve particular care, because an agent that can run arbitrary code inherits every permission the process holds. The question to answer in writing is simple. If this agent behaves as badly as it possibly could within the permissions granted, what is the worst outcome, and can the business absorb it?
Before model selection, document the regulatory perimeter. Which authority decides what data can move where, whether that is PDPL, DIFC Data Protection Law, ADGM Data Protection Regulations, GDPR, HIPAA, or sector-specific banking rules. For each data class the agent will touch, write down where it can be processed, where it can be stored, and where traces and embeddings can land. The output is a one-page residency document signed off by the compliance lead before the build starts. Skip it and you will discover the limits during procurement review, weeks before launch, when the architecture can no longer bend.
Route by task rather than standardising on one model everywhere. A cheap fast model handles classification and extraction. A frontier model handles the planning step where a wrong turn costs an hour of downstream work. Open weights running inside your own boundary handle whatever the residency document says cannot leave. The criteria are inference cost at your real volume, latency at your p95, and measured quality on your tasks rather than on a public leaderboard. Hybrid is usually the right answer and rarely the first one proposed.
Most useful agents read something before they act, and the retrieval layer is where correctness is won or lost. Decide the citation surface before you choose the embedding model. Every output that makes a claim should carry a link to the source it came from, enforced programmatically rather than requested in a prompt. Reject any citation that does not appear in the retrieval results. This is partly hallucination control and partly the thing that lets an auditor trust the system, because a claim with a source is reviewable and a claim without one is a guess wearing a confident tone.
Build a golden dataset of 100 to 200 real production-shaped tasks with outcomes labelled by a domain expert. Grade deterministically wherever you can, which for agents means tool call correctness, argument validity, output schema, and whether the run terminated. Use LLM-as-judge for the genuinely subjective parts, calibrated against the human grader. Run the suite before every prompt change, every model change, and every tool definition change. Without an eval suite, every change is a hope. With one, every change is a measurement.
Decide what the agent does when it is not confident, when a tool fails twice, or when it hits a decision above its authority from item one. The answer should be a named queue with an owner and a response time, not a log line nobody reads. Build the hand-off so the human receives the full context of the run rather than a bare alert, because an escalation that requires ten minutes of archaeology gets ignored by the second week. An agent with no escalation path does not fail safely. It fails quietly.
Log the full trace. Every prompt, every tool call with its arguments and its result, every retry, every model response, and the token count at each step. You need to replay a run months later and understand exactly why the agent did what it did. Standard tracing formats and existing observability tooling handle this well, and the discipline matters more than the vendor. The teams who add tracing after the first production surprise always say they wish they had added it on day one, and they are always right.
Set explicit budgets per run, per user, per tenant. The first month after launch is the most expensive month, because edge cases trigger long-context reasoning, debugging burns tokens, and a loop in the agent graph can run an order of magnitude over baseline before anyone notices. Hard limits that fail fast beat silent degradation. Cap the number of steps a single run may take, not just the spend, since a runaway agent usually announces itself as a step count long before the invoice arrives.
Connect the trace stream to the pipeline your security team already operates, whether that is Splunk, ELK, or Sentinel. Match retention to the regulator's requirement, which is often longer than the engineering team expects. The most common failure is logging into a separate vendor portal the security team cannot reach. An agent without audit is a compliance failure, and audit running in a parallel system nobody monitors is an integration failure wearing a dashboard.
Write the failure modes down, in a table, with the eval or guardrail that catches each one. Hallucination, caught by enforcing citation from retrieval. Prompt injection through retrieved content or tool output, caught by treating everything the agent reads as untrusted input and never letting it reach the system prompt. Tool misuse, caught by the access boundary from item two and by schema validation on every argument. Runaway loops, caught by the step cap from item ten. Authorisation leakage, caught by enforcing identity at the retrieval layer rather than asking the agent to respect it. The list is short and well understood. The mitigations are all pattern matches. Skipping any of them is what creates the incident.
Decide who may change a prompt, who may upgrade a model, who may add a tool, and how each change gets reviewed. The pattern that works keeps prompts, model versions, and tool schemas in Git alongside the rest of the code, sends changes through pull request review, runs the eval suite in CI, and makes roll-forward and rollback one command. The pattern that fails edits prompts live in a vendor console with no version history. The first time a regression ships, that team cannot say what changed.
For every external dependency, meaning the model provider, the vector store, the observability platform, and the agent framework itself, write down what happens if it disappears tomorrow. What data do you keep, in what format, and how long would a swap take? The answer should fit on a page. The exercise usually surfaces a dependency nobody realised was load-bearing, and occasionally a contract tighter than the team remembered. The point is not vendor pessimism. The point is engineering hygiene.
The deeper engineering work behind this list lives in Agent Harness & Deployment.