Devspedia

Tool Schemas Are Prompts: Designing Tools AI Agents Can't Misuse

2026-09-05T09:30:00+00:00

Here are two tool definitions an agent can be given. Same backend, same underlying capability.

{
  "name": "search_orders",
  "description": "Searches orders.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": { "type": "string" },
      "all": { "type": "boolean" },
      "full": { "type": "boolean" }
    }
  }
}

Ship this to an agent and watch the traffic. You will see all: true paired with an empty query. You will see full: true on every call, because the model wants to be thorough. You will see complete order histories pulled into context until the window is gone. Nothing in this schema is broken — every one of those calls is allowed, and the model is doing exactly what the documentation invites.

Now the same capability, designed differently:

{
  "name": "list_orders",
  "description": "Lists orders, newest first. Use for browsing and lookup; use get_order for a single known id. Returns at most `limit` order summaries — never line items. Pass `status` instead of guessing date ranges.",
  "parameters": {
    "type": "object",
    "properties": {
      "status": {
        "type": "string",
        "enum": ["open", "shipped", "delivered", "refunded", "cancelled"],
        "description": "Filter by fulfilment state. Omit for all statuses."
      },
      "customer_email": {
        "type": "string",
        "description": "Exact email address, lowercased. Not a substring search."
      },
      "limit": {
        "type": "integer",
        "minimum": 1,
        "maximum": 50,
        "default": 20,
        "description": "Page size. Raise only if the first page was not enough."
      }
    }
  }
}

Agents call this version correctly far more often — and when they drift, the drift is small and correctable. The difference is not model quality. The second schema is written as instructions, and the model reads it as instructions.

The schema is the only documentation the model reads

When a model chooses a tool, it sees four things: the tool name, the description, the parameter names and types, and whatever the conversation contains. It never sees your internal wiki, your ADRs, or the intent behind the code. Whatever you put in the schema is the entire documentation set, and the model consumes it the way a new hire consumes a runbook — literally, and without the shared context that lets humans read past imprecision.

That makes every vagueness a prompt. A description like "Searches orders" does not say what matches, what order results come back in, how big a result can be, or when the tool is the wrong choice. The model resolves all of it from priors, and priors are how you get all: true. The fix is not a smarter model. The fix is writing the description as if it were an instruction to a capable person on their first shift: what the tool does, when not to use it, what the units and timezones are, and what comes back.

Name the capability, not the implementation

Tool names carry semantics into every prompt. Verb-first names — list_, get_, create_, cancel_ — tell the model what kind of action it is considering before it reads anything else. orders_handler_2 tells it nothing, so it guesses, and the guess shows up as a wrong call at the worst time.

One tool should be one capability. A manage_orders tool that creates, cancels, and refunds based on an action enum makes every invocation a composition problem: pick the verb, then pick the parameters that verb actually accepts. Splitting it into create_order, cancel_order, and refund_order turns one judgment call into a selection, and gives each tool a description that can be precise instead of a taxonomy of caveats.

The same logic applies at the read/write boundary. Keeping read-only and mutating tools on separate names — list_orders versus cancel_order — is what lets a policy layer say "this agent may call anything starting with list_ and get_, nothing else" without parsing arguments. That separation is the seam sandboxing and approval gates attach to; a naming convention is load-bearing infrastructure, not style.

Constrain with types, not prose

Every enum is an instruction the runtime enforces. status with five allowed values eliminates the whole family of malformed filters a free-form query string invites, and a model that wants "closed a while ago" must map its intent onto delivered or refunded — a much smaller step than inventing date-range syntax.

Booleans are where misuse breeds. all, full, force, include_everything — models set them true to be safe, and true is the one direction you cannot take back. Replace them with typed alternatives: an enum status instead of include_closed, a fields selection instead of full, explicit ranges with minimum and maximum instead of force.

The rest of JSON Schema is cheap insurance. pattern, minLength, maximum fail a bad call at the client boundary with a message the model can correct from, instead of failing mid-query inside your database where the only recovery is a timeout.

Split preview from apply

One tool should not both show what would happen and make it happen. preview_refund computes and returns; apply_refund commits, and takes the token the preview returned. This mirrors the two-step patterns humans already trust, and it produces two concrete benefits:

The destructive path becomes structurally narrow. apply_refund cannot be called sensibly without a preview token, so the accidental invocation has nothing to stand on.
Policy gets a natural seam. Autonomy can allow previews freely while applies route through an approval gate — the gate sees a concrete, already-computed intent rather than a hypothetical.

For destructive tools, make the dangerous input structurally required. delete_expenses takes ids — an explicit array, capped — and has no "all" mode at all. A confirm: true boolean is weaker than it feels, because models pass true; a required explicit id list cannot be satisfied by enthusiasm.

Return errors the agent can act on

An error message is a prompt too. The model will read it and decide what to do next, so an error that says only Error: 500 invites a retry loop, and a raw stack trace invites the model to paste fragments of itself into the next attempt.

Design the error for the reader:

{
  "error": {
    "code": "order_not_refundable",
    "message": "Order 8412 was delivered 92 days ago, past the 60-day refund window.",
    "recovery": "Offer store credit via grant_store_credit, or escalate to the support queue."
  }
}

The stable code lets harnesses assert on behavior — the agent saw order_not_refundable and did not retry the same call. The recovery field tells the model what to do instead, which is the difference between a dead end and a decision.

Make idempotency a parameter, not documentation

If a tool creates or mutates something that matters — payments, messages, state transitions — take an idempotency_key parameter and honor it server side. Prose like "safe to retry" is addressed to the exact agent that just watched a timeout fire and must now decide whether calling again doubles the charge. A required key removes the decision entirely: the retry replays the same key and the backend folds it into the original operation. Command ledgers and workflow engines can then reason about side effects from the key alone.

Budget the output

Context is the scarce resource in an agent run, and a tool that dumps 40 MB of JSON in response to "recent orders" has harmed the run even though every byte was accurate. Output design is interface design:

Default page sizes, with limit capped.
Summaries by default; line items only through a dedicated get_order.
A truncation contract the model can work with — a next_cursor, not a silent cutoff.
Field selection where reads are wide.

The general pattern is the same one that keeps runaway agents affordable: every tool should have a defensible answer to "what is the most context this call can consume?"

Version like an API, because it is one

Additive changes only. Adding an optional parameter with a documented default is safe; repurposing what an existing parameter means is not, because prompts and recorded traces bake in the old semantics and will keep arriving long after the server has moved on. When a tool must change shape, deprecate in the description and nudge in errors: a call using a deprecated parameter succeeds, but the response carries a notice the model can read and adapt to. That is slower than a breaking change and it is supposed to be.

Test the contract like one

Two layers, and they catch different things:

Schema tests. Plain unit tests against your validation layer: limit: 10000 is rejected, empty ids is rejected, unknown status values are rejected. These are fast, deterministic, and fail before any model is involved.
Behavioral assertions on replays. Recorded agent traces replayed against the tool layer with assertions attached: given this ticket, the agent called preview_refund before apply_refund; it never passed limit above 50; after order_not_refundable it did not retry. Replayable harnesses make these assertions regression tests rather than vibes.

The schema tests catch the contract you wrote. The replay assertions catch the contract the model understood. They are not the same document until both are green.

The checklist

Verb-first name, one capability per tool.
Description states what it does, when not to use it, units, and defaults.
Enums over booleans; JSON Schema bounds on everything free-form.
Preview and apply split; destructive inputs structurally required.
Errors carry a stable code, a specific message, and a recovery hint.
Mutating tools take an idempotency key.
Outputs paginated, capped, and summarizable.
Additive versioning; deprecation nudged in description and errors.
Schema tests and replay assertions both cover the contract.

The model is trying to obey you. The schema is where you actually say how.

Human Approval Gates for Tool-Using AI Agents

2026-08-29T15:16:01+00:00

Introduction

Tool-using AI agents become useful when they can do real work: create tickets, send messages, open pull requests, run scripts, update configuration, or trigger deployments. They also become risky at exactly that point. A model can select the right tool for the wrong target, retry a write after the user has changed their mind, or resume from stale context after waiting for a person to respond.

Human approval is the usual answer, but a vague "ask before doing dangerous things" rule is not enough. Approval has to be a first-class workflow state. The runtime must know what exact action is waiting, who is allowed to approve it, when the approval expires, which policy produced the decision, and how to resume without repeating a side effect.

This article shows how to design approval gates for tool-using AI agents. The examples use TypeScript and SQL, but the pattern applies to queues, workflow engines, serverless jobs, background workers, and long-running agent runtimes.

Classify Which Actions Need Approval

Start by deciding which tool calls are allowed automatically, denied outright, or paused for approval. Do this in deterministic application code, not in the model prompt. The model can propose an action, but the runtime owns authorization.

A practical classifier looks at four inputs:

The tool effect: read, write, external communication, destructive change, or privileged operation.
The target: production, customer data, billing, infrastructure, source control, or an internal sandbox.
The actor: user role, tenant policy, service account scope, and current incident state.
The workflow context: budget remaining, deadline, deploy freeze, current step, and prior approvals.

type ToolEffect = "read" | "write" | "notify" | "deploy" | "delete";

type ProposedToolCall = {
  runId: string;
  toolName: string;
  effect: ToolEffect;
  target: string;
  arguments: Record<string, unknown>;
};

type RuntimeContext = {
  actorId: string;
  actorRole: "viewer" | "operator" | "admin";
  environment: "sandbox" | "staging" | "production";
  deployFreezeActive: boolean;
  remainingBudgetUsd: number;
};

type GateDecision =
  | { kind: "allow"; policyVersion: string }
  | { kind: "deny"; policyVersion: string; reason: string }
  | {
      kind: "requires_approval";
      policyVersion: string;
      approverRoles: RuntimeContext["actorRole"][];
      expiresInSeconds: number;
      reason: string;
    };

function classifyToolCall(
  call: ProposedToolCall,
  context: RuntimeContext,
): GateDecision {
  const policyVersion = "agent-tools-2026-08-29";

  if (context.remainingBudgetUsd <= 0) {
    return { kind: "deny", policyVersion, reason: "budget_exhausted" };
  }

  if (call.effect === "read" && context.environment !== "production") {
    return { kind: "allow", policyVersion };
  }

  if (call.effect === "deploy" && context.deployFreezeActive) {
    return { kind: "deny", policyVersion, reason: "deploy_freeze" };
  }

  if (call.effect === "delete" || context.environment === "production") {
    return {
      kind: "requires_approval",
      policyVersion,
      approverRoles: ["operator", "admin"],
      expiresInSeconds: 900,
      reason: "high_impact_side_effect",
    };
  }

  return { kind: "allow", policyVersion };
}

The policy does not need to be clever at first. It needs to be explicit, versioned, and testable. That gives you a stable place to add new rules after incidents, security reviews, and real denied requests.

Approval should be narrow

An approval should grant permission for one normalized action, not a broad future capability. "Approve sending this exact email to this exact customer with this exact template" is safer than "the agent can send emails for the next hour." Narrow approvals are easier to audit and easier to invalidate when context changes.

Store Approval Requests as Durable Workflow State

Approval requests should live in the same durable store as the run, steps, tool calls, and command ledger. A chat message or notification is only a delivery mechanism. The source of truth is the request record.

create type agent_approval_status as enum (
  'pending',
  'approved',
  'denied',
  'expired',
  'cancelled'
);

create table agent_approval_requests (
  id uuid primary key,
  run_id uuid not null,
  step_id text not null,
  command_id uuid not null,
  request_key text not null,
  status agent_approval_status not null default 'pending',
  policy_version text not null,
  reason text not null,
  approver_roles text[] not null,
  requested_by text not null,
  approved_by text,
  denied_by text,
  normalized_tool_name text not null,
  normalized_target text not null,
  normalized_arguments jsonb not null,
  arguments_sha256 text not null,
  expires_at timestamptz not null,
  decided_at timestamptz,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (run_id, request_key)
);

create index agent_approval_requests_pending_idx
  on agent_approval_requests (status, expires_at);

The request_key should be derived from the logical action: run id, step id, command id, normalized tool name, target, and an argument hash. If a worker crashes after creating the request but before sending the notification, the resumed worker can find the same pending request instead of creating a duplicate approval card.

Store the facts the approver needs

The request should contain enough structured context for a human to make a decision without reading the model transcript. Store the normalized tool call, the risk reason, the policy version, the target environment, and a concise explanation produced by application code. Keep raw prompts and large payloads behind separate links or redacted references.

An approval screen can then render the record directly:

Action: deployService
Target: production/search-api
Reason: high_impact_side_effect
Requested by: agent-run:3e86...
Expires: 15 minutes
Arguments hash: 4d2b...

The hash is not for humans to read. It is for the runtime to prove that the approved action is the same action it later executes.

Present Exact Intent and Recheck Before Resuming

The user should approve an exact, stable intent. By the time they click approve, the world may have changed: the incident may be resolved, the budget may be spent, the deploy freeze may have started, or the user's role may have changed. Approval is one input to the final authorization decision, not a replacement for it.

async function approveRequest(
  db: Database,
  requestId: string,
  approver: { id: string; role: string },
) {
  return db.tx(async (tx) => {
    const request = await tx.one(
      `
      select *
      from agent_approval_requests
      where id = $1
      for update
      `,
      [requestId],
    );

    if (request.status !== "pending") {
      throw new Error(`approval is already ${request.status}`);
    }

    if (new Date(request.expires_at).getTime() <= Date.now()) {
      await markApprovalExpired(tx, request.id);
      throw new Error("approval expired");
    }

    if (!request.approver_roles.includes(approver.role)) {
      throw new Error("approver is not allowed for this request");
    }

    await tx.none(
      `
      update agent_approval_requests
      set status = 'approved',
          approved_by = $2,
          decided_at = now(),
          updated_at = now()
      where id = $1
      `,
      [request.id, approver.id],
    );

    await markCommandApproved(tx, request.command_id, request.id);
    return request.command_id;
  });
}

async function resumeApprovedCommand(db: Database, commandId: string) {
  const command = await loadCommand(commandId);
  const approval = await loadApproval(command.approvalId);

  if (approval.status !== "approved") {
    throw new Error("command has no approved request");
  }

  if (approval.arguments_sha256 !== sha256(stableJson(command.arguments))) {
    throw new Error("approved arguments do not match command arguments");
  }

  const decision = classifyToolCall(command, await loadRuntimeContext(command.runId));
  if (decision.kind !== "allow" && decision.kind !== "requires_approval") {
    await blockCommand(command.id, decision.reason);
    return;
  }

  await executeCommandWithIdempotency(command);
}

Two details matter here. First, the approval update and command update happen in one transaction, so a crash cannot approve the request without marking the command resumable. Second, the resume path rechecks policy and verifies the argument hash before executing the tool.

Do not ask the model to remember the approval

After approval, the worker should resume from stored workflow state. It can call the model again if the next step needs reasoning, but the approved side effect should not depend on the model restating what the human approved. The runtime already has the command, target, arguments, policy version, approval id, and idempotency key. Use those.

Handle Expiry, Denial, and Partial Outcomes

Approval gates need failure behavior that is as explicit as the happy path. A request can expire, be denied, be cancelled because the run was cancelled, or become irrelevant because the underlying command was superseded. Each case should produce a workflow state, not an ambiguous timeout in a worker log.

Use short expiration windows for risky writes. Fifteen minutes is reasonable for many operational actions; financial, permission, and infrastructure changes may need even shorter windows. If the request expires, mark the command blocked or cancelled and require the agent to re-plan against current state. Do not let an old approval wake up a run hours later.

Denial should also be a terminal signal for that exact request. The model may propose an alternative, but it should not keep repackaging the same denied action until a person gives up. Record the denial reason and feed a concise version back into the planning loop:

type ApprovalOutcome =
  | { kind: "approved"; commandId: string }
  | { kind: "denied"; reason: string }
  | { kind: "expired" }
  | { kind: "cancelled"; reason: string };

function nextStateAfterApproval(outcome: ApprovalOutcome): AgentRunStatus {
  switch (outcome.kind) {
    case "approved":
      return "ready_to_resume";
    case "denied":
      return "needs_replan";
    case "expired":
      return "needs_replan";
    case "cancelled":
      return "cancelled";
  }
}

Partial outcomes happen after approval, not before it. The user may approve a deployment, the worker may call the deployment API, and the connection may drop before the response returns. That is no longer an approval problem. It is a command execution problem. The command should move to uncertain, and a reconciler should query the external system using the idempotency key or natural key before retrying.

Test the Gate as a Failure Boundary

Approval gates are easy to demo and easy to get wrong in production. The valuable tests are crash, replay, and race tests. They prove that the runtime can pause safely, resume once, and avoid executing stale or changed intent.

it("does not execute a changed command after approval", async () => {
  const run = await startRun({ goal: "deploy search-api to production" });
  const command = await reserveCommand(run.id, {
    toolName: "deployService",
    target: "production/search-api",
    arguments: { version: "2026.08.29-1" },
  });

  const approval = await createApprovalRequest(command.id);
  await approveRequest(db, approval.id, { id: "u_123", role: "operator" });

  await tamperWithCommandArguments(command.id, { version: "latest" });

  await expect(resumeApprovedCommand(db, command.id)).rejects.toThrow(
    "approved arguments do not match command arguments",
  );

  await expect(deployService).not.toHaveBeenCalled();
});

Add tests for the less dramatic cases too:

A worker crashes after creating the approval request but before notifying the user.
The same approval callback is delivered twice.
The approval expires while the run is asleep.
A user's role changes after the request is created.
A deploy freeze starts after approval but before execution.
Two workers try to resume the approved command at the same time.
The external tool times out after applying the side effect.

Those tests keep the approval gate honest. Without them, the system may look safe in the product surface while still allowing duplicate writes, stale approvals, or policy bypasses in the worker.

Conclusion and Next Steps

Human approval is not a button attached to a chat transcript. It is a durable authorization boundary around a specific side effect. A reliable approval gate classifies risk in code, stores a narrow request, shows exact intent, expires quickly, rechecks policy before execution, and resumes through the command ledger with idempotency.

Start with the highest-impact tool your agent can call. Write the classifier, store approval requests as durable records, and add one test that tampers with the command after approval. Once that fails safely, add expiry, duplicate callback handling, and reconciliation for uncertain tool outcomes. The result is an agent that can ask for help without losing control of what it is allowed to do.

The Static Accessor TypeScript Won't Protect

2026-08-21T17:45:00+00:00

Here are two classes. One of them silently does nothing.

class Base {
  static set config(v: unknown) { register(this.name, v); }
  static get config(): unknown { return lookup(this.name); }
}

class A extends Base {
  static config = { retries: 3 };   // ← the setter never runs
}

class B extends Base {}
B.config = { retries: 3 };          // ← the setter runs

A compiles without a single diagnostic. A.config even reads back { retries: 3 }, so it looks like it worked. But register was never called, because that read is hitting an own property that now sits in front of the inherited getter.

If register is what wires up your library, A is broken and nothing told you.

Why the field wins

A class field is not an assignment. Since ES2022 — and in TypeScript whenever useDefineForClassFields is on, which is the default for target: ES2022 and above — a field declaration is installed with [[DefineOwnProperty]], not [[Set]].

The distinction is the whole bug:

[[Set]] walks the prototype chain. It finds the inherited setter and calls it.
[[DefineOwnProperty]] does not walk anything. It stamps a new own data property directly onto the object, shadowing whatever the chain had.

So static config = … doesn't call Base's setter. It replaces it, on that subclass.

You can watch the emit flip:

class B { static set outbound(h: unknown) { console.log("SETTER RAN"); } }
class C extends B { static outbound = () => {}; }

`target`	emitted	semantics	result
`ES2021`	`C.outbound = () => {};` after the class	`[[Set]]`	prints `SETTER RAN`
`ES2022`+	`static outbound = () => {};` inside the class	`[[DefineOwnProperty]]`	silence

Same source. Same library. One config line between working and not — which means this can arrive during a routine target bump, in a commit that touches no application code.

TypeScript already has an error for this

Here is the part that surprised me. TypeScript does diagnose this pattern:

class Base { set thing(v: unknown) {} get thing(): unknown { return 1; } }
class Sub extends Base { thing = { a: 1 }; }

error TS2610: 'thing' is defined as an accessor in class 'Base',
but is overridden here in 'Sub' as an instance property.

That is exactly the right error, and it names exactly the right hazard.

Now make both members static:

class Base { static set thing(v: unknown) {} static get thing(): unknown { return 1; } }
class Sub extends Base { static thing = { a: 1 }; }

(no output)

No error. No warning. Not under --strict. I checked this on TypeScript 6.0.2.

The compiler knows this shape is dangerous enough to have a dedicated diagnostic for it, and that diagnostic covers instance members only. Static members get nothing from it.

The closest thing they get is TS4114, under noImplicitOverride — not part of --strict, but set by @tsconfig/strictest, so plenty of projects have it on:

error TS4114: This member must have an 'override' modifier because it
overrides a member in the base class 'Base'.

That is not shadow detection. It is generic override hygiene, it names the wrong hazard, and the fix it demands makes things worse: add override and the compiler goes quiet while the registration is still being skipped. The instance case cannot be dismissed that way — TS2610 keeps firing with override present, because it is diagnosing the actual problem.

What this looks like in a real library

@cloudflare/containers lets you intercept a container's outbound network traffic by putting a handler on the container class. Three of its configuration points — outbound, outboundByHost and outboundHandlers — are static accessors, and their setters are the only writers of the handler registry the runtime proxy reads from.

The README documents all three as class fields:

static outbound = (req: Request) => {
  return new Response(`Hi ${req.url}, I can't handle you`);
};

Follow that on a modern tsconfig and no handler is ever registered. The proxy looks up the registry, finds nothing, and falls through — and where it falls through to depends on how the container is configured. On the package default (enableInternet is true) it calls fetch(request), so the traffic you meant to intercept goes out unmediated. Follow the README's own example, which pairs enableInternet = false with an allowedHosts list, and the allowlist gate hands your configured hosts straight to the real origin for the same reason. Only with internet disabled and no allowlist does it reach the default-deny path and answer 520 Origin is disallowed.

The failure has an unpleasant shape:

It is silent. No throw, no warning, no type error.
It usually fails open. The handler you wrote to redact, block or log never runs, and the request goes out anyway — which is the more dangerous half of this. Total egress denial is the loud version, and you only get it in the narrow case of internet disabled with no allowlist.
It looks correct. MyContainer.outbound returns your function, because you defined it as an own property. Every reflexive check you'd run says the handler is there.
It is far from its cause. The registration happens at module-eval time; the failure shows up on a request, in a different component, later.

The tell is the mismatch between the two sides. MyContainer.outbound reads back your handler, but the registry the proxy consults is empty — because the only thing that writes that registry is the setter you shadowed.

I filed this as cloudflare/containers#247 with a runnable reproduction, and opened #248 to fix the docs.

The detail I found most interesting: the same repository already documents the correct form. docs/egress.md — linked from the README — uses the assignment form in all six places it sets a handler:

MyContainer.outboundByHost = { /* … */ };

Two documents in one repo, disagreeing, and one of them costs you all your egress. Nothing caught the drift, because nothing could: no test, no type, no lint rule sees the difference between prose that works and prose that doesn't.

If you maintain a library

Don't put a static accessor in your public API if you can avoid it. A static method (MyContainer.setOutbound(handler)) cannot be shadowed by a field, and the failure mode simply doesn't exist. This is the fix that removes the class of bug rather than documenting around it.

If you keep the accessor, detect the shadow. You know what a correct registration looks like. At the point where you'd read the registry, check whether the constructor carries an own property that a registration would have consumed:

const shadowed = Object.getOwnPropertyDescriptor(ctor, "outbound");
// A data property is a shadowing field. An accessor is the class that declares
// the accessor in the first place, which is not a bug.
if (shadowed && !shadowed.get && !shadowed.set && !registry.has(ctor.name)) {
  throw new Error(
    `${ctor.name}.outbound was declared as a static class field, which shadows the ` +
    `inherited setter and skips registration. Delete the field and assign after the ` +
    `class body instead: ${ctor.name}.outbound = handler;`
  );
}

That converts an unmediated request in production into a one-line diagnosis at startup. Note the wording of the remedy: the field has to be deleted, not merely followed by an assignment. Appending MyContainer.outbound = handler below a static outbound = … field is a [[Set]] that finds the own data property and overwrites it in place, never reaching the inherited setter — the same trap one level down.

Grep your own docs. If your API is an accessor, every static yourThing = in your README is a broken example. It costs one CI grep to keep them honest, and — as above — your docs can disagree with each other for a long time without anyone noticing.

If you consume one

Assume every static x = … in a README is untested prose. Documentation examples are rarely type-checked and almost never executed. That one is a class field is invisible to the person who wrote it, because it worked when they wrote it.

Check the type declarations, not the docs. static get x() / static set x() in the .d.ts means assign after the class body. It takes ten seconds and it is authoritative in a way the README is not.

Check that your interception is happening at all, not just that nothing broke. A handler that silently never registered looks like traffic flowing normally, because it is — your redaction, blocking or logging simply isn't in the path. If a uniform failure did appear instead, that is the same bug wearing its loud face. Either way, if a target bump landed near the incident, look there before you audit your network rules.

The general shape

Class fields and accessors are both ways of writing x = value in a class body, and they do fundamentally different things. Where a base class exposes a static accessor with side effects in the setter, a subclass field silently replaces it — the compiler will warn you on instance members and — apart from a generic override warning that its own fix silences — stay quiet on static ones.

Any library whose configuration is "set this static property and we'll register it" is exposed. It's worth checking whether yours is.

Spend Ceilings for Public Endpoints

2026-08-21T16:40:00+00:00

Introduction

Most endpoints cost CPU. A few cost money every time they succeed. A subscription form sends an email. A password reset sends an SMS. A "notify me" button fires a webhook into a metered queue. The request is free to make, expensive to serve, and the person making it is anonymous.

Rate limiting is the usual answer, and it is necessary, but it is not the interesting part. The interesting failures are the ones that survive a correct rate limiter: the concurrent pair that both pass a limit of one, the window that resets at midnight so two messages land sixty seconds apart, the failed retry that burns a slot without sending anything, and the configuration typo that multiplies the ceiling tenfold without anyone noticing.

This article is about bounding that spend so the worst case is a number you can state in advance, rather than a number you discover on an invoice. The examples use TypeScript and SQL against a single-primary relational store, but the ideas apply to any endpoint whose success has a per-unit cost.

Separate the Throttle From the Ceiling

These look like the same mechanism and are not.

A throttle protects the service. It is per-caller, measured in seconds or minutes, and approximate is fine — nobody minds if a limit of sixty requests a minute occasionally admits sixty-two.

A ceiling protects the budget. It is global, measured over a billing period, and approximate is useless. The entire value of a ceiling is that it is a hard number; one that admits "roughly" the right amount of spend has not bounded anything.

That difference decides where each lives. A throttle can sit in an eventually consistent store — a cache with tens of seconds of propagation delay is fine for smoothing traffic. A ceiling cannot. Propagation delay is exactly long enough for a burst to walk past a limit whose only job is stopping bursts. Put the ceiling somewhere a counter read back in the same statement is the real count: a single-primary database, a durable object, a store with atomic increments and a return value.

Write both down as separate numbers with separate windows before you write either one in code. Conflating them is how people end up with a per-IP limit doing a job it cannot do, because any attacker with a second IP address has doubled your bill.

Reserve the Spend, Do Not Merely Permit It

Here is the shape almost everybody writes first.

const lastSend = await db.lastSendAt(recipient);
if (lastSend && now - lastSend < COOLDOWN) {
  return accepted();
}

await sendEmail(recipient);                 // 200ms, sometimes 8s
await db.recordSend(recipient, now);        // the proof, written last

It reads correctly and it is broken. The gap between the check and the record is the duration of a network call. Every request that arrives inside that window reads "never sent", judges itself eligible, and proceeds. Two concurrent requests send two messages against a limit of one.

You cannot close this by wrapping it in a transaction, because the expensive part is a third-party API call and that cannot live inside a database transaction. The fix is to change what the write means. Stop writing a record that the work happened and start writing a reservation that the work is about to happen — before the call, in the same atomic statement that decides eligibility.

INSERT INTO senders (recipient, sends, last_send_at)
VALUES (?1, 1, ?2)
ON CONFLICT(recipient) DO UPDATE SET
  sends = CASE
    WHEN senders.last_send_at IS NULL OR senders.last_send_at <= ?3
    THEN senders.sends + 1 ELSE senders.sends END,
  last_send_at = CASE
    WHEN senders.last_send_at IS NULL OR senders.last_send_at <= ?3
    THEN ?2 ELSE senders.last_send_at END
RETURNING sends, last_send_at

Parameter ?2 is now, ?3 is the cooldown boundary. RETURNING hands back the post-update row, which tells the caller whether it is the one holding the reservation.

const row = await reserve(recipient, now, now - COOLDOWN);
if (row.last_send_at !== now) {
  return accepted();          // somebody else got there first
}

await sendEmail(recipient);

The reservation is the permission. There is no second check, because a second check is where the race lives. A concurrent request runs the same statement, sees a last_send_at that is inside the cooldown, and stands down — without ever reading a value that another request is still in the middle of earning.

Release What You Reserved

Reserving before the work means the reservation can outlive work that never happened. The provider returns a 500, the request times out, the process is killed mid-flight — and the recipient now carries a cooldown for a message they never received.

So the failure path has to give the reservation back.

UPDATE senders
SET sends = CASE WHEN sends > 0 THEN sends - 1 ELSE 0 END,
    last_send_at = NULL
WHERE recipient = ?1 AND last_send_at = ?2

The last_send_at = ?2 guard is the important half. It matches only the exact reservation this request wrote, so a slow release arriving after a newer reservation has legitimately taken its place cannot undo it.

This is easy to skip and expensive to omit. Without a release, a single bad minute at your provider costs every affected user a full cooldown window — and their retry, thirty seconds later, is refused by a reservation held for a message that does not exist. The failure mode is invisible in testing and infuriating in production, because the people it hits are exactly the people who wanted your product enough to try twice.

Put the Ceiling Outside the Write

Ordering matters as much as the mechanism.

If you reserve first and consult the global ceiling second, then a request the ceiling refuses has already mutated state. In a system I worked on recently, that meant rotating a confirmation token for a message that was then never sent — quietly invalidating the live link already sitting in the recipient's inbox, and mailing no replacement.

The rule that avoids this: every gate that can refuse the work runs before the write that commits to it. The write is what happens once all the gates have said yes. Anything that fails after that point releases what it took.

free checks        method, content type, body size, origin, honeypot, syntax
per-caller throttle    counters keyed on the caller
bot verification       the first thing that costs a network call
per-recipient window   is this destination inside its cooldown?
global ceiling         is there budget left at all?
-------------------------------------------------- gates end here
reserve                the atomic write
send                   the expensive part
on failure             release everything reserved above

Note the secondary ordering inside the gates: nothing that costs a network call runs before something free can reject the request. A malformed body should never reach your bot-verification provider. That is not an optimisation, it is the difference between a flood costing you nothing and a flood costing you one upstream call per request.

Choose Windows That Cannot Be Walked Around

Two mistakes recur, and both produce limits that look right in tests.

Calendar buckets. A key like recipient:2026-08-21 with a limit of one per day resets at midnight UTC rather than twenty-four hours after the send. An attacker sends at 23:59:30 and again at 00:00:30: two messages, a minute apart, repeatable every night. Use a stable key whose row carries its own expiry, so the window starts at the first hit and rolls from there.

// The key does not encode the date. The row's own expiry defines the window.
await consume(`inbox:${canonical(recipient)}`, { limit: 1, windowSeconds: 86_400 });

Uncanonicalised identity. A limit keyed on the literal string the user submitted is a limit on strings, not on people. For email, victim+1@example.com, victim+2@example.com and vic.tim@example.com may all arrive in one inbox. Fold the address to a canonical form for the key, and keep the original for delivery.

const canonical = (email: string): string => {
  const at = email.lastIndexOf('@');
  let local = email.slice(0, at);
  const domain = ALIASES[email.slice(at + 1)] ?? email.slice(at + 1);

  const plus = local.indexOf('+');
  if (plus > 0) local = local.slice(0, plus);
  if (DOT_INSENSITIVE.has(domain)) local = local.split('.').join('');

  return local.length > 0 ? `${local}@${domain}` : email;
};

Be honest about the limits of this. Folding rules are provider-specific and incomplete; somebody with an inbox whose aliasing scheme you have not encoded still gets one message per alias. What stops that from being unbounded is the global ceiling sitting behind it, which is precisely why the ceiling is not optional.

The general lesson outlives the email example. Every limit key is an identity claim. Before you ship one, ask what else maps to the same real-world resource — a phone number with punctuation, a URL with a trailing slash, a customer with two accounts.

Make the Ceiling a Number You Can Defend

Pick the ceiling from the bill, not from comfort.

If your plan includes three thousand messages a month, a hundred a day is not an arbitrary round number — it is thirty days at a hundred, which is the included allowance exactly. A ceiling derived that way is defensible in a review, and the worst case is a sentence rather than a shrug.

Two properties make it operationally useful. A ceiling of zero should be a working kill switch that leaves the endpoint answering normally, so the response to abuse is a configuration change rather than a deploy. And the refusal should be logged loudly while the caller sees the ordinary success response — the person hitting your form is not the person you are defending against.

Configuration deserves more suspicion than it usually gets, because the ceiling is only as good as the number that reaches it.

const boundedInt = (raw: string | undefined, fallback: number, min: number, max: number, name: string) => {
  const text = String(raw ?? '').trim();
  if (text.length === 0) return fallback;

  // parseInt reads "1e9" as 1 and "500abc" as 500 — a plausible-looking
  // limit nobody asked for. Demand the whole string.
  if (!/^\d+$/.test(text)) {
    console.warn(`${name} is not a whole number ("${text}"); using ${fallback}`);
    return fallback;
  }

  const parsed = Number.parseInt(text, 10);
  const clamped = Math.min(Math.max(parsed, min), max);
  if (clamped !== parsed) console.warn(`${name} of ${parsed} clamped to ${clamped}`);
  return clamped;
};

Clamping bounds a typo; it does not make one safe. A ceiling of 1000 typed instead of 100 sits comfortably inside a generous range and costs ten times as much, so keep the range close to the default and log anything that gets adjusted. A range wide enough to feel flexible is a range wide enough for a stray keystroke to matter.

Fail Closed, and Answer Uniformly

Three properties protect the endpoint itself.

Fail closed. If bot verification times out, returns a 500, or hands back something unparseable, refuse the request. Treating an unreachable verifier as a pass converts one bad minute upstream into an open relay for anyone watching.

Answer uniformly. Every outcome that is not an outright error should return the same body: new recipient, recipient already known, recipient inside a cooldown, ceiling exhausted. Otherwise the endpoint answers a question it was never meant to answer, which is whether a given address is in your database.

Answer at the same moment. Identical bodies are not enough if one path waits on a network round trip and the others return immediately; response time is the oracle. Defer the side effect and return straight away.

ctx.waitUntil(deliver(recipient, token));
return accepted();

Deferring also removes an error branch that only a not-yet-known recipient could ever reach, which is the same leak wearing a different hat.

Test the Ceiling by Removing It

A passing test suite proves nothing about a control until you have watched it fail without it.

The procedure is mechanical: delete the control, run the test that claims to cover it, confirm the test fails, restore the control. Anything that still passes was never testing what its name says. This catches two distinct problems — a test that asserts at the wrong layer, where some other mechanism was quietly doing the work, and a test that never ran at all, because removing the control broke compilation and a suite that does not build reports no failures.

Concurrency needs its own test, because sequential tests cannot see the bug that matters here.

test('two overlapping requests produce exactly one send', async () => {
  const [a, b] = await Promise.all([subscribe(address), subscribe(address)]);
  expect(sendsFrom(a) + sendsFrom(b)).toBe(1);
});

It is worth internalising how ordinary the failure is. A suite can pass completely against code with a live spend bug, because every test in it exercised one request at a time and the bug needs two. Thoroughness about the failure modes you thought of is not coverage of the ones you did not.

Conclusion and Next Steps

Bounding spend on a public endpoint comes down to a few decisions that are cheap to make early and expensive to retrofit:

Keep the throttle and the ceiling separate, and put the ceiling somewhere strongly consistent.
Reserve the spend in one atomic statement before the side effect, and treat that reservation as the permission.
Release the reservation whenever the work does not happen, guarded on the exact reservation you wrote.
Run every refusing gate before the write that commits to the work.
Use rolling windows and canonical keys, so neither the clock nor an alias walks around the limit.
Derive the ceiling from the bill, clamp the configuration, and make zero a kill switch.
Verify each control by removing it, and test concurrency explicitly.

If you are building this alongside other reliability work, the natural neighbours are idempotency keys for the retry path, a transactional outbox if the side effect must survive a crash, and per-tenant budgets if the spend is attributable to accounts rather than to anonymous callers. The mechanism is the same in each case: decide what the write means before you decide where to put it.

Observability for Tool-Using AI Agents

2026-08-14T10:20:00+00:00

Introduction

A failing web request is usually easy to explain. There is one entry point, a handful of downstream calls, and a status code at the end. A failing agent run is different. It may have made nine model calls, selected six tools, retried two of them, waited eleven minutes for an approval, spent a few dollars in tokens, and stopped in a state nobody can describe without reading raw logs.

Traditional application logging does not survive that shape. Prompt dumps are too large to keep, too sensitive to export, and too unstructured to query. Meanwhile the questions operators actually ask are boringly specific: which tool failed, how much did this tenant spend today, why is this run stuck, and did the agent write anything before it died.

This article covers the instrumentation that answers those questions. The examples use TypeScript and OpenTelemetry-style primitives, but the design applies to any runtime that executes model-selected tools: queue workers, workflow engines, serverless jobs, or a long-lived agent service.

Model a Run as a Trace, Not a Log Stream

The unit of debugging for an agent is the run, not the request. A run has a lifetime, a budget, a terminal status, and a tree of decisions inside it. That maps naturally onto a trace, so start by giving every run a trace ID and every decision a span.

A four-level hierarchy covers most agent runtimes:

Run — the whole task, from goal to terminal status.
Step — one iteration of the agent loop, holding whatever the model decided this time around.
Model call — a single request to the model, with its token usage and chosen tool.
Tool call — one validated tool invocation, including retries and policy decisions.

type SpanKind = "run" | "step" | "model" | "tool" | "policy";

type AttributeValue = string | number | boolean | string[];

type SpanStart = {
  kind: SpanKind;
  name: string;
  attributes: Record<string, AttributeValue>;
};

type SpanEnd = {
  status: "ok" | "error";
  attributes?: Record<string, AttributeValue>;
};

interface ActiveSpan {
  setAttributes(attributes: Record<string, AttributeValue>): void;
  end(result: SpanEnd): void;
}

interface AgentTracer {
  start(span: SpanStart): ActiveSpan;
}

async function withSpan<T>(
  tracer: AgentTracer,
  start: SpanStart,
  body: (span: ActiveSpan) => Promise<T>,
): Promise<T> {
  const span = tracer.start(start);

  try {
    const result = await body(span);
    span.end({ status: "ok" });
    return result;
  } catch (error) {
    span.end({
      status: "error",
      attributes: {
        "error.type": error instanceof Error ? error.name : "unknown",
        "error.message": error instanceof Error ? error.message : String(error),
      },
    });
    throw error;
  }
}

The important detail is that the span closes on the error path too. Agent runtimes are full of await chains that abandon work on timeout, and an unfinished span is worse than no span: it looks like the step is still running long after the worker died.

Keep the trace ID when the process does not

Agent runs outlive processes. A run can pause for an approval, get picked up by another worker, or resume after a deploy. If the trace context lives only in memory, the second half of the run becomes an orphan trace and the interesting failure is exactly at the seam.

Persist the trace ID and the current step's span ID alongside the run's durable state, then resume as a child of the stored context. The result is one trace that spans hours and several processes, which is what you need when the question is "what happened before it got stuck."

Choose Attributes That Answer Real Questions

Spans are only as useful as the fields you can filter on. Decide what an operator will type into a query box at 2 a.m., then make sure those fields exist on every relevant span with the same name and the same type.

type ModelSpanAttributes = {
  "gen_ai.operation.name": "chat";
  "gen_ai.system": string;
  "gen_ai.request.model": string;
  "gen_ai.response.model": string;
  "gen_ai.usage.input_tokens": number;
  "gen_ai.usage.output_tokens": number;
  "gen_ai.response.finish_reasons": string[];
  "agent.run.id": string;
  "agent.step.index": number;
  "agent.cost.micros": number;
};

type ToolSpanAttributes = {
  "agent.run.id": string;
  "agent.step.index": number;
  "agent.tool.name": string;
  "agent.tool.effect": "read" | "write";
  "agent.tool.idempotency_key": string;
  "agent.tool.attempt": number;
  "agent.policy.decision": "allowed" | "denied" | "needs_approval";
  "agent.tenant.id": string;
};

Three of these carry most of the debugging weight. agent.tool.effect lets you separate harmless reads from writes that may need reconciliation. agent.tool.idempotency_key links the span to the durable command record, so a trace and a database row can be joined without guesswork. agent.policy.decision records that the runtime made a choice, which is how you prove a denied call never executed.

Use the GenAI conventions, but pin your own namespace

OpenTelemetry's generative-AI semantic conventions give you gen_ai.* names for model calls, and adopting them means vendor dashboards understand your traces without custom mapping. They are also still evolving, and attribute names in that namespace have changed between releases.

The practical compromise is to emit the conventional names for model-level facts and keep your own agent.* namespace for workflow facts the spec does not cover: run IDs, budgets, approvals, idempotency keys, and policy decisions. When the convention shifts, you update one adapter instead of every dashboard and alert you own.

Record the decision, not the prose

It is tempting to attach the model's reasoning text to the span, and it is almost always the wrong default. The text is large, sensitive, expensive to index, and rarely the thing you filter on. Attach the structured outcome instead — selected tool, argument hash, finish reason, retry count, policy verdict — and keep the prose in a payload store you can open on demand.

Redact at the Boundary, Not in the Backend

Agent spans carry tool arguments, and tool arguments carry whatever the user was talking about: customer records, ticket bodies, file contents, occasionally an access token the model helpfully copied from earlier context. Once that leaves your process it is in a third-party index, subject to their retention policy rather than yours.

Redact in the exporter path, before anything is serialized for transport.

const SENSITIVE_KEYS = new Set([
  "password",
  "secret",
  "token",
  "api_key",
  "authorization",
  "access_token",
  "refresh_token",
  "cookie",
]);

const MAX_STRING = 256;

function redactValue(value: unknown): unknown {
  if (Array.isArray(value)) {
    return value.map(redactValue);
  }

  if (value && typeof value === "object") {
    return Object.fromEntries(
      Object.entries(value as Record<string, unknown>).map(([key, nested]) => [
        key,
        SENSITIVE_KEYS.has(key.toLowerCase()) ? "[redacted]" : redactValue(nested),
      ]),
    );
  }

  if (typeof value === "string" && value.length > MAX_STRING) {
    return `${value.slice(0, MAX_STRING)}[truncated ${value.length - MAX_STRING}]`;
  }

  return value;
}

A denylist alone is not a compliance story. It catches the obvious keys and nothing else, so pair it with two stronger controls: an allowlist of argument fields that specific tools may export in full, and a size cap that pushes anything large out of the span entirely.

Store big payloads by reference

Spans should stay small enough to sample and index cheaply. When a tool result is a 40 KB document, write the payload to object storage under a run-scoped key and put only the pointer and a hash on the span.

const MAX_INLINE_PAYLOAD = 1024;

async function attachPayload(
  span: ActiveSpan,
  store: PayloadStore,
  runId: string,
  stepIndex: number,
  label: string,
  payload: unknown,
) {
  const body = JSON.stringify(redactValue(payload));

  if (body.length <= MAX_INLINE_PAYLOAD) {
    span.setAttributes({ [`agent.payload.${label}`]: body });
    return;
  }

  const key = `runs/${runId}/${stepIndex}/${label}.json`;
  await store.put(key, body);

  span.setAttributes({
    [`agent.payload.${label}.ref`]: key,
    [`agent.payload.${label}.bytes`]: body.length,
    [`agent.payload.${label}.sha256`]: sha256(body),
  });
}

The hash is what makes this worth the extra call. It lets you compare two runs that "did the same thing" and prove whether the arguments were actually identical, without ever exporting the arguments themselves.

Measure What Operators Page On

Traces explain one run. Metrics tell you whether the fleet is healthy, and they are what alerts should read from, because they are cheap, complete, and unaffected by sampling.

const runsFinished = meter.createCounter("agent.runs.finished");
const runDuration = meter.createHistogram("agent.run.duration", { unit: "s" });
const runSteps = meter.createHistogram("agent.run.steps");
const toolCalls = meter.createCounter("agent.tool.calls");
const tokensUsed = meter.createCounter("agent.tokens.used");
const costMicros = meter.createCounter("agent.cost.micros");

function recordToolCall(tool: string, outcome: "ok" | "error" | "denied") {
  toolCalls.add(1, { tool, outcome });
}

function recordRunFinished(run: FinishedRun) {
  const labels = {
    workflow: run.workflow,
    status: run.status,
    reason: run.reason,
  };

  runsFinished.add(1, labels);
  runDuration.record(run.durationMs / 1000, { workflow: run.workflow });
  runSteps.record(run.steps, { workflow: run.workflow });
  tokensUsed.add(run.tokens.input, { workflow: run.workflow, direction: "input" });
  tokensUsed.add(run.tokens.output, { workflow: run.workflow, direction: "output" });
  costMicros.add(run.costMicros, { workflow: run.workflow, model: run.model });
}

Watch the label sets. workflow, status, reason, tool, and model are bounded and safe. run_id, tenant_id, user_id, and anything derived from model output are not — they turn one metric into millions of time series and will eventually cost more than the agent. Keep unbounded identifiers on spans, where the storage model expects high cardinality.

Attribute cost to something you can act on

Token counters are only interesting when they roll up to a decision. Record cost in integer micros rather than floating-point dollars, so the numbers stay exact through aggregation, and attach the workflow and model as labels so you can answer the two questions that matter: which workflow got more expensive after the last deploy, and which model change actually paid for itself.

For per-tenant spend, aggregate from the durable run records rather than the metrics pipeline. Billing needs exact numbers over a billing period, not sampled telemetry with a retention window.

Sample Long Runs Without Losing Failures

Head-based sampling is a poor fit for agents. The decision gets made when the run starts, which is exactly when you know the least about whether it will be interesting. A run that looked routine for eight steps and then double-charged a customer is the one trace you cannot afford to have dropped at step one.

Decide at the end of the run instead, and bias hard toward keeping anything unusual.

type RunSummary = {
  runId: string;
  workflow: string;
  status: "completed" | "failed" | "blocked" | "cancelled";
  steps: number;
  durationMs: number;
  costMicros: number;
  wroteSideEffect: boolean;
  hadRetry: boolean;
};

function shouldKeepTrace(run: RunSummary, keptWorkflows: Set<string>): boolean {
  if (run.status !== "completed") return true;
  if (run.wroteSideEffect || run.hadRetry) return true;
  if (run.steps > 20 || run.durationMs > 60_000) return true;
  if (run.costMicros > 50_000) return true;

  if (!keptWorkflows.has(run.workflow)) {
    keptWorkflows.add(run.workflow);
    return true;
  }

  return hashToUnitInterval(run.runId) < 0.05;
}

The last line uses a hash of the run ID rather than a random number on purpose. A deterministic function gives every component that sees the run — the worker, the collector, a replay job — the same answer, so a trace is never half-kept because two processes rolled different dice.

Keeping at least one full trace per workflow is the cheap part of this policy and often the most useful. It gives you a known-good reference to diff against when someone reports that a workflow "used to work."

Close the Loop with a Terminal Event

Every run should end by writing one durable, structured record that explains itself. This is the piece teams skip, and it is the reason support tickets turn into archaeology.

type RunFinishedEvent = {
  type: "run_finished";
  runId: string;
  traceId: string;
  workflow: string;
  status: "completed" | "failed" | "blocked" | "cancelled";
  reason:
    | "goal_reached"
    | "budget_exhausted"
    | "approval_expired"
    | "policy_denied"
    | "tool_unavailable"
    | "operator_cancelled"
    | "unrecoverable_error";
  steps: number;
  costMicros: number;
  tokens: { input: number; output: number };
  sideEffects: Array<{
    tool: string;
    idempotencyKey: string;
    outcome: "applied" | "skipped" | "unknown";
  }>;
};

Two fields here do disproportionate work. reason separates failures that need engineering attention from stops that were the system working correctly — a run that halted on an exhausted budget is not an incident, and it should never page anyone. And an unknown side-effect outcome is the flag that a write may or may not have landed, which is the signal a reconciliation job should consume before anyone retries by hand.

Alert on stalls and duplicates, not on error rate

Agent failures are rarely loud. The symptoms that actually correlate with user pain are stalls and repeats, so build the first alerts around those:

Runs in a non-terminal state past their deadline, which means a worker died holding the lease.
Approval requests older than their expiry with nobody notified.
Two applied side effects sharing an idempotency key, which is a correctness bug by definition.
A rise in unknown outcomes, meaning reconciliation is falling behind.
Cost per completed run climbing while completion rate stays flat, the usual signature of a retry loop.

Each of these is answerable from the terminal event and the metrics above, without touching a sampled trace. Traces are for the follow-up question: once an alert says a run stalled, the trace shows which tool it was waiting on.

Conclusion and Next Steps

Instrumenting an agent is not the same job as instrumenting a service. The run is the unit of work, the interesting state is durable rather than in-process, and the most expensive failures are silent ones where something got written twice or nothing got written at all.

Start with the run trace and the terminal event, since together they explain almost every support question you will get. Add token and cost counters next, with bounded labels and integer micros. Then tighten the export path: redact by allowlist, push large payloads to storage by reference, and switch to tail sampling that always keeps failures, retries, and writes. Once those are in place, an operator can answer what the agent did, what it spent, and whether it left anything half-finished, without reading a single prompt.

Replayable Test Harnesses for AI Agents

2026-08-11T17:41:35+00:00

Introduction

Tool-using AI agents are difficult to test because the interesting behavior crosses several boundaries at once. The model may choose a tool, the runtime may validate arguments, the tool may call an external API, and the workflow may persist state before deciding what to do next. A normal unit test can cover one function, but it rarely proves that the whole agent survives a timeout, resumes after a crash, avoids duplicate writes, and explains why it stopped.

The answer is not to hope the model makes the same decision every time. The useful pattern is a replayable test harness: record the runtime boundary during a real or scripted run, then replay that boundary deterministically with fake clocks, fake tools, and deliberate failures. The model can remain probabilistic in production while the workflow contract stays testable in CI.

This article shows how to build that harness for AI agent workflows. The examples use TypeScript, but the same design works in queues, workflow engines, serverless jobs, background workers, and durable agent runtimes.

Test the Runtime Boundary, Not the Prompt

Prompt snapshots are a weak test surface. They are useful for review, but they do not tell you whether the runtime protected a write tool, persisted a checkpoint, honored a budget, or resumed without duplicating work. A replayable harness should wrap the boundary where nondeterministic decisions become operational events.

For most agents, that boundary has four parts:

Model calls, including the normalized request and the chosen response.
Tool calls, including the validated arguments and returned result.
State writes, including checkpoints, command records, approvals, and terminal status.
Time, including deadlines, leases, retry delays, and scheduled wakeups.

The production runtime and the test harness should use the same interface. That keeps replay from becoming a parallel implementation that silently drifts away from reality.

type ModelRequest = {
  runId: string;
  stepId: string;
  messages: Array<{ role: "system" | "user" | "assistant"; content: string }>;
  tools: Array<{ name: string; schema: unknown }>;
};

type ModelResponse = {
  message: string;
  toolCall?: {
    name: string;
    arguments: Record<string, unknown>;
  };
};

type ToolRequest = {
  runId: string;
  stepId: string;
  toolName: string;
  arguments: Record<string, unknown>;
  idempotencyKey: string;
};

type AgentIO = {
  now(): Date;
  callModel(request: ModelRequest): Promise<ModelResponse>;
  callTool(request: ToolRequest): Promise<unknown>;
  writeEvent(event: AgentEvent): Promise<void>;
};

The agent loop depends on AgentIO, not on a concrete model SDK, HTTP client, or wall clock. Production code passes real adapters. Tests pass a recorder, a replayer, or a fault-injecting wrapper. That single seam is what makes the workflow inspectable.

Keep validation outside the model

The model can propose a tool call, but validation should live in deterministic code before the call becomes executable. Tests should assert that invalid proposals become rejected events, not that the model never makes them. This distinction matters because a prompt change can improve suggestions, but only runtime policy can guarantee safety.

Record Canonical Traces

A trace is not a raw log dump. It is the minimum contract needed to replay the run and prove the same runtime decisions still happen. Store entries in a stable order, remove volatile fields, normalize arguments, and keep enough metadata to diagnose failures later.

type TraceEntry =
  | {
      kind: "model";
      stepId: string;
      requestHash: string;
      response: ModelResponse;
    }
  | {
      kind: "tool";
      stepId: string;
      toolName: string;
      argumentHash: string;
      idempotencyKey: string;
      result: unknown;
    }
  | {
      kind: "event";
      stepId: string;
      eventType: string;
      payload: Record<string, unknown>;
    }
  | {
      kind: "clock";
      label: string;
      isoTime: string;
    };

function stableJson(value: unknown): string {
  if (Array.isArray(value)) {
    return `[${value.map(stableJson).join(",")}]`;
  }

  if (value && typeof value === "object") {
    return `{${Object.entries(value as Record<string, unknown>)
      .sort(([left], [right]) => left.localeCompare(right))
      .map(([key, nested]) => `${JSON.stringify(key)}:${stableJson(nested)}`)
      .join(",")}}`;
  }

  return JSON.stringify(value);
}

async function recordToolCall(
  trace: TraceWriter,
  request: ToolRequest,
  callRealTool: () => Promise<unknown>,
) {
  const result = await callRealTool();

  await trace.append({
    kind: "tool",
    stepId: request.stepId,
    toolName: request.toolName,
    argumentHash: sha256(stableJson(request.arguments)),
    idempotencyKey: request.idempotencyKey,
    result,
  });

  return result;
}

Hashes keep traces compact while still detecting drift. If a tool argument changes, the replay fails at the exact boundary instead of continuing with a stale fixture. When debugging, store the full normalized request in a private artifact, but keep public or committed traces free of secrets and customer data.

Version the trace format

Trace files should include a schema version and runtime version. Agent systems change quickly: a new tool argument, a renamed event, or a stricter policy can make an old trace fail for a legitimate reason. Versioning lets the test harness distinguish "the workflow regressed" from "this fixture needs migration."

Replay Without Real Side Effects

Replay mode should make the agent believe it is running normally while every external boundary is served from the trace. That means model responses come from recorded entries, tools return recorded results, and time advances according to recorded clock entries or an injected fake clock.

class TraceReplayer implements AgentIO {
  private cursor = 0;

  constructor(
    private readonly entries: TraceEntry[],
    private readonly events: AgentEvent[] = [],
  ) {}

  now(): Date {
    const entry = this.next("clock");
    return new Date(entry.isoTime);
  }

  async callModel(request: ModelRequest): Promise<ModelResponse> {
    const entry = this.next("model");
    const actualHash = sha256(stableJson(request));

    if (entry.requestHash !== actualHash) {
      throw new Error(
        `model request drift at ${request.stepId}: expected ${entry.requestHash}, got ${actualHash}`,
      );
    }

    return entry.response;
  }

  async callTool(request: ToolRequest): Promise<unknown> {
    const entry = this.next("tool");
    const actualHash = sha256(stableJson(request.arguments));

    if (entry.toolName !== request.toolName) {
      throw new Error(`tool drift: expected ${entry.toolName}, got ${request.toolName}`);
    }
    if (entry.argumentHash !== actualHash) {
      throw new Error(`argument drift for ${request.toolName}`);
    }
    if (entry.idempotencyKey !== request.idempotencyKey) {
      throw new Error(`idempotency key drift for ${request.toolName}`);
    }

    return entry.result;
  }

  async writeEvent(event: AgentEvent): Promise<void> {
    this.events.push(event);
  }

  private next<TKind extends TraceEntry["kind"]>(
    kind: TKind,
  ): Extract<TraceEntry, { kind: TKind }> {
    const entry = this.entries[this.cursor++];
    if (!entry || entry.kind !== kind) {
      throw new Error(`trace drift: expected ${kind}, got ${entry?.kind ?? "end"}`);
    }
    return entry as Extract<TraceEntry, { kind: TKind }>;
  }
}

The strict cursor is deliberate. If the workflow adds an extra model call, skips a policy event, or calls a different tool, the replay fails immediately. That is the point: an agent test should catch control-flow drift before a changed prompt or refactor reaches production.

Do not assert on every token

Replay tests should be strict at operational boundaries and flexible around explanatory text. Exact model prose is usually the wrong invariant. Better assertions include the selected tool, normalized arguments, state transition, command key, approval requirement, idempotency key, and terminal status.

Inject the Failures You Fear

A passing replay proves the happy path still works. A useful harness also mutates the boundary so the workflow experiences production-shaped failures: model refusal, malformed tool arguments, tool timeout, partial success, process crash after a side effect, expired approval, exhausted budget, or duplicate worker execution.

Do this with small wrappers around AgentIO, not with sleeps or real external dependencies.

type Fault =
  | { at: "model"; stepId: string; error: Error }
  | { at: "tool"; toolName: string; error: Error }
  | { at: "afterEvent"; eventType: string; error: Error };

class FaultInjectingIO implements AgentIO {
  constructor(
    private readonly inner: AgentIO,
    private readonly faults: Fault[],
  ) {}

  now() {
    return this.inner.now();
  }

  async callModel(request: ModelRequest) {
    const fault = this.faults.find(
      (candidate) => candidate.at === "model" && candidate.stepId === request.stepId,
    );
    if (fault?.at === "model") throw fault.error;
    return this.inner.callModel(request);
  }

  async callTool(request: ToolRequest) {
    const fault = this.faults.find(
      (candidate) => candidate.at === "tool" && candidate.toolName === request.toolName,
    );
    if (fault?.at === "tool") throw fault.error;
    return this.inner.callTool(request);
  }

  async writeEvent(event: AgentEvent) {
    await this.inner.writeEvent(event);
    const fault = this.faults.find(
      (candidate) => candidate.at === "afterEvent" && candidate.eventType === event.type,
    );
    if (fault?.at === "afterEvent") throw fault.error;
  }
}

The afterEvent fault is especially valuable. It simulates the uncomfortable production case where the runtime persisted intent or called a tool, then crashed before advancing the workflow. Recovery should reload durable state, observe that the event already happened, and continue without replaying unsafe work.

Test recovery as a two-run scenario

Crash tests should usually run the workflow twice. The first run injects the failure. The second run starts from the persisted database state and uses the same idempotency keys. The assertion is not "the first run returned success." The assertion is that the combined result is correct after recovery.

it("resumes after a crash without sending a duplicate message", async () => {
  const db = await createTestDatabase();
  const trace = await loadTrace("approve-and-notify.json");

  const firstIO = new FaultInjectingIO(new TraceReplayer(trace), [
    {
      at: "afterEvent",
      eventType: "tool_succeeded",
      error: new Error("simulated worker crash"),
    },
  ]);

  await expect(runAgentWorkflow(db, firstIO)).rejects.toThrow("simulated worker crash");

  const secondIO = new TraceReplayer(trace);
  await runAgentWorkflow(db, secondIO);

  const sends = await db.messages.findByRunId("run_123");
  expect(sends).toHaveLength(1);
  expect(sends[0].idempotency_key).toBe("run_123:notify-user:approved");

  const run = await db.agentRuns.findById("run_123");
  expect(run.status).toBe("completed");
});

This is the test that catches the expensive class of agent bugs: duplicate comments, duplicate emails, duplicate tickets, repeated deployments, and workflows that get stuck in an uncertain state after restart.

Decide What Makes a Trace Pass

Replay without clear assertions becomes a snapshot test with a nicer name. Before adding a trace to CI, decide which invariants the trace is meant to protect. A small number of explicit assertions is usually better than comparing a giant JSON blob.

Useful invariants include:

Every write tool has a stable idempotency key.
Approval-required tools cannot execute before an approval event exists.
A crash after recording a side effect does not create a second side effect.
A timeout moves the run to a retryable or blocked state, not silent success.
Budget exhaustion stops the workflow at the next safe boundary.
Reconciliation is attempted before retrying an uncertain write.
Final events contain enough context for support and observability.

Separate these from quality evaluations. A replay harness tells you whether the workflow is reliable. It does not prove the model made the best plan, wrote the clearest summary, or selected the highest-quality patch. Those checks belong in evals, code review, or product-specific acceptance tests.

Keep traces small and named by risk

Trace names should describe the production risk they protect: crash-after-ticket-create, approval-expired-before-deploy, tool-timeout-before-checkpoint, or budget-exhausted-during-research. Avoid a single mega-trace that tries to cover everything. Small traces fail with better error messages and are easier to update when the workflow evolves.

Conclusion and Next Steps

Reliable agents need more than prompt tests. They need deterministic pressure around the runtime boundary where model output becomes state, tool calls, approvals, retries, and side effects. A replayable harness gives you that pressure without depending on live model calls or real external systems.

Start with one risky workflow and record a happy-path trace. Add strict checks for tool arguments, idempotency keys, and state transitions. Then add two failure traces: one timeout before a tool returns and one crash after a side effect is recorded. Once those tests run in CI, every prompt change and runtime refactor has to preserve the recovery contract before it reaches users.

Taming Tail Latency with Hedged Requests

2026-07-04T10:30:00+00:00

Introduction

Average latency is a comforting lie. A service can report a 40 ms mean while a meaningful slice of requests take 500 ms or more, and it is those slow requests — the tail — that users actually feel. The problem compounds the moment one operation depends on many others. A request that fans out to ten backends waits for the slowest of the ten, so a rare per-call delay becomes a common per-request delay. Tool-using AI agents feel this acutely: a single run issues many sequential model and tool calls, and one slow call per step adds up into a run that drags.

Retries do not fix this. A retry helps when a call fails, but a slow-yet-successful call never triggers one — you just wait. Hedged requests target exactly that gap. The idea, popularized by Google's "The Tail at Scale," is simple: send the request, and if it has not answered within a short delay, send a second copy to another replica. Take whichever responds first and cancel the rest. You trade a little extra load for a dramatically shorter tail.

This article shows how to implement request hedging correctly: how it works, how to pick the hedge delay, why you must only hedge idempotent work, and how to cap the extra load so hedging never amplifies an incident. The examples use TypeScript and apply to API gateways, backend services, and AI agent runtimes alike.

Why Tail Latency Dominates Real Systems

The tail matters because it is contagious under fan-out. Suppose each backend responds slowly — say above 200 ms — only 1% of the time. That sounds negligible until a request has to wait on several of them. A request touching 10 such backends is slow whenever any of them is slow, which happens with probability 1 - 0.99^10 ≈ 9.6%. Touch 100 backends and the chance climbs to 1 - 0.99^100 ≈ 63%. A one-in-a-hundred event at the component level becomes the common case at the request level.

AI agents recreate this pattern in time rather than in parallel. An agent that runs twenty steps, each making one model call and one tool call, is a chain of forty dependencies. If each call has a small chance of landing in its own slow tail, the odds that at least one step stalls across the whole run are high, and the user watches the agent hang on that single unlucky call.

This is why tuning the average is the wrong target. Shaving 5 ms off the median does nothing for the request that waited 800 ms on one straggling replica. Stragglers come from causes you cannot fully eliminate: garbage collection pauses, a cold cache, a noisy neighbor on shared hardware, a queue that briefly backed up, a slow disk seek. Hedging accepts that stragglers exist and routes around them instead of trying to prevent every one.

How Request Hedging Works

Hedging runs on a timer. Send the primary request and start a clock. If the primary answers before the hedge delay elapses — which it usually will — you never send a second request and there is no extra cost. If the delay passes with no answer, send a duplicate to a different replica and race them. The first success wins; the loser is cancelled.

The mechanics in TypeScript lean on AbortController to cancel the loser and Promise.any to resolve on the first success while ignoring a straggler that merely fails:

type Attempt<T> = (signal: AbortSignal) => Promise<T>;

async function hedgedCall<T>(attempt: Attempt<T>, hedgeDelayMs: number): Promise<T> {
  const controllers = new Set<AbortController>();
  let hedgeTimer: ReturnType<typeof setTimeout> | undefined;

  const launch = (): Promise<T> => {
    const controller = new AbortController();
    controllers.add(controller);
    return attempt(controller.signal);
  };

  const hedge = new Promise<T>((resolve, reject) => {
    hedgeTimer = setTimeout(() => launch().then(resolve, reject), hedgeDelayMs);
  });

  try {
    // Resolves on the first success; rejects only if every attempt fails.
    return await Promise.any([launch(), hedge]);
  } finally {
    if (hedgeTimer) clearTimeout(hedgeTimer);
    for (const controller of controllers) controller.abort();
  }
}

Two details make this correct. Promise.any resolves as soon as either attempt fulfills and only rejects — with an AggregateError — if both fail, so a fast primary failure does not short-circuit a hedge that would have succeeded. The finally block both clears the timer (so the hedge never launches if the primary already answered) and aborts every controller (so the losing in-flight request is cancelled rather than left running and billing you for compute).

Each attempt must honor the AbortSignal: pass it to fetch, your database driver, or the model SDK so a cancellation actually stops the work. Hedging without propagating cancellation still improves latency, but it doubles load on every hedge because the loser keeps running to completion.

Tune the Hedge Delay to Your Latency Curve

The hedge delay is the whole ballgame. Set it too low and you hedge almost every request, doubling load for little gain. Set it too high and slow requests finish before the hedge ever fires, so the tail stays fat. The sweet spot is a high percentile of your own latency distribution — commonly p95. At p95, roughly 95% of requests answer before the hedge fires and cost nothing extra, while the slowest 5% get a second chance.

That means you cannot hardcode a number; you have to measure it and let it move as your service does. A lightweight rolling window is enough to derive the delay at runtime:

class LatencyWindow {
  private readonly samples: number[] = [];

  constructor(private readonly capacity = 2048) {}

  record(ms: number): void {
    this.samples.push(ms);
    if (this.samples.length > this.capacity) this.samples.shift();
  }

  percentile(p: number): number {
    if (this.samples.length === 0) return 0;
    const sorted = [...this.samples].sort((a, b) => a - b);
    const index = Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length));
    return sorted[index];
  }
}

const window = new LatencyWindow();
// After each completed primary call: window.record(elapsedMs)
const hedgeDelayMs = Math.max(window.percentile(95), 25);

Sorting on every read is fine for illustration but wasteful at scale; a production system should use a streaming estimator such as a t-digest or an HDR histogram. Whatever you use, keep the delay adaptive per route or per dependency. A fast cache lookup and a slow analytics query have completely different curves, and a single global delay will over-hedge one while under-hedging the other. The floor (25 ms above) prevents a temporarily quiet service from setting an absurdly small delay that hedges everything.

Only Hedge Idempotent Work

Hedging duplicates a request, so it is only safe when running the request twice is indistinguishable from running it once. Reads qualify: a database SELECT, a cache fetch, a search query, an object GET, and — importantly for agents — a model inference call, which has no external side effects. Writes do not. Hedging "charge the customer" or "send the email" or "create the ticket" can execute the side effect twice, and cancelling the loser does not help, because the remote system may already have committed the change before your abort arrived.

For an AI agent, this maps cleanly onto the tool boundary. Read-only tools and the model call itself are safe to hedge; write tools are not. A small guard keeps the two paths honest:

type ToolCall = {
  name: string;
  idempotent: boolean;
  run: (signal: AbortSignal) => Promise<unknown>;
};

async function callTool(tool: ToolCall, hedgeDelayMs: number): Promise<unknown> {
  if (!tool.idempotent) {
    // A write must run exactly once; hedging could duplicate the effect.
    const controller = new AbortController();
    return tool.run(controller.signal);
  }
  return hedgedCall(tool.run, hedgeDelayMs);
}

There is a middle ground for writes that carry an idempotency key. If both the primary and the hedge send the same key, a well-behaved downstream deduplicates them and the duplicate is harmless — this is the same guarantee that protects retries. If you already run agent write tools through idempotency keys or a command ledger, you can extend hedging to them deliberately. Absent that protection, treat every non-idempotent call as un-hedgeable and leave its tail to other tactics such as timeouts and circuit breakers.

Cap the Cost with a Hedge Budget

Hedging is self-defeating if it fires too often. A service that hedges 40% of requests is doing 40% more work, and if it starts hedging because a dependency is already overloaded, those duplicates pour fuel on the fire. The guardrail is a hedge budget: allow only a small fraction of requests — often 1% to 5% — to spawn a hedge, and refuse the rest.

A token-style budget enforces the cap cheaply. Each request contributes a fraction of a token, and each hedge costs a whole token, so the long-run hedge rate can never exceed the ratio you set:

class HedgeBudget {
  private tokens = 0;

  constructor(private readonly maxHedgeRatio = 0.05) {}

  onRequest(): void {
    this.tokens = Math.min(1, this.tokens + this.maxHedgeRatio);
  }

  tryClaim(): boolean {
    if (this.tokens >= 1) {
      this.tokens -= 1;
      return true;
    }
    return false;
  }
}

Call onRequest() for every request and only hedge when tryClaim() returns true, so a burst of slow requests cannot each spawn a duplicate. Combine the budget with load awareness: skip hedging entirely when a dependency's circuit breaker is open, when its queue is deep, or when you are already shedding load. Hedging assumes spare capacity on the replicas; when there is none, the correct move is to stop hedging, not to double down. The p95-based delay and the budget work together here — a healthy service rarely trips the delay, so it rarely spends budget, and a degraded service should back off on both fronts.

Measure Whether Hedging Helps

Hedging is easy to misconfigure and invisible when it goes wrong, so instrument it from the start. Track the hedge rate (fraction of requests that spawned a duplicate), the win rate (fraction of hedges where the second request actually beat the first), and the p99 latency with hedging on versus off. These three numbers tell you almost everything.

Read them together. A hedge rate near your budget ceiling means the delay is too low or a dependency is genuinely slow — investigate before raising the budget. A high win rate means the hedge frequently wins, which is good, unless the rate is so high it suggests the delay is set below the real p95 and you are hedging healthy requests. Most importantly, if p99 barely moves with hedging enabled, the tail is not coming from stragglers you can route around — it may be a systemic bottleneck like a saturated database or a lock, and hedging is just adding load without buying latency. Alert on a hedge rate that pins to the budget and on extra load that climbs without a matching drop in p99, because both signal that hedging has stopped helping and started hurting.

Conclusion and Next Steps

Hedged requests are a precise tool for a specific problem: successful-but-slow calls whose tail you cannot design away. Send a backup after a p95-length delay, take the first success, cancel the loser — and the slowest few percent of requests stop dominating your latency and your agent run times. The correctness rules are what keep it safe: propagate cancellation, hedge only idempotent work, and cap the extra load with a budget that backs off under stress.

Start with one read-heavy dependency or one slow agent tool. Measure its latency distribution, set the hedge delay at p95, and cap hedging at a few percent. Watch p99 and the win rate for a week, tune the delay, and only then extend hedging to the next dependency. Paired with timeouts, retries, and circuit breakers, hedging rounds out a resilience toolkit that treats slowness — not just failure — as a first-class fault.

Resource Budgets for Tool-Using AI Agents

2026-07-01T09:20:00+00:00

Introduction

A tool-using AI agent is a loop: the model proposes an action, the runtime executes a tool, the result feeds back into the next decision, and the loop repeats until the agent believes it is done. That belief is the problem. A model can loop far longer than you intended, retry a failing tool forever, fan out into dozens of parallel sub-tasks, or call an expensive search endpoint on every iteration. None of that shows up as a crash. It shows up as a surprising invoice, a saturated dependency, or a run that never terminates.

Runaway resource use is a reliability failure, not just a cost problem. An agent stuck in a retry loop holds worker capacity, keeps database connections open, and delays every other run in the queue. Treating budgets as an afterthought — a dashboard you check the next morning — means the damage is already done by the time you notice.

This article shows how to give an agent an explicit resource budget and enforce it at runtime. It sits alongside patterns like durable checkpoints, command ledgers, and compensating actions: those keep an agent's side effects recoverable, while budgets keep the agent's consumption bounded. The examples use TypeScript and SQL, but the ideas apply to any agent runtime, workflow engine, or background worker.

Define the Budget as an Explicit Contract

Vague limits produce vague behavior. Before you can enforce anything, decide exactly which resources a run is allowed to consume and give each one a number. A useful agent budget bounds several dimensions at once, because any single limit can be gamed by the others. A step cap alone does not stop one step from spending a fortune; a cost cap alone does not stop a fast infinite loop.

The dimensions worth bounding for most agents:

Cost in real currency, derived from token usage and tool prices. This is the limit finance cares about.
Steps, meaning model-plus-tool iterations. This bounds loops even when each step is cheap.
Tool calls, sometimes with per-tool sub-limits, because a single expensive tool deserves a tighter ceiling.
Wall-clock time, expressed as an absolute deadline. This bounds runs that stall on slow dependencies.

type BudgetLimits = {
  maxUsd: number;
  maxSteps: number;
  maxToolCalls: number;
  deadlineAt: number; // absolute epoch milliseconds, not a duration
  perToolMaxCalls?: Record<string, number>;
};

type BudgetUsage = {
  spentUsd: number;
  reservedUsd: number;
  usedSteps: number;
  usedToolCalls: number;
  toolCalls: Record<string, number>;
};

type BudgetState = {
  runId: string;
  limits: BudgetLimits;
  usage: BudgetUsage;
  status: "active" | "exhausted" | "cancelled" | "completed";
};

Note that deadlineAt is an absolute timestamp, not a number of seconds. Durations are convenient in a request handler and dangerous in an agent: if you store "300 seconds" and the process restarts, the clock silently resets and the run gets a fresh five minutes on every crash. An absolute deadline computed once, at run creation, survives restarts and means the same thing to every worker that picks up the run.

Separate soft limits from hard limits

Not every limit should behave the same way. A hard limit is a wall: crossing it stops the run. A soft limit is a warning line: crossing it changes the run's behavior — smaller outputs, a cheaper model, no new sub-tasks — while still allowing it to finish cleanly. Most production agents want both. The hard cost cap might be one dollar, and the soft cap seventy cents, so the agent has room to wind down gracefully instead of being killed mid-thought. The next sections build both.

Meter Spend in a Durable Ledger

In-memory counters are fine until the worker restarts, and agents restart often: deploys, crashes, autoscaling, and preemption all interrupt long runs. If the budget lives only in process memory, a crashed run resumes with its spend reset to zero, which quietly defeats the entire mechanism. Store the budget where the run's durable state already lives.

create table agent_budgets (
  run_id uuid primary key,
  max_usd numeric(12, 4) not null,
  max_steps integer not null,
  max_tool_calls integer not null,
  deadline_at timestamptz not null,
  spent_usd numeric(12, 4) not null default 0,
  reserved_usd numeric(12, 4) not null default 0,
  used_steps integer not null default 0,
  used_tool_calls integer not null default 0,
  status text not null default 'active',
  updated_at timestamptz not null default now()
);

Charging the budget is a write with a guard. Do the check and the increment in one atomic statement so two concurrent tool calls cannot both squeeze past the limit. The where clause enforces the ceiling; if the update touches no row, the charge would have exceeded the budget and you must not proceed.

async function chargeCost(
  db: Database,
  runId: string,
  usd: number,
): Promise<{ ok: boolean; spentUsd?: number }> {
  const row = await db.oneOrNone(
    `
    update agent_budgets
    set spent_usd = spent_usd + $2,
        used_tool_calls = used_tool_calls + 1,
        updated_at = now()
    where run_id = $1
      and status = 'active'
      and spent_usd + reserved_usd + $2 <= max_usd
    returning spent_usd
    `,
    [runId, usd],
  );

  return row ? { ok: true, spentUsd: row.spent_usd } : { ok: false };
}

Reserve before you spend, reconcile after

There is a timing gap in token accounting: you know the input token count before a model call, but you learn the output token count only after it returns. If you charge nothing up front, a burst of concurrent calls can all pass the check and collectively blow the budget. The fix is a two-phase charge. Reserve a conservative estimate before the call, then reconcile to the actual cost afterward.

async function reserveCost(db: Database, runId: string, estimateUsd: number) {
  return db.oneOrNone(
    `
    update agent_budgets
    set reserved_usd = reserved_usd + $2, updated_at = now()
    where run_id = $1
      and status = 'active'
      and spent_usd + reserved_usd + $2 <= max_usd
    returning run_id
    `,
    [runId, estimateUsd],
  );
}

async function settleCost(
  db: Database,
  runId: string,
  estimateUsd: number,
  actualUsd: number,
) {
  await db.none(
    `
    update agent_budgets
    set reserved_usd = greatest(reserved_usd - $2, 0),
        spent_usd = spent_usd + $3,
        updated_at = now()
    where run_id = $1
    `,
    [runId, estimateUsd, actualUsd],
  );
}

Every enforcement decision then reads spent_usd + reserved_usd, which represents committed plus in-flight cost. Reservations make the budget honest under concurrency instead of only correct in a single-threaded ideal.

Enforce Budgets at Every Decision Boundary

A budget you only check at the end is a report, not a control. Enforcement has to happen at the boundaries where the agent is about to consume something: before each model call, before each tool call, and before spawning a sub-task. The guard is deliberately boring — it reads current state and throws when any limit is crossed.

type BudgetReason = "cost" | "steps" | "tool_calls" | "deadline" | "cancelled";

class BudgetExceededError extends Error {
  constructor(public readonly reason: BudgetReason) {
    super(`agent budget exceeded: ${reason}`);
    this.name = "BudgetExceededError";
  }
}

function assertCanContinue(state: BudgetState, now: number): void {
  const { limits, usage, status } = state;

  if (status === "cancelled") throw new BudgetExceededError("cancelled");
  if (now >= limits.deadlineAt) throw new BudgetExceededError("deadline");
  if (usage.usedSteps >= limits.maxSteps) throw new BudgetExceededError("steps");
  if (usage.usedToolCalls >= limits.maxToolCalls) {
    throw new BudgetExceededError("tool_calls");
  }
  if (usage.spentUsd + usage.reservedUsd >= limits.maxUsd) {
    throw new BudgetExceededError("cost");
  }
}

The agent loop calls this at the top of every iteration. Because the guard throws a typed error, the surrounding workflow can catch it, mark the run exhausted, and route to a finalization step rather than letting the exception bubble up as an unhandled crash.

Propagate the deadline into the tool call itself

An absolute deadline is only useful if it reaches the slow operation. A tool that ignores the deadline can hang for minutes past the budget while the guard waits patiently for it to return. Convert the remaining budget into a per-call timeout and pass a cancellation signal down to the tool.

async function callToolWithinBudget<T>(
  state: BudgetState,
  now: number,
  tool: (signal: AbortSignal) => Promise<T>,
): Promise<T> {
  const remainingMs = state.limits.deadlineAt - now;
  if (remainingMs <= 0) throw new BudgetExceededError("deadline");

  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), remainingMs);
  try {
    return await tool(controller.signal);
  } finally {
    clearTimeout(timer);
  }
}

The tool receives the signal and is responsible for aborting its own network request. Deadline propagation turns "the run should stop in thirty seconds" into "this HTTP call must also finish within the run's remaining time," which is what actually prevents a stall from outliving the budget.

Detect No Progress, Not Just Hard Caps

Hard caps stop a runaway agent eventually, but "eventually" can mean twenty wasted steps and several dollars spent going in circles. A better signal is progress: an agent that keeps issuing the same tool call with the same arguments, or whose working state stops changing, has stalled and will not benefit from more budget. Catch that early and stop before the hard cap.

import { createHash } from "node:crypto";

function fingerprint(toolName: string, args: unknown): string {
  // Canonicalize argument key order in production so that equivalent
  // calls hash identically and do not read as fresh progress.
  return createHash("sha1")
    .update(`${toolName}:${JSON.stringify(args)}`)
    .digest("hex");
}

class ProgressMonitor {
  private readonly recent: string[] = [];

  constructor(
    private readonly windowSize = 8,
    private readonly maxRepeats = 3,
  ) {}

  record(toolName: string, args: unknown): void {
    this.recent.push(fingerprint(toolName, args));
    if (this.recent.length > this.windowSize) this.recent.shift();
  }

  isStalled(): boolean {
    const counts = new Map<string, number>();
    for (const key of this.recent) {
      const next = (counts.get(key) ?? 0) + 1;
      if (next >= this.maxRepeats) return true;
      counts.set(key, next);
    }
    return false;
  }
}

The monitor keeps a small sliding window of recent tool fingerprints and flags a stall when the same call repeats too often. You can extend the same idea to the agent's working memory: hash the relevant state after each step, and if the fingerprint has not changed across several steps, the agent is spinning without making progress. A stall should end the run through the same finalization path as a hard cap, with a distinct reason so your dashboards can tell "ran out of budget" apart from "got stuck."

Degrade Gracefully Before You Hit the Wall

Hitting a hard limit mid-step is the worst possible moment to stop, because the agent has no chance to summarize what it found or leave a clean handoff. Soft limits let the run wind down on its own terms. As the budget tightens, change behavior instead of continuing at full throttle: prefer a cheaper model, cap output length, stop opening new sub-tasks, and steer the agent toward finishing rather than exploring.

type ModelChoice = { model: string; maxOutputTokens: number; allowFanOut: boolean };

function chooseModel(state: BudgetState, now: number): ModelChoice {
  const remainingUsd =
    state.limits.maxUsd - state.usage.spentUsd - state.usage.reservedUsd;
  const remainingMs = state.limits.deadlineAt - now;
  const pressured = remainingUsd < state.limits.maxUsd * 0.3 || remainingMs < 20_000;

  if (pressured) {
    return {
      model: "claude-haiku-4-5-20251001",
      maxOutputTokens: 512,
      allowFanOut: false,
    };
  }

  return { model: "claude-opus-4-8", maxOutputTokens: 2048, allowFanOut: true };
}

When even the soft budget is gone, force a finalization turn: tell the model it has no remaining budget and must return its best answer from what it already knows, with tools disabled. A truthful partial answer plus a note about what was skipped is far more useful than a hard abort that discards the work in progress.

Give operators a kill switch

Automated limits handle the expected cases; a human needs a manual override for the unexpected one. A cancel control that flips a run's status to cancelled is enough, because the boundary guard already checks status on every iteration and will stop at the next step. Support cancellation at several scopes — a single run, a tenant, or a global flag that pauses all agents during an incident.

async function cancelRun(db: Database, runId: string, reason: string) {
  await db.none(
    `update agent_budgets set status = 'cancelled', updated_at = now()
     where run_id = $1 and status = 'active'`,
    [runId],
  );
  await recordAudit(db, runId, "run_cancelled", reason);
}

Cancellation is cooperative, not a forced kill: the run stops cleanly at its next boundary instead of being torn down mid-side-effect, which keeps it compatible with durable checkpoints and compensation logic. Pair it with observability so budgets are visible, not just enforced. Emit metrics for spend, steps, and time as a fraction of each run's limits, and alert on runs that terminate by exhaustion, stall out, or repeatedly brush the soft cap — those are the ones worth investigating.

Test Budget Enforcement Before Incidents

Budget logic is exactly the kind of code that is never exercised in a happy-path demo and is load-bearing during an incident. Test it directly, with adversarial agents designed to overspend. A deterministic test needs an injected clock and stubbed tools so you can simulate loops, slow calls, and cost spikes without real model traffic.

import { describe, it, expect } from "vitest";

describe("agent budget enforcement", () => {
  it("refuses to continue once the deadline passes", () => {
    const state = makeState({ deadlineAt: 1_000 });
    expect(() => assertCanContinue(state, 1_500)).toThrow(BudgetExceededError);
  });

  it("flags a stall before the hard step cap is reached", () => {
    const monitor = new ProgressMonitor(8, 3);
    for (let i = 0; i < 5; i += 1) {
      monitor.record("web_search", { query: "same query" });
    }
    expect(monitor.isStalled()).toBe(true);
  });

  it("rejects a charge that would exceed the cost ceiling", () => {
    const state = makeState({ maxUsd: 1, spentUsd: 0.95 });
    expect(() => assertCanContinue(state, 0)).not.toThrow();
    state.usage.reservedUsd = 0.1;
    expect(() => assertCanContinue(state, 0)).toThrow(BudgetExceededError);
  });
});

Cover the paths that matter under failure: a tool that always returns the same result until the monitor stops it, a tool that never resolves until the deadline aborts it, and a sequence of expensive calls that trips the cost ceiling. Assert not only that the run stops, but that it stops for the right reason and reaches finalization instead of throwing an unhandled error. Those assertions are what let you trust an agent to run unattended.

Conclusion and Next Steps

An agent without a budget is a loop that trusts a language model to know when to stop. A budget turns that open-ended loop into a bounded, observable run: explicit limits on cost, steps, tool calls, and time; a durable meter that survives restarts; enforcement at every decision boundary; progress checks that catch stalls early; and graceful degradation that lets a run finish instead of crashing into a wall.

Start small. Add the durable budget table and a hard cost-and-step guard to one agent, and confirm it terminates a deliberately looping run in a test. Then layer in reservations for honest concurrent accounting, deadline propagation into your slowest tool, and a soft limit that downshifts the model before the hard cap. Once the boring path is in place, every new tool you expose to the agent inherits the same guardrails — and unattended autonomy stops being a gamble.

Compensating Actions for Tool-Using AI Agents

2026-06-19T21:20:15+00:00

Introduction

Tool-using AI agents are most valuable when they can do real work: open pull requests, update tickets, send messages, change configuration, start deployments, or call internal APIs. Those actions are also where reliability stops being a prompt-engineering problem and becomes a distributed-systems problem. A model can choose a sensible action, the tool can partly succeed, and the workflow can still end in a state that nobody intended.

Retries and idempotency keys reduce duplicate writes, but they do not answer every recovery question. What should happen if an agent created the wrong ticket, moved a customer to the wrong plan, posted a message to the wrong channel, or started a deployment that later failed validation? Some outcomes need a compensating action: a deliberate follow-up step that repairs, reverses, neutralizes, or clearly escalates the side effect.

This article shows how to design compensating actions for tool-using AI agents. The examples use TypeScript and SQL, but the pattern applies to workflow engines, queues, background workers, serverless jobs, and long-running agent runtimes.

Treat Every Write Tool as a Side-Effect Contract

A compensating action is not an afterthought bolted onto a failed workflow. It starts with the contract for each write tool. Before an agent can call a tool, the runtime should know what the tool changes, whether the change can be reversed, how to prove the outcome, and who must approve a repair.

Useful tool metadata includes:

effect: the business effect of the tool, not just the function name.
impact: whether the action is low-risk, customer-visible, financial, destructive, or security-sensitive.
idempotency: how repeated execution is deduplicated.
reconcile: how to prove whether the side effect happened.
compensate: what action can repair or neutralize the effect.
approval: whether the forward action or compensation needs human approval.

Here is a compact TypeScript shape:

type ToolImpact = "read" | "write" | "customer_visible" | "destructive";

type CompensationMode =
  | "automatic"
  | "approval_required"
  | "manual_runbook"
  | "not_supported";

type WriteToolContract<TArgs, TResult> = {
  name: string;
  impact: ToolImpact;
  validate(args: unknown): TArgs;
  execute(args: TArgs, context: ToolContext): Promise<TResult>;
  reconcile(effect: StoredEffect): Promise<ReconciliationResult>;
  compensation: {
    mode: CompensationMode;
    reason: string;
    buildArgs?: (effect: StoredEffect) => unknown;
    execute?: (args: unknown, context: ToolContext) => Promise<unknown>;
  };
};

This contract forces design decisions into code review. If a tool says not_supported, that is acceptable only when the team has consciously accepted the operational cost. For example, sending an email cannot be undone. The compensation may be a follow-up correction message, a support task, and an audit note rather than a literal rollback.

Prefer forward recovery over fake rollback

Distributed systems rarely roll back perfectly. A payment authorization, user notification, deployment, or third-party API update may have already escaped your boundary. In those cases, the better compensation is forward recovery: create a correcting event, move the workflow to a safe state, and make the user-visible outcome explicit.

For an AI agent, this distinction matters because the model may describe a compensation as "undo that change" even when undo is impossible. The runtime should expose precise actions such as closeTicket, restorePreviousConfig, postCorrection, revertPullRequest, or requestHumanReview. The model can propose intent, but the workflow chooses from concrete recovery tools.

Store Forward Effects and Compensation State

Compensations need durable facts. A chat transcript may explain why the agent acted, but recovery code needs structured records: what tool ran, which arguments were used, what external resource changed, whether the outcome is proven, and which compensation has already been attempted.

A relational schema can separate the forward effect from the compensation lifecycle:

create type agent_effect_status as enum (
  'planned',
  'executing',
  'succeeded',
  'failed',
  'uncertain',
  'compensation_pending',
  'compensating',
  'compensated',
  'manual_review'
);

create table agent_effects (
  id uuid primary key,
  run_id uuid not null,
  step_id text not null,
  tool_name text not null,
  effect_key text not null,
  idempotency_key text not null,
  arguments jsonb not null,
  result jsonb,
  external_ref text,
  status agent_effect_status not null,
  compensation_mode text not null,
  compensation_args jsonb,
  compensation_attempts integer not null default 0,
  last_error text,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (run_id, effect_key)
);

create index agent_effects_recovery_idx
  on agent_effects (status, updated_at);

The effect_key is the logical identity of the side effect within one run. It should be stable across retries and worker restarts. A common pattern is to combine the step id, tool name, target resource, and a stable hash of validated arguments. If the agent re-plans the same action after a crash, the unique constraint returns the existing effect record instead of creating another one.

Record enough context to compensate safely

The compensation should not depend on a model remembering what happened. Store the fields needed to repair the effect before or immediately after executing the forward action.

For example, a configuration update tool should record the previous value, new value, config version, and resource id. A ticket update tool should record the prior status and comment id. A pull request tool should record the branch, commit, PR number, and repository. If you discover during recovery that the saved context is incomplete, route the effect to manual_review instead of asking the model to guess.

Decide When to Compensate

Not every failure should trigger compensation. If the forward action never ran, there is nothing to undo. If the outcome is uncertain, the system should reconcile first. If a compensation could make the situation worse, the right move is operator review.

Use a small decision table:

Forward outcome	Example	Recovery action
Not started	Worker crashed before tool call	Retry forward action
Succeeded but later invalid	Deployment succeeded, smoke test failed	Run approved compensation
Uncertain	API timeout after request was sent	Reconcile before retry or compensate
Irreversible	Email sent to user	Create correction task or follow-up message
High impact	Billing, security, deletion	Require human approval

The workflow should apply that table deterministically. The model may help summarize context for a human, but it should not be the only component deciding whether to reverse a customer-visible change.

Here is one way to encode the decision:

type RecoveryDecision =
  | { action: "retry_forward" }
  | { action: "reconcile_first" }
  | { action: "compensate_now" }
  | { action: "request_approval"; reason: string }
  | { action: "manual_review"; reason: string };

function chooseRecovery(effect: StoredEffect): RecoveryDecision {
  if (effect.status === "planned" || effect.status === "failed") {
    return { action: "retry_forward" };
  }

  if (effect.status === "uncertain") {
    return { action: "reconcile_first" };
  }

  if (effect.compensationMode === "not_supported") {
    return {
      action: "manual_review",
      reason: "tool_has_no_automatic_compensation",
    };
  }

  if (effect.compensationMode === "approval_required") {
    return {
      action: "request_approval",
      reason: "compensation_requires_human_approval",
    };
  }

  if (effect.status === "compensation_pending") {
    return { action: "compensate_now" };
  }

  return {
    action: "manual_review",
    reason: `unsupported_recovery_state:${effect.status}`,
  };
}

This function is intentionally boring. Recovery logic should be predictable, testable, and visible in logs. The agent can still explain why a compensation is needed, but the runtime should enforce the rules.

Execute Compensations with the Same Discipline as Forward Actions

A compensation is also a side effect. It needs idempotency, policy checks, leases, audit events, and observability. Otherwise the repair path becomes another source of production incidents.

The compensation worker should:

Acquire one eligible effect with a lease.
Reconcile uncertain outcomes before changing anything.
Re-check policy and approval state.
Build compensation arguments from stored facts.
Execute with a compensation idempotency key.
Mark the effect as compensated, manual review, or failed with a clear reason.

async function compensateEffect(effect: StoredEffect, registry: ToolRegistry) {
  const contract = registry.getWriteTool(effect.toolName);
  const decision = chooseRecovery(effect);

  if (decision.action === "reconcile_first") {
    const result = await contract.reconcile(effect);
    await storeReconciliation(effect.id, result);
    return;
  }

  if (decision.action === "request_approval") {
    await requestCompensationApproval(effect.id, decision.reason);
    return;
  }

  if (decision.action !== "compensate_now") {
    await markManualReview(effect.id, decision.reason ?? decision.action);
    return;
  }

  if (!contract.compensation.execute || !contract.compensation.buildArgs) {
    await markManualReview(effect.id, "missing_compensation_implementation");
    return;
  }

  const args = contract.compensation.buildArgs(effect);
  const idempotencyKey = `${effect.id}:compensation`;

  await contract.compensation.execute(args, {
    runId: effect.runId,
    idempotencyKey,
    reason: "agent_effect_compensation",
  });

  await markCompensated(effect.id, { idempotencyKey });
}

The idempotency key for compensation should be different from the forward action key. Retrying the repair should deduplicate with previous repair attempts, not with the original action.

Do not compensate stale state blindly

Before a compensation changes an external system, check whether the resource has changed since the agent touched it. If a human already fixed the ticket, a deploy pipeline already rolled back, or a newer workflow updated the same config value, automatic compensation may overwrite good work.

Common stale-state checks include:

Version numbers or ETags.
Last-modified timestamps.
Current owner or status fields.
The exact comment, branch, deployment, or message id created by the agent.
A compare-and-swap update when the external API supports it.

If the current state no longer matches the expected post-action state, stop and create a manual review item. "Do no further damage" is more important than making every recovery path automatic.

Make Operators Part of the Recovery Design

Some compensations should be automatic. Many should not. Human review is not a failure of agent design; it is the right boundary for actions with ambiguous state, irreversible consequences, or customer impact.

An operator-ready review item should include:

The original objective and run id.
The model decision that proposed the action.
The validated tool arguments.
The external resource that changed.
The reconciliation result.
The proposed compensation and its risk.
The reason approval is required.
Links to logs, traces, and audit records.

Avoid presenting only a raw transcript. Transcripts are useful context, but operators need structured facts and a small set of allowed actions. A good review screen offers commands such as "approve compensation", "mark already repaired", "retry reconciliation", or "escalate to incident".

Measure compensation quality

Compensation paths should have their own metrics:

Effects by status and tool.
Time from failed validation to compensation completion.
Automatic versus manual compensation rate.
Compensation failures by reason.
Stale-state conflicts.
Irreversible effects created by agents.
Human approval latency.

These metrics show where the system is too risky, too manual, or missing enough context to recover. If a particular tool often lands in manual review because it cannot prove outcomes, the next engineering task is not a better prompt. It is a better reconciliation endpoint, resource identifier, or side-effect contract.

Test Compensation Paths Before Production Incidents

Compensation code often sits idle until something has already gone wrong. That makes it easy for schemas, permissions, or third-party APIs to drift. Treat compensations as first-class behavior and test them alongside forward actions.

Start with unit tests for recovery decisions:

import assert from "node:assert/strict";

const effect = {
  status: "compensation_pending",
  compensationMode: "approval_required",
} as StoredEffect;

assert.deepEqual(chooseRecovery(effect), {
  action: "request_approval",
  reason: "compensation_requires_human_approval",
});

Then add integration tests around representative tools:

Forward action succeeds, validation fails, compensation succeeds.
Forward action times out, reconciliation proves success, compensation is requested.
Compensation retry uses the same compensation idempotency key.
Resource state changed externally, so the effect moves to manual review.
Tool has no automatic compensation, so the operator item has enough context.

For agent workflows, also test that model retries cannot create multiple compensation records for the same effect. The runtime should own that invariant through database constraints and idempotency keys.

Conclusion and Next Steps

Compensating actions make tool-using AI agents safer because they acknowledge a basic truth: side effects do not always line up with workflow state. A useful agent runtime records what it intended to change, proves what actually changed, and has a deliberate path for repairing, reversing, or escalating the result.

For your next write-capable agent tool, define the side-effect contract before exposing it to the model. Store the forward effect, give it a reconciliation strategy, decide whether compensation is automatic or approval-gated, and test the repair path under failure. That work is less glamorous than a smarter prompt, but it is what turns agent autonomy into an operationally survivable system.

Agent Command Ledgers for Reliable AI Workflows

2026-06-13T21:03:03+00:00

Introduction

AI agents become operationally interesting when they stop at nothing more than drafting text and start changing state. They can create tickets, open pull requests, update configuration, send messages, call billing APIs, rotate incidents, or trigger deployments. Those actions are useful, but they also create a reliability problem: the agent may decide correctly, the worker may crash halfway through execution, and the external system may still apply the change.

A chat transcript is not enough to recover from that failure. It records what the model said, not necessarily what the runtime authorized, reserved, executed, retried, or reconciled. Production agent systems need a separate command ledger: an append-friendly record of every intended side effect and every state transition around it.

This article shows how to design an agent command ledger for reliable AI workflows. The examples use TypeScript and SQL, but the pattern applies to workflow engines, queues, serverless jobs, background workers, and long-running agent runtimes.

Separate Reasoning Events from Commands

An agent workflow usually has many internal events: prompts, model responses, tool suggestions, memory reads, policy decisions, human approvals, and status messages. Do not treat all of those as executable work. A command is narrower. It is a durable request to perform one side effect against one target with one set of validated arguments.

A good command record answers:

What action did the agent intend to perform?
Which workflow run and step produced the command?
Which policy version authorized or denied it?
Which human approval, if any, allowed it to continue?
Which external idempotency key protects the action?
What external identifier came back after execution?
Is the outcome complete, failed, uncertain, or awaiting reconciliation?

That separation keeps the model transcript useful for explanation while making the command ledger useful for recovery. You can replay reasoning if you need context, but you should replay commands only through explicit recovery tooling.

Here is a compact relational schema for command records:

create type agent_command_status as enum (
  'pending',
  'blocked',
  'approved',
  'leased',
  'succeeded',
  'failed',
  'uncertain',
  'cancelled'
);

create table agent_commands (
  id uuid primary key,
  run_id uuid not null,
  step_id text not null,
  command_key text not null,
  tool_name text not null,
  target text not null,
  arguments jsonb not null,
  status agent_command_status not null default 'pending',
  policy_version text not null,
  approval_id uuid,
  idempotency_key text not null,
  external_id text,
  leased_by text,
  lease_expires_at timestamptz,
  attempt_count integer not null default 0,
  last_error text,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (run_id, command_key)
);

create index agent_commands_recovery_idx
  on agent_commands (status, lease_expires_at, updated_at);

The command_key is the logical identity of the intended side effect inside one run. It might include the step id, tool name, target resource, and a stable hash of the validated arguments. If the worker restarts and the agent reaches the same step again, the unique constraint prevents a second command from being created for the same logical action.

Model output is not a command contract

The model can propose sendCustomerEmail or createPullRequest, but the runtime should convert that proposal into a command only after validation. The command should contain normalized arguments, not raw model text. For example, a recipient should be a verified user id or email address from your database, not an untrusted string copied from a prompt.

This gives you a clean boundary: model output is advisory, command records are operational.

Reserve Commands Before Executing Side Effects

The most important rule is simple: record intent before the side effect. If you call the external API first and write the ledger later, a crash can leave you with an applied action and no durable record. The recovery worker will not know whether to retry, reconcile, or alert a human.

Reserve the command in a transaction, then let a worker execute reserved commands:

import { createHash, randomUUID } from "node:crypto";

type CommandInput = {
  runId: string;
  stepId: string;
  toolName: string;
  target: string;
  arguments: Record<string, unknown>;
  policyVersion: string;
};

function stableStringify(value: unknown): string {
  if (Array.isArray(value)) {
    return `[${value.map(stableStringify).join(",")}]`;
  }

  if (value && typeof value === "object") {
    return `{${Object.entries(value)
      .sort(([left], [right]) => left.localeCompare(right))
      .map(([key, nested]) => `${JSON.stringify(key)}:${stableStringify(nested)}`)
      .join(",")}}`;
  }

  return JSON.stringify(value);
}

function stableHash(value: unknown): string {
  return createHash("sha256")
    .update(stableStringify(value))
    .digest("hex")
    .slice(0, 24);
}

function buildCommandKey(input: CommandInput): string {
  const argHash = stableHash(input.arguments);
  return `${input.stepId}:${input.toolName}:${input.target}:${argHash}`;
}

async function reserveCommand(db: Database, input: CommandInput) {
  const commandKey = buildCommandKey(input);
  const idempotencyKey = `${input.runId}:${commandKey}`;

  return db.one(
    `
    insert into agent_commands (
      id, run_id, step_id, command_key, tool_name, target,
      arguments, policy_version, idempotency_key
    )
    values ($1, $2, $3, $4, $5, $6, $7, $8, $9)
    on conflict (run_id, command_key)
    do update set updated_at = agent_commands.updated_at
    returning *
    `,
    [
      randomUUID(),
      input.runId,
      input.stepId,
      commandKey,
      input.toolName,
      input.target,
      input.arguments,
      input.policyVersion,
      idempotencyKey,
    ]
  );
}

The on conflict clause is intentional. A repeated planning step returns the existing command instead of creating a duplicate. The worker can then inspect the command status and decide whether it already succeeded, still needs approval, or is eligible for recovery.

Idempotency is necessary but not sufficient

An external idempotency key protects the remote system from duplicate writes, but it does not explain your local workflow state. The ledger still needs status, attempts, leases, policy decisions, and reconciliation results. Treat the external key as one field in the command contract, not as the entire reliability design.

For tools that do not support idempotency keys, the command ledger becomes even more important. You may need to query by natural key before retrying, restrict retries to read-after-write reconciliation, or route ambiguous outcomes to a human operator.

Fence Execution with Policy and Leases

Commands should not execute just because they exist. A worker should acquire a lease, re-check policy, verify approvals, and only then call the external tool. This prevents two workers from racing, and it protects workflows whose permissions changed after the command was reserved.

One practical flow is:

Select a pending or approved command whose lease is empty or expired.
Atomically move it to leased with a short lease timeout.
Re-run deterministic policy checks using the stored arguments.
Execute the tool with the stored idempotency key.
Store the external identifier and final status.

type WorkerContext = {
  workerId: string;
  now: Date;
};

async function acquireCommand(db: Database, context: WorkerContext) {
  return db.oneOrNone(
    `
    update agent_commands
    set
      status = 'leased',
      leased_by = $1,
      lease_expires_at = $2,
      attempt_count = attempt_count + 1,
      updated_at = now()
    where id = (
      select id
      from agent_commands
      where status in ('pending', 'approved', 'uncertain')
        and (lease_expires_at is null or lease_expires_at < now())
      order by created_at
      for update skip locked
      limit 1
    )
    returning *
    `,
    [context.workerId, new Date(context.now.getTime() + 60_000)]
  );
}

async function executeLeasedCommand(command: AgentCommand) {
  const decision = await authorizeStoredCommand(command);
  if (!decision.ok) {
    await markBlocked(command.id, decision.reason);
    return;
  }

  if (decision.approvalRequired && !command.approval_id) {
    await markBlocked(command.id, "approval_required");
    return;
  }

  await toolRegistry.call(command.tool_name, {
    target: command.target,
    arguments: command.arguments,
    idempotencyKey: command.idempotency_key,
  });
}

The worker does not ask the model whether the command is still allowed. It asks deterministic application code. That code can evaluate tenant policy, user permissions, workflow state, approval records, maintenance windows, budget limits, and tool-specific rules.

Use short leases and explicit terminal states

Leases should be short enough that crashed workers do not block recovery for long. Terminal states should be explicit: succeeded, failed, cancelled, and sometimes blocked. Avoid a vague done value that hides whether the command completed externally, failed locally, or was intentionally stopped.

The uncertain state deserves special attention. It means the worker cannot prove whether the side effect happened. That can occur after a timeout, dropped connection, process crash, or partial response from an external API. Uncertainty should trigger reconciliation, not blind retry.

Reconcile Ambiguous Outcomes

Reliable agent systems assume that some outcomes will be ambiguous. The runtime may lose the response after the remote system accepts the request. The external API may return a timeout while still completing the action. A deployment tool may start a rollout and fail before returning the deployment id.

The command ledger should make ambiguity visible and recoverable:

async function executeWithUncertainty(command: AgentCommand) {
  try {
    const result = await toolRegistry.call(command.tool_name, {
      target: command.target,
      arguments: command.arguments,
      idempotencyKey: command.idempotency_key,
    });

    await markSucceeded(command.id, {
      externalId: result.externalId,
      response: result.safeResponse,
    });
  } catch (error) {
    if (isAmbiguousTransportError(error)) {
      await markUncertain(command.id, describeError(error));
      return;
    }

    await markFailed(command.id, describeError(error));
  }
}

async function reconcileCommand(command: AgentCommand) {
  const evidence = await toolRegistry.lookup(command.tool_name, {
    target: command.target,
    idempotencyKey: command.idempotency_key,
    arguments: command.arguments,
  });

  if (evidence.found) {
    await markSucceeded(command.id, {
      externalId: evidence.externalId,
      response: evidence.safeResponse,
    });
    return;
  }

  if (command.attempt_count >= 3) {
    await requestHumanReview(command.id, "external_outcome_not_found");
    return;
  }

  await releaseForRetry(command.id);
}

The reconciliation path needs tool-specific lookup logic. A pull request tool can search by branch name, title marker, or idempotency metadata. A ticket tool can search by a hidden correlation field. A deployment tool can query recent deployments for the commit and environment. A payment or billing tool may require stricter support from the provider before retrying safely.

Prefer evidence over optimism

Do not mark a command as successful because the agent is confident. Mark it successful because the external system returned an identifier, a reconciliation lookup found matching evidence, or a human operator recorded a verified outcome.

The ledger can still preserve the model's reasoning, but it should not let reasoning replace operational proof.

Make Replay a Controlled Operation

Replay is not the same thing as re-running the prompt. A replay tool should operate on command records, not on free-form model output. It should show the original intent, current status, policy decision, approval status, attempts, errors, and external evidence before allowing another execution attempt.

For low-risk commands, replay can be automatic after reconciliation proves the side effect did not happen. For high-risk commands, replay should require review. Examples include sending customer emails, modifying access controls, charging money, deleting data, or merging code.

Useful replay controls include:

A dry-run mode that rechecks policy and validates arguments without calling the tool.
A maximum attempt count per command and per workflow run.
A reason field for every manual retry, cancellation, or override.
Rate limits so recovery does not overload a dependency.
Separate permissions for viewing commands and replaying commands.
Alerts when commands stay uncertain, blocked, or leased too long.

Observability should treat command health as a first-class signal. Track command counts by status, tool, tenant, workflow type, attempt count, and age. Alert on growing uncertain counts, expired leases, repeated policy denials, and replay failure rates. Those signals tell you whether the agent runtime is actually durable or merely optimistic.

Conclusion and Next Steps

AI agents that change state need more than prompts, logs, and retries. They need a durable command ledger that records intent before execution, fences work with policy and leases, preserves idempotency keys, and reconciles uncertain outcomes before retrying.

Start with one risky write tool. Add a command table, generate a stable command key, reserve commands before execution, and store the external identifier on success. Then add lease recovery, reconciliation lookup, and a small replay console. Once that path is boring, extend the same ledger pattern to the rest of your agent tools.