One agent request branches into two completed purchases after the first response is lost.

Scenario: the purchase order that disappeared

Hypothetical scenario. A finance employee asks an agent to create a purchase order for a supplier and an approved total of $18,400. The agent calls the procurement API. The service validates the request, commits the order, and begins returning PO-78421. Before that response reaches the agent, a gateway times out. The orchestration layer records a tool error. On resume, the model sees no order number and asks the tool to create the order again. The second request succeeds as PO-78422.

From the agent’s point of view, the first attempt failed. From the business system’s point of view, it succeeded. This is not an unusual contradiction in a distributed system. A timeout says that the caller did not receive a result within a time limit; it does not establish that the operation did not commit.

The recovery problem is therefore not “how many times should the agent retry?” It is “how can the workflow preserve one intended business effect while the outcome of an earlier attempt is uncertain?”

Three outcomes that must not be collapsed

A tool call can end in at least three materially different states:

  1. Failed before commit. Validation or authorization rejected the operation, or the request never reached the service. Retrying the same intent may be safe after the cause is addressed.
  2. Completed, response lost. The service committed the action, but the caller did not receive or persist the result. Repeating a non-idempotent command may create a second effect.
  3. Outcome unknown. The caller cannot prove either of the above. The safe next step is usually to query or reconcile the operation, not to invent a new identity and issue another command.

The distinction matters because retries can originate at several layers. An agent graph may retry a node; a tool wrapper may retry an HTTP request; an SDK may retry a transient response; a queue consumer may redeliver a message; and the downstream API may itself repeat work. A single “retry count” at the orchestration layer does not necessarily count all attempts that can reach the business system.

AWS’s Builders’ Library describes this network ambiguity directly: callers need a way to make retries safe when they cannot tell whether the prior request succeeded. Its idempotent API pattern uses a caller-provided request token so the service can recognize repeated requests and return the earlier result, while also distinguishing a retry from a new request with similar parameters. Making retries safe with idempotent APIs

Idempotency, deduplication, and reconciliation

These terms address related but different controls.

Idempotency is a property of an operation under repeated execution: repeating the same logical request produces the same business effect as executing it once. For a purchase-order creation endpoint, an idempotency key can identify one intended create operation. When a duplicate request arrives with the same key and equivalent payload, the service returns the recorded outcome instead of creating another order.

Deduplication is the mechanism that recognizes and suppresses repeated delivery. A database uniqueness constraint, an inbox table, or a processed-message record can implement a piece of this behavior. Deduplication usually has a bounded scope and retention window; it is not automatically an end-to-end guarantee.

Reconciliation resolves uncertainty by comparing workflow intent with authoritative business state. Given an operation ID, the caller may ask whether the purchase order exists, whether it is pending, or whether the original request was rejected. Reconciliation is valuable when the original result is unavailable, when an idempotency window expired, or when a downstream system lacks a safe retry contract.

None of these substitutes for authorization. An idempotency key can prevent duplicate creation of a forbidden order just as effectively as it can prevent a permitted one. The service still needs to check who is acting, which tenant and supplier are in scope, what amount is allowed, and whether approval is valid for the exact transaction.

A stable identity for the intended operation

A robust workflow creates an operation identity when the business intent is accepted, before calling the external service. For example, it can derive or allocate an opaque operation ID associated with the initiating request, then carry that identity unchanged through orchestration, tool adapters, queues, and service calls. A retry is another attempt to complete that operation; it is not a new operation.

The identity should be bound to the correct context. A safe idempotency record commonly includes the authenticated caller or delegated principal, tenant, operation type, target resource, and a normalized payload fingerprint. The service can then reject a request that reuses a key with a materially different payload instead of silently treating it as the original request. A key scoped only by globally predictable text such as “create-order” risks collisions or cross-tenant confusion. A key scoped only by the agent’s transient run ID may change when work resumes in a new process.

Generating a fresh key on every retry defeats the mechanism. If attempt one uses k-101 and attempt two uses k-102, the service sees two independent create requests and is entitled to create two orders. Conversely, reusing a key forever can also be unsafe if a genuine later purchase is accidentally attached to an old operation. The workflow needs a clear boundary: one user-approved intent receives one stable operation ID, and a genuinely new intent receives another.

Identical parameters are not a sufficient substitute for that identity. A buyer may intentionally create two orders with the same supplier, items, and amount on separate days. Conversely, a user may revise a quantity or delivery date while still talking about the same order draft. The business meaning comes from intent and workflow state, not a blind comparison of JSON bodies. Treat payload matching as a consistency check on a known operation, not as the definition of the operation itself.

Timeline: unsafe replay and controlled recovery

UNSAFE
10:02:00  Agent sends create(order, key=A)
10:02:01  Procurement commits PO-78421
10:02:06  Response is lost; orchestration records timeout
10:02:08  Resume creates a new key B and sends the same payload
10:02:09  Procurement commits PO-78422

CONTROLLED
10:02:00  Workflow records operation O-55; sends create(order, key=O-55)
10:02:01  Procurement commits PO-78421 and stores O-55 → PO-78421
10:02:06  Response is lost; caller records outcome unknown
10:02:08  Resume queries O-55, or repeats key O-55 with the same payload
10:02:09  Service returns PO-78421; no second order is created

The controlled path depends on service support. If the API does not retain idempotency records, the caller needs a reconciliation endpoint or a separate business constraint that prevents duplicates. A local “sent” flag is not enough: the process can crash after the remote commit but before the flag is durably written.

Retry outcome depends on stable operation identityThe unsafe path creates key A, loses the response after commit, then generates key B and creates a duplicate. The controlled path reuses operation O-55 and receives the original order result.UNSAFEkey A → commitPO-78421response lostoutcome unknownnew key Bsecond createduplicatePO-78422CONTROLLEDoperation O-55stable across attemptsreconcile or replaysame key + same intentoriginal resultone order · PO-78421
Source: author-created hypothetical timeline based on the AWS idempotency pattern and LangGraph resume semantics.

The local checkpoint is not the external transaction

Agent frameworks can persist graph state and resume interrupted work, but that persistence does not create an atomic transaction with a procurement system. LangGraph documents that an interrupted node can run again when execution resumes; side effects before the interrupt must therefore be designed to tolerate replay. Durable checkpoints preserve the workflow cursor, not the exactly-once effect of a separate API call. LangGraph interrupts

A useful state machine makes uncertainty explicit:

Prepared → Submitted → Confirmed
              ↓             ↑
          Outcome unknown → Reconcile
              ↓
           Rejected

Persist the stable operation ID and the request intent before submission. After a response, persist the downstream identifier and result. If the process fails between remote commit and local persistence, resume by querying with the operation ID or replaying the same key. Avoid placing an irreversible call before a checkpoint and then assuming that a resumed node will know it already happened.

The same issue appears with queues. “At least once” delivery means a consumer may receive a message again after it has performed a side effect but before acknowledging the message. A transactional inbox can record message processing alongside a local database update. If the side effect occurs in an external procurement service, the consumer still needs a stable downstream idempotency key or a reconciliation path. Exactly-once delivery claims at one layer do not automatically mean exactly-once business effects across several systems.

Key scope, concurrency, and time

Two workers can submit the same operation concurrently before either observes a result. The idempotency store must perform an atomic claim on the operation key, not a check-then-write sequence that allows both attempts to proceed. A typical design stores a state such as in_progress, the payload fingerprint, and eventually the canonical response. A concurrent duplicate can wait, receive a conflict/retryable status, or be given the existing result, depending on the API contract.

A duplicate key with a different payload should fail explicitly. Do not return the first response as though the altered request had succeeded; that can mislead the agent and the human reviewing the result. Record the mismatch and require a new authorized operation identity if the user intended a revised purchase.

Retention matters. Services often keep idempotency records for a finite period. For example, Stripe documents that API v1 keys can be pruned after they are at least 24 hours old; reusing a pruned key can start a new request, and parameter mismatches are rejected while the key is retained. That is a provider-specific contract, not a universal retention rule. If an agent retries after a service’s retention period, it may treat the old key as new. Stripe idempotent requests. The caller must know the window, cap automatic retry duration accordingly, and use reconciliation or manual review after expiry. Delayed queue messages also matter: an old message arriving after a record has expired can recreate the effect unless the business domain has a longer-lived uniqueness rule.

An idempotency record is also not a universal cache. If an operation includes a changing price quote, exchange rate, inventory reservation, or policy decision, returning an old response may be correct for the same intent—or harmful if the business meaning has expired. Bind the operation to relevant versions or state and define whether a retry returns the original result, rejects as stale, or requires reauthorization.

Approval must attach to the operation

A human approval for “create a purchase order” should identify the order being approved, its supplier, amount, currency, cost center, and relevant terms. If the payload changes after approval, the system should invalidate or renew that approval according to policy. Otherwise an agent may obtain approval for one request and use the same workflow token to execute another.

Authorization should be checked at the action boundary, ideally by the service that creates the order or a policy layer that the service cannot bypass. Check the requester, delegated agent identity, tenant, allowed amount, supplier, approval binding, and operation status. This control remains necessary on every execution attempt. A retry with the same key may be eligible to retrieve an existing result, but it should not become an opportunity to use stale credentials or bypass a revoked authorization. Design carefully whether authorization is reevaluated before returning an earlier result and what evidence is recorded.

Compensation can help when a duplicate or partial action must be reversed, but it is not equivalent to undo. Cancelling a purchase order may trigger notifications, reservations, accounting entries, or supplier work that cannot be erased. Some actions—sending a disclosure, issuing a payment, or changing an external record—may be irreversible. Prevention and reconciliation should be primary; compensation is a recovery option with its own authority and audit trail.

Test matrix for recovery behavior

Scenario Setup Expected observable outcome
Timeout after commit Let the API commit, then drop the response A retry with the same operation ID returns the original order ID; exactly one order exists
Concurrent retries Submit the same operation ID from two workers at once One canonical order is committed; the other attempt waits, conflicts, or receives the canonical result
Resume after interruption Interrupt after the remote commit and before local result persistence Resume reconciles or repeats the same key and records the original order; no second order appears
Changed parameters Reuse an existing key with a changed amount or supplier The API rejects the mismatch and records it; no silent substitution or second order occurs
Expired idempotency record Deliver a delayed duplicate beyond the documented retention window The workflow refuses automatic create and uses authoritative reconciliation or review
Authorization revoked Revoke the principal after submission but before retry The service follows a documented retrieval/retry policy and records the decision; no new unauthorized order is created

Measure outcomes at the business system, not only at the agent runtime. Correlate operation ID, attempt ID, caller, tenant, payload fingerprint, authorization result, approval reference, downstream order ID, and status transitions. The evidence should allow an operator to distinguish “never submitted,” “committed but response lost,” “duplicate suppressed,” “rejected,” and “still unknown.”

Recover uncertainty without repeating intent

Reliable agent recovery needs a durable identity for business intent, an idempotent or reconcilable action boundary, and an explicit state for unknown outcomes. Persistence and retries can make a workflow resume; they cannot independently guarantee that an external action happens once. ORBYNTIS’s assurance approach emphasizes evaluating how agents interact with tools and business workflows, then testing observable controls against realistic failure paths. The relevant evidence is the resulting business state and the policy that allowed or denied each attempt.

References