Idempotency, retries, and human gates for n8n AI nodes

An n8n workflow that calls a model looks finished when the happy path works once. Production breaks on the second delivery of the same webhook, the timeout that retries after the first call already succeeded, and the draft that auto-sent because nobody owned the approval step.

What follows is the hardening layer for AI-bearing workflows: idempotency, retry policy, human gates, and logging. It pairs with your first AI agent in n8n and the review patterns in human-in-the-loop design.

Enabling retry on a node that may already have created a CRM note, sent a message, or queued email can duplicate side effects when the result is unknown. Treat every external write as non-repeatable until the provider’s idempotency or reconciliation behavior is proved.

Why AI steps need different failure handling

Ordinary HTTP and model calls can fail through status codes, timeouts, malformed responses, or unknown commit state. Model-bearing steps add failure modes such as:

• Timeouts on slow local inference (local OpenAI-compatible endpoints).
• Parse failures when the model returns prose instead of JSON.
• Soft failures: valid JSON that is wrong.
• Partial success: the model answered, but a later tool write failed.

A blind retry fixes some timeouts. It amplifies the others. Separate transport retries (safe if the server never committed work) from business retries (only safe with an idempotency key).

n8n allows operators to retry failed executions from execution history (n8n execution documentation). That operator feature does not prove a side effect is safe to repeat; the workflow still needs the claim, reconciliation, and outbox controls below.

Idempotency starts with an atomic claim

Pick a stable key as early as the trigger allows:

Do not implement SELECT key followed by INSERT key, and do not use a spreadsheet row as a lock. Two n8n workers can both observe “missing” and proceed. Use a database uniqueness constraint and one atomic statement; PostgreSQL documents unique constraints as the mechanism that guarantees key uniqueness (PostgreSQL constraints).

Minimal PostgreSQL shape (adapt types, retention, and migrations to your system):

CREATE TABLE workflow_runs (
  idempotency_key text PRIMARY KEY,
  state text NOT NULL CHECK (state IN (
    'processing', 'awaiting_human', 'approved',
    'completed', 'failed_retryable', 'failed_terminal'
  )),
  payload_hash text NOT NULL,
  lease_owner uuid,
  lease_expires_at timestamptz,
  version bigint NOT NULL DEFAULT 0,
  result jsonb,
  updated_at timestamptz NOT NULL DEFAULT now()
);

Generate a random lease_owner UUID per n8n execution. Claim a new key, or recover only an explicitly retryable/expired lease, in one statement:

INSERT INTO workflow_runs (
  idempotency_key, state, payload_hash, lease_owner, lease_expires_at
)
VALUES ($1, 'processing', $2, $3, now() + interval '5 minutes')
ON CONFLICT (idempotency_key) DO UPDATE
SET lease_owner = EXCLUDED.lease_owner,
    lease_expires_at = EXCLUDED.lease_expires_at,
    state = 'processing',
    version = workflow_runs.version + 1,
    updated_at = now()
WHERE workflow_runs.payload_hash = EXCLUDED.payload_hash
  AND (workflow_runs.state = 'failed_retryable'
       OR (workflow_runs.state = 'processing'
           AND workflow_runs.lease_expires_at < now()))
RETURNING idempotency_key, lease_owner, version;

Zero returned rows means another execution owns the key or the run already reached a non-retryable state: fetch its status and no-op/return the prior outcome. If the same key arrives with a different payload_hash, stop for investigation; silently treating changed business input as the same event hides upstream corruption.

The lease must be bounded and renewed only by its owner. Every state transition uses compare-and-set:

UPDATE workflow_runs
SET state = $4, version = version + 1, updated_at = now()
WHERE idempotency_key = $1
  AND lease_owner = $2
  AND version = $3
  AND lease_expires_at > now()
RETURNING version;

If no row returns, this execution lost ownership and must not act. Size the initial lease from measured work duration, renew before expiry, cap total lease lifetime, and alert on repeated lease steals. A lease prevents abandoned work from blocking forever; it does not make a non-idempotent external send safe.

Webhook delivery and worker execution are normally at least once. A database claim makes concurrent ownership deterministic. It does not create exactly-once email, payment, or CRM effects across a network boundary; those require a downstream idempotency key or an outbox/dispatcher that can reconcile an unknown result.

Retry policy for AI nodes

Use a short matrix and encode it in the workflow, not in tribal memory:

Keep max iterations on agent nodes finite. A retry wrapper around an agent that already loops tools is how token bills and duplicate tool calls explode.

For local endpoints, size timeouts from measured latency; do not stack “retry three times at 60s each” on a synchronous customer webhook.

Human gates that block side effects

A human gate is not a Slack message that says “FYI.” It is a state where no customer-visible or irreversible action runs until an explicit approve signal.

Three patterns that work in n8n:

1. Approve-before-act

AI node → validate schema → write draft + key to store → create a single-use approval challenge → only an authenticated, unexpired approval transaction can enqueue the send.

2. Act-with-window

Queue send-later with a cancel window. Use only when the action is reversible enough that a late cancel is meaningful.

3. Approve-by-exception

Auto-act only for narrow, reversible cases whose deterministic eligibility rules and calibrated evaluation evidence meet an approved threshold; sample and monitor them, and escalate or abstain on uncertainty. A model’s self-reported confidence is not an enforcement control.

Map gate choice to consequence — the same decision model as human-in-the-loop design. Customer email, refunds, account or CRM changes, and ordinary operational financial actions stay approve-before-act until measured evidence and policy permit otherwise. Medical treatment, legal advice, regulated financial advice, child-safety decisions, and structural/construction decisions require a qualified professional; automation may prepare or route records but must not replace that review.

Example gate checklist on the approval card:

• Idempotency key
• Source record link
• Model output (draft / label / scores)
• Validation errors if any
• Approver identity to log
• Expiry time for the pending state

Approval links are bearer credentials

Never send https://n8n.example/webhook/approve?id=ticket-42&action=approve. Anyone who guesses, forwards, scans, or replays that URL can act. Generate at least 256 bits of cryptographically random token material, send the opaque token only over HTTPS, and store only its SHA-256 hash with:

• the run key and permitted decision;
• intended approver/audience or SSO policy;
• absolute expiry;
• consumed_at, decision, and approver identity;
• a one-use constraint.

A GET should display a confirmation page, not mutate state. Submit the decision with POST after authentication and CSRF protection. For low-complexity cases, current n8n nodes can pause and request approval; n8n itself recommends the Wait node for more complex approvals (n8n Gmail approval operation). Verify the actual authentication, expiry, forwarding, and audit semantics of the node/version you deploy; an emailed button is not automatically suitable for a payment or legal approval.

Create an approval record linked to the immutable business idempotency key:

CREATE TABLE approvals (
  approval_id uuid PRIMARY KEY,
  idempotency_key text NOT NULL REFERENCES workflow_runs(idempotency_key),
  token_hash bytea NOT NULL UNIQUE,
  allowed_decisions text[] NOT NULL,
  expires_at timestamptz NOT NULL,
  consumed_at timestamptz,
  decision text,
  approver_subject text,
  created_at timestamptz NOT NULL DEFAULT now()
);

Hash the raw token in the application and pass only the digest as $1. Consume it atomically:

UPDATE approvals
SET consumed_at = now(), decision = $2, approver_subject = $3
WHERE token_hash = $1
  AND consumed_at IS NULL
  AND expires_at > now()
  AND $2 = ANY (allowed_decisions)
RETURNING idempotency_key;

Zero returned rows means expired, invalid, already used, or wrong decision: do not send. Run this statement inside a transaction that then locks the matching workflow_runs row, verifies it is still awaiting_human, updates it to approved, and inserts the unique outbox row. Roll back the whole transaction if any step fails. For high-consequence actions, require logged-in SSO plus role/segregation-of-duties checks; possession of an email link alone is insufficient.

Do not let the model choose auto_reply and then honour that choice without a workflow-enforced threshold. Prompts suggest; nodes enforce.

Transactional outbox for external effects

Consuming approval, changing the run state, and recording the intended external effect should happen in one database transaction. Do not send from inside the approval webhook. A minimal outbox constraint:

CREATE TABLE effect_outbox (
  effect_id uuid PRIMARY KEY,
  idempotency_key text NOT NULL REFERENCES workflow_runs(idempotency_key),
  effect_type text NOT NULL,
  target text NOT NULL,
  payload jsonb NOT NULL,
  state text NOT NULL CHECK (state IN ('pending', 'sending', 'completed', 'unknown', 'failed')),
  lease_owner uuid,
  lease_expires_at timestamptz,
  provider_id text,
  created_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (idempotency_key, effect_type, target)
);

An outbox worker claims pending rows with a bounded lease (PostgreSQL FOR UPDATE SKIP LOCKED is designed for queue-like consumers; see locking clause documentation), calls the provider with the same idempotency key when supported, stores the provider’s external ID, then marks the row complete by compare-and-set.

If the worker times out after the provider may have accepted a non-idempotent action, mark the effect unknown and reconcile with the provider before retrying. SMTP send, for example, cannot be made exactly once by a local database transaction. Automatic resend after an unknown result is how duplicate customer email happens.

Logging that survives an incident

n8n execution history is a start. It is not a compliance archive by itself. For AI steps, log a structured event per key:

• Timestamp and workflow version / commit id if you version workflows
• Idempotency key and trigger source
• Redacted input hash or allowed fields (not raw secrets)
• Approved provider/endpoint class plus model and revision identity; avoid exposing internal hosts or credentials in broadly accessible logs
• Approved, minimized model output fields or a controlled pointer; raw output capture requires its own purpose, access, and retention decision
• Validation result
• Gate decision and actor
• Downstream writes with external ids
• Error class and retry count

Do not store private chain-of-thought dumps “for debugging” in a shared channel. Store decision summaries and tool arguments you would be willing to audit.

Execution logs often contain personal data from tickets and emails. Define retention, access, and redaction before you enable verbose logging on production AI nodes. Local models do not exempt you from GDPR-style accountability if you process personal data.

When something goes wrong, you need to answer: Did we process this key? Did we send? Who approved? Which model version drafted?

Reference sequence for a lead or ticket path

1. Webhook receives payload → validate schema (first AI agent in n8n style gate).
2. Compute key + payload hash → atomically claim a bounded processing lease.
3. Call AI node / agent with structured output contract.
4. Validate JSON (enum, required fields, max length).
5. If invalid after limited repair → failed_terminal + human alert.
6. If valid and high-risk action → CAS to awaiting_human; create a hashed, expiring, single-use approval challenge.
7. On authenticated approval POST → atomically consume challenge, update state, and insert the unique outbox effect.
8. Dispatcher leases the outbox row, calls the provider with the same key when supported, stores the provider ID, and marks both effect and run complete by CAS.
9. On reject → mark terminal with reason; do not enqueue.
10. On duplicate delivery → return prior outcome or report current state; never repeat the model/send path silently.

Optional: hand judgment-heavy drafting to Hermes through its bearer-authenticated API server, or deliberately use the separate webhook adapter when its event-ingress and configured-delivery contract fits the workflow. In either case, n8n or the business system keeps durable keys, gates, and connectors. See the illustrative n8n → Hermes webhook handoff.

Forced re-runs without breaking idempotency

Operators will re-run failed executions from the n8n UI. That is healthy — unless the re-run silently creates a second CRM note because the key is still completed from a partial success, or worse, re-sends mail because the key was never written.

Define an explicit re-run protocol:

1. Retryable recovery — only failed_retryable or an expired processing lease can be reclaimed by the atomic claim shown above. The same business key is retained.
2. Terminal or completed replay forbidden — failed_terminal, awaiting_human, approved, and completed keys return their prior/current state and do not start again.
3. Intentional correction or replacement — create a new business event with its own upstream-issued idempotency key, link it to the original key and external result, record the operator and reason, and send it through a fresh approval/outbox path. Do not invent an ad-hoc suffix or mutate the original run in place.

Surface the protocol on the approval card so night-shift operators are not inventing policy under pressure.

Observability metrics worth watching

You do not need a full observability platform on day one. Track weekly:

• Duplicate webhook rate (same key seen twice)
• Gate wait time (p50 / p95 — labeled as your measurements)
• Validation failure rate after the AI node
• Auto-act vs human-approved ratio
• Retry exhaustion count

Spikes in validation failures warrant investigation of model, prompt, schema, input-distribution, or integration changes. Duplicate spikes warrant investigation of upstream redelivery, claim failures, replays, or provider result ambiguity; the metric alone does not diagnose the cause.

Kill switch and ownership

Enforce a deny-by-default kill switch at the side-effect boundary or dispatcher, not only at the first workflow node: AI_ACTIONS_ENABLED=false must prevent every external send even when a run resumes mid-flow or bypasses an early branch. Test the disabled state against queued and in-flight effects, define what still logs, and name an authorized owner who can operate and verify the control.

Also define:

• Who may approve
• Who may force a re-run, and how a replacement event receives a new upstream-issued key linked to the original without an ad hoc suffix
• What “done” means for support SLAs when the gate is waiting

Ship checklist

• Idempotency key chosen and persisted before AI call
• Ten concurrent deliveries of the same key produce exactly one active lease
• Expired-lease recovery and stale-owner CAS rejection tested
• Retry rules documented per failure class
• Approval token hash, expiry, SSO/role, POST/CSRF, and single-use replay tested
• Human gate inserts an outbox row; it cannot call the send node directly
• Provider timeout after possible acceptance enters unknown and does not auto-resend
• Structured logs include key, validation, approver, external ids
• Kill switch tested
• Privacy retention set on logs

AI nodes earn their place when they are boring under failure. Idempotency keeps retries from lying. Human gates keep wrong outputs from becoming customer facts. Logging makes both claims checkable.