LLM Security Review Checklist

Use this before shipping an LLM feature that reads untrusted content, retrieves private data, or calls tools.

1. Workflow Boundary

• The workflow owner is named.
• The workflow’s allowed actions are documented.
• External, destructive, financial, legal, HR, or customer-visible actions are identified.
• There is a documented way to disable the workflow quickly.
• The workflow has a rollback or manual fallback path.

2. Untrusted Input Inventory

• Direct user input is treated as untrusted.
• Retrieved documents are treated as untrusted data, not instructions.
• Tool outputs are treated as untrusted unless generated inside the trusted boundary.
• Uploaded PDFs, images, audio, spreadsheets, and transcripts are treated as untrusted.
• Web pages and browser-agent observations are treated as untrusted.
• Admin-editable prompts, workflow templates, CMS content, and knowledge sources are reviewed before production use.

3. Data And Retrieval Boundary

• Secrets, credentials, raw tokens, and private URLs are never sent to the model.
• Retrieval filters by tenant, user, role, and source permissions before ranking.
• Retrieved chunks preserve source ID, tenant ID, visibility, owner, version, and review timestamp.
• The answer path can cite or log which source IDs were used.
• Stale, unreviewed, or low-trust sources are excluded or flagged.
• Logs redact personal data and credentials.

4. Prompt And Context Contract

• System/developer instructions are versioned and reviewed like code.
• Untrusted content is wrapped and labeled as data.
• The model is told what to do when untrusted content conflicts with task instructions.
• The model receives only the minimum context required for the task.
• High-risk workflows use a narrow extraction step before the main agent sees content.
• Prompt canary phrases are used only for detection, not as the primary defense.

5. Tool And Action Design

• Tools are narrow and task-specific.
• Tools derive user, tenant, and role from server-side auth context, not model arguments.
• Every tool validates arguments with a schema and business rules.
• Every tool enforces authorization independently of the model.
• Write tools are idempotent where possible.
• External side effects require human approval or deterministic policy checks.
• Rate limits, quotas, and cost limits are configured.
• Tool calls are logged with user, tenant, workflow, prompt version, model, arguments summary, and result.

6. Output Validation

• Model output is parsed with a strict schema before use.
• Unknown fields are rejected when the contract should be closed.
• URLs, Markdown, HTML, filenames, and code blocks are sanitized where relevant.
• Outputs cannot include data classes outside the task’s allowed scope.
• Factual answers over private sources require source IDs or source snippets.
• Malformed output fails closed instead of falling back to free text.

7. Regression Tests

• Direct injection test: user input asks the model to ignore instructions.
• Indirect injection test: retrieved content asks the model to reveal data or call a tool.
• Cross-tenant test: a request tries to access another tenant’s data.
• Unsafe tool-call test: the model requests an unavailable or disallowed action.
• Malformed output test: extra fields or invalid enum values are rejected.
• Exfiltration test: output tries to include secrets, private data, or prompt text.
• Persistence test: stored content contains malicious instructions that are later retrieved.
• Multi-modal test, if relevant: OCR-visible or image-visible instructions are treated as untrusted.

8. Monitoring And Incident Response

• Prompt-extraction attempts are detectable.
• Repeated validator failures are alertable.
• Unusual retrieval breadth, tool-call volume, outbound recipients, cost, or rate spikes are alertable.
• Prompt-injection attempts can be linked to user, tenant, workflow, source IDs, and tool calls.
• The team knows how to disable the workflow or individual tools.
• The team knows how to revoke provider keys and rotate affected credentials.
• There is an owner for customer, legal, security, and regulator notification decisions.

Launch Gate

Do not mark the workflow production-ready until:

• All high-consequence actions have a gate.
• At least one adversarial document test has passed.
• At least one unauthorized-access attempt has failed safely.
• At least one unsafe tool-call attempt has failed safely.
• At least one malformed-output test has failed closed.
• At least one workflow-disable path has been tested.