Secure document ingestion for RAG: PDFs, OCR, metadata, and retention

RAG failures can begin before retrieval: stale documents, missed tables, ownerless sources, lost permission metadata, undeleted vectors, contradictory hidden text, or unsafe files.

Secure document ingestion is source control for company knowledge. It decides what enters the retrieval system, who may see it, how freshness is tracked, how deletion works, and how bad inputs are caught.

This article covers the ingestion layer: PDFs, OCR, metadata, permissions, retention, and operational checks.

If a user should not access a document in the source system, they should not access its chunks, embeddings, summaries, or cached answers in the RAG system. Permission metadata is not optional.

The ingestion pipeline

A production pipeline should have explicit stages:

1. Source registration.
2. Data classification.
3. File safety checks.
4. Text extraction and OCR.
5. Structure preservation.
6. Metadata attachment.
7. Permission mapping.
8. Chunking and embedding.
9. Quality checks.
10. Index publication.
11. Retention and deletion handling.
12. Operational ownership.

The exact tools can vary. The control points should not.

Stage 1: source registration

Do not ingest random folders because they are easy to connect.

For each source, record:

• source name,
• system of record,
• source owner,
• data owner,
• allowed users or roles,
• document types,
• sensitivity level,
• retention rule,
• update frequency,
• deletion behavior,
• review schedule.

Example sources:

• public help center,
• internal support playbook,
• sales collateral,
• customer contracts,
• HR policies,
• engineering runbooks,
• product documentation,
• meeting transcripts.

These sources should not all land in the same index with the same permissions.

Stage 2: classify data before extraction

Classify the source before the model or embedding provider sees the content.

Useful classes:

Classification is not just compliance paperwork. It decides whether content can be sent to a hosted embedding API, stored in a shared vector database, included in logs, or used for eval examples.

Stage 3: file safety checks

Documents can be hostile or simply broken.

Before extraction:

• check file type against an allowlist,
• enforce file size limits,
• scan for malware where your environment requires it,
• reject encrypted files unless there is an approved decrypt path,
• reject files with unsupported embedded objects,
• normalize filenames,
• store original file hash,
• record who uploaded or connected the file.

This is especially important if non-admin users can upload documents. Admin-only ingestion lowers one threat path but does not make a compromised, malicious, oversized, or malformed document safe. Connector and URL ingestion also require allowlisted origins, redirect and DNS controls, authentication scoping, rate limits, and SSRF testing.

If untrusted users can upload files, implement a file-safety layer before parsing. Use the maintained OWASP File Upload Cheat Sheet as a baseline and threat-model the selected parsers and storage path. For the broader retrieval threat model—including poisoning, permission leakage, prompt injection, and insecure output—also use the OWASP RAG Security Cheat Sheet.

Candidate controls include allowlisted types verified from file content, size and archive-expansion limits, randomized storage names, signature scanning, isolated no-egress parsing with resource limits, and content disarm and reconstruction where the threat model justifies it. No single scanner establishes that a file is safe.

Stage 4: text extraction and OCR

PDFs are not one format in practice. Some contain selectable text. Some are scans. Some have columns, tables, footnotes, forms, comments, stamps, or hidden text layers.

Choose the extraction path by document type: compare text-first extractors for born-digital PDFs, layout-aware converters for structured documents, and OCR for scans. Tool choice must follow benchmarked layout/table accuracy, language support, license, patch process, and the data boundary.

On thresholds: do not adopt a universal OCR-confidence cutoff from a blog post — calibrate on a sample of your own documents. The useful shape is two bands: below the lower band the page is rejected outright; between the bands it is queued for human review; above the upper band it flows through. Where those bands sit depends on your scanner, your document age, and your language.

For Estonian and mixed-language archives, include diacritics, older typewritten pages, tables, stamps, and Estonian-Russian examples in the benchmark. Verify that the chosen OCR engine has the required language data, then measure character, word, field, and table accuracy on representative pages. Do not promise a universal pilot duration.

Track extraction quality:

• extraction method,
• OCR confidence,
• page count,
• extracted character count,
• table extraction status,
• language detected,
• pages with no text,
• parser warnings.

Low-quality extraction should not quietly enter the index. Route it to review or mark it as low confidence.

Common problems:

• columns read in the wrong order,
• table rows merged incorrectly,
• headers repeated in every chunk,
• scanned pages missing entirely,
• handwritten notes ignored,
• hidden text layer contradicting the visible scan,
• OCR converting account numbers incorrectly.

For high-value documents, spot-check the rendered page against extracted text.

Stage 5: preserve structure

RAG systems need more than text. They need enough structure to produce useful, source-grounded answers.

Preserve:

• title,
• heading path,
• section number,
• page number,
• table captions,
• list boundaries,
• document version,
• effective date,
• source URL or storage path.

Chunk text with headings and page references. A chunk that says “The following applies” without the preceding heading is weak evidence.

For tables, decide whether to:

• keep the table as Markdown,
• convert it to structured JSON,
• store both text and structured rows,
• exclude it until a better parser is available.

Do not pretend table extraction is solved if your use case depends on exact prices, dates, limits, or thresholds.

Stage 6: attach metadata

Every chunk should carry metadata that can survive retrieval:

{
  "sourceId": "policy-2026-expenses",
  "documentId": "doc_123",
  "tenantId": "tenant_a",
  "visibility": "internal",
  "allowedRoles": ["finance", "leadership"],
  "sensitivity": "confidential",
  "sourceOwner": "Finance",
  "version": "2026-02",
  "lastReviewedAt": "2026-02-10",
  "effectiveFrom": "2026-03-01",
  "page": 7,
  "headingPath": ["Travel", "Hotel limits"],
  "contentHash": "sha256:..."
}

Metadata carries inputs to policy enforcement; it does not enforce policy by itself. The application must validate trusted metadata provenance and enforce authorization at publication, retrieval, cache, citation, generation, and export boundaries.

Stage 7: permission mapping

Permission enforcement must happen before retrieval results reach the model and again at any answer, citation, cache, or export boundary where identity or policy may have changed.

Good pattern:

1. User asks a question.
2. Application derives tenant, user, roles, groups, and data permissions from auth.
3. Retriever filters candidate chunks by permissions.
4. Ranking happens within allowed chunks.
5. Model receives only allowed chunks.

Bad pattern:

1. Retrieve broadly.
2. Send all likely chunks to the model.
3. Prompt says “only answer using chunks the user may access.”

The bad pattern has already exposed data to the model context.

If source permissions are complex, start narrower. It is better to miss an answer than leak a confidential document.

Stage 8: chunking and embedding

Chunking is a security and quality decision, not only a search tuning decision.

Guidelines:

• Keep chunks inside permission boundaries.
• Do not merge public and confidential text into one chunk.
• Include headings and source references.
• Avoid giant chunks that contain unrelated sections.
• Avoid tiny chunks that lose context.
• Re-embed when source text or metadata changes.
• Store embedding model and version.

For sensitive sources, confirm whether your embedding provider, vector store, and logs are approved for that data class.

Stage 9: quality gates

Before publishing a source into production retrieval, run checks:

• all documents have owners,
• all chunks have permission metadata,
• stale documents are flagged,
• pages with failed extraction are excluded or reviewed,
• sample questions retrieve expected sources,
• unauthorized users retrieve zero restricted chunks,
• deleted documents disappear from search,
• citations point to valid source locations,
• suspicious instructions in documents are isolated as content, not followed.

The last point matters. Documents can contain prompt injection. The ingestion pipeline should not silently remove all such text, because users may need to know what a document says and removal can damage evidence. The runtime must keep retrieved content in a data channel, prevent it from granting tools or changing policy, validate tool arguments independently, and require human approval for consequential effects.

Stage 10: publish an index version atomically

Do not let partially ingested documents or a mixture of old and new permissions leak into the active index. Build a candidate index or versioned namespace, then:

1. freeze the source/version manifest and content hashes;
2. verify document counts, extraction failures, permission coverage, duplicate policy, embedding model/revision, and representative authorized/unauthorized queries;
3. record the candidate version, source versions, test result, approver, and rollback target;
4. atomically switch an alias or application pointer to the approved version where the store supports it;
5. invalidate or version retrieval and answer caches; and
6. monitor errors, access denials, empty retrieval, and stale-version traffic after promotion.

If the vector store cannot switch versions atomically, implement an application-side publication state that excludes incomplete records and test concurrent readers during promotion. Permission revocation and urgent source deletion must not wait for a full rebuild: enforce current authorization outside the index, deny the affected source immediately, purge relevant caches, and complete derivative cleanup through the lifecycle workflow.

Rollback should restore the last approved index pointer without restoring revoked access or deleted data. Test that rollback does not resurrect a superseded ACL, deleted document, poisoned source, or stale cached answer.

Stage 11: retention and deletion

RAG systems often accidentally keep data longer than the source system.

GDPR Article 17 defines a right to erasure with conditions and exceptions. Qualified counsel must determine the applicable obligation and any lawful retention. The system still needs an inventory and lifecycle operation for every copy and derivative:

• original file cache,
• extracted text,
• chunks,
• embeddings,
• summaries,
• thumbnails or rendered pages,
• eval samples,
• logs, with an approved minimization and retention basis,
• backups, with documented expiry and restore behavior.

When a document is deleted or access is revoked, retrieval and response caches should stop returning it within the documented target. The deletion workflow must cover derived stores and prevent an old backup or delayed ingestion job from resurrecting the record.

Track:

• deletedAt,
• deletedBy or source event,
• deletion reason,
• downstream cleanup status,
• verification result.

Do not rely on “we removed it from the UI.” Vector stores and caches are easy to forget.

Stage 12: operational ownership

Every production source needs an accountable owner and a review trigger or cadence derived from its change rate and impact.

For each source, define:

• who approves ingestion,
• who approves permission changes,
• who reviews stale documents,
• who handles extraction failures,
• who responds to data deletion requests,
• who investigates retrieval mistakes.

If nobody owns a source, it should not be in a production RAG system.

Source governance, not guarantees

Secure RAG ingestion is deliberate source governance. It makes retrieval behavior and failures more observable and testable; it does not make generated answers inherently correct or safe.

The core controls:

• register sources,
• classify data,
• check files before parsing,
• measure extraction quality,
• preserve structure,
• attach metadata,
• enforce permissions before retrieval,
• test unauthorized access,
• support deletion,
• assign owners.

Source quality and source controls are necessary inputs to answer quality and safety, not guarantees. Validate retrieval, authorization, grounding, refusal, tool use, and lifecycle behavior end to end.