Document ingestion for retrieval systems: extracting text from real files, stripping boilerplate, deduplication, keeping an index fresh, and deletions.https://contextaugmentedllm.com/https://contextaugmentedllm.com/blogs/a-website-has-no-file-list/https://contextaugmentedllm.com/blogs/a-website-has-no-file-list/A share can be enumerated. A site cannot: its document set is whatever the traversal reached, and nothing on the far end will report what was missed.Wed, 12 Aug 2026 00:00:00 GMTpipelinesfreshnessmetadatahttps://contextaugmentedllm.com/blogs/the-page-that-checks-who-is-asking/https://contextaugmentedllm.com/blogs/the-page-that-checks-who-is-asking/Four hundred documents in the corpus say 'Verify you are human'. Every fetch succeeded; the source served the barrier instead of the page.Sat, 08 Aug 2026 00:00:00 GMTextractionboilerplatepipelineshttps://contextaugmentedllm.com/blogs/what-the-converter-threw-away/https://contextaugmentedllm.com/blogs/what-the-converter-threw-away/HTML becomes markdown or plain text and the conversion never errors. Tables, alt text, heading levels and withdrawn clauses go missing on the way.Tue, 04 Aug 2026 00:00:00 GMTextractionmetadatapipelineshttps://contextaugmentedllm.com/blogs/counting-what-you-did-not-ingest/https://contextaugmentedllm.com/blogs/counting-what-you-did-not-ingest/Every corpus has documents that never made it in. Without a category for each way that happens, they look identical to documents that never existed.Tue, 28 Jul 2026 00:00:00 GMTpipelinesmetadatafreshnesshttps://contextaugmentedllm.com/blogs/reprocessing-a-corpus-you-already-ingested/https://contextaugmentedllm.com/blogs/reprocessing-a-corpus-you-already-ingested/An extraction fix only runs on documents ingested after it shipped. Everything already in the index keeps the old damage until something goes back for it.Mon, 27 Jul 2026 00:00:00 GMTpipelinesmetadatafreshnesshttps://contextaugmentedllm.com/blogs/deletion-is-the-hard-one/https://contextaugmentedllm.com/blogs/deletion-is-the-hard-one/Removing a source file removes nothing downstream. Everything derived from it persists, stays retrievable, and answers questions as if it still existed.Sun, 26 Jul 2026 00:00:00 GMTdeletionfreshnessmetadatapipelineshttps://contextaugmentedllm.com/blogs/keeping-an-index-fresh/https://contextaugmentedllm.com/blogs/keeping-an-index-fresh/A full rebuild is correct and unaffordable. Incremental sync is affordable and drifts. What decides it is how you detect that something changed.Sat, 25 Jul 2026 00:00:00 GMTfreshnessmetadatapipelineshttps://contextaugmentedllm.com/blogs/the-same-document-three-times/https://contextaugmentedllm.com/blogs/the-same-document-three-times/Identical copies, near-copies and successive versions look alike and need opposite treatment. Deduplicating a version history destroys the record.Fri, 24 Jul 2026 00:00:00 GMTdeduplicationmetadatapipelineshttps://contextaugmentedllm.com/blogs/boilerplate-is-not-content/https://contextaugmentedllm.com/blogs/boilerplate-is-not-content/Navigation, footers and disclaimers extract perfectly and index as fact. Repeated across a corpus, they crowd out the pages they were attached to.Thu, 23 Jul 2026 00:00:00 GMTboilerplateextractionpipelineshttps://contextaugmentedllm.com/blogs/extracting-text-from-real-documents/https://contextaugmentedllm.com/blogs/extracting-text-from-real-documents/A PDF that extracts as scrambled reading order does not raise an error. It produces text, reports success, and poisons everything downstream of it.Wed, 22 Jul 2026 00:00:00 GMTextractionocrpipelineshttps://contextaugmentedllm.com/blogs/ingesting-the-same-document-twice-at-once/https://contextaugmentedllm.com/blogs/ingesting-the-same-document-twice-at-once/Retries and queues deliver documents more than once. If a rerun adds records instead of replacing them, one document answers with two of everything.Wed, 15 Jul 2026 00:00:00 GMTpipelinesdeduplicationfreshnesshttps://contextaugmentedllm.com/blogs/capturing-who-may-see-a-document/https://contextaugmentedllm.com/blogs/capturing-who-may-see-a-document/Permissions live on folders and groups, not on documents. If ingestion does not record them, nothing downstream can reconstruct who may read what.Mon, 13 Jul 2026 00:00:00 GMTmetadatapipelinesfreshnesshttps://contextaugmentedllm.com/blogs/the-reconciliation-pass/https://contextaugmentedllm.com/blogs/the-reconciliation-pass/Incremental sync has no way to prove it kept up. A periodic comparison of the index against the source is the only thing that finds what it missed.Sat, 11 Jul 2026 00:00:00 GMTfreshnessdeletionpipelineshttps://contextaugmentedllm.com/blogs/finding-one-bad-document-a-month-later/https://contextaugmentedllm.com/blogs/finding-one-bad-document-a-month-later/Someone reports a wrong answer citing a specific file. Ingestion logs are keyed by job and time, so nothing can say what happened to that one document.Thu, 09 Jul 2026 00:00:00 GMTmetadatapipelinesfreshnesshttps://contextaugmentedllm.com/blogs/replacing-what-an-edit-invalidated/https://contextaugmentedllm.com/blogs/replacing-what-an-edit-invalidated/A document was edited and reprocessed correctly. The index now holds the new text and some of the old, because the previous records were never removed.Tue, 07 Jul 2026 00:00:00 GMTfreshnessdeletionpipelineshttps://contextaugmentedllm.com/blogs/formats-with-no-reading-order/https://contextaugmentedllm.com/blogs/formats-with-no-reading-order/Spreadsheets and slide decks have no sequence to extract. Flattening them produces values and phrases detached from the thing that gave them meaning.Mon, 06 Jul 2026 00:00:00 GMTextractionmetadatapipelineshttps://contextaugmentedllm.com/blogs/deciding-which-documents-need-ocr/https://contextaugmentedllm.com/blogs/deciding-which-documents-need-ocr/Recognition is the most expensive stage in ingestion, and the documents that need it do not announce themselves. Routing has to be inferred per page.Sun, 05 Jul 2026 00:00:00 GMTocrextractionpipelineshttps://contextaugmentedllm.com/blogs/normalising-text-before-anyone-reads-it/https://contextaugmentedllm.com/blogs/normalising-text-before-anyone-reads-it/Invisible characters, split accents and soft hyphens break matching without changing how text looks. The damage is silent and the fix is one pass.Sat, 04 Jul 2026 00:00:00 GMTextractionmetadatapipelineshttps://contextaugmentedllm.com/blogs/separating-extraction-from-indexing/https://contextaugmentedllm.com/blogs/separating-extraction-from-indexing/One pass that reads a file and writes records means every fix re-reads the whole corpus. Keeping the extracted text makes reprocessing affordable.Fri, 03 Jul 2026 00:00:00 GMTpipelinesextractionmetadatahttps://contextaugmentedllm.com/blogs/a-document-id-that-survives-a-move/https://contextaugmentedllm.com/blogs/a-document-id-that-survives-a-move/Path-based identity breaks the moment a file moves. One rename produces a duplicate and an orphan at the same time, and neither of them errors.Thu, 02 Jul 2026 00:00:00 GMTmetadatadeduplicationpipelineshttps://contextaugmentedllm.com/blogs/which-date-does-a-document-have/https://contextaugmentedllm.com/blogs/which-date-does-a-document-have/A file offers half a dozen dates and none of them is when the document was written. A share migration set most of them to the day the migration ran.Mon, 29 Jun 2026 00:00:00 GMTmetadatafreshnesspipelineshttps://contextaugmentedllm.com/blogs/the-listing-that-was-not-complete/https://contextaugmentedllm.com/blogs/the-listing-that-was-not-complete/Everything downstream of enumeration is scoped by it. A listing that stopped early produces a smaller corpus that behaves exactly like a correct one.Fri, 26 Jun 2026 00:00:00 GMTpipelinesfreshnessmetadatahttps://contextaugmentedllm.com/blogs/which-language-is-this-document-in/https://contextaugmentedllm.com/blogs/which-language-is-this-document-in/Language is a fact nothing records by default. Without it, translations look like duplicates and quality checks report clean text as damaged.Wed, 24 Jun 2026 00:00:00 GMTmetadataextractiondeduplicationhttps://contextaugmentedllm.com/blogs/documents-inside-documents/https://contextaugmentedllm.com/blogs/documents-inside-documents/Archives, attachments and embedded files hold most of what people search for. A pipeline that reads only the outer file ingests the wrapper and none of it.Mon, 22 Jun 2026 00:00:00 GMTextractionmetadatapipelineshttps://contextaugmentedllm.com/blogs/the-documents-that-will-not-ingest/https://contextaugmentedllm.com/blogs/the-documents-that-will-not-ingest/Every corpus has files that error, time out, or crash the parser. Handled as log lines they leave the corpus quietly and never come back.Fri, 19 Jun 2026 00:00:00 GMTpipelinesfreshnessmetadatahttps://contextaugmentedllm.com/blogs/pages-that-are-empty-until-something-runs-them/https://contextaugmentedllm.com/blogs/pages-that-are-empty-until-something-runs-them/A web source ingested cleanly and every page holds the same nav and a loading message. The fetch returned a shell that assembles its content elsewhere.Wed, 17 Jun 2026 00:00:00 GMTextractionboilerplatepipelineshttps://contextaugmentedllm.com/blogs/a-thread-is-not-a-document/https://contextaugmentedllm.com/blogs/a-thread-is-not-a-document/A twenty-message email thread contains the first message twenty times. Ingest it naively and the oldest, most-quoted text becomes the loudest claim.Mon, 15 Jun 2026 00:00:00 GMTextractionboilerplatededuplicationhttps://contextaugmentedllm.com/blogs/the-order-the-stages-run-in/https://contextaugmentedllm.com/blogs/the-order-the-stages-run-in/Hashing before normalising, fingerprinting before stripping, writing before deciding identity. Each is a stage in the wrong place and none of them error.Thu, 11 Jun 2026 00:00:00 GMTpipelinesdeduplicationmetadata