Project-Thoth/codex/rag-ingestion/0001-establish-rag-ingestion-foundation.md

Work Order 0001 — Establish the RAG Ingestion Foundation

Status

Planned

Work Stream

RAG Ingestion

Application

rag-ingestion

Location

Work order:

./codex/rag-ingestion/0001-establish-rag-ingestion-foundation.md

Implementation package:

./processors/rag_ingestion/

---

Objective

Establish the first working slice of the Project Thoth RAG ingestion pipeline.

This work order should create a small, explicit, inspectable Python application that can:

1. Accept the path to a single Markdown source file.
2. Read the source file without modifying it.
3. Capture basic mechanical source information from the filesystem.
4. Assign a deterministic document identifier.
5. Produce an in-memory document object representing the source.
6. Display the resulting document record in a human-readable form for inspection.

This work order intentionally stops before chunking, embeddings, PostgreSQL, pgvector, or recursive folder ingestion.

The purpose is to establish and understand the ingestion boundary before adding downstream RAG behavior.

---

Learning Objective

This work order is not only an implementation task.

The implementation must make the ingestion process understandable enough that the developer can explain:

• what "ingestion" means in a RAG pipeline;
• what information exists before any AI processing occurs;
• which source properties can be obtained deterministically from the filesystem;
• why a stable document identifier is needed;
• what information belongs to the original source versus a derived ingestion record;
• where ingestion ends and chunking begins.

Codex should prefer explicit code over framework abstractions when the abstraction would hide these concepts.

The finished implementation should be easy to read in VS Code and suitable for walking through line by line.

---

Architectural Context

Project Thoth preserves Primary Sources as authoritative records.

The RAG ingestion pipeline is a consumer of those sources. It must not modify, rewrite, summarize, enrich, or replace them.

For this baseline RAG implementation, distinguish between:

Mechanical Metadata

Information obtained directly from the file or filesystem, such as:

• source path;
• filename;
• file size;
• modified timestamp;
• source format.

Knowledge Metadata

Information that requires classification or interpretation, such as:

• primary topics;
• reasoning contexts;
• content types;
• entities;
• controlled vocabulary;
• relationships.

Knowledge Metadata is explicitly out of scope for this work order.

Project Thoth already has separate Source Metadata and Conversation Manifest processors. This baseline RAG pipeline must not silently reproduce their responsibilities.

---

Problem

The Project Thoth vault already contains many saved conversations as Markdown files.

Before those files can participate in a RAG pipeline, the system needs a deterministic way to discover and represent a source document.

Most RAG frameworks combine loading, parsing, chunking, metadata extraction, and indexing behind high-level abstractions. That is useful for rapid application development, but it can obscure the mechanics that this implementation is intended to teach.

The first step should therefore be implemented directly in Python with minimal dependencies.

---

Decision

Create a dedicated Python package at:

./processors/rag_ingestion/

The package will initially contain only the code necessary for single-file Markdown ingestion.

Use Python package naming conventions (rag_ingestion) even though the application and work-order directory use the name rag-ingestion.

The source Markdown file remains untouched.

The ingestion result is a derived in-memory representation.

Do not introduce LangChain, LlamaIndex, Haystack, or another RAG framework in this work order.

---

Proposed Package Structure

processors/
└── rag_ingestion/
    ├── __init__.py
    ├── models.py
    ├── loader.py
    └── ingest.py

The exact internal organization may vary slightly if repository conventions require it, but responsibilities should remain separated.

models.py

Defines the document representation used by the ingestion layer.

loader.py

Contains the logic for reading a Markdown source and gathering filesystem metadata.

ingest.py

Provides the command-line entry point for ingesting one Markdown file and displaying the resulting document record.

---

Document Record

The initial document representation should include, at minimum:

• document_id
• source_path
• filename
• source_format
• file_size
• modified_at
• raw_text

The implementation may add a small number of additional mechanical fields if required by existing project conventions.

Do not add semantic or AI-generated fields.

---

Deterministic Document Identifier

The document_id must be deterministic.

Running ingestion repeatedly against the same logical source should produce the same identifier.

Do not use a random UUID.

Select and document a simple deterministic strategy.

Examples of acceptable inputs to the identifier calculation include:

• normalized source path;
• repository-relative or configured corpus-relative path.

Do not base the identifier solely on file contents because editing a source should not automatically turn the same logical source into a different document identity.

The implementation should make the chosen identity rule obvious in the code.

---

Command-Line Behavior

Provide a simple invocation for one file.

For example:

python -m processors.rag_ingestion.ingest /path/to/conversation.md

The exact argument syntax may follow existing repository conventions.

The command should:

1. validate that the path exists;
2. validate that the source is a Markdown file;
3. load the source;
4. construct the document record;
5. print the document record in a readable form;
6. exit successfully.

Invalid inputs should fail clearly with an actionable error.

---

Inspectability Requirement

The output must make it possible to verify what the ingestion process learned from the source.

At minimum, the displayed output should make the following visible:

Document ID:
Source Path:
Filename:
Source Format:
File Size:
Modified At:
Raw Text Length:

It is not necessary to print the entire Markdown document by default.

The implementation should allow the developer to inspect the raw text easily in code or during debugging.

---

Requirements

1. Create the rag_ingestion Python package under ./processors.
2. Implement ingestion for one Markdown file only.
3. Preserve the source file exactly as it exists.
4. Capture only deterministic/mechanical metadata.
5. Generate a deterministic document ID.
6. Keep source loading separate from the document model.
7. Provide a CLI entry point.
8. Produce readable inspection output.
9. Use clear type annotations.
10. Include docstrings where they improve understanding.
11. Keep dependencies minimal.
12. Do not hide the ingestion process behind a RAG framework.
13. Follow existing Project Thoth repository conventions.
14. Do not perform opportunistic refactoring outside this work order.

---

Non-Goals

Do not implement any of the following:

• recursive directory traversal;
• Nextcloud integration;
• Markdown semantic parsing;
• conversation-specific parsing;
• chunking;
• token counting for chunking;
• embeddings;
• embedding models;
• vector databases;
• PostgreSQL;
• pgvector;
• lexical indexing;
• retrieval;
• reranking;
• prompt construction;
• Oracle communication;
• LLM inference;
• Source Metadata generation;
• Conversation Manifest generation;
• knowledge extraction;
• taxonomy;
• relationship extraction;
• web UI;
• background watchers;
• automatic re-ingestion.

These belong to later work orders.

---

Manual Test Plan

Use at least one real Markdown conversation from the Project Thoth archive for the manual test.

Test 1 — Valid Markdown source

Run the ingestion command against a real .md conversation.

Verify:

• the file loads successfully;
• the document ID is produced;
• source path is correct;
• filename is correct;
• source format identifies Markdown;
• file size corresponds to the filesystem;
• modified timestamp is captured;
• raw text length is greater than zero.

Test 2 — Deterministic identity

Run the command against the same source multiple times.

Verify:

• document_id is identical on each run.

Test 3 — Different source

Run the command against a second Markdown file.

Verify:

• the second source receives a different document_id.

Test 4 — Missing source

Run the command against a path that does not exist.

Verify:

• the program fails clearly;
• the error identifies the missing path.

Test 5 — Wrong file type

Run the command against a non-Markdown file.

Verify:

• the program rejects the input;
• the error explains that this ingestion path currently supports Markdown only.

Test 6 — Source preservation

Record the source file hash before and after ingestion.

Verify:

• the source file hash is unchanged.

---

Acceptance Criteria

This work order is complete when:

• ./processors/rag_ingestion/ exists as an importable Python package;
• a developer can invoke the ingestion pipeline against one Markdown file;
• the source is represented by a typed document model;
• all required mechanical fields are populated;
• the document ID is deterministic;
• the source file is not modified;
• no chunking or AI processing occurs;
• the implementation contains no RAG orchestration framework;
• all manual tests pass;
• the developer can trace the execution from CLI input to document record without encountering a hidden framework pipeline.

---

Learning Verification

After implementation, Codex should include a short implementation report that answers the following questions using the actual code that was built:

1. Where does ingestion begin in this implementation?
2. Which values come directly from the filesystem?
3. Which value is generated by the ingestion system?
4. How is document identity made deterministic?
5. Why is raw_text retained even though later stages will operate on chunks?
6. What information has deliberately not been inferred from the document?
7. At what exact point will the next work order begin adding derived retrieval units?

The answers should reference the relevant files/functions in the implementation.

This section is required because Project Thoth work orders are being used both to build the system and to develop a deep technical understanding of RAG.

---

Definition of Done

The first RAG ingestion boundary exists and is operational.

A real Project Thoth Markdown conversation can be passed into the application and converted into an inspectable deterministic document record without modifying the Primary Source or performing any semantic processing.

The implementation is sufficiently transparent that the developer can explain every field in the record, where it came from, and why it exists.

The next work order can begin with chunking from this known document representation rather than combining source ingestion and retrieval processing into a single opaque step.

---

Codex Execution Guidance

Before making changes:

1. Read the repository bootstrap and architectural guidance.
2. Inspect existing ./processors conventions.
3. Inspect existing Python project/dependency configuration.
4. Do not assume a new dependency manager or application layout if the repository already defines one.
5. Plan the narrowest set of files required for this work order.

During implementation:

1. Keep the code explicit and educational.
2. Prefer standard-library functionality where practical.
3. Preserve separation between source loading and the document model.
4. Do not implement future work-order functionality "while already in the code."
5. Add only the tests or supporting files necessary to validate this work order.

At completion, report:

• root cause / need addressed;
• files created or changed;
• implementation decisions;
• deterministic ID strategy;
• validation performed;
• manual test results;
• learning verification answers;
• anything that could not be verified.