refactored the entire structure for the new product line

Revised Project Plan.md

@@ -0,0 +1,814 @@
+Yes. The plan is directionally sound, but there are several gaps I would close **before we begin rewriting architecture**. Most of them arise because Project Thoth has evolved considerably beyond what the current `.thoth` documents describe.
+
+The existing material gives us a strong foundation. It already establishes preservation, modularity, platform independence, deterministic processing, Markdown canonicality, self-description, ADRs, work orders, specifications, and the distinction between methodology and implementation.   The problem is that the existing architecture and roadmap still describe the earlier capture-first Project Thoth rather than the complete application you now intend to build.  
+
+I would therefore make the **current project an explicit re-foundation of Project Thoth**, rather than incrementally editing the old plan.
+
+## 1. Add a Project Reconciliation phase before SDLC design
+
+Your first activity should not actually be "Select an SDLC methodology."
+
+It should be:
+
+> **Establish the authoritative state of Project Thoth.**
+
+Right now there are multiple generations of Project Thoth documentation.
+
+For example, ADR-001 correctly establishes Project Thoth as a platform whose methodology is authoritative over any particular software implementation.  That remains highly compatible with what you just described.
+
+But the current architecture describes a primarily linear knowledge-preservation system:
+
+> External Sources → Capture → Archive → Processing → Repository → Search → AI Consumers. 
+
+That is no longer sufficient. The new application must also provide an **interactive AI operating environment** in which a user can choose Oracle/Ollama or a frontier provider, apply the Thinking Methodology, retrieve knowledge, continue a task across engines, and produce artifacts.
+
+Likewise, `00-project-state.md` still says the project is at version 0.0.1 and focused on the ChatGPT Capture Connector. 
+
+So add:
+
+### Phase 0 — Project Reconciliation
+
+* Inventory all `.thoth` documents.
+* Inventory specifications.
+* Inventory ADRs.
+* Inventory existing application/source code.
+* Inventory previous experiments and abandoned implementations.
+* Classify each artifact as:
+
+  * Current
+  * Requires Revision
+  * Superseded
+  * Historical
+  * Candidate for removal
+* Explicitly review every accepted ADR.
+* Preserve valid decisions rather than silently rewriting history.
+* Create superseding ADRs where architectural decisions have changed.
+* Update `00-project-state.md` only after this audit.
+
+This prevents an AI developer three months from now from encountering two equally "canonical" descriptions of Project Thoth.
+
+---
+
+# 2. Separate the Project Thoth **product** from the Project Thoth **application**
+
+You have effectively defined a product line:
+
+```text
+Project Thoth
+│
+├── Methodology
+│   ├── Knowledge Architecture
+│   ├── Archiving Methodology
+│   └── Thinking Methodology
+│
+├── Intellectual Property
+│   ├── Project Thoth Book
+│   ├── Consulting Methodology
+│   └── Specifications
+│
+├── Software Platform
+│   ├── Project Thoth Application
+│   ├── Capture Connectors
+│   ├── Processors
+│   └── Engine Adapters
+│
+└── Engineering System
+    ├── .thoth SDLC standard
+    ├── Studio Runbook
+    └── Development Runbook
+```
+
+This actually reinforces ADR-001 rather than replacing it. ADR-001 already says the software is an implementation of the methodology rather than the definition of Project Thoth itself. 
+
+I would formalize this distinction now.
+
+Otherwise "Project Thoth" will alternately mean:
+
+* the philosophy,
+* the methodology,
+* the application,
+* the repository,
+* the corpus,
+* the development project,
+
+depending upon which document one is reading.
+
+Your glossary is already intended to prevent exactly this kind of semantic drift. 
+
+---
+
+# 3. Add explicit Product Vision and Product Scope documents
+
+Before requirements, create a short **Product Definition**.
+
+Your current message contains most of it.
+
+It should define:
+
+**Vision**
+
+A knowledge-centered generative AI environment in which knowledge and reasoning persist independently of the AI engine consuming them.
+
+**Primary differentiator**
+
+Engine independence.
+
+**Primary user**
+
+Initially you.
+
+**Primary operating model**
+
+Single-user, locally operated, internally designated system.
+
+**Primary infrastructure**
+
+```text
+Forge   → development/client workstation
+Gateway → application runtime
+Oracle  → replaceable inference appliance
+Frontier Providers → external replaceable inference services
+Vault   → durable knowledge corpus
+```
+
+**Critical product promise**
+
+> Changing inference engines may change capability and depth, but it must not fundamentally change how Project Thoth works.
+
+That last statement should eventually become both a requirement and a testable acceptance criterion.
+
+---
+
+# 4. Add actors and system boundaries before identifying use cases
+
+"Identify all Use Cases" is correct, but there is a missing preceding step.
+
+First identify the **actors**.
+
+At minimum I expect:
+
+```text
+Human User
+
+Project Thoth Application
+
+Local Inference Engine
+    Ollama today
+    Spark/Ollama or successor tomorrow
+
+Frontier AI Provider
+    OpenAI
+    Anthropic
+    Google
+    etc.
+
+Knowledge Vault
+
+Capture Connector
+
+External Knowledge Source
+
+Source Control / CI System
+```
+
+Possibly later:
+
+```text
+Administrator
+Additional Project Thoth User
+Processor
+Automation
+External Application
+```
+
+Then use cases can be organized around actor goals rather than features.
+
+This will greatly improve the requirements document.
+
+---
+
+# 5. Add Non-Functional Requirements
+
+This is the largest requirements gap.
+
+Use cases tell us **what Project Thoth does**.
+
+Architecture will be driven just as heavily by **how it must behave**.
+
+You need explicit requirements for at least:
+
+* Engine independence
+* Data portability
+* Deployment portability
+* Replaceability
+* Availability
+* Performance
+* Security
+* Privacy
+* Auditability
+* Provenance
+* Recoverability
+* Backup
+* Testability
+* Maintainability
+* Observability
+* Configuration management
+* Upgradeability
+* Browser support
+* Accessibility
+* Data integrity
+
+Some already exist philosophically. For example, the current philosophy says AI providers must be interchangeable, canonical artifacts human-readable, generated artifacts reproducible, and architecture capable of outliving implementations. 
+
+Those principles now need to become engineering requirements.
+
+For example:
+
+> **NFR-ENGINE-001:** Project Thoth SHALL expose inference through a provider-independent application interface.
+
+> **NFR-ENGINE-002:** Replacement of Oracle SHALL NOT require modification of knowledge storage, user interface, Thinking Methodology, or conversation persistence components.
+
+That turns philosophy into something architecture can enforce.
+
+---
+
+# 6. Add the Thinking Methodology as a first-class product capability
+
+This is especially important.
+
+The current architecture barely contains the system you have developed around:
+
+```text
+Constitution
+    ↓
+Perspective
+    ↓
+Protocol
+    ↓
+Channel
+    ↓
+Voice
+```
+
+Those concepts now exist as substantive Project Thoth intellectual property. The Constitution of Inquiry establishes the reasoning framework and explicitly treats conversations as primary sources of reasoning.  Your Perspective and Protocol artifacts already demonstrate that these are modular behavioral components rather than merely prompts.   
+
+Therefore your methodology phase should include:
+
+### Thinking Methodology Definition
+
+* Constitution
+* Perspectives
+* Protocols
+* Voices
+* Channels
+* Artifact definitions
+* Precedence/order rules
+* Composition rules
+* Activation rules
+* Override/supersession rules
+* Storage format
+* Versioning
+* Runtime assembly
+* Engine-independent behavior
+* Validation/testing methodology
+
+This should become a requirements source for the application.
+
+---
+
+# 7. Add Knowledge Architecture requirements explicitly
+
+"Knowledge Management" is presently implied rather than decomposed.
+
+You should define requirements around:
+
+```text
+Capture
+Preservation
+Cataloging
+Classification
+Collections
+Retrieval
+Reasoning Context
+Processing
+Transformation
+Provenance
+Publishing
+Archiving
+```
+
+The existing Source Metadata and Manifest specifications are useful inputs because they already establish the distinction between objective source description and downstream analytical interpretation.  
+
+But the application now needs requirements around how those artifacts are:
+
+* created,
+* stored,
+* viewed,
+* edited,
+* regenerated,
+* related,
+* searched,
+* fed into inference.
+
+That bridge between **knowledge methodology** and **application behavior** is currently missing.
+
+---
+
+# 8. Add an Engine Abstraction requirements workstream
+
+Given your new product definition, I would make this one of the central requirements workstreams.
+
+The architecture must not think in terms of:
+
+```text
+Project Thoth → Ollama
+```
+
+It should eventually resemble conceptually:
+
+```text
+                  Project Thoth
+                        │
+                 Inference Contract
+                        │
+          ┌─────────────┼─────────────┐
+          ▼             ▼             ▼
+       Ollama        OpenAI       Anthropic
+          │
+        Oracle
+```
+
+Oracle itself is **not an architectural dependency**.
+
+Oracle is a deployment of one provider.
+
+That distinction is crucial because you already expect Oracle to evolve from the current machine to one Spark and potentially several Sparks.
+
+The application should not care.
+
+This should therefore be settled as a requirement before architecture begins.
+
+---
+
+# 9. Add an Environment and Toolchain Decision phase
+
+Your SDLC section currently jumps from methodology selection to coding standards and CI/CD.
+
+Before Sprint 0, decide the standard engineering workstation.
+
+Questions include:
+
+* VS Code vs Visual Studio Community
+* Git workflow
+* Gitea project organization
+* Package management
+* Container development approach
+* Local debugging strategy
+* Testing framework
+* Documentation tooling
+* Diagram standard
+* API documentation standard
+* secrets management
+* environment configuration
+* dependency update strategy
+
+I would **not switch from VS Code merely because this becomes a larger application**. That decision should follow the eventual technology stack rather than precede it.
+
+If, for example, architecture leads toward TypeScript/React/Python, VS Code remains entirely sensible.
+
+If you deliberately choose a heavily .NET-centric architecture, Visual Studio Community becomes more compelling.
+
+So defer that decision until the technology architecture is clearer.
+
+---
+
+# 10. Expand Sprint 0 beyond CI/CD
+
+You describe Sprint 0 as creation of the SDLC pipeline.
+
+I would make Sprint 0:
+
+> **Prove that the software factory works before using it to build Project Thoth.**
+
+It should include:
+
+```text
+Repository
+    ↓
+Branch / change workflow
+    ↓
+Work Order / issue
+    ↓
+Developer or AI implementation
+    ↓
+Local validation
+    ↓
+Commit
+    ↓
+Push to Gitea
+    ↓
+Automated build
+    ↓
+Automated tests
+    ↓
+Container build
+    ↓
+Artifact/version creation
+    ↓
+Deployment to Gateway test environment
+    ↓
+Smoke test
+```
+
+The deliverable should be a trivial application running through the **entire pipeline**.
+
+Not Project Thoth functionality.
+
+Something as simple as:
+
+```text
+Project Thoth
+Version: 0.0.1
+Build: <git-sha>
+Status: Healthy
+```
+
+would be enough.
+
+That proves the studio before the studio begins building the product.
+
+---
+
+# 11. Add software quality and test strategy before development
+
+Your coding standard already calls for unit, integration, E2E, and regression testing. 
+
+But the project plan needs a **Testing Strategy**.
+
+Particularly important for Project Thoth will be:
+
+### Deterministic tests
+
+For parsers, serializers, configuration, retrieval, and storage.
+
+### Contract tests
+
+For engine adapters.
+
+Every inference provider should satisfy the same Project Thoth engine contract.
+
+### Behavioral fidelity tests
+
+This is unusual and important.
+
+You have already defined the requirement conceptually:
+
+> Results may differ in depth but fundamentally should be similar.
+
+Eventually you need a test corpus that can ask:
+
+```text
+Same:
+  conversation
+  knowledge context
+  Constitution
+  Perspective
+  Protocol
+  Voice
+  user request
+
+Run against:
+  frontier engine
+  local engine
+
+Evaluate:
+  Did each follow the methodology?
+  Did each use the supplied knowledge?
+  Did each preserve provenance?
+  Did each satisfy the task?
+```
+
+That is likely to become one of Project Thoth's most distinctive engineering practices.
+
+---
+
+# 12. Add security and trust boundaries
+
+This is missing completely from the plan and will influence architecture.
+
+At minimum define:
+
+* Where API keys live.
+* Whether frontier providers may receive all knowledge.
+* Which corpus material may leave the local network.
+* How retrieved documents inherit access restrictions.
+* Whether logs may contain prompts or documents.
+* How application secrets are stored.
+* Authentication assumptions.
+* Network trust boundaries between Forge, Gateway, and Oracle.
+* How imported/captured documents are treated as untrusted content.
+* Prompt-injection boundaries.
+
+Given that Project Thoth is fundamentally a Knowledge Architecture product, security cannot be retrofitted after retrieval is built.
+
+---
+
+# 13. Add configuration and environment separation
+
+Even an internal project should define:
+
+```text
+Development
+Test
+Production
+```
+
+They might initially all live on the same physical hardware, but the logical distinction matters.
+
+You will want containerized configuration such as:
+
+```text
+thoth-dev
+thoth-test
+thoth-prod
+```
+
+or equivalent.
+
+This makes your studio runbook applicable later to GW and future software projects.
+
+---
+
+# 14. Add architectural decision governance
+
+You already have ADRs, which is excellent.
+
+But define their lifecycle:
+
+```text
+Proposed
+Accepted
+Rejected
+Deprecated
+Superseded
+```
+
+And establish the rule:
+
+> Existing accepted ADRs are never silently rewritten to reflect a new decision.
+
+Instead:
+
+```text
+ADR-004 supersedes ADR-00X
+```
+
+That preserves the intellectual history of the project, which is especially appropriate for Project Thoth.
+
+---
+
+# 15. Add requirements traceability
+
+Because AI will participate heavily in development, I would establish this from the beginning.
+
+Something lightweight:
+
+```text
+Business Goal
+    ↓
+Use Case
+    ↓
+Functional Requirement
+    ↓
+Non-Functional Requirement
+    ↓
+Architecture Component
+    ↓
+Work Order
+    ↓
+Test
+```
+
+You do not need heavyweight enterprise tooling.
+
+Markdown IDs are enough:
+
+```text
+UC-007
+FR-021
+NFR-008
+ADR-012
+WO-034
+TEST-091
+```
+
+That makes the repository extraordinarily legible to both humans and AI.
+
+---
+
+# 16. Separate the two runbooks
+
+Your deliverables identify:
+
+1. Software Development Studio Runbook
+2. SDLC Runbook
+
+Keep those separate.
+
+### Software Development Studio Runbook
+
+Answers:
+
+> How do I create the environment in which software gets built?
+
+For example:
+
+```text
+Forge configuration
+Gitea
+Gateway
+Docker
+CI runners
+SSH
+credentials
+repository creation
+developer tooling
+backup
+deployment environment
+```
+
+### SDLC Runbook
+
+Answers:
+
+> Once the studio exists, how do we build software?
+
+For example:
+
+```text
+Product definition
+Requirements
+Architecture
+ADR
+Sprint planning
+Work Orders
+Development
+Code review
+Testing
+CI
+Release
+Deployment
+Maintenance
+Retirement
+```
+
+That distinction will make these deliverables reusable outside Project Thoth.
+
+---
+
+# Revised Project Plan
+
+I would therefore restructure your current plan like this:
+
+### 0. Reconcile Project Thoth
+
+* Audit existing repository.
+* Audit `.thoth`.
+* Audit ADRs.
+* Audit specifications.
+* Audit existing source code.
+* Identify obsolete experiments.
+* Establish authoritative current state.
+* Record superseded decisions without deleting history.
+
+### 1. Establish the Software Development Studio
+
+* Define Forge/Gateway/Gitea development topology.
+* Select standard development tools.
+* Define repository creation procedure.
+* Define environment strategy.
+* Define secrets/configuration strategy.
+* Define backup/recovery expectations.
+* Draft Software Development Studio Runbook.
+
+### 2. Establish SDLC Practices
+
+* Select/adapt methodology.
+* Define lifecycle phases.
+* Define roles.
+* Define ADR process.
+* Define Work Order process.
+* Define Git/branch/change process.
+* Define testing strategy.
+* Define Definition of Ready.
+* Define Definition of Done.
+* Define release/versioning conventions.
+* Define requirements traceability.
+* Audit/refactor coding standards.
+* Draft SDLC Runbook.
+
+### 3. Redesign `.thoth`
+
+* Define purpose of `.thoth`.
+* Define required files.
+* Define optional files.
+* Define numbering system.
+* Create `.thoth/README.md`.
+* Define canonical-vs-generated documentation rules.
+* Define AI onboarding instructions.
+* Reorganize existing documents.
+* Update `00-project-state.md`.
+
+### 4. Establish Sprint 0
+
+* Gitea repository.
+* CI runner.
+* Build pipeline.
+* Test pipeline.
+* Container pipeline.
+* Versioning.
+* Deployment to Gateway.
+* Health check.
+* Smoke test.
+* Prove end-to-end SDLC with trivial application.
+
+### 5. Define Project Thoth Product
+
+* Product vision.
+* Product boundaries.
+* Actors.
+* Infrastructure assumptions.
+* Product terminology.
+* Engine-independence principle.
+* Product success measures.
+* Relationship among methodology, application, archive, connectors, processors, and book.
+
+### 6. Define Project Thoth Methodology
+
+* Review philosophy.
+* Define Knowledge Architecture.
+* Define Archiving Methodology.
+* Define Thinking Methodology.
+* Define Constitutions.
+* Define Perspectives.
+* Define Protocols.
+* Define Voices.
+* Define Channels.
+* Define artifact specifications.
+* Define composition and precedence.
+* Identify which concepts require application support.
+
+### 7. Develop Requirements
+
+* Actor catalog.
+* Use-case catalog.
+* Detailed use cases.
+* Functional requirements.
+* Non-functional requirements.
+* Security/privacy requirements.
+* Knowledge requirements.
+* Engine abstraction requirements.
+* Deployment requirements.
+* Data requirements.
+* Operational requirements.
+* Behavioral-fidelity requirements.
+* Acceptance criteria.
+* Requirements traceability matrix.
+
+### 8. Architecture Readiness Review
+
+This is where **this project plan ends**.
+
+Confirm:
+
+* Product boundary is stable.
+* Major use cases are understood.
+* NFRs exist.
+* Thinking Methodology requirements exist.
+* Knowledge Architecture requirements exist.
+* Engine independence is formally defined.
+* SDLC is operational.
+* Sprint 0 pipeline works.
+* `.thoth` can onboard a new human or AI developer.
+* Major architectural unknowns are identified.
+* Existing ADR conflicts are resolved.
+
+Then:
+
+> **Formal Project Thoth Architecture begins.**
+
+---
+
+## One important change in emphasis
+
+I would not call what happens before architecture "pre-development."
+
+You are already developing Project Thoth.
+
+The **software implementation** has not begun, but requirements, specifications, ADRs, methodology, test strategy, development infrastructure, and the `.thoth` system are all engineering artifacts.
+
+That distinction matters because one of the things you are practicing here is precisely what gets lost in many AI-driven projects: **software engineering is not synonymous with writing code**.
+
+And that may ultimately be one of the most reusable lessons from Project Thoth for the studio model.
+
+The immediate next artifact I would create is **not the requirements document**. It should be the new `.thoth/README.md`, because that document can establish the document taxonomy, numbering scheme, authority hierarchy, and instructions governing all the work that follows.

Structure.md

@@ -0,0 +1,50 @@
+Project Thoth
+│
+├── .thoth
+│   └── Canonical project self-description
+│
+├── Methodologies
+│   │
+│   ├── Source Capture Methodology
+│   │   ├── Principles
+│   │   ├── Processes
+│   │   └── Specifications
+│   │
+│   ├── Archive Methodology
+│   │   ├── Principles
+│   │   ├── Processes
+│   │   └── Specifications
+│   │
+│   └── Thinking Methodology
+│       ├── Constitution
+│       ├── Perspectives
+│       ├── Protocols
+│       ├── Voices
+│       ├── Channels
+│       ├── Artifact Definitions
+│       └── Specifications
+│
+├── Application
+│   │
+│   ├── Source Capture
+│   │   └── Implementation determined by requirements
+│   │
+│   ├── Archive
+│   │   └── Implementation determined by requirements
+│   │
+│   ├── Thinking
+│   │   └── Implementation determined by requirements
+│   │
+│   ├── Retrieval
+│   │
+│   ├── Inference
+│   │
+│   └── User Interface
+│
+├── Knowledge Corpus
+│   └── Durable source and derived knowledge
+│
+└── Engineering
+    ├── Software Development Studio Runbook
+    ├── SDLC Runbook
+    └── Development Standards

applications/rag-ingestion-poc/.gitignore

@@ -0,0 +1,7 @@
+.env
+__pycache__/
+*.py[cod]
+.venv/
+build/
+dist/
+*.egg-info/

applications/rag-ingestion-poc/README.md

@@ -0,0 +1,45 @@
+# Thoth RAG Ingestion POC
+
+This directory is a self-contained Python application for loading, chunking,
+embedding, and inspecting Markdown documents. Runtime code uses only the Python
+standard library.
+
+## Layout
+
+```text
+rag-ingestion-poc/
+├── README.md
+├── src/
+│   └── rag_ingestion/
+├── tests/
+└── work-orders/
+```
+
+The implementation uses the conventional `src` package layout. Historical
+implementation work orders are kept under `work-orders`.
+
+## Configure
+
+Copy `.env.example` to `.env` and provide the Oracle embedding endpoint and
+model. The application always reads the `.env` beside this README, regardless
+of the shell's current working directory. Operating-system environment values
+override values from the file.
+
+## Set up and run
+
+From this directory:
+
+```powershell
+python -m venv .venv
+.venv\Scripts\python -m pip install -e .
+.venv\Scripts\python -m rag_ingestion.ingest path\to\conversation.md --show-chunks
+.venv\Scripts\python -m rag_ingestion.ingest --check-embedding-config
+```
+
+The editable installation also provides `thoth-rag-ingest` as a console command.
+
+## Test
+
+```powershell
+.venv\Scripts\python -m unittest discover -s tests -v
+```

applications/rag-ingestion-poc/pyproject.toml

@@ -0,0 +1,19 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "thoth-rag-ingestion"
+version = "0.1.0"
+description = "Self-contained Markdown RAG ingestion proof of concept"
+requires-python = ">=3.10"
+
+[project.scripts]
+thoth-rag-ingest = "rag_ingestion.ingest:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["rag_ingestion*"]
+
+[tool.unittest]
+start-directory = "tests"

applications/rag-ingestion-poc/src/rag_ingestion/config.py

@@ -9,6 +9,8 @@ from urllib.parse import urlsplit
 ORACLE_BASE_URL_VARIABLE = "THOTH_ORACLE_BASE_URL"
 EMBEDDING_MODEL_VARIABLE = "THOTH_EMBEDDING_MODEL"
 EMBEDDING_DIMENSION_VARIABLE = "THOTH_EMBEDDING_DIMENSION"
+APPLICATION_ROOT = Path(__file__).resolve().parents[2]
+DEFAULT_ENV_PATH = APPLICATION_ROOT / ".env"
 
 
 @dataclass(frozen=True, slots=True)
@@ -65,9 +67,9 @@ def read_env_file(path: Path) -> dict[str, str]:
 
 def load_embedding_config(
     environ: Mapping[str, str] | None = None,
-    env_path: str | Path = ".env",
+    env_path: str | Path = DEFAULT_ENV_PATH,
 ) -> EmbeddingConfig:
-    """Load `.env`, then override it with operating-system environment values."""
+    """Load the application's `.env`, then apply OS environment overrides."""
 
     values = read_env_file(Path(env_path))
     values.update(os.environ if environ is None else environ)

applications/rag-ingestion-poc/tests/test_chunking.py

@@ -6,10 +6,10 @@ import unittest
 from datetime import datetime, timezone
 from pathlib import Path
 
-from processors.rag_ingestion.chunking import ChunkConfiguration, chunk_document
-from processors.rag_ingestion.ingest import format_chunks
-from processors.rag_ingestion.loader import load_markdown_document
-from processors.rag_ingestion.models import Document
+from rag_ingestion.chunking import ChunkConfiguration, chunk_document
+from rag_ingestion.ingest import format_chunks
+from rag_ingestion.loader import load_markdown_document
+from rag_ingestion.models import Document
 
 
 def document_with_text(text: str) -> Document:

applications/rag-ingestion-poc/tests/test_config.py

@@ -5,15 +5,15 @@ import unittest
 from pathlib import Path
 from unittest.mock import patch
 
-from processors.rag_ingestion.config import EmbeddingConfig, load_embedding_config
-from processors.rag_ingestion.embeddings import embed_chunk, provider_from_config
-from processors.rag_ingestion.embeddings import (
+from rag_ingestion.config import EmbeddingConfig, load_embedding_config
+from rag_ingestion.embeddings import embed_chunk, provider_from_config
+from rag_ingestion.embeddings import (
     EmbeddingConnectionError,
     EmbeddingRequestError,
 )
-from processors.rag_ingestion.ingest import check_embedding_config
+from rag_ingestion.ingest import check_embedding_config
 from tests.test_chunking import document_with_text
-from processors.rag_ingestion.chunking import ChunkConfiguration, chunk_document
+from rag_ingestion.chunking import ChunkConfiguration, chunk_document
 
 
 class EmbeddingConfigurationTests(unittest.TestCase):
@@ -121,7 +121,7 @@ class ConfigurationDiagnosticTests(unittest.TestCase):
     def test_request_failure_distinguishes_successful_connection(self) -> None:
         config = EmbeddingConfig("http://oracle:11434", "model:v1", None)
         with patch(
-            "processors.rag_ingestion.ingest.provider_from_config"
+            "rag_ingestion.ingest.provider_from_config"
         ) as provider_factory:
             provider_factory.return_value.embed.side_effect = EmbeddingRequestError(
                 "model does not support embeddings"
@@ -134,7 +134,7 @@ class ConfigurationDiagnosticTests(unittest.TestCase):
     def test_connection_failure_is_reported_separately(self) -> None:
         config = EmbeddingConfig("http://oracle:11434", "model:v1", None)
         with patch(
-            "processors.rag_ingestion.ingest.provider_from_config"
+            "rag_ingestion.ingest.provider_from_config"
         ) as provider_factory:
             provider_factory.return_value.embed.side_effect = EmbeddingConnectionError(
                 "unreachable"

applications/rag-ingestion-poc/tests/test_embeddings.py

@@ -5,10 +5,10 @@ import tempfile
 import unittest
 from pathlib import Path
 
-from processors.rag_ingestion.chunking import ChunkConfiguration, chunk_document
-from processors.rag_ingestion.embeddings import embed_chunk
-from processors.rag_ingestion.loader import load_markdown_document
-from processors.rag_ingestion.similarity import cosine_similarity
+from rag_ingestion.chunking import ChunkConfiguration, chunk_document
+from rag_ingestion.embeddings import embed_chunk
+from rag_ingestion.loader import load_markdown_document
+from rag_ingestion.similarity import cosine_similarity
 
 
 class FixedEmbeddingProvider:

applications/rag-ingestion-poc/tests/test_rag_ingestion.py

@@ -7,8 +7,8 @@ import tempfile
 import unittest
 from pathlib import Path
 
-from processors.rag_ingestion.ingest import format_document
-from processors.rag_ingestion.loader import load_markdown_document
+from rag_ingestion.ingest import format_document
+from rag_ingestion.loader import load_markdown_document
 
 
 class MarkdownIngestionTests(unittest.TestCase):
@@ -93,7 +93,7 @@ class MarkdownIngestionTests(unittest.TestCase):
         missing = self.directory / "absent.md"
 
         result = subprocess.run(
-            [sys.executable, "-m", "processors.rag_ingestion.ingest", str(missing)],
+            [sys.executable, "-m", "rag_ingestion.ingest", str(missing)],
             capture_output=True,
             text=True,
             check=False,

iteration-plan.md

@@ -1,26 +0,0 @@
-I agree that the presentation layer should be a self-hosted web application.
-
-I agree with using an n-tier architecture where the functions exist in the business logic layer and interfaces to external services (like NextCloud) exist in the data layer.
-
-The iterations of the MVP should be much smaller than proposed:
-
-Iteration 0:
-- establish CI/CD pipeline to use with project development
-- create docker compose to deploy self-hosted application
-- display home page that says "Project Thoth"
-
-Iteration 1:
-- create application shell
-- navigation bar with placeholders including link to the Vault
-- header title and menu bar
-- placeholder for profile icon, and identity management menu
-- footer content
-- informational home page with boiler plate content
-
-Iteration 2:
-- a Vault page to show nextcloud content
-- nextcloud interface that will successfully connect to an instance
-- display top level nextcloud folder structure in Vault page
-
-Iteration 3:
-- allow user to browse through the Vault

methodologies/thinking-methodology/contitutions/metadata specification.md

@@ -0,0 +1,582 @@
+---
+
+id: metadata.source.standard
+name: Standard Source Metadata
+version: 1.0
+type: metadata-schema
+status: canonical
+scope: external-and-reference-sources
+-------------------------------------
+
+# Standard Source Metadata
+
+## PURPOSE
+
+This metadata schema provides a common provenance structure for source material used by Project Thoth.
+
+It is intended for:
+
+* downloaded articles;
+* social media posts;
+* news stories;
+* Substack articles;
+* magazine and journal publications;
+* book excerpts;
+* essays;
+* reports;
+* transcripts;
+* reference documents;
+* archived web content;
+* and similar source material.
+
+The schema should identify:
+
+1. what the source is;
+2. who created it;
+3. where it came from;
+4. when it was published or captured;
+5. what relationship the user has to it;
+6. how Thoth should treat it during inquiry.
+
+---
+
+# REQUIRED FIELDS
+
+Every source should contain the following fields when known.
+
+```yaml
+---
+title: ""
+author: ""
+publication: ""
+publication_date: ""
+source_type: ""
+canonical_url: ""
+
+user_authored: false
+editable: false
+role: evidence
+---
+```
+
+---
+
+# FIELD DEFINITIONS
+
+## title
+
+The title of the source.
+
+Examples:
+
+```yaml
+title: Myth of the Day: Witch
+```
+
+```yaml
+title: The Left Hand of Darkness
+```
+
+```yaml
+title: LinkedIn post by Jane Smith on AI governance
+```
+
+For social media posts without a formal title, create a descriptive archival title.
+
+---
+
+## author
+
+The person or organization primarily responsible for the source.
+
+Examples:
+
+```yaml
+author: Ursula K. Le Guin
+```
+
+```yaml
+author: Mythology: Gods and Monsters
+```
+
+```yaml
+author: Jane Smith
+```
+
+Do not infer an individual author when only an organization is known.
+
+---
+
+## publication
+
+The publication, platform, publisher, or container in which the source appeared.
+
+Examples:
+
+```yaml
+publication: Substack
+```
+
+```yaml
+publication: The Atlantic
+```
+
+```yaml
+publication: LinkedIn
+```
+
+```yaml
+publication: Reddit
+```
+
+```yaml
+publication: Harper & Row
+```
+
+---
+
+## publication_date
+
+The original publication date when known.
+
+Prefer ISO format:
+
+```yaml
+publication_date: 2026-09-01
+```
+
+If the exact day is unknown, preserve only what is known.
+
+```yaml
+publication_date: 1969
+```
+
+Do not invent precision.
+
+---
+
+## source_type
+
+Classify the source by its original form.
+
+Recommended controlled values:
+
+```yaml
+source_type: news_article
+source_type: web_article
+source_type: substack_article
+source_type: social_media_post
+source_type: journal_article
+source_type: magazine_article
+source_type: book
+source_type: book_excerpt
+source_type: report
+source_type: transcript
+source_type: forum_post
+source_type: blog_post
+source_type: reference_document
+source_type: archival_document
+```
+
+Use the narrowest accurate value.
+
+---
+
+## canonical_url
+
+The original or preferred source URL when available.
+
+Example:
+
+```yaml
+canonical_url: https://example.com/article
+```
+
+For offline or print sources, omit or leave blank.
+
+---
+
+# PROVENANCE FIELDS
+
+## captured_from
+
+Describes how the source entered Thoth.
+
+Recommended values:
+
+```yaml
+captured_from: web
+captured_from: browser_export
+captured_from: manual_copy
+captured_from: pdf
+captured_from: ebook
+captured_from: screenshot
+captured_from: social_platform
+captured_from: transcript_export
+```
+
+---
+
+## captured_at
+
+Date or timestamp when the source was captured into Thoth.
+
+Example:
+
+```yaml
+captured_at: 2026-09-04T05:30:00-05:00
+```
+
+This is distinct from publication date.
+
+---
+
+## source_status
+
+Describes the source's relationship to the original.
+
+Recommended values:
+
+```yaml
+source_status: complete
+source_status: excerpt
+source_status: partial
+source_status: archived_copy
+source_status: transcript
+source_status: quoted_fragment
+```
+
+This field is important because Thoth should not silently treat an excerpt as though it represents a whole work.
+
+---
+
+# USER RELATIONSHIP
+
+## user_authored
+
+Indicates whether the source was written by the user.
+
+```yaml
+user_authored: true
+```
+
+or:
+
+```yaml
+user_authored: false
+```
+
+This field should be explicit.
+
+Do not infer authorship merely because the user uploaded the document.
+
+---
+
+## editable
+
+Indicates whether the source should normally be treated as working material that may be revised.
+
+```yaml
+editable: true
+```
+
+or:
+
+```yaml
+editable: false
+```
+
+Examples:
+
+A user's article draft:
+
+```yaml
+user_authored: true
+editable: true
+```
+
+A downloaded news article:
+
+```yaml
+user_authored: false
+editable: false
+```
+
+A copied public-domain text:
+
+```yaml
+user_authored: false
+editable: false
+```
+
+---
+
+# SOURCE ROLE
+
+## role
+
+Defines how the source should normally function during inquiry.
+
+Recommended controlled values:
+
+```yaml
+role: evidence
+role: reference
+role: working_draft
+role: canonical
+role: example
+role: context
+role: quotation_source
+role: background
+```
+
+### evidence
+
+The source is being examined as evidence.
+
+Typical behavior:
+
+* analyze claims;
+* inspect assumptions;
+* compare with other sources;
+* test interpretations;
+* cite when needed.
+
+Do not rewrite unless requested.
+
+### reference
+
+The source supplies factual or contextual information.
+
+Typical behavior:
+
+* retrieve information;
+* preserve terminology;
+* use as supporting context.
+
+### working_draft
+
+The source is material intended for revision.
+
+Typical behavior:
+
+* critique;
+* rewrite;
+* restructure;
+* edit;
+* develop.
+
+Usually:
+
+```yaml
+user_authored: true
+editable: true
+role: working_draft
+```
+
+### canonical
+
+The source defines an authoritative internal rule or specification.
+
+Examples:
+
+* Constitution of Inquiry;
+* canonical Voice specification;
+* Protocol definition;
+* architecture document.
+
+Treat its definitions as governing within its declared scope.
+
+Do not casually rewrite or contradict it.
+
+### example
+
+The source is illustrative rather than authoritative.
+
+Use it to understand form, style, structure, or behavior.
+
+### context
+
+The source exists primarily to provide situational background.
+
+### quotation_source
+
+The source is preserved primarily to support exact quotations or citation.
+
+### background
+
+The source may inform inquiry but should not be treated as primary evidence.
+
+---
+
+# OPTIONAL TRUST AND EVIDENCE FIELDS
+
+## source_level
+
+Recommended values:
+
+```yaml
+source_level: primary
+source_level: secondary
+source_level: tertiary
+source_level: unknown
+```
+
+This describes evidentiary position, not quality.
+
+A primary source may be wrong.
+
+A tertiary source may be accurate.
+
+---
+
+## verification_status
+
+Recommended values:
+
+```yaml
+verification_status: unverified
+verification_status: partially_verified
+verification_status: verified
+verification_status: disputed
+```
+
+Do not mark a source verified merely because it appears authoritative.
+
+---
+
+## reliability_notes
+
+Free-text notes about known limitations.
+
+Example:
+
+```yaml
+reliability_notes: >
+  Author provides no primary-source citations for several historical claims.
+```
+
+---
+
+# OPTIONAL CONTENT FIELDS
+
+## language
+
+```yaml
+language: en
+```
+
+## topics
+
+```yaml
+topics:
+  - witchcraft
+  - mythology
+  - folklore
+  - anthropology
+```
+
+## people
+
+```yaml
+people:
+  - E. E. Evans-Pritchard
+  - Heinrich Kramer
+```
+
+## places
+
+```yaml
+places:
+  - Europe
+  - Central Africa
+```
+
+These fields support discovery and indexing.
+
+They should not substitute for interpretation.
+
+---
+
+# OPTIONAL RIGHTS FIELDS
+
+## rights
+
+Recommended values:
+
+```yaml
+rights: copyrighted
+rights: public_domain
+rights: licensed
+rights: unknown
+```
+
+## license
+
+Example:
+
+```yaml
+license: CC BY 4.0
+```
+
+These fields describe usage rights, not evidentiary value.
+
+---
+
+# COMPLETE EXAMPLE
+
+```yaml
+---
+title: Myth of the Day: Witch
+author: Mythology: Gods and Monsters
+publication: Substack
+publication_date: 2026-09-01
+source_type: substack_article
+canonical_url: https://substack.com/home/post/p-204248117
+
+captured_from: browser_export
+captured_at: 2026-09-04T05:30:00-05:00
+source_status: complete
+
+user_authored: false
+editable: false
+role: evidence
+
+source_level: secondary
+verification_status: unverified
+
+language: en
+
+topics:
+  - witchcraft
+  - mythology
+  - folklore
+  - anthropology
+
+rights: copyrighted
+---
+```
+
+---
+
+# MINIMUM PRACTICAL PROFILE
+
+When complete metadata is unavailable, use at minimum:
+
+```yaml
+---
+title: ""
+author: ""
+publication: ""
+publication_date: ""
+source_type: ""
+canonical_url: ""
+
+user_authored: false
+editable: false
+role: evidence
+source_status: complete
+---
+```
+
+Do not invent missing values merely to complete the schema.