Initial commit containing architetural scaffolding and mvp

This commit is contained in:
2026-07-15 16:55:37 -05:00
commit 2cb04c77a9
57 changed files with 8933 additions and 0 deletions
+31
View File
@@ -0,0 +1,31 @@
Current Version
0.0.1
Current Focus
ChatGPT Capture Connector
Completed
✓ Reference Architecture
✓ ADR-001
✓ ADR-002
✓ ADR-003
✓ Metadata Generator
In Progress
• Capture Service
Blocked
• Browser Extension
Next Milestone
Capture MVP
+341
View File
@@ -0,0 +1,341 @@
# Project Thoth Architecture
**Version:** 1.0
**Status:** Living Document
---
# Purpose
This document describes the canonical architecture of Project Thoth.
It is intended to answer a single question:
> **How does information flow through the Project Thoth ecosystem?**
This document describes architecture, not implementation.
Current implementation status is maintained separately in `.thoth/00-project-state.md`.
---
# Architectural Principles
Project Thoth is built upon the following principles:
- Source artifacts are permanent.
- Generated artifacts are disposable.
- Every transformation is deterministic.
- AI providers are interchangeable.
- Markdown is the canonical document format.
- Knowledge is owned by the archive, not the AI.
- Processing stages should be loosely coupled.
- Every stage should be independently testable.
---
# High-Level Architecture
```
External Sources
Capture Connectors
Canonical Source Archive
Processing Pipeline
┌──────────────┼──────────────┐
▼ ▼ ▼
Metadata Manifest Harvests
Generation Generation Generation
│ │ │
└──────────────┼──────────────┘
Knowledge Repository
Search / Retrieval Layer
AI Consumers
```
---
# Layers
## 1. External Sources
Project Thoth captures knowledge from external systems.
Examples include:
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
- Local LLMs
- Documentation systems
- Markdown repositories
- Future knowledge sources
These systems are considered transient.
They are never considered the canonical repository.
---
## 2. Capture Connectors
Capture Connectors preserve source material.
Responsibilities include:
- Capture conversations
- Preserve ordering
- Preserve formatting
- Preserve attachments
- Produce canonical source documents
Capture Connectors do **not**:
- Summarize
- Interpret
- Classify
- Generate metadata
- Perform AI reasoning
They are intentionally deterministic.
---
## 3. Canonical Source Archive
The archive contains the preserved source material.
Typical artifacts include:
- conversation.md
- source assets
- attachments
The archive is immutable.
Once source material has been captured it should not be modified.
If processing changes in the future, artifacts are regenerated from the archived source.
---
## 4. Processing Pipeline
Processors transform source artifacts into derived artifacts.
Typical processors include:
- Metadata generation
- Manifest generation
- Harvest generation
- Search indexing
- Embedding generation
Processors never modify source artifacts.
They only generate derived artifacts.
---
## 5. Knowledge Repository
The repository contains:
- Source artifacts
- Derived artifacts
- Metadata
- Relationships
This becomes the canonical knowledge base for Project Thoth.
---
## 6. Consumers
Consumers interact with the repository.
Examples:
- Local LLMs
- Search engines
- Knowledge assistants
- Documentation systems
- Human readers
Consumers should never depend on the original AI platform.
---
# Capture Connector Architecture
Every Capture Connector implements the same logical pipeline.
```
Optional Preprocessing
Conversation Discovery
Intermediate Representation
Content Transformation
Markdown Serialization
conversation.md
```
This architecture is defined by ADR-002.
Only the Discovery stage is expected to be platform-specific.
---
# Processing Philosophy
Project Thoth separates preservation from interpretation.
```
Capture Once
Preserve Forever
Process Many Times
```
This allows improvements to processors without requiring the original source to be recaptured.
---
# Canonical Artifacts
Project Thoth distinguishes between canonical artifacts and generated artifacts.
## Canonical
Examples:
- conversation.md
- Source attachments
- Images
- Audio
- Video
Canonical artifacts are permanent.
---
## Generated
Examples:
- Metadata
- Manifests
- Harvests
- Embeddings
- Search indexes
Generated artifacts may be deleted and regenerated at any time.
---
# Repository Structure
A typical Project Thoth repository contains:
```
.thoth/
Project self-description
applications/
Executable applications
processors/
Artifact processors
specifications/
Canonical document specifications
docs/
Architecture and ADRs
codex/
Engineering work orders
archive/
Captured source material
generated/
Derived artifacts
```
Implementation details may evolve while preserving the architectural boundaries described in this document.
---
# Architectural Boundaries
Each layer has a single responsibility.
| Layer | Responsibility |
|---------|----------------|
| Capture Connectors | Preserve source material |
| Archive | Store canonical source |
| Processors | Generate derived artifacts |
| Repository | Organize knowledge |
| Consumers | Read knowledge |
No layer should assume the responsibilities of another.
---
# Design Goals
The architecture is intended to achieve:
- Long-term preservation
- Platform independence
- Deterministic processing
- Reproducibility
- Testability
- Portability
- AI interoperability
---
# Relationship to Other Documents
This document defines **how Project Thoth is organized**.
Related documents include:
- **00-project-state.md** — Current implementation status.
- **20-philosophy.md** — Guiding principles.
- **30-glossary.md** — Canonical terminology.
- **40-coding-standards.md** — Development standards.
- **ADR-001** — Project structure.
- **ADR-002** — Capture connector pipeline.
- **ADR-003** — Capture connector architecture.
Together these documents form the canonical self-description of the project.
+321
View File
@@ -0,0 +1,321 @@
# Project Thoth Philosophy
**Version:** 1.0
**Status:** Living Document
---
# Purpose
Project Thoth is founded on a simple belief:
> **Knowledge should outlive the systems that create it.**
Artificial intelligence systems, software platforms, file formats, and vendors will continue to evolve.
Human knowledge should not depend on any of them.
Project Thoth exists to preserve human reasoning in a portable, deterministic, and vendor-independent form.
---
# Core Philosophy
Project Thoth is not an AI project.
It is a knowledge preservation project that happens to use AI.
Artificial intelligence is a consumer of knowledge, not its owner.
The archive is the product.
Everything else is replaceable.
---
# Guiding Principles
## 1. The Archive is Canonical
The archive is the source of truth.
Not ChatGPT.
Not Claude.
Not Gemini.
Not a vector database.
Not an embedding model.
Every generated artifact should be reproducible from the archive.
---
## 2. Source Artifacts are Permanent
Original captured material should never be destroyed.
Examples include:
- Conversations
- Images
- Audio
- Video
- Attachments
Source artifacts represent historical evidence.
They cannot be recreated once lost.
---
## 3. Generated Artifacts are Disposable
Metadata.
Embeddings.
Search indexes.
Harvests.
Summaries.
These exist only to improve access to the archive.
They should always be considered disposable.
If a better algorithm is developed tomorrow, they should simply be regenerated.
---
## 4. Capture Once. Process Forever.
Knowledge capture should occur only once.
Processing should occur as many times as necessary.
New processors should never require recollecting historical conversations.
---
## 5. Determinism Over Magic
Project Thoth prefers deterministic transformations over opaque intelligence.
The same input should produce the same output.
When AI reasoning is introduced, it should be clearly separated from preservation.
---
## 6. AI Providers are Interchangeable
No AI vendor should become part of the archive.
ChatGPT.
Claude.
Gemini.
Copilot.
Open WebUI.
Future models.
These are simply sources of knowledge.
The archive must remain usable regardless of which systems disappear.
---
## 7. Markdown is the Universal Language
Project Thoth prefers open, human-readable formats.
Markdown provides:
- longevity
- portability
- version control
- interoperability
- readability
Whenever practical, proprietary formats should be transformed into Markdown.
---
## 8. Knowledge Should be Human-Readable
Every canonical artifact should be understandable without specialized software.
A person should be able to browse the archive with nothing more than a text editor.
Software enhances the archive.
It should never become a prerequisite for understanding it.
---
## 9. Preserve Before Improving
Improvement can always happen later.
Preservation cannot.
When faced with a choice between:
- perfect processing
- complete preservation
Project Thoth chooses preservation.
Incomplete but preserved information is preferable to perfectly formatted information that has silently lost content.
---
## 10. Separation of Responsibilities
Every component should have one responsibility.
Capture preserves.
Processors transform.
Indexes organize.
AI reasons.
Mixing these concerns creates unnecessary complexity.
---
## 11. Reproducibility Matters
Every generated artifact should be reproducible.
A future developer should be able to regenerate:
- metadata
- manifests
- harvests
- search indexes
- embeddings
from the canonical archive without requiring access to the original AI platform.
---
## 12. Preserve Context, Not Just Content
Facts without context lose meaning.
Whenever possible, preserve:
- ordering
- timestamps
- relationships
- authorship
- source platform
- surrounding discussion
Knowledge is more than isolated documents.
It is the reasoning that connects them.
---
## 13. The Project Describes Itself
Project Thoth repositories should be self-describing.
A developer.
An AI assistant.
An automated process.
All should be able to understand the project by reading its canonical documents.
Knowledge about the project belongs inside the project.
---
## 14. Architecture Should Outlive Implementation
Applications will be rewritten.
Frameworks will change.
Programming languages will evolve.
The architecture should remain recognizable.
Implementation serves architecture.
Architecture serves philosophy.
---
# What Project Thoth is Not
Project Thoth is not:
- another note-taking application
- another vector database
- another chatbot
- another RAG framework
- another AI assistant
These technologies may become components of a Project Thoth ecosystem.
They are not its purpose.
---
# Long-Term Vision
Project Thoth seeks to become a durable knowledge architecture.
A place where:
- conversations survive software vendors
- ideas survive AI models
- archives survive databases
- knowledge survives technology shifts
The objective is not simply to remember information.
It is to preserve reasoning.
---
# Decision Filter
When making architectural decisions, ask:
1. Does this strengthen the archive?
2. Is the result deterministic?
3. Can it be regenerated?
4. Does it reduce vendor dependence?
5. Will this still make sense in ten years?
If the answer to these questions is "yes," the decision is likely aligned with the philosophy of Project Thoth.
---
# Final Principle
Technology changes.
Knowledge endures.
Project Thoth exists to ensure that the second is never held hostage by the first.
+461
View File
@@ -0,0 +1,461 @@
# Project Thoth Glossary
**Version:** 1.0
**Status:** Canonical Terminology
---
# Purpose
This glossary defines the canonical vocabulary used throughout Project Thoth.
The purpose of this document is to ensure that humans, AI assistants, and automated tooling use the same terminology consistently.
Where possible, each concept has exactly one preferred term.
---
# A
## Archive
The permanent collection of canonical source artifacts captured by Project Thoth.
The archive is the authoritative source of knowledge for the project.
Examples include:
- conversations
- images
- attachments
- audio
- video
The archive is immutable.
---
## Artifact
Any file produced or managed by Project Thoth.
Artifacts fall into two categories:
- Canonical Artifacts
- Generated Artifacts
---
# C
## Canonical Artifact
An artifact considered the permanent source of truth.
Examples:
- `conversation.md`
- Original attachments
- Images
- Audio
- Video
Canonical artifacts are preserved indefinitely.
---
## Capture
The act of preserving information from an external source.
Capture is deterministic.
Capture never interprets the information being preserved.
---
## Capture Connector
A platform-specific adapter responsible for preserving source material.
Responsibilities include:
- Discovering conversations
- Preserving formatting
- Preserving ordering
- Producing canonical source artifacts
Capture Connectors never perform semantic analysis or AI reasoning.
Examples:
- ChatGPT Connector
- Claude Connector
- Gemini Connector
- Open WebUI Connector
---
## Capture Pipeline
The sequence of stages that transforms external content into canonical Project Thoth artifacts.
Defined in ADR-002.
```
Discovery
Intermediate Representation
Transformation
Serialization
```
---
## Collection
A logical grouping of related canonical artifacts.
Collections may represent:
- projects
- research topics
- books
- clients
- domains
Collections organize knowledge without altering source artifacts.
---
## Conversation
A chronological exchange between one or more participants.
A conversation is preserved as a canonical artifact.
---
## Conversation Turn
A single exchange within a conversation.
A turn represents exactly one logical contribution from a participant.
A turn may contain:
- paragraphs
- lists
- tables
- images
- code blocks
- attachments
A turn is never divided into multiple Project Thoth messages.
---
# D
## Derived Artifact
An artifact generated from canonical source material.
Derived artifacts may always be regenerated.
Examples include:
- metadata
- manifests
- harvests
- embeddings
- search indexes
---
## Discovery
The process of locating logical content within an external system.
Discovery is platform-specific.
Discovery does not perform formatting or interpretation.
---
# G
## Generated Artifact
See **Derived Artifact**.
---
# H
## Harvest
A synthesized knowledge artifact produced from one or more canonical sources.
Harvests represent interpretation rather than preservation.
Harvests are always generated.
---
# I
## Intermediate Representation (IR)
The platform-neutral model produced after discovery.
The IR separates platform-specific extraction from platform-independent processing.
The IR is not normally stored permanently.
---
# K
## Knowledge Repository
The organized collection of canonical and generated artifacts managed by Project Thoth.
The repository represents the project's complete preserved knowledge.
---
# M
## Manifest
A generated document describing a collection of artifacts.
A manifest typically records:
- included files
- relationships
- metadata
- processing status
Manifests are generated and may be regenerated at any time.
---
## Markdown
The canonical document format used by Project Thoth.
Markdown is preferred because it is:
- human-readable
- portable
- version-control friendly
- widely supported
---
## Metadata
Structured information describing an artifact.
Examples include:
- title
- author
- capture date
- source platform
- tags
- relationships
Metadata is generated from canonical artifacts.
---
# P
## Philosophy
The enduring principles that guide architectural decisions.
The philosophy explains *why* Project Thoth is designed the way it is.
---
## Processor
A platform-independent component that transforms canonical artifacts into generated artifacts.
Examples include:
- Metadata Generator
- Manifest Generator
- Harvest Generator
Processors never modify canonical artifacts.
---
## Project Self-Description
The collection of documents contained within the `.thoth` directory.
These documents allow humans and AI systems to understand the project without relying on external knowledge.
---
# R
## Repository
The complete Project Thoth project, including:
- applications
- archive
- processors
- documentation
- specifications
- work orders
- canonical project description
---
# S
## Serialization
The process of writing transformed content into a canonical Project Thoth document.
Examples include:
- `conversation.md`
- metadata files
- manifests
---
## Source Artifact
A canonical artifact captured directly from an external source.
Source artifacts are never regenerated.
---
## Source Platform
The external system from which information was captured.
Examples:
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
---
## Specification
A document defining the canonical structure of Project Thoth artifacts.
Specifications describe formats rather than implementations.
---
# T
## Transformation
The process of converting source content into canonical Project Thoth formats.
Examples include:
- HTML → Markdown
- DOM → Intermediate Representation
Transformation does not alter meaning.
---
## Turn Renderer
A component responsible for converting one Conversation Turn into canonical Markdown.
A Turn Renderer never discovers conversation turns.
---
# V
## Vault
A user-managed knowledge repository containing canonical Project Thoth artifacts.
The vault is tool-independent.
Applications such as Obsidian, VS Code, or future tools may operate on the same vault.
The vault is not defined by any particular application.
---
# W
## Work Order
A structured engineering document describing a discrete unit of implementation work.
A Work Order contains:
- context
- objectives
- constraints
- acceptance criteria
- architectural guidance
Work Orders are intended to be executable by either human developers or AI assistants.
Work Orders are stored under the `codex/` directory.
---
# Canonical Terminology
The following preferred terms should be used consistently throughout Project Thoth.
| Preferred | Avoid |
|-----------|-------|
| Archive | Database (when referring to preserved knowledge) |
| Capture Connector | Scraper |
| Conversation Turn | Message Block |
| Derived Artifact | Output File |
| Discovery | DOM Parsing |
| Intermediate Representation | Temporary Object |
| Markdown Serialization | Export |
| Processor | Script |
| Source Artifact | Raw File |
| Vault | Obsidian Vault |
---
# Naming Principles
When introducing new terminology:
1. Prefer nouns over verbs.
2. Use one canonical term for each concept.
3. Avoid vendor-specific language.
4. Prefer platform-neutral terminology.
5. Preserve consistency over novelty.
---
# Final Principle
A shared vocabulary is part of the project's architecture.
When every contributor—human or AI—uses the same words to describe the same concepts, communication becomes more precise, architectural decisions become easier to understand, and knowledge becomes more durable.
+429
View File
@@ -0,0 +1,429 @@
# Project Thoth Coding Standards
**Version:** 1.0
**Status:** Canonical Development Standard
---
# Purpose
This document defines the engineering standards used throughout Project Thoth.
The objective is not merely to produce working software, but to produce software that remains understandable, maintainable, and portable over time.
These standards apply equally to:
- Human developers
- AI-assisted development
- Autonomous development agents
When implementation choices conflict with these standards, architectural decisions (ADRs) take precedence.
---
# Core Principles
## 1. Architecture Before Code
Every significant implementation should begin with architecture.
If a design decision is expected to persist beyond a single implementation, document it in an Architecture Decision Record (ADR) before coding.
Code should implement architecture—not define it.
---
## 2. Prefer Refactoring Over Patching
When repeated fixes indicate a flawed design, refactor rather than accumulate patches.
Signs that refactoring is warranted include:
- duplicate logic
- increasing conditional complexity
- repeated bug fixes in the same area
- unclear module responsibilities
- growing technical debt
The goal is long-term simplicity, not short-term completion.
---
## 3. Single Responsibility
Every module should have one clear responsibility.
Examples:
Good:
- Conversation Discovery
- HTML-to-Markdown Conversion
- Markdown Serialization
- Metadata Generation
Poor:
- ExtractConversationAndGenerateMarkdownAndDownload()
If a component's name requires multiple conjunctions ("and", "or"), it likely has more than one responsibility.
---
## 4. Separation of Concerns
Capture, transformation, processing, and presentation are independent concerns.
Each should exist in its own layer.
No layer should assume the responsibilities of another.
---
## 5. Deterministic Behavior
Given the same input, Project Thoth components should produce the same output.
Randomness, hidden state, and side effects should be avoided unless explicitly required.
---
# Repository Organization
The repository should separate concerns by purpose rather than technology.
Typical structure:
```text
.thoth/
applications/
archive/
codex/
docs/
processors/
specifications/
```
Each top-level directory should represent a distinct architectural concern.
---
# Module Design
Modules should be:
- cohesive
- loosely coupled
- independently testable
Modules should expose small, well-defined interfaces.
Avoid exposing internal implementation details.
---
# Function Design
Functions should:
- perform one task
- have descriptive names
- avoid unnecessary side effects
- return predictable results
Prefer composition over deeply nested logic.
---
# Error Handling
Errors should be:
- explicit
- actionable
- recoverable when practical
Never silently discard data.
When partial processing is possible:
- preserve available information
- report the failure
- continue processing where safe
Preservation takes precedence over perfection.
---
# Logging
Logs should assist debugging without becoming part of the application's behavior.
Log messages should answer:
- What happened?
- Why did it happen?
- What should the developer investigate next?
Avoid excessive or repetitive logging.
---
# Documentation
Document architecture rather than implementation details.
Prefer:
- ADRs
- specifications
- interfaces
- module responsibilities
Avoid comments that merely repeat what the code already expresses.
Comments should explain *why*, not *what*.
---
# Naming
Choose names that reflect business concepts rather than implementation details.
Prefer:
- ConversationTurn
- CaptureConnector
- ManifestGenerator
Avoid:
- TempData
- Helper
- Utils
- Stuff
Names should communicate intent.
---
# Interfaces
Define interfaces at architectural boundaries.
Examples include:
- Capture Connector
- Processor
- Serializer
Interfaces should remain stable even when implementations change.
---
# Dependencies
Minimize external dependencies.
Before adding a dependency, ask:
- Does the standard library already solve this?
- Does this dependency simplify architecture?
- Is it actively maintained?
- Can the dependency be removed later without significant effort?
Favor long-lived, well-supported libraries.
---
# File Formats
Project Thoth prefers open formats.
Preferred formats include:
- Markdown
- YAML
- JSON
- Plain text
Avoid introducing proprietary formats unless there is a compelling architectural reason.
---
# Testing
Testing should occur at multiple levels.
Examples:
- Unit tests
- Integration tests
- End-to-end workflow tests
Where practical, preserve representative fixtures for regression testing.
Regression tests are especially important for parsers and capture connectors.
---
# AI-Assisted Development
AI is a development assistant, not an architectural authority.
AI-generated code should be reviewed for:
- correctness
- simplicity
- architectural alignment
- maintainability
AI should implement established architecture rather than invent new architecture during implementation.
---
# Work Orders
Implementation work should be described using Work Orders.
A Work Order should include:
- context
- objective
- constraints
- deliverables
- acceptance criteria
- architectural references
Work Orders should be executable with minimal additional explanation.
---
# Refactoring
Refactoring should preserve observable behavior while improving internal structure.
Typical reasons include:
- simplifying design
- improving readability
- reducing duplication
- improving testability
- aligning implementation with updated architecture
Refactoring should not introduce unrelated feature work.
---
# Code Reviews
Reviews should evaluate:
1. Architectural alignment
2. Correctness
3. Readability
4. Maintainability
5. Simplicity
The primary question is not:
> "Does it work?"
Instead ask:
> "Will another developer understand this in two years?"
---
# Performance
Optimize for clarity first.
Optimize for performance when:
- measurement identifies a bottleneck
- scalability requires it
- architectural goals demand it
Avoid premature optimization.
---
# Security
Treat external input as untrusted.
Validate:
- user input
- downloaded content
- file paths
- URLs
- serialized data
Avoid embedding secrets in source code or repositories.
---
# Backward Compatibility
When changing canonical formats:
- preserve compatibility where practical
- document breaking changes
- update specifications
- record architectural decisions in ADRs
---
# Technical Debt
Technical debt should be visible.
Known debt should be documented rather than ignored.
Temporary solutions should include:
- rationale
- limitations
- expected future resolution
---
# Definition of Done
Work is considered complete when:
- implementation satisfies the Work Order
- acceptance criteria are met
- architecture remains consistent
- documentation is updated if required
- tests pass (where applicable)
- no known regressions have been introduced
Completion is measured by quality, not by the number of lines of code written.
---
# Coding Philosophy
Project Thoth values software that is:
- understandable
- deterministic
- modular
- portable
- maintainable
- testable
The objective is to create software that remains useful long after the original implementation has evolved.
---
# Final Principle
Good software preserves knowledge.
Great software preserves understanding.
Every design decision, module, and line of code should make the project easier for the next developer—or the next AI assistant—to understand.
+419
View File
@@ -0,0 +1,419 @@
# Project Thoth Roadmap
**Version:** 1.0
**Status:** Living Document
---
# Purpose
This roadmap describes the long-term evolution of Project Thoth.
Unlike `00-project-state.md`, which describes the project's current implementation, this document describes the intended direction of the platform.
The roadmap is intentionally high-level.
Implementation details belong in Work Orders, ADRs, and project-specific documentation.
---
# Vision
Project Thoth will become a platform for preserving, organizing, processing, and retrieving human knowledge independent of any AI provider or software vendor.
Its primary objective is to create a durable knowledge architecture where information can be captured once and continuously reprocessed as technology evolves.
---
# Guiding Principles
The roadmap follows several principles:
- Build vertically before horizontally.
- Preserve before processing.
- Prefer deterministic systems.
- Keep platform-specific code isolated.
- Favor reusable processors over application-specific logic.
- Introduce AI only after preservation is complete.
---
# Phase 1 — Foundation
## Objective
Establish the architectural and technical foundations of Project Thoth.
### Deliverables
- Reference Architecture
- Project structure
- ADR framework
- Repository standards
- Canonical specifications
- `.thoth` project self-description
- Initial Work Order framework
### Success Criteria
Project architecture is stable and documented.
---
# Phase 2 — Capture MVP
## Objective
Capture conversations from ChatGPT into canonical Project Thoth artifacts.
### Deliverables
- Browser extension
- ChatGPT connector
- Conversation discovery
- DOM-to-Markdown transformation
- Markdown serialization
- Download support
- Basic user feedback
### Success Criteria
Conversations can be captured with high fidelity into `conversation.md`.
---
# Phase 3 — Capture Framework
## Objective
Generalize the ChatGPT implementation into a reusable connector framework.
### Deliverables
- Capture Connector interface
- Canonical Conversation Model
- Shared HTML-to-Markdown engine
- Shared Markdown serializer
- Connector testing framework
### Target Connectors
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
### Success Criteria
New connectors primarily require implementation of platform-specific discovery.
---
# Phase 4 — Processor Framework
## Objective
Build reusable processors that operate on canonical source artifacts.
### Initial Processors
- Metadata Generator
- Manifest Generator
- Harvest Generator
### Future Processors
- Relationship extraction
- Citation generation
- Cross-reference generation
- Knowledge graph generation
### Success Criteria
Processors are independent of capture connectors.
---
# Phase 5 — Archive Management
## Objective
Create a durable archive for canonical artifacts.
### Features
- Archive organization
- Collection management
- Version tracking
- Attachment management
- Integrity validation
### Success Criteria
The archive becomes the canonical source of knowledge.
---
# Phase 6 — Knowledge Repository
## Objective
Organize captured knowledge into a coherent repository.
### Features
- Cross-linking
- Collections
- Relationships
- Metadata indexing
- Search preparation
### Success Criteria
Knowledge can be navigated independently of its original source platform.
---
# Phase 7 — Search
## Objective
Provide deterministic search across the repository.
### Features
- Full-text search
- Metadata search
- Collection search
- Relationship search
- Citation search
### Success Criteria
Users can reliably locate preserved knowledge without AI assistance.
---
# Phase 8 — AI Integration
## Objective
Enable AI systems to consume the repository.
### Features
- Local LLM integration
- Context packaging
- Prompt generation
- Retrieval interfaces
- Knowledge injection
### Target Platforms
- Ollama
- Open WebUI
- ChatGPT
- Claude
- Gemini
### Success Criteria
AI systems become consumers of the archive rather than owners of knowledge.
---
# Phase 9 — Knowledge Operations
## Objective
Support organizational knowledge management.
### Features
- Team repositories
- Shared collections
- Governance
- Retention policies
- Workflow automation
- Knowledge health metrics
### Success Criteria
Organizations can manage institutional knowledge using Project Thoth.
---
# Phase 10 — Project Thoth Platform
## Objective
Transform Project Thoth into a complete knowledge platform.
### Features
- Native applications
- Plugin architecture
- Processor marketplace
- Connector marketplace
- Automation framework
- Scheduled processing
- Background services
### Success Criteria
Project Thoth becomes an extensible ecosystem.
---
# Parallel Work Streams
Several work streams evolve throughout the roadmap.
---
## Capture Connectors
Examples include:
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
- Cursor
- GitHub Copilot Chat
- Future AI platforms
---
## Processors
Examples include:
- Metadata
- Manifest
- Harvest
- Summaries
- Search indexing
- Embeddings
- Relationship analysis
---
## Specifications
Examples include:
- Conversation Specification
- Metadata Specification
- Manifest Specification
- Harvest Specification
- Package Specification
---
## Applications
Potential applications include:
- Browser Capture
- Desktop Archive Manager
- Knowledge Explorer
- Search Portal
- Processor Runner
- Administration Console
---
# Future Capabilities
Potential future capabilities include:
- Automatic conversation synchronization
- Email capture
- Document ingestion
- PDF processing
- Audio transcription
- Video indexing
- OCR
- Knowledge graph visualization
- Timeline reconstruction
- Citation management
- Research workspaces
- Multi-user collaboration
These features are intentionally deferred until the core preservation architecture is mature.
---
# Non-Goals
Project Thoth is not intended to become:
- another chatbot
- another note-taking application
- another vector database
- another proprietary knowledge silo
These technologies may be integrated where appropriate, but they are not the project's primary purpose.
---
# Milestone Philosophy
Each milestone should deliver independently valuable functionality.
The project should remain useful even if later phases are never completed.
Earlier phases should minimize assumptions about later implementation choices.
---
# Success Measures
Progress should be measured by:
- Preservation fidelity
- Architectural stability
- Deterministic processing
- Platform independence
- Ease of extension
- Long-term maintainability
Feature count is not a primary measure of success.
---
# Living Roadmap
This roadmap is expected to evolve.
New phases may be added.
Existing phases may be reordered.
However, changes should preserve the project's guiding philosophy:
1. Capture.
2. Preserve.
3. Process.
4. Organize.
5. Retrieve.
6. Reason.
Reasoning is the final stage—not the first.
---
# Final Vision
Project Thoth is not simply a collection of tools.
It is an architecture for preserving human reasoning.
The long-term goal is to ensure that knowledge remains accessible, portable, and understandable regardless of which AI systems, software platforms, or technologies come and go.
Technology will continue to evolve.
The archive should not have to.
+7
View File
@@ -0,0 +1,7 @@
They should be able to answer:
What is Project Thoth?
Why does it exist?
Is this something I want to use?
How do I build it?
How do I contribute?
+33
View File
@@ -0,0 +1,33 @@
# Project Thoth Capture
This folder contains a minimal Manifest V3 browser extension skeleton for Chrome and Edge.
## Local installation
1. Open Chrome or Edge.
2. Navigate to `chrome://extensions` or `edge://extensions`.
3. Enable "Developer mode".
4. Click "Load unpacked".
5. Select the `applications/chatgpt-capture` folder in this repository.
## Current behavior
- The toolbar button is visible in the browser toolbar.
- Clicking it checks whether the active tab is a ChatGPT page.
- On a supported ChatGPT tab, the extractor runs in the page context and sends a capture payload back to the service worker.
- On unsupported tabs, the extension logs a simple warning and does not inject the script.
- Future extraction logic can be expanded in `src/chatgptExtractor.js` and `src/background.js`.
## How to test against a live ChatGPT page
1. Load the unpacked extension from this folder in Chrome or Edge developer mode.
2. Open a real ChatGPT conversation in a tab.
3. Open the extension's service worker console:
- Visit `chrome://extensions` or `edge://extensions`.
- Find "Project Thoth Capture" and click "service worker" or "inspect views".
4. Click the extension toolbar button while the ChatGPT tab is active.
5. Inspect the console for:
- `ChatGPT extractor payload:`
- `Received capture payload:`
6. If the page is detected as a conversation, the payload should include `title`, `url`, `capturedAt`, and `messages`.
- If no messages are found, the payload will include an `error` field instead.
@@ -0,0 +1 @@
placeholder
@@ -0,0 +1 @@
placeholder
@@ -0,0 +1 @@
placeholder
@@ -0,0 +1,41 @@
{
"manifest_version": 3,
"name": "Project Thoth Capture",
"description": "Capture ChatGPT conversations for Project Thoth.",
"version": "0.1.0",
"permissions": ["activeTab", "scripting", "downloads"],
"host_permissions": [
"https://chatgpt.com/*",
"https://chat.openai.com/*"
],
"action": {
"default_title": "Save to Project Thoth",
"default_icon": {
"16": "icons/icon16.png",
"48": "icons/icon48.png",
"128": "icons/icon128.png"
}
},
"background": {
"service_worker": "src/background.js",
"type": "module"
},
"web_accessible_resources": [
{
"resources": [
"src/chatgptConnectorRuntime.js",
"src/connectors/chatgpt/chatgptConnector.js",
"src/connectors/chatgpt/chatgptDiscovery.js",
"src/connectors/chatgpt/chatgptPreprocessor.js",
"src/connectors/chatgpt/chatgptTurnRenderer.js",
"src/core/conversationModel.js",
"src/core/htmlToMarkdown.js",
"src/core/markdownSerializer.js"
],
"matches": [
"https://chatgpt.com/*",
"https://chat.openai.com/*"
]
}
]
}
@@ -0,0 +1,162 @@
import { createConversationFilename } from "./filename.js";
import { serializeConversationToMarkdown } from "./core/markdownSerializer.js";
const CHATGPT_HOSTS = ["https://chatgpt.com/", "https://chat.openai.com/"];
const pendingDownloadIds = new Set();
const USER_MESSAGES = {
NOT_ON_CHATGPT: "Open a ChatGPT conversation before saving to Project Thoth.",
NO_CONVERSATION: "No ChatGPT conversation was detected on this page.",
CAPTURE_FAILED: "Capture failed. Try scrolling through the conversation and capturing again.",
DOWNLOAD_FAILED: "The conversation was captured, but the file could not be saved.",
SUCCESS: "Saved to Downloads."
};
function showStatus(tabId, message, badgeText = "") {
if (typeof tabId === "number") {
chrome.action.setTitle({ tabId, title: message });
chrome.action.setBadgeText({ tabId, text: badgeText });
if (badgeText) {
setTimeout(() => {
chrome.action.setBadgeText({ tabId, text: "" });
chrome.action.setTitle({ tabId, title: "Save to Project Thoth" });
}, 3000);
}
} else {
console.log(`Status: ${message}`);
}
}
function isChatGPTUrl(url) {
if (!url) {
return false;
}
return CHATGPT_HOSTS.some((host) => url.startsWith(host));
}
chrome.action.onClicked.addListener(async (tab) => {
const tabUrl = tab?.url || "unknown";
const tabTitle = tab?.title || "unknown";
console.log("Project Thoth capture clicked");
console.log(`Tab URL: ${tabUrl}`);
console.log(`Tab title: ${tabTitle}`);
if (!tab?.id || !isChatGPTUrl(tabUrl)) {
console.warn("Capture skipped: active tab is not a ChatGPT page.");
showStatus(tab.id, USER_MESSAGES.NOT_ON_CHATGPT, "ERR");
return;
}
try {
console.log("[Thoth] capture requested");
await chrome.scripting.executeScript({
target: { tabId: tab.id },
func: async (runtimeUrl) => {
const { captureChatGPTConversation } = await import(runtimeUrl);
await captureChatGPTConversation();
},
args: [chrome.runtime.getURL("src/chatgptConnectorRuntime.js")]
});
console.log("Injected ChatGPT capture script.");
} catch (error) {
console.error("Failed to inject capture script:", error);
showStatus(tab.id, USER_MESSAGES.CAPTURE_FAILED, "ERR");
}
});
chrome.runtime.onMessage.addListener((message, sender) => {
const tabId = sender?.tab?.id;
if (message?.type === "CHATGPT_CAPTURE_STATUS") {
const messageText = message?.payload?.message || "Preparing conversation for capture...";
showStatus(tabId, messageText, "...");
return;
}
if (message?.type === "CHATGPT_CAPTURE_PAYLOAD") {
console.log("Received capture payload:", message.payload);
console.log("Sender tab URL:", sender?.tab?.url || "unknown");
if (message.payload?.error || !Array.isArray(message.payload?.messages) || message.payload.messages.length === 0) {
console.warn("No ChatGPT conversation was detected on this page.", message.payload);
showStatus(tabId, USER_MESSAGES.NO_CONVERSATION, "ERR");
return;
}
console.log("[Thoth] extraction completed");
try {
const markdown = serializeConversationToMarkdown({
title: message.payload.title,
sourcePlatform: message.payload.sourcePlatform,
url: message.payload.url,
capturedAt: message.payload.capturedAt,
warning: message.payload.warning,
turns: (message.payload.messages || []).map((messageItem) => ({
role: messageItem.role,
markdown: messageItem.content,
captureStatus: messageItem.captureStatus || "captured"
}))
});
const filename = createConversationFilename(message.payload);
const blob = new Blob([markdown], { type: "text/markdown;charset=utf-8" });
const downloadUrl =
typeof URL.createObjectURL === "function"
? URL.createObjectURL(blob)
: `data:text/markdown;charset=utf-8,${encodeURIComponent(markdown)}`;
console.log("Normalized markdown output:\n" + markdown);
console.log("Generated filename:", filename);
console.log("[Thoth] markdown generated");
console.log("[Thoth] download requested");
chrome.downloads.download(
{
url: downloadUrl,
filename,
saveAs: false,
conflictAction: "uniquify"
},
(downloadId) => {
if (typeof URL.revokeObjectURL === "function" && downloadUrl.startsWith("blob:")) {
URL.revokeObjectURL(downloadUrl);
}
if (chrome.runtime.lastError || !downloadId) {
console.error(
"[Thoth] download failed:",
chrome.runtime.lastError?.message || "No download ID was returned."
);
showStatus(tabId, USER_MESSAGES.DOWNLOAD_FAILED, "ERR");
return;
}
console.log("Markdown download started with ID:", downloadId);
pendingDownloadIds.add(downloadId);
showStatus(tabId, USER_MESSAGES.SUCCESS, "OK");
}
);
} catch (error) {
console.error("Failed to normalize capture payload:", error);
showStatus(tabId, USER_MESSAGES.CAPTURE_FAILED, "ERR");
}
}
});
chrome.downloads.onChanged.addListener((delta) => {
if (!pendingDownloadIds.has(delta.id)) {
return;
}
if (delta.state?.current === "complete") {
console.log(`[Thoth] download completed: ${delta.id}`);
pendingDownloadIds.delete(delta.id);
} else if (delta.error?.current) {
console.error(`[Thoth] download failed: ${delta.error.current}`);
pendingDownloadIds.delete(delta.id);
}
});
@@ -0,0 +1,49 @@
import { captureConversation } from "./connectors/chatgpt/chatgptConnector.js";
export async function captureChatGPTConversation() {
console.log("[Thoth] connector runtime loaded");
try {
const result = await captureConversation({
title: document.title,
url: window.location.href,
capturedAt: new Date().toISOString()
});
if (typeof chrome !== "undefined" && chrome.runtime && chrome.runtime.sendMessage) {
chrome.runtime.sendMessage({
type: "CHATGPT_CAPTURE_PAYLOAD",
payload: {
sourcePlatform: result.conversation.sourcePlatform,
title: result.conversation.title,
url: result.conversation.url,
capturedAt: result.conversation.capturedAt,
warning: result.conversation.warning,
messages: result.conversation.turns.map((turn) => ({
role: turn.role,
content: turn.markdown,
captureStatus: turn.captureStatus,
turnIndex: turn.turnIndex
})),
debug: result.conversation.debug
}
});
}
} catch (error) {
console.error("[Thoth] extraction failed:", error);
if (typeof chrome !== "undefined" && chrome.runtime && chrome.runtime.sendMessage) {
chrome.runtime.sendMessage({
type: "CHATGPT_CAPTURE_PAYLOAD",
payload: {
sourcePlatform: "ChatGPT",
title: document.title || "ChatGPT Conversation",
url: window.location.href,
capturedAt: new Date().toISOString(),
messages: [],
error: error.message || "ChatGPT capture failed."
}
});
}
}
}
@@ -0,0 +1,994 @@
(function () {
const MESSAGE_CONTAINER_SELECTORS = [
'[data-message-author-role]',
'[data-testid*="message"]',
'[data-testid*="conversation-turn"]',
'[data-testid*="assistant"]',
'[data-testid*="user"]',
'[data-message-id]',
'.message',
'.conversation-turn',
'.assistant-message',
'.user-message'
];
const SKIP_SELECTORS = [
'nav',
'aside',
'header',
'footer',
'form',
'input',
'textarea',
'button',
'[role="button"]',
'[role="textbox"]',
'[contenteditable="true"]',
'select',
'svg',
'script',
'style'
];
const UI_CHROME_SELECTORS = [
'button',
'[role="button"]',
'[aria-label*="copy" i]',
'[aria-label*="clipboard" i]',
'[aria-label*="feedback" i]',
'[data-testid*="copy" i]',
'[data-testid*="feedback" i]',
'[data-testid*="toolbar" i]',
'[data-testid*="model" i]',
'[data-testid*="timestamp" i]',
'[class*="copy" i]',
'[class*="feedback" i]',
'[class*="toolbar" i]',
'[class*="timestamp" i]'
];
function normalizeWhitespace(text) {
if (!text) {
return "";
}
return text
.replace(/\u00a0/g, " ")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.replace(/[ \t]{2,}/g, " ")
.trim();
}
function isHiddenOrInvisible(node) {
if (!node || typeof node.matches !== "function") {
return true;
}
if (node.matches('[aria-hidden="true"], [hidden], .hidden, .sr-only')) {
return true;
}
if (typeof node.getBoundingClientRect !== "function") {
return false;
}
try {
const rect = node.getBoundingClientRect();
return rect.width === 0 && rect.height === 0;
} catch (error) {
return false;
}
}
function hasMeaningfulText(node) {
if (!node) {
return false;
}
const text = (node.textContent || "").replace(/\s+/g, "").trim();
return text.length >= 6;
}
function isChromeUi(node) {
if (!node || typeof node.matches !== "function") {
return true;
}
if (node.matches(SKIP_SELECTORS.join(", ")) || node.closest(SKIP_SELECTORS.join(", "))) {
return true;
}
return false;
}
function getConversationTitle() {
const titleSelectors = [
'[data-testid*="title"]',
'[data-testid*="conversation-title"]',
'main h1',
'main h2',
'[role="heading"]'
];
for (const selector of titleSelectors) {
const element = document.querySelector(selector);
if (element && hasMeaningfulText(element)) {
return normalizeWhitespace(element.textContent || "");
}
}
return document.title || "ChatGPT Conversation";
}
function getConversationRoot() {
const possibleRoots = [
document.querySelector('main[role="main"]'),
document.querySelector('main'),
document.querySelector('[role="main"]'),
document.querySelector('[data-testid*="conversation"]'),
document.querySelector('[data-testid*="messages"]'),
document.querySelector('[data-testid*="thread"]'),
document.querySelector('[data-testid*="chat"]'),
document.querySelector('[class*="conversation"]'),
document.querySelector('[class*="messages"]'),
document.querySelector('[class*="thread"]'),
document.body
];
return possibleRoots.find(Boolean) || document.body;
}
function getElementFingerprint(node) {
return normalizeWhitespace((node.textContent || "").replace(/\s+/g, " ")).toLowerCase();
}
function isShellContainer(node, root) {
if (!node || !root || node === root) {
return false;
}
const label = [
node.getAttribute("aria-label"),
node.getAttribute("data-testid"),
node.className,
node.textContent || ""
].join(" ").toLowerCase();
const shellHints = [
"sidebar",
"history",
"nav",
"navigation",
"composer",
"input",
"textarea",
"search",
"account",
"project",
"share",
"settings",
"footer",
"menu",
"toolbar",
"new chat",
"skip to content",
"window.__oai_loghtml"
];
if (shellHints.some((hint) => label.includes(hint))) {
return true;
}
const shellSelector = [
'nav',
'aside',
'header',
'footer',
'form',
'input',
'textarea',
'button',
'[role="button"]',
'[role="textbox"]',
'[role="navigation"]',
'[role="menu"]',
'[role="tooltip"]',
'[contenteditable="true"]',
'[data-testid*="sidebar"]',
'[data-testid*="history"]',
'[data-testid*="composer"]',
'[data-testid*="input"]',
'[data-testid*="toolbar"]',
'[data-testid*="share"]',
'[data-testid*="account"]'
].join(", ");
if (node.matches(shellSelector)) {
return true;
}
const shellAncestor = node.closest(shellSelector);
return Boolean(shellAncestor && shellAncestor !== root && root.contains(shellAncestor));
}
function looksLikeTranscriptNode(node, root) {
if (!node || !root || node === root) {
return false;
}
if (isChromeUi(node) || isHiddenOrInvisible(node) || isShellContainer(node, root)) {
return false;
}
const text = normalizeWhitespace(node.textContent || "");
if (!text) {
return false;
}
if (/window\.__oai_loghtml|<script|script>/i.test(text)) {
return false;
}
const label = [
node.getAttribute("data-testid"),
node.getAttribute("aria-label"),
node.className,
text
].join(" ").toLowerCase();
const shellWords = /sidebar|history|composer|input|toolbar|settings|share|footer|menu|account|project|navigation|new chat|skip to content/i.test(label);
if (shellWords) {
return false;
}
const roleAncestor = node.closest('[data-message-author-role], [data-testid*="message"], [data-testid*="conversation"], [data-testid*="assistant"], [data-testid*="user"], [data-testid*="turn"]');
const hasAncestorRoleSignal = Boolean(roleAncestor && root.contains(roleAncestor) && roleAncestor !== node);
const messageSignals = /message|conversation|turn|assistant|user|chatgpt|you\s+said|chatgpt\s+said/i.test(label);
const tagName = (node.tagName || "").toLowerCase();
const isBlockLike = ["div", "section", "article", "p", "li", "td", "th", "pre", "blockquote"].includes(tagName);
const childCount = Array.from(node.children || []).filter((child) => child.nodeType === Node.ELEMENT_NODE).length;
const hasBlockStructure = childCount <= 12 && (node.querySelector("p, li, pre, blockquote, table") || text.length >= 8);
const hasMeaningfulLength = text.length >= 8 && text.length <= 4000;
return (messageSignals || hasAncestorRoleSignal || (isBlockLike && hasBlockStructure && hasMeaningfulLength)) && !shellWords;
}
function splitIntoTranscriptTurns(text, fallbackRole) {
const normalized = normalizeWhitespace(text || "");
if (!normalized) {
return [];
}
const parts = normalized.split(/(?=\b(?:you|user|assistant|chatgpt)\b)/i).map((part) => part.trim()).filter(Boolean);
if (parts.length <= 1) {
return [{ role: fallbackRole || "unknown", content: normalized }];
}
return parts.map((part) => {
const roleText = part.match(/^(you|user|assistant|chatgpt)\b/i);
const role = roleText ? (roleText[1].toLowerCase().includes("assistant") || roleText[1].toLowerCase() === "chatgpt" ? "assistant" : roleText[1].toLowerCase() === "user" || roleText[1].toLowerCase() === "you" ? "user" : "unknown") : (fallbackRole || "unknown");
const content = part.replace(/^(you|user|assistant|chatgpt)\b(?:\s*(?:said|:|-))?\s*/i, "").trim();
return {
role,
content: content || part
};
}).filter((item) => Boolean(item.content));
}
function collectMessageCandidates() {
const root = getConversationRoot();
const candidates = [];
const shellSkipped = [];
const duplicateSkipped = [];
const orphanChildBlocksSkipped = [];
const possibleNodes = Array.from(root.querySelectorAll("*"));
possibleNodes.forEach((element) => {
if (!element || !hasMeaningfulText(element)) {
return;
}
if (isChromeUi(element) || isHiddenOrInvisible(element)) {
return;
}
if (isShellContainer(element, root)) {
shellSkipped.push(element);
return;
}
if (!looksLikeTranscriptNode(element, root)) {
return;
}
const text = getElementFingerprint(element);
if (!text) {
return;
}
candidates.push(element);
});
const orderedCandidates = candidates.sort((left, right) => {
const leftDepth = left && left.tagName !== "BODY" ? (left.parentElement ? Array.from(left.parentElement.children).indexOf(left) : 0) : 0;
const rightDepth = right && right.tagName !== "BODY" ? (right.parentElement ? Array.from(right.parentElement.children).indexOf(right) : 0) : 0;
if (leftDepth !== rightDepth) {
return leftDepth - rightDepth;
}
const leftTextLength = (left.textContent || "").length;
const rightTextLength = (right.textContent || "").length;
return rightTextLength - leftTextLength;
});
const deduped = [];
orderedCandidates.forEach((candidate) => {
const hasAncestorCandidate = deduped.some((existing) => existing !== candidate && existing.contains(candidate));
if (hasAncestorCandidate) {
orphanChildBlocksSkipped.push(candidate);
return;
}
const fingerprint = getElementFingerprint(candidate);
const isDuplicate = deduped.some((existing) => {
const existingFingerprint = getElementFingerprint(existing);
const sameText = existingFingerprint === fingerprint;
const nested = existing.contains(candidate) || candidate.contains(existing);
return sameText && nested;
});
if (isDuplicate) {
duplicateSkipped.push(candidate);
return;
}
deduped.push(candidate);
});
return {
candidates: deduped,
debug: {
candidateContainersFound: orderedCandidates.length,
afterShellFilter: deduped.length,
afterDeduplication: deduped.length,
skippedShellContainers: shellSkipped.length,
skippedDuplicateContainers: duplicateSkipped.length,
orphanChildBlocksSkipped: orphanChildBlocksSkipped.length
}
};
}
function detectMessageRole(element) {
const parts = [];
let current = element;
while (current && current !== document.body) {
parts.push([
current.getAttribute("data-message-author-role"),
current.getAttribute("data-testid"),
current.getAttribute("aria-label"),
current.className
].join(" "));
current = current.parentElement;
}
const roleText = parts.join(" ").toLowerCase();
if (roleText.includes("assistant") || roleText.includes("chatgpt")) {
return "assistant";
}
if (roleText.includes("user") || roleText.includes("you")) {
return "user";
}
return "unknown";
}
function classifyContainer(container) {
const role = detectMessageRole(container);
const text = normalizeWhitespace(container.textContent || "");
const hasStrongSignals = /assistant|chatgpt|you|user/i.test(container.getAttribute("data-testid") || "") || /assistant|chatgpt|you|user/i.test(container.className || "");
return {
role,
type: hasStrongSignals ? "conversation" : "unknown",
rawText: text,
htmlSnippet: (container.outerHTML || "").slice(0, 240)
};
}
function extractContainerFallback(container) {
const text = normalizeWhitespace(container.textContent || "");
return {
role: detectMessageRole(container),
rawText: text,
htmlSnippet: (container.outerHTML || "").slice(0, 240)
};
}
function findConversationContainers() {
const discovery = collectMessageCandidates();
const containers = discovery.candidates.map((container) => ({
element: container,
...classifyContainer(container)
}));
return {
containers,
debug: {
...discovery.debug,
candidateContainersFound: discovery.debug.candidateContainersFound,
afterShellFilter: discovery.debug.afterShellFilter,
afterDeduplication: discovery.debug.afterDeduplication,
skippedShellContainers: discovery.debug.skippedShellContainers,
skippedDuplicateContainers: discovery.debug.skippedDuplicateContainers,
orphanChildBlocksSkipped: discovery.debug.orphanChildBlocksSkipped
}
};
}
function getCodeLanguage(node) {
const className = (node.className || "").toString();
const languageMatch = className.match(/language-([a-z0-9_+-]+)/i) || className.match(/lang-([a-z0-9_+-]+)/i);
const dataLanguage = node.getAttribute("data-language") || node.getAttribute("data-lang");
if (languageMatch) {
return languageMatch[1];
}
return dataLanguage || "";
}
function normalizeInlineText(text) {
return (text || "")
.replace(/\u00a0/g, " ")
.replace(/\s+/g, " ")
.replace(/\s+([,.;:!?])/g, "$1")
.trim();
}
function escapeMarkdownLinkText(text) {
return String(text || "")
.replace(/\[/g, "\\[")
.replace(/\]/g, "\\]")
.trim();
}
function escapeMarkdownLinkUrl(url) {
return String(url || "")
.replace(/\)/g, "%29")
.trim();
}
function isSafeHref(href) {
if (!href) {
return false;
}
const safeHref = href.trim();
if (!safeHref || /^javascript:/i.test(safeHref) || /^data:/i.test(safeHref) || /^blob:/i.test(safeHref)) {
return false;
}
return /^https?:\/\//i.test(safeHref) || /^mailto:/i.test(safeHref);
}
function linkToMarkdown(anchorNode) {
const href = anchorNode.getAttribute("href") || "";
const textContent = normalizeInlineText(childrenToInlineMarkdown(anchorNode));
const visibleText = textContent || normalizeInlineText(anchorNode.textContent || "");
const label = visibleText || href;
if (!isSafeHref(href)) {
return visibleText;
}
return `[${escapeMarkdownLinkText(label)}](${escapeMarkdownLinkUrl(href)})`;
}
function isInlineElement(node) {
return Boolean(
node &&
node.nodeType === Node.ELEMENT_NODE &&
node.matches("strong, b, em, i, code, span, a, small, sub, sup, u, s")
);
}
function isBlockElement(node) {
return Boolean(
node &&
node.nodeType === Node.ELEMENT_NODE &&
node.matches("p, ul, ol, li, pre, blockquote, h1, h2, h3, h4, h5, h6, table, tr, td, th, div, section, article, main, hr")
);
}
function escapeTableCell(value) {
return String(value || "")
.replace(/\|/g, "\\|")
.replace(/\r?\n/g, " <br> ")
.replace(/\s+/g, " ")
.trim();
}
function tableToMarkdown(tableNode) {
const rows = Array.from(tableNode.querySelectorAll("tr"));
if (rows.length === 0) {
return "";
}
const headerRow = rows.find((row) => row.querySelector("th")) || rows[0];
const bodyRows = rows.filter((row) => row !== headerRow);
const headerCells = Array.from(headerRow.querySelectorAll("th, td"))
.map((cell) => escapeTableCell(childrenToInlineMarkdown(cell)))
.filter((cell) => cell.length > 0);
if (headerCells.length === 0) {
return "";
}
const body = bodyRows.map((row) => {
const cells = Array.from(row.querySelectorAll("td"))
.map((cell) => escapeTableCell(childrenToInlineMarkdown(cell)))
.filter((cell) => cell.length > 0);
if (cells.length === 0) {
return "";
}
return `| ${cells.join(" | ")} |`;
}).filter(Boolean);
const headerLine = `| ${headerCells.join(" | ")} |`;
const separatorLine = `| ${headerCells.map(() => "---").join(" | ")} |`;
return [headerLine, separatorLine, ...body].join("\n");
}
function codeBlockToMarkdown(preNode) {
const codeNode = preNode.querySelector("code") || preNode;
const language = getCodeLanguage(codeNode);
const content = (codeNode.textContent || "")
.replace(/\r\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
return [language ? `\`\`\`${language}` : "```", content, "```"].filter(Boolean).join("\n");
}
function childrenToInlineMarkdown(node) {
const parts = [];
Array.from(node.childNodes).forEach((child) => {
if (child.nodeType === Node.TEXT_NODE) {
const text = normalizeInlineText(child.textContent || "");
if (text) {
parts.push(text);
}
return;
}
if (child.nodeType !== Node.ELEMENT_NODE) {
return;
}
if (child.matches("strong, b")) {
parts.push(`**${childrenToInlineMarkdown(child)}**`);
} else if (child.matches("em, i")) {
parts.push(`*${childrenToInlineMarkdown(child)}*`);
} else if (child.matches("code")) {
parts.push(`\`${normalizeInlineText(child.textContent || "")}\``);
} else if (child.matches("a")) {
parts.push(linkToMarkdown(child));
} else {
parts.push(nodeToMarkdown(child));
}
});
return parts.join(" ").trim();
}
function listItemToMarkdown(liNode, depth = 0) {
const prefix = depth > 0 ? `${" ".repeat(depth)}` : "";
const inlineContent = childrenToInlineMarkdown(liNode).trim();
const nestedItems = Array.from(liNode.children || []).filter((child) => child.matches("ul, ol"));
let content = inlineContent || normalizeInlineText(liNode.textContent || "");
if (!content) {
return "";
}
const nestedMarkdown = nestedItems
.map((nestedList) => listToMarkdown(nestedList, depth + 1))
.filter(Boolean)
.join("\n");
return [
`${prefix}- ${content}`,
nestedMarkdown ? `${prefix} ${nestedMarkdown.replace(/\n/g, `\n${prefix} `)}` : ""
].filter(Boolean).join("\n");
}
function listToMarkdown(listNode, depth = 0) {
const listType = listNode.tagName.toLowerCase();
const items = Array.from(listNode.children || []).filter((child) => child.matches("li"));
return items
.map((item, index) => {
const marker = listType === "ol" ? `${index + 1}. ` : "- ";
const inlineContent = childrenToInlineMarkdown(item).trim();
const nestedItems = Array.from(item.children || []).filter((child) => child.matches("ul, ol"));
const nestedMarkdown = nestedItems
.map((nestedList) => listToMarkdown(nestedList, depth + 1))
.filter(Boolean)
.join("\n");
const content = inlineContent || normalizeInlineText(item.textContent || "");
const prefix = depth > 0 ? `${" ".repeat(depth)}` : "";
return [
`${prefix}${marker}${content}`,
nestedMarkdown ? `${prefix}${" ".repeat(depth + 1)}${nestedMarkdown.replace(/\n/g, `\n${prefix}${" ".repeat(depth + 1)}`)}` : ""
].filter(Boolean).join("\n");
})
.filter(Boolean)
.join("\n");
}
function blockquoteToMarkdown(blockquoteNode) {
return Array.from(blockquoteNode.childNodes)
.map((child) => {
if (child.nodeType === Node.TEXT_NODE) {
return `> ${normalizeInlineText(child.textContent || "")}`;
}
if (child.nodeType !== Node.ELEMENT_NODE) {
return "";
}
const markdown = nodeToMarkdown(child);
return markdown
.split(/\n/)
.map((line) => (line ? `> ${line}` : ">"))
.join("\n");
})
.filter(Boolean)
.join("\n");
}
function cleanupMarkdownOutsideCodeFences(markdown) {
const lines = (markdown || "").split(/\r?\n/);
const cleaned = [];
let inCodeFence = false;
lines.forEach((line) => {
const trimmed = line.trim();
const isFence = /^```/.test(trimmed);
if (isFence) {
inCodeFence = !inCodeFence;
cleaned.push(line);
return;
}
if (inCodeFence) {
cleaned.push(line);
return;
}
const normalizedLine = line.replace(/\s+([,.;:!?])/g, "$1");
const punctuationOnly = /^[\s\p{P}\u2014\u2013]+$/u.test(normalizedLine);
if (normalizedLine.trim() === "" || punctuationOnly) {
if (cleaned.length === 0 || cleaned[cleaned.length - 1].trim() !== "") {
cleaned.push("");
}
return;
}
cleaned.push(normalizedLine);
});
return cleaned.join("\n").replace(/\n{3,}/g, "\n\n").trim();
}
function blockChildrenToMarkdown(node) {
const parts = [];
let paragraphBuffer = [];
const flushParagraph = () => {
const paragraph = paragraphBuffer.join(" ").trim();
if (paragraph) {
parts.push(paragraph);
}
paragraphBuffer = [];
};
Array.from(node.childNodes).forEach((child) => {
if (child.nodeType === Node.TEXT_NODE) {
const text = normalizeInlineText(child.textContent || "");
if (text) {
paragraphBuffer.push(text);
}
return;
}
if (child.nodeType !== Node.ELEMENT_NODE) {
return;
}
if (child.matches("p")) {
flushParagraph();
parts.push(nodeToMarkdown(child));
return;
}
if (child.matches("ul, ol")) {
flushParagraph();
parts.push(nodeToMarkdown(child));
return;
}
if (child.matches("pre")) {
flushParagraph();
parts.push(nodeToMarkdown(child));
return;
}
if (child.matches("blockquote")) {
flushParagraph();
parts.push(nodeToMarkdown(child));
return;
}
if (child.matches("table")) {
flushParagraph();
parts.push(nodeToMarkdown(child));
return;
}
if (child.matches("h1, h2, h3, h4, h5, h6")) {
flushParagraph();
parts.push(nodeToMarkdown(child));
return;
}
if (child.matches("br")) {
paragraphBuffer.push("\n");
return;
}
if (isInlineElement(child)) {
paragraphBuffer.push(childrenToInlineMarkdown(child));
return;
}
const nestedMarkdown = nodeToMarkdown(child);
if (nestedMarkdown) {
flushParagraph();
parts.push(nestedMarkdown);
}
});
flushParagraph();
return parts.join("\n\n");
}
function nodeToMarkdown(node) {
if (!node) {
return "";
}
if (node.nodeType === Node.TEXT_NODE) {
return normalizeInlineText(node.textContent || "");
}
if (node.nodeType !== Node.ELEMENT_NODE) {
return "";
}
if (node.matches('script, style, svg, img, video, audio')) {
return "";
}
if (node.matches("pre")) {
return codeBlockToMarkdown(node);
}
if (node.matches("code")) {
return `\`${normalizeInlineText(node.textContent || "")}\``;
}
if (node.matches("br")) {
return "\n";
}
if (node.matches("blockquote")) {
return blockquoteToMarkdown(node);
}
if (node.matches("table")) {
return tableToMarkdown(node);
}
if (node.matches("ul, ol")) {
return listToMarkdown(node);
}
if (node.matches("li")) {
return listItemToMarkdown(node);
}
if (node.matches("h1, h2, h3, h4, h5, h6")) {
const level = parseInt(node.tagName.charAt(1), 10) || 1;
return `${"#".repeat(level)} ${childrenToInlineMarkdown(node)}`;
}
if (node.matches("p")) {
return childrenToInlineMarkdown(node);
}
if (isInlineElement(node)) {
return childrenToInlineMarkdown(node);
}
return blockChildrenToMarkdown(node);
}
function inlineChildrenToMarkdown(node) {
return childrenToInlineMarkdown(node);
}
function containerToMarkdown(container) {
const element = container?.element || container;
if (!element) {
return "";
}
const rawText = normalizeWhitespace(element.textContent || "");
const htmlText = (element.outerHTML || "").toLowerCase();
const unsupportedKeywords = /(image generation|generated image|image|artifact|file upload|uploaded file|tool output|browser|code interpreter|analysis)/i;
if (unsupportedKeywords.test(rawText) || unsupportedKeywords.test(htmlText)) {
const typeMatch = rawText.match(/image generation|artifact|file upload|tool output|browser|code interpreter|analysis/i);
const type = typeMatch ? typeMatch[0] : "unsupported content";
const placeholder = `[Unsupported ChatGPT content: ${type}]`;
return rawText ? `${placeholder}\n\n${rawText}` : placeholder;
}
const clone = prepareExtractableNode(element);
const markdown = extractNodeAsMarkdown(clone);
return markdown || rawText;
}
function extractNodeAsMarkdown(node) {
if (!node) {
return "";
}
const markdown = nodeToMarkdown(node);
return cleanupMarkdownOutsideCodeFences(markdown);
}
function prepareExtractableNode(node) {
const clone = node.cloneNode(true);
clone.querySelectorAll(UI_CHROME_SELECTORS.join(", ")).forEach((child) => child.remove());
clone.querySelectorAll('script, style, svg, img, video, audio, [aria-hidden="true"], [hidden], .hidden, .sr-only').forEach((child) => child.remove());
return clone;
}
function extractMessageContent(element) {
return containerToMarkdown(element);
}
function findMessageElements() {
return collectMessageCandidates().filter((element) => {
const content = extractMessageContent(element);
return Boolean(content);
});
}
function extractChatGPTConversation() {
const title = getConversationTitle();
const discovery = findConversationContainers();
const discoveredContainers = discovery.containers;
const parsedMessages = [];
const seenFingerprints = new Set();
discoveredContainers.forEach((container) => {
const rawContent = containerToMarkdown(container);
const content = normalizeWhitespace(rawContent || container.rawText || "");
if (!content) {
return;
}
const role = container.role || detectMessageRole(container.element);
const fingerprint = `${role}:${content.toLowerCase()}`;
if (seenFingerprints.has(fingerprint)) {
return;
}
seenFingerprints.add(fingerprint);
parsedMessages.push({
role,
content,
rawText: container.rawText,
type: container.type,
htmlSnippet: container.htmlSnippet
});
});
const messages = parsedMessages
.map((message) => ({
role: message.role,
content: message.content || message.rawText
}))
.filter((message) => Boolean(message.content));
const debug = {
turnContainersFound: discoveredContainers.length,
messagesReturned: messages.length,
duplicateTurnsSkipped: Math.max(0, discoveredContainers.length - parsedMessages.length),
orphanChildBlocksSkipped: discovery.debug.orphanChildBlocksSkipped || 0,
unknownTurnsReturned: parsedMessages.filter((message) => message.role === "unknown").length,
candidateContainersFound: discovery.debug.candidateContainersFound,
afterShellFilter: discovery.debug.afterShellFilter,
afterDeduplication: discovery.debug.afterDeduplication,
skippedShellContainers: discovery.debug.skippedShellContainers,
skippedDuplicateContainers: discovery.debug.skippedDuplicateContainers
};
if (messages.length === 0) {
return {
sourcePlatform: "ChatGPT",
title,
url: window.location.href,
capturedAt: new Date().toISOString(),
messages: [],
error: "No ChatGPT conversation messages were detected.",
debug
};
}
return {
sourcePlatform: "ChatGPT",
title,
url: window.location.href,
capturedAt: new Date().toISOString(),
messages,
debug
};
}
window.extractChatGPTConversation = extractChatGPTConversation;
(async function () {
try {
const payload = await extractChatGPTConversation();
console.log("ChatGPT extractor payload:", payload);
if (typeof chrome !== "undefined" && chrome.runtime && chrome.runtime.sendMessage) {
chrome.runtime.sendMessage({
type: "CHATGPT_CAPTURE_PAYLOAD",
payload
});
}
} catch (error) {
const fallbackPayload = {
sourcePlatform: "ChatGPT",
title: getConversationTitle(),
url: window.location.href,
capturedAt: new Date().toISOString(),
messages: [],
error: error.message || "ChatGPT extraction failed."
};
if (typeof chrome !== "undefined" && chrome.runtime && chrome.runtime.sendMessage) {
chrome.runtime.sendMessage({
type: "CHATGPT_CAPTURE_PAYLOAD",
payload: fallbackPayload
});
}
}
})();
})();
@@ -0,0 +1,51 @@
import { prepareChatGPTForCapture } from "./chatgptPreprocessor.js";
import { discoverChatGPTTurns } from "./chatgptDiscovery.js";
import { renderChatGPTTurnToMarkdown } from "./chatgptTurnRenderer.js";
import { createConversationModel } from "../../core/conversationModel.js";
import { serializeConversationToMarkdown } from "../../core/markdownSerializer.js";
export async function captureConversation(options = {}) {
const root = options.root || document;
const turns = discoverChatGPTTurns(root);
const preprocessed = await prepareChatGPTForCapture({ root, turns });
const renderedTurns = turns.map((turn) => {
const markdown = renderChatGPTTurnToMarkdown(turn);
const captureStatus = markdown.includes("[Turn not rendered") ? "not_rendered" : "captured";
return {
...turn,
captureStatus,
contentType: "text",
markdown,
debug: {
rendered: captureStatus === "captured"
}
};
});
const conversation = createConversationModel({
sourcePlatform: "ChatGPT",
title: options.title || document.title || "ChatGPT Conversation",
url: options.url || window.location.href,
capturedAt: options.capturedAt || new Date().toISOString(),
warning: renderedTurns.some((turn) => turn.captureStatus === "not_rendered")
? "Some conversation turns were not rendered in the DOM. Scroll through the conversation and capture again."
: undefined,
turns: renderedTurns,
debug: {
turnSectionsFound: renderedTurns.length,
renderedTurnsCaptured: renderedTurns.filter((turn) => turn.captureStatus === "captured").length,
notRenderedTurns: renderedTurns.filter((turn) => turn.captureStatus === "not_rendered").length,
userTurns: renderedTurns.filter((turn) => turn.role === "user").length,
assistantTurns: renderedTurns.filter((turn) => turn.role === "assistant").length,
unknownTurns: renderedTurns.filter((turn) => turn.role === "unknown").length
}
});
const markdown = serializeConversationToMarkdown(conversation);
return {
conversation,
markdown,
debug: conversation.debug,
preprocessed
};
}
@@ -0,0 +1,49 @@
const TURN_SECTION_SELECTOR = 'section[data-testid^="conversation-turn-"][data-turn]';
function normalizeRole(role) {
const value = (role || "unknown").toLowerCase();
if (value === "user" || value === "you") {
return "user";
}
if (value === "assistant" || value === "chatgpt") {
return "assistant";
}
if (value === "system") {
return "system";
}
if (value === "tool") {
return "tool";
}
return "unknown";
}
function parseTurnNumber(section) {
const match = (section?.getAttribute("data-testid") || "").match(/conversation-turn-(\d+)/i);
return match ? Number(match[1]) : NaN;
}
export function discoverChatGPTTurns(root = document) {
const sections = Array.from(root.querySelectorAll(TURN_SECTION_SELECTOR));
const orderedSections = sections
.map((section, index) => ({
section,
sortIndex: Number.isNaN(parseTurnNumber(section)) ? index : parseTurnNumber(section)
}))
.sort((left, right) => left.sortIndex - right.sortIndex);
return orderedSections.map((entry, index) => {
const section = entry.section;
const explicitRole = normalizeRole(section?.getAttribute("data-turn") || "");
const authorRoleNode = section?.querySelector('[data-message-author-role]');
const fallbackRole = authorRoleNode ? normalizeRole(authorRoleNode.getAttribute("data-message-author-role") || "") : "unknown";
const role = explicitRole !== "unknown" ? explicitRole : fallbackRole;
return {
turnIndex: index,
turnId: section?.getAttribute("data-testid") || `conversation-turn-${index + 1}`,
role,
sourceElement: section,
sourcePlatform: "ChatGPT"
};
});
}
@@ -0,0 +1,16 @@
export async function prepareChatGPTForCapture(options = {}) {
const turns = options.turns || [];
if (turns.length > 0 && typeof options.root?.querySelectorAll === "function") {
turns.forEach((turn) => {
if (turn?.sourceElement?.scrollIntoView) {
turn.sourceElement.scrollIntoView({ block: "center", inline: "nearest" });
}
});
}
return {
prepared: true,
message: "Preparing conversation for capture...",
scrolledTurns: turns.length
};
}
@@ -0,0 +1,79 @@
import { htmlToMarkdown } from "../../core/htmlToMarkdown.js";
function normalizeWhitespace(text) {
if (!text) {
return "";
}
return text
.replace(/\u00a0/g, " ")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.replace(/[ \t]{2,}/g, " ")
.trim();
}
function getTurnContentRoot(turn) {
const section = turn?.sourceElement;
if (!section) {
return null;
}
if (turn.role === "assistant") {
const assistantMessage = section.querySelector('[data-message-author-role="assistant"]');
return assistantMessage?.querySelector(".markdown") || assistantMessage || section.querySelector(".markdown") || section;
}
if (turn.role === "user") {
const userMessage = section.querySelector('[data-message-author-role="user"]');
return userMessage || section;
}
return section;
}
function isRenderCandidate(node) {
if (!node || node.nodeType !== Node.ELEMENT_NODE) {
return false;
}
if (node.matches('button, [role="button"], [aria-hidden="true"], [hidden], .hidden, .sr-only, nav, aside, header, footer, form, input, textarea, select, svg, script, style')) {
return false;
}
return true;
}
export function renderChatGPTTurnToMarkdown(turn) {
const section = turn?.sourceElement;
const role = turn?.role || "unknown";
const contentRoot = getTurnContentRoot(turn);
const placeholder = "[Turn not rendered in DOM. Scroll this turn into view and recapture.]";
if (!section || !contentRoot) {
return placeholder;
}
const rect = typeof section.getBoundingClientRect === "function" ? section.getBoundingClientRect() : null;
const isVirtualized = section.getAttribute("data-is-intersecting") === "false";
const hasRenderableContent = Boolean(contentRoot && Array.from(contentRoot.querySelectorAll("p, pre, ul, ol, li, blockquote, h1, h2, h3, h4, h5, h6, table, img, [data-message-author-role]"))
.filter(isRenderCandidate)
.some((node) => (node.textContent || "").trim().length > 0));
if (isVirtualized || (rect && rect.height <= 0 && rect.width <= 0) || (!hasRenderableContent && !(contentRoot.textContent || "").trim())) {
return placeholder;
}
const clone = contentRoot.cloneNode(true);
clone.querySelectorAll('button, [role="button"], [aria-hidden="true"], [hidden], .hidden, .sr-only, nav, aside, header, footer, form, input, textarea, select, svg, script, style').forEach((child) => child.remove());
if (role === "user") {
const userMessage = clone.querySelector('[data-message-author-role="user"]') || clone;
const markdown = htmlToMarkdown(userMessage, { role });
return normalizeWhitespace(markdown || placeholder);
}
const assistantMessage = clone.querySelector('[data-message-author-role="assistant"]') || clone.querySelector(".markdown") || clone;
const markdown = htmlToMarkdown(assistantMessage, { role });
return normalizeWhitespace(markdown || placeholder);
}
@@ -0,0 +1,32 @@
export function createConversationTurn(turn) {
return {
turnIndex: turn?.turnIndex ?? 0,
turnId: turn?.turnId || "",
role: turn?.role || "unknown",
captureStatus: turn?.captureStatus || "captured",
sourcePlatform: turn?.sourcePlatform || "ChatGPT",
contentType: turn?.contentType || "text",
sourceElement: turn?.sourceElement || null,
markdown: turn?.markdown || "",
debug: turn?.debug || {}
};
}
export function createConversationModel(payload = {}) {
return {
sourcePlatform: payload?.sourcePlatform || "ChatGPT",
title: payload?.title || "ChatGPT Conversation",
url: payload?.url || "",
capturedAt: payload?.capturedAt || new Date().toISOString(),
warning: payload?.warning,
turns: (payload?.turns || []).map(createConversationTurn),
debug: {
turnSectionsFound: payload?.debug?.turnSectionsFound || 0,
renderedTurnsCaptured: payload?.debug?.renderedTurnsCaptured || 0,
notRenderedTurns: payload?.debug?.notRenderedTurns || 0,
userTurns: payload?.debug?.userTurns || 0,
assistantTurns: payload?.debug?.assistantTurns || 0,
unknownTurns: payload?.debug?.unknownTurns || 0
}
};
}
@@ -0,0 +1,232 @@
function normalizeWhitespace(text) {
if (!text) {
return "";
}
return text
.replace(/\u00a0/g, " ")
.replace(/[ \t]+\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.replace(/[ \t]{2,}/g, " ")
.trim();
}
function normalizeInlineText(text) {
return (text || "")
.replace(/\u00a0/g, " ")
.replace(/\s+/g, " ")
.replace(/\s+([,.;:!?])/g, "$1")
.trim();
}
function escapeMarkdownLinkText(text) {
return String(text || "")
.replace(/\[/g, "\\[")
.replace(/\]/g, "\\]")
.trim();
}
function escapeMarkdownLinkUrl(url) {
return String(url || "")
.replace(/\)/g, "%29")
.trim();
}
function isSafeHref(href) {
const value = (href || "").trim();
if (!value || /^javascript:/i.test(value) || /^data:/i.test(value) || /^blob:/i.test(value)) {
return false;
}
return /^https?:\/\//i.test(value) || /^mailto:/i.test(value);
}
function isInlineElement(node) {
return Boolean(
node &&
node.nodeType === Node.ELEMENT_NODE &&
node.matches("strong, b, em, i, code, span, a, small, sub, sup, u, s")
);
}
function getCodeLanguage(node) {
const className = (node?.className || "").toString();
const languageMatch = className.match(/language-([a-z0-9_+-]+)/i) || className.match(/lang-([a-z0-9_+-]+)/i);
const dataLanguage = node?.getAttribute?.("data-language") || node?.getAttribute?.("data-lang");
return languageMatch ? languageMatch[1] : dataLanguage || "";
}
function anchorToMarkdown(anchorNode, role) {
const href = anchorNode.getAttribute("href") || "";
const label = normalizeInlineText(childrenToMarkdown(anchorNode, role)) || normalizeInlineText(anchorNode.textContent || "") || href;
if (!isSafeHref(href)) {
return label;
}
return `[${escapeMarkdownLinkText(label)}](${escapeMarkdownLinkUrl(href)})`;
}
function imageToMarkdown(img, role) {
const alt = normalizeInlineText(img.getAttribute("alt") || "");
const src = img.getAttribute("src") || "";
const label = alt || src || "image";
if (role === "user") {
return `[Attached image: ${label}](${src || ""})`;
}
if (src) {
return `![Generated image: ${label}](${src})`;
}
return `![Generated image: ${label}]`;
}
function listToMarkdown(listNode, role, depth = 0) {
const listType = listNode.tagName.toLowerCase();
const items = Array.from(listNode.children || []).filter((child) => child.matches("li"));
const prefix = depth > 0 ? " ".repeat(depth) : "";
return items
.map((item, index) => {
const marker = listType === "ol" ? `${index + 1}. ` : "- ";
const inlineContent = childrenToMarkdown(item, role).trim();
const nestedItems = Array.from(item.children || []).filter((child) => child.matches("ul, ol"));
const nestedMarkdown = nestedItems.map((nestedList) => listToMarkdown(nestedList, role, depth + 1)).filter(Boolean).join("\n");
const content = inlineContent || normalizeInlineText(item.textContent || "");
return [
`${prefix}${marker}${content}`,
nestedMarkdown ? `${prefix} ${nestedMarkdown.replace(/\n/g, `\n${prefix} `)}` : ""
].filter(Boolean).join("\n");
})
.filter(Boolean)
.join("\n");
}
function tableToMarkdown(tableNode, role) {
const rows = Array.from(tableNode.querySelectorAll("tr"));
if (rows.length === 0) {
return "";
}
const headerRow = rows.find((row) => row.querySelector("th")) || rows[0];
const bodyRows = rows.filter((row) => row !== headerRow);
const headerCells = Array.from(headerRow.querySelectorAll("th, td"))
.map((cell) => normalizeInlineText(childrenToMarkdown(cell, role)))
.filter(Boolean);
if (headerCells.length === 0) {
return "";
}
const body = bodyRows.map((row) => {
const cells = Array.from(row.querySelectorAll("td"))
.map((cell) => normalizeInlineText(childrenToMarkdown(cell, role)))
.filter(Boolean);
return cells.length > 0 ? `| ${cells.join(" | ")} |` : "";
}).filter(Boolean);
const headerLine = `| ${headerCells.join(" | ")} |`;
const separatorLine = `| ${headerCells.map(() => "---").join(" | ")} |`;
return [headerLine, separatorLine, ...body].join("\n");
}
function codeBlockToMarkdown(preNode) {
const codeNode = preNode.querySelector("code") || preNode;
const language = getCodeLanguage(codeNode);
const content = (codeNode.textContent || "")
.replace(/\r\n/g, "\n")
.replace(/\n{3,}/g, "\n\n")
.trim();
return [language ? `\`\`\`${language}` : "```", content, "```"].filter(Boolean).join("\n");
}
function blockquoteToMarkdown(blockquoteNode, role) {
return Array.from(blockquoteNode.childNodes)
.map((child) => {
if (child.nodeType === Node.TEXT_NODE) {
return `> ${normalizeInlineText(child.textContent || "")}`;
}
if (child.nodeType !== Node.ELEMENT_NODE) {
return "";
}
const markdown = nodeToMarkdown(child, role);
return markdown.split(/\n/).map((line) => (line ? `> ${line}` : ">")).join("\n");
})
.filter(Boolean)
.join("\n");
}
function childrenToMarkdown(node, role) {
const parts = [];
Array.from(node.childNodes).forEach((child) => {
if (child.nodeType === Node.TEXT_NODE) {
const text = normalizeInlineText(child.textContent || "");
if (text) {
parts.push(text);
}
return;
}
if (child.nodeType !== Node.ELEMENT_NODE) {
return;
}
if (child.matches("strong, b")) {
parts.push(`**${childrenToMarkdown(child, role)}**`);
} else if (child.matches("em, i")) {
parts.push(`*${childrenToMarkdown(child, role)}*`);
} else if (child.matches("code")) {
parts.push(`\`${normalizeInlineText(child.textContent || "")}\``);
} else if (child.matches("a")) {
parts.push(anchorToMarkdown(child, role));
} else if (child.matches("img")) {
parts.push(imageToMarkdown(child, role));
} else {
parts.push(nodeToMarkdown(child, role));
}
});
return parts.join(" ").trim();
}
function nodeToMarkdown(node, role) {
if (!node) {
return "";
}
if (node.nodeType === Node.TEXT_NODE) {
return normalizeInlineText(node.textContent || "");
}
if (node.nodeType !== Node.ELEMENT_NODE) {
return "";
}
if (node.matches("pre")) {
return codeBlockToMarkdown(node);
}
if (node.matches("code")) {
return `\`${normalizeInlineText(node.textContent || "")}\``;
}
if (node.matches("br")) {
return "\n";
}
if (node.matches("blockquote")) {
return blockquoteToMarkdown(node, role);
}
if (node.matches("table")) {
return tableToMarkdown(node, role);
}
if (node.matches("ul, ol")) {
return listToMarkdown(node, role);
}
if (node.matches("h1, h2, h3, h4, h5, h6")) {
const level = parseInt(node.tagName.charAt(1), 10) || 1;
return `${"#".repeat(level)} ${childrenToMarkdown(node, role)}`;
}
if (node.matches("p")) {
return childrenToMarkdown(node, role);
}
if (isInlineElement(node)) {
return childrenToMarkdown(node, role);
}
return childrenToMarkdown(node, role);
}
function cleanupWhitespace(markdown) {
return normalizeWhitespace(markdown || "").replace(/\n{3,}/g, "\n\n");
}
export function htmlToMarkdown(root, options = {}) {
const role = options.role || "unknown";
if (!root) {
return "";
}
return cleanupWhitespace(nodeToMarkdown(root, role));
}
@@ -0,0 +1,52 @@
function normalizeRoleHeading(role) {
switch ((role || "unknown").toLowerCase()) {
case "user":
return "## User";
case "assistant":
return "## Assistant";
case "system":
return "## System";
case "tool":
return "## Tool";
default:
return "## Unknown";
}
}
function normalizeMessageContent(content) {
return (content || "").trim();
}
export function serializeConversationToMarkdown(conversation) {
const lines = [];
lines.push(`# ${conversation?.title || "ChatGPT Conversation"}`);
lines.push("");
if (conversation?.capturedAt) {
lines.push(`Captured: ${conversation.capturedAt}`);
}
lines.push(`Source Platform: ${conversation?.sourcePlatform || "ChatGPT"}`);
if (conversation?.url) {
lines.push(`Source URL: ${conversation.url}`);
}
if (conversation?.warning) {
lines.push("");
lines.push(`Warning: ${conversation.warning}`);
}
lines.push("");
lines.push("---");
lines.push("");
const turns = Array.isArray(conversation?.turns) ? conversation.turns : [];
turns.forEach((turn, index) => {
lines.push(normalizeRoleHeading(turn?.role));
lines.push("");
lines.push(normalizeMessageContent(turn?.markdown));
if (index < turns.length - 1) {
lines.push("");
lines.push("---");
lines.push("");
}
});
return `${lines.join("\n").trim()}\n`;
}
@@ -0,0 +1,44 @@
function formatDateForFilename(dateValue) {
const date = dateValue ? new Date(dateValue) : new Date();
if (Number.isNaN(date.getTime())) {
return new Date().toISOString().slice(0, 10);
}
return date.toISOString().slice(0, 10);
}
function sanitizeFilenamePart(value) {
return String(value || "")
.normalize("NFKD")
.replace(/[\u0000-\u001F\u007F]/g, "")
.replace(/[<>:"/\\|?*]/g, " ")
.replace(/\s+/g, " ")
.replace(/^\.+|\.+$/g, "")
.replace(/^\s+|\s+$/g, "")
.trim();
}
function truncateFilename(filename, maxLength) {
if (!filename) {
return filename;
}
if (filename.length <= maxLength) {
return filename;
}
const extension = filename.slice(filename.lastIndexOf("."));
const base = filename.slice(0, filename.lastIndexOf("."));
const safeExtension = extension || ".md";
const safeBase = base.slice(0, Math.max(1, maxLength - safeExtension.length - 1)).trimEnd();
return `${safeBase}${safeExtension}`;
}
export function createConversationFilename(capturePayload) {
const date = formatDateForFilename(capturePayload?.capturedAt);
const platform = sanitizeFilenamePart(capturePayload?.sourcePlatform || "ChatGPT") || "ChatGPT";
const title = sanitizeFilenamePart(capturePayload?.title || "Conversation") || "Conversation";
const baseFilename = `${date} - ${platform} - ${title}.md`;
return truncateFilename(baseFilename, 180);
}
@@ -0,0 +1,120 @@
function normalizeTitle(title) {
const normalized = (title || "ChatGPT Conversation")
.replace(/\s+/g, " ")
.trim();
return normalized || "ChatGPT Conversation";
}
function normalizeRoleHeading(role) {
switch ((role || "unknown").toLowerCase()) {
case "user":
return "## User";
case "assistant":
return "## Assistant";
case "system":
return "## System";
default:
return "## Unknown";
}
}
function collapseExcessBlankLinesOutsideCodeBlocks(text) {
const lines = (text || "").split(/\r?\n/);
const result = [];
let inCodeBlock = false;
for (const line of lines) {
const trimmed = line.trim();
const fence = trimmed.startsWith("```");
if (fence) {
if (!inCodeBlock) {
inCodeBlock = true;
} else {
inCodeBlock = false;
}
result.push(line);
continue;
}
if (inCodeBlock) {
result.push(line);
continue;
}
if (trimmed === "") {
if (result.length === 0 || result[result.length - 1].trim() !== "") {
result.push("");
}
continue;
}
result.push(line);
}
return result.join("\n").trim();
}
function normalizeMessageContent(content) {
return collapseExcessBlankLinesOutsideCodeBlocks((content || "").trim());
}
export function normalizeConversationToMarkdown(capturePayload) {
const payload = capturePayload || {};
const title = normalizeTitle(payload.title);
const capturedAt = payload.capturedAt || "";
const sourcePlatform = payload.sourcePlatform || "Unknown";
const sourceUrl = payload.url || "";
const messages = Array.isArray(payload.messages) ? payload.messages : [];
const lines = [];
lines.push(`# ${title}`);
lines.push("");
if (capturedAt) {
lines.push(`Captured: ${capturedAt}`);
}
lines.push(`Source Platform: ${sourcePlatform}`);
if (sourceUrl) {
lines.push(`Source URL: ${sourceUrl}`);
}
lines.push("");
lines.push("---");
lines.push("");
if (payload.error) {
lines.push("## Capture Error");
lines.push("");
lines.push(payload.error);
lines.push("");
lines.push("---");
lines.push("");
}
if (messages.length === 0) {
lines.push("## Capture Notice");
lines.push("");
lines.push("No conversation messages were detected.");
lines.push("");
lines.push("---");
lines.push("");
} else {
messages.forEach((message, index) => {
lines.push(normalizeRoleHeading(message?.role));
lines.push("");
lines.push(normalizeMessageContent(message?.content));
if (index < messages.length - 1) {
lines.push("");
lines.push("---");
lines.push("");
}
});
}
return `${lines.join("\n").trim()}\n`;
}
globalThis.normalizeConversationToMarkdown = normalizeConversationToMarkdown;
View File
+25
View File
@@ -0,0 +1,25 @@
# ChatGPT Capture MVP
Current Phase
Implementation
Current Task
0003c-render-all-turns
Completed
✓ 0001
✓ 0002
✓ 0003
✓ 0003a
✓ 0003b
Blocked
None
Next
0004-markdown-normalizer
@@ -0,0 +1,86 @@
Fix ChatGPT DOM extractor formatting defects discovered during manual testing.
Observed defects:
1. Bold text is being extracted with comma-like artifacts instead of Markdown emphasis.
Example bad output:
what we know
,
what is plausible
, and
what is not supported by the evidence
.
Expected output:
**what we know**, **what is plausible**, and **what is not supported by the evidence**.
2. Quoted text is being surrounded by em-dash artifacts.
Example bad output:
So the author's central educational point—
"Jet streams matter."
—is absolutely correct.
Expected output:
So the author's central educational point — "Jet streams matter." — is absolutely correct.
Likely cause:
The extractor is walking DOM nodes and treating presentational or punctuation-only nodes incorrectly, especially around strong/bold tags and quotation/punctuation boundaries.
Requirements:
1. Update src/chatgptExtractor.js only unless integration requires a small change elsewhere.
2. Improve inline formatting extraction:
- Convert <strong> and <b> to Markdown bold: **text**
- Convert <em> and <i> to Markdown italics: *text*
- Convert inline <code> to `text`
- Preserve surrounding punctuation in natural sentence order.
- Do not emit isolated punctuation artifacts on separate lines.
3. Fix punctuation spacing:
- Do not place commas, periods, semicolons, colons, exclamation marks, or question marks on their own line.
- Attach closing punctuation to the preceding token.
- Preserve spaces between words.
- Avoid inserting line breaks between inline elements unless the source node is block-level.
4. Fix quote/em-dash handling:
- Do not treat em dashes as block separators.
- Keep em dashes inline with surrounding text.
- Normalize this pattern:
text—
"quote"
—text
into readable inline prose where possible.
5. Preserve block formatting:
- Paragraphs should remain paragraphs.
- Lists should remain readable.
- Code blocks should still use fenced Markdown.
- Headings should remain Markdown headings where already implemented.
6. Add helper functions if useful:
- isInlineElement(node)
- isBlockElement(node)
- appendInlineText(parts, text)
- normalizeInlineMarkdown(text)
- cleanupPunctuationSpacing(text)
7. Add lightweight manual test fixtures or comments with examples:
- bold phrase followed by comma
- bold phrase followed by comma and another bold phrase
- quoted phrase surrounded by em dashes
- assistant message with paragraph + list + code block
8. Do not add AI processing.
9. Do not change Markdown normalizer behavior unless absolutely necessary.
10. Do not change download writer behavior.
Acceptance criteria:
- Bold text exports as **bold text**.
- Italics export as *italic text*.
- Inline code exports as `inline code`.
- Commas remain attached to the correct phrase.
- Em dashes remain inline and readable.
- No punctuation-only lines are introduced by inline formatting.
- Code blocks remain intact.
- Message order and role detection still work.
@@ -0,0 +1,100 @@
Fix ChatGPT DOM extractor block-structure defects discovered during manual testing.
Observed defect:
Large assistant responses are being flattened into plain lines. Paragraph breaks, list structure, indentation, and inline emphasis boundaries are being lost.
Example bad output:
There's a real pattern here, but it's useful to separate
public narratives
from
the underlying issues
.
Over the past several decades...
Expected behavior:
There's a real pattern here, but it's useful to separate **public narratives** from **the underlying issues**.
Over the past several decades, there have indeed been many highly publicized predictions of catastrophic outcomes:
- Nuclear war during the Cold War.
- Concerns about global cooling in the 1970s...
- Y2K causing widespread infrastructure failures.
Root cause:
The extractor is likely using innerText/textContent or recursively joining nodes without distinguishing inline elements from block elements.
Requirements:
1. Update src/chatgptExtractor.js.
2. Implement DOM-to-Markdown extraction that treats block and inline elements differently:
- Inline elements must remain inline.
- Block elements must create paragraph/list/code boundaries.
- Do not insert line breaks around inline tags like strong, em, span, a, code.
3. Preserve paragraphs:
- Each <p> becomes one paragraph.
- Paragraphs separated by one blank line.
- Inline formatting inside paragraphs must remain inline.
4. Preserve unordered lists:
- <ul><li>Item</li></ul> becomes:
- Item
- Preserve nested list indentation where practical.
5. Preserve ordered lists:
- <ol><li>Item</li></ol> becomes:
1. Item
2. Item
6. Preserve blockquotes where detectable:
- Prefix quoted block lines with >.
7. Preserve headings:
- h1-h6 become Markdown headings.
8. Preserve code blocks:
- <pre><code>...</code></pre> becomes fenced code.
- Do not wrap or reformat code content.
9. Preserve inline formatting:
- <strong>/<b> → **text**
- <em>/<i> → *text*
- inline <code> → `text`
- <a> may preserve readable text only for MVP.
10. Add a cleaner that operates after Markdown generation:
- Remove punctuation-only lines caused by extraction.
- Collapse 3+ blank lines to 2.
- Remove spaces before punctuation.
- Preserve content inside fenced code blocks unchanged.
11. Do not use innerText for whole message containers as the primary extraction method.
- It may be used only as a fallback when DOM parsing fails.
12. Add small local test fixtures or comments for:
- paragraph with bold inline phrases
- paragraph followed by unordered list
- paragraph followed by numbered list
- nested list if easy
- code block
- quote/em dash sentence
Suggested helper functions:
- nodeToMarkdown(node, context)
- childrenToInlineMarkdown(node, context)
- blockChildrenToMarkdown(node, context)
- listToMarkdown(listNode, context)
- listItemToMarkdown(liNode, context)
- codeBlockToMarkdown(preNode)
- cleanupMarkdownOutsideCodeFences(markdown)
Acceptance criteria:
- The sample assistant response preserves paragraphs.
- Catastrophe examples become a Markdown bullet list.
- The “important distinctions” examples become separate paragraphs or list items if ChatGPT rendered them as list items.
- Bold phrases remain inline, not on separate lines.
- Periods and commas remain attached to the correct sentence.
- Code blocks remain intact.
- Message order and role detection are unchanged.
- No LLM calls.
- No Project Thoth application dependency.
@@ -0,0 +1,76 @@
Fix ChatGPT DOM extractor table handling.
Observed defect:
Tables in ChatGPT responses are being flattened into line-by-line text instead of Markdown tables.
Example bad output:
Goal
Good Plant Choices
Attract deer
Hostas, daylilies, many ornamentals
Feed pollinators
Native flowering perennials, shrubs, trees
Expected output:
| Goal | Good Plant Choices |
|---|---|
| Attract deer | Hostas, daylilies, many ornamentals |
| Feed pollinators | Native flowering perennials, shrubs, trees |
| Prevent deer damage | Deer-resistant species |
| Support overall biodiversity | A mix of native plants with different bloom times |
Requirements:
1. Update src/chatgptExtractor.js.
2. Add table-to-Markdown support:
- Detect <table> elements.
- Extract rows from <tr>.
- Extract header cells from <th>.
- Extract data cells from <td>.
- Preserve cell order.
- Trim cell text.
- Convert inline formatting inside cells using existing inline extractor.
3. Markdown table rules:
- First row should become the header row.
- If the table has <thead>, use that as the header.
- If no <thead> exists but the first row uses <th>, use that row as the header.
- If no header exists, use the first row as the header for MVP.
- Add separator row using `---`.
- Escape pipe characters inside cells as `\|`.
- Replace internal newlines in cells with `<br>` or a single space.
4. Example output format:
| Goal | Good Plant Choices |
|---|---|
| Attract deer | Hostas, daylilies, many ornamentals |
5. Ensure table extraction is treated as a block element:
- Tables should be separated from surrounding paragraphs by blank lines.
- Do not also extract duplicate flattened table text from child nodes.
6. Preserve other existing behavior:
- Paragraphs remain paragraphs.
- Lists remain Markdown lists.
- Code blocks remain fenced code.
- Bold/italic/inline code remain inline.
- Message order and role detection unchanged.
Suggested helper:
function tableToMarkdown(tableNode, context) {
// return markdown table string
}
Acceptance criteria:
- ChatGPT tables export as valid Markdown tables.
- Headers and rows are preserved.
- Pipe characters inside cells are escaped.
- Tables are not duplicated as flattened text.
- Surrounding prose remains intact.
- No LLM calls.
- No Project Thoth application dependency.
@@ -0,0 +1,68 @@
Fix ChatGPT DOM extractor external link handling.
Observed defect:
External links in ChatGPT responses are being extracted as plain text only.
Example bad output:
Services such as:
Amazon Prime Video
Apple TV
Fandango at Home (Vudu)
Expected behavior:
Preserve links as Markdown links when href is available:
- [Amazon Prime Video](https://...)
- [Apple TV](https://...)
- [Fandango at Home (Vudu)](https://...)
Requirements:
1. Update src/chatgptExtractor.js.
2. Add Markdown link support for <a> elements:
- Extract visible link text using the existing inline extractor.
- Extract href.
- If href is valid, output `[text](href)`.
- If href is missing or unsafe, fall back to plain visible text.
3. Preserve surrounding inline formatting:
- Links inside paragraphs remain inline.
- Links inside lists remain list items.
- Links inside table cells remain Markdown links.
- Do not force every link onto its own line unless the original block structure does.
4. URL handling:
- Preserve http:// and https:// links.
- Preserve mailto: links if encountered.
- Ignore or flatten javascript:, data:, blob:, and empty href values.
- Decode or normalize only as needed for readable Markdown.
- Escape closing parentheses in URLs if necessary.
5. Link text handling:
- Trim excessive whitespace.
- If link text is empty, use the href as the visible text.
- Escape square brackets in link text.
- Do not duplicate the URL if ChatGPT already displays the raw URL as the link text.
6. Preserve existing behavior:
- Paragraphs remain paragraphs.
- Lists remain Markdown lists.
- Tables remain Markdown tables.
- Code blocks remain fenced code.
- Bold, italics, and inline code remain inline.
- Message order and role detection unchanged.
Suggested helper functions:
- linkToMarkdown(anchorNode, context)
- isSafeHref(href)
- escapeMarkdownLinkText(text)
- escapeMarkdownLinkUrl(url)
Acceptance criteria:
- External links export as valid Markdown links.
- Plain text without links remains unchanged.
- Unsafe href values are not preserved as Markdown links.
- Links in lists, paragraphs, and tables remain readable.
- No LLM calls.
- No Project Thoth application dependency.
@@ -0,0 +1,47 @@
Fix Task 3A conversation discovery over-capture and duplication.
Problem:
The extractor is now capturing the ChatGPT application shell, sidebar, navigation, composer UI, and duplicate parent/child containers. It also captures large flattened blocks containing both "You said" and "ChatGPT said" instead of discrete messages.
Goal:
Discover only the conversation transcript stream and return one message object per actual conversation turn.
Requirements:
1. Exclude ChatGPT app shell:
- Skip sidebar/nav/history/project/account areas.
- Skip composer/input area.
- Skip buttons, menus, share controls, footer text, scripts, and hidden UI.
2. Scope discovery to the main conversation region:
- Prefer main[role="main"] or the primary conversation scroll container.
- Do not search the entire document body unless used only as a fallback.
3. Deduplicate containers:
- Do not capture both a parent and its child if both contain the same message text.
- Prefer the smallest stable container that represents one message.
- Track DOM containment and normalized text fingerprints to remove duplicates.
4. Split transcript turns:
- Do not allow one captured block to contain multiple "You said:" / "ChatGPT said:" turns.
- Each actual user or assistant turn should become a separate message.
5. Keep fallback behavior, but only after filtering:
- Unknown containers are allowed, but they must come from the conversation region, not the app shell.
- Unknown should mean unknown conversation content, not random ChatGPT UI.
6. Add debug counts:
- candidateContainersFound
- afterShellFilter
- afterDeduplication
- messagesReturned
- skippedShellContainers
- skippedDuplicateContainers
Acceptance criteria:
- No "Skip to content" block.
- No sidebar chat history.
- No repeated full transcript blocks.
- No window.__oai_logHTML or script text.
- Actual user/assistant turns are separate messages.
- Image generation/edit placeholders may remain as unknown or unsupported, but only once.
- Completeness remains favored, but not by capturing the whole app shell.
@@ -0,0 +1,47 @@
Fix ChatGPT extractor message granularity and role inheritance.
Problem:
The extractor is now discovering conversation content, but it is treating paragraphs, list items, links, inline fragments, and code blocks as separate messages. This creates many Unknown blocks and misclassified User/Assistant fragments.
Goal:
Return one message object per actual ChatGPT turn, not one message per child DOM block.
Requirements:
1. Identify the outer message turn container first.
- A single user prompt should produce one User message.
- A single assistant response should produce one Assistant message.
- Child paragraphs, lists, tables, links, and code blocks must be content within that message, not separate messages.
2. Role must be assigned from the outer message container.
- Child nodes must inherit the parent message role.
- Do not classify inline fragments independently.
- Do not classify text fragments like "re", "can", "own", "have" as User messages.
3. Deduplicate repeated turns.
- The same user prompt should not appear twice.
- Use normalized text fingerprinting after role assignment.
4. Fix link serialization.
- A Markdown link must be emitted as one inline unit.
- Do not split `[MakeMKV](https://www.makemkv.com)` into separate Unknown/Assistant blocks.
5. Keep unsupported placeholders inside the parent turn.
- `[Unsupported ChatGPT content: ...]` should not be split across Unknown and Assistant blocks.
6. Debug output should include:
- turnContainersFound
- messagesReturned
- duplicateTurnsSkipped
- orphanChildBlocksSkipped
- unknownTurnsReturned
Acceptance criteria:
- The Jellyfin conversation starts with exactly one User message for the opening question.
- The first assistant answer is one Assistant message containing all options, lists, links, and code blocks.
- The Tiny Media Manager question is one User message.
- The SmartScreen answer is one Assistant message.
- The .nfo question is one User message.
- The .nfo answer is one Assistant message.
- No `.com)` standalone Assistant message.
- No `re`, `can`, `own`, `have`, or similar fragments as User messages.
- Unknown messages are allowed only for actual unsupported ChatGPT turn-level content, not paragraph/list/link fragments.
@@ -0,0 +1,142 @@
# Work Order: Resolve connectorRuntime Module-Loading Regression
## Current defect
The browser extension no longer creates a Markdown file in Downloads.
The extension console reports:
```text
chrome-extension://.../connectorRuntime.js:1
Uncaught SyntaxError: Cannot use import statement outside a module
```
A separate page-level error also appears:
```text
QuotaExceededError: Failed to execute 'setItem' on 'Storage':
Setting the value of
'cache/user-JY5yPfTiyFSprhKcrYXMA2gG/9e03f04b-55ee-4773-bb7b-47df25276387/system-connectors'
exceeded the quota.
```
Treat the `QuotaExceededError` as originating from the ChatGPT application unless repository evidence shows that the extension interacts with that storage key.
## Objective
Restore the extension to the point where invoking capture reliably creates a Markdown file in the browser's Downloads location.
## Required investigation
1. Identify where `connectorRuntime.js` is loaded.
2. Identify the `import` statement or statements causing the parse failure.
3. Determine whether `connectorRuntime.js` is intended to be:
* an ES module,
* a bundled traditional script, or
* imported by another module.
4. Compare the current loading configuration with the last working implementation or repository history, where available.
5. Determine whether the regression was introduced by:
* adding an `import` statement to a non-module content script,
* changing the manifest entry,
* changing script injection logic,
* moving code without updating module boundaries, or
* failing to include the file in the build/bundle process.
## Constraints
* Make the smallest change necessary to restore the intended module-loading behavior.
* Do not redesign the DOM extractor.
* Do not change conversation-selection logic.
* Do not modify Markdown normalization except where directly required by the module-loading repair.
* Do not suppress the syntax error without addressing its cause.
* Do not treat the ChatGPT `QuotaExceededError` as the extension root cause without evidence.
* Preserve Manifest V3 compatibility.
## Possible correction paths
Evaluate the repository and choose the correction consistent with the existing architecture. Possible approaches include:
### Module-based loading
If `connectorRuntime.js` is intended to be an ES module, ensure it is loaded through a supported module mechanism.
For an extension service worker, this may require:
```json
{
"background": {
"service_worker": "background.js",
"type": "module"
}
}
```
For a page or extension HTML document, this may require:
```html
<script type="module" src="connectorRuntime.js"></script>
```
Do not assume that declaring a content script as a module is supported in the same way. Verify how the file is being executed.
### Bundled or non-module loading
If `connectorRuntime.js` is a content script or injected traditional script, remove direct runtime `import` syntax by using the repository's intended bundling process or by restoring the prior dependency-loading pattern.
Do not simply concatenate files unless that is already the projects build design.
### Dynamic import
Use `import()` only if it is valid in the execution context and consistent with the extension architecture. Do not use it merely to hide a structural module problem.
## Diagnostics
Add or retain narrowly scoped diagnostic logging around:
```text
[Thoth] connector runtime loaded
[Thoth] capture requested
[Thoth] extraction completed
[Thoth] markdown generated
[Thoth] download requested
[Thoth] download completed: <download ID>
```
Errors should be logged with the failed stage and exception.
## Acceptance criteria
1. The extension loads without:
```text
Cannot use import statement outside a module
```
2. Clicking the capture action reaches `connectorRuntime.js`.
3. The extension creates a `.md` file in Downloads.
4. The resulting file is non-empty.
5. The extension console contains no uncaught extension-originated exception during capture.
6. The ChatGPT page-level `QuotaExceededError`, if still present, does not prevent extension capture.
7. Document:
* the root cause,
* the file or manifest entry responsible,
* the corrective change,
* why the selected module-loading approach is correct,
* and the verification performed.
## Out of scope
* Improving DOM selectors
* Solving missing historical turns
* Improving table conversion
* Improving citation conversion
* Changing the conversation intermediate model
* General refactoring
@@ -0,0 +1,50 @@
Create a minimal Manifest V3 browser extension for Chrome/Edge named "Project Thoth Capture".
Goal:
Build only the extension skeleton for Sprint Task 1. Do not implement ChatGPT extraction yet.
Requirements:
1. Use Manifest V3.
2. Add a browser toolbar button.
3. Add a background service worker.
4. When the toolbar button is clicked, log a message to the console: "Project Thoth capture clicked".
5. Include placeholder extension icons.
6. Include a README with local installation instructions for Chrome/Edge developer mode.
7. Keep the project simple and dependency-free. Use plain JavaScript, HTML only if needed, and no build tools.
Project structure:
project-thoth/applications/chatgpt-capture/
manifest.json
src/
background.js
icons/
icon16.png
icon48.png
icon128.png
README.md
Manifest details:
- name: "Project Thoth Capture"
- description: "Capture ChatGPT conversations for Project Thoth."
- version: "0.1.0"
- manifest_version: 3
- permissions: ["activeTab", "scripting", "downloads"]
- host_permissions:
- "https://chatgpt.com/*"
- "https://chat.openai.com/*"
- action:
- default_title: "Save to Project Thoth"
- default_icon using the three icon files
- background:
- service_worker: "src/background.js"
Implementation notes:
- The background service worker should register chrome.action.onClicked.
- On click, log the current tab URL and title if available.
- Do not inject scripts yet.
- Do not download files yet.
- Do not access page DOM yet.
- Add TODO comments for future capture flow.
Return the complete file tree and the contents of each file.
@@ -0,0 +1,50 @@
Implement Task 2 — Active Tab Capture Flow for the Project Thoth ChatGPT Capture Connector MVP.
Context:
- This is a Manifest V3 Chrome/Edge extension.
- Capture Connectors must only capture source material; they must not reason, call LLMs, generate metadata, or interact with the Project Thoth application yet.
- The MVP action is a browser toolbar button labeled “Save to Project Thoth.”
- The extension should use minimal permissions: activeTab, scripting, and downloads as needed later.
- This task focuses only on toolbar click → active tab lookup → script injection → service worker/content script message passing.
Architecture requirements:
1. Use the Manifest V3 service worker as the orchestration point.
2. On toolbar button click, get the currently active tab.
3. Verify the current tab URL is ChatGPT:
- https://chatgpt.com/*
- https://chat.openai.com/*
4. Inject or execute a content script using chrome.scripting.
5. The content script should send a message back to the service worker with a placeholder capture payload.
6. The service worker should receive and log the payload.
7. Do not implement the real ChatGPT DOM extractor yet.
8. Do not implement Markdown normalization yet.
9. Do not implement downloads yet except leaving clear TODO boundaries.
10. Keep ChatGPT-specific extraction isolated for the next task.
Expected placeholder payload:
{
sourcePlatform: "ChatGPT",
title: document.title || "ChatGPT Conversation",
url: window.location.href,
capturedAt: new Date().toISOString(),
messages: []
}
Files to update or create:
- manifest.json
- src/background.js
- src/chatgptExtractor.js if needed as a placeholder injected module
- README.md if local testing instructions need updating
Acceptance criteria:
- Extension loads locally in Chrome/Edge.
- Toolbar button click triggers the service worker.
- On a ChatGPT tab, the content script executes successfully.
- A placeholder capture payload is returned to the service worker.
- On a non-ChatGPT tab, the extension does not inject and logs or displays a simple error.
- No LLM calls.
- No Project Thoth application dependency.
- No vault writes.
Keep the implementation simple, readable, and modular so Task 3 can replace the placeholder with the real ChatGPT DOM extractor.
@@ -0,0 +1,124 @@
Implement Task 3 — ChatGPT DOM Extractor for the Project Thoth ChatGPT Capture Connector MVP.
Context:
- This is a Manifest V3 Chrome/Edge extension.
- Task 1 extension skeleton is complete.
- Task 2 activeTab capture flow is complete.
- The service worker can inject or invoke a content script and receive a placeholder payload.
- This task replaces the placeholder capture logic with a ChatGPT-specific DOM extractor.
- Capture Connectors must only capture source material. Do not reason, summarize, classify, call LLMs, generate Project Thoth metadata, or write to the vault.
Goal:
Create a ChatGPT-specific DOM extractor module that returns an ordered conversation capture payload with title, URL, capture timestamp, and messages.
Files to update or create:
- src/chatgptExtractor.js
- src/background.js only if needed to integrate the extractor result
- README.md only if testing instructions need updating
Expected output shape:
{
sourcePlatform: "ChatGPT",
title: "Detected conversation title",
url: "https://chatgpt.com/...",
capturedAt: "2026-07-08T...",
messages: [
{
role: "user",
content: "..."
},
{
role: "assistant",
content: "..."
}
]
}
Extractor requirements:
1. Extract the conversation title if available.
- Prefer a visible conversation title if one exists.
- Fall back to document.title.
- Fall back to "ChatGPT Conversation".
2. Extract ordered message blocks.
- Preserve the page order of messages.
- Return messages in the same order they appear in the conversation.
- Ignore navigation, sidebar, composer/input box, buttons, menus, and unrelated UI chrome.
3. Detect speaker role when possible.
- Detect user messages as role: "user".
- Detect assistant messages as role: "assistant".
- If role cannot be determined, use role: "unknown" rather than guessing too aggressively.
- Keep role detection logic isolated and easy to revise because ChatGPT DOM structure may change.
4. Extract text content.
- Extract the readable content of each message.
- Trim leading/trailing whitespace.
- Collapse excessive blank lines where appropriate.
- Do not include copy buttons, feedback controls, model labels, timestamps, or hidden UI text.
5. Preserve basic formatting where practical.
- Preserve paragraph breaks.
- Preserve markdown-like headings.
- Preserve bullet and numbered lists as readable text.
- Preserve code blocks using fenced markdown when detectable.
- Preserve inline code as readable text when detectable.
- Do not attempt perfect HTML-to-Markdown conversion in this task.
6. Handle code blocks.
- Detect pre/code blocks inside assistant messages.
- Output fenced code blocks.
- Include a language tag if available from the DOM; otherwise use a plain triple-backtick fence.
- Avoid duplicating code block content in the surrounding extracted text.
7. Be defensive against ChatGPT DOM instability.
- Use multiple selector strategies where reasonable.
- Prefer semantic attributes when available, such as data-testid, role, aria-label, or known message container patterns.
- Keep selectors centralized near the top of the extractor.
- Add comments explaining selector assumptions.
8. Return useful failure states.
- If no messages are found, return a structured payload with messages: [] and an error field such as:
{
error: "No ChatGPT conversation messages were detected."
}
- Do not throw unhandled exceptions from normal extraction failure.
9. No browser download logic in this task.
10. No markdown normalizer in this task.
11. No Project Thoth application dependency.
12. No LLM calls.
Suggested module API:
export function extractChatGPTConversation() {
return {
sourcePlatform: "ChatGPT",
title,
url: window.location.href,
capturedAt: new Date().toISOString(),
messages
};
}
Suggested helper functions:
- getConversationTitle()
- findMessageElements()
- detectMessageRole(element)
- extractMessageContent(element)
- extractNodeAsMarkdown(node)
- normalizeWhitespace(text)
Integration:
- The activeTab capture flow should invoke extractChatGPTConversation() from the injected content context.
- The service worker should receive and log the returned payload.
- Existing Task 2 message passing should remain intact.
Acceptance criteria:
- On an open ChatGPT conversation, clicking the extension button returns a payload with title, URL, capturedAt, and ordered messages.
- User and assistant roles are detected when possible.
- Message content is readable and excludes obvious UI chrome.
- Paragraphs, lists, headings, and code blocks remain usable in the extracted content.
- If no conversation is detected, the payload clearly reports that no messages were found.
- The implementation remains modular so Task 4 can consume the payload and convert it into canonical conversation.md format.
@@ -0,0 +1,34 @@
Implement Task 3A — ChatGPT Conversation Discovery.
Goal:
Refactor the extractor so it reliably discovers all conversation-bearing containers before attempting high-quality Markdown conversion.
Primary requirement:
Completeness is more important than formatting.
Deliverables:
- findConversationContainers()
- classifyContainer(container)
- extractContainerFallback(container)
- debug reporting
Rules:
- Never silently skip a candidate container.
- If role cannot be determined, mark role: "unknown".
- If type cannot be determined, mark type: "unknown".
- Preserve rawText for every discovered container.
- Preserve a short htmlSnippet for unknown containers.
- Return debug counts:
- containersFound
- parsedMessages
- userMessages
- assistantMessages
- unknownContainers
Do not solve tables, links, bold, lists, or Markdown formatting in this task.
Those belong to Task 3B.
Acceptance criteria:
- Previously lost conversation sections are captured as either user, assistant, or unknown.
- Unknown sections are included in output rather than dropped.
- Debug output makes it clear what was discovered and what was not confidently parsed.
@@ -0,0 +1,87 @@
Implement Task 3B — ChatGPT DOM-to-Markdown Conversion.
Context:
- Task 3A discovers all conversation-bearing containers.
- 3A prioritizes completeness and returns containers/messages with role, type, rawText, htmlSnippet, and debug data.
- This task improves formatting quality for discovered ChatGPT message containers.
- Do not change the 3A discovery strategy unless absolutely necessary.
Goal:
Convert discovered ChatGPT message DOM content into readable Markdown while preserving source structure.
Primary rule:
Do not lose content. If structured conversion fails, fall back to rawText.
Requirements:
1. Preserve paragraphs:
- <p> becomes one paragraph.
- Paragraphs separated by one blank line.
- Inline elements stay inline.
2. Preserve inline formatting:
- <strong>/<b> → **text**
- <em>/<i> → *text*
- inline <code> → `text`
- <a href> → [text](url)
3. Preserve lists:
- <ul><li> → - item
- <ol><li> → 1. item
- Preserve nested indentation where practical.
4. Preserve code blocks:
- <pre><code> → fenced Markdown.
- Preserve code exactly.
- Include language tag if detectable.
5. Preserve tables:
- <table> → Markdown table.
- Use <th> or first row as header.
- Escape pipe characters inside cells.
- Do not duplicate flattened table text.
6. Preserve blockquotes:
- Prefix lines with >.
7. Preserve headings:
- h1-h6 → Markdown headings.
8. Fix spacing and punctuation:
- Do not put punctuation on its own line.
- Do not break around bold/italic/link spans.
- Keep em dashes inline.
- Collapse excessive blank lines outside code fences only.
9. Preserve attachments/tool/artifact placeholders:
- If a container represents image generation, file upload, artifact, or tool output and cannot be fully converted, emit a readable placeholder plus rawText.
- Example:
[Unsupported ChatGPT content: image generation]
<raw text>
10. Keep conversion separate from discovery:
- discovery decides what containers exist.
- conversion decides how each container becomes Markdown.
Suggested functions:
- containerToMarkdown(container)
- nodeToMarkdown(node, context)
- blockChildrenToMarkdown(node, context)
- inlineChildrenToMarkdown(node, context)
- listToMarkdown(listNode, context)
- tableToMarkdown(tableNode, context)
- codeBlockToMarkdown(preNode)
- linkToMarkdown(anchorNode, context)
- cleanupMarkdownOutsideCodeFences(markdown)
Acceptance criteria:
- Previously discovered messages are still all present.
- Bold text exports as **bold text**.
- Links export as Markdown links.
- Tables export as Markdown tables.
- Lists remain lists.
- Code blocks remain fenced and unchanged.
- Paragraph breaks are preserved.
- No punctuation-only lines are introduced.
- Unknown/unsupported containers are not dropped.
- No LLM calls.
- No Project Thoth application dependency.
@@ -0,0 +1,19 @@
Task 3C — Render All Turns Before Capture
Goal:
Before extraction, automatically scroll through the ChatGPT conversation so virtualized turns are rendered in the DOM.
Approach:
1. Find all section[data-testid^="conversation-turn-"][data-turn].
2. Scroll each turn into view in ascending order.
3. Wait briefly after each scroll for React to render content.
4. After reaching the bottom, run the turn-section extractor.
5. If any turns are still empty, mark them not_rendered.
Requirements:
- Do not click buttons.
- Do not submit messages.
- Do not alter conversation content.
- Use scrollIntoView only.
- Add timeout protection.
- Show status: "Preparing conversation for capture..."
@@ -0,0 +1,146 @@
Implement Task 4 — Markdown Normalizer for the Project Thoth ChatGPT Capture Connector MVP.
Context:
- Task 3 now returns a ChatGPT capture payload with:
- sourcePlatform
- title
- url
- capturedAt
- messages[]
- This task converts that extracted payload into canonical Project Thoth conversation.md format.
- Capture Connectors still only capture and transmit source material.
- Do not reason, summarize, classify, call LLMs, generate Project Thoth Source Metadata, generate Manifests, generate Harvests, or write to the vault.
Goal:
Create a markdown normalization module that converts extracted ChatGPT messages into a single Markdown string.
Files to create or update:
- src/markdownNormalizer.js
- src/background.js only as needed to call the normalizer
- README.md only if testing instructions need updating
Expected input shape:
{
sourcePlatform: "ChatGPT",
title: "Detected conversation title",
url: "https://chatgpt.com/...",
capturedAt: "2026-07-08T...",
messages: [
{
role: "user",
content: "..."
},
{
role: "assistant",
content: "..."
}
]
}
Expected Markdown output format:
# Conversation Title
Captured: 2026-07-08T15:30:00.000Z
Source Platform: ChatGPT
Source URL: https://chatgpt.com/...
---
## User
Message content
---
## Assistant
Message content
---
Requirements:
1. Export a function named normalizeConversationToMarkdown(capturePayload).
2. Preserve message order exactly as received in capturePayload.messages.
3. Add a metadata header containing:
- Title as H1
- Captured timestamp
- Source Platform
- Source URL
4. Use speaker boundaries:
- role: "user" → ## User
- role: "assistant" → ## Assistant
- role: "system" → ## System
- any unknown or missing role → ## Unknown
5. Preserve message content as already extracted by the DOM extractor.
- Do not summarize.
- Do not rewrite.
- Do not classify.
- Do not run AI processing.
- Do not alter code block contents.
6. Normalize only structural Markdown concerns:
- Trim leading/trailing whitespace from each message.
- Ensure exactly one blank line after headings.
- Ensure `---` separators between messages.
- Ensure final output ends with a newline.
- Avoid excessive blank lines caused by extraction artifacts.
- Do not collapse intentional line breaks inside code fences.
7. Escape or sanitize the title only as needed for Markdown safety.
- Remove excessive whitespace.
- If no title exists, use "ChatGPT Conversation".
8. If messages is empty or missing:
- Still produce a valid Markdown file.
- Include the metadata header.
- Add:
## Capture Notice
No conversation messages were detected.
9. Include capture errors if present:
- If capturePayload.error exists, include:
## Capture Error
<error text>
10. Keep the module generic enough that later connectors can reuse the same normalizer with other source platforms.
Suggested implementation:
export function normalizeConversationToMarkdown(capturePayload) {
// validate payload
// normalize title
// write metadata header
// write optional error
// write messages in order
// return markdown string
}
Suggested helper functions:
- normalizeTitle(title)
- normalizeRoleHeading(role)
- normalizeMessageContent(content)
- collapseExcessBlankLinesOutsideCodeBlocks(text)
Integration:
- In src/background.js, after receiving the capture payload from the content script, call normalizeConversationToMarkdown(payload).
- For now, log the Markdown output or hold it for Task 5 download writing.
- Do not implement the downloads API in this task unless it already exists from a previous step.
Acceptance criteria:
- A valid Markdown string is produced from a normal ChatGPT capture payload.
- Metadata header appears at the top.
- Message order is preserved.
- User and assistant sections are clearly separated.
- Code blocks remain intact.
- Empty captures produce a useful Markdown file with a Capture Notice.
- No LLM calls.
- No Project Thoth application dependency.
- No vault writes.
@@ -0,0 +1,126 @@
Implement Task 5 — Download Writer for the Project Thoth ChatGPT Capture Connector MVP.
Context:
- Task 3 extracts a ChatGPT conversation payload.
- Task 4 converts that payload into canonical Project Thoth Markdown.
- This task saves the Markdown file to the users Downloads folder.
- Capture Connectors still only capture and transmit source material.
- Do not reason, summarize, classify, call LLMs, generate Source Metadata, generate Manifests, generate Harvests, or write directly to the vault.
Goal:
Implement Markdown download using the browser downloads API with safe filename generation.
Files to create or update:
- src/filename.js
- src/background.js
- manifest.json if the downloads permission is not already present
- README.md only if testing instructions need updating
Manifest requirement:
Ensure manifest.json includes:
"permissions": [
"activeTab",
"scripting",
"downloads"
]
Filename requirements:
1. Generate a safe filename using:
- capture date
- source platform
- conversation title
2. Preferred format:
YYYY-MM-DD - ChatGPT - Conversation Title.md
3. If title is missing, use:
YYYY-MM-DD - ChatGPT - Conversation.md
4. Sanitize invalid filename characters:
- < > : " / \ | ? *
- control characters
- excessive whitespace
- leading/trailing dots or spaces
5. Keep filename length reasonable.
- Limit final filename to approximately 180 characters including `.md`.
- Preserve the date, platform, and extension.
- Truncate the title portion if needed.
6. If capturedAt is invalid or missing:
- Use the current date.
7. Add a timestamp suffix only when needed to reduce collision risk or when using downloads API conflict handling.
Suggested module API:
export function createConversationFilename(capturePayload) {
return "2026-07-08 - ChatGPT - Conversation Title.md";
}
Suggested helpers:
- formatDateForFilename(dateValue)
- sanitizeFilenamePart(value)
- truncateFilename(filename, maxLength)
Download requirements:
1. Export or implement a function that downloads Markdown as a `.md` file.
2. Use chrome.downloads.download.
3. Create a Blob from the Markdown content.
4. Create an object URL using URL.createObjectURL(blob).
5. Pass the object URL to chrome.downloads.download.
6. Use the generated filename.
7. Save to Downloads by using only the filename, not an absolute path.
8. Use conflictAction: "uniquify" to avoid overwriting existing captures.
9. Revoke the object URL after download is started or after the callback returns.
Suggested implementation shape in background.js:
const blob = new Blob([markdown], { type: "text/markdown;charset=utf-8" });
const url = URL.createObjectURL(blob);
chrome.downloads.download(
{
url,
filename,
saveAs: false,
conflictAction: "uniquify"
},
(downloadId) => {
URL.revokeObjectURL(url);
if (chrome.runtime.lastError || !downloadId) {
// Handle download failure
}
// Handle success
}
);
Integration:
- After activeTab capture succeeds, call normalizeConversationToMarkdown(payload).
- Generate filename from the same payload.
- Download the Markdown file.
- Log success/failure clearly.
- Do not add complex UI yet unless already present.
- Task 6 will handle user-facing feedback.
Error handling:
- If Markdown generation succeeds but download fails, report or log:
"The conversation was captured, but the file could not be saved."
- If filename generation fails, fall back to:
conversation.md
Acceptance criteria:
- Clicking the extension button on a ChatGPT conversation downloads a `.md` file.
- File is saved to the browsers default Downloads folder.
- Filename is human-readable and safe.
- Invalid filename characters are removed or replaced.
- Existing files are not overwritten.
- Markdown content matches Task 4 output.
- No LLM calls.
- No Project Thoth application dependency.
- No vault writes.
@@ -0,0 +1,101 @@
Implement Task 6 — User Feedback for the Project Thoth ChatGPT Capture Connector MVP.
Context:
- Task 3 extracts the ChatGPT conversation.
- Task 4 normalizes the capture to Markdown.
- Task 5 downloads the Markdown file to Downloads.
- This task adds simple user-facing feedback for success and failure states.
- Keep the implementation lightweight.
- Do not add AI processing, Project Thoth app integration, vault writes, manifests, metadata generation, or harvest generation.
Goal:
Show clear feedback to the user when capture succeeds or fails.
Required user-facing messages:
1. Success:
"Saved to Downloads."
2. Not on ChatGPT:
"Open a ChatGPT conversation before saving to Project Thoth."
3. No ChatGPT conversation detected:
"No ChatGPT conversation was detected on this page."
4. Capture failed:
"Capture failed. Try scrolling through the conversation and capturing again."
5. Download failed:
"The conversation was captured, but the file could not be saved."
Implementation options:
Use the simplest reliable feedback mechanism for a Manifest V3 Chrome/Edge extension.
Preferred approach:
- Use chrome.action.setBadgeText and chrome.action.setTitle for lightweight status feedback.
- Badge examples:
- Success: "OK"
- Error: "ERR"
- Clear badge text after a short timeout.
- Also log detailed errors to the service worker console.
Optional approach if already using a popup:
- Display the same messages in the popup.
- Do not create a complex UI just for this task unless necessary.
Files to update:
- src/background.js
- manifest.json only if required
- README.md only if behavior/testing instructions need updating
Suggested helper function:
function showStatus(tabId, message, badgeText = "") {
chrome.action.setTitle({ tabId, title: message });
chrome.action.setBadgeText({ tabId, text: badgeText });
if (badgeText) {
setTimeout(() => {
chrome.action.setBadgeText({ tabId, text: "" });
chrome.action.setTitle({ tabId, title: "Save to Project Thoth" });
}, 3000);
}
}
Behavior requirements:
1. When the user clicks the toolbar button on a non-ChatGPT page:
- Do not inject the content script.
- Show the not-on-ChatGPT message.
- Log the reason.
2. When extraction returns no messages:
- Do not download an empty conversation unless the current architecture already intentionally supports capture notice files.
- Show the no-conversation-detected message.
- Log the payload for debugging.
3. When content script injection or extraction throws:
- Show the capture-failed message.
- Log the full error.
4. When Markdown normalization succeeds but downloads API fails:
- Show the download-failed message.
- Log chrome.runtime.lastError if available.
5. When download succeeds:
- Show the success message.
- Log filename and download id.
6. Keep all user-facing text centralized as constants.
7. Do not use alert() inside the page unless no extension-native feedback mechanism works.
Acceptance criteria:
- Success case displays "Saved to Downloads."
- Non-ChatGPT page displays "Open a ChatGPT conversation before saving to Project Thoth."
- No detected messages displays "No ChatGPT conversation was detected on this page."
- Capture/injection failure displays "Capture failed. Try scrolling through the conversation and capturing again."
- Download failure displays "The conversation was captured, but the file could not be saved."
- Badge/title feedback clears after a short delay.
- Console logs contain enough diagnostic information for manual testing.
- No LLM calls.
- No Project Thoth application dependency.
- No vault writes.
@@ -0,0 +1,123 @@
Refactor the Project Thoth ChatGPT Capture Connector around ChatGPT turn sections.
Problem:
The current extractor is guessing message boundaries and captures fragments, duplicates, app chrome, and out-of-order content.
The actual ChatGPT DOM exposes canonical turn containers:
section[data-testid^="conversation-turn-"][data-turn]
Examples:
- section[data-testid="conversation-turn-1"][data-turn="user"]
- section[data-testid="conversation-turn-2"][data-turn="assistant"]
Goal:
Use these turn sections as the only source of conversation turns.
Primary rule:
One section[data-testid^="conversation-turn-"][data-turn] becomes at most one captured message.
Do not recursively discover message-like descendants as separate messages.
Files to update:
- src/chatgptExtractor.js
- related tests or debug helpers if present
Discovery requirements:
1. Find turn sections with:
section[data-testid^="conversation-turn-"][data-turn]
2. Sort turns by their conversation-turn number.
- Parse N from data-testid="conversation-turn-N".
- Fall back to DOM order if parsing fails.
3. Determine role from:
- section.getAttribute("data-turn")
- fallback: descendant [data-message-author-role]
- allowed roles: user, assistant, system, tool, unknown
4. Extract only rendered turns.
- If the section has no meaningful content because it is virtualized/offscreen, return a placeholder message:
{
role,
content: "[Turn not rendered in DOM. Scroll this turn into view and recapture.]",
captureStatus: "not_rendered",
turnId,
turnIndex
}
- Do not search outside the turn to compensate.
5. For user turns:
- Prefer [data-message-author-role="user"].
- Extract text from user message bubble/content.
- Preserve attached image references as placeholders:
[Attached image: <alt text or src>]
- Exclude buttons and message actions.
6. For assistant turns:
- Prefer [data-message-author-role="assistant"] .markdown
- Convert the `.markdown` subtree to Markdown.
- If no `.markdown` exists but image generation content exists, emit:
[Generated image: <alt text>]
<src if available>
- Exclude response actions, buttons, Sources button, copy buttons, feedback controls, and hidden sr-only labels.
7. Do not capture:
- nav/sidebar
- composer
- action bars
- copy/edit/share buttons
- "Sources" button
- "ChatGPT can make mistakes"
- sr-only labels such as "You said:" or "ChatGPT said:"
DOM-to-Markdown requirements:
1. Preserve paragraphs, headings, lists, blockquotes, links, inline code, bold, italics, and fenced code blocks.
2. Code blocks in ChatGPT may contain nested CodeMirror markup:
- Prefer pre code textContent.
- Preserve whitespace.
- Detect language from visible code block label if available, otherwise omit language.
3. Links:
- Convert safe href links to [text](url).
- Ignore aria-hidden SVG/icon text inside links.
4. Images:
- For generated images, capture alt text and src as a Markdown image or placeholder:
![Generated image: ...](src)
- For user-uploaded images, emit:
[Attached image: <alt>](src)
- Do not duplicate thumbnail images from image galleries.
Virtualization requirement:
The DOM shows many offscreen turns as empty placeholder sections with data-is-intersecting="false" and only last-known-height. The extractor must report these as not_rendered instead of pretending the conversation was fully captured.
Add capture-level warning:
If any turns are not_rendered, include:
warning: "Some conversation turns were not rendered in the DOM. Scroll through the conversation and capture again."
Debug output:
Return debug fields:
- turnSectionsFound
- renderedTurnsCaptured
- notRenderedTurns
- userTurns
- assistantTurns
- imageTurns
- unknownTurns
- firstTurnIndex
- lastTurnIndex
Acceptance criteria:
- No sidebar/app shell text.
- No duplicated user prompts.
- No one paragraph/list item per message.
- One rendered ChatGPT turn becomes one Project Thoth message.
- The first Jellyfin question is one User message.
- The first Jellyfin answer is one Assistant message.
- The SmartScreen question is one User message.
- The SmartScreen answer is one Assistant message.
- The .nfo question is one User message.
- The .nfo answer is one Assistant message.
- Offscreen placeholder turns are explicitly marked not_rendered.
- Markdown formatting remains readable.
- No LLM calls.
- No Project Thoth app dependency.
- No vault writes.
@@ -0,0 +1,190 @@
Refactor the Project Thoth ChatGPT Capture Connector MVP to align with ADR-002 and ADR-003.
Goal:
Replace the current mixed extractor with the canonical Capture Connector pipeline:
1. Optional preprocessing
2. Conversation discovery
3. Intermediate Representation
4. Content transformation
5. Markdown serialization
6. Download output
Primary reason:
Current extraction still treats block-level content inside assistant responses as separate messages. The refactor must make one ChatGPT turn section equal one Project Thoth message.
Required architecture:
src/
background.js
connectors/
chatgpt/
chatgptConnector.js
chatgptDiscovery.js
chatgptPreprocessor.js
chatgptTurnRenderer.js
core/
conversationModel.js
markdownSerializer.js
htmlToMarkdown.js
filename.js
Pipeline:
background.js
→ chatgptConnector.captureConversation()
→ optional preprocessing
→ discoverConversation()
→ renderTurnsToMarkdown()
→ serializeConversationToMarkdown()
→ download Markdown
Discovery requirements:
1. Find only canonical ChatGPT turn sections:
section[data-testid^="conversation-turn-"][data-turn]
2. One section equals at most one ConversationTurn.
3. Do not discover paragraphs, list items, headings, links, code blocks, or child divs as messages.
4. Sort turns by the number in data-testid:
conversation-turn-1, conversation-turn-2, etc.
5. Role comes from:
- section.getAttribute("data-turn")
- fallback: descendant [data-message-author-role]
6. If a turn section is virtualized/offscreen and empty:
- include the turn with captureStatus: "not_rendered"
- content: "[Turn not rendered in DOM. Scroll this turn into view and recapture.]"
- do not invent content
Intermediate Representation:
Create a platform-neutral Conversation model.
Suggested shape:
{
sourcePlatform: "ChatGPT",
title,
url,
capturedAt,
warning,
turns: [
{
turnIndex,
turnId,
role,
captureStatus,
sourcePlatform: "ChatGPT",
contentType,
sourceElement,
markdown,
debug
}
],
debug: {
turnSectionsFound,
renderedTurnsCaptured,
notRenderedTurns,
userTurns,
assistantTurns,
unknownTurns
}
}
Turn rendering requirements:
1. chatgptTurnRenderer.js receives one discovered turn section.
2. It returns exactly one Markdown string for that turn.
3. It must never emit multiple Project Thoth messages.
4. For user turns:
- extract text from [data-message-author-role="user"]
- include attached images as placeholders
- exclude user action buttons
5. For assistant turns:
- prefer [data-message-author-role="assistant"] .markdown
- convert that subtree to Markdown using core/htmlToMarkdown.js
- if image generation content exists, emit generated image placeholders or Markdown image links
- exclude response actions, buttons, sources button, copy/edit/share UI, and sr-only labels
HTML to Markdown requirements:
Preserve:
- paragraphs
- headings
- unordered lists
- ordered lists
- nested lists where practical
- blockquotes
- tables
- links
- images
- bold
- italics
- inline code
- fenced code blocks
- horizontal rules
Code blocks:
- ChatGPT code blocks may contain nested CodeMirror markup.
- Use pre code textContent as the source.
- Preserve whitespace.
- Detect language from visible code block label when available.
- Do not include copy buttons or UI labels as code content.
Link handling:
- Convert safe href links to [text](url).
- Exclude aria-hidden SVG/icon text.
- If href is missing, preserve visible link text.
Image handling:
- For assistant generated images:
![Generated image: alt](src)
- For user attached images:
[Attached image: alt](src)
- Avoid duplicate thumbnail captures.
Markdown serialization:
1. markdownSerializer.js consumes the Conversation IR.
2. It writes:
- H1 title
- Captured timestamp
- Source Platform
- Source URL
- optional warning
- one section per turn
3. Role headings:
- user → ## User
- assistant → ## Assistant
- system → ## System
- tool → ## Tool
- unknown → ## Unknown
4. Preserve turn order.
5. Include not_rendered placeholders where needed.
Preprocessing:
Create chatgptPreprocessor.js but keep it minimal for now.
It may expose:
prepareChatGPTForCapture(options)
For this refactor, it can be a no-op or simple scroll-current-view helper.
Full automatic scroll-through is deferred to Task 3C.
Acceptance criteria:
- No app shell/sidebar/composer text.
- No duplicated user prompts.
- No fragmented assistant responses.
- The first Jellyfin question is one User message.
- The first Jellyfin response is one Assistant message.
- Headings/lists/code/links stay inside that Assistant message.
- SmartScreen question is one User message.
- SmartScreen response is one Assistant message.
- .nfo question is one User message.
- .nfo response is one Assistant message.
- Virtualized empty turns are represented as not_rendered placeholders.
- The downloaded conversation.md is generated through the new pipeline.
- No LLM calls.
- No Project Thoth app dependency.
- No vault writes.
Important:
Do not try to preserve the old extractor design.
This is an architectural refactor, not another patch.
@@ -0,0 +1,360 @@
# Work Order 0003 --- ChatGPT Turn Section Pipeline
## Status
Ready
## Work Stream
Refactoring
## Application
ChatGPT Capture Connector MVP
------------------------------------------------------------------------
# Objective
Refactor the ChatGPT Capture Connector MVP so conversation capture is
based on canonical ChatGPT turn sections while preserving the existing
extension shell.
The existing extraction logic should be replaced with the canonical
capture pipeline defined by ADR-002 and ADR-003.
------------------------------------------------------------------------
# Architectural References
- ADR-002 --- Canonical Capture Connector Pipeline
- ADR-003 --- Capture Connector Architecture
- `.thoth/10-architecture.md`
- `.thoth/20-philosophy.md`
- `.thoth/40-coding-standards.md`
------------------------------------------------------------------------
# Problem Statement
The MVP successfully:
- Loads as a Manifest V3 extension
- Injects a content script
- Captures page content
- Generates Markdown
- Downloads a `.md` file
However, the extracted conversations are not reliable.
Observed issues include:
- Fragmented assistant responses
- Duplicate content
- Incorrect role detection
- Application chrome captured as conversation
- Virtualized (off-screen) turns omitted
- Headings, lists, links, tables and code emitted as separate messages
The root cause is that discovery is based on content blocks rather than
canonical ChatGPT conversation turns.
------------------------------------------------------------------------
# Key DOM Finding
Canonical turn boundaries are exposed as:
``` css
section[data-testid^="conversation-turn-"][data-turn]
```
Examples:
``` html
<section data-testid="conversation-turn-1" data-turn="user">
<section data-testid="conversation-turn-2" data-turn="assistant">
```
One turn section shall produce one Project Thoth conversation turn.
------------------------------------------------------------------------
# Required Pipeline
``` text
background.js
chatgptConnector.captureConversation()
chatgptPreprocessor.prepareChatGPTForCapture()
chatgptDiscovery.discoverConversation()
chatgptTurnRenderer.renderTurn()
htmlToMarkdown.convertHtmlToMarkdown()
markdownSerializer.serializeConversationToMarkdown()
download writer
```
------------------------------------------------------------------------
# Proposed File Structure
``` text
src/
background.js
connectors/
chatgpt/
chatgptConnector.js
chatgptPreprocessor.js
chatgptDiscovery.js
chatgptTurnRenderer.js
core/
conversationModel.js
htmlToMarkdown.js
markdownSerializer.js
filename.js
```
------------------------------------------------------------------------
# Preserve Existing MVP
Retain:
- Manifest V3
- Service worker
- Toolbar button
- activeTab workflow
- Downloads integration
- Filename generation
- User feedback
Replace:
- Existing mixed extractor
- Fragment-oriented discovery
- Message heuristics
------------------------------------------------------------------------
# Discovery Requirements
Use only:
``` css
section[data-testid^="conversation-turn-"][data-turn]
```
Rules:
1. One section equals one conversation turn.
2. Sort by numeric suffix in `conversation-turn-N`.
3. Determine role from `data-turn`, falling back to
`data-message-author-role`.
4. Never create separate turns for paragraphs, headings, list items,
links or code blocks.
5. Ignore sidebar, composer, buttons, action bars, "Sources", "ChatGPT
can make mistakes", and other application chrome.
------------------------------------------------------------------------
# Virtualized Turns
If a discovered turn is present but not rendered:
``` json
{
"captureStatus": "not_rendered",
"content": "[Turn not rendered in DOM. Scroll this turn into view and recapture.]"
}
```
Add a conversation-level warning whenever one or more turns are not
rendered.
Automatic scrolling is deferred to a future work order.
------------------------------------------------------------------------
# Intermediate Representation
``` javascript
Conversation {
sourcePlatform,
title,
url,
capturedAt,
warning,
turns[],
debug
}
ConversationTurn {
turnIndex,
turnId,
role,
captureStatus,
contentType,
markdown,
debug
}
```
------------------------------------------------------------------------
# Turn Rendering
## User
- Extract only the user turn content.
- Preserve uploaded-image placeholders.
- Ignore edit/copy/action controls.
## Assistant
- Render the `.markdown` subtree.
- Preserve paragraphs, headings, lists, tables, links, blockquotes,
images, inline formatting and fenced code blocks.
- Ignore response controls and hidden accessibility labels.
One turn renderer call must return one Markdown string.
------------------------------------------------------------------------
# HTML to Markdown
Preserve:
- Paragraphs
- Headings
- Ordered lists
- Unordered lists
- Nested lists (where practical)
- Tables
- Blockquotes
- Links
- Images
- Bold
- Italics
- Inline code
- Fenced code blocks
- Horizontal rules
Do not emit UI chrome.
------------------------------------------------------------------------
# Markdown Serialization
Produce canonical `conversation.md`:
- H1 title
- Captured timestamp
- Source platform
- Source URL
- Warning (if needed)
Then:
``` text
## User
...
---
## Assistant
...
```
Preserve turn order exactly.
------------------------------------------------------------------------
# Debug Information
Return:
- turnSectionsFound
- renderedTurnsCaptured
- notRenderedTurns
- userTurns
- assistantTurns
- unknownTurns
- firstTurnIndex
- lastTurnIndex
------------------------------------------------------------------------
# Acceptance Criteria
Using the Jellyfin Server Setup Options conversation:
- Opening question is one User turn.
- First response is one Assistant turn.
- SmartScreen question is one User turn.
- SmartScreen response is one Assistant turn.
- `.nfo` question is one User turn.
- `.nfo` response is one Assistant turn.
The capture shall not contain:
- Sidebar
- Composer
- App shell
- Duplicate turns
- Fragmented headings
- Standalone list items
- Standalone links
- Standalone code fragments
------------------------------------------------------------------------
# Non-Goals
This work order does not implement:
- Automatic scroll-through
- Metadata generation
- Manifest generation
- Harvest generation
- Vault integration
- Project Thoth application integration
- Other platform connectors
------------------------------------------------------------------------
# Manual Test Plan
Validate against:
1. Jellyfin Server Setup Options
2. El Niño Forecast Analysis
3. Deer and Garden Goals
4. Code-heavy conversation
5. Table-heavy conversation
6. Image generation conversation
7. Short conversation
8. Non-ChatGPT page
------------------------------------------------------------------------
# Definition of Done
Complete when:
- Extension loads successfully.
- Downloads still function.
- The new discovery pipeline is used.
- One ChatGPT turn becomes one Project Thoth message.
- Markdown preserves useful structure.
- Virtualized turns are explicitly reported.
- Implementation aligns with ADR-002 and ADR-003.
@@ -0,0 +1,185 @@
# Project Thoth Work Order Template
**Work Order ID:** XXXX\
**Title:** `<Concise descriptive title>`{=html}
------------------------------------------------------------------------
# Status
Draft \| Ready \| In Progress \| Blocked \| Complete
# Work Stream
- Implementation
- Refactoring
- Debugging
- Research
- Documentation
- Testing
# Application / Component
Name of the application, connector, processor, or specification
affected.
------------------------------------------------------------------------
# Objective
A one-paragraph description of the desired outcome.
------------------------------------------------------------------------
# Background
Describe the current situation and why this work is needed.
Include links or references to previous work orders when appropriate.
------------------------------------------------------------------------
# Architectural References
List all governing architectural documents.
Example:
- Reference Architecture
- ADR-001
- ADR-002
- ADR-003
- .thoth/10-architecture.md
- .thoth/20-philosophy.md
- .thoth/40-coding-standards.md
------------------------------------------------------------------------
# Problem Statement
Describe the observable problem.
Focus on facts rather than implementation assumptions.
------------------------------------------------------------------------
# Scope
## In Scope
- Item
- Item
## Out of Scope
- Item
- Item
------------------------------------------------------------------------
# Requirements
Describe the functional and technical requirements.
Organize by logical area.
Example sections:
- Discovery
- Transformation
- Serialization
- User Interface
- Error Handling
- Performance
- Security
------------------------------------------------------------------------
# Deliverables
List the expected outputs.
Examples:
- Source code
- Documentation
- Tests
- Configuration
- Specifications
------------------------------------------------------------------------
# Proposed Implementation
Describe the preferred implementation strategy.
Include suggested modules, interfaces, file structure, or pipeline where
helpful.
------------------------------------------------------------------------
# Constraints
Document important limitations.
Examples:
- Maintain backward compatibility
- No LLM calls
- Preserve deterministic behavior
- Do not modify canonical artifacts
- No vendor lock-in
------------------------------------------------------------------------
# Acceptance Criteria
Provide objective, testable criteria.
Each criterion should be independently verifiable.
------------------------------------------------------------------------
# Manual Test Plan
Describe how the work will be validated.
Include representative test cases and expected outcomes.
------------------------------------------------------------------------
# Risks
Document known risks and mitigation strategies.
------------------------------------------------------------------------
# Non-Goals
Explicitly identify work that is intentionally excluded.
------------------------------------------------------------------------
# Definition of Done
The work order is complete when:
- Acceptance criteria are satisfied.
- Relevant documentation has been updated.
- Code aligns with applicable ADRs.
- No known regressions have been introduced.
- Outstanding technical debt is documented.
------------------------------------------------------------------------
# Notes
Optional implementation notes, observations, or future follow-up items.
------------------------------------------------------------------------
# Revision History
Date Author Description
------------ ---------- ---------------
YYYY-MM-DD Initials Initial draft
+157
View File
@@ -0,0 +1,157 @@
# ADR-001 — Project Thoth is a Platform
**Status:** Accepted
**Date:** 2026-07-07
**Version:** 1.0
---
# Context
Project Thoth began as an investigation into preserving conversations with generative AI.
During architectural exploration, it became clear that the underlying methodology extended far beyond AI conversations.
The methodology addresses the broader problem of acquiring, organizing, transforming, and reusing human knowledge regardless of its origin.
Potential implementations include:
* Browser capture connectors
* Office integrations
* Desktop applications
* Local AI reasoning
* Frontier AI reasoning
* Knowledge transformation pipelines
* Publication workflows
Initially these appeared to be independent projects.
Further architectural analysis showed they are different implementations of a common platform.
---
# Decision
Project Thoth shall be treated as a **platform** rather than a single software application.
The platform consists of multiple complementary components.
```text
Project Thoth
├── Methodology
├── Book
├── Reference Architecture
├── Specifications
├── Processor Library
├── Capture Connectors
├── Desktop Application
├── Knowledge Corpus
└── Publications
```
The software is an implementation of the methodology.
The methodology remains the authoritative definition of Project Thoth.
---
# Guiding Principles
1. The methodology is independent of any software implementation.
2. Software exists to implement the methodology rather than define it.
3. Multiple implementations may coexist while remaining compliant with the same specifications.
4. Specifications define behavior.
5. Processors implement specifications.
6. Capture occurs where knowledge is created.
7. Curation occurs within Project Thoth.
---
# Rationale
Treating Project Thoth as a platform provides several advantages.
The book and software evolve together rather than competing for direction.
Specifications remain implementation-independent.
Capture connectors become interchangeable.
Multiple applications may be developed without changing the underlying methodology.
Commercial offerings can focus on implementation while preserving a stable conceptual foundation.
---
# Consequences
Positive:
* Clear separation between methodology and implementation.
* Stable architectural foundation.
* Easier long-term maintenance.
* Multiple connector types become possible.
* New processors can be added without redesigning the platform.
* Book, consulting, and software reinforce one another.
Trade-offs:
* Higher initial architectural effort.
* More specifications must be maintained.
* Greater emphasis on documentation before implementation.
These trade-offs are accepted because long-term maintainability is a primary design objective.
---
# Future Implications
Future work may include:
* Additional Capture Connectors
* Desktop applications
* Mobile applications
* Local LLM integration
* Frontier LLM integration
* Knowledge graph construction
* Automated transformation pipelines
* Publishing workflows
* Commercial editions
* Cognitive support editions
All future implementations should conform to the Project Thoth methodology and reference architecture.
---
# Related Decisions
None.
---
# References
* Project Thoth Design Principles
* Project Thoth Reference Architecture
* Project Thoth Implementation Guide
---
# Notes
This Architectural Decision Record establishes Project Thoth as a platform composed of methodology, specifications, processors, applications, and publications.
Future ADRs should assume this architectural foundation unless explicitly superseded.
---
# End
+337
View File
@@ -0,0 +1,337 @@
---
adr: 002
title: Canonical Capture Connector Pipeline
status: Accepted
date: 2026-07-09
authors:
- Ken Schaefer
---
# ADR-002: Canonical Capture Connector Pipeline
## Status
Accepted
---
# Context
Project Thoth includes a family of Capture Connectors responsible for preserving conversations and other source material from external systems.
Initial implementation of the ChatGPT connector attempted to identify individual messages while simultaneously converting DOM content into Markdown.
During MVP development, testing revealed several classes of defects:
- Missing conversation content
- Duplicate content
- Fragmented assistant responses
- Incorrect role detection
- Loss of tables, links, and formatting
- Virtualized (non-rendered) conversation sections
- ChatGPT application chrome being captured as conversation content
Investigation showed that these defects were not primarily caused by Markdown generation. They resulted from attempting to perform conversation discovery and content transformation simultaneously.
Modern web applications such as ChatGPT are built using React and other component frameworks that expose deeply nested and frequently changing DOM structures. A single logical conversation turn may consist of dozens of nested DOM elements.
Attempting to infer conversation boundaries while simultaneously rendering Markdown creates unnecessary complexity and makes debugging difficult.
---
# Decision
All Project Thoth Capture Connectors SHALL implement a three-stage pipeline:
```
Conversation Discovery
Conversation Turn Model
Content Transformation
Markdown Serialization
```
Each stage has a single responsibility.
---
# Stage 1 — Conversation Discovery
Purpose:
Identify the canonical conversation turns for a source platform.
Responsibilities:
- Locate conversation root
- Locate conversation turn containers
- Determine turn ordering
- Determine speaker role
- Detect unsupported content
- Detect partially rendered or virtualized content
- Produce a platform-neutral intermediate representation
This stage SHALL NOT:
- Generate Markdown
- Normalize formatting
- Generate metadata
- Summarize
- Invoke LLMs
Output:
```text
Conversation
Turn
Turn
Turn
```
---
# Stage 2 — Content Transformation
Purpose:
Transform a single conversation turn into normalized content.
Responsibilities:
- Convert HTML to Markdown
- Preserve paragraphs
- Preserve headings
- Preserve lists
- Preserve tables
- Preserve links
- Preserve images
- Preserve code blocks
- Preserve inline formatting
This stage SHALL NOT:
- Discover conversation turns
- Infer ordering
- Generate files
Output:
```text
Conversation Turn
Markdown
```
---
# Stage 3 — Markdown Serialization
Purpose:
Produce the canonical Project Thoth `conversation.md` document.
Responsibilities:
- Write capture metadata
- Preserve conversation order
- Emit User / Assistant boundaries
- Write final Markdown document
This stage SHALL NOT:
- Parse HTML
- Discover DOM elements
- Modify extracted content
Output:
```
conversation.md
```
---
# Intermediate Representation
Conversation discovery SHALL produce a platform-neutral model.
Example:
```typescript
interface Conversation {
sourcePlatform: string;
title: string;
url: string;
capturedAt: Date;
turns: ConversationTurn[];
}
interface ConversationTurn {
turnIndex: number;
role:
| "user"
| "assistant"
| "system"
| "tool"
| "unknown";
captureStatus:
| "rendered"
| "not_rendered"
| "unsupported";
sourceElement: HTMLElement;
markdown?: string;
}
```
The Intermediate Representation (IR) becomes the contract between discovery and transformation.
---
# Rationale
Separating discovery from rendering provides several advantages.
## Separation of Concerns
Each stage performs one responsibility.
Conversation discovery determines *what* exists.
Content transformation determines *how* it is represented.
Markdown serialization determines *how* it is packaged.
---
## Testability
Each stage can be independently tested.
Examples:
- Discovery tests verify turn detection.
- Transformation tests verify Markdown fidelity.
- Serialization tests verify document format.
Failures can be isolated without affecting unrelated stages.
---
## Maintainability
Modern web applications frequently change DOM structure.
When a platform changes, only Conversation Discovery should typically require modification.
Markdown rendering remains reusable across platforms.
---
## Reuse
Most Capture Connectors share identical downstream behavior.
Expected connectors include:
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
- Perplexity
- Future browser-based AI systems
Only Conversation Discovery is expected to be platform-specific.
---
## Debuggability
The IR enables inspection before Markdown generation.
Developers can validate:
- turn count
- ordering
- role detection
- unsupported content
- rendering completeness
without involving Markdown generation.
---
# Consequences
## Positive
- Cleaner architecture
- Easier debugging
- Platform independence
- Improved testability
- Reduced coupling
- Higher long-term maintainability
## Negative
- Additional abstraction layer
- Slightly more implementation effort
- Requires maintenance of an Intermediate Representation
---
# Alternatives Considered
## Single-Pass DOM → Markdown
Rejected.
Although initially simpler, this approach couples conversation discovery with formatting.
Testing demonstrated that defects become difficult to isolate and frequently require heuristic patches.
---
## Platform-Specific End-to-End Connectors
Rejected.
Embedding discovery, rendering, and serialization into a single connector creates duplication across platforms and limits reuse.
---
# Future Considerations
Future Capture Connectors may introduce an optional preprocessing stage before Conversation Discovery.
Examples include:
- Automatic scrolling to render virtualized conversation turns
- Lazy-loading attachments
- Expansion of collapsed content
These preprocessing activities remain outside the canonical pipeline and serve only to improve completeness of the source material.
---
# References
- ADR-001: Project Structure
- Project Thoth MVP — ChatGPT Capture Connector
- Empirical findings from ChatGPT DOM extraction (July 2026)
+383
View File
@@ -0,0 +1,383 @@
---
adr: 003
title: Capture Connector Architecture
status: Accepted
date: 2026-07-09
authors:
- Ken Schaefer
---
# ADR-003: Capture Connector Architecture
## Status
Accepted
---
# Context
Project Thoth is intended to preserve knowledge from a growing ecosystem of AI assistants and digital systems.
Initially, the first connector targeted ChatGPT. During development it became clear that the ChatGPT connector was not unique. Although each platform exposes different APIs and DOM structures, every connector ultimately performs the same high-level task:
1. Capture source material.
2. Transform it into a canonical representation.
3. Deliver the result to Project Thoth.
The platform-specific logic lies almost entirely in *how the source material is discovered*. Once discovered, the remainder of the processing pipeline is largely identical.
Rather than implementing each connector as a monolithic application, Project Thoth should define a common connector architecture with clearly defined extension points.
---
# Decision
All Project Thoth Capture Connectors SHALL implement the same logical architecture.
```
Source Platform
Platform Discovery Layer
Canonical Conversation Model
Platform-Neutral Transformation
Canonical conversation.md
Project Thoth Pipeline
```
Only the Discovery Layer is expected to be platform-specific.
---
# Connector Responsibilities
Capture Connectors are responsible only for preserving source material.
They SHALL:
- Capture conversations
- Capture attachments
- Preserve ordering
- Preserve formatting where practical
- Preserve metadata supplied by the source platform
- Produce canonical Project Thoth documents
They SHALL NOT:
- Summarize
- Classify
- Generate tags
- Generate YAML metadata
- Generate manifests
- Generate harvests
- Perform semantic analysis
- Invoke LLMs
- Modify user content
Connectors are intentionally "dumb."
Their responsibility is faithful preservation.
---
# Canonical Connector Pipeline
Every connector SHALL implement the following stages.
```
Preprocessing (Optional)
Discovery
Intermediate Representation
Transformation
Serialization
Output
```
---
# Stage 0 — Preprocessing (Optional)
Purpose:
Prepare the source for capture.
Examples:
- Render virtualized content
- Expand collapsed sections
- Load lazy content
- Wait for streaming responses to complete
Preprocessing SHALL NOT modify user content.
---
# Stage 1 — Discovery
Purpose:
Locate the logical content exposed by the source platform.
Responsibilities include:
- Locate conversation root
- Locate conversation turns
- Determine ordering
- Determine speaker
- Detect unsupported content
- Detect partial rendering
Discovery is platform-specific.
---
# Stage 2 — Intermediate Representation
Purpose:
Represent captured information in a platform-neutral model.
Example:
```typescript
interface Conversation {
sourcePlatform: string;
title: string;
url: string;
capturedAt: Date;
turns: ConversationTurn[];
}
interface ConversationTurn {
turnIndex: number;
role: string;
captureStatus: string;
content: DocumentFragment | HTMLElement | string;
}
```
The Intermediate Representation (IR) is the contract between Discovery and Transformation.
---
# Stage 3 — Transformation
Purpose:
Convert platform-specific content into canonical Project Thoth Markdown.
Responsibilities include:
- HTML → Markdown
- Paragraph preservation
- Lists
- Tables
- Links
- Images
- Code blocks
- Inline formatting
Transformation is expected to be reusable across platforms.
---
# Stage 4 — Serialization
Purpose:
Generate the canonical output artifacts.
Current artifacts include:
- `conversation.md`
Future artifacts may include:
- Attachments
- Asset manifests
- Conversation package formats
Serialization SHALL NOT reinterpret content.
---
# Platform Independence
The connector architecture intentionally separates platform-specific logic from platform-neutral logic.
Examples:
| Component | Platform Specific |
|-----------|-------------------|
| Discovery | Yes |
| Preprocessing | Mostly |
| Transformation | No |
| Serialization | No |
This minimizes duplication across connectors.
---
# Supported Connector Types
The architecture is intended to support connectors including, but not limited to:
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
- Perplexity
- Cursor
- GitHub Copilot Chat
- Future browser-based AI assistants
Additional connectors should primarily require implementation of Discovery and, where necessary, Preprocessing.
---
# Design Principles
## Fidelity Over Intelligence
Connectors preserve information.
They do not interpret information.
---
## Platform Neutrality
Internal Project Thoth formats are independent of any external platform.
No downstream component should need to know whether content originated from ChatGPT, Gemini, Claude, or another system.
---
## Composability
Each stage should be independently testable and replaceable.
This enables improvements to one stage without affecting others.
---
## Deterministic Output
Running the connector multiple times against the same rendered conversation should produce equivalent output.
---
## Fail Gracefully
When unsupported content is encountered:
- Preserve placeholders
- Preserve ordering
- Record warnings
- Never silently discard content
---
# Error Handling
Capture is considered successful when source material is faithfully preserved.
If content cannot be rendered or extracted:
- Report the issue
- Preserve available context
- Continue processing remaining content
Partial capture is preferred over silent failure.
---
# Consequences
## Positive
- Uniform architecture across all connectors
- Reduced duplication
- Easier testing
- Improved maintainability
- Simplified onboarding for new connector development
- Reusable transformation and serialization components
## Negative
- Additional abstraction layers
- Requires maintenance of a shared Intermediate Representation
- Slightly higher initial implementation effort
---
# Alternatives Considered
## Monolithic Platform Connectors
Rejected.
Embedding discovery, transformation, and serialization into a single implementation results in duplicated logic and inconsistent behavior across connectors.
---
## Direct Platform-to-Markdown Conversion
Rejected.
Coupling discovery with rendering makes debugging difficult and limits reuse.
---
# Relationship to Other ADRs
- **ADR-001** establishes the overall Project Thoth repository and project structure.
- **ADR-002** defines the canonical capture pipeline used within connectors.
- **ADR-003** defines the architectural responsibilities, lifecycle, and composition of Capture Connectors as reusable platform adapters.
Together, these ADRs establish the foundation for a connector ecosystem rather than a collection of independent integrations.
---
# Future Considerations
Future enhancements may include:
- Native API-based connectors where supported
- Hybrid API + browser capture
- Incremental conversation synchronization
- Background monitoring of supported platforms
- Signed connector packages
- Connector capability negotiation
- Automated regression testing against captured DOM snapshots
The architectural principles defined in this ADR are expected to remain stable even as individual source platforms evolve.
+631
View File
@@ -0,0 +1,631 @@
# Project Thoth Application
# ChatGPT Capture Connector MVP
## Architecture and Sprint Plan
## Version 0.1
---
# Purpose
This document defines the architecture and implementation plan for the first Project Thoth Capture Connector MVP.
The MVP will create a browser extension that captures the currently open ChatGPT conversation and saves it as a Markdown file to the user's local Downloads folder.
This MVP validates the Capture Connector boundary before the Project Thoth application exists.
---
# Product Boundary
Project Thoth has two systems.
```text
System 1: Capture Connectors
System 2: Project Thoth Application
```
This sprint implements only the first MVP Capture Connector.
---
# MVP Definition
The MVP browser extension must:
1. Present a browser toolbar button.
2. Assume the user is already logged into ChatGPT.
3. Assume a ChatGPT conversation is open in the current browser tab.
4. Capture the visible/current conversation content.
5. Preserve message order as well as practical.
6. Preserve user and assistant content as well as practical.
7. Export the captured conversation as Markdown.
8. Save the file to the local Downloads folder.
9. Avoid all AI processing.
10. Avoid all Project Thoth application dependencies.
---
# Non-Goals
The MVP will not:
- Generate Source Metadata.
- Generate Conversation Manifests.
- Generate Harvest artifacts.
- Call a local LLM.
- Call a frontier LLM.
- Write directly to the vault.
- Sync with Open WebUI.
- Manage Knowledge Collections.
- Support Gemini.
- Support Copilot.
- Support Outlook.
- Support PDFs.
- Provide a full desktop application.
---
# Architectural Principle
> Capture Connectors do not reason. They only capture and transmit source material.
For this MVP, "transmit" means saving a Markdown file to Downloads.
---
# Technical Platform
The MVP should use a Manifest V3 browser extension.
Manifest V3 is the current Chrome extensions platform, and every extension requires a `manifest.json` file that defines extension metadata, permissions, and behavior.
The MVP should initially target:
```text
Google Chrome / Microsoft Edge
```
because both use the Chromium extension model.
---
# High-Level Architecture
```text
User opens ChatGPT conversation
User clicks Project Thoth browser button
Extension receives active tab permission
Content script extracts conversation content
Extension normalizes content into Markdown
Extension saves file to Downloads
```
Chrome's `activeTab` permission grants temporary access to the current tab when the user invokes the extension, which matches the intended "click to capture this page" UX.
Content scripts can run in the context of a web page and read page content through standard Web APIs, which is the mechanism the extension will use to inspect the ChatGPT page.
---
# Component Architecture
## 1. Browser Action
Purpose:
- Provide the toolbar button.
- Trigger capture.
User-facing label:
```text
Save to Project Thoth
```
Codex implementation required.
---
## 2. Manifest Configuration
Purpose:
- Define extension metadata.
- Request minimal permissions.
- Register the service worker.
- Configure content script execution or script injection.
Likely permissions:
```json
"permissions": ["activeTab", "scripting", "downloads"]
```
Likely host permissions:
```json
"host_permissions": ["https://chatgpt.com/*", "https://chat.openai.com/*"]
```
Chrome extensions must declare permissions in the manifest to use extension APIs.
Codex implementation required.
---
## 3. Service Worker
Purpose:
- Listen for toolbar button clicks.
- Inject or invoke the content script.
- Receive extracted content.
- Create Markdown file.
- Save through browser download API.
Codex implementation required.
---
## 4. ChatGPT Content Extractor
Purpose:
- Inspect the current ChatGPT conversation page.
- Extract message blocks.
- Preserve ordering.
- Detect speaker role when possible.
- Preserve Markdown-like formatting where possible.
Expected output:
```json
{
"sourcePlatform": "ChatGPT",
"title": "Detected conversation title",
"url": "https://chatgpt.com/...",
"capturedAt": "2026-07-07T...",
"messages": [
{
"role": "user",
"content": "..."
},
{
"role": "assistant",
"content": "..."
}
]
}
```
Codex implementation required.
---
## 5. Markdown Normalizer
Purpose:
Convert extracted message data into canonical Markdown.
MVP output format:
```markdown
# Conversation Title
Captured: 2026-07-07
Source Platform: ChatGPT
Source URL: https://chatgpt.com/...
---
## User
Message content
---
## Assistant
Message content
---
```
Codex implementation required.
---
## 6. Download Writer
Purpose:
- Generate a safe filename.
- Save Markdown file to Downloads.
- Avoid overwriting when possible.
Example filename:
```text
2026-07-07 - ChatGPT - Conversation Title.md
```
Codex implementation required.
---
# MVP File Output
The MVP produces exactly one file:
```text
conversation.md
```
Saved to:
```text
Downloads/
```
Later versions may produce:
```text
conversation.raw.json
conversation.manifest.md
conversation.harvest.md
```
but those are explicitly deferred.
---
# Error Handling
The extension should show simple user-facing messages.
## Success
```text
Saved to Downloads.
```
## No ChatGPT Conversation Detected
```text
No ChatGPT conversation was detected on this page.
```
## Capture Failed
```text
Capture failed. Try scrolling through the conversation and capturing again.
```
## Download Failed
```text
The conversation was captured, but the file could not be saved.
```
Codex implementation required.
---
# Key Technical Risk
The largest technical risk is ChatGPT DOM instability.
The ChatGPT page structure may change without notice.
Therefore, the extractor should be isolated in its own module:
```text
chatgptExtractor.js
```
Do not mix ChatGPT-specific DOM logic with generic extension logic.
---
# Proposed Project Structure
```text
project-thoth-chatgpt-capture/
manifest.json
src/
background.js
chatgptExtractor.js
markdownNormalizer.js
filename.js
icons/
icon16.png
icon48.png
icon128.png
README.md
```
Codex implementation required.
---
# Sprint Plan
## Sprint Goal
Build and manually test a browser extension that captures the currently open ChatGPT conversation and saves it as Markdown to Downloads.
---
## Task 1 — Create Extension Skeleton
Deliverables:
- `manifest.json`
- background service worker
- toolbar button
- placeholder icon
- local install instructions
Codex prompt needed:
```text
Prompt Codex to create a minimal Manifest V3 browser extension skeleton for Chrome/Edge with a toolbar button and service worker.
```
---
## Task 2 — Implement Active Tab Capture Flow
Deliverables:
- Toolbar click handler
- active tab lookup
- script injection
- message passing between service worker and content script
Codex prompt needed:
```text
Prompt Codex to implement activeTab-based capture flow using Manifest V3 service worker, scripting API, and content script message passing.
```
---
## Task 3 — Implement ChatGPT DOM Extractor
Deliverables:
- Extract conversation title if available
- Extract ordered message blocks
- Detect user vs assistant messages when possible
- Extract text content
- Preserve basic formatting
Codex prompt needed:
```text
Prompt Codex to write a ChatGPT-specific DOM extractor module that returns ordered conversation messages with role and content fields.
```
---
## Task 4 — Implement Markdown Normalizer
Deliverables:
- Convert extracted messages to canonical Markdown
- Include capture metadata header
- Preserve message order
- Add `## User` and `## Assistant` boundaries
Codex prompt needed:
```text
Prompt Codex to create a markdown normalization module that converts extracted ChatGPT messages into Project Thoth conversation.md format.
```
---
## Task 5 — Implement Download Writer
Deliverables:
- Generate filename
- Save Markdown to Downloads
- Sanitize invalid filename characters
- Add timestamp if needed
Codex prompt needed:
```text
Prompt Codex to implement Markdown download using the browser downloads API with safe filename generation.
```
---
## Task 6 — Add User Feedback
Deliverables:
- Success message
- Error message when not on ChatGPT
- Error message when no conversation is detected
- Error message when download fails
Codex prompt needed:
```text
Prompt Codex to add simple user feedback for successful capture and failure states.
```
---
## Task 7 — Manual Test Pass
Test cases:
1. Short ChatGPT conversation.
2. Long ChatGPT conversation.
3. Conversation with code blocks.
4. Conversation with markdown lists.
5. Conversation with headings.
6. Conversation with uploaded file references.
7. Conversation where only part of the conversation has been scrolled into view.
8. Non-ChatGPT page.
Codex prompt may be needed after defects are found:
```text
Prompt Codex to fix extraction or formatting defects discovered during manual testing.
```
---
# Acceptance Criteria
The MVP is complete when:
- Extension installs locally in Chrome or Edge.
- Toolbar button appears.
- User can open a ChatGPT conversation and click the button.
- A Markdown file is downloaded.
- The Markdown file contains the conversation content.
- Message order is preserved.
- Speaker boundaries are present when detectable.
- The extension does not call any LLM.
- The extension does not require Project Thoth application installation.
---
# Deferred Architecture
The following belong to later sprints:
```text
Project Thoth desktop application
Native messaging
Vault writing
Git integration
Source Metadata generation
Conversation Manifest generation
Open WebUI integration
Local LLM integration
Gemini connector
Copilot connector
Office connector
Adobe connector
General web clipper
```
---
# Architectural Finding to Validate
This MVP should answer one question:
> Can a browser extension reliably capture a complete ChatGPT conversation into a usable Project Thoth Primary Source?
If yes, the connector architecture is viable.
If no, Project Thoth must use a different acquisition path for ChatGPT conversations.
---
# End
+282
View File
@@ -0,0 +1,282 @@
# Project Thoth Reference Architecture
## Version 0.2 — Capture Boundary
# Core Definition
Project Thoth consists of two primary systems:
```text
System 1: Capture Connectors
System 2: Project Thoth Application
```
The boundary is strict:
> Capture Connectors capture source material.
> Project Thoth stores, processes, reasons over, and transforms that material.
---
# System 1 — Capture Connectors
## Purpose
Capture Connectors operate where knowledge is created or encountered.
Examples:
- ChatGPT
- Gemini
- Copilot
- Open WebUI
- Outlook
- Word
- Excel
- PowerPoint
- Adobe Acrobat
- Web pages
- Slack
- Teams
- Email
- YouTube
## Responsibilities
Capture Connectors may:
- detect the active source
- extract source content
- preserve structure when possible
- capture technical metadata
- send the captured payload to Project Thoth
## Non-Responsibilities
Capture Connectors must not:
- reason over the source
- generate Source Metadata
- generate Manifests
- generate Harvests
- decide canonical knowledge
- call LLM processors
- manage the vault
## MVP Connector
The MVP Capture Connector is:
```text
ChatGPT Capture Connector
```
Its single user-facing action is:
```text
Save to Project Thoth
```
---
# System 2 — Project Thoth Application
## Purpose
The Project Thoth Application manages the knowledge corpus.
It is the place where captured content is stored, processed, searched, reasoned over, transformed, and published.
## Responsibilities
Project Thoth may:
- receive captured payloads
- save Primary Sources to the vault
- display raw captured content
- manage vault structure
- generate Source Metadata
- generate Conversation Manifests
- run Harvest processors
- call local LLMs
- call frontier LLMs
- sync with Knowledge Collections
- manage reasoning contexts
- support search and review
- generate downstream artifacts
## MVP Application
The MVP Project Thoth Application should:
- receive captured ChatGPT content
- save the raw capture
- save a normalized markdown file
- display captured content
- provide scaffolding for future metadata, manifest, and harvest processing
---
# Boundary Contract
Capture Connectors send captured source packages to Project Thoth.
A capture package may contain:
```text
raw_capture
normalized_markdown
capture_metadata
attachments
source_references
```
## Capture Metadata
Capture metadata is technical, not interpretive.
Examples:
- source platform
- source URL
- capture date
- connector version
- detected title
- message count
- source format
Capture metadata is not the same as Project Thoth Source Metadata.
---
# Processing Pipeline
```text
Knowledge Source
Capture Connector
Captured Source Package
Project Thoth Application
Vault Storage
Source Metadata Processor
Manifest Processor
Purpose-Specific Transformations
Canonical Knowledge / Publications / Runbooks / Courses
```
---
# Design Principles
1. Capture connectors do not reason.
2. Project Thoth owns processing.
3. The vault is the durable source repository.
4. Tools are replaceable.
5. Specifications define artifacts.
6. Processors implement specifications.
7. Capture happens where knowledge occurs.
8. Curation happens inside Project Thoth.
9. A source has one physical location, many classifications, and many possible reasoning contexts.
10. The value of an artifact is determined by expected retrieval behavior.
---
# MVP Scope
## Included
```text
ChatGPT Capture Connector
Project Thoth Application Scaffold
Raw Capture Viewer
Normalized Markdown Viewer
Vault Save Workflow
```
## Deferred
```text
Gemini Connector
Copilot Connector
Office Connector
Adobe Connector
General Web Clipper
Source Metadata Automation
Manifest Automation
Harvest Processors
Knowledge Collection Sync
Embedded LLM
Local LLM Integration
```
---
# End
@@ -0,0 +1,96 @@
# Project Thoth Conversation Manifest Generator
## Role
You are acting as the **Project Thoth Conversation Indexer**.
Your responsibility is to examine a completed conversation and generate a Project Thoth Conversation Manifest.
The conversation is the **Primary Source**.
The Manifest is a **Secondary Source**.
Do not modify, rewrite, summarize, or replace the Primary Source.
Your responsibility is limited to generating a Conversation Manifest.
---
# Source Selection Rule
This processor supports two operating modes.
## Mode 1 — Active Conversation
If no source file is attached, treat the current conversation as the Primary Source.
## Mode 2 — Attached Source
If one or more source documents are attached, treat the attached conversation document as the Primary Source.
Ignore the surrounding conversation except for these instructions.
If multiple sources are attached, process only the source explicitly identified by the user.
---
# Project Thoth Specification
Generate a Conversation Manifest that conforms to the **Project Thoth Conversation Manifest Specification**.
The specification defines:
* Required sections
* Field meanings
* Catalog Metadata
* Analytical Metadata
* Architectural principles
* Relationship to other Project Thoth artifacts
Do not redefine or reinterpret the specification.
Implement it exactly.
---
# Processing Rules
1. Produce a Conversation Manifest only.
2. Treat the Primary Source as authoritative.
3. Prefer omission over invention.
4. Use objective description in Catalog Metadata.
5. Use faithful interpretation in Analytical Metadata.
6. Do not generate Source Metadata.
7. Do not generate Harvest documents.
8. Do not generate Canonical Knowledge notes.
9. Do not generate blog ideas, articles, curriculum, or publication drafts.
10. Return only the completed Manifest in Markdown.
---
# Architectural Responsibility
This processor performs one stage of the Project Thoth pipeline.
Primary Source
Conversation Manifest Generator
Conversation Manifest
Subsequent processors—including Harvesting, Knowledge Graph construction, Curriculum Generation, and Publication—operate on the Primary Source, Source Metadata, and Conversation Manifest independently.
This processor must not perform the responsibilities of downstream processors.
View File
+102
View File
@@ -0,0 +1,102 @@
# Project Thoth Source Metadata Generator
## Role
You are acting as the **Project Thoth Source Cataloger**.
Your responsibility is to examine a single Primary Source and generate Project Thoth Source Metadata.
The Primary Source remains authoritative.
Do not modify the source.
Do not summarize the source except where explicitly requested by the Project Thoth Source Metadata Schema Specification.
Do not generate analysis, editorial commentary, blog ideas, harvested knowledge, manifests, or canonical knowledge.
Your responsibility is limited to generating Source Metadata.
---
# Source Selection Rule
This processor supports two operating modes.
### Mode 1 — Active Conversation
If no source file is attached, treat the current conversation as the Primary Source.
---
### Mode 2 — Attached Source
If one or more source documents are attached, treat the attached source as the Primary Source.
Ignore the surrounding conversation except for these instructions.
If multiple sources are attached, process only the source explicitly identified by the user.
---
# Project Thoth Specification
Generate Source Metadata that conforms to the **Project Thoth Source Metadata Schema Specification**.
The schema specification defines:
* Required fields
* Optional fields
* Field meanings
* Serialization requirements
* YAML formatting rules
* Validation requirements
* Canonical examples
Do not redefine or reinterpret the specification.
Implement it exactly.
---
# Processing Rules
1. Produce Source Metadata only.
2. Treat the Primary Source as authoritative.
3. Prefer omission over invention.
4. Use objective description rather than interpretation.
5. Preserve provenance whenever it can be reasonably determined.
6. Optimize the metadata for retrieval, indexing, and reasoning.
7. Do not produce Manifests.
8. Do not produce Harvest documents.
9. Do not produce Canonical Knowledge.
10. Return only the YAML frontmatter defined by the Source Metadata Schema Specification.
---
# Architectural Responsibility
This processor performs only one stage of the Project Thoth pipeline.
Primary Source
Source Metadata Generator
Source Metadata
Subsequent processors—including Conversation Manifest Generation, Harvesting, Knowledge Graph construction, Curriculum Generation, and Publication—operate on the Primary Source and its associated metadata independently.
This processor must not perform the responsibilities of downstream processors.
@@ -0,0 +1,330 @@
# Project Thoth
# Conversation Manifest Specification
## Version 0.1 (Draft)
---
# Purpose
This specification defines the canonical structure of a Project Thoth Conversation Manifest.
A Conversation Manifest is a **Secondary Source** derived from a Primary Source conversation.
Its purpose is to improve:
* discoverability
* classification
* semantic retrieval
* corpus navigation
* future harvesting
* AI reasoning
The Manifest is descriptive.
It is not intended to replace the Primary Source.
---
# Design Principles
The Manifest answers:
> **Why is this conversation important, and how should it be found?**
The Manifest does **not** answer:
* What knowledge should become canonical?
* What articles should be written?
* What curriculum should be produced?
Those belong to downstream Project Thoth processors.
---
# Relationship to Primary Sources
Every Manifest shall reference exactly one Primary Source.
The Manifest is always considered a Secondary Source.
If the Primary Source changes substantially, the Manifest may be regenerated.
---
# Manifest Structure
A Conversation Manifest consists of two logical sections.
---
# Part I — Catalog Metadata
The purpose of Catalog Metadata is objective classification.
Whenever possible, prefer observation over interpretation.
---
## Conversation Title
A concise descriptive title.
---
## Conversation Purpose
One or two paragraphs describing why the conversation occurred.
Remain descriptive.
Avoid interpretation.
---
## Conversation Type
One or more classifications.
Examples include:
* discussion
* brainstorming
* architecture
* planning
* philosophy
* research
* implementation
* editorial
* reflection
* learning
* decision_making
---
## Primary Topics
Major subjects discussed.
---
## Secondary Topics
Supporting or incidental subjects.
---
## Entities
Group entities by category.
Typical categories include:
* people
* organizations
* software
* technologies
* projects
* books
* games
* methodologies
* places
* other
Only include populated categories.
---
## Related Domains
Academic or professional disciplines represented.
Examples include:
* Knowledge Management
* Library Science
* Sociology
* Anthropology
* Information Science
* Artificial Intelligence
* Software Architecture
---
## Controlled Vocabulary
Preferred indexing terms selected from the evolving Project Thoth taxonomy.
These should remain relatively stable across the corpus.
---
## Free Keywords
Additional retrieval keywords derived directly from the conversation.
---
## Suggested Tags
Obsidian-compatible tags.
---
## Related Sources
Known related Primary Sources or significant Secondary Sources.
Do not invent relationships.
---
## Source Classification
Choose one.
Examples include:
* Permanent Primary Source
* Reference Conversation
* Temporary Discussion
* Administrative
* Ephemeral
Include a brief justification.
---
# End of Catalog Metadata
---
# Part II — Analytical Metadata
The purpose of Analytical Metadata is to assist future harvesting.
Interpretation is appropriate within this section.
Interpretations should always remain faithful to the Primary Source.
Different reasoning systems may legitimately produce different analytical metadata.
---
## Key Concepts
Important concepts introduced, refined, or clarified.
---
## Significant Discoveries
Insights that emerged through the conversation itself.
Only include discoveries resulting from reasoning.
---
## Decisions Reached
Agreed conclusions.
---
## Assumptions Challenged
Assumptions that were questioned, revised, or replaced.
---
## Open Questions
Questions intentionally left unresolved.
---
## Future Exploration
Research directions naturally suggested by the conversation.
These are not implementation task lists.
---
## Proposed Artifacts
Artifacts that could reasonably be derived from this conversation.
Examples include:
* methodology
* article
* book chapter
* prompt
* specification
* architecture document
* plugin
* software
* diagram
* presentation
* course
---
## Relationship Candidates
Potential knowledge graph relationships that may become useful during future curation.
These are suggestions rather than established facts.
---
## Harvest Recommendation
Assign one.
* Low
* Medium
* High
* Foundational
Include a concise justification.
---
## Confidence
Assign:
* High
* Medium
* Low
This represents confidence that the Manifest accurately reflects the conversation.
---
# Architectural Principles
1. A Manifest is always a Secondary Source.
2. A Manifest catalogs rather than replaces the Primary Source.
3. Catalog Metadata should remain relatively stable.
4. Analytical Metadata may evolve as reasoning systems improve.
5. Interpretation must remain faithful to the evidence.
6. The Manifest exists to support retrieval and future harvesting.
7. The Manifest should optimize corpus navigation rather than content generation.
8. The specification is implementation independent.
---
# Relationship to Other Specifications
This specification should be used together with:
* Source Metadata Schema Specification
* Harvest Specification
* Canonical Knowledge Specification (future)
* Knowledge Graph Specification (future)
Together these specifications define the Project Thoth knowledge processing pipeline.
@@ -0,0 +1,463 @@
# Project Thoth
# Source Metadata Schema Specification
## Version 0.1 (Draft)
---
# Purpose
This specification defines the canonical metadata schema for all Primary Sources managed within Project Thoth.
Its objectives are to:
* provide consistent metadata across heterogeneous sources
* improve semantic retrieval
* support AI reasoning
* preserve provenance
* enable corpus compilation
* remain independent of any specific software platform
This specification defines the metadata schema.
It does **not** define harvesting, interpretation, manifests, or canonical knowledge extraction.
---
# Design Principles
The metadata should answer only one question:
> **What is this source?**
It should not answer:
* What does this source mean?
* What conclusions should be drawn?
* What knowledge should be extracted?
Those responsibilities belong to downstream Project Thoth processors.
---
# Supported Source Types
Examples include:
* conversation
* meeting
* interview
* article
* book
* paper
* transcript
* journal
* email
* web_page
* image
* notes
The schema is intentionally source-independent.
---
# Required Fields
## title
A concise descriptive title.
Do not use sensational language.
Always use double quotes.
Example
```yaml
title: "Knowledge Harvesting Methodology"
```
---
## source_type
Type of primary source.
Example
```yaml
source_type: "conversation"
```
---
## status
Classification of the source.
Allowed values include:
* primary_source
* reference_source
* archival_source
---
# Optional Fields
## source_platform
Examples
* ChatGPT
* Gemini
* Claude
* Open WebUI
* Teams
* Slack
* Gmail
* YouTube
* Local Markdown
---
## conversation_type
One or more values.
Examples
* discussion
* architecture
* brainstorming
* research
* philosophy
* planning
* implementation
* reflection
---
## creator
One or more creators or participants.
---
## created
Creation date.
Use ISO format when known.
---
## language
Language identifier.
Example
```yaml
language: "en-US"
```
---
## reasoning_contexts
Long-lived reasoning contexts in which this source is useful.
These are reasoning contexts.
They are **not** folder names.
Examples
* Personal
* Project Thoth
* Fractional Insight
* Withered Sanctum
* Pyramid
* AI Lab
---
## primary_topics
Major subjects.
---
## secondary_topics
Supporting subjects.
---
## entities
Group entities by category.
Typical categories include
* people
* organizations
* software
* technologies
* books
* games
* projects
* methodologies
* places
* other
Only include populated categories.
---
## content_types
High-level description of the source.
Examples
* discussion
* architecture
* methodology
* philosophy
* tutorial
* research
* historical_analysis
* technical_analysis
* technical_history
* personal_history
* design
---
## keywords
Fifteen to forty concise retrieval keywords.
---
## summary
A brief objective description of the source.
Maximum one paragraph.
Avoid interpretation.
---
## related_sources
Titles of known related sources.
If none are known
```yaml
related_sources: []
```
---
## manifest
Expected filename of the corresponding manifest.
---
## harvest
Expected filename of the corresponding harvest document.
---
# YAML Serialization Rules
The schema shall be serialized using valid YAML.
## Strings
All scalar string values shall be enclosed in double quotes.
Correct
```yaml
title: "Project Thoth"
```
---
## Lists
Lists shall always use block sequence notation.
Correct
```yaml
primary_topics:
- "Knowledge Architecture"
- "Harvesting"
```
Incorrect
```yaml
primary_topics: - Knowledge Architecture
```
Incorrect
```yaml
primary_topics: [Knowledge Architecture, Harvesting]
```
---
## Nested Objects
Nested structures shall use indentation.
Correct
```yaml
entities:
software:
- "Obsidian"
- "VS Code"
```
---
## Empty Lists
Represent empty lists explicitly.
```yaml
related_sources: []
```
---
## Null Values
Omit unknown values.
Do not generate
* null
* Unknown
* N/A
---
## Indentation
Use two spaces.
Do not use tabs.
---
## Quoting
When uncertain, quote the value.
---
## Validation
Generated YAML should parse successfully using a standard YAML parser without modification.
---
# Canonical Example
```yaml
---
title: "Knowledge Harvesting Methodology"
source_type: "conversation"
source_platform: "ChatGPT"
conversation_type:
- "architecture"
- "discussion"
creator:
- "Ken Schaefer"
- "ChatGPT"
created: "2026-07-03"
language: "en-US"
status: "primary_source"
reasoning_contexts:
- "Project Thoth"
primary_topics:
- "Knowledge Harvesting"
- "Metadata"
secondary_topics:
- "Prompt Engineering"
entities:
software:
- "Obsidian"
- "VS Code"
content_types:
- "architecture"
- "methodology"
keywords:
- "knowledge"
- "metadata"
- "vault"
summary: "Discussion establishing the metadata schema for Project Thoth primary sources."
related_sources: []
manifest: "Knowledge Harvesting Methodology.manifest.md"
harvest: "Knowledge Harvesting Methodology.harvest.md"
---
```
---
# Architectural Principles
1. Metadata identifies the source.
2. Metadata is descriptive rather than interpretive.
3. Metadata should remain relatively stable over time.
4. Metadata supports retrieval rather than analysis.
5. Provenance should be preserved whenever possible.
6. The schema is implementation independent.
7. Different processors may generate the metadata provided they conform to this specification.
8. Improvements in reasoning should modify Manifests and Harvests rather than Source Metadata.
---
# Relationship to Other Specifications
This specification defines only Source Metadata.
It should be used together with:
* Conversation Manifest Specification
* Harvest Specification
* Canonical Knowledge Specification
* Knowledge Graph Specification (future)
These specifications collectively define the Project Thoth knowledge processing pipeline.