Initial commit containing architetural scaffolding and mvp
This commit is contained in:
+157
@@ -0,0 +1,157 @@
|
||||
# ADR-001 — Project Thoth is a Platform
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
**Date:** 2026-07-07
|
||||
|
||||
**Version:** 1.0
|
||||
|
||||
---
|
||||
|
||||
# Context
|
||||
|
||||
Project Thoth began as an investigation into preserving conversations with generative AI.
|
||||
|
||||
During architectural exploration, it became clear that the underlying methodology extended far beyond AI conversations.
|
||||
|
||||
The methodology addresses the broader problem of acquiring, organizing, transforming, and reusing human knowledge regardless of its origin.
|
||||
|
||||
Potential implementations include:
|
||||
|
||||
* Browser capture connectors
|
||||
* Office integrations
|
||||
* Desktop applications
|
||||
* Local AI reasoning
|
||||
* Frontier AI reasoning
|
||||
* Knowledge transformation pipelines
|
||||
* Publication workflows
|
||||
|
||||
Initially these appeared to be independent projects.
|
||||
|
||||
Further architectural analysis showed they are different implementations of a common platform.
|
||||
|
||||
---
|
||||
|
||||
# Decision
|
||||
|
||||
Project Thoth shall be treated as a **platform** rather than a single software application.
|
||||
|
||||
The platform consists of multiple complementary components.
|
||||
|
||||
```text
|
||||
Project Thoth
|
||||
│
|
||||
├── Methodology
|
||||
├── Book
|
||||
├── Reference Architecture
|
||||
├── Specifications
|
||||
├── Processor Library
|
||||
├── Capture Connectors
|
||||
├── Desktop Application
|
||||
├── Knowledge Corpus
|
||||
└── Publications
|
||||
```
|
||||
|
||||
The software is an implementation of the methodology.
|
||||
|
||||
The methodology remains the authoritative definition of Project Thoth.
|
||||
|
||||
---
|
||||
|
||||
# Guiding Principles
|
||||
|
||||
1. The methodology is independent of any software implementation.
|
||||
|
||||
2. Software exists to implement the methodology rather than define it.
|
||||
|
||||
3. Multiple implementations may coexist while remaining compliant with the same specifications.
|
||||
|
||||
4. Specifications define behavior.
|
||||
|
||||
5. Processors implement specifications.
|
||||
|
||||
6. Capture occurs where knowledge is created.
|
||||
|
||||
7. Curation occurs within Project Thoth.
|
||||
|
||||
---
|
||||
|
||||
# Rationale
|
||||
|
||||
Treating Project Thoth as a platform provides several advantages.
|
||||
|
||||
The book and software evolve together rather than competing for direction.
|
||||
|
||||
Specifications remain implementation-independent.
|
||||
|
||||
Capture connectors become interchangeable.
|
||||
|
||||
Multiple applications may be developed without changing the underlying methodology.
|
||||
|
||||
Commercial offerings can focus on implementation while preserving a stable conceptual foundation.
|
||||
|
||||
---
|
||||
|
||||
# Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
* Clear separation between methodology and implementation.
|
||||
* Stable architectural foundation.
|
||||
* Easier long-term maintenance.
|
||||
* Multiple connector types become possible.
|
||||
* New processors can be added without redesigning the platform.
|
||||
* Book, consulting, and software reinforce one another.
|
||||
|
||||
Trade-offs:
|
||||
|
||||
* Higher initial architectural effort.
|
||||
* More specifications must be maintained.
|
||||
* Greater emphasis on documentation before implementation.
|
||||
|
||||
These trade-offs are accepted because long-term maintainability is a primary design objective.
|
||||
|
||||
---
|
||||
|
||||
# Future Implications
|
||||
|
||||
Future work may include:
|
||||
|
||||
* Additional Capture Connectors
|
||||
* Desktop applications
|
||||
* Mobile applications
|
||||
* Local LLM integration
|
||||
* Frontier LLM integration
|
||||
* Knowledge graph construction
|
||||
* Automated transformation pipelines
|
||||
* Publishing workflows
|
||||
* Commercial editions
|
||||
* Cognitive support editions
|
||||
|
||||
All future implementations should conform to the Project Thoth methodology and reference architecture.
|
||||
|
||||
---
|
||||
|
||||
# Related Decisions
|
||||
|
||||
None.
|
||||
|
||||
---
|
||||
|
||||
# References
|
||||
|
||||
* Project Thoth Design Principles
|
||||
* Project Thoth Reference Architecture
|
||||
* Project Thoth Implementation Guide
|
||||
|
||||
---
|
||||
|
||||
# Notes
|
||||
|
||||
This Architectural Decision Record establishes Project Thoth as a platform composed of methodology, specifications, processors, applications, and publications.
|
||||
|
||||
Future ADRs should assume this architectural foundation unless explicitly superseded.
|
||||
|
||||
---
|
||||
|
||||
# End
|
||||
+337
@@ -0,0 +1,337 @@
|
||||
---
|
||||
adr: 002
|
||||
title: Canonical Capture Connector Pipeline
|
||||
status: Accepted
|
||||
date: 2026-07-09
|
||||
authors:
|
||||
- Ken Schaefer
|
||||
---
|
||||
|
||||
# ADR-002: Canonical Capture Connector Pipeline
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
---
|
||||
|
||||
# Context
|
||||
|
||||
Project Thoth includes a family of Capture Connectors responsible for preserving conversations and other source material from external systems.
|
||||
|
||||
Initial implementation of the ChatGPT connector attempted to identify individual messages while simultaneously converting DOM content into Markdown.
|
||||
|
||||
During MVP development, testing revealed several classes of defects:
|
||||
|
||||
- Missing conversation content
|
||||
- Duplicate content
|
||||
- Fragmented assistant responses
|
||||
- Incorrect role detection
|
||||
- Loss of tables, links, and formatting
|
||||
- Virtualized (non-rendered) conversation sections
|
||||
- ChatGPT application chrome being captured as conversation content
|
||||
|
||||
Investigation showed that these defects were not primarily caused by Markdown generation. They resulted from attempting to perform conversation discovery and content transformation simultaneously.
|
||||
|
||||
Modern web applications such as ChatGPT are built using React and other component frameworks that expose deeply nested and frequently changing DOM structures. A single logical conversation turn may consist of dozens of nested DOM elements.
|
||||
|
||||
Attempting to infer conversation boundaries while simultaneously rendering Markdown creates unnecessary complexity and makes debugging difficult.
|
||||
|
||||
---
|
||||
|
||||
# Decision
|
||||
|
||||
All Project Thoth Capture Connectors SHALL implement a three-stage pipeline:
|
||||
|
||||
```
|
||||
Conversation Discovery
|
||||
│
|
||||
▼
|
||||
Conversation Turn Model
|
||||
│
|
||||
▼
|
||||
Content Transformation
|
||||
│
|
||||
▼
|
||||
Markdown Serialization
|
||||
```
|
||||
|
||||
Each stage has a single responsibility.
|
||||
|
||||
---
|
||||
|
||||
# Stage 1 — Conversation Discovery
|
||||
|
||||
Purpose:
|
||||
|
||||
Identify the canonical conversation turns for a source platform.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Locate conversation root
|
||||
- Locate conversation turn containers
|
||||
- Determine turn ordering
|
||||
- Determine speaker role
|
||||
- Detect unsupported content
|
||||
- Detect partially rendered or virtualized content
|
||||
- Produce a platform-neutral intermediate representation
|
||||
|
||||
This stage SHALL NOT:
|
||||
|
||||
- Generate Markdown
|
||||
- Normalize formatting
|
||||
- Generate metadata
|
||||
- Summarize
|
||||
- Invoke LLMs
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
Conversation
|
||||
Turn
|
||||
Turn
|
||||
Turn
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Stage 2 — Content Transformation
|
||||
|
||||
Purpose:
|
||||
|
||||
Transform a single conversation turn into normalized content.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Convert HTML to Markdown
|
||||
- Preserve paragraphs
|
||||
- Preserve headings
|
||||
- Preserve lists
|
||||
- Preserve tables
|
||||
- Preserve links
|
||||
- Preserve images
|
||||
- Preserve code blocks
|
||||
- Preserve inline formatting
|
||||
|
||||
This stage SHALL NOT:
|
||||
|
||||
- Discover conversation turns
|
||||
- Infer ordering
|
||||
- Generate files
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
Conversation Turn
|
||||
│
|
||||
▼
|
||||
Markdown
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Stage 3 — Markdown Serialization
|
||||
|
||||
Purpose:
|
||||
|
||||
Produce the canonical Project Thoth `conversation.md` document.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Write capture metadata
|
||||
- Preserve conversation order
|
||||
- Emit User / Assistant boundaries
|
||||
- Write final Markdown document
|
||||
|
||||
This stage SHALL NOT:
|
||||
|
||||
- Parse HTML
|
||||
- Discover DOM elements
|
||||
- Modify extracted content
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
conversation.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Intermediate Representation
|
||||
|
||||
Conversation discovery SHALL produce a platform-neutral model.
|
||||
|
||||
Example:
|
||||
|
||||
```typescript
|
||||
interface Conversation {
|
||||
|
||||
sourcePlatform: string;
|
||||
|
||||
title: string;
|
||||
|
||||
url: string;
|
||||
|
||||
capturedAt: Date;
|
||||
|
||||
turns: ConversationTurn[];
|
||||
}
|
||||
|
||||
interface ConversationTurn {
|
||||
|
||||
turnIndex: number;
|
||||
|
||||
role:
|
||||
| "user"
|
||||
| "assistant"
|
||||
| "system"
|
||||
| "tool"
|
||||
| "unknown";
|
||||
|
||||
captureStatus:
|
||||
| "rendered"
|
||||
| "not_rendered"
|
||||
| "unsupported";
|
||||
|
||||
sourceElement: HTMLElement;
|
||||
|
||||
markdown?: string;
|
||||
}
|
||||
```
|
||||
|
||||
The Intermediate Representation (IR) becomes the contract between discovery and transformation.
|
||||
|
||||
---
|
||||
|
||||
# Rationale
|
||||
|
||||
Separating discovery from rendering provides several advantages.
|
||||
|
||||
## Separation of Concerns
|
||||
|
||||
Each stage performs one responsibility.
|
||||
|
||||
Conversation discovery determines *what* exists.
|
||||
|
||||
Content transformation determines *how* it is represented.
|
||||
|
||||
Markdown serialization determines *how* it is packaged.
|
||||
|
||||
---
|
||||
|
||||
## Testability
|
||||
|
||||
Each stage can be independently tested.
|
||||
|
||||
Examples:
|
||||
|
||||
- Discovery tests verify turn detection.
|
||||
- Transformation tests verify Markdown fidelity.
|
||||
- Serialization tests verify document format.
|
||||
|
||||
Failures can be isolated without affecting unrelated stages.
|
||||
|
||||
---
|
||||
|
||||
## Maintainability
|
||||
|
||||
Modern web applications frequently change DOM structure.
|
||||
|
||||
When a platform changes, only Conversation Discovery should typically require modification.
|
||||
|
||||
Markdown rendering remains reusable across platforms.
|
||||
|
||||
---
|
||||
|
||||
## Reuse
|
||||
|
||||
Most Capture Connectors share identical downstream behavior.
|
||||
|
||||
Expected connectors include:
|
||||
|
||||
- ChatGPT
|
||||
- Claude
|
||||
- Gemini
|
||||
- Microsoft Copilot
|
||||
- Open WebUI
|
||||
- Perplexity
|
||||
- Future browser-based AI systems
|
||||
|
||||
Only Conversation Discovery is expected to be platform-specific.
|
||||
|
||||
---
|
||||
|
||||
## Debuggability
|
||||
|
||||
The IR enables inspection before Markdown generation.
|
||||
|
||||
Developers can validate:
|
||||
|
||||
- turn count
|
||||
- ordering
|
||||
- role detection
|
||||
- unsupported content
|
||||
- rendering completeness
|
||||
|
||||
without involving Markdown generation.
|
||||
|
||||
---
|
||||
|
||||
# Consequences
|
||||
|
||||
## Positive
|
||||
|
||||
- Cleaner architecture
|
||||
- Easier debugging
|
||||
- Platform independence
|
||||
- Improved testability
|
||||
- Reduced coupling
|
||||
- Higher long-term maintainability
|
||||
|
||||
## Negative
|
||||
|
||||
- Additional abstraction layer
|
||||
- Slightly more implementation effort
|
||||
- Requires maintenance of an Intermediate Representation
|
||||
|
||||
---
|
||||
|
||||
# Alternatives Considered
|
||||
|
||||
## Single-Pass DOM → Markdown
|
||||
|
||||
Rejected.
|
||||
|
||||
Although initially simpler, this approach couples conversation discovery with formatting.
|
||||
|
||||
Testing demonstrated that defects become difficult to isolate and frequently require heuristic patches.
|
||||
|
||||
---
|
||||
|
||||
## Platform-Specific End-to-End Connectors
|
||||
|
||||
Rejected.
|
||||
|
||||
Embedding discovery, rendering, and serialization into a single connector creates duplication across platforms and limits reuse.
|
||||
|
||||
---
|
||||
|
||||
# Future Considerations
|
||||
|
||||
Future Capture Connectors may introduce an optional preprocessing stage before Conversation Discovery.
|
||||
|
||||
Examples include:
|
||||
|
||||
- Automatic scrolling to render virtualized conversation turns
|
||||
- Lazy-loading attachments
|
||||
- Expansion of collapsed content
|
||||
|
||||
These preprocessing activities remain outside the canonical pipeline and serve only to improve completeness of the source material.
|
||||
|
||||
---
|
||||
|
||||
# References
|
||||
|
||||
- ADR-001: Project Structure
|
||||
- Project Thoth MVP — ChatGPT Capture Connector
|
||||
- Empirical findings from ChatGPT DOM extraction (July 2026)
|
||||
+383
@@ -0,0 +1,383 @@
|
||||
---
|
||||
adr: 003
|
||||
title: Capture Connector Architecture
|
||||
status: Accepted
|
||||
date: 2026-07-09
|
||||
authors:
|
||||
- Ken Schaefer
|
||||
---
|
||||
|
||||
# ADR-003: Capture Connector Architecture
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
---
|
||||
|
||||
# Context
|
||||
|
||||
Project Thoth is intended to preserve knowledge from a growing ecosystem of AI assistants and digital systems.
|
||||
|
||||
Initially, the first connector targeted ChatGPT. During development it became clear that the ChatGPT connector was not unique. Although each platform exposes different APIs and DOM structures, every connector ultimately performs the same high-level task:
|
||||
|
||||
1. Capture source material.
|
||||
2. Transform it into a canonical representation.
|
||||
3. Deliver the result to Project Thoth.
|
||||
|
||||
The platform-specific logic lies almost entirely in *how the source material is discovered*. Once discovered, the remainder of the processing pipeline is largely identical.
|
||||
|
||||
Rather than implementing each connector as a monolithic application, Project Thoth should define a common connector architecture with clearly defined extension points.
|
||||
|
||||
---
|
||||
|
||||
# Decision
|
||||
|
||||
All Project Thoth Capture Connectors SHALL implement the same logical architecture.
|
||||
|
||||
```
|
||||
Source Platform
|
||||
│
|
||||
▼
|
||||
Platform Discovery Layer
|
||||
│
|
||||
▼
|
||||
Canonical Conversation Model
|
||||
│
|
||||
▼
|
||||
Platform-Neutral Transformation
|
||||
│
|
||||
▼
|
||||
Canonical conversation.md
|
||||
│
|
||||
▼
|
||||
Project Thoth Pipeline
|
||||
```
|
||||
|
||||
Only the Discovery Layer is expected to be platform-specific.
|
||||
|
||||
---
|
||||
|
||||
# Connector Responsibilities
|
||||
|
||||
Capture Connectors are responsible only for preserving source material.
|
||||
|
||||
They SHALL:
|
||||
|
||||
- Capture conversations
|
||||
- Capture attachments
|
||||
- Preserve ordering
|
||||
- Preserve formatting where practical
|
||||
- Preserve metadata supplied by the source platform
|
||||
- Produce canonical Project Thoth documents
|
||||
|
||||
They SHALL NOT:
|
||||
|
||||
- Summarize
|
||||
- Classify
|
||||
- Generate tags
|
||||
- Generate YAML metadata
|
||||
- Generate manifests
|
||||
- Generate harvests
|
||||
- Perform semantic analysis
|
||||
- Invoke LLMs
|
||||
- Modify user content
|
||||
|
||||
Connectors are intentionally "dumb."
|
||||
|
||||
Their responsibility is faithful preservation.
|
||||
|
||||
---
|
||||
|
||||
# Canonical Connector Pipeline
|
||||
|
||||
Every connector SHALL implement the following stages.
|
||||
|
||||
```
|
||||
Preprocessing (Optional)
|
||||
│
|
||||
▼
|
||||
Discovery
|
||||
│
|
||||
▼
|
||||
Intermediate Representation
|
||||
│
|
||||
▼
|
||||
Transformation
|
||||
│
|
||||
▼
|
||||
Serialization
|
||||
│
|
||||
▼
|
||||
Output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Stage 0 — Preprocessing (Optional)
|
||||
|
||||
Purpose:
|
||||
|
||||
Prepare the source for capture.
|
||||
|
||||
Examples:
|
||||
|
||||
- Render virtualized content
|
||||
- Expand collapsed sections
|
||||
- Load lazy content
|
||||
- Wait for streaming responses to complete
|
||||
|
||||
Preprocessing SHALL NOT modify user content.
|
||||
|
||||
---
|
||||
|
||||
# Stage 1 — Discovery
|
||||
|
||||
Purpose:
|
||||
|
||||
Locate the logical content exposed by the source platform.
|
||||
|
||||
Responsibilities include:
|
||||
|
||||
- Locate conversation root
|
||||
- Locate conversation turns
|
||||
- Determine ordering
|
||||
- Determine speaker
|
||||
- Detect unsupported content
|
||||
- Detect partial rendering
|
||||
|
||||
Discovery is platform-specific.
|
||||
|
||||
---
|
||||
|
||||
# Stage 2 — Intermediate Representation
|
||||
|
||||
Purpose:
|
||||
|
||||
Represent captured information in a platform-neutral model.
|
||||
|
||||
Example:
|
||||
|
||||
```typescript
|
||||
interface Conversation {
|
||||
|
||||
sourcePlatform: string;
|
||||
|
||||
title: string;
|
||||
|
||||
url: string;
|
||||
|
||||
capturedAt: Date;
|
||||
|
||||
turns: ConversationTurn[];
|
||||
}
|
||||
|
||||
interface ConversationTurn {
|
||||
|
||||
turnIndex: number;
|
||||
|
||||
role: string;
|
||||
|
||||
captureStatus: string;
|
||||
|
||||
content: DocumentFragment | HTMLElement | string;
|
||||
}
|
||||
```
|
||||
|
||||
The Intermediate Representation (IR) is the contract between Discovery and Transformation.
|
||||
|
||||
---
|
||||
|
||||
# Stage 3 — Transformation
|
||||
|
||||
Purpose:
|
||||
|
||||
Convert platform-specific content into canonical Project Thoth Markdown.
|
||||
|
||||
Responsibilities include:
|
||||
|
||||
- HTML → Markdown
|
||||
- Paragraph preservation
|
||||
- Lists
|
||||
- Tables
|
||||
- Links
|
||||
- Images
|
||||
- Code blocks
|
||||
- Inline formatting
|
||||
|
||||
Transformation is expected to be reusable across platforms.
|
||||
|
||||
---
|
||||
|
||||
# Stage 4 — Serialization
|
||||
|
||||
Purpose:
|
||||
|
||||
Generate the canonical output artifacts.
|
||||
|
||||
Current artifacts include:
|
||||
|
||||
- `conversation.md`
|
||||
|
||||
Future artifacts may include:
|
||||
|
||||
- Attachments
|
||||
- Asset manifests
|
||||
- Conversation package formats
|
||||
|
||||
Serialization SHALL NOT reinterpret content.
|
||||
|
||||
---
|
||||
|
||||
# Platform Independence
|
||||
|
||||
The connector architecture intentionally separates platform-specific logic from platform-neutral logic.
|
||||
|
||||
Examples:
|
||||
|
||||
| Component | Platform Specific |
|
||||
|-----------|-------------------|
|
||||
| Discovery | Yes |
|
||||
| Preprocessing | Mostly |
|
||||
| Transformation | No |
|
||||
| Serialization | No |
|
||||
|
||||
This minimizes duplication across connectors.
|
||||
|
||||
---
|
||||
|
||||
# Supported Connector Types
|
||||
|
||||
The architecture is intended to support connectors including, but not limited to:
|
||||
|
||||
- ChatGPT
|
||||
- Claude
|
||||
- Gemini
|
||||
- Microsoft Copilot
|
||||
- Open WebUI
|
||||
- Perplexity
|
||||
- Cursor
|
||||
- GitHub Copilot Chat
|
||||
- Future browser-based AI assistants
|
||||
|
||||
Additional connectors should primarily require implementation of Discovery and, where necessary, Preprocessing.
|
||||
|
||||
---
|
||||
|
||||
# Design Principles
|
||||
|
||||
## Fidelity Over Intelligence
|
||||
|
||||
Connectors preserve information.
|
||||
|
||||
They do not interpret information.
|
||||
|
||||
---
|
||||
|
||||
## Platform Neutrality
|
||||
|
||||
Internal Project Thoth formats are independent of any external platform.
|
||||
|
||||
No downstream component should need to know whether content originated from ChatGPT, Gemini, Claude, or another system.
|
||||
|
||||
---
|
||||
|
||||
## Composability
|
||||
|
||||
Each stage should be independently testable and replaceable.
|
||||
|
||||
This enables improvements to one stage without affecting others.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Output
|
||||
|
||||
Running the connector multiple times against the same rendered conversation should produce equivalent output.
|
||||
|
||||
---
|
||||
|
||||
## Fail Gracefully
|
||||
|
||||
When unsupported content is encountered:
|
||||
|
||||
- Preserve placeholders
|
||||
- Preserve ordering
|
||||
- Record warnings
|
||||
- Never silently discard content
|
||||
|
||||
---
|
||||
|
||||
# Error Handling
|
||||
|
||||
Capture is considered successful when source material is faithfully preserved.
|
||||
|
||||
If content cannot be rendered or extracted:
|
||||
|
||||
- Report the issue
|
||||
- Preserve available context
|
||||
- Continue processing remaining content
|
||||
|
||||
Partial capture is preferred over silent failure.
|
||||
|
||||
---
|
||||
|
||||
# Consequences
|
||||
|
||||
## Positive
|
||||
|
||||
- Uniform architecture across all connectors
|
||||
- Reduced duplication
|
||||
- Easier testing
|
||||
- Improved maintainability
|
||||
- Simplified onboarding for new connector development
|
||||
- Reusable transformation and serialization components
|
||||
|
||||
## Negative
|
||||
|
||||
- Additional abstraction layers
|
||||
- Requires maintenance of a shared Intermediate Representation
|
||||
- Slightly higher initial implementation effort
|
||||
|
||||
---
|
||||
|
||||
# Alternatives Considered
|
||||
|
||||
## Monolithic Platform Connectors
|
||||
|
||||
Rejected.
|
||||
|
||||
Embedding discovery, transformation, and serialization into a single implementation results in duplicated logic and inconsistent behavior across connectors.
|
||||
|
||||
---
|
||||
|
||||
## Direct Platform-to-Markdown Conversion
|
||||
|
||||
Rejected.
|
||||
|
||||
Coupling discovery with rendering makes debugging difficult and limits reuse.
|
||||
|
||||
---
|
||||
|
||||
# Relationship to Other ADRs
|
||||
|
||||
- **ADR-001** establishes the overall Project Thoth repository and project structure.
|
||||
- **ADR-002** defines the canonical capture pipeline used within connectors.
|
||||
- **ADR-003** defines the architectural responsibilities, lifecycle, and composition of Capture Connectors as reusable platform adapters.
|
||||
|
||||
Together, these ADRs establish the foundation for a connector ecosystem rather than a collection of independent integrations.
|
||||
|
||||
---
|
||||
|
||||
# Future Considerations
|
||||
|
||||
Future enhancements may include:
|
||||
|
||||
- Native API-based connectors where supported
|
||||
- Hybrid API + browser capture
|
||||
- Incremental conversation synchronization
|
||||
- Background monitoring of supported platforms
|
||||
- Signed connector packages
|
||||
- Connector capability negotiation
|
||||
- Automated regression testing against captured DOM snapshots
|
||||
|
||||
The architectural principles defined in this ADR are expected to remain stable even as individual source platforms evolve.
|
||||
@@ -0,0 +1,631 @@
|
||||
# Project Thoth Application
|
||||
|
||||
# ChatGPT Capture Connector MVP
|
||||
|
||||
## Architecture and Sprint Plan
|
||||
|
||||
## Version 0.1
|
||||
|
||||
---
|
||||
|
||||
# Purpose
|
||||
|
||||
This document defines the architecture and implementation plan for the first Project Thoth Capture Connector MVP.
|
||||
|
||||
The MVP will create a browser extension that captures the currently open ChatGPT conversation and saves it as a Markdown file to the user's local Downloads folder.
|
||||
|
||||
This MVP validates the Capture Connector boundary before the Project Thoth application exists.
|
||||
|
||||
---
|
||||
|
||||
# Product Boundary
|
||||
|
||||
Project Thoth has two systems.
|
||||
|
||||
```text
|
||||
System 1: Capture Connectors
|
||||
System 2: Project Thoth Application
|
||||
```
|
||||
|
||||
This sprint implements only the first MVP Capture Connector.
|
||||
|
||||
---
|
||||
|
||||
# MVP Definition
|
||||
|
||||
The MVP browser extension must:
|
||||
|
||||
1. Present a browser toolbar button.
|
||||
|
||||
2. Assume the user is already logged into ChatGPT.
|
||||
|
||||
3. Assume a ChatGPT conversation is open in the current browser tab.
|
||||
|
||||
4. Capture the visible/current conversation content.
|
||||
|
||||
5. Preserve message order as well as practical.
|
||||
|
||||
6. Preserve user and assistant content as well as practical.
|
||||
|
||||
7. Export the captured conversation as Markdown.
|
||||
|
||||
8. Save the file to the local Downloads folder.
|
||||
|
||||
9. Avoid all AI processing.
|
||||
|
||||
10. Avoid all Project Thoth application dependencies.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Non-Goals
|
||||
|
||||
The MVP will not:
|
||||
|
||||
- Generate Source Metadata.
|
||||
|
||||
- Generate Conversation Manifests.
|
||||
|
||||
- Generate Harvest artifacts.
|
||||
|
||||
- Call a local LLM.
|
||||
|
||||
- Call a frontier LLM.
|
||||
|
||||
- Write directly to the vault.
|
||||
|
||||
- Sync with Open WebUI.
|
||||
|
||||
- Manage Knowledge Collections.
|
||||
|
||||
- Support Gemini.
|
||||
|
||||
- Support Copilot.
|
||||
|
||||
- Support Outlook.
|
||||
|
||||
- Support PDFs.
|
||||
|
||||
- Provide a full desktop application.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Architectural Principle
|
||||
|
||||
> Capture Connectors do not reason. They only capture and transmit source material.
|
||||
|
||||
For this MVP, "transmit" means saving a Markdown file to Downloads.
|
||||
|
||||
---
|
||||
|
||||
# Technical Platform
|
||||
|
||||
The MVP should use a Manifest V3 browser extension.
|
||||
|
||||
Manifest V3 is the current Chrome extensions platform, and every extension requires a `manifest.json` file that defines extension metadata, permissions, and behavior.
|
||||
|
||||
The MVP should initially target:
|
||||
|
||||
```text
|
||||
Google Chrome / Microsoft Edge
|
||||
```
|
||||
|
||||
because both use the Chromium extension model.
|
||||
|
||||
---
|
||||
|
||||
# High-Level Architecture
|
||||
|
||||
```text
|
||||
User opens ChatGPT conversation
|
||||
↓
|
||||
User clicks Project Thoth browser button
|
||||
↓
|
||||
Extension receives active tab permission
|
||||
↓
|
||||
Content script extracts conversation content
|
||||
↓
|
||||
Extension normalizes content into Markdown
|
||||
↓
|
||||
Extension saves file to Downloads
|
||||
```
|
||||
|
||||
Chrome's `activeTab` permission grants temporary access to the current tab when the user invokes the extension, which matches the intended "click to capture this page" UX.
|
||||
|
||||
Content scripts can run in the context of a web page and read page content through standard Web APIs, which is the mechanism the extension will use to inspect the ChatGPT page.
|
||||
|
||||
---
|
||||
|
||||
# Component Architecture
|
||||
|
||||
## 1. Browser Action
|
||||
|
||||
Purpose:
|
||||
|
||||
- Provide the toolbar button.
|
||||
|
||||
- Trigger capture.
|
||||
|
||||
|
||||
User-facing label:
|
||||
|
||||
```text
|
||||
Save to Project Thoth
|
||||
```
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
## 2. Manifest Configuration
|
||||
|
||||
Purpose:
|
||||
|
||||
- Define extension metadata.
|
||||
|
||||
- Request minimal permissions.
|
||||
|
||||
- Register the service worker.
|
||||
|
||||
- Configure content script execution or script injection.
|
||||
|
||||
|
||||
Likely permissions:
|
||||
|
||||
```json
|
||||
"permissions": ["activeTab", "scripting", "downloads"]
|
||||
```
|
||||
|
||||
Likely host permissions:
|
||||
|
||||
```json
|
||||
"host_permissions": ["https://chatgpt.com/*", "https://chat.openai.com/*"]
|
||||
```
|
||||
|
||||
Chrome extensions must declare permissions in the manifest to use extension APIs.
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
## 3. Service Worker
|
||||
|
||||
Purpose:
|
||||
|
||||
- Listen for toolbar button clicks.
|
||||
|
||||
- Inject or invoke the content script.
|
||||
|
||||
- Receive extracted content.
|
||||
|
||||
- Create Markdown file.
|
||||
|
||||
- Save through browser download API.
|
||||
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
## 4. ChatGPT Content Extractor
|
||||
|
||||
Purpose:
|
||||
|
||||
- Inspect the current ChatGPT conversation page.
|
||||
|
||||
- Extract message blocks.
|
||||
|
||||
- Preserve ordering.
|
||||
|
||||
- Detect speaker role when possible.
|
||||
|
||||
- Preserve Markdown-like formatting where possible.
|
||||
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"sourcePlatform": "ChatGPT",
|
||||
"title": "Detected conversation title",
|
||||
"url": "https://chatgpt.com/...",
|
||||
"capturedAt": "2026-07-07T...",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "..."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
## 5. Markdown Normalizer
|
||||
|
||||
Purpose:
|
||||
|
||||
Convert extracted message data into canonical Markdown.
|
||||
|
||||
MVP output format:
|
||||
|
||||
```markdown
|
||||
# Conversation Title
|
||||
|
||||
Captured: 2026-07-07
|
||||
Source Platform: ChatGPT
|
||||
Source URL: https://chatgpt.com/...
|
||||
|
||||
---
|
||||
|
||||
## User
|
||||
|
||||
Message content
|
||||
|
||||
---
|
||||
|
||||
## Assistant
|
||||
|
||||
Message content
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
## 6. Download Writer
|
||||
|
||||
Purpose:
|
||||
|
||||
- Generate a safe filename.
|
||||
|
||||
- Save Markdown file to Downloads.
|
||||
|
||||
- Avoid overwriting when possible.
|
||||
|
||||
|
||||
Example filename:
|
||||
|
||||
```text
|
||||
2026-07-07 - ChatGPT - Conversation Title.md
|
||||
```
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
# MVP File Output
|
||||
|
||||
The MVP produces exactly one file:
|
||||
|
||||
```text
|
||||
conversation.md
|
||||
```
|
||||
|
||||
Saved to:
|
||||
|
||||
```text
|
||||
Downloads/
|
||||
```
|
||||
|
||||
Later versions may produce:
|
||||
|
||||
```text
|
||||
conversation.raw.json
|
||||
conversation.manifest.md
|
||||
conversation.harvest.md
|
||||
```
|
||||
|
||||
but those are explicitly deferred.
|
||||
|
||||
---
|
||||
|
||||
# Error Handling
|
||||
|
||||
The extension should show simple user-facing messages.
|
||||
|
||||
## Success
|
||||
|
||||
```text
|
||||
Saved to Downloads.
|
||||
```
|
||||
|
||||
## No ChatGPT Conversation Detected
|
||||
|
||||
```text
|
||||
No ChatGPT conversation was detected on this page.
|
||||
```
|
||||
|
||||
## Capture Failed
|
||||
|
||||
```text
|
||||
Capture failed. Try scrolling through the conversation and capturing again.
|
||||
```
|
||||
|
||||
## Download Failed
|
||||
|
||||
```text
|
||||
The conversation was captured, but the file could not be saved.
|
||||
```
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
# Key Technical Risk
|
||||
|
||||
The largest technical risk is ChatGPT DOM instability.
|
||||
|
||||
The ChatGPT page structure may change without notice.
|
||||
|
||||
Therefore, the extractor should be isolated in its own module:
|
||||
|
||||
```text
|
||||
chatgptExtractor.js
|
||||
```
|
||||
|
||||
Do not mix ChatGPT-specific DOM logic with generic extension logic.
|
||||
|
||||
---
|
||||
|
||||
# Proposed Project Structure
|
||||
|
||||
```text
|
||||
project-thoth-chatgpt-capture/
|
||||
|
||||
manifest.json
|
||||
|
||||
src/
|
||||
background.js
|
||||
chatgptExtractor.js
|
||||
markdownNormalizer.js
|
||||
filename.js
|
||||
|
||||
icons/
|
||||
icon16.png
|
||||
icon48.png
|
||||
icon128.png
|
||||
|
||||
README.md
|
||||
```
|
||||
|
||||
Codex implementation required.
|
||||
|
||||
---
|
||||
|
||||
# Sprint Plan
|
||||
|
||||
## Sprint Goal
|
||||
|
||||
Build and manually test a browser extension that captures the currently open ChatGPT conversation and saves it as Markdown to Downloads.
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — Create Extension Skeleton
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `manifest.json`
|
||||
|
||||
- background service worker
|
||||
|
||||
- toolbar button
|
||||
|
||||
- placeholder icon
|
||||
|
||||
- local install instructions
|
||||
|
||||
|
||||
Codex prompt needed:
|
||||
|
||||
```text
|
||||
Prompt Codex to create a minimal Manifest V3 browser extension skeleton for Chrome/Edge with a toolbar button and service worker.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Implement Active Tab Capture Flow
|
||||
|
||||
Deliverables:
|
||||
|
||||
- Toolbar click handler
|
||||
|
||||
- active tab lookup
|
||||
|
||||
- script injection
|
||||
|
||||
- message passing between service worker and content script
|
||||
|
||||
|
||||
Codex prompt needed:
|
||||
|
||||
```text
|
||||
Prompt Codex to implement activeTab-based capture flow using Manifest V3 service worker, scripting API, and content script message passing.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — Implement ChatGPT DOM Extractor
|
||||
|
||||
Deliverables:
|
||||
|
||||
- Extract conversation title if available
|
||||
|
||||
- Extract ordered message blocks
|
||||
|
||||
- Detect user vs assistant messages when possible
|
||||
|
||||
- Extract text content
|
||||
|
||||
- Preserve basic formatting
|
||||
|
||||
|
||||
Codex prompt needed:
|
||||
|
||||
```text
|
||||
Prompt Codex to write a ChatGPT-specific DOM extractor module that returns ordered conversation messages with role and content fields.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — Implement Markdown Normalizer
|
||||
|
||||
Deliverables:
|
||||
|
||||
- Convert extracted messages to canonical Markdown
|
||||
|
||||
- Include capture metadata header
|
||||
|
||||
- Preserve message order
|
||||
|
||||
- Add `## User` and `## Assistant` boundaries
|
||||
|
||||
|
||||
Codex prompt needed:
|
||||
|
||||
```text
|
||||
Prompt Codex to create a markdown normalization module that converts extracted ChatGPT messages into Project Thoth conversation.md format.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — Implement Download Writer
|
||||
|
||||
Deliverables:
|
||||
|
||||
- Generate filename
|
||||
|
||||
- Save Markdown to Downloads
|
||||
|
||||
- Sanitize invalid filename characters
|
||||
|
||||
- Add timestamp if needed
|
||||
|
||||
|
||||
Codex prompt needed:
|
||||
|
||||
```text
|
||||
Prompt Codex to implement Markdown download using the browser downloads API with safe filename generation.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6 — Add User Feedback
|
||||
|
||||
Deliverables:
|
||||
|
||||
- Success message
|
||||
|
||||
- Error message when not on ChatGPT
|
||||
|
||||
- Error message when no conversation is detected
|
||||
|
||||
- Error message when download fails
|
||||
|
||||
|
||||
Codex prompt needed:
|
||||
|
||||
```text
|
||||
Prompt Codex to add simple user feedback for successful capture and failure states.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7 — Manual Test Pass
|
||||
|
||||
Test cases:
|
||||
|
||||
1. Short ChatGPT conversation.
|
||||
|
||||
2. Long ChatGPT conversation.
|
||||
|
||||
3. Conversation with code blocks.
|
||||
|
||||
4. Conversation with markdown lists.
|
||||
|
||||
5. Conversation with headings.
|
||||
|
||||
6. Conversation with uploaded file references.
|
||||
|
||||
7. Conversation where only part of the conversation has been scrolled into view.
|
||||
|
||||
8. Non-ChatGPT page.
|
||||
|
||||
|
||||
Codex prompt may be needed after defects are found:
|
||||
|
||||
```text
|
||||
Prompt Codex to fix extraction or formatting defects discovered during manual testing.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Acceptance Criteria
|
||||
|
||||
The MVP is complete when:
|
||||
|
||||
- Extension installs locally in Chrome or Edge.
|
||||
|
||||
- Toolbar button appears.
|
||||
|
||||
- User can open a ChatGPT conversation and click the button.
|
||||
|
||||
- A Markdown file is downloaded.
|
||||
|
||||
- The Markdown file contains the conversation content.
|
||||
|
||||
- Message order is preserved.
|
||||
|
||||
- Speaker boundaries are present when detectable.
|
||||
|
||||
- The extension does not call any LLM.
|
||||
|
||||
- The extension does not require Project Thoth application installation.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Deferred Architecture
|
||||
|
||||
The following belong to later sprints:
|
||||
|
||||
```text
|
||||
Project Thoth desktop application
|
||||
Native messaging
|
||||
Vault writing
|
||||
Git integration
|
||||
Source Metadata generation
|
||||
Conversation Manifest generation
|
||||
Open WebUI integration
|
||||
Local LLM integration
|
||||
Gemini connector
|
||||
Copilot connector
|
||||
Office connector
|
||||
Adobe connector
|
||||
General web clipper
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Architectural Finding to Validate
|
||||
|
||||
This MVP should answer one question:
|
||||
|
||||
> Can a browser extension reliably capture a complete ChatGPT conversation into a usable Project Thoth Primary Source?
|
||||
|
||||
If yes, the connector architecture is viable.
|
||||
|
||||
If no, Project Thoth must use a different acquisition path for ChatGPT conversations.
|
||||
|
||||
---
|
||||
|
||||
# End
|
||||
@@ -0,0 +1,282 @@
|
||||
# Project Thoth Reference Architecture
|
||||
|
||||
## Version 0.2 — Capture Boundary
|
||||
|
||||
# Core Definition
|
||||
|
||||
Project Thoth consists of two primary systems:
|
||||
|
||||
```text
|
||||
System 1: Capture Connectors
|
||||
System 2: Project Thoth Application
|
||||
```
|
||||
|
||||
The boundary is strict:
|
||||
|
||||
> Capture Connectors capture source material.
|
||||
> Project Thoth stores, processes, reasons over, and transforms that material.
|
||||
|
||||
---
|
||||
|
||||
# System 1 — Capture Connectors
|
||||
|
||||
## Purpose
|
||||
|
||||
Capture Connectors operate where knowledge is created or encountered.
|
||||
|
||||
Examples:
|
||||
|
||||
- ChatGPT
|
||||
|
||||
- Gemini
|
||||
|
||||
- Copilot
|
||||
|
||||
- Open WebUI
|
||||
|
||||
- Outlook
|
||||
|
||||
- Word
|
||||
|
||||
- Excel
|
||||
|
||||
- PowerPoint
|
||||
|
||||
- Adobe Acrobat
|
||||
|
||||
- Web pages
|
||||
|
||||
- Slack
|
||||
|
||||
- Teams
|
||||
|
||||
- Email
|
||||
|
||||
- YouTube
|
||||
|
||||
|
||||
## Responsibilities
|
||||
|
||||
Capture Connectors may:
|
||||
|
||||
- detect the active source
|
||||
|
||||
- extract source content
|
||||
|
||||
- preserve structure when possible
|
||||
|
||||
- capture technical metadata
|
||||
|
||||
- send the captured payload to Project Thoth
|
||||
|
||||
|
||||
## Non-Responsibilities
|
||||
|
||||
Capture Connectors must not:
|
||||
|
||||
- reason over the source
|
||||
|
||||
- generate Source Metadata
|
||||
|
||||
- generate Manifests
|
||||
|
||||
- generate Harvests
|
||||
|
||||
- decide canonical knowledge
|
||||
|
||||
- call LLM processors
|
||||
|
||||
- manage the vault
|
||||
|
||||
|
||||
## MVP Connector
|
||||
|
||||
The MVP Capture Connector is:
|
||||
|
||||
```text
|
||||
ChatGPT Capture Connector
|
||||
```
|
||||
|
||||
Its single user-facing action is:
|
||||
|
||||
```text
|
||||
Save to Project Thoth
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# System 2 — Project Thoth Application
|
||||
|
||||
## Purpose
|
||||
|
||||
The Project Thoth Application manages the knowledge corpus.
|
||||
|
||||
It is the place where captured content is stored, processed, searched, reasoned over, transformed, and published.
|
||||
|
||||
## Responsibilities
|
||||
|
||||
Project Thoth may:
|
||||
|
||||
- receive captured payloads
|
||||
|
||||
- save Primary Sources to the vault
|
||||
|
||||
- display raw captured content
|
||||
|
||||
- manage vault structure
|
||||
|
||||
- generate Source Metadata
|
||||
|
||||
- generate Conversation Manifests
|
||||
|
||||
- run Harvest processors
|
||||
|
||||
- call local LLMs
|
||||
|
||||
- call frontier LLMs
|
||||
|
||||
- sync with Knowledge Collections
|
||||
|
||||
- manage reasoning contexts
|
||||
|
||||
- support search and review
|
||||
|
||||
- generate downstream artifacts
|
||||
|
||||
|
||||
## MVP Application
|
||||
|
||||
The MVP Project Thoth Application should:
|
||||
|
||||
- receive captured ChatGPT content
|
||||
|
||||
- save the raw capture
|
||||
|
||||
- save a normalized markdown file
|
||||
|
||||
- display captured content
|
||||
|
||||
- provide scaffolding for future metadata, manifest, and harvest processing
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Boundary Contract
|
||||
|
||||
Capture Connectors send captured source packages to Project Thoth.
|
||||
|
||||
A capture package may contain:
|
||||
|
||||
```text
|
||||
raw_capture
|
||||
normalized_markdown
|
||||
capture_metadata
|
||||
attachments
|
||||
source_references
|
||||
```
|
||||
|
||||
## Capture Metadata
|
||||
|
||||
Capture metadata is technical, not interpretive.
|
||||
|
||||
Examples:
|
||||
|
||||
- source platform
|
||||
|
||||
- source URL
|
||||
|
||||
- capture date
|
||||
|
||||
- connector version
|
||||
|
||||
- detected title
|
||||
|
||||
- message count
|
||||
|
||||
- source format
|
||||
|
||||
|
||||
Capture metadata is not the same as Project Thoth Source Metadata.
|
||||
|
||||
---
|
||||
|
||||
# Processing Pipeline
|
||||
|
||||
```text
|
||||
Knowledge Source
|
||||
↓
|
||||
Capture Connector
|
||||
↓
|
||||
Captured Source Package
|
||||
↓
|
||||
Project Thoth Application
|
||||
↓
|
||||
Vault Storage
|
||||
↓
|
||||
Source Metadata Processor
|
||||
↓
|
||||
Manifest Processor
|
||||
↓
|
||||
Purpose-Specific Transformations
|
||||
↓
|
||||
Canonical Knowledge / Publications / Runbooks / Courses
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Design Principles
|
||||
|
||||
1. Capture connectors do not reason.
|
||||
|
||||
2. Project Thoth owns processing.
|
||||
|
||||
3. The vault is the durable source repository.
|
||||
|
||||
4. Tools are replaceable.
|
||||
|
||||
5. Specifications define artifacts.
|
||||
|
||||
6. Processors implement specifications.
|
||||
|
||||
7. Capture happens where knowledge occurs.
|
||||
|
||||
8. Curation happens inside Project Thoth.
|
||||
|
||||
9. A source has one physical location, many classifications, and many possible reasoning contexts.
|
||||
|
||||
10. The value of an artifact is determined by expected retrieval behavior.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# MVP Scope
|
||||
|
||||
## Included
|
||||
|
||||
```text
|
||||
ChatGPT Capture Connector
|
||||
Project Thoth Application Scaffold
|
||||
Raw Capture Viewer
|
||||
Normalized Markdown Viewer
|
||||
Vault Save Workflow
|
||||
```
|
||||
|
||||
## Deferred
|
||||
|
||||
```text
|
||||
Gemini Connector
|
||||
Copilot Connector
|
||||
Office Connector
|
||||
Adobe Connector
|
||||
General Web Clipper
|
||||
Source Metadata Automation
|
||||
Manifest Automation
|
||||
Harvest Processors
|
||||
Knowledge Collection Sync
|
||||
Embedded LLM
|
||||
Local LLM Integration
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# End
|
||||
Reference in New Issue
Block a user