6.4 KiB
adr, title, status, date, authors
| adr | title | status | date | authors | |
|---|---|---|---|---|---|
| 002 | Canonical Capture Connector Pipeline | Accepted | 2026-07-09 |
|
ADR-002: Canonical Capture Connector Pipeline
Status
Accepted
Context
Project Thoth includes a family of Capture Connectors responsible for preserving conversations and other source material from external systems.
Initial implementation of the ChatGPT connector attempted to identify individual messages while simultaneously converting DOM content into Markdown.
During MVP development, testing revealed several classes of defects:
- Missing conversation content
- Duplicate content
- Fragmented assistant responses
- Incorrect role detection
- Loss of tables, links, and formatting
- Virtualized (non-rendered) conversation sections
- ChatGPT application chrome being captured as conversation content
Investigation showed that these defects were not primarily caused by Markdown generation. They resulted from attempting to perform conversation discovery and content transformation simultaneously.
Modern web applications such as ChatGPT are built using React and other component frameworks that expose deeply nested and frequently changing DOM structures. A single logical conversation turn may consist of dozens of nested DOM elements.
Attempting to infer conversation boundaries while simultaneously rendering Markdown creates unnecessary complexity and makes debugging difficult.
Decision
All Project Thoth Capture Connectors SHALL implement a three-stage pipeline:
Conversation Discovery
│
▼
Conversation Turn Model
│
▼
Content Transformation
│
▼
Markdown Serialization
Each stage has a single responsibility.
Stage 1 — Conversation Discovery
Purpose:
Identify the canonical conversation turns for a source platform.
Responsibilities:
- Locate conversation root
- Locate conversation turn containers
- Determine turn ordering
- Determine speaker role
- Detect unsupported content
- Detect partially rendered or virtualized content
- Produce a platform-neutral intermediate representation
This stage SHALL NOT:
- Generate Markdown
- Normalize formatting
- Generate metadata
- Summarize
- Invoke LLMs
Output:
Conversation
Turn
Turn
Turn
Stage 2 — Content Transformation
Purpose:
Transform a single conversation turn into normalized content.
Responsibilities:
- Convert HTML to Markdown
- Preserve paragraphs
- Preserve headings
- Preserve lists
- Preserve tables
- Preserve links
- Preserve images
- Preserve code blocks
- Preserve inline formatting
This stage SHALL NOT:
- Discover conversation turns
- Infer ordering
- Generate files
Output:
Conversation Turn
│
▼
Markdown
Stage 3 — Markdown Serialization
Purpose:
Produce the canonical Project Thoth conversation.md document.
Responsibilities:
- Write capture metadata
- Preserve conversation order
- Emit User / Assistant boundaries
- Write final Markdown document
This stage SHALL NOT:
- Parse HTML
- Discover DOM elements
- Modify extracted content
Output:
conversation.md
Intermediate Representation
Conversation discovery SHALL produce a platform-neutral model.
Example:
interface Conversation {
sourcePlatform: string;
title: string;
url: string;
capturedAt: Date;
turns: ConversationTurn[];
}
interface ConversationTurn {
turnIndex: number;
role:
| "user"
| "assistant"
| "system"
| "tool"
| "unknown";
captureStatus:
| "rendered"
| "not_rendered"
| "unsupported";
sourceElement: HTMLElement;
markdown?: string;
}
The Intermediate Representation (IR) becomes the contract between discovery and transformation.
Rationale
Separating discovery from rendering provides several advantages.
Separation of Concerns
Each stage performs one responsibility.
Conversation discovery determines what exists.
Content transformation determines how it is represented.
Markdown serialization determines how it is packaged.
Testability
Each stage can be independently tested.
Examples:
- Discovery tests verify turn detection.
- Transformation tests verify Markdown fidelity.
- Serialization tests verify document format.
Failures can be isolated without affecting unrelated stages.
Maintainability
Modern web applications frequently change DOM structure.
When a platform changes, only Conversation Discovery should typically require modification.
Markdown rendering remains reusable across platforms.
Reuse
Most Capture Connectors share identical downstream behavior.
Expected connectors include:
- ChatGPT
- Claude
- Gemini
- Microsoft Copilot
- Open WebUI
- Perplexity
- Future browser-based AI systems
Only Conversation Discovery is expected to be platform-specific.
Debuggability
The IR enables inspection before Markdown generation.
Developers can validate:
- turn count
- ordering
- role detection
- unsupported content
- rendering completeness
without involving Markdown generation.
Consequences
Positive
- Cleaner architecture
- Easier debugging
- Platform independence
- Improved testability
- Reduced coupling
- Higher long-term maintainability
Negative
- Additional abstraction layer
- Slightly more implementation effort
- Requires maintenance of an Intermediate Representation
Alternatives Considered
Single-Pass DOM → Markdown
Rejected.
Although initially simpler, this approach couples conversation discovery with formatting.
Testing demonstrated that defects become difficult to isolate and frequently require heuristic patches.
Platform-Specific End-to-End Connectors
Rejected.
Embedding discovery, rendering, and serialization into a single connector creates duplication across platforms and limits reuse.
Future Considerations
Future Capture Connectors may introduce an optional preprocessing stage before Conversation Discovery.
Examples include:
- Automatic scrolling to render virtualized conversation turns
- Lazy-loading attachments
- Expansion of collapsed content
These preprocessing activities remain outside the canonical pipeline and serve only to improve completeness of the source material.
References
- ADR-001: Project Structure
- Project Thoth MVP — ChatGPT Capture Connector
- Empirical findings from ChatGPT DOM extraction (July 2026)