464 lines
5.9 KiB
Markdown
464 lines
5.9 KiB
Markdown
# Project Thoth
|
|
|
|
# Source Metadata Schema Specification
|
|
|
|
## Version 0.1 (Draft)
|
|
|
|
---
|
|
|
|
# Purpose
|
|
|
|
This specification defines the canonical metadata schema for all Primary Sources managed within Project Thoth.
|
|
|
|
Its objectives are to:
|
|
|
|
* provide consistent metadata across heterogeneous sources
|
|
* improve semantic retrieval
|
|
* support AI reasoning
|
|
* preserve provenance
|
|
* enable corpus compilation
|
|
* remain independent of any specific software platform
|
|
|
|
This specification defines the metadata schema.
|
|
|
|
It does **not** define harvesting, interpretation, manifests, or canonical knowledge extraction.
|
|
|
|
---
|
|
|
|
# Design Principles
|
|
|
|
The metadata should answer only one question:
|
|
|
|
> **What is this source?**
|
|
|
|
It should not answer:
|
|
|
|
* What does this source mean?
|
|
* What conclusions should be drawn?
|
|
* What knowledge should be extracted?
|
|
|
|
Those responsibilities belong to downstream Project Thoth processors.
|
|
|
|
---
|
|
|
|
# Supported Source Types
|
|
|
|
Examples include:
|
|
|
|
* conversation
|
|
* meeting
|
|
* interview
|
|
* article
|
|
* book
|
|
* paper
|
|
* transcript
|
|
* journal
|
|
* email
|
|
* web_page
|
|
* image
|
|
* notes
|
|
|
|
The schema is intentionally source-independent.
|
|
|
|
---
|
|
|
|
# Required Fields
|
|
|
|
## title
|
|
|
|
A concise descriptive title.
|
|
|
|
Do not use sensational language.
|
|
|
|
Always use double quotes.
|
|
|
|
Example
|
|
|
|
```yaml
|
|
title: "Knowledge Harvesting Methodology"
|
|
```
|
|
|
|
---
|
|
|
|
## source_type
|
|
|
|
Type of primary source.
|
|
|
|
Example
|
|
|
|
```yaml
|
|
source_type: "conversation"
|
|
```
|
|
|
|
---
|
|
|
|
## status
|
|
|
|
Classification of the source.
|
|
|
|
Allowed values include:
|
|
|
|
* primary_source
|
|
* reference_source
|
|
* archival_source
|
|
|
|
---
|
|
|
|
# Optional Fields
|
|
|
|
## source_platform
|
|
|
|
Examples
|
|
|
|
* ChatGPT
|
|
* Gemini
|
|
* Claude
|
|
* Open WebUI
|
|
* Teams
|
|
* Slack
|
|
* Gmail
|
|
* YouTube
|
|
* Local Markdown
|
|
|
|
---
|
|
|
|
## conversation_type
|
|
|
|
One or more values.
|
|
|
|
Examples
|
|
|
|
* discussion
|
|
* architecture
|
|
* brainstorming
|
|
* research
|
|
* philosophy
|
|
* planning
|
|
* implementation
|
|
* reflection
|
|
|
|
---
|
|
|
|
## creator
|
|
|
|
One or more creators or participants.
|
|
|
|
---
|
|
|
|
## created
|
|
|
|
Creation date.
|
|
|
|
Use ISO format when known.
|
|
|
|
---
|
|
|
|
## language
|
|
|
|
Language identifier.
|
|
|
|
Example
|
|
|
|
```yaml
|
|
language: "en-US"
|
|
```
|
|
|
|
---
|
|
|
|
## reasoning_contexts
|
|
|
|
Long-lived reasoning contexts in which this source is useful.
|
|
|
|
These are reasoning contexts.
|
|
|
|
They are **not** folder names.
|
|
|
|
Examples
|
|
|
|
* Personal
|
|
* Project Thoth
|
|
* Fractional Insight
|
|
* Withered Sanctum
|
|
* Pyramid
|
|
* AI Lab
|
|
|
|
---
|
|
|
|
## primary_topics
|
|
|
|
Major subjects.
|
|
|
|
---
|
|
|
|
## secondary_topics
|
|
|
|
Supporting subjects.
|
|
|
|
---
|
|
|
|
## entities
|
|
|
|
Group entities by category.
|
|
|
|
Typical categories include
|
|
|
|
* people
|
|
* organizations
|
|
* software
|
|
* technologies
|
|
* books
|
|
* games
|
|
* projects
|
|
* methodologies
|
|
* places
|
|
* other
|
|
|
|
Only include populated categories.
|
|
|
|
---
|
|
|
|
## content_types
|
|
|
|
High-level description of the source.
|
|
|
|
Examples
|
|
|
|
* discussion
|
|
* architecture
|
|
* methodology
|
|
* philosophy
|
|
* tutorial
|
|
* research
|
|
* historical_analysis
|
|
* technical_analysis
|
|
* technical_history
|
|
* personal_history
|
|
* design
|
|
|
|
---
|
|
|
|
## keywords
|
|
|
|
Fifteen to forty concise retrieval keywords.
|
|
|
|
---
|
|
|
|
## summary
|
|
|
|
A brief objective description of the source.
|
|
|
|
Maximum one paragraph.
|
|
|
|
Avoid interpretation.
|
|
|
|
---
|
|
|
|
## related_sources
|
|
|
|
Titles of known related sources.
|
|
|
|
If none are known
|
|
|
|
```yaml
|
|
related_sources: []
|
|
```
|
|
|
|
---
|
|
|
|
## manifest
|
|
|
|
Expected filename of the corresponding manifest.
|
|
|
|
---
|
|
|
|
## harvest
|
|
|
|
Expected filename of the corresponding harvest document.
|
|
|
|
---
|
|
|
|
# YAML Serialization Rules
|
|
|
|
The schema shall be serialized using valid YAML.
|
|
|
|
## Strings
|
|
|
|
All scalar string values shall be enclosed in double quotes.
|
|
|
|
Correct
|
|
|
|
```yaml
|
|
title: "Project Thoth"
|
|
```
|
|
|
|
---
|
|
|
|
## Lists
|
|
|
|
Lists shall always use block sequence notation.
|
|
|
|
Correct
|
|
|
|
```yaml
|
|
primary_topics:
|
|
- "Knowledge Architecture"
|
|
- "Harvesting"
|
|
```
|
|
|
|
Incorrect
|
|
|
|
```yaml
|
|
primary_topics: - Knowledge Architecture
|
|
```
|
|
|
|
Incorrect
|
|
|
|
```yaml
|
|
primary_topics: [Knowledge Architecture, Harvesting]
|
|
```
|
|
|
|
---
|
|
|
|
## Nested Objects
|
|
|
|
Nested structures shall use indentation.
|
|
|
|
Correct
|
|
|
|
```yaml
|
|
entities:
|
|
software:
|
|
- "Obsidian"
|
|
- "VS Code"
|
|
```
|
|
|
|
---
|
|
|
|
## Empty Lists
|
|
|
|
Represent empty lists explicitly.
|
|
|
|
```yaml
|
|
related_sources: []
|
|
```
|
|
|
|
---
|
|
|
|
## Null Values
|
|
|
|
Omit unknown values.
|
|
|
|
Do not generate
|
|
|
|
* null
|
|
* Unknown
|
|
* N/A
|
|
|
|
---
|
|
|
|
## Indentation
|
|
|
|
Use two spaces.
|
|
|
|
Do not use tabs.
|
|
|
|
---
|
|
|
|
## Quoting
|
|
|
|
When uncertain, quote the value.
|
|
|
|
---
|
|
|
|
## Validation
|
|
|
|
Generated YAML should parse successfully using a standard YAML parser without modification.
|
|
|
|
---
|
|
|
|
# Canonical Example
|
|
|
|
```yaml
|
|
---
|
|
title: "Knowledge Harvesting Methodology"
|
|
|
|
source_type: "conversation"
|
|
|
|
source_platform: "ChatGPT"
|
|
|
|
conversation_type:
|
|
- "architecture"
|
|
- "discussion"
|
|
|
|
creator:
|
|
- "Ken Schaefer"
|
|
- "ChatGPT"
|
|
|
|
created: "2026-07-03"
|
|
|
|
language: "en-US"
|
|
|
|
status: "primary_source"
|
|
|
|
reasoning_contexts:
|
|
- "Project Thoth"
|
|
|
|
primary_topics:
|
|
- "Knowledge Harvesting"
|
|
- "Metadata"
|
|
|
|
secondary_topics:
|
|
- "Prompt Engineering"
|
|
|
|
entities:
|
|
software:
|
|
- "Obsidian"
|
|
- "VS Code"
|
|
|
|
content_types:
|
|
- "architecture"
|
|
- "methodology"
|
|
|
|
keywords:
|
|
- "knowledge"
|
|
- "metadata"
|
|
- "vault"
|
|
|
|
summary: "Discussion establishing the metadata schema for Project Thoth primary sources."
|
|
|
|
related_sources: []
|
|
|
|
manifest: "Knowledge Harvesting Methodology.manifest.md"
|
|
|
|
harvest: "Knowledge Harvesting Methodology.harvest.md"
|
|
---
|
|
```
|
|
|
|
---
|
|
|
|
# Architectural Principles
|
|
|
|
1. Metadata identifies the source.
|
|
2. Metadata is descriptive rather than interpretive.
|
|
3. Metadata should remain relatively stable over time.
|
|
4. Metadata supports retrieval rather than analysis.
|
|
5. Provenance should be preserved whenever possible.
|
|
6. The schema is implementation independent.
|
|
7. Different processors may generate the metadata provided they conform to this specification.
|
|
8. Improvements in reasoning should modify Manifests and Harvests rather than Source Metadata.
|
|
|
|
---
|
|
|
|
# Relationship to Other Specifications
|
|
|
|
This specification defines only Source Metadata.
|
|
|
|
It should be used together with:
|
|
|
|
* Conversation Manifest Specification
|
|
* Harvest Specification
|
|
* Canonical Knowledge Specification
|
|
* Knowledge Graph Specification (future)
|
|
|
|
These specifications collectively define the Project Thoth knowledge processing pipeline.
|