Project-Thoth/specifications/source-metadata-specification.md

Project Thoth

Source Metadata Schema Specification

Version 0.1 (Draft)

---

Purpose

This specification defines the canonical metadata schema for all Primary Sources managed within Project Thoth.

Its objectives are to:

• provide consistent metadata across heterogeneous sources
• improve semantic retrieval
• support AI reasoning
• preserve provenance
• enable corpus compilation
• remain independent of any specific software platform

This specification defines the metadata schema.

It does not define harvesting, interpretation, manifests, or canonical knowledge extraction.

---

Design Principles

The metadata should answer only one question:

What is this source?

It should not answer:

• What does this source mean?
• What conclusions should be drawn?
• What knowledge should be extracted?

Those responsibilities belong to downstream Project Thoth processors.

---

Supported Source Types

Examples include:

• conversation
• meeting
• interview
• article
• book
• paper
• transcript
• journal
• email
• web_page
• image
• notes

The schema is intentionally source-independent.

---

Required Fields

title

A concise descriptive title.

Do not use sensational language.

Always use double quotes.

Example

title: "Knowledge Harvesting Methodology"

---

source_type

Type of primary source.

Example

source_type: "conversation"

---

status

Classification of the source.

Allowed values include:

• primary_source
• reference_source
• archival_source

---

Optional Fields

source_platform

Examples

• ChatGPT
• Gemini
• Claude
• Open WebUI
• Teams
• Slack
• Gmail
• YouTube
• Local Markdown

---

conversation_type

One or more values.

Examples

• discussion
• architecture
• brainstorming
• research
• philosophy
• planning
• implementation
• reflection

---

creator

One or more creators or participants.

---

created

Creation date.

Use ISO format when known.

---

language

Language identifier.

Example

language: "en-US"

---

reasoning_contexts

Long-lived reasoning contexts in which this source is useful.

These are reasoning contexts.

They are not folder names.

Examples

• Personal
• Project Thoth
• Fractional Insight
• Withered Sanctum
• Pyramid
• AI Lab

---

primary_topics

Major subjects.

---

secondary_topics

Supporting subjects.

---

entities

Group entities by category.

Typical categories include

• people
• organizations
• software
• technologies
• books
• games
• projects
• methodologies
• places
• other

Only include populated categories.

---

content_types

High-level description of the source.

Examples

• discussion
• architecture
• methodology
• philosophy
• tutorial
• research
• historical_analysis
• technical_analysis
• technical_history
• personal_history
• design

---

keywords

Fifteen to forty concise retrieval keywords.

---

summary

A brief objective description of the source.

Maximum one paragraph.

Avoid interpretation.

---

related_sources

Titles of known related sources.

If none are known

related_sources: []

---

manifest

Expected filename of the corresponding manifest.

---

harvest

Expected filename of the corresponding harvest document.

---

YAML Serialization Rules

The schema shall be serialized using valid YAML.

Strings

All scalar string values shall be enclosed in double quotes.

Correct

title: "Project Thoth"

---

Lists

Lists shall always use block sequence notation.

Correct

primary_topics:
  - "Knowledge Architecture"
  - "Harvesting"

Incorrect

primary_topics: - Knowledge Architecture

Incorrect

primary_topics: [Knowledge Architecture, Harvesting]

---

Nested Objects

Nested structures shall use indentation.

Correct

entities:
  software:
    - "Obsidian"
    - "VS Code"

---

Empty Lists

Represent empty lists explicitly.

related_sources: []

---

Null Values

Omit unknown values.

Do not generate

• null
• Unknown
• N/A

---

Indentation

Use two spaces.

Do not use tabs.

---

Quoting

When uncertain, quote the value.

---

Validation

Generated YAML should parse successfully using a standard YAML parser without modification.

---

Canonical Example

---
title: "Knowledge Harvesting Methodology"

source_type: "conversation"

source_platform: "ChatGPT"

conversation_type:
  - "architecture"
  - "discussion"

creator:
  - "Ken Schaefer"
  - "ChatGPT"

created: "2026-07-03"

language: "en-US"

status: "primary_source"

reasoning_contexts:
  - "Project Thoth"

primary_topics:
  - "Knowledge Harvesting"
  - "Metadata"

secondary_topics:
  - "Prompt Engineering"

entities:
  software:
    - "Obsidian"
    - "VS Code"

content_types:
  - "architecture"
  - "methodology"

keywords:
  - "knowledge"
  - "metadata"
  - "vault"

summary: "Discussion establishing the metadata schema for Project Thoth primary sources."

related_sources: []

manifest: "Knowledge Harvesting Methodology.manifest.md"

harvest: "Knowledge Harvesting Methodology.harvest.md"
---

---

Architectural Principles

1. Metadata identifies the source.
2. Metadata is descriptive rather than interpretive.
3. Metadata should remain relatively stable over time.
4. Metadata supports retrieval rather than analysis.
5. Provenance should be preserved whenever possible.
6. The schema is implementation independent.
7. Different processors may generate the metadata provided they conform to this specification.
8. Improvements in reasoning should modify Manifests and Harvests rather than Source Metadata.

---

Relationship to Other Specifications

This specification defines only Source Metadata.

It should be used together with:

• Conversation Manifest Specification
• Harvest Specification
• Canonical Knowledge Specification
• Knowledge Graph Specification (future)

These specifications collectively define the Project Thoth knowledge processing pipeline.