# 1.8 million words, freed from Word 97 and made searchable We’ll explore how to **leverage a corpus of 1,800 articles (~1.8 million words, comparable in volume to *A Song of Ice and Fire*)** to build an **AI-driven discovery platform** that makes reading, exploration, and print publication smarter and more meaningful.
1,800
Oral lectures
~1.8M
Total words
37
Legacy Word files
The central challenge is not merely searching text, but transforming a monumental chronological archive into a living knowledge base: ## Objective: building an AI-enabled knowledge platform from 1,800 articles The goal is to transform a vast, heterogeneous collection of articles into a **structured, AI-enabled knowledge platform**.
Organize the heterogeneous archive into 1,800 clean, version-controlled Markdown files with standardized YAML frontmatter. Enrich each lecture with thematic tags, Gaussian-normalized relevance scores, full-text indexation, and vector embeddings. Enable conversational exploration, terminology clarification, contextual reading recommendations, and automated print curation.
![Labels and index tabs organizing physical documents](https://redaction-technique.org/images/blog/transforming-meditation-classes-ai-discovery-large.webp)
Structuring a vast corpus: metadata, tags, and classification transform static text files into an organized knowledge system.
> **The editorial principle:** Make historical articles discoverable and approachable without losing the depth, subtlety, or authentic voice of the original teachings. The end-to-end system architecture spans five connected layers: --- ## Analysis and cleaning Before AI can meaningfully process the corpus, the data must be **clean and consistent**: - **Deduplication:** Identify repeated articles and remove redundant content across files. - **Formatting normalization:** Standardize headers, timestamps, and text encoding. - **Quality checks:** Flag incomplete or corrupted articles for human review. - **Consistency enforcement:** Align style, terminology, and structural headings for machine readability. > **Infrastructure before AI:** AI does not magically fix poor data quality. Cleaning and normalization form the indispensable foundation for reliable metadata, dependable indexing, and trustworthy AI interaction. --- ## Metadata and tagging Metadata transforms a large corpus into a **discoverable and analyzable knowledge base**:
Generate titles, meta descriptions, and keyword sets optimized for search engines and web-based exploration. Automatically assign pedagogical themes: meditation, cause and effect, lineage, training, and daily life. Assign interest scores and thematic relevance indicators based on user priorities and contextual intensity. Insert cross-references, summaries, and definitions for technique-related terminology to improve searchability.
AI assists here by reading each article and suggesting thematic tags or highlights while maintaining consistency across the entire collection. --- ## Indexing and search A robust indexing system enables **fast, accurate retrieval**, moving progressively from exact matches to conceptual connections: - **Full-text search:** Index content to allow keyword-based queries across 1,800 texts. - **Thematic navigation:** Enable filtering by topics, scores, or temporal context. - **Semantic search:** Use embeddings or vector search to capture meaning beyond exact keywords, letting users find related concepts even if phrased differently. - **Cross-linking:** Automatically connect related articles to create a **web of knowledge** that mirrors article interconnections. This ensures that users can **explore the corpus efficiently**, whether seeking a specific lesson or discovering related concepts. --- ## Interactive AI layer The AI layer transforms static content into an **intelligent, interactive discovery platform**:
Suggest articles or themes based on user interests or previous reading paths. Summarize complex teachings or provide background on lineage and philosophical terminology. Allow users to ask questions, clarify concepts, or explore nuances dynamically. Generate summaries, highlight patterns, and identify frequently referenced masters or concepts.
> **Preservation of integrity:** The original lecture transcripts remain intact and untouched. AI-generated metadata, embeddings, and summaries serve strictly as an enrichment layer to guide readers back to the source text. --- ## Deploying the AI discovery platform for web and API access Finally, the platform is designed as an extensible content platform rather than a collection of one-off scripts: - **Web interface:** User-friendly dashboards for browsing, filtering, and exploring articles. - **API access:** Allow other applications or AI systems to query the corpus programmatically. - **Scalability:** Ensure smooth operation even as new articles or supplementary materials are added. - **Preservation of integrity:** Maintain original text alongside enriched metadata to honor the authenticity of the articles. --- ## Automated thematic scoring of Markdown files with GPT We can use a Python script to automate the evaluation and enrichment of Markdown files by assigning a **semantic score** related to any theme, for instance, *daily life*. The scoring process operates in three explicit stages: ### 1. AI-based Evaluation Each `.md` file is read and sent to an OpenAI model. The model assigns a raw score from **1 to 10**, estimating how strongly the text relates to the target theme.
![Number of articles for each combined score graph showing a bell curve distribution](https://redaction-technique.org/images/blog/compound.webp)
Distribution of articles across combined scores, normalized as a Gaussian curve to spread values evenly across the 1–10 range.
### 2. Gaussian Normalization To ensure a smooth and realistic distribution of results, the raw scores are rebalanced using a **Gaussian (normal) curve**, spreading the final values evenly across the 1–10 range. This avoids clusters of identical scores and makes large-scale data more insightful. > **Normalization vs. underlying truth:** There's no law that thematic relevance is normally distributed: for a corpus built around core themes, it may genuinely be bimodal or heavily skewed. Forcing raw scores onto a bell curve spreads them out for browsing, but it manufactures gradations that may not exist and flattens real clusters. Normalize for presentation, but keep raw scores too: the raw distribution reveals the true density of the corpus. ### 3. Frontmatter Update The script then writes the final score back into each file’s **YAML frontmatter** as a new or updated field. **Example: AI-generated metadata record** The metadata for this article was generated from an AI-assisted analysis of hundreds of articles. By identifying recurring themes of awareness, balance, and mindful attention, the AI distilled key insights into how stillness can nurture clarity and calm in everyday life: ```yaml title: The Power of Stillness – Deepening Awareness Through Mindful Sitting sourceLanguage: en description: Explore how mindful sitting cultivates presence, clarity, and balance in everyday life through sustained attention and inner calm. lastUpdated: 2009-10-31T11:00:00Z wordCount: 1586 keywords: - Mindfulness - Meditation - Awareness - Focus - Presence - Inner Calm - Clarity - Balance - Attention - Well-being interest_score: 10 daily_life: 8 mindful_posture: 8 intentional_action: 7 continuity_of_practice: 10 ``` > **From text file to structured record:** Each article remains readable Markdown, but now carries machine-readable keywords, timestamps, word counts, and multi-thematic scores that power programmatic curation. --- ## From Markdown to LaTeX: automating the creation of various anthologies > **The curation problem:** No one could realistically read 1.8 million words of articles in print: even a skilled reader would need about 120 hours—roughly two full weeks of non-stop reading—to get through them all. Since the full corpus is available online, the real challenge is curation: selecting and assembling themed anthologies tailored to reader interests. A Python script automates the **conversion and assembly of Markdown articles into a single, beautifully formatted LaTeX book**, ready to compile with *XeLaTeX* or *LuaLaTeX* on Overleaf. It performs the entire workflow in three stages: 1. **Markdown Cleaning and Conversion** Given a Markdown file, the script extracts its title from the YAML frontmatter, removes HTML markup and Markdown links, and converts the text into **LaTeX** using *Pandoc*. Each `.md` file becomes a clean, self-contained `.tex` chapter with a proper `\chapter{Title}` heading.