Skip to content

Download Pipeline

A technical reference for how a raw Project Gutenberg text becomes a structured Markdown file with full author provenance under corpus/.

This document covers download only — the ingest pipeline (DocKG build + KGRAG registration) is described in README.md.

1. End-to-End Flow

                        ┌─────────────────────────────────┐
                        │  Project Gutenberg (network)    │
                        │  ─────────────────────────────  │
                        │   OPDS feed .opds               │
                        │   RDF catalog pg<id>.rdf        │
                        │   Plain text pg<id>.txt         │
                        └────────────────┬────────────────┘
                                         │
                                         ▼
     ┌─────────────────────────────────────────────────────────────────┐
     │ fetch_metadata(id)   ──┐                                        │
     │   ├─ OPDS → title, author, subjects, summary, language, rights  │
     │   └─ RDF  → author_birth, author_death, author_url,             │
     │            author_agent_id                                      │
     └─────────────────────────────────────────────────────────────────┘
                                         │
                                         ▼
     ┌─────────────────────────────────────────────────────────────────┐
     │ download_book(id)                                                │
     │   1. idempotence check  (skip if <slug>.md already exists)       │
     │   2. fetch plain text                                            │
     │   3. strip_boilerplate()    — remove PG header/footer markers    │
     │   4. text_to_markdown()     — detect structure, emit headings    │
     │   5. write_reference()      — metadata sidecar                   │
     └─────────────────────────────────────────────────────────────────┘
                                         │
                                         ▼
     ┌─────────────────────────────────────────────────────────────────┐
     │ corpus/<genre>/<Title>/                                         │
     │   ├── <slug>.md        — full text with Markdown heading tree   │
     │   └── reference.md     — author provenance + Gutenberg metadata │
     └─────────────────────────────────────────────────────────────────┘
                                         │
                 (periodically, after a batch of downloads)
                                         │
                                         ▼
     ┌─────────────────────────────────────────────────────────────────┐
     │ gutenkg authors  (gutenberg_kg.authors.build)                    │
     │   ├─ parse every corpus/*/*/reference.md                        │
     │   ├─ [--refresh] re-fetch RDF for every book and rewrite the     │
     │   │             Author section of its reference.md               │
     │   ├─ write corpus/authors/<slug>/author.md    (one per author)   │
     │   └─ write corpus/authors/index.md            (master table)     │
     └─────────────────────────────────────────────────────────────────┘

2. Entry Points

Command Does
gutenkg download book <id> [--genre G] Single book
gutenkg download catalog <file> [--genre G] Batch from a catalog file
gutenkg download search "<query>" [--author A] [--title T] [--subject S] [--language L] Gutenberg catalog search
gutenkg download fetch-genre <G> Search + confirm + download, whole genre
gutenkg download survey [--genre G] Show what's downloaded/indexed
gutenkg authors [--refresh] [--dry-run] Rebuild author pages

The gutenkg CLI is the authoritative interface. Scripts are thin wrappers that delegate to it:

3. Data Sources

Every book pulls from up to three Gutenberg endpoints:

Endpoint URL pattern Used for
OPDS feed https://www.gutenberg.org/ebooks/{id}.opds Title, author name, subjects, summary, language, rights, published date
RDF catalog https://www.gutenberg.org/cache/epub/{id}/pg{id}.rdf Author birth year, death year, Wikipedia URL, Gutenberg agent ID
Plain text https://www.gutenberg.org/cache/epub/{id}/pg{id}.txt The book itself (UTF-8 with BOM)
Search https://www.gutenberg.org/ebooks/search.opds/?query=… Used only by download search / fetch-genre

All four live in module-level constants at the top of src/gutenberg_kg/gutenberg.py.

4. Downloading One Book

4.1 Metadata fetch — fetch_metadata(ebook_id)

Fetches the OPDS atom entry and extracts:

  • title — <atom:title>
  • authors -- every <atom:author><atom:name>, as a fallback only: the feed lists co-authors in the reverse of the catalog's order, so the RDF creators (4.2) replace them whenever the RDF has any
  • published — <atom:published> (date Gutenberg added the ebook, not original publication)
  • rights — <atom:rights>
  • subjects — all <atom:category term="...">
  • language — <dcterms:language>
  • summary, note — parsed from the XHTML block inside <atom:content>
  • gutenberg_url — synthesized (/ebooks/{id})

Immediately after the OPDS parse, fetch_metadata calls _fetch_rdf_author(id) and merges the result into the same meta dict (silent failure -- a missing RDF doesn't break the download). It then sets author, the byline, from authors with authors.credit: "A", "A and B", or "A, B, and C". A book whose feed and RDF both name no author, such as the King James Bible, is credited to authors.NO_AUTHOR, "Various", rather than left blank.

4.2 RDF enrichment — _fetch_rdf_author(ebook_id)

The RDF catalog exposes the data the OPDS feed omits. The parsing is in rdf_authors(xml_text). The authors are the <dcterms:creator> agents, in the record's order, which puts the primary author first (Dumas before Maquet). Translators and editors sit under relator terms such as marcrel:trl, not dcterms:creator, so they are never authors. Each name goes through authors.display_name, which turns Gutenberg's "Surname, Forenames, Title" headings into reading order: "Tolstoy, Leo, graf" becomes "Leo Tolstoy", and "Wells, H. G. (Herbert George)" becomes "H. G. Wells". Its docstring lists every rule.

The provenance fields describe the first creator:

Field Source
authors Every <dcterms:creator> agent's <pgterms:name>, as display names
author_birth <pgterms:birthdate> (year as string, may be negative for BCE)
author_death <pgterms:deathdate>
author_url The English Wikipedia <pgterms:webpage>, else any Wikipedia one
author_agent_id Integer extracted from rdf:about="2009/agents/N"

In reference.md the first creator is the **Name** line, followed by that person's Born/Died/Wikipedia/Agent ID lines, and every further creator gets a **Co-author** line. authors.parse_reference rebuilds authors and the author byline from those lines. An Internet Archive reference has no Name line; it records the item's creator as **Author** under ## Publication, and parse_reference falls back to that.

BCE years are stored as negative integers (Homer: -750 / -650).

4.3 Idempotence check

Before downloading text, download_book() checks whether corpus/<genre>/<title>/<slug>.md already exists. If yes, it prints [=] already downloaded and returns early. Use --force to override.

Metadata fetch happens before the idempotence check because it provides the title needed to compute the output path.

4.4 Text download

Hits pg{id}.txt with retries + exponential backoff (fetch_url(): 3 attempts, 2×/4× backoff). Always decodes as utf-8-sig to transparently strip the Gutenberg BOM.

4.5 Boilerplate stripping — strip_boilerplate()

Removes everything before *** START OF THE PROJECT GUTENBERG EBOOK … *** and everything after *** END OF THE PROJECT GUTENBERG EBOOK … ***. If markers aren't found the full text is kept (better to have noise than to lose content).

4.6 Structure detection — text_to_markdown()

Two pre-passes before heading detection:

  1. Front-matter skip — drops "Produced by…", "Transcribed by…", "E-text prepared by…" credit blocks if they appear at the top.
  2. Table-of-contents skip — detects a CONTENTS / TABLE OF CONTENTS block in the first ~200 lines and excludes it from the output (redundant once headings are emitted structurally).

Heading detection (_is_heading()) walks the HEADING_PATTERNS list in priority order. Supported patterns:

Pattern Example Level
THE FIRST/SECOND/… BOOK THE FIRST BOOK (Meditations) ##
VOLUME/BOOK/PART + numeral BOOK THE FIRST, VOLUME II, PART III ##
Book the First — Recalled to Life Dickens style ##
ACT + numeral ACT III ##
CHAPTER + numeral CHAPTER XIV., Chapter 3 ##
SCENE + numeral SCENE II ###
LETTER + numeral Letter 4 (epistolary novels) ##
Bible books The First Book of Moses: Called Genesis ##
Old/New Testament divider The Old Testament ##
STAVE + numeral STAVE I (A Christmas Carol) ##
I. A SCANDAL IN BOHEMIA Roman + ALL-CAPS titled ##
Bare roman numeral IV. ###
Standalone ALL-CAPS line THE END (with filters to reject sentences) ###

Multi-line titles are merged: e.g. CHAPTER I. followed by HOW MANY KINDS OF ANIMALS… becomes one ## with the two lines joined. Short Title-Case lines following a heading become italic subtitles.

Illustration markers ([Illustration …]) are stripped throughout.

4.7 Reference generation — write_reference()

Writes corpus/<genre>/<title>/reference.md with sections:

  • Source — Project Gutenberg ID, URL, rights
  • Author — Name, Born, Died, Wikipedia, Gutenberg Agent ID
  • Gutenberg Published — ebook submission date
  • Language
  • Subjects — bulleted list
  • Summary — if OPDS provided one
  • Notes — e.g. "Wikipedia page about this book: …"

All Author fields except Name are emitted conditionally — if the RDF didn't yield a birth year for that agent, no **Born** line appears.

5. Batch Download

Catalog files live in scripts/catalogs/ and use a simple tab-separated format:

# Ancient & Classical
1727    The Odyssey
6130    The Iliad
228     The Aeneid
…
  • Column 1: Gutenberg ebook ID (required)
  • Column 2: Title override (optional)
  • Column 3: Genre override (optional; otherwise taken from --genre)
  • Lines starting with # are comments

fetch-genre is the interactive variant: it searches Gutenberg for the genre keyword, presents the results, prompts per book, downloads the confirmed set, and writes a Markdown report to reports/fetch_genre_<genre>_<timestamp>.md.

6. Output Layout

corpus/
├── <genre>/                         # ancient-classical, shakespeare, …
│   └── <Book Title>/
│       ├── <slug>.md                # full text with Markdown heading tree
│       ├── reference.md             # author provenance + Gutenberg metadata
│       └── .dockg/                  # created later by ingest (gitignored)
│           ├── graph.sqlite            # Graph database
│           └── vectors.sqlite          # Vector index (sqlite-vec)
└── authors/                         # built by `gutenkg authors`
    ├── index.md                     # master alphabetical table
    ├── jane_austen/
    │   └── author.md
    ├── homer/
    │   └── author.md
    └── …

Slug rules — slugify() lowercases, strips non-alphanumerics, collapses runs of whitespace/hyphens into a single _. For both book titles and author names. (Book titles themselves remain cased and spaced — only the filename inside the dir is slugified.)

7. Author Index Build

gutenkg authors (implemented in src/gutenberg_kg/authors.py) is a post-processor. It does not fetch anything on its own unless --refresh is set.

Steps (in order):

  1. Scan — glob corpus/*/*/reference.md
  2. Parse -- parse_reference(path) extracts title, ebook_id, genre, authors, the author byline, author_birth, author_death, author_url and author_agent_id by regex
  3. Refresh (only with --refresh) -- for every book with a Gutenberg ID, call gutenberg._fetch_rdf_author(ebook_id) and rewrite_author_section() to replace the reference.md author lines with the RDF's. A failed fetch, or an RDF with no creator, leaves the file untouched
  4. Group -- each person in authors gets the book, so a co-written book is listed under every author. Born/Died/Wikipedia describe the first author only, so a co-author's entry carries none of them
  5. Write per-author pages -- corpus/authors/<slug>/author.md with # Name, era line (born - died), Wikipedia, agent ID, and a "Works in Corpus" table
  6. Remove stale pages -- a page directory whose author is no longer in the corpus, and which holds nothing but a generated author.md, is removed
  7. Write index -- corpus/authors/index.md with a master table

Rate-limiting: --refresh sleeps 0.3 s between RDF fetches to be polite.

8. Refresh Workflow

When you add a book manually, the new download_book() call already fetches the RDF, so new books land with full provenance in reference.md.

For any book whose reference.md predates the RDF fetch, or after a change to how authors are parsed, run:

gutenkg authors --refresh --dry-run   # list what would change
gutenkg authors --refresh

This: 1. Re-fetches the RDF for every book with a Gutenberg ID 2. Rewrites the author lines of each reference.md whose authors or provenance differ, and leaves every other line of the file as it was 3. Regenerates corpus/authors/ and removes pages for authors no longer in the corpus

Safe to run repeatedly: a second run rewrites nothing. The book text files are never touched, since the index's character offsets depend on them.

A bundle carries authors in its catalog.json, which gutenkg build-corpus writes at the end of a full build. To refresh only the catalog, for the same books, without rebuilding the index:

poetry run python -c '
import json; from pathlib import Path
from gutenberg_kg.build_corpus import build_catalog
out = Path("bundles/gutenberg-all/.dockg")
keys = frozenset(json.loads((out / "catalog.json").read_text()))
build_catalog(sorted({k.split("/")[0] for k in keys}), out, catalog_keys=keys)'
make export-swift

export-swift reads authors from that catalog, so the app's packs pick up the new names without re-embedding anything.

9. Failure Modes

Failure Behavior
OPDS 404 / network error fetch_metadata returns {"ebook_id": id} — title falls back to Book_<id>
RDF 404 / network error _fetch_rdf_author returns {} — book still downloads, just without Born/Died/Wikipedia
Plain-text 404 fetch_url raises after 3 retries; the download CLI catches and reports an error per book
START/END markers missing Boilerplate not stripped — full text (including Gutenberg header) is kept
TOC not detected No skip — TOC ends up in the body. Cheap to ignore; the heading detector still works downstream
Author field missing from RDF Book downloads normally; author_birth/author_death/author_url simply aren't in the reference. Happens for e.g. The Bible (ID 10)
Idempotence collision [=] already downloaded — skipped unless --force
Bad title characters for filesystem slugify() strips them; the book dir name itself uses the raw title

10. Adding a Single New Book

# 1. Find it
gutenkg download search --author "Herman Melville"

# 2. Download into the right genre
gutenkg download book 2701 --genre american-literature

# 3. (optional) refresh the author index if you care about the author page
gutenkg authors

The book dir corpus/american-literature/Moby Dick/ appears immediately with a fully-populated reference.md — no separate refresh needed for new downloads.

11. Genre Registry

Genres live in corpus/genres.json — the single source of truth. The file is split into "gutenberg" and "ia" source lists. src/gutenberg_kg/genres.py loads it at import time and exports GUTENBERG_GENRES, IA_GENRES, and ALL_GENRES.

To add a genre without touching any Python:

gutenkg genres init                              # one-time seed (creates genres.json)
gutenkg genres add medieval-literature --source gutenberg
gutenkg genres list                              # verify

genres add creates genres.json automatically if it doesn't exist.