Gutenberg source¶

`gutenberg_kg.gutenberg` ¶

Download books from Project Gutenberg as structured Markdown.

Public API (used by the Click CLI in gutenberg_kg.cli.cmd_download): download_book() — fetch one book by Gutenberg ID run_search() — search the Gutenberg OPDS catalog run_catalog() — download all books from a catalog file run_fetch_genre() — interactive genre fetch (search → confirm → download) run_survey() — scan corpus and print download/ingest status

Entry point: gutenkg download [options]

Output structure (per book): corpus/// <slug>.md — structured Markdown with chapter/section headings reference.md — Gutenberg metadata, subjects, and source info <div class="doc doc-children"> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.download_book" class="doc doc-heading"> <code class="highlight language-python">download_book(ebook_id, title=None, genre=None, force=False, dry_run=False)</code> <a href="#gutenberg_kg.gutenberg.download_book" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Download a book, convert to Markdown, and save with reference. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>ebook_id</code> </td> <td> <code>int</code> </td> <td> <div class="doc-md-description"> Project Gutenberg numeric book ID. </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>title</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Optional title override (auto-detected from metadata if omitted). </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>genre</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Optional genre subdirectory name (one of ALL_GENRES). When provided the book is written to <code>REPO_ROOT/<genre>/<Title>/</code>; when omitted, it is written to <code>REPO_ROOT/<Title>/</code> as before. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>force</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Re-download even when the Markdown file already exists. </div> </td> <td> <code>False</code> </td> </tr> <tr class="doc-section-item"> <td> <code>dry_run</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Print what would be done without writing any files. </div> </td> <td> <code>False</code> </td> </tr> </tbody> </table> Returns: <table> <thead> <tr> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>str</code> </td> <td> <div class="doc-md-description"> Absolute path to the primary Markdown file. </div> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739</pre></div></td><td class="code"><div><pre><code>def download_book( ebook_id: int, title: str | None = None, genre: str | None = None, force: bool = False, dry_run: bool = False, ) -> str: """Download a book, convert to Markdown, and save with reference. :param ebook_id: Project Gutenberg numeric book ID. :param title: Optional title override (auto-detected from metadata if omitted). :param genre: Optional genre subdirectory name (one of ALL_GENRES). When provided the book is written to ``REPO_ROOT/<genre>/<Title>/``; when omitted, it is written to ``REPO_ROOT/<Title>/`` as before. :param force: Re-download even when the Markdown file already exists. :param dry_run: Print what would be done without writing any files. :returns: Absolute path to the primary Markdown file. """ # ID-based idempotence check: a book already present under ANY directory # name (e.g. downloaded under its OPDS title before a catalog override was # curated) must not be fetched again. Runs before the metadata fetch so a # skip costs no network round-trip. if not force: search_root = os.path.join(CORPUS_ROOT, genre) if genre else CORPUS_ROOT existing = _find_book_by_id(ebook_id, search_root) if existing: existing_dir = os.path.basename(os.path.dirname(existing)) print(f" [=] already downloaded (Gutenberg #{ebook_id}): {existing_dir} — skipping") # Catalogue on the skip path too: a book already on disk but absent # from the catalog is exactly the pre-existing drift, and re-running # the download is the natural way to repair it. record_in_catalog(genre, ebook_id, existing_dir, dry_run=dry_run) return existing # Fetch metadata (needed for the title before we can check idempotence) print(f" Fetching metadata for Gutenberg #{ebook_id}...") meta = fetch_metadata(ebook_id) title = title or meta.get("title") or f"Book_{ebook_id}" meta["title"] = title # Resolve output directory if genre: book_dir = os.path.join(CORPUS_ROOT, genre, title) else: book_dir = os.path.join(CORPUS_ROOT, title) slug = slugify(title) out_path = os.path.join(book_dir, f"{slug}.md") # Idempotence check if not force and os.path.exists(out_path): print(f" [=] already downloaded: {title} — skipping") record_in_catalog(genre, ebook_id, title, dry_run=dry_run) return out_path if dry_run: dest = os.path.join(genre, title) if genre else title print(f" [dry] would download: {title} (Gutenberg #{ebook_id}) → {dest}/") return out_path os.makedirs(book_dir, exist_ok=True) # Download plain text print(f" Downloading: {title} (Gutenberg #{ebook_id})") url = GUTENBERG_TXT_URL.format(ebook_id=ebook_id) raw_text = fetch_url(url) # Strip boilerplate clean_text = strip_boilerplate(raw_text) # Convert to structured Markdown markdown = text_to_markdown(clean_text, meta) with open(out_path, "w", encoding="utf-8") as f: f.write(markdown) size_kb = os.path.getsize(out_path) / 1024 print(f" Saved: {out_path} ({size_kb:.1f} KB)") # Write reference file ref_path = write_reference(book_dir, meta) print(f" Saved: {ref_path}") record_in_catalog(genre, ebook_id, title, dry_run=dry_run) return out_path </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.fetch_metadata" class="doc doc-heading"> <code class="highlight language-python">fetch_metadata(ebook_id)</code> <a href="#gutenberg_kg.gutenberg.fetch_metadata" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Fetch book metadata from the Gutenberg OPDS feed. <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206</pre></div></td><td class="code"><div><pre><code>def fetch_metadata(ebook_id: int) -> dict: """Fetch book metadata from the Gutenberg OPDS feed.""" url = GUTENBERG_OPDS_URL.format(ebook_id=ebook_id) xml_text = fetch_url(url) root = ET.fromstring(xml_text) entry = root.find("atom:entry", NS) if entry is None: return {"ebook_id": ebook_id} meta: dict[str, object] = {"ebook_id": ebook_id} # Title title_el = entry.find("atom:title", NS) if title_el is not None and title_el.text: meta["title"] = title_el.text.strip() # Authors: only a fallback. The feed lists co-authors in the reverse of # the catalog's order, which is how Auguste Maquet came to be credited # with The Count of Monte Cristo; the RDF creators below replace these. meta["authors"] = [ display_name(el.text) for el in entry.findall("atom:author/atom:name", NS) if el.text and el.text.strip() ] # Published date pub_el = entry.find("atom:published", NS) if pub_el is not None and pub_el.text: meta["published"] = pub_el.text.strip()[:10] # Rights rights_el = entry.find("atom:rights", NS) if rights_el is not None and rights_el.text: meta["rights"] = rights_el.text.strip() # Subjects from <category> elements subjects = [] for cat in entry.findall("atom:category", NS): term = cat.get("term", "") if term: subjects.append(term) if subjects: meta["subjects"] = subjects # Language lang_el = entry.find("dcterms:language", NS) if lang_el is not None and lang_el.text: meta["language"] = lang_el.text.strip() # Summary and other details from content block content_el = entry.find("atom:content", NS) if content_el is not None: # The content is XHTML — extract text from elements div = content_el.find("xhtml:div", NS) if div is not None: for p in div.findall("xhtml:p", NS): text = "".join(p.itertext()).strip() if text.startswith("Summary:"): meta["summary"] = text[len("Summary:") :].strip() elif text.startswith("Note:"): meta["note"] = text[len("Note:") :].strip() meta["gutenberg_url"] = GUTENBERG_PAGE_URL.format(ebook_id=ebook_id) # The RDF catalog's creators and the first creator's provenance # (birth/death/Wikipedia) replace the feed's authors when it has any. # A record with no creator at all (scripture, mostly) is credited to # NO_AUTHOR rather than left blank. meta.update(_fetch_rdf_author(ebook_id)) if not meta["authors"]: meta["authors"] = [NO_AUTHOR] meta["author"] = credit(meta["authors"]) return meta </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.fetch_url" class="doc doc-heading"> <code class="highlight language-python">fetch_url(url, retries=3, backoff=2.0)</code> <a href="#gutenberg_kg.gutenberg.fetch_url" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Fetch a URL with retries and exponential backoff. <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>111 112 113 114 115 116 117 118 119 120 121 122 123 124</pre></div></td><td class="code"><div><pre><code>def fetch_url(url: str, retries: int = 3, backoff: float = 2.0) -> str: """Fetch a URL with retries and exponential backoff.""" for attempt in range(retries): try: req = urllib.request.Request(url, headers={"User-Agent": "GreatBooksProject/1.0"}) with urllib.request.urlopen(req, timeout=30) as resp: return resp.read().decode("utf-8-sig") except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError) as exc: if attempt == retries - 1: raise wait = backoff * (2**attempt) print(f" Retry {attempt + 1}/{retries} after error: {exc} (waiting {wait:.0f}s)") time.sleep(wait) return "" </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.parse_catalog" class="doc doc-heading"> <code class="highlight language-python">parse_catalog(catalog_path)</code> <a href="#gutenberg_kg.gutenberg.parse_catalog" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Parse a catalog file. Each line: <ebook_id>[\t<title>] <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857</pre></div></td><td class="code"><div><pre><code>def parse_catalog(catalog_path: str) -> list[tuple[int, str | None]]: """Parse a catalog file. Each line: <ebook_id>[\\t<title>]""" entries = [] with open(catalog_path, encoding="utf-8") as f: for line_num, line in enumerate(f, 1): line = line.strip() if not line or line.startswith("#"): continue parts = line.split("\t") try: ebook_id = int(parts[0].strip()) except ValueError: print(f" Warning: skipping non-numeric ID on line {line_num}: {parts[0]!r}") continue title = parts[1].strip() if len(parts) >= 2 else None entries.append((ebook_id, title)) return entries </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.rdf_authors" class="doc doc-heading"> <code class="highlight language-python">rdf_authors(xml_text)</code> <a href="#gutenberg_kg.gutenberg.rdf_authors" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Read a book's authors from its Gutenberg RDF record. The authors are the <code>dcterms:creator</code> agents, in the record's order, which puts the primary author first: Dumas before Maquet, Polo before Rusticiano. Translators and editors are recorded under relator terms, not as creators, so they are never counted as authors. Provenance describes the first creator only. It used to be read from the first agent anywhere in the record, which could be a translator. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>xml_text</code> </td> <td> <code>str</code> </td> <td> <div class="doc-md-description"> The RDF document. </div> </td> <td> required </td> </tr> </tbody> </table> Returns: <table> <thead> <tr> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>dict</code> </td> <td> <div class="doc-md-description"> <code>authors</code> (display names) plus whichever of <code>author_birth</code>, <code>author_death</code>, <code>author_url</code> and <code>author_agent_id</code> the first creator has. Empty when the record has no creator or does not parse. </div> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271</pre></div></td><td class="code"><div><pre><code>def rdf_authors(xml_text: str) -> dict: """Read a book's authors from its Gutenberg RDF record. The authors are the ``dcterms:creator`` agents, in the record's order, which puts the primary author first: Dumas before Maquet, Polo before Rusticiano. Translators and editors are recorded under relator terms, not as creators, so they are never counted as authors. Provenance describes the first creator only. It used to be read from the first agent anywhere in the record, which could be a translator. :param xml_text: The RDF document. :return: ``authors`` (display names) plus whichever of ``author_birth``, ``author_death``, ``author_url`` and ``author_agent_id`` the first creator has. Empty when the record has no creator or does not parse. """ try: root = ET.fromstring(xml_text) except ET.ParseError: return {} agents = [ agent for creator in root.iter(f"{{{RDF_NS['dcterms']}}}creator") if (agent := creator.find("pgterms:agent", RDF_NS)) is not None ] names = [ display_name(name.text) for agent in agents if (name := agent.find("pgterms:name", RDF_NS)) is not None and name.text and name.text.strip() ] if not names: return {} result: dict[str, object] = {"authors": names} first = agents[0] m = re.search(r"/agents/(\d+)$", first.get(_RDF_ABOUT, "")) if m: result["author_agent_id"] = int(m.group(1)) birth = first.find("pgterms:birthdate", RDF_NS) if birth is not None and birth.text: result["author_birth"] = birth.text.strip() death = first.find("pgterms:deathdate", RDF_NS) if death is not None and death.text: result["author_death"] = death.text.strip() # English Wikipedia when the record has it, else whichever language it # lists; the record's own order is not stable enough to choose by. wikis = [ href for webpage in first.findall("pgterms:webpage", RDF_NS) if "wikipedia.org" in (href := webpage.get(_RDF_RESOURCE, "")) ] english = [href for href in wikis if "//en.wikipedia.org" in href] if english or wikis: result["author_url"] = (english or wikis)[0] return result </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.record_in_catalog" class="doc doc-heading"> <code class="highlight language-python">record_in_catalog(genre, ebook_id, title, dry_run=False)</code> <a href="#gutenberg_kg.gutenberg.record_in_catalog" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Record a downloaded book in its genre catalog, if not already there. The catalog files under <code>scripts/catalogs/</code> were originally batch input manifests: <code>download catalog</code> read them and nothing ever wrote them. That left the three ad-hoc paths -- <code>download book</code>, <code>download fetch-genre</code> and <code>ia download</code> -- adding books to the corpus that no catalog recorded, permanently and silently. The drift is invisible to <code>gutenkg audit</code>, which only checks the catalog-to-corpus direction, and it means a fresh clone replayed through <code>download catalog</code> reproduces a strict subset of the corpus. Calling this on every successful download closes that loop. <code>title</code> is written as the override column and must be the book's directory name, since :func:<code>gutenberg_kg.audit.audit_corpus</code> reports a catalog title that differs from the directory as an error. Appending is idempotent on the ID, so re-running a download repairs a missing catalog line rather than duplicating an existing one. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>genre</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Genre whose catalog to write; <code>None</code> means the book was downloaded to the corpus root, which no catalog covers, so nothing is written. </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>ebook_id</code> </td> <td> <code>int</code> </td> <td> <div class="doc-md-description"> Project Gutenberg numeric book ID. </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>title</code> </td> <td> <code>str</code> </td> <td> <div class="doc-md-description"> The book's directory name, written as the title override. </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>dry_run</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> When true, report what would be written but write nothing. </div> </td> <td> <code>False</code> </td> </tr> </tbody> </table> Returns: <table> <thead> <tr> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> True when a line was appended, or would have been under <code>dry_run</code> -- so a caller can count pending repairs without writing. </div> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796</pre></div></td><td class="code"><div><pre><code>def record_in_catalog( genre: str | None, ebook_id: int, title: str, dry_run: bool = False, ) -> bool: """Record a downloaded book in its genre catalog, if not already there. The catalog files under ``scripts/catalogs/`` were originally batch *input* manifests: ``download catalog`` read them and nothing ever wrote them. That left the three ad-hoc paths -- ``download book``, ``download fetch-genre`` and ``ia download`` -- adding books to the corpus that no catalog recorded, permanently and silently. The drift is invisible to ``gutenkg audit``, which only checks the catalog-to-corpus direction, and it means a fresh clone replayed through ``download catalog`` reproduces a strict subset of the corpus. Calling this on every successful download closes that loop. ``title`` is written as the override column and must be the book's directory name, since :func:`gutenberg_kg.audit.audit_corpus` reports a catalog title that differs from the directory as an error. Appending is idempotent on the ID, so re-running a download repairs a missing catalog line rather than duplicating an existing one. :param genre: Genre whose catalog to write; ``None`` means the book was downloaded to the corpus root, which no catalog covers, so nothing is written. :param ebook_id: Project Gutenberg numeric book ID. :param title: The book's directory name, written as the title override. :param dry_run: When true, report what would be written but write nothing. :returns: True when a line was appended, or would have been under ``dry_run`` -- so a caller can count pending repairs without writing. """ if genre is None: return False catalog_path = os.path.join(CATALOG_ROOT, f"{genre}.txt") needs_newline = False if os.path.exists(catalog_path): if any(eid == ebook_id for eid, _ in parse_catalog(catalog_path)): return False with open(catalog_path, encoding="utf-8") as f: needs_newline = not f.read().endswith("\n") if dry_run: print(f" [dry] would catalogue: scripts/catalogs/{genre}.txt <- {ebook_id}\t{title}") return True os.makedirs(CATALOG_ROOT, exist_ok=True) with open(catalog_path, "a", encoding="utf-8") as f: if needs_newline: f.write("\n") f.write(f"{ebook_id}\t{title}\n") print(f" Catalogued: scripts/catalogs/{genre}.txt <- {ebook_id}\t{title}") return True </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.run_catalog" class="doc doc-heading"> <code class="highlight language-python">run_catalog(catalog_file, genre=None, force=False, dry_run=False)</code> <a href="#gutenberg_kg.gutenberg.run_catalog" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Download all books from a catalog file. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>catalog_file</code> </td> <td> <code>str</code> </td> <td> <div class="doc-md-description"> Path to a tab-separated catalog file (ebook_id [title]). </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>genre</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Genre subdirectory for all books. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>force</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Re-download even if already present. </div> </td> <td> <code>False</code> </td> </tr> <tr class="doc-section-item"> <td> <code>dry_run</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Print actions without writing files. </div> </td> <td> <code>False</code> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027</pre></div></td><td class="code"><div><pre><code>def run_catalog( catalog_file: str, genre: str | None = None, force: bool = False, dry_run: bool = False, ) -> None: """Download all books from a catalog file. :param catalog_file: Path to a tab-separated catalog file (ebook_id [title]). :param genre: Genre subdirectory for all books. :param force: Re-download even if already present. :param dry_run: Print actions without writing files. """ if genre and genre not in ALL_GENRES: print(f"ERROR: unknown genre {genre!r}.", file=sys.stderr) sys.exit(1) entries = parse_catalog(catalog_file) if not entries: print("No valid entries found in catalog file.") sys.exit(1) print(f"{'[dry] ' if dry_run else ''}Catalog loaded: {len(entries)} book(s)\n") succeeded = 0 for ebook_id, title in entries: try: download_book(ebook_id, title=title, genre=genre, force=force, dry_run=dry_run) succeeded += 1 except Exception as exc: # noqa: BLE001 print(f" ERROR downloading #{ebook_id}: {exc}", file=sys.stderr) print() if dry_run: print(f"[dry] {succeeded}/{len(entries)} book(s) would be downloaded.") else: print(f"Done. {succeeded}/{len(entries)} book(s) downloaded.") </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.run_catalog_sync" class="doc doc-heading"> <code class="highlight language-python">run_catalog_sync(genres, dry_run=False)</code> <a href="#gutenberg_kg.gutenberg.run_catalog_sync" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Record every downloaded book in its genre catalog. The repair counterpart to :func:<code>record_in_catalog</code>, for books downloaded before write-back existed. It is the analogue of <code>gutenkg re-register</code>: both rebuild a derived record from what is already on disk, without redoing the expensive work that produced it. No downloading is involved. Each book's <code>reference.md</code> -- written by every download path, Gutenberg and IA alike -- carries the Gutenberg ID, so the catalog can be reconstructed by reading the corpus. The directory name is written as the title override, matching what <code>audit</code> compares against. IA genres are skipped: they have no catalog files, and what an IA catalog should contain is a separate design question. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>genres</code> </td> <td> <code>list[str]</code> </td> <td> <div class="doc-md-description"> Genres to process. </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>dry_run</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Report what would be written without writing. </div> </td> <td> <code>False</code> </td> </tr> </tbody> </table> Returns: <table> <thead> <tr> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>int</code> </td> <td> <div class="doc-md-description"> Number of catalog entries added (or that would be added). </div> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838</pre></div></td><td class="code"><div><pre><code>def run_catalog_sync(genres: list[str], dry_run: bool = False) -> int: """Record every downloaded book in its genre catalog. The repair counterpart to :func:`record_in_catalog`, for books downloaded before write-back existed. It is the analogue of ``gutenkg re-register``: both rebuild a derived record from what is already on disk, without redoing the expensive work that produced it. No downloading is involved. Each book's ``reference.md`` -- written by every download path, Gutenberg and IA alike -- carries the Gutenberg ID, so the catalog can be reconstructed by reading the corpus. The directory name is written as the title override, matching what ``audit`` compares against. IA genres are skipped: they have no catalog files, and what an IA catalog should contain is a separate design question. :param genres: Genres to process. :param dry_run: Report what would be written without writing. :returns: Number of catalog entries added (or that would be added). """ from gutenberg_kg.genres import IA_GENRES added = 0 for genre in genres: if genre in IA_GENRES: continue genre_dir = os.path.join(CORPUS_ROOT, genre) if not os.path.isdir(genre_dir): continue for book in sorted(os.listdir(genre_dir)): book_dir = os.path.join(genre_dir, book) ref = os.path.join(book_dir, "reference.md") if book.startswith(".") or not os.path.isfile(ref): continue ebook_id = parse_reference(Path(ref)).get("ebook_id") if ebook_id is None: continue if record_in_catalog(genre, int(ebook_id), book, dry_run=dry_run): added += 1 return added </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.run_fetch_genre" class="doc doc-heading"> <code class="highlight language-python">run_fetch_genre(genre, query=None, max_results=25, yes=False, force=False, dry_run=False)</code> <a href="#gutenberg_kg.gutenberg.run_fetch_genre" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Search, confirm, and download an entire genre in one step. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>genre</code> </td> <td> <code>str</code> </td> <td> <div class="doc-md-description"> Genre directory name (e.g. <code>science-fiction</code>). </div> </td> <td> required </td> </tr> <tr class="doc-section-item"> <td> <code>query</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Optional additional keyword refinement. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>max_results</code> </td> <td> <code>int</code> </td> <td> <div class="doc-md-description"> Maximum books to search for. </div> </td> <td> <code>25</code> </td> </tr> <tr class="doc-section-item"> <td> <code>yes</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Skip interactive confirmation. </div> </td> <td> <code>False</code> </td> </tr> <tr class="doc-section-item"> <td> <code>force</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Re-download even if already present. </div> </td> <td> <code>False</code> </td> </tr> <tr class="doc-section-item"> <td> <code>dry_run</code> </td> <td> <code>bool</code> </td> <td> <div class="doc-md-description"> Print actions without writing files. </div> </td> <td> <code>False</code> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149</pre></div></td><td class="code"><div><pre><code>def run_fetch_genre( genre: str, query: str | None = None, max_results: int = 25, yes: bool = False, force: bool = False, dry_run: bool = False, ) -> None: """Search, confirm, and download an entire genre in one step. :param genre: Genre directory name (e.g. ``science-fiction``). :param query: Optional additional keyword refinement. :param max_results: Maximum books to search for. :param yes: Skip interactive confirmation. :param force: Re-download even if already present. :param dry_run: Print actions without writing files. """ import datetime q = query or genre.replace("-", " ") genre_dir = os.path.join(CORPUS_ROOT, genre) already_present: set[str] = set() if os.path.isdir(genre_dir): already_present = { e.name for e in os.scandir(genre_dir) if e.is_dir() and not e.name.startswith(".") } print(f"Searching Gutenberg for: {q!r} (max {max_results} results)\n") results = search_gutenberg(q, max_results=max_results) if not results: print("No results found.") return print(f"{'':>2}{'ID':>6} {'Title':<50} {'Author':<30} {'Status'}") print("-" * 100) for r in results: title_s = r["title"][:48] author_s = (r["author"] or "—")[:28] status = "[already downloaded]" if r["title"] in already_present else "" print(f" {r['ebook_id']:>6} {title_s:<50} {author_s:<30} {status}") print() to_download = [] for r in results: if r["title"] in already_present and not force: continue if yes or dry_run: to_download.append(r) else: try: ans = ( input(f" Download {r['title']!r} (#{r['ebook_id']})? [Y/n/q] ").strip().lower() ) except (EOFError, KeyboardInterrupt): print("\n Aborted.") break if ans == "q": print(" Aborted.") break if ans in ("", "y"): to_download.append(r) if not to_download: print("Nothing to download.") return print(f"\n{'[dry] ' if dry_run else ''}Downloading {len(to_download)} book(s) into {genre}/\n") report_rows = [] downloaded = 0 failed = 0 for r in to_download: ebook_id = r["ebook_id"] try: download_book(ebook_id, genre=genre, force=force, dry_run=dry_run) status = "dry-run" if dry_run else "downloaded" downloaded += 1 except Exception as exc: # noqa: BLE001 print(f" ERROR #{ebook_id}: {exc}", file=sys.stderr) status = f"failed: {exc}" failed += 1 report_rows.append( {"id": ebook_id, "title": r["title"], "author": r.get("author", ""), "status": status} ) print() reports_dir = os.path.join(REPO_ROOT, "reports") os.makedirs(reports_dir, exist_ok=True) ts = datetime.datetime.now(datetime.UTC).strftime("%Y-%m-%d_%H%M%S") report_path = os.path.join(reports_dir, f"fetch_genre_{genre}_{ts}.md") lines = [ f"# Fetch-Genre Report: {genre}", "", f"**Date:** {datetime.datetime.now(datetime.UTC).strftime('%Y-%m-%d %H:%M:%S UTC')} ", f"**Query:** `{q}` ", f"**Genre:** `{genre}` ", f"**Dry run:** {'yes' if dry_run else 'no'} ", f"**Downloaded:** {downloaded} ", f"**Failed:** {failed} ", "", "## Results", "", "| ID | Title | Author | Status |", "|----|-------|--------|--------|", ] for row in report_rows: lines.append(f"| {row['id']} | {row['title']} | {row['author']} | {row['status']} |") lines += ["", "## Already Present (skipped)", ""] for t in sorted(already_present): lines.append(f"- {t}") if not already_present: lines.append("_(none)_") lines.append("") with open(report_path, "w", encoding="utf-8") as f: f.write("\n".join(lines)) print(f"{'[dry] ' if dry_run else ''}Done. {downloaded} downloaded, {failed} failed.") print(f"Report saved: {report_path}") </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.run_search" class="doc doc-heading"> <code class="highlight language-python">run_search(query='', author=None, title=None, subject=None, language=None, max_results=25)</code> <a href="#gutenberg_kg.gutenberg.run_search" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Search Gutenberg and print results. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>query</code> </td> <td> <code>str</code> </td> <td> <div class="doc-md-description"> General keyword query. </div> </td> <td> <code>''</code> </td> </tr> <tr class="doc-section-item"> <td> <code>author</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Filter by author name. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>title</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Filter by title keyword. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>subject</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Filter by subject. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>language</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Filter by language code. </div> </td> <td> <code>None</code> </td> </tr> <tr class="doc-section-item"> <td> <code>max_results</code> </td> <td> <code>int</code> </td> <td> <div class="doc-md-description"> Maximum results to display. </div> </td> <td> <code>25</code> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989</pre></div></td><td class="code"><div><pre><code>def run_search( query: str = "", author: str | None = None, title: str | None = None, subject: str | None = None, language: str | None = None, max_results: int = 25, ) -> None: """Search Gutenberg and print results. :param query: General keyword query. :param author: Filter by author name. :param title: Filter by title keyword. :param subject: Filter by subject. :param language: Filter by language code. :param max_results: Maximum results to display. """ parts = [] if author: parts.append(author) if title: parts.append(title) if subject: parts.append(subject) if language: parts.append(f"l.{language}") if query: parts.append(query) if not parts: print("Error: Provide at least one search term (--author, --title, --subject, or keyword).") sys.exit(1) q = " ".join(parts) print(f"Searching Gutenberg for: {q}\n") results = search_gutenberg(q, max_results=max_results) if not results: print("No results found.") return print(f"{'ID':>6} {'Title':<55} {'Author':<30}") print("-" * 95) for r in results: title_s = r["title"][:53] author_s = (r["author"] or "—")[:28] subjects_s = ", ".join(r.get("subjects", [])[:3]) print(f"{r['ebook_id']:>6} {title_s:<55} {author_s:<30}") if subjects_s: print(f"{'':>8}Subjects: {subjects_s}") print(f"\n{len(results)} result(s). Download with: gutenkg download book <ID>") </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.run_survey" class="doc doc-heading"> <code class="highlight language-python">run_survey(genre=None)</code> <a href="#gutenberg_kg.gutenberg.run_survey" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Scan the corpus and print download/ingest status by genre. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>genre</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> Optional genre filter. </div> </td> <td> <code>None</code> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1152 1153 1154 1155 1156 1157</pre></div></td><td class="code"><div><pre><code>def run_survey(genre: str | None = None) -> None: """Scan the corpus and print download/ingest status by genre. :param genre: Optional genre filter. """ survey_repo(genre_filter=genre) </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.search_gutenberg" class="doc doc-heading"> <code class="highlight language-python">search_gutenberg(query, max_results=25)</code> <a href="#gutenberg_kg.gutenberg.search_gutenberg" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Search Gutenberg via the OPDS feed. Returns list of book dicts. <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356</pre></div></td><td class="code"><div><pre><code>def search_gutenberg(query: str, max_results: int = 25) -> list[dict]: """Search Gutenberg via the OPDS feed. Returns list of book dicts.""" params = urllib.parse.quote(query) url = f"{GUTENBERG_SEARCH_URL}?query={params}" xml_text = fetch_url(url) root = ET.fromstring(xml_text) results = [] for entry in root.findall("atom:entry", NS): entry_id = entry.find("atom:id", NS) if entry_id is None or entry_id.text is None: continue # Skip non-book entries (Authors, Subjects facets) id_text = entry_id.text if "/authors/" in id_text or "/subjects/" in id_text: continue # Extract ebook ID from the entry id URL m = re.search(r"/ebooks/(\d+)", id_text) if not m: # Try the urn format m = re.search(r"gutenberg:(\d+)", id_text) if not m: continue ebook_id = int(m.group(1)) title_el = entry.find("atom:title", NS) title = ( title_el.text.strip() if title_el is not None and title_el.text else f"Unknown (#{ebook_id})" ) # In search results, author is in <content type="text">, not <author> author = "" author_el = entry.find("atom:author/atom:name", NS) if author_el is not None and author_el.text: name = author_el.text.strip() parts = [p.strip() for p in name.split(",", 1)] author = " ".join(reversed(parts)) if len(parts) == 2 else name if not author: content_el = entry.find("atom:content", NS) if content_el is not None and content_el.text: text = content_el.text.strip() # Filter out download count strings like "996 downloads" if not re.match(r"^\d+\s+downloads?$", text, re.IGNORECASE): author = text # Subjects subjects = [] for cat in entry.findall("atom:category", NS): term = cat.get("term", "") if term: subjects.append(term) results.append( { "ebook_id": ebook_id, "title": title, "author": author, "subjects": subjects, } ) if len(results) >= max_results: break return results </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.slugify" class="doc doc-heading"> <code class="highlight language-python">slugify(title)</code> <a href="#gutenberg_kg.gutenberg.slugify" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Convert a title to a filesystem-friendly slug. <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>103 104 105 106 107 108</pre></div></td><td class="code"><div><pre><code>def slugify(title: str) -> str: """Convert a title to a filesystem-friendly slug.""" slug = title.lower().strip() slug = re.sub(r"[^\w\s-]", "", slug) slug = re.sub(r"[\s-]+", "_", slug) return slug </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.strip_boilerplate" class="doc doc-heading"> <code class="highlight language-python">strip_boilerplate(text)</code> <a href="#gutenberg_kg.gutenberg.strip_boilerplate" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Remove Project Gutenberg header and footer boilerplate. <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>364 365 366 367 368 369 370 371 372 373 374 375 376</pre></div></td><td class="code"><div><pre><code>def strip_boilerplate(text: str) -> str: """Remove Project Gutenberg header and footer boilerplate.""" start = START_MARKER.search(text) end = END_MARKER.search(text) if start: text = text[start.end() :] if end: end2 = END_MARKER.search(text) if end2: text = text[: end2.start()] return text.strip() + "\n" </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.survey_repo" class="doc doc-heading"> <code class="highlight language-python">survey_repo(genre_filter=None)</code> <a href="#gutenberg_kg.gutenberg.survey_repo" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Print a survey table of downloaded books organized by genre. Parameters: <table> <thead> <tr> <th>Name</th> <th>Type</th> <th>Description</th> <th>Default</th> </tr> </thead> <tbody> <tr class="doc-section-item"> <td> <code>genre_filter</code> </td> <td> <code>str | None</code> </td> <td> <div class="doc-md-description"> When provided, only the matching genre is shown. </div> </td> <td> <code>None</code> </td> </tr> </tbody> </table> <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930</pre></div></td><td class="code"><div><pre><code>def survey_repo(genre_filter: str | None = None) -> None: """Print a survey table of downloaded books organized by genre. :param genre_filter: When provided, only the matching genre is shown. """ genres_to_show = [genre_filter] if genre_filter else ALL_GENRES genre_set = set(ALL_GENRES) for genre in genres_to_show: genre_dir = os.path.join(CORPUS_ROOT, genre) books = [] if os.path.isdir(genre_dir): for entry in sorted(os.scandir(genre_dir), key=lambda e: e.name): if entry.is_dir() and not entry.name.startswith("."): books.append(_survey_book_dir(entry.path, entry.name)) count = len(books) print(f"\n=== {genre} ({count} book{'s' if count != 1 else ''}) ===") if not books: print(" (none)") else: for b in books: title_col = b["title"][:50].ljust(50) print( f" {title_col}" f" md={_check_mark(b['md'])}" f" ref={_check_mark(b['ref'])}" f" kg={_check_mark(b['kg'])}" ) # Ungrouped books (corpus-level dirs that aren't genre dirs and don't start with '.') if not genre_filter: ungrouped = [] for entry in sorted(os.scandir(CORPUS_ROOT), key=lambda e: e.name): if entry.is_dir() and not entry.name.startswith(".") and entry.name not in genre_set: ungrouped.append(_survey_book_dir(entry.path, entry.name)) if ungrouped: count = len(ungrouped) print(f"\n=== (ungrouped) ({count} book{'s' if count != 1 else ''}) ===") for b in ungrouped: title_col = b["title"][:50].ljust(50) print( f" {title_col}" f" md={_check_mark(b['md'])}" f" ref={_check_mark(b['ref'])}" f" kg={_check_mark(b['kg'])}" ) </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.text_to_markdown" class="doc doc-heading"> <code class="highlight language-python">text_to_markdown(text, meta)</code> <a href="#gutenberg_kg.gutenberg.text_to_markdown" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Convert plain Gutenberg text to structured Markdown. Strategy: 1. Add YAML-style front matter from metadata 2. Detect and convert chapter/section headings to Markdown headings 3. Preserve paragraph structure 4. Skip table of contents (redundant with heading structure) <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556</pre></div></td><td class="code"><div><pre><code>def text_to_markdown(text: str, meta: dict) -> str: """Convert plain Gutenberg text to structured Markdown. Strategy: 1. Add YAML-style front matter from metadata 2. Detect and convert chapter/section headings to Markdown headings 3. Preserve paragraph structure 4. Skip table of contents (redundant with heading structure) """ lines = text.split("\n") total = len(lines) # Skip producer credits at the very start start_idx = _skip_front_matter(lines, 0) # Detect and skip table of contents # Where the contents list ends comes from the structural profile, not # from counting blank lines; detect_toc anchors it to the marker line. toc = _detect_toc(lines, start_idx, min(start_idx + 200, total), contents_regions(lines)) toc_range = range(toc[0], toc[1]) if toc else range(0) # Title-Case work titles, honoured only where they recur (see # _repeated_title_lines). File-level knowledge, so it cannot live in the # per-line _is_heading. title_lines = _repeated_title_lines(lines, start_idx, toc_range) title_lines |= _bare_bible_title_lines(lines, start_idx, toc_range) roman_lines = bare_roman_heading_lines(lines, start_idx, toc_range) # Build the markdown md_lines = [] # Front matter: title as h1 title = meta.get("title", "Untitled") author = meta.get("author", "Unknown") md_lines.append(f"# {title}") md_lines.append("") md_lines.append(f"**{author}**") md_lines.append("") md_lines.append("---") md_lines.append("") prev_blank = True prev_line = "" i = start_idx while i < total: # Skip TOC region if i in toc_range: i += 1 continue line = lines[i] stripped = line.strip() # Blank line if not stripped: if not prev_blank: md_lines.append("") prev_blank = True i += 1 continue # Strip illustration markers stripped = ILLUSTRATION_RE.sub("", stripped).strip() if not stripped: if not prev_blank: md_lines.append("") prev_blank = True i += 1 continue # Check for heading heading = _is_heading(stripped) if heading is None and i in title_lines: heading = (2, stripped) # Level 3, the level ALL-CAPS already gives III and XII in the same books. if heading is None and i in roman_lines: heading = (3, stripped) if heading and (prev_blank or _breaks_before_heading(prev_line)): level, heading_text = heading # Look ahead for subtitle / multi-line chapter title: # "CHAPTER I." followed by "HOW MANY KINDS OF..." on next line(s) subtitle_lines = [] j = i + 1 # Skip blank lines between heading and potential subtitle while j < total and not lines[j].strip(): j += 1 if j < total: next_line = lines[j].strip() next_line_clean = ILLUSTRATION_RE.sub("", next_line).strip() # Multi-line ALL CAPS title continuation (e.g. The Prince) # But NOT bare roman numerals like "I." — those are sub-sections is_bare_roman = bool(re.match(r"^[IVXLCDM]+\.\s*$", next_line_clean)) if ( next_line_clean and next_line_clean.isupper() and heading_text.isupper() and len(next_line_clean) < 100 and not is_bare_roman ): subtitle_lines.append(next_line_clean) k = j + 1 while k < total and lines[k].strip() and lines[k].strip().isupper(): subtitle_lines.append(lines[k].strip()) k += 1 heading_text = heading_text + " " + " ".join(subtitle_lines) j = k # Short, Title-Case subtitle (not a paragraph start) # Must be very short (< 50 chars) and look like a title elif ( next_line_clean and len(next_line_clean) < 50 and not _is_heading(next_line_clean) and not next_line_clean[0].islower() and next_line_clean[0].isupper() and next_line_clean[-1] not in ".!?;," ): subtitle_lines.append(next_line_clean) j = j + 1 # A subtitle ending in a colon introduces the next line. # When that line looks like a heading it is still the # subtitle's: the KJV's "Otherwise Called:" / "The First # Book of the Kings" under 1 Samuel used to open a # spurious section. Prose after a colon stays body text. if next_line_clean.endswith(":"): k = j while k < total and not lines[k].strip(): k += 1 if k < total and _is_heading(lines[k].strip()): subtitle_lines[0] += " " + lines[k].strip() j = k + 1 md_lines.append(f"{'#' * level} {heading_text}") if subtitle_lines and not subtitle_lines[0].isupper(): md_lines.append("") md_lines.append(f"*{subtitle_lines[0]}*") md_lines.append("") i = j if subtitle_lines else i + 1 prev_blank = True continue # Normal text line md_lines.append(stripped) prev_blank = False prev_line = stripped i += 1 return "\n".join(md_lines).strip() + "\n" </code></pre></div></td></tr></table></div> </details> </div> </div> <div class="doc doc-object doc-function"> <h3 id="gutenberg_kg.gutenberg.write_reference" class="doc doc-heading"> <code class="highlight language-python">write_reference(book_dir, meta)</code> <a href="#gutenberg_kg.gutenberg.write_reference" class="headerlink" title="Permanent link">¶</a></h3> <div class="doc doc-contents "> Write a reference.md file with Gutenberg metadata. <details class="mkdocstrings-source"> <summary>Source code in <code>src/gutenberg_kg/gutenberg.py</code></summary> <div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620</pre></div></td><td class="code"><div><pre><code>def write_reference(book_dir: str, meta: dict) -> str: """Write a reference.md file with Gutenberg metadata.""" ref_path = os.path.join(book_dir, "reference.md") lines = [ f"# Reference: {meta.get('title', 'Unknown')}", "", "## Source", "", f"- **Project Gutenberg ID**: {meta.get('ebook_id', '?')}", f"- **URL**: {meta.get('gutenberg_url', 'N/A')}", f"- **Rights**: {meta.get('rights', 'Public domain in the USA.')}", "", ] authored = author_lines(meta) if authored: lines += ["## Author", "", *authored, ""] if meta.get("published"): lines += [f"- **Gutenberg Published**: {meta['published']}", ""] if meta.get("language"): lines += [ "## Language", "", f"- {meta['language']}", "", ] if meta.get("subjects"): lines += ["## Subjects", ""] for s in meta["subjects"]: lines.append(f"- {s}") lines.append("") if meta.get("summary"): lines += [ "## Summary", "", meta["summary"], "", ] if meta.get("note"): lines += [ "## Notes", "", meta["note"], "", ] content = "\n".join(lines) with open(ref_path, "w", encoding="utf-8") as f: f.write(content) return ref_path </code></pre></div></td></tr></table></div> </details> </div> </div> </div> </div> </div> </article> </div> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type="button" class="md-top md-icon" data-md-component="top" hidden> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class="md-footer"> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class="md-copyright"> Made with <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener"> Material for MkDocs </a> </div> </div> </div> </footer> </div> <div class="md-dialog" data-md-component="dialog"> <div class="md-dialog__inner md-typeset"></div> </div> <script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": ["navigation.tabs", "navigation.top", "content.code.copy", "content.action.edit"], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script src="../../assets/javascripts/bundle.d7400e89.min.js"></script> </body> </html>

Gutenberg source¶

gutenberg_kg.gutenberg ¶

`gutenberg_kg.gutenberg` ¶