Skip to content

The Pustakālaya Book Format, version 1.2

This document is the specification of the format in which a book is handed to Pustakālaya, a library of books that are read page by page, online or offline. It is written for two readers: an author who writes a book by hand, and a program that generates books (a converter, a course tool, a script that turns notes into chapters). A book that follows this format can be attached to any Pustakālaya library and is published without further work.

A book is a git repository (or a plain folder). The library never changes it. Two things describe it:

  • pustakalaya.yml, which says what the repository contains (the book's id, title, author, cover, language);
  • the book itself, in one of three formats: Markdown (the normalized Markdown book, described in full below, the format to generate), LaTeX or HTML (described briefly, with pointers).

To check a book, run python tools/books.py check <folder or git address> in the product repository. To start one, run python tools/books.py new <folder> --title "...", or copy the template repository (book-template). To attach one, run python tools/books.py attach <folder or git address> --collection <id>.

Words in this document: must is checked by books.py check as an error (the book would not build or would display wrongly); should is a warning.

Contents

  1. Repository layout
  2. pustakalaya.yml
  3. The Markdown book: book.yml and the chapter files
  4. Markdown conventions
  5. Images and covers
  6. Languages
  7. LaTeX and HTML books
  8. What makes a book build cleanly
  9. Attaching a book
  10. Generating a book: a minimal example
  11. Animations and visualizations (new in 1.1)
  12. Category, level and kind (new in 1.2)
  13. Slides (new in 1.2)
  14. Pronunciation, quizzes and language markup (new in 1.2)

1. Repository layout

A repository with one Markdown book:

my-book/
  pustakalaya.yml        what the repository contains (section 2)            required
  book.yml               the chapter list, in reading order (section 3)       required
  chapters/              one Markdown file per chapter                        required
    01_preface.md
    02_first-chapter.md
    03_second-chapter.md
  images/                figures and photographs the chapters use             optional
    workflow.svg
  README.md              for people browsing the repository                   optional
  .gitignore             editor files and the like                            optional
  • File and folder names use letters, digits, -, _ and ., no spaces. Names are case-sensitive: a book that works on Windows must also work on Linux, so images/Fig1.png is not images/fig1.png.
  • All text files are UTF-8 (a byte order mark is tolerated), with \n or \r\n line ends.
  • The repository holds no build output. The library builds the book; nothing in the repository is executed.
  • A repository may hold several books, each in a folder of its own, listed in one pustakalaya.yml with a path: for each (section 2). The layout above is then repeated per folder.

2. pustakalaya.yml

pustakalaya.yml is at the root of the repository. It is a YAML mapping with a list of books.

pustakalaya: 1                  # schema version
format: markdown                # default format of the books below
author: Ada Example             # defaults for the books below
language: en
books:
  - id: my_book
    path: .
    title: My Book
    subtitle: A short line under the title
    description: One or two sentences for the cover card and the book's first page.
    chapter_label: Chapter
    subjects: [programming, algorithms]   # section 12: subcategories from the list, the main one first
    level: beginner
    kind: textbook
    cover:
      background: "#1f3a5f"
      accent: "#f2b84b"
      text: "#f6f8fb"

Top-level keys

Key Type Default Meaning
pustakalaya integer The schema version. Must be 1 when present; should be present.
format latex | markdown | html latex Default format of the books. A Markdown book must say markdown, here or on the book.
main text main.tex Default LaTeX main file.
author, affiliation, generated_by, language, chapter_label, conventions, subjects, level, kind, tags as on a book Defaults for every book that does not set its own.
collection text The collection the books would like to be in. A suggestion: the library decides.
books list Required. At least one book.

Keys of a book

Key Type Default Meaning
id text Required. Letters a-z, digits, _ or -; starts with a letter or digit. Unique in the repository and in the library. It is the book's folder in the site (<collection>/<id>/), so it is never changed after publication: readers' saved places and links depend on it. The names assets, search, img, index, about, offline, sitemap, library, sw, install, publish, format, books, subjects, lists are reserved.
path text . The book's folder inside the repository. Must stay inside the repository.
format latex | markdown | html the top-level format See sections 3 to 7.
title text the id, in capitals The book's title. Should be set.
subtitle text empty A line under the title, on the cover and the first page.
description text empty One or two sentences, on the cover card and under the title. Should be set; keep it under 500 characters.
author text the owner of the library Author's name, on the cover. Always set it.
affiliation text empty The author's institution.
generated_by text empty For a book whose author is shown as "AI generated": the name of the person behind it. The engine writes "Generated by " inside the book: under the author on the book's first page (and at the top of its "About this book" section), in the reader's contents panel, and on the title slide of the presentation. It is not shown on covers, shelf cards or in search; they show the author.
language text en Language code (en, hi, sa, ta, ...). Sets the page language, fonts and hyphenation (section 6).
chapter_label text Chapter What a numbered chapter is called in headings and contents: Chapter, Lesson, Unit, Part, or a word in the book's language.
cover mapping maroon cover Section 5.
conventions mapping LaTeX books only (section 7).
main text main.tex LaTeX books only: the root file.
importer, edition, file text HTML books only (section 7).
contents list of paths Plain Markdown books only: the chapter files, in order, instead of detecting them.
collection text the library entry's Overrides the collection for this book.
order integer order of listing Position of the book in its collection. Books are ordered as listed unless order is set.
public boolean true false keeps the book out of the published site (it can still be read in a local private build).
published_elsewhere boolean false With public: false: the book is published on another portal (a book kept on a portal's reserve shelf, for example), so its title may be cited in the text of the books of this portal; check.py then looks only for its addresses.
subjects list of subject ids Section 12. The categories of the book: ids of subcategories (or of a category) from the list in section 12, the main one first. Should be set.
level beginner | intermediate | advanced Section 12. Who the book is written for.
kind textbook | course | field-guide | reference | workbook | series-volume Section 12. What sort of book it is.
tags list of words Section 12. Words that help a search find the book (names, abbreviations, other spellings).

Unknown keys are reported as warnings: they are usually typos.

A repository that cannot hold the file (it is not yours to change) is described by an inline manifest: in the library's library.yml, with the same keys. When there is neither a file nor a manifest, the repository is auto-detected: a normalized Markdown book (book.yml and chapters/), a LaTeX main file, plain Markdown, or an HTML book. Auto-detection is a convenience; write the file, so that the id, the title and the cover are yours.

3. The Markdown book: book.yml and the chapter files

This is the format to generate. A book folder holds book.yml, chapters/ and images/.

book.yml

title: My Book                  # informational; the library takes title and author from pustakalaya.yml
subtitle: A short line
author: Ada Example
language: en
description: >-
  A fuller text about the book. It is shown on the book's first page, under "About this book".
numbering: sections
chapters:
  - file: chapters/01_preface.md
    title: Preface
    kind: front
    sections: 3
  - file: chapters/02_first-chapter.md
    title: The First Chapter
    kind: chapter
    number: 1
    label: Part I
    sections: 5
  - file: chapters/09_glossary.md
    title: Glossary
    kind: back
    sections: 1
Key Type Meaning
chapters list Required, not empty. The pages of the book in reading order. The order of this list is the order of the book; the file names do not decide it.
description text The fuller description on the book's first page. The one-line description on the cover card is description in pustakalaya.yml.
title, subtitle, author, language, numbering Informational. numbering: sections says that sections carry the numbers written in their headings.
id, collection, importer, source Written by the converters of the library; ignored.

Each entry of chapters:

Key Type Meaning
file path Required. The chapter's file, relative to the book folder; normally chapters/NN_slug.md. Must exist, and be listed once.
title text Required. The title in the contents and the navigation. Should equal the # heading of the file.
kind front | chapter | back Default chapter. front: unnumbered pages before the chapters (preface, how to read this book). chapter: a numbered chapter. back: unnumbered pages after the chapters (glossary, bibliography, solutions).
number integer Required for chapter, ignored otherwise. The chapter's number: 3 makes the page heading "Chapter 3: Title". Numbers should increase.
label text A label for the contents: Unit 2, Part III. See Grouping below.
sections integer The number of ## sections of the file. Informational.

Chapter file names. Name the files NN_slug.md: two digits (three for a book of more than 99 pages), an underscore, a lowercase slug of letters, digits and hyphens. The slug, without the number, is the page's address (<book>/first-chapter.html), so slugs should be unique; a repeated slug gets -2. Do not use index, about, bibliography or offline as a slug. Use ASCII slugs even for books in other scripts.

Kinds. A book needs at least one chapter. The contents list the pages under Front matter, Chapters and Back matter.

Grouping. When two or more chapters share a label, the contents group the chapters by label ("Unit 1", "Part II"). Chapters with a label of their own then form a group of one. Without shared labels, the label is not used.

The first page. The library writes the book's first page itself, from the descriptions and the contents (every chapter, with its sections and subsections). The front chapters follow it as ordinary pages; the reader opens at the first front chapter or, without one, at the first chapter.

4. Markdown conventions

The text is Markdown as understood by Python-Markdown with these extensions: tables, fenced code with highlighting, attribute lists, definition lists, footnotes, admonitions, folded details, Markdown inside HTML, and MathJax mathematics. Each convention below has an example; the template book (book-template) uses all of them.

Chapter structure

A chapter file starts with one title line and then sections:

# Querying with SELECT

*Filtering, operators and ordering*

## 3.1 The SELECT statement

Text...

### Basic forms

Text...

## 3.2 The WHERE clause
  • # Title is the first line of the file (blank lines may come before it). It has no number: the build writes the chapter's number (from book.yml) into the page heading. Only one # heading.
  • A chapter must have at least one ## section. Sections of a numbered chapter are written ## N.M Title, with the chapter number N and a running number M, typed by hand: the library does not renumber them. Front and back matter use unnumbered ## Title.
  • ### subsections and deeper headings are not numbered. The contents and the reader's "Go to" box list ## and ###.
  • Headings get anchors made from their text. To keep a link stable when a title changes, give the heading an explicit anchor: ## 3.1 The SELECT statement {#select} (letters, digits, -, _).
  • Blank lines separate blocks. Paragraphs are plain lines; *italic*, **bold**, `code`, lists with - or 1.. A list or block that follows a paragraph needs a blank line before it.

Boxes (admonitions)

!!! warning "Common mistake"

    The body is indented by four spaces. It may hold paragraphs, lists, code and mathematics.
  • The header is !!!, a type and an optional title in double quotes. A blank line before and after the box.
  • Types: note, tip, info, warning, example, quote, abstract, success, question, failure, danger, bug. (summary, hint, important, caution, error and a few more are accepted as aliases.) Other types are shown as plain boxes and reported as a warning.
  • Use abstract for "In brief" and chapter summaries, note for learning outcomes, example for worked examples and exercises, quote for quotations (give the source in the box), warning for common mistakes, tip for advice.
  • ??? instead of !!! makes the box folded until the reader opens it; ???+ makes it open at first.

Exercises with folded answers

An exercise is an example box titled "Exercise N"; its answer is a folded box that follows it directly.

!!! example "Exercise 1"

    Name the file in which the list of chapters is kept.

??? note "Answer"

    **`book.yml`.** It lists the chapters in reading order.

A short answer can also be typed and checked by the reader (??? note typed "Answer", section 14), and a whole set of questions with text boxes, choices and matching is written as a quiz block (section 14).

Figures with captions and labels

![The caption of the figure, as a full sentence.](images/workflow.svg "Figure 1.1 Short label")
  • The figure goes on a line of its own. The text in brackets is the caption; the optional quoted text after the file name is the label, printed in bold before the caption ("Figure 1.1 Short label. The caption..."). Type the figure number into the label; it is not counted for you.
  • The path is images/name.ext. Write the caption: it is also the text for readers who cannot see the picture.
  • An image inside a sentence, or with any other path form, is shown as an ordinary image without caption.

Tables

| Element | Written as | Appears as |
|---|:---:|---|
| Section | `## 1.1 Title` | numbered section |
| Box | `!!! note "Title"` | coloured box |

*Table 1.1. The main elements of a chapter.*

Pipe tables with a header row. The second row sets alignment (:---, :---:, ---:). A cell is one line; use <br> for a line break. A table wider than the page scrolls inside the page. Put the caption on a line of its own after the table, in italics, with the number typed into it. A blank line before the table.

Code

```python
def mean(values):
    return sum(values) / len(values)
```

Fence code with three backticks (or ~~~) and a language: any Pygments name (python, c, cpp, java, sql, bash, json, yaml, verilog, ...); use text for output and anything that is not a programming language. To show a fence inside a fence, use a longer outer fence (four backticks). Inline code is `like this`.

Mathematics

Write TeX between dollar signs. MathJax typesets it in the page.

A formula in a sentence, $E = mc^2$, and one on a line of its own:

$$
\int_0^1 x^2\,dx = \frac{1}{3}
$$
  • $...$ is inline; $$...$$ is display. \begin{align}...\end{align} and the other amsmath environments work in display mode. \( ... \) and \[ ... \] are accepted too.
  • Delimiters must be balanced in a paragraph. A dollar sign that is only a dollar sign is written &#36; ("the price was $5").
  • Mathematics is typeset from the MathJax CDN; the page shows the TeX until it has loaded.

Footnotes

A claim that needs a source.[^src]

[^src]: Author, *Title*, chapter 3.

The marker [^name] and its definition [^name]: text are on the same page; names are local to the page. Every marker needs a definition and every definition a marker. The reader shows footnotes as pop-ups.

Cross-references

Link to another chapter by its file name, and to a heading by its anchor:

See [Writing a Chapter](02_writing-a-chapter.md), and [the section on figures](02_writing-a-chapter.md#figures).
Within a page: [section 2.1](#exercises).
  • file.md is the file name from book.yml (with or without chapters/ or ./); the link also works when the repository is browsed on a git host. The link must name a chapter that is in book.yml.
  • Prefer explicit anchors ({#figures}) for targets of links. The anchor of a heading without one is its text in lowercase, with punctuation removed and spaces turned into - (## 1.3 Figures becomes #13-figures).
  • References to numbered figures, tables and equations are typed as text ("Figure 1.1"); the label is the number.
  • Links to other sites are ordinary Markdown links.

Glossary (definition lists)

Admonition
:   A box with a title, written with `!!!` and a body indented by four spaces.

Front matter
:   The unnumbered chapters before the first numbered one.

The term on a line of its own, then a line that starts with a colon and three spaces. A glossary is a back chapter with one or more ## sections (## A, ## B, ...).

Blocks written in HTML

A converter that needs what the conventions above cannot say (a table with joined cells, a figure whose caption is not its alt text, a table or figure inside a box) may write the block in HTML with markdown="1"; the Markdown inside is then read as usual. The block starts at the beginning of a line, after a blank line; every element between the block and a cell that holds Markdown carries markdown="1", and the cell itself markdown="span". A box is the HTML of an admonition, a folded box that of ???. An id="..." on such a block is an anchor for links, like {#id}.

<figure class="fig fig-diagram" id="fig-2-1" markdown="1">
![What the picture shows, for readers who cannot see it.](images/cell.webp){: loading="lazy" }
<figcaption markdown="span">**Figure 2.1** The caption, with *emphasis* and $x^2$.</figcaption>
</figure>

<div class="admonition example" markdown="1">
<p class="admonition-title">Example 2.1 A box with a table</p>

<table markdown="1">
<tbody markdown="1">
<tr markdown="1"><td markdown="span" colspan="2">a cell with $y_1$</td></tr>
</tbody>
</table>

</div>

Hand-written books should keep to the conventions above; tools/importers/cnxml.py (open textbooks in CNXML) writes these blocks.

Not allowed

  • Raw <script> and <style> elements. Nothing in a book is executed, and styles belong to the library.
  • <iframe>, <object> and <embed> do not work offline and in the reader; avoid them. For a program that the reader can use, write a viz block (section 11).
  • Remote images (https://...): they are not available offline. Put a copy in images/.

5. Images and covers

  • Folder and names. Images are in images/, referenced as images/name.ext. Use lowercase names with hyphens. Subfolders are allowed.
  • Formats. SVG for diagrams (small, sharp at every size, theme-neutral colours); PNG for screenshots; JPEG or WebP for photographs; GIF only when unavoidable. MP4 or WebM for video, and the other moving figures of section 11.
  • Size. A raster image should be at most 1600 pixels wide and a file at most 2 MB; larger files slow the page and the offline copy. books.py check warns about both. Figures are scaled to the page.
  • Photographs and licences. For Markdown books every file in images/ is published as it is. Only include images that you may publish, and put the credit in the caption: ![Sanchi stupa. Photo: A. Photographer, CC BY-SA 4.0.](...).
  • Cover. The cover on the shelf is typographic: the collection name, the book's number if the collection numbers its books, the title, the subtitle and the author on a coloured background.
cover:
  background: "#1f3a5f"   # the cover
  accent: "#f2b84b"       # the rule and small details
  text: "#f6f8fb"         # the title and author

Colours are # followed by six hexadecimal digits, in quotes. Choose a text colour with a contrast of at least 4.5:1 against the background (the checker warns). The defaults are maroon, gold and cream.

A picture can be the cover instead: cover: {image: images/cover.jpg}, a JPEG, PNG or WebP file, relative to the book folder, portrait (it is scaled to 600 pixels wide). With an image the title text is hidden on the shelf, so the picture should carry the title.

6. Languages

Set language in pustakalaya.yml (en, hi, sa, ta, ...). It sets the lang of every page and of the reader, which selects the fonts and the hyphenation rules: Devanagari text (hi, sa, mr) uses Noto Serif Devanagari when the reader is online and the installed Devanagari faces otherwise, with hyphenation off and a larger line spacing.

  • Write the text, the titles and the chapter_label (for example अध्याय, पाठ) in the language of the book.
  • Keep file names, slugs and the id in ASCII: they become addresses.
  • Headings in a non-Latin script get automatic anchors that are empty or identical. Give the headings that you link to an explicit anchor (## परिचय {#parichay}).
  • Mixed text (Sanskrit words in an English book) needs no markup; give transliteration in IAST if the book uses it.
  • The words that the library itself shows (Contents, Read, Library) are in English.

7. LaTeX and HTML books

The Markdown book is the format to generate. The library also reads two other kinds of repository; they are described in README.md, which is the reference for them.

LaTeX (format: latex). A book class (book, report, memoir, scrbook) with a main file (main.tex or main: name.tex), \include or \input files, \frontmatter, \mainmatter, \appendix, \backmatter, and an optional BibLaTeX database. Each \chapter becomes a page; boxes, TikZ figures, tables, code listings, mathematics and bibliography are converted, and the numbers are those of the print edition. conventions: in pustakalaya.yml switches on the generic conversion for books with their own macros and preamble (expand_macros, equation_numbering, slugs, box_titles, strip_words, bibliography, preface_title, boxes). See "LaTeX conventions" and "Books of any layout" in README.md. pandoc and a TeX distribution must be installed where the library is built.

pustakalaya: 1
format: latex
books:
  - id: hardware_security
    main: hardware_security_book.tex
    title: Hardware Security
    author: Ada Example
    conventions: {expand_macros: true, equation_numbering: chapter, slugs: title}

HTML (format: html). Two layouts are understood: importer: english with edition: en | hinglish | hi (a folder per edition with assets/toc.js; one repository gives three books) and importer: toc-book with edition: en | hinglish | hi (the same, for a course whose English edition is at the root of the repository and whose other editions are in hinglish/ and hi/) and importer: guide with file: guide.html (a single page with a cover section). Both are converted into a normalized Markdown book when the library is built, with the phrases to hear, quizzes, practice tables and colours of section 14. A new HTML book should rather be generated as Markdown.

Plain Markdown (format: markdown without book.yml). An mkdocs, mdBook or GitBook layout, or numbered chapter files (01-intro.md, ...), with an optional contents: list. It is converted to a normalized Markdown book when the library is built, and the result is only as good as the guesses made from the layout. Generating book.yml and chapters/ is better.

8. What makes a book build cleanly

python tools/books.py check <folder> tests the points marked (c). The others are for you.

Repository and manifest

  • [ ] pustakalaya.yml is valid YAML, has pustakalaya: 1, format: markdown and a books: list (c)
  • [ ] every book has an id that matches [a-z0-9][a-z0-9_-]*, is unique and is not the id of another book in the library (c)
  • [ ] title, author, description and language are set (c, as warnings)
  • [ ] the cover colours are #rrggbb in quotes, and the text is readable on the background (c)
  • [ ] path exists and lies inside the repository (c)

book.yml and chapters

  • [ ] chapters lists every chapter file once, in reading order; no file exists in chapters/ that is not listed (c)
  • [ ] every entry has file (existing) and title; kind is front, chapter or back; every chapter has a whole number, increasing (c)
  • [ ] every file starts with # Title and has at least one ## section (c)
  • [ ] section numbers ## N.M agree with the chapter number (c)
  • [ ] file names are ASCII, and case matches (c)

Content

  • [ ] every image link is images/... and the file exists, with the same upper and lower case (c)
  • [ ] raster images are at most 1600 px wide and 2 MB; remote images are not used (c)
  • [ ] no raw <script> or <style> (c)
  • [ ] every fenced code block has a language, and every fence is closed (c)
  • [ ] every box is written !!! type "Title" with a body indented by four spaces (c)
  • [ ] $ and $$ are balanced; literal dollar signs are &#36; (c)
  • [ ] every footnote marker has a definition on the same page (c)
  • [ ] every link to another chapter names a chapter in book.yml, and its anchor exists (c)
  • [ ] a table has a header row and a separator row, with a blank line before it
  • [ ] quotations are attributed and figures have captions
  • [ ] every viz block names an existing viz/<name>/index.html, every steps frame exists, videos are MP4 or WebM, and nothing in viz/ loads from another site (c, section 11)

Beyond the checker, the library's own build (python tools/build.py, then python tools/check.py --build) must report no issues for the whole site.

9. Attaching a book

From the product repository (the one with library.yml):

python tools/books.py check ../my-book                         # validate
python tools/books.py attach ../my-book --collection courses   # a folder: used where it is
python tools/books.py attach https://github.com/me/my-book.git --collection courses
python tools/build.py                                          # pages
python -m mkdocs serve                                         # read it at http://127.0.0.1:8000

attach fetches the repository (a git address is cloned into books/), reads pustakalaya.yml, adds an entry under repos: in library.yml (with the collection), checks the library and prints the books it found with their order and address. If the check fails, library.yml is put back. --private attaches the books as not public. The collection is one of the collections of library.yml; without --collection, the repository's own suggestion or the collection more is used. To update a book, commit to its repository and run python tools/books.py sync, then python tools/build.py, or run python tools/publish.py, which also checks and deploys.

10. Generating a book: a minimal example

A generator writes three kinds of file: the chapters, book.yml, and pustakalaya.yml. This script makes a complete book from a list of titles and texts, then it can be checked and attached like any other.

#!/usr/bin/env python3
"""Write a Pustakālaya book from a list of (title, [(section title, markdown text), ...])."""
import json
import re
from pathlib import Path


def slug(text):
    return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") or "chapter"


def q(text):                                  # a YAML string
    return json.dumps(text, ensure_ascii=False)


def write_book(folder, book_id, title, author, chapters, description="", language="en"):
    root = Path(folder)
    (root / "chapters").mkdir(parents=True, exist_ok=True)
    (root / "images").mkdir(exist_ok=True)
    entries = []
    for n, (ctitle, sections) in enumerate(chapters, 1):
        fname = "chapters/%02d_%s.md" % (n, slug(ctitle))
        body = ["# " + ctitle, ""]
        for m, (stitle, text) in enumerate(sections, 1):
            body += ["## %d.%d %s" % (n, m, stitle), "", text.strip(), ""]
        (root / fname).write_text("\n".join(body), encoding="utf-8", newline="\n")
        entries.append("  - file: %s\n    title: %s\n    kind: chapter\n    number: %d\n    sections: %d\n"
                       % (fname, q(ctitle), n, len(sections)))
    (root / "book.yml").write_text(
        "title: %s\nauthor: %s\nlanguage: %s\ndescription: %s\nnumbering: sections\nchapters:\n%s"
        % (q(title), q(author), q(language), q(description), "".join(entries)),
        encoding="utf-8", newline="\n")
    (root / "pustakalaya.yml").write_text(
        "pustakalaya: 1\nformat: markdown\nbooks:\n  - id: %s\n    path: .\n    title: %s\n    author: %s\n"
        "    language: %s\n    description: %s\n    cover:\n      background: \"#1f3a5f\"\n"
        "      accent: \"#f2b84b\"\n      text: \"#f6f8fb\"\n"
        % (book_id, q(title), q(author), q(language), q(description)),
        encoding="utf-8", newline="\n")


write_book("my-book", "my_book", "My Book", "Ada Example",
           [("First Steps", [("Beginning", "Some text with $x^2$."), ("Going on", "!!! note \"Note\"\n\n    A box.")])],
           description="A book made by a script.")

Then:

python tools/books.py check my-book
python tools/books.py attach my-book --collection courses

Rules of thumb for generators:

  • Write pustakalaya.yml last, after the chapters, so that the section counts and the file list are known.
  • Escape the text of the source: a literal $ becomes &#36;, a line that begins with #, -, + or 1. and is not a heading or a list item gets a backslash, | inside a table cell becomes &#124;, < and > in text become &lt; and &gt; unless they are real HTML you intend.
  • Fence code with a language and use a fence longer than any run of backticks inside it.
  • Give each box a type from section 4 and indent its body by four spaces, blank lines included (an empty line in a box body may be empty).
  • Write images to images/ with ASCII names, scale rasters to 1600 px, and reference them as images/name.ext.
  • Keep ids stable between runs: the id is the book's address.
  • Run books.py check on the output as part of the generator's tests, and treat errors as bugs of the generator.

11. Animations and visualizations

Added in version 1.1. Optional, and nothing in sections 1 to 10 changes: a book without these blocks is a version 1 book.

A Markdown book can hold four kinds of moving figure. All of them work on the chapter pages and in the paginated reader, in light and dark, and are kept with the book for offline reading. Examples of each are in the template book, section 1.8.

You write You get Needs
![caption](images/clip.mp4) a video (muted, looping, with controls) an .mp4, .webm, .m4v or .ogv file in images/
a steps block frames with previous, next and play controls and a counter images in images/
a viz block an interactive visualization in a sandboxed frame a folder viz/<name>/ with an index.html
a mermaid block a diagram drawn from text nothing

Animated GIF, APNG, animated WebP and SVG files with animation (SMIL or CSS) are ordinary images (section 5): write them as figures and they move. Prefer a video for anything longer than a few seconds.

Video

![A short caption for the clip.](images/pipeline-fill.mp4 "Figure 4.2 Short label")
![A clip that plays while it is on screen.](images/loop.webm#autoplay "Figure 4.3 Short label")
  • The figure line is written as for an image (section 5). A file ending in .mp4, .webm, .m4v or .ogv becomes a <video> with controls, muted, looping, playsinline and preload="metadata".
  • A picture with the same name and the extension .jpg, .png or .webp (pipeline-fill.jpg) is the poster.
  • #autoplay after the file name plays the video while at least half of it is on screen and pauses it otherwise. It is never played automatically for readers who ask their device for reduced motion.
  • Use H.264 with AAC (MP4) or VP9 (WebM), at most 1280 pixels wide and 15 MB. MOV, AVI, MKV and the like are errors: browsers do not play them. A video is part of the offline copy of the book when the reader keeps the book, so keep clips short.

Step-through figures

```steps
title: Building a K-map grouping
label: Figure 2.4 Grouping the minterms
interval: 2000
- images/kmap-1.svg: Plot the minterms.
- images/kmap-2.svg: Group the pairs.
- images/kmap-3.svg: Group the quad and read the terms.
```
  • The block starts at a line of three backticks and the word steps, and ends at the closing fence.
  • The header is key: value lines: title (names the figure for screen readers), label and caption (printed under the figure, like a figure's label and caption; caption defaults to the title), interval (milliseconds between frames when playing, default 1800) and loop: true (play round and round). The value is the rest of the line, so it may hold colons.
  • Each frame is a line - images/name.svg: text. The text is the frame's caption and its description for screen readers. At least two frames. A frame may be SVG, PNG, JPEG, GIF or WebP; use frames of the same size.
  • The reader has previous, play and next buttons, a counter and a slider; the left and right arrow keys step through the figure when it has the keyboard focus. Without scripts all frames are shown one under the other.

Interactive visualizations

A visualization is a small web page that belongs to the book. It is kept in its own folder:

my-book/
  viz/
    pipeline/
      index.html          the visualization: HTML, CSS and JavaScript, self-contained
      poster.png          optional picture shown before it is opened

and is embedded with a viz block:

```viz
src: viz/pipeline/index.html
title: Five-stage instruction pipeline
label: Figure 13.4 Hazards and stalls
caption: Step through the pipeline cycle by cycle.
height: 420
poster: viz/pipeline/poster.png
```
Key Meaning
src Required. viz/<name>/index.html; the file must exist.
title Names the figure (the card, the frame, the full-screen view). Should be set.
caption, label Printed under the figure, like a figure's caption and label. Say what the reader can do.
height The height of the frame in pixels, 200 to 1200, default 420. A visualization that calls VizKit.autoHeight() (below) is as tall as its content, and height only reserves the space until it has loaded.
poster An image in viz/ or images/, shown on the card before the visualization is opened.

How it is shown. On a chapter page the visualization is a frame that is loaded when it comes near the screen, with the buttons Full screen and Restart under it. In the paginated reader it is a card (the poster, the title and a Play button), because a reading page cannot hold a program; Play opens the visualization full screen. Escape, the Close button and the back button of a phone close it. Nothing runs until the reader asks for it.

What it may do. The frame is sandboxed (sandbox="allow-scripts", without allow-same-origin). The page has an opaque origin: it cannot read or write the library's storage or cookies, cannot reach the page around it, and cannot open windows or submit forms. So:

  • put everything the visualization needs in its folder: scripts, styles, images, data. The library fetches index.html, puts the files it names with <script src>, <link rel="stylesheet"> and <img src> into the page, and shows the result, so that it works offline. Write the script tags at the end of <body>;
  • load nothing from another site (no CDN, no web font, no https:// address). It would fail offline and is reported by the checker;
  • do not use localStorage, cookies or IndexedDB (they throw in the frame) and do not call fetch or XMLHttpRequest (the frame cannot reach the book's files; put data in the script);
  • keep it small: each file under 600 KB, the folder under 4 MB. It is part of the offline copy of the book;
  • make it work from the keyboard, from 320 pixels wide, and in light and dark.

The helper viz-kit.js. Include it with <script src="viz-kit.js"></script>. The library puts its own copy there (copy tools/assets/viz-kit.js next to your file to try the page in a plain browser; the copy in the book is never used). It provides, as window.VizKit:

VizKit.palette, VizKit.theme the colours of the page that shows the figure (bg, fg, muted, line, surface, accent, accent2, ok, bad, warn) and the theme, "light" or "dark". The same colours are the CSS variables --viz-bg, --viz-fg, ... and VizKit.series(i) gives distinguishable colours for charts.
VizKit.onTheme(fn) fn(palette) now and whenever the reader changes the theme or the type size. Redraw canvases here; HTML and SVG that use the variables need nothing.
VizKit.full, VizKit.reader whether the figure is shown full screen, or inside the paginated reader.
VizKit.autoHeight() the frame takes the height of the content.
VizKit.stepper(el, {count, interval, loop, onStep}) restart, previous, play/pause and next buttons with a counter and a speed slider, for anything that runs in steps; returns {go, next, prev, play, pause, index}.
VizKit.el(tag, attrs, children) a small DOM helper.

The kit also adds base styles for the body, buttons, inputs and tables (using the colours above) and sets the type size of the page to follow the reader's type size, so sizes written in rem scale with it.

The protocol (for a visualization that does not use the kit). The page and the frame talk with postMessage. Every message is an object with bjpViz: 1 and a type. The frame must check that event.source === window.parent.

Direction Message
page to frame {type: "theme", theme: "light" or "dark", bg, fg, muted, line, surface, accent, accent2, fontSize, full, reader}: colours as CSS colours, fontSize in pixels. Sent when the frame has loaded, when it answers ready, and whenever the theme or type size changes. The first theme is also in the data-viz-init attribute (JSON) of <html>.
frame to page {type: "ready"} when the frame is ready for the theme.
frame to page {type: "height", value: 640}: the height wanted, in pixels (160 to 1600; ignored full screen).
frame to page {type: "close"}: close the full-screen view (the kit sends it on Escape).

Diagrams as text

```mermaid
%% label: Figure 1.6 The path of a book
%% caption: A book is written, checked, attached and read.
flowchart LR
    W[Write] --> C[Check] --> A[Attach] --> R[Read]
```

The block is mermaid syntax: flowcharts, sequence diagrams, state machines, class diagrams, Gantt charts. The first lines, if they are %% label: and %% caption:, caption the figure. The diagram is drawn in the reader's browser with a copy of mermaid that the library ships (it is loaded only on a page with a diagram, and kept with the book for offline reading). The text is shown as it is if the diagram cannot be drawn. For a figure that has to look exactly right, draw an SVG.

What books.py check tests

  • Errors: a viz block without src, with a src that is not viz/.../*.html or does not exist, or with a poster that does not exist; a steps block with fewer than two frames, a frame that is not written - images/file: text or whose file does not exist; a video in a format that browsers do not play; a folder viz/<name>/ without index.html; a script, style sheet or image of a visualization that does not exist.
  • Warnings: unknown keys; a viz block without title or caption, or with a height outside 200 to 1200; a frame without a caption; a video over 15 MB; a visualization file over 600 KB or a folder over 4 MB; a visualization that loads anything from another site, uses browser storage, or makes network requests.

Blocks written as examples inside a longer code fence (four backticks, as above) are not blocks and are not checked.

12. Category, level and kind

Added in version 1.2. Optional and backward compatible: a book without these keys builds as before. A library asks for at least one category per book, and can give it in its own library.yml when the repository does not.

Readers find books by category and subcategory: the Categories button in the header of every page, Browse categories on the front page, a page for each category and subcategory (subjects/<id>.html) listing its books as covers or as a compact list with filters for level, kind and language, and All books (books.html), where a reader chooses a category, then a subcategory. Three keys of a book in pustakalaya.yml describe it:

books:
  - id: linear_algebra
    title: Linear Algebra
    subjects: [algebra-geometry, statistics]   # subcategories from the list below, the main one first
    level: intermediate
    kind: textbook
    tags: [matrices, eigenvalues]

Choosing the category. Find the category of the book in the list below, then the subcategory that fits it best, and write its id (the code in backticks) first under subjects. A book that belongs to more than one field may name one or two more subcategories, of the same category or of others: it is then listed under each. Name a category itself (mathematics) only when none of its subcategories fits. The key is called subjects for compatibility with earlier versions; its values are these ids.

Key Values Meaning
subjects ids from the list below The categories of the book, the main one first: the book is listed under each subcategory named and under its category. Usually one to three.
level beginner, intermediate, advanced Beginner: no background needed beyond school; Intermediate: for readers who know the basics of the field, such as university students; Advanced: for specialists and practitioners.
kind textbook, course, field-guide, reference, workbook, series-volume, classic-text Textbook: teaches a subject chapter by chapter, usually with exercises; Course: follows a course or syllabus lesson by lesson; Field guide: a broad survey in plain language, to look things up and get oriented; Reference: a comprehensive work to consult; Workbook: mostly exercises and practice; Series volume: one volume of a numbered series; Classic text: a classic or primary text, such as a scripture, an epic or an older literary work.
tags words Free words a reader might type: names, abbreviations, other spellings (TPM, Ramayana). They are searched, not shown.

The categories and their subcategories

Readers see only the categories and subcategories that have books. A library can add categories or subcategories of its own (see "Categories and reading lists" in README.md of the product repository); a book may name them once the library has them. The list is kept in tools/subjects.yml.

Category Its subcategories (id: title)
computer-science: Computers & Technology programming: Programming
algorithms: Algorithms & Data Structures
web-development: Web Development
computer-architecture: Computer Architecture & Chips
operating-systems: Operating Systems
cloud-data-centres: Cloud & Data Centres
networks: Networks & Telecom
databases: Databases
artificial-intelligence: AI & Machine Learning
cyber-security: Cyber Security
quantum: Quantum Computing
science: Science physics: Physics
chemistry: Chemistry
biology: Biology
astronomy: Astronomy & Space
earth-environment: Earth & Environment
agriculture: Agriculture & Food
mathematics: Mathematics algebra-geometry: Algebra & Geometry
calculus: Calculus & Analysis
discrete-mathematics: Discrete Mathematics & Logic
statistics: Statistics & Data Science
optimization: Optimization
engineering: Engineering electronics: Electronics & Electrical
mechanical-engineering: Mechanical Engineering
civil-engineering: Civil Engineering
metallurgy: Materials & Metallurgy
aerospace-defence: Aerospace & Defence
medicine: Health & Medicine human-body: The Human Body
public-health: Public Health & Nutrition
psychology: Psychology & Mind
traditional-medicine: Traditional Medicine
history: History indian-history: Indian History
world-history: World History
history-of-science: History of Science & Technology
military-history: War & Military History
geography: Geography & Travel
biography: Lives & Memoirs
philosophy-religion: Philosophy & Religion philosophy: Philosophy
religion: Religion & Spirituality
sacred-texts: Scriptures & Sacred Texts
yoga: Yoga & Meditation
language-literature: Language & Literature english: English
french: French
indian-languages: Hindi & Indian Languages
sanskrit: Sanskrit & Classical Languages
languages: Linguistics & Grammar
literature: Novels, Plays & Stories
poetry: Poetry
childrens-books: Children's Books
world-languages: World Languages
epics: Epics & Mythology
arts: Arts & Music visual-arts: Painting & Sculpture
crafts: Crafts & Textiles
architecture: Architecture & Design
music-performance: Music, Dance & Theatre
social-sciences: Society, Law & Economics society-culture: Society & Culture
sociology: Sociology
law-polity: Law, Politics & Government
economics: Economics & Business
trade: Trade & Seafaring
anthropology: Tribal & Folk Cultures
education: Education teaching: Teaching & Learning
history-of-education: History of Education
iks: Indian Knowledge Systems iks-veda: Veda & Upaniṣad
iks-darsana: Darśana (Philosophy)
iks-traditions: Buddhist, Jain & Tantric Traditions
iks-yoga: Yoga
iks-itihasa: Itihāsa & Purāṇa
iks-kavya: Kāvya (Literature)
iks-sabda: Language & Grammar
iks-ganita: Jyotiṣa & Gaṇita
iks-ayurveda: Āyurveda
iks-science: Science, Farming & Geography
iks-arts: Arts & Crafts
iks-polity: Polity & Society
iks-mind: Mind & Education

Retired ids that are still accepted (the book is listed under the new one): chip-design (now computer-architecture), cryptography (now cyber-security), ecology (now earth-environment), embedded-systems (now electronics), hardware-security (now cyber-security), memory-storage (now computer-architecture), quantum-machine-learning (now quantum), security-operations (now cyber-security), telecommunications (now networks).

The same keys at the top of pustakalaya.yml are defaults for every book of the repository. A library may also set them in library.yml, for a collection (defaults for its books) or for one book (an override), without changing the repository.

What books.py check tests

  • Errors: subjects that is not a list of ids, a level or kind that is not one of the values above, tags that is not a list of words.
  • Warnings: an id that is neither in the list above nor among the library's own subjects, with the nearest id ("did you mean cyber-security (Cyber Security)?"); the library may still add it.
  • Notes: a book without subjects; a retired id (it still works: the book is listed under the id that replaced it).

13. Slides

Added with version 1.2 ("1.2-slides"). Optional and backward compatible: a book without these conventions is presented from its text, and nothing in sections 1 to 12 changes.

Every chapter of every book (Markdown, LaTeX or HTML) can be shown as slides: Present in the header of a chapter, in the reader's toolbar (or the key P), or on the book's page. An address with ?slides=1 (?slides=12 for slide 12) opens a chapter as slides directly. The slides are made in the reader's browser from the chapter as it is written:

The chapter has The slides show
the # title, the italic line under it, the book's title and author a title slide in the book's cover colours
three or more ## sections a slide with the chapter's sections
a ## section a section slide, with its ### subsections
paragraphs the key sentence of each paragraph as a bullet point, at most about six bullets and sixty words on a slide; the full paragraphs are the speaker's notes
a list, a definition list its items (long items shortened to their first sentence)
a figure, a table, a code block, a large equation a slide each; a long table or listing goes on over several slides
a viz, steps or mermaid block, a video a slide each; a visualization runs in place, a step-through figure steps with the arrow keys
a box (!!!) a callout slide; an example box titled "Example ..." is revealed piece by piece
an exercise and its folded answer (section 4) a question slide; the answer appears with the next key press (and the right option of a multiple-choice question is marked)
a section titled Summary, Key points, Conclusion ... the closing slide, with the next chapter

So a book that follows sections 3 and 4 needs nothing more. Write a paragraph so that its first sentence states its point: that sentence is the bullet. Give tables and figures captions, and keep a code block to the lines that matter.

An author who wants more control writes one of the following.

A slide marker

<!-- slide: The four steps -->

1. Write the chapters.
2. Check the book.
3. Attach it.

<!-- /slide -->

The blocks after <!-- slide --> (or <!-- slide: Title -->), up to the next marker, a <!-- /slide -->, a ## or ### heading or a ::: slide block, are put together on one slide as written (not shortened), under the title given or the heading of the section. The comment is invisible on the page and in the reader.

A slide for the presentation only

::: slide "The four steps"

1. Write the chapters.
2. Check the book.
3. Attach it.

Notes: Point at the figure of section 1.3 while you go through the steps.

:::
  • The block starts at a line ::: slide, with an optional title in double quotes, and ends at a line :::.
  • It is a slide of its own at that place in the deck, and is not shown on the page or in the reader.
  • Inside, anything of section 4 may be used: paragraphs, lists, figures, tables, code, mathematics, boxes, viz and steps blocks. A first # or ## heading is the slide's title when the opening line gives none; other headings become bold lines.
  • A paragraph that starts with Notes: begins the speaker's notes, to the end of the block. They are shown in the speaker view (key S), with a timer and the next slide.

A deck of its own

A file in slides/ with the name of a chapter file replaces that chapter's slides:

my-book/
  chapters/
    03_querying.md
  slides/
    03_querying.md        the slides of chapter 3
# Querying with SELECT

Notes: Start with the question the chapter answers.

---

## The SELECT statement

- choose the columns
- filter with WHERE

![A query and its result.](images/select.svg)

---

## Ordering

`ORDER BY` sorts the result.
  • Slides are separated by lines of three dashes (---).
  • A slide's first # or ## heading is its title; a paragraph that starts with Notes: begins its notes.
  • Pictures are in images/, as for the chapters. The deck opens with the chapter's title slide and ends with its closing slide (the summary and the next chapter).

The template book (book-template) has a slide marker and a ::: slide block in section 1.9, and a deck of its own for the preface (slides/01_preface.md).

How it is built

tools/slides.py in the product repository writes a ::: slide block into the page as a hidden <div class="pk-slide" data-title="..."> (with a <div class="pk-slide-notes"> for the notes), and a slides file as a hidden <div class="pk-slides"> at the end of the chapter's page. assets/slides-deck.js reads them. A program that generates books may write these elements directly (section 10), with Markdown inside them.

What books.py check tests

  • Errors: a ::: slide block without its closing :::.
  • Warnings: a file in slides/ without a chapter file of the same name in book.yml; a slides file without slides.

LaTeX and HTML books are presented from their text; the conventions of this section are for Markdown books.


Version 1.2. The format changes only by adding optional keys and conventions; pustakalaya: 1 stays valid, and a book written for version 1 builds unchanged. Version 1.1 added section 11 (video, steps, viz and mermaid blocks); version 1.2 added section 12 (subjects, level, kind and tags). tools/importers/common.py, tools/mdbook.py and tools/vizblocks.py in the product repository are the reference implementation.

13. Linked books

A linked book is a freely licensed book that lives on another site and is shown in the library in its own format and reader, without its text being stored in the library. The library keeps:

  • the catalogue entry in library.yml (title, author, cover colours, description, licence, attribution);
  • the contents: the chapters' titles, the source's names of their pages and their section headings. This is metadata, fetched when the site is built (tools/remote.py) and cached in the git-ignored .cache/remote/<id>.json;
  • one empty page per chapter (a shell) at the library's own address, with the chapter's heading and the headings of its sections, so that the navigation, the book's contents, the reader's "Whole book" tab and the shelf work as for any other book.

When a reader opens a chapter, assets/remote.js fetches its text from the source's API in the reader's browser, removes the source site's chrome (edit links, navigation boxes, templates, page numbers), converts it to the library's chapter HTML and fills the shell; the page and the reader then show it like any other chapter. The address stays on the library's site: nothing of the source site is embedded or opened, and no reader is sent there. Sections are numbered as in the contents, figures get their captions, tables, code (with its language), mathematics (typeset by MathJax), footnotes and pictures (loaded from the source's image servers, with their sizes) are kept, links to other chapters of the book lead to the library's pages, and other links open in a new tab when they are clicked. Each chapter ends with its attribution: the page it comes from, its contributors and history, the licence and the note "Modified: converted to this format". These links, and those of the badge on the book's page, are the only links to the source.

Sources. Only sources whose API allows a page of another site to read it (CORS) can be linked:

type Site Contents rules
wikibooks <lang>.wikibooks.org (CC BY-SA 4.0) collection, links, subpages, list
wikisource <lang>.wikisource.org (public-domain texts) links, subpages, list
standardebooks the public GitHub repository of a Standard Ebooks edition (CC0) toc
archive scanned books of archive.org (public domain in India, judged title by title) page ranges, or chapters

Project Gutenberg and LibreTexts do not allow it and cannot be linked.

EPUBs on GitHub (standardebooks). The editions of Standard Ebooks (carefully produced public-domain texts, many of them transcriptions first made by Project Gutenberg) are dedicated to the public domain (CC0), and the repository of each one serves its files with Access-Control-Allow-Origin: * at raw.githubusercontent.com/standardebooks/<repo>/master/src/epub/. A linked book of this type needs only the repository's name:

  - id: pride-and-prejudice
    collection: literature-classics
    title: Pride and Prejudice
    author: Jane Austen
    description: One sentence for the cover card.
    subjects: [literature]
    source: {type: standardebooks, repo: jane-austen_pride-and-prejudice}
    attribution: "Jane Austen, Pride and Prejudice (1813)"

The licence (CC0 1.0) and the book's page on Standard Ebooks (the url) have defaults. The contents are read at build time from the repository's toc.xhtml and content.opf: a table-of-contents entry with entries of other files below it is a group (a part or book of the work), entries that point into the entry's own file are its sections (the scenes of an act), entries before the half-title page are front matter and those after the body back matter; the title page, imprint, colophon and licence pages are left out. A chapter known only by its Roman number is called "Chapter I". Keys of source: repo, branch (default master), skip (file names, without extension, to leave out), numbering (none for plays: "Act I" without a number). When a chapter is opened, its XHTML file (and the book's endnotes.xhtml when the chapter has notes) is fetched in the browser and converted: the file's own heading is dropped (the page shows the chapter's title), verse and stage directions keep their layout, a table of speakers and speeches becomes a list of speeches, endnotes become footnotes, pictures are loaded from the repository, and links to the book's other chapters lead to the library's pages. Each chapter ends with its attribution: the edition, its source file on GitHub, "CC0 1.0 (public domain dedication)" and the original transcription it names (Project Gutenberg, Wikisource, ...), with links that open only when they are clicked. "Keep for offline reading" keeps the chapters' files and pictures on the device.

linked:
  - id: wb-data-structures               # the book's id (its address: <collection path>/<id>/)
    collection: open-textbooks
    title: Data Structures
    subtitle: Fundamental tools
    author: Wikibooks contributors        # or the historical author and the translator
    description: One sentence for the cover card.
    about: >-                             # optional: the longer text of "About this book"
      ...
    language: en                          # default: the source's language
    chapter_label: Chapter                # "Section", "अध्यायः", ...; empty for the title alone
    cover: {background: "#1d4e5f", accent: "#f2b544", text: "#f4f1e8"}
    subjects: [algorithms]                # level, kind and tags as for every book (section 12)
    source:
      type: wikibooks                     # wikibooks | wikisource
      lang: en                            # the language of the site
      root: Data Structures               # the book's main page on the site
      contents: collection                # see below
    licence: {id: CC BY-SA 4.0, url: "https://creativecommons.org/licenses/by-sa/4.0/"}
    attribution: "Data Structures, a Wikibooks textbook, by its contributors"
    url: "https://en.wikibooks.org/wiki/Data_Structures"     # the book on its site (for the attribution only)

licence (an id and a URL), attribution and url are required: the licences of these books ask that their authors are credited and the licence is named. The badge on the cover and the book's page reads "From Wikibooks · CC BY-SA 4.0".

Contents rules (source.contents):

  • collection: the book's saved collection, Wikibooks:Collections/<root>: a ;Heading line groups the pages listed after it (:[[Page|Title]]); a group of two or more chapters becomes a part of the contents.
  • links: the links of the root page to its subpages, in order; a level-2 heading or a line in bold names a group.
  • subpages: all the subpages of the root, ordered by their number (सेवासदन/१, सेवासदन/२, ...).
  • list: an explicit list, chapters: [{page: ..., title: ..., group: ...}].

Other keys of source: exclude (pages to leave out), front (pages shown unnumbered before the chapters), numbering (sequence, the default; group: I.1, I.2, ... within each group; none: chapters by title only), digits: devanagari (chapter numbers in Devanagari), parts: links (a chapter whose page links to two or more of its own subpages is made of them: each subpage becomes a section of the chapter, as the hymns of a book), sections: false (no section headings in the contents), math: true (the book has mathematics: MathJax is loaded early and kept for offline reading) and drop (CSS selectors of elements of the source's pages to remove). A chapter whose title is only a number is called by the label and the number ("अध्याय १").

Offline. "Keep for offline reading" keeps the shells like any book's pages; for a linked book the source's answers for every chapter, and its pictures, are then kept on the reader's device as well (never in the library), and removed with the book.

Search. The library holds no text of a linked book: the titles of its chapters and sections are found by the search, and the reader's search of the book's text is not offered.

Commands. python tools/remote.py list | refresh [id ...] | show <id>; python tools/build.py fetches the contents of a linked book whose cache is missing or older than 30 days, and python tools/books.py sync fetches them all again. python tools/check.py verifies that every linked book has its licence and attribution, that its chapter pages are shells (no text of the book in the site) and that its search index holds only headings.

Scanned books: the archive source

A book that exists as scanned pages on the Internet Archive (the books of the Archaeological Survey of India and the Digital Library of India, in.gov.ignca.* and in.ernet.dli.*) is linked in the same way, with source.type: archive. Its catalogue entry carries the basis of its public-domain status; its licence and attribution are filled in from the item (the licence is "Public domain in India", the attribution the digitisers and the host), so the entry needs neither. Only titles in the public domain in India are linked: published by 1940 and, where the author or editor is known, dead before 1965 (life plus sixty years); a title that is in doubt is left out.

linked:
  - id: ayurvedasutram
    collection: rare-books
    title: Āyurvedasūtram
    author: R. Shama Sastry (editor)
    language: sa
    chapter_label: ""                     # chapters are page ranges: "Pages 1-25"
    kind: rare-book                       # the kind of a scanned book (section 12)
    subjects: [traditional-medicine, sanskrit]
    source:
      type: archive
      identifier: in.gov.ignca.8087       # the item on archive.org
      pd_basis: "Published 1922; edited by R. Shama Sastry (died 1944); public domain in India"
      display: scans                      # text | scans (default: automatic, see below)
      chapter_pages: 25                   # scanned pages in a chapter (default 25)
      chapters:                           # optional: the chapters by scan number, instead of ranges
        - {title: Introduction, from: 1, to: 30}
  • identifier and pd_basis are required. The build asks archive.org/metadata/<identifier> for the title, creator, date, language, the names of the files and the number of scanned pages, and the image service for the size of a page; the results are cached in .cache/remote/<id>.json. Nothing else of the item is kept: no image, and no text but the library's own reading of an item that has none (below).
  • Chapters are page ranges ("Pages 1–25", "Pages 26–50", ...), each a shell like those of the other linked books, or the chapters the entry lists in chapters (title, from, to: scan numbers, the first scan being 1).
  • Display. In scans mode a chapter is its pages as images; in text mode it is the OCR text of its pages, cleaned lightly (running heads and folios removed, hyphenated words joined, lines made paragraphs) and set in the library's typography, with a "View page scan" button on every page. text needs the item's *_hocr_searchtext.txt.gz and *_hocr_pageindex.json.gz; by default a book is shown as text when its language is in the Latin script (English, Latin, French, German and the like), it has them, and a sample of its text reads well: at least 92% clean words (letters with a vowel) and at most 1.5% broken ones (stray signs, letters mixed with digits: what the OCR makes of Devanagari, of diacritics it cannot read and of formulas); otherwise as scans (Sanskrit and Hindi in Devanagari cannot be read by the OCR; a book that mixes English with Devanagari, or tables of figures, read better as scans). display: overrides the choice (for a book whose language is wrongly recorded).
  • The library's own reading. Many English items (most of those of the Digital Library of India) have no OCR text on the Internet Archive. python tools/ocr_archive.py --list | --book <id> | --all reads their scans with Tesseract (English, the LSTM engine, automatic page layout; each page fetched from the image service at about 2,000 pixels wide, at most one request a second, read and not kept; small scans are enlarged first; lines read with little confidence are dropped; the lines read poorly are read again with the Sanskrit model as well, and runs of words in Devanagari are taken from that reading, when --tessdata names a folder with eng and san; a book whose first forty pages read again show none is read in English only from then on) and keeps the text in a private repository of its own, named by app.ocr_repo in library.yml (its folder in library.local.yml local:, like the audio repository): <repo>/<book id>/pages.json, the text of every scan as read (a list, scan 1 first), and meta.json (the item, the engine and its settings, the date, the mean word confidence of each page and of the book, the shares of clean and of broken words). A run resumes where the last one stopped. Such a book is shown as text when the reading covers its scans and is good (the same 92% clean words, counted on the text in the Latin script, and 1.5% broken words as above, and a mean word confidence of 70), whatever language its item records; otherwise as scans. The build copies the text of each chapter into assets/ocr/<book id>/<from>-<to>.json (a list of the pages' texts) and assets/scans.js reads it from there instead of the Internet Archive's OCR files, cleaning it in the same way; the book's page and every chapter say that the text was read by machine for this library. A library without the repository shows these books as scans.
  • The viewer (assets/scans.js, scans.css) is the library's own: page images come from the image service of the Internet Archive (IIIF, which answers pages of another site), at a size that fits the screen and a larger one on zoom, with pinch, Ctrl+wheel, button and double-tap zoom, pan, fit to width or to the page, a jump to a page number, a strip of thumbnails, the keyboard (arrows, space, plus, minus, 0, f, t, g, Home, End, Escape), swipes, and light, dark or sepia surroundings. The Internet Archive's own reader is never embedded or opened; the links to the item (on the book's page and at the end of every chapter) open only when they are clicked.
  • Credit. The book's page and each chapter carry the digitisers' credit ("Digitised by the Indira Gandhi National Centre for the Arts / Archaeological Survey of India; hosted by the Internet Archive"), the item's own details and the public-domain basis (pd_basis).
  • Offline. "Keep these pages offline" on a chapter (the viewer has "Keep this page offline") stores the chapter's page images, their thumbnails and, for a text book, the text of the book in the browser's cache of the reader's device only (pk-scans-kept), after saying how much room it takes (about 0.2 MB a page); "Remove" takes them out again. The text read by this library (assets/ocr/) is also kept with the book by "Keep for offline reading".
  • Search. The library holds no text of these books (except the text it read itself, which is not indexed): their titles and authors are in the search of titles (search/titles.json), and the chapters' titles in the book's own index.
  • A text read from the scans. A scanned book of which a Markdown book holds the machine-read text (a book of the collection "Rare Texts") is paired with it under editions: in library.yml, by site paths (<collection path>/<book id>):
editions:
  - {scans: rare-books/my-scanned-book, text: rare-texts/my-scanned-book-text}

The first page of the scanned book then has the link "Read as text", and that of the text book "View the scans" (tools/editions.py). Inside the text book, a page opens its scan in the library's own viewer with a button <button type="button" class="pk-sc-view" data-book="<id of the scanned book>" data-n="<scan number>">View scan of page N</button> (put in a <div class="pk-sc-pn" markdown="0"> with the page number). The scan number is the position of the scan, the first scan being 1. The text of such a book says that it was read by machine and may contain errors.

14. Pronunciation, quizzes and language markup

Added with version 1.2 ("1.2-language"). Optional and backward compatible: a book without this markup builds as before.

Language books (and any other book) can let the reader hear words and sentences, answer questions in text boxes, choices and matching lists, and practise tables from memory. Everything works on the chapter page, in the paginated reader (typing in a box never turns a page) and in slides, offline, with the keyboard and with screen readers. tools/assets/speak.js, quiz.js, speak.css and quiz.css in the product repository bring it to life; tools/quizblocks.py writes the quizzes.

Phrases to hear

<span data-say lang="fr">Bonjour, madame !</span>
<span data-say="l'arbre, un arbre" lang="fr">l’arbre → un arbre</span>
<span data-say="Le samedi matin, Léa va à la boulangerie." lang="fr" data-rate="0.8">▶ the whole story</span>
  • A span with data-say gets a Listen (speaker) button after it. Without a value it speaks its own text; with a value it speaks the value (the spelling a voice reads best, or a longer passage behind a short label).
  • lang is the language of the voice (fr, en, hi, de, es, or a full tag such as en-GB). Without it the nearest lang around the span, or the page's language, is used.
  • data-rate slows a phrase down (1 is the normal speed of the language; French is spoken at 0.82, English at 0.9).
  • Recorded voices. The button plays a recording of the phrase in a native voice, the same on every phone and browser (many devices have no French voice of their own). A second tap on the button plays the phrase slowly (playbackRate 0.75, the pitch kept). The recordings are made offline by python tools/tts.py (see "Audio pipeline" below). The device's own voice (the Web Speech API) is used only if the reader chooses Device voice in "Voice and speed" above the chapter (also: a slower or faster speed, kept in the browser), or for a phrase that has no recording. The button is hidden only where there is neither a recording nor a device voice for the language; write the pronunciation in the text ([luh LEEVR]), so that the reader still has it.
  • Microphone. Where the browser can recognise speech, a microphone button (say the phrase, see how close it was) is shown only inside speaking exercises: an exercise box titled "Say it", "Speak" or "Say", and the say questions of a quiz. It is not put beside every phrase.
  • Play all. A list of three or more phrases (a dialogue, a word list) or a table with four or more phrases in its rows gets a Play all button above it, which plays them one after another (tap again to stop).
  • In a table, put the phrase in its cell; one small speaker per item, not a column of buttons. The importers of the two language books mark the language taught in the tables of words, phrases and examples, in the rows of conjugation tables (je parle, with the person) and in the lists of phrases (tools/importers/speakable.py); a cell in English, Hindi or Hinglish is never marked, so a French voice never reads English.
  • After a quiz question is checked, a speaker beside the feedback plays the model sentence (the answer, or the question with its blanks filled in) when it is in the language and has a recording.

Audio pipeline (recorded voices)

tools/tts.py collects every phrase of the built site (data-say spans and the quiz questions that are spoken: say, listen, dictation and the model answers), normalises the text (Unicode NFC, one space between words, the 🔊 mark removed) and makes one MP3 per unique (language, text) with Piper (MIT, offline, CPU): mono, 32 kbit/s, 22.05 kHz, named by the hash of the text, audio/<lang>/<sha1-12>.mp3. The recordings are kept in a separate repository of book assets, language-audio (library.yml app.audio_repo: language-audio, its folder in the git-ignored library.local.yml local:), so that a phrase is made once and a rerun makes only the new phrases. The build copies the recordings a site needs into site/assets/audio/<lang>/ with assets/audio/index.json ({lang: [hash, ...]}, read once by speak.js, which hashes a phrase itself with SHA-1 and looks it up). Without app.audio_repo, or without the folder, nothing is copied and every phrase plays with the device's voice.

Language Voice Source and licence
French fr_FR-siwis-medium rhasspy/piper-voices on Hugging Face; SIWIS French Speech Synthesis Database (University of Edinburgh); CC BY 4.0
English en_GB-cori-medium rhasspy/piper-voices; Hi-Fi TTS (NVIDIA, OpenSLR 109), the voice of the LibriVox reader Cori Samuel; CC BY 4.0

CC BY 4.0 allows the recordings to be redistributed with attribution: the voices and their sources are credited in the audio repository's README, which travels with the recordings.

To regenerate: install Piper and the MP3 encoder in a virtual environment (nothing system-wide): python -m venv D:/books/.work/tts/venv, venv/Scripts/python -m pip install piper-tts lameenc; build the site (python tools/build.py, python -m mkdocs build), then venv/Scripts/python tools/tts.py (--jobs N for the number of processes, --list to only count, --copy to copy the recordings into site/assets/audio/), and build again (python -m mkdocs build) so that the offline lists of library.json include them. The voices are fetched on first use into $PK_TTS_VOICES (default D:/books/.work/tts/voices).

Offline reading. A played recording is kept in the browser (Cache Storage pk-audio-v1) and plays offline the next time. "Keep for offline reading" also keeps a book's recordings when they total no more than app.audio_offline_mb (default 12 MB); a larger book (a whole language course in three editions) keeps what the reader has played. * The span may hold other inline Markdown (**bold**, other spans). Do not put a span around a whole paragraph of explanation: mark the phrase.

Quizzes

A fenced block quiz holds the questions of a quiz, written in YAML:

```quiz
title: Practice: gender and plurals
lang: fr
items:
  - type: choice
    q: "___ télévision"
    hint: Look at the ending -sion.
    options: [le, la, les, "l'"]
    answer: la
    why: -tion and -sion nouns are feminine.
  - type: blanks
    q: un gâteau → des ___
    answer: [gâteaux, gateaux]
  - type: text
    q: The French for "the book"?
    answer: [le livre]
  - type: match
    q: Match each singular to its plural.
    pairs: [[un œil, des yeux], [un journal, des journaux], [un prix, des prix]]
  - type: say
    q: Say it aloud, with the liaison.
    answer: Les amis sont dans la maison.
```
Key Meaning
title A line above the questions (optional; the section heading usually names the quiz).
lang The language of the answers and of the phrases to hear (say, listen, dictation).
labels The words of the page version in the book's language: exercise, answer, hint, words, wrong_word, corrected, say, aloud, dictation, transcript (English by default).
ui The words of the live quiz (check, show, retry, right, wrong, ...), for a language other than English and Hindi, which the library has.
number The number of the first exercise (default 1).
items Required. The questions, in order.

Each question has a type and the keys of its type; hint and why (shown after the check and in the answer) are optional for all.

type Keys The reader
text q, answer types the answer in a box
blanks q with ___ for each blank, answer types into boxes inside the sentence. With one blank, answer is the answer or a list of accepted answers; with several, a list with one entry (an answer or a list) per blank
choice q, options, answer picks one option
match q, pairs chooses, for each left item, its partner from a list
order q, answer (the sentence), optional tokens taps the words into the right order
spot q, sentence, answer (the wrong word), optional correction taps the wrong word
say q, answer hears the model, says it; the browser checks it where it can recognise speech, otherwise the reader marks it
listen q, audio, options, answer hears a passage (hidden until answered) and picks an option
dictation q, audio, optional answer hears a passage and writes it down; the words missed are marked

mcq, fill, reorder and speak are accepted for choice, blanks (or text, without a blank), order and say: the types of the quizzes of The English Book and The French Book.

Checking. Answers are compared without regard to case, punctuation and spaces. An answer typed without its accents is accepted, and the reader is told which accent was missing ("Correct, but mind the accents: gâteaux"); a slip of one letter in a long word is accepted with the exact spelling shown. Every question has Check (or is checked when an option or word is tapped), Show answer (which opens the folded answer) and Try again; the quiz shows its score, and the answers given and the best score are kept in the browser (localStorage), so the quiz is as the reader left it.

On the page without scripts, in print and in slides each question is an exercise box with its answer folded under it, as in section 4 (so a slide shows the question and reveals the answer with the next key press).

What books.py check tests. Errors: a block that is not valid YAML, without items, a question of an unknown type, without the keys of its type, a blanks question without ___ or with a wrong number of answers, a choice question whose answer is not an option, a match question with fewer than two pairs. Unknown keys are errors too.

Typed answers for exercises

An exercise and its folded answer (section 4) can get a Type your answer box above the answer, checked on demand (the folded answer stays):

??? note typed "Answer"

    **Le livre.**
  • The class typed after the box type asks for the box. In a book whose kind is course (section 12) every exercise gets it, unless its answer box has the class untyped.
  • The box is offered only when the answer is short: the first paragraph of the answer, or its bold part, is a word, a number or a phrase of up to four words (alternatives separated by / or or). Longer answers are left as they are.

Practice tables

<div class="pk-drill" markdown="1">

| Person | parler | finir |
|---|---|---|
| je | parl**e** | fin**is** |
| tu | parl**es** | fin**is** |

</div>

A table inside <div class="pk-drill" markdown="1"> gets a Practice switch. It hides the forms that the table highlights (bold text, verb colours) or, without highlights, the cells after the first column (not those of pronunciation, meaning or examples), and turns them into text boxes; Check marks each box, Show answer fills in the rest. A blank line after the opening tag and before </div> is needed.

Colours of language books

Class For Shown
g-m, g-f, g-pl masculine, feminine, plural words blue, pink, violet
r-noun, r-verb, r-adj, r-adv, r-prep, r-pron, r-conj, r-interj word classes a coloured highlight each
snd-mute, snd-liaison silent letters, liaisons faded and struck through; bold in the accent colour

Write them as spans: <span class="g-f">la table</span>. A key to the gender colours is a paragraph <p class="g-legend"><span class="g-m">masculine</span><span class="g-f">feminine</span><span class="g-pl">plural</span></p>. The colours follow the light and dark themes and the reader's page colours.

The chapter map

A figure <figure class="fig pk-cmap"> with an SVG drawing (a hub and the chapter's sections, each a link <a href="#anchor">) and an ordered list of the same links shows the drawing on a wide page and the list on a phone. The importers of The English Book and The French Book write one at the start of every chapter, as their readers drew it.

HTML books: what the importers add

The importers english and toc-book (section 7) turn every Listen button of the source into a phrase to hear (the text spoken is the button's), the microphone exercises into say questions, the quizzes into quiz blocks with their questions and answers as written, conjugation tables (a person in the first column) into practice tables, the colour classes of the source into the classes above, and the reader's chapter map into a figure. Visualizations made for these books (a chart of sounds, timelines of tenses, a sentence builder, a numbers and time trainer) are kept in the product repository, in tools/importers/extras/<book>/ (map.yml names the chapter and the section of each), and are added to the book as viz blocks (section 11), so that the source repositories stay as they are.


Back to Publish your book