The Pustakālaya Book Format, version 1.2¶
This document is the specification of the format in which a book is handed to Pustakālaya, a library of books that are read page by page, online or offline. It is written for two readers: an author who writes a book by hand, and a program that generates books (a converter, a course tool, a script that turns notes into chapters). A book that follows this format can be attached to any Pustakālaya library and is published without further work.
A book is a git repository (or a plain folder). The library never changes it. Two things describe it:
pustakalaya.yml, which says what the repository contains (the book's id, title, author, cover, language);- the book itself, in one of three formats: Markdown (the normalized Markdown book, described in full below, the format to generate), LaTeX or HTML (described briefly, with pointers).
To check a book, run python tools/books.py check <folder or git address> in the product repository. To start one, run
python tools/books.py new <folder> --title "...", or copy the template repository (book-template). To attach one,
run python tools/books.py attach <folder or git address> --collection <id>.
Words in this document: must is checked by books.py check as an error (the book would not build or would display
wrongly); should is a warning.
Contents
- Repository layout
- pustakalaya.yml
- The Markdown book: book.yml and the chapter files
- Markdown conventions
- Images and covers
- Languages
- LaTeX and HTML books
- What makes a book build cleanly
- Attaching a book
- Generating a book: a minimal example
- Animations and visualizations (new in 1.1)
- Category, level and kind (new in 1.2)
- Slides (new in 1.2)
- Pronunciation, quizzes and language markup (new in 1.2)
1. Repository layout¶
A repository with one Markdown book:
my-book/
pustakalaya.yml what the repository contains (section 2) required
book.yml the chapter list, in reading order (section 3) required
chapters/ one Markdown file per chapter required
01_preface.md
02_first-chapter.md
03_second-chapter.md
images/ figures and photographs the chapters use optional
workflow.svg
README.md for people browsing the repository optional
.gitignore editor files and the like optional
- File and folder names use letters, digits,
-,_and., no spaces. Names are case-sensitive: a book that works on Windows must also work on Linux, soimages/Fig1.pngis notimages/fig1.png. - All text files are UTF-8 (a byte order mark is tolerated), with
\nor\r\nline ends. - The repository holds no build output. The library builds the book; nothing in the repository is executed.
- A repository may hold several books, each in a folder of its own, listed in one
pustakalaya.ymlwith apath:for each (section 2). The layout above is then repeated per folder.
2. pustakalaya.yml¶
pustakalaya.yml is at the root of the repository. It is a YAML mapping with a list of books.
pustakalaya: 1 # schema version
format: markdown # default format of the books below
author: Ada Example # defaults for the books below
language: en
books:
- id: my_book
path: .
title: My Book
subtitle: A short line under the title
description: One or two sentences for the cover card and the book's first page.
chapter_label: Chapter
subjects: [programming, algorithms] # section 12: subcategories from the list, the main one first
level: beginner
kind: textbook
cover:
background: "#1f3a5f"
accent: "#f2b84b"
text: "#f6f8fb"
Top-level keys¶
| Key | Type | Default | Meaning |
|---|---|---|---|
pustakalaya |
integer | The schema version. Must be 1 when present; should be present. |
|
format |
latex | markdown | html |
latex |
Default format of the books. A Markdown book must say markdown, here or on the book. |
main |
text | main.tex |
Default LaTeX main file. |
author, affiliation, generated_by, language, chapter_label, conventions, subjects, level, kind, tags |
as on a book | Defaults for every book that does not set its own. | |
collection |
text | The collection the books would like to be in. A suggestion: the library decides. | |
books |
list | Required. At least one book. |
Keys of a book¶
| Key | Type | Default | Meaning |
|---|---|---|---|
id |
text | Required. Letters a-z, digits, _ or -; starts with a letter or digit. Unique in the repository and in the library. It is the book's folder in the site (<collection>/<id>/), so it is never changed after publication: readers' saved places and links depend on it. The names assets, search, img, index, about, offline, sitemap, library, sw, install, publish, format, books, subjects, lists are reserved. |
|
path |
text | . |
The book's folder inside the repository. Must stay inside the repository. |
format |
latex | markdown | html |
the top-level format |
See sections 3 to 7. |
title |
text | the id, in capitals | The book's title. Should be set. |
subtitle |
text | empty | A line under the title, on the cover and the first page. |
description |
text | empty | One or two sentences, on the cover card and under the title. Should be set; keep it under 500 characters. |
author |
text | the owner of the library | Author's name, on the cover. Always set it. |
affiliation |
text | empty | The author's institution. |
generated_by |
text | empty | For a book whose author is shown as "AI generated": the name of the person behind it. The engine writes "Generated by author. |
language |
text | en |
Language code (en, hi, sa, ta, ...). Sets the page language, fonts and hyphenation (section 6). |
chapter_label |
text | Chapter |
What a numbered chapter is called in headings and contents: Chapter, Lesson, Unit, Part, or a word in the book's language. |
cover |
mapping | maroon cover | Section 5. |
conventions |
mapping | LaTeX books only (section 7). | |
main |
text | main.tex |
LaTeX books only: the root file. |
importer, edition, file |
text | HTML books only (section 7). | |
contents |
list of paths | Plain Markdown books only: the chapter files, in order, instead of detecting them. | |
collection |
text | the library entry's | Overrides the collection for this book. |
order |
integer | order of listing | Position of the book in its collection. Books are ordered as listed unless order is set. |
public |
boolean | true |
false keeps the book out of the published site (it can still be read in a local private build). |
published_elsewhere |
boolean | false |
With public: false: the book is published on another portal (a book kept on a portal's reserve shelf, for example), so its title may be cited in the text of the books of this portal; check.py then looks only for its addresses. |
subjects |
list of subject ids | Section 12. The categories of the book: ids of subcategories (or of a category) from the list in section 12, the main one first. Should be set. | |
level |
beginner | intermediate | advanced |
Section 12. Who the book is written for. | |
kind |
textbook | course | field-guide | reference | workbook | series-volume |
Section 12. What sort of book it is. | |
tags |
list of words | Section 12. Words that help a search find the book (names, abbreviations, other spellings). |
Unknown keys are reported as warnings: they are usually typos.
A repository that cannot hold the file (it is not yours to change) is described by an inline manifest: in the
library's library.yml, with the same keys. When there is neither a file nor a manifest, the repository is
auto-detected: a normalized Markdown book (book.yml and chapters/), a LaTeX main file, plain Markdown, or an HTML
book. Auto-detection is a convenience; write the file, so that the id, the title and the cover are yours.
3. The Markdown book: book.yml and the chapter files¶
This is the format to generate. A book folder holds book.yml, chapters/ and images/.
book.yml¶
title: My Book # informational; the library takes title and author from pustakalaya.yml
subtitle: A short line
author: Ada Example
language: en
description: >-
A fuller text about the book. It is shown on the book's first page, under "About this book".
numbering: sections
chapters:
- file: chapters/01_preface.md
title: Preface
kind: front
sections: 3
- file: chapters/02_first-chapter.md
title: The First Chapter
kind: chapter
number: 1
label: Part I
sections: 5
- file: chapters/09_glossary.md
title: Glossary
kind: back
sections: 1
| Key | Type | Meaning |
|---|---|---|
chapters |
list | Required, not empty. The pages of the book in reading order. The order of this list is the order of the book; the file names do not decide it. |
description |
text | The fuller description on the book's first page. The one-line description on the cover card is description in pustakalaya.yml. |
title, subtitle, author, language, numbering |
Informational. numbering: sections says that sections carry the numbers written in their headings. |
|
id, collection, importer, source |
Written by the converters of the library; ignored. |
Each entry of chapters:
| Key | Type | Meaning |
|---|---|---|
file |
path | Required. The chapter's file, relative to the book folder; normally chapters/NN_slug.md. Must exist, and be listed once. |
title |
text | Required. The title in the contents and the navigation. Should equal the # heading of the file. |
kind |
front | chapter | back |
Default chapter. front: unnumbered pages before the chapters (preface, how to read this book). chapter: a numbered chapter. back: unnumbered pages after the chapters (glossary, bibliography, solutions). |
number |
integer | Required for chapter, ignored otherwise. The chapter's number: 3 makes the page heading "Chapter 3: Title". Numbers should increase. |
label |
text | A label for the contents: Unit 2, Part III. See Grouping below. |
sections |
integer | The number of ## sections of the file. Informational. |
Chapter file names. Name the files NN_slug.md: two digits (three for a book of more than 99 pages), an underscore,
a lowercase slug of letters, digits and hyphens. The slug, without the number, is the page's address
(<book>/first-chapter.html), so slugs should be unique; a repeated slug gets -2. Do not use index, about,
bibliography or offline as a slug. Use ASCII slugs even for books in other scripts.
Kinds. A book needs at least one chapter. The contents list the pages under Front matter, Chapters and
Back matter.
Grouping. When two or more chapters share a label, the contents group the chapters by label ("Unit 1", "Part II").
Chapters with a label of their own then form a group of one. Without shared labels, the label is not used.
The first page. The library writes the book's first page itself, from the descriptions and the contents (every
chapter, with its sections and subsections). The front chapters follow it as ordinary pages; the reader opens at the
first front chapter or, without one, at the first chapter.
4. Markdown conventions¶
The text is Markdown as understood by Python-Markdown with these extensions: tables, fenced code with highlighting,
attribute lists, definition lists, footnotes, admonitions, folded details, Markdown inside HTML, and MathJax
mathematics. Each convention below has an example; the template book (book-template) uses all of them.
Chapter structure¶
A chapter file starts with one title line and then sections:
# Querying with SELECT
*Filtering, operators and ordering*
## 3.1 The SELECT statement
Text...
### Basic forms
Text...
## 3.2 The WHERE clause
# Titleis the first line of the file (blank lines may come before it). It has no number: the build writes the chapter's number (frombook.yml) into the page heading. Only one#heading.- A chapter must have at least one
##section. Sections of a numbered chapter are written## N.M Title, with the chapter numberNand a running numberM, typed by hand: the library does not renumber them. Front and back matter use unnumbered## Title. ###subsections and deeper headings are not numbered. The contents and the reader's "Go to" box list##and###.- Headings get anchors made from their text. To keep a link stable when a title changes, give the heading an explicit
anchor:
## 3.1 The SELECT statement {#select}(letters, digits,-,_). - Blank lines separate blocks. Paragraphs are plain lines;
*italic*,**bold**,`code`, lists with-or1.. A list or block that follows a paragraph needs a blank line before it.
Boxes (admonitions)¶
!!! warning "Common mistake"
The body is indented by four spaces. It may hold paragraphs, lists, code and mathematics.
- The header is
!!!, a type and an optional title in double quotes. A blank line before and after the box. - Types:
note,tip,info,warning,example,quote,abstract,success,question,failure,danger,bug. (summary,hint,important,caution,errorand a few more are accepted as aliases.) Other types are shown as plain boxes and reported as a warning. - Use
abstractfor "In brief" and chapter summaries,notefor learning outcomes,examplefor worked examples and exercises,quotefor quotations (give the source in the box),warningfor common mistakes,tipfor advice. ???instead of!!!makes the box folded until the reader opens it;???+makes it open at first.
Exercises with folded answers¶
An exercise is an example box titled "Exercise N"; its answer is a folded box that follows it directly.
!!! example "Exercise 1"
Name the file in which the list of chapters is kept.
??? note "Answer"
**`book.yml`.** It lists the chapters in reading order.
A short answer can also be typed and checked by the reader (??? note typed "Answer", section 14), and a whole set of
questions with text boxes, choices and matching is written as a quiz block (section 14).
Figures with captions and labels¶

- The figure goes on a line of its own. The text in brackets is the caption; the optional quoted text after the file name is the label, printed in bold before the caption ("Figure 1.1 Short label. The caption..."). Type the figure number into the label; it is not counted for you.
- The path is
images/name.ext. Write the caption: it is also the text for readers who cannot see the picture. - An image inside a sentence, or with any other path form, is shown as an ordinary image without caption.
Tables¶
| Element | Written as | Appears as |
|---|:---:|---|
| Section | `## 1.1 Title` | numbered section |
| Box | `!!! note "Title"` | coloured box |
*Table 1.1. The main elements of a chapter.*
Pipe tables with a header row. The second row sets alignment (:---, :---:, ---:). A cell is one line; use
<br> for a line break. A table wider than the page scrolls inside the page. Put the caption on a line of its own after
the table, in italics, with the number typed into it. A blank line before the table.
Code¶
```python
def mean(values):
return sum(values) / len(values)
```
Fence code with three backticks (or ~~~) and a language: any Pygments name (python, c, cpp, java, sql,
bash, json, yaml, verilog, ...); use text for output and anything that is not a programming language. To show a
fence inside a fence, use a longer outer fence (four backticks). Inline code is `like this`.
Mathematics¶
Write TeX between dollar signs. MathJax typesets it in the page.
A formula in a sentence, $E = mc^2$, and one on a line of its own:
$$
\int_0^1 x^2\,dx = \frac{1}{3}
$$
$...$is inline;$$...$$is display.\begin{align}...\end{align}and the other amsmath environments work in display mode.\( ... \)and\[ ... \]are accepted too.- Delimiters must be balanced in a paragraph. A dollar sign that is only a dollar sign is written
$("the price was $5"). - Mathematics is typeset from the MathJax CDN; the page shows the TeX until it has loaded.
Footnotes¶
A claim that needs a source.[^src]
[^src]: Author, *Title*, chapter 3.
The marker [^name] and its definition [^name]: text are on the same page; names are local to the page. Every marker
needs a definition and every definition a marker. The reader shows footnotes as pop-ups.
Cross-references¶
Link to another chapter by its file name, and to a heading by its anchor:
See [Writing a Chapter](02_writing-a-chapter.md), and [the section on figures](02_writing-a-chapter.md#figures).
Within a page: [section 2.1](#exercises).
file.mdis the file name frombook.yml(with or withoutchapters/or./); the link also works when the repository is browsed on a git host. The link must name a chapter that is inbook.yml.- Prefer explicit anchors (
{#figures}) for targets of links. The anchor of a heading without one is its text in lowercase, with punctuation removed and spaces turned into-(## 1.3 Figuresbecomes#13-figures). - References to numbered figures, tables and equations are typed as text ("Figure 1.1"); the label is the number.
- Links to other sites are ordinary Markdown links.
Glossary (definition lists)¶
Admonition
: A box with a title, written with `!!!` and a body indented by four spaces.
Front matter
: The unnumbered chapters before the first numbered one.
The term on a line of its own, then a line that starts with a colon and three spaces. A glossary is a back chapter with
one or more ## sections (## A, ## B, ...).
Blocks written in HTML¶
A converter that needs what the conventions above cannot say (a table with joined cells, a figure whose caption is
not its alt text, a table or figure inside a box) may write the block in HTML with markdown="1"; the Markdown inside is
then read as usual. The block starts at the beginning of a line, after a blank line; every element between the block
and a cell that holds Markdown carries markdown="1", and the cell itself markdown="span". A box is the HTML of an
admonition, a folded box that of ???. An id="..." on such a block is an anchor for links, like {#id}.
<figure class="fig fig-diagram" id="fig-2-1" markdown="1">
{: loading="lazy" }
<figcaption markdown="span">**Figure 2.1** The caption, with *emphasis* and $x^2$.</figcaption>
</figure>
<div class="admonition example" markdown="1">
<p class="admonition-title">Example 2.1 A box with a table</p>
<table markdown="1">
<tbody markdown="1">
<tr markdown="1"><td markdown="span" colspan="2">a cell with $y_1$</td></tr>
</tbody>
</table>
</div>
Hand-written books should keep to the conventions above; tools/importers/cnxml.py (open textbooks in CNXML) writes
these blocks.
Not allowed¶
- Raw
<script>and<style>elements. Nothing in a book is executed, and styles belong to the library. <iframe>,<object>and<embed>do not work offline and in the reader; avoid them. For a program that the reader can use, write avizblock (section 11).- Remote images (
https://...): they are not available offline. Put a copy inimages/.
5. Images and covers¶
- Folder and names. Images are in
images/, referenced asimages/name.ext. Use lowercase names with hyphens. Subfolders are allowed. - Formats. SVG for diagrams (small, sharp at every size, theme-neutral colours); PNG for screenshots; JPEG or WebP for photographs; GIF only when unavoidable. MP4 or WebM for video, and the other moving figures of section 11.
- Size. A raster image should be at most 1600 pixels wide and a file at most 2 MB; larger files slow the page and
the offline copy.
books.py checkwarns about both. Figures are scaled to the page. - Photographs and licences. For Markdown books every file in
images/is published as it is. Only include images that you may publish, and put the credit in the caption:. - Cover. The cover on the shelf is typographic: the collection name, the book's number if the collection numbers its books, the title, the subtitle and the author on a coloured background.
cover:
background: "#1f3a5f" # the cover
accent: "#f2b84b" # the rule and small details
text: "#f6f8fb" # the title and author
Colours are # followed by six hexadecimal digits, in quotes. Choose a text colour with a contrast of at least 4.5:1
against the background (the checker warns). The defaults are maroon, gold and cream.
A picture can be the cover instead: cover: {image: images/cover.jpg}, a JPEG, PNG or WebP file, relative to the book
folder, portrait (it is scaled to 600 pixels wide). With an image the title text is hidden on the
shelf, so the picture should carry the title.
6. Languages¶
Set language in pustakalaya.yml (en, hi, sa, ta, ...). It sets the lang of every page and of the reader,
which selects the fonts and the hyphenation rules: Devanagari text (hi, sa, mr) uses Noto Serif Devanagari when the
reader is online and the installed Devanagari faces otherwise, with hyphenation off and a larger line spacing.
- Write the text, the titles and the
chapter_label(for exampleअध्याय,पाठ) in the language of the book. - Keep file names, slugs and the
idin ASCII: they become addresses. - Headings in a non-Latin script get automatic anchors that are empty or identical. Give the headings that you link to an
explicit anchor (
## परिचय {#parichay}). - Mixed text (Sanskrit words in an English book) needs no markup; give transliteration in IAST if the book uses it.
- The words that the library itself shows (Contents, Read, Library) are in English.
7. LaTeX and HTML books¶
The Markdown book is the format to generate. The library also reads two other kinds of repository; they are described in
README.md, which is the reference for them.
LaTeX (format: latex). A book class (book, report, memoir, scrbook) with a main file (main.tex or
main: name.tex), \include or \input files, \frontmatter, \mainmatter, \appendix, \backmatter, and an
optional BibLaTeX database. Each \chapter becomes a page; boxes, TikZ figures, tables, code listings, mathematics and
bibliography are converted, and the numbers are those of the print edition. conventions: in pustakalaya.yml switches
on the generic conversion for books with their own macros and preamble (expand_macros, equation_numbering, slugs,
box_titles, strip_words, bibliography, preface_title, boxes). See "LaTeX conventions" and "Books of any layout"
in README.md. pandoc and a TeX distribution must be installed where the library is built.
pustakalaya: 1
format: latex
books:
- id: hardware_security
main: hardware_security_book.tex
title: Hardware Security
author: Ada Example
conventions: {expand_macros: true, equation_numbering: chapter, slugs: title}
HTML (format: html). Two layouts are understood: importer: english with edition: en | hinglish | hi (a folder
per edition with assets/toc.js; one repository gives three books) and importer: toc-book with edition: en | hinglish | hi (the same, for a course whose English edition is at the root of the
repository and whose other editions are in hinglish/ and hi/) and importer: guide with file: guide.html (a
single page with a cover section). Both are converted into a normalized Markdown book when the library is built, with
the phrases to hear, quizzes, practice tables and colours of section 14. A new HTML book should rather be generated as
Markdown.
Plain Markdown (format: markdown without book.yml). An mkdocs, mdBook or GitBook layout, or numbered chapter files
(01-intro.md, ...), with an optional contents: list. It is converted to a normalized Markdown book when the library
is built, and the result is only as good as the guesses made from the layout. Generating book.yml and chapters/ is
better.
8. What makes a book build cleanly¶
python tools/books.py check <folder> tests the points marked (c). The others are for you.
Repository and manifest
- [ ]
pustakalaya.ymlis valid YAML, haspustakalaya: 1,format: markdownand abooks:list (c) - [ ] every book has an
idthat matches[a-z0-9][a-z0-9_-]*, is unique and is not the id of another book in the library (c) - [ ]
title,author,descriptionandlanguageare set (c, as warnings) - [ ] the cover colours are
#rrggbbin quotes, and the text is readable on the background (c) - [ ]
pathexists and lies inside the repository (c)
book.yml and chapters
- [ ]
chapterslists every chapter file once, in reading order; no file exists inchapters/that is not listed (c) - [ ] every entry has
file(existing) andtitle;kindisfront,chapterorback; everychapterhas a wholenumber, increasing (c) - [ ] every file starts with
# Titleand has at least one##section (c) - [ ] section numbers
## N.Magree with the chapter number (c) - [ ] file names are ASCII, and case matches (c)
Content
- [ ] every image link is
images/...and the file exists, with the same upper and lower case (c) - [ ] raster images are at most 1600 px wide and 2 MB; remote images are not used (c)
- [ ] no raw
<script>or<style>(c) - [ ] every fenced code block has a language, and every fence is closed (c)
- [ ] every box is written
!!! type "Title"with a body indented by four spaces (c) - [ ]
$and$$are balanced; literal dollar signs are$(c) - [ ] every footnote marker has a definition on the same page (c)
- [ ] every link to another chapter names a chapter in
book.yml, and its anchor exists (c) - [ ] a table has a header row and a separator row, with a blank line before it
- [ ] quotations are attributed and figures have captions
- [ ] every
vizblock names an existingviz/<name>/index.html, everystepsframe exists, videos are MP4 or WebM, and nothing inviz/loads from another site (c, section 11)
Beyond the checker, the library's own build (python tools/build.py, then python tools/check.py --build) must report
no issues for the whole site.
9. Attaching a book¶
From the product repository (the one with library.yml):
python tools/books.py check ../my-book # validate
python tools/books.py attach ../my-book --collection courses # a folder: used where it is
python tools/books.py attach https://github.com/me/my-book.git --collection courses
python tools/build.py # pages
python -m mkdocs serve # read it at http://127.0.0.1:8000
attach fetches the repository (a git address is cloned into books/), reads pustakalaya.yml, adds an entry under
repos: in library.yml (with the collection), checks the library and prints the books it found with their order and
address. If the check fails, library.yml is put back. --private attaches the books as not public. The collection is
one of the collections of library.yml; without --collection, the repository's own suggestion or the collection
more is used. To update a book, commit to its repository and run python tools/books.py sync, then
python tools/build.py, or run python tools/publish.py, which also checks and deploys.
10. Generating a book: a minimal example¶
A generator writes three kinds of file: the chapters, book.yml, and pustakalaya.yml. This script makes a complete
book from a list of titles and texts, then it can be checked and attached like any other.
#!/usr/bin/env python3
"""Write a Pustakālaya book from a list of (title, [(section title, markdown text), ...])."""
import json
import re
from pathlib import Path
def slug(text):
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") or "chapter"
def q(text): # a YAML string
return json.dumps(text, ensure_ascii=False)
def write_book(folder, book_id, title, author, chapters, description="", language="en"):
root = Path(folder)
(root / "chapters").mkdir(parents=True, exist_ok=True)
(root / "images").mkdir(exist_ok=True)
entries = []
for n, (ctitle, sections) in enumerate(chapters, 1):
fname = "chapters/%02d_%s.md" % (n, slug(ctitle))
body = ["# " + ctitle, ""]
for m, (stitle, text) in enumerate(sections, 1):
body += ["## %d.%d %s" % (n, m, stitle), "", text.strip(), ""]
(root / fname).write_text("\n".join(body), encoding="utf-8", newline="\n")
entries.append(" - file: %s\n title: %s\n kind: chapter\n number: %d\n sections: %d\n"
% (fname, q(ctitle), n, len(sections)))
(root / "book.yml").write_text(
"title: %s\nauthor: %s\nlanguage: %s\ndescription: %s\nnumbering: sections\nchapters:\n%s"
% (q(title), q(author), q(language), q(description), "".join(entries)),
encoding="utf-8", newline="\n")
(root / "pustakalaya.yml").write_text(
"pustakalaya: 1\nformat: markdown\nbooks:\n - id: %s\n path: .\n title: %s\n author: %s\n"
" language: %s\n description: %s\n cover:\n background: \"#1f3a5f\"\n"
" accent: \"#f2b84b\"\n text: \"#f6f8fb\"\n"
% (book_id, q(title), q(author), q(language), q(description)),
encoding="utf-8", newline="\n")
write_book("my-book", "my_book", "My Book", "Ada Example",
[("First Steps", [("Beginning", "Some text with $x^2$."), ("Going on", "!!! note \"Note\"\n\n A box.")])],
description="A book made by a script.")
Then:
python tools/books.py check my-book
python tools/books.py attach my-book --collection courses
Rules of thumb for generators:
- Write
pustakalaya.ymllast, after the chapters, so that the section counts and the file list are known. - Escape the text of the source: a literal
$becomes$, a line that begins with#,-,+or1.and is not a heading or a list item gets a backslash,|inside a table cell becomes|,<and>in text become<and>unless they are real HTML you intend. - Fence code with a language and use a fence longer than any run of backticks inside it.
- Give each box a type from section 4 and indent its body by four spaces, blank lines included (an empty line in a box body may be empty).
- Write images to
images/with ASCII names, scale rasters to 1600 px, and reference them asimages/name.ext. - Keep ids stable between runs: the id is the book's address.
- Run
books.py checkon the output as part of the generator's tests, and treat errors as bugs of the generator.
11. Animations and visualizations¶
Added in version 1.1. Optional, and nothing in sections 1 to 10 changes: a book without these blocks is a version 1 book.
A Markdown book can hold four kinds of moving figure. All of them work on the chapter pages and in the paginated reader, in light and dark, and are kept with the book for offline reading. Examples of each are in the template book, section 1.8.
| You write | You get | Needs |
|---|---|---|
 |
a video (muted, looping, with controls) | an .mp4, .webm, .m4v or .ogv file in images/ |
a steps block |
frames with previous, next and play controls and a counter | images in images/ |
a viz block |
an interactive visualization in a sandboxed frame | a folder viz/<name>/ with an index.html |
a mermaid block |
a diagram drawn from text | nothing |
Animated GIF, APNG, animated WebP and SVG files with animation (SMIL or CSS) are ordinary images (section 5): write them as figures and they move. Prefer a video for anything longer than a few seconds.
Video¶


- The figure line is written as for an image (section 5). A file ending in
.mp4,.webm,.m4vor.ogvbecomes a<video>with controls, muted, looping,playsinlineandpreload="metadata". - A picture with the same name and the extension
.jpg,.pngor.webp(pipeline-fill.jpg) is the poster. #autoplayafter the file name plays the video while at least half of it is on screen and pauses it otherwise. It is never played automatically for readers who ask their device for reduced motion.- Use H.264 with AAC (MP4) or VP9 (WebM), at most 1280 pixels wide and 15 MB. MOV, AVI, MKV and the like are errors: browsers do not play them. A video is part of the offline copy of the book when the reader keeps the book, so keep clips short.
Step-through figures¶
```steps
title: Building a K-map grouping
label: Figure 2.4 Grouping the minterms
interval: 2000
- images/kmap-1.svg: Plot the minterms.
- images/kmap-2.svg: Group the pairs.
- images/kmap-3.svg: Group the quad and read the terms.
```
- The block starts at a line of three backticks and the word
steps, and ends at the closing fence. - The header is
key: valuelines:title(names the figure for screen readers),labelandcaption(printed under the figure, like a figure's label and caption;captiondefaults to the title),interval(milliseconds between frames when playing, default 1800) andloop: true(play round and round). The value is the rest of the line, so it may hold colons. - Each frame is a line
- images/name.svg: text. The text is the frame's caption and its description for screen readers. At least two frames. A frame may be SVG, PNG, JPEG, GIF or WebP; use frames of the same size. - The reader has previous, play and next buttons, a counter and a slider; the left and right arrow keys step through the figure when it has the keyboard focus. Without scripts all frames are shown one under the other.
Interactive visualizations¶
A visualization is a small web page that belongs to the book. It is kept in its own folder:
my-book/
viz/
pipeline/
index.html the visualization: HTML, CSS and JavaScript, self-contained
poster.png optional picture shown before it is opened
and is embedded with a viz block:
```viz
src: viz/pipeline/index.html
title: Five-stage instruction pipeline
label: Figure 13.4 Hazards and stalls
caption: Step through the pipeline cycle by cycle.
height: 420
poster: viz/pipeline/poster.png
```
| Key | Meaning |
|---|---|
src |
Required. viz/<name>/index.html; the file must exist. |
title |
Names the figure (the card, the frame, the full-screen view). Should be set. |
caption, label |
Printed under the figure, like a figure's caption and label. Say what the reader can do. |
height |
The height of the frame in pixels, 200 to 1200, default 420. A visualization that calls VizKit.autoHeight() (below) is as tall as its content, and height only reserves the space until it has loaded. |
poster |
An image in viz/ or images/, shown on the card before the visualization is opened. |
How it is shown. On a chapter page the visualization is a frame that is loaded when it comes near the screen, with the buttons Full screen and Restart under it. In the paginated reader it is a card (the poster, the title and a Play button), because a reading page cannot hold a program; Play opens the visualization full screen. Escape, the Close button and the back button of a phone close it. Nothing runs until the reader asks for it.
What it may do. The frame is sandboxed (sandbox="allow-scripts", without allow-same-origin). The page has an opaque
origin: it cannot read or write the library's storage or cookies, cannot reach the page around it, and cannot open windows
or submit forms. So:
- put everything the visualization needs in its folder: scripts, styles, images, data. The library fetches
index.html, puts the files it names with<script src>,<link rel="stylesheet">and<img src>into the page, and shows the result, so that it works offline. Write the script tags at the end of<body>; - load nothing from another site (no CDN, no web font, no
https://address). It would fail offline and is reported by the checker; - do not use
localStorage, cookies orIndexedDB(they throw in the frame) and do not callfetchorXMLHttpRequest(the frame cannot reach the book's files; put data in the script); - keep it small: each file under 600 KB, the folder under 4 MB. It is part of the offline copy of the book;
- make it work from the keyboard, from 320 pixels wide, and in light and dark.
The helper viz-kit.js. Include it with <script src="viz-kit.js"></script>. The library puts its own copy there (copy
tools/assets/viz-kit.js next to your file to try the page in a plain browser; the copy in the book is never used). It
provides, as window.VizKit:
VizKit.palette, VizKit.theme |
the colours of the page that shows the figure (bg, fg, muted, line, surface, accent, accent2, ok, bad, warn) and the theme, "light" or "dark". The same colours are the CSS variables --viz-bg, --viz-fg, ... and VizKit.series(i) gives distinguishable colours for charts. |
VizKit.onTheme(fn) |
fn(palette) now and whenever the reader changes the theme or the type size. Redraw canvases here; HTML and SVG that use the variables need nothing. |
VizKit.full, VizKit.reader |
whether the figure is shown full screen, or inside the paginated reader. |
VizKit.autoHeight() |
the frame takes the height of the content. |
VizKit.stepper(el, {count, interval, loop, onStep}) |
restart, previous, play/pause and next buttons with a counter and a speed slider, for anything that runs in steps; returns {go, next, prev, play, pause, index}. |
VizKit.el(tag, attrs, children) |
a small DOM helper. |
The kit also adds base styles for the body, buttons, inputs and tables (using the colours above) and sets the type size of the
page to follow the reader's type size, so sizes written in rem scale with it.
The protocol (for a visualization that does not use the kit). The page and the frame talk with postMessage. Every
message is an object with bjpViz: 1 and a type. The frame must check that event.source === window.parent.
| Direction | Message |
|---|---|
| page to frame | {type: "theme", theme: "light" or "dark", bg, fg, muted, line, surface, accent, accent2, fontSize, full, reader}: colours as CSS colours, fontSize in pixels. Sent when the frame has loaded, when it answers ready, and whenever the theme or type size changes. The first theme is also in the data-viz-init attribute (JSON) of <html>. |
| frame to page | {type: "ready"} when the frame is ready for the theme. |
| frame to page | {type: "height", value: 640}: the height wanted, in pixels (160 to 1600; ignored full screen). |
| frame to page | {type: "close"}: close the full-screen view (the kit sends it on Escape). |
Diagrams as text¶
```mermaid
%% label: Figure 1.6 The path of a book
%% caption: A book is written, checked, attached and read.
flowchart LR
W[Write] --> C[Check] --> A[Attach] --> R[Read]
```
The block is mermaid syntax: flowcharts, sequence diagrams, state machines, class diagrams, Gantt
charts. The first lines, if they are %% label: and %% caption:, caption the figure. The diagram is drawn in the reader's
browser with a copy of mermaid that the library ships (it is loaded only on a page with a diagram, and kept with the book for
offline reading). The text is shown as it is if the diagram cannot be drawn. For a figure that has to look exactly right,
draw an SVG.
What books.py check tests¶
- Errors: a
vizblock withoutsrc, with asrcthat is notviz/.../*.htmlor does not exist, or with a poster that does not exist; astepsblock with fewer than two frames, a frame that is not written- images/file: textor whose file does not exist; a video in a format that browsers do not play; a folderviz/<name>/withoutindex.html; a script, style sheet or image of a visualization that does not exist. - Warnings: unknown keys; a
vizblock without title or caption, or with a height outside 200 to 1200; a frame without a caption; a video over 15 MB; a visualization file over 600 KB or a folder over 4 MB; a visualization that loads anything from another site, uses browser storage, or makes network requests.
Blocks written as examples inside a longer code fence (four backticks, as above) are not blocks and are not checked.
12. Category, level and kind¶
Added in version 1.2. Optional and backward compatible: a book without these keys builds as before. A library asks for at
least one category per book, and can give it in its own library.yml when the repository does not.
Readers find books by category and subcategory: the Categories button in the header of every page, Browse
categories on the front page, a page for each category and subcategory (subjects/<id>.html) listing its books as covers
or as a compact list with filters for level, kind and language, and All books (books.html), where a reader chooses a
category, then a subcategory. Three keys of a book in pustakalaya.yml describe it:
books:
- id: linear_algebra
title: Linear Algebra
subjects: [algebra-geometry, statistics] # subcategories from the list below, the main one first
level: intermediate
kind: textbook
tags: [matrices, eigenvalues]
Choosing the category. Find the category of the book in the list below, then the subcategory that fits it best, and
write its id (the code in backticks) first under subjects. A book that belongs to more than one field may name one or
two more subcategories, of the same category or of others: it is then listed under each. Name a category itself
(mathematics) only when none of its subcategories fits. The key is called subjects for compatibility with earlier
versions; its values are these ids.
| Key | Values | Meaning |
|---|---|---|
subjects |
ids from the list below | The categories of the book, the main one first: the book is listed under each subcategory named and under its category. Usually one to three. |
level |
beginner, intermediate, advanced |
Beginner: no background needed beyond school; Intermediate: for readers who know the basics of the field, such as university students; Advanced: for specialists and practitioners. |
kind |
textbook, course, field-guide, reference, workbook, series-volume, classic-text |
Textbook: teaches a subject chapter by chapter, usually with exercises; Course: follows a course or syllabus lesson by lesson; Field guide: a broad survey in plain language, to look things up and get oriented; Reference: a comprehensive work to consult; Workbook: mostly exercises and practice; Series volume: one volume of a numbered series; Classic text: a classic or primary text, such as a scripture, an epic or an older literary work. |
tags |
words | Free words a reader might type: names, abbreviations, other spellings (TPM, Ramayana). They are searched, not shown. |
The categories and their subcategories¶
Readers see only the categories and subcategories that have books. A library can add categories or subcategories of its
own (see "Categories and reading lists" in README.md of the product repository); a book may name them once the library
has them. The list is kept in tools/subjects.yml.
| Category | Its subcategories (id: title) |
|---|---|
computer-science: Computers & Technology |
programming: Programmingalgorithms: Algorithms & Data Structuresweb-development: Web Developmentcomputer-architecture: Computer Architecture & Chipsoperating-systems: Operating Systemscloud-data-centres: Cloud & Data Centresnetworks: Networks & Telecomdatabases: Databasesartificial-intelligence: AI & Machine Learningcyber-security: Cyber Securityquantum: Quantum Computing |
science: Science |
physics: Physicschemistry: Chemistrybiology: Biologyastronomy: Astronomy & Spaceearth-environment: Earth & Environmentagriculture: Agriculture & Food |
mathematics: Mathematics |
algebra-geometry: Algebra & Geometrycalculus: Calculus & Analysisdiscrete-mathematics: Discrete Mathematics & Logicstatistics: Statistics & Data Scienceoptimization: Optimization |
engineering: Engineering |
electronics: Electronics & Electricalmechanical-engineering: Mechanical Engineeringcivil-engineering: Civil Engineeringmetallurgy: Materials & Metallurgyaerospace-defence: Aerospace & Defence |
medicine: Health & Medicine |
human-body: The Human Bodypublic-health: Public Health & Nutritionpsychology: Psychology & Mindtraditional-medicine: Traditional Medicine |
history: History |
indian-history: Indian Historyworld-history: World Historyhistory-of-science: History of Science & Technologymilitary-history: War & Military Historygeography: Geography & Travelbiography: Lives & Memoirs |
philosophy-religion: Philosophy & Religion |
philosophy: Philosophyreligion: Religion & Spiritualitysacred-texts: Scriptures & Sacred Textsyoga: Yoga & Meditation |
language-literature: Language & Literature |
english: Englishfrench: Frenchindian-languages: Hindi & Indian Languagessanskrit: Sanskrit & Classical Languageslanguages: Linguistics & Grammarliterature: Novels, Plays & Storiespoetry: Poetrychildrens-books: Children's Booksworld-languages: World Languagesepics: Epics & Mythology |
arts: Arts & Music |
visual-arts: Painting & Sculpturecrafts: Crafts & Textilesarchitecture: Architecture & Designmusic-performance: Music, Dance & Theatre |
social-sciences: Society, Law & Economics |
society-culture: Society & Culturesociology: Sociologylaw-polity: Law, Politics & Governmenteconomics: Economics & Businesstrade: Trade & Seafaringanthropology: Tribal & Folk Cultures |
education: Education |
teaching: Teaching & Learninghistory-of-education: History of Education |
iks: Indian Knowledge Systems |
iks-veda: Veda & Upaniṣadiks-darsana: Darśana (Philosophy)iks-traditions: Buddhist, Jain & Tantric Traditionsiks-yoga: Yogaiks-itihasa: Itihāsa & Purāṇaiks-kavya: Kāvya (Literature)iks-sabda: Language & Grammariks-ganita: Jyotiṣa & Gaṇitaiks-ayurveda: Āyurvedaiks-science: Science, Farming & Geographyiks-arts: Arts & Craftsiks-polity: Polity & Societyiks-mind: Mind & Education |
Retired ids that are still accepted (the book is listed under the new one): chip-design (now computer-architecture), cryptography (now cyber-security), ecology (now earth-environment), embedded-systems (now electronics), hardware-security (now cyber-security), memory-storage (now computer-architecture), quantum-machine-learning (now quantum), security-operations (now cyber-security), telecommunications (now networks).
The same keys at the top of pustakalaya.yml are defaults for every book of the repository. A library may also set them in
library.yml, for a collection (defaults for its books) or for one book (an override), without changing the repository.
What books.py check tests¶
- Errors:
subjectsthat is not a list of ids, alevelorkindthat is not one of the values above,tagsthat is not a list of words. - Warnings: an id that is neither in the list above nor among the library's own subjects, with the nearest id
("did you mean
cyber-security(Cyber Security)?"); the library may still add it. - Notes: a book without subjects; a retired id (it still works: the book is listed under the id that replaced it).
13. Slides¶
Added with version 1.2 ("1.2-slides"). Optional and backward compatible: a book without these conventions is presented from its text, and nothing in sections 1 to 12 changes.
Every chapter of every book (Markdown, LaTeX or HTML) can be shown as slides: Present in the header of a chapter, in
the reader's toolbar (or the key P), or on the book's page. An address with ?slides=1 (?slides=12 for slide 12) opens
a chapter as slides directly. The slides are made in the reader's browser from the chapter as it is written:
| The chapter has | The slides show |
|---|---|
the # title, the italic line under it, the book's title and author |
a title slide in the book's cover colours |
three or more ## sections |
a slide with the chapter's sections |
a ## section |
a section slide, with its ### subsections |
| paragraphs | the key sentence of each paragraph as a bullet point, at most about six bullets and sixty words on a slide; the full paragraphs are the speaker's notes |
| a list, a definition list | its items (long items shortened to their first sentence) |
| a figure, a table, a code block, a large equation | a slide each; a long table or listing goes on over several slides |
a viz, steps or mermaid block, a video |
a slide each; a visualization runs in place, a step-through figure steps with the arrow keys |
a box (!!!) |
a callout slide; an example box titled "Example ..." is revealed piece by piece |
| an exercise and its folded answer (section 4) | a question slide; the answer appears with the next key press (and the right option of a multiple-choice question is marked) |
| a section titled Summary, Key points, Conclusion ... | the closing slide, with the next chapter |
So a book that follows sections 3 and 4 needs nothing more. Write a paragraph so that its first sentence states its point: that sentence is the bullet. Give tables and figures captions, and keep a code block to the lines that matter.
An author who wants more control writes one of the following.
A slide marker¶
<!-- slide: The four steps -->
1. Write the chapters.
2. Check the book.
3. Attach it.
<!-- /slide -->
The blocks after <!-- slide --> (or <!-- slide: Title -->), up to the next marker, a <!-- /slide -->, a ## or
### heading or a ::: slide block, are put together on one slide as written (not shortened), under the title given
or the heading of the section. The comment is invisible on the page and in the reader.
A slide for the presentation only¶
::: slide "The four steps"
1. Write the chapters.
2. Check the book.
3. Attach it.
Notes: Point at the figure of section 1.3 while you go through the steps.
:::
- The block starts at a line
::: slide, with an optional title in double quotes, and ends at a line:::. - It is a slide of its own at that place in the deck, and is not shown on the page or in the reader.
- Inside, anything of section 4 may be used: paragraphs, lists, figures, tables, code, mathematics, boxes,
vizandstepsblocks. A first#or##heading is the slide's title when the opening line gives none; other headings become bold lines. - A paragraph that starts with
Notes:begins the speaker's notes, to the end of the block. They are shown in the speaker view (key S), with a timer and the next slide.
A deck of its own¶
A file in slides/ with the name of a chapter file replaces that chapter's slides:
my-book/
chapters/
03_querying.md
slides/
03_querying.md the slides of chapter 3
# Querying with SELECT
Notes: Start with the question the chapter answers.
---
## The SELECT statement
- choose the columns
- filter with WHERE

---
## Ordering
`ORDER BY` sorts the result.
- Slides are separated by lines of three dashes (
---). - A slide's first
#or##heading is its title; a paragraph that starts withNotes:begins its notes. - Pictures are in
images/, as for the chapters. The deck opens with the chapter's title slide and ends with its closing slide (the summary and the next chapter).
The template book (book-template) has a slide marker and a ::: slide block in section 1.9, and a deck of its own for
the preface (slides/01_preface.md).
How it is built¶
tools/slides.py in the product repository writes a ::: slide block into the page as a hidden
<div class="pk-slide" data-title="..."> (with a <div class="pk-slide-notes"> for the notes), and a slides file as a
hidden <div class="pk-slides"> at the end of the chapter's page. assets/slides-deck.js reads them. A program that
generates books may write these elements directly (section 10), with Markdown inside them.
What books.py check tests¶
- Errors: a
::: slideblock without its closing:::. - Warnings: a file in
slides/without a chapter file of the same name inbook.yml; a slides file without slides.
LaTeX and HTML books are presented from their text; the conventions of this section are for Markdown books.
Version 1.2. The format changes only by adding optional keys and conventions; pustakalaya: 1 stays valid, and a book written
for version 1 builds unchanged. Version 1.1 added section 11 (video, steps, viz and mermaid blocks); version 1.2 added section 12 (subjects, level,
kind and tags).
tools/importers/common.py, tools/mdbook.py and tools/vizblocks.py in the product repository are the reference implementation.
13. Linked books¶
A linked book is a freely licensed book that lives on another site and is shown in the library in its own format and reader, without its text being stored in the library. The library keeps:
- the catalogue entry in library.yml (title, author, cover colours, description, licence, attribution);
- the contents: the chapters' titles, the source's names of their pages and their section headings. This is metadata,
fetched when the site is built (
tools/remote.py) and cached in the git-ignored.cache/remote/<id>.json; - one empty page per chapter (a shell) at the library's own address, with the chapter's heading and the headings of its sections, so that the navigation, the book's contents, the reader's "Whole book" tab and the shelf work as for any other book.
When a reader opens a chapter, assets/remote.js fetches its text from the source's API in the reader's browser, removes
the source site's chrome (edit links, navigation boxes, templates, page numbers), converts it to the library's chapter
HTML and fills the shell; the page and the reader then show it like any other chapter. The address stays on the library's
site: nothing of the source site is embedded or opened, and no reader is sent there. Sections are numbered as in the
contents, figures get their captions, tables, code (with its language), mathematics (typeset by MathJax), footnotes and
pictures (loaded from the source's image servers, with their sizes) are kept, links to other chapters of the book lead to
the library's pages, and other links open in a new tab when they are clicked. Each chapter ends with its attribution: the
page it comes from, its contributors and history, the licence and the note "Modified: converted to this format". These
links, and those of the badge on the book's page, are the only links to the source.
Sources. Only sources whose API allows a page of another site to read it (CORS) can be linked:
type |
Site | Contents rules |
|---|---|---|
wikibooks |
<lang>.wikibooks.org (CC BY-SA 4.0) |
collection, links, subpages, list |
wikisource |
<lang>.wikisource.org (public-domain texts) |
links, subpages, list |
standardebooks |
the public GitHub repository of a Standard Ebooks edition (CC0) | toc |
archive |
scanned books of archive.org (public domain in India, judged title by title) |
page ranges, or chapters |
Project Gutenberg and LibreTexts do not allow it and cannot be linked.
EPUBs on GitHub (standardebooks). The editions of Standard Ebooks (carefully produced public-domain texts, many of them
transcriptions first made by Project Gutenberg) are dedicated to the public domain (CC0), and the repository of each one serves its
files with Access-Control-Allow-Origin: * at raw.githubusercontent.com/standardebooks/<repo>/master/src/epub/. A linked book of
this type needs only the repository's name:
- id: pride-and-prejudice
collection: literature-classics
title: Pride and Prejudice
author: Jane Austen
description: One sentence for the cover card.
subjects: [literature]
source: {type: standardebooks, repo: jane-austen_pride-and-prejudice}
attribution: "Jane Austen, Pride and Prejudice (1813)"
The licence (CC0 1.0) and the book's page on Standard Ebooks (the url) have defaults. The contents are read at build time from
the repository's toc.xhtml and content.opf: a table-of-contents entry with entries of other files below it is a group (a part
or book of the work), entries that point into the entry's own file are its sections (the scenes of an act), entries before the
half-title page are front matter and those after the body back matter; the title page, imprint, colophon and licence pages are
left out. A chapter known only by its Roman number is called "Chapter I". Keys of source: repo, branch (default master),
skip (file names, without extension, to leave out), numbering (none for plays: "Act I" without a number). When a chapter is
opened, its XHTML file (and the book's endnotes.xhtml when the chapter has notes) is fetched in the browser and converted:
the file's own heading is dropped (the page shows the chapter's title), verse and stage directions keep their layout, a table of
speakers and speeches becomes a list of speeches, endnotes become footnotes, pictures are loaded from the repository, and links to
the book's other chapters lead to the library's pages. Each chapter ends with its attribution: the edition, its source file on
GitHub, "CC0 1.0 (public domain dedication)" and the original transcription it names (Project Gutenberg, Wikisource, ...), with
links that open only when they are clicked. "Keep for offline reading" keeps the chapters' files and pictures on the device.
linked:
- id: wb-data-structures # the book's id (its address: <collection path>/<id>/)
collection: open-textbooks
title: Data Structures
subtitle: Fundamental tools
author: Wikibooks contributors # or the historical author and the translator
description: One sentence for the cover card.
about: >- # optional: the longer text of "About this book"
...
language: en # default: the source's language
chapter_label: Chapter # "Section", "अध्यायः", ...; empty for the title alone
cover: {background: "#1d4e5f", accent: "#f2b544", text: "#f4f1e8"}
subjects: [algorithms] # level, kind and tags as for every book (section 12)
source:
type: wikibooks # wikibooks | wikisource
lang: en # the language of the site
root: Data Structures # the book's main page on the site
contents: collection # see below
licence: {id: CC BY-SA 4.0, url: "https://creativecommons.org/licenses/by-sa/4.0/"}
attribution: "Data Structures, a Wikibooks textbook, by its contributors"
url: "https://en.wikibooks.org/wiki/Data_Structures" # the book on its site (for the attribution only)
licence (an id and a URL), attribution and url are required: the licences of these books ask that their authors
are credited and the licence is named. The badge on the cover and the book's page reads "From Wikibooks · CC BY-SA 4.0".
Contents rules (source.contents):
collection: the book's saved collection,Wikibooks:Collections/<root>: a;Headingline groups the pages listed after it (:[[Page|Title]]); a group of two or more chapters becomes a part of the contents.links: the links of the root page to its subpages, in order; a level-2 heading or a line in bold names a group.subpages: all the subpages of the root, ordered by their number (सेवासदन/१,सेवासदन/२, ...).list: an explicit list,chapters: [{page: ..., title: ..., group: ...}].
Other keys of source: exclude (pages to leave out), front (pages shown unnumbered before the chapters),
numbering (sequence, the default; group: I.1, I.2, ... within each group; none: chapters by title only),
digits: devanagari (chapter numbers in Devanagari), parts: links (a chapter whose page links to two or more of its own
subpages is made of them: each subpage becomes a section of the chapter, as the hymns of a book), sections: false (no
section headings in the contents), math: true (the book has mathematics: MathJax is loaded early and kept for offline
reading) and drop (CSS selectors of elements of the source's pages to remove). A chapter whose title is only a number
is called by the label and the number ("अध्याय १").
Offline. "Keep for offline reading" keeps the shells like any book's pages; for a linked book the source's answers for every chapter, and its pictures, are then kept on the reader's device as well (never in the library), and removed with the book.
Search. The library holds no text of a linked book: the titles of its chapters and sections are found by the search, and the reader's search of the book's text is not offered.
Commands. python tools/remote.py list | refresh [id ...] | show <id>; python tools/build.py fetches the contents of
a linked book whose cache is missing or older than 30 days, and python tools/books.py sync fetches them all again.
python tools/check.py verifies that every linked book has its licence and attribution, that its chapter pages are
shells (no text of the book in the site) and that its search index holds only headings.
Scanned books: the archive source¶
A book that exists as scanned pages on the Internet Archive (the books of the Archaeological Survey of India and the Digital
Library of India, in.gov.ignca.* and in.ernet.dli.*) is linked in the same way, with source.type: archive. Its
catalogue entry carries the basis of its public-domain status; its licence and attribution are filled in from the
item (the licence is "Public domain in India", the attribution the digitisers and the host), so the entry needs neither.
Only titles in the public domain in India are linked: published by 1940 and, where the author or editor is known, dead
before 1965 (life plus sixty years); a title that is in doubt is left out.
linked:
- id: ayurvedasutram
collection: rare-books
title: Āyurvedasūtram
author: R. Shama Sastry (editor)
language: sa
chapter_label: "" # chapters are page ranges: "Pages 1-25"
kind: rare-book # the kind of a scanned book (section 12)
subjects: [traditional-medicine, sanskrit]
source:
type: archive
identifier: in.gov.ignca.8087 # the item on archive.org
pd_basis: "Published 1922; edited by R. Shama Sastry (died 1944); public domain in India"
display: scans # text | scans (default: automatic, see below)
chapter_pages: 25 # scanned pages in a chapter (default 25)
chapters: # optional: the chapters by scan number, instead of ranges
- {title: Introduction, from: 1, to: 30}
identifierandpd_basisare required. The build asksarchive.org/metadata/<identifier>for the title, creator, date, language, the names of the files and the number of scanned pages, and the image service for the size of a page; the results are cached in.cache/remote/<id>.json. Nothing else of the item is kept: no image, and no text but the library's own reading of an item that has none (below).- Chapters are page ranges ("Pages 1–25", "Pages 26–50", ...), each a shell like those of the other linked books, or the
chapters the entry lists in
chapters(title,from,to: scan numbers, the first scan being 1). - Display. In
scansmode a chapter is its pages as images; intextmode it is the OCR text of its pages, cleaned lightly (running heads and folios removed, hyphenated words joined, lines made paragraphs) and set in the library's typography, with a "View page scan" button on every page.textneeds the item's*_hocr_searchtext.txt.gzand*_hocr_pageindex.json.gz; by default a book is shown as text when its language is in the Latin script (English, Latin, French, German and the like), it has them, and a sample of its text reads well: at least 92% clean words (letters with a vowel) and at most 1.5% broken ones (stray signs, letters mixed with digits: what the OCR makes of Devanagari, of diacritics it cannot read and of formulas); otherwise as scans (Sanskrit and Hindi in Devanagari cannot be read by the OCR; a book that mixes English with Devanagari, or tables of figures, read better as scans).display:overrides the choice (for a book whose language is wrongly recorded). - The library's own reading. Many English items (most of those of the Digital Library of India) have no OCR text on
the Internet Archive.
python tools/ocr_archive.py --list | --book <id> | --allreads their scans with Tesseract (English, the LSTM engine, automatic page layout; each page fetched from the image service at about 2,000 pixels wide, at most one request a second, read and not kept; small scans are enlarged first; lines read with little confidence are dropped; the lines read poorly are read again with the Sanskrit model as well, and runs of words in Devanagari are taken from that reading, when--tessdatanames a folder withengandsan; a book whose first forty pages read again show none is read in English only from then on) and keeps the text in a private repository of its own, named byapp.ocr_repoin library.yml (its folder in library.local.ymllocal:, like the audio repository):<repo>/<book id>/pages.json, the text of every scan as read (a list, scan 1 first), andmeta.json(the item, the engine and its settings, the date, the mean word confidence of each page and of the book, the shares of clean and of broken words). A run resumes where the last one stopped. Such a book is shown as text when the reading covers its scans and is good (the same 92% clean words, counted on the text in the Latin script, and 1.5% broken words as above, and a mean word confidence of 70), whatever language its item records; otherwise as scans. The build copies the text of each chapter intoassets/ocr/<book id>/<from>-<to>.json(a list of the pages' texts) andassets/scans.jsreads it from there instead of the Internet Archive's OCR files, cleaning it in the same way; the book's page and every chapter say that the text was read by machine for this library. A library without the repository shows these books as scans. - The viewer (
assets/scans.js,scans.css) is the library's own: page images come from the image service of the Internet Archive (IIIF, which answers pages of another site), at a size that fits the screen and a larger one on zoom, with pinch, Ctrl+wheel, button and double-tap zoom, pan, fit to width or to the page, a jump to a page number, a strip of thumbnails, the keyboard (arrows, space, plus, minus, 0, f, t, g, Home, End, Escape), swipes, and light, dark or sepia surroundings. The Internet Archive's own reader is never embedded or opened; the links to the item (on the book's page and at the end of every chapter) open only when they are clicked. - Credit. The book's page and each chapter carry the digitisers' credit ("Digitised by the Indira Gandhi National
Centre for the Arts / Archaeological Survey of India; hosted by the Internet Archive"), the item's own details and the
public-domain basis (
pd_basis). - Offline. "Keep these pages offline" on a chapter (the viewer has "Keep this page offline") stores the chapter's
page images, their thumbnails and, for a text book, the text of the book in the browser's cache of the reader's device
only (
pk-scans-kept), after saying how much room it takes (about 0.2 MB a page); "Remove" takes them out again. The text read by this library (assets/ocr/) is also kept with the book by "Keep for offline reading". - Search. The library holds no text of these books (except the text it read itself, which is not indexed): their titles and authors are in the search of titles
(
search/titles.json), and the chapters' titles in the book's own index. - A text read from the scans. A scanned book of which a Markdown book holds the machine-read text (a book of the
collection "Rare Texts") is paired with it under
editions:in library.yml, by site paths (<collection path>/<book id>):
editions:
- {scans: rare-books/my-scanned-book, text: rare-texts/my-scanned-book-text}
The first page of the scanned book then has the link "Read as text", and that of the text book "View the scans"
(tools/editions.py). Inside the text book, a page opens its scan in the library's own viewer with a button
<button type="button" class="pk-sc-view" data-book="<id of the scanned book>" data-n="<scan number>">View scan of page N</button>
(put in a <div class="pk-sc-pn" markdown="0"> with the page number). The scan number is the position of the scan, the
first scan being 1. The text of such a book says that it was read by machine and may contain errors.
14. Pronunciation, quizzes and language markup¶
Added with version 1.2 ("1.2-language"). Optional and backward compatible: a book without this markup builds as before.
Language books (and any other book) can let the reader hear words and sentences, answer questions in text boxes,
choices and matching lists, and practise tables from memory. Everything works on the chapter page, in the paginated
reader (typing in a box never turns a page) and in slides, offline, with the keyboard and with screen readers.
tools/assets/speak.js, quiz.js, speak.css and quiz.css in the product repository bring it to life;
tools/quizblocks.py writes the quizzes.
Phrases to hear¶
<span data-say lang="fr">Bonjour, madame !</span>
<span data-say="l'arbre, un arbre" lang="fr">l’arbre → un arbre</span>
<span data-say="Le samedi matin, Léa va à la boulangerie." lang="fr" data-rate="0.8">▶ the whole story</span>
- A span with
data-saygets a Listen (speaker) button after it. Without a value it speaks its own text; with a value it speaks the value (the spelling a voice reads best, or a longer passage behind a short label). langis the language of the voice (fr,en,hi,de,es, or a full tag such asen-GB). Without it the nearestlangaround the span, or the page's language, is used.data-rateslows a phrase down (1 is the normal speed of the language; French is spoken at 0.82, English at 0.9).- Recorded voices. The button plays a recording of the phrase in a native voice, the same on every phone and
browser (many devices have no French voice of their own). A second tap on the button plays the phrase slowly
(
playbackRate0.75, the pitch kept). The recordings are made offline bypython tools/tts.py(see "Audio pipeline" below). The device's own voice (the Web Speech API) is used only if the reader chooses Device voice in "Voice and speed" above the chapter (also: a slower or faster speed, kept in the browser), or for a phrase that has no recording. The button is hidden only where there is neither a recording nor a device voice for the language; write the pronunciation in the text ([luh LEEVR]), so that the reader still has it. - Microphone. Where the browser can recognise speech, a microphone button (say the phrase, see how close it was)
is shown only inside speaking exercises: an exercise box titled "Say it", "Speak" or "Say", and the
sayquestions of a quiz. It is not put beside every phrase. - Play all. A list of three or more phrases (a dialogue, a word list) or a table with four or more phrases in its rows gets a Play all button above it, which plays them one after another (tap again to stop).
- In a table, put the phrase in its cell; one small speaker per item, not a column of buttons. The importers of the two
language books mark the language taught in the tables of words, phrases and examples, in the rows of conjugation
tables (
je parle, with the person) and in the lists of phrases (tools/importers/speakable.py); a cell in English, Hindi or Hinglish is never marked, so a French voice never reads English. - After a quiz question is checked, a speaker beside the feedback plays the model sentence (the answer, or the question with its blanks filled in) when it is in the language and has a recording.
Audio pipeline (recorded voices)¶
tools/tts.py collects every phrase of the built site (data-say spans and the quiz questions that are spoken:
say, listen, dictation and the model answers), normalises the text (Unicode NFC, one space between words,
the 🔊 mark removed) and makes one MP3 per unique (language, text) with Piper
(MIT, offline, CPU): mono, 32 kbit/s, 22.05 kHz, named by the hash of the text, audio/<lang>/<sha1-12>.mp3.
The recordings are kept in a separate repository of book assets, language-audio (library.yml
app.audio_repo: language-audio, its folder in the git-ignored library.local.yml local:), so that a phrase is made
once and a rerun makes only the new phrases. The build copies the recordings a site needs into
site/assets/audio/<lang>/ with assets/audio/index.json ({lang: [hash, ...]}, read once by speak.js, which
hashes a phrase itself with SHA-1 and looks it up). Without app.audio_repo, or without the folder, nothing is copied and
every phrase plays with the device's voice.
| Language | Voice | Source and licence |
|---|---|---|
| French | fr_FR-siwis-medium |
rhasspy/piper-voices on Hugging Face; SIWIS French Speech Synthesis Database (University of Edinburgh); CC BY 4.0 |
| English | en_GB-cori-medium |
rhasspy/piper-voices; Hi-Fi TTS (NVIDIA, OpenSLR 109), the voice of the LibriVox reader Cori Samuel; CC BY 4.0 |
CC BY 4.0 allows the recordings to be redistributed with attribution: the voices and their sources are credited in the audio repository's README, which travels with the recordings.
To regenerate: install Piper and the MP3 encoder in a virtual environment (nothing system-wide):
python -m venv D:/books/.work/tts/venv, venv/Scripts/python -m pip install piper-tts lameenc; build the site
(python tools/build.py, python -m mkdocs build), then venv/Scripts/python tools/tts.py (--jobs N for the number
of processes, --list to only count, --copy to copy the recordings into site/assets/audio/), and build again
(python -m mkdocs build) so that the offline lists of library.json include them. The voices are fetched on first use
into $PK_TTS_VOICES (default D:/books/.work/tts/voices).
Offline reading. A played recording is kept in the browser (Cache Storage pk-audio-v1) and plays offline the next
time. "Keep for offline reading" also keeps a book's recordings when they total no more than app.audio_offline_mb
(default 12 MB); a larger book (a whole language course in three editions) keeps what the reader has played.
* The span may hold other inline Markdown (**bold**, other spans). Do not put a span around a whole paragraph of
explanation: mark the phrase.
Quizzes¶
A fenced block quiz holds the questions of a quiz, written in YAML:
```quiz
title: Practice: gender and plurals
lang: fr
items:
- type: choice
q: "___ télévision"
hint: Look at the ending -sion.
options: [le, la, les, "l'"]
answer: la
why: -tion and -sion nouns are feminine.
- type: blanks
q: un gâteau → des ___
answer: [gâteaux, gateaux]
- type: text
q: The French for "the book"?
answer: [le livre]
- type: match
q: Match each singular to its plural.
pairs: [[un œil, des yeux], [un journal, des journaux], [un prix, des prix]]
- type: say
q: Say it aloud, with the liaison.
answer: Les amis sont dans la maison.
```
| Key | Meaning |
|---|---|
title |
A line above the questions (optional; the section heading usually names the quiz). |
lang |
The language of the answers and of the phrases to hear (say, listen, dictation). |
labels |
The words of the page version in the book's language: exercise, answer, hint, words, wrong_word, corrected, say, aloud, dictation, transcript (English by default). |
ui |
The words of the live quiz (check, show, retry, right, wrong, ...), for a language other than English and Hindi, which the library has. |
number |
The number of the first exercise (default 1). |
items |
Required. The questions, in order. |
Each question has a type and the keys of its type; hint and why (shown after the check and in the answer) are
optional for all.
type |
Keys | The reader |
|---|---|---|
text |
q, answer |
types the answer in a box |
blanks |
q with ___ for each blank, answer |
types into boxes inside the sentence. With one blank, answer is the answer or a list of accepted answers; with several, a list with one entry (an answer or a list) per blank |
choice |
q, options, answer |
picks one option |
match |
q, pairs |
chooses, for each left item, its partner from a list |
order |
q, answer (the sentence), optional tokens |
taps the words into the right order |
spot |
q, sentence, answer (the wrong word), optional correction |
taps the wrong word |
say |
q, answer |
hears the model, says it; the browser checks it where it can recognise speech, otherwise the reader marks it |
listen |
q, audio, options, answer |
hears a passage (hidden until answered) and picks an option |
dictation |
q, audio, optional answer |
hears a passage and writes it down; the words missed are marked |
mcq, fill, reorder and speak are accepted for choice, blanks (or text, without a blank), order and
say: the types of the quizzes of The English Book and The French Book.
Checking. Answers are compared without regard to case, punctuation and spaces. An answer typed without its accents
is accepted, and the reader is told which accent was missing ("Correct, but mind the accents: gâteaux"); a slip of
one letter in a long word is accepted with the exact spelling shown. Every question has Check (or is checked when an
option or word is tapped), Show answer (which opens the folded answer) and Try again; the quiz shows its score,
and the answers given and the best score are kept in the browser (localStorage), so the quiz is as the reader left it.
On the page without scripts, in print and in slides each question is an exercise box with its answer folded under it, as in section 4 (so a slide shows the question and reveals the answer with the next key press).
What books.py check tests. Errors: a block that is not valid YAML, without items, a question of an unknown
type, without the keys of its type, a blanks question without ___ or with a wrong number of answers, a choice
question whose answer is not an option, a match question with fewer than two pairs. Unknown keys are errors too.
Typed answers for exercises¶
An exercise and its folded answer (section 4) can get a Type your answer box above the answer, checked on demand (the folded answer stays):
??? note typed "Answer"
**Le livre.**
- The class
typedafter the box type asks for the box. In a book whosekindiscourse(section 12) every exercise gets it, unless its answer box has the classuntyped. - The box is offered only when the answer is short: the first paragraph of the answer, or its bold part, is a word, a
number or a phrase of up to four words (alternatives separated by
/oror). Longer answers are left as they are.
Practice tables¶
<div class="pk-drill" markdown="1">
| Person | parler | finir |
|---|---|---|
| je | parl**e** | fin**is** |
| tu | parl**es** | fin**is** |
</div>
A table inside <div class="pk-drill" markdown="1"> gets a Practice switch. It hides the forms that the table
highlights (bold text, verb colours) or, without highlights, the cells after the first column (not those of
pronunciation, meaning or examples), and turns them into text boxes; Check marks each box, Show answer fills in
the rest. A blank line after the opening tag and before </div> is needed.
Colours of language books¶
| Class | For | Shown |
|---|---|---|
g-m, g-f, g-pl |
masculine, feminine, plural words | blue, pink, violet |
r-noun, r-verb, r-adj, r-adv, r-prep, r-pron, r-conj, r-interj |
word classes | a coloured highlight each |
snd-mute, snd-liaison |
silent letters, liaisons | faded and struck through; bold in the accent colour |
Write them as spans: <span class="g-f">la table</span>. A key to the gender colours is a paragraph
<p class="g-legend"><span class="g-m">masculine</span><span class="g-f">feminine</span><span class="g-pl">plural</span></p>.
The colours follow the light and dark themes and the reader's page colours.
The chapter map¶
A figure <figure class="fig pk-cmap"> with an SVG drawing (a hub and the chapter's sections, each a link <a
href="#anchor">) and an ordered list of the same links shows the drawing on a wide page and the list on a phone. The
importers of The English Book and The French Book write one at the start of every chapter, as their readers drew it.
HTML books: what the importers add¶
The importers english and toc-book (section 7) turn every Listen button of the source into a phrase to hear (the
text spoken is the button's), the microphone exercises into say questions, the quizzes into quiz blocks with their
questions and answers as written, conjugation tables (a person in the first column) into practice tables, the colour
classes of the source into the classes above, and the reader's chapter map into a figure. Visualizations made for
these books (a chart of sounds, timelines of tenses, a sentence builder, a numbers and time trainer) are kept in the
product repository, in tools/importers/extras/<book>/ (map.yml names the chapter and the section of each), and are
added to the book as viz blocks (section 11), so that the source repositories stay as they are.