A command-line tool that scans a directory tree for ebook files, reads their metadata (author and title), and reports the results. It searches for ebooks both directly on disk and inside archives, following nested archives up to a configurable depth.
Requires Go 1.26.8+.
make build # builds ./build/margaret
make test # runs the test suite
make vet # runs go vet
make run ARGS="scan <path>"Or without the Makefile:
go build -o build/margaret ./cmd/margaret-cliThe tool has three commands: scan walks a directory recursively and
processes every ebook it finds, report regenerates the JSON reports from
an existing database, and search finds books and authors in the database
without scanning.
margaret scan <path> [flags]
margaret search <query> [flags]
<path> must be a directory. The scan is recursive: ebooks in subdirectories
are found too.
margaret scan ./books<path> can also be an ftp:// URL. The FTP tree is walked and read over the
network; ebooks and archives are spooled lazily instead of being downloaded to
disk first.
margaret scan ftp://192.168.178.1/network/backupIf the server requires credentials, pass them with --ftp-user and
--ftp-pass. When neither is given, the anonymous login is used:
margaret scan ftp://192.168.178.1/network/backup --ftp-user backup --ftp-pass secretNote for FTP scans: one FTP connection is opened per worker, and each connection can transfer one file at a time. Many consumer routers and NAS devices (for example the FRITZ!Box that often lives at
192.168.178.1) only accept one or two simultaneous FTP connections and reject any further ones, which shows up as connection errors during the scan. If that happens, lower the worker count:margaret scan ftp://192.168.178.1/network/backup --workers 1
--workers 1is the safe choice for such devices, but it is a recommendation, not a hard limit — if your server accepts multiple connections, keep the default worker count.
Ebooks (metadata is read from these):
.epub.mobi.azw3.azw.prc.txt.text.rtf.pdf(only when Poppler'spdftotextandpdfinfoare available; see below)
Archives (searched for ebooks inside):
.zip.tar.tgz/.tar.gz.gz.bz2.tbz2/.tar.bz2.rar.7z
Ebooks nested inside archives are handled in place — they are read from the
archive without being extracted to disk. Archives may be nested inside other
archives, up to the depth given by --archive-depth.
Plain-text books are read with a dedicated parser. It detects the encoding
(UTF-8 and UTF-16 BOMs, and legacy code pages from the detector's candidates,
validated by language identification), takes a title and authors from a header
block at the start of the text (Title:, Author:, Authors:, legacy
TI:/AU: header keys, YAML front matter or a Project Gutenberg preamble) or
from Calibre's title-page layout, and extracts at most one validated ISBN from
the first 64 KiB. The whole decoded text feeds the same 512-hash MinHash
fingerprint as the other formats; the content hash, the decoded head and the
fingerprint come from a single pass over the file. RTF books are decoded to
plain text the same way: the \info block supplies the title, authors,
description, subjects and publisher (following Calibre's field mapping), while
documents without \info metadata fall back to Calibre's title-page layout.
Files smaller than --min-doc-size (15 KiB by default) are treated as
distribution metadata or fragments rather than books and are counted as
skipped. Empty files are counted as skipped as well. Binary files are
reported as failures.
PDF books are read through the system-installed Poppler tools: pdfinfo
supplies the Info dictionary and XMP metadata, and pdftotext supplies the
plain text that feeds the MinHash fingerprint. Both tools must be on PATH
(or in the directory given by --pdf-tools-dir). If either is missing, the
scan prints a warning, skips PDF files, and continues with every other
supported format. PDF support is therefore conditional. The same
--min-doc-size limit applies to both the raw PDF file and the extracted
text: PDFs whose cleaned text is smaller than the limit are treated as having
no usable text and are skipped. Poppler's pdftotext does not perform OCR, so
image-only/scanned PDFs are skipped as well.
Poppler is not bundled. Install it with the operating system package manager, for example:
# Debian / Ubuntu
sudo apt install poppler-utils
# Fedora
sudo dnf install poppler-utils
# macOS (Homebrew)
brew install popplerEvery ebook's language is verified from its reading-order text with language-agnostic detection. A missing language is filled in, and a declared language is replaced only when the detected language is clearly different and the detector is confident; regional tags that agree with the detection are kept unchanged.
Metadata and encoding detection are language- and country-independent. Structural signals (Unicode categories, line shape, file names) come first; keyword lists are additive hints that no feature requires, so a language without hints still works.
| Flag | Type | Default | Description |
|---|---|---|---|
--workers |
int |
number of CPUs | Number of worker goroutines used for scanning and parsing. |
--archive-depth |
int |
2 |
Maximum archive nesting depth to unpack. |
--archive-password |
string |
"" |
Password for encrypted rar and 7z archives. |
--spool-mem-limit |
int |
67108864 |
Max bytes per archive member buffered in memory before spilling to a temp file. |
--min-doc-size |
int |
15360 |
Minimum size in bytes for a document-format file (txt, rtf, pdf) to be treated as a book; for PDF it also applies to the extracted text (0 = default, negative = disable). |
--pdf-tools-dir |
string |
"" |
Directory containing Poppler's pdftotext and pdfinfo (default: search PATH). |
--ftp-user |
string |
anonymous |
FTP username for ftp:// scans. |
--ftp-pass |
string |
anonymous |
FTP password for ftp:// scans. |
--failures-out |
string |
reports/failures.json |
Path to write a JSON report of the failed items. |
--skipped-out |
string |
reports/skipped.json |
Path to write a JSON report of the skipped items. |
--duplicates-out |
string |
reports/duplicates.json |
Path to write a JSON report of the duplicate books. |
--books-out |
string |
reports/books.json |
Path to write a JSON report of the grouped books. |
--report |
bool |
false |
Write the books and duplicates JSON reports (failures and skipped reports are always written). |
--db |
string |
margaret.db |
SQLite database file to use. |
--json |
bool |
false |
Write search results in JSON format (search only). |
--json-out |
string |
reports/search.json |
Path to write the JSON search results (search only). |
--author |
bool |
false |
Search authors instead of both titles and authors (search only). |
--title |
bool |
false |
Search titles instead of both titles and authors (search only). |
--limit |
int |
0 |
Limit the number of search results (0 = unlimited, search only). |
--fresh |
bool |
false |
Delete existing scan data and start from a clean slate. |
--resume |
bool |
false |
Keep existing scan data and only process never-seen paths. |
--ext-stats |
bool |
false |
Print per-extension file counts (supported vs unsupported) after the scan. |
--cpuprofile |
string |
"" |
Write a pprof CPU profile of the command to this file. |
-h, --help |
Show help. | ||
-v, --version |
Print version. |
Sets the number of concurrent workers for both the directory scan and the
metadata parsing pipeline. The queue size is derived from this value
(workers * 2). When omitted, the number of CPUs is used.
For FTP scans one connection is opened per worker (see
Scanning over FTP). If the server rejects concurrent
connections, lower this to 1.
Controls how many levels of nested archives are unpacked. A depth of 0 means
the archive itself is opened but nothing inside is processed; an ebook directly
inside an archive is at depth 1. If an archive is nested deeper than this
limit, the scan records it as a failure.
Supplies the password for encrypted rar and 7z archives. Both formats are supported: members are decrypted in memory while being read. AES-encrypted zip members are not supported yet; they are reported as failures.
Controls how much of each ebook or archive found inside an archive is buffered
in memory before the rest is spilled to a temporary file (see
Memory usage). The value is the limit per member in bytes;
a value of 0 restores the default of 64 MiB. Only members that exceed the
limit create a temp file, and those files are deleted as soon as the member is
done processing.
Sets the minimum size in bytes for a document-format file (plain text, RTF
and PDF) to be treated as a book. Document collections often contain tiny
side files (nncl.txt, olvasd_el.txt, _adatlap.txt, chapter fragments)
next to the actual books; these are skipped and counted in Skipped instead
of becoming books on their own. The default is 15 KiB (15360 bytes) because
side files (readmes, blurbs, file lists) are typically smaller than real
books. A value of 0 uses the default; a negative value disables the size
filter entirely.
For PDFs the same limit is applied twice: the raw PDF must be at least this large, and the cleaned extracted text must be at least this large. PDFs below either threshold are treated as empty or fragmentary and skipped.
# keep only document files of at least 32 KiB
margaret scan ./books --min-doc-size 32768
# disable the size filter
margaret scan ./books --min-doc-size -1Robust content checks still reject HTML sources, HTTP cache dumps, credit pages, blurbs and tables of contents for plain-text and RTF files.
Sets the directory that contains Poppler's pdftotext and pdfinfo. By
default both are looked up on PATH. When either is missing, the scan prints
a warning and skips PDF files; all other supported formats are still
processed. This is useful on systems where Poppler is installed outside the
standard PATH.
The path where the JSON report of failed items is written. Defaults to
reports/failures.json.
The path where the JSON report of skipped items is written. Defaults to
reports/skipped.json. It lists the files that were passed
over without being parsed because they are not books, are below
--min-doc-size, or cannot be processed (for example PDFs while Poppler is
unavailable). Files skipped by a resume run because they were already
processed are not listed: there is nothing left to inspect for those.
They are counted separately as already processed.
The path where the JSON report of duplicate books is written. Defaults to
reports/duplicates.json. Each entry lists the original book
(authors, title, path) and every other path that holds the same content
(duplicates). When no duplicates are found the file is not created. Only
written when --report is set.
The path where the JSON report of the grouped books is written. Defaults to
reports/books.json. See
How books are grouped and described.
When no books were found the file is not created. Only written when
--report is set.
Writes the two bulk JSON reports — the grouped books (--books-out) and the
duplicate books (--duplicates-out). Off by default: at hundreds of thousands
of books these files grow to hundreds of megabytes, while the database itself
is already kept as a stable, queryable artifact (see --db), so
there is no reason to pay that disk cost on every run.
failures.json and skipped.json are not affected: they are the error and
skip channels, normally small, and are written on every run, as is the stdout
summary.
margaret scan ./books --reportCredentials for ftp:// scans. Both default to the anonymous login
(anonymous / anonymous@). They are optional: an FTP URL may embed
credentials directly (ftp://user:pass@host/path), but using the flags keeps
the password out of the command line.
The SQLite database file used during the scan. Defaults to margaret.db in
the current directory. Pending schema migrations are applied automatically
with goose, so an existing database file is reused and only migrated forward.
A scan with --fresh (or over an empty database) starts with a clean slate:
previous scan data is deleted first (the schema and migration history are
kept), so re-running a scan never mixes results from earlier runs. With
--resume existing rows are kept instead (see
--fresh / --resume). Pass a custom path to use a
different database file, or ":memory:" for an ephemeral database that is
discarded when the scan finishes:
margaret scan ./books --db /tmp/test.db
margaret scan ./books --db ":memory:"If the parent directory of a custom path does not exist, the scan fails with an error. Do not run two scans against the same database file in parallel.
Controls what happens when the database already holds scanned data:
--freshdeletes the existing data first (same as the default behavior on an empty database) and rescans everything.--resumekeeps the existing rows: files whose path is already recorded are skipped without being read again — even if they changed on disk — while never-seen paths are processed. Failures leave no row behind, so they are retried on every run. The scan summary counts already processed files as a separate line, and they are not listed in the skipped report.
With neither flag and a non-empty database, an interactive terminal asks
whether to resume (the default) or start fresh, showing the database path,
the recorded book/file counts and the previous scan's root and time. Without
a terminal (scripts, CI, cron) the scan fails instead with an error telling
you to pass --fresh or --resume — so automation can never silently wipe
or mix data. The two flags are mutually exclusive.
Resuming a different tree into the same database warns explicitly on a
terminal and is refused without one: mixing two trees is almost never
intended, so use --fresh or a different --db file instead.
Every successfully parsed file is committed to the database immediately (one
transaction per file), so interrupting a scan with Ctrl-C keeps everything
processed so far — only the in-progress files are lost. Re-running with
--resume continues where the interrupted run stopped: finished files are
skipped, failed files (which leave no row behind) are retried. Each run is
recorded with its root, mode and outcome; the re-run prompt reads the latest
entry.
Writes a pprof CPU profile of the whole command (scan, consolidation and
report writing included) to the given file. It works for every subcommand, and
the profile is flushed even when the run is canceled with Ctrl-C or fails.
The flag has no effect on the results.
margaret scan /home/attila/books --db scan.db --cpuprofile cpu.prof
go tool pprof -top margaret cpu.profEbooks found inside archives are read in place — they are never extracted to
disk. Instead, each archive member is spooled lazily: the part of the file the
metadata reader actually needs (for most formats only a small prefix) is read
straight from the archive. If the metadata lives at the end of the file (as
for .epub, whose central directory sits at the end), the whole member has to
be pulled from the archive, but it is kept in memory up to
--spool-mem-limit (default 64 MiB) per member; anything beyond that is
spilled to a temporary file that is removed when the member is done.
The memory limit is per member, and several members are processed concurrently
(one per --workers), so peak memory can be up to
workers × spool-mem-limit. Lower the limit if you scan with many workers or
have little RAM; raise it to avoid temp files on a slow disk:
# keep at most 16 MiB of each member in memory
margaret scan ./books --spool-mem-limit 16777216
# keep at most 256 MiB in memory (avoids temp files for large ebooks)
margaret scan ./books --spool-mem-limit 268435456During a run, a single-line progress bar is drawn to stderr (it shows a spinner, a progress bar, a counter, and the file currently being processed). It is rewritten in place and never wraps, so your terminal stays clean.
When the scan finishes, the final report is printed to stdout:
Total 2 ebook(s)
Succeeded 2
Failed 0
Duration 1ms
- Total — total number of ebook files encountered
(
succeeded + failed + skipped + already processed). - Succeeded — ebooks whose metadata was read successfully.
- Failed — items that could not be processed.
- Skipped — actionable skips: document files that are too small or
otherwise not books, empty files, and PDFs skipped because Poppler is
unavailable or their extracted text is too short. These are listed in
skipped.json. - Already processed — files a resume run passed over because they were
recorded in an earlier run. These are not listed in
skipped.json. - Duration — wall-clock time of the scan.
With --ext-stats the summary is followed by per-extension file counts,
split into supported and unsupported. Supported archives are treated as
folders and excluded; files on disk and inside archives are both counted.
The percentages sum to 100% over supported + unsupported:
Extension stats (10 files):
Supported:
epub: 3 (30%)
pdf: 3 (30%)
mobi: 2 (20%)
txt: 1 (10%)
Unsupported:
docx: 1 (10%)
margaret scan ./books --ext-statsThe report is followed by the failures and skipped JSON files. Empty output is written even when nothing failed or was skipped:
[]When items fail, the report lists each item and the reason:
[
{
"path": "books/secret.zip!book.epub",
"error": "password-protected zip member; --archive-password is not supported yet"
}
]Note that paths of items found inside archives are shown as
<archive-path>!<member-name>.
The skipped report follows the same conventions and is written even when nothing was skipped:
[]Skipped items list the path and the reason:
[
{
"path": "books/short.txt",
"reason": "below minimum document size (1200 bytes, minimum 15360)"
},
{
"path": "books/bundle.zip!scanned.pdf",
"reason": "no extractable text"
}
]Duplicate books are reported in duplicates.json. Each entry describes the
original book and every other path that holds the same content:
[
{
"authors": "Herman Melville",
"title": "Moby Dick",
"path": "books/mobydick.epub",
"duplicates": ["backup/mobydick.epub", "mirror/mobydick.epub"]
}
]When no duplicates are found the file is not created.
Grouped books are reported in books.json (see
How books are grouped and described).
Each entry holds the chosen book-level metadata plus every grouped file with
its own metadata:
[
{
"authors": ["Herman Melville"],
"title": "Moby Dick",
"description": "The story of Captain Ahab's hunt for the white whale.",
"series": "",
"series_index": "",
"subjects": ["Sea stories", "Adventure"],
"languages": ["en"],
"identifiers": [
{"scheme": "ISBN", "value": "9780142437247", "source": "METADATA_AND_TEXT"}
],
"publications": [
{"publisher": "Harper & Brothers", "publication_date": "1851"},
{"publisher": "Penguin Classics", "publication_date": "2003"}
],
"files": [
{
"path": "books/mobydick.epub",
"authors": ["Herman Melville"],
"title": "Moby Dick",
"identifiers": [
{"scheme": "ISBN", "value": "9780142437247", "source": "METADATA_AND_TEXT"}
]
},
{
"path": "backup/mobydick.mobi",
"authors": ["Herman Melville"],
"title": "Moby Dick; Or, The Whale",
"identifiers": [
{"scheme": "ISBN", "value": "9780142437247", "source": "METADATA_AND_TEXT"}
]
}
]
}
]When no books were found the file is not created.
File entries also carry the claim provenance: legacy (the pre-claim title
and author list), title_status/author_status, title_set/author_set
and the full claims object with source, trust, witness, junk, attested and
attestation fields. Book entries carry the synthesized
title_status/author_status, title_candidates/author_candidates,
grouping_suspect and the runtime-only legacy_book object used for the
transition diff.
Files with byte-identical content (same content hash) are exact duplicates.
The first path seen wins; the rest are listed under it in duplicates.json.
Exact duplicates never get a book of their own.
Every other file carries a 512-number MinHash content fingerprint. During the scan each new file is grouped as follows:
- The first 125 values of its MinHash signature are split into 25 bands of 5 numbers (LSH banding). Files sharing at least one band hash become candidates.
- Candidates are verified with Jaccard similarity estimated from the full 512-value signature in Go.
- At Jaccard ≥ 75% the file joins the best-matching candidate's book (greedy best match, no transitive merging of books); otherwise a new book is created.
Every file belongs to exactly one book. Grouping runs inside the scan transaction, so results do not depend on worker scheduling.
books.json picks one title, one author list and the remaining book-level
metadata per book; the per-file entries keep their original titles
(whitespace-normalized), their placeholder-filtered authors and their
identifiers with provenance. The chosen values are computed by Consolidate
after a successful scan and stored in the books and book_* tables; the
raw book_files metadata is preserved. The reports read the consolidated
book-level values; file entries read the stored per-file identifiers. All
comparisons below use normalized keys:
- Normalization. Comparisons use trimmed, whitespace-collapsed,
lower-cased keys; diacritics are folded (NFD mark-stripping for Latin
script only, so Cyrillic
йand Indic scripts survive, plusø→o,ß→ss,æ→ae,œ→oe,ł→l,đ→d,þ→th,ı→i). - PDF metadata.
pdfinforeads the Info dictionary and, when present, the XMP packet. XMP wins unless the InfoModDateis newer. The supported fields map to title, authors, description (Subject/dc:description), subjects (Keywords/dc:subject/pdf:Keywords), publisher, publication date, series, languages and ISBN/DOI identifiers. Cover images and Calibre-specific custom metadata are not read. - Author placeholders are dropped. Converter defaults such as
Unknown,Ismeretlen,Inconnu,Unbekannt,Desconocido,Sconosciuto,Onbekend,Nieznany,Okänd,Bilinmiyor,Неизвестный,مجهول,نامعلوم,अज्ञात,未知,佚名,不明,미상,Névtelen,User,N/A(full list in code). An author name identical to its file's title is also dropped (converters sometimes copy the title into the author field). - Author vote. The most common full author list wins;
Last, Firstspellings vote together withFirst Lastones (display keeps the first-seen spelling and order). Ties break towards more authors, then more characters, then more space-containing names, then first-seen. - Title placeholders are dropped.
Untitled,Sans titre,Ohne Titel,Sin título,Senza titolo,Sem título,Zonder titel,Bez tytułu,Utan titel,Başlıksız,Без названия,بدون عنوان,无标题,無題,제목 없음,Névtelen(full list in code). - Title vote. The most common title wins. Ties break towards fewer underscores (machine-stamped titles use them, clean ones use spaces), then towards titles not mentioning a known author, then longer, then first-seen.
- Author-in-title penalty (tie-break only). A title containing every
distinctive token of a known author (
Steven Saylor - X) loses ties. Tokens of 1–2 letters and particles (van,von,del, …) do not count; single-token names can still match on their one word. - Filename fallback. With no usable title at all, the first file's name is used (archive prefixes and directories dropped, known ebook extension stripped, underscores to spaces). File entries keep their empty titles.
- Calibre
Title - Authorfilenames. With no usable authors, member filenames are scanned in path order: the strict form needs a comma (X - Last, First) and can also supply the title; next to a known-good title a comma-less two-word author (X - First Last) is accepted for the authors only. Authors recovered this way are stored verbatim. - Credit-affix wash. With known authors, a leading
Author -or trailing- Author/by Authoris stripped from the picked title (whole-word, order-free, diacritic-folded match; the result must stay non-empty). Genuine titles such asStephen King Goes to the Moviesare unaffected. - Description. HTML markup is converted to plain text before voting
(tags become spaces, entities are decoded, and
script/style/headcontents are dropped). The most common normalized text wins; ties prefer the longer text. The rawbook_files.descriptionis preserved as extracted. - Publications. Every distinct publisher/publication-date pair is kept, so different files may contribute different editions of the same book. Publisher names are grouped case-insensitively (diacritics folded) and the first spelling is kept; pairs are deduplicated and emitted in first-seen order. An undated occurrence does not create a separate blank-date entry when the same publisher already has at least one dated entry; if the publisher has no valid date at all, one blank-date entry is kept. A valid date without a publisher is kept as a pair with an empty publisher.
- Publication date validation. Dates are normalized to
YYYY,YYYY-MMorYYYY-MM-DD; RFC3339/ISO datetimes are truncated to their date. Impossible dates, free-form values such ascirca 1999or1999. május, and implausible years outside1000..current year + 1 (for example0101-01-01) are dropped. A publisher with no valid date keeps an empty date. - Series. The most common normalized series name wins.
series_indexis voted only among files that carry the chosen series, so an index can never be paired with a different series. - Subjects. The union of all files' subjects, deduplicated case-insensitively; the first spelling and order are kept.
- Languages. Only valid BCP 47 tags are kept, canonicalized (for
example
en_USbecomesen-US). Empty values, invalid codes such asenglish, and the meaningless basesund,mul,zxxandmisare dropped. The result is the union of the remaining tags in first-seen order. - Title and author claims. Each file stores every title and author
statement extracted from metadata, leading text and its file name in
book_file_claims, together with the source, the extraction hypothesis (set), junk/attestation flags and the text-support score. The resolvedbook_files.titleandbook_file_authorscome from these claims; the pre-claim values remain inlegacy_titleandbook_file_legacy_authorsfor comparison and rollback. A metadata title that appears as a title-page-like phrase in the first 8 KiB of text isattested; a filename-only title or author isweakand stays out of the stored title/author fields, so the book-level filename fallback still applies. - Book-level claim synthesis. The book title and author list are resolved
from all file claims, not by voting the already-resolved file values. A
metadata agreement across independent content witnesses is
cross_file; a metadata claim attested in a title-page window iscorroborated. Weak filename or title-page evidence is accepted at book level withweakstatus (the matcher treats it as neutral), and alternative hypotheses are kept inbook_claim_candidates. Conflicting metadata can setgrouping_suspectwithout splitting the book automatically. The oldpickTitle/pickAuthorsresult is still computed for thelegacy_bookdiff and used as a fallback when a book has no claims. - Identifiers and source. Every identifier carries a
source:METADATA,TEXTorMETADATA_AND_TEXT. Book-level identifiers store only ISBNs; file-level identifiers keep other schemes as well. Values are validated, ISBN-10 values are converted to ISBN-13, and the result is deduplicated by value. If the same ISBN comes from both metadata and text, it becomes one entry withMETADATA_AND_TEXT. - ISBN fallback from text. The parser always scans plain text for ISBN candidates, even when metadata already has an ISBN. It captures the first 64 KiB of the linear reading order; if that yields no ISBN, it checks the first 64 KiB of each non-linear spine item and manifest-only textual resource. The end of the linear text is not scanned. The extraction follows the isbnx approach (ISBN markers, ISBN-10/ISBN-13 patterns, 978/979 fallback), converts Arabic-Indic and Persian digits to ASCII, and validates every candidate with the same ISBN parser used for metadata. At most one unique ISBN-13 is added as a file-level ISBN identifier; the linear scan is a separate pass over the reading order.
The rules above naturally cannot guarantee fully correct data: metadata incorrectness occurs in a huge variety of forms, so the reports are worth checking by hand.
Root-level commands:
scan— scan a directory for ebook files.report— regenerate JSON reports from the database.search— search books and authors in the database (see Searching).enrich— prepare local OpenLibrary dump data (see OpenLibrary data).help— show help for any command.completion— generate shell autocompletion scripts.
Run margaret --help or margaret scan --help for details.
margaret enrich prepares the local OpenLibrary bulk data dumps that are
used to enrich scanned book metadata. The dumps are read from disk; no
OpenLibrary web API calls are made.
The dumps are published monthly on the OpenLibrary dumps page as gzipped TSV. The three families used by Margaret are:
| Dump | Approx. compressed size |
|---|---|
| editions | ~9.2 GB |
| works | ~2.9 GB |
| authors | ~0.5 GB |
enrich searches for the dump in this order:
--openlibrary-dir DIRMARGARET_OPENLIBRARY_DIR./data/openlibrary$XDG_DATA_HOME/margaret/openlibrary(default~/.local/share/margaret/openlibrary)data/openlibrarynext to the executable
If several directories hold a valid dump, the newest one wins. A combined
ol_dump_latest.txt.gz dump is also accepted and satisfies all three
families.
A combined dump is processed in one combined pass plus a small temporary author sidecar; work-level metadata is deferred in that mode. The sidecar is removed when the run finishes, but it may need about 1 GB of temporary disk. For full work-level enrichment, the three family dumps are recommended.
Use --status to see the resolved paths, the ISBN and content-head coverage
of the scan database, the in-memory matching cache and the stored match status
counts:
margaret enrich --statusThe cache summary counts the books that have a usable title, ISBN or author.
Books without any of these will end up as no_data when matching runs.
Use --download to fetch missing dumps. Without it, an interactive terminal
is asked whether to download when no dump is found; scripts fail with an
error instead of downloading implicitly.
margaret enrich --download
margaret enrich --openlibrary-dir /mnt/dumps/openlibraryDownloads are resumable: an interrupted transfer is kept as a .part file
and continued from where it stopped. Existing valid dumps are skipped.
The dumps are regenerated monthly. To move to the current month, run:
margaret enrich --refresh--refresh re-resolves the rolling dump URLs, downloads the newer dated
files, reprocesses every book against them, and refreshes the enrichment.
Old dump files are not deleted automatically; the newest valid file in the
dump directory is used.
The OpenLibrary catalog is contributed by libraries and dedicated to the public domain. The Internet Archive does not assert new copyright or proprietary rights over the database, and contributions are provided under the CC0 1.0 Universal license. Individual records may still carry rights in some jurisdictions, so review the OpenLibrary licensing page and the dump page before redistributing derived data. Margaret only reads the dumps locally; it does not call the OpenLibrary web API.
enrich matches the scanned books against the local dump, stores the match
status, confidence and top candidates in margaret.db, applies OpenLibrary
metadata to confirmed books, and runs a local metadata/content conflict
check. Enriched values are stored in the separate book_enrichment_* tables;
the raw scan metadata is left untouched. The enrichment pass reads the dump
again, but only for the selected editions, works and authors.
By default only books without a current match are processed. --force
reprocesses every book and refreshes the enrichment.
Scanned metadata is often wrong even when the ISBN is right. The scanner
stores a compact fingerprint of the first 64 KB of each book's text in
book_files.head_bloom. When an ISBN match disagrees with the scanned title
or author, enrich checks whether the OpenLibrary title and author appear in
that head. If they do, the ISBN is trusted and the book is confirmed with
method isbn+content; the scanned metadata is assumed to be wrong. The
confirmation is positive-only: a missing head or a low score never rejects a
candidate.
The fingerprint is added during scan. Files scanned by an older version do
not have one; run margaret scan <path> once more to backfill them. The
resume pass re-parses only the files whose fingerprint is missing. Use
margaret enrich --status to see how many are still missing.
| Flag | Type | Default | Description |
|---|---|---|---|
--db |
string |
margaret.db |
SQLite database file to read. |
--openlibrary-dir |
string |
"" |
Directory containing the dump files. |
--download |
bool |
false |
Download missing dumps before enriching. |
--refresh |
bool |
false |
Download the current dumps and reprocess every book (implies --download and --force). |
--status |
bool |
false |
Show resolved dump files, scan statistics, content-head coverage and match status counts, then exit. |
--force |
bool |
false |
Reprocess every book even when a stored match is current. |
--enriched-out |
string |
reports/books_enriched.json |
Write a JSON report of confirmed enriched books. |
--review-out |
string |
reports/review.json |
Write a JSON report of books needing review. |
Generated JSON reports are written under reports/ by default, relative to
the current directory. The parent directory is created automatically. Passing
an explicit --*-out path writes there instead.
| Report | Default path |
|---|---|
| failures | reports/failures.json |
| skipped | reports/skipped.json |
| duplicates | reports/duplicates.json |
| grouped books | reports/books.json |
| confirmed enriched books | reports/books_enriched.json |
| books needing review | reports/review.json |
| search hits | reports/search.json |
books_enriched.json contains only confirmed matches. Its book fields are
merged with the OpenLibrary work/edition/author metadata where available, and
it carries match (including content_score), enrichment (source, keys,
matcher/dump versions) and author_details objects for provenance.
review.json groups the uncertain books into separate arrays: probable,
ambiguous, needs_review, not_found and no_data. Empty categories are
written as []; when no book needs review, the file is not created.
enrich also runs a local conflict check. Two books with the same normalized
title and author whose content MinHash Jaccard is below 75% are marked
needs_review with reason metadata_content_mismatch: the metadata claims
they are the same book, but the content says otherwise. Both books appear in
review.json with their peer IDs and content similarity, and they are
excluded from books_enriched.json even when the OpenLibrary match itself was
confirmed.
scan writes the books.json / duplicates.json reports only with
--report, but the database always holds the full data. report regenerates
the book, duplicate, enriched and review reports from an existing database
without scanning — after a report-less scan, when the JSON files were
deleted, or when they are wanted at different paths:
margaret report
margaret report --db /tmp/test.db --books-out /tmp/books.jsonreport reads the consolidated book-level metadata but does not recompute
it; the values come from the last successful scan. Databases created before
the metadata columns existed need a fresh scan (--fresh) to populate them.
Databases containing 128-value MinHash signatures from older versions also
need a fresh scan; --resume refuses them because the signature length no
longer matches the 512-value format.
failures.json and skipped.json cannot be regenerated: they are produced
during the scan and are not stored in the database. Against an empty or
missing database the command fails with an error pointing to scan, instead
of writing empty reports or creating an empty database file.
search answers "how do I find a book or an author in this collection?"
from the database, without running a scan. Both titles and authors are
matched with a full-text index (trigram, diacritic-insensitive), so partial
words and accented names match:
margaret search "Asimov"
margaret search --author "King"
margaret search --title "Foundation" --limit 5By default both titles and authors are searched and hits are ordered by
relevance. --author restricts the search to authors, --title to titles
(both flags together mean the default). --limit caps the number of hits;
0 (the default) means unlimited. Only letters and digits count towards
the query: punctuation and FTS5 syntax characters (", *, parentheses,
:) are ignored and bare AND/OR/NOT words are dropped, so they can
never break the search; an empty query is an error.
Two matching rules follow from the trigram index: every word must match
within a single field — all words have to occur in the title or all in one
author name, so Asimov Foundation finds nothing unless one field holds
both words — and words shorter than 3 characters never match, so e.g. Ed
alone finds nothing.
The human-readable table (authors, title, file paths) goes to stdout.
With --json the hits are written in the books.json shape (same entries,
restricted to the hits in relevance order) to --json-out (default
reports/search.json):
margaret search "Asimov" --json --json-out /tmp/hits.jsonWith no hits nothing is printed and no file is created. Like report,
search is read-only: against an empty or missing database it fails with an
error naming the database file and pointing to scan, and it never creates
a missing database file.
- The command exits non-zero if the root path is inaccessible or not a directory, or if the failures report cannot be written.
- The scan can be interrupted with
Ctrl-C; it cancels gracefully, keeps the files processed so far, marks the run as canceled and writes the failures report before exiting. - A second
Ctrl-Cforces an immediate exit with code 130 if an in-flight operation cannot be canceled. This is the escape hatch for a parser or archive extractor stuck outside the cancellation path.
Scan a bookshelf with the default settings:
margaret scan ./booksScan using 8 workers and allow deep archive nesting:
margaret scan ./books --workers 8 --archive-depth 5Write the failures report to a custom location:
margaret scan ./books --failures-out /tmp/failures.jsonScan into a custom database file instead of the default ./margaret.db:
margaret scan ./books --db /tmp/test.dbUnpack password-protected rar and 7z archives:
margaret scan ./books --archive-password letmeinScan with a small in-memory spool limit, e.g. 32 MiB per member:
margaret scan ./books --spool-mem-limit 33554432Scan a bookshelf on an FTP server with the default (anonymous) login:
margaret scan ftp://192.168.178.1/network/backupScan over FTP with a username and password:
margaret scan ftp://192.168.178.1/network/backup --ftp-user backup --ftp-pass secretScan over FTP with a single worker, for routers and NAS devices that only accept one concurrent FTP connection:
margaret scan ftp://192.168.178.1/network/backup --workers 1Scan over FTP with credentials and a single worker:
margaret scan ftp://192.168.178.1/network/backup --ftp-user backup --ftp-pass secret --workers 1Search books and authors from a previous scan:
margaret search "Asimov"Search only authors, at most 5 hits:
margaret search --author "King" --limit 5Write the hits in the books.json shape:
margaret search "Foundation" --json --json-out /tmp/hits.jsonMIT — see LICENSE.