klea-stores-create¶
Create stores from documents (vector stores and BM25 stores).
The build and store commands accept --bm25-store <path>,
which writes the combined chunked documents to a single pickle that can
be configured as a bm25_stores entry in the RAG config. When
--bm25-store is omitted, the corpus is written to
<collection>.pkl in the current directory.
Three options deserve special attention:
--collection– the collection name inside the store. It must match thenameof the store’svector_stores/bm25_storesentry in the RAG config file (e.g.klea.json); retrieval looks stores up by name, so a mismatch silently returns no results.--store– for local Chroma stores this points at the store folder; the database file inside it is always namedchroma.sqlite3(the filename is not configurable), so passing the path of an existing file is rejected. A folder that does not exist yet is created. One Chroma store file can hold several collections, so--collectionselects which collection within the file is used.--bm25-store– path to the combined corpus pickle (see above).
Usage: klea-stores-create [OPTIONS] COMMAND [ARGS]...
Create stores from documents
╭─ Options ──────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current │
│ shell. │
│ --show-completion Show completion for the current │
│ shell, to copy it or customize the │
│ installation. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────╮
│ build Full pipeline: chunk, embed, and write to a vector store. │
│ chunk Chunk and cache documents without writing to a vector │
│ store. │
│ store Write cached document chunks to a vector store. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create build [OPTIONS] SOURCE_DIR
Full pipeline: chunk, embed, and write to a vector store.
Processes all files in SOURCE_DIR: converts them with Docling,
chunks them, embeds them, and writes to the vector store.
Processed chunks are cached in ``<source_dir>/.klea-cache/`` so
subsequent runs (e.g. with ``--metadata-map``) skip conversion.
The ``--bm25-store`` option (default ``<collection>.pkl`` in the
current directory) writes the combined chunked documents to a single
pickle file that can be used as a BM25 store.
The optional ``--metadata-map`` / ``-M`` flag accepts a JSON file
organised by source file. Within each file entry, the most specific
heading chain match wins; a ``DEFAULT`` entry provides fallback for
any heading not listed.
Example metadata-map.json::
{
"PrimerOnCElegans.md": {
"DEFAULT": {},
"C. elegans tissue morphology": {
"url": "https://example.com/worm"
}
},
"c302-paper.pdf": {
"DEFAULT": {
"url": "https://example.com/c302"
}
}
}
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ * --collection -n TEXT Collection name │
│ for the vector │
│ store. Must match │
│ the 'name' of the │
│ corresponding │
│ vector_stores/bm2… │
│ entry in the RAG │
│ config file (e.g. │
│ klea.json); a │
│ different name on │
│ an existing store │
│ file creates a new │
│ collection │
│ [required] │
│ * --store -s TEXT Vector store URI │
│ (e.g. │
│ chroma:/path/to/s… │
│ For local Chroma │
│ stores, point at │
│ the store folder: │
│ the database file │
│ inside it is │
│ always named │
│ chroma.sqlite3 │
│ [required] │
│ --model -m TEXT Embedding model │
│ identifier │
│ [default: │
│ ollama:bge-m3:lat… │
│ --max-tokens INTEGER Maximum tokens per │
│ chunk │
│ [default: 450] │
│ --ocr --no-ocr Whether to perform │
│ optical character │
│ recognition (OCR) │
│ during PDF │
│ conversion │
│ (default: on). │
│ Keep for │
│ scanned/image │
│ PDFs; disable for │
│ text-based PDFs to │
│ speed up │
│ conversion │
│ significantly │
│ [default: ocr] │
│ --metadata-map -M TEXT JSON file keyed by │
│ source filename; │
│ each file entry │
│ maps heading │
│ chains to metadata │
│ dicts (with │
│ per-file DEFAULT │
│ fallback) │
│ --bm25-store TEXT Write the combined │
│ document corpus to │
│ this path for BM25 │
│ retrieval (a │
│ pickle of all │
│ chunked │
│ documents). │
│ Defaults to │
│ <collection>.pkl │
│ in the current │
│ directory; the │
│ file can be moved │
│ after creation │
│ [default: │
│ (<collection>.pkl… │
│ --embed-batch-size INTEGER Number of chunks │
│ embedded per store │
│ write call. │
│ Smaller values │
│ report progress │
│ more frequently; │
│ larger values │
│ reduce per-request │
│ overhead on very │
│ large corpora │
│ [default: 256] │
│ --force -f Re-process all │
│ files even if │
│ unchanged │
│ --help Show this message │
│ and exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create chunk [OPTIONS] SOURCE_DIR
Chunk and cache documents without writing to a vector store.
Converts all files in SOURCE_DIR with Docling, chunks them, and
caches the result in ``<source_dir>/.klea-cache/``. Also writes a
``metadata-map.template.json`` file organised by source file, with
empty ``{}`` placeholders for each heading chain. Fill in the
metadata values and pass the file to
``klea-stores-create store --metadata-map``.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ --max-tokens INTEGER Maximum tokens per chunk │
│ [default: 450] │
│ --ocr --no-ocr Whether to perform optical │
│ character recognition (OCR) │
│ during PDF conversion │
│ (default: on). Keep for │
│ scanned/image PDFs; disable │
│ for text-based PDFs to │
│ speed up conversion │
│ significantly │
│ [default: ocr] │
│ --force -f Re-process all files even │
│ if unchanged │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create store [OPTIONS] SOURCE_DIR
Write cached document chunks to a vector store.
Reads previously cached chunks from ``<source_dir>/.klea-cache/``,
optionally applies a metadata map (per-file format), and writes
them to the vector store. Unseen files are converted and chunked
on the fly.
The ``--bm25-store`` option (default ``<collection>.pkl`` in the
current directory) writes the combined chunked documents to a single
pickle file that can be used as a BM25 store.
Run ``klea-stores-create chunk`` first to populate the cache and
generate a ``metadata-map.template.json``.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ * --collection -n TEXT Collection name │
│ for the vector │
│ store. Must match │
│ the 'name' of the │
│ corresponding │
│ vector_stores/bm2… │
│ entry in the RAG │
│ config file (e.g. │
│ klea.json); a │
│ different name on │
│ an existing store │
│ file creates a new │
│ collection │
│ [required] │
│ * --store -s TEXT Vector store URI │
│ (e.g. │
│ chroma:/path/to/s… │
│ For local Chroma │
│ stores, point at │
│ the store folder: │
│ the database file │
│ inside it is │
│ always named │
│ chroma.sqlite3 │
│ [required] │
│ --model -m TEXT Embedding model │
│ identifier │
│ [default: │
│ ollama:bge-m3:lat… │
│ --max-tokens INTEGER Maximum tokens per │
│ chunk (for files │
│ not yet cached) │
│ [default: 450] │
│ --ocr --no-ocr Whether to perform │
│ optical character │
│ recognition (OCR) │
│ during PDF │
│ conversion │
│ (default: on). │
│ Keep for │
│ scanned/image │
│ PDFs; disable for │
│ text-based PDFs to │
│ speed up │
│ conversion │
│ significantly │
│ [default: ocr] │
│ --metadata-map -M TEXT JSON file keyed by │
│ source filename; │
│ each file entry │
│ maps heading │
│ chains to metadata │
│ dicts (with │
│ per-file DEFAULT │
│ fallback) │
│ --bm25-store TEXT Write the combined │
│ document corpus to │
│ this path for BM25 │
│ retrieval (a │
│ pickle of all │
│ chunked │
│ documents). │
│ Defaults to │
│ <collection>.pkl │
│ in the current │
│ directory; the │
│ file can be moved │
│ after creation │
│ [default: │
│ (<collection>.pkl… │
│ --embed-batch-size INTEGER Number of chunks │
│ embedded per store │
│ write call. │
│ Smaller values │
│ report progress │
│ more frequently; │
│ larger values │
│ reduce per-request │
│ overhead on very │
│ large corpora │
│ [default: 256] │
│ --force -f Re-process all │
│ files even if │
│ unchanged │
│ --help Show this message │
│ and exit. │
╰────────────────────────────────────────────────────────────────────╯