UI — store creation CLI¶

CLI for creating stores from documents

File: klea_utils/ui/stores_create.py

klea_utils.ui.stores_create.build(source_dir: str = <typer.models.ArgumentInfo object>, collection_name: str = <typer.models.OptionInfo object>, store_path: str = <typer.models.OptionInfo object>, embedding_model: str = <typer.models.OptionInfo object>, max_tokens: int = <typer.models.OptionInfo object>, ocr: bool = <typer.models.OptionInfo object>, metadata_map_path: str = <typer.models.OptionInfo object>, bm25_store: str = <typer.models.OptionInfo object>, embed_batch_size: int = <typer.models.OptionInfo object>, force: bool = <typer.models.OptionInfo object>)[source]¶

Full pipeline: chunk, embed, and write to a vector store.

Processes all files in SOURCE_DIR: converts them with Docling, chunks them, embeds them, and writes to the vector store. Processed chunks are cached in <source_dir>/.klea-cache/ so subsequent runs (e.g. with --metadata-map) skip conversion.

The --bm25-store option (default <collection>.pkl in the current directory) writes the combined chunked documents to a single pickle file that can be used as a BM25 store.

The optional --metadata-map / -M flag accepts a JSON file organised by source file. Within each file entry, the most specific heading chain match wins; a DEFAULT entry provides fallback for any heading not listed.

Example metadata-map.json:

{
    "PrimerOnCElegans.md": {
        "DEFAULT": {},
        "C. elegans tissue morphology": {
            "url": "https://example.com/worm"
        }
    },
    "c302-paper.pdf": {
        "DEFAULT": {
            "url": "https://example.com/c302"
        }
    }
}

klea_utils.ui.stores_create.chunk(source_dir: str = <typer.models.ArgumentInfo object>, max_tokens: int = <typer.models.OptionInfo object>, ocr: bool = <typer.models.OptionInfo object>, force: bool = <typer.models.OptionInfo object>)[source]¶

Chunk and cache documents without writing to a vector store.

Converts all files in SOURCE_DIR with Docling, chunks them, and caches the result in <source_dir>/.klea-cache/. Also writes a metadata-map.template.json file organised by source file, with empty {} placeholders for each heading chain. Fill in the metadata values and pass the file to klea-stores-create store --metadata-map.

klea_utils.ui.stores_create.store(source_dir: str = <typer.models.ArgumentInfo object>, collection_name: str = <typer.models.OptionInfo object>, store_path: str = <typer.models.OptionInfo object>, embedding_model: str = <typer.models.OptionInfo object>, max_tokens: int = <typer.models.OptionInfo object>, ocr: bool = <typer.models.OptionInfo object>, metadata_map_path: str = <typer.models.OptionInfo object>, bm25_store: str = <typer.models.OptionInfo object>, embed_batch_size: int = <typer.models.OptionInfo object>, force: bool = <typer.models.OptionInfo object>)[source]¶

Write cached document chunks to a vector store.

Reads previously cached chunks from <source_dir>/.klea-cache/, optionally applies a metadata map (per-file format), and writes them to the vector store. Unseen files are converted and chunked on the fly.

The --bm25-store option (default <collection>.pkl in the current directory) writes the combined chunked documents to a single pickle file that can be used as a BM25 store.

Run klea-stores-create chunk first to populate the cache and generate a metadata-map.template.json.