Bibliographic metadata extraction (`klea\_utils.biblio`)¶

Automatic extraction of bibliographic metadata (title, authors, keywords, DOI, URL) from ingested documents, used to pre-fill the per-file DEFAULT entries of metadata-map.template.json. See RAG for a description of the extraction cascade.

The modules in this package are reusable utilities: the PDF, regex and DOI-resolution parts do not depend on Docling and can be used on their own (e.g. from a bundled tool).

PDF Info dict¶

PDF bibliographic metadata extraction

File: klea_utils/biblio/pdf.py

klea_utils.biblio.pdf.extract_pdf_info(path: str) → dict[str, str][source]¶

Extract bibliographic metadata from a PDF’s Info dict.

Reads the document’s metadata fields (Title, Author, Keywords, Subject) with pypdfium2, which is already installed as a Docling dependency. A DOI or URL is also picked up from the values of the standard fields (journals commonly embed the DOI in Subject). Only non-empty fields are returned, keyed lower-case (title, author, keywords, subject, doi, url).

Returns an empty dict for non-PDF files, files without metadata, and when the backend is unavailable, so callers can simply fall through to the next extraction tier.

Parameters:: path – Path to the PDF file
Returns:: Lower-case mapping of bibliographic PDF metadata fields to their string values (only non-empty fields)

Regex extraction¶

Regex-based bibliographic metadata extraction

File: klea_utils/biblio/regex.py

klea_utils.biblio.regex.DEFAULT_SCAN_LIMIT = 3000¶: Default number of leading characters scanned. Bibliographic headers (authors, keywords, DOI, URL) live on the first page, so scanning the whole document is unnecessary and would raise false-positive noise.

klea_utils.biblio.regex.DOI_RE = re.compile('\\b10\\.\\d{4,9}/[^\\s,;]+')¶

a DOI begins with “10.” followed by a 4-9 digit registrant prefix and a slash.

Type:: Loose DOI pattern

klea_utils.biblio.regex.URL_RE = re.compile('https?://[^\\s,;]+')¶: Loose URL pattern.

klea_utils.biblio.regex.extract_regex_metadata(text: str, limit: int = 3000) → dict[str, Any][source]¶

Extract bibliographic fields from text with regex heuristics.

Scans the first limit characters of text for labeled keyword and author lists, and for a DOI (labeled or loose pattern) and a labeled URL. This is a pre-population aid, not robust extraction: regex matches are acknowledged to be noisy, and the caller (the metadata extraction cascade) falls back to it only when more authoritative sources have failed. Only non-empty keys are returned.

Parameters:

text – Text to scan (e.g. the joined body text of a document)
limit – Number of leading characters to scan

Returns:

Mapping with any of keywords (list), authors (list), doi (str), url (str)

DOI resolution¶

DOI resolution via Crossref, OpenAlex and Semantic Scholar

File: klea_utils/biblio/doi.py

Bases: BaseModel

Normalised bibliographic record shared across the DOI services.

model_config = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class klea_utils.biblio.doi.DoiResolver(cache_dir: str | Path, mailto: str | None = None, timeout: float = 10.0, transport: BaseTransport | None = None, logger_obj: Logger | None = None)[source]¶

Bases: object

Resolve DOIs to bibliographic records via three web services.

Services are queried in round-robin order across calls, so a bulk ingestion does not hammer a single API; when the primary is rate-limited (HTTP 429) or fails, the other services are tried as a fallback. Successful records are cached to a JSON file on disk, so re-ingests never re-query the APIs.

Polite-pool attribution is sent when KLEA_INGEST_MAILTO (or the mailto argument) is set – Crossref and OpenAlex both honour a mailto parameter to raise their rate limits.

SERVICE_ORDER = ('crossref', 'openalex', 'semantic_scholar')¶: Services, in round-robin order.

close() → None[source]¶: Close the underlying HTTP client.

resolve(doi: str) → BiblioRecord | None[source]¶

Resolve doi to a normalised bibliographic record.

Returns the cached record immediately when present. Otherwise queries the services in round-robin order, falling back to the remaining services when one is rate-limited or fails. A success is cached to disk. Returns None when the DOI is invalid or no service could resolve it.

Parameters:: doi – DOI string (URL/prefix wrappers are stripped)
Returns:: Normalised record, or None

klea_utils.biblio.doi.normalize_doi(doi: str) → str[source]¶

Normalise a DOI by stripping common URL/prefix wrappers.

https://doi.org/10.x/y, http://dx.doi.org/10.x/y and doi: 10.x/y all become 10.x/y. Trailing punctuation is removed.

Parameters:: doi – DOI string, possibly wrapped in a URL or prefix
Returns:: Normalised DOI, or "" when doi is empty

Docling structured signals¶

Docling-based bibliographic metadata extraction

File: klea_utils/biblio/docling.py

klea_utils.biblio.docling.extract_docling_structured(dl_doc, file_path: str) → dict[source]¶

Extract the free, reliably-structured bibliographic signals from a Docling document.

These come straight from Docling’s layout model and document origin, so they do not depend on regex heuristics:

title – the first TitleItem’s text, falling back to the source file’s stem
source_type – the document’s origin.mimetype
source_url – the origin.uri when it is an http(s) URL (web-sourced inputs only)
urls – deduplicated http(s) hyperlink URLs found on any text item (from markdown/HTML links)

Parameters:

dl_doc – A Docling DoclingDocument
file_path – Path of the source file (for the title fallback)

Returns:

Dict with any of title, source_type, source_url, urls

klea_utils.biblio.docling.extract_layout_region(dl_doc, page: int = 1, frac: float = 0.35) → str | None[source]¶

Return the text of the top frac of page as a single string.

Filters the document’s text items to those whose bounding box starts within the top frac of page (TOPLEFT origin) and joins their text. The first-page header is where authors, keywords and the DOI live, so this gives the regex tier a focused region to scan rather than the whole document.

Parameters:

dl_doc – A Docling DoclingDocument
page – Page number to inspect
frac – Fraction of the page height that counts as the header

Returns:

Joined region text, or None when there is nothing in the region

Extraction cascade¶

Bibliographic metadata extraction cascade

File: klea_utils/biblio/extract.py

class klea_utils.biblio.extract.Resolver(*args, **kwargs)[source]¶

Bases: Protocol

Protocol for objects that can resolve a DOI to a record.

DoiResolver implements this; tests and other callers may substitute any object with a compatible resolve method.

resolve(doi: str) → BiblioRecord | None[source]¶: Resolve doi to a record, or None on failure.

klea_utils.biblio.extract.extract_metadata(dl_doc, file_path: str, pdf_path: str | None = None, resolver: Resolver | None = None) → dict[source]¶

Extract bibliographic metadata for a converted document.

The full-cascade entry point, used when a Docling document is available (fresh conversion). Tiers are applied in precedence order, most authoritative first; each tier only fills fields the tiers above it have not already set (gap-fill), so a resolved record always wins:

doi-service – a DOI discovered by any tier below is resolved via Crossref/OpenAlex/Semantic Scholar (round-robin across calls, fallback on rate limits, disk-cached). Its title, authors, year, venue and DOI override everything else. Skipped when resolver is None.
pdf-info – the PDF Info dict (title, authors, keywords, doi, url), read with pypdfium2. Local and fast, but often empty: many publishers ship no bibliographic fields in it.
docling – the free structured signals from Docling’s layout model: the title item, the origin mimetype/URI, and the hyperlinks on text items.
layout-regex – regex over the focused first-page header region (the top fraction of page one, selected via the layout bounding boxes), where authors, keywords and the DOI live.
regex – regex over the first DEFAULT_SCAN_LIMIT characters of the whole document text. A broader net than the layout region, so it still catches a DOI/URL sitting in a first-page footer.

Output – a flat dict of non-empty fields (title, authors, keywords, year, venue, doi, url, source_type, source_url, urls) plus two internal keys:

_metadata_complete – True only when a full DOI record (title + authors + year) or a full PDF Info dict (title + author + keywords) was obtained; otherwise False, signalling the researcher to review the pre-populated map.
_sources – the tiers that contributed at least one field, in precedence order (e.g. ["doi-service", "regex"]).

The abstract of a resolved record is used only as completeness evidence and is deliberately NOT included in the output: the abstract is already part of the chunked document, so persisting it per-chunk would duplicate it.

Parameters:

dl_doc – A Docling DoclingDocument
file_path – Path of the source file
pdf_path – Path to a PDF file, when the source is a PDF
resolver – Optional Resolver; when None the DOI-service tier is skipped (no network)

Returns:

Flat metadata dict including the _metadata_complete and _sources internal keys

klea_utils.biblio.extract.extract_metadata_from_text(text: str, file_path: str, pdf_path: str | None = None, resolver: Resolver | None = None) → dict[source]¶

Extract bibliographic metadata from document text.

The text-only entry point of the cascade, used when no Docling document is available (e.g. cached chunks). Tiers are applied in precedence order, most authoritative first; each tier only fills fields not already set:

doi-service – a DOI found by a tier below is resolved, and its record overrides everything else. Skipped when resolver is None.
pdf-info – the PDF Info dict, when pdf_path is a PDF.
regex – the first DEFAULT_SCAN_LIMIT characters of text (the first chunks: the title and front matter).

Parameters:

text – Document text (e.g. the joined cached chunk text)
file_path – Path of the source file
pdf_path – Path to a PDF file, when the source is a PDF
resolver – Optional Resolver; when None the DOI-service tier is skipped (no network)

Returns:

Flat metadata dict including the _metadata_complete and _sources internal keys

Bibliographic metadata extraction (klea\_utils.biblio)¶

PDF Info dict¶

Regex extraction¶

DOI resolution¶

Docling structured signals¶

Extraction cascade¶

Bibliographic metadata extraction (`klea\_utils.biblio`)¶