09798eb27a
Ensures text<->gold agreement for the text->triple dataset: - fetch: newest full prospectus book + later 497 supplements (time-aligned with the current N-CEN gold; fixes stale 'largest book' picking 2-5yr-old filings) - grounding: fast alias matcher (name present, variant-tolerant) AND an LLM role-check (llm_extract.py match mode, via local Ollama or remote vLLM server) that verifies the entity plays that ROLE -- catches right-name/wrong-role cases a lexical matcher over-keeps. Validated with a strong model; ~93% of gold names are present once granularity+currency are fixed. - llm_extract.py: extract (baseline) + match (grounding) modes, sliding-window with retrieval pre-filter, claim dedup, retry, ollama/vllm backends - build_rdf_dataset.py: --grounding alias|llm|name|context|none, whole-trust samples now filtered, MIN_PROSE stub guard Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
12 lines
457 B
Plaintext
12 lines
457 B
Plaintext
requests>=2.31.0
|
|
beautifulsoup4>=4.12.0
|
|
lxml>=5.1.0
|
|
pandas>=2.2.0
|
|
tqdm>=4.66.0
|
|
# SQLite3 is part of Python stdlib — no extra package needed
|
|
#
|
|
# build_rdf_dataset.py / llm_extract.py / score_baseline.py use only the Python
|
|
# stdlib (urllib, json, csv, re), so no extra packages are required for the RDF
|
|
# dataset pipeline. The LLM extraction baseline needs a running Ollama server
|
|
# (https://ollama.com) with a model pulled, e.g.: ollama pull qwen3.6:35b
|