09798eb27a
Ensures text<->gold agreement for the text->triple dataset: - fetch: newest full prospectus book + later 497 supplements (time-aligned with the current N-CEN gold; fixes stale 'largest book' picking 2-5yr-old filings) - grounding: fast alias matcher (name present, variant-tolerant) AND an LLM role-check (llm_extract.py match mode, via local Ollama or remote vLLM server) that verifies the entity plays that ROLE -- catches right-name/wrong-role cases a lexical matcher over-keeps. Validated with a strong model; ~93% of gold names are present once granularity+currency are fixed. - llm_extract.py: extract (baseline) + match (grounding) modes, sliding-window with retrieval pre-filter, claim dedup, retry, ollama/vllm backends - build_rdf_dataset.py: --grounding alias|llm|name|context|none, whole-trust samples now filtered, MIN_PROSE stub guard Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
47 lines
918 B
Plaintext
47 lines
918 B
Plaintext
# ---- Large / derived data (reproducible via build_rdf_dataset.py) ----
|
|
# Raw prospectus prose fetched from EDGAR (GBs)
|
|
data/rdf_poc/prose/
|
|
# Generated training samples and splits (embed raw SEC text, 100s of MB)
|
|
data/rdf_poc/samples.jsonl
|
|
data/rdf_poc/samples_full.jsonl
|
|
data/rdf_poc/train.jsonl
|
|
data/rdf_poc/val.jsonl
|
|
data/rdf_poc/test.jsonl
|
|
# LLM match inputs/outputs (embed raw text / large)
|
|
data/rdf_poc/match_input.jsonl
|
|
data/rdf_poc/match_all.jsonl
|
|
# diagnostic / scratch files
|
|
data/rdf_poc/_*.jsonl
|
|
# prose backups from iterative fetch-strategy changes
|
|
data/rdf_poc/prose_*_bak/
|
|
# Raw SEC bulk downloads (re-downloadable from sec.gov)
|
|
data/ncen/
|
|
data/nport/
|
|
data/xbrl_rr/
|
|
|
|
# SQLite working DB
|
|
fund_data.db
|
|
fund_data.db-shm
|
|
fund_data.db-wal
|
|
|
|
# Archives
|
|
*.zip
|
|
|
|
# Python
|
|
__pycache__/
|
|
*.pyc
|
|
*.pyo
|
|
|
|
# LaTeX build artifacts
|
|
*.aux
|
|
*.log
|
|
*.out
|
|
*.toc
|
|
*.fls
|
|
*.fdb_latexmk
|
|
*.synctex.gz
|
|
|
|
# OS / editor
|
|
.DS_Store
|
|
.claude/
|