991715ab76
- llm_extract.py: match mode now window-parallel with retrieval pre-filter,
claim dedup, retry, and enable_thinking=false (vLLM) -> ~36x faster per call;
n_failed_windows/ok flags so an interrupted run never records bogus 0s.
- build_rdf_dataset.py:
- gold now includes the share-class level (hasShareClass/ticker/className)
- grounding modes alias|llm|name|context|none (--grounding); llm reads the
role-check verdicts from match_all.jsonl
- label stage: per-triple extractable + per-sample FULL/PARTIAL/NONE
- trainset stage: combines GROUNDED triples with focused TEXT EXCERPTS cut
around the actual provider statement (evidence), not the multi-MB book
- split --src to split trainset.jsonl (trust-level, no leakage)
- helper scripts: watch_match.sh, resume_match.sh (crash/sleep-safe resume),
finalize_dataset.sh
- final dataset: 335/335 trusts, 85% text<->gold agreement, 334 samples,
10,689 grounded triples, train/val/test 264/35/35
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
59 lines
1.3 KiB
Plaintext
59 lines
1.3 KiB
Plaintext
# ---- Large / derived data (reproducible via build_rdf_dataset.py) ----
|
|
# Raw prospectus prose fetched from EDGAR (GBs)
|
|
data/rdf_poc/prose/
|
|
# Generated training samples and splits (embed raw SEC text, 100s of MB)
|
|
data/rdf_poc/samples.jsonl
|
|
data/rdf_poc/samples_full.jsonl
|
|
data/rdf_poc/train.jsonl
|
|
data/rdf_poc/val.jsonl
|
|
data/rdf_poc/test.jsonl
|
|
# LLM match inputs/outputs (embed raw text / large)
|
|
data/rdf_poc/match_input.jsonl
|
|
data/rdf_poc/match_all.jsonl
|
|
# diagnostic / scratch files
|
|
data/rdf_poc/_*.jsonl
|
|
# prose backups from iterative fetch-strategy changes
|
|
data/rdf_poc/prose_*_bak/
|
|
# Raw SEC bulk downloads (re-downloadable from sec.gov)
|
|
data/ncen/
|
|
data/nport/
|
|
data/xbrl_rr/
|
|
|
|
# SQLite working DB
|
|
fund_data.db
|
|
fund_data.db-shm
|
|
fund_data.db-wal
|
|
|
|
# Archives
|
|
*.zip
|
|
|
|
# Python
|
|
__pycache__/
|
|
*.pyc
|
|
*.pyo
|
|
|
|
# LaTeX build artifacts
|
|
*.aux
|
|
*.log
|
|
*.out
|
|
*.toc
|
|
*.fls
|
|
*.fdb_latexmk
|
|
*.synctex.gz
|
|
|
|
# OS / editor
|
|
.DS_Store
|
|
.claude/
|
|
data/rdf_poc/samples_labelled.jsonl
|
|
data/rdf_poc/samples_full.jsonl
|
|
data/rdf_poc/labelled_partial.jsonl
|
|
data/rdf_poc/labelled_none.jsonl
|
|
data/rdf_poc/labelled_full.jsonl
|
|
data/rdf_poc/trainset.jsonl
|
|
data/rdf_poc/match_input_remaining.jsonl
|
|
data/rdf_poc/match_remaining*.jsonl
|
|
data/rdf_poc/match_all_clean*.jsonl
|
|
data/rdf_poc/match_input_sc.jsonl
|
|
data/rdf_poc/match_remaining.bak
|
|
data/rdf_poc/match_all.jsonl
|