09798eb27a
Ensures text<->gold agreement for the text->triple dataset: - fetch: newest full prospectus book + later 497 supplements (time-aligned with the current N-CEN gold; fixes stale 'largest book' picking 2-5yr-old filings) - grounding: fast alias matcher (name present, variant-tolerant) AND an LLM role-check (llm_extract.py match mode, via local Ollama or remote vLLM server) that verifies the entity plays that ROLE -- catches right-name/wrong-role cases a lexical matcher over-keeps. Validated with a strong model; ~93% of gold names are present once granularity+currency are fixed. - llm_extract.py: extract (baseline) + match (grounding) modes, sliding-window with retrieval pre-filter, claim dedup, retry, ollama/vllm backends - build_rdf_dataset.py: --grounding alias|llm|name|context|none, whole-trust samples now filtered, MIN_PROSE stub guard Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
17 lines
752 B
Bash
17 lines
752 B
Bash
#!/usr/bin/env bash
|
|
# Live-Fortschritt des laufenden Prospekt-Downloads anzeigen.
|
|
# bash watch_fetch.sh -> live mitlaufen (Strg-C zum Beenden)
|
|
# bash watch_fetch.sh count -> nur kurzer Stand (X/435 + Groesse)
|
|
LOG="/private/tmp/claude-501/-Users-florianherzog-development-AISE-Admin-Thesis--Florian-Code/745ff40e-0ecf-4430-886c-33fbc724df69/tasks/bp1wdlneq.output"
|
|
DIR="data/rdf_poc/prose"
|
|
|
|
if [ "${1:-}" = "count" ]; then
|
|
n=$(ls "$DIR"/*.txt 2>/dev/null | wc -l | tr -d ' ')
|
|
last=$(grep -oE '\[[0-9]+/435\]' "$LOG" 2>/dev/null | tail -1)
|
|
echo "Fertige Trusts: $n | letzte Logzeile: $last"
|
|
du -sh "$DIR" 2>/dev/null
|
|
else
|
|
echo "Live-Download (Strg-C zum Beenden). Jede Zeile = ein Trust [n/435]:"
|
|
tail -f "$LOG"
|
|
fi
|