herzogfloria/Portfolio_Risk_Management
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Setup: exact candidate symbols and a quiet run on students' Sparks

Browse Source 
On a student Spark the installer's data step printed hundreds of red
yfinance "Quote not found for symbol" lines and built a universe of 652
instead of ~745 stocks. Cause: the iShares holdings download (German site)
answers 404 since October 2026, and setup fell back to Wikipedia's STOXX
600 table, whose Reuters-style codes give Yahoo symbols that do not exist
(BNPP.PA for BNP.PA, NOV-N.SW for NOVN.SW, VESTAS.CO for VWS.CO).

- sources: the iShares file is read from the Swiss site (same German CSV,
  same parser), the German address stays as a spare; when iShares fails,
  setup uses the snapshot shipped in quantcore/data/static and not the
  Wikipedia guesses (now a last resort when no snapshot ships); only a
  primary source refreshes the snapshot
- yahoo: yfinance's per-symbol ERROR lines are silenced during the
  metadata step and summarised in one line
- install.sh: builds the data with `qc setup --offline`, so every student
  gets the universe of the shipped lists (the course data set's); new
  --rebuild-data moves the existing data aside and builds it anew
- qc.sh: the image build attempt logs to ~/quantcore/build.log; a Spark
  that allows no build shows one plain line, then the no-build setup
- tests for the source order, the address fallback and the summary

Tested on a stripped-down machine (no git/curl/wget/python3, broken
APPTAINER_TMPDIR): install, then install.sh --rebuild-data built 746
stocks (382 US, 364 Europe) from the 1117 shipped candidates in under
10 minutes with no yfinance error line; self-test passes; qc upgrade
rebuilt and restarted. 142 tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
herzogflorian 2026-10-07 14:03:20 +02:00
parent fdbb69ca98
commit 772aa2de84
6 changed files with 206 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
deploy/apptainer/qc.sh
View File

@ -329,9 +329,8 @@ get_quantcore_image() { # force(0|1)
say " present: virtualenv $QC_VENV (no-build fallback; reinstall: $0 build)"; return 0 say " present: virtualenv $QC_VENV (no-build fallback; reinstall: $0 build)"; return 0
fi fi
build_sif || { build_sif || {
say "" say " -> using the no-build setup instead: a plain Python image plus a virtualenv in $QC_VENV"
say " The image build is not possible on this Spark without root. Falling back to the no-build" say " (same libraries, same results)"
say " setup: a plain Python image plus a virtualenv in $QC_VENV (same libraries, same results)."
build_venv build_venv
} ;; } ;;
venv) venv)
@ -348,15 +347,18 @@ get_quantcore_image() { # force(0|1)
} }
build_sif() { # build quantcore.sif from the definition; non-zero when this Apptainer cannot build build_sif() { # build quantcore.sif from the definition; non-zero when this Apptainer cannot build
local dst="$QC_IMAGES/quantcore.sif" local dst="$QC_IMAGES/quantcore.sif" log="$QC_HOME/build.log" reason
mkdir -p "$QC_IMAGES" mkdir -p "$QC_IMAGES"
say " building $dst from $REPO_DIR (a few minutes)" say " building $dst from $REPO_DIR (a few minutes; log: $log)"
prepare_apptainer_dirs prepare_apptainer_dirs
rm -f "$dst.part" rm -f "$dst.part"
# %files paths in the definition are relative to the repository root. Without root Apptainer # %files paths in the definition are relative to the repository root. Without root Apptainer
# builds through --fakeroot (user namespaces); a Spark may not allow that. # builds through --fakeroot (user namespaces); many Sparks do not allow that. The output goes to a
if ! (cd "$REPO_DIR" && "$APPTAINER" build "$dst.part" deploy/apptainer/quantcore.def); then # log: on such a Spark Apptainer's red ERROR line is expected, and the caller falls back.
if ! (cd "$REPO_DIR" && "$APPTAINER" build "$dst.part" deploy/apptainer/quantcore.def) >"$log" 2>&1; then
rm -f "$dst.part" rm -f "$dst.part"
reason="$(grep -E 'FATAL|ERROR' "$log" | tail -n 1 | sed -e 's/\x1b\[[0-9;]*m//g' -e 's/^[A-Z]* *: *//')"
say " no image build on this machine${reason:+ ($reason)}"
return 1 return 1
fi fi
mv -f "$dst.part" "$dst" mv -f "$dst.part" "$dst"
@ -391,8 +393,8 @@ cmd_build() { # rebuild after a code change, in the mode in use
if [[ "$mode" == venv ]]; then if [[ "$mode" == venv ]]; then
build_venv build_venv
elif [[ "$QC_IMAGE" == build ]]; then elif [[ "$QC_IMAGE" == build ]]; then
build_sif || die "the image build failed (see above). Without root it needs Apptainer's --fakeroot. Use the build_sif || die "the image build failed (log: $QC_HOME/build.log). Without root it needs Apptainer's
no-build fallback instead: set QC_IMAGE=venv in $CONFIG and run: $0 build" --fakeroot. Use the no-build setup instead: set QC_IMAGE=venv in $CONFIG and run: $0 build"
else else
die "QC_IMAGE=$QC_IMAGE is a prebuilt image; nothing to build" die "QC_IMAGE=$QC_IMAGE is a prebuilt image; nothing to build"
fi fi

16
docs/SPARK_APPTAINER.md
View File

@ -45,7 +45,9 @@ The installer takes 20 to 30 minutes the first time and prints each step:
2. It downloads the code to `~/Portfolio_Risk_Management`. If the Spark has no git, it uses git 2. It downloads the code to `~/Portfolio_Risk_Management`. If the Spark has no git, it uses git
from a small container. from a small container.
3. It gets the cuOpt image and the quantcore image. 3. It gets the cuOpt image and the quantcore image.
4. It builds the market data from Yahoo Finance, which takes about 15 minutes. 4. It builds the market data from Yahoo Finance, which takes about 15 minutes. The candidate
stocks come from the list shipped in the repository, so every student gets the same universe.
Companies that Yahoo no longer lists are left out, and setup names them in one summary line.
5. It starts the service, runs the self-test and installs the command `qc`. 5. It starts the service, runs the self-test and installs the command `qc`.
With a prepared data set from your lecturer, the data step takes seconds instead. Pass the file With a prepared data set from your lecturer, the data step takes seconds instead. Pass the file
@ -78,6 +80,7 @@ qc status # what runs, whether cuOpt is reachable, how much data
qc logs # API log; `qc logs cuopt` for the solver, add -f to follow qc logs # API log; `qc logs cuopt` for the solver, add -f to follow
qc update # the newest prices and FX rates from Yahoo Finance qc update # the newest prices and FX rates from Yahoo Finance
qc upgrade # the newest version of the code: download, rebuild, restart qc upgrade # the newest version of the code: download, rebuild, restart
~/Portfolio_Risk_Management/install.sh --rebuild-data # set the data aside and build it anew
qc stop # stop both containers qc stop # stop both containers
``` ```
@ -215,6 +218,7 @@ and restarts the service.
| Yahoo errors during `qc setup` or `qc update` | Yahoo limits how fast it answers. Run the command again later; stored data stays intact. | | Yahoo errors during `qc setup` or `qc update` | Yahoo limits how fast it answers. Run the command again later; stored data stays intact. |
| `FATAL: ... failed to create build parent dir: stat /scratch/...` | Your account sets `APPTAINER_TMPDIR` to a folder that does not exist. The installer and `qc` now detect this, use a folder that works, and save the choice in `~/quantcore/quantcore.env`. Run the installer again. | | `FATAL: ... failed to create build parent dir: stat /scratch/...` | Your account sets `APPTAINER_TMPDIR` to a folder that does not exist. The installer and `qc` now detect this, use a folder that works, and save the choice in `~/quantcore/quantcore.env`. Run the installer again. |
| `git: command not found` | Use the one-line installer. It needs no git. | | `git: command not found` | Use the one-line installer. It needs no git. |
| Hundreds of red `Quote not found for symbol` lines during setup | Your installation predates the fix. Its setup used guessed symbols, so many European companies are missing from your data. Run `qc upgrade`, then `~/Portfolio_Risk_Management/install.sh --rebuild-data`. |
| `qc: command not found` right after the installation | Open a new terminal, or run `export PATH="$HOME/.local/bin:$PATH"`. | | `qc: command not found` right after the installation | Open a new terminal, or run `export PATH="$HOME/.local/bin:$PATH"`. |
| `qc install` stops with "no 'instance run'" | The Spark's Apptainer is older than 1.1. Ask the administrator to update it. | | `qc install` stops with "no 'instance run'" | The Spark's Apptainer is older than 1.1. Ask the administrator to update it. |
| The cuOpt pull fails with "no space left on device" | Apptainer unpacks images in `/tmp`. Point it at a larger directory: `APPTAINER_TMPDIR=~/quantcore/tmp qc pull-cuopt`. | | The cuOpt pull fails with "no space left on device" | Apptainer unpacks images in `/tmp`. Point it at a larger directory: `APPTAINER_TMPDIR=~/quantcore/tmp qc pull-cuopt`. |
@ -278,6 +282,10 @@ code with git from a container, installed, started and passed the self-test. `qc
a new commit, rebuilt quantcore and restarted it in under a minute. `qc setup` built a data set from a new commit, rebuilt quantcore and restarted it in under a minute. `qc setup` built a data set from
scratch inside the container. scratch inside the container.
`qc setup` takes the STOXX 600 candidates from the iShares holdings file. In October 2026 that link **Where the candidate stocks come from.** The installer runs `qc setup --offline`. It takes the
answered 404, and setup fell back to Wikipedia's list of 465 names. A data set built by a student can candidates from the lists shipped in `quantcore/data/static/`: the S&P 500 and the iShares STOXX
therefore have fewer European stocks than the prepared one. Europe 600 holdings of September 2026, from which the course data set was built. `qc setup` without
`--offline` fetches the current lists instead. In October 2026 iShares moved the German download, so
setup now reads the same file from iShares' Swiss site. If that fails too, setup uses the shipped
list. It no longer uses Wikipedia's STOXX 600 table, whose codes mostly do not exist on Yahoo, for
example `BNPP.PA` instead of `BNP.PA`.

24
install.sh
View File

@ -9,7 +9,8 @@
# #
# Options go after `bash -s --`, e.g. ... | bash -s -- --data ~/Downloads/quantcore_data.tar.gz # Options go after `bash -s --`, e.g. ... | bash -s -- --data ~/Downloads/quantcore_data.tar.gz
# --data FILE|URL import this prepared data set instead of building one from Yahoo Finance # --data FILE|URL import this prepared data set instead of building one from Yahoo Finance
# --no-data skip the data step (later: qc setup, or qc data-import FILE) # --no-data skip the data step (later: qc setup --offline, or qc data-import FILE)
# --rebuild-data move the existing data set aside (to ~/quantcore/data.<date>) and build a new one
# --no-start install only, do not start the services # --no-start install only, do not start the services
# --dir DIR where the code goes (default ~/Portfolio_Risk_Management) # --dir DIR where the code goes (default ~/Portfolio_Risk_Management)
# #
@ -18,7 +19,10 @@
# point to a directory that does not exist (seen on managed machines: /scratch/$USER/...) # point to a directory that does not exist (seen on managed machines: /scratch/$USER/...)
# 2. downloads the code with git -- or, without git on the machine, with git in a small container # 2. downloads the code with git -- or, without git on the machine, with git in a small container
# 3. deploy/apptainer/qc.sh install: the cuOpt image and the quantcore image (or its no-build fallback) # 3. deploy/apptainer/qc.sh install: the cuOpt image and the quantcore image (or its no-build fallback)
# 4. market data: imports --data, or builds the data set from Yahoo Finance (~15 min) when there is none # 4. market data: imports --data, or builds the data set from Yahoo Finance (~15 min) when there is none.
# The candidate stocks come from the list shipped in the repository (the iShares STOXX Europe 600
# and S&P 500 constituents the course data was built from), so every installation gets the same
# universe; prices and company data come from Yahoo.
# 5. starts cuOpt and the API, runs the self-test, and installs the command `qc` # 5. starts cuOpt and the API, runs the self-test, and installs the command `qc`
# #
# Everything is wrapped in main(), which runs only after bash has read the whole script: with # Everything is wrapped in main(), which runs only after bash has read the whole script: with
@ -29,7 +33,7 @@ main() {
local repo_url="${QC_REPO_URL:-https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git}" local repo_url="${QC_REPO_URL:-https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git}"
local branch="${QC_BRANCH:-main}" local branch="${QC_BRANCH:-main}"
local dir="${QC_DIR:-$HOME/Portfolio_Risk_Management}" local dir="${QC_DIR:-$HOME/Portfolio_Risk_Management}"
local data="" do_data=1 do_start=1 local data="" do_data=1 do_start=1 rebuild_data=0
QC_HOME="${QC_HOME:-$HOME/quantcore}" QC_HOME="${QC_HOME:-$HOME/quantcore}"
APPTAINER="${APPTAINER:-apptainer}" APPTAINER="${APPTAINER:-apptainer}"
GIT_IMAGE="${QC_GIT_IMAGE:-docker://alpine/git:2.54.0}" GIT_IMAGE="${QC_GIT_IMAGE:-docker://alpine/git:2.54.0}"
@ -43,7 +47,8 @@ main() {
--no-start) do_start=0 ;; --no-start) do_start=0 ;;
--dir) dir="${2:?--dir needs a directory}"; shift ;; --dir) dir="${2:?--dir needs a directory}"; shift ;;
--dir=*) dir="${1#--dir=}" ;; --dir=*) dir="${1#--dir=}" ;;
-h|--help) sed -n '2,26p' "${BASH_SOURCE[0]:-/dev/null}" 2>/dev/null || true; return 0 ;; --rebuild-data) rebuild_data=1 ;;
-h|--help) sed -n '2,30p' "${BASH_SOURCE[0]:-/dev/null}" 2>/dev/null || true; return 0 ;;
*) die "unknown option: $1 (see the top of install.sh)" ;; *) die "unknown option: $1 (see the top of install.sh)" ;;
esac esac
shift shift
@ -88,6 +93,14 @@ main() {
fi fi
step "4/5 Market data" step "4/5 Market data"
if [[ "$rebuild_data" == 1 ]] && "$qc" has-data; then
local data_dir aside
data_dir="$("$qc" env | sed -n 's|^QC_DATA=||p')"
aside="$data_dir.$(date +%Y%m%d-%H%M%S)"
"$qc" stop >/dev/null 2>&1 || true
mv "$data_dir" "$aside"
say " moved the existing data set to $aside"
fi
if [[ -n "$data" ]]; then if [[ -n "$data" ]]; then
local file="$data" local file="$data"
if [[ "$data" == http://* || "$data" == https://* ]]; then if [[ "$data" == http://* || "$data" == https://* ]]; then
@ -100,7 +113,8 @@ main() {
say " present -- newest prices later with: qc update" say " present -- newest prices later with: qc update"
elif [[ "$do_data" == 1 ]]; then elif [[ "$do_data" == 1 ]]; then
say " no data yet: building the data set from Yahoo Finance (~15 min, needs internet)" say " no data yet: building the data set from Yahoo Finance (~15 min, needs internet)"
"$qc" setup say " candidates: the list shipped in the repository; companies Yahoo no longer knows are left out"
"$qc" setup --offline
else else
say " skipped (--no-data). Later: qc setup or qc data-import FILE" say " skipped (--no-data). Later: qc setup or qc data-import FILE"
fi fi

34
quantcore/data/yahoo.py
View File

@ -141,6 +141,9 @@ class YahooClient:
# -- metadata -------------------------------------------------------------------------- # -- metadata --------------------------------------------------------------------------
def ticker_info(self, symbol: str) -> dict: def ticker_info(self, symbol: str) -> dict:
"""Selected ``Ticker.info`` fields for one symbol (``{}`` when Yahoo has nothing).""" """Selected ``Ticker.info`` fields for one symbol (``{}`` when Yahoo has nothing)."""
return normalise_info(symbol, self._info_fields(symbol))
def _info_fields(self, symbol: str) -> dict:
import yfinance as yf import yfinance as yf
def _info(): def _info():
@ -155,15 +158,32 @@ class YahooClient:
log.warning("no info for %s: %s", symbol, exc) log.warning("no info for %s: %s", symbol, exc)
data = {} data = {}
time.sleep(self.settings.yahoo_info_pause) time.sleep(self.settings.yahoo_info_pause)
return normalise_info(symbol, data) return data
def ticker_infos(self, symbols: list[str], progress=None) -> pd.DataFrame: def ticker_infos(self, symbols: list[str], progress=None) -> pd.DataFrame:
"""``ticker_info`` for many symbols as a DataFrame indexed by symbol.""" """``ticker_info`` for many symbols as a DataFrame indexed by symbol.
rows = []
for i, s in enumerate(symbols, 1): yfinance logs an ERROR line for every symbol Yahoo does not know -- a delisted, renamed or
rows.append(self.ticker_info(s)) merged company. They are expected among ~1100 candidates and the selection drops them, so
if progress and (i % 25 == 0 or i == len(symbols)): they are collected and reported in one line instead."""
progress(i, len(symbols)) rows, unknown = [], []
yf_log = logging.getLogger("yfinance")
previous = yf_log.level
yf_log.setLevel(logging.CRITICAL)
try:
for i, s in enumerate(symbols, 1):
data = self._info_fields(s)
if not data:
unknown.append(s)
rows.append(normalise_info(s, data))
if progress and (i % 25 == 0 or i == len(symbols)):
progress(i, len(symbols))
finally:
yf_log.setLevel(previous)
if unknown:
shown = ", ".join(unknown[:12]) + (f" and {len(unknown) - 12} more" if len(unknown) > 12 else "")
log.info("Yahoo has no quote for %d of %d candidates (delisted, renamed or merged); "
"the selection leaves them out: %s", len(unknown), len(symbols), shown)
df = pd.DataFrame(rows).set_index("symbol") df = pd.DataFrame(rows).set_index("symbol")
return df return df

95
quantcore/universe/sources.py
View File

@ -47,12 +47,17 @@ USER_AGENT = "Mozilla/5.0 (compatible; quantcore-setup/0.1; +https://fhgr.ch)"
SP500_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies" SP500_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
STOXX600_WIKI_URL = "https://en.wikipedia.org/wiki/STOXX_Europe_600" STOXX600_WIKI_URL = "https://en.wikipedia.org/wiki/STOXX_Europe_600"
# iShares STOXX Europe 600 UCITS ETF (DE), holdings download of the German retail site # iShares STOXX Europe 600 UCITS ETF (DE), holdings file in German, tried in this order. The Swiss
ISHARES_STOXX600_URL = ( # site serves it; the German site's download answered 404 from October 2026 on and stays as a spare.
"https://www.ishares.com/de/privatanleger/de/produkte/251931/" ISHARES_STOXX600_URLS = (
"ishares-stoxx-europe-600-ucits-etf-de-fund/1478358465952.ajax" ("https://www.ishares.com/ch/privatkunden/de/produkte/251931/"
"?fileType=csv&fileName=EXSA_holdings&dataType=fund" "ishares-stoxx-europe-600-ucits-etf-de-fund/1495092304805.ajax"
"?fileType=csv&fileName=EXSA_holdings&dataType=fund"),
("https://www.ishares.com/de/privatanleger/de/produkte/251931/"
"ishares-stoxx-europe-600-ucits-etf-de-fund/1478358465952.ajax"
"?fileType=csv&fileName=EXSA_holdings&dataType=fund"),
) )
ISHARES_STOXX600_URL = ISHARES_STOXX600_URLS[0]
# German labels of the iShares file -> English # German labels of the iShares file -> English
ISHARES_SECTOR_EN = { ISHARES_SECTOR_EN = {
@ -209,15 +214,30 @@ def parse_ishares_holdings(text: str, source_date: str | None = None) -> pd.Data
def fetch_stoxx600_ishares(timeout: float = 30.0) -> pd.DataFrame: def fetch_stoxx600_ishares(timeout: float = 30.0) -> pd.DataFrame:
"""STOXX Europe 600 constituents from the iShares ETF holdings (complete, with weights).""" """STOXX 600 constituents from the iShares holdings file: exchange and ticker per line, so the
raw = _get(ISHARES_STOXX600_URL, timeout).decode("utf-8-sig", errors="replace") Yahoo symbols are exact. Tries every address in :data:`ISHARES_STOXX600_URLS`."""
df = parse_ishares_holdings(raw) last: Exception | None = None
log.info("STOXX 600 candidates from iShares: %d", len(df)) for url in ISHARES_STOXX600_URLS:
return df try:
raw = _get(url, timeout).decode("utf-8-sig", errors="replace")
df = parse_ishares_holdings(raw)
except Exception as exc: # noqa: BLE001 - next address
log.info("iShares holdings not available at %s: %s", url.split("?")[0], exc)
last = exc
continue
if len(df) >= 100:
log.info("STOXX 600 candidates from iShares: %d (as of %s)", len(df), df["source_date"].iloc[0])
return df
last = ValueError(f"only {len(df)} rows in the iShares file at {url.split('?')[0]}")
raise RuntimeError(f"iShares holdings file not available: {last}")
def fetch_stoxx600_wikipedia(timeout: float = 30.0) -> pd.DataFrame: def fetch_stoxx600_wikipedia(timeout: float = 30.0) -> pd.DataFrame:
"""Partial STOXX 600 list from Wikipedia (fallback; no exchange, so the suffix is by country).""" """Partial STOXX 600 list from Wikipedia -- the last resort, used only when no snapshot ships.
The table gives Reuters-style codes without the exchange, so the Yahoo suffix is guessed from the
country; for a large share of the index the guess does not exist on Yahoo (BNPP.PA for BNP.PA,
NOV-N.SW for NOVN.SW, VESTAS.CO for VWS.CO, UC.MI for UCG.MI)."""
html = _get(STOXX600_WIKI_URL, timeout) html = _get(STOXX600_WIKI_URL, timeout)
tables = pd.read_html(io.BytesIO(html)) tables = pd.read_html(io.BytesIO(html))
table = next(t for t in tables if "Ticker" in t.columns and "Country" in t.columns) table = next(t for t in tables if "Ticker" in t.columns and "Country" in t.columns)
@ -277,26 +297,43 @@ def save_static_candidates(df: pd.DataFrame, region: str, path: Path | None = No
def fetch_candidates(region: str, allow_web: bool = True, timeout: float = 30.0) -> pd.DataFrame: def fetch_candidates(region: str, allow_web: bool = True, timeout: float = 30.0) -> pd.DataFrame:
"""Best-effort candidate list: web sources first, then the static snapshot. """Candidate list of a region: the primary web source, else the snapshot shipped with the package.
Every web source is wrapped: a failing source is logged and the next one is tried US: the S&P 500 table on Wikipedia. Europe: the iShares STOXX Europe 600 holdings. When the
(timeout, retry and fallback -- the resilience the platform requires on external calls). primary source fails, the shipped snapshot (taken from the same sources) is used -- not the
Wikipedia STOXX 600 table, whose guessed symbols miss a large share of the index; that table is
the last resort when no snapshot ships. ``allow_web=False`` (``quantcore setup --offline``) uses
the snapshot directly, so every installation builds its universe from the same candidates.
Only a primary source refreshes the snapshot. Every web call is wrapped: a failure is logged
and the next option is used (timeout, retry and fallback, as the platform requires).
""" """
region = "US" if region.upper() == "US" else "Europe" region = "US" if region.upper() == "US" else "Europe"
fetchers = [fetch_sp500_wikipedia] if region == "US" else [fetch_stoxx600_ishares, fetch_stoxx600_wikipedia] primary = fetch_sp500_wikipedia if region == "US" else fetch_stoxx600_ishares
if allow_web: if allow_web:
for fetch in fetchers: try:
df = primary(timeout)
except Exception as exc: # noqa: BLE001 - any failure -> the snapshot
log.warning("Candidate source %s failed: %s", primary.__name__, exc)
df = _empty()
if len(df) >= 100:
if region == "Europe":
df = df[df["country"].isin(EUROPE_COUNTRIES)].reset_index(drop=True)
try: try:
df = fetch(timeout) save_static_candidates(df, region)
except Exception as exc: # noqa: BLE001 - any failure -> next source except OSError as exc: # read-only install
log.warning("Candidate source %s failed: %s", fetch.__name__, exc) log.debug("Could not refresh static snapshot: %s", exc)
continue return df
if len(df) >= 100: static = load_static_candidates(region)
if region == "Europe": if len(static):
df = df[df["country"].isin(EUROPE_COUNTRIES)].reset_index(drop=True) if allow_web:
try: log.warning("Using the %s candidates shipped with the package instead (%d, as of %s)",
save_static_candidates(df, region) region, len(static), static["source_date"].iloc[0])
except OSError as exc: # read-only install return static
log.debug("Could not refresh static snapshot: %s", exc) if allow_web and region == "Europe":
return df try:
return load_static_candidates(region) df = fetch_stoxx600_wikipedia(timeout)
return df[df["country"].isin(EUROPE_COUNTRIES)].reset_index(drop=True)
except Exception as exc: # noqa: BLE001
log.warning("Candidate source fetch_stoxx600_wikipedia failed: %s", exc)
return static

71
tests/test_data.py
View File

@ -79,3 +79,74 @@ def test_parse_ishares_holdings():
assert df.iloc[0]["gics_sector"] == "Information Technology" and df.iloc[0]["country"] == "Netherlands" assert df.iloc[0]["gics_sector"] == "Information Technology" and df.iloc[0]["country"] == "Netherlands"
assert df.iloc[0]["index_weight"] == pytest.approx(4.23) assert df.iloc[0]["index_weight"] == pytest.approx(4.23)
assert df.iloc[0]["source_date"] == "2026-09-18" assert df.iloc[0]["source_date"] == "2026-09-18"
# -- candidate sources: which list a setup builds its universe from ---------------------------------
def _frame(symbols, source):
from quantcore.universe.sources import CANDIDATE_COLUMNS
rows = [{c: "" for c in CANDIDATE_COLUMNS} | {"symbol": s, "country": "Switzerland", "region": "Europe",
"source": source, "source_date": "2026-10-06"} for s in symbols]
return pd.DataFrame(rows, columns=CANDIDATE_COLUMNS)
def test_europe_falls_back_to_the_shipped_list_not_to_wikipedia(monkeypatch):
"""iShares down (it answered 404 in October 2026): the shipped snapshot, with exact symbols, is used.
Wikipedia's table yields guesses such as BNPP.PA or NOV-N.SW that Yahoo does not know."""
from quantcore.universe import sources as S
calls = []
monkeypatch.setattr(S, "fetch_stoxx600_ishares", lambda timeout=30.0: (_ for _ in ()).throw(RuntimeError("404")))
monkeypatch.setattr(S, "fetch_stoxx600_wikipedia", lambda timeout=30.0: calls.append("wiki") or _frame(["X"] * 200, "w"))
monkeypatch.setattr(S, "save_static_candidates", lambda *a, **k: calls.append("save"))
df = S.fetch_candidates("Europe")
assert calls == [] and len(df) > 500 and set(df["source"]) == {"ishares_stoxx600"}
assert {"BNP.PA", "NOVN.SW", "VWS.CO", "UCG.MI"} <= set(df["symbol"])
def test_offline_setup_uses_the_shipped_lists_without_the_web(monkeypatch):
from quantcore.universe import sources as S
def no_web(*a, **k):
raise AssertionError("no web access with allow_web=False")
monkeypatch.setattr(S, "_get", no_web)
assert len(S.fetch_candidates("US", allow_web=False)) > 400
assert len(S.fetch_candidates("Europe", allow_web=False)) > 500
def test_ishares_tries_the_next_address(monkeypatch):
from quantcore.universe import sources as S
tried = []
def fake_get(url, timeout=30.0):
tried.append(url)
if len(tried) == 1:
raise RuntimeError("404 Client Error")
return b"csv"
monkeypatch.setattr(S, "_get", fake_get)
monkeypatch.setattr(S, "parse_ishares_holdings", lambda raw: _frame([f"S{i}.SW" for i in range(150)], "ishares"))
df = S.fetch_stoxx600_ishares()
assert tried == list(S.ISHARES_STOXX600_URLS[:2]) and len(df) == 150
def test_unknown_symbols_are_summarised_not_logged_one_by_one(monkeypatch, caplog):
import logging
from quantcore.config import Settings
from quantcore.data.yahoo import YahooClient
client = YahooClient(Settings(yahoo_info_pause=0.0))
known = {"quoteType": "EQUITY", "currency": "CHF", "longName": "Known AG", "exchange": "EBS"}
monkeypatch.setattr(YahooClient, "_info_fields",
lambda self, s: logging.getLogger("yfinance").error("HTTP Error 404 %s", s) or
({} if s.startswith("GONE") else known))
with caplog.at_level(logging.INFO):
df = client.ticker_infos(["KNOWN.SW", "GONE1.PA", "GONE2.SW"])
assert list(df.index) == ["KNOWN.SW", "GONE1.PA", "GONE2.SW"] and df.loc["KNOWN.SW", "currency"] == "CHF"
assert not [r for r in caplog.records if r.name == "yfinance"] # no per-symbol ERROR lines
summary = [r.getMessage() for r in caplog.records if "no quote for 2 of 3" in r.getMessage()]
assert summary and "GONE1.PA, GONE2.SW" in summary[0]
assert logging.getLogger("yfinance").level == logging.NOTSET # restored afterwards