On a student Spark the API answered with 746 stocks while `quantcore
risk` found no data, and after a restart the test bench answered 503
"no market data yet". Cause: `install.sh --rebuild-data` renamed the data
folder; a container that still has a folder mounted keeps using it under
its new name, and the data jobs ran inside the API -- so the rebuilt data
went into ~/quantcore/data.<date> while the commands read the new, empty
~/quantcore/data. Pasting the README's command block made it visible: it
contained `qc stop / qc start`, which stopped the service.
- install.sh --rebuild-data stops the service, checks that it stopped,
and moves the folder's contents, never the folder itself
- data jobs (setup, update) always run in a fresh container on the data
folder; a running API is restarted afterwards (stopped during setup);
`qc data-import` restarts it too; stop waits and forces if needed
- `qc status` warns when the API serves a different data set than the
folder, and lists earlier data sets with the command to restore one;
`qc start` does the same when the folder is empty
- the CLI prints three lines instead of a traceback when there is no data
- README and guide: command blocks only read, so they can be pasted whole;
start, stop, update, upgrade and --rebuild-data are in tables
- lookups that may find nothing are guarded under pipefail (one ended
`qc status` silently)
Tested on a stripped-down machine: the README block pasted whole keeps the
service running; `quantcore update` restarts the API and status agrees;
the broken state is detected and restored with `qc data-import`;
`install.sh --rebuild-data` with the service running rebuilt 745 stocks,
and API, CLI and folder agree afterwards. 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the one-line installation, following the README on a student Spark
failed: `quantcore ...` was "command not found" (the software lives in
the container) and the laptop route's `pip install -e .` stopped with
Ubuntu's "externally-managed-environment".
- install.sh writes `quantcore` and `quantcore-api` into ~/.local/bin next
to `qc`: they run the CLI and the API client inside the container;
`quantcore serve` starts the background service, `setup`/`update` go
through the running API; a user's own file of that name is kept
- install.sh adds ~/.local/bin to PATH in ~/.bashrc (once, guarded):
Ubuntu's ~/.profile only adds it when it existed at login
- the closing message says to open a new terminal and shows commands to try
- install.sh and `qc status` flag a data set built from the old Wikipedia
candidates and give the --rebuild-data command
- README: a numbered Spark quick start (install, new terminal, use) with
the commands that work there, and the laptop quick start marked as
such, with a virtual environment; SPARK_APPTAINER.md and USAGE.md match
- cli: `risk` and `performance` round only numeric columns (pandas warned
about the date rows in every call)
Tested on a stripped-down machine: after the installation, a new
interactive shell ran every Spark command of the README (quantcore risk,
performance, optimize on cuOpt, quantcore-api health, quantcore serve);
qc upgrade rebuilt and restarted; 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a student Spark the installer's data step printed hundreds of red
yfinance "Quote not found for symbol" lines and built a universe of 652
instead of ~745 stocks. Cause: the iShares holdings download (German site)
answers 404 since October 2026, and setup fell back to Wikipedia's STOXX
600 table, whose Reuters-style codes give Yahoo symbols that do not exist
(BNPP.PA for BNP.PA, NOV-N.SW for NOVN.SW, VESTAS.CO for VWS.CO).
- sources: the iShares file is read from the Swiss site (same German CSV,
same parser), the German address stays as a spare; when iShares fails,
setup uses the snapshot shipped in quantcore/data/static and not the
Wikipedia guesses (now a last resort when no snapshot ships); only a
primary source refreshes the snapshot
- yahoo: yfinance's per-symbol ERROR lines are silenced during the
metadata step and summarised in one line
- install.sh: builds the data with `qc setup --offline`, so every student
gets the universe of the shipped lists (the course data set's); new
--rebuild-data moves the existing data aside and builds it anew
- qc.sh: the image build attempt logs to ~/quantcore/build.log; a Spark
that allows no build shows one plain line, then the no-build setup
- tests for the source order, the address fallback and the summary
Tested on a stripped-down machine (no git/curl/wget/python3, broken
APPTAINER_TMPDIR): install, then install.sh --rebuild-data built 746
stocks (382 US, 364 Europe) from the 1117 shipped candidates in under
10 minutes with no yfinance error line; self-test passes; qc upgrade
rebuilt and restarted. 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A student Spark can be barebone: no git, no sudo, Apptainer only, and an
account that exports APPTAINER_TMPDIR=/scratch/$USER/... for a directory
that does not exist, so every image pull failed ("failed to create build
parent dir"). Students now run one line:
wget -qO- .../raw/branch/main/install.sh | bash
install.sh (wrapped in main() so a piped run cannot lose its tail):
- repairs APPTAINER_TMPDIR / APPTAINER_CACHEDIR (and SINGULARITY_*) that
point nowhere and records the choice in ~/quantcore/quantcore.env
- gets the code with git, or with git from a container (alpine/git) when
the machine has none; Gitea's archive download answers 502
- runs qc.sh install, imports --data FILE|URL or builds the data from
Yahoo, starts the services, runs the self-test, links ~/.local/bin/qc
- running it again updates: rebuild and restart only when the code changed
qc.sh:
- the same directory repair before every pull or build, with one helper
for all pulls and a clear message when a download fails
- health checks over bash's /dev/tcp when neither curl nor python3 exists
- instance logs found for domain accounts and shared network homes
- works when called through the ~/.local/bin/qc link; new commands
`upgrade` (runs install.sh) and `has-data`
- the virtualenv build copies the sources under QC_HOME, not $TMPDIR
Tested on a stripped-down machine (no git/curl/wget/python3, home
/home/name@edu.local, APPTAINER_TMPDIR on a missing /scratch): install,
start and self-test pass; `qc upgrade` rebuilt and restarted after a new
commit; `qc setup` built a data set inside the container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The portfolio data, risk and mean-CVaR optimisation service, prepared for
students who run it on their own DGX Spark without root, Docker or Tailscale.
- deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python,
data import/export and Yahoo setup/update. cuOpt and quantcore run as two
Apptainer instances of the user and talk over 127.0.0.1; the API and the
test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or
falls back to a plain Python image plus a virtualenv when the Spark allows
no unprivileged builds. Detects taken ports and suggests free ones.
- deploy/apptainer/quantcore.def + constraints.txt: the image, with the
library versions of the verified production container pinned.
- cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a
SIF, so the solver died on its first job; qc.sh gives it a writable home.
- solvers: a cuOpt that disappears during a solve now falls back to HiGHS
instead of answering 500 (requests' ConnectionError escaped the fallback).
- client: idempotent GETs retry a dropped keep-alive connection, which made
`quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout.
- config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to
127.0.0.1; the Docker images keep passing their own values.
- docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE
and SPARK_DEPLOYMENT point to it.
Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both
modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update,
port conflicts; 139 tests pass in the container against the live cuOpt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
