strategy='auto' now routes every scenario-mode CVaR optimisation through
quantcore.optimization.large_scale (LP relaxation on the tail rows, screening,
round-and-repair, certified against the relaxation bound, escalation to the
integer MILP otherwise); strategy='milp' is the explicit opt-out, and
row_generation applies to that path only. When the escalated MILP proves the
rounded portfolio optimal within its gap, the result is Optimal even where the
relaxation bound is loose (small portfolios with a real integrality gap).
UI: a Strategy selector and the driver's bound, gap, screened size, rows and
solver calls in the provenance card. Deploy env examples document the new
cuOpt settings (presolve off, dual simplex for LPs, 50 ms polling). The
service-equivalence test asks for strategy='milp' where it checks the
integer row-generation path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A third deployment next to the two on the DGX Spark, which stay as they
are: deploy/k8s/ has its own image and Kustomize manifests. The API is a
Deployment with probes, the data a PersistentVolumeClaim, the nightly
update a CronJob; overlays/gpu adds cuOpt on a GPU of the cluster.
Five assumptions of the single-container setup did not hold for pods, so
the service changed (on the Spark nothing changes that one would notice):
- probes: GET /health/live (async, no I/O) and GET /health/ready (data
directory readable); neither needs cuOpt or market data, so a cuOpt
outage never gets the API restarted. /health is unchanged
- DataStore replaces every file atomically (temporary file renamed into
place) and saves the manifest last; the API reloads its market data
when the manifest changes, so it serves what a CronJob wrote
- one writer at a time across processes and pods: an flock on
/data/.write.lock, taken by API jobs and the CLI's data commands
(setup, reselect, update, clean, snapshot); a second writer gets 409 or
exit status 3, with the holder's name. `clean` now updates the manifest
- job records live in /data/jobs/ and are written at every status
change: every replica answers GET /jobs/{id}, and a job whose process
has gone is reported failed ("interrupted")
- the image runs as uid 10001 under a read-only root file system, with
the library versions pinned to deploy/apptainer/constraints.txt (SciPy
1.18 releases the GIL while HiGHS solves, so the liveness probe is
answered during an optimisation)
docs/KUBERNETES.md is the guide (build, deploy, load the data, operate,
troubleshooting); README, USAGE and ARCHITECTURE point to it.
Both kustomizations render (kubectl kustomize); the k3s test of the
deployment is recorded in docs/KUBERNETES.md. 149 tests (148 pass, the
cuOpt round trip skipped without a server).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a student Spark the API answered with 746 stocks while `quantcore
risk` found no data, and after a restart the test bench answered 503
"no market data yet". Cause: `install.sh --rebuild-data` renamed the data
folder; a container that still has a folder mounted keeps using it under
its new name, and the data jobs ran inside the API -- so the rebuilt data
went into ~/quantcore/data.<date> while the commands read the new, empty
~/quantcore/data. Pasting the README's command block made it visible: it
contained `qc stop / qc start`, which stopped the service.
- install.sh --rebuild-data stops the service, checks that it stopped,
and moves the folder's contents, never the folder itself
- data jobs (setup, update) always run in a fresh container on the data
folder; a running API is restarted afterwards (stopped during setup);
`qc data-import` restarts it too; stop waits and forces if needed
- `qc status` warns when the API serves a different data set than the
folder, and lists earlier data sets with the command to restore one;
`qc start` does the same when the folder is empty
- the CLI prints three lines instead of a traceback when there is no data
- README and guide: command blocks only read, so they can be pasted whole;
start, stop, update, upgrade and --rebuild-data are in tables
- lookups that may find nothing are guarded under pipefail (one ended
`qc status` silently)
Tested on a stripped-down machine: the README block pasted whole keeps the
service running; `quantcore update` restarts the API and status agrees;
the broken state is detected and restored with `qc data-import`;
`install.sh --rebuild-data` with the service running rebuilt 745 stocks,
and API, CLI and folder agree afterwards. 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the one-line installation, following the README on a student Spark
failed: `quantcore ...` was "command not found" (the software lives in
the container) and the laptop route's `pip install -e .` stopped with
Ubuntu's "externally-managed-environment".
- install.sh writes `quantcore` and `quantcore-api` into ~/.local/bin next
to `qc`: they run the CLI and the API client inside the container;
`quantcore serve` starts the background service, `setup`/`update` go
through the running API; a user's own file of that name is kept
- install.sh adds ~/.local/bin to PATH in ~/.bashrc (once, guarded):
Ubuntu's ~/.profile only adds it when it existed at login
- the closing message says to open a new terminal and shows commands to try
- install.sh and `qc status` flag a data set built from the old Wikipedia
candidates and give the --rebuild-data command
- README: a numbered Spark quick start (install, new terminal, use) with
the commands that work there, and the laptop quick start marked as
such, with a virtual environment; SPARK_APPTAINER.md and USAGE.md match
- cli: `risk` and `performance` round only numeric columns (pandas warned
about the date rows in every call)
Tested on a stripped-down machine: after the installation, a new
interactive shell ran every Spark command of the README (quantcore risk,
performance, optimize on cuOpt, quantcore-api health, quantcore serve);
qc upgrade rebuilt and restarted; 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a student Spark the installer's data step printed hundreds of red
yfinance "Quote not found for symbol" lines and built a universe of 652
instead of ~745 stocks. Cause: the iShares holdings download (German site)
answers 404 since October 2026, and setup fell back to Wikipedia's STOXX
600 table, whose Reuters-style codes give Yahoo symbols that do not exist
(BNPP.PA for BNP.PA, NOV-N.SW for NOVN.SW, VESTAS.CO for VWS.CO).
- sources: the iShares file is read from the Swiss site (same German CSV,
same parser), the German address stays as a spare; when iShares fails,
setup uses the snapshot shipped in quantcore/data/static and not the
Wikipedia guesses (now a last resort when no snapshot ships); only a
primary source refreshes the snapshot
- yahoo: yfinance's per-symbol ERROR lines are silenced during the
metadata step and summarised in one line
- install.sh: builds the data with `qc setup --offline`, so every student
gets the universe of the shipped lists (the course data set's); new
--rebuild-data moves the existing data aside and builds it anew
- qc.sh: the image build attempt logs to ~/quantcore/build.log; a Spark
that allows no build shows one plain line, then the no-build setup
- tests for the source order, the address fallback and the summary
Tested on a stripped-down machine (no git/curl/wget/python3, broken
APPTAINER_TMPDIR): install, then install.sh --rebuild-data built 746
stocks (382 US, 364 Europe) from the 1117 shipped candidates in under
10 minutes with no yfinance error line; self-test passes; qc upgrade
rebuilt and restarted. 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A student Spark can be barebone: no git, no sudo, Apptainer only, and an
account that exports APPTAINER_TMPDIR=/scratch/$USER/... for a directory
that does not exist, so every image pull failed ("failed to create build
parent dir"). Students now run one line:
wget -qO- .../raw/branch/main/install.sh | bash
install.sh (wrapped in main() so a piped run cannot lose its tail):
- repairs APPTAINER_TMPDIR / APPTAINER_CACHEDIR (and SINGULARITY_*) that
point nowhere and records the choice in ~/quantcore/quantcore.env
- gets the code with git, or with git from a container (alpine/git) when
the machine has none; Gitea's archive download answers 502
- runs qc.sh install, imports --data FILE|URL or builds the data from
Yahoo, starts the services, runs the self-test, links ~/.local/bin/qc
- running it again updates: rebuild and restart only when the code changed
qc.sh:
- the same directory repair before every pull or build, with one helper
for all pulls and a clear message when a download fails
- health checks over bash's /dev/tcp when neither curl nor python3 exists
- instance logs found for domain accounts and shared network homes
- works when called through the ~/.local/bin/qc link; new commands
`upgrade` (runs install.sh) and `has-data`
- the virtualenv build copies the sources under QC_HOME, not $TMPDIR
Tested on a stripped-down machine (no git/curl/wget/python3, home
/home/name@edu.local, APPTAINER_TMPDIR on a missing /scratch): install,
start and self-test pass; `qc upgrade` rebuilt and restarted after a new
commit; `qc setup` built a data set inside the container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unanchored `data/` rule, meant for the market-data directory, also
matched the Python subpackage quantcore/data/, so a fresh clone could not
import quantcore.data. `/data/`, `/build/` and `/dist/` now match only at
the repository root.
qc.sh: the virtualenv install stops when the installed package does not
start, and the self-test runs from / so that it imports the installed
package rather than the sources in the working directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The portfolio data, risk and mean-CVaR optimisation service, prepared for
students who run it on their own DGX Spark without root, Docker or Tailscale.
- deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python,
data import/export and Yahoo setup/update. cuOpt and quantcore run as two
Apptainer instances of the user and talk over 127.0.0.1; the API and the
test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or
falls back to a plain Python image plus a virtualenv when the Spark allows
no unprivileged builds. Detects taken ports and suggests free ones.
- deploy/apptainer/quantcore.def + constraints.txt: the image, with the
library versions of the verified production container pinned.
- cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a
SIF, so the solver died on its first job; qc.sh gives it a writable home.
- solvers: a cuOpt that disappears during a solve now falls back to HiGHS
instead of answering 500 (requests' ConnectionError escaped the fallback).
- client: idempotent GETs retry a dropped keep-alive connection, which made
`quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout.
- config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to
127.0.0.1; the Docker images keep passing their own values.
- docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE
and SPARK_DEPLOYMENT point to it.
Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both
modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update,
port conflicts; 139 tests pass in the container against the live cuOpt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
