strategy='auto' now routes every scenario-mode CVaR optimisation through
quantcore.optimization.large_scale (LP relaxation on the tail rows, screening,
round-and-repair, certified against the relaxation bound, escalation to the
integer MILP otherwise); strategy='milp' is the explicit opt-out, and
row_generation applies to that path only. When the escalated MILP proves the
rounded portfolio optimal within its gap, the result is Optimal even where the
relaxation bound is loose (small portfolios with a real integrality gap).
UI: a Strategy selector and the driver's bound, gap, screened size, rows and
solver calls in the provenance card. Deploy env examples document the new
cuOpt settings (presolve off, dual simplex for LPs, 50 ms polling). The
service-equivalence test asks for strategy='milp' where it checks the
integer row-generation path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The solver benchmark (GPU_Optimization/cuopt_benchmark, November 2026) showed
that the integer mean-CVaR MILP is an LP in disguise: HiGHS closes it at the
root node, only the tail scenario rows bind, and the cardinality constraint is
the only hard combinatorics. quantcore.optimization.large_scale exploits that:
LP relaxation by tail-row generation (evaluated at the continuous point, its
objective a valid bound), screening to the top names under a cardinality limit,
a boxed round-and-repair MILP, certification against the bound with escalation
to the integer row generation otherwise. Solver choice by size (HiGHS below
16 000 rows, cuOpt above).
Request option `strategy` ('auto' | 'milp' | 'large_scale'; auto from 200
candidates or 32 768 scenarios) in the service, the API schema and both CLIs.
Settings from the benchmark: cuOpt presolve off (its asynchronous GPU heuristic
otherwise holds every MILP for 5-25 s), dual simplex for pure LPs (concurrent
returns PDLP's inexact point on the L40S), 50 ms polling. rowgen takes an
initial working set.
Measured with HiGHS (M4 Max): N=300 S=4096 5.2 s -> 0.7 s, N=300 S=16384
45 s -> 2.9 s, N=750 S=16384 15.6 s -> 7.4 s, K=40 of 750 24.6 s -> 7.3 s,
objectives within the 1e-4 gap. Tests in tests/test_large_scale.py; benchmark
script scripts/experiments/large_scale.py; docs updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A third deployment next to the two on the DGX Spark, which stay as they
are: deploy/k8s/ has its own image and Kustomize manifests. The API is a
Deployment with probes, the data a PersistentVolumeClaim, the nightly
update a CronJob; overlays/gpu adds cuOpt on a GPU of the cluster.
Five assumptions of the single-container setup did not hold for pods, so
the service changed (on the Spark nothing changes that one would notice):
- probes: GET /health/live (async, no I/O) and GET /health/ready (data
directory readable); neither needs cuOpt or market data, so a cuOpt
outage never gets the API restarted. /health is unchanged
- DataStore replaces every file atomically (temporary file renamed into
place) and saves the manifest last; the API reloads its market data
when the manifest changes, so it serves what a CronJob wrote
- one writer at a time across processes and pods: an flock on
/data/.write.lock, taken by API jobs and the CLI's data commands
(setup, reselect, update, clean, snapshot); a second writer gets 409 or
exit status 3, with the holder's name. `clean` now updates the manifest
- job records live in /data/jobs/ and are written at every status
change: every replica answers GET /jobs/{id}, and a job whose process
has gone is reported failed ("interrupted")
- the image runs as uid 10001 under a read-only root file system, with
the library versions pinned to deploy/apptainer/constraints.txt (SciPy
1.18 releases the GIL while HiGHS solves, so the liveness probe is
answered during an optimisation)
docs/KUBERNETES.md is the guide (build, deploy, load the data, operate,
troubleshooting); README, USAGE and ARCHITECTURE point to it.
Both kustomizations render (kubectl kustomize); the k3s test of the
deployment is recorded in docs/KUBERNETES.md. 149 tests (148 pass, the
cuOpt round trip skipped without a server).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a student Spark the installer's data step printed hundreds of red
yfinance "Quote not found for symbol" lines and built a universe of 652
instead of ~745 stocks. Cause: the iShares holdings download (German site)
answers 404 since October 2026, and setup fell back to Wikipedia's STOXX
600 table, whose Reuters-style codes give Yahoo symbols that do not exist
(BNPP.PA for BNP.PA, NOV-N.SW for NOVN.SW, VESTAS.CO for VWS.CO).
- sources: the iShares file is read from the Swiss site (same German CSV,
same parser), the German address stays as a spare; when iShares fails,
setup uses the snapshot shipped in quantcore/data/static and not the
Wikipedia guesses (now a last resort when no snapshot ships); only a
primary source refreshes the snapshot
- yahoo: yfinance's per-symbol ERROR lines are silenced during the
metadata step and summarised in one line
- install.sh: builds the data with `qc setup --offline`, so every student
gets the universe of the shipped lists (the course data set's); new
--rebuild-data moves the existing data aside and builds it anew
- qc.sh: the image build attempt logs to ~/quantcore/build.log; a Spark
that allows no build shows one plain line, then the no-build setup
- tests for the source order, the address fallback and the summary
Tested on a stripped-down machine (no git/curl/wget/python3, broken
APPTAINER_TMPDIR): install, then install.sh --rebuild-data built 746
stocks (382 US, 364 Europe) from the 1117 shipped candidates in under
10 minutes with no yfinance error line; self-test passes; qc upgrade
rebuilt and restarted. 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The portfolio data, risk and mean-CVaR optimisation service, prepared for
students who run it on their own DGX Spark without root, Docker or Tailscale.
- deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python,
data import/export and Yahoo setup/update. cuOpt and quantcore run as two
Apptainer instances of the user and talk over 127.0.0.1; the API and the
test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or
falls back to a plain Python image plus a virtualenv when the Spark allows
no unprivileged builds. Detects taken ports and suggests free ones.
- deploy/apptainer/quantcore.def + constraints.txt: the image, with the
library versions of the verified production container pinned.
- cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a
SIF, so the solver died on its first job; qc.sh gives it a writable home.
- solvers: a cuOpt that disappears during a solve now falls back to HiGHS
instead of answering 500 (requests' ConnectionError escaped the fallback).
- client: idempotent GETs retry a dropped keep-alive connection, which made
`quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout.
- config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to
127.0.0.1; the Docker images keep passing their own values.
- docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE
and SPARK_DEPLOYMENT point to it.
Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both
modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update,
port conflicts; 139 tests pass in the container against the live cuOpt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
