The solver benchmark (GPU_Optimization/cuopt_benchmark, November 2026) showed
that the integer mean-CVaR MILP is an LP in disguise: HiGHS closes it at the
root node, only the tail scenario rows bind, and the cardinality constraint is
the only hard combinatorics. quantcore.optimization.large_scale exploits that:
LP relaxation by tail-row generation (evaluated at the continuous point, its
objective a valid bound), screening to the top names under a cardinality limit,
a boxed round-and-repair MILP, certification against the bound with escalation
to the integer row generation otherwise. Solver choice by size (HiGHS below
16 000 rows, cuOpt above).
Request option `strategy` ('auto' | 'milp' | 'large_scale'; auto from 200
candidates or 32 768 scenarios) in the service, the API schema and both CLIs.
Settings from the benchmark: cuOpt presolve off (its asynchronous GPU heuristic
otherwise holds every MILP for 5-25 s), dual simplex for pure LPs (concurrent
returns PDLP's inexact point on the L40S), 50 ms polling. rowgen takes an
initial working set.
Measured with HiGHS (M4 Max): N=300 S=4096 5.2 s -> 0.7 s, N=300 S=16384
45 s -> 2.9 s, N=750 S=16384 15.6 s -> 7.4 s, K=40 of 750 24.6 s -> 7.3 s,
objectives within the 1e-4 gap. Tests in tests/test_large_scale.py; benchmark
script scripts/experiments/large_scale.py; docs updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A third deployment next to the two on the DGX Spark, which stay as they
are: deploy/k8s/ has its own image and Kustomize manifests. The API is a
Deployment with probes, the data a PersistentVolumeClaim, the nightly
update a CronJob; overlays/gpu adds cuOpt on a GPU of the cluster.
Five assumptions of the single-container setup did not hold for pods, so
the service changed (on the Spark nothing changes that one would notice):
- probes: GET /health/live (async, no I/O) and GET /health/ready (data
directory readable); neither needs cuOpt or market data, so a cuOpt
outage never gets the API restarted. /health is unchanged
- DataStore replaces every file atomically (temporary file renamed into
place) and saves the manifest last; the API reloads its market data
when the manifest changes, so it serves what a CronJob wrote
- one writer at a time across processes and pods: an flock on
/data/.write.lock, taken by API jobs and the CLI's data commands
(setup, reselect, update, clean, snapshot); a second writer gets 409 or
exit status 3, with the holder's name. `clean` now updates the manifest
- job records live in /data/jobs/ and are written at every status
change: every replica answers GET /jobs/{id}, and a job whose process
has gone is reported failed ("interrupted")
- the image runs as uid 10001 under a read-only root file system, with
the library versions pinned to deploy/apptainer/constraints.txt (SciPy
1.18 releases the GIL while HiGHS solves, so the liveness probe is
answered during an optimisation)
docs/KUBERNETES.md is the guide (build, deploy, load the data, operate,
troubleshooting); README, USAGE and ARCHITECTURE point to it.
Both kustomizations render (kubectl kustomize); the k3s test of the
deployment is recorded in docs/KUBERNETES.md. 149 tests (148 pass, the
cuOpt round trip skipped without a server).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a student Spark the API answered with 746 stocks while `quantcore
risk` found no data, and after a restart the test bench answered 503
"no market data yet". Cause: `install.sh --rebuild-data` renamed the data
folder; a container that still has a folder mounted keeps using it under
its new name, and the data jobs ran inside the API -- so the rebuilt data
went into ~/quantcore/data.<date> while the commands read the new, empty
~/quantcore/data. Pasting the README's command block made it visible: it
contained `qc stop / qc start`, which stopped the service.
- install.sh --rebuild-data stops the service, checks that it stopped,
and moves the folder's contents, never the folder itself
- data jobs (setup, update) always run in a fresh container on the data
folder; a running API is restarted afterwards (stopped during setup);
`qc data-import` restarts it too; stop waits and forces if needed
- `qc status` warns when the API serves a different data set than the
folder, and lists earlier data sets with the command to restore one;
`qc start` does the same when the folder is empty
- the CLI prints three lines instead of a traceback when there is no data
- README and guide: command blocks only read, so they can be pasted whole;
start, stop, update, upgrade and --rebuild-data are in tables
- lookups that may find nothing are guarded under pipefail (one ended
`qc status` silently)
Tested on a stripped-down machine: the README block pasted whole keeps the
service running; `quantcore update` restarts the API and status agrees;
the broken state is detected and restored with `qc data-import`;
`install.sh --rebuild-data` with the service running rebuilt 745 stocks,
and API, CLI and folder agree afterwards. 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the one-line installation, following the README on a student Spark
failed: `quantcore ...` was "command not found" (the software lives in
the container) and the laptop route's `pip install -e .` stopped with
Ubuntu's "externally-managed-environment".
- install.sh writes `quantcore` and `quantcore-api` into ~/.local/bin next
to `qc`: they run the CLI and the API client inside the container;
`quantcore serve` starts the background service, `setup`/`update` go
through the running API; a user's own file of that name is kept
- install.sh adds ~/.local/bin to PATH in ~/.bashrc (once, guarded):
Ubuntu's ~/.profile only adds it when it existed at login
- the closing message says to open a new terminal and shows commands to try
- install.sh and `qc status` flag a data set built from the old Wikipedia
candidates and give the --rebuild-data command
- README: a numbered Spark quick start (install, new terminal, use) with
the commands that work there, and the laptop quick start marked as
such, with a virtual environment; SPARK_APPTAINER.md and USAGE.md match
- cli: `risk` and `performance` round only numeric columns (pandas warned
about the date rows in every call)
Tested on a stripped-down machine: after the installation, a new
interactive shell ran every Spark command of the README (quantcore risk,
performance, optimize on cuOpt, quantcore-api health, quantcore serve);
qc upgrade rebuilt and restarted; 142 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The portfolio data, risk and mean-CVaR optimisation service, prepared for
students who run it on their own DGX Spark without root, Docker or Tailscale.
- deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python,
data import/export and Yahoo setup/update. cuOpt and quantcore run as two
Apptainer instances of the user and talk over 127.0.0.1; the API and the
test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or
falls back to a plain Python image plus a virtualenv when the Spark allows
no unprivileged builds. Detects taken ports and suggests free ones.
- deploy/apptainer/quantcore.def + constraints.txt: the image, with the
library versions of the verified production container pinned.
- cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a
SIF, so the solver died on its first job; qc.sh gives it a writable home.
- solvers: a cuOpt that disappears during a solve now falls back to HiGHS
instead of answering 500 (requests' ConnectionError escaped the fallback).
- client: idempotent GETs retry a dropped keep-alive connection, which made
`quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout.
- config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to
127.0.0.1; the Docker images keep passing their own values.
- docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE
and SPARK_DEPLOYMENT point to it.
Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both
modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update,
port conflicts; 139 tests pass in the container against the live cuOpt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
