b5da424dfe
The portfolio data, risk and mean-CVaR optimisation service, prepared for students who run it on their own DGX Spark without root, Docker or Tailscale. - deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python, data import/export and Yahoo setup/update. cuOpt and quantcore run as two Apptainer instances of the user and talk over 127.0.0.1; the API and the test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or falls back to a plain Python image plus a virtualenv when the Spark allows no unprivileged builds. Detects taken ports and suggests free ones. - deploy/apptainer/quantcore.def + constraints.txt: the image, with the library versions of the verified production container pinned. - cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a SIF, so the solver died on its first job; qc.sh gives it a writable home. - solvers: a cuOpt that disappears during a solve now falls back to HiGHS instead of answering 500 (requests' ConnectionError escaped the fallback). - client: idempotent GETs retry a dropped keep-alive connection, which made `quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout. - config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to 127.0.0.1; the Docker images keep passing their own values. - docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE and SPARK_DEPLOYMENT point to it. Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update, port conflicts; 139 tests pass in the container against the live cuOpt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Experiments behind the scenario-sampling section
These scripts produced the tables in docs/optimization_model.pdf, Section "Scenario sets:
sampling error, variance reduction and large sets". They run against the local data store
(quantcore setup first) and print their tables; they are not part of the package or the tests.
| script | what it measures |
|---|---|
scenario_noise.py N S SEEDS [cont|int] [kinds] [beta] |
GBM model: plain MC vs antithetic / moment matching / LHS / Sobol' at equal S, continuous (weights) or integer problem, against the exact Gaussian optimum (cutting planes) — Table "Sampling noise" |
empirical_noise.py N S SEEDS BETA [kinds] |
empirical-innovation model: bootstrap vs moment-matched / balanced / rank-stratified resampling, evaluated on a 500 000-draw set |
is_noise.py N SIZES SEEDS BETAS |
importance sampling (two-stage tilt, defensive mixture) vs moment-matched Sobol' — Table "Importance sampling" |
scaling.py N SIZES highs|cuopt |
solve time versus the number of scenarios (HiGHS locally / cuOpt on the Spark) — Table "Solve times" |
rowgen.py N SIZES [full] |
tail-row generation versus the full MILP |
gurobi_ref.py N BETA |
the exact Gaussian MISOCP solved directly with Gurobi (needs gurobipy and a licence large enough for N) — validation of the cutting-plane reference |
Example: python scripts/experiments/scenario_noise.py 30 2048 8 cont mc,antithetic,mc+mm,sobol,sobol+mm 0.95