b5da424dfe
The portfolio data, risk and mean-CVaR optimisation service, prepared for students who run it on their own DGX Spark without root, Docker or Tailscale. - deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python, data import/export and Yahoo setup/update. cuOpt and quantcore run as two Apptainer instances of the user and talk over 127.0.0.1; the API and the test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or falls back to a plain Python image plus a virtualenv when the Spark allows no unprivileged builds. Detects taken ports and suggests free ones. - deploy/apptainer/quantcore.def + constraints.txt: the image, with the library versions of the verified production container pinned. - cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a SIF, so the solver died on its first job; qc.sh gives it a writable home. - solvers: a cuOpt that disappears during a solve now falls back to HiGHS instead of answering 500 (requests' ConnectionError escaped the fallback). - client: idempotent GETs retry a dropped keep-alive connection, which made `quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout. - config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to 127.0.0.1; the Docker images keep passing their own values. - docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE and SPARK_DEPLOYMENT point to it. Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update, port conflicts; 139 tests pass in the container against the live cuOpt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
111 lines
5.8 KiB
Markdown
111 lines
5.8 KiB
Markdown
# Running the quant service on the DGX Spark (Docker, with root)
|
||
|
||
> This is the setup of the lecturer's Spark, which has root rights, Docker Compose and Tailscale.
|
||
> **Students** run the service on their own Spark with Apptainer and without root:
|
||
> see [SPARK_APPTAINER.md](SPARK_APPTAINER.md).
|
||
|
||
The Spark (`spark-6d4a`, reachable over Tailscale) already runs cuOpt, Gurobi and JupyterLab as
|
||
Docker containers defined in `~/services/docker-compose.yml`. The quant service joins that
|
||
compose file as the container **`quantcore`** on port **5001**; it reaches cuOpt over the compose
|
||
network (`http://cuopt:5000`) and keeps its data on the host in `~/services/work/quantcore_data`.
|
||
|
||
| Service | Address | What it is |
|
||
|---|---|---|
|
||
| **quantcore** | `http://spark-6d4a:5001` (`/docs` for Swagger, `/ui/` for the test bench) | portfolio data, risk/performance measurement, models, mean-CVaR MILP |
|
||
| cuOpt | `http://spark-6d4a:5000` | the GPU MILP solver the service uses |
|
||
|
||
## Deploy (from the Mac)
|
||
|
||
```bash
|
||
deploy/deploy_to_spark.sh --with-data # first time: also ships the locally built data set
|
||
deploy/deploy_to_spark.sh # later: code only
|
||
```
|
||
|
||
The script rsyncs the repository to `~/services/work/Portfolio_Risk_Optimization`, appends the
|
||
service block from `deploy/docker-compose.quantcore.yml` to the compose file (once; a backup of the
|
||
compose file is kept), builds the image (`python:3.12-slim`, aarch64) and starts the container.
|
||
|
||
The block is appended **only if `quantcore:` is not already in `~/services/docker-compose.yml`**, so
|
||
later edits to `deploy/docker-compose.quantcore.yml` (a new environment variable, a changed port) do
|
||
not travel with a redeploy — apply them to the Spark's compose file by hand, or delete the block
|
||
there and re-run the script. Code and image changes need no such step.
|
||
|
||
Without `--with-data` the container starts empty; run the setup once through the API:
|
||
|
||
```bash
|
||
curl -X POST http://spark-6d4a:5001/jobs/setup # ~15 min: 1 117 tickers' metadata + history
|
||
curl http://spark-6d4a:5001/jobs/<id> # status: queued | running | done | failed
|
||
```
|
||
|
||
## Operate
|
||
|
||
```bash
|
||
ssh user_fh_fhgr@spark-6d4a
|
||
cd ~/services
|
||
docker compose ps quantcore # status (healthcheck on /health)
|
||
docker compose logs -f quantcore # live logs
|
||
docker compose restart quantcore
|
||
docker compose up -d --build quantcore # rebuild after a code change (or re-run the deploy script)
|
||
|
||
# inside the container: the CLI works against the same data volume
|
||
docker exec -it quantcore quantcore status
|
||
docker exec -it quantcore quantcore update # newest prices (incremental)
|
||
docker exec -it quantcore quantcore snapshot --name demo
|
||
```
|
||
|
||
Nightly data refresh (host cron on the Spark), e.g. `crontab -e`:
|
||
|
||
```
|
||
15 23 * * 1-5 curl -s -X POST http://localhost:5001/jobs/update >/dev/null
|
||
```
|
||
|
||
## Use from anywhere on the tailnet
|
||
|
||
```bash
|
||
curl http://spark-6d4a:5001/health
|
||
curl "http://spark-6d4a:5001/universe?region=Europe§or=Health%20Care"
|
||
curl -s http://spark-6d4a:5001/risk/assets -H 'Content-Type: application/json' \
|
||
-d '{"symbols": ["NESN.SW", "ROP.SW", "AAPL"], "horizon": "week"}'
|
||
curl -s http://spark-6d4a:5001/optimize -H 'Content-Type: application/json' \
|
||
-d '{"cash": 1000000, "max_assets": 100, "cvar_limit": 40000, "horizon": "week"}'
|
||
```
|
||
|
||
The test bench is served from the same container (`QUANTCORE_ENABLE_UI`, default `true`, in the
|
||
compose block). Set it to `"false"` and rebuild for a deployment that exposes the JSON API and
|
||
`/docs` only — for users who build their own front end against these endpoints.
|
||
`deploy/docker-compose.students.yml` is that deployment, standalone and with the bench already
|
||
off; `docs/USAGE.md` is the guide its users need.
|
||
|
||
Python clients simply point `QUANTCORE_CUOPT_URL` / the API base URL at the Spark; the package
|
||
itself also runs on a laptop (`quantcore serve`) and uses the Spark's cuOpt remotely
|
||
(`QUANTCORE_CUOPT_URL=http://spark-6d4a:5000`), falling back to HiGHS — which needs no service of its
|
||
own: it is compiled into SciPy inside the `quantcore` image (SciPy 1.18.1 → HiGHS 1.12.0) and runs on
|
||
the container's CPU — when the
|
||
Spark is unreachable.
|
||
|
||
## Sizing
|
||
|
||
| Candidates | Scenarios | MILP size (vars / rows / nnz) | cuOpt solve |
|
||
|---|---|---|---|
|
||
| 100 | 2 000 | 2 101 / 2 013 / 206 k | ~6–10 s |
|
||
| 300 | 2 000 | 2 301 / 2 013 / 607 k | ~12 s |
|
||
| 745 (all) | 1 000 | 1 746 / 1 013 / 749 k | ~17 s |
|
||
| 100, rebalancing with fixed costs | 2 000 | 2 401 / 2 215 / 808 k (400 integers) | ~33 s |
|
||
|
||
The time limit (`QUANTCORE_CUOPT_TIME_LIMIT`, request field `time_limit`) bounds the solve; a
|
||
`FeasibleFound` status with a small `mip_gap` is a valid, slightly sub-optimal answer.
|
||
|
||
BLAS threads are pinned to 4 (`OPENBLAS_NUM_THREADS`): the empirical-innovation filter performs
|
||
thousands of small eigendecompositions, which OpenBLAS runs ~50× slower with all 20 cores.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause / fix |
|
||
|---|---|
|
||
| `/health` shows `cuopt.reachable: false` | cuOpt container down or unhealthy: `docker compose restart cuopt` (see the Spark README §5). Optimisations fall back to HiGHS and say so in `solver` / `warnings`. |
|
||
| `/optimize` answers 503 "no market data yet" | run `POST /jobs/setup` or deploy with `--with-data`. |
|
||
| Yahoo errors in the setup/update job | rate limiting; the client retries with back-off and opens a circuit after 5 failures. Re-run the job later; stored data stays intact. |
|
||
| `FeasibleFound` instead of `Optimal` | time limit hit; raise `time_limit` or reduce `n_scenarios` / `max_assets`. |
|
||
| `Infeasible` | the constraints contradict each other (e.g. `min_invested` with a tiny CVaR limit, or a rebalancing turnover cap that prevents reaching the risk budget). |
|
||
| Tailscale SSH asks for a browser check | `ssh user_fh_fhgr@spark-6d4a` prints a login URL; open it once, then re-run the deploy script. |
|