# Running the quant service on the DGX Spark (Docker, with root)

> This is the setup of the lecturer's Spark, which has root rights, Docker Compose and Tailscale.
> **Students** run the service on their own Spark with Apptainer and without root:
> see [SPARK_APPTAINER.md](SPARK_APPTAINER.md).

The Spark (`spark-6d4a`, reachable over Tailscale) already runs cuOpt, Gurobi and JupyterLab as
Docker containers defined in `~/services/docker-compose.yml`. The quant service joins that
compose file as the container **`quantcore`** on port **5001**; it reaches cuOpt over the compose
network (`http://cuopt:5000`) and keeps its data on the host in `~/services/work/quantcore_data`.

| Service | Address | What it is |
|---|---|---|
| **quantcore** | `http://spark-6d4a:5001` (`/docs` for Swagger, `/ui/` for the test bench) | portfolio data, risk/performance measurement, models, mean-CVaR MILP |
| cuOpt | `http://spark-6d4a:5000` | the GPU MILP solver the service uses |

## Deploy (from the Mac)

```bash
deploy/deploy_to_spark.sh --with-data        # first time: also ships the locally built data set
deploy/deploy_to_spark.sh                    # later: code only
```

The script rsyncs the repository to `~/services/work/Portfolio_Risk_Optimization`, appends the
service block from `deploy/docker-compose.quantcore.yml` to the compose file (once; a backup of the
compose file is kept), builds the image (`python:3.12-slim`, aarch64) and starts the container.

The block is appended **only if `quantcore:` is not already in `~/services/docker-compose.yml`**, so
later edits to `deploy/docker-compose.quantcore.yml` (a new environment variable, a changed port) do
not travel with a redeploy — apply them to the Spark's compose file by hand, or delete the block
there and re-run the script. Code and image changes need no such step.

Without `--with-data` the container starts empty; run the setup once through the API:

```bash
curl -X POST http://spark-6d4a:5001/jobs/setup            # ~15 min: 1 117 tickers' metadata + history
curl http://spark-6d4a:5001/jobs/<id>                     # status: queued | running | done | failed
```

## Operate

```bash
ssh user_fh_fhgr@spark-6d4a
cd ~/services
docker compose ps quantcore                  # status (healthcheck on /health)
docker compose logs -f quantcore             # live logs
docker compose restart quantcore
docker compose up -d --build quantcore       # rebuild after a code change (or re-run the deploy script)

# inside the container: the CLI works against the same data volume
docker exec -it quantcore quantcore status
docker exec -it quantcore quantcore update            # newest prices (incremental)
docker exec -it quantcore quantcore snapshot --name demo
```

Nightly data refresh (host cron on the Spark), e.g. `crontab -e`:

```
15 23 * * 1-5  curl -s -X POST http://localhost:5001/jobs/update >/dev/null
```

## Use from anywhere on the tailnet

```bash
curl http://spark-6d4a:5001/health
curl "http://spark-6d4a:5001/universe?region=Europe&sector=Health%20Care"
curl -s http://spark-6d4a:5001/risk/assets -H 'Content-Type: application/json' \
     -d '{"symbols": ["NESN.SW", "ROP.SW", "AAPL"], "horizon": "week"}'
curl -s http://spark-6d4a:5001/optimize -H 'Content-Type: application/json' \
     -d '{"cash": 1000000, "max_assets": 100, "cvar_limit": 40000, "horizon": "week"}'
```

The test bench is served from the same container (`QUANTCORE_ENABLE_UI`, default `true`, in the
compose block). Set it to `"false"` and rebuild for a deployment that exposes the JSON API and
`/docs` only — for users who build their own front end against these endpoints.
`deploy/docker-compose.students.yml` is that deployment, standalone and with the bench already
off; `docs/USAGE.md` is the guide its users need.

Python clients simply point `QUANTCORE_CUOPT_URL` / the API base URL at the Spark; the package
itself also runs on a laptop (`quantcore serve`) and uses the Spark's cuOpt remotely
(`QUANTCORE_CUOPT_URL=http://spark-6d4a:5000`), falling back to HiGHS — which needs no service of its
own: it is compiled into SciPy inside the `quantcore` image (SciPy 1.18.1 → HiGHS 1.12.0) and runs on
the container's CPU — when the
Spark is unreachable.

## Sizing

| Candidates | Scenarios | MILP size (vars / rows / nnz) | cuOpt solve |
|---|---|---|---|
| 100 | 2 000 | 2 101 / 2 013 / 206 k | ~6–10 s |
| 300 | 2 000 | 2 301 / 2 013 / 607 k | ~12 s |
| 745 (all) | 1 000 | 1 746 / 1 013 / 749 k | ~17 s |
| 100, rebalancing with fixed costs | 2 000 | 2 401 / 2 215 / 808 k (400 integers) | ~33 s |

The time limit (`QUANTCORE_CUOPT_TIME_LIMIT`, request field `time_limit`) bounds the solve; a
`FeasibleFound` status with a small `mip_gap` is a valid, slightly sub-optimal answer.

BLAS threads are pinned to 4 (`OPENBLAS_NUM_THREADS`): the empirical-innovation filter performs
thousands of small eigendecompositions, which OpenBLAS runs ~50× slower with all 20 cores.

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `/health` shows `cuopt.reachable: false` | cuOpt container down or unhealthy: `docker compose restart cuopt` (see the Spark README §5). Optimisations fall back to HiGHS and say so in `solver` / `warnings`. |
| `/optimize` answers 503 "no market data yet" | run `POST /jobs/setup` or deploy with `--with-data`. |
| Yahoo errors in the setup/update job | rate limiting; the client retries with back-off and opens a circuit after 5 failures. Re-run the job later; stored data stays intact. |
| `FeasibleFound` instead of `Optimal` | time limit hit; raise `time_limit` or reduce `n_scenarios` / `max_assets`. |
| `Infeasible` | the constraints contradict each other (e.g. `min_invested` with a tiny CVaR limit, or a rebalancing turnover cap that prevents reaching the risk budget). |
| Tailscale SSH asks for a browser check | `ssh user_fh_fhgr@spark-6d4a` prints a login URL; open it once, then re-run the deploy script. |