Portfolio_Risk_Management/docs/SPARK_DEPLOYMENT.md

Running the quant service on the DGX Spark (Docker, with root)

This is the setup of the lecturer's Spark, which has root rights, Docker Compose and Tailscale. Students run the service on their own Spark with Apptainer and without root: see SPARK_APPTAINER.md.

The Spark (spark-6d4a, reachable over Tailscale) already runs cuOpt, Gurobi and JupyterLab as Docker containers defined in ~/services/docker-compose.yml. The quant service joins that compose file as the container quantcore on port 5001; it reaches cuOpt over the compose network (http://cuopt:5000) and keeps its data on the host in ~/services/work/quantcore_data.

Service	Address	What it is
quantcore	http://spark-6d4a:5001 (/docs for Swagger, /ui/ for the test bench)	portfolio data, risk/performance measurement, models, mean-CVaR MILP
cuOpt	http://spark-6d4a:5000	the GPU MILP solver the service uses

Deploy (from the Mac)

deploy/deploy_to_spark.sh --with-data        # first time: also ships the locally built data set
deploy/deploy_to_spark.sh                    # later: code only

The script rsyncs the repository to ~/services/work/Portfolio_Risk_Optimization, appends the service block from deploy/docker-compose.quantcore.yml to the compose file (once; a backup of the compose file is kept), builds the image (python:3.12-slim, aarch64) and starts the container.

The block is appended only if quantcore: is not already in ~/services/docker-compose.yml, so later edits to deploy/docker-compose.quantcore.yml (a new environment variable, a changed port) do not travel with a redeploy — apply them to the Spark's compose file by hand, or delete the block there and re-run the script. Code and image changes need no such step.

Without --with-data the container starts empty; run the setup once through the API:

curl -X POST http://spark-6d4a:5001/jobs/setup            # ~15 min: 1 117 tickers' metadata + history
curl http://spark-6d4a:5001/jobs/<id>                     # status: queued | running | done | failed

Operate

ssh user_fh_fhgr@spark-6d4a
cd ~/services
docker compose ps quantcore                  # status (healthcheck on /health)
docker compose logs -f quantcore             # live logs
docker compose restart quantcore
docker compose up -d --build quantcore       # rebuild after a code change (or re-run the deploy script)

# inside the container: the CLI works against the same data volume
docker exec -it quantcore quantcore status
docker exec -it quantcore quantcore update            # newest prices (incremental)
docker exec -it quantcore quantcore snapshot --name demo

Nightly data refresh (host cron on the Spark), e.g. crontab -e:

15 23 * * 1-5  curl -s -X POST http://localhost:5001/jobs/update >/dev/null

Use from anywhere on the tailnet

curl http://spark-6d4a:5001/health
curl "http://spark-6d4a:5001/universe?region=Europe&sector=Health%20Care"
curl -s http://spark-6d4a:5001/risk/assets -H 'Content-Type: application/json' \
     -d '{"symbols": ["NESN.SW", "ROP.SW", "AAPL"], "horizon": "week"}'
curl -s http://spark-6d4a:5001/optimize -H 'Content-Type: application/json' \
     -d '{"cash": 1000000, "max_assets": 100, "cvar_limit": 40000, "horizon": "week"}'

The test bench is served from the same container (QUANTCORE_ENABLE_UI, default true, in the compose block). Set it to "false" and rebuild for a deployment that exposes the JSON API and /docs only — for users who build their own front end against these endpoints. deploy/docker-compose.students.yml is that deployment, standalone and with the bench already off; docs/USAGE.md is the guide its users need.

Python clients simply point QUANTCORE_CUOPT_URL / the API base URL at the Spark; the package itself also runs on a laptop (quantcore serve) and uses the Spark's cuOpt remotely (QUANTCORE_CUOPT_URL=http://spark-6d4a:5000), falling back to HiGHS — which needs no service of its own: it is compiled into SciPy inside the quantcore image (SciPy 1.18.1 → HiGHS 1.12.0) and runs on the container's CPU — when the Spark is unreachable.

Sizing

Candidates	Scenarios	MILP size (vars / rows / nnz)	cuOpt solve
100	2 000	2 101 / 2 013 / 206 k	~6–10 s
300	2 000	2 301 / 2 013 / 607 k	~12 s
745 (all)	1 000	1 746 / 1 013 / 749 k	~17 s
100, rebalancing with fixed costs	2 000	2 401 / 2 215 / 808 k (400 integers)	~33 s

The time limit (QUANTCORE_CUOPT_TIME_LIMIT, request field time_limit) bounds the solve; a FeasibleFound status with a small mip_gap is a valid, slightly sub-optimal answer.

BLAS threads are pinned to 4 (OPENBLAS_NUM_THREADS): the empirical-innovation filter performs thousands of small eigendecompositions, which OpenBLAS runs ~50× slower with all 20 cores.

Troubleshooting

Symptom	Cause / fix
/health shows cuopt.reachable: false	cuOpt container down or unhealthy: docker compose restart cuopt (see the Spark README §5). Optimisations fall back to HiGHS and say so in solver / warnings.
/optimize answers 503 "no market data yet"	run POST /jobs/setup or deploy with --with-data.
Yahoo errors in the setup/update job	rate limiting; the client retries with back-off and opens a circuit after 5 failures. Re-run the job later; stored data stays intact.
FeasibleFound instead of Optimal	time limit hit; raise time_limit or reduce n_scenarios / max_assets.
Infeasible	the constraints contradict each other (e.g. min_invested with a tiny CVaR limit, or a rebalancing turnover cap that prevents reaching the risk budget).
Tailscale SSH asks for a browser check	ssh user_fh_fhgr@spark-6d4a prints a login URL; open it once, then re-run the deploy script.