Portfolio_Risk_Management/docs/SPARK_APPTAINER.md

quantcore on your DGX Spark (Apptainer, no root)

This guide sets up the quant service of the project on your own DGX Spark. The service covers market data, risk measurement, Black–Litterman expected returns and the mean-CVaR optimiser. Everything runs on the Spark itself:

• cuOpt, NVIDIA's GPU solver, runs in its own Apptainer container.
• quantcore, this repository, runs in a second Apptainer container. It calls cuOpt at http://127.0.0.1:5000.
• The API and the browser test bench listen on http://127.0.0.1:5001. You reach them from the Spark: its desktop browser, a terminal, a notebook, or the platform you build. They are not reachable from the network.

You need no root rights, no Docker and no VPN. One script, deploy/apptainer/qc.sh, does everything.

What you need

Requirement	Check
Apptainer 1.1 or newer	apptainer --version
The GPU	nvidia-smi -L lists the NVIDIA GB10. Without it, optimisation runs on HiGHS on the CPU.
About 5 GB in your home directory	The cuOpt image is 3.3 GB, quantcore 0.2 GB, the data 40 MB. Pulling cuOpt needs about 12 GB for a few minutes.
Internet access	Docker Hub for the images, PyPI for the Python libraries, Yahoo Finance for market data.

Quick start

git clone https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git
cd Portfolio_Risk_Management

deploy/apptainer/qc.sh install      # once, ~10 min: settings, cuOpt image, quantcore image, self-test
deploy/apptainer/qc.sh data-import /path/to/quantcore_data.tar.gz   # the prepared data set
#   or build the data set from Yahoo Finance yourself (~15 min):  deploy/apptainer/qc.sh setup
deploy/apptainer/qc.sh start        # starts cuOpt and the API

Then open http://127.0.0.1:5001/ui/ in the Spark's browser. The interactive API documentation is at http://127.0.0.1:5001/docs.

Working on the Spark over SSH from your laptop? Forward the port, then open the same address in the laptop's browser. Nothing else on the network can reach the service:

ssh -L 5001:127.0.0.1:5001 <your-user>@<your-spark>

A shorter command saves typing. Put this line in your ~/.bashrc:

alias qc="$HOME/Portfolio_Risk_Management/deploy/apptainer/qc.sh"

The rest of this guide writes qc for the script.

Daily use

qc start                 # cuOpt + API; after a reboot, run it again
qc status                # what runs, whether cuOpt is reachable, how much data there is
qc logs                  # API log; `qc logs cuopt` for the solver, add -f to follow
qc update                # the newest prices and FX rates from Yahoo Finance
qc stop                  # stop both containers

The containers keep running after you log out. They stop when the Spark reboots.

Using the service

In the browser. The test bench at http://127.0.0.1:5001/ui/ shows every service: the stock universe, risk and scenarios, expected returns and views, constraints, and the optimisation with a before/after comparison.

From your own code. This is how the platform you build uses the service. The API speaks JSON over HTTP, so any language works. Your code reaches it at http://127.0.0.1:5001. That address also works inside your own Apptainer containers, because Apptainer containers share the Spark's network. For Python there is a ready-made client:

from quantcore.client import QuantcoreClient

qc = QuantcoreClient("http://127.0.0.1:5001")
print(qc.health()["cuopt"]["reachable"])
risk = qc.asset_risk(["NESN.SW", "AAPL"], horizon="week")

Install the client into your own environment with pip install "git+https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git". USAGE.md describes every endpoint and the request fields.

From the command line, through the API. qc api runs the API client against your service:

qc api health
qc api selftest                      # calls every endpoint, pass/fail table
qc api risk NESN.SW AAPL --horizon week
qc api optimize --cash 1000000 --cvar-limit 40000 --horizon week

From the command line, without the API. qc cli runs the services directly in the container. The API does not need to be running:

qc cli status
qc cli risk NESN.SW AAPL --horizon week
qc cli optimize --cash 1000000 --cvar-limit 40000 --horizon week --max-assets 50

Settings

qc install writes the settings file ~/quantcore/quantcore.env. Edit it, then run qc restart. A value set in the shell wins over the file, for example QC_API_PORT=5011 qc start.

Setting	Default	Meaning
QC_API_PORT	5001	Port of the API and the test bench.
QC_CUOPT_PORT	5000	Port of the cuOpt server.
QC_CUOPT_START	auto	auto uses a cuOpt that already answers on the port, or starts one. yes always starts its own. no never starts one, so HiGHS solves every model.
QC_CUOPT_IMAGE	docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14	Where the cuOpt image comes from. Set the path of a .sif file if your Spark already provides one.
QC_IMAGE	build	How quantcore gets its container; see the next section.
QUANTCORE_*		Passed into the container unchanged. Examples are QUANTCORE_CUOPT_TIME_LIMIT and QUANTCORE_ENABLE_UI. All of them are listed in quantcore/config.py.

Several users on one Spark. Ports belong to the whole machine. If someone else already uses port 5001, qc start stops and suggests a free port to put into your settings file. A cuOpt that another user already runs on port 5000 is simply used: cuOpt keeps no state between requests.

How quantcore gets its container

QC_IMAGE=build is the default. qc install then builds the image quantcore.sif from this repository with Apptainer's unprivileged build. Some Spark installations do not allow unprivileged builds. In that case qc install falls back by itself to a setup that needs no build. It pulls a plain Python image and installs quantcore into a virtual environment in ~/quantcore/venv. The libraries and the results are the same, because both setups install the pinned versions in deploy/apptainer/constraints.txt. You can choose the fallback directly with QC_IMAGE=venv.

QC_IMAGE also accepts the path of a prebuilt quantcore.sif, or a docker:// or oras:// address to pull one from.

After git pull, rebuild and restart:

git pull && qc build && qc restart

How it works

 DGX Spark, your user account, no root
 ┌────────────────────────────────────────────────────────────────────────────┐
 │  browser / your platform / qc api                                          │
 │        │ HTTP                                                              │
 │        ▼                                                                   │
 │  127.0.0.1:5001   instance "quantcore-api"    quantcore.sif                │
 │                   FastAPI + test bench        ~/quantcore/data -> /data    │
 │        │ HTTP (msgpack)                                                    │
 │        ▼                                                                   │
 │  127.0.0.1:5000   instance "quantcore-cuopt"  cuopt.sif, started with --nv  │
 │                   cuOpt server                ──► GB10 GPU                 │
 └────────────────────────────────────────────────────────────────────────────┘

• An Apptainer instance is a container that runs in the background, like a service. apptainer instance list shows yours.
• Apptainer containers use the Spark's network directly, with no port mapping. A server inside a container that listens on 127.0.0.1 is therefore reachable from the Spark, and only from it.
• The images are read-only. Everything that changes lives in ~/quantcore/data, which is mounted into the container as /data.
• The containers start with --cleanenv. Variables from your shell, such as an activated virtual environment or PYTHONPATH, cannot interfere. qc.sh passes only the settings above, plus proxy variables if you have them.
• NVIDIA's cuOpt image sets its home directory to /opt/cuopt. That folder is writable under Docker but read-only in an Apptainer image, and the solver would crash on its first job when CuPy writes its kernel cache there. qc.sh therefore starts the cuOpt server with its home directory in ~/quantcore/cuopt-home.
• When cuOpt is not reachable, or stops during a solve, every optimisation falls back to HiGHS on the CPU. The result says which solver produced it.

Path	Content
~/quantcore/quantcore.env	your settings
~/quantcore/data/	universe, prices, FX, snapshots
~/quantcore/images/cuopt.sif	the cuOpt image
~/quantcore/images/quantcore.sif	the quantcore image; in the fallback, python.sif plus ~/quantcore/venv/
~/quantcore/cuopt-home/	cuOpt's caches (CuPy kernels, CUDA)
~/.apptainer/instances/logs/	instance logs, also shown by qc logs
~/.apptainer/cache/	downloaded image layers; qc clean-cache frees the space

Troubleshooting

Symptom	Cause and fix
port 5001 is taken	Another user or program uses the port. Set the suggested QC_API_PORT in ~/quantcore/quantcore.env, then run qc start.
qc status shows cuOpt as not answering	Run qc logs cuopt. Check that nvidia-smi works. Then qc restart. Optimisations keep working on HiGHS in the meantime.
The cuOpt log says Read-only file system: '/opt/cuopt/.cupy'	cuOpt was started without qc.sh. Start it with qc start, or give it a writable home directory as described in "Running cuOpt by hand".
/optimize answers 503, "no market data yet"	Run qc data-import <file> or qc setup.
Yahoo errors during qc setup or qc update	Yahoo limits how fast it answers. Run the command again later; stored data stays intact.
qc install stops with "no 'instance run'"	The Spark's Apptainer is older than 1.1. Ask the administrator to update it.
The cuOpt pull fails with "no space left on device"	Apptainer unpacks images in /tmp. Point it at a larger directory: APPTAINER_TMPDIR=~/quantcore/tmp qc pull-cuopt.
Model status FeasibleFound instead of Optimal	The time limit was reached. Raise time_limit in the request or QUANTCORE_CUOPT_TIME_LIMIT, or reduce n_scenarios or max_assets.

Running cuOpt by hand

qc start does this for you. If you want to run the cuOpt image yourself, for example for another project, start it like this:

apptainer pull cuopt.sif docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14
mkdir -p ~/cuopt-home
apptainer instance run --nv --cleanenv --bind ~/cuopt-home cuopt.sif my-cuopt \
    env HOME=$HOME/cuopt-home python -m cuopt_server.cuopt_service -i 127.0.0.1 -p 5000
curl http://127.0.0.1:5000/cuopt/health
apptainer instance stop my-cuopt

For the lecturer

Data set for the students. Many students running qc setup at the same time from one network can hit Yahoo's rate limits. A prepared data set avoids that, and every student then works on the same data. Pack it on a machine that has the data:

qc data-export quantcore_data.tar.gz              # or, from any data directory:
tar -czf quantcore_data.tar.gz -C <data dir> --exclude ./jobs --exclude '*.log' .

The archive is about 30 MB. It is not part of the public repository: Yahoo's terms do not allow redistributing their data.

Prebuilt image. Build quantcore.sif on any aarch64 Linux machine where you can build images, then hand out the file:

apptainer build quantcore.sif deploy/apptainer/quantcore.def    # from the repository root

Students set QC_IMAGE=/path/to/quantcore.sif in their settings file. A copy built and tested on the lecturer's Spark is described in the verification note below.

The lecturer's own Spark runs the Docker setup in SPARK_DEPLOYMENT.md. That setup needs root and Docker. Students do not need it.

Verification. The setup was tested end to end on 7 October 2026 on a DGX Spark: NVIDIA GB10, driver 580.159, Ubuntu 24.04, Apptainer 1.5.4, as a user without root. That Spark allowed no unprivileged image builds, so qc install took the virtualenv fallback by itself. The image mode was then tested with a prebuilt quantcore.sif. In both modes:

• qc api selftest --full passed on every endpoint, and the optimisation ran on cuOpt on the GPU.
• qc update fetched the newest Yahoo prices through the running API.
• A taken API port stopped qc start with a suggested free port, and that port worked.
• The test suite passed inside the container, including the round trip to the live cuOpt server.