Portfolio_Risk_Management/docs/SPARK_APPTAINER.md

# quantcore on your DGX Spark (Apptainer, no root)

This guide sets up the quant service of the project on your own DGX Spark. The service covers
market data, risk measurement, Black–Litterman expected returns and the mean-CVaR optimiser.
Everything runs on the Spark itself:

- **cuOpt**, NVIDIA's GPU solver, runs in its own Apptainer container.
- **quantcore**, this repository, runs in a second Apptainer container. It calls cuOpt at
  `http://127.0.0.1:5000`.
- The API and the browser test bench listen on `http://127.0.0.1:5001`. You reach them from the
  Spark: its desktop browser, a terminal, a notebook, or the platform you build. They are not
  reachable from the network.

You need no root rights, no Docker and no VPN. One script, `deploy/apptainer/qc.sh`, does
everything.

## What you need

| Requirement | Check |
|---|---|
| Apptainer 1.1 or newer | `apptainer --version` |
| The GPU | `nvidia-smi -L` lists the NVIDIA GB10. Without it, optimisation runs on HiGHS on the CPU. |
| About 5 GB in your home directory | The cuOpt image is 3.3 GB, quantcore 0.2 GB, the data 40 MB. Pulling cuOpt needs about 12 GB for a few minutes. |
| Internet access | Docker Hub for the images, PyPI for the Python libraries, Yahoo Finance for market data. |

## Quick start

```bash
git clone https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git
cd Portfolio_Risk_Management

deploy/apptainer/qc.sh install      # once, ~10 min: settings, cuOpt image, quantcore image, self-test
deploy/apptainer/qc.sh data-import /path/to/quantcore_data.tar.gz   # the prepared data set
#   or build the data set from Yahoo Finance yourself (~15 min):  deploy/apptainer/qc.sh setup
deploy/apptainer/qc.sh start        # starts cuOpt and the API
```

Then open **http://127.0.0.1:5001/ui/** in the Spark's browser. The interactive API
documentation is at **http://127.0.0.1:5001/docs**.

Working on the Spark over SSH from your laptop? Forward the port, then open the same address in
the laptop's browser. Nothing else on the network can reach the service:

```bash
ssh -L 5001:127.0.0.1:5001 <your-user>@<your-spark>
```

A shorter command saves typing. Put this line in your `~/.bashrc`:

```bash
alias qc="$HOME/Portfolio_Risk_Management/deploy/apptainer/qc.sh"
```

The rest of this guide writes `qc` for the script.

## Daily use

```bash
qc start                 # cuOpt + API; after a reboot, run it again
qc status                # what runs, whether cuOpt is reachable, how much data there is
qc logs                  # API log; `qc logs cuopt` for the solver, add -f to follow
qc update                # the newest prices and FX rates from Yahoo Finance
qc stop                  # stop both containers
```

The containers keep running after you log out. They stop when the Spark reboots.

## Using the service

**In the browser.** The test bench at `http://127.0.0.1:5001/ui/` shows every service: the
stock universe, risk and scenarios, expected returns and views, constraints, and the
optimisation with a before/after comparison.

**From your own code.** This is how the platform you build uses the service. The API speaks
JSON over HTTP, so any language works. Your code reaches it at `http://127.0.0.1:5001`. That
address also works inside your own Apptainer containers, because Apptainer containers share
the Spark's network. For Python there is a ready-made client:

```python
from quantcore.client import QuantcoreClient

qc = QuantcoreClient("http://127.0.0.1:5001")
print(qc.health()["cuopt"]["reachable"])
risk = qc.asset_risk(["NESN.SW", "AAPL"], horizon="week")
```

Install the client into your own environment with
`pip install "git+https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git"`.
[USAGE.md](USAGE.md) describes every endpoint and the request fields.

**From the command line, through the API.** `qc api` runs the API client against your
service:

```bash
qc api health
qc api selftest                      # calls every endpoint, pass/fail table
qc api risk NESN.SW AAPL --horizon week
qc api optimize --cash 1000000 --cvar-limit 40000 --horizon week
```

**From the command line, without the API.** `qc cli` runs the services directly in the
container. The API does not need to be running:

```bash
qc cli status
qc cli risk NESN.SW AAPL --horizon week
qc cli optimize --cash 1000000 --cvar-limit 40000 --horizon week --max-assets 50
```

## Settings

`qc install` writes the settings file `~/quantcore/quantcore.env`. Edit it, then run
`qc restart`. A value set in the shell wins over the file, for example
`QC_API_PORT=5011 qc start`.

| Setting | Default | Meaning |
|---|---|---|
| `QC_API_PORT` | `5001` | Port of the API and the test bench. |
| `QC_CUOPT_PORT` | `5000` | Port of the cuOpt server. |
| `QC_CUOPT_START` | `auto` | `auto` uses a cuOpt that already answers on the port, or starts one. `yes` always starts its own. `no` never starts one, so HiGHS solves every model. |
| `QC_CUOPT_IMAGE` | `docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14` | Where the cuOpt image comes from. Set the path of a `.sif` file if your Spark already provides one. |
| `QC_IMAGE` | `build` | How quantcore gets its container; see the next section. |
| `QUANTCORE_*` | | Passed into the container unchanged. Examples are `QUANTCORE_CUOPT_TIME_LIMIT` and `QUANTCORE_ENABLE_UI`. All of them are listed in `quantcore/config.py`. |

**Several users on one Spark.** Ports belong to the whole machine. If someone else already uses
port 5001, `qc start` stops and suggests a free port to put into your settings file. A cuOpt
that another user already runs on port 5000 is simply used: cuOpt keeps no state between
requests.

## How quantcore gets its container

`QC_IMAGE=build` is the default. `qc install` then builds the image `quantcore.sif` from this
repository with Apptainer's unprivileged build. Some Spark installations do not allow
unprivileged builds. In that case `qc install` falls back by itself to a setup that needs no
build. It pulls a plain Python image and installs quantcore into a virtual environment in
`~/quantcore/venv`. The libraries and the results are the same, because both setups install the
pinned versions in `deploy/apptainer/constraints.txt`. You can choose the fallback directly
with `QC_IMAGE=venv`.

`QC_IMAGE` also accepts the path of a prebuilt `quantcore.sif`, or a `docker://` or `oras://`
address to pull one from.

**After `git pull`**, rebuild and restart:

```bash
git pull && qc build && qc restart
```

## How it works

```
 DGX Spark, your user account, no root
 ┌────────────────────────────────────────────────────────────────────────────┐
 │  browser / your platform / qc api                                          │
 │        │ HTTP                                                              │
 │        ▼                                                                   │
 │  127.0.0.1:5001   instance "quantcore-api"    quantcore.sif                │
 │                   FastAPI + test bench        ~/quantcore/data -> /data    │
 │        │ HTTP (msgpack)                                                    │
 │        ▼                                                                   │
 │  127.0.0.1:5000   instance "quantcore-cuopt"  cuopt.sif, started with --nv  │
 │                   cuOpt server                ──► GB10 GPU                 │
 └────────────────────────────────────────────────────────────────────────────┘
```

- An Apptainer **instance** is a container that runs in the background, like a service.
  `apptainer instance list` shows yours.
- Apptainer containers use the Spark's network directly, with no port mapping. A server inside a
  container that listens on `127.0.0.1` is therefore reachable from the Spark, and only from it.
- The images are read-only. Everything that changes lives in `~/quantcore/data`, which is
  mounted into the container as `/data`.
- The containers start with `--cleanenv`. Variables from your shell, such as an activated
  virtual environment or `PYTHONPATH`, cannot interfere. `qc.sh` passes only the settings above,
  plus proxy variables if you have them.
- NVIDIA's cuOpt image sets its home directory to `/opt/cuopt`. That folder is writable under
  Docker but read-only in an Apptainer image, and the solver would crash on its first job when
  CuPy writes its kernel cache there. `qc.sh` therefore starts the cuOpt server with its home
  directory in `~/quantcore/cuopt-home`.
- When cuOpt is not reachable, or stops during a solve, every optimisation falls back to HiGHS on
  the CPU. The result says which solver produced it.

| Path | Content |
|---|---|
| `~/quantcore/quantcore.env` | your settings |
| `~/quantcore/data/` | universe, prices, FX, snapshots |
| `~/quantcore/images/cuopt.sif` | the cuOpt image |
| `~/quantcore/images/quantcore.sif` | the quantcore image; in the fallback, `python.sif` plus `~/quantcore/venv/` |
| `~/quantcore/cuopt-home/` | cuOpt's caches (CuPy kernels, CUDA) |
| `~/.apptainer/instances/logs/` | instance logs, also shown by `qc logs` |
| `~/.apptainer/cache/` | downloaded image layers; `qc clean-cache` frees the space |

## Troubleshooting

| Symptom | Cause and fix |
|---|---|
| `port 5001 is taken` | Another user or program uses the port. Set the suggested `QC_API_PORT` in `~/quantcore/quantcore.env`, then run `qc start`. |
| `qc status` shows cuOpt as not answering | Run `qc logs cuopt`. Check that `nvidia-smi` works. Then `qc restart`. Optimisations keep working on HiGHS in the meantime. |
| The cuOpt log says `Read-only file system: '/opt/cuopt/.cupy'` | cuOpt was started without `qc.sh`. Start it with `qc start`, or give it a writable home directory as described in "Running cuOpt by hand". |
| `/optimize` answers 503, "no market data yet" | Run `qc data-import <file>` or `qc setup`. |
| Yahoo errors during `qc setup` or `qc update` | Yahoo limits how fast it answers. Run the command again later; stored data stays intact. |
| `qc install` stops with "no 'instance run'" | The Spark's Apptainer is older than 1.1. Ask the administrator to update it. |
| The cuOpt pull fails with "no space left on device" | Apptainer unpacks images in `/tmp`. Point it at a larger directory: `APPTAINER_TMPDIR=~/quantcore/tmp qc pull-cuopt`. |
| Model status `FeasibleFound` instead of `Optimal` | The time limit was reached. Raise `time_limit` in the request or `QUANTCORE_CUOPT_TIME_LIMIT`, or reduce `n_scenarios` or `max_assets`. |

## Running cuOpt by hand

`qc start` does this for you. If you want to run the cuOpt image yourself, for example for another
project, start it like this:

```bash
apptainer pull cuopt.sif docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14
mkdir -p ~/cuopt-home
apptainer instance run --nv --cleanenv --bind ~/cuopt-home cuopt.sif my-cuopt \
    env HOME=$HOME/cuopt-home python -m cuopt_server.cuopt_service -i 127.0.0.1 -p 5000
curl http://127.0.0.1:5000/cuopt/health
apptainer instance stop my-cuopt
```

## For the lecturer

**Data set for the students.** Many students running `qc setup` at the same time from one
network can hit Yahoo's rate limits. A prepared data set avoids that, and every student then
works on the same data. Pack it on a machine that has the data:

```bash
qc data-export quantcore_data.tar.gz              # or, from any data directory:
tar -czf quantcore_data.tar.gz -C <data dir> --exclude ./jobs --exclude '*.log' .
```

The archive is about 30 MB. It is not part of the public repository: Yahoo's terms do not allow
redistributing their data.

**Prebuilt image.** Build `quantcore.sif` on any aarch64 Linux machine where you can build
images, then hand out the file:

```bash
apptainer build quantcore.sif deploy/apptainer/quantcore.def    # from the repository root
```

Students set `QC_IMAGE=/path/to/quantcore.sif` in their settings file. A copy built and tested on
the lecturer's Spark is described in the verification note below.

**The lecturer's own Spark** runs the Docker setup in [SPARK_DEPLOYMENT.md](SPARK_DEPLOYMENT.md).
That setup needs root and Docker. Students do not need it.

**Verification.** The setup was tested end to end on 7 October 2026 on a DGX Spark: NVIDIA GB10,
driver 580.159, Ubuntu 24.04, Apptainer 1.5.4, as a user without root. That Spark allowed no
unprivileged image builds, so `qc install` took the virtualenv fallback by itself. The image mode
was then tested with a prebuilt `quantcore.sif`. In both modes:

- `qc api selftest --full` passed on every endpoint, and the optimisation ran on cuOpt on the GPU.
- `qc update` fetched the newest Yahoo prices through the running API.
- A taken API port stopped `qc start` with a suggested free port, and that port worked.
- The test suite passed inside the container, including the round trip to the live cuOpt server.