Portfolio_Risk_Management/docs/SPARK_APPTAINER.md

# quantcore on your DGX Spark (Apptainer, no root)

This guide sets up the quant service of the project on your own DGX Spark. The service covers
market data, risk measurement, Black–Litterman expected returns and the mean-CVaR optimiser.
Everything runs on the Spark itself:

- **cuOpt**, NVIDIA's GPU solver, runs in its own Apptainer container.
- **quantcore**, this repository, runs in a second Apptainer container. It calls cuOpt at
  `http://127.0.0.1:5000`.
- The API and the browser test bench listen on `http://127.0.0.1:5001`. You reach them from the
  Spark: its desktop browser, a terminal, a notebook, or the platform you build. They are not
  reachable from the network.

You need no root rights, no Docker and no VPN. One script, `deploy/apptainer/qc.sh`, does
everything.

## What you need

| Requirement | Check |
|---|---|
| Apptainer 1.1 or newer | `apptainer --version` |
| The GPU | `nvidia-smi -L` lists the NVIDIA GB10. Without it, optimisation runs on HiGHS on the CPU. |
| About 5 GB in your home directory | The cuOpt image is 3.3 GB, quantcore 0.2 GB, the data 40 MB. Pulling cuOpt needs about 12 GB for a few minutes. |
| Internet access | Docker Hub for the images, PyPI for the Python libraries, Yahoo Finance for market data. |

## Quick start

Open a terminal on the Spark and run this one line:

```bash
wget -qO- https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash
```

If the Spark has no `wget`, one of these does the same:

```bash
curl -fsSL https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash
python3 -c "import urllib.request as u;print(u.urlopen('https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh').read().decode())" | bash
```

The installer takes 20 to 30 minutes the first time and prints each step:

1. It checks Apptainer and the GPU. It also repairs Apptainer's temporary and cache folders if
   your account points them at a folder that does not exist.
2. It downloads the code to `~/Portfolio_Risk_Management`. If the Spark has no git, it uses git
   from a small container.
3. It gets the cuOpt image and the quantcore image.
4. It builds the market data from Yahoo Finance, which takes about 15 minutes. The candidate
   stocks come from the list shipped in the repository, so every student gets the same universe.
   Companies that Yahoo no longer lists are left out, and setup names them in one summary line.
5. It starts the service, runs the self-test and installs the command `qc`.

With a prepared data set from your lecturer, the data step takes seconds instead. Pass the file
with `--data`:

```bash
wget -qO- https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash -s -- --data ~/Downloads/quantcore_data.tar.gz
```

Running the installer again is safe. It skips what is done and updates the rest.

When it is done, open **http://127.0.0.1:5001/ui/** in the Spark's browser. The interactive API
documentation is at **http://127.0.0.1:5001/docs**.

Working on the Spark over SSH from your laptop? Forward the port, then open the same address in
the laptop's browser. Nothing else on the network can reach the service:

```bash
ssh -L 5001:127.0.0.1:5001 <your-user>@<your-spark>
```

The installer puts the command `qc` into `~/.local/bin`. Open a new terminal once so that the
shell finds it. `qc` is the same as `deploy/apptainer/qc.sh` in the code folder.

## Daily use

```bash
qc start                 # cuOpt + API; after a reboot, run it again
qc status                # what runs, whether cuOpt is reachable, how much data there is
qc logs                  # API log; `qc logs cuopt` for the solver, add -f to follow
qc update                # the newest prices and FX rates from Yahoo Finance
qc upgrade               # the newest version of the code: download, rebuild, restart
~/Portfolio_Risk_Management/install.sh --rebuild-data   # set the data aside and build it anew
qc stop                  # stop both containers
```

The containers keep running after you log out. They stop when the Spark reboots.

## Using the service

**In the browser.** The test bench at `http://127.0.0.1:5001/ui/` shows every service: the
stock universe, risk and scenarios, expected returns and views, constraints, and the
optimisation with a before/after comparison.

**From your own code.** This is how the platform you build uses the service. The API speaks
JSON over HTTP, so any language works. Your code reaches it at `http://127.0.0.1:5001`. That
address also works inside your own Apptainer containers, because Apptainer containers share
the Spark's network. For Python there is a ready-made client:

```python
from quantcore.client import QuantcoreClient

qc = QuantcoreClient("http://127.0.0.1:5001")
print(qc.health()["cuopt"]["reachable"])
risk = qc.asset_risk(["NESN.SW", "AAPL"], horizon="week")
```

Install the client into your own environment with
`pip install "git+https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git"`.
[USAGE.md](USAGE.md) describes every endpoint and the request fields.

**From the command line, through the API.** `qc api` runs the API client against your
service:

```bash
qc api health
qc api selftest                      # calls every endpoint, pass/fail table
qc api risk NESN.SW AAPL --horizon week
qc api optimize --cash 1000000 --cvar-limit 40000 --horizon week
```

**From the command line, without the API.** `qc cli` runs the services directly in the
container. The API does not need to be running:

```bash
qc cli status
qc cli risk NESN.SW AAPL --horizon week
qc cli optimize --cash 1000000 --cvar-limit 40000 --horizon week --max-assets 50
```

## Settings

`qc install` writes the settings file `~/quantcore/quantcore.env`. Edit it, then run
`qc restart`. A value set in the shell wins over the file, for example
`QC_API_PORT=5011 qc start`.

| Setting | Default | Meaning |
|---|---|---|
| `QC_API_PORT` | `5001` | Port of the API and the test bench. |
| `QC_CUOPT_PORT` | `5000` | Port of the cuOpt server. |
| `QC_CUOPT_START` | `auto` | `auto` uses a cuOpt that already answers on the port, or starts one. `yes` always starts its own. `no` never starts one, so HiGHS solves every model. |
| `QC_CUOPT_IMAGE` | `docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14` | Where the cuOpt image comes from. Set the path of a `.sif` file if your Spark already provides one. |
| `QC_IMAGE` | `build` | How quantcore gets its container; see the next section. |
| `QUANTCORE_*` | | Passed into the container unchanged. Examples are `QUANTCORE_CUOPT_TIME_LIMIT` and `QUANTCORE_ENABLE_UI`. All of them are listed in `quantcore/config.py`. |

**Several users on one Spark.** Ports belong to the whole machine. If someone else already uses
port 5001, `qc start` stops and suggests a free port to put into your settings file. A cuOpt
that another user already runs on port 5000 is simply used: cuOpt keeps no state between
requests.

## How quantcore gets its container

`QC_IMAGE=build` is the default. `qc install` then builds the image `quantcore.sif` from this
repository with Apptainer's unprivileged build. Some Spark installations do not allow
unprivileged builds. In that case `qc install` falls back by itself to a setup that needs no
build. It pulls a plain Python image and installs quantcore into a virtual environment in
`~/quantcore/venv`. The libraries and the results are the same, because both setups install the
pinned versions in `deploy/apptainer/constraints.txt`. You can choose the fallback directly
with `QC_IMAGE=venv`.

`QC_IMAGE` also accepts the path of a prebuilt `quantcore.sif`, or a `docker://` or `oras://`
address to pull one from.

**New versions of the code** come with `qc upgrade`. It downloads the changes, rebuilds quantcore
and restarts the service.

## How it works

```
 DGX Spark, your user account, no root
 ┌────────────────────────────────────────────────────────────────────────────┐
 │  browser / your platform / qc api                                          │
 │        │ HTTP                                                              │
 │        ▼                                                                   │
 │  127.0.0.1:5001   instance "quantcore-api"    quantcore.sif                │
 │                   FastAPI + test bench        ~/quantcore/data -> /data    │
 │        │ HTTP (msgpack)                                                    │
 │        ▼                                                                   │
 │  127.0.0.1:5000   instance "quantcore-cuopt"  cuopt.sif, started with --nv  │
 │                   cuOpt server                ──► GB10 GPU                 │
 └────────────────────────────────────────────────────────────────────────────┘
```

- An Apptainer **instance** is a container that runs in the background, like a service.
  `apptainer instance list` shows yours.
- Apptainer containers use the Spark's network directly, with no port mapping. A server inside a
  container that listens on `127.0.0.1` is therefore reachable from the Spark, and only from it.
- The images are read-only. Everything that changes lives in `~/quantcore/data`, which is
  mounted into the container as `/data`.
- The containers start with `--cleanenv`. Variables from your shell, such as an activated
  virtual environment or `PYTHONPATH`, cannot interfere. `qc.sh` passes only the settings above,
  plus proxy variables if you have them.
- NVIDIA's cuOpt image sets its home directory to `/opt/cuopt`. That folder is writable under
  Docker but read-only in an Apptainer image, and the solver would crash on its first job when
  CuPy writes its kernel cache there. `qc.sh` therefore starts the cuOpt server with its home
  directory in `~/quantcore/cuopt-home`.
- When cuOpt is not reachable, or stops during a solve, every optimisation falls back to HiGHS on
  the CPU. The result says which solver produced it.

| Path | Content |
|---|---|
| `~/quantcore/quantcore.env` | your settings |
| `~/quantcore/data/` | universe, prices, FX, snapshots |
| `~/quantcore/images/cuopt.sif` | the cuOpt image |
| `~/quantcore/images/quantcore.sif` | the quantcore image; in the fallback, `python.sif` plus `~/quantcore/venv/` |
| `~/quantcore/cuopt-home/` | cuOpt's caches (CuPy kernels, CUDA) |
| `~/.apptainer/instances/logs/` | instance logs, also shown by `qc logs` |
| `~/.apptainer/cache/` | downloaded image layers; `qc clean-cache` frees the space |

## Troubleshooting

| Symptom | Cause and fix |
|---|---|
| `port 5001 is taken` | Another user or program uses the port. Set the suggested `QC_API_PORT` in `~/quantcore/quantcore.env`, then run `qc start`. |
| `qc status` shows cuOpt as not answering | Run `qc logs cuopt`. Check that `nvidia-smi` works. Then `qc restart`. Optimisations keep working on HiGHS in the meantime. |
| The cuOpt log says `Read-only file system: '/opt/cuopt/.cupy'` | cuOpt was started without `qc.sh`. Start it with `qc start`, or give it a writable home directory as described in "Running cuOpt by hand". |
| `/optimize` answers 503, "no market data yet" | Run `qc data-import <file>` or `qc setup`. |
| Yahoo errors during `qc setup` or `qc update` | Yahoo limits how fast it answers. Run the command again later; stored data stays intact. |
| `FATAL: ... failed to create build parent dir: stat /scratch/...` | Your account sets `APPTAINER_TMPDIR` to a folder that does not exist. The installer and `qc` now detect this, use a folder that works, and save the choice in `~/quantcore/quantcore.env`. Run the installer again. |
| `git: command not found` | Use the one-line installer. It needs no git. |
| Hundreds of red `Quote not found for symbol` lines during setup | Your installation predates the fix. Its setup used guessed symbols, so many European companies are missing from your data. Run `qc upgrade`, then `~/Portfolio_Risk_Management/install.sh --rebuild-data`. |
| `qc: command not found` right after the installation | Open a new terminal, or run `export PATH="$HOME/.local/bin:$PATH"`. |
| `qc install` stops with "no 'instance run'" | The Spark's Apptainer is older than 1.1. Ask the administrator to update it. |
| The cuOpt pull fails with "no space left on device" | Apptainer unpacks images in `/tmp`. Point it at a larger directory: `APPTAINER_TMPDIR=~/quantcore/tmp qc pull-cuopt`. |
| Model status `FeasibleFound` instead of `Optimal` | The time limit was reached. Raise `time_limit` in the request or `QUANTCORE_CUOPT_TIME_LIMIT`, or reduce `n_scenarios` or `max_assets`. |

## Running cuOpt by hand

`qc start` does this for you. If you want to run the cuOpt image yourself, for example for another
project, start it like this:

```bash
apptainer pull cuopt.sif docker://nvidia/cuopt:26.8.0-cuda13.3-py3.14
mkdir -p ~/cuopt-home
apptainer instance run --nv --cleanenv --bind ~/cuopt-home cuopt.sif my-cuopt \
    env HOME=$HOME/cuopt-home python -m cuopt_server.cuopt_service -i 127.0.0.1 -p 5000
curl http://127.0.0.1:5000/cuopt/health
apptainer instance stop my-cuopt
```

## For the lecturer

**Data set for the students.** Many students running `qc setup` at the same time from one
network can hit Yahoo's rate limits. A prepared data set avoids that, and every student then
works on the same data. Pack it on a machine that has the data:

```bash
qc data-export quantcore_data.tar.gz              # or, from any data directory:
tar -czf quantcore_data.tar.gz -C <data dir> --exclude ./jobs --exclude '*.log' .
```

The archive is about 30 MB. It is not part of the public repository: Yahoo's terms do not allow
redistributing their data.

**Prebuilt image.** Build `quantcore.sif` on any aarch64 Linux machine where you can build
images, then hand out the file:

```bash
apptainer build quantcore.sif deploy/apptainer/quantcore.def    # from the repository root
```

Students set `QC_IMAGE=/path/to/quantcore.sif` in their settings file. A copy built and tested on
the lecturer's Spark is described in the verification note below.

**The lecturer's own Spark** runs the Docker setup in [SPARK_DEPLOYMENT.md](SPARK_DEPLOYMENT.md).
That setup needs root and Docker. Students do not need it.

**Verification.** The setup was tested end to end on 7 October 2026 on a DGX Spark: NVIDIA GB10,
driver 580.159, Ubuntu 24.04, Apptainer 1.5.4, as a user without root. That Spark allowed no
unprivileged image builds, so `qc install` took the virtualenv fallback by itself. The image mode
was then tested with a prebuilt `quantcore.sif`. In both modes:

- `qc api selftest --full` passed on every endpoint, and the optimisation ran on cuOpt on the GPU.
- `qc update` fetched the newest Yahoo prices through the running API.
- A taken API port stopped `qc start` with a suggested free port, and that port worked.
- The test suite passed inside the container, including the round trip to the live cuOpt server.

The one-line installer was then tested on a stripped-down machine. It had no git, curl, wget or
python3, a home folder named like a domain account (`/home/name@edu.local`), and `APPTAINER_TMPDIR`
pointing to a `/scratch` folder that does not exist. The installer repaired the folders, fetched the
code with git from a container, installed, started and passed the self-test. `qc upgrade` picked up
a new commit, rebuilt quantcore and restarted it in under a minute. `qc setup` built a data set from
scratch inside the container.

**Where the candidate stocks come from.** The installer runs `qc setup --offline`. It takes the
candidates from the lists shipped in `quantcore/data/static/`: the S&P 500 and the iShares STOXX
Europe 600 holdings of September 2026, from which the course data set was built. `qc setup` without
`--offline` fetches the current lists instead. In October 2026 iShares moved the German download, so
setup now reads the same file from iShares' Swiss site. If that fails too, setup uses the shipped
list. It no longer uses Wikipedia's STOXX 600 table, whose codes mostly do not exist on Yahoo, for
example `BNPP.PA` instead of `BNP.PA`.