Portfolio_Risk_Management/README.md

# quantcore — Portfolio Risk Measurement & Optimisation

The **deterministic quant core** of the AI-Augmented Portfolio Intelligence Platform (AISE502):
data, risk and performance measurement, stock-price models, Black–Litterman expected returns and
mean–CVaR portfolio optimisation in whole shares, solved as a MILP on **NVIDIA cuOpt** on the DGX Spark.

Everything here is *deterministic* — same input, same output — and contains **no LLM call**. The AI
layer of the platform (news → insights → views) consumes these services through their contracts and
never computes a number itself.

| Service | Module | What it does |
|---|---|---|
| **SetupService** | `quantcore.universe` | Builds the investment universe **once**: largest 500 US (S&P 500) and 500 European (STOXX 600; EU, CH, NO, UK) stocks, Yahoo metadata (name, country, listing currency, exchange, GICS sector/industry, market cap, shares outstanding), full price history since 2004, FX rates; keeps only stocks with prices since 2005. |
| **DataDownloadService** | `quantcore.data` | Fetches the newest Yahoo prices and FX rates (incremental with dividend-adjustment splicing, or full), snapshots for reproducible demos. |
| **MarketDataService** | `quantcore.data.market_data` | Aligned adjusted closes and log-returns, daily/weekly/monthly, in CHF or local currency; market caps, cap-weighted broad market index. |
| **RiskService** | `quantcore.measurement.risk` | Volatility, correlation, covariance, VaR and CVaR (95 %, 99 %) per asset and per share-based portfolio — moving windows and exponentially weighted (EWM), Gaussian and historical, for a day / week / month, as absolute CHF amounts. |
| **PerformanceService** | `quantcore.measurement.performance` | Annualised return, volatility, Sharpe, Sortino, drawdown, Calmar, skew/kurtosis, hit ratio; rolling and EWM series; portfolios in shares. |
| **Stock-price models** | `quantcore.models.simulation` | Geometric Brownian motion (multivariate normal) and the empirical-innovation model `eps_t = Sigma_{t-1}^{-1/2} r_t`, `r(j|T) = Sigma_T^{1/2} eps_j` with the latest rolling or EWM covariance. |
| **Expected returns** | `quantcore.models.black_litterman` | Market-cap equilibrium returns (broad-market average, market weights), Black–Litterman on a 10-year weekly covariance with **Idzorek confidence** views (diagonal Ω), sentiment → view helper. |
| **OptimizationService** | `quantcore.optimization` | Mean–CVaR MILP in integer share numbers: **new portfolio** (from cash) or **rebalancing** (existing holdings, transaction costs); constraints: max 10 % per stock, max per sector, no shorts, cost/turnover limits, optional cardinality; scenario sets with variance reduction (Sobol', moment matching, market-direction importance sampling) that stay identical when risk limits, constraints, costs or views change, tail-row generation for large sets, and an exact Gaussian (mean–variance-equivalent) mode; solved on cuOpt (GPU, over HTTP) with a HiGHS fallback (CPU, compiled into SciPy — runs in-process, nothing to deploy). |
| **HTTP API** | `quantcore.api` | FastAPI app (`/docs`) exposing all of the above; setup/update as background jobs. Runs as a container on the DGX Spark next to cuOpt. |

Documentation: **[docs/USAGE.md](docs/USAGE.md) (start here if you are building on this package: the API, the
Python client, the `quantcore-api` command line, the conventions and the traps)**,
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (services, contracts, data flow, resilience),
[docs/METHODOLOGY.md](docs/METHODOLOGY.md) (every formula and convention),
[docs/optimization_model.pdf](docs/optimization_model.pdf) (derivation of the MILP from the Rockafellar–Uryasev
model in weights to whole shares with rebalancing and costs; sampling error of scenario sets, quasi-Monte Carlo,
moment matching, importance sampling, row generation, the exact Gaussian/mean–variance case and reproducibility;
source `docs/optimization_model.tex`),
[docs/measurement_and_models.pdf](docs/measurement_and_models.pdf) (risk and performance measures, rolling/EWM
estimators and shrinkage, GBM and empirical-innovation scenarios, Black–Litterman with Idzorek confidence;
source `docs/measurement_and_models.tex`),
[docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md) (**students: running it on your own DGX Spark** with
Apptainer, no root), [docs/SPARK_DEPLOYMENT.md](docs/SPARK_DEPLOYMENT.md) (the Docker setup on a Spark with root).

---

## Quick start on your DGX Spark (students)

The service and cuOpt run as two Apptainer containers on your Spark and talk over `127.0.0.1`.
No root rights, no Docker, no git, nothing exposed to the network.

**1. Install.** One line in a terminal on the Spark installs everything, builds the market data
from Yahoo Finance (~15 min) and starts the service:

```bash
wget -qO- https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash
```

Without `wget`: `curl -fsSL https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash`.
With a prepared data set: `wget -qO- https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash -s -- --data /path/to/quantcore_data.tar.gz`.

**2. Open a new terminal.** The installer adds the commands `qc`, `quantcore` and
`quantcore-api` to `~/.local/bin`; a terminal that was open during the installation does not know
them yet.

**3. Use it.** The test bench is at `http://127.0.0.1:5001/ui/` in the Spark's browser, the API
documentation at `http://127.0.0.1:5001/docs`. Your platform calls the API at
`http://127.0.0.1:5001`. On the command line:

```bash
qc status                        # what runs, cuOpt, how much data
quantcore risk NESN.SW AAPL ASML.AS --horizon week --window 200 --lam 0.94
quantcore performance NESN.SW AAPL --freq weekly
quantcore optimize --cash 1000000 --cvar-limit 40000 --confidence 0.95 --horizon week --max-assets 100
quantcore-api health             # the same service through its HTTP API
quantcore update                 # newest prices / FX
qc stop  /  qc start             # the service runs in the background; after a reboot: qc start
qc upgrade                       # newest version of this repository
```

On the Spark, `quantcore` and `quantcore-api` run inside the container. Do **not** run `pip install`
there: the software is already installed in the container, and Ubuntu refuses system-wide
`pip install` anyway ("externally-managed-environment"). The full guide, with settings, several
users on one Spark and troubleshooting, is [docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md).

## Quick start on your own computer (not the Spark)

With Python 3.11+ on a laptop or workstation, install the package into a virtual environment. Not
needed on a Spark that ran the installer above.

```bash
git clone https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management.git
cd Portfolio_Risk_Management
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env             # optional: data dir, cuOpt URL, ...

quantcore setup                  # once: universe + full history (~15 min, ~1100 Yahoo tickers)
quantcore status                 # what is stored, cuOpt health
quantcore update                 # later: newest prices / FX (incremental)
quantcore snapshot --name demo   # freeze the data for a reproducible demo

quantcore risk NESN.SW AAPL ASML.AS --horizon week --window 200 --lam 0.94
quantcore performance NESN.SW AAPL --freq weekly
quantcore optimize --cash 1000000 --cvar-limit 40000 --confidence 0.95 --horizon week --max-assets 100
quantcore serve                  # HTTP API on :5001  ->  http://localhost:5001/docs
                                 # and the test bench ->  http://localhost:5001/ui/
```

Against a *running* service -- from any machine, without the data directory -- use the API client
instead (`quantcore-api`, or `quantcore.client.QuantcoreClient` from Python):

```bash
export QUANTCORE_API_URL=http://127.0.0.1:5001   # or the address of the machine that runs it
quantcore-api health
quantcore-api selftest           # call every endpoint, pass/fail table, exit code
quantcore-api endpoints          # the API surface, from its own OpenAPI schema
quantcore-api risk NESN.SW AAPL --horizon week --drift zero --space value
quantcore-api optimize --cash 1000000 --cvar-limit 40000 --horizon week --risk-drift zero
```

`quantcore setup` fetches the candidate lists from the web (Wikipedia S&P 500, iShares STOXX 600
holdings); `--offline` uses the snapshots shipped in `quantcore/data/static/`. All data lands in
`QUANTCORE_DATA_DIR` (default `./data`): `universe/universe.csv`, `prices/adj_close.parquet`,
`fx/fx_chf.parquet`, `manifest.json`, `snapshots/`.

## Using the services from Python

```python
from quantcore import get_settings
from quantcore.data.market_data import MarketDataService
from quantcore.measurement.risk import RiskService
from quantcore.measurement.performance import PerformanceService
from quantcore.optimization.service import OptimizationService, OptimizationRequest

settings = get_settings()
md = MarketDataService(settings)                      # or MarketDataService(settings, snapshot="demo")

risk = RiskService(md)
risk.asset_risk(["NESN.SW", "AAPL"], horizon="week", window=200, lam=0.94)     # per-asset table
risk.correlation(["NESN.SW", "AAPL", "ASML.AS"], method="ewm", lam=0.99)
risk.portfolio_risk({"NESN.SW": 500, "AAPL": 100}, cash=20_000, horizon="month", method="historical")

PerformanceService(md).asset_performance(["NESN.SW", "AAPL"], freq="weekly")

opt = OptimizationService(settings, md)
res = opt.optimize(OptimizationRequest(
    mode="new", cash=1_000_000, max_assets=100, horizon="week", confidence=0.95, cvar_limit=40_000,
    scenario_model="empirical", covariance={"method": "ewm", "lam": 0.99},
    views=[{"symbol": "NESN.SW", "sentiment": 0.7, "confidence": 0.6, "source": "insight-1234"}],
))
res.positions_frame()          # shares, value, weight, trade per stock
res.cvar, res.expected_pnl     # CHF over the horizon
res.scenario_hash, res.reproducibility   # same inputs -> same scenarios; 'exact' (HiGHS) | 'within_gap' (cuOpt)
res.scenario_key["id"]         # varying the risk limit, constraints, costs or views keeps this id: one
                               # scenario set for a whole family of runs (and it is cached in the service)

# scenario options: 4096 moment-matched Sobol' scenarios by default; importance sampling for the tail
# switches on automatically at beta = 0.99; large sets are solved by tail-row generation
res99 = opt.optimize(OptimizationRequest(cash=1_000_000, confidence=0.99, cvar_limit=60_000, n_scenarios=16_384))
# the exact Gaussian (mean-variance-equivalent) solution: no scenarios, no sampling error
res_g = opt.optimize(OptimizationRequest(cash=1_000_000, cvar_limit=40_000, scenario_model="gaussian_exact"))
# n_scenarios defaults to 16 per candidate (4096-16384); the result warns if that is low for the
# candidate count or if the covariance estimate is ill-conditioned for it

# rebalancing: existing holdings, transaction costs, turnover limit
res2 = opt.optimize(OptimizationRequest(
    mode="rebalance", cash=res.cash_end, holdings={p["symbol"]: p["shares"] for p in res.positions},
    cvar_limit=30_000, costs={"proportional_buy": 0.001, "proportional_sell": 0.001, "fixed_per_trade": 10},
    constraints={"max_weight": 0.10, "max_sector_weight": 0.25, "max_turnover": 0.3},
))
```

## HTTP API (the platform's contract)

| Method | Path | Purpose |
|---|---|---|
| GET | `/health` | store status, cuOpt health |
| GET | `/universe?region=&sector=&country=` · `/universe/{symbol}` | universe with metadata |
| GET | `/prices?symbols=A,B&freq=weekly&currency=base` · `/returns?...` | prices / log-returns |
| GET | `/market/index?freq=weekly` · `/market/weights` | cap-weighted market index, weights |
| POST | `/risk/assets` (`space: log｜value`, `drift: estimated｜zero`) · `/risk/series` · `/risk/covariance` · `/risk/correlation` · `/risk/portfolio` | risk measurement |
| POST | `/performance/assets` · `/performance/rolling` · `/performance/portfolio` | performance measurement |
| GET | `/data/symbols?region=&sector=&country=&search=` | one table of the universe (metadata, market cap, price) plus the filter facets |
| POST | `/model/expected-returns` · `/model/covariance` · `/model/scenarios` | Black–Litterman posterior, the estimator the models use, scenario set (+ the P&L distribution of a portfolio) |
| POST | `/optimize` | mean–CVaR MILP (new / rebalance; a rebalancing also reports the starting portfolio under `before`) |
| POST | `/jobs/setup` · `/jobs/update?full=` · GET `/jobs/{id}` | background setup / data update |
| POST/GET | `/snapshots` | freeze / list data snapshots |

Example:

```bash
curl -s http://127.0.0.1:5001/optimize -H 'Content-Type: application/json' -d '{
  "mode": "new", "cash": 1000000, "max_assets": 100, "horizon": "week",
  "confidence": 0.95, "cvar_limit": 40000, "scenario_model": "empirical",
  "views": [{"symbol": "NESN.SW", "sentiment": 0.7, "confidence": 0.6}]
}' | python -m json.tool
```

## Test bench (browser UI)

`http://127.0.0.1:5001/ui/` serves a single-page test bench — the
one place to drive every service by hand and judge what it produces. It is a pure client of the HTTP
API above: no numbers are computed in the browser.

Because it is a client and not a part of the API, it can be left out of a deployment:
`QUANTCORE_ENABLE_UI=false` (read once at startup) serves the JSON API and `/docs` only, `/ui/`
answers 404 and `/` no longer redirects. The same image therefore also suits an installation whose
users build their own front end against these endpoints.

The left panel holds the state every tab shares — the candidate stocks, the portfolio (new or
rebalancing, cash, holdings) and the models (frequency, horizon, stock-price model, scenario count,
seed, sampling, tail tilt, covariance estimator, expected-return model). Five tabs use it:

| Tab | What it answers |
|---|---|
| **Stocks & data** | Which stocks exist (filter by region, sector, country, name) and what their history looks like: indexed prices, drawdowns, risk/return, market caps, and a per-stock table of volatility, VaR/CVaR (normal and historical), Sharpe, drawdown, skew and kurtosis. |
| **Risk & scenarios** | Per-stock scenario moments and quantiles in **price returns** (bounded below by −100 %) or, with one click, in the log-returns the model works in — a quarterly log-return of −2.0 is a price return of −86 %, and mixing the two is the easiest way to misread a simulation. The covariance the optimiser actually sees: correlation heat map, volatilities, eigenvalue spectrum, shrinkage and conditioning warnings. Then the scenario set: the horizon P&L distribution of a chosen portfolio (cap-weighted, equally weighted, your holdings or the last proposal) with VaR/CVaR markers, per-stock moments and quantiles, a single-stock distribution, the scenario hash, and a sampling-noise check across three seeds. |
| **Expected returns** | The market-implied (equilibrium) returns from reverse optimisation, the views you add (sentiment or explicit, with Idzorek confidence), and the posterior — side by side, with the change per stock and how each view was translated into `q` and `ω`. |
| **Constraints** | Weight, sector, cardinality, turnover, cost and investment-band limits, transaction costs, plus a feasibility check against the candidate set (are there enough names, do the sector caps leave room, is any stock unaffordable at one share?) and the sector composition against the caps. |
| **Optimisation** | Objective, confidence, CVaR limit, the risk convention (`risk_drift`), solver and time limit; the proposal in integer shares with trades and costs; for a rebalancing the **before and after** of the same portfolio on the same scenarios; the weights and sector exposure against the limits; the provenance (scenario key and hash, solver, MIP gap, row-generation passes, reproducibility); a run history and a risk-limit sweep that traces the risk/return trade-off. |

A third: **the CVaR limit can stop constraining anything over a long horizon**. CVaR₉₉ ≈ −μₕ + 2.67 σₕ,
so with a one-year horizon and an expected return of 40 % against a portfolio volatility of ~13 %, the whole
1 % tail is a gain, the limit is slack and mean-CVaR degenerates into "buy the highest forecast returns up to
the weight caps". The result then warns, and `risk_drift="zero"` measures the CVaR on the centred scenarios
(the risk figure is then independent of the return forecast) while the objective keeps it. The `historical`
expected-return model makes this extreme: on this data it gives up to 821 % p.a. to a single stock.

Two things to know while using it. The **time limit is the budget of the whole run**, not of each
solver call: row generation and the Gaussian outer approximation solve a sequence of MILPs, and each
one gets what is left of the budget (the default is 300 s; an empty field means no limit, and HiGHS
will then run until it can prove optimality -- hours on a large candidate set). And the service
answers **one request at a time** while a solve runs: the MILP holds the interpreter, so the other
tabs wait until it finishes. The progress panel says both, and a run can be cancelled.

Typical checks it is built for: put 1 m or 50 m CHF into a new portfolio and see how the composition
changes with wealth (integer shares start to matter); restrict the universe to European stocks and
compare; add a view on one stock and watch the posterior and the proposal move; switch the
stock-price model from GBM to empirical innovations (or to the exact Gaussian mode, which has no
sampling error at all) and see the risk figures change; run the same request twice and confirm that
the scenario hash and the shares are identical.

## On the DGX Spark

There are two setups. Both run the same code and give the same results.

* **Apptainer, no root** (the students' Sparks): `deploy/apptainer/qc.sh` runs cuOpt and the API as
  two Apptainer instances of the user, bound to `127.0.0.1`. See
  [docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md).
* **Docker Compose, with root** (the lecturer's Spark): `deploy/deploy_to_spark.sh --with-data`
  adds a `quantcore` service next to the `cuopt` service of an existing compose file and reaches
  cuOpt at `http://cuopt:5000`. See [docs/SPARK_DEPLOYMENT.md](docs/SPARK_DEPLOYMENT.md).

## Tests

```bash
pytest            # 127 tests: reference values for VaR/CVaR, Idzorek closed form vs. numeric, brute-force MILP optimality,
                  # exact Gaussian mode vs. brute force, row generation vs. full MILP, sampling/moment matching/tilt,
                  # weighted quantiles and scenario distributions, API (incl. the before/after of a rebalancing),
                  # the HTTP client and every `quantcore-api` command against the app in-process
```

The cuOpt round-trip test runs only when the server is reachable.

## Layout

```
quantcore/
  config.py            settings (env QUANTCORE_*, .env)
  universe/            candidate sources, Yahoo symbol mapping, SetupService
  data/                YahooClient (timeouts/retry/breaker), FX, DataStore, DataDownloadService, MarketDataService, static/
  measurement/         returns, rolling & EWM estimators, risk (VaR/CVaR), performance
  models/              CovarianceModel + matrix roots, GBM & empirical-innovation simulators, Black-Litterman/Idzorek
  optimization/        MILP builder (problem.py), cuOpt/HiGHS solvers, OptimizationService
  api/                 FastAPI app + pydantic schemas
  api/ui/              the browser test bench (static HTML/CSS/ES modules, no build step, no CDN);
                       QUANTCORE_ENABLE_UI=false serves the JSON API and /docs only
  client.py            QuantcoreClient -- the API as a Python object (needs only a URL)
  cli.py               `quantcore` command (in-process services, needs the data directory)
  apicli.py            `quantcore-api` command (HTTP client of a running service, incl. `selftest`)
install.sh             one-step installer/updater for a DGX Spark (wget ... | bash): code, images, data, start
deploy/apptainer/       qc.sh (install/start/stop/cli on a Spark without root), quantcore.def, pinned constraints.txt
deploy/                Dockerfile, compose service blocks (Spark + a bench-free one), deploy_to_spark.sh
docs/                  USAGE.md, ARCHITECTURE.md, METHODOLOGY.md, SPARK_APPTAINER.md, SPARK_DEPLOYMENT.md
tests/                 pytest suite (synthetic data store fixture)
```