fdbb69ca98
A student Spark can be barebone: no git, no sudo, Apptainer only, and an
account that exports APPTAINER_TMPDIR=/scratch/$USER/... for a directory
that does not exist, so every image pull failed ("failed to create build
parent dir"). Students now run one line:
wget -qO- .../raw/branch/main/install.sh | bash
install.sh (wrapped in main() so a piped run cannot lose its tail):
- repairs APPTAINER_TMPDIR / APPTAINER_CACHEDIR (and SINGULARITY_*) that
point nowhere and records the choice in ~/quantcore/quantcore.env
- gets the code with git, or with git from a container (alpine/git) when
the machine has none; Gitea's archive download answers 502
- runs qc.sh install, imports --data FILE|URL or builds the data from
Yahoo, starts the services, runs the self-test, links ~/.local/bin/qc
- running it again updates: rebuild and restart only when the code changed
qc.sh:
- the same directory repair before every pull or build, with one helper
for all pulls and a clear message when a download fails
- health checks over bash's /dev/tcp when neither curl nor python3 exists
- instance logs found for domain accounts and shared network homes
- works when called through the ~/.local/bin/qc link; new commands
`upgrade` (runs install.sh) and `has-data`
- the virtualenv build copies the sources under QC_HOME, not $TMPDIR
Tested on a stripped-down machine (no git/curl/wget/python3, home
/home/name@edu.local, APPTAINER_TMPDIR on a missing /scratch): install,
start and self-test pass; `qc upgrade` rebuilt and restarted after a new
commit; `qc setup` built a data set inside the container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
251 lines
19 KiB
Markdown
251 lines
19 KiB
Markdown
# quantcore — Portfolio Risk Measurement & Optimisation
|
||
|
||
The **deterministic quant core** of the AI-Augmented Portfolio Intelligence Platform (AISE502):
|
||
data, risk and performance measurement, stock-price models, Black–Litterman expected returns and
|
||
mean–CVaR portfolio optimisation in whole shares, solved as a MILP on **NVIDIA cuOpt** on the DGX Spark.
|
||
|
||
Everything here is *deterministic* — same input, same output — and contains **no LLM call**. The AI
|
||
layer of the platform (news → insights → views) consumes these services through their contracts and
|
||
never computes a number itself.
|
||
|
||
| Service | Module | What it does |
|
||
|---|---|---|
|
||
| **SetupService** | `quantcore.universe` | Builds the investment universe **once**: largest 500 US (S&P 500) and 500 European (STOXX 600; EU, CH, NO, UK) stocks, Yahoo metadata (name, country, listing currency, exchange, GICS sector/industry, market cap, shares outstanding), full price history since 2004, FX rates; keeps only stocks with prices since 2005. |
|
||
| **DataDownloadService** | `quantcore.data` | Fetches the newest Yahoo prices and FX rates (incremental with dividend-adjustment splicing, or full), snapshots for reproducible demos. |
|
||
| **MarketDataService** | `quantcore.data.market_data` | Aligned adjusted closes and log-returns, daily/weekly/monthly, in CHF or local currency; market caps, cap-weighted broad market index. |
|
||
| **RiskService** | `quantcore.measurement.risk` | Volatility, correlation, covariance, VaR and CVaR (95 %, 99 %) per asset and per share-based portfolio — moving windows and exponentially weighted (EWM), Gaussian and historical, for a day / week / month, as absolute CHF amounts. |
|
||
| **PerformanceService** | `quantcore.measurement.performance` | Annualised return, volatility, Sharpe, Sortino, drawdown, Calmar, skew/kurtosis, hit ratio; rolling and EWM series; portfolios in shares. |
|
||
| **Stock-price models** | `quantcore.models.simulation` | Geometric Brownian motion (multivariate normal) and the empirical-innovation model `eps_t = Sigma_{t-1}^{-1/2} r_t`, `r(j|T) = Sigma_T^{1/2} eps_j` with the latest rolling or EWM covariance. |
|
||
| **Expected returns** | `quantcore.models.black_litterman` | Market-cap equilibrium returns (broad-market average, market weights), Black–Litterman on a 10-year weekly covariance with **Idzorek confidence** views (diagonal Ω), sentiment → view helper. |
|
||
| **OptimizationService** | `quantcore.optimization` | Mean–CVaR MILP in integer share numbers: **new portfolio** (from cash) or **rebalancing** (existing holdings, transaction costs); constraints: max 10 % per stock, max per sector, no shorts, cost/turnover limits, optional cardinality; scenario sets with variance reduction (Sobol', moment matching, market-direction importance sampling) that stay identical when risk limits, constraints, costs or views change, tail-row generation for large sets, and an exact Gaussian (mean–variance-equivalent) mode; solved on cuOpt (GPU, over HTTP) with a HiGHS fallback (CPU, compiled into SciPy — runs in-process, nothing to deploy). |
|
||
| **HTTP API** | `quantcore.api` | FastAPI app (`/docs`) exposing all of the above; setup/update as background jobs. Runs as a container on the DGX Spark next to cuOpt. |
|
||
|
||
Documentation: **[docs/USAGE.md](docs/USAGE.md) (start here if you are building on this package: the API, the
|
||
Python client, the `quantcore-api` command line, the conventions and the traps)**,
|
||
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (services, contracts, data flow, resilience),
|
||
[docs/METHODOLOGY.md](docs/METHODOLOGY.md) (every formula and convention),
|
||
[docs/optimization_model.pdf](docs/optimization_model.pdf) (derivation of the MILP from the Rockafellar–Uryasev
|
||
model in weights to whole shares with rebalancing and costs; sampling error of scenario sets, quasi-Monte Carlo,
|
||
moment matching, importance sampling, row generation, the exact Gaussian/mean–variance case and reproducibility;
|
||
source `docs/optimization_model.tex`),
|
||
[docs/measurement_and_models.pdf](docs/measurement_and_models.pdf) (risk and performance measures, rolling/EWM
|
||
estimators and shrinkage, GBM and empirical-innovation scenarios, Black–Litterman with Idzorek confidence;
|
||
source `docs/measurement_and_models.tex`),
|
||
[docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md) (**students: running it on your own DGX Spark** with
|
||
Apptainer, no root), [docs/SPARK_DEPLOYMENT.md](docs/SPARK_DEPLOYMENT.md) (the Docker setup on a Spark with root).
|
||
|
||
---
|
||
|
||
## Quick start on your DGX Spark (students)
|
||
|
||
The service and cuOpt run as two Apptainer containers on your Spark and talk over `127.0.0.1`.
|
||
No root rights, no Docker, no git, nothing exposed to the network. One line in a terminal on the
|
||
Spark installs everything, builds the market data from Yahoo Finance and starts the service:
|
||
|
||
```bash
|
||
wget -qO- https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash
|
||
```
|
||
|
||
Without `wget`, use `curl -fsSL https://gitea.fhgr.ch/herzogfloria/Portfolio_Risk_Management/raw/branch/main/install.sh | bash`. With a prepared data set, add
|
||
`bash -s -- --data /path/to/quantcore_data.tar.gz` in place of the final `bash`. Afterwards the command `qc`
|
||
runs the service (`qc status`, `qc stop`, `qc start`, `qc upgrade`), the test bench is at
|
||
`http://127.0.0.1:5001/ui/`, and your platform calls the API at `http://127.0.0.1:5001`. The full guide, with settings,
|
||
several users on one Spark and troubleshooting, is [docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md).
|
||
|
||
## Quick start (local)
|
||
|
||
```bash
|
||
pip install -e ".[dev]" # Python 3.11+
|
||
cp .env.example .env # optional: data dir, cuOpt URL, ...
|
||
|
||
quantcore setup # once: universe + full history (~15 min, ~1100 Yahoo tickers)
|
||
quantcore status # what is stored, cuOpt health
|
||
quantcore update # later: newest prices / FX (incremental)
|
||
quantcore snapshot --name demo # freeze the data for a reproducible demo
|
||
|
||
quantcore risk NESN.SW AAPL ASML.AS --horizon week --window 200 --lam 0.94
|
||
quantcore performance NESN.SW AAPL --freq weekly
|
||
quantcore optimize --cash 1000000 --cvar-limit 40000 --confidence 0.95 --horizon week --max-assets 100
|
||
quantcore serve # HTTP API on :5001 -> http://localhost:5001/docs
|
||
# and the test bench -> http://localhost:5001/ui/
|
||
```
|
||
|
||
Against a *running* service -- from any machine, without the data directory -- use the API client
|
||
instead (`quantcore-api`, or `quantcore.client.QuantcoreClient` from Python):
|
||
|
||
```bash
|
||
export QUANTCORE_API_URL=http://127.0.0.1:5001 # or the address of the machine that runs it
|
||
quantcore-api health
|
||
quantcore-api selftest # call every endpoint, pass/fail table, exit code
|
||
quantcore-api endpoints # the API surface, from its own OpenAPI schema
|
||
quantcore-api risk NESN.SW AAPL --horizon week --drift zero --space value
|
||
quantcore-api optimize --cash 1000000 --cvar-limit 40000 --horizon week --risk-drift zero
|
||
```
|
||
|
||
`quantcore setup` fetches the candidate lists from the web (Wikipedia S&P 500, iShares STOXX 600
|
||
holdings); `--offline` uses the snapshots shipped in `quantcore/data/static/`. All data lands in
|
||
`QUANTCORE_DATA_DIR` (default `./data`): `universe/universe.csv`, `prices/adj_close.parquet`,
|
||
`fx/fx_chf.parquet`, `manifest.json`, `snapshots/`.
|
||
|
||
## Using the services from Python
|
||
|
||
```python
|
||
from quantcore import get_settings
|
||
from quantcore.data.market_data import MarketDataService
|
||
from quantcore.measurement.risk import RiskService
|
||
from quantcore.measurement.performance import PerformanceService
|
||
from quantcore.optimization.service import OptimizationService, OptimizationRequest
|
||
|
||
settings = get_settings()
|
||
md = MarketDataService(settings) # or MarketDataService(settings, snapshot="demo")
|
||
|
||
risk = RiskService(md)
|
||
risk.asset_risk(["NESN.SW", "AAPL"], horizon="week", window=200, lam=0.94) # per-asset table
|
||
risk.correlation(["NESN.SW", "AAPL", "ASML.AS"], method="ewm", lam=0.99)
|
||
risk.portfolio_risk({"NESN.SW": 500, "AAPL": 100}, cash=20_000, horizon="month", method="historical")
|
||
|
||
PerformanceService(md).asset_performance(["NESN.SW", "AAPL"], freq="weekly")
|
||
|
||
opt = OptimizationService(settings, md)
|
||
res = opt.optimize(OptimizationRequest(
|
||
mode="new", cash=1_000_000, max_assets=100, horizon="week", confidence=0.95, cvar_limit=40_000,
|
||
scenario_model="empirical", covariance={"method": "ewm", "lam": 0.99},
|
||
views=[{"symbol": "NESN.SW", "sentiment": 0.7, "confidence": 0.6, "source": "insight-1234"}],
|
||
))
|
||
res.positions_frame() # shares, value, weight, trade per stock
|
||
res.cvar, res.expected_pnl # CHF over the horizon
|
||
res.scenario_hash, res.reproducibility # same inputs -> same scenarios; 'exact' (HiGHS) | 'within_gap' (cuOpt)
|
||
res.scenario_key["id"] # varying the risk limit, constraints, costs or views keeps this id: one
|
||
# scenario set for a whole family of runs (and it is cached in the service)
|
||
|
||
# scenario options: 4096 moment-matched Sobol' scenarios by default; importance sampling for the tail
|
||
# switches on automatically at beta = 0.99; large sets are solved by tail-row generation
|
||
res99 = opt.optimize(OptimizationRequest(cash=1_000_000, confidence=0.99, cvar_limit=60_000, n_scenarios=16_384))
|
||
# the exact Gaussian (mean-variance-equivalent) solution: no scenarios, no sampling error
|
||
res_g = opt.optimize(OptimizationRequest(cash=1_000_000, cvar_limit=40_000, scenario_model="gaussian_exact"))
|
||
# n_scenarios defaults to 16 per candidate (4096-16384); the result warns if that is low for the
|
||
# candidate count or if the covariance estimate is ill-conditioned for it
|
||
|
||
# rebalancing: existing holdings, transaction costs, turnover limit
|
||
res2 = opt.optimize(OptimizationRequest(
|
||
mode="rebalance", cash=res.cash_end, holdings={p["symbol"]: p["shares"] for p in res.positions},
|
||
cvar_limit=30_000, costs={"proportional_buy": 0.001, "proportional_sell": 0.001, "fixed_per_trade": 10},
|
||
constraints={"max_weight": 0.10, "max_sector_weight": 0.25, "max_turnover": 0.3},
|
||
))
|
||
```
|
||
|
||
## HTTP API (the platform's contract)
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| GET | `/health` | store status, cuOpt health |
|
||
| GET | `/universe?region=§or=&country=` · `/universe/{symbol}` | universe with metadata |
|
||
| GET | `/prices?symbols=A,B&freq=weekly¤cy=base` · `/returns?...` | prices / log-returns |
|
||
| GET | `/market/index?freq=weekly` · `/market/weights` | cap-weighted market index, weights |
|
||
| POST | `/risk/assets` (`space: log｜value`, `drift: estimated｜zero`) · `/risk/series` · `/risk/covariance` · `/risk/correlation` · `/risk/portfolio` | risk measurement |
|
||
| POST | `/performance/assets` · `/performance/rolling` · `/performance/portfolio` | performance measurement |
|
||
| GET | `/data/symbols?region=§or=&country=&search=` | one table of the universe (metadata, market cap, price) plus the filter facets |
|
||
| POST | `/model/expected-returns` · `/model/covariance` · `/model/scenarios` | Black–Litterman posterior, the estimator the models use, scenario set (+ the P&L distribution of a portfolio) |
|
||
| POST | `/optimize` | mean–CVaR MILP (new / rebalance; a rebalancing also reports the starting portfolio under `before`) |
|
||
| POST | `/jobs/setup` · `/jobs/update?full=` · GET `/jobs/{id}` | background setup / data update |
|
||
| POST/GET | `/snapshots` | freeze / list data snapshots |
|
||
|
||
Example:
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:5001/optimize -H 'Content-Type: application/json' -d '{
|
||
"mode": "new", "cash": 1000000, "max_assets": 100, "horizon": "week",
|
||
"confidence": 0.95, "cvar_limit": 40000, "scenario_model": "empirical",
|
||
"views": [{"symbol": "NESN.SW", "sentiment": 0.7, "confidence": 0.6}]
|
||
}' | python -m json.tool
|
||
```
|
||
|
||
## Test bench (browser UI)
|
||
|
||
`http://127.0.0.1:5001/ui/` serves a single-page test bench — the
|
||
one place to drive every service by hand and judge what it produces. It is a pure client of the HTTP
|
||
API above: no numbers are computed in the browser.
|
||
|
||
Because it is a client and not a part of the API, it can be left out of a deployment:
|
||
`QUANTCORE_ENABLE_UI=false` (read once at startup) serves the JSON API and `/docs` only, `/ui/`
|
||
answers 404 and `/` no longer redirects. The same image therefore also suits an installation whose
|
||
users build their own front end against these endpoints.
|
||
|
||
The left panel holds the state every tab shares — the candidate stocks, the portfolio (new or
|
||
rebalancing, cash, holdings) and the models (frequency, horizon, stock-price model, scenario count,
|
||
seed, sampling, tail tilt, covariance estimator, expected-return model). Five tabs use it:
|
||
|
||
| Tab | What it answers |
|
||
|---|---|
|
||
| **Stocks & data** | Which stocks exist (filter by region, sector, country, name) and what their history looks like: indexed prices, drawdowns, risk/return, market caps, and a per-stock table of volatility, VaR/CVaR (normal and historical), Sharpe, drawdown, skew and kurtosis. |
|
||
| **Risk & scenarios** | Per-stock scenario moments and quantiles in **price returns** (bounded below by −100 %) or, with one click, in the log-returns the model works in — a quarterly log-return of −2.0 is a price return of −86 %, and mixing the two is the easiest way to misread a simulation. The covariance the optimiser actually sees: correlation heat map, volatilities, eigenvalue spectrum, shrinkage and conditioning warnings. Then the scenario set: the horizon P&L distribution of a chosen portfolio (cap-weighted, equally weighted, your holdings or the last proposal) with VaR/CVaR markers, per-stock moments and quantiles, a single-stock distribution, the scenario hash, and a sampling-noise check across three seeds. |
|
||
| **Expected returns** | The market-implied (equilibrium) returns from reverse optimisation, the views you add (sentiment or explicit, with Idzorek confidence), and the posterior — side by side, with the change per stock and how each view was translated into `q` and `ω`. |
|
||
| **Constraints** | Weight, sector, cardinality, turnover, cost and investment-band limits, transaction costs, plus a feasibility check against the candidate set (are there enough names, do the sector caps leave room, is any stock unaffordable at one share?) and the sector composition against the caps. |
|
||
| **Optimisation** | Objective, confidence, CVaR limit, the risk convention (`risk_drift`), solver and time limit; the proposal in integer shares with trades and costs; for a rebalancing the **before and after** of the same portfolio on the same scenarios; the weights and sector exposure against the limits; the provenance (scenario key and hash, solver, MIP gap, row-generation passes, reproducibility); a run history and a risk-limit sweep that traces the risk/return trade-off. |
|
||
|
||
A third: **the CVaR limit can stop constraining anything over a long horizon**. CVaR₉₉ ≈ −μₕ + 2.67 σₕ,
|
||
so with a one-year horizon and an expected return of 40 % against a portfolio volatility of ~13 %, the whole
|
||
1 % tail is a gain, the limit is slack and mean-CVaR degenerates into "buy the highest forecast returns up to
|
||
the weight caps". The result then warns, and `risk_drift="zero"` measures the CVaR on the centred scenarios
|
||
(the risk figure is then independent of the return forecast) while the objective keeps it. The `historical`
|
||
expected-return model makes this extreme: on this data it gives up to 821 % p.a. to a single stock.
|
||
|
||
Two things to know while using it. The **time limit is the budget of the whole run**, not of each
|
||
solver call: row generation and the Gaussian outer approximation solve a sequence of MILPs, and each
|
||
one gets what is left of the budget (the default is 300 s; an empty field means no limit, and HiGHS
|
||
will then run until it can prove optimality -- hours on a large candidate set). And the service
|
||
answers **one request at a time** while a solve runs: the MILP holds the interpreter, so the other
|
||
tabs wait until it finishes. The progress panel says both, and a run can be cancelled.
|
||
|
||
Typical checks it is built for: put 1 m or 50 m CHF into a new portfolio and see how the composition
|
||
changes with wealth (integer shares start to matter); restrict the universe to European stocks and
|
||
compare; add a view on one stock and watch the posterior and the proposal move; switch the
|
||
stock-price model from GBM to empirical innovations (or to the exact Gaussian mode, which has no
|
||
sampling error at all) and see the risk figures change; run the same request twice and confirm that
|
||
the scenario hash and the shares are identical.
|
||
|
||
## On the DGX Spark
|
||
|
||
There are two setups. Both run the same code and give the same results.
|
||
|
||
* **Apptainer, no root** (the students' Sparks): `deploy/apptainer/qc.sh` runs cuOpt and the API as
|
||
two Apptainer instances of the user, bound to `127.0.0.1`. See
|
||
[docs/SPARK_APPTAINER.md](docs/SPARK_APPTAINER.md).
|
||
* **Docker Compose, with root** (the lecturer's Spark): `deploy/deploy_to_spark.sh --with-data`
|
||
adds a `quantcore` service next to the `cuopt` service of an existing compose file and reaches
|
||
cuOpt at `http://cuopt:5000`. See [docs/SPARK_DEPLOYMENT.md](docs/SPARK_DEPLOYMENT.md).
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
pytest # 127 tests: reference values for VaR/CVaR, Idzorek closed form vs. numeric, brute-force MILP optimality,
|
||
# exact Gaussian mode vs. brute force, row generation vs. full MILP, sampling/moment matching/tilt,
|
||
# weighted quantiles and scenario distributions, API (incl. the before/after of a rebalancing),
|
||
# the HTTP client and every `quantcore-api` command against the app in-process
|
||
```
|
||
|
||
The cuOpt round-trip test runs only when the server is reachable.
|
||
|
||
## Layout
|
||
|
||
```
|
||
quantcore/
|
||
config.py settings (env QUANTCORE_*, .env)
|
||
universe/ candidate sources, Yahoo symbol mapping, SetupService
|
||
data/ YahooClient (timeouts/retry/breaker), FX, DataStore, DataDownloadService, MarketDataService, static/
|
||
measurement/ returns, rolling & EWM estimators, risk (VaR/CVaR), performance
|
||
models/ CovarianceModel + matrix roots, GBM & empirical-innovation simulators, Black-Litterman/Idzorek
|
||
optimization/ MILP builder (problem.py), cuOpt/HiGHS solvers, OptimizationService
|
||
api/ FastAPI app + pydantic schemas
|
||
api/ui/ the browser test bench (static HTML/CSS/ES modules, no build step, no CDN);
|
||
QUANTCORE_ENABLE_UI=false serves the JSON API and /docs only
|
||
client.py QuantcoreClient -- the API as a Python object (needs only a URL)
|
||
cli.py `quantcore` command (in-process services, needs the data directory)
|
||
apicli.py `quantcore-api` command (HTTP client of a running service, incl. `selftest`)
|
||
install.sh one-step installer/updater for a DGX Spark (wget ... | bash): code, images, data, start
|
||
deploy/apptainer/ qc.sh (install/start/stop/cli on a Spark without root), quantcore.def, pinned constraints.txt
|
||
deploy/ Dockerfile, compose service blocks (Spark + a bench-free one), deploy_to_spark.sh
|
||
docs/ USAGE.md, ARCHITECTURE.md, METHODOLOGY.md, SPARK_APPTAINER.md, SPARK_DEPLOYMENT.md
|
||
tests/ pytest suite (synthetic data store fixture)
|
||
```
|